Space-time adaptive image representations : Data Structures , Hardware and Algorithms

Simultuaneous spatial identification and rapid tracking of moving objects is difficult with conventional video formats, which do not have enough space-time bandwidth. Increasing the spatial or temporal sampling rates tends to run into problems with available light and/or with inundating computational and communication channels with to much data. In this paper, we explore alternate sensing strategies, based on abandoning the usual space-invariant, temporally synchronous video stream. Part of the motivation for this idea comes from biological vision systems, which are both space-variant and asynchronous, and whose architectures can provide very large performance enhancements. We summarize past experience with space-variant (i.e. foveal) architectures, and provide speculations and simulations based on some very recent ideas concerning temporally asynchronous vision sensors. Two critical resources limit the performance of computational vision systems. They result from the basic physics of vision. They will still be present in the future, even with unlimited improvements in computational power: 1. ) Pixel sensitivity: physically small pixels are required to obtain high density sensors, but pixels can be no smaller than a wavelength of visible light. Biological pixels already have reached this constraint(foveal cones-0:5 ), while VLSI sensors are within jonnyreb@bu.edu eric@bu.edu 1 an order of magnitude of it (typical commercial CCD sensors-5 ). But, small pixels have small photon ”catch”. It is not possible to run high density sensors at high temporal frequencies. The limited availability of visible photons clamps the space-time bandwidth of sensors. 2. ) Limited bandwidth: Bandwidth for communication is, and always will be, a scarce, limiting resource. As local communications achieve higher computational power, their ability to communicate globally becomes the limiting factor. This is seen in the brain (white matter fibers dominate the volume of the skull) and computation (wiring area dominates transistor area in VLSI and parallel computers). It follows that precise motion tracking suffers the same physical limits. High precision tracking in time, for targets which must be distinguished by high precision in space, has limited performance given current pixel-frame based video standards. In the context of image based motion tracking, e.g. for purposes of target acquisition or robotic control, one cannot separate the question of temporal sampling and compression from spatial sampling and compression. In the light of this discussion, this papers considers the following question: What sensor architectures, given fixed illumination and communication bandwidth, optimise space-time performance? In the following discussion, this question will be addressed by a a critical comparison of current machine vision standards, based on synchronous space-invariant sampling, and biological visual representations, which are space-variant and asynchronous. Computer vision is currently based on video standards that were developed in the early days of broadcast television, roughly fifty years ago. “Pixels and frames” represent a constant sampling strategy in space and time. It has been simple to build sensor readout based on this strategy, and sampling theory, as developed fifty years ago by Shannon, provides a framework for signal analysis. For these reasons, the dominant current architecture is space-invariant (constant sized pixels) and synchronous (constant “clock” for pixels and frames). Biological vision is based on standards that were developed roughly fifty million years ago (origin of primates). There are no pixels and frames. There is no clock. Resolution is space-variant and adaptive. It will be argued below that there is much to learn from the biological strategy, and, more to the point, a significant part of it can be “harvested” by technology. In order to do so, a series of hardware and computational problems need to be stated, and then solved. The key idea is that adaptive sampling, in space and time, can provide one form of optimization of vision applications, particularly where high spatial accuracy (e.g. mega-pixel sensors) and/or high temporal accuracy (motion tracking, target acquisition, robotic control) is required in a situation of limited lighting and limited communications bandwidth. The conclusion of this analysis will be that a series of research problems must be solved to realize this strategy. These are: VLSI Adaptive sensors that provide high density, space-variant and temporally asynchronous output, must be developed. It is likely that AER (Address Event Representation) techniques (Mahowald, 1994) will provide the basis for this technology, 2 although an alternate (short term) approach could be based on high rate conventional sensors (e.g.240 Frame/sec), with associated DSP or gate-array processing at the sensor. Sampling Non-uniform sampling strategies (post-Shannon) must be developed for sensor parallel video signal processing. Algorithms Current computer vision algorithms are largely based on a finite-difference approach. Non-uniform sensors require mesh-based(triangulation) representations, and therefore point to finite element methods (FEM) and graph-theoretic approaches as a replacement for finite difference methods. CPU High performance mesh based processors will be needed. Current graphics chips (e.g. NVIDIA) already have transistor counts rivaling the Pentium, but have largely been developed for gaming applications. Re-directing this existing technology towards computer vision will be associated with the sensor-parallel paradigm. Applications/Simulations Applications demonstrations and proof-of-concept (developed perhaps via sensor simulation) must be studied to understand the advantages (and/or disadvantages) of space-variant and asynchronous sensing. Some examples of this will be shown in this paper. The space-variant aspect of this argument is very well understood, has been studied for more than twenty years, and has resulted in a small body of VLSI sensor research and computer vision applications in the area of ”space-variant active vision” (see (Schwartz, 1994a) for review). The temporally asynchronous part of this argument is much less explored. In the next section of this paper, space-variant sampling will be outlined, reviewed, and demonstrated. The final section of this paper will provide a speculative discussion of temporally asynchronous sampling. 1 Examples of space-domain sensor parallel architecture: log polar mapping and primate visual cortex Human vision is space-variant. This is a fact known to the (relatively) small community of vision scientists, but is surprisingly unknown to the vast majority of humans who live in a strongly space-variant visual world that they perceive as Euclidean in geometry and constant in resolution. You, the reader can (and should, at this point) perform a simple experiment. Hold your two thumbs at arms length. Fix your eyes on one thumb, and move the other thumb further and further away from the first thumb. Resist the tendency to move your eyes. Determine that the second moving thumb begins to disappear. First, the thumb-nail is gone; then the whole thumb, then the entire hand. This illustrate the loss of resolution associated with increasing distance from the foveal (fixated) point. 3 Many scientists (but few layman) have an awareness of this phenomenon, but even in this select group, there is little understanding of the huge amount of lossy compression that occurs due to space-variant representation. In order to demonstrate this phenomenon, a few ideas need to be presented concerning visual anatomy. The retinal receptor layers is a thin photo-sensitive layer of cells arranged on a spherical surface (the eyeball). A wide-angle optical system provided by the lens of the eye, as well as refraction in the vitreous layers of the eyes, provides a very wide field of view (roughly 200 degrees horizontally and 140 degrees vertically). At the center of the retina (the fovea) there is extremely high resolution (about 1 arc minute). This is performed by sensors (retinal cones) whose size is about one wavelength of visible light(in the fovea), and therefore are at the physical limit (diffraction limit) for optical sensing. The retina is ”mapped” on to the surface of primary visual cortex. The term mapped refers to the fact that nearby points of stimulation on the retina lead to nearby stimulation of cortical points. The cortex is a thin, (2-3 mm) layer of cells whose surface area is about 2000 mm) in humans. It is about the size of a business card. Within this surface is an image of the retina, but one which is highly distorted, because it is sampled in a highly non-uniform manner. (There are additional details of spatial representation that are not relevant to the present discussion, (see (Schwartz, 1994a) for review). V1 is then mapped in a slightly different way onto a second visual area (V2), a third (V3). More than twenty such areas have been described in the monkey, so that visual cortex occupies more than half of the entire cortical area! Current understanding of the internal representation of the retina at the level of primary visual cortex is based on fifty years of anatomical and physiological experiment at measuring the location of point sources in both visual field, and cortical representation. A mathematical approximation to this representation has been developed, which is in good agreement with the collected data, using a complex analytic (conformal) mapping to approximate the visual projection to primary visual cortex. The function w = log(z+a) was originally introduced by (Schwartz, 1977; Schwartz, 1994a), and later extended to a function of the form w = log(z+a)=(z+ b)(Schwartz, 1983), where w (complex variable) represents a point location in the (flattened) cortex, and complex z represents a stereographic projection of a point of the (spherical)