Query: The silicon retina.

Candidate papers:

abstract: The USB Revolution Robotic vision is the most fascinating and feasible application for neuromorphic engineering, since processing images in real time with low power consumption is the field's most critical requirement. Conventional machine vision systems have been implemented using CMOS (complimentary metal oxide semiconductor) imagers or CCD (charge coupled device) cameras that are interfaced to digital processing systems operating with serial algorithms. These systems often consume too much power, are too large, and compute with too high a cost.1 Though neuromorphic technology has advantages in these areas, there are some disadvantages too: current implementations have less programmable architectures, for example, than digital processing technologies. In addition, digital image processing has a long history and highly developed hardware and software for pattern recognition are readily available. We therefore think it is practical at least at the current stage of progress in neuromorphic engineering to combine neuromorphic sensors with conventional digital technology to implement, for robot vision, the computational essence of what the brain does. On this basis, we designed a neuromorphic vision system consisting of analog VLSI (very large silicon integration) neuromorphic chips and field programmable gate array (FPGA) circuits. Figure 1 shows a block diagram of the system, which consists of silicon retinas, 'simple cell' chips (named after the simple cell in the V1 area of the brain) and FPGA circuits. The silicon retina is implemented with active pixel sensors (conventionallysampled photo sensors)2 and has a concentric center surround Laplacian Gaussian like receptive field.2 Its output image is transferred to the simple cell chips serially. These chips then aggregate analog pixel outputs from the silicon retina to generate an orientation selective response similar to the simple cell response in the primary visual cortex.3 The architecture mimics the feed forward model proposed by Hubel and Wiesel,4 and efficiently computes the two dimensional Gabor like receptive field using the concentric center surround receptive field. The signal transfer from the silicon retina to the simple cell chip is performed using analog voltage, aided by analog memories embedded in each pixel of the simple cell chip. The output image of the simple cell chip is then converted into a digital signal and fed into the FPGA circuits, where the image is further processed with programReceptive field Silicon retina
author_names: Misha A Mahowald

Carver Mead
title: The silicon retina.
venue: Scientific American
year: 1991

abstract: A new rotation invariant pattern recognition system is proposed and analyzed. In this system, silicon retina cells capable of image sensing and edge extraction are used so that the system can directly process images from the real world without an extra edge detector. The rotation invariant discrete correlation function is modified and implemented in the silicon retina structure by using the current summation. Simulation results have verified the correct function of the proposed system. Moreover, an experimental chip to implement the proposed system with a 32/spl times/32 cell array has been designed and fabricated in 0.8 /spl mu/m n well CMOS process. Experimental results have successfully shown that the system works well for the arbitrary orientation pattern recognition.
author_names: Chin-Fong Chiu

Chung-Yu Wu
title: The design of rotation invariant pattern recognition using the silicon retina
year: 1997

abstract: In this paper a new stereo matching concept for event driven silicon retinae is presented. The main contribution of the proposed approach is the correlation of incoming events. As a novelty, not only the spatial information is used, but also the time of occurrence of the events as a part of the similarity measure. Stereo matching is used in depth generating camera systems for solving the correspondence problem and for 3D reconstruction of the sensed environment. In fact, using conventionally frame based cameras, this is a time consuming and computationally expensive task, especially for high frame rates and spatial resolutions. An event based silicon retina delivers events only on illumination changes and completely asynchronous in time. The sensor provides no frames, but a time continuous data stream of intensity differences and thus inherently reduces the visual information to a minimum. This paper focuses on an event based stereo matching algorithm implemented in hardware on a field programmable gate array (FPGA) that allows a reliable matching of the sparse input event data. Furthermore, the approach is compared to other standard frame based and event driven stereo methods. The results show that the achieved depth map outperforms other algorithms in terms of accuracy and the calculation performance of the hardware architecture is in the range or still higher than state of the art computing platforms.
author_names: Florian Eibensteiner

Hans Georg Brachtendorf

Josef Scharinger
title: Event driven stereo vision algorithm based on silicon retina sensors
venue: 2017 27th International Conference Radioelektronika (RADIOELEKTRONIKA)
year: 2017

abstract: In this paper, we present: (i) a novel analog silicon retina featuring auto adaptive pixels that obey the Michaelis Menten law, i.e. V=V(m) I(n)/I(n)+s(n) (ii) a method of characterizing silicon retinas, which makes it possible to accurately assess the pixels' response to transient luminous changes in a 3 decade range, as well as changes in the initial steady state intensity in a 7 decade range. The novel pixel, called M(2)APix, which stands for Michaelis Menten Auto Adaptive Pixel, can auto adapt in a 7 decade range and responds appropriately to step changes up to 3 decades in size without causing any saturation of the Very Large Scale Integration (VLSI) transistors. Thanks to the intrinsic properties of the Michaelis Menten equation, the pixel output always remains within a constant limited voltage range. The range of the Analog to Digital Converter (ADC) was therefore adjusted so as to obtain a Least Significant Bit (LSB) voltage of 2.35mV and an effective resolution of about 9 bits. The results presented here show that the M(2)APix produced a quasi linear contrast response once it had adapted to the average luminosity. Differently to what occurs in its biological counterparts, neither the sensitivity to changes in light nor the contrast response of the M(2)APix depend on the mean luminosity (i.e. the ambient lighting conditions) Lastly, a full comparison between the M(2)APix and the Delbruck auto adaptive pixel is provided.
author_names: Stefano Mafrica

Stephanie Godiot

Mohsine Menouni

Marc Boyron

Fabien Expert

Raphael Juston

Nicolas Marchand

Franck Ruffier

Stephane Viollet
title: A bio inspired analog silicon retina with Michaelis Menten auto adaptive pixels sensitive to small and large changes in light.
venue: Optics express
year: 2015

abstract: A new silicon retina structure in CMOS is proposed and analyzed, which incorporates a parasitic phototransistor as an isolated photoreceptor with many phototransistors in a common well region. The smooth function is achieved by the diffusion of photogenerated carriers in the common well. Experimental results have verified the correct functions of the new structure as a silicon retina. Small chip area and compact wiring make the new structure quite realizable with the associated neural networks in VLSI.<ETX>
author_names: Chung-Yu Wu

Chin-Fong Chiu
title: A new structure for the silicon retina
venue: 1992 International Technical Digest on Electron Devices Meeting
year: 1992

abstract: Optical flow sensors have been a long running theme in neuromorphic vision sensors which include circuits that implement the local background intensity adaptation mechanism seen in biological retinas. This paper reports a bio inspired optical motion sensor aimed towards miniature robotic and aerial platforms. It combines a 20 x 20 continuous time CMOS silicon retina vision sensor with a DSP microcontroller. The retina sensor has pixels that have local gain control and adapt to background lighting. The system allows the user to validate various motion algorithms without building dedicated custom solutions. Measurements are presented to show that the system can compute global 2D translational motion from complex natural scenes using one particular algorithm: the image interpolation algorithm (I2A) With this algorithm, the system can compute global translational motion vectors at a sample rate of 1 kHz, for speeds up to 1000 pixels/s, using less than 5 k instruction cycles (12 instructions per pixel) per frame. At 1 kHz sample rate the DSP is 12% occupied with motion computation. The sensor is implemented as a 6 g PCB consuming 170 mW of power.
author_names: Shih-Chii Liu

Minhao Yang

Andreas Steiner

Rico Moeckel

Tobi Delbruck
title: 1 kHz 2D Visual Motion Sensor Using 20$\times\$20 Silicon Retina Optical Sensor and DSP Microcontroller
venue: IEEE Transactions on Biomedical Circuits and Systems
year: 2015

abstract: Abstract In the Background section of the proposal I described the basic conceptual elements which underlie the Silicon Retina (SR) and those which underlie passive mammalian hearing. My original intention was to utilize the frequency discrimination capability manifested in mammalian hearing as the mechanism on which to base an acoustic analogue to the SR concept. Specifically, mammals have sophisticated frequency dependent sensors which transmit responses to discrete and well resolved frequencies upward to higher processing levels. Each intermediate processor transmits the precise frequency information upstream. Thus one may think of acoustic frequencies as being analogous to visual positional information, since in the ray limit of optics there is a one to one correspondence between the relative locations of external objects and the retinal sensors which are stimulated at any given time. The SR mechanism was to be implemented by comparing individual frequencies with running averages over groups of nearby frequencies, eliciting a response when the intensities of these differed by a prescribed amount. The next step in the program was to remain open minded regarding the implementation of directional acoustic arrays, with the idea pf steering an array making use of the 'alert' response created by the introduction of a non background signal. In this way it was envisioned that the 'cocktail party effect' would be achieved.
author_names: Jorge F Willemsen
title: Feasibility Assessment of a Transient Sound Sensor Based on the Silicon Retina Architecture
year: 1992

abstract: In this paper we present a new approach for the evaluation of event based Silicon Retina stereo matching results. The evaluation of stereo matching algorithm results is a necessary task for the development, comparison, and improvement of depth generating camera systems. In contrast to conventional frame based cameras, the silicon retina sensors delivers asynchronous events instead of synchronous intensity or color images. The polarity of the events represents either an increase (on event) or a decrease (off event) of the brightness of the projected scene point. This is the reason why existing ground truth data and evaluation platforms are not suitable for testing silicon retina stereo camera systems. For the analysis of the introduced novel evaluation method, we use an area based (sum of absolute difference) algorithm for the event driven sensor system. A conventional video camera stereo vision system is used to produce reference data. The results show that the presented method offers new opportunities for the evaluation of stereo matching results computed from silicon retina stereo data.
author_names: Jurgen Kogler

Florian Eibensteiner

M Humenberger

Margrit Gelautz

Josef Scharinger
title: Ground Truth Evaluation for Event Based Silicon Retina Stereo Data
venue: 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops
year: 2013

abstract: Silicon retina, modeling a biological architecture, can implement parallel operations with low power. To implement a compact, fast image processing system, this paper describes a circuit with FPGA (field programmable gate array) and silicon retina combined together. Circuit optimization by DCT (discrete cosine transform) is proposed.
author_names: Motoki Amagasaki

Fuminori Kobayashi

Minoru Watanabe

Tetsuya Yagi
title: Motion image compression circuit using the silicon retina as active sensor
venue: Proceedings of the 41st SICE Annual Conference. SICE 2002.
year: 2002

abstract: Abstract. We present two improvement techniques for stereo matching algorithms using silicon retina sensors. We verify the results with ground truth data. In contrast to conventional monochrome/color cameras, silicon retina sensors deliver an asynchronous flow of events instead of common framed and discrete intensity or color images. While using this kind of sensor in a stereo setup to enable new fields of applications, it also introduces new challenges in terms of stereo image analysis. Using this type of sensor, stereo matching algorithms have to deal with sparse event data, thus, less information. This affects the quality of the achievable disparity results and renders improving the stereo matching algorithms a necessary task. For this reason, we introduce two techniques for increasing the accuracy of silicon retina stereo results, in the sense that the average distance error is reduced. The first method is an adapted belief propagation approach optimizing the initial matching cost volume, and the second is an innovative two stage postfilter for smoothing and outlier rejection. The evaluation shows that the proposed techniques increase the accuracy of the stereo matching and constitute a useful extension for using silicon retina sensors for depth estimation.
author_names: Jurgen Kogler

Florian Eibensteiner

M Humenberger

Christoph Sulzbachner

Margrit Gelautz

Josef Scharinger
title: Enhancement of sparse silicon retina based stereo matching using belief propagation and two stage postfiltering
venue: J. Electronic Imaging
year: 2014