Vision circuits architecture with on-chip spatial window operations.
Authors:   , LÓPEZ-PORTILLA VIGIL BARBARO MAYKEL 
Published: November 2024

CMOS Image Sensors (CIS) are crucial components in multimedia applications for the Internet of the Things (IoT). They allow IoT devices to perceive, interpret, and respond to the physical world, improving efficiency, safety, and convenience across various industries. The images captured by these sensors are sent to an external digital device for processing. Additionally, the images produced by CMOS sensors may contain defective pixels due to noise, manufacturing errors, or device malfunction. These defects need to be detected and corrected close to the sensor before the images are processed, which is essential for ensuring the images are helpful for human users as well as for image-processing and machine-vision algorithms.
This thesis reports the design and evaluation of two Smart Image Sensor (SIS) of 128×128 pixels using a 0.35 μm CMOS process, one for kernel-based spatial filtering calculation and the other for detecting and correcting defective pixels. The SISs are based on custom smart pixels that can perform some image processing algorithms in the analog domain. The remaining processing, including machinevision algorithms and machine-learning methods, is handled by an external digital device. The focus of this thesis does not extend to this latter aspect.
The first SIS enables spatial filtering using a single kernel, two different kernels on the same image, or two cascaded kernels. The circuit calculates the absolute value of each convolution on-pixel in parallel at each pixel using simple neighborhood arithmetic during photocurrent integration. This process is repeated when the filter requires more than one kernel. According to post-layout simulation, this SIS processes images at frame rates ranging from 743 to 752 frames per second (fps) and achieves a fill factor of 20.6 %. Our SIS outperforms state-of-the-art circuits in denoising, being the best result with a Mean Square Error (MSE) of 0.90, Peak Signal to Noise Ratio (PSNR) of 48.55 dB, and Structural Similarity Index Measure (SSIM) of 0.99, configured as the mean filter. Additionally, it demonstrates comparable edge detection performance to state-of-the-art circuits. The best results were achieved with a MSE of 4.68, PSNR of 41.42 dB, and SSIM of 0.97, using the y-direction Robert filter. Our proposed circuit demonstrated good edge-location accuracy of 0.99 % with the Sobel filter, which is one of the best results according to the Pratt Figure of Merit.
During photocurrent integration, the second SIS detects defective pixels in parallel at each pixel using simple arithmetic operations within a neighborhood. At the column level, the circuit replaces the defective pixels with the median value of their neighborhood. According to post-layout simulation, this SIS processes images at 694 fps, achieving a fill factor of 35.4 %. Our SIS produces better results than current state-of-the-art algorithms: it achieves a PSNR and Image Enhancement Factor (IEF) of 45 dB and 198.4, respectively, in images with 0.5 % random defective pixels, and a PSNR of 44.4 dB and IEF of 194.2, respectively, in images with 1.0 % random defective pixels.
The results demonstrate that, in a vision system, transferring part of the computation to the CIS can be beneficial. In this setup, an analog circuit performs computations at the pixel level using arithmetic operations within a pixel window while integrating photodiode current. This approach leverages the parallel architecture of the pixel array, enabling high frame rates, low power consumption, and compact size. Additionally, it minimizes the impact on fill factor and data accuracy.
Thanks to the results obtained in this thesis, our contributions are that a SIS is developed with on-pixel computing capabilities to perform kernel-based spatial filtering, allowing external configuration for various kernels and executing computations during photocurrent integration, achieving performance comparable to other SISs. Additionally, an algorithm from the literature for detecting defective pixels is modified for analog hardware implementation, yielding superior results compared to existing methods. Furthermore, another SIS is developed to detect defective pixels at the pixel level during photocurrent integration, implementing correction at the column level. This SIS is the first instance reported in the state of the art that combines detection and correction processes in the analog domain using CMOS technology on the same chip, achieving better outcomes than other implementations.