A 2 pJ / pixel / direction MIMO Processing based CMOS Image Sensor for Omnidirectional Local Binary Pattern Extraction and Edge Detection

Abstract This paper presents an energy-efficient multi-mode CMOS image sensor featuring omnidirectional local binary pattern extraction (LBP-E), edge detection (ED) and normal imaging. A mixed-signal 4-pixel simultaneous group computation (GC) scheme is developed to extract complete 8-direction LBP and edge. High-speed and energy-efficient column-parallel GC is achieved with the proposed group-switchable multi-input multi-output (MIMO) comparator. The reconfigurable mixedsignal processing circuits ensure multi-mode operations with minimum area overhead. Fabricated in 0. 18μm CMOS, the prototype sensor consumes 6. 5μW and 12. 7μW at 30fps for full 8-direction LBP-E and ED, achieving the state-of-the-art FoMs of 2.0 and 3.9 pJ/pixel/direction, respectively. Measurement results also demonstrate high-accuracy on-chip LBP-E, with a histogram similarity of up to 97.5% compared to the DSP one. Introduction Spatial contrast features such as LBP and edge have been widely adopted for various vision applications such as object detection and recognition [1-2]. Embedding mixed-signal LBP-E/ED on chip has enabled the image sensors to reduce both power and bandwidth, so as to extend the system lifetime [1-5]. Theoretically, LBP-E/ED should support full 8 feature directions to ensure high detection/recognition performance [6]. However, existing works are limited to 2 [1-3] or 4 [4-5] feature directions due to power/area constraints, degrading the system accuracy. Increasing feature directions inevitably causes significant system penalties, namely larger power/pixel size for in-pixel simultaneous comparison [4] and reduced speed/larger buffer memory for column serial processing [5]. In this work, we propose a low-power area-efficient multimode CMOS image sensor supporting LBP-E, ED and normal imaging. Complete omnidirectional feature extraction is achieved through the decentralized 4-pixel GC in Fig. 1. Compared with the conventional 8-direction pixel processing, the proposed decentralized scheme enables independent GCs, allowing column-parallel simultaneous processing. Therefore, pixel computation is effectively reduced by 2X and addressing complexity is greatly relaxed. High-speed 8-diretion extraction is supported by the proposed group-switchable MIMO (4I6O) dynamic comparator, which simultaneously processes all pairwise GC comparisons with minimum overhead. Both high energy and area efficiencies are achieved with the reconfigurable mixed-signal processing circuits. Architecture & Operation Fig. 2 shows the complete sensor system supporting 3 operation modes: LBP-E, ED and normal imaging. The pixel array consists of 128x108 groups with 2x2 pixels each, featuring 1.75T/pixel. The 2x2 pixels are binned for LBP-E/ED and separated for full-resolution 256x216 normal imaging. Two separate pixel readout channels are implemented in one single column to enable simultaneous oddand even-row pixel processing. Each column mainly consists of a MIMO comparator unit (COMP) and 2 reconfigurable binary capacitor arrays (Capp/Capn), which serve as the memories for analog processing or the capacitive DAC for ADC. As illustrated in Fig. 3, a 4I6O MIMO comparator is formed by 2 adjacent column-parallel three-stage COMPs. Each COMP can be connected to either of the neighboring columns to perform either odd-even (OE) or even-odd (EO) column GC processing. Fig. 4 shows the configurations at different operation modes. For LBP-E and ED, Capp and Capn alternatively store Vsig from the previous and current rows. In-pixel double sampling is employed to cancel pixel mismatch without using powerhungry amplifiers. In ED, a programmable LSB section of Capp is utilized for variable threshold generation. Capp/Capn and COMP are reconfigured as a low-power area-efficient 8b SAR-Single Slope (SS) ADC for normal imaging (6b SAR through Capn and 3b SS with 1b redundancy by ramping Capp). Fig. 5 shows the timing diagrams for LBP-E and ED. To support the row-rolling MIMO GC processing, a specific lowpower asynchronous offset cancellation scheme is developed, which independently stores the offset of each input transistor in different Vsig sampling periods. During each row operation, OE-GC and EO-GC are processed in turns. With the proposed MIMO comparator, singleand two-cycle GC processing are enabled for LBP-E and ED respectively. Consequently, omnidirectional feature extraction is realized in high speed with low power. Experimental Results A prototype sensor has been fabricated in 0. 18μm CMOS. The chip size is 2.5x2.0 mm2 and its microphotograph is shown in Fig. 6. The sensor dissipates 6. 5μW and 12. 7μW at 30fps for LBP-E and ED respectively, and 32μW at 15fps for normal imaging. Fig. 7 shows the captured sample images under different operation modes. Omnidirectional LBP and edge have been successfully extracted. The LBP image in Fig.7 is divided into 3x3 blocks to generate the 81-dimension LBP histogram in Fig. 8. Compared with the DSP results, the onchip LBP-E exhibits a high histogram similarity of up to 97.5%, demonstrating high-accuracy feature extraction with low power. Table I compares this work with the state of the art. This work features dual 8-direction feature extraction modes while achieving the best FoM1 of 16 and 31 pJ/pixel/frame for LBP-E and ED respectively. It also accomplishes the state-ofthe-art FoM2 of 2.0 and 3.9 pJ/pixel/direction. Conclusion In this paper, we present a low-power area-efficient multimode CMOS image sensor with embedded LBP-E, ED and normal imaging. Mixed-signal 4-pixel GC is adopted for omnidirectional feature extraction. A group-switchable MIMO comparator with a specific asynchronous offset cancellation scheme is proposed for column-parallel simultaneous GC processing. Through reconfigurable column-parallel MIMO processing, accurate 8-direction LBP and edge are extracted with high energy efficiency and low area overhead. The prototype sensor accomplishes the best FoM1 and FoM2 when compared with the state of the art.