Abstract:
A CMOS imager in which a CMOS image sensor, a color image processing module and an image compression module are all provided on a single die. Both the color image processing module and the image compression module incorporate pipelined architectures to process the image data at a video rate in a massively parallel fashion.

Description:
FIELD OF THE INVENTION 
   The invention relates generally to improved semiconductor imaging devices and, more specifically, to a CMOS imager provided with an on-chip data compression module. 
   BACKGROUND OF THE INVENTION 
   A number of different types of semiconductor-based imagers exist, including charge coupled devices (CCDs), CMOS arrays, photodiode arrays, charge injection devices and hybrid focal plane arrays. Recently, however, CMOS imagers have gained popularity in use in a wide variety of electronic devices, because CMOS imagers offer a number of advantages over other types of imagers. CMOS imagers, for example, are compatible with integrated on-chip electronics (control logic and timing, image processing, and signal conditioning such as A/D conversion). CMOS imagers allow random access to the image data. CMOS imagers have lower fabrication costs as compared with the conventional CCD imagers, since standard CMOS processing techniques can be used. Additionally, CMOS imagers have low power consumption because-only one row of pixels at a time needs to be active during the readout and there is no charge transfer (and associated switching) from pixel to pixel during image acquisition. On-chip integration of electronics is particularly advantageous because of the potential to perform many signal conditioning functions in the digital domain (versus analog signal processing) as well as to achieve a reduction in system size and cost. 
   CMOS imagers as discussed above are generally known as discussed, for example, in Nixon et al., “256×.256 CMOS Active Pixel Sensor Camera-on-a-Chip,” IEEE Journal of Solid-State Circuits, Vol. 31(12) pp. 2046-2050, 1996; Mendis et al, “CMOS Active Pixel Image Sensors,” IEEE Transactions on Electron Devices, Vol. 41(3) pp. 452-453, 1994 as well as U.S. Pat. Nos. 5,708,263, 5,471,515, and 6,204,524, which are herein incorporated by reference. 
   Recent advances in CMOS image sensor technology include the integration of the imager and sophisticated image processing modules on a single die, as mentioned above. These systems-on-a-chip (SOCs) usually have cost, power consumption and form-factor advantages over the multi-chip solutions with the same functionality. Furthermore, image processing modules integrated with CMOS imagers can be fine tuned to the specific properties of the given imager and to the needs of the targeted applications. One segment of the market where the low power of CMOS imagers is most advantageous is the mobile devices market. Many cell-phone designs are incorporating image sensors in their architectures. However, bandwidth limitations imposed by wireless transmission, together with the desire to employ image sensors with large pixel counts, necessitates the use of image compression in the system. Many of the existing designs draw on the ability of the on-chip CPU to perform image compression, but at speeds far below video rates. Other available solutions rely on an additional image processing chip to perform color processing and compression. 
   All of the known solutions require a frame buffer memory to allow for rate conversion between video rate of incoming uncompressed data and the rate at which compression can be performed by either system or dedicated CPU. This leads to increased cost of the imager module and reduced video throughput of the system. Accordingly, it would be desirable to provide a CMOS imager with on-board image compression circuitry which processes image data in real time, and thus eliminates the need for a frame buffer memory. 
   SUMMARY OF THE INVENTION 
   The present invention provides a CMOS imager in which a CMOS image sensor, a color image processing module and an image compression module are all provided on a single die. Both the color image processing module and the image compression module incorporate pipelined architectures to process the image data at a video rate in a massively parallel fashion. 
   Additional advantages and features of the present invention will be apparent from the following detailed description and drawings, which illustrate preferred embodiments of the invention. 

   
     BRIEF DESCRIPTION OF THE DRAWINGS 
       FIG. 1  is a block diagram of a typical CMOS pixel sensor chip. 
       FIG. 2  is a block diagram of a preferred embodiment of the present invention. 
       FIG. 3  is a block diagram of the baseline JPEG compression module of the present invention. 
       FIG. 4  is a block diagram of the SRAM Addressing Scheme of the present invention. 
       FIG. 5  is a block diagram of the FIFO &amp; Register Control module of the present invention. 
       FIG. 6  is an illustration of a computer system having a CMOS imager according to the present invention. 
   

   DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
   In the following detailed description, reference is made to the accompanying drawings that form a part hereof, and in which are shown by way of illustration specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized, and that structural, logical and electrical changes may be made without departing from the spirit and scope of the present invention. 
     FIG. 1  illustrates a block diagram for a typical CMOS imager  10  having a pixel array  100  with each pixel cell being constructed in the manner disclosed, for example, in U.S. Pat. No. 6,204,524, the disclosure of which is herein incorporated by reference. Pixel array  100  comprises a plurality of pixels arranged in a predetermined number of columns and rows. The pixels of each row in array  100  are all turned on at the same time by a row select line, and the pixels of each column are selectively output by a column select line. A plurality of rows and column lines are provided for the entire array  100 . The row lines are selectively activated by the row driver  110  in response to row address decoder  120  and the column select lines are selectively activated by the column driver  160  in response to column address decoder  170 . Thus, a row and column address is provided for each pixel. The CMOS imager is operated by the control circuit  150  which controls address decoders  120 ,  170  for selecting the appropriate row and column lines for pixel readout, and row and column driver circuitry  110 ,  160  which apply driving voltage to the drive transistors of the selected row and column lines. 
   In the operation of the  FIG. 1  CMOS imager, one of the rows of pixels is selected for readout by using row driver  110  as described above. All pixels on a selected row are processed simultaneously and sampled onto a capacitor at the bottom of their respective columns. The column-parallel sampling process typically takes 1-10 mμ sec, and preferably occurs during the so-called horizontal blanking interval of a video image. Each column is then successively selected for read-out of the voltage stored in the capacitor of that column using column driver  160 . The output data from CMOS imager  10  is then processed in accordance with the present invention as described below. 
   With reference to the block diagram of  FIG. 2 , in a preferred embodiment of the present invention, color image processor  202  converts digitized imager data (output from CMOS imager  10  of  FIG. 1 ) from its native format (such as RGB) into a format more suitable for image compression, such as YCrCb (according to ISO CCIR-601). Once the image data is converted into appropriate for compression format, the data stream is passed to highly pipelined compression module. In the preferred embodiment of the present invention, compression module  204  implements JPEG compression at the video rate. In this embodiment, there are a number of line buffers  206 ,  208  at the front end of the compression designed to allow two-dimensional processing. In the preferred embodiment of the present invention, JPEG compression module  204  implements baseline JPEG compression, as described below in more detail with reference to  FIG. 3 . FIFO &amp; Register &amp; CPU Control module  205  implements all of the interface control and communicates with each of the modules and the outside system data/memory bus. 
     FIG. 3  provides a more detailed block of baseline JPEG compression module  204 . The circuitry is arranged in such a way as to pass the data from one stage to the next, allowing the entire compression operation to be performed at the video pixel rate. JPEG compression module  204  divides up the image into 8 by 8 pixel blocks in module  300 , and then calculates the fast discrete cosine transform (FDCT) of each block in module  301 . Quantizer module  302  rounds off the FDCT coefficients according to a specified quantization matrix. This step produces the “lossy” nature of JPEG, but allows for large compression ratios. Zigzag module  303  then performs variable length coding on these coefficients, followed by Huffman coding in module  305 , which then outputs the compressed binary bitstream. For decompression, JPEG recovers the quantized DCT coefficients from the compressed data stream, takes the inverse transforms and displays the image. 
   The present invention utilizes a unique SRAM addressing scheme to reduce the memory area and improve the utilization rate of memory to maximum. Referring back to  FIG. 2 , the output from Color Image Processing module  202  is continually written to the corresponding SRAM  206 ,  208  line by line. Since JPEG compression module  204  works on 8*8 blocks of data, 8 lines of data have to be ready before compression. The challenge is how to read and write to the SRAM at the same time. There are two addressing schemes to address this problem. The first scheme uses a “ping-pong” algorithm, which uses two SRAMs, one to store the data from Color Image Processing module, and the other for JPEG compression. This scheme switches SRAM banks after every interval of 8 lines of data. The second scheme uses only one SRAM. The JPEG compression module  204  begins to read after 8 lines of data have been written to the SRAM  206 ,  208 . Then, the output from Color Image Processing module  202  is written to the location in the SRAM from where the data has just been read. Compared with the ping-pong scheme, the latter scheme uses only one-half of the amount of SRAM. In the present invention, the second scheme is utilized, thereby greatly reducing the chip area. 
   With reference to the block diagram of  FIG. 4 , the detail of the SRAM addressing is shown. To reduce the chip area, the SRAM uses the minimum size to hold 8 lines of image data for JPEG compression. Module  401  automatically detects the video size and self-adjusts the addressing scheme. Module  402  delays reading for eight lines so that full 8 lines of data are ready for JPEG processing. Module  404  extracts two dimensional data from the linear SRAM, separates Y, Cb and Cr to reconstruct 8*8 blocks, and outputs the data in block order of Y Y Cb Cr, while new video data is simultaneously written to the SRAM. In each block, the data is output line by line. To provide real time video, the calculation of read and write addresses is performed “on-the-fly” without overwriting video data. All addresses including Y read address, Cr read address, Cb read address, Y write address, Cr write address, and Cb write address are calculated, pixel by pixel, which adds to the complexity of the scheme. The following formulas are used for all address calculations. The coefficients are adjusted every 8 lines. LineSize denotes the number of columns in the video frame.
 
Address_next=Clip (Address_current+Delta_current)
 
Delta_next=Delta_current*(LineSize/8)−(LineSize−1)*[Delta_current*(LineSize/8)/(LineSize−1)]
 
   
     
       
         
           
             
               
                 
                   Clip 
                   ⁡ 
                   
                     ( 
                     A 
                     ) 
                   
                 
                 = 
                   
                 ⁢ 
                 
                   
                     A 
                     - 
                     LineSize 
                     - 
                     
                       1 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       when 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       A 
                     
                   
                   &gt; 
                   
                     ( 
                     
                       LineSize 
                       - 
                       1 
                     
                     ) 
                   
                 
               
             
           
           
             
               
                 = 
                   
                 ⁢ 
                 
                   
                     2 
                     * 
                     
                       ( 
                       
                         LineSize 
                         - 
                         1 
                       
                       ) 
                     
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     when 
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     A 
                   
                   &gt; 
                   
                     2 
                     * 
                     
                       ( 
                       
                         LineSize 
                         - 
                         1 
                       
                       ) 
                     
                   
                 
               
             
           
         
       
     
   
   Due to the nature of JPEG encoding, the output data rate of the encoder is variable (intermittent). This might present a problem for the overall system architecture, as in the absence of the frame buffer there would be a need to constantly monitor the availability of the compressed data for transfer to the system memory or direct transmission to the remote site. This type of system behavior would unnecessarily tie-up system resources (such as CPU cycles and data bus bandwidth). In order to address this problem, the present invention incorporates a relatively small memory buffer, FIFO  210 , at the output of the encoder. This buffer, being much smaller that a full frame buffer, preserves the cost advantages of the system and allows for periods of fixed-rate data output interspersed with periods of inactivity. 
   In the preferred embodiment of the present invention, output buffer  210  is a dual-ported memory together with pointer control block, allowing the buffer to function as a FIFO (first-in-first-out). In this embodiment, the pointer control block allows for storage of the output of the encoder as soon as it becomes available, while allowing independent retrieval of the data based on the external requests, as long as FIFO is not empty. In yet another embodiment of this invention, the FIFO control block generates Half-Full and/or Almost Empty/Over X bytes flags describing the state of the content of the FIFO. These signals can then be used as an interrupt for external controller, prompting data retrieval cycle and relieving the external controller from the need to constantly monitor data availability in the FIFO. 
     Fig. 5  shows a more detailed block diagram of the FIFO &amp; Register &amp; CPU Control module, which implements the communication with the other modules. CPU Interface Control Module  502  communicates with the system data/memory bus through control signals and bi-directional data bus for various operations, such as register read/write, FIFO data read, FIFO current usage, INT status, etc. Decoder module  503  handles control signal decoding. Module  506  generates an INT signal according to the current FIFO status. Reading or writing to a specified register via the Register Bus ( FIG. 2 ) is coordinated through Register Interface Control module  500 . FIFO Interface Control module  504  controls the reading process from FIFO. All interfaces are controlled and synchronized by Overall Interface Control module  501 . 
   In the preferred embodiment of the present invention, the INT signal is supplied to access the internal FIFO over the system data/memory bus. To read video data correctly from the internal FIFO, the CPU must know the current state of FIFO. The following FIFO conditions will generate the INT signal transition and will also be reflected as the corresponding INT status register bits. The CPU inquiries the INT status register to get the current FIFO states. Based on the FIFO status, the CPU initiates or stops the video data read process or processes the data. 
   
     
       
             
             
             
           
         
             
                 
                 
             
           
           
             
                 
               Bit 0: 
               End of video frame is in FIFO 
             
             
                 
               Bit 1: 
               FIFO overflow 
             
             
                 
               Bit 2: 
               FIFO empty 
             
             
                 
               Bit 3: 
               End of Frame is read from FIFO 
             
             
                 
               Bit 4: 
               X bytes in FIFO where X is programmable 
             
             
                 
                 
             
           
        
       
     
   
   In a preferred embodiment of the present invention, the FIFO outputs are connected directly to the system data/memory bus, allowing for the access to the imager system to be performed over standard system communication channel in a way similar to the memory access. In yet another embodiment of the invention, the access to the registers/control functions of the imager system itself is also performed through the same bus interface. In this embodiment, the control signals are provided to allow distinguishing between various traffic over the output pins: image access, register write and register read access. 
   In addition to providing real-time compressed data stream, the imager system of the present invention may also need to provide uncompressed video stream either in full frame format or in decimated format (such as VGA image decimated to CIF resolution). Accordingly, in another embodiment of the present invention, the encoder output (compressed data stream) and the uncompressed video can be multiplexed to the input port of the output memory buffer (FIFO). 
   A typical processor based system that includes a CMOS imager device according to the present invention is illustrated generally at  600  in  FIG. 6 . A processor based system is exemplary of a system having digital circuits which could include CMOS imager devices. Without being limiting, such a system could include a computer system, camera system, scanner, machine vision system, vehicle navigation system, video telephone, surveillance system, auto focus system, star tracker system, motion detection system, image stabilization system and data compression system for high-definition television, all of which can utilize the present invention. 
   A processor system, such as a computer system, for example generally comprises a central processing unit (CPU)  644  that communicates with an input/output (I/O) device  646  over a bus  652 . The CMOS imager  610  also communicates with the system over bus  652 . The computer system  600  also includes random access memory (RAM)  648 , and, in the case of a computer system may include peripheral devices such as a floppy disk drive  654  and a compact disk (CD) ROM drive  656  which also communicate with CPU  644  over the bus  652 . As described above, CMOS imager  610  is combined with a pipelined JPEG compression module in a single integrated circuit. 
   The above description and drawings illustrate a preferred embodiment which achieves the objects, features and advantages of the present invention. It is not intended that the present invention be limited to the illustrated embodiments. Any modification of the present invention which comes within the spirit and scope of the following claims should be considered part of the present invention.