Patent Publication Number: US-2003222998-A1

Title: Digital still camera system and method

Description:
CROSS REFERENCE TO RELATED APPLICATIONS  
     [0001] This application claims priority from provisional application Serial No. 60/293,912, filed May 25, 2001 and is a continuation-in-part of application Ser. Nos. 09/742,258, 09/745,132, 09/745,134, 09/745,135, and 9/745,136, all filed Dec. 20, 2000. The following pending patent applications disclose related subject matter and have a common assignee with the present application: Ser. No. 09/490,813, filed Jan. 26, 2000. 
    
    
     
       BACKGROUND OF THE INVENTION  
       [0002] This invention relates to integrated circuits, and more particularly, to integrated circuits and methods for use with digital cameras.  
       [0003] Recently, Digital Still Cameras (DSCS) have become a very popular consumer appliance appealing to a wide variety of users ranging from photo hobbyists, web developers, real estate agents, insurance adjusters, photo-journalists to everyday photography enthusiasts. Recent advances in large resolution CCD arrays coupled with the availability of low-power digital signal processors (DSPs) has led to the development of DSCs that come quite close to the resolution and quality offered by traditional film cameras. These DSCs offer several additional advantages compared to traditional film cameras in terms of data storage, manipulation, and transmission. The digital representation of captured images enables the user to easily incorporate the images into any type of electronic media and transmit them over any type of network. The ability to instantly view and selectively store captured images provides the flexibility to minimize film waste and instantly determine if the image needs to be captured again. With its digital representation the image can be corrected, altered, or modified after its capture. See for example, Venkataraman et al, “Next Generation Digital Camera Integration and Software Development Issues” in Digital Solid State Cameras: Design and Applications, 3302 Proc. SPIE (1998). Similarly, U.S. Pat. No. 5,528,293 and U.S. Pat. No. 5,412,425 disclose aspects of digital still camera systems including storage of images on memory cards and power conservation for battery-powered cameras.  
       SUMMARY OF THE INVENTION  
       [0004] The invention provides a digital still camera architecture with a programmable preview engine.  
       [0005] This has advantages including flexibility, adaptability, and efficiency. 
     
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
     [0006]FIGS. 1 a - 1   c  show a preferred embodiment system in functional block format and image processing steps.  
     [0007] FIGS.  2 - 6  illustrate data flows.  
     [0008]FIGS. 7 a - 7   b  show CFA arrangements.  
     [0009]FIG. 8 is a functional diagram for white balance.  
     [0010]FIGS. 9 a - 9   c  show gamma correction.  
     [0011]FIGS. 10 a -101 illustrates CFA interpolation.  
     [0012]FIGS. 11 a - 11   b  show color conversion.  
     [0013]FIGS. 12 a - 12   b  show a memory controller data flow.  
     [0014]FIGS. 13 a - 13   b  show burst compression/decompression.  
     [0015]FIG. 14 is a functional block diagram of a preview engine.  
     [0016]FIG. 15 is an on screen display block diagram.  
     [0017]FIG. 16 is an on screen display window.  
     [0018]FIG. 17 shows a hardware cursor.  
     [0019]FIGS. 18 a - 18   b  illustrate a DSP subsystem.  
     [0020]FIG. 19 shows parallel multiply-accumulate datapath.  
     [0021]FIG. 20 shows a coprocessor architecture.  
     [0022]FIG. 21 illustrates a look-up table accelerator.  
     [0023]FIG. 22 is a block diagram of a variable length coder.  
     [0024]FIGS. 23 a - 23   c  show a bridge.  
     [0025]FIG. 24 shows multiprocessor debugging support.  
     [0026]FIG. 25 illustrates UART connections.  
     [0027]FIG. 26 is a block diagram of flash card/smart card interface.  
     [0028]FIG. 27 shows image pipeline processing blocks.  
     [0029] FIGS.  28 - 38  illustrate color filter array interpolations.  
     [0030]FIGS. 39 a - 39   b  and  40  show white balancing.  
     [0031]FIGS. 41 a - 41   b  and  42   a - 42   e  indicate image resizing.  
     [0032] FIGS.  43 - 45  illustrate tone-scaling.  
     [0033]FIGS. 46 a - 46   b  and  47 - 48  show frame synchronization.  
     [0034] FIGS.  49 - 52  show decoding buffering.  
     [0035] FIGS.  53 - 71  illustrate preview engine aspects.  
    
    
     DESCRIPTION OF THE PREFERRED EMBODIMENTS  
     [0036] 1. System overview  
     [0037]FIGS. 1 a - 1   b  show the various high-level functional blocks in a preferred embodiment digital still camera (DSC) and systems with FIG. 1 b  providing more detail than FIG. 1 a . In particular, preferred embodiment integrated circuit  100  includes the following items: CCD Controller  102  interfaced with either CCD or CMOS imager  150 ; preview engine block  104  to convert the data from CCD controller  102  into a format suitable for display using NTSC encoder  106  or a digital LCD interface; burst mode compression-decompression engine  108  to compress the raw image data from CCD controller  102  using a lossless (or lossy, as selected by the user) compression and to write the compressed data to external SDRAM  160  via SDRAM controller  110 . This data can then be decompressed by the decompression engine under DSP  122  control, processed, and displayed or stored back to SDRAM  160 . DSP subsystem block  120  (DSP  122  and iMX  124  plus Variable Length Coder  126  and buffers  128 ) performs all the processing of the image data in the capture mode. The data is fetched from SDRAM  160  into image buffer  128  by DSP  122  through requests to SDRAM controller  110 , and DSP  122  performs all the image processing and compression required in the capture mode. The Image Extension processor (iMX)  124  acts as a dedicated accelerator to DSP  122  to increase the performance of DSP  122  for the imaging applications.  
     [0038] RISC microprocessor subsystem (ARM  130  plus memory  132 ) supports the in-camera Operating Systems (OS). Various OSes and other real-time kernels such as VxWorks, Microitron, Nucleus, and PSOS may be supported on circuit  100 .  
     [0039] SDRAM controller block  110  acts as the main interface between SDRAM  160  and all the function blocks such as the processors (ARM  130 , DSP  122 ), CCD controller  102 , TV encoder  106 , preview engine  104 , etc. SDRAM controller  110  may support up to 80 MHz SDRAM timing and also provide a low overhead for continuous data accesses. It also has the ability to prioritize the access units to support the real-time data stream of CCD data in and TV display data out.  
     [0040] Camera shot-to-shot delay is the time it takes for DSC engine  100  to read the data from CCD  150 , process it and write it to SDRAM  160 . The processing includes the image pipeline stages and also JPEG compression.  
     [0041] In order to support real-time preview, DSC engine  100  will set CCD  150  in “fast readout” mode, process the data, convert the data to NTSC format, and display the data on a built-in LCD screen (not shown in FIG. 1) or TV monitor as the case may be.  
     [0042] Auto focus, auto exposure and auto white balance (the 3A functions) are performed by DSP  122  while DSC  100  is in the preview mode of operation. DSP  122  reads the image data from SDRAM  160 , performs the 3A functions in real-time. The algorithms for the 3A functions are programmable.  
     [0043] Both interlace and progressive CCD and CMOS imagers  150  interface directly to DSC engine  100  using the built-in CCD/CMOS controller  102 .  
     [0044] In-camera operating systems such as Microitron will be supported efficiently on ARM processor  130  in DSC engine  100 . DSC engine  100  also has the capability to support capturing of a rapid sequence of images in the “burst mode” of operation. Bursts at up to 10 frames/sec of 2 Megapixel images will be supported. The duration of the burst sequence is only limited by the size of SDRAM  160  of the DSC system. Also, MPEG compression may be used for short clips. And capabilities for playback of audio-video include circular buffering.  
     [0045] DSC circuit  100  also includes I/O block  140  with USB core  142  for programming and interrupt processing with ARM  130 .  
     [0046] CCD module  150  includes a CCD imager to sense the images, driver electronics and a timing generator for the necessary signals to clock the CCD, correlated double sampling and automatic gain control electronics. This CCD data is then digitized and fed into the DSC Engine  100 .  
     [0047] SDRAM  160  may be any convenient size and speed SDRAM.  
     [0048] DSC systems may be even more versatile with the ability to annotate images with text/speech. The preferred embodiment programmable DSP allows easy inclusion of a modem and/or a TCP/IP interface for direct connection to the Internet. DSCs may run complex multi-tasking operating systems to schedule the various real-time tasks.  
     [0049] Thus the preferred embodiments provide platforms for programmable camera functions, dual processors (ARM and DSP) plus an image coprocessor, burst mode compression/decompression engine, programmable preview engine, and integration of all camera peripherals including IrDA, USB, NTSC/PAL encoder, DACs for RGB, UART, and compact flash card/smart media card interface. Further, the platforms can provide both camera functions and digital audio playback on the same integrated circuit.  
     [0050] The following sections provide more detail of the functions and modules.  
     [0051] 2. DSC operating modes  
     [0052] The preferred embodiment systems have (1) Preview mode, (2) Capture mode, (3) Playback mode, and (4) Burst mode of operation as follows.  
     [0053] (1) Preview mode has data flow as illustrated in FIG. 2. ARM  130  sets CCD  150  into high-frame-rate readout mode (reduced vertical resolution). ARM  130  enables preview engine  104  and sets the appropriate registers for the default parameters. The raw CCD data is streamed into preview engine  104  and, after preview engine processing, is streamed into SDRAM  160 . ARM  130  enables TV encoder  106  to display the preview engine output. Preview engine  104  processing (hardware) includes gain control, white balance, CFA interpolation, down-sampling, gamma correction, and RGB to YUV conversion. ARM  130  commands DSP  122  to perform auto exposure and auto white balance whenever required. DSP  122  processing includes auto exposure, auto white balance, and auto focus. ARM  130  receives new parameters for preview engine  104  and loads the preview engine hardware with these parameters. The output is full resolution CCIR 601 NTSC/PAL and real-time updating of gain, white balance, and auto focus.  
     [0054] (2) Capture mode has data flow as illustrated in FIG. 3 a . ARM  130  sets CCD  150  in “fine” readout mode, full resolution. The CCD data is read directly into SDRAM  160  through SDRAM controller  110 . ARM  130  commands DSP  122  (plus IMX  124  and VLC engine  126 ) perform capture processing: black clamp, fault pixel correction, shading compensation, white balancing, gamma correction, CFA interpolation, color space conversion, edge enhancement, false color suppression, 4:2:0 down-sampling, and JPEG compression. The DSP stores compressed data in the SDRAM. ARM  130  writes the compressed data to compact flash/smart media  182 .  
     [0055] The computation is scheduled as two threads: iMX on one thread, the other units on the other thread. FIG. 3 b  shows timing and data flow with threads related to buffers A and B.  
     [0056] (3) Playback mode has data flow as illustrated in FIG. 4. ARM  130  reads the compressed data from CFC/Smartmeda  182  into SDRAM  160  through the SDRAM controller  110  using DMA  162 . ARM commands DSP  122  to do “playback”. DSP processing (DSP  122  plus IMX  124  and VLC engine  126 ) includes JPEG decode (bitstream parsing, IDCT, VLD, and down-sampling for aspect ratio) and store uncompressed image data in SDRAM. ARM enables TV encoder  106  to display the image on TV/LCD display. Note that also audio plus video (e.g., MPEG compressed) clips may be played back. (4) Burst capture mode has data flow as illustrated in FIG. 5, and FIG. 6 shows offline data processing. ARM  130  sets CCD  150  into fine resolution mode. ARM sets up the burst compression parameters, burst length, number of frames/second, compression ratio (lossy, lossless), etc. ARM enables burst compression engine  108  to write the raw CCD data to SDRAM  160 . ARM signals DSP to process each of the stored raw CCD images in the burst. Burst mode decompression engine  108  decompresses each of the burst captured images. DSP processes each of the images as in normal capture and writes the JPEG bitstream to SDRAM  160 .  
     [0057] Burst capture mode is achieved by repeated calls to the regular playback routine with a different JPEG bitstream each time by ARM  130 .  
     [0058] The preferred embodiment also has MPEG1 capture mode and playback mode.  
     [0059] 3. Image Acquisition  
     [0060] A DSC usually has to perform multiple processing steps before a high quality image can be stored. The first step is the image acquisition. The intensity distribution reflected from the scene is mapped by an optical system onto the imager. The preferred embodiments use CCDs, but a shift to CMOS does not alter the image processing principles. To provide a color image the imager (CCD or CMOS) has each pixel masked by a color filter (such as a deposited dye on each CCD photosite). This raw imager data is normally referred as a Color-Filtered Array (CFA). The masking pattern of the array of pixels in the CCD as well as the filter color primaries vary between different manufactures. In DSC applications, the CFA pattern that is most commonly used is an RGB Bayer pattern that consists of 2×2 cell elements which are tiled across the entire CCD-array. FIG. 7 a  depicts a subset of this Bayer pattern in the matrix block following the CCD camera. Note that half of the pixels are sensitive to green and that the red and blue are balanced to green. FIG. 7 b  shows a subset of the alternative complementary color CFA pattern with yellow, cyan, green, and magenta pixels. Each pixel in the final color image has three (or four) color values, such as a red, a green, and a blue value for RGB images. The red values alone could be called the “red plane” or “red channel” or “red array”, and the raw data from the CFA (where each pixel has only one color value) may be separated into the “red subarray”, “green subarray”, and “blue subarray” with the subarray either considered alone or as embedded in a full array (or plane or channel) with the other pixels&#39; values as Os.  
     [0061] 4. Image Pipeline  
     [0062] CFA data needs to undergo a significant amount of image processing before the image can be finally presented in a usable format for compression or display. All these processing stages are collectively called the “image pipeline”. The preferred embodiment DSC may perform multiple processing steps before a high quality image can be stored, and FIG. 1 c  illustrates a possible set of processing steps. Most of the image pipeline processing tasks are multiply-accumulate (MAC) intensive operations, making a DSP a preferred platform. The various image pipeline processing stages are described in the following sections.  
     [0063] A/D Converters  
     [0064] The A/D converter digitizing the CCD imager data may have a resolution of 10 to 12 bits. This allows for a good dynamic range in representing the input image values. Of course, higher resolution implies higher quality images but more computations and slower processing, and lower resolution implies the converse. The A/D converter may be part of the CCD module.  
     [0065] Black Clamp  
     [0066] After A/D conversion the “black” pixels do not necessarily have a 0 value due to a CCD which may still record some current (charge accumulation) at these pixel locations. In order to optimize the dynamic range of the pixel values represented by the CCD imager, the pixels representing black should have a 0 value. The black clamp function adjusts for this by subtracting an offset from each pixel value. Note that there is only one color channel per pixel at this stage of the processing.  
     [0067] Fault Pixel Interpolation  
     [0068] CCD-arrays may have defective (missing) pixels, especially arrays with more than 500,000 elements. The missing pixel values are filled by simple interpolation. A high order interpolation may not be necessary because an interpolation is also performed in the CFA interpolation stage. Therefore, the main reason for this preliminary interpolation step is to make the image processing regular by eliminating missing data.  
     [0069] Typically, the locations of the missing pixels are obtained from the CCD manufacturer. The faulty pixel locations can also be computed by the DSC engine offline. For example, during camera initialization operation, an image with the lens cap closed is captured. The faulty pixels appear as “white spots” while the rest of the image is dark. The faulty pixel locations can then be identified with a simple threshold detector and stored in memory as a bitmap.  
     [0070] During the normal operation of the DSC the image values at the faulty pixel locations are filled by a simple bilinear interpolation technique.  
     [0071] Lens Distortion Compensation  
     [0072] Due to non-linearities introduced by imperfections in lenses, the brightness of the image decreases from the center of the image to the borders of the image. The effects of these lens distortions are compensated by adjustment of the brightness of each pixel as a function fo its spatial location. The parameters describing the lens distortions need to be measured with the final system, supported by information supplied by the lens manufacturer.  
     [0073] The lens adjustment can be accomplished by multiplying the pixel intensity with a constant, where the value of the constant varies with the pixel location. The adjustment needs to be done for both horizontal and vertical directions.  
     [0074] White Balance  
     [0075] White balancing tries to transform the tristimulus values sensed under a certain light condition such that if displayed white appears again as white. In general the colors as captured by the camera do not appear on an output device as they were seen when capturing the scene. A couple of reasons account for that.  
     [0076] First, the sensitivity of the color filters over the spectral range are slightly different. If exposed with a perfect white light source (constant light spectrum) the tristimulus values sensed by the CCD are slightly different.  
     [0077] Second, the design of the entire CCD module and the optical system add to the imbalance of the tristimulus values.  
     [0078] Third, typical illuminants present while recording a scene are not constant. The illuminants have a certain “color”, which is typically characterised as “color temperature” (or correlated color temperature). If an image captured under illuminant 1 is displayed under a different illuminant the color appearance changes. This causes a white area to turn a little bit red or a little bit blue.  
     [0079] Several different approaches for white balancing are known. Most of them multiply the red and blue channels with a factor such that the resulting tristimuls value for a white patch has identical values:  
         [           R   ′               G   ′               B   ′           ]     =       [         a1       0       0           0       1       0           0       0       a2         ]                [         R           G           B         ]               R   ′     =       G   ′     =       B   ′                   for                 a                 neutral                   (   gray   )                   patch                     
 
     [0080] However, as explained later, this approach does not provide correction for changes of the illuminant. Therefore, the white balancing implementation in preferred embodiment system corrects imbalances of the sensor module. The illumination correction is handled at a later stage in the color correction section.  
     [0081] Typical techniques to calculate the gain factors are  
     [0082] (1) equal energy  
       a 1=Σ (x,y)   g   2 ( x,y )/Σ (x,y)   r   2 ( x,y )  
     [0083] (2) Gray World Assumption  
       a 1=Σ (x,y)   g ( x,y )/Σ (x,y)   r ( x,y )  
     [0084] (3) Maximum Value in an Image is White  
       a 1=max (x,y)   g ( x,y )/max (x,y)   r ( x,y )  
     [0085] All of them do not hold in every case. Therefore, by defining the white balancing mainly as a correction of imager module characteristics, the algorithms to obtain the correction values can be made almost scene independent.  
     [0086] The FIG. 8 depicts the simplified realization of the preview engine, giving good results as long as the CCD sensor operates in the linear range. The white balance section below discusses a more sophisticated method.  
     [0087] Gamma Correction  
     [0088] Display devices (TV monitors) used to display images and printers used to print images have a non-linear mapping between the image gray value and the actual displayed pixel intensities. Hence, in the preferred embodiment DSC Gamma correction stage compensates the CCD images to adjust them for eventual display/printing.  
     [0089] Gamma correction is a non-linear operation. The preferred embodiments implement the corrections as table look ups. The advantages of table look up are high speed and high flexibility. The look-up table data might even be provided by the camera manufacturer.  
     [0090] With 12-bit data, a full look-up table would have 4K entries, with each entry 8 to 12 bits. For a smaller look-up table, a piecewise linear approximation to the correction curves could be used. For example, the 6 most significant bits could address a 64-entry look-up table whose entries are pairs of values: a base value (8 to 12 bits) and a slope (6 bits). Then the product of the 6 least significant bits and the slope is added to the base value to yield the final corrected value of 8 to 12 bits. FIG. 9 b  illustrates a piecewise linear approximation curve, and FIG. 9 c  the corresponding operations.  
     [0091] Note that LCD displays can be considered to be linear, making gamma compensation unnecessary. However, LCD display modules usually expect an NTSC input (which is already gamma compensated) and hence perform some “gamma uncorrection” (inverse gamma correction) to compensate for this expected gamma correction. So in the preferred embodiment DSCs using such LCD preview modules, still perform Gamma correction and then NTSC encode the signal before feeding it to the LCD module.  
     [0092] Gamma correction may be performed at the end of the all the stages of the image pipeline processing and just before going to the display. Alternatively, the image pipeline could perform the Gamma correction earlier in the pipeline: before the CFA interpolation stage.  
     [0093] CFA Interpolation  
     [0094] Due to the use of a color-filtered array (CFA), the effective resolution of each of the color planes is reduced. At any given pixel location there is only one color pixel information (either of R, G, or B in the case of RGB color primaries). However, it is required to generate a full color resolution (R, G, and B) at each pixel in the DSC. To be able to do this, the missing pixel values (R and B at the G location, etc.) are reconstructed by interpolation from the values in a local neighborhood in the CFA interpolation. To take advantage of the DSP in this system a FIR-kernel is employed as interpolation filter. The length of the filter and the weights vary from one implementation to the other. Also the interband relationship has to be considered. FIG. 10 describes the realization of the CFA interpolation in the hardwired preview engine module. It basically employs a 1 D FIR kernel for horizontal followed by vertical interpolation.  
     [0095] The implementation in the DSP subsystem for high quality image processing is different in that it is fully programmable and able to utilize 2D filter kernels. Some background information and a proposal for an improved CFA interpolation technique is given in subsequent sections.  
     [0096] Color Correction  
     [0097] Changes in the color appearance caused by differing illuminants between capture and playback/print cannot be corrected just by balancing the red, green and blue channels independently. To compensate for this, a tone (color) correction matrix maps the RGB pixel values to corrected RGB pixel values that take the illuminant into account.  
     [0098] The principle is as follows. Let I1 denote an N×N diagonal matrix describing the recording illuminant, S the N×3 matrix denoting the spectral characteristics of the imager module with one column vector for each color, and R the 1×N column vector describing the reflectance of the scene. The measured tristimulus value X1 at a pixel location is given by:  
       X 1 T   =R   T   *I 1* S    
     [0099] Denoting  
     
       SS=S*S 
       T  
     
     [0100] we can transform the measured tristimulus value X1 into X2, we would have been measured if the scene would have been illuminated by I2:  
       X 2 T   =X 1 T   *S   T   *SS   −1   *I 1 −1   *I 2 *S    
     [0101] The 3×3 transform matrix S T *SS −1 *I1 −1 *I2*S can be calculated offline, assuming that the spectral response of the sensor can be measured. Thus it is sufficient to store a set of color correction matrices for different illuminants in the camera.  
     [0102] Since the subjective preferences of the color appearance changes among users, it is easily possible to include these into the color correction matrix or add a separate step to the image processing pipeline (e.g. “tone scale”).  
     [0103] Color Space Conversion  
     [0104] After the CFA interpolation and color correction, the pixels are typically in the RGB color space. Since the compression algorithm (JPEG) is based on the YCbCr color space, a color space transformation must be carried out. Also the preferred embodiment DSC generates a NTSC signal output for display on the TV and also to feed into the LCD preview. Hence an RGB to YCbCr color space conversion needs to be carried out. This is a linear transformation and each Y, Cb, Cr value is a weighted sum of the R, G, B values at that pixel location. FIG. 11 a  illustrates the color conversion as realized in the hardwired preview engine. The DSP (playback) implementation is similar in principle but allows a higher precision conversion:  
         [         Y           Cb           Cr         ]     =       [         α1         α                 2           α                 3               α                 4           α                 5           α                 6               α                 7           α                 8           α                 9           ]                [         R           G           B         ]                   
 
     [0105] Edge Enhancement  
     [0106] After CFA interpolation the images appear a little “smooth” due to the low pass filtering effect of the interpolation filters. To sharpen the images it is sufficient to operate on the Y-component only. At each pixel location we compute the edge magnitude using an edge detector, which is typically a two-dimensional FIR filter. The preferred embodiment uses a 3×3 Laplace-Operator. The edge magnitude is thresholded and scaled and before being added to the original luminance (Y) image to enhance the sharpness of the image.  
     [0107] The edge enhancement is a high pass filter; this high pass filter also amplifies the noise. To avoid this amplified noise, a threshold mechanism is used to only enhance those portion of the image lying on an edge. The amplitude of the amplified edge may vary. The threshold operation is necessary to reduce amplification of noise. Therefore, only those pixels get enhanced which are an element of an edge. The enhancement signal added to the luminance channel can be represented graphically as in FIG. 11 b ; the parameters t 1 , t 2 , and the slope s 1  can be chosen as seen necessary to obtain the best quality.  
     [0108] False Color Suppression  
     [0109] Note that the edge enhancement is only performed in the Y image. At edges the interpolated images of the color channels may not be aligned well. This causes annoying rainbow-like artifacts at sharp edges. Therefore, by suppressing the color components Cb and Cr at edges in the Y-component, these artifacts can be reduced. Depending on the output of the edge detector, the color components Cb and Cr are multiplied by a factor less than 1 on a per pixel basis to suppress the false color artifacts.  
     [0110] Image Compression  
     [0111] The image compression step compresses the image, typically by about 10:1 to 15:1. The preferred embodiment DSC uses JPEG compression. This is a DCT-based image compression technique that gives good performance.  
     [0112] Auto Exposure  
     [0113] Due to the varying scene brightness, to get a good overall image quality, it is necessary to control the exposure of the CCD to maximize the dynamic range of the digitized image. The main task of exposure control is to keep the sensor operating in the linear range by controlling the shutter speed, and if possible the aperture of the optical system. Since closing the iris and slowing down the shutter speed compensates each other, there exists a certain parameter range in which the exposure remains unchanged. It is obvious that this can be accomplished only to a certain extent, as other constraints such as capturing fast moving scenes may be desired by the user.  
     [0114] Besides trying to keep the sensor operating in the linear range it is desirable to maximize the dynamic range of the ADC and hence the digitized image. This is done by controlling the PGA in the AFE. The processing necessary to obtain the relevant control parameters is performed on the DSP.  
     [0115] Auto Focus  
     [0116] It is also possible to automatically adjust the lens focus in a DSC through image processing. Similar to Auto Exposure, these auto focus mechanisms operate also in a feed back loop. They perform image processing to detect the quality of lens focus and move the lens motor iteratively till the image comes sharply into focus. Auto focus may rely on edge measurements from the edge enhancement previously described.  
     [0117] Playback  
     [0118] The preferred embodiment DSCs also provide the ability for the user to view the captured images on LCD screen on the camera or on an external TV monitor. Since the captured images are stored in SDRAM (or on compact flash memory) as JPEG bitstreams, playback mode software is also provided on the DSP. This playback mode software decodes the JPEG bitstream, scales the decoded image to the appropriate spatial resolution, and displays it on the LCD screen and/or the external TV monitor.  
     [0119] Down-Sampling  
     [0120] In the preferred embodiment DSC system the image during the playback mode after decoding the JPEG data is at the resolution of the CCD sensor, e.g. 2 Megapixels (1600×1200). This image can even be larger depending on the resolution of the CCD sensor. However, for the display purposes, this decoded data has to be down-sampled to NTSC resolution (720×480) before it can be fed into the NTSC encoder. Hence, the DSC should implement a down-sampling filter at the tail end of the playback mode thereby requiring additional DSP computation.  
     [0121] The preferred embodiment solves this problem of additional DSP computations by a DCT-domain down-sampling scheme that is included as part of the JPEG decompression module. Note that the JPEG decompression essentially involves three stages: first an entropy decoding stage, followed by an inverse quantization stage, and finally an IDCT stage. In JPEG the IDCT is performed on a block of 8×8 pixels. The preferred embodiments down sample a 2 Megapixel image to NTSC resolution (a 4/8 down-sampling) in the IDCT domain by employing a 4×4 IDCT to the top left 4×4 DCT coefficients (out of a 8×8 DCT coefficient block) and hence effectively achieving both the IDCT and the 4/8 down-sampling in one step. The sampling ratio can be varied between 1/8 (smallest image) to 8/8 (full resolution image).  
     [0122] A separable two-dimensional 4-point IDCT is applied to obtain a 4×4 block of image pixels from the top-left (low spatial frequency) 4×4 DCT coefficients. By this low-order IDCT we effectively combine anti-aliasing filtering and 8-to-4 decimation. The employed anti-aliasing filter corresponds to a simple operation of preserving only the 16 lowest frequency components in the DCT domain without scaling the preserved DCT coefficients. Though this simple filter is effective in reducing aliasing effect, the preferred embodiments may have a lowpass filter with better frequency response to further reduce aliasing. The use of other lowpass filters will lead to scaling of the preserved coefficients where the scaling factor is the location of each DCT coefficient.  
     [0123] Note that the DCT domain down-sampling technique does not increase the computational complexity. In fact, it reduces the computation since the JPEG decoding stages after entropy decoding does not need to deal with the whole 8×8 DCT coefficients except the top-left 4×4 coefficients. Use of other anti-aliasing filters also does not add any complexity since the coefficient scaling operation can be merged into the low-order IDCT operation. Also note that this DCT domain down-sampling idea technique can offer n/8 down-sampling ratios, n=1, . . . 7, for other CCD sensor resolutions.  
     [0124] Up-Sampling  
     [0125] Displaying cropped images for zooming of images also uses an up-sampling scheme. The inverse approach to the down-sampling provides an elegant tool. In the first case the 8×8 DCT coefficients are (virtually) vertically and horizontally extended with zeroes to form a block of N×M coefficients (N,M&gt;8). On this block an IDCT of size N×M is executed yielding N×M samples in the spatial domain.  
     [0126] Currently, most image pipeline operations are non-standardized. Having a programmable DSC engine offers the ability to upgrade the software to conform to new standards or improve image pipeline quality. Unused performance can be dedicated to other tasks, such as human interface, voice annotation, audio recording/compression, modem, wireless communication, etc.  
     [0127]FIG. 27 shows a preprocessing functional block diagram including CFA interpolation, white balance, color correction, tone scaling, gamma correction, conversion of RGB to YCrCb, edge enhancement, edge detection, color boost, and false color suppression in preparation of JPEG compression. The following sections describe preferred embodiments relating to CFA interpolations.  
     [0128] 5. CFA Interpolation with Reduced Aliasing  
     [0129] A preferred embodiment CFA interpolation for a Bayer pattern (FIG. 7 a ) uses the high-frequency from the green channel to modify the red and blue channel interpolations to reduce the aliasing components at edges within the image by utilizing the signal of the other color channels. By this means artifacts are reduced, sharpness improved, and additional post-processing avoided. Indeed, proceed as follows.  
     [0130] (1) apply interpolation to green channel (any interpolation method); this yields the green plane.  
     [0131] (2) detect edges in the green channel (by gradient or other method).  
     [0132] (3) compute high-pass component of the green channel (filter with any high-pass filter).  
     [0133] (4) apply interpolation to the red channel (any interpolation method); this yields the red plane.  
     [0134] (5) add high-pass component of (3) (with a weighting factor) to red channel.  
     [0135] (6) apply interpolation to the blue channel (any interpolation method); this yields the blue plane.  
     [0136] (7) add high-pass component of (3) (with a weighting factor) to the blue channel.  
     [0137] So the final image consists of three color planes: the green plane from step (1), the red plane from step (5), and the blue plane from step (7). That is, for a pixel in the final image the green intensity is taken to be the value of the corresponding pixel of the green plane from step (3), the red intensity is taken to be the value of the corresponding pixel of the modified red plane from step (5), and the blue intensity is taken to be the value of the corresponding pixel of the modified blue plane from step (7)  
     [0138] Theoretical analysis of the foregoing: Each CCD pixel averages the incident optical signal over the spatial extent of the pixel; thus the CCD effectively provides a low-pass filtering of the incident optical signal with a cutoff frequency the reciprocal of the pixel size. Further, the subsampling of the pixel array by the color filters on the pixels leads to aliasing in each color plane. Indeed, for red and blue the subsampling is by a factor of 2 in each direction; so the frequency spectrum folds at half the maximum frequency in each direction. Thus the red and blue baseband spectra areas are each one-quarter of the original array spectrum area (reflecting that the red and blue samplings are each one-quarter of the original array). For green the subsampling is only half as bad in that the spectrum folding is in the diagonal directions and at a distance {square root}2 as large as for the red and blue. The green baseband spectrum is one-half the area of the original array spectrum.  
     [0139] Color fringing at edges is an aliasing problem. In addition, dissimilar baseband spectra lead to color fringing as well, even if no aliasing is present. Indeed, aliasing is not necessarily visible in a single color band image, but the effect becomes obvious upon combination of the three color components into one color image. The shift of the sampling grids between red, green, and blue causes a phase shift of the aliasing signal components. A one-dimensional example clarifies this: presume a one-dimensional discrete signal f(n) and two subsamplings, each by a factor of 2 but one of even-numbered samples and one of odd-numbered samples (so there is a shift of the sampling grids by one sample):  
       f   even (2 m )= f (2 m )  
       f   even (2 m +1)=0  
       f   odd (2 m )=0  
       f   odd (2 m +1)= f (2 m +1)  
     [0140] Of course, f(n)=f even (n)+f odd (n). Let F(z) be the z-transform of f(n), F even (z) the z-transform of f even (n), and F odd (z) the z-transform of f odd (n). Then noting that F even (z) is an even function of z (only even powers of z) and F odd (z) an odd function of z (only odd powers of z):  
       F   even ( z )={ F ( z )+ F (− z )}/2  
       F   odd ( z )={ F ( z )− F (− z )}/2  
     [0141] The F(−z) corresponds to the aliasing and appears with opposite signs; that is, a phase shift of π.  
     [0142] The color fringing can be reduced by a phase shift of π of the aliased components. However, this is very difficult to achieve, because the only available signal is the sum of the original signal with the aliasing signal. Therefore, the preferred embodiments have another approach.  
     [0143] As long as two (or more) subsampled signals (i.e., red, green, and blue) have identical characteristics (such as for a gray scale image), a perfect reconstruction of the original image can be achieved by just adding the subsampled signals. However, in CFA interpolation generally the subsampled signals stem from different color bands. Aliasing errors become visible especially at edges where the interpolated signals of the different color bands are misaligned. Therefore, the preferred embodiments counter color fringing at edges by reducing the aliasing components only at edges through utilization of other ones of the subsampled signals. This reduces artifacts, improves sharpness, and avoids additional postprocessing.  
     [0144] In particular, for Bayer pattern CFA the green channel has a higher cutoff frequency than that of the red and blue channels; thus the green channel has less significant aliasing. The aliasing signal to be compensated is a high-pass signal, which is now estimated as the high-pass component of the green channel; and this is added (rather than subtracted due to the phase shift due to the offset of the red and blue subsampling grids relative to the green subsampling grid) to the red and blue channels. The high-pass green component could be multiplied by a scale factor prior to addition to the red and blue subsamplings. The signals are added while interpolating red, blue or afterwards.  
     [0145] 6. CFA Interpolation with Inter-Hue Adaptation  
     [0146] Alternative CFA interpolation preferred embodiments first interpolate Bayer pattern greens using a 5×5 FIR filter, and then use the interpolated green to interpolate red and blue each with two steps: first interpolate diagonally to form a pattern analogous to the original green pattern (this interpolation uses a normalization by the green to estimate high frequencies), and then apply a four-nearest neighbor interpolation (again using green normalization to estimate high frequencies) to complete the red or blue plane.  
     [0147] More explicitly, denote the CFA value for pixel location (y,x), where y is the row number and x the column number of the array, as follows: red values R(y,x) at pixel locations (y,x) where y and x are both even integers, blue values B(y,x) where y and x are both odd integers, and green values g(y,x) elsewhere, that is, where y+x is an odd integer.  
     [0148] First, let G{circumflex over ( )}(y,x) denote the green value at pixel location (y,x) resulting from the green plane interpolation; this is defined for all pixel locations (y,x). This interpolation can be done by various methods, including the edge preservation interpolation of the following section. Note that many interpolations do not change the original green values; that is, G{circumflex over ( )}(y,x)=G(y,x) may be true for (y,x) where G was originally defined (i.e., y+x is an odd integer).  
     [0149] Next, define the red and blue interpolations each in two steps as illustrated in FIG. 28 which is labeled for blue and uses arrows to show interpolation contributions.  
     [0150] First red step: R(y,x) is already defined for pixel locations (y,x) with y=2m, and x=2n with m and n integers; so first for y=2 m+1 and x=2n+1, define R{circumflex over ( )}(y,x): 
     [0151]               R   ^     (     y   ,   x     )       =              G   ^     (     y   ,   x     )            {         R        (       y   -   1     ,     x   -   1       )       /     G   ^     (       y   -   1     ,     x   -   1       )         +                                  R        (       y   -   1     ,     x   +   1       )       /     G   ^     (       y   -   1     ,     x   +   1       )         +                              R        (       y   +   1     ,     x   -   1       )       /     G   ^     (       y   +   1     ,     x   -   1       )         +                              R        (       y   +   1     ,     x   +   1       )       /     G   ^     (       y   +   1     ,     x   +   1       )         }     /   4                         
     [0152] This interpolates the red plane to the pixels where B(y,x) was defined. (FIG. 28 illustrates the analogous interpolation for blue.) Note that the this interpolation essentially averages the red values at the four corners of the 3×3 square about (y,x) with the values normalized at each location by the corresponding green values. If any of the green values are below a threshold, then omit the normalization and just average the red values.  
     [0153] Perform the first blue step in parallel with the first red step because the same green values are being used.  
     [0154] First blue step: B(y,x) is already defined for pixel locations (y,x) with y=2 m+1, and x=2n+1 with m and n integers, so first for y=2m and x=2n, define B{circumflex over ( )}(y,x):  
     [0155]         B   ^     (     y   ,   x     )       =       G   ^     (     y   ,   x     )              {         B        (       y   -   1     ,     x   -   1       )       /     G   ^     (       y   -   1     ,     x   -   1       )         +       B        (       y   -   1     ,     x   +   1       )       /     G   ^     (       y   -   1     ,     x   +   1       )         +       B        (       y   +   1     ,     x   -   1       )       /     G   ^     (       y   +   1     ,     x   -   1       )         +       B        (       y   +   1     ,     x   +   1       )       /     G   ^     (       y   +   1     ,     x   +   1       )           }     /   4                       
     [0156] This interpolates the blue plane to the pixels where R(y,x) was defined as illustrated in the lefthand portion of FIG. 28. Again, this interpolation essentially averages the blue values at the four corners of the 3×3 square about (y,x) with the values normalized at each location by the corresponding green values.  
     [0157] Second red step: define R{circumflex over ( )}(y,x) where y+x is an odd integer (either y=2m and x=2n+1 or y=2 m+1 and x=2n)  
     [0158]               R   ^     (     y   ,   x     )       =            G   ^       (     y   ,   x     )     [         R   ^     (       y   -   1     ,   x     )       /     G   ^     (       y   -   1     ,   x     )         +                                  R   ^     (     y   ,     x   +   1       )       /     G   ^     (     y   ,     x   +   1       )         +                              R   ^     (       y   +   1     ,   x     )       /     G   ^     (       y   +   1     ,   x     )         +                              R   ^     (     y   ,     x   +   1       )       /     G   ^     (     y   ,     x   +   1       )         ]     /   4                         
     [0159] This second step interpolates the red plane portion defined by the first step to the pixels where G(y,x) is defined. Again, this interpolation essentially averages the red values at four neighboring pixels of (y,x) with the values normalized at each location by the corresponding green values.  
     [0160] Second blue step: define for y+x an odd integer (either y=2m and x=2n+1 or y=2 m+1 and x=2n)  
               B   ^     (     y   ,   x     )       =              G   ^     (     y   ,   x     )            {         B   ^     (       y   -   1     ,   x     )       /     G   ^     (       y   -   1     ,   x     )         +                                  B   ^     (     y   ,     x   +   1       )       /     G   ^     (     y   ,     x   +   1       )         +                              B   ^     (       y   +   1     ,   x     )       /     G   ^     (       y   +   1     ,   x     )         +                              B   ^     (     y   ,     x   +   1       )       /     G   ^     (     y   ,     x   +   1       )         }     /   4                       
 
     [0161] This second step interpolates the blue plane portion defined by the first step to the pixels where G(y,x) is defined. Again, this interpolation essentially averages the blue values at four neighboring pixels of (y,x) with the values normalized at each location by the corresponding green values.  
     [0162] The final color image is defined by the three interpolated color planes: G{circumflex over ( )}(y,x), R{circumflex over ( )}(y,x), and B{circumflex over ( )}(y,x). The particular interpolation used for G{circumflex over ( )}(y,x) will be reflected in the normalizations for the two-step interpolations used for R{circumflex over ( )}(y,x) and B{circumflex over ( )}(y,x).  
     [0163] 7. CFA Interpolation with Edge Preservation  
     [0164] Alternative CFA interpolation preferred embodiments interpolate Bayer pattern greens by a (small) FIR filter plus preserve edges by a comparison of an interpolated pixel green value with the nearest-neighbor pixel green values and a replacement of the interpolated value with a neighbor value if the interpolated value is out of range. FIG. 29 illustrates the green interpolation. After this green interpolation, interpolate the red and blue planes.  
     [0165] In particular, first at each pixel (y,x) apply the following 5×5 FIR filter to G(y,x) defined on the pixels (y,x) where x+y is odd to yield G1 (y,x) defined for all (y,x):  
         1   /   200                              0         -   11         0         -   11         0             -   11         0       72       0         -   11             0       72       200       72       0             -   11         0       72       0         -   11             0         -   11         0         -   11         0                            
 
     [0166] The 200 center entry just implies for (y,x) where G(y,x) is defined in the CFA, G1(y,x)=G(y,x). Note that green values are in the range of 0-255, and negative values are truncated to 0. Of course, other FIR filters could be used, but this one is simple and effective.  
     [0167] Next, for the (y,x) where G1(y,x) is interpolated, consider the four nearest neighbors&#39; values G(y±1,x), G(y,x±1) and discard the largest and smallest values. Let A and B be the remaining two nearest-neighbor values with B greater than or equal to A. Then define the final interpolated green value G{circumflex over ( )}(y,x) as follows:  
         G   ^     (     y   ,   x     )       =     {         A           if                   G1        (     y   ,   x     )         &lt;   A               G1        (     y   ,   x     )               if                 A     ≤     G1        (     y   ,   x     )       ≤   B             B           if                 B     &lt;     G1        (     y   ,   x     )                               
 
     [0168] This clamps the interpolated value to midrange of the neighboring pixel values and prevents a single beyond-the-edge nearest-neighbor pixel from diluting the interpolated pixel value. FIG. 29 shows the overall green interpolation.  
     [0169] Complete the image by red and blue interpolations. The red and blue interpolations may each be a single step interpolation, or each be a two-step interpolation as described in the foregoing section which uses the edge-preserved green values, or each be some other type of interpolation.  
     [0170] 8. CFA Interpolation Plus Noise Filtering  
     [0171] Preferred embodiments save on line memory required for CFA interpolation followed by lowpass filtering to limit noise with an integrated approach. In particular, CFA interpolation typically contains a horizontal interpolation block and a vertical interpolation block with line memories in between as illustrated in FIG. 30. The horizontal interpolation block has an input of a row of CFA signals, two toggle switches, two zero insertion subblocks, two three-tap FIR filters (coefficients 0.5, 1.0, 0.5), and two outputs: one output for each color. Each of the FIR filters just reproduces the input color values and puts the average of successive input color values in place of the inserted zeros. The zero-insertion and toggle timing of two subblocks alternate with each other. The block diagram of the horizontal interpolation block is shown in FIG. 31 with a row of raw data R/G/R/G/R . . . ; in this block row-interpolated Red and Green signals are output. In case the row of raw data input is B/G/B/G/B . . . interpolated Blue and Green signals are output.  
     [0172] A line (row) memory delays the data by one CFA line (row) period in order to interpolate the data in the vertical interpolation block. FIG. 32 shows the four line memories and the input/output data of the memories. In the case of an input row of R/G/R/G/ . . . raw data with m indicating the (even) row number and n the column number which increments as the row data enters, the input and output data are:  
     [0173] Input_A=R(m,n)  
     [0174] Output_A1=Input_A=R(m,n)  
     [0175] Output_A2=G(m−1,n) which was the interpolated green from the previous row of raw data, a G/B/G/B . . . row  
     [0176] Output_A3=R(m−2,n) which was the interpolated red from the second previous row of raw data, a R/G/R/G/ . . . row  
     [0177] Input_B=G(m,n)  
     [0178] Output_B1=Input_B=G(m,n)  
     [0179] Output_B2=B(m−1,n) which was the interpolated blue from the previous row of raw data, a G/B/G/B/ . . . row  
     [0180] Output_B3=G(m−2,n) which was the interpolated green from the second previous row of raw data, a R/G/R/G/ . . . row This provides the two rows of red, R(m,n) and R(m−2,n), for vertical interpolation to create the m−1 row of red and also provides the green rows G(m,n), G(m−1,n), and G(m−2,n) which do not need vertical interpolation.  
     [0181] The next input row (row m+1) of G/B,!G/B/ . . . raw data leads to the following input and output data:  
     [0182] Input_A=G(m+1,n)  
     [0183] Output_A1=Input_A=G(m+1,n)  
     [0184] Output_A2 R(m,n) which was the interpolated red from the previous row of raw data, a R/G/R/G/ . . . row  
     [0185] Output_A3=G(m−1,n) which was the interpolated green from the second previous row of raw data, a G/B/G/B/ . . . row  
     [0186] Input_B=B(m+1,n)  
     [0187] Output_B1=Input_B=B(m+1,n)  
     [0188] Output_B2=G(m,n) which was the interpolated green from the previous row of raw data, a R/G/R/G/ . . . row  
     [0189] Output_B3=B(m−1,n) which was the interpolated blue from the second previous row of raw data, a G/B/G/B/ . . . row  
     [0190] This provides the two rows of blue, B(m+1,n) and B(m−1,n), for vertical interpolation to define the m row blue and also provides the green rows G(m+1,n), G(m,n), and G(m−1,n) which do not need vertical interpolation.  
     [0191]FIG. 33 shows the combinations for vertical interpolations. In particular, for row m output (row m+1 input) the combinations are (FIG. 33 b ):  
     [0192] green is G(m,n)  
     [0193] red is R(m,n)  
     [0194] blue is (B(m−1,n)+B(m+1,n))/2  
     [0195] And for row m−1 output (row m input) the combinations are (FIG. 33 a ):  
     [0196] green is G(m−1,n)  
     [0197] red is (R(m,n)+R(m−2,n))/2  
     [0198] blue is B(m−1,n)  
     [0199] As FIG. 33 illustrates, a vertical low-pass noise filter can be applied directly to the three green outputs (G(m−2,n), G(m−1,n), and G(m,n) for row m input and G(m−1,n), G(m,n), and G(m+1,n) for row m+1 input), but red and blue cannot be vertically filtered because the four line memories of FIG. 32 do not output enough lines (rows). Rather, eight line memories are needed as illustrated in FIG. 34.  
     [0200]FIGS. 35 a - 35   b  illustrate the preferred embodiment combination vertical interpolation and low-pass noise filtering including green vertical noise reduction filter block A, green-noise block B, blue/red green-noise difference block C, and red/blue green-noise sum block D. The six inputs for the preferred embodiments of FIGS. 35 a - 35   b  are the outputs of the horizontal interpolations and four line memories of FIGS.  30 - 32  and thus the same as the inputs to the known vertical interpolation filter of FIG. 34.  
     [0201] For an implementation of this interpolation plus noise filtering on a programmable processor the eight line memories in FIG. 34 would take up twice as much processor memory space as the four line memories of FIGS.  30 - 32 , and this can be significant memory space. For a large CFA such as a 2 megapixel (1920 by 1080 pixels) CCD, a line memory would be 1-2 kbytes, so the difference would be 4-8 kbytes of processor memory.  
     [0202] In more detail, FIG. 35 a  illustrates the noise reduction and vertical interpolation for the case of input row m with m an even integer (raw CFA data R/G/R/G/ . . . ) into the horizontal interpolator plus four line memories of FIG. 32: the six (horizontally interpolated) inputs at the lefthand edge of FIG. 35 a  are R(m,n), G(m−1,n), R(m−2,n), G(m,n), B(m−1,n), and G(m−2,n) (i.e., the outputs in FIG. 32); and the output will be noise-reduced colors for row m−1: R″(m−1,n), G″(m−1,n), and B″(m−1,n). First, the vertical interpolation (lefthand portion of FIG. 35 a ) averages R(m,n) and R(m−2,n) to create R(m−1,n); G(m−1,n) and B(m−1,n) already exist as inputs.  
     [0203] Then the noise reduction filter (block A in the righthand portion of FIG. 35 a ) creates and outputs the vertically low-pass filtered green G″(m−1,n) as:  
       G ″( m− 1 ,n )=[ G ( m,n )+2* G ( m− 1 ,n )+ G ( m− 2 ,n )]/4  
     [0204] Next, block B creates Delta_G as the difference between G and G″; that is, Delta_G is the vertical high-frequency part of G:  
       Delta   —   G ( m− 1 ,n )= G ( m− 1 ,n )− G ″( m− 1 ,n )  
     [0205] Because G is sampled twice as frequently as B and R in the Bayer CFA, direct high-frequency estimation of G will likely be better than that of B and R, and thus the preferred embodiment uses Delta_G to subtract for noise reduction. Note that the difference between the vertical average [G(m+1,n)−G(m−1,n)]/2 and G″(m,n) equals−Delta_G(m,n), so for R and B which are to be vertically interpolated (averaged) plus low-pass filtered, the high-frequency estimate provided by G which is to be subtracted from R and B will have opposite sign.  
     [0206] Thus block C subtracts Delta_G from B to create B″ for row m−1 because B is not vertically interpolated for m−1:  
       B ″( m− 1 ,n )= B ( m− 1 ,n )− Delta   —   G ( m− 1 ,n )  
     [0207] Essentially, the vertical high-frequency part of G is used as an estimate for the vertical high-frequency part of B, and no direct vertical low-pass filtering of B is applied.  
     [0208] Then block D adds Delta_G to R to create R″ for row m−1 because R was vertically interpolated:  
       R ″( m− 1 ,n )= R ( m− 1 ,n )+ Delta   —   G ( m− 1 ,n )  
     [0209] Again, the vertical high-frequency part of G is used in lieu of the high-frequency part of R, and because an vertical averaging creates R(m−1,n), the opposite sign of Delta_G is used to subtract the high-frequency estimate.  
     [0210] Thus the noise-reduced filtered three color output row m−1 are the foregoing G″(m−1,n), R″(m−1,n), and B″(m−1,n).  
     [0211] Similarly, for output row m from input row m+1 (again with m an even integer) and raw CFA data G/B/G/B/ . . . the six (horizontally interpolated) inputs are G(m+1,n), R(m,n), G(m−1,n), B(m+1,n), G(m,n), and B(m−1,n), and the output will be noise-reduced colors for row m: R″(m,n), G″(m,n), and B″(m,n). The vertical interpolation (lefthand portion of FIG. 35 b ) averages B(m+1,n) and B(m−1,n) to create B(m,n); G(m,n) and R(m,n) already exist as inputs. Then the noise reduction filter (righthand portion of FIG. 35 b ) block A again creates vertically low-pass filtered green G″(m,n) as:  
       G ″( m,n )={ G ( m+ 1 ,n )+2* G ( m,n )+ G ( m− 1 ,n )}/4  
     [0212] Next, block B again creates the vertical high-frequency portion of G, called Delta_G, as the difference between G and G″:  
       Delta   —   G ( m,n )= G ( m,n )− G ″( m,n )  
     [0213] Then block C again subtracts Delta_G but from R (rather than B as for row m−1 outputs) to create R″:  
       R ″( m,n )= R ( m,n )− Delta   —   G ( m,n )  
     [0214] Thus the high-frequency part of G is again used as an estimate for the noisy part of R, and no direct noise filtering of R is applied, but for row m the Delta_G is subtracted rather than added as for row m−1. Indeed, for R even rows have Delta_G subtracted and odd rows have Delta_G added because the odd rows have R defined as a vertical average.  
     [0215] Lastly, block D adds Delta-G to B to create B″:  
       B ″( m,n )= B ( m,n )+ Delta   —   G ( m,n )  
     [0216] Thus as with R, the Delta_G vertical high-frequency estimate is row-by-row alternately added to and subtracted from B instead of a direct vertical low-pass filtering of B. Note that for a given row the Delta_G terms for R and B have opposite signs because one of R and B will be an average of preceding and succeeding rows.  
     [0217] In short, the preferred embodiments are able to emulate the CFA horizontal interpolation, vertical interpolation, and low-pass filtering with only four line memories by using a high-frequency estimate based on G.  
     [0218]FIGS. 36 a - 36   b  and  37   a - 37   b  illustrate an alternative embodiment in which the vertical low-pass filtering of G differs from the ¼, ½, ¼ weighting of the preferred embodiments of FIGS. 35 a - 35   b.    
     [0219] 9. CFA Interpolation for Complementary Color CCD  
     [0220] Preferred embodiment CFA interpolations for a complementary color pattern CFA (illustrated in FIG. 7 b ) combine a simple interpolation followed by an image quality enhancement by detection and adjustment for color imbalance. In particular, presume initial interpolation as defined at each pixel all four complementary color values, and denote the color values as Ye (yellow), Cy (cyan), Mg (magneta), and G (green).  
     [0221] First, at each pixel compute an imbalance factor p:  
     μ= Ye+Cy− 2* G−Mg    
     [0222] This imbalance factor represents the difference between ideal and actual pixel color values. Indeed, the definitions of the complementary color values in terms of red value (R), green value (G), and blue value (B) are Ye=R+G, Cy=G+B, and Mg=B+G. Hence, the following relation always holds for a pixel&#39;s color values:  
       Ye+Cy= 2* G+Mg    
     [0223] Thus the imbalance factor t ideally vanishes. When an edge is near a pixel, imbalance can arise due to the spatial difference of each of the four color samples in the CFA. The preferred embodiments detect the imbalance and adjust by modifying each color value:  
       Ye′=Ye−μ/ 4  
       Cy′=Cy−μ/ 4  
       Mg′=Mg+μ/ 4  
       G′=G+μ/ 8  
     [0224] Then these modified complementary colors are used to form the final image.  
     [0225]FIG. 38 illustrates the overall flow for the enhancement using the imbalance factor. Of course, scale factors other than −¼, −¼, ¼, and ⅛ could be applied to the imbalance factor provided that Ye′+Cy′=2*G′+Mg′.  
     [0226] 10. White Balance  
     [0227] The term “white balancing” is typically used to describe algorithms, which correct the white point of the camera with respect to the light source under which the camera currently operates. Since the estimation of the true light spectrum is very difficult, the aim of most approaches is to correct the output of the red and blue channel (assuming CCDs based on the RGB color filters), such that for a gray object the pixel intensities for all color channels are almost identical. The most common technique basically calculates the average energy or simply the mean for each channel. The calculation of averages may be carried out in N local windows W j , j=1, 2, . . . , N, as for red:  
       R   j =Σ k∈Wj   r ( k )  
     [0228] with r(k) denoting the digital signal for the red channel. Similar averages B j  and G j  are calculated for the blue and green color channels. The imbalance between the channels, given by the green-to-red and green-to-blue ratios  
       WBR=Σ   j   G   j /Σ j   R   j    
       WBB=Σ   j   G   j /Σ j   B   j    
     [0229] are used as correction multiplier for the red and blue channels, respectively  
       r ′( k )= WBR r ( k )  
       b ′( k )= WBB b ( k )  
     [0230] There exist many different flavors of this approach, which all calculate intensity-independent multiplication factors WBR and WBB.  
     [0231] This approach works only if several assumptions are valid. First, it is assumed that the sensor responses are well aligned over the input intensity range; in other words, the green response curve equals the red (blue) response curve multiplied by a factor. Looking at sensor (CCD) characteristics indicates that this assumption does not hold. For high light intensities, the sensor saturates; while at very low light intensities, the sensor response (especially for the blue channel) is very small. Furthermore, non-linearities of the sensor, as well as imbalances of the color channels related to the sensor response and the light source, are handled simultaneously. Resulting artifacts include magenta colors in very bright areas, where the “color” should turn white, or wrong colors in dark areas.  
     [0232] The pixel intensity at the sensor output, e.g. for the red color channel, can be modeled as  
       r ( k )=∫ l (λ)β( k ,λ) f   R (λ)α( l ,λ) dλ   
     [0233] where λ denotes the wavelength, l(λ) the spectrum of the light source, β(x,λ) the reflectance of the object under observation, f R (λ) the spectral sensitivity of the red color filter covering the CCD pixels, and α(l,λ) the intensity- and wavelength-dependent efficiency of the CCD in converting photons into electrons.  
     [0234] Regarding only the spectral response curves of the color filters f R (λ) (and also f G (λ) and f B (λ)) of a typical CCD sensor, the output signals differ:  
       WBR=∫f   G (λ) dλ/∫f   R (λ) dλ= 1.09  
       WBB=∫f   G (λ) dλ/∫f   B (λ) dλ= 1.34  
     [0235] The values are obtained using the response of a typical CCD and assuming perfect white light source (the spectrum l(λ) is flat), a perfectly white object (the spectrum of the reflected light is identical to the spectrum of the illuminating light which means β(k,λ)=1), and neglecting α(l,λ) (no wavelength dependent quantum efficiency). Especially the blue channel shows a smaller response than green or red at the same intensity. The non-linear quantum efficiency of the sensor is another effect. A typical s-shaped sensor response over the input intensity is shown in FIG. 39 a . Furthermore, the sensor response in each channel depends on spectrum of the light source.  
     [0236] Thus, preferred embodiment white balancing takes into account the misalignment as well as the non-linearity. Typical light sources are not flat over the visible spectrum but tend to have a higher energy in certain spectral bands. This effect influences the observed sensor response; ideally it should be corrected by white point compensation, which may be based on a correction matrix. An independent balancing of the channels cannot handle this effect as previously outlined. For ease of mathematical description, approximate the s-shaped response curve in FIG. 39 a  by piecewise linear segments. Three segments separate the light conditions into three categories: very low intensity, normal intensity, and very bright light. FIG. 39 b  shows the effect of applying a single multiplier. With respect to the green signal, the amplification of the blue signal is too small in low light conditions, whereas in very bright conditions the multiplier is too large. Reducing the factor leaves an offset between the components, visible as wrong colors. Therefore, the correction terms for aligning all three response curves must look different and reflect the sensor characteristics.  
     [0237] The preferred embodiment white balancing splits into two separate schemes, one accounts for imager dependent adjustments, while the other one is related to light sources.  
     [0238] Without any restrictions on generality, the s-shape response curve is approximated in the following by three piecewise linear segments. More segments increase the accuracy but do not change the basic concept. For the first region (very low intensity) and the blue channel, the model reads with s the response and x the input intensity:  
     
       s 
       B,1 
       =a 
       B,1 
       x  
     
     [0239] Modeling the second region requires a multiplier and an offset  
     
       s 
       B,2 
       =a 
       B,2 
       x+b 
       B,2  
     
     [0240] The offset term is determined by the constraint that the response curve needs to be contiguous at the transition point x 1  from region 1 to region 2:  
       s   B,1 ( x   1 )= s   B,2 ( x   1 )  
     so  b   B,2 =( a   B,1   −a   B,2 ) x   1    
     [0241] The parameters for the linear model of region 3  
     
       s 
       B,3 
       =a 
       B,3 
       x+b 
       B,3  
     
     [0242] are completely determined because the maximum output has to be identical to the maximum input x max , and the the response curve needs to be contiguous at the joint point x 2 :  
     
       x 
       max 
       =a 
       B,3 
       x 
       max 
       +b 
       B,3  
     
       s   B,2 ( x   2 )= s   B,3 ( x   2 )  
       a   B,3 =( s   B,2 ( x   2 )− x   max )/( x   2   −x   max )  
       b   B,3 =(1 −a   B,3 ) x   max    
     [0243] Thus the parameters to specify the approximation of the response curve for each color component are a 1 , a 2 , x 1 , and x 2 , x max  is not a free parameter, because it is specified by the bit resolution of the input signal.  
     [0244] The preferred embodiment white balancing now applies different multipliers for each region. For continuous transition from one region to the next, an additional offset is required. Although the number of regions is arbitrary, without loss of generality only three regions are considered in the following equations. The correction term for blue with respect to green for region 1 has to be:  
     
       WBB 
       1 
       =a 
       G,1 
       /a 
       B,1 
       ≈G 
       1 
       /B 
       1  
     
     [0245] where window 1 (for G 1  and B 1 ) has pixels with intensities in region 1.  
     [0246] Thus, an input intensity value lying in region 1 gets the corrected output  
       b ′( k )= WBB   1   b ( k )  
     [0247] Based on the balancing multiplier for region 2  
     
       WBB 
       2 
       =a 
       G,2 
       /a 
       B,2 
       ≈G 
       2 
       /B 
       2  
     
     [0248] the white balancing must consider an additional offset for values in region 2  
       b ′( k )= WBB   2   b ( k )+ WBOB   2    
     [0249] with  
       WBOB   2 =( WBB   1   −WBB   2 ) x   1    
     [0250] For the third region the calculation is basically the same, except that no explicit WBB 3  can be specified, but the amplification is determined by the maximum value x max .  
       b ′( k )= WBB   3   b ( k )+ WBOB   3    
     [0251] with  
       WBB   3 =( x   max −( WBB   2   x   2   +WBOB   2 ))/( x   max   −x   2 )  
       WBOB   3 =(1 −a   B,3 ) x   max    
     [0252] For an implementation, the system must determine appropriate white balancing multipliers WBB i  for N−1 regions. Based on these values, the remaining offset values WBOB and the multiplier for the last regions are calculated. The locations of the transition points are specified a priori. The white balancing itself selects the region based on the intensity value of the input pixel and applies the appropriate gain and offset to that value  
           b   ′          (   k   )       =     {             WBB   1   *          b        (   k   )                 b        (   k   )       ≤     x   1                     WBB   2   *          b        (   k   )         +     WBOB   2               x   1     &lt;     b        (   k   )       ≤     x   2                     WBB   3   *          b        (   k   )         +     WBOB   3               x   2     &lt;     b        (   k   )                               
 
     [0253] Plus a similar multiplier for the red channel.  
     [0254] The total dynamic range of the CCD output signal is independent of aperture, and shutter, since they affect the number of photons captured in the CCD. An analog gain however, or any digital gain prior to processing shifts the signal and should be avoided. In case a gain (digital) (x needs to be applied, this gain can be included into the white balancing method. A gain maps the maximum input value x max  to the output value α*x max    
     [0255] The scaled response curves behave identical to the non-scaled one, meaning that the scaled signal saturates at α*x max . Substituting  
     
       WBB 
       1 
       :=α*WBB 
       1  
     
     
       WBB 
       2 
       :=α*WBB 
       2  
     
     [0256] In that way the equation in the previous section remain unchanged, except  
       WBOB   3 =(α− a   B,3 ) x   max    
     [0257] After linearization the signal can undergo an adjustment reflecting the light source. This is also known as white point adjustment. Here the input signal is transformed such that it looks like as if it has been captured under a different light source. For example, an image has been captured in bright sunlight (D65), but the color characteristics should be as if it has been captured under indoor conditions (D 50  tungsten).  
     [ R,G,B]D   65   T   =l   D65   T   *β*[f   R   ,f   G   ,f   B ] T    
     [ R,G,B]D   50   T   =l   D50   T   *β*[f   R   ,f   G   ,f   B ] T    
     [0258] Here, l Dxx  denotes a vector sampling the light spectrum, β is a diagonal matrix describing the reflectance of the objects, and f R , f G , and f B  denote the spectral response of the CCD light filters. Based on these equations a 3×3 transformation matrix can be calculated relating the signal under D65 to D50:  
     [ R,G,B]D   50   T   =l   D50   T   *l   D65   −T   *[R,G,B]D   65   T    
     [0259] The 3×3 transformation matrix  
     
       M 
       D 
       =l 
       D50 
       T 
       *l 
       D65 
       −T  
     
     [0260] can be calculated offline.  
     [0261] In real systems it is almost impossible to determine averages for the different response regions. Therefore a simple solution is to calculate overall values as in the foregoing ratio of integrals, and modify them with fixed values based on predetermined sensor measurements  
       WBB   1 =α 1   *WBB    
       WBB   2 =α 2   *WBB    
     [0262] And similarly for WBR.  
     [0263] The transition points can be fixed in advance, too. There is just one exception for the transition point x 2 . In rare situations the WBR-value may be so large that it exceeds the maximum output value at the transition point x 2 . In that situation, either the WBR needs to be decreased or the transition point is reduced. The diagram in FIG. 40 shows an example of the effectiveness of this technique. The red components is adjusted with respect to the green component. Using a single multiplier exceeds the green signal in bright areas, and is less effective in low light areas, whereas the segmented white balancing matches the green curve for all intensities.  
     [0264] 11. Resizing Preferred Embodiments  
     [0265] Frequently images captured in one size (e.g., 320×240 pixels) have to be converted to another size (e.g., about 288×216) to match various storage or input/output formats. In general this requires a fractional up-sampling or down-sampling by a rational factor, N/M; for example, a resizing from 320×240 to 288×216 would be a 9/10 resizing. Theoretically, resizing amounts to cascaded interpolation by N, anti-aliasing filter, and decimation by M. In practice the resizing may be achieved with an M-phase, K-tap filtering plus selection of N outputs per M inputs.  
     [0266] For example, preliminarily consider a resizing by a ratio of 63/64 using a 3-tap filter as illustrated in FIG. 41 a  in which the top horizontal line represents pixel inputs and the horizontal length-three braces represent the 3-tap filter kernel applied to the indicated three inputs and producing the indicated outputs. Indeed, presume the filter kernel is a continuous function f(t) with support of length 3-1/63 so that at most three inputs can be involved; see FIG. 41 b . Note the slight shifting to the right of successive braces in FIG. 41 a : this represents the resizing from 64 inputs down to 63 outputs because the center of the filter kernel (and thus the non-rounded-off output position) must increment 1+1/63 (=64/63) pixel positions for each output in order for the 63 outputs to match the 64 inputs. Output[0] (represented by the farthest left brace in FIG. 41 a ) is centered at the position of input, and the non-rounded-off output position j, denoted outp_pos[j], thus equals 1+j*64/63. The filter kernel is represented as a symmetrical continuous function f(t) centered at time 0. Output[0] for example, needs three kernel values: f(−1), f(0), and f(1). Each output point is computed as the inner product of three kernel coefficient values with three input pixel values. The center input point for the output[j] is positioned at round (outp_pos[j]) where round( ) is the round off function. The other two input points are offset from this center point by ±1. The center filter kernel coefficient value is f(round(outp_pos[j])−outp_pos[j]) and the other are f( ) at the ±1 offsets of this center value point. Thus the following table shows the output position, coefficient kernel values, and input points needed for each output:  
                                                   center coeff               outp_pos   position   input points                                                            0   1   0   0, 1, 2           1    2 1/63   −1/63     1, 2, 3           2    3 2/63   −2/63     2, 3, 4           .   .   .   .           .   .   .   .           .   .   .   .           31   32 31/63    −31/63         31, 32, 33           32   33 32/63   31/63    33, 34, 35           33   34 33/63   30/63    34, 35, 36           .   .   .   .           .   .   .   .           .   .   .   .           61   62 61/63   2/63   62, 63, 64           62   63 62/63   1/63   63, 64, 65           63   65   0   64, 65, 66           .   .   .   .           .   .   .   .           .   .   .   .                      
 
     [0267] The table shows the desired coefficient position as well as the inputs involved in each output. Note the j=63 case is similar to the j=0 case in that the kernel center aligns with the input, but with the output position and input indices shifted by 64. Notice that at j= 32  there is a change in the input pattern: for j≦31, output[j] uses input j, j+1, and j+2; whereas for j≧32, output[j] uses inputs j+1, j+2, and j+3.  
     [0268] The preferred embodiments partition the filtering computations for resizing a two-dimensional array (image) between iMX  124  and DSP  122  and limit memory use as follows. First iMX  124  performs the 3-tap row filtering with 64 banks of coefficients and then 3-tap column filtering with 64 banks of coefficients. First consider the row filtering. 3-tap row filtering on iMX  124  has the input/output  
                                                   iMX output j   input points                                                    0   0, 1, 2           1   1, 2, 3           2   2, 3, 4           .   .           .   .           .   .           31   31, 32, 33           32   32, 33, 34           33   33, 34, 35           .   .           .   .           .   .           61   61, 62, 63           62   62, 63, 64           63   63, 64, 65           64   64, 65, 66           .   .           .   .           .   .                      
 
     [0269] Comparing this table with the prior 63/64 resizing table shows that the only difference is the iMX produces one extra point, namely, IPP_output[32]. Thus the preferred embodiments produce the 64 output points with iMX  124 , and then use DSP  122  to pick the 63 valid points:  
     [0270] output[j]=IPP_output[j] for j=1,2, . . . 31  
     [0271] IPP_outpt[j+1] for j=32,33, . . . , 62  
     [0272] In general, N/M resizing when N/M is less than 1 involves deleting M-N outputs of every M outputs. Thus the preferred embodiments generally perform the filter operations on the M input points in an accelerator such as the iMX and then use a processor such as the DSP to discard the unneeded outputs. (iMX can also handle larger-than-unity resizing up to N/M=3.)  
     [0273] iMX can produce 8 outputs of 3-tap row filter in 3 cycles. Basically, 8 adjacent outputs are computed in parallel using the 8 MAC units. At time 0, pull out input points 0,1,2,3, . . . 7, multiply with appropriate coefficients (each can be different), and accumulate into 8 accumulators. At time 1 pull out input points 1,2, . . . 8, do the same, and at time 2, pull out input points 2,3, . . . 9, accumulate the products, and write out 8 outputs, j=0,1, . . . 7. Next, shift over 8 input points to compute j=8,9, . . . 15.  
     [0274] For the vertical direction, iMX computes 8 outputs in parallel as well. These are 8 horizontally adjacent output points, and every fetch of input array also bundles 8 horizontally adjacent output points. Therefore, all 8 MAC units share the same coefficient values for each cycle. For vertical direction there is less data reuse in iMX, so input/output memory conflicts slow down the computation to 4 cycles/8 outputs. Total filtering time is 7 cycles/8 outputs, or 7/8 cycle per output. Input data is of size 320×240×3. Thus, the filtering of iMX takes 320×240×3.7/8 201,600 cycles, or 1.7 msec with iMX running at 120 MHz.  
     [0275] After filtering, DSP picks correct outputs. Basically, one row out of every 64 rows and one column out of every 64 columns should be discarded. A DSP assembly loop moves the valid iMX output points to a separate output area. iMX and DSP may run in parallel if there is sufficient local memory for both. An entire input image likely is too large to fit into local memory; even the natural choice, 63×63 output points, may be too large. In such a case partition the image, such as 63 wide×16 tall, and deal with extra bookkeeping in the vertical direction. With just 3×64=192 coefficients, it would be economical to pre-compute and store them. DSP should keep track of the phase of each processing block, and point iMX to the correct starting address of coefficients. If the colors are interleaved, this allows interleaved filtering as well. iMX deals with strides in getting input points. The following table shows interleaved 3-tap filtering.  
                                                   j   input points                          0   0, 3, 6           1   1, 4, 7           2   2, 5, 8           .   .           .   .           .   .                      
 
     [0276] However, interleaving consumes three times more memory for the same output block size for each color. Thus it si possible to partition the task into smaller size, such as 63×5 on each color plane, and eal with extra overhead in the vertical direction. If the color format is not 4;4:4 (say, 4:2:2), and input is color-interleaved, the DSP will need to spend some additional time separating color planes.  
     [0277] Performing resizing totally in DSP  122  is time-consuming if implemented with straightforward fractional addressing. The preferred embodiments streamline the computation by requiring filter coefficients to be reordered and padded with dummy words. iMX  124  performs the main processing concurrently with DSP  122  computing the coefficients. This efficiently realizes high throughput resizing.  
     [0278] In more detail, the preferred embodiments perform an NIM resizing of an image by using iMX  124  to perform M-phase, K-tap filtering (which produces redundant output points) and DSP  122  to select the correct output points. Further, DSP  122  computes needed coefficients from a fewer-subsample coefficient template to reduce memory usage to 8*K; otherwise memory usage up to 2*M*K coefficient words would be needed. DSP  122  can compute the rounded position for the coefficients, and build up the coefficient memory for iMX  124 .  
     [0279] For processing wide and short blocks of pixels (i.e., 16×64) the horizontal direction requires more computation in that horizontal coefficients are updated more often than vertical coefficients. However, the coefficients constructed by DSP  122  can be reused many times within the short block, so the load on DSP  122  should not be excessive.  
     [0280] In particular, preferred embodiments proceed with the following steps which are illustrated in FIGS. 42 a - 42   e  for a 3-tap filter and a 10-to-9 resizing (e.g., resizing from 320×240 to 288×216 in 30 frames/sec)(presume 4:4:4 interleaved, for 4:2:2 or 4:1 ;1 do subsampling after resizing):  
     [0281] 1. select input/output pattern: every 10 inputs leads to 9 outputs as per FIG. 42 a.    
     [0282] 2. draw coefficient pattern for a processing unit, one color first. Arrows in FIG. 42 b  indicate which input points are used: connected arrows form the same output point, and gray (open head) arrows indicate zero coefficients. Thus three input points determine the first output point, only two input points determine each of the next eight output points, and then a tenth ignored output (no non-zero input points); and this repeats every ten. This pattern suggests use of a polyphase 3-tap filter, and drop the last output in every group of 10 outputs.  
     [0283] 3. consider interleaved input/output. See FIG. 42 c  which shows a set of three groups of ten input points interleaved so that the three input points determining the first output point from the original first group of ten input points are now at locations 1, 4, and 7; the three input points determining the first output point from the original second group of ten input points are now at locations 2, 5, and 8; and the three input points determining the first output point from the original third group of ten input points are now at locations 3, 6, and 9; and so forth. This interleave implies that sets of three adjacent output points use all different input points and do not require simultaneous memory accesses.  
     [0284] 4. Consider 8-way parallelism and iMX, add more dummy outputs if necessary. See FIG. 42 d  which shows the output points partitioned into four groups of 8 for parallel computations.  
     [0285] 5. Compute coefficients and order as grouped. iMX will process one group at a time, using coefficient order from left-to-right, then up-to-down, then next group. Coefficients need to be arranged to the same order. If the iMX coefficient memory and the flash memory can accommodate all these coefficients, these coefficients can be included in the DSP code as constant data, and this step is done once in the software development. If the iMX coefficient memory can hold these coefficients all the time, but these take up too much room in the flash memory, this step can be performed once during system initialization. Likely the SDRAM can hold all these coefficients, but iMX coefficient memory cannot hold them all the time. this step should be performed once in the system initialization, an the coefficient image should be stored in SDRAM. When needed, these coefficients are swapped in from the SDRAM. If it is not desirable to store all these coefficients at any time, especially when M is very large (100+), compute needed “window” of coefficients with DSP concurrently with iMX processing. Just make sure the iMX coefficient memory can hold the necessary coefficients for a computation block.  
     [0286] 6. Start computation on iMX. In this case, it takes about 12 cycles in the inner loop to produce the 27 valid output points. Each iMX command can produce a 2-D output block, so producing 16×27 output points will take about 10+16*12=202 cycles.  
     [0287] 7. When iMX is done, have DSP pick the correct output points. In this example, 276 points are picked out of every group of 32 output points. This task will be easier to code if the width of output matches or is a multiple of 3*M. DSP only has to touch each valid outaput once, so the loading of the DSP should not be significant.  
     [0288] In vertical resizing, iMX works in SIMD mode. Every group of 8 adjacent data input are processed in parallel. Coefficient are used one value per cycle, and this value should apply to all color components. Even if resizing factors are the same for horizontal and vertical, how iMX uses coefficients is different, so there needs to be a separate vertical resizing coefficient storage (which takes 1/3 of horizontal coefficients). See FIG. 42 e . Again, there is the option to keep all vertical coefficients in iMX, swap in and out, or have DSP compute on the fly. DSP may need to pick valid output rows after iMX completes processing.  
     [0289] 12. Tone-Scaling Preferred Embodiments  
     [0290] Tone-scaling operates on the dynamic range of the luminance signal (or the color signals) of an image to make details more clear. For example, a picture taken against the light or in a very bright environment typically has high brightness levels. Tone-scaling commonly relies on luminance (or color) histogram equalization as illustrated in block form by FIG. 43. Indeed, converter block  430  converts the input luminance levels (in the range 0 to 255 for 8-bit or 0 to 4095 for 12-bit) to output luminance levels in the same range using a look-up table. The look-up table consists of the pairs that are the input level and the corresponding output level with the output levels calculated in histogram equalization block  432  as follows. First, find the cumulative distribution function of the input luminance levels of the image to which the tone-scaling will apply; that is, find F(r) such that F(r)=(the number of pixels with level≦r)/(total number of pixels in the image). Next, create the look-up table function T(r) through multiplication of F(r) by the maximum pixel level and round-off to the nearest integer. Then the look-up table is just the pairs of levels (r,s) where s=T(r). FIG. 45 illustrates T(r) for an under-developed image (the majority of pixels have a low level as reflected by the large slope of T(r) for small r) in which fine details in dark parts are difficult to perceive. Also as FIG. 45 shows for this under-developed image, the tone-scaling converts the level r=500 to s=2000; and thus in the tone-scaled image the differences of the luminance levels will be emphasized for the low levels and de-emphasized for the high levels. Thus the tone-scaling enhances detail in dark portions.  
     [0291] However, the tone-scaled image may look unnatural in that the colors are too clear, as if the tone-scaled image were painted in oil paints. Thus this tone-scaling is sometimes too strong for consumer use because of the unnatural character even if the fine details are clearer; although other applications such as medical and night vision demand the fine detail despite unnaturalness.  
     [0292] The preferred embodiments provide tone-scaling by using a linear combination of the histogram equalization function T(r) and the original image level r. That is, for a parameter α with 0&lt;a &lt;1 define a tone-scaling function by  
       s =Round(α T ( r )+( 1−α)   r )  
     [0293] where T(r) is as previously described except that the round off to the nearest integer is not needed in the definition of T(r) because of the subsequent multiplication by U plus addition of (1−α)r and round off. FIG. 45 illustrates the preferred embodiment for α=0.3 between the curve s=T(r) and the identity line s=r.  
     [0294]FIG. 44 shows preferred embodiment tone-scaling in functional block form: again define a histogram equalization function T( ) for the luminance (or color) levels in block  442 , and then define the rounded-off linear combination with weight α of T( ) and the identity in block  444  to yield the final look-up table for the tone-scaling in converter  440 . When the weight α equals 0, then there is no tone-scaling and a natural look, but when the weight α equals 1, the tone-scaling is with T( ) and fine details are enhanced. The value of weight α can be selected according to the application. All of the computations are programmable.  
     [0295] Preferred embodiment hardware structures supporting the foregoing functions include the following.  
     [0296] 13. SDRAM Controller  
     [0297] SDRAM controller block  110  acts as the main interface between SDRAM  160  and all the function blocks such as processors (ARM  130 , DSP  122 ), CCD controller  102 , TV encoder  106 , preview engine  104 , etc. It supports up to 80 MHz SDRAM timing. It also provides low overhead for continuous data accesses. It also has the ability to prioritize the access units to support the real-time data stream of CCD data in and TV display data out. It also provides power down control for external SDRAM. DSP  122  can inhibit CKE signal of SDRAM  160  during no data access.  
     [0298] SDRAM controller block  110  supports 16/64/128/256 MB SDRAMs, 32-bit width or 2×16-bit width SDRAMs, maximum 80 MHz (e.g., 10-80 MHz) operation, availability of word, half-word, or byte access (ARM), commands: mode setting, power down and self refresh, programmable refresh interval, 2 or 3 CAS latency can be selectable, 2 Chip Select Output (maximum SDRAM size is 1G bit), authorizes and manages DMA transfers, manages the data flow between processors SDRAM, CCD data buffer to SDRAM, preview engine to SDRAM, burst compression to/from SDRAM, video encoder from SDRAM, OSD from SDRAM, ARM to/from SDRAM, DSP image buffer to/from SDRAM. FIG. 12 a  shows the data flow managed by the SDRAM controller. The signals and priorities are:  
                                   Signal Name   Signal Description                  Clk   SDRAM clock (10-80 MHz)       Req   Data read/write request signal       req_en   Request enable (acknowledge) signal from SDRAM Controller           When the peripheral modules require a data IN/OUT, the req signal shall be           asserted and when the req_en signal is asserted, the req signal shall be negated       Address   Start address of read or write           CCDC, PREVIEW, BURSTC, ENC, OSD, DSP: 22-bit width           ARM: 25-bit width       Odata   output data to SDRAM (32-bit)       Idata   Input data from SDRAM (32-bit)       Rw   Read or Write signal           0: Write / 1: Read       Dten   Data write enable signal for DSP IF       Ds   Bus Select (4-bit) for ARM IF                  
 
     [0299] The Priority list of access units is as follows,  
                                                   Priority   Access Unit                          1 (highest)   ENC out           2   CCD in           3   OSD out           4   PRVW in           5   BURST in           6   DSP I/O           7   ARM I/O                      
 
     [0300] 14 . Preview Engine Preferred Embodiments  
     [0301]FIG. 14 is a block diagram of first preferred embodiment preview engine  104  which provides image data with YCbCr in 4:2:2 format from CCD raw data from CCD-controller  102  and has the following main functions.  
     [0302] _Available for both RBG CCDs and complementary (YeCyMgG) CCDs (FIGS. 7 a - 7   b  show these CCD patterns)  
     [0303] _Digital gain adjustment  
     [0304] _White balance  
     [0305] _Vertical and horizontal noise filter  
     [0306] _RGB gain adjustment for complementary CCDs  
     [0307] _Independent gamma correction for RGB colors  
     [0308] _YCbCr-42:2 formatted data output  
     [0309] Sync module  1402  generates control signals for other modules such as a sync signal for a starting point of an image and an enable signal for down sampling. In this module, no image processing is executed. White balance module  1404  executes digital gain adjustment and white balance for CCD raw data. CFA interpolation module  1406  has many important sub-modules such as a horizontal noise filter, a horizontal interpolation, a vertical noise filter, a vertical interpolation, a down sampling, etc. This module outputs RGB formatted data irrespective of CCD mode (RGB CCD or complementary CCD). RGB gain modules  1408  for complementary CCD allow adjustment to white balance by RGB color format for complementary CCD. Gamma correction modules  1410  execute gamma correction with an approximated gamma curve having 4 linear segments. This module exists for each color to permit the independent adjustment to RGB. RGB2YCbCr conversion module  1412  converts RGB formatted data into YCbCr formatted data and adjusts offsets of Cb and Cr. 4:2:2 conversion module  1414  converts YCbCr-4:4:4 formatted data into 4:2:2 format and outputs them on a 32-bit data bus. SDRAM interface module  1416  communicates with SDRAM controller  110  (FIG. 1 b ) and requests it to store YCbCr-4:2:2 formatted image data.  
     [0310] The following describes the modules.  
     [0311] White balance module  1404  executes digital gain adjustment and white balance for CCD raw data. Digital gain adjusts for total brightness of the image and white balance adjusts the ratio of colors existing in a CFA pattern.  
     [0312]FIG. 8 is a block diagram of white balance module  1404 . There are two multipliers for the two gain adjustments and clip circuits to reduce the size of circuits. A gain value for digital gain named PVGAIN in this figure uses data in a PVGAIN register, and white balance is selected automatically by setting the CFA pattern register.  
     [0313] CFA interpolation module  1406  include both sub-modules for horizontal and vertical interpolation and for horizontal and vertical noise filtering, down sampling, color adjustment and complementary color to RGB color conversion. FIG. 10 a  is a block diagram of CFA interpolation module  1406 . Horizontal noise filter sub-module  1002  executes a three-tap low pass filter horizontal filter; see FIG. 10 b.    
     [0314] Horizontal interpolation filter sub-module  1004  prepares two types of filters and interpolates horizontally using one of them. The outputs signal “L” and “R” means a left data and a right data on the line. For example, a processed line starts the following CFA pattern, GBGBGBGBGB . . . , the output signal “L” is G and “R″ is B. Therefore, these two outputs change the colors each line. Horizontal down-sampling sub-module  1006  outputs only data on valid pixels based on register settings of horizontal decimation pattern. Vertical interpolation sub-module  1008  processes a three-tap vertical interpolation filter using two line-memories  1010  outside the preview engine module and outputs data of all colors existing in the CFA pattern. And this sub-module also executes a vertical noise filter. Color selection sub-module  1012  extracts data by each color in the CFA pattern and outputs RGB color formatted data in RGB CCD mode or complementary color formatted data in complementary CCD mode. In this figure, “g” signal is temporal data regarding G and used for recalculating R and B in the next color adjustment sub-module  1014 . The color formatted data is processed color adjustment in color adjustment sub-module  1014  and the processing is different depending on CCD mode. This image processing from vertical interpolation sub-module  1008  to color adjustment sub-module  1014  has a strong correlation depending on CCD mode and vertical interpolation mode. Therefore, the processing should be considered as a sequence of vertical interpolation processing as described below. Comp2RGB conversion sub-module  1016  converts complementary color format into RGB color format in complementary CCD mode. In RGB CCD mode, the data bypass this sub-module.  
     [0315] The following sections describe these sub-modules.  
     [0316] Horizontal noise filter  1002  executes three-tap horizontal low pass filter and can reduce random noise effectively. Actually, when the center of data is set to X 0 , the following calculation is executed depending on the CFA pattern and its processed line.  
         X   0     =     {             (       X     -   2       +     2        X   0       +     X   2       )     /   4           (     two                 colors                 in                 processed                 line     )                 (       X     -   1       +     2        X   0       +     X   1       )     /   4           (     one                 color                 in                 processed                 line     )                           
 
     [0317] An on/off switching of this filter can be controlled by a register setting.  
     [0318]FIG. 10 b  is a block diagram of horizontal noise filter sub-module  1002 . The two types of filter are implemented by using two adders and a switch named “three_taps_sw” in this figure. If there is one color in the processed line, the switch is set to on (High in the figure). This switch is automatically controlled depending on a register setting of the CFA pattern and a position of the line in the processed image. Before the output, noise-filtered data or bypassed data is selected by a register setting.  
     [0319] In horizontal interpolation sub-module  1004 , there are two modes of filtering and the data from horizontal noise filter  1002  is interpolated horizontally by either a two-tap or five-tap interpolation filter. The two-tap filter utilizes the average the two data at the adjacent pixels on the left and right to interpolate the center data. This mode is called “simple mode”. The five-tap horizontal interpolation filter utilizes the information of another color on the processed line so that a false color around an edge in processed image can be reduced effectively. This mode is called “normal mode”. These modes are selectable by a register setting. Actually, when the center of data is set to X 0 , the following calculation is executed depending upon the interpolation mode.  
         x   0     =     {             (       -     X     -   2         +     2        X     -   1         +     2        X   0       +     2        X   1       -     X   2       )     /   4           (     normal                 mode     )                 (       X   1     +     X   1       )     /   2           (     simple                 mode     )                           
 
     [0320]FIG. 10 c  shows an example of this horizontal interpolation processing in RGB Bayer CCD mode. In this figure, interpolated data is represented by small letters.  
     [0321]FIG. 10 d  is a block diagram of horizontal interpolation module  1004 . Two adders, one subtracter and a filter mode switch are implemented for executing one of these two types of filters. The filter mode switch is controlled by setting a register.  
     [0322] Vertical interpolation sub-module  1008  processes either a two-tap or three-tap vertical interpolation filter using two line-memories outside the preview engine module and outputs the information of all colors existing in the CFA pattern. And this sub-module also executes a vertical noise filter. An image processing in this module is a little complicated and the outputs from this sub-module is varied depending on a processed line, CCD mode, CFA pattern, filter mode and noise filter on/off. As explained in the following, the image processing from vertical interpolation sub-module  1008  to color adjustment sub-module  1014  has a strong correlation and this processing flow of them should be considered as a sequence of vertical interpolation processing. Therefore, this sequence of the vertical interpolation processing is explained first. The sequence may be called “vertical interpolation sequence”.  
     [0323] As with horizontal interpolation, vertical interpolation processing also has two types of interpolation mode, that is “simple mode” and “normal mode”. An interpolation filter in simple mode utilizes the average two data at the next pixels on the upper and lower to interpolate the center of data. In normal mode, the processing differs between RGB CCD mode and complementary CCD mode. The interpolation filter in normal mode in RGB CCD mode utilizes the data of one of the others color same as horizontal interpolation filter. Actually, when the data of a certain color to be interpolated is set to X (mainly R,B) and the data of a color utilized as a reference is set to Y (mainly G), the following calculation is executed depending on the interpolation mode through this vertical interpolation sequence and it is the output from color adjustment sub-module.  
         x   0     =     {               (       X     -   1       -     Y     -   1       +     X   1     -     Y   1       )     /   2     +     Y   0             (     normal                 mode     )                 (       X     -   1       +     X   1       )     /   2           (     simple                 mode     )                           
 
     [0324]FIG. 10 e  shows an example of this vertical interpolation sequence for the RGB Bayer CCD pattern.  
     [0325] In complementary CCD mode, normal mode means “simple interpolation with color adjustment”. That is, data of all colors which is processed by simple vertical interpolation is adjusted based on the formula in complementary color space. Actually, when the data of a certain color to be interpolated is set to X and the data of the others color is set to W, Y, and Z, the following calculations are executed in normal mode in complementary CCD mode.  
         x   0     =     {               (       X     -   1       +     X   1       )     /   2     ±     a                   (       w   0     ,     x   0     ,     y   0     ,     z   0       )               (     normal                 mode     )                 (       X     -   1       +     X   1       )     /   2           (     simple                 mode     )                           
 
     [0326] As to the calculation of a=a(w 0 , x 0 , y 0 , z 0 ), see below.  
     [0327] In this vertical interpolation sequence, main roles of vertical interpolation sub-module 1008 are to execute a part of vertical interpolation sequence and vertical noise filter. The part of vertical interpolation sequence means preparing data for normal vertical interpolation mode. As shown in FIGS. 10 e  and  10   b  (for RGB and complementary CCD patterns, respectively), in simple mode, an output data of this vertical interpolation sub-module bypasses color adjustment sub-module. Therefore, in simple mode, the output from this sub-module is used as the output of vertical interpolation sequence. In any case of interpolation mode, this sub-module calculates the following equation for vertical interpolation sequence.  
     x 0 =(X −1 −X 1 )/2  
     [0328] Vertical noise filter . . . which executes the following 3 taps vertical low pass filter is also processed in this sub-module depending on the CFA pattern.  
     x 0 =(X −1 −2 X   0   −X   1 )4  
     [0329] However, for this filtering, data of same color on processed 3 lines must be prepared. Therefore, a function of the vertical noise filter mainly executes only G in RGB Bayer CCD. FIG. 10 g  shows an example of the output of this vertical interpolation sub-module for a RGB Bayer CCD. When the vertical noise filter can be applied and it is set on, original data (R in this figure) is also adjusted in order to keep a correlation to the others color (G in this figure).  
     [0330]FIG. 10 h  is a block diagram of vertical interpolation sub-module  1008 . Six adders and two subtracters are implemented for executing vertical interpolation and noise filtering. Especially, a calculation process of L — 121 and R — 121 is so complicated that switching operation for L — 121 and R — 121 is not shown to simplify this figure.  
     [0331] Color selection sub-module  1012  arranges the inputs from vertical interpolation sub-modules in order of color format, that is R, G and B in RGB CCD mode or Ye, Cy, Mg, G in complementary CCD mode. This arrangement is executed automatically by setting register of the CFA pattern. FIG. 10 i  shows an example of this color selection processing in RGB Bayer CCD of FIG. 10 g . The outputs named “g” in this figure is a temporal data of G and is used for recalculation of R or B in RGB CCD mode in color adjustment sub-module.  
     [0332]FIG. 10 j  is a block diagram of color selection sub-module  1012 . Four color extractors switch and select independently correct colors from four inputs from vertical interpolation sub-module  1008 .  
     [0333] Color adjustment sub-module  1014  executes the rest of calculation for vertical interpolation sequence. In RGB CCD mode such as RGB Bayer CCD, R or B is recalculated using the temporal data of G. When data of R or B from color selection sub-module is set to X, the following calculation is executed in RGB CCD mode.  
     
       x=X−G 
       temp 
       +G  
     
     [0334] In the example of FIG. 10 i , when noise filter is off,  
       X =( b   02   −b   22 )/2  
       G   temp =( G   02   +G   22 )/2  
     
       G=g 
       12  
     
     [0335] Therefore,  
             x   =   B               =         (       b   02     -     b   22       )     /   2     -       (       G   02     +     G   22       )     /   2     +     g   12                   =         (       (       b   02     -     G   02       )     +     (       b   22     -     G   22       )       )     /   2     +     g   12                           
 
     [0336] This is the output B of the color adjustment module and also the output of vertical interpolation sequence. That is, vertical interpolation sequence in RGB CCD mode utilizes the average of differences between data of color to be interpolated and reference data of the others color.  
     [0337] In complementary CCD mode, color adjustment is processed to data of all colors from color selection sub-module. First, value a is calculated at each pixel based on a formula in complementary color space Ye+Cy=G+Mg.  
     
       a=G+Mg−Ye−Cy  
     
     [0338] That is, the value a can be considered as the amount of an error value of four colors. Therefore, in complementary CCD mode, to data of all colors, Ye, Cy, Mg and G, the following adjustment is processed to satisfy the above formula.  
       ye=Ye+a/ 4  
       cy=Cy+a/ 4  
       g=G−a/ 4  
       mg=Mg−a/ 4  
     [0339]FIG. 10 k  is a block diagram of color adjustment sub-module  1014 . Six adders and three subtracters are implemented for executing the two types of calculations described above. A switcher named CCDMOD in this figure selects correct outputs depending on CCD mode and is controlled by setting a register.  
     [0340] Comp2RGB conversion sub-modules  1016  converts complementary color formatted data to RGB formatted data in complementary CCD mode. Especially for G, data from color adjustment and data calculated by conversion formula can be blended by 5 types of blending ratio. Actually, the following calculation is executed based on the conversion formula:  
     
       R=Ye−Cy+Mg  
     
       G=rG   input +(1 −r )( Ye+Cy−Mg )( r= 0,1/4,2/4,3/4,1)  
     
       B=Mg−Ye+Cy  
     
     [0341] In RGB CCD mode, data from color adjustment sub-module bypass this sub-module.  
     [0342]FIG. 10 l  is a block diagram of comp2RGB conversion sub-module 1016 . Three adders, three subtractors, and two multipliers are implemented for executing the calculations above. A gain adjuster for G named “green_ratio” in this figure is adjustable by setting a register. In RGB CCD mode, a CCDMOD switcher selects off (high in this figure) for bypassing this module.  
     [0343] RGB gain for complementary CCD module allows adjustment of white balance by RGB color format even for complementary CCD module. This module is also available in RGB CCD mode.  
     [0344]FIG. 9 a  is a block diagram of complementary white balance module  1408 . One multiplier and clip circuit is implemented for this operation. Each gain for RGB is set by a register.  
     [0345] Gamma correction modules  1410  execute gamma correction for each color data in RGB color format. For this operation, prepare in advance three types of data for approximating the gamma curve by four linear segments. Those are area, offset and gain shown in FIG. 9 b . As shown in FIG. 14, this module exists for each color so that the independent adjustment to RGB may be made.  
     [0346]FIG. 9 c  is a block diagram of gamma correction module  1410 . Area detector selects correct gain and offset for input data based on area data. The data regarding gain, offset, and area are set in three registers.  
     [0347] RGB2YCbCr conversion module  1412  converts RGB formatted data to YCbCr formatted data and adjusts offsets to Cb and Cr based on the following matrix calculation.  
         [         Y           Cb           Cr         ]     =         [         COEF1       COEF2       COEF3           COEF4       COEF5       COEF6           COEF7       COEF8       COEF9         ]                [         R           G           B         ]     +     [         0           OFFSET_Cb           OFFSET_Cr         ]                     
 
     [0348] A register sets each coefficient in this matrix so that variable setting for this conversion is available.  
     [0349]FIG. 11 a  is a block diagram of this RGB2YCbCr conversion module  1412 . Nine multipliers and five adders are implemented for the foregoing matrix calculation. After multiplying RGB data with coefficients, the six least significant bits of each data from the multipliers is cut in order to reduce size of circuits. As to Cb and Cr, YCbCr conversion circuit follows additional circuit for offset adjustment. Clip circuits for Cb and Cr includes conversion circuits from two&#39;s complement to offset binary.  
     [0350] 15. Alternative Preview Engine Preferred Embodiments  
     [0351]FIG. 53 is a block diagram of another preview engine preferred embodiment. The preview engine is a hardware video processor module which generates image data with YCbCr-4:2:2 format from CCD raw data with an RGB color space or a complementary color space format. The followings are the main features.  
     [0352] Dual input ports (CCD controller and SDRAM controller)  
     [0353] Noise filter with noise coring  
     [0354] Digital gain  
     [0355] White balance  
     [0356] Two steps smoothing  
     [0357] Horizontal and vertical seamless down sampling (x1/64˜1)  
     [0358] Horizontal and vertical seamless zoom (x 1 ˜x4) (SDRAM input mode)  
     [0359] RGB2RGB blending matrix  
     [0360] Gamma correction  
     [0361] Chroma offset &amp; suppression  
     [0362] One shot preview  
     [0363] Support 16 bit data bus SDRAM  
     [0364] Resizing function of YCbCr4:2:2 image  
     [0365] As more fully described in the following, the preferred embodiment has various modules. The input interface module has two input ports which are connected to a CCD controller (CCDC) and an SDRAM controller (SDRAMC), and extract valid raw data from CCD raw data. A noise filter module reduces an impulse noise in the raw data using a noise coring technique. A white balance module has two gain adjusters, a digital gain adjuster and a white balance adjuster. In the digital gain adjuster, the raw data is multiplied by a fixed value of gain regardless of a color space of a CCD image sensor. In the white balance gain adjuster, the raw data is multiplied by an selected value of gain corresponding to the color space of the CCD image sensor. A CFA interpolation module has many sub-modules which process an interpolation, a smoothing and a resizing. The output from the CFA interpolation module is always formatted into RGB color space regardless of the type of the color space of the CCD. An RGB2RGB blending module prepares a general 3×3 square matrix and redefines the RGB data from the CFA interpolation module. It can be used as a function of a color correction. A gamma correction module performs a gamma correction independently to each color by approximating an ideal gamma curve with four piece wise linear lines. An RGB2YCbCr conversion module has a 3×3 square matrix circuit and converts the RGB color space of the image data into a YCbCr color space. In a 4:2:2 conversion module, the color data of Cb and Cr are down-sampled into 4:2:2 format. An output interface module has a data buffer for a burst transfer of an SDRAM and requests an arbiter module to store the processed image data. The arbiter module selects one of requests from the input interface of the SDRAMC and the output interface, and requests the SDRAMC to read a source data or to store the processed image data. An ARM interface module is connected to an external ARM bus and sends necessary information to each module in the preview engine. The preview engine can be controlled through the ARM bus and the ARM interface.  
     [0366] The ports of the preview engine are  
                                       Port Name   Type   Description                  vsync_ccd   Input   VSYNC(VD) input from CCD controller       hsync_ccd   Input   HSYNC(HD) input from CCD controller       ccdc_wrt   Input   Write enable input from CCD controller       ccd_data[9..0]   Input   Data input from CCD controller       sdc_req   Output   Request output to SDRAM controller       sdc_reqen   Input   Request enable input from SDRAM controller       sdc_rw   Output   Read/Write request output to SDRAM controller               0: Write               1: Read       sdc_16bit   Input   SDRAM data width selection input from SDRAM controller               0: 16bit               1: 32bit       sdc_address[21..0]   Output   SDRAM address output to SDRAM controller       sdc_data_in[31..0]   Input   SDRAM data input from SDRAM controller       sdc_data_out[31.0]   Output   SDRAM data output to SDRAM controller       arm_data_in[15..0]   Input   ARM data input through ARM bus       arm_add[6..0]   Input   ARM address input through ARM bus       chip_sel   Input   Chip select input through ARM bus       arm_write   Input   Write enable input through ARM bus       arm_data_out[15..0]   Output   ARM data output to ARM bus       rst_n   Input   Reset input       clk   Input   Main clock input. Depending on the input mode, one of these two types of clock               must be supplied.               CCDC input mode: clk_ccd               SDRAMC input mode: clk_sdr       clk_ccd   Input   CCD clock input       clk_sdr   Input   SDRAM clock input       clk_arm   Input   ARM clock input                  
 
     [0367] Except for selecting a width of the SDRAM data bus (32 bits or 16 bits), an activation of the preview engine is determined by registers which are written through the ARM bus (the selection of the width of SDRAM data bus is determined by a special input signal from the SDRAMC). In this description, some basic operations of the preview engine are explained citing examples, and to simplify an expression, the following two commands “write” and “read” are used.  
     [0368] write NN XXXX :write 0xXXXX to a register at offset 0xNN through the ARM bus  
     [0369] read NN :read a value of a register at offset 0xNN through the ARM bus  
     [0370] The following paragraphs describe the operation of the preview engine in terms of registers; detailed register information appears at the end of the description.  
     [0371] Activate the Preview Engine by setting a PVEN register at offset 0x00 to 1. And by setting the register to 0, the preview engine will be inactive. Actually, if we set the register to 1, the preview engine becomes active at the next rising edge of the VD signal from the CCDC. And if we set it to 0 when the preview engine is active, the preview engine becomes inactive after the last burst transfer of a processed image data to SDRAMC is finished; see FIG. 54.  
     [0372] The following is an example of an operation to start the preview engine. By reading the PVEN bit, a status of the preview engine can be verified.  
     [0373] &gt;write 00 0001 // start the preview  
     [0374] &gt;read 00 // check the PVEN bit  
     [0375] &gt;0001 // preview is active  
     [0376] &gt;write 00 0000 // stop the preview  
     [0377] &gt;read 00 // check the PVEN bit  
     [0378] &gt;0000 // preview is inactive  
     [0379] Preferably the preview engine is inactive when any of the following registers is changed: PVSET1, H_SRT, H_SIZE, V_SRT, V_SIZE, SMTH, H_RSZ, V_RSZ  
     [0380] Select Input Mode; the preview engine has two selectable input ports which are connected to the CCDC and the SDRAMC. Note that there are some difference of available functions between in the CCDC input mode and in the SDRAMC input mode. The following table shows the main difference of available functions between the two input modes. A subsequent paragraph considers resize-only mode in the SDRAMC input mode.  
                              Comparison of the two input modes                             CCDC input mode   SDRAMC input mode                                 Functions   Normal   Resize Only   Normal   Resize Only               Noise filter   ◯   X   ◯   X       Digital gain   ◯   X   ◯   X       White balance   ◯   X   ◯   X       Smoothing   ◯   ◯   ◯   ◯       Down sample   ◯   ◯   ◯   ◯       Zoom   X   X   ◯   ◯       RGB blend   ◯   X   ◯   X       Gamma   ◯   X   ◯   X       Chroma suppress   ◯   X   ◯   X       Chroma offset   ◯   X   ◯   X       1 shot preview   ◯   ◯   ◯   ◯                         Main clock   CCD clock   SDRAM clock                  
 
     [0381] By setting an INMOD bit in a PVSET1 register at offset 0x01 to 0, the preview engine selects the CCDC input port. And by setting the INMOD bit to 1, the preview engine selects the SDRAMC input port. The following are examples of an operation to select the input mode.  
     [0382] &gt;write 00 0000 // stop the preview  
     [0383] &gt;write 01 0000 // select the CCDC input mode  
     [0384] &gt;write 04 0000 // set write address 0x000000  
     [0385] &gt;write 05 0000  
     [0386] &gt;write 00 0001 // start the preview  
     [0387] &gt;write 00 0000 // stop the preview  
     [0388] &gt;write 01 0004 // select the SDRAMC input mode  
     [0389] &gt;write 02 0000 // set read address 0×006000  
     [0390] &gt;write 03 6000  
     [0391] &gt;write 04 0000 // set write address 0x000000  
     [0392] &gt;write 05 0000  
     [0393] &gt;write 00 0001 // start the preview  
     [0394] Define the image size; a size of the output image from the preview engine is determined by the following parameters.  
     [0395] a valid horizontal image size (a H-SIZE register at offset 0×07)  
     [0396] a valid vertical image size (a V_SIZE register at offset 0×09)  
     [0397] a horizontal resize ratio (a H_RSZ register at offset 0x1 E)  
     [0398] a vertical resize ratio (a V_RSZ register at offset 0x1 F)  
     [0399] And the horizontal and vertical sizes of the output image, H out  and V out , are expressed by the following equations  
               H   out     =         (   H_SIZE   )                     (     16   /   H_RSZ     )                   if                 H_RSZ     &gt;     0   ×   0010                   =         (     H_SIZE   -   1     )                     (     16   /   H_RSZ     )                   if                 H_RSZ     &lt;     0   ×   0010                       V   out     =       (   V_SIZE   )                     (     16   /   V_RSZ     )                     
 
     [0400] The horizontal size of the output image must be lower than 720 pixels due to the size of implemented line memories. For example, when we apply a ×4 zoom and want the size of output image to be 720×480, that is, H_RSZ=0×0004, V_RSZ=0×0004, H out =720 and V out =480, an appropriate H_SIZE and V_SIZE can be calculated from the foregoing equations as follows.  
     [0401] 720=(H-SIZE−1)(16/0x0004)  
     [0402] so H_SIZE=181=0x00B5  
     [0403] 480=(V_SIZE)(16/0x0004)  
     [0404] so V_SIZE=120=0x0078  
     [0405] The following is an example operation to apply this ×4 zoom function.  
     [0406] &gt;write 00 0000 // stop the preview  
     [0407] &gt;write 01 0004 // SDRAMC input mode  
     [0408] &gt;write 07 00B5 // Set a valid H-size  
     [0409] &gt;write 09 0078 // Set a valid V-size  
     [0410] &gt;write 1 E 0004 // Set a H-resize ratio  
     [0411] &gt;write 1 F 0004 // Set a V-resize ratio  
     [0412] &gt;write 00 0001 // start the preview  
     [0413] One shot preview; by setting a PVOS bit in the PVSET1 register at offset 0x01 to 1, the preview engine processes only one frame of a CCD raw data. When the preview engine finishes the processing and transferring the image data to the SDRAMC, the preview engine stops by itself as illustrated in FIG. 55.  
     [0414] The following is an example to perform the one shot preview mode and check a status of the PVEN bit.  
     [0415] &gt;write 00 0000 // stop the preview  
     [0416] &gt;write 01 0010 // 1 shot preview mode  
     [0417] // CCDC input mode  
     [0418] &gt;write 00 0001 // start the preview  
     [0419] &gt;read 00 // read PVEN register  
     [0420] &gt;0000 // preview is stopped automatically  
     [0421] Resizing function for YCbCr image data; the preview engine has another data path to utilize only a resizing function in the CFA interpolation module. Therefore, we can use the preview engine as an only resizing function to image data with YCbCr-4:2:2 format. FIG. 56 a  shows the data path of the preview engine in this “resizing only mode”.  
     [0422] In the CCDC input mode, note that the preview engine considers the sequence of input data from CCDC to CbYCrYCbYCrY . . . according to a H_SRT register at offset 0x06. FIG. 56 b  and the following example show this resizing only mode in the CCDC input mode.  
     [0423] &gt;write 00 0000 // stop the preview  
     [0424] &gt;write 01 0008 // resample only mode  
     [0425] // CCDC input mode  
     [0426] &gt;write 06 0002 // set horizontal start point 0x0002  
     [0427] &gt;write 07 0010 // set horizontal size 0x0010 (16 pixel)  
     [0428] &gt;write 00 0001 // start the preview  
     [0429] In the SDRAM input mode, size of the resized image can be calculated the same as in the normal SDRAM input mode described previously.  
     [0430] The following paragraphs provide functional descriptions of the blocks of FIG. 53.  
     [0431] Noise Filter is the first image processing in the preview engine. It is useful to reduce an impulse noise included in the raw data from the CCD image sensor using a noise coring technique. When an original pixel to be applied is set to X 0 , the following calculation is made.  
         x   0     =       X   0     -     a   ×         X     -   2       -     2        X   0       +     X   2       4                 (       a   =     1   8       ,     1   4     ,     1   2     ,   1     )                 
 
     [0432]FIG. 57 shows the block diagram of the noise filter module. It is composed of a high pass filter, a shifter, a subtracter and a output selector. When a NF_EN bit in a N_FIL register is 1, this noise filter is valid. And NF_RT[1.0] bits in the N_FIL register controls the gain of the high pass filter. As to the N-FIL register, see below.  
     [0433] A white balance module has two gain adjusters, a digital gain adjuster and a white balance adjuster. In the digital gain adjuster, the raw data is multiplied by a fixed value gain regardless of a color pixel to be processed. In the white balance gain adjuster, the raw data is multiplied by an selected gain corresponding to the color of the processed pixel. As to the white balance, three different gains corresponding to the value of the processed pixel can be set. FIG. 58 a  shows this example.  
     [0434]FIG. 58 b  shows the block diagram of the write balance module. It is composed of two multipliers for the digital gain and the white balance gain and one adder for the offset of the white balance. An appropriate gain and offset are—selected automatically by setting WB-GAIN, WB_AREA and WB_OFST registers as described below.  
     [0435] The CFA interpolation module, includes many important sub-modules for an image processing, such as a horizontal and vertical interpolation, a smoothing and a horizontal and vertical resizing. FIGS. 59 a - 59   b  show the block diagram and the data path of the CFA interpolation module in the normal mode and the resize only mode. A horizontal interpolation sub-module processes a horizontal interpolation using a two-tap or a five-tap interpolation filter. A smoother sub-module performs a selectable low pass filter which has two different levels of a band-width. A horizontal resampler sub-module performs a seamless horizontal down-sampling or up-sampling (zoom) by ×4 over-sampling. Similarly, a vertical interpolation &amp; resampler sub-module performs a vertical interpolation and a seamless vertical down-sampling or up-sampling by ×4 over-sampling using two line memories. A color selector sub-module extracts an appropriate color from the 4 input from the vertical interpolation and resampler sub-module. A color adjustment sub-module executes the color adjustment to an interpolated data of 4 colors in the complementary color space. In case of the RGB CCD mode, the data bypass this sub-module. A comp2RGB conversion sub-module converts the data from 4 color in the complementary color space to the RGB color space. Therefore, the output of this CFA interpolation module is formatted to RGB color space regardless of a type of the color space of CCD image sensor. As shown in FIG. 59 b , in the resample only mode, only the smoother, the horizontal and vertical resampler are processed.  
     [0436] A horizontal interpolation sub-module performs a horizontal interpolation with a two-tap or a five-tap interpolation filter. The two-tap interpolation filter uses a simple average of the next pixels on the left and right side of the pixel to be interpolated. On the other hand, the 5 taps interpolation filter utilizes the information of another color on the processed line so that a false color due to this interpolation  
         x   0     =     {               X     -   1       +     X   1       2           (     2                 taps     )                   -     X     -   2         +     2        X     -   1         +     2        X   0       +     2        X   1       -     X   2       4           (     5                 taps     )                           
 
     [0437] process can be reduced effectively. Actually, when a pixel at the position to be interpolated is set to X 0 , this horizontal interpolation sub-module performs the following calculation.  
     [0438]FIG. 60 shows a block diagram of the horizontal interpolation sub-module. It is composed three adders for calculating the above expression. A type of the two interpolation processes depends on a H_INTP bit in PVSET2 register as described below.  
     [0439] A smoother sub-module is an anti-aliasing filter which is composed of two kinds of FIR filters. By combining the two filter, one of two levels of horizontal low pass filter can be performed. Actually, one of the following calculations is made depending on the register setting.  
         x   0     =     {           X   0           (   bypass   )                     X     -   1       +     2        X   0       +     X   1       4     ≡     x   0   ′             (   weak   )                   x     -   2     ′     +     2        x   0   ′       +     x   2   ′       4           (   strong   )                           
 
     [0440]FIG. 61 shows a block diagram of the smoother sub-module. It is composed of the two FIR filter and an output selector. The output is determined by a SMTH_EN bit and a SMTH_LV bit in a SMTH register as described below.  
     [0441] A horizontal resampler sub-module performs a seamless horizontal down-sampling or up-sampling (zoom). This seamless resampling processing is done by a bi-linear interpolation of data from the smoother sub-module. FIG. 62 a  shows an example of this resampling process (×2 zoom). Actually, one of the following four calculations is processed depending on a resampled position.  
         x   0     =     {           X   0           (     position                 0     )                   X   0     +     3        X   1         4           (     position                 1     )                   X   0     +     X   1       2           (     position                 2     )                   3        X   0       +     X   1       4           (     position                 3     )                           
 
     [0442]FIG. 62 b  shows a block diagram of the horizontal resampler sub-module. The calculation of the resampled position above is done by a position detector which accumulates H_RSZ bits in a H_RSZ register periodically.  
     [0443] A vertical interpolation &amp; resampler sub-module performs a vertical interpolation and a seamless vertical down-sampling or up-sampling. Same as the horizontal resampler module, this vertical resampling is processed by a bi-linear interpolation of a data from the horizontal interpolation sub-module and two line memories. And this sub-module executes two different calculation depending on the color pattern of CCD image sensor. FIG. 63 a  shows an example of this two types of the CFA pattern.  
     [0444] If there is a color which exists in every line, this sub-module activates in “Bayer mode”, otherwise it activates in “non-Bayer mode”. In the Bayer mode, the vertical interpolation and resampling processing are based on the information of a main color (G or Ye in FIG. 63 a ). As shown in FIG. 63 b , in the Bayer mode, one main color (G in this figure) exists in every line. Therefore, in this mode, the other colors can be calculated in order to keep relationship with the main color. Actually in this example of the Bayer mode, the following calculation is made in this sub-module  
             Position                 0               r   =     R   1                 g   =     G   1                 b   =           B   0     +     B   2       2     -         G   0     +     G   2       2     +     G   1                           Position                 1               r   =       R   1     -     G   1     +   g                 g   =         3        G   1       +     G   2       4                 b   =           3        B   0       +     5        B   2         8     -         3        G   0       +     5        G   2         8     +   g                         Position                 2               r   =       R   1     -     G   1     +   g                 g   =         G   1     +     G   2       2                 b   =           B   0     +     3        B   2         4     -         G   0     +     3        G   2         4     +   g                         Position                 3               r   =       R   1     -     G   1     +   g                 g   =         G   1     +     3        G   2         4                 b   =           B   0     +     7        B   2         8     -         G   0     +     7        G   2         8     +   g                         
 
     [0445] On the other hand, in the non-Bayer mode, there is no main color in the CFA pattern. Therefore, the vertical interpolation and resampling processing is performed in order to keep relationship with upper or lower color (in FIG. 63 b : Ye and G, Cy and Mg). Actually, the following calculations (for Ye, Gr, Cy, and Mg) are made.  
             Position                 0               ye   =         Ye   0     +     Ye   2       2                 cy   =         Cy   0     +     Cy   2       2                 gr   =     Gr   1                 mg   =     Mg   1                         Position                 1               ye   =         3        Ye   0       +     5        Ye   2         8                 cy   =         3        Cy   0       +     5        Cy   2         8                 gr   =       Gr   1     -         Ye   0     +     Ye   2       2     +   ye                 mg   =       Mg   1     -         Cy   0     +     Cy   2       2     +   Cy                         Position                 2               ye   =         Ye   0     +     3        Ye   2         4                 cy   =         Cy   0     +     3        Cy   2         4                 gr   =       Gr   1     -         Ye   0     +     Ye   2       2     +   ye                 mg   =       Mg   1     -         Cy   0     +     Cy   2       2     +   Cy                         Position                 3               ye   =         Ye   0     +     7        Ye   2         8                 cy   =         Cy   0     +     7        Cy   2         8                 gr   =       Gr   1     -         Ye   0     +     Ye   2       2     +   ye                 mg   =       Mg   1     -         Cy   0     +     Cy   2       2     +   Cy                         
 
     [0446]FIGS. 64 a - 64   c  show a block diagram of the vertical interpolation and resampler sub-module. There are two steps of the processing for supporting the two types of calculation in the same circuit. The first step is to prepare all terms of expression above calculating the resampled position as in the horizontal resampling sub-module. In the second step, the final outputs are calculated using the results of the first step operation.  
     [0447] A color selector sub-module extracts an appropriate color from the four input signals which the vertical interpolation and resampler sub-module generates. FIG. 65 shows a block diagram of the color selector sub-module. These extractor for each color select an appropriate signal automatically using the sync signal and CFA[7 . . . 0] bits in the PVSET2 register as described below.  
     [0448] A color adjustment sub-module performs the adjustment to interpolated data of 4 colors in the complementary color space in order to satisfy an equation of the following complementary color space.  
     
       Ye+Cy=Gr+Mg  
     
     [0449] To keep this relationship, firstly, the following value e is calculated.  
     
       e=G+Mg−Ye−Cy  
     
     [0450] This value e can be defined as the amount of errors of 4 colors from the interpolation process. Therefore, the result from the interpolation process, Ye, Cy, Gr and Mg, is redefined to satisfy the above formula.  
       {           ye   =     Ye   +     e   4                   cy   =     Cy   +     e   4                   gr   =     Gr   -     e   4                   mg   =     Mg   -     e   4                             
 
     [0451]FIG. 66 shows a block diagram of the color adjustment sub-module. It is composed of six adders and one subtracter for executing the above calculation. If a CCDMOD bit in the PVSET2 register is zero, the data bypass this sub-module.  
     [0452] A comp2RGB conversion sub-module converts the data from four colors in the complementary color space to the RGB color space. Especially for G, a data from the interpolation process and data calculated in this module using the same formula in the complementary color space can be blended. Actually, the following calculation is made based on the conversion formula.  
     [0453]FIG. 67 shows a block diagram of the color adjustment sub-module. Same as in the color adjustment sub-module, if the CCDMOD bit is 0, the data bypass this module.  
       {             r   =     Ye   -   Cy   +   Mg                 g   =     rG   +       (     1   -   r     )                     (     Ye   +   Cy   -   Mg     )                     b   =     Mg   -   Ye   +   Cy                  
          (       r   =   0     ,     1   4     ,     1   2     ,     3   4     ,   1     )                     
 
     [0454] An RGB2RGB blending module has a general 3×3 square matrix and redefines the RGB data from the CFA interpolation module, which can be used as a function of a color correction. In this module, the following calculation is made.  
         [           R   out               G   out               B   out           ]     =       [         MTX_RR       MTX_GR       MTX_BR           MTX_RG       MTX_GG       MTX_BG           MTX_RB       MTX_GB       MTX_BB         ]                [           R   in               G   in               B   in           ]                   
 
     [0455]FIG. 68 shows a block diagram of the RGB2RGB blending module. Nine multipliers and three adders are required for performing this matrix operation.  
     [0456] A gamma correction module performs a gamma correction independently for each color in the RGB color space by approximating an ideal gamma curve with four piece wise linear lines. As shown in FIG. 53, this module exists independently for each color so that the independent setting is possible. FIG. 69 a  shows this example  
     [0457]FIG. 69 b  shows a block diagram of the gamma correction module. It is composed of one multiplier for the gain and one adder for the offset. An area detector selects an appropriate gain and offset value using the data in a GMGAIN, a GMAREA and a GMOFST register as detailed below.  
     [0458] An RGB2YCbCr conversion module has a 3×3 square matrix and converts the RGB color space of the image data into a YCbCr color space. In addition to the conversion matrix operation, the functions of a chroma suppression and a chroma offset are performed in this module.  
     [0459]FIG. 70 shows a block diagram of the RGB2YcbCr conversion matrix. It is composed of nine multipliers and three adders for the basic conversion matrix, two multipliers for the chroma suppression and 2 adders for the chroma offset.  
     [0460] Details of the registers referred to in the foregoing follow.  
                                                  1.1  PVEN (offset=0x00, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       PVEN   X   X   X   X   X   X   X   X       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x00   X   X   X   X   X   X   X   PVEN                             D0: PVEN   Set the preview engine to On/Off.                         0: Off                         1: On                      
 
     [0461]                                                  1.2  PVSET1 (offset=0x01, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       PVSET1   X   X   X   X   X   X   X   X       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x01   X   X   X   PVOS   RSOLY   INMOD   BSTAL   WEN                             D4: PVOS   Set a number of frames to be processed.                         0: Normal (Continuous)                         1: One shot only                             D3: RSOLY   Select a type of an input data.                         0: Normal (Raw data input)                         1: Resizing only mode (YCbCr-4:2:2 data input)                             D2: INMOD   Select an input interface.                         0: CCDC interface                         1: SDRAMC interface                             D1: BSTAL   Apply a burst aligned transfer of an output image to               SDRAMC. If it is set to 1, the horizontal size is aligned               to the burst length and surplus pixels are disregarded.                         0: Normal                         1: Burst align mode                             D0: WEN   Select a handling of a write enable signal from the               CCDC. If it is set to 1 and the write enable signal is               low, the preview engine is suspended in the frame.                         0: Ignore                         1: Valid                        
     [0462]                                                  1.3  RADDU (offset=0x02, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       RADDU   X   X   X   X   X   X   X   X       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x02   X   X   RADD21   RADD20   RADD19   RADD18   RADD17   RADD16                             D[5..0]: RADD[21..16]   Set a start address where an input data is stored in               SDRAMC input mode.                        
     [0463]                                                  1.4  RADDL (offset=0x03, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       RADDL   RADD15   RADD14   RADD13   RADD12   RADD11   RADD10   RADD9   RADD8       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x03   RADD7   RADD6   RADD5   RADD4   RADD3   RADD2   RADD1   RADD0                             D[15..0]: RADD[15..0]   Set a start address where an input data is stored in SDRAMC               input mode.                        
     [0464]                                                  1.5  WADDU (offset=0x04, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       WADDU   X   X   X   X   X   X   X   X       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x04   X   X   WADD21   WADD20   WADD19   WADD18   WADD17   WADD16                             D[5..0]: WADD[21..16]   Set a start address of an output image to be stored.                        
     [0465]                                                  1.6  WADDL (offset=0x05, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       WADDL   WADD15   WADD14   WADD13   WADD12   WADD11   WADD10   WADD9   WADD8       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x05   WADD7   WADD6   WADD5   WADD4   WADD3   WADD2   WADD1   WADD0                             D[15..0]: WADD[15..0]   Set a start address of an output image to be stored.                        
     [0466]                                                  1.7  H_SRT (offset=0x06, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       H_SRT   X   X   X   X   H_SRT11   H_SRT10   H_SRT9   H_SRT8       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x06   H_SRT7   H_SRT6   H_SRT5   H_SRT4   H_SRT3   H_SRT2   H_SRT1   H_SRT0                             D[11..0]: H_SRT[11..0]   If the INMOD=0, set a horizontal start position of a valid               input image. In this mode, the data must be lower than the a               total number of horizontal pixel from CCDC. If the               INMOD=1, set a total horizontal image size of a input image               stored in the SDRAM.                        
     [0467]                                                  1.8  H_SIZE (offset=0x07, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       H_SIZE   X   X   X   X   H_SIZE11   H_SIZE10   H_SIZE9   H_SIZE8       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x07   H_SIZE7   H_SIZE6   H_SIZE5   H_SIZE4   H_SIZE3   H_SIZE2   H_SIZE1   H_SIZE0                             D[11..0]: H_SIZE[11..0]   Set a horizontal image size of a valid input image. If the               INMOD=0, the data must be 32 clock lower than the total               number of horizontal pixel from CCDC.                        
     [0468]                                                  1.9  V_SRT (offset=0x08, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       V_SRT   X   X   X   X   V_SRT11   V_SRT10   V_SRT9   V_SRT8       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x08   V_SRT7   V_SRT6   V_SRT5   V_SRT4   V_SRT3   V_SRT2   V_SRT1   V_SRT0                             D[11..0]: V_SRT[11..0]   Set a vertical start position of a valid input image.                        
     [0469]                                                  1.10  V_SIZE (offset=0x09, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       V_SIZE   X   X   X   X   V_SIZE11   V_SIZE10   V_SIZE9   V_SIZE8       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x09   V_SIZE7   V_SIZE6   V_SIZE5   V_SIZE4   V_SIZE3   V_SIZE2   V_SIZE1   V_SIZE0                             D[11..0]: V_SIZE[11..0]   Set a vertical image size of a valid input image.                        
     [0470]                                                  1.11 PVSET2 (offset=0x0A, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       PVSET2   H_INTP   V_INTP   V_NFIL   CCDMOD   X   G_BLND2   G_BLND1   G_BLND0       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x0A   CFA7   CFA6   CFA5   CFA4   CFA3   CFA2   CFA1   CFA0                             D15: H_INTP   Select a type of a horizontal interpolation.                         0: Hue interpolation (5 taps)           1: Simple interpolation (2 taps)                             D14: V_INTP   Select a type of a vertical interpolation.                         0: Hue interpolation           1: Simple interpolation                             D13: V_NFIL   Apply a vertical noise filter.                         0: Off           1: On                             D12: CCDMOD   Select a type of a CCD image sensor.                         0: RGB color space           1: Complementary color space                             D[10..8]: G_BLND[2..0]   In a complementary CCD mode, adjust a blending ratio               between an original green and a calculated green from the               other colors.                         G output  = rG org  + (1−r)(Ye+Cy−Mg)                         0: r=1           1: r=3/4           2: r=1/2           3: r=1/4           4: r=0                             D[7..6]: CFA[7..6]   Set a color of a raw data on even pixel and on even line.           D[5..4]: CFA[5..4]   Set a color of a raw data on odd pixel and on even line.           D[3..2]: CFA[3..2]   Set a color of a raw data on even pixel and on odd line.           D[1..0]: CFA[1..0]   Set a color of a raw data on odd pixel and on odd line.                         0: G/Ye           1: B/Cy           2: R/Mg           3: −/Gr                        
     [0471]                                                  1.12 N_FIL (offset=0x0B, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       N_FIL   X   X   X   X   X   X   X   X       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x0B   X   X   X   NF_EN   X   X   NF_RT1   NF_RT0                             D4: NF_EN   Apply a horizontal noise filter.           D[1..0]: NF_RT[1..0]   Select a level of an applied noise filter.                         0: Weak           1: A little weak           2: A little strong           3: Strong                        
     [0472]                                                  1.13 D_GAIN (offset=0x0C, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       D_GAIN   X   X   X   X   X   X   D_GAIN9   D_GAIN8       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x0C   D_GAIN7   D_GAIN6   D_GAIN5   D_GAIN4   D_GAIN3   D_GAIN2   D_GAIN1   D_GAIN0                             D[9..0]: D_GAIN[9..0]   Set a value of a digital gain process.                             D_GAIN   Gain           11 1111 1111   X 3.99609375           11 1111 1110   X 3.9921875           11 1111 1101   X 3.98828125           01 0000 0000   X 1           00 1000 0000   X 0.5           00 0000 0010   X 0.0078125           00 0000 0001   X 0.00390625           00 0000 0000   X 0                 Table 5.13 D_GAIN                    
     [0473]                                                  1.14 WB_GAIN0 (offset=0x0D, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       WB_GAIN0   WG0_GY7   WG0_GY6   WG0_GY5   WG0_GY4   WG0_GY3   WG0_GY2   WG0_GY1   WG0_GY0       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x0D   WG1_GY7   WG1_GY6   WG1_GY5   WG1_GY4   WG1_GY3   WG1_GY2   WG1_GY1   WG1_GY0                             D[15..8]: WG0_GY[7..0]   For Gb/Ye, set a gain value of a white balance in area 0 of               FIG. 58a.           D[7..0]: WG1_GY[7..0]    For Gb/Ye, set a gain value of a white balance in area 1.                             WG0_GY   Gain           1111 1111   X 7.96875           1111 1110   X 7.9375           1111 1101   X 7.90625           0010 0000   X 1           0001 0000   X 0.5           0000 0010   X 0.0625           0000 0001   X 0.003125           0000 0000   X 0                 WG0_GY                    
     [0474]                                                  1.15 WB_GAIN1 (offset=0x0E, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       WB_GAIN1   WG2_GY7   WG2_GY6   WG2_GY5   WG2_GY4   WG2_GY3   WG2_GY2   WG2_GY1   WG2_GY0       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x0E   WG0_BC7   WG0_BC6   WG0_BC5   WG0_BC4   WG0_BC3   WG0_BC2   WG0_BC1   WG0_BC0                             D[15..8]: WG2_GY[7..0]   For Gb/Ye, set a gain value of a white balance in area 2.           D[7..0]: WG0_BC[7..0]   For B/Cy, set a gain value of a white balance in area 0.                        
     [0475]                                                  1.16 WB_GAIN2 (offset=0x0F, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       WB_GAIN2   WG1_BC7   WG1_BC6   WG1_BC5   WG1_BC4   WG1_BC3   WG1_BC2   WG1_BC1   WG1_BC0       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x0F   WG2_BC7   WG2_BC6   WG2_BC5   WG2_BC4   WG2_BC3   WG2_BC2   WG2_BC1   WG2_BC0                             D[15..8]: WG1_BC[7..0]   For B/Cy, set a gain value of a white balance in area 1.           D[7..0]: WG2_BC[7..0]   For B/Cy, set a gain value of a white balance in area 2.                        
     [0476]                                                  1.17 WB_GAIN3 (offset=0x10, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       WB_GAIN3   WG0_GG7   WG0_GG6   WG0_GG5   WG0_GG4   WG0_GG3   WG0_GG2   WG0_GG1   WG0_GG0       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x10   WG1_GG7   WG1_GG6   WG1_GG5   WG1_GG4   WG1_GG3   WG1_GG2   WG1_GG1   WG1_GG0                             D[15..8]: WG0_GG[7..0]   For Gr/Gr, set a gain value of a white balance in area 0.           D[7..0]: WG1_GG[7..0]   For Gr/Gr, set a gain value of a white balance in area 1.                        
     [0477]                                                  1.18 WB_GAIN4 (offset=0x11, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       WB_GAIN4   WG2_GG7   WG2_GG6   WG2_GG5   WG2_GG4   WG2_GG3   WG2_GG2   WG2_GG1   WG2_GG0       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x11   WG0_RM7   WG0_RM6   WG0_RM5   WG0_RM4   WG0_RM3   WG0_RM2   WG0_RM1   WG0_RM0                             D[15..8]: WG2_GG[7..0]   For Gr/Gr, set a gain value of a white balance in area 2.           D[7..0]: WG0_RM[7..0]   For R/Mg, set a gain value of a white balance in area 0.                        
     [0478]                                                  1.19 WB_GAIN5 (offset=0x12, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       WB_GAIN5   WG1_RM7   WG1_RM6   WG1_RM5   WG1_RM4   WG1_RM3   WG1_RM2   WG1_RM1   WG1_RM0       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x12   WG2_RM7   WG2_RM6   WG2_RM5   WG2_RM4   WG2_RM3   WG2_RM2   WG2_RM1   WG2_RM0                             D[15..8]: WG1_RM[7..0]   For R/Mg, set a gain value of a white balance in area 1.           D[7..0]: WG2_RM[7..0]   For R/Mg, set a gain value of a white balance in area 2.                        
     [0479]                                                  1.20 WB_AREA0 (offset=0x13, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       WB_AREA0   X   X   X   X   X   X   WB0_A9   WB0_A8       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x13   WB0_A7   WB0_A6   WB0_A5   WB0_A4   WB0_A3   WB0_A2   WB0_A1   WB0_A0                             D[9..0]: WB0_A[9..0]   Set a boundary point of parameters for a white balance               between area 0 and area 1.                        
     [0480]                                                  1.21  WB_AREA1 (offset=0x14, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       WB_AREA1   X   X   X   X   X   X   WB1_A9   WB1_A8       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x14   WB1_A7   WB1_A6   WB1_A5   WB1_A4   WB1_A3   WB1_A2   WB1_A1   WB1_A0                             D[9..0]: WB1_A[9..0]   Set a boundary point of parameters for a white balance               between area 1 and area 2.                        
     [0481]                                                  1.22 WB_OFST0 (offset=0x15, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       WB_OFST0   X   X   X   X   X   WO1_GY10   WO1_GY9   WO1_GY8       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x15   WO1_GY7   WO1_GY6   WO1_GY5   WO1_GY4   WO1_GY3   WO1_GY2   WO1_GY1   WO1_GY0                             D[10..0]: WO1_GY[10..0]   For Gb/Ye, set a offset value of a white balance in area1.                        
     [0482]                                                  1.23 WB_OFST1 (offset=0x16, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       WB_OFST1   X   X   X   X   X   WO2_GY10   WO2_GY9   WO2_GY8       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x16   WO2_GY7   WO2_GY6   WO2_GY5   WO2_GY4   WO2_GY3   WO2_GY2   WO2_GY1   WO2_GY0                             D[10..0]: WO2_GY[10..0]   For Gb/Ye, set a offset value of a white balance in area2.                        
     [0483]                                                  1.24 WB_OFST2 (offset=0x17, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       WB_OFST2   X   X   X   X   X   WO1_BC10   WO1_BC9   WO1_BC8       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x17   WO1_BC7   WO1_BC6   WO1_BC5   WO1_BC4   WO1_BC3   WO1_BC2   WO1_BC1   WO1_BC0                             D[10..0]: WO1_BC[10..0]   For B/Cy, set a offset value of a white balance in area1.                        
     [0484]                                                  1.25 WB_OFST3 (offset=0x18, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       WB_OFST3   X   X   X   X   X   WO2_BC10   WO2_BC9   WO2_BC8       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x18   WO2_BC7   WO2_BC6   WO2_BC5   WO2_BC4   WO2_BC3   WO2_BC2   WO2_BC1   WO2_BC0                             D[10..0]: WO2_BC[10..0]   For B/Cy, set a offset value of a white balance in area2.                        
     [0485]                                                  1.26 WB_OFST4 (offset=0x19, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       WB_OFST4   X   X   X   X   X   WO1_GG10   WO1_GG9   WO1_GG8       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x19   WO1_GG7   WO1_GG6   WO1_GG5   WO1_GG4   WO1_GG3   WO1_GG2   WO1_GG1   WO1_GG0                             D[10..0]: WO1_GG[10..0]   For Gr/Gr, set a offset value of a white balance in area1.                        
     [0486]                                                  1.27 WB_OFST5 (offset=0x1A, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       WB_OFST5   X   X   X   X   X   WO2_GG10   WO2_GG9   WO2_GG8       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x1A   WO2_GG7   WO2_GG6   WO2_GG5   WO2_GG4   WO2_GG3   WO2_GG2   WO2_GG1   WO2_GG0                             D[10..0]: WO2_GG[10..0]   For Gr/Gr, set a offset value of a white balance in area2.                        
     [0487]                                                  1.28 WB_OFST6 (offset=0x1B, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       WB_OFST6   X   X   X   X   X   WO1_RM10   WO1_RM9   WO1_RM8       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x1B   WO1_RM7   WO1_RM6   WO1_RM5   WO1_RM4   WO1_RM3   WO1_RM2   WO1_RM1   WO1_RM0                             D[10..0]: WO1_RM[10..0]   For R/Mg, set a offset value of a white balance in area1.                        
     [0488]                                                  1.29 WB_OFST7 (offset=0x1C, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       WB_OFST7   X   X   X   X   X   WO2_RM10   WO2_RM9   WO2_RM8       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x1C   WO2_RM7   WO2_RM6   WO2_RM5   WO2_RM4   WO2_RM3   WO2_RM2   WO2_RM1   WO2_RM0                             D[10..0]: WO2_RM[10..0]   For R/Mg, set a offset value of a white balance in area2.                        
     [0489]                                                  1.30 SMTH (offset=0x1D, R/W)                                                 Register Name   D15   D14   D13   D12   D11   D10   D9   D8       SMTH   X   X   X   X   X   X   X   X       Offset   D7   D6   D5   D4   D3   D2   D1   D0       0x1D   X   X   X   X   X   X   SMTH_EN   SMTH_LV                             D1: SMTH_EN   Apply a smoothing function to a processed image.                         0: Off           1: On                             D0: SMTH_LV   Select a level of the smoothing function.                         0: Weak           1: Strong                        
     [0490]                               1.31 H_RSZ (offset=0×1E, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               H_RSZ   X   X   X   X   X   X   H_RSZ9   H_RSZ8               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0×1E   H_RSZ7   H_RSZ6   H_RSZ5   H_RSZ   H_RSZ3   H_RSZ2   H_RSZ1   H_RSZ0                             D[9..0[: H_RSZ[9..0]   Set a parameter for a horizontal resizing. An actual resizing           ratio R H  is calculated by the following equation.                             R   H     =     16   H_RSZ                                
     [0491]                               1.32 V_RSZ (offset=0×1F, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               H_RSZ   X   X   X   X   X   X   H_RSZ9   H_RSZ8               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0×1F   H_RSZ7   H_RSZ6   H_RSZ5   H_RSZ   H_RSZ3   H_RSZ2   H_RSZ1   H_RSZ0                             D[9..0[: H_RSZ[9..0]   Set a parameter for a horizontal resizing. An actual resizing           ratio R V  is calculated by the following equation.                             R   V     =     16   V_RSZ                                
     [0492]                               1.33 MTX_GAIN0 (offset=0×20, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               BLD_GAIN0   MTX_RR7   MTX_RR6   MTX_RR5   MTX_RR4   MTX_RR3   MTX_RR2   MTX_RR1   MTX_RR0               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0×20   MTX —GR7     MTX_GR6   MTX_GR5   MTX_GR4   MTX_GR3   MTX_GR2   MTX_GR1   MTX_GR0                             D[15..8]: MTX_RR[7..0]   Set a coefficient of R in  for calculating R out  in a RGB matrix.       D[7..0]: MTX_GR[7..0]   Set a coefficient of G in  for calculating R out  in a RGB matrix.                             [           R   out               G   out               B   out           ]     =       [         MTX_RR       MTX_GR       MTX_BR           MTX_RG       MTX_GG       MTX_BG           MTX_RB       MTX_GB       MTX_BB         ]                [           R   in               G   in               B   in           ]                                                         MTX —RR     Gain                                       1111 1111   X 3.984375           1111 1110   X 3.96875           1111 1101   X 3.953125           0100 0000   X 1           0010 0000   X 0.5           0000 0010   X 0.03125           0000 0001   X 0 015625           0000 0000   X 0                             MTX_RR                    
     [0493]                               1.34 MTX_GAIN1 (offset = 0 × 21, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               BLD_GAIN1   MTX_BR7   MTX_BR6   MTX_BR5   MTX_BR4   MTX_BR3   MTX_BR2   MTX_BR1   MTX_BR0               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 21   MTX_RG7   MTX_RG6   MTX_RG5   MTX_RG4   MTX_RG3   MTX_RG2   MTX_RG1   MTX_RG0                                    
     [0494]                               1.35 MTX_GAIN2 (offset = 0 × 22, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               BLD_GAIN2   MTX_GG7   MTX_GG6   MTX_GG5   MTX_GG4   MTX_GG3   MTX_GG2   MTX_GG1   MTX_GG0               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 22   MTX_BG7   MTX_BG6   MTX_BG5   MTX_BG4   MTX_BG3   MTX_BG2   MTX_BG1   MTX_BG0                                    
     [0495]                               1.36 MTX_GAIN3 (offset = 0 × 23, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               BLD_GAIN3   MTX_RB7   MTX_RB6   MTX_RB5   MTX_RB4   MTX_RB3   MTX_RB2   MTX_RB1   MTX_RB0               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 23   MTX_GB7   MTX_GB6   MTX_GB5   MTX_GB4   MTX_GB3   MTX_GB2   MTX_GBl   MTX_GB0                                    
     [0496]                               1.37 MTX_GAIN4 (offset = 0 × 24, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               BLD_GAIN4   MTX_BB7   MTX_BB6   MTX_BB5   MTX_BB4   MTX_BB3   MTX_BB2   MTX_BB1   MTX_BB0               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 24   X   X   X   X   X   X   X   X                            
     [0497]                               1.38 GMGAIN_R0 (offset = 0 × 25, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMGAIN_R0   GMG0_R7   GMG0_R6   GMG0_R5   GMG_R4   GMG_R3   GMG0_R2   GMG0_R1   GMG0_R0               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 25   GMG1_R7   GMG1_R6   GMG1_R5   GMG1_R4   GMG1_R3   GMG1_R2   GMG1_R1   GMG1_R0                                     D[15 . . . 8]: GMG0_R[7 . . . 0]   For R, set a gain value of a gamma correction in area0 of FIG. 69a.           D[7 . . . 0]: GMG1_R[7 . . . 0]   For R, set a gain value of a gamma correction in area1.                                         GMG0_R   Gain                       1111 1111   X 15.9375           1111 1110   X 15.875           1111 1101   X 15.8125           0010 0000   X 2           0001 0000   X 1           0000 0010   X 0.125           0000 0001   X 0.0625           0000 0000   X 0                 GMGO_R                    
     [0498]                               1.39 GMGAIN_R1 (offset = 0 × 26, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMGAIN_R1   GMG2_R7   GMG2_R6   GMG2_R5   GMG2_R4   GMG2_R3   GMG2_R2   GMG2_R1   GMG2_R0               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 26   GMG3_R7   GMG3_R6   GMG3_R5   GMG3_R4   GMG3_R3   GMG3_R2   GMG3_R1 GMG3_R0                                    
     [0499]                               1.40 GMGAIN_G0 (offset = 0 × 27, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMGAIN_G0   GMG0_G7   GMG0_G6   GMG0_G5   GMG0_G4   GMG0_G3   GMG0_G2   GMG0_G1   GMG0_G0               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 27   GMG1_G7   GMG1_G6   GMG1_G5   GMG1_G4   GMG1_G3   GMG1_G2   GMG1_G1   GMG1_G0                                    
     [0500]                               1.41 GMGAIN_G1 (offset = 0 × 28, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMGAIN_G1   GMG2_G7   GMG2_G6   GMG2_G5   GMG2_G4   GMG2_G3   GMG2_G2   GMG2_G1   GMG2_G0               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 28   GMG3_G7   GMG3_G6   GMG3_G5   GMG3_G4   GMG3_G3   GMG3_G2   GMG3_G1   GMG3_G0                                    
     [0501]                               1.42 GMGAIN_B0 (offset = 0 × 29, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMGAIN_B0   GMG0_B7   GMG0_B6   GMG0_B5   GMG0_B4   GMG0_B3   GMG00_B2   GMG0_B1   GMG0_B0               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 29   GMG1_B7   GMG1_B6   GMG1_B5   GMG1_B4   GMG1_B3   GMG1_B2   GMG1_B1   GMG1_B0                                    
     [0502]                               1.43 GMGAIN_B1 (offset = 0 × 29, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMGAIN_B1   GMG2_B7   GMG2_B6   GMG2_B5   GMG2_B4   GMG2_B3   GMG2_B2   GMG2_B1   GMG2_B0               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 2A   GMG3_B7   GMG3_B6   GMG3_B5   GMG3_B4   GMG3_B3   GMG3_B2   GMG3_B1   GMG3_B0                                    
     [0503]                               1.44 GMAREA_R0 (offset = 0 × 2B, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMAREA_R0   X   X   X   X   X   X   GMA0_R9   GMA0_R8               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 2B   GMA0_R7   GMA0_R6   GMA0_R5   GMA0_R4   GMA0_R3   GMA0_R2   GMA0_R1   GMA_R0                            
     [0504]                               1.45 GMAREA_R1 (offset = 0 × 2C, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMAREA_R1   X   X   X   X   X   X   GMA1_R9   GMA1_R8               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 2C   GMA1_R7   GMA1_R6   GMA1_R5   GMA1_R4   GMA1_R3   GMA1_R2   GMA1_R1   GMA1_R0                            
     [0505]                               1.46 GMAREA_R2 (offset = 0 × 2D, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMAREA_R2   X   X   X   X   X   X   GMA2_R9   GMA2_R8               0 × 2D   GMA2_R7   GMA2_R6   GMA2_R5   GMA2_R4   GMA2_R3   GMA2_R2   GMA2_R1   GMA2_R0                            
     [0506]                               1.47 GMAREA_G0 (offset = 0 × 2E, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMAREA_G0   X   X   X   X   X   X   GMA0_G9   GMA0_G8               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 2E   GMA0_G7   GMA0_G6   GMA0_G5   GMA0_G4   GMA0_G3   GMA0_G2   GMA0_G1   GMA0_G0                            
     [0507]                               1.48 GMAREA_G1 (offset = 0 × 2F, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMAREA_G1   X   X   X   X   X   X   GMA1_G9   GMA1_G8               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 2F   GMA1_G7   GMA1_G6   GMA1_G5   GMA1_G4   GMA1_G3   GMA1_G2   GMA1_G1   GMA1_G0                            
     [0508]                               1.49 GMAREA_G2 (offset = 0 × 30, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMAREA_G2   X   X   X   X   X   X   GMA2_G9   GMA2_G8               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 30   GMA2_G7   GMA2_G6   GMA2_G5   GMA2_G4   GMA2_G3   GMA2_G2   GMA2_G1   GMA2_G0                            
     [0509]                               1.50 GMAREA_B0 (offset = 0 × 3l, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMAREA_B0   X   X   X   X   X   X   GMA0_B9   GMA0_B8               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 31   GMA0_B7   GMA0_B6   GMA0_B5   GMA0_B4   GMA0_B3   GMA0_B2   GMA0_B1   GMA0_B0                            
     [0510]                               1.51 GMAREA_B1 (offset = 0 × 32, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMAREA_B1   X   X   X   X   X   X   GMA1_B9   GMA1_B8               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 32   GMA1_B7   GMA1_B6   GMA1_B5   GMA1_B4   GMA1_B3   GMA1_B2   GMA1_B1   GMA1_B0                            
     [0511]                               1.52 GMAREA_B2 (offset = 0 × 33, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMAREA_B2   X   X   X   X   X   X   GMA2_B9   GMA2_B8               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 33   GMA2_B7   GMA2_B6   GMA2_B5   GMA2_B4   GMA2_B3   GMA2_B2   GMA2_B1   GMA_B0                            
     [0512]                               1.53 GMOFST_R0 (offset = 0 × 34, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMOFST_R0   X   X   X   X   X   GMO0_R10   GMO0_R9   GMO0_R8               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 34   GMO0_R7   GMO0_R6   GMO0_R5   GMO0_R4   GMO0_R3   GMO0_R2   GMO0_R1   GMO0_R0                            
     [0513]                               1.54 GMOFST_R1 (offset = 0 × 35, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMOFST_R1   X   X   X   X   X   GMO1_R10   GMO1_R9   GMO1_R8               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 35   GMO1_R7   GMO1_R6   GMO1_R5   GMO1_R4   GMO1_R3   GMO1_R2   GMO1_R1   GMO1_R0                            
     [0514]                               1.55 GMOFST_R2 (offset = 0 × 36, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMOFST_R2   X   X   X   X   X   GMO2_R10   GMO2_R9   GMO2_R8               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 36   GMO2_R7   GMO2_R6   GMO2_R5   GMO2_R4   GMO2_R3   GMO2_R2   GMO2_R1   GMO2_R0                            
     [0515]                               1.56 GMOFST_R3 (offset = 0 × 37, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMOFST_R3   X   X   X   X   X   GMO3_R10   GMO3_R9   GMO3_R8               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 37   GMO3_R7   GMO3_R6   GMO3_R5   GMO3_R4   GMO3_R3   GMO3_R2   GMO3_R1   GMO3_R0                            
     [0516]                               1.57 GMOFST_G0 (offset = 0 × 38, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMOFST_G0   X   X   X   X   X   GMO0_G10   GMO0_G9   GMO0_G8               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 38   GMO0_G7   GMO0_G6   GMO0_G5   GMO0_G4   GMO0_G3   GMO0_G2   GMO0_G1   GMO0_G0                            
     [0517]                               1.58 GMOFST_G1 (offset = 0 × 39, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMOFST_G1   X   X   X   X   X   GMO1_G10   GMO1_G9   GMO1_G8               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 39   GMO1_G7   GMO1_G6   GMO1_G5   GMO1_G4   GMO1_G3   GMO1_G2   GMO1_G1   GMO1_G0                            
     [0518]                               1.59 GMOFST_G2 (offset = 0 × 3A, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMOFST_G2   X   X   X   X   X   GMO2_G10   GMO2_G9   GMO2_G8               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 3A   GMO2_G7   GMO2_G6   GMO2_G5   GMO2_G4   GMO2_G3   GMO2_G2   GMO2_G1   GMO2_G0                            
     [0519]                               1.60 GMOFST_G3 (offset = 0 × 3B, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMOFST_G3   X   X   X   X   X   GMO3_G10   GMO3_G9   GMO3_G8               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 3B   GMO3_G7   GMO3_G6   GMO3_G5   GMO3_G4   GMO3_G3   GMO3_G2   GMO3_G1   GMO3_G0                            
     [0520]                               1.61 GMOFST_B0 (offset = 0 × 3C, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMOFST_B0   X   X   X   X   X   GMO0_B10   GMO0_B9   GMO_B8               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 3C   GMO0_B7   GMO0_B6   GMO0_B5   GMO0_B4   GMO0_B3   GMO0_B2   GMO0_B1   GMO0_B0                            
     [0521]                               1.62 GMOFST_B1 (offset = 0 × 3D, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMOFST_B1   X   X   X   X   X   GMO1_B10   GMO1_B9   GMO1_B8               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 3D   GMO1_B7   GMO1_B6   GMO1_B5   GMO1_B4   GMO1_B3   GMO1_B2   GMO1_B1   GMO1_B0                            
     [0522]                               1.63 GMOFST_B2 (offset = 0 × 3E, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMOFST_B2   X   X   X   X   X   GMO2_B10   GMO2_B9   GMO2_B8               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 3E   GMO2_B7   GMO2_B6   GMO2_B5   GMO2_B4   GMO2_B3   GMO2_B2   GMO2_B1   GMO2_B0                            
     [0523]                               1.64 GMOFST_B3 (offset = 0 × 3F, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               GMOFST_B3   X   X   X   X   X   GMO3_B10   GMO3_B9   GMO3_B8               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 3F   GMO3_B7   GMO3_B6   GMO3_B5   GMO3_B4   GMO3_B3   GMO3_B2   GMO3_B1   GMO3_B0                            
     [0524]                               1.65 CSC0 (offset=0x40, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               CSC0   CSC_RY7   CSC_RY6   CSC_RY5   CSC_RY4   CSC_RY3   CSC_RY2   CSC_RY1   CSC_RY0               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0×40   CSC_GY7   CSC_GY6   CSC_GY5   CSC_GY4   CSC_GY3   CSC_GY2   CSC_GY1   CSC_GY0                             D[15..8]:CSC_RY[7..0]   Set a coefficient of R in  for calculating Y in a RGB2YCbCr           conversion matrix.       D[7..0]:CSC_GY[7..0]   Set a coefficient of G in  for calculating Y in a RGB2YCbCr           conversion matrix.                             [         Y           Cb           Cr         ]     =       [         CSC_RY       CSC_GY       CSC_BY           CSC_RCB       CSC_GCB       CSC_BCB           CSC_RCR       CSC_GCR       CSC_BCB         ]          [           R   in               G   in               B   in           ]                                  
     [0525]                               1.66 CSC1 (offset = 0 × 41, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               CSC1   CSC_BY7   CSC_BY6   CSC_BY5   CSC_BY4   CSC_BY3   CSC_BY2   CSC_BY1   CSC_BY0               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 41   CSC_RCB7   CSC_RCB6   CSC_RCB5   CSC_RCB4   CSC_RCB3   CSC_RCB2   CSC_RCB1   CSC_RCB0                                    
     [0526]                               1.67 CSC2 (offset = 0 × 42, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               CSC2   CSC_GCB7   CSC_GCB6   CSC_GCB5   CSC_GCB4   CSC_GCB3   CSC_GCB2   CSC_GCB1   CSC_GCB0               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 42   CSC_BCB7   CSC_BCB6   CSC_BCB5   CSC_BCB4   CSC_BCB3   CSC_BCB2   CSC_BCB1   CSC_BCB0                                    
     [0527]                               1.68 CSC3 (offset = 0 × 43, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               CSC3   CSC_RCR7   CSC_RCR6   CSC_RCR5   CSC_RCR4   CSC_RCR3   CSC_RCR2   CSC_RCR1   CSC_RCR0               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 43   CSC_GCR7   CSC_GCR6   CSC_GCR5   CSC_GCR4   CSC_GCR3   CSC_GCR2   CSC_GCR1   CSC_GCR0                                    
     [0528]                               1.69 CSC4 (offset = 0 × 44, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               CSC4   CSC_BCR7   CSC_BCR6   CSC_BCR5   CSC_BCR4   CSC_BCR3   CSC_BCR2   CSC_BCR1   CSC_BCR0               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 44   X   X   X   X   X   X   X   X                            
     [0529]                               1.70 C_SUP0 (offset = 0 × 45, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               C_SUP0   X   X   X   X   X   X   X   X               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 45   X   X   X   X   X   X   X   CSUP_EN                                            
     [0530]                               1.71 C_SUP1 (offset = 0 × 46, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               C_SUP1   CSUP_TH7   CSUP_TH6   CSUP_TH5   CSUP_TH4   CSUP_TH3   CSUP_TH2   CSUP_TH1   CSUP_TH0               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 46   CSUP_G7   CSUP_G6   CSUP_G5   CSUP_G4   CSUP_G3   CSUP_G2   CSUP_G1   CSUP_G0                                    
     [0531]                               1.72 C_OFST (offset = 0 × 47, R/W)                                                                    Register Name   D15   D14   D13   D12   D11   D10   D9   D8               C_OFST   OFST_CB7   OFST_CB6   OFST_CB5   OFST_CB4   OFST_CB3   OFST_CB2   OFST_CB1   OFST_CB0               Offset   D7   D6   D5   D4   D3   D2   D1   D0               0 × 47   OFST_CR7   OFST_CR6   OFST_CR5   OFST_CR4   OFST_CR3   OFST_CR2   OFST_CR1   OFST_CR0                                    
     [0532] 16. Burst Mode Compression/Decompression Engine Preferred Embodiments  
     [0533] The preferred embodiment DSC engine includes an improved Burst Capture function with real-time processing, without compromise in the image resolution as compared to the regular capture mode. The Burst Capture Mode is the use of dedicated compression and decompression engine  108  for an increased burst capture sequence length. A sequence of CCD raw image frames is first stored in SDRAM  160  by using Compression engine  108 . Then, as an off-line process, the image pipeline of regular capture mode retrieves the CCD raw images from SDRAM  160 , processes them sequentially, and finally stores them back as JPEG files in the SDRAM. The Animated Playback Mode can display these JPEG files.  
     [0534] Burst mode compression/decompression engine  108  includes differential pulse code modulation (DPCM) and Huffman coding using the same tables as the entropy-coding of DC coefficients in baseline JPEG compression. Engine  108  uses the sample Huffman table in the JPEG standard for chrominance DC differential data. Engine  108  also provides the inverse transforms as illustrated in FIG. 13. Fixed Huffman Table (JPEG Huffman table for Chrominance DC coefficients):  
                                           Category (SSSS)   {circumflex over (D)} l     Code Length   Codeword                                                0   0   2   00       1   −1, 1   2   01       2   −3, −2, 2, 3   2   10       3   −7, . . . , −4, 4, . . . , 7   3   110       4   −15, . . . , −8, 8, . . . , 15   4   1110       5   −31, . . . , −16, 16, . . . , 31   5   11110       6   −63, . . . , −32, 32, . . . , 63   6   111110       7   −127, . . . , −64, 64, . . . , 127   7   11111110       8   −255, . . . , −128, 128, . . . , 128   8   111111110       9   −511, . . . , −256, 256, . . . , 511   9   1111111110       10   −1023, . . . , −512, 512, . . . , 1023   10   11111111110       11   −2047, . . . , −1024, 1024, . . . , 2047   11   111111111110       12   −4095, . . . , −2048, 2048, . . . , 4095   12   1111111111110                  
 
     [0535] The encoder has four look-up tables: Huffman code (13×2-byte entries), Huffman code length table (13×1-byte entries), low bit mask to generate variable-length bit stream (32×4-byte entries), nad log table (256×1-byte entries). The Huffman tables are not programmable for simplicity, although alternative embodiments could include programmable Huffman tables.  
     [0536] The Huffman decoder performs the inverse function of the Huffman encoder and has five look-up tables: max code comparison table (13×2-byte entries), Min code comparison table (13×2-byte entries), decoded Huffman symbol pointer (13×1-byte entries), decoded Huffman symbol table (13×1-byte entries), and bit position mask (32×4-byte entries).  
     [0537] The lossy mode compression just discards the least significant bit (LSB) or the two least significant bits of each coefficient.  
     [0538] 17. Playback Synchronization  
     [0539] A problem involved in playback of audio-visual bitstreams is how to synchronize audio with video signal. The preferred embodiments play the audio bitstream seamlessly in the background in real-time with the audio encoded by using the simple coding standards like ITU-T G.711 and Microsoft 16-bit PCM. By using an interrupt service routine, about 0.1% of the DSP resources is enough to output audio in real time through (multichannel) buffered serial ports; see FIG. 1 b . Therefore, the preferred embodiment must realize the video decoding in synchronization to the audio playback.  
     [0540] For clarity, assume that both audio and video are captured in full speed (real-time with 8K sample/s for audio and 30 frame/s for video). Audio is played back as samples. However, video is displayed in the granularity of frames. Thus the synchronization problem is caused by the fact that the video decoding could be faster or slower than the real-time requirement. If the video decoding is too fast, a certain amount of delay slots has to be inserted to slow down the decoding. Contrarily, if the video decoding is too slow, some video frames must be skipped to catch up with the real-time audio playback.  
     [0541] The preferred embodiments handle both cases. Especially in the case of slow video decoding, the preferred embodiments can properly select and skip the frames in an optimal manner. Note that the preferred embodiment is described for video bitstreams without bi-directional coded frames (B-frames).  
     [0542]FIG. 46 a  depicts the synchronization between audio and video. The first video frame is pre-decoded before beginning audio-video playback. Since the video is displayed in the granularity of frames, the synchronization points are located at the video frame boundaries, i.e. {t=0, ΔT, 2ΔT, 3ΔT . . . }. Here AT is the duration of a frame, which is defined as:  
     ΔT=1fp  (1)  
     [0543] where fp is the frame-rate used for the video sequence.  
     [0544] Audio and video could lose synchronization when the video decoding speed is not fast enough. As illustrated in FIG. 46 a , when the decoding of video frame 2 has not finished in time (Td2&gt;ΔT), the audio-video playback loses synchronization after displaying video frame 1. Here {Tdm, m=0,1,2, . . .} denotes the decoding time used for decoding video frame m.  
     [0545] With insufficient video playback speed, the only way to maintain a reasonable synchronization between audio and video is to skip video frames properly. In FIG. 46 b , video frame  2  is skipped (and frame  1  repeated) so that synchronization can be reacquired at frame  3 .  
     [0546] A preferred embodiment circular buffer scheme is illustrated in FIG. 47. The video decoder is connected to one side of the circular buffer, the display is connected to the other side. The circular buffer has a size of N video frames. There are two registers associated with each frame buffer of the circular buffer: the first register contains TP n , n=0, 1, 2, 3, . . . , N−1 which indicates the presumptive presentation time of the video frame stored in buffer n, and the second register contains S n , n=0, 1, 2, 3, . . . , N−1 which signals whether the frame in buffer n is ready for display (1 for ready, 0 for not ready). Of course, the value of TP n  is a multiple of ΔT. Buffer switching for display also occurs at frame boundaries (i.e. at time t=mΔT, m=0, 1, 2, . . . ). Because the preferred embodiments use a circular buffer containing N frames, all the indices ( . . . , n−1, n, n+1, . . . ) should be regarded as modulo-N indices.  
     [0547] Suppose the time after decoding the current video frame is T. The decoded current frame is stored in buffer n−1 in FIG. 47. Therefore, the buffer to be used for storing the next frame in FIG. 47 is buffer n.  
     [0548] Determine the current position in the bitstream: the frame index m of the current decoded frame is defined as  
       m=TP   n−1   /ΔT   (2)  
     [0549] Determe the decoding starting time of the next frame: since the frame in the buffer n is to be displayed during the time interval of {TP n ≦t&lt;TP n+1 }, buffer n is not available for decoding the next frame until TP n+1 . Therefore, the decoding starting time of the next frame Ts is:  
     Ts=max{T,TP n+1 }  (3)  
     [0550] Determine the next frame to be decoded: let {circumflex over (T)}d be the estimated time for decoding the next frame, the presentation time of the next frame must satisfy:  
       {             TP   n     &gt;     Ts   +       T   ^        d                     TP   n     ≥       TP     n   -   1       +     Δ                 T                             
 
     [0551] The above conditions imply that the decoding of the next frame is finished before its presentation time, and the next frame is located at least a frame after the current frame in the bitstream. Because TP n  must be a multiple of ΔT, the next frame that can be synchronized to audio satisfies the conditions:  
       {             TP   n     =     Δ                   T              [         Ts   +       T   ^        d         Δ                 T       +   0.5     ]                     TP   n     ≥       TP     n   -   1       +     Δ                 T                             
 
     [0552] where [•] denotes integer part by truncation.  
     [0553] Therefore, the presentation time of the next frame is determined by:  
               TP   n     =     max                   {       Δ                   T              [         Ts   +       T   ^        d         Δ                 T       +   0.5     ]       ,       TP     n   -   1       +     Δ                 T         }               (   4   )                       
 
     [0554] There are different methods to estimate {circumflex over (T)}d, such as using statistical estimation based on prior decodings or frame parameters. One preferred embodiment simply uses the actual decoding time of the most recently decoded frame of the same picture coding type (I-frame or P-frame) plus a certain amount of safety margin as the estimated decoding time for the next frame.  
     [0555] The frame index m′ of the next frame to be decoded can thus be computed as:  
       m′=TP   n   /ΔT   (5)  
     [0556] Then the number of frames Δm to be skipped from the current position is determined by:  
     Δ m=m′−m− 1  (6)  
     [0557] Equation (2) to (6) make up of the basic control operations for updating the circular buffer.  
     [0558] The preferred embodiments use the circular buffer scheme to realize synchronization. There are two parts: the video decoder buffer switch control and the display buffer switch control. FIG. 48 demonstrates the flowchart of the video decoder buffer switch control, which contains two stages: initialization and playback.  
     [0559] Initialization: in the circular buffer initialization, N f (1≦N f ≦N) video frames are decoded before starting playback. As shown in the dashed box in FIG. 48, there are four steps for the initialization:  
     [0560] step 0: set all the presentation time registers {TP n , n=0, 1, 2, 3, . . . , N−1} and the status registers {S n ,n=0, 1, 2, 3, . . . , N−1} to zero, switch the video decoder to buffer 0 (i.e. n=0), point to the beginning of the video bitstream (i.e. m′=Δm=0), set time to zero (i.e. t=0)  
     [0561] step 1: set the related status register S n  to 1, skip Δm video frames, decode frame m′, store the decoded frame in buffer n. (Recall on the first pass through the loop, n=0, m′=0, so the first frame is decoded and stored in buffer 0.)  
     [0562] step 2: set the decoding start time Ts to t, switch to the next buffer (i.e. n++), update TP n , m′, Δm according to equations (4), (5), and (6).  
     [0563] step 3: check whether the number of decoded frames reaches the pre-set frame number N f . If true, go to playback, otherwise, loop to step 1.  
     [0564] Playback: there are six steps involved in updating the circular buffer during the playback.  
     [0565] step 0: switch display to buffer 0, enable display, reset time to zero (i.e. t=T=0), switch the video decoder to buffer N f (i.e. n=N f )  
     [0566] step 1: if the whole video sequence is decoded, stop decoding, otherwise, go to step 2.  
     [0567] step 2: update Ts, TP n , m′ and Δm according to equations (3), (4), (5), and (6).  
     [0568] step 3: wait until time reaches Ts (i.e. t&gt;Ts), go to step 4.  
     [0569] step 4: set the related status register Sn to 0, skip Δm video frames, decode frame m′, store the decoded frame in buffer n.  
     [0570] step 5: if the frame decoding finishes in time (i.e. t&lt;TP n ), set S, to 1 to indicate the decoded frame is ready for display, set T to t, switch the video decoder to the next buffer (i.e. n++). Otherwise, set T to t, add DT to the estimated  {circumflex over (T)} d (i.e.  {circumflex over (T)} d+= DT  with  DT = N   d Δ T , intentionally skip N d (0≦N d ) more frames in the next stage), set the current frame index m to m′. Go to step 1. Note that N d  is a parameter to control the screen freezing time before resuming the synchronization.  
     [0571] Users can freely decide the circular buffer size (N), the initial time delay (N f ) for the playback as well as the screen freezing time (N d ). Obviously, the minimum buffer size is 3 video frames (i.e. N=3), the least time delay is one video frame (i.e. N f =1). However, in the case of insufficient video decoding speed, it is strongly recommended to decode N−1 frames (i.e. N f =N−1) during the circular buffer initialization, so that the video decoder can gain the maximal room to catch up with the audio real time playback.  
     [0572] Display buffer switch control: the display buffer switch control is carried out in parallel to the video decoder buffer switch. The preferred embodiment checks the display buffer switch at video frame boundaries: t=mΔT, m=0, 1, 2, . . . . Suppose the display is currently showing the video frame in buffer n−1, it switches to the next buffer, i.e. buffer n, if and only if the current time (t≧TP n ) and (S n =1) holds. Otherwise, it is connected to buffer n−1. Here, if (t≧TP n ) and (S n =0), it means the decoder has not finished decoding of the frame in time. In this case, the video frame in buffer n has been discarded, the decoder is decoding the conservatively selected next frame to update buffer n again, the display should keep displaying the frame in buffer n−1 until (t≧TP n ) and (S n =1) holds.  
     [0573] In summary, the preferred embodiment provides a way to realize the synchronization between audio and video when playing back by using software or firmware.  
     [0574] 18. Variable Length Decoding  
     [0575] Variable Length Decoding (VLD) is involved in decoding bitstreams which are generated by using Variable Length Encoding (VLC) at encoder; see FIG. 1 b  item  126 . Because of VLC, the number of bits used for coding units varies from unit to unit. Therefore, a decoder does not know the number of bits used for a coding unit before having decoded it. This makes it essential for a decoder to use a bitstream buffer during the decoding process.  
     [0576] In video coding, for example, a frame to be encoded is decomposed into a set of macroblocks (see FIG. 49). Under the consideration of the smallest memory requirement, a coding unit here is normally defined as macroblock, which consists of a 16×16 pixel luminance area and the corresponding chrominance areas depending on the chroma format (4:2:0, 4:2:2:, or 4:4:4). Certainly, a slice (a row of macroblocks in a frame) or even the frame itself can be treated a coding unit if there is enough memory.  
     [0577]FIG. 50 depicts the video playback on a preferred embodiment digital still camera (DSC). In DSC applications, the video bitstream is pre-captured and stored on the high-capacity SDRAM, and the video decoder is built on the DSP. Since it is extremely expensive for the decoder to directly access the SDRAM, an on-chip bitstream buffer is opened on the DSP internal memory. The bitstream is first loaded from SDRAM to the bitstream buffer through the SDRAM, then the decoder uses the bitstream in the bitstream buffer to reconstruct video. Since the bitstream loading is achieved by using DMA (Direct Memory Access), which can run in the background without intervention of a CPU, the bitstream loading overhead is mainly due to time used for setting up registers for the DMA transfer.  
     [0578] There are two basic requirements in terms of bitstream buffer management. First of all, the buffer size should be big enough to cover the worst case. For example, in video coding, the theoretically maximal number of bits for encoding a macroblock could be 256 words (one word here is defined as two bytes). Although this worst case is very rare, the bitstream buffer size has to be 256 words in order to be at the safe side. Secondly, the bitstream buffer should never underflow, that is, the buffer management should guarantee that the bitstream for a coding unit is available when it is being decoded.  
     [0579] There are different schemes to satisfy the second requirement. The simplest one would be to check the decoding position in the bitstream buffer at each buffer access. The bitstream buffer is re-filled whenever the decoding position is out of the valid buffer range. Because the decoding is a bit by bit operation, this scheme is not realistic: it spends too much overhead in deciding when to re-fill the buffer.  
     [0580] A realistic scheme is the linear shifting buffer scheme as shown in FIG. 51 a . In this scheme, the bitstream buffer is linearly accessed by the decoder from left to right, after decoding a unit the rest of the bitstream is shifted forward to the beginning of the buffer, then the buffer is re-filled to “full” before decoding the next unit. In FIG. 51 a , Ps and Pd denote the current decoding position and the bitstream end position in the bitstream buffer, respectively.  
     [0581] This buffer scheme has two disadvantages. First, since the buffer size is much larger than the average number of bits of the decoding units, a lot of time will be spent on the bitstream shifting. For instance, in video decoding the buffer size is 256 words to cover the worst case, but on average a unit may only use 16 words, this means about 240 words of shifting for each unit. The second disadvantage is that it requires a bitstream loading after decoding each unit; this costs additional overhead because time has to spent on issuing the DMA transfers.  
     [0582] A better buffer management scheme is so-called quasi-circular buffer scheme as shown in FIG. 51 b . In this scheme, the decoder accesses the bitstream buffer in a circular manner. This avoids the bitstream shifting required by the linear buffer scheme. There are two cases after decoding a unit. This first case is in the lefthand portion of FIG. 51 b : the rest of bitstream is located in the middle of the buffer. In this case, the buffer is filled by loading the bitstream twice, one for the right end followed by the other one for loading the left end. (Note: if the bitstream loading can write the bitstream into the bitstream buffer in a circular manner, only one load is needed; however, this is not always the case.) The second case is shown in the righthand portion of FIG. 51 b , in which only the middle of the buffer needs to be filled.  
     [0583] The quasi-circular buffer scheme is much more efficient than the linear shifting buffer because it avoids bitstream shifting, but it still suffers from a disadvantage that one or two bitstream loads are needed after decoding each unit. The following preferred embodiment hybrid circular-double buffer scheme solves this problem.  
     [0584]FIG. 52 status 0 shows a hybrid circular-double buffer containing two buffers of equal size; namely, the left buffer and the right buffer. There is a flag for each buffer to indicate the buffer fullness (“full”/“not-full”). Ps points to the current decoding position after decoding a unit. In terms of buffer size, each buffer covers the worst case of decoding coding units, this makes the hybrid buffer size twice of a linear shifting buffer or a quasi-circular buffer. Unlike a traditional double buffer, the two buffers here have a continual memory allocation, i.e. the left buffer is directly followed by the right buffer in the memory map. The decoder accesses the hybrid buffer in a circular manner.  
     [0585] The preferred embodiment hybrid buffer operates through the following four statuses:  
     [0586] Status 0: the initialization status, both the left and right buffers are fully loaded and set to “full”, Ps points to the beginning of the hybrid buffer.  
     [0587] Status 1: after decoding the first unit, change the left buffer flag to “not-full”.  
     [0588] Status 2: after decoding a unit, if the current decoding position Ps is in the right buffer and the left buffer flag is “not-full”, fully load the left buffer and set the left buffer flag to “full”. In addition, if the right buffer flag is “full”, change it to “not-full”. Otherwise, no action is taken.  
     [0589] Status 3: after decoding a unit, if the current decoding position Ps is in the left buffer and the right buffer flag is “not-full”, fully load the right buffer and set the right buffer flag to “full”. If the left buffer flag is “full”, change it to “not-full”. Otherwise, no action is taken.  
     [0590] Taking the preferred embodiment platform (e.g., FIG. 1 b ) as an example (where data is in 16-bit units), define the following data type:  
                                      typedef struct           bitstream {       SInt bit_ptr;   /* current bit position (0 ˜ 16)       */       SInt Ps;   /* current decoding position in bitstream       buffer */       SInt left_flag   /* left buffer flag “full / not-full”       */       SInt right_flag   /* right buffer flag “full / not-full”       */       USInt *databuf;   /* bitstream buffer       */       Long Addr_SDRAM;   /* bitstream address in SURAM       */       }Bitstream;                  
 
     [0591] The pseudo code shown in Table 1. describes the hybrid circular-double buffer scheme. Function BufferInitialization( ) is called only once at the beginning of decoding, while function BitstreamBufferUpdate( ) is called after decoding each coding unit, it automatically updates the buffer flags and re-loads the buffers if the conditions become true. In Table 1 BUFSIZE stands for the buffer size of the hybrid circular-double buffer. 
 
 
 
     [0592] As it can be seen in BitstreamBufferUpdate( ) in Table 1, the left buffer or right buffer is not reloaded after decoding each unit, but is loaded only if the opposite buffer (left/right) is in use and its buffer flag is “not-full”. This greatly reduces the number of buffer loads. Consider the video coding as an example. This needs BUFSIZE of 512 words if a macroblock is the unit, the average bitstream size of a unit is assumed to be 16 words. Because the linear shifting buffer and the quasi-circular buffer re-fill the buffer after decoding each unit, the average loading length for those two schemes is also 16 words. Compared with the fixed loading length of 256 words in the hybrid circular-double buffer scheme, the preferred embodiment reduces the loading overhead by a factor about 16 (i.e. 256/16).  
     [0593] Mini-experiments compared the three buffer schemes discussed above. The video sequence used was coastguard (352×288, 300 frames, 4:2:0). The bitstream is generated by using a MPEG1 video encoder. The target bit-rate is 3 Mbit/s, I-frame only. The same decoder with three different buffer schemes are used to decode the same bitstream, the buffer loading count and word shifting count are recorded during the decoding. The performance comparison among the three buffer schemes is listed in Table 2. As shown in Table 2, for each macroblock the linear shifting buffer scheme requires one buffer load, and on average about 240 words of shifting. The quasi-circular buffer scheme needs slightly more buffer loads (1.06 load/macroblock) but no shifting. The preferred embodiment hybrid circular-double buffer scheme used only about 0.0619 buffer load per macroblock. On the preferred embodiment platform of FIG. 1 b  in particular, the preferred embodiment scheme provides a cycle count reduction ratio of about 113 and 17 in comparison to the linear shifting buffer scheme and the quasi-circular buffer scheme, respectively.  
               TABLE 2                          Performance comparison among three buffer schemes on TMS320DSC21 platform                                     Quasi-circular   Hybrid circular-double           Linear shifting buffer   buffer   buffer                                         Buffer size (words)   256   256   512       Number of loads per   1.00   1.06   0.0619       macroblock       Number of word shifting per   240.15   0   0       macroblock       Overhead per load (cycles)   80   80   80       Cycle count per word   2   2   2       shifting       Total cycles used for   560.30   84.72   4.95       bitstream buffer per       macroblock       Cycle count ratio vs. the   113.19   17.12   1.00       hybrid circular-double       buffer scheme                  
 
     [0594] 19. Onscreen Display and Graphics Acceleration  
     [0595] The Onscreen display (OSD) module  105  is responsible for managing OSD data from different OSD windows and blending it with the video. It reads OSD data from SDRAM  160 , and outputs to NTSC/PAL encoder  106 . The OSD module defaults to standby mode, in which it simply sends video to NTSC/PAL encoder  106 . After being configured and activated by ARM CPU  130 , the OSD module reads OSD data and mixes it with the video output. ARM CPU  130  is responsible for turning on and off OSD operations and writing the OSD data to the SDRAM. FIG. 15 shows the block diagram of the OSD module and affiliated other items. The various functions of the OSD are described in the following paragraphs.  
     [0596] OSD data storage. The OSD data has variable size. In the bitmap window, each pixel can be 1, 2, 4, or 8 bits wide. In the YCrCb 4:2:2 window, it takes 8-bit per components, and the components are arranged according to 4:2:2 (CbNY/CrY . . . ) format. In the case where RGB graphics data needs to be used as OSD, the application should perform software conversion to Y/Cr/Cb before storing it. The OSD data is always packed into 32-bit words and left justified. Starting from the upper left corner of the OSD window, all data will be packed into adjacent 32-bit words.  
     [0597] Setting up an OSD window. An OSD window is defined by its attributes. Besides storing OSD data for a window into SDRAM by ARM CPU  130 , the application program also needs to update window attributes and other setup in the OSD module as described in the following subsections.  
     [0598] Location register. The Location register contains X and Y locations of the upper left and lower right corners of each window. The application program needs to set up the CAM and enable selected OSD windows; see FIG. 16.  
     [0599] Color look up tables. The OSD has the fixed 256-entry color look up table (CLUT). The CLUT is used to convert bitmap data into Y/Cr/Cb components. In the case of 1,2 or 4 bitmap pixels, the CLUT can be determined by CLUT registers.  
     [0600] Blending and transparency. Color blending on the pixel level is also supported. This feature is available for the bitmap displays only (Window1,2). If the window color blending is enabled, the amount of blending of each pixel is determined by the blending factor. As shown in the following table, the window blending supports 5 different levels, according to the selected blending factor. The hardware also supports a transparency mode with bitmap. If transparency is enabled, then any pixel on the bitmap display that has a value of 0 will allow video to be displayed. Essentially, 0-valued pixels are considered the transparent color, i.e. the background color will show through the bitmap. The Table shows the connection between transparency and blending on the same window.  
                                                   Blend   OSD window               Transparency   Factor   contribution   Video contribution                                                    OFF   0   0   1               1   ¼   ¾           2   ½   ½           3   ¾   ¼           4   1   0       ON       if pixel value = 0   if pixel value = 0           0   0   1           1   ¼   ¾           2   ½   ½           3   ¾   ¼           4   1   0                  
 
     [0601] Hardware cursor. A rectangular shape is provided using hardware window1. With window1, the cursor always appears on top of other OSD Windows. The user can specify the size, color of the shape. When hardware window1 is designated as the cursor, only two windows are available for the OSD application. If a hardware cursor is not used, then the application can use window1 as a regular hardware window. FIG. 17 shows an example of the hardware cursor.  
     [0602] 20. DSP Subsystem  
     [0603] The DSP subsystem consists of C54x DSP, local memory blocks, iMX and VLC accelerators, shared image buffers, and the multiplexers implementing the sharing.  
     [0604] C54x is a high performance, low power, and market proven DSP. cDSP hardware and software development tools for C54x are also very mature.  
     [0605] The DSP carries out auto exposure, auto focus, auto white-balancing (AE/AF/AWB) and part of the image pipeline tasks. It also handles SDRAM transfer and drives the accelerators to implement the rest of image processing and image compression tasks. Flexibility and ease of programming in the DSP enables camera makers to refine the image processing flow, adjust quality-performance tradeoffs, and introduce additional features to the camera.  
     [0606] The configurable DSP (cDSP) design flow is adopted to allow flexibility and design reuse. The memory blocks time-shared among DSP and accelerators are large enough for one processing unit (16×16 pixels) and provide zero-wait state access to DSP.  
     [0607] Features  
     [0608] Fixed-point Digital Signal Processor  
     [0609] 100 MIPs LEAD2.0 CPU  
     [0610] On-module RAM 32Kx16 bit  
     [0611] (4 blocks of 8K×16 bit dual access program/data RAM)  
     [0612] Multi-Channel Buffered Serial Ports (McBSPs)  
     [0613] ARM can access RAM via Enhanced 8-bit Host Port Interface  
     [0614] One hardware timer  
     [0615] On-chip Programmable PLL  
     [0616] Software Programmable Wait-State Generator  
     [0617] Scan-based emulation and JTAG boundary scan logic  
     [0618]FIG. 18 a  shows more details on the DSP subsystem and in particular the details of the connection between the DSP and the iMX and VLC. FIG. 18 b  is the memory map.  
     [0619] The shared memory blocks A and B occupy two 2 Kword banks on the DSP&#39;s data memory space. Each block can be accessed by DSP, iMX, VLC, and SDRAM controller depending on static switching controlled by DSP. No dynamic, cycle-by-cycle, memory arbitration is planned. DSP&#39;s program should get seamless access of these memory blocks through zero-wait-state external memory interface.  
     [0620] The configuration memory blocks, for iMX coefficient, iMX command, VLC Q-matrix, and VLC Huffman table, also connect to DSP&#39;s external memory interface. They are also statically switched between the specific module and DSP. Typically at power-up or at initial stage of camera operation mode, these memory blocks are switched to DSP side so DSP can set up the appropriate configuration information for the operation. Then, they are switched over to iMX and VLC for the duration of operation.  
     [0621] Imaging Extension (iMX)  
     [0622] iMX, imaging extension, is a parallel MAC engine with flexible control and memory interface for extending image processing performance of programmable DSPs. iMX is conceived to work well in a shared memory configuration with a DSP processor, such that flexibility, memory utilization, and ease of programming are achieved. The architecture covers generic 1-D and 2-D FIR filtering, array scaling/addition, matrix multiplications (for color space transform), clipping, and thresholding operations.  
     [0623] For digital still cameras, iMX can be used to speed up  
     [0624] CFA interpolation,  
     [0625] color space conversion,  
     [0626] chroma down-sampling,  
     [0627] edge enhancement,  
     [0628] color suppression,  
     [0629] DCT and IDCT,  
     [0630] Table lookup.  
     [0631] iMX methodology originates from the discipline of parallel processing and high performance computer architecture. The design comprehends the need for a scalable MAC engine. iMX in the first preferred embodiment incorporates 4 MAC units; see FIG. 19. Alternative preferred embodiments upgrade to 8 MAC units or more. Software can be structured so that the hardware upgrade will not incur substantial software changes.  
     [0632] Much flexibility of iMX is due to parameter-driven address generation and looping control. Overall efficiency comes from efficient pipelining control inside iMX as well as the system-level memory buffering scheme. iMX works best for block-based processing. To facilitate this, the datapath needs to connect to data input/output and coefficient memory. iMX contains data input, data output, and coefficient memory ports, and allows arbitration among these ports. This eliminates the need for dedicated memory blocks, and brings more flexibility and better memory utilization on the system level. These memory blocks are accessible as DSP data memory to facilitate data exchange.  
     [0633] There is a separate command memory that feeds a command decode unit in iMX. The command memory should be specified to fit all the accelerated steps in our reference image pipeline algorithm, so that this sequence of commands can be executed with little intervention from DSP.  
     [0634] iMX block diagram appears in FIG. 20. A command decode subblock reads and decodes commands, and drives static parameters, one set per command, to the address generator. Address generator then computes looping variables and data/coefficient/output pointers, and coordinates with execution control, which handles cycle-by-cycle pipelining control. Address generator sends data and coefficient read requests to the arbiter. Arbiter forwards the requests to the data/coefficient memory. Data read back from memory go to the input formatter, which takes care of data alignment and replication. Formatted data and coefficients are then provided to the datapath, which mainly consists of the 4 MAC units. Output from datapath is routed to arbiter for memory write.  
     [0635] iMX communicates to DSP via shared memory (for data input, coefficient, data output, command) and via memory-mapped registers (start command, completion status). All data buffers and memory blocks are single-ported, and are switched to one party or another via static control, rather than on-line arbitration.  
     [0636] In a typical application, DSP would place filter coefficients, DCT/IDCT cosine constants, and lookup tables in the coefficient memory, and put iMX commands in the command memory. DSP then turns over access to these memory blocks to iMX. These memory blocks are sized adequately for our reference design to fit all needed coefficients and commands for a major camera operation mode (e.g., image capture). Any update/reload should occur very infrequently. In case either or both memory blocks run out of space, paging can be performed.  
     [0637] DSP manages the switch network so that, to iMX, there is only one data buffer. During run time, DSP switched the A/B buffers among itself, iMX. VLC, and SDRAM controller to implement data passing.  
     [0638]FIG. 21 illustrates a simple table lookup accelerator with input rounding/clipping capability used to speed up the image pipeline on the DSP. This is carried out with a very simple control structure and datapath.  
     [0639] 21. VLC Engine  
     [0640] VLC accelerator is a coprocessor optimized for quantization and Huffman encode in the context of JPEG compression and MPEG compression. It operates with quantizer matrices and Huffman tables preloaded by DSP, via shared memory blocks. Aggressive pipelining in the design achieves very high throughput rate, above 30 million DCT coefficients for compression.  
     [0641] VLC&#39;s working memory, including quantizer matrices, Huffman tables, and data input/output memory, are all shared memory blocks.  
     [0642] VLC Functionality  
     [0643] Basically, VLC covers Quantization, zigzag scan, and Huffman encode for JPEG encode (baseline DCT, 8-bit sample), with up to 4 quantizer matrices (stored as invq[i,j]=2 16 /q[i,j]) and 2 encode Huffman tables all loadable. Can process one MCU that contains up to 10 blocks. Each block consists of 8×8=64 samples.  
     [0644] Quantization, zigzag scan, and Huffman encode for MPEG-1 video encode. One macroblock, with up to six 8×8 blocks, can be processed. Number of blocks and within them, number of luminance blocks, can be specified. Huffman encode can be bypassed to produce quantized and zigzag-ordered levels.  
     [0645] The accelerator requires memory blocks for input/output buffer, quantization matrices and Huffman encode tables. The memory configuration should be sufficient to support normal encode operations, one JPEG MCU (minimum coding unit), or MPEG macroblock per call.  
     [0646] Both input and output must fit the 2K words (1 word=16-bit) shared memory buffer (A or B). MCU or macroblock has maximally ten 8×8 blocks, or 640 input words. Compressed output data is typically smaller than input size.  
     [0647] JPEG Huffman encode table takes up (12×176)×32-bit, or 384 words per table. JPEG standard allows 2 tables, so taking totally 768 memory words. MPEG tables are hard-wired into VLC and do not take up memory. We have allocated 2K words for the Huffman tables.  
     [0648] The quantizer matrix memory, 512 words by 16-bit, allow for 8 quantizer matrices to coexist, each taking 64×16-bit. JPEG allows for 4 matrices, and MPEG encode requires 2 matrices.  
     [0649]FIG. 22 shows the major subblocks of VLC. Only the encode path is implemented in one preferred embodiment VLC module; alternative preferred embodiments incorporate the decode path into the module.  
     [0650] 22. ARM Subsystem  
     [0651] ARM microprocessor  130  handles system-level initialization, configuration, user interface, user command execution, connectivity functions, and overall system control. ARM  130  has a larger memory space, better context switching capability, and is thus more suitable for complex, multi-tasking, and general processing than DSP  122 . Preferred embodiments integrate an ARM7 cTDMI core; see FIG. 1 b . ARM7 core is specified up to at least 40 MHz. The ARM subsystem will also have a 32 Kbytes local static RAM  132 .  
     [0652] ARM processor  130  is connected to all the DSC peripherals including CCD Controller, TV encoder, preview engine, IrDA, USB, Compact Flash/Smart Media, UART, etc.  
     [0653] ARM processor  130  is involved with the management of CCD incoming raw data and intermediate data to the SDRAM and LCD. Connected to all I/O devices, the ARM manages and is responsible for the smart devices such as USB, IrDA, Compact Flash/Smart Media, and UARTS. The four basic operation modes of PREVIEW, CAPTURE, PLAYBACK, and BURST are initiated by requests from the ARM. The ARM will then monitor the device for completion of the request and in some cases will manage data after the request is completed.  
     [0654] After RESET and before any of the camera operations can occur, the ARM must perform several housekeeping tasks. The initial task is known as the BOOT operation task. This function not only initializes the I/O and peripherals to a known state, it also must prepare, load and start DSP  122 . This sequence begins by reading the DSP boot code from the flash, loading the DSP code memory and then releasing the DSP from its HOLD state. Additional DSP code is loaded into the SDRAM in a format the DSP can then read and overlay into its code space without ARM intervention.  
     [0655] ARM SDRAM Interface  
     [0656] ARM has two types of access to the SDRAM (1) through SDRAM buffer (burst read/write) and (2) direct access to the SDRAM with a higher latency-4 cycle READ, 6 cycle WRITE. The direct access to memory can be word, half word or byte access.  
     [0657] The ARM/SDRAM controller interface also has a 32 byte buffer. The SDRAM burst request first fills this buffer and ARM reads and writes from/to this buffer.  
     [0658] ARM External Memory Interface  
     [0659] ARM  130  connects to the external memory through the External memory interface module. ARM  130  connects to the Compact Flash/Smart media through this interface. ARM  130  also connects to the off chip flash memory through this interface. DMA block (FIG. 1 b ) enhances the ARM to CF/Smart media transfer.  
     [0660] ARM/DSP BOOT Sequence  
     [0661] The DSP BOOT sequence begins after a power up or after a COLD START. In this state, DSP  122  is in a HOLD condition waiting on initialization from ARM  130  . The ARM checks DSP status registers to assure the DSP is in a HOLD state. The ARM programs the DSP boot code data to the DSP code memory from—” the FLASH. The code is organized in logical overlays that allow the ARM to select the proper code for the function needed, in this case BOOT code.  
     [0662] The ARM loads the DSP code using the HPI Bridge (HPIB) interface. This interface can be programmed to access in either 8- or 16-bit width. For BOOT purposes, this will always be a 16-bit access.  
     [0663] After the code is loaded, the ARM signals the DSP to begin by releasing the HOLD. The DSP then begins its reset sequence from an address of DSP 7F80h which is in the DSP RESET vector area. Upon completion of the RESET sequence, the DSP then branches to DSP FF80h, which is the beginning of the BOOT program loaded by the ARM.  
     [0664]FIG. 23 a  shows the data paths used in the ARM/DSP boot sequence as well as data, request and command exchanges discussed later.  
     [0665] Capture Mode  
     [0666] ARM  130  programs CCD controller  102  to capture an image. The CCD controller auto transfers the image data to SDRAM and interrupts the ARM using IRQL when the transfer is complete. The ARM then notifies the DSP the RAW picture data is available to crunch. When the processing of the raw data is complete, the DSP signals the ARM the task is finished.  
     [0667] Preview Mode  
     [0668] The CCD will be programmed for a 30 fps high frame rate but reduced resolution vertically. The reconfiguration of the CCD and TG (timing generator) will cause the raw picture data to go to preview engine  104 . The DSP will post process the data in SDRAM and prepare parameters for FOCUS, EXPOSURE and WHITE BALANCE. The ARM is signaled by the DSP when new adjustment parameters are ready and those corrections are applied by the ARM. The transferring of the correction parameters use the same communication interrupt architecture as previously mentioned and are expected to be at the current frame rate.  
     [0669] Burst Mode  
     [0670] The burst mode timing is based on the ARM clocking the picture rate from application parameters. Similar to a cross between Capture and Preview modes, the ARM programs the CCD for a capture that stores a compressed image into SDRAM through the compression engine. As in Preview mode, the ARM receives adjustment parameters from the DSP to make corrections of FOCUS, EXPOSURE and WHITE BALANCE.  
     [0671] Idle Mode  
     [0672] ARM may use an idle mode to receive correction parameters from the DSP during periods preceding other camera modes. If not in a power down situation, this time of 10-15 frames will allow the DSP-to-ARM correction loop to make auto corrections on FOCUS, EXPOSURE and WHITE BALANCE. This idle mode will simulate Preview mode for the purposes of obtaining a stable correction.  
     [0673] 23. ARM/DSP Communication  
     [0674] The communication between ARM  130  and DSP  122  is via the HPIB (Host Port Interface Bridge). The HPIB physically connects the DSP (a C5409 type DSP) ports and BUSC (BUS Controller)  134 . The ARM accesses the DSP memory by programming the HPIB, opening a 32k-word window into the DSP memory map. The map contains the data structures shared by the ARM and DSP for command request&#39;s, acknowledgements and datagrams.  
     [0675] The HPIB contains five sub-blocks. They are the interface, timing generator, DSP control registers, and interrupt hold sections.  
     [0676] The interface section receives and stores data from BUSC 134 and transfers it to and from the C5409. This interface can be an 8- or 16-bit data path to the C5409 and is 16-bit to the BUSC. An added feature is the ability to exchange the upper and lower byte if programmed to do so.  
     [0677] The timing generator makes signals HBIL and HDS and detects signal HRDY. HBIL is the HPI byte identification signal to the C5409. The HDS is the data strobe signal to the C5409 and the HRDY is the ready signal read from the C5409.  
     [0678] The interrupt hold section will detect the HINT level and make the INTC pulse synchronized with the ARM clock. The module will also set the HOLD port of the C5409 and detect HOLDA.  
     [0679] In 8-bit mode, address data from the ARM will not reach the C5409. The address is used only if the C5409 internal memory is selected. Therefore, the ARM must set the address in the HPIA register before sending or receiving data to the 32 Kword DARAM. The 8-bit mode may also be used for ARM&lt;-&gt;DSP handshaking. The ARM will use the HINT bit in the HPIC register to interrupt the C5409.  
     [0680] In 16-bit mode, the HPIA/HPIC/HPID are not used. The ARM can access the C5409 internal memory as if it exists in the HPIB module. This mode will deliver faster performance, but does not support the HANDSHAKE signals because of these are routed in the HPIC register.  
     [0681]FIG. 23 b  shows the signals and paths for the ARM to reach the C5409 DARAM.  
     [0682]FIG. 23 c  indicates the shared memory map between the ARM (HOST) and the C5409 processor. When the ARM selects the memory area, “DSP Memory”, BUSC takes cs_hpib signal active. The ARM can now access the DSP internal memory (32 kword DARAM+HPIA+HPIC+HPID).  
     [0683] When the ARM selects the “DSP Controller” area, BUSC takes cs_dspc signal active. The ARM is now accessing registers related to the C5409.  
     [0684] 24. Multi-Processing Debugging Environment  
     [0685] The preferred embodiment integrates ARM  130  and DSP  122  and thus multi-processing and thus requires debugging and development support. The preferred embodiment accomplishes this with a single JTAG connector  170  with additional emulation logic as illustrated in FIG. 24.  
     [0686] 25. Input/Output Modules  
     [0687] The input/output module provides the different interfaces with the DSC peripherals as follows.  
     [0688] TV encoder  106  produces NTSC/PAL and RGB outputs for the LCD display and TV.  
     [0689] CCD/CMOS controller  102  generates timing signals VDIHD, can synchronize on externally generated HDND signals (#0 of MODESET register, #0 of SYNCEN register), supports progressive scan and interlaced CCDs, generates black clamping ontrol signals, programmable culing pattern 9CULH, CULV registers), 1 line/2 line alternating fields, MCLK (generated by CCD module), WEN (WRQ on TG, active-high) indicates CCD controller writing data to SDRAM, TG serial port interface (clk, data, TG chip select) is controlled by GIO pins, Iris, mechanical shutter, focus and zoom are controlled by GIO pins.  
     [0690] USB 142 from programmer&#39;s perspective consists of three main parts: FIFO controllers, UDC controller, and UDC core. USB configuration: INTERFACED0 ALT0 ENDPOINT0: CONTROL; INTERFACE0 ALT0 ENDPOINT1: BULKIN; INTERFACE0 ALT0 ENDPOINT1: BULKOUT; INTERFACE1 ALTO ENDPOINT2: ISOIN; INTERFACE2 ALT0 ENDPOINT3: INTERRUPT IN. Buffer configuration: SUB module has six FIFOs inside; each FIFO is of the same construction, except for direction and buffer size; USB module has only one unified memory for all endpoints; buffer sizes are programmable as long as all buffers fit inside the memory.  
     [0691] UART part of I/O block  140 , supports start/stop communication protocol, detects parity errors (supporting dta length of 7 or 8 bits with even, odd, or no parity and 1 or 2 stop bits), has 32 bytes of FIFO for both transmitter and receiver, generates interrupts for a FIFO overflow or a time-out is detected on data receiving. ARM  130  control UART modules. There are seven 16-bit width registers which are accessible from ARM  130  : data transmitter/receiver register (FIFO), bit rate register, mode register, FIFO control register for receiver, FIFO control register for transmitter, line control register, and status register. FIG. 25 is a block diagram.  
     [0692] Compact Flash/Smart Media interface  180  is used to save/store image or user&#39;s data to a compact flash card or smart media; see FIG. 26. The interface supports two kinds of operation modes for register setting and data transfer: memory mapped mode and I/O mode. An ARM  130  interrupt is generated for card detection while a compact flash card is being plugged or unplugged. The pins for both the smart media and the compact flash control interfaces are overlapped and can be switched by ARM  130  depending on product needs; see FIG. 26.  
     [0693] In particular, the compact flash controller has registers mapped to the ARM memory space. The compact flash controller is responsible for generating the related control signals to the interface pins, and writes at 420 KB/s and reads at 2.0 MB/s. SDRAM can be utilized for storing at least one picture and an attempt to write to the compact flash with a big sector count, as done in a DOS machine, will invoke the fast write performance.  
     [0694] In contrast, the smart media controller has five register settings: command register, address1 register, address2 register, address3 register, and data port register. These five registers are mapped to the ARM memory space, and smart media controller will generate the related signals for different register access automatically.  
     [0695] Audio input/output may be through the serial port of I/O block  140  with DSP buffering.  
     [0696] Infrared data access (IrDA) is supported by a fast FIR core and part of I/O block  140 .  
     [0697] Block  140  also contains general purpose input/output which can support items such as CCD/CMOS imager module control for tuning AGC gain and electronic shutter, RTC control, battery power detection which can generate inner interrupt to the ARM for appropriate system response, camera lens motor control for focus and zoom, a user keypad input, LED indicators, flash light control, and power management control.  
     [0698] iMX Programming  
     [0699] DSP  122  instructs iMX  124  to perform tasks by sending iMX commands. These commands can be complex jto understand and contain many parameters that are fixed in the inner loops. The dieal model is to provide separate command building and command-transfer routines to the DSP progarmmer, so that the commands can be pre-constructed outside the loop, and transferred to iMX as generaic data memory moves inside the loop. Commonly used iMX commands are prepackaged in C code to ease the programming.  
     [0700] ARM/DSP Task Allocation  
     [0701] ARM  130  runs an operating system such as Windows CE, controls low frequency, synchronous input/output (such as to a compact flash card (CFC), and controls user interactions which also are slow and all the peripheral modules control preview engine, burst mode compression, 1V encoder, CCD controller, USB, CF, IrDA, etc.  
     [0702] DSP  122  runs an operating system such as SPOX, controls all real-time functions (auto focus, auto exposure, auto white balance), real-time input/output (audio IO, modem IO), real-time applications (e.g., audio player), computational expensive signal processing tasks (image pipeline, JPEG 2000, image stitching).  
     [0703] Audio Player  
     [0704] Portable digital audio players are expected to be one of the most popular consumer products. Currently the MP-3 player based on MPEG-1 Layer 3 ausio compression standard is growing rapidly in portable audio market while MPEG-2 MC and Doby AC-3 are alternative digital audio coding formats to be considered as emerging standards. Thus the preferred embodiments&#39;s programmability permits inclusion of digital audio player functions. The audio can be input via flash memory, PC, etc. and the decoded can be output on the serial port. The decoding program can be loaded from flash memory, ROM, etc.