Patent Publication Number: US-7221378-B2

Title: Memory efficient method and apparatus for displaying large overlaid camera images

Description:
This invention relates to a memory efficient method and apparatus for displaying large overlaid camera images, particularly for use in an electronic computing device such as a cellular telephone. 
   BACKGROUND 
   It is common in electronic computing systems or devices having graphics displays, such as cell phones, to overlay one image on top of another. Typically, the main, or underlying image is covered by the overlay image so that, where the main and overlay images overlap, the main image is not visible. One particular example of overlaying occurs where it is desired to provide a picture frame or border around an image. Similarly, other geometric shapes or text may be superimposed over an image. 
   In such applications, the overlay image defines pixels that are transparent and pixels that are not transparent, or opaque. Where the overlay image is transparent, the underlying main image is visible, and where the overlay image is opaque, only the overlay image is visible. Pixels in the overlay image can be defined as being transparent or opaque by the use of appropriate coding. For example, particular colors can be defined as being transparent while other colors can be defined as being opaque. 
   In many cell phones, the main image is produced by a camera. A graphics controller includes a camera interface for interfacing to the camera. The graphics controller resizes the camera data and converts the camera data from YUV to RGB color format for storage in an on-board display buffer. The graphics controller also receives an overlay image from a host CPU and stores the overlay image data in the display buffer as well. The overlay image and the main image may or may not be the same size. 
   The graphics controller fetches a main image pixel and a corresponding overlay image pixel from the display buffer and combines the two pixels. This sequence is performed for every pixel in the main image, to form composite image data The composite image data are provided to an interface for interfacing to a display device. 
   One problem with the methodology described above is that it requires a significant amount of memory, because both images are stored in the display buffer. This problem is exacerbated if the overlay image is the same size as the main image, so that twice the amount of memory required for the main image is required for the composite. Another problem with the methodology is that a main image which is too large to fit within the display buffer cannot be displayed. 
   Accordingly, there is a need for a memory efficient method and apparatus for displaying large overlaid camera images that solves these problems. 
   SUMMARY 
   A memory efficient method and apparatus for displaying large overlaid camera images according to the invention combines main image data and overlay image data to form composite image data for rendering on a graphics display device. According to one aspect of the invention, the overlay image data are stored in a memory, fetched, up-scaled, and then combined with the main image data to form the composite image data. According to another aspect of the invention, the overlay image data are stored in a memory, fetched, and then combined with main image data streamed from a source of the main image data. 

   
     BRIEF DESCRIPTION OF THE DRAWINGS 
       FIG. 1  is block diagram of a prior art electronic computing system. 
       FIG. 2  is a block diagram of an electronic computing system for displaying overlaid camera images according to the present invention. 
       FIG. 3  is a block diagram of an alternative electronic computing system for displaying large overlaid camera images according to the present invention. 
       FIG. 4  is a schematic representation of up-scaling down-scaled overlay image data according to the present invention. 
   

   DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
     FIG. 1  shows a prior art electronic computing system  10  for comparison with systems described herein according to the present invention. The system  10  includes a graphics controller  12  that interfaces to (a) a camera  14 , (b) a host CPU  16 , and (c) a graphics display device  18  such as an LCD panel. The graphics controller manages image data streaming from the camera and the CPU so as to combine the image data into a composite, and provides the composite data to the graphics display device. 
   The composite image is particularly a main image overlaid with an overlay image. Main image data defining the main image is provided by the camera  14 , and overlay image data defining the overlay image is provided by the CPU  16 . The overlay image data function as a template through which the main image data are to be viewed. 
   Main image data obtained from the camera are received by the camera interface (IF)  20 . The received data are provided to a resizer  22  and, in turn, a color format converter  24  for converting the (typically) YUV camera output to RGB format for storage in a first memory portion “M 1 ” of a display buffer  26 . Other color conversions may also be performed, and these functions can be performed in alternative sequences. 
   The graphics controller  12  stores overlay image data received from the host CPU  16  directly to a second memory portion “M 2 ” of the display buffer  26 , separate from the first portion. The graphics controller includes a fetching module  23  for fetching, pixel by pixel, each of the main image data and the corresponding overlay image data from the display buffer  26 . The fetched data are provided to a graphics display interface  28 . The graphics display interface includes a combining circuit  29  for forming a composite image from the main and overlay image data The graphics display interface transmits the composite image to the graphics display device for display. 
   The interface  28  receives, for each pixel location on the display, a main image pixel and an overlay image pixel. Where the overlay image pixel is transparent, the combining circuit  29  selects the main image pixel for the composite image. Where the overlay image pixel is opaque, the combining circuit selects the overlay image pixel for the composite image. 
   Turning to  FIG. 2 , an electronic computing system  30  for displaying overlaid camera images according to the present invention is shown. Like the system  10 , the system  30  includes a graphics controller  32  that interfaces to a camera  34 , a host CPU  36 , and a graphics display device  38 . In the context of the present invention, a graphics display device may be any device adapted for rendering image data. A preferred graphics display device is an LCD panel, but the graphics display device may be, for example, a CRT, plasma display, OLED, printer, or equivalent device. 
   As in the graphics controller  12  of the system  10 , an objective of the system  30  is to form a composite image from a main image and an overlay image. In the preferred cell phone context of the invention, the overlay image data are generated or stored in the CPU  36  while the main image data are generated by the camera  34 ; however, any source of graphics output may be used to produce either type of image data without departing from the principles of the invention. 
   Also as in the graphics controller  12 , main image data obtained from the camera are received by a camera interface (IF)  40 . The received data are streamed through a transmitting pipe  41  to a graphics display interface  48  for interfacing to the graphics display device  38 . The transmitting pipe  41  typically includes processing units such as a resizer  42  and a color format converter  44  as in the controller  12 ; however, such processing units are not essential to the invention. 
   The graphics controller  32  stores overlay image data received from the host CPU  36  in an internal display buffer  46 . However, in contradistinction to the graphics controller  12  and in accordance with principles of the present invention, the main image data are transmitted by the transmitting pipe  41  from the camera interface  40  to the graphics display interface  48  without fetching the data as in the prior art. Such transmitting is referred to herein as “streaming.” For example, the aforementioned processing units receive ordered pixel data, so that there is no need to address a memory to obtain the pixel data Accordingly, to perform streaming according to the present invention, the transmitting pipe  41  does not require a fetching module. 
   The graphics controller includes a fetching module  43  for fetching overlay image pixels from the display buffer  46  and provides the overlay image pixels to the interface  48  as corresponding main image pixels are streamed from the color converter  44  to the interface  48 . A composite image is formed at the interface using a combining circuit  49  that is the same as or similar to that described above in connection with the graphics controller  12 . However, because the main image data are not stored, e.g., in the display buffer  46 , the display buffer can be smaller, providing a commensurate reduction in power consumption and increased speed of operation. 
   The main image data correspond to main image pixels and the overlay image data correspond to overlay image pixels. The display interface  48  includes a counter for counting the main image pixels corresponding to main image data received from the transmitting pipe  41  and communicates with the fetching module  43  so as to fetch the corresponding overlay image pixel from the overlay image data stored in the display buffer  46 . Synchronization of the main and overlay image pixels may be accomplished by any means known in the art as will be readily appreciated by persons of ordinary skill. 
   The graphics display interface  48  typically transmits the composite image to the graphics display device for display; however, this not essential. 
   Streaming the main image data within the graphics controller  32  from the camera interface  40  to the graphics display interface  48  for combining with the overlay image for provision of the composite data to the graphics display device is generally practical only if the data meet the timing requirements of the graphics display device. That is, a timing adaptation would typically be required if the graphics display device does not have its own memory. However, in the preferred cell phone context of the invention as well as in other contexts, LCD panels are now being provided with internal RAM ( 38   a ). Accordingly, the main image data can be provided to the panels, where the data are written to the internal RAM, so that the panels can read the data from the internal RAM with the appropriate timing. 
   Turning now to  FIG. 3 , an enhanced electronic computing system  50  according to the present invention that is particularly advantageous for displaying large overlaid camera images is shown. Like the system  30 , the system  50  includes a graphics controller  52  that interfaces to a camera  54 , a host CPU  56 , and a graphics display device  58 . 
   As for the system  30 , in the preferred cell phone context of the invention, the overlay image data are generated or stored in the CPU  56  while the main image data are generated by the camera  54 ; however, any source of graphics output may be used to produce either type of image data without departing from the principles of the invention. 
   As in the graphics controller  32 , main image data obtained from the camera are received by the camera interface (IF)  60  and streamed through a transmitting pipe  61  to a graphics display interface  68  provided for interfacing to the graphics display device  58 . The transmitting pipe  61  typically includes processing units such as a resizer  62  and a color format converter  64 , however, such processing units are not essential to the invention. 
   Also as in the graphics controller  32 , overlay image data received from the host CPU  56  are stored in the graphics controller  52  in an internal display buffer  56 . However, according to principles of the invention, it is recognized that the resolution of the overlay image data is not critical, since they represent artificial features, such as borders or text, that typically do not have or require a high resolution. In accord with this recognition, the overlay image according to the invention is provided as a scaled down version of the desired overlay image as displayed on the display device. This permits a further reduction in the size of the display buffer and the performance consequences thereof. 
   Accordingly, a fetching module  63  fetches the scaled down overlay image data from the display buffer  66  and the fetched data are provided to an scaling circuit  67  that up-scales the overlay image data before providing the overlay image data to the graphics display interface  68 . The amount that the overlay image data are down-scaled for storage is typically known in advance and the scaling circuit  67 , in up-scaling the overlay image data, restores the overlay image data to full size with reference to a suitable scale factor. Subsequently, as in the graphics controller  32 , a composite (main+overlay) image is formed at the interface using a combining circuit  69  that is the same as or similar to that described above in connection with the graphics controller  12 . The graphics display interface  68  typically transmits the composite image to the graphics display device for display; however, this not essential. 
     FIG. 4  shows an example of a border defining, for illustrative purposes only, a 16 pixel scaled down overlay image of a border comprising 12 border pixels that are opaque (“O”) and 4 interior pixels that are transparent (“T”). The 16 pixel scaled down overlay image scales up in this example to a 64 pixel image. Particularly, a single corner pixel C( 1 , 1 ) 16  is expanded by up-scaling to become the group of 4 pixels indicated as C( 1 - 2 ,  1 - 2 ) 64  and a single interior pixel “T( 2 , 2 ) 16 ” is expanded to become the group of pixels indicated as C( 3 - 4 ,  3 - 4 ) 64 . In this example and as is typical, it can be readily appreciated that no loss of resolution occurs as a result of the scaling. It will also be appreciated that scaling may be by any known method, e.g., any integral or fractional zoom. 
   As noted above, the system  50  is particularly advantageous where the overlay image is large, and may even be necessary where the overlay image is larger than the main image. However, the system  50  provides a similar advantage for displaying overlay images that are the same size as, or that are smaller than, the corresponding main image. Additional processing may be performed on the composite image data without departing from the principles of the invention. 
   It should be recognized that, while specific memory efficient methods and apparatus for displaying large overlaid camera images have been shown and described as preferred, other configurations and methods could be utilized, in addition to those already mentioned, without departing from the principles of the invention. For example, wherever it is desired, overlay image data may be substituted for main image data and the reverse. 
   The terms and expressions which have been employed in the foregoing specification are used therein as terms of description and not of limitation, and there is no intention in the use of such terms and expressions to exclude equivalents of the features shown and described or portions thereof, it being recognized that the scope of the invention is defined and limited only by the claims that follow.