Abstract:
In an image-frame processing method, the image frame is outputted from and image sensor by an image processor via the buffering of a memory buffer. The method includes the following steps of: defining at least two storage spaces in the memory buffer; dividing the image frame into a plurality of image portions, each of which has a size corresponding to the size of one of said at least two storage spaces; sequentially storing the image portions into the storage spaces in turn; and sequentially processing said image portions stored in the memory buffer. This method is applicable to processing the image frame with the use of a small-sized memory buffer.

Description:
FIELD OF THE INVENTION 
   The present invention relates to a method for processing an image frame, and more particularly to a method for processing an image frame with the use of a small-sized memory buffer. 
   BACKGROUND OF THE INVENTION 
   With highly development and improvement of image-processing techniques, various electronic apparatuses including web camera, digital camera, cellular and picture phone, personal digital assistant (PDA), multimedia computer, etc., take advantages of these techniques to get well developed and enhanced.  FIG. 1  shows a typical real-time image transmission architecture for implementing a digital image processing technique, wherein an image sensor  11 , an image processor  12  with digital signal processing (DSP) function, and a memory buffer  13  are shown. The image sensor  11  picks up image of an object (now shown) under exposure and outputs the resulting image frame within a specified period of time for image-frame processing. Then the image processor  12  stores the image frame into the memory buffer  13  for subsequent processing. The image frame processed by the image processor  12  is optionally outputted to a computer or an image storage apparatus for further processing. 
   In the above-descried architecture, the image sensor  11  is generally implemented with a CMOS sensor or a CCD sensor, and the image processor  12  can be a digital signal processor (DSP). The image frame is generally presented in either RGB or YCbCr format depending on the choice of the image sensor  11 . When the DSP  12  is to do a JPEG compression of the image frame, however, the image frame has to be in YCbCr format. Hence, it is necessary to use an image sensor capable of generating image data in YCbCr format to serve as the image sensor  11 . Otherwise, if an RGB-type image sensor is used as the image sensor  11 , additional RGB to YCbCr conversion has to be performed before the image data can be compressed by the image processor  12 . 
   Conventionally, an image frame is stored in the memory buffer  13  and accessed by the image processor  12  as a whole. Therefore, the capacity of the memory buffer  13  should be large enough to accommodate the entire image frame. Moreover, under the requirement of receiving next image frame while processing currently stored image frame, the capacity of the memory buffer  13  should be enlarged to a level able to accommodate at least two image frames so as to prevent from improperly overwriting image data. Taking a VGA image frame consisting of 640×480 pixels for example, the storage capacity of the memory buffer  13  should be equal to or larger than (640×480 bytes)×3×2, which is about 1.8 MB. It is difficult to integrate such a large-sized memory buffer  13  with the image processor  12 , and thus the memory buffer  13  generally stands alone. It is apparent that such arrangement is inefficient in manufacturing process and production cost. 
   SUMMARY OF THE INVENTION 
   Therefore, the present invention provides a method for processing an image frame with the use of a small-sized memory buffer. 
   The present invention provides a method for processing an image frame which is outputted from an image sensor by an image processor via the buffering of a memory buffer. The method includes step of: defining at least two storage spaces in the memory buffer; dividing the image frame into a plurality of image portions, each of which has a size corresponding to the size of one of the storage spaces; sequentially storing the image portions into the storage spaces in turn; and sequentially processing the image portions stored in the memory buffer. 
   In an embodiment, the processing step for processing a previously stored image portion and the storing step for storing a subsequent image portion are performed simultaneously. 
   In an embodiment, the processing step performs a JPEG or MPEG compression operation based on sub-sampling algorithm. 
   In an embodiment, the image frame has a size of m×n and the memory buffer consists of two storage spaces. Each of the image portions has a size of m×p, and each of the storage spaces has a size of m×p, where m, n and p are positive integers, and p is less than n/2. The processing unit for the JPEG or MPEG compression operation is 8×8 or 16×16, and the parameter p is a corresponding value, i.e. 8 or 16. 
   In another embodiment, the image frame has a size of m×n and the memory buffer consists of three storage spaces. Each of the image portions has a size of m×q, and each of the storage spaces has a size of m×q, where m and q are positive integers, and q is less than n/3. 
   For example, the image frame is presented in RGB or YCbCr format. The image frame may has a size of 352×288, 640×480, 704×480, 704×576, 1024×768, 1280×1024, 1440×900, 1600×1200, or 1920×1440. 
   The present invention also provides an image-frame processing method. The image frame includes at least first, second and third image portions, and the memory buffer including at least first and second storage spaces. The method includes the following steps of: storing the first image portion into the first storage space; storing the second image portion into the second storage space while performing a first image-processing operation of the first image portion stored in the first storage space; and storing the third image portion into the memory buffer while performing a second image-processing operation of the second image portion stored in the second storage space. 
   In an embodiment, the first storage space, the second storage space, the first image portion, the second image portion, and the third image portion all have the same size. 
   For example, the first storage space and the second storage space both have a size of 640×16. The first image portion, the second image portion and the third image portion all have a size of 1280×8. A half of the first storage space and a half of the second storage space combined to store one of the first image portion, the second image portion and the third image portion. The image portions are sequentially stored into the storage spaces in turn until the entire image frame is completely processed. 
   The present invention further provides an image-frame processing method. At first, the memory buffer is patterned to consist of a plurality of storage parts, and the image frame is divided into a plurality of image portions. Then the storage parts are optionally combined to provide at least two storage spaces, each of which has a size corresponding to the size of the image portion. By sequentially storing the image portions into the storage spaces in turn and sequentially processing the image portions stored in the memory buffer, the image fame can be completely processed. 
   In one embodiment, each storage space consists of at least two storage parts, while each storage part has a size of 640×8. 

   
     BRIEF DESCRIPTION OF THE DRAWINGS 
     The above contents of the present invention will become more readily apparent to those ordinarily skilled in the art after reviewing the following detailed description and accompanying drawings, in which: 
       FIG. 1  is a schematic diagram illustrating a typical real-time image transmission architecture; 
       FIG. 2  is a schematic diagram illustrating the use of a 640×32 memory buffer to process a 640×480 image frame under a dual-buffer architecture according to the first embodiment of the present invention; 
       FIG. 3  is a schematic diagram illustrating the use of a 640×48 memory buffer to process a 640×480 image frame under a tri-buffer architecture according to the second embodiment of the present invention; 
       FIGS. 4   a  and  4   b  are schematic diagrams illustrating the use of a 1280×16 memory buffer to process image frames with various sizes according to the third embodiment of the present invention; 
       FIG. 5   a ˜ 5   c  are schematic diagrams illustrating the use of a 1920×24 memory buffer to process image frames with various sizes according to the fifth embodiment of the present invention; 
       FIG. 6   a  is a flowchart illustrating an image frame processing method under a dual-buffer architecture according to the present invention; and 
       FIG. 6   b  is a flowchart illustrating an image frame processing method under a tri-buffer architecture according to the present invention. 
   

   DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
   The method for processing an image frame according to the present invention will be illustrated hereinafter with reference to the architecture of  FIG. 1 . The memory buffer  13  of  FIG. 1 , however, has a size significantly smaller than that used conventionally. The image sensor  11  used in the present invention can also be a CMOS sensor or a CCD sensor, which results in an image frame in RGB or YCbCr format. The image processor  12  used in the present invention can also be a digital signal processor (DSP). For JPEG compression, an RGB to YCbCr conversion is performed in advance if the image sensor  11  generates an image frame in RGB format. In the JPEG compression procedure, image blocks having a size of 16×16 pixels or 8×8 pixels are generally taken as processing units for the sub-sampling procedure. That is, the image processor  12  will read 8 or 16 rows of image data from the memory buffer  13  at a time to meet the sub-sampling algorithm. Hence, according to an embodiment of the present invention applicable to JPEG compression, the memory buffer  13  includes a plurality of storage spaces and each storage space is set to be capable of accommodating 8 or 16 rows of image data. The plurality of storage spaces can be used to respectively store portions of the image frame and output the stored portions for JPEG compression sequentially. Basically, two or three storage spaces are enough for image-frame processing according to the present invention by alternating the data-transfer and data-compression procedures of these storage spaces. The sub-sampling algorithm used herein is well known to those skilled in the art, which for example can be 4:4:4, 4:2:2, 4:2:0 or 2:1:1 sub-sampling, and will not be described in detail herein. 
   Please refer to  FIG. 2  which is a schematic diagram illustrating the method for processing a 640×480 image frame F 1  according to the present invention, which is implemented with a dual-buffer architecture. In this embodiment, a 640×32 memory buffer  13  is patterned to consist of two 640×16 volatile storage spaces  13   a  and  13   b , and the image frame F 1  is transferred into the two storage spaces  13   a  and  13   b  portion by portion for JPEG compression. 
   For processing the image frame F 1 , a 640×16 image portion F 11  is first stored into the first storage space  13   a . Then the next 640×16 image portion F 12  is received and stored into the second storage space  13   b  while the image processor  12  is carrying out a JPEG compression procedure on the image portion F 11  stored in the first storage space  13   a . In this embodiment, the basic processing unit u 1  for JPEG compression is 16×16 pixels. Afterwards, a subsequent 640×16 image portion F 13  is received and stored into the first storage space  13   a  and overwrites the image portion F 11  having completed JPEG compression. Meanwhile, the image processor  12  carries out a JPEG compression procedure on the image portion F 12  stored in the second storage space  13   b . By repeating the above steps to alternately update and process the image portions in the two storage spaces  13   a  and  13   b , the image frame F 1  can be completely processed by the image processor  12  with the buffering of the small-sized memory buffer. 
   Alternatively, the processing of the 640×480 image frame F 1  can be implemented with a tri-buffer architecture, wherein a 640×48 memory buffer  23  is used and patterned to consist of three 640×16 volatile storage spaces  23   a ,  23   b  and  23   c , as shown in  FIG. 3 . The image frame F 1  is transferred into the three storage spaces  23   a ,  23   b  and  23   c  portion by portion for JPEG compression. 
   For processing the image frame F 1 , two 640×16 image portions F 11  and F 12  are first stored into the first storage space  23   a  and the second storage space  23   b , respectively. Then next 640×16 image portion F 13  is received and stored into the third storage space  23   c  while the image processor  12  is carrying out a JPEG compression procedure on the image portion F 11  stored in the first storage space  23   a . Like the previous embodiment, the basic processing unit u 1  for JPEG compression is 16×16 pixels. Afterwards, a subsequent 640×16 image portion F 14  is received and stored into the first storage space  23   a  and overwrites the image portion F 11  having completed JPEG compression. Meanwhile, the image processor  12  carries out a JPEG compression procedure on the image portion F 12  stored in the second storage space  23   b . By repeating the above steps to alternately update and process the image portions in the three storage spaces  23   a ,  23   b  and  23   c , the image frame F 1  can be completely processed by the image processor  12  with the buffering of the small-sized memory buffer. 
   When a larger-size frame, e.g. 1280×1024, is to be processed, a 1280×16 memory buffer  33  is properly used. By patterning the memory buffer  33  to consist of four 640×8 storage portions, either the larger 1280×1024 frame or the smaller 640×480 frame can be processed under a dual-buffer architecture. Please refer to  FIG. 4   a  which is a schematic diagram illustrating the method for processing a 640×480 image frame F 1  according to the present invention. The memory buffer  33  consists of four 640×8 storage parts  33   a   1 ,  33   a   2 ,  33   b   1  and  33   b   2 . For receiving and storing the 640×16 image portions F 11 , F 12 , F 13 , etc., the storage parts  33   a   1  and  33   a   2  are combined to provide a 640×16 storage space  33   a , and the other two storage parts  33   b   1  and  33   b   2  are combined to provide another 640×16 storage space  33   b . Hence, the image frame F 1  is transferred into the two storage spaces  33   a  and  33   b  portion by portion for JPEG compression. 
   In more detail, a 640×16 image portion F 11  is first stored to occupy the storage space  33   a . Then next 640×16 image portion F 12  is received and stored to occupy the second storage space  33   b  while the image processor  12  is carrying out a JPEG compression procedure on the image portion F 11 . In this embodiment, the basic processing unit u 1  for JPEG compression is 16×16 pixels. Afterwards, a subsequent 640×16 image portion F 13  is received and overwrites the image portion F 11  having completed JPEG compression to take over the first storage space  33   a . Meanwhile, the image processor  12  carries out a JPEG compression procedure on the image portion F 12  occupying the second storage space  33   b . By repeating the above steps to take advantages of the two storage spaces  33   a  and  33   b  and sequentially process the image portions, the image frame F 1  can be completely processed by the image processor  12  with the buffering of the small-sized memory buffer. 
   On the other hand, the memory buffer  33  consisting of the four 640×8 storage parts  33   a   1 ,  33   a   2 ,  33   b   1  and  33   b   2  is used for processing a 1280×1024 image frame F 2 , as shown in  FIG. 4   b . In this embodiment, 2:1:1 sub-sampling is adopted, and the basic processing unit u 2  for JPEG compression is 8×8 pixels. For receiving and storing the 1280×8 image portions F 21 , F 22 , F 23 , etc., the storage parts  33   a   1  and  33   b   1  are combined to provide a 1280×8 storage space  33   x , and the other two storage parts  33   a   2  and  33   b   2  are combined to provide another 1280×8 storage space  33   y . The two 1280×8 storage spaces  33   x  and  33   y  alternately receive 1280×8 image portions and output them for JPEG compression in sequence. As described in the above embodiments, by repeating the steps to take advantages of the dual-buffer architecture of the memory buffer  33  and sequentially store and process the image portions F 21 , F 22 , F 23 , etc., the image frame F 2  can be completely processed by the image processor  12  with the buffering of the small-sized memory buffer. 
   When an even larger-size frame, e.g. 1920×1440, is to be processed, a 1920×24 memory buffer  43  is properly used. By patterning the memory buffer  43  to consist of nine 640×8 storage portions, the 1920×1440 frame can be processed under a tri-buffer architecture. Moreover, the smaller 640×480 frame or 1280×1024 frame can also be processed under a tri-buffer architecture with this 1920×24 memory buffer  43 . Please refer to  FIG. 5   a  which is a schematic diagram illustrating the method for processing a 640×480 image frame F 1  according to the present invention. The memory buffer  43  consists of nine 640×8 storage parts  43   a   1 ,  43   a   2 ,  43   a   3 ,  43   b   1 ,  43   b   2 ,  43   b   3 ,  43   c   1 ,  43   c   2  and  43   c   3 . For receiving and storing the 640×16 image portions F 11 , F 12 , F 13 , F 14 , etc., the two storage parts  43   a   1  and  43   a   2  are combined to provide a 640×16 storage space  43   a , another two storage parts  43   b   1  and  43   b   2  are combined to provide another 640×16 storage space  43   b , and further two storage parts  43   c   1  and  43   c   2  are combined to provide a further 640×16 storage space  43   c . Accordingly, the image frame F 1  is transferred into the three storage spaces  43   a ,  43   b  and  43   c  portion by portion for JPEG compression. 
   For processing the image frame F 1 , two 640×16 image portions F 11  and F 12  are first stored into the first storage space  43   a  and the second storage space  43   b , respectively. Then the next 640×16 image portion F 13  is received and stored into the third storage space  43   c  while the image processor  12  is carrying out a JPEG compression procedure on the image portion F 11  stored in the first storage space  43   a . In this embodiment, the basic processing unit u 1  for JPEG compression is 16×16 pixels. Afterwards, a subsequent 640×16 image portion F 14  is received and stored into the first storage space  43   a  and overwrites the image portion F 11  having completed JPEG compression. Meanwhile, the image processor  12  carries out a JPEG compression procedure on the image portion F 12  stored in the second storage space  43   b . By repeating the above steps to alternately update and process the image portions in the three storage spaces  43   a ,  43   b  and  43   c , the image frame F 1  can be completely processed by the image processor  12  with the buffering of the small-sized memory buffer. 
   The memory buffer  43  consisting of nine 640×8 storage parts  43   a   1 ,  43   a   2 ,  43   a   3 ,  43   b   1 ,  43   b   2 ,  43   b   3 ,  43   c   1 ,  43   c   2  and  43   c   3  is also used for processing a 1280×1024 image frame F 2 , as shown in  FIG. 5   b . In this embodiment, 2:1:1 sub-sampling is adopted, and the basic processing unit u 2  for JPEG compression is 8×8 pixels. For receiving and storing the 1280×8 image portions F 21 , F 22 , F 23 , F 24 , etc., the storage parts  43   a   1  and  43   b   1  are combined to provide a 1280×8 storage space  43   m , another two storage parts  43   a   2  and  43   b   2  are combined to provide another 1280×8 storage space  43   n , and further two storage parts  43   a   3  and  43   b   3  are combined to provide a further 1280×8 storage space  43   k . The three 1280×8 storage spaces  43   m ,  43   n  and  43   k  alternately receive 1280×8 image portions F 21 , F 22 , F 23 , F 24 , etc. and output them for JPEG compression in sequence. By repeating the steps similar to that described above with reference to  FIG. 5   a  to take advantages of the tri-buffer architecture of the memory buffer  43  and sequentially store and process the image portions F 21 , F 22 , F 23 , F 24 , etc., the image frame F 2  can be completely processed by the image processor  12  with the buffering of the small-sized memory buffer. 
   Furthermore, the memory buffer  43  consisting of nine 640×8 storage parts  43   a   1 ,  43   a   2 ,  43   a   3 ,  43   b   1 ,  43   b   2 ,  43   b   3 ,  43   c   1 ,  43   c   2  and  43   c   3  is also applicable to process a 1920×1440 image frame F 3 , as shown in  FIG. 5   c . In this embodiment, 2:1:1 sub-sampling is also adopted, and the basic processing unit u 2  for JPEG compression is 8×8 pixels. For receiving and storing the 1920×8 image portions F 31 , F 32 , F 33 , F 34  etc., the three storage parts  43   a   1 ,  43   b   1 , and  43   c   1  are combined to provide a 1920×8 storage space  43   x , another three storage parts  43   a   2 ,  43   b   2  and  43   c   2  are combined to provide another 1920×8 storage space  43   y , and the other three storage parts  43   a   3 ,  43   b   3  and  43   c   3  are combined to provide the other 1920×8 storage space  43   z . The three 1920×8 storage spaces  43   x ,  43   y  and  43   z  alternately receive 1920×8 image portions F 31 , F 32 , F 33 , F 34 , etc. and output them for JPEG compression in sequence. By repeating the steps similar to that described above with reference to  FIG. 5   a  to take advantages of the tri-buffer architecture of the memory buffer  43  and sequentially store and process the image portions F 31 , F 32 , F 33 , F 34 , etc., the image frame F 3  can be completely processed by the image processor  12  with the buffering of the small-sized memory buffer. 
   Although not specifically described, it should be understood that the 1920×1440 image frame F 3  can also be processed under a dual-buffer architecture by providing a 1920×16 memory buffer patterned to consist of two 1920×8 storage spaces while selecting the basic processing unit u 2  for JPEG compression as 8×8 pixels. 
   In the above embodiments, 640×480, 1280×1024 and 1920×1440 image frames are exemplified for illustrating the present image-frame processing method. It is to be noted that the present invention can also be used to process other size of image frame such as 352×288, 704×480, 704×576, 1024×768, 1152×864, 1440×900, 1600×1200, etc. Furthermore, though the image frames are subjected to JPEG compression in the above embodiments, the present invention can be similarly applied for MPEG processing. From the above embodiments, it is understood that the required storage capacity of the volatile memory  13  is significantly reduced as compared with the prior art. The reduced size will facilitate the integration of the volatile memory  13  with the image processor  12 . 
   Though the above embodiments are exemplified to be applied to JPEG compression, the present image processing method can be applied to other suitable fields. A flowchart shown in  FIG. 6   a  is used to summarize an image processing method according to the present invention wherein two storage spaces of the memory buffer are used to store an image frame consisting of m columns and n rows of image data, i.e. dual-buffer architecture. First, the first i rows (row 1˜row i) of image data are stored into a first storage space of the memory buffer where 1&lt;i&lt;n/2 (Step  61 ). Then, next i rows (row i+1˜row 2i) of image data are stored into a second storage space of the memory buffer. In the meantime, the image processor processes the first i rows of image data (Step  62 ). After the first i rows of image data complete processing (Step  63 ), it is the turn of the second i rows to be processed by the image processor. Meanwhile, the third i rows (row 2i+1˜row 3i) of image data overwrite the first i rows stored in the first storage space (Step  64 ). After the second i rows of image data complete processing (Step  65 ), it is the turn of the third i rows to be processed by the image processor (Step  67 ). If there are still rows of image data not transferred to the memory buffer yet (Step  66 ), the fourth i rows (row 3i+1˜row 4i) of image data overwrite the second i rows stored in the second storage space while the third i rows is being processed by the image processor (Step  68 ). The alternate updating and processing procedures of image data described above are repeated till the entire image frame is completely processed (Step  69 ). Likewise, a tri-buffer architecture can be derived according to the above principles and illustrated in  FIG. 6   b.    
   While the invention has been described in terms of what is presently considered to be the most practical and preferred embodiments, it is to be understood that the invention needs not be limited to the disclosed embodiment. On the contrary, it is intended to cover various modifications and similar arrangements included within the spirit and scope of the appended claims that are to be accorded with the broadest interpretation so as to encompass all such modifications and similar structures.