Abstract:
A video frame buffer may enable a high resolution image sensor video stream to be transmitted on a standard communication link at a high frame rate. The buffer may include an image stream input for receiving image frames from the image sensor, and a memory for buffering image frames. A frame processor manipulates portions of each image frame before passing the modified image frame to the memory which selectively buffers the frames before outputting the frames to a communication output. The video frame buffer may be implemented in a FPGA along with a DDR SDRAM memory. The communication link may be a USB2 communication link.

Description:
FIELD OF THE INVENTION  
       [0001]     The present invention relates to video frame buffers, and more particularly, but not exclusively, to a video frame buffer for receiving images from an image stream from an image sensor and converting the image stream to an alternative format.  
       BACKGROUND OF THE INVENTION  
       [0002]     Images, as well as video streams, direct from image sensors are usually provided in one of several formats. These formats include Compact Camera Port (CCP), CCP Joint Photographic Experts Group (JPEG), International Telecommunications Union (ITU) and ITU JPEG. An image frame that is output from an image sensor in one of these formats contains additional information beyond that of simply the data of the image.  
         [0003]     For example, referring to  FIG. 1 , a typical CCP formatted frame  10  is shown. The frame  10  begins with a start of frame data SOF indicating to any serial reader of the frame where the first line of the frame begins. The first line is then followed by end of line data EOL so the serial reader now knows that it has the first line of the frame. Following the end of line data EOL, checksum data is provided to verify the integrity of the image data contained in the first line. Following the checksum data is a blank period separating the data associated with the first line to that of the second. This data area is known as the line blanking period. Start of line data SOL then indicates the beginning of a second line of data. This is then followed by the image data, the end of line data EOL, a checksum and then the line blanking period. The same order continues for each line until the last line, which instead of the end of line data EOL has the end of frame data EOF.  
         [0004]     Following the last line, including its line blanking period, is another blank period separating the current frame from the next one, which is known as the frame blanking period. Blanking periods are introduced into image frames from an image sensor to allow for image scaling. The blank period provides a buffer for timing and scaling, and for delays in electronics in the image sensor or coprocessor to allow time for the electronics to colorize the image (raw outputs from an image sensor are bayer images).  
         [0005]     When image streams are received by a computer over a communication link, such as a USB2 (Universal Serial Bus 2) communication link, in one of these formats at least two problems effect the quality of the video image stream displayed. First, insufficient bandwidth is available in a USB2 communication link to transmit all image frames at the rate at which the sensor is capable (for a relatively high resolution image sensor, such as one having 1600×1200 pixels), which is typically 30 frames per second. As such, the image sensor has to be de-rated by reducing the output frame rate so that all produced images can be received.  
         [0006]     As a result of the slower frame rate, motion blur occurs in the displayed video, and therefore, does not represent the quality of the image stream that can be produced. Second, when images are transmitted in one of these formats over a communication channel, such as a USB2 link, the software in the receiving system is required to manipulate the image data for removing the portions which are not required for display. This operation further limits the frame rate at which images can be displayed on the receiving system.  
       SUMMARY OF THE INVENTION  
       [0007]     According to a first aspect, a communication link video frame buffer for an image sensor comprises an image stream input for receiving image frames from the image sensor, memory means or a memory for buffering image frames, and frame processing means or a frame processor. The frame processing means may manipulate portions of each image frame before passing the modified image frame to the memory means which selectively buffers (i.e., where necessary) the frames before outputting the frames to a communication output.  
         [0008]     The frame processing means may remove any blanking or padding periods before passing the modified image frame to the memory means. The frame processing means may comprise an input for receiving an image frame stream, selection means or a selector for selecting from a set of pre-defined output frame formats, and memory means or a memory for storing pre-defined sets of characters. Search means or a searcher may identify in the image frame stream one or more of the pre-defined sets of characters according to the selected output frame format, and modification means or a modifier may remove the identified sets of characters or replace them with an alternative pre-defined set of characters.  
         [0009]     The set of pre-defined output frame formats may be compatible with USB video formats. The frame processing means may be implemented in a Field Programmable Gate Array. The communication link may be a Universal Serial Bus 2 (USB2) communication link.  
         [0010]     According to a second aspect, video frame processing means or a video frame processor may comprise an input for receiving an image frame stream, selection means or a selector for selecting from a set of pre-defined output frame formats, and memory means or a memory for storing pre-defined sets of characters. Search means or a searcher may identify in the image frame stream one or more of the pre-defined sets of characters according to the selected output frame format, and modification means or a modifier may remove the identified sets of characters or replace with an alternative pre-defined set of characters.  
         [0011]     The set of pre-defined output frame formats may be compatible with USB video formats. The frame processing means may be implemented in a Field Programmable Gate Array. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0012]     Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which;  
         [0013]      FIG. 1  illustrates data organization in a CCP image frame according to the prior art;  
         [0014]      FIG. 2  illustrates a block diagram of a video frame buffer according to the present invention;  
         [0015]      FIG. 3  illustrates a Field Programmable Gate Array (FPGA) implementation according to the present invention;  
         [0016]      FIG. 4  illustrates a CCP image frame before and after processing according to the present invention;  
         [0017]      FIG. 5  illustrates a CCP JPEG8 image frame or ITU JPEG8 image frame before and after processing according to the present invention;  
         [0018]      FIG. 6  illustrates an ITU External image frame before and after processing according to the present invention;  
         [0019]      FIG. 7  illustrates an ITU Embedded 1 image frame before and after processing according to the present invention; and  
         [0020]      FIG. 8  illustrates an ITU Embedded 2 image frame before and after processing according to the present invention. 
     
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS  
       [0021]     The present invention is particularly applicable where full frame rate video from a high resolution image sensor is required to be transmitted over a communication link. As image sensor resolution continues to gets larger, the size of each individual image in storage and transmission terms has increased dramatically. For example, Compact Camera Port (CCP) images cannot be transmitted over a USB2 (Universal Serial Bus 2) link at a full frame rate of 30 frames per second for an image sensor having a resolution of 1600×1200 without any management of the image frames. Furthermore, even at a lower frame rate from a de-rated image sensor, current computing processor power cannot manipulate the image frames quickly enough to provide a simple video stream without the blanking periods, or other frame information on a computer connected to an image sensor over a USB2 communication link.  
         [0022]     As such, the illustrated video frame buffer  10  overcomes the problems associated with limited bandwidth and insufficient computing processor speed to display full frame rate video from an image sensor. Referring to  FIG. 2 , a video frame buffer  10  is shown having a video image stream input  12  from an image sensor (not shown). Frame processing means  14  processes the video image stream as described below. In this example, the frame processing means also converts 10 and 12 bit data to 16 bit data before the 16 bit data is passed to an input width conversion module  16  which converts the 16 bit data to 32 bit data. The input width conversion module  16  passes the 32 bit data to an input WR FIFO (WRite First In First Out) module  18  which, in turn, passes the 32 bit data to a memory controller  20 .  
         [0023]     The input width conversion module  16  and input WR FIFO module  18  manipulate the data into the correct format for the memory controller  20 . In this example, the memory controller  20  is a DDR (Double Data Rate) memory controller which accepts and outputs 32 bit data. Associated with the memory controller  20  is a memory driver  24 , which controls the timing of writes and reads to/from memory, and a memory  22  which in this case is a DDR SDRAM (Synchronous Dynamic Random Access Memory). Data received by the memory controller  20  is buffered in the memory  22 .  
         [0024]     The memory  22 , and associated driver  24  and controller  20  enable the full frame rate of the image sensor to be captured. The video frames captured in the memory  22  are then read out as fast as the communication link allows.  
         [0025]     Data is output from the memory  22  when required by the memory controller  20 , which in turn outputs the 32 bit data to an output RD FIFO  30  (ReaD FIFO) module  26  and then to an output width conversion module  28 , which converts the 32 bits data back to 16 bit. The 16 bit data is then passed to a USB interface  30 , which in this case is a Cypress FX2 USB interface.  
         [0026]     In an alternative embodiment, buffering is removed from the process altogether, as shown by the dashed lines of the alternative embodiment, and the frame processing means  32  simply performs image stream processing before outputting the data to the USB interface  30 . The operations of the frame processing means is described below.  
         [0027]     Referring now to  FIG. 3 , a FPGA (Field Programmable Gate Array) implementation is shown. Components common with that of  FIG. 2  are identically numbered. In this example, three different parallel data inputs are shown: CCP format from an image sensor/co-processor via a deserializer, such as the STCCP832 deserializer device; ITU format from an image sensor/coprocessor; and external from an external system also providing CCP and ITU data.  
         [0028]     Data flow through the FPGA is controlled by MUXs (multiplexers) and DEMUXs (demultiplexers) via an on chip  12 C (Inter-integrated Circuit) and Regmap block. The Regmap block contains all of the  12 C registers used to control the design.  
         [0029]     Frame processing means  14  and frame processing means  32  perform the same functions although the latter bypasses the memory means  20 ,  22 ,  24 . As such, the following description will apply to both frame processing means  14 ,  32 .  
         [0030]     The frame processing means is enabled to perform formatting on image frames from a first format to a second format. For example, as shown in  FIG. 4  (CCP Embedded (3)), a CCP input can be formatted to reduce the overall amount of information of the image frame, and the frame processing means removes blanking period from the image frame. This step can be performed in a blanking removal block implemented in a FPGA. The frame processing means is set to the appropriate in coming frame format and the desired outgoing frame format.  
         [0031]     Furthermore, the frame processing means can also remove or alter other parts of the image frame, as required. For example, checksum data at the end of each line can be removed, end of line and start of line data can be removed or USB synchronization codes can be inserted. The frame processing means processes image frames by searching the incoming bit streams for pre-defined sets of characters or markers, and either removing or replacing those markers according to the desired output image frame.  
         [0032]     Referring specifically to  FIG. 4 , a CCP frame shown in the input column can be processed in one of four ways. First, as shown in the first row, the image frame can be allowed to pass through without modification. Secondly, as shown in the second row, line and frame blanking periods as well as checksum data is removed to leave an simple image frame.  
         [0033]     Thirdly, as shown in the third row, frame and line blanking periods, checksum data and start of line and end of line data are removed. In this case, as the end of line data has been removed, USB synchronization codes are inserted. Fourthly, as shown in the fourth row, line and frame blanking periods are removed.  
         [0034]     Referring now to  FIG. 5 , a CCP JPEG8 frame, which corresponds to a JPEG8 frame, is shown in the input column. JPEG8 image frames do not contain line and frame blanking periods or start of frame and end of frame markers. However, they do contain padding bytes, which are inserted during encoding where the data stream contains a CCP synchronization code to prevent a genuine bit sequence from being interpreted as a marker (due to its serial nature, the encoder checks the data at a bit boundary), as well as other undefined data regions. As such, the frame processing means is enabled to remove the undefined data regions as well as the padding bytes to produce the output frame shown in the output column for both CCP JPEG8 and ITU JPEG8.  
         [0035]     Referring to  FIG. 6 , an ITU External frame is shown, which has additional vertical and horizontal synchronization lines rather than markers as in other ITU formats. In this example, the synchronization information is removed and is replaced with USB synchronization code.  
         [0036]     Referring to  FIG. 7 , an ITU embedded 1 frame is shown, in which there are embedded codes but no unique SOF/EOF markers. As shown in the first row, frame and line blanking are removed and a SOF marker is inserted into the frame.  
         [0037]     Referring to  FIG. 8 , an ITU embedded 2 frame is shown, which has the same structure as a CCP frame but is presented in parallel rather than in serial, as a CCP frame is. The first row shows that a frame can be generated with all frame and line blanking periods removed as well as EOL, SOL and EOF markers and the checksum data. The second row shows that an alternative frame output frame can be generated with only the frame and line blanking periods and the checksum data removed.  
         [0038]     The above described video frame buffer provides a significant speed advantage over the prior art. This is particularly the case where large resolution image sensors are providing high frame rate video. The video frame buffer enables the image quality and frame rate of the image sensor to be fully utilized. Improvements and modifications may be incorporated without departing from the scope of the present invention.