Abstract:
An image sensor and a pre-processing circuit adapted to associate weights respectively with cells of a grid. The cells respectively may include multiple contiguous picture elements of the image sensor. The pre-processing circuit may be configured to adaptively change the weights responsive to changes in the scene being imaged by the image sensor. The number of the cells per distance on the surface of the image sensor may be substantially less than the resolution of the image sensor. The cells which share the same weights may specify a region of interest on the image sensor.

Description:
BACKGROUND 
       [0001]    1. Technical Field 
         [0002]    The present invention relates to pre-processing in a camera. 
         [0003]    2. Description of Related Art 
         [0004]    Image histogram calculations are present on many modern digital cameras. Photographers may use the image histogram as an aid to show the distribution of tones captured, and whether image detail has been lost. In a histogram, the horizontal axis of the graph may represent tonal variations, red/green/blue and/or gray scale. The vertical axis may represent the number of pixels in each tone. 
         [0005]    During the last few years camera based driver assistance systems (DAS) have been entering the market; including lane departure warning (LDW), automatic high-beam control (AHC), traffic sign recognition (TSR) forward collision warning (FCW) and pedestrian detection. 
       BRIEF SUMMARY 
       [0006]    Various camera systems are provided for herein including an image sensor and a pre-processing circuit adapted to associate weights respectively with cells of a grid. The cells respectively may include multiple contiguous picture elements of the image sensor. The pre-processing circuit may be configured to adaptively change the weights responsive to changes in the scene being imaged by the image sensor. The number of the cells per distance on the surface of the image sensor may be substantially less than the resolution of the image sensor. The cells which share the same weights may specify a region of interest on the image sensor. The weights associated with the cells of the region of interest may be non-zero and the weights associated with cells of the grid outside the region of interest may be zero. The camera is mountable on a vehicle, and the region of interest may include an image of a road in the environment of the vehicle. 
         [0007]    An image processor may be connected to the image sensor. The pre-processing circuit may calculate a pre-processing result over the region of interest. The image processor may be configured to receive the pre-processing result from the pre-processing circuit. The image processor responsive to the pre-processing result may reset a camera parameter such as gain, exposure time and/or aperture. 
         [0008]    The pre-processing circuit may perform a weighted histogram by summing instances of a key over the cells of the grid. The key may include the weights as a multiplicative factor. The key may be a function of an image parameter such as image intensity, color intensity of at least one color, gradient of image intensity and/or gradient of color intensity. 
         [0009]    The image processor may estimate ego-motion of the image sensor. The image processor, responsive to the estimated ego-motion of the camera, may reset one or more weights of one or more of the cells. The image processor may estimate image motion of an object in the field of view of the image sensor and responsive to the estimated image motion reset one or more of the weights of one or more of the cells. 
         [0010]    Various computerized methods are provided herein for pre-processing image data of an image frame in an image sensor. Weights are associated respectively with cells of a grid. The cells respectively may include multiple contiguous picture elements of the image sensor. The weights may be adaptively changed responsive to changes in the scene being imaged by the image sensor. Multiple picture elements of the image sensor may be grouped contiguously into the cells of the grid with a number of cells per distance on the surface of the image sensor substantially less than the resolution of the image sensor. The image sensor may be mounted on a vehicle as a part of a driver assistance system. A region of interest may be determined to include cells of non-zero weight which image at least a portion of a road in the environment of the vehicle when traveling. Responsive to the weights, a pre-preprocessing result may be calculated. A camera parameter may be reset responsive to the pre-processing result. A weighted histogram may be performed by summing instances of a key over the cells of the grid. The key may include the weights as a multiplicative factor. The key may be a function of at least one image parameter: image intensity, color intensity of at least one color, gradient of image intensity and/or gradient of color intensity. Ego-motion of the image sensor may be estimated and responsive to the estimated ego-motion one or more of the weights of one or more of the cells may be reset. Image motion of an object being tracked in the field of view of the image sensor may be estimated and responsive to the estimated image motion, one or more of the weights of one or more of the cells may be reset. 
         [0011]    The foregoing and/or other aspects will become apparent from the following detailed description when considered in conjunction with the accompanying drawing figures. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0012]    The invention is herein described, by way of example only, with reference to the accompanying drawings, wherein: 
           [0013]      FIGS. 1 and 2  illustrate a system including a camera or image sensor mountable in a vehicle, according to an aspect of the present invention. 
           [0014]      FIG. 3  shows an image frame from a road scene according to a feature of the present invention. 
           [0015]      FIG. 4   a  illustrates region of interest superimposed onto an image frame respectively, according to a feature of the present invention. 
           [0016]      FIG. 4   b  illustrates the region of interest re-projected onto the grid of the image frame due to ego-motion of the camera or image motion of an object being tracked, according to a feature of the present invention. 
           [0017]      FIG. 5 , shows a simplified block diagram of pre-processing circuitry according to aspects of the present invention. 
           [0018]      FIG. 6  shows a flow diagram of a process according to aspects of the present invention. 
       
    
    
     DETAILED DESCRIPTION 
       [0019]    Reference will now be made in detail to features of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. The features are described below to explain the present invention by referring to the figures. 
         [0020]    Before explaining features of the invention in detail, it is to be understood that the invention is not limited in its application to the details of design and the arrangement of the components set forth in the following description or illustrated in the drawings. The invention is capable of other features or of being practiced or carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein is for the purpose of description and should not be regarded as limiting. 
         [0021]    By way of introduction, various embodiments of the present invention are directed to defining a region of interest in one or more image frames. In general, the region of interest may be selected by including an image portion having the high variation in dynamic range over time from image frame to image frame. Portions of the image having little variation in dynamic range such as sky or landscape may be excluded from the region of interest. The region of interest in image frames of camera based driver assistance systems may include the image of the road shown below the horizon, where images of other vehicles, traffic signs pedestrians and road obstructions are found. The position, shape and size of the region of interest may be specific for one or more monitor and/or control algorithms. The image may be pre-processed in the region of interest such as by computing a histogram which may be used by the monitor/control algorithm. A feature of the present invention includes a simple and efficient adjustment of the position of the region of interest over the image plane by keeping the topology of the region of interest similar to a canonical topology selected as appropriate for the monitor/control algorithm. The region of interest may be specified using multiplicative weighting factors or weights over a spatially constant grid of reduced resolution in the image area. The weights provide flexibility of preserving the predefined topology of region of interest as well as a simple ability to adjust the position of the region of interest by varying the values of the weights typically over a relatively small number of cells at the periphery of the grid. The fine details of the periphery of the region may be smoothed to the level of the reduced resolution by scaling the weights of the peripheral cells to be proportional to the area of the cells within the region of interest. 
         [0022]    Embodiments of the present invention may be useful to provide flexible and efficient method of controlling the location of the region of interest over the image surface as a result of ego-motion of the camera or image motion of an object of interest being tracked. In driver assistance systems, improved obstruction detection in the road environment may be achieved when the obstruction and road have minimal image contrast and/or in images which include the sun above the horizon outside the region of interest. 
         [0023]    Thus there is a need for and it would be advantageous to have a camera in which the region of interest of the image may be specified by weights over a low resolution grid of cells in the image frames. 
         [0024]    Referring now to the drawings, reference is now made to  FIGS. 1 and 2  which illustrate a system  16  including a camera or image sensor  12  mountable in a vehicle  18 , according to an aspect of the present invention. Image sensor  12 , images a field of view in the forward direction. Image sensor  12  may be monochrome or black-white, i.e. without color separation or image sensor  12  may be color sensitive. Image frames  15  from camera  12  are captured and processed by image processor  35 . Camera  12  includes a pre-processing circuit  19  which preprocesses image frames  15  and may provide a monitor signal or may receive a control signal over a monitor/control bus  27  to or from image processor  35  as a result of the pre-preprocessing. The monitor signal may include histogram information being provided to image processor  35 . 
         [0025]    Image processor  35  may be used to process image frames  15  simultaneously and/or in parallel to serve a number of driver assistance systems/applications. Image processor  35  may be used to process image frames  15  to detect and recognize an image or portions of the image in the forward field of view of camera  12 . The driver assistance systems may be implemented using specific hardware circuitry (not shown) with on board software and/or software monitor/control algorithms in storage  13 . By way of example in  FIG. 2 , image frames  15  are used to serve pedestrian detection  20 , traffic sign recognition (TSR)  21  and forward collision warning (FCW)  22  according to features of the present invention. 
         [0026]    In some cases, image frames  15  are partitioned between different driver assistance applications and in other cases the image frames  15  may be shared between the different driver assistance applications. 
         [0027]    Reference is now made to  FIG. 3  which shows an image frame  15  from a road scene according to a feature of the present invention. Image frame  15  in  FIG. 3  shows a road scene as captured by image sensor  12 . A region of interest  30  is shown as a trapezoid and indicates an image of road in front of vehicle  18  as a region of image frame  15  which is of particular importance for driver assistance functions such as pedestrian detection  20 , traffic sign recognition (TSR)  21 , forward collision warning (FCW)  22 , detection of lane markers or obstructions in the road surface like manhole covers. For other driver assistance applications or for image processing in a field other than driver assistance, a region of interest  30  of different size, shape, position and/or orientation in image space may be selected a priori. 
         [0028]    Reference is now made to  FIG. 4   a  which illustrates region of interest  30  superimposed onto image frame  15   a  respectively, according to a feature of the present invention. Picture elements in an image frame  15   a  captured by image sensor  12  are shown as parametrized by a coarse grid  41 , e.g. 50 by 50 pixels. Coarse grid  41  may be used to specify region of interest  30 . In order to specify region of interest  30 , 8 cells labeled individually with letters A-H are entirely within region of interest  30 . The perimeter of region of interest, e.g. trapezoid  30  crosses 16 cells, labeled with letter ‘p’. 
         [0029]    Reference is now made also to  FIG. 5 , which is a simplified block diagram of preprocessing circuitry  19  and to  FIG. 6  which is a flow diagram  501  of a process according to aspects of the present invention. In  FIG. 5 , grouping circuitry  40  functions to group pixels (step  503 ) into cells of a low resolution grid, for instance grid  41  shown in  FIG. 4   a . By selecting cells of grid  41 , a region of arbitrary shape may be provided (step  505 ). The region of arbitrary shape may be contiguous as in region of interest  30  as shown in  FIGS. 3 and 4   a , or non-contiguous. Weighting circuitry  42 , provides (step  507 ) a weight e.g. one, to all the pixels of the selected cells, for instance cells A-H in region of interest  30  of  FIG. 4   a . For cells p which lie on the perimeter of region of interest  30 , weighting circuitry  42  may assign a weight between zero and one which is proportional to the number of pixels within region of interest  30 , or proportional to the cell area within region of interest  30  divided by the total cell area. Outside the region of interest, the cells of grid  41  are normally weighted with smaller weights such as zeros. Histogram circuitry  44  may pre-process (step  509 ) image frames  15  responsive to the weights of the cells. Examples of pre-processing may include calculating histogram information, average and/or median of grey scale intensities or average color in Red/Green/Blue intensity values. The image information may be output (step  513 ) to image processor  35  over monitor signal line  27 . A camera control parameter, e.g. gain, exposure time, for controlling camera  12  may be reset (step  515 ) via control line  27  responsive to the information received in step  513 . 
         [0030]    In step  511 , ego-motion of vehicle  18  may be detected and/or an image of an object of interest may be tracked by image processor  35 . Based on the detected ego-motion and/or tracked image of an object of interest, region of interest  30  may be adjusted by signaling of control line  27 . On receiving the signal, pre-processing circuit re-adjusts weights (step  507 ) of the cells of grid  41  in order to maintain the tracked image within region of interest  30  independent of the ego-motion of camera  12 . For instance, the horizon image may be detected (step  511 ) which varies in image height dependent on the pitch angle of vehicle  18 . Region of interest  30  may be maintained within 50 pixels for instance at the same vertical image distance relative to the imaged horizon independent of the changing pitch angle of vehicle  18 , by adjusting weights (step  507 ) according cells of grid  41 . 
         [0031]    Reference is now also made to  FIG. 4   b  which illustrates an example of region of interest  30  re-projected onto grid  41  of image frame  15   b  responsive to detected ego-motion of camera  12  and/or image motion of an object being tracked, according to a feature of the present invention. Image frame  15   b  of  FIG. 4   b  illustrates for instance a case that the pitch angle of vehicle  18  is positive (upward) relative to the pitch angle of vehicle  18  when image frame  15   a  of  FIG. 4   a  is captured. In order to maintain tracking and/or region of interest  30  on the road at the same distance always from vehicle  18 , weights of cells of grid  41  are re-adjusted (step  507 ). In the specific example of image frame  15   b  of  FIG. 4   b , weights of cells labelled A,B,C,D,E,F,G,H remain the same as the weights assigned to cells A-H in image frame  15   a  of  FIG. 4   a . Region of interest  30  is updated and relocated by adjusting peripheral cells labelled I,J,K,L and additional cells p partially within region of interest  30 . The use of low resolution grid  41  thus allows for a calculationally efficient mechanism for maintaining region of interest  30  in the desired portion of images frames  15  since normally only a small number of peripheral cells need to have respective weights updated in step  507 . 
         [0032]    Although embodiments of the present invention are presented in the context of driver assistance applications, embodiments of the present invention may be equally applicable in other real time signal processing applications and/or digital processing applications, such as communications, machine vision, audio and/or speech processing as examples. 
         [0033]    The term “resolution” as used herein refers to a number of picture elements per distance for instance along a horizontal and/or vertical line on the surface of the image sensor. 
         [0034]    The term “weight” as used herein refers to a multiplicative scale factor for performing image processing over images. A weight of zero generally means that the pixel or group of pixels is not included in the processing. The higher the non-zero weight the more the pixel or group of pixels is considered in the processing. 
         [0035]    The term “contiguous” as used herein is defined herein by selecting two points of a geometric region and if the line segment connecting the two points wholly lies in the geometric region, then the geometric region is contiguous. 
         [0036]    The term “region of interest” as used herein refers to a region in image space the shape and dimensions of which remain essentially unchanged over multiple image frames. The term “region of interest” as used herein is NOT an image of an object being tracked. 
         [0037]    The term “ego-motion” as used herein refers to at least the angular re-orientation of the image sensor over angles” pitch, yaw and roll. 
         [0038]    The term “key” as used herein refers to a parameter of interest selected for pre-processing an image. The “key” is used to represent a horizontal axis and instances of particular values of the key are summed to generate a histogram. 
         [0039]    The indefinite articles “a”, “an” is used herein, such as “an image” has the meaning of “one or more” that is “one or more images”. 
         [0040]    Although selected features of the present invention have been shown and described, it is to be understood the present invention is not limited to the described features. Instead, it is to be appreciated that changes may be made to these features without departing from the principles and spirit of the invention, the scope of which is defined by the claims and the equivalents thereof.