Abstract:
Provided is a device for monitoring the surrounding area of a vehicle. The device separates and extracts an object around the vehicle in better precision from the background. The device detects the ambient temperature of the vehicle, and determines a temperature difference between the ambient temperature and an object surface temperature estimated on the basis of the ambient temperature. On the basis of the brightness value of the background of a captured image obtained by an infrared camera and a brightness difference corresponding to the temperature difference, the brightness value of the object is calculated. The captured image is binarized by using the brightness value of the object as a threshold value, and the object is extracted. Since the binary threshold value is set on the basis of the relation between the ambient temperature and the object surface temperature, the object can be separated and extracted in the better precision from the background.

Description:
TECHNICAL FIELD 
       [0001]    The present invention relates to an apparatus for monitoring a periphery of a vehicle using an image captured by one or more infrared cameras, more specifically relates to a vehicle periphery monitoring apparatus for extracting an object by a binarization process of the captured image. 
       BACKGROUND ART 
       [0002]    Conventionally, an apparatus for capturing an image around a vehicle by an infrared camera that is mounted on the vehicle, and binirizing the captured image to extract an object having higher temperature such as a pedestrian and animal has been proposed. In the patent document 1 below, a method for generating a luminance histogram of an image captured by an infrared camera and determining a threshold value that divides the captured image into a background image and an object image based on the luminance histogram. Through a binarization process using such a threshold value, an object having higher temperature is distinguished and extracted from a background. 
         [0000]    Patent Document 1: Japanese patent publication laid-open No. 2003-216949 
       DISCLOSURE OF THE INVENTION 
     Problem to be Solved by the Invention 
       [0003]    In addition to living bodies such as pedestrians and animals, artificial structures such as utility poles and walls may exist around a vehicle. In order to distinguish and extract living bodies such as pedestrians and animals from the background as higher temperature objects, it is desirable that such artificial structures are classified into the background in the binarization process. However, depending on the kinds and placement of the artificial structures and the environment such as temperature around the vehicle, the artificial structures may be classified into the higher temperature objects even if the above conventional method is used. Therefore, a technique for distinguishing and extracting a desired object from a background with better accuracy in the binarization process, independently of the environment around the vehicle, is desired. 
       Means for Solving Problem 
       [0004]    According to one aspect of the present invention, a vehicle periphery monitoring apparatus for monitoring a periphery of a vehicle using an image captured by an infrared camera mounted on the vehicle detects an outside temperature of the vehicle. A temperature difference between a surface temperature of an object estimated based on the outside temperature and the outside temperature is calculated. Based on a luminance value of the background in the captured image and a luminance difference corresponding to the temperature difference, a luminance value of an object in the captured image is determined. The captured image obtained by the infrared camera is binarized by using the luminance value of the object as a threshold value to extract the object. 
         [0005]    A relationship between a surface temperature of an object and an outside temperature is previously determined, and hence the surface temperature can be estimated from the outside temperature. This invention is based on this findings. A temperature difference between the detected outside temperature and a surface temperature of an object estimated based on the outside temperature is calculated. Because the luminance value of the background can be considered as corresponding to the outside temperature, a luminance value corresponding to an object can be determined based on the luminance value of the background and a luminance difference corresponding to the temperature difference. By conducting the binarization using the luminance value thus determined as a threshold value, the object can be separated and extracted well from the background portion that is other than the object. For example, when a pedestrian is extracted as an object, it is prevented that an object such as an artificial structure is erroneously extracted, by previously determining a relationship between the surface temperature of the pedestrian and the outside temperature. 
         [0006]    Other features and advantages of the present invention will be apparent from the following detailed description of the present invention and the accompanying drawings. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0007]      FIG. 1  is a block diagram showing a structure of a periphery monitoring apparatus in accordance with one embodiment of the present invention; 
           [0008]      FIG. 2  is a diagram for explaining an attachment position of cameras in accordance with one embodiment of the present invention; 
           [0009]      FIG. 3  is a flowchart of a process by an image processing unit in accordance with one embodiment of the present invention; 
           [0010]      FIG. 4  indicates a map for defining a relationship between an outside temperature and a surface temperature of an object in accordance with one embodiment of the present invention; 
           [0011]      FIG. 5  is a diagram for explaining an establishment of threshold values for a binarization process in accordance with one embodiment of the present invention; 
           [0012]      FIG. 6  shows a comparison between a conventional binary image and a binary image according to one embodiment of the present invention; 
           [0013]      FIG. 7  is a diagram for explaining an estimation of a size of an object in a captured image in accordance with one embodiment of the present invention; 
           [0014]      FIG. 8  is a diagram for explaining a setting of an object region and an object determination in accordance with one embodiment of the present invention; 
           [0015]      FIG. 9  is a diagram for explaining another technique for setting an object region in accordance with one embodiment of the present invention; and 
           [0016]      FIG. 10  is a diagram for explaining a technique for establishing threshold values for a binarization process using a road surface temperature in accordance with one embodiment of the present invention. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0017]    Preferred embodiments of the present invention will be described referring to the attached drawings. 
         [0018]      FIG. 1  is a block diagram showing a structure of a periphery monitoring apparatus of a vehicle in accordance with one embodiment of the present invention. The apparatus is mounted on the vehicle and comprises two infrared cameras  1 R,  1 L capable of detecting far-infrared rays, a sensor  5  for detecting an outside temperature in a periphery of the vehicle, an image processing unit  2  for detecting an object in front of the vehicle based on image data obtained by the cameras  1 R,  1 L, a speaker  3  for issuing a warning with voice based on the detected result, and a head-up display (hereinafter referred to as a “HUD”)  4  for displaying an image obtained by the camera  1 R or  1 L and outputting a display to cause a driver of the vehicle to recognize the object in front of the vehicle. 
         [0019]    As shown in  FIG. 2 , the cameras  1 R,  1 L are arranged in a front portion of the vehicle  10  at locations symmetric with respect to the longitudinal central axis of the vehicle  10 , and rigidly fixed to the vehicle such that the two cameras  1 R,  1 L have optical axes in parallel with each other and equal heights from a road surface. The infrared cameras  1 R,  1 L have a characteristic that the output signal level becomes higher (that is, the luminance in a captured image becomes larger) as the temperature of the object becomes higher. 
         [0020]    The image processing unit  2  includes an A/D converter circuit for converting input analog signals to digital signals, an image memory for storing digitized image signals, a CPU (central processing unit) for carrying out arithmetic operations, a RAM (Random access memory) used by the CPU for storing data being processed in the arithmetic operations, a ROM (Read Only memory) storing programs executed by the CPU and data (including tables and maps) to be used by the programs, and an output circuit for outputting driving signals to the speaker  3 , display signals to the HUD  4 , and the like. Output signals from the cameras  1 R,  1 L and the sensor  5  are converted to digital signals and input into the CPU. As shown in  FIG. 2 , the HUD  4  is arranged such that a screen  4   a  thereof is displayed in a front window at a location ahead of the driver. Thus, the driver can view the screen displayed on the HUD  4 . 
         [0021]      FIG. 3  is a flowchart of a process executed by the image processing unit  2 . This process is executed at predetermined time intervals. 
         [0022]    In steps S 11  through S 13 , output signals (that is, data of captured images) from the cameras  1 R,  1 L are received, A/D converted and stored in the image memory. Data of images thus stored are gray scale images having luminance values information. 
         [0023]    The following steps S 14  through S 19  are a process for distinguishably extracting a desired object from the background in a binarization process. This embodiment will be described for a case where the desired object is a pedestrian. 
         [0024]    In step S 14 , an outside temperature i (° C.) detected by the outside temperature sensor  5  is obtained. In step S 15 , a luminance value Tb of the background is determined 
         [0025]    The luminance value of the background may be determined by any technique. In this embodiment, a luminance value histogram is created based on the gray scale image. A luminance value having the highest frequency is used as the luminance value Tb of the background. This is because an area occupied by the background is generally largest in the captured image. 
         [0026]    In step S 16 , a map as shown in  FIG. 4  is referred to based on the detected outside temperature i. Here, the map will be described. In the head portion of a pedestrian, the skin of the face is generally exposed to the air, and hence the head portion has almost nothing that blocks the heat source. Therefore, the present invention focuses on the surface of the head portion that is exposed to the air. As a result of examining a relationship between the temperature of the surface of the head portion (hereinafter referred to as surface temperature) fa (° C.) and the outside temperature i (° C.) through experiments or simulations, it was found that there is a relationship as shown in  FIG. 4  between the both. In the figure, the horizontal axis indicates the outside temperature i (° C.) whereas the vertical axis indicates the surface temperature fa (° C.). As shown in this figure, the surface temperature fa can be estimated from the outside temperature i. 
         [0027]    The surface temperature fa changes as indicated by a curve  101  with respect to the outside temperature i. The surface temperature fa is higher as the outside temperature i is higher. For a given outside temperature i, a difference of the surface temperature fa(i) with respect to the outside temperature i is indicated by a difference between the curve  101  and a line  103  (which is a straight line indicating fa=i), and is referred to as a surface temperature difference, which is represented by F(i). That is, F(i)=surface temperature fa(i)−outside temperature i. As shown in the figure, there is a tendency that the surface temperature difference F(i) is smaller as the outside temperature i is higher. 
         [0028]    In order to improve the accuracy of extracting an object, a predetermined margin range T (° C.) with respect to F(i) is set in this embodiment. An upper limit of the margin range is indicated by a dotted line  101 U. A difference between the upper limit and the outside temperature i is represented by F(i)max. A lower limit of the margin range is indicated by a dotted line  101 L. A difference between the lower limit and the outside temperature i is represented by F(i)min. 
         [0029]    The map as shown in  FIG. 4  is pre-stored in a memory of the image processing unit  2 . The image processing unit  2  refers to the map based on the detected outside temperature i (° C.) to determine a surface temperature fa corresponding to the outside temperature i. The processing unit  2  calculates a surface temperature difference F(i) between the surface temperature fa and the outside temperature i, and uses the margin range T to calculate an upper limit value F(i)max and a lower limit value F(i)min with respect to the surface temperature difference F(i). Here, the margin range T may be changed in accordance with the outside temperature i or may be constant. 
         [0030]    Alternatively, the upper limit value F(i)max and the lower limit value F(i)min for the surface temperature difference F(i) corresponding to each outside temperature i may be stored in a memory. In this case, determining the surface temperature fa from the outside temperature i can be skipped. The upper limit value F(i)max and the lower limit value F(i)min can be directly determined from the outside temperature i. 
         [0031]    Referring back to  FIG. 3 , in step S 17 , a luminance difference corresponding to the upper limit value F(i)max and the lower limit value F(i)min of the surface temperature difference F(i) is calculated. A ratio of a change in the luminance value with respect to a change in the temperature is predetermined according to the specification of the infrared camera. Such a ratio is represented by a parameter SiTF. Thus, an upper limit value dTmax and a lower limit value dTmin of the luminance difference corresponding to the upper limit value F(i)max and the lower limit value F(i)min of the surface temperature difference F(i) are calculated as shown in the equation (1). 
         [0000]        dT max= SiTF×F ( i )max 
         [0000]        dT min= SiTF×F ( i )min  (1)
 
         [0032]    In step S 18 , a threshold value for the binarization process is determined. Referring to  FIG. 5 , one example of a luminance value histogram for the gray scale image obtained in step S 13  is shown. As described above, in step S 15 , a luminance value having the highest frequency (peak luminance value) is set in the luminance value Tb of the background. Therefore, as shown in the following equation (2), a luminance value Tcmax of the surface temperature having the upper limit value F(i)max of the surface temperature difference with respect to the outside temperature i has the upper limit value dTmax of the luminance difference with respect to the background luminance value Tb. Similarly, a luminance value Tcmin of the surface temperature having the lower limit value F(i)min of the surface temperature difference with respect to the outside temperature i has the lower limit value dTmax of the luminance difference with respect to the background luminance value Tb. 
         [0000]        Tc max= Tb +dT max 
         [0000]        Tc min= Tb +dT min  (2)
 
         [0033]    The upper limit luminance value Tcmax and the lower limit luminance value Tcmin are set in the threshold values for the binarization process. A region  111  defined by these two threshold values is shown in  FIG. 5 . The region  111  is a luminance region of an object to be extracted. 
         [0034]    In step S 19 , by using the threshold values thus set in step S 18 , the binarization process is applied to the gray scale image obtained in step S 13  (in this embodiment, the image captured by the camera  1 R is used, but alternatively, the image captured by the camera  1 L may be used). For each pixel in the captured image, when a luminance value of the pixel is within the luminance region  111 , the pixel is set to a white region having a value of 1 because the pixel is determined as constituting an object to be extracted. When a luminance value of the pixel is not within the luminance region  111 , the pixel is set to a black region having zero because the pixel is determined as constituting the background. 
         [0035]    Here, referring to  FIG. 6 , a diagram that schematically represents images is shown. (a) indicates a gray scale image (captured image), where differences in the gradation are represented by different types of hatching. (b) indicates an image obtained by the binarization process according to a conventional method. (c) indicates an image obtained by the binarization process using the above-described method shown in steps S 14  through S 19 . In the figure, the black region is represented by a hatched region. 
         [0036]    In the gray scale image, in addition to a pedestrian  121 , artificial structures such as an electric pole  125  and an automobile  127  are captured. According to a conventional method, not only a pedestrian but also these artificial structures  125  and  127  may be extracted as an object or a white region, as shown in (b), depending on threshold values used in the binarization. 
         [0037]    In contrast, according to the above technique of the present invention, the surface temperature of an object (in this embodiment, a pedestrian) with respect to the outside temperature is estimated, and a luminance region of the object is established based on a temperature difference of the estimated surface temperature with respect to the outside temperature. Therefore, only a head portion of the pedestrian  121  can be extracted as shown by the white region  131  of (c) (this region is referred to as a head portion region, hereinafter). Even when the artificial structure  125  and the pedestrian  121  are overlapped in the captured image as shown in (a), only the pedestrian  121  can be easily extracted as shown by the white region  131  of (c). Thus, according to the present invention, an object can be better distinguished and extracted from the background portion that is other than the object. 
         [0038]    Referring back to  FIG. 3 , in step S 20 , a size of the full-body of the pedestrian is estimated based on the extracted head portion region. This estimation can be implemented by any technique. Here, one example of the estimation technique will be specifically described. 
         [0039]    Referring to  FIG. 7(   a ), the head portion region extracted in the binary image is represented by a black region. The width of the head portion region is w (expressed in terms of the number of pixels). The width w is calculated by, for example, setting a rectangle circumscribing the head portion region and calculating the width of the rectangle. Portions other than the head of the pedestrian are indicated by the dotted lines, which have not yet been extracted. In step S 20 , the aim is to estimate the height h (expressed in terms of the number of pixels) of the pedestrian in the captured image. 
         [0040]    In order to achieve this estimation, a general size of a pedestrian in the real space, that is, the width Wa of the head and the height Ha are predetermined. Wa and Ha may be set based on the average value for adults (for example, Wa is 20 centimeters and Ha is a value within a range from 160 to 170 centimeters). 
         [0041]    Furthermore, (c) is a diagram where a placement relationship between the camera  1 R and the object is represented on an XZ plane. (d) is a diagram where a placement relationship between the camera  1 R and the object is represented on a YZ plane. Here, X indicates a direction of the width of the vehicle  10 . Y indicates a direction of the height of the vehicle  10 . Z indicates a direction of a distance from the vehicle  10  to the object. The camera  1 R comprises an imaging element  11 R and a lens  12 R. f indicates a focal distance of the lens  12 R. 
         [0042]    Let the distance to the object be Z (centimeters). From the diagram of (c), the distance Z is calculated as shown by the equation (3). Here, pcw indicates an interval between pixels in X direction, that is, a length (centimeters) per pixel in X direction. 
         [0000]        Z=Wa×f /( w×pcw )  (3)
 
         [0043]    From the diagram of (d), the height h (centimeters) of the pedestrian in the captured image is calculated using the distance Z, as shown by the equation (4). Here, pch indicates an interval between pixels in Y direction, that is, a length (centimeters) per pixel in Y direction. 
         [0000]        h =( Ha/pch )× f/Z   (4)
 
         [0044]    Thus, the size of the pedestrian in the captured image can be estimated as having the width w and the height h. Alternatively, the fact that the width of the body is generally larger than the width of the head can be taken into account. In this case, a value obtained by adding a predetermined margin value to the width w of the head portion region may be used in place of the above-described width w. 
         [0045]    Referring back to  FIG. 3 , in step S 21 , an object region is set on the captured image (may be the gray scale image, or the binary image) according to the size of the pedestrian estimated in step S 20 . Here, referring to  FIG. 8(   a ), the head portion region  131  extracted as described above is shown. As indicated by the bold frame, an object region  141  that has the width w of the head portion region  131  and the height h from the top (y coordinate value is yu in the figure) of the head portion region  131  is set. Thus, the position of the object in the captured image is identified. 
         [0046]    Referring back to  FIG. 3 , in this embodiment, the step S 22  is performed, in which an object determination process is performed on the object region  141  thus set to determine whether the object captured in the object region  141  is a pedestrian or not. An arbitrary appropriate object determination technique, for example, using a well known shape matching method, can be used to determine a pedestrian (for example, see Japanese patent publication laid-open 2007-264778). This process is performed using the gray scale image. In  FIG. 8(   b ), the pedestrian  151  whose shape is thus determined is shown. 
         [0047]    If the object is determined as a pedestrian in step S 22 , the process proceeds to step S 23  where a warning determination process is performed. In this process, it is determined whether a warning should be actually output or not to a driver. If the result of this determination is affirmative, the warning is output. 
         [0048]    For example, it is determined whether a brake operation is being performed by a driver of the vehicle from the output of a brake sensor (not shown in the figure). If the brake operation is not being performed, the warning may be output. The warning output may be implemented by issuing a warning with voice through the speaker  3  while displaying the image obtained by, for example, the camera  1 R on the screen  4   a  in which the pedestrian is emphatically displayed. The emphatic display is implemented by any technique. For example, the object is emphatically displayed by surrounding the object by a colored frame. Thus, the driver can more surely recognize the pedestrian in front of the vehicle. Alternatively, any one of the warning voice and the image display may be used to implement the warning output. 
         [0049]    As another method of the step S 20 , for example, the height h of the pedestrian in the captured image may be calculated from a height of the head portion region (a height of the rectangle circumscribing the head portion region  131  can be used and is expressed in terms of the number of pixels) and the number of heads tall. For example, if the height of the head portion region  131  is hb and the average height for adults is seven-heads tall, then the height h of the pedestrian can be estimated as h=7×hb. 
         [0050]    As yet another method of steps S 20  and S 21 , a method for determining a road surface from luminance values in a region below the head portion region  131  to identify the object region  141  may be employed. This method will be briefly described referring to  FIG. 9 . (a) is a gray scale image (regions other than the head portion region  131  are omitted in the figure). A mask  161  having a predetermined size is set below the extracted head portion region  131 . The variance of luminance values in a region covered by the mask is calculated (alternatively, the standard deviation that is the square root of the variance may be used in place of the variance). It can be considered that the road surface is captured as an image region whose luminance values are almost uniform. Therefore, If the variance is higher than a predetermined value, it is determined that the region on which the mask is set is not the road surface. The mask  161  is moved downward as shown in (b) and the variance of a region covered by the mask is calculated again. This process is repeated while moving the mask  161  downward. If a region covered by the mask  161  includes only the road surface, the variance presents a low value. As shown in (c), if the position of the mask  161  where the variance becomes lower than the predetermined value, a boundary (y coordinate value is yb) between the position of the mask  161  and the previous position (indicated by the dotted line) of the mask  161  can be determined as a bottom edge of the object region  141 . Thus, the object region  141  having the width w and the height from the top (y coordinate value is yu) of the head portion region to the boundary is extracted. 
         [0051]    In the above embodiments, a luminance value having the highest frequency in the luminance value histogram is set in the luminance value Tb of the background, which is brought into correspondence with the outside temperature. Alternatively, the outside temperature and the temperature of the road surface may be distinguished to determine the luminance value Tb of the background. More specifically, in the case where the camera is placed in the front portion of the vehicle as shown in  FIG. 2 , because an area occupied by the road surface in the captured image is larger, a luminance value having the highest frequency can be generally brought into correspondence with the temperature of the road surface. Therefore, a relationship between the temperature of the road surface and the outside temperature is predetermined in a map (not shown), which is stored in a memory. This relationship may be obtained by experiments and/or simulations. 
         [0052]    The map is referred to based on the detected outside temperature i to determine a corresponding temperature R of the road surface. A temperature difference between the road surface temperature R and the outside temperature i is calculated. The parameter SiTF as described above is used to convert the temperature difference into a luminance difference dTi. Here, referring to  FIG. 10 , a luminance value histogram similar to  FIG. 5  is shown. A luminance value Tr having the highest frequency is brought into correspondence with the road surface temperature R. Because the temperature R of the road surface is generally higher than the outside temperature i, the calculated luminance difference dTi is subtracted from the luminance value Tr of the road surface to calculate a luminance value corresponding to the outside temperature i. The luminance value thus calculated is set in the background luminance value Tb in step S 15  of  FIG. 3 . In a case where the outside temperature i is higher than the road surface temperature R, the luminance difference dTi is added to the road surface luminance value Tr to calculate the luminance value Ti corresponding to the outside temperature. The luminance region  111  of the object is identified by the luminance values Tcmax and Tcmin having the luminance difference dTmax and dTmin, respectively, with respect to the luminance value Tb of the background. Thus, by distinguishing between the outside temperature and the road surface temperature, the luminance value of the background can be more accurately determined. Therefore, the threshold values used for the binarization process can be more appropriately established, thereby improving the accuracy of extracting the object. 
         [0053]    More preferably, because the temperature difference between the road surface temperature and the outside temperature may vary depending on a value of one or more external environment parameters such as a weather condition (sunny or not, wind speed, amount of rainfall, etc.) and/or time passed from sunset, a map may be created and stored for each value of a predetermined external environment parameter. A map according to the external environment parameter value on that day is selected and used. 
         [0054]    Similarly, the map of  FIG. 4  may be established for each value of the external environment parameter such as a weather condition and stored in a memory. For example, a map for a day where the wind speed is greater than a predetermined value and a map for a day where the wind speed is not greater than the predetermined value are separately created and stored. A map according to the wind speed on that day may be selected and used. 
         [0055]    In the above embodiments, the upper limit value F(i)max and the lower limit value F(i)min that define the margin range T are established for the surface temperature difference F(i) as described referring to  FIG. 4 . By establishing such a margin range, the threshold values for the binarization process can be set such that an object is more surely and accurately extracted. However, alternatively, without establishing such a margin range T, the luminance difference dT corresponding to the surface temperature difference F(i) is calculated, and the luminance difference dT is added to the background luminance Tb to calculate the luminance value Tc of the object. Pixels having a luminance value that matches the luminance value Tc are determined as indicating an object and hence are set to a white region. Pixels having a luminance value that does not match the luminance value Tc are determined as not indicating an object and hence are set to a black region. Furthermore, a predetermined range with respect to the luminance value Tc may be set to a luminance region of the object. 
         [0056]    In the above embodiments, a case where an object to be extracted in the binarization process is a pedestrian is described as one example. Alternatively, an object to be extracted may be another living body such as an animal. For example, a map as shown in  FIG. 3  may be previously created for a predetermined animal via experiments and/or simulations. The map may be used to set threshold values for the binarization process as described above. In the case of an animal, its full-body is often exposed to the air. Therefore, the steps S 20  and S 21  may be skipped. In step S 22 , for example, a shape determination is performed on a region extracted in step S 19  to determine whether an object is an animal or not. If it is determined that the object is an animal, a warning determination is made. 
         [0057]    The present invention is applicable to an object having a surface temperature that can be pre-defined with respect to the outside temperature via experiments and/or simulations as shown by a map of  FIG. 3 . Therefore, the present invention is not necessarily limited to a living body such as a human being and animal.