Object sensing device

In order to provide an object sensing device whereby, among limited computation resources, performance is improved in sensing an object when sense processing a plurality of objects to be sensed, an object sensing device includes image capture units which capture images external to a host vehicle, and a processing device which sense processes objects to be sensed from the images which are captured by the image capture units, said processing device further including: a scene analysis unit which analyzes a travel scene of the host vehicle; a process priority change unit which changes a sensing process priority of the object to be sensed on the basis of the travel scene which is analyzed by the scene analysis unit; and an object to be sensed sensing unit which carries out a sensing of the object to be sensed on the basis of the sensing process priority which is changed by the process priority change unit.

TECHNICAL FIELD

The present invention relates to an object sensing device that detects objects in a vehicle's surroundings from image information outside of the vehicle.

BACKGROUND ART

In order to realize safe travel of a vehicle, research and development is underway regarding devices that detect dangerous phenomena around a vehicle and automatically control the steering, accelerator, and brakes of the vehicle to avoid any dangerous phenomena that has been detected, and such devices have already been installed in some vehicles. Among such devices, a system that senses a pedestrian crossing in front of the vehicle with a sensor installed in the vehicle and warns the driver or automatically applies the brakes if there is a possibility of colliding with the pedestrian is effective in terms of enhancing the vehicle safety.

A camera or radar and a processing device that processes signals therefrom are used to sense a pedestrian in front of the vehicle with a sensor installed in the vehicle. In order to improve the sensing performance thereof, it is necessary to execute more detailed processes in the processing device. However, the computation resources of such a processing device are limited, and the processing device must simultaneously process other objects to be sensed in addition to the process for sensing a pedestrian. Thus, it is necessary to assign a priority to the processes and intensively execute the calculation processes. In order to achieve this, PTL 1 discloses one effective means for intensively executing processes in a scene in which there is a high possibility that a pedestrian exists, and PTL 1 further discloses an existence probability indicating the possibility that a pedestrian exists.

CITATION LIST

Patent Literature

SUMMARY OF INVENTION

Technical Problem

PTL 1 discloses finding an existence probability that a pedestrian is likely to move after a pedestrian has been detected, but does not disclose improving the performance itself of detecting a pedestrian. Therefore, in order to improve the pedestrian sensing performance itself, the pedestrian existence probability must be calculated before sensing a pedestrian to determine whether to intensively process the pedestrian.

An object of the present invention is to provide an object sensing device that improves the sensing performance of an object when processing to sense a plurality of objects to be sensed given limited computation resources.

Solution to Problem

To achieve the above object, an object sensing device of the present invention includes: an image capture unit that captures surroundings of a host vehicle; and a processing device that executes a sensing process of an object to be sensed from an image captured by the image capture unit, wherein the processing device includes: a scene analysis unit that analyzes a travel scene of the host vehicle; a process priority change unit that changes a sensing process priority of the object to be sensed based on the travel scene analyzed by the scene analysis unit; and an object-to-be-sensed sensing unit that senses the object to be sensed based on the sensing process priority changed by the process priority change unit.

Advantageous Effects of Invention

According to the invention, an object sensing device that improves the sensing performance of an object when processing to sense a plurality of objects to be sensed given limited computation resources can be provided.

DESCRIPTION OF EMBODIMENTS

Embodiments of the present invention are hereinafter described with reference to the drawings.

An embodiment of a stereo camera, which is an object sensing device, of the present invention will be explained below. Specifically, an embodiment of an object sensing device that senses a pedestrian using images of a stereo camera installed in a vehicle will be explained.

First, an overview of the object sensing device of the present invention will be explained usingFIG. 1.

FIG. 1is a block diagram realizing the object sensing device of the present invention. The object sensing device includes a stereo camera100, a left image capture unit101of the stereo camera, and a right image capture unit102of the stereo camera. The left image capture unit101and the right image capture unit102capture images of the front of the vehicle in which the stereo camera is installed. The captured images are processed in a processing device111.

The processing device111will now be explained in detail below.

An image from the right image capture unit102is input into a scene analysis unit103, and the scene analysis unit103analyzes the scene regarding what is captured in the image. The following explanation will focus on the processing of an image from the right image capture unit102, but an image from the left image capture unit101may also be processed in this way.

Next, in an external information acquisition unit104, information for calculating an existence probability of an object to be detected (pedestrian) is input from an external device such as a car navigation device installed in the vehicle.

Next, in an existence probability calculation unit105, an existence probability of an object to be detected (pedestrian) in the image captured by the right image capture unit102is calculated based on a scene of a subsequent image acquired in the scene analysis unit103and the information for calculating the existence probability acquired in the external information acquisition unit104.

Next, if the pedestrian existence probability is higher than a predetermined value, or for a portion in the image in which the pedestrian existence probability is higher than a predetermined value, a process priority change unit106changes a process priority so that the pedestrian sensing process is executed with priority over other objects to be detected (a preceding vehicle, a sign, a lane, etc.).

In a parameter changing unit107, the sensing process parameters are changed so that the pedestrian sensing process in a portion in which the pedestrian existence probability is high is executed in more detail. In a scene in which the pedestrian existence probability is high, the exposure control parameters of the camera (right image capture unit102) are changed to make adjustments so as to acquire an image in which a pedestrian can be easily sensed. Further, the image processing parameters of a portion in which the pedestrian existence probability is high within the image acquired by the right image capture unit102are changed to produce an image in which a pedestrian can be easily detected.

In a vehicle speed determination unit108, in a scene in which the pedestrian existence probability is higher than a predetermined value, a command for executing speed control by suppressing acceleration of the vehicle is generated and output to a vehicle speed control device.

Meanwhile, in a distance calculation unit103, an image captured by the left image capture unit101and an image captured by the right image capture unit102are input, and a distance to an object is calculated from a deviation in the images between the same object captured by the left image capture unit101and the right image capture unit102. In an object-to-be-sensed sensing unit110, a process is executed to sense an object to be sensed (pedestrian) using the prior image from the right image capture unit102and the distance information to the object calculated in the distance calculation unit109. Therein, the sensing process is executed based on the priority that was previously changed by the process priority change unit106, and the sensing process is executed using the parameters changed in the parameter changing unit107.

Next, the processes executed in the scene analysis unit103of the stereo camera100, which is the object sensing device, will be explained.

FIG. 2is a processing flow that is executed in the scene analysis unit103of the stereo camera100.

First, in a left-right image acquisition process201, images of the front of the vehicle captured by the left image capture unit101and the right image capture unit102of the stereo camera100are acquired. Next, in a distance data acquisition process202, data regarding the distance information of the images capturing the front of the vehicle that was calculated in the distance calculation unit109of the stereo camera100is acquired. The details of the distance calculation unit109will be explained later.

Next, in a road region extraction process203, a road region in the image is extracted using the two images of the front of the vehicle captured by the left image capture unit101and the right image capture unit102that were acquired in the left-right image acquisition process201. A road region is the portion outlined by the dotted line (road region301) in the image capturing the front of the vehicle (processing image300inFIG. 3), excluding other vehicles (parked vehicle302), structures outside the road (guard rail or shrubbery303) or (sidewalk without guard rail304), etc., and is a region in which the vehicle can travel.

The road region301can be extracted from the two images captured by the stereo camera100by the method disclosed in JP 2005-217883 A.

Next, in a parked vehicle detection process204, a parked vehicle302is detected from the processing image300capturing the front of the vehicle. In order to detect the parked vehicle302from the processing image300, first, the size of three-dimensional objects that exist is calculated in regions outside of the road region301previously extracted in the road region extraction process203using the distance information previously acquired in the distance data acquisition process202.

Herein, the distance information is the distance from the stereo camera100(vehicle) of objects captured in each pixel of the processing image300. From this distance information, for example, a vehicle height305, a vehicle width306, and a vehicle depth307of the parked vehicle302inFIG. 3can be calculated.

Next, among the three-dimensional objects whose size was calculated, those having height, width, and depth values near those of a vehicle are extracted. With regard to the height, width, and depth values of a vehicle, the value ranges of height, width, and depth of vehicles in the market are investigated in advance, and if the height, width, and depth values of a three-dimensional object are within these ranges, then the object is deemed to have a size equivalent to that of a vehicle.

Next, it is determined whether a side surface (vehicle side surface308inFIG. 3) of the three-dimensional object equivalent to a vehicle extracted previously has a texture similar to that of a vehicle. In this determination method, the textures of side surfaces of vehicles in the market are learned in advance, and it is determined whether the vehicle side surface in the processing image300and this learned data are similar. If it is determined that the three-dimensional object is a vehicle as a result of the vehicle side surface determination, it is then determined whether the vehicle is stopped.

In order to determine whether the vehicle is stopped, the processes indicated in the processing flow ofFIG. 2are similarly executed for the images of the previous frame and the frame before the previous frame, and a movement trajectory is calculated regarding where the same vehicle detected in the frame before the previous frame and the previous frame has moved in the image.

At this time, in determining whether the same vehicle exists in the frame before the previous frame, the previous frame, and the current frame, the vehicle side surface textures in each frame are compared using the vehicle side surface texture used when previously determining whether the three-dimensional object is a vehicle, and it is determined to be the same vehicle if the similarly of the side surface textures is high. Finally, the movement trajectory in the frame before the previous frame, the previous frame, and the current frame of the vehicle in the image calculated previously is compared to the speed of the host vehicle, and it is determined that the vehicle in the image is stopped if the movement of the background of the processing image300estimated from the speed of the host vehicle matches the movement of the trajectory of the vehicle in the image.

By the above-described processes, the parked vehicle302can be detected from the processing image300.

Next, in a road side condition determination process205, the attributes of the road shoulders outside of the road region301previously extracted in the road region extraction process203besides the portion of the parked vehicle302previously detected in the parked vehicle detection process204are determined. The attributes include the guard rail or shrubbery303, a building309, and the sidewalk without guard rail304. Herein, in determining whether an object is a guard rail or shrubbery303, the size of three-dimensional objects that exist is calculated using the distance information previously acquired in the distance data acquisition process202in regions outside of the road region301previously extracted in the road region extraction process203besides the portion of the parked vehicle302previously detected in the parked vehicle detection process204.

Herein, the distance information is the distance from the stereo camera100(vehicle) of objects captured in each pixel of the processing image300. From this distance information, the height of the three-dimensional objects is estimated. As a result, if the height of a three-dimensional object is within a certain fixed value, it is determined that the three-dimensional object is a guard rail or shrubbery. The certain fixed value is prepared as learned data by learning data regarding a typical guard rail and shrubbery in advance.

In determining whether an object is a building309, the size of three-dimensional objects that exist is calculated using the distance information previously acquired in the distance data acquisition process202in regions outside of the road region301previously extracted in the road region extraction process203besides the portion of the parked vehicle302previously detected in the parked vehicle detection process204and the portion determined to be a guard rail or shrubbery303in the road side condition determination process205. As a result, if the height of a three-dimensional object is equal to or greater than a certain fixed value, it is determined that the three-dimensional object is a building. The certain fixed value is prepared as learned data by learning data regarding the height of a typical building in advance.

In determining whether there is a sidewalk with no guard rail304, first image processing is executed outside of the road region301to extract a road border line310(solid white line). The road border line can be detected by the method disclosed in JP 2012-155399 A. If no stationary three-dimensional objects exist between the road border line310that was detected and the portion that was determined to be the building309in the road side condition determination process205, then it is determined that the sidewalk with no guard rail304exists. In determining whether a three-dimensional object is a stationary three-dimensional object, the trajectory of the target three-dimensional object in the frame before the previous frame, the previous frame, and the current frame is calculated, and if this trajectory matches the movement of the background of the processing image300estimated from the speed of the host vehicle, the three-dimensional object is determined to be a stationary three-dimensional object.

Next, in a crosswalk detection process206, it is determined whether there are road surface markings of a crosswalk within the road region301previously extracted in the road region extraction process203. A crosswalk can be detected from within the road region301by the method disclosed in JP 2011-192071 A, etc.

Finally, in a scene analysis diagram production process207, a scene analysis diagram of the road region301as shown inFIG. 3is produced. A scene analysis diagram is a diagram that describes what kind of objects exist in which regions within an image as shown inFIG. 4based on the results extracted in the parked vehicle detection process204, the road side condition determination process205, and the crosswalk detection process206described above.

FIG. 4illustrates guard rail or shrubbery regions401and402, a gap region403between a guard rail or shrubbery, a crosswalk region404, sidewalk regions without a guard rail405and406, parked vehicle regions407and408, and a gap region409between parked vehicles.

Next, the processes executed in the external information acquisition unit104of the stereo camera100will be explained.

Herein, external information is a car navigation device installed in the vehicle or a device outside the vehicle such as a sensor or other vehicle. A device outside the vehicle acquires information by a road-to-vehicle communication device called DSRC (Dedicated Short Range Communication), a mobile telephone, or a wireless LAN.

Herein, an example of acquiring information from a car navigation device will be explained. Attributes of a place where the host vehicle is traveling are delivered to the stereo camera100from the car navigation device.

Herein, the attributes of a place where the host vehicle is traveling are the attributes of urban area, residential area, commercial facility, school, road with few vehicles, and place where the density of intersection is high, which are places where the probability that a pedestrian exists is high, and conversely, the attributes of highway, elevated road, road with many vehicles, place with few buildings, mountainous area, and road with few intersections, which are places where the probability that a pedestrian exists is low.

The car navigation device specifies the location of the host vehicle on map data within the car navigation device based on GPS (Global Positioning System) position data, and transmits the above-described place attributes regarding the probability of excessive pedestrians around the host vehicle to the stereo camera100.

Next, the processing in the existence probability calculation unit105of the stereo camera100will be explained in detail.

In the existence probability calculation unit105, an existence probability regarding whether the possibility that a pedestrian exists in the image captured by the right image capture unit102is high or low is calculated based on the image scene acquired in the scene analysis unit103and the information regarding the attributes of the place where the host vehicle is traveling acquired in the external information acquisition unit104as described above.

Herein, in calculating the existence probability, learned data as shown inFIG. 5is prepared based on the results of investigations conducted in advance, and the pedestrian existence probability is calculated by referring to this learned data. In the table of learned data shown inFIG. 5, image scene types501are given on the vertical axis, including guard rail or shrubbery, gaps between guard rail or shrubbery, crosswalks, sidewalks without guard rail, parked vehicles, and gaps between parked vehicles, which are elements of the scene captured by the stereo camera100in the scene analysis diagram production process207among the processes executed in the scene analysis unit103of the stereo camera100.

Meanwhile, the horizontal axis502shows attributes of the places where the vehicle is traveling acquired in the external information acquisition unit104of the stereo camera100, including urban area, residential area, commercial facility, school, highway, elevated road, mountainous area, and road with few intersections.

The numbers listed in the table as the values503of the pedestrian existence probability indicate the pedestrian existence probability. For example, if the image scene is a guard rail/shrubbery and the place attribute is an urban area, the probability that a pedestrian exists is 10%.

Herein, in calculating the probability of the values503of the pedestrian existence probability, pre-acquired images are investigated to actually check the probability that a pedestrian exists, and thereby probability values are prepared as empirical values.

Next, a pedestrian existence probability is assigned to the scene analysis diagram ofFIG. 4produced in the scene analysis diagram production process207executed in the scene analysis unit103of the stereo camera100based on the learned data regarding the pedestrian existence probability as shown inFIG. 5. Considering an example when the place attribute of the scene acquired in the external information acquisition unit104of the stereo camera100is a commercial facility, for example, referring to the table inFIG. 5, the pedestrian existence probability in the gap region403between a guard rail or shrubbery is 90% based on a value504of the pedestrian existence probability inFIG. 5. Similarly, with regard to the other guard rail or shrubbery regions401and402, the crosswalk region404, the sidewalk regions without a guard rail405and406, the parked vehicle regions407and408, and the gap region409between parked vehicles, a pedestrian existence probability is assigned to each of the above in the scene analysis diagram ofFIG. 4as shown inFIG. 6referring to the existence probabilities from the table inFIG. 5.

InFIG. 6, the regions601indicated with a thick solid line frame are regions with a pedestrian existence probability of 90% (the regions403,404,405,406, and409inFIG. 6), the regions602indicated with a thin solid line frame are regions with a pedestrian existence probability of 60% (the regions407and408), and the regions603indicated with a thin dotted line frame are regions with a pedestrian existence probability of 30% (the regions401and402).

Next, the processes executed in the process priority change unit106of the stereo camera100will be explained in detail.

In the process priority change unit106, if the pedestrian existence probability is higher than a predetermined value, or for a portion in the image in which the pedestrian existence probability is higher than a predetermined value, the process priority is changed so that the pedestrian sensing process is executed with priority over other objects to be detected (a preceding vehicle, a sign, a lane, etc.).

FIG. 7shows an overview of the process priority changing. As a result of calculating the pedestrian existence probability of the current scene in the existence probability calculation unit105of the stereo camera100, if there is a region in the scene in which the probability is at or above a certain fixed value, the process priority is changed as shown inFIG. 7.FIG. 7illustrates a process schedule704before the priority change and a process schedule705before the priority change.

In the process schedule704before the priority change, considering an example in which a pedestrian sensing process, a vehicle sensing process, and a sign sensing process are executed in the stereo camera100, a pedestrian sensing process701, a vehicle sensing process702, and a sign sensing process703are all executed sequentially in a period of 90 ms, such that the pedestrian sensing process701is executed first at 0 ms, the vehicle sensing process702is executed next, the sign sensing process703is executed last, and then a pedestrian sensing process706is executed again at 90 ms.

As a result of calculating the pedestrian existence probability of the current scene, if there is a region in the scene in which the probability is at or above a certain fixed value, the process priority is changed to the process schedule705before the priority change ofFIG. 7. In other words, the process priority is changed such that a pedestrian sensing process707is executed first at 0 ms, a vehicle sensing process708is executed next, then a pedestrian sensing process709is executed again, a sign sensing process710is executed last, and then a pedestrian sensing process711is executed again at time 120 ms.

Thereby, the pedestrian sensing process is executed in a 60 ms period, and the vehicle sensing process and the sign sensing process are executed in a period of 120 ms. By repeatedly executing the pedestrian sensing process with priority, the pedestrian sensing performance can be improved.

Next, the processes executed in the parameter changing unit107of the stereo camera100will be explained in detail.

FIG. 8illustrates a processing flow of the processes that are executed in the parameter changing unit107. First, in a region of high probability of pedestrians extraction process801, the regions having an existence probability at or above a certain fixed value are extracted from the pedestrian existence probabilities (FIG. 6) calculated in the existence probability calculation unit105of the stereo camera100. If the certain fixed value is 80%, the regions601indicated with a thick solid line frame inFIG. 6(the regions403,404,405,406, and409) will be extracted.

Next, in a pedestrian sensing logic changing process802, if there are existence probability values that are at or above the certain fixed value among the pedestrian existence probabilities (FIG. 6) calculated in the existence probability calculation unit105of the stereo camera100, a logic of the pedestrian sensing process is changed so that the pedestrian sensing process in the regions with a high pedestrian existence probability extracted previously in the region of high probability of pedestrians extraction process801is executed in more detail. Herein, in the pedestrian sensing process, pedestrians can be detected by making a determination in comparison with data resulting from learning many pedestrian images using an image feature quantity called HOG (Histograms of Oriented Gradients), which is described in the following Non-Patent Literature: “N. Dalal and B. Triggs, ‘Histograms of Oriented Gradients for Human Detection’, IEEE Symposium on Intelligent Vehicle, pp. 206-212, June, 2006”. In this process, the determination process can be made more detailed by adding another sensing process using a second feature quantity other than HOG, thereby improving the detection performance.

Further, in this process, the pedestrian sensing performance can be improved by, when making a determination in comparison to data that has been learned in advance using image feature quantities, lowering the determination threshold almost to the point of oversensitivity, and then adding a detailed determination regarding whether the movement of each part of the pedestrian, i.e. the head, shoulders, and legs of the pedestrian, resemble that of a pedestrian.

Next, in a pedestrian sensing region changing process803, the regions having an existence probability at or above the certain fixed value among the pedestrian existence probabilities (FIG. 6) calculated in the existence probability calculation unit105of the stereo camera100are extracted and set as processing regions in which a pedestrian viewpoint is executed with priority. For example, inFIG. 6, settings are implemented so that the pedestrian sensing process is executed at a high frequency of a period of 60 ms for the regions including the regions601and602in which the pedestrian existence probability is 60% or greater, whereas the pedestrian sensing process is not executed or executed at a low frequency in the other regions.

Next, in an image preprocessing parameter changing process804, the regions having an existence probability at or above the certain fixed value among the pedestrian existence probabilities (FIG. 6) calculated in the existence probability calculation unit105of the stereo camera100are extracted, and preprocessing parameters for these regions in which the pedestrian existence probability is high are changed to produce images in which a pedestrian can be easily sensed. Herein, in a case in which the image is overexposed in white or darkened in portions where the pedestrian existence probability is high such that it is difficult to detect a pedestrian, the entire image is subjected to gradation correction so that pedestrians are displayed with good contrast. Alternatively, gradation correction is conducted only in portions including the regions in which the pedestrian existence probability is at or above the certain fixed value that were previously extracted so that pedestrians are displayed with good contrast.

Finally, in an exposure control parameter changing process805, in a scene in which the pedestrian existence probability is high, the exposure control parameters of the left image capture unit101and the right image capture unit102of the stereo camera100are changed and adjusted so as to acquire an image in which a pedestrian can be easily sensed. Herein, the regions having an existence probability at or above the certain fixed value among the pedestrian existence probabilities (FIG. 6) calculated in the existence probability calculation unit105of the stereo camera100are extracted and the brightness in these portions of the image is extracted, and then the exposure control parameters are changed so that the exposure time is shortened if the portions are bright and the exposure time is lengthened if the portions are dark.

Next, the processes executed in the vehicle speed determination unit108of the stereo camera100will be explained.

In the vehicle speed determination unit108, in a scene in which the pedestrian existence probability is high, a command for executing speed control by suppressing acceleration of the vehicle is generated and output to a vehicle speed control device.

In other words, if there are existence probability values that are at or above the certain fixed value among the pedestrian existence probabilities (FIG. 6) calculated in the existence probability calculation unit105of the stereo camera100, even if the vehicle speed is lower than a user set speed of an ACC (Adaptive Cruise Control) of the vehicle, control is performed to suppress the vehicle speed without allowing the vehicle to accelerate to the set speed.

Further, the speed limit of the road where the vehicle is currently traveling is acquired from the car navigation device in the external information acquisition unit104of the stereo camera100, and if the vehicle speed is higher than the speed limit, deceleration control is performed to decelerate the vehicle to the speed limit.

Next, the processes executed in the distance calculation unit109of the stereo camera100will be explained in detail using the flowchart ofFIG. 9.

In the flowchart ofFIG. 9, first, in a left image input process901, image data captured by the left image capture unit101is received. Next, in a right image input process902, image data captured by the right image capture unit102is received. Herein, the left image input process901and the right image input process902can be simultaneously executed as parallel processes.

Next, in a corresponding point calculation process903, the two left and right images acquired in the left image input process901and the right image input process902are compared, and portions where the same object is captured are specified. As shown inFIG. 10, when an object1001, which is an object on the travel path, is captured by the stereo camera100, the images captured by the left image capture unit101and the right image capture unit102appear as the left image1002and the right image1003. Herein, the identical object1001is captured at an object position1004in the left image1002and is captured at an object position1005in the right image1003, and thus a deviation d1in the horizontal direction occurs between the images. Therefore, it is necessary to specify where the object captured at the object position1004of the left image1002is captured in the right image1003.

A method for specifying where the specific object captured in the left image1002is captured in the right image1003will now be explained usingFIG. 11.

InFIG. 11, in the coordinate system of the left image1002and the right image1003, the horizontal axis is a u-axis1101and the vertical axis is a v-axis1102. First, in the left image1002, a rectangular region1103defined by (u1, v1), (u1, v2), (u2, v1), (u2, v2) in the uv coordinate system is set.

Next, in the right image1003, the U value is increased from u=0 to u=u3 so that a region defined by (U, v1), (U, v2), (U+(u2−u1), v1), (U+(u2−u1), v2) is scanned up to a rectangular region1104in the rightward direction of the image. When scanning, the correlation values of the image within the rectangular region1103and the image within the rectangular region1104are compared, and it is determined that an object which is identical to the object captured in the rectangular region1103is captured at a position (u4, v1), (u4, v2), (u4+(u2−u1), v1), (u4+(u2−u1), v2) of a rectangular region1105in the right image1003where the correlation with the rectangular region1103of the left image1002is the highest. Herein, the pixels within the rectangular region1103are regarded as corresponding to the pixels within the rectangular region1105. Herein, when scanning the rectangular region1104of the right image1003, if there are no rectangles in which the correlation value is at or above a certain fixed value, it is determined that there are no corresponding points in the right image1003that correspond to the rectangular region1103of the left image1002.

Next, the rectangular region1103of the left image1002is shifted to the position of a rectangular region1106, and the same process is executed.

In this way, rectangular regions in the left image1002are scanned throughout the entire left image1002, and corresponding points within the right image1003are found for all of the pixels in the left image1002. If no corresponding points are found, then it is determined that no corresponding points exist.

In the distance calculation process904, with regard to the corresponding points of the left image1002and the right image1003capturing the same object found in the corresponding point calculation process903described above, the distance from the stereo camera100of the corresponding points is calculated.

First, usingFIG. 12, a method for calculating a distance from the camera of an object point1201in the left image1002and the right image1003will be explained.

InFIG. 12, the left image capture unit101is a camera having focal length f and optical axis1208of the left image capture unit, and consisting of a lens1202and an image capture surface1203, and the right image capture unit102is a camera having focal length f and optical axis1209of the right image capture unit, and consisting of a lens1204and an image capture surface1205. The object point1201in front of the cameras is captured at a point1206(a distance d2from the optical axis1208) on the image capture surface1203of the left image capture unit101, and becomes the point1206(a position of d4pixels from the optical axis1208) in the left image1002. Similarly, the object point1201in front of the cameras is captured at a point1207(a distance d3from the optical axis1209) on the image capture surface1205of the right image capture unit102, and becomes the point1207(a position of d5pixels from the optical axis1209) in the right image1003.

In this way, the object point1201of the same object is captured at a position of d4pixels toward the left from the optical axis1208in the left image1002, and is captured at a position of d5toward the right from the optical axis1209in the right image1003. Thus, a parallax of d4+d5pixels is generated.

Therefore, if the distance between the optical axis1208of the left image capture unit101and the object point1201is x, a distance D from the stereo camera100to the object point1201can be calculated by the following formulas.

From the relationship between the object point1201and the left image capture unit101d2:f=x:D

From the relationship between the object point1201and the right image capture unit102d3:f=(d−x):D

Thus, D=f×d/(d2+d3)=f×d/{(d4+d5)×a}. Herein, a is the size of the image capture elements of the image capture surface1203and the image capture surface1205.

The distance calculation described above is carried out for all of the corresponding points calculated in the corresponding point calculation process903described above. As a result, a distance image expressing the distance from the stereo camera100to the object can be found.

In a distance information output process905of the flowchart ofFIG. 9, this distance image is output and saved.

Finally, at a branch906of the flowchart ofFIG. 9, if there are image input signals from the left image capture unit101and the right image capture unit102, the process returns to the left image input process901. At the branch906, if there are no image input signals from the left image capture unit101and the right image capture unit102, the process enters standby until image input signals are input into the distance calculation unit109.

Finally, the processes executed in the object-to-be-sensed sensing unit110of the stereo camera100will be explained. In the object-to-be-sensed sensing unit110, each sensing process is initiated following the process schedule shown in the process schedule705before the priority change ofFIG. 7determined in the process priority change unit106of the stereo camera100as described above. In the example ofFIG. 7, a pedestrian sensing process, a vehicle sensing process, and a sign sensing process are executed sequentially. The results of sensing are output from the stereo camera100to an external device.

Here, another embodiment in which the present invention is applied to a system for sensing a pedestrian using images of a stereo camera installed in a vehicle is shown inFIG. 1.

A stereo camera1300, which is an image capture device, has a left image capture unit1301and a right image capture unit1302. A processing device1311has the scene analysis unit103, the external information acquisition unit104, the existence probability calculation unit105, the process priority change unit106, the parameter changing unit107, the vehicle speed determination unit108, the distance calculation unit109, and the object-to-be-sensed sensing unit110. The processing content in each unit from the scene analysis unit103to the object-to-be-sensed sensing unit110is the same as that described in Embodiment 1.

In the present embodiment, the stereo camera1300and the processing device1311can be in separate housings. The stereo camera1300and the processing device1311are connected by a single or a plurality of signal lines, and the image captured by the left image capture unit1301and the image captured by the right image capture unit1302are sent to the processing device1311.

REFERENCE SIGNS LIST

100stereo camera101left image capture unit102right image capture unit103scene analysis unit104external information acquisition unit105existence probability calculation unit106process priority change unit107parameter changing unit108vehicle speed determination unit109distance calculation unit110object-to-be-sensed sensing unit111processing device201left-right image acquisition process202distance data acquisition process203road region extraction process204parked vehicle detection process205road side condition determination process206crosswalk detection process207scene analysis diagram production process