Patent Description:
As a background technology in this technical field, PTL <NUM> proposes a technique of suppressing a decrease in a calculation time and an increase in a circuit scale in distance detection by a stereo camera, for example.

Specifically, PTL <NUM> describes a technique of narrowing a search range in an upper area than a parallax search range in a lower area on a screen in a case where there are no neighboring objects.

<CIT> shows how stereoscopic images are acquired to compute parallax wherein a first arithmetic processing method having high distance resolution to a far-distance region; a second arithmetic processing method having lower distance resolution but higher processing speed than the first arithmetic processing method, to an intermediate-distance region; and a third arithmetic processing method having lower distance resolution but higher processing speed than the second arithmetic processing method, to a near-distance region.

In <CIT> shows an obstruction measuring method that measures the distance to the obstruction by parallax calculating through a stereo image.

In <CIT> a roadside object detection apparatus is capable of detecting a roadside object such as a far-off curbstone.

<CIT> shows how to detect roughness or an obstacle on a road surface from images captured by using a stereo camera.

<CIT> provides an environment recognition device which recognizes an arrow signal of a traffic light and recognizes the arrow signal on a local vehicle at a position far away from the traffic light.

<CIT> discloses that based on images from a stereoscopic camera and parallax calculations, the distance to an object is obtained. First, in a low resolution image, rough distance to an object is computed. Then, depending on the obtained rough distance, a pixel size for the parallax pixel window size and a computation method are selected in a more precise high resolution image area.

In the technology described in PTL <NUM>, a calculation load can be decreased in the case where there is no object to be detected in the lower area on the screen. However, typically, the object to be detected exists in the neighbor and there are many cases where the search range in the upper area on the screen cannot be narrowed, and the calculation load may not be able to be efficiently decreased.

Therefore, the purpose of the present invention is to provide an object distance detection device capable of both improving object detection accuracy and reducing a calculation load.

The present invention that solves the above issue is an object distance detection device according to claim <NUM>. Dependent claims relate to preferred embodiments of present invention.

The present invention can provide an imaging device capable of both improving object detection accuracy and reducing a calculation load.

<FIG> is a diagram illustrating a configuration of an embodiment of an imaging device of the present invention. An imaging device <NUM> of the present embodiment is mounted in the front of a vehicle, for example, and constitutes a part of a safety system that recognizes signals, obstacles, and the like to assist a driver.

Imaging units <NUM> and <NUM> has an optical lens mounted on an image sensor. These imaging units repeat imaging of one image at predetermined timing and output an imaged image.

The imaging unit <NUM> and the imaging unit <NUM> are installed apart from each other in a left and right direction with a predetermined distance and can calculate a distance to an object from a displacement between the images captured by the imaging unit <NUM> and the imaging unit <NUM>, a so-called parallax.

Note that <FIG> illustrates an example in which constituent elements of the imaging device <NUM> are accommodated in the same housing. However, for example, the imaging units <NUM> and <NUM> may be collectively accommodated in a different housing from other constituent elements (the dotted frame <NUM> in <FIG>) or may be respectively housed in different housings and attached to the vehicle. In this case, image signals may be connected by connection cables (not illustrated). An example of a method of transmitting an image using a connection cable includes a transmission method using a differential transmission line of a low voltage differential signaling (LVDS) system.

Further, color image sensors are adopted as image sensors of the imaging unit <NUM> and the imaging unit <NUM>, thereby to acquire color information of the captured images.

An image correction unit <NUM> takes in the images from the imaging units <NUM> and <NUM>, performs correction to adjust luminance of the images with a correction value measured in advance, and further corrects distortion of the images by a lens and performs correction to adjust horizontal positions of the images of the imaging units <NUM> and <NUM> with a correction value measured in advance. Measurement of the correction values is performed in a process of manufacturing the imaging device. For each device before application of collection values, a specific object is imaged, a luminance correction value of each pixel, which makes the luminance of the acquired images uniform, and a geometric correction value of each pixel, which cancels lens distortion and makes the images positioned horizontal, are obtained, and the correction values are stored in a nonvolatile memory (not illustrated) for each device as correction tables.

An object information acquisition unit <NUM> acquires object information including distance information of an object. According to the invention, the object information acquisition unit <NUM> detects an object from an image acquired from at least one of the plurality of imaging units, and detects a distance of the object. That is, the object information acquisition unit <NUM> detects an object by monocular processing and thus can be referred to as a monocular distance detection unit. Further, the object information acquisition unit <NUM> can be specified as a first distance detection unit from a relationship with a stereo distance detection unit to be described below.

The object information acquisition unit <NUM> inputs the image of either the imaging unit <NUM> or the imaging unit <NUM> and detects a distance to an area of the object. As a method of detecting an object, there is a following method, for example. The object information acquisition unit <NUM> takes in an image of either the imaging unit <NUM> or the imaging unit <NUM> and detects an assumed traffic signal, road sign, or the like, in the taken image. An example of the detection method includes a method of detecting an object such as a traffic signal or a road sign from a similarity amount between luminance distribution or edge shape information in the image and pattern data held as reference data. The object in the image and the position of the object on the screen can be grasped by the method. Furthermore, the object information acquisition unit <NUM> can roughly detect the distance of the object from the height and size of the detected object on the screen, for example. The object information acquisition unit <NUM> outputs a detection result to a processing area setting unit <NUM> and a search range setting unit <NUM> to be described below.

A stereo distance detection unit <NUM> is another distance detection unit (that is, a second distance detection unit), and inputs images from the image correction unit <NUM> and detects a distance of an object. As a method of detecting a distance, there is a following method, for example. The stereo distance detection unit <NUM> takes in the images from the image correction unit <NUM>, and calculates a parallax. As described above, since the imaging unit <NUM> and the imaging unit <NUM> are installed apart from each other in the left and right direction with a predetermined distance, the imaged images have a parallax. So-called stereo processing of calculating the parallax is performed. An example of the parallax calculation method includes a block matching method. The stereo distance detection unit <NUM> detects the distance of an area of an image specified by the processing area setting unit <NUM> to be described below, of the images from the image correction selection unit <NUM>, for example. Specifically, first, the stereo distance detection unit <NUM> searches an area having the same object appear on the image of the imaging unit <NUM>, the area corresponding to a small block area having a predetermined size cut out from a specified image area of the imaging unit <NUM>, by shifting one pixel at a time in a horizontal direction. At that time, the stereo distance detection unit <NUM> searches a search range of the number of pixels specified by the search range setting unit <NUM> to be described below, as the search range in the horizontal direction. Then, a difference in position between the matched block areas in the imaging unit <NUM> and the imaging unit <NUM> becomes the parallax. The distance in a real environment of the object appearing in the block area can be obtained using this parallax. Note that, in this example, the block area is adopted as an image element of which the distance is to be obtained. As a matching and comparison method, for example, a position at which the sum of differences in luminance of pixels in the block area becomes small is employed as the parallax. Note that it is known that the detected distance can be obtained from lens focal lengths of the imaging unit <NUM> and the imaging unit <NUM>, the distance between the imaging unit <NUM> and the imaging unit <NUM>, the above-obtained parallax, and a pixel pitch of the imaging sensors. However, the distance calculation method is not limited to this example. Further, the image element of which the distance is to be obtained is not limited to the above-described block area, and individual pixels constituting the imaging sensors may be adopted.

A search condition setting unit <NUM> sets a condition for searching, by the stereo distance detection unit <NUM>, for an image element corresponding to a specific image element in a standard image captured by one of the plurality of imaging units, a reference image captured by another imaging unit. Specifically, the search condition setting unit <NUM> includes the processing area setting unit <NUM> and the search range setting unit <NUM>.

The processing area setting unit <NUM> specifies an area of an image of which the distance is to be detected by the stereo distance detection unit <NUM> to be described below and specifies a position of an image of which an object is to be recognized by a recognition unit <NUM> to be described below on the basis of the result of the object information acquisition unit <NUM>.

The search range setting unit <NUM> sets a search range for detecting a distance by the stereo distance detection unit <NUM> on the basis of the result of the object information acquisition unit <NUM>.

The stereo distance detection unit <NUM> detects the distance of the specified area of the image from the correction unit <NUM> as described above and outputs a result to the recognition unit <NUM> to be described below. By the processing, limitation of the area of the distance detection by the distance stereo distance detection unit <NUM> and the search range becomes possible, and the increase in the processing load can be avoided.

The recognition unit <NUM> receives the detection result from the stereo distance detection unit <NUM> and the area specification from the processing area setting unit <NUM>, recognizes the object on the image, and outputs information of a recognition result to the outside of the imaging device <NUM>. The recognition unit <NUM> recognizes the object on the basis of the distance information obtained by the stereo distance detection unit <NUM>, of the area specified by the processing area setting unit <NUM>. As a method of recognizing an object, in a case where pieces of distance information indicating nearly the same distance exist in the vicinity, for example, the pieces of distance information are made into one group and is recognized as the object when the size of the group has a fixed value or more. Then, the recognition unit <NUM> detects that the object is a vehicle or a pedestrian, for example, on the basis of the size and shape of the detected group. There is a method of detecting the size and shape of the object from comparison with pattern data held as reference data in advance. According to this processing method, the distance from a user's own vehicle to an object in front such as a pedestrian or a vehicle can be obtained with high accuracy. Therefore, the obtained distance is used as information for avoiding a collision, such as for deceleration and stop of the user's own vehicle.

Note that, in the imaging device <NUM>, the imaging units <NUM> and <NUM>, the image correction unit <NUM>, and the stereo distance detection unit <NUM> in the dotted frame <NUM> are constituted by an electronic circuit, and the other constituent elements are realized by software processing by a microcomputer (not illustrated), for example.

<FIG> is a diagram illustrating an example of imaged image imaged in an embodiment of the imaging device of the present invention. <FIG> illustrates an imaged image <NUM> imaged by the imaging unit <NUM> and corrected by the correction unit <NUM>, and an imaged image <NUM> imaged by the imaging unit <NUM> and corrected by the correction unit <NUM>. <FIG> illustrates objects <NUM>, <NUM>, and <NUM>.

Further, <FIG> illustrates commonly imaged areas <NUM> and <NUM> that are commonly imaged areas of the imaged image <NUM> and the imaged image <NUM>. As described above, there is a displacement of the commonly imaged area between the imaged image <NUM> and the imaged image <NUM>, and the distance of the object is calculated by the displacement amount, that is, the parallax.

<FIG> is a diagram illustrating an example of an imaged image imaged in an embodiment of the imaging device of the present invention and area control. An area <NUM> in <FIG> indicates an area of the imaged image imaged by the imaging unit <NUM> and corrected by the correction unit <NUM>, the area <NUM> having been commonly imaged with the image imaged by the imaging unit <NUM> as described above, for example.

Processing areas <NUM>, <NUM>, and <NUM> are processing areas specified by the processing area setting unit <NUM>, of the imaged image <NUM>, and are processing areas for which the distance detection processing is performed by the stereo distance detection unit <NUM>. That is, the processing areas <NUM>, <NUM>, and <NUM> are partial areas of the common imaging area, which are specified by the processing area setting unit <NUM> from results of detection of a pedestrian <NUM>, an oncoming vehicle <NUM>, and a traffic signal <NUM> and of rough distance detection processing of the detected objects by the object information acquisition unit <NUM>. In each of these areas, the distance of the image element in the partial area is calculated on the basis of the parallax between the plurality of captured images by the stereo distance detection unit <NUM>.

<FIG> is a diagram illustrating a relationship between the distance and the parallax of an embodiment of the imaging device of the present invention, and an example of a method of setting the search range by the search range setting unit <NUM>. The parallax is expressed by a unit of pixels, and indicates the displacement amount of the image captured by the imaging unit <NUM> from the image captured by the imaging unit <NUM>. For example, in the case where the distance of the pedestrian <NUM> is detected as about <NUM> by the object information acquisition unit <NUM>, the parallax in this case is <NUM> pixels from <FIG>. Therefore, the search range setting unit <NUM> sets a range from <NUM> to <NUM> pixels around the <NUM> pixels to the processing area <NUM> as the search range of the distance of the pedestrian <NUM>. Similarly, in the case where the distances of about <NUM> and about <NUM> are respectively detected for the processing areas <NUM> and <NUM> that are the areas of the traffic signal <NUM> and the oncoming vehicle <NUM>, the search range setting unit <NUM> sets a range from <NUM> to <NUM> pixel and a range from <NUM> to <NUM> pixels as the search ranges, respectively. With the setting, the range of the limited number of pixels around the distance detected by the object information acquisition unit <NUM> is simply searched in each processing area for the parallax, and the processing load is reduced. Further, in the case where the distance cannot be detected in the set search range, the set search range is expanded and the search processing for the expanded range is performed, whereby the processing load can be minimized. For example, a case in which a character on a roadside poster is erroneously detected as a person, and the object information acquisition unit <NUM> detects a distance that is different from an actual distance is expected.

<FIG> is a diagram illustrating processing timing of an embodiment of the imaging device of the present invention. In <FIG>, (<NUM>-<NUM>) illustrates processing timing of the object information acquisition unit <NUM> and (<NUM>-<NUM>) illustrates processing timing of the stereo distance detection unit <NUM>.

In (<NUM>-<NUM>), the object detection and rough distance detection processing for the object are performed for the imaged image <NUM> by the object information acquisition unit <NUM> as described above. Further, in (<NUM>-<NUM>), the distance detection is performed by the stereo distance detection unit <NUM>, for the search range of each processing area specified by the search range setting unit <NUM>, for each of the processing areas <NUM>, <NUM>, and <NUM> specified by the processing area setting unit <NUM>. As the order of processing, an area including an object close to the user's own vehicle is processed first, whereby early recognition by the subsequent recognition unit <NUM> becomes possible and safety can be secured.

In this manner, the distance detection processing by the stereo distance detection unit <NUM> is performed for only the specified necessary processing area in the minimum search range necessary in the processing area. Therefore, distance search of the entire range is not necessary for all the areas of the imaged image, and the processing load can be decreased.

<FIG> is a diagram illustrating a processing flow of an embodiment of the imaging device of the present invention. First, images are captured by the imaging units <NUM> and <NUM> (S601: S represents a step). The image correction unit <NUM> performs the luminance correction, lens distortion correction, and horizontal alignment for the captured images (S602). Next, the object information acquisition unit <NUM> detects the object and its approximate distance (S603). Among the detection results, the processing area setting unit <NUM> outputs area information to be processed from position information of the detected object to the stereo distance detection unit <NUM> (S604), and the search range setting unit <NUM> determines the search range to be searched in the area including the object from the distance information of the detected object and outputs the search range to the stereo distance detection unit <NUM> (S605).

Next, the stereo distance detection unit <NUM> detects a detailed distance in the search range in each specified area on the basis of the obtained detection result (S406).

Finally, the recognition unit <NUM> performs object recognition processing on the basis of the distance detection result of the object in each processing area, and outputs the recognition result (S607). These processes are repeated, for example, every frame.

<FIG> is a diagram illustrating an example of a captured image and a recognition result in an embodiment of the imaging device of the present invention. An image <NUM> is imaged by the imaging unit <NUM> at a certain point of time, and the imaging unit <NUM> also captures and acquires a substantially similar image. Further, recognition results <NUM>, <NUM>, and <NUM> are recognition results of objects. The frames and distance display in the image are not imaged images and are explicitly superimposed on the image.

The pedestrian <NUM> detected from the processing area <NUM> being positioned at a distance <NUM>, the oncoming vehicle <NUM> detected from the processing area <NUM> being positioned at a distance <NUM>, and the traffic signal <NUM> detected from the processing area <NUM> being positioned at a distance <NUM>. <NUM> are illustrated. As described above, the object distance detection with high accuracy can be realized over the entire captured image.

According to the present embodiment, the object and the distance of the object are roughly detected from the captured image, and the accurate distance detection processing is performed for the area including the object on the basis of the detection result, limiting the search range of the distance. Therefore, the object recognition of the entire imaged image becomes possible without the increase in the processing load.

<FIG> is a diagram illustrating a configuration of another embodiment of an imaging device of the present invention. An imaging device <NUM> is mounted on a vehicle such as an automobile, and a vehicle control unit <NUM> is illustrated in <FIG>. An output of a recognition unit <NUM> is input to the vehicle control unit <NUM>. Further, the recognition unit <NUM> includes a road surface detection unit <NUM> that separates a road surface portion from other objects and detects the road surface portion as a road surface area from distance information in a captured image, and outputs the area to an object information acquisition unit <NUM>. The recognition unit <NUM> further includes a travel route prediction unit <NUM> that inputs vehicle information such as a vehicle speed and a steering angle although not illustrated and predicts a travel route of a user's own vehicle, and outputs the travel route to the object information acquisition unit <NUM>.

The object information acquisition unit <NUM> inputs the road surface information detected by the road surface detection unit <NUM> and preferentially processes an object on the road surface, and a processing area setting unit <NUM>, a search range setting unit <NUM>, and a stereo distance detection unit <NUM> detect an accurate distance, and the recognition unit <NUM> performs recognition, whereby an obstacle of the user's own vehicle can be promptly recognized and safe traveling can be maintained. Further, the object information acquisition unit <NUM> preferentially processes an object on the road surface to travel from the travel route prediction unit <NUM> and recognizes an obstacle on the travel route of the user's own vehicle, as described above, to realize the safe traveling.

Further, the search range setting unit <NUM> inputs a vehicle speed of the user's own vehicle, and sets a wider search range in the case of a fast speed and sets a narrower search range in the case of a slow speed, thereby to enable reliable distance detection processing with a minimum necessary processing amount.

Further, as illustrated in <FIG>, the processing area setting unit <NUM> sets a processing area <NUM> that is an entire image as a processing area, for example, and at that time, a search control unit sets a range from <NUM> to <NUM> pixels around <NUM> pixels as a search range, as described above, and a near distance such as <NUM> to <NUM> of the user's own vehicle is detected, whereby an entire screen can be searched for distance, for the vicinity of the user's own vehicle, and safety can be enhanced.

Further, the object information acquisition unit <NUM> collectively processes a plurality of adjacent objects, the processing area setting unit <NUM> sets the plurality of objects as one area, and the search range setting unit <NUM> sets the search range that includes distances of the objects within the one area collectively detected by the object information acquisition unit <NUM>, whereby the case of a plurality of objects can be handled.

The vehicle control unit <NUM> in <FIG> receives a recognition result by the recognition unit <NUM>, and controls devices (not illustrated) of the vehicle. The control of the vehicle includes lighting of a warning lamp to a driver due to detection of approach of a pedestrian, a red light signal, or a road sign, generation of a warning sound, deceleration and stop control by braking, throttle and brake control at the time of following a vehicle ahead, steering angle control for collision avoidance and lane keeping, and the like. These pieces of vehicle control information is output from the imaging device <NUM> to other devices (not illustrated) via an in-vehicle network.

Note that <FIG> illustrates the example in which the vehicle control unit <NUM> is accommodated in the same housing as the imaging device <NUM>. However, an embodiment is not limited to the example, and imaging units <NUM> and <NUM> may be accommodated in a separate housing, as described above.

<FIG> is a diagram illustrating a configuration of still another embodiment of an imaging device of the present invention. <FIG> illustrates a network imaging unit <NUM>, a local area network (LAN) <NUM>, and a control unit <NUM>. The network imaging unit <NUM> is connected with the control unit <NUM> via the LAN <NUM>. Further, <FIG> illustrates an image compression/interface unit <NUM>, a network interface unit <NUM>, and an image decompression unit <NUM>.

Images captured by an imaging unit <NUM> and an imaging unit <NUM> undergo luminance correction, lens distortion correction, and horizontal alignment by an image correction unit <NUM>. Next, the image compression/interface unit <NUM> compresses the images from the image correction unit <NUM> and transmits the images to the LAN <NUM>. An example of an image compression method includes a method using an in-screen compression method of performing compression in one image without using temporal correlation of a plurality of images to reduce a processing time. Alternatively, a video compression coding method may be selected and the image compression method may be switched.

The image compression/interface unit <NUM> generates compression coded data and transmits the data according to a predetermined network protocol. Note that achievement of high efficiency and high image quality is expected by having the processing of the image correction unit <NUM> at a front stage of the image compression/interface unit <NUM>, such as performing compression after correction of lens distortion. However, the image correction unit <NUM> may be provided at a subsequent stage of the image decompression unit <NUM> of the control unit <NUM>.

In the control unit <NUM>, the network interface unit <NUM> receives the compressed image data via the LAN <NUM>. The compressed image data received by the network interface unit <NUM> of the control unit <NUM> is decompressed into the original image by the image decompression unit <NUM>, and the above-described processing is performed by a processing area setting unit <NUM> and a search range setting unit <NUM> for an object detected by an object information acquisition unit <NUM>. Subsequent processing is as described above.

According to the present embodiment, since the image and imaging timing information are exchanged via the LAN <NUM>, the processing amount on the imaging unit side can be reduced, and dimensional restrictions for vehicle installation can be decreased by weight reduction, low power consumption, and downsizing of the housing on the imaging unit side.

<FIG> is a diagram illustrating a configuration of another embodiment of the present invention. In the present embodiment, input information of an object information acquisition unit <NUM> is obtained from a sensor capable of obtaining distance information other than imaging units <NUM> and <NUM>. An example of the input information includes information obtained from the sensor such as a radar or an infrared sensor (not illustrated), and a distance of an object in a target range can be obtained. Subsequent operations are as described above.

Note that the present invention is not limited to the above-described embodiments and includes various modifications.

For example, the above embodiments have been described in detail for easy understanding of the present invention, and the present invention is not necessarily limited to one including all the described configurations. Further, a part of the configuration of a certain embodiment can be replaced with the configuration of another embodiment. Further, the configuration of another embodiment can be added to the configuration of a certain embodiment. Further, another configuration can be added to/deleted from/replaced with a part of the configurations of the embodiments.

Claim 1:
An object distance detection device comprising:
a plurality of imaging units (<NUM>, <NUM>) including a first imaging unit (<NUM>) and a second imaging unit (<NUM>);
an object information acquisition unit (<NUM>) configured to acquire object information including distance information of an object and to detect the object from an image acquired from either the first imaging unit (<NUM>) or second imaging unit (<NUM>), and to detect a distance of the object by monocular processing;
a search condition setting unit (<NUM>) configured to set a condition for searching for an image element, corresponding to a specific image element in a standard image captured by one of the plurality of imaging units (<NUM>, <NUM>), inside a reference image captured by another one of the plurality of imaging units (<NUM>, <NUM>), wherein the condition for searching includes a search range and a processing area (<NUM>, <NUM>, <NUM>, <NUM>), and
wherein the search condition setting unit (<NUM>) includes:
a processing area setting unit (<NUM>) configured to set, within the reference image, the processing area (<NUM>, <NUM>, <NUM>, <NUM>) including the object obtained by the object information acquisition unit (<NUM>), and
a search range setting unit (<NUM>) configured to set the search range of the number of pixels for the search on the basis of the distance information of the object included in the object information detected by the object information acquisition unit (<NUM>); and
a stereo distance detection unit (<NUM>) configured to perform the search on the basis of the condition for searching set by the search condition setting unit (<NUM>), and to detect a detailed distance of the object on the basis of a parallax obtained by the search.