Patent Description:
In recent years, a camera device has been mounted on a vehicle and has become widespread as an in-vehicle camera device. With the widespread use of in-vehicle camera devices, there is an increasing demand for various recognition functions for safe driving and automatic driving. Among them, since the stereo camera device measures the visual information by images and the distance information to an object at the same time, it is possible to grasp various objects around the car (a person, a car, a three-dimensional object, a road surface, a road marker, a sign marker, etc.) in detail, so that it is said that the stereo camera device contributes to improving safety during driving assistance.

The demand for the marker recognition function in stereo camera devices is increasing in Japan and overseas, and improvement in recognition accuracy is required.

As one of the factors that reduce the accuracy of stereo camera devices, there is a problem that a random pattern with many edges, such as tree branches, is mistakenly detected as a circle, and is erroneously recognized as a marker in identification.

In the related art, various technologies and devices related to an in-vehicle camera device mounted on a vehicle and recognizing the situation in front of the vehicle have been proposed. For example, with regard to improving the performance of marker recognition, PTL <NUM> discloses a technique focusing on improving the accuracy of recognition.

The marker recognition technique described in PTL <NUM> executes the center estimation process, the radius estimation process, and the identification process of the imaged target circle in order. Then, in the center estimation process, voting is performed for a predetermined length in the normal direction from the edge, and the pixel in which a certain number of votes are collected is set as the estimation center.

However, in the technique described in PTL <NUM>, there is a possibility that as one of the factors of accuracy reduction, a random pattern with many edges, such as tree branches, is mistakenly detected as a circle, and erroneously recognized as a marker in identification.

In other words when the target is a circle, votes for a predetermined length in the normal direction from the edge on the circumference are collected in the center, and as in a large number of tree branches, the number of votes may exceed a certain number even in a pattern with dense edges. In such a case, there is a possibility that an erroneous determination is made in the radius estimation process and the identification process described above, and a large number of tree branches are erroneously recognized as a marker.

An object of the present invention has been made in view of the above points, and is to provide a marker recognition method for a camera device and a marker recognition device capable of improving the marker recognition accuracy by excluding the random pattern such as tree branches, and suppressing incorrect determination of the random pattern as a marker in the circle detection process.

In order to achieve the above object, the present invention is configured as follows.

In a marker identification method for a stereo device, the method includes estimating, from an image captured by the stereo camera device, a circle and a center of the circle, detecting, based on a change in shading of the image, or an intensity image, or the image and the intensity image, a number of edges within a predetermined width in a radial direction of the circle from the center and a radius which is a distance from the center to each of the edges to create a histogram including the number of edges and the radius, selecting, as a radius candidate from the histogram, a radius at which the number of edges is equal to or greater than a predetermined number of edges threshold value, calculating a gradient of the histogram, from the selected radius candidate and a value of the histogram in a vicinity of the selected radius candidate, and performing an identification process on a histogram in which the calculated gradient is equal to or greater than a predetermined gradient threshold value to identify a marker.

In addition, a marker identification device includes an image input unit that receives an image captured by a stereo camera device, an image processing unit that corrects the image received by the image input unit, and an arithmetic processing unit that determines whether the image corrected by the image processing unit is a marker to identify the determined marker, wherein the arithmetic processing unit estimates, from the corrected image, a circle and a center of the circle, detects, based on a change in shading of the image, or an intensity image, or the angle image and the intensity image, a number of edges within a predetermined width in a radial direction of the circle from the center and a radius which is a distance from the center to each of the edges to create a histogram including the number of edges and the radius, wherein the arithmetic processing unit selects a radius at which the number of edges is equal to or greater than a predetermined number of edges threshold value as a radius candidate from the histogram, and calculates a gradient of the histogram from the selected radius candidate and a value of the histogram in a vicinity of the selected radius candidate, and performs an identification process on a histogram in which the calculated gradient is equal to or greater than a predetermined gradient threshold value to identify the marker.

According to the present invention, it is possible to provide a marker recognition method for a camera device and a marker recognition device capable of improving the marker recognition accuracy by excluding a random pattern such as tree branches and suppressing incorrect determination of the random pattern as a marker.

<FIG> is a block diagram showing the overall configuration of an image processing device in an in-vehicle camera system according to an embodiment of the present invention.

In <FIG>, an in-vehicle stereo camera device <NUM> according to an embodiment is mounted on a vehicle and recognizes the environment outside the vehicle based on the image information of the region to be imaged in front of the vehicle.

The in-vehicle stereo camera device <NUM> recognizes, for example, a white line on a road, a pedestrian, a vehicle, another three-dimensional object, a signal, a marker, a lighting lamp, and the like, and adjusts the brakes, the steering, etc. of the vehicle (own vehicle) equipped with the in-vehicle stereo camera device <NUM>.

The in-vehicle stereo camera device <NUM> includes a left camera <NUM> and a right camera <NUM>, which are two cameras disposed on the left and right to acquire image information, and an image input interface <NUM> for controlling the imaging operation of the left camera <NUM> and the right camera <NUM>, and taking in a captured image. The data of the image captured through this image input interface <NUM> is transmitted through a bus <NUM>, and is processed by an image processing unit <NUM> and an arithmetic processing unit <NUM>, and the result during the processing, the image data as the final result, and the like are stored in a storage unit <NUM>.

The image processing unit <NUM> compares a first image obtained from the imaging element of the left camera <NUM> with a second image obtained from the imaging element of the right camera <NUM> to perform an image correction process such as a correction of device-specific deviations caused by the imaging element and a noise interpolation on each image, and stores the result in the storage unit <NUM>. Further, the image processing unit <NUM> calculates the portion corresponding to each other between the first and second images to calculate parallax information, and stores the result in the storage unit <NUM> as in the above.

The arithmetic processing unit <NUM> recognizes various objects necessary for perceiving the environment around the own vehicle by using the image and the parallax information (distance information for each point on the image) stored in the storage unit <NUM>. The various objects include a person, a car, another obstacle, a signal, a marker, a car tail lamp and head lights, and the like. Part of these recognition results and intermediate calculation results is stored in the storage unit <NUM> in the same manner as described above. After performing various objects recognition on the captured image, the arithmetic processing unit <NUM> calculates the control of the vehicle using these recognition results.

The vehicle control policy obtained as a result of the calculation, and part of the object recognition result are transmitted to an in-vehicle network CAN <NUM> through a CAN interface <NUM>, and this brakes the vehicle.

Also, with respect to these operations, a control processing unit <NUM> monitors whether each processing unit is operating abnormally or whether an error has occurred during data transfer, and has a mechanism to prevent abnormal operation.

The image processing unit <NUM> is connected to the control processing unit <NUM>, the storage unit <NUM>, the arithmetic processing unit <NUM>, the image input interface <NUM> between the imaging elements of the left camera <NUM> and the right camera <NUM>, and an input/output unit <NUM> with an external in-vehicle network <NUM> via an internal bus <NUM>.

The image processing unit <NUM>, the storage unit <NUM>, the arithmetic processing unit <NUM>, the output unit <NUM>, and the control processing unit <NUM> are configured by a single computer unit or a plurality of computer units.

The storage unit <NUM> is configured by a memory or the like that stores, for example, image information obtained by the image processing unit <NUM> and image information created by the arithmetic processing unit <NUM> as a result of scanning.

The input/output unit <NUM> with the external in-vehicle network outputs the information output from the in-vehicle stereo camera device <NUM> to the control system of the own vehicle via the in-vehicle network CAN <NUM>.

<FIG> is a diagram showing a processing flow of a stereo camera <NUM> according to an embodiment of the present invention.

First, an image is captured by the left and right cameras <NUM> and <NUM> in the in-vehicle stereo camera device <NUM>, an image process (step S205) such as correction for absorbing the peculiarity of each imaging element is performed on each of the image data <NUM> and <NUM> captured by the left and right cameras <NUM> and <NUM>, respectively.

The processing result of image processing step S205 is stored in the storage unit <NUM> by image buffer step S206. The image buffer is provided in the storage unit <NUM> shown in <FIG>. Further, the image processing unit <NUM> collates the images with each other by using the two corrected images (corrected images from the cameras <NUM> and <NUM>), and as a result, the parallax information of the images obtained by the left and right cameras <NUM> and <NUM> is obtained (parallax processing step S207).

Due to the parallax of the left and right images it became clear where a certain point of interest on the target object corresponds to on the images of the left and right cameras <NUM> and <NUM>, and the distance to the object will be obtained by the principle of triangulation. This is done in parallax processing step S207. Image processing step S205, image buffer step S206, and parallax processing step S207 are performed by the image processing unit <NUM> shown in <FIG>, and the finally obtained image and the parallax information is stored in the storage unit <NUM>.

Further, various objects recognition process (step S209) is performed using the above-mentioned stored image and parallax information. Examples of the object to be recognized include a person, a car, another three-dimensional object, a marker, a signal, a tail lamp, and the like, and at the time of recognition, recognition dictionary use step S210 (stored in the storage unit <NUM>) is used as necessary. The recognition dictionary is stored in the storage unit <NUM>.

Furthermore, considering the result of object recognition and the condition of the own vehicle (speed, steering angle, etc.), in vehicle control processing step S211, a policy to, for example, warn the occupant, perform braking such as braking of the own vehicle and adjustment of the steering angle, or control avoidance of the object thereby is determined, and output the result through CAN interface <NUM> (CAN IF step S212).

The various objects recognition process <NUM>, recognition dictionary use step S210, and vehicle control processing step S211 are performed by the arithmetic processing unit <NUM> shown in <FIG>, and the output to the in-vehicle network CAN <NUM> is performed through the CAN interface <NUM>. Each of these processes and means is configured by, for example, a single computer unit or a plurality of computer units, and is configured so that data can be exchanged with each other.

<FIG> is a diagram showing a timing chart of the process in the stereo camera device.

In the timing chart of <FIG>, the processing flows of two systems are shown as a processing flow <NUM> and a processing flow <NUM>.

The processing flow <NUM> indicates the processing timing in the image processing unit <NUM> shown in <FIG>, and the processing flow <NUM> indicates the processing timing in the arithmetic processing unit <NUM> shown in <FIG>.

In <FIG>, first, right image input step S303 is performed in the processing flow <NUM>. This corresponds to the process in which the right camera <NUM> captures an image in <FIG>, and after that, through the image processing <NUM>, the right image is stored in image buffer step S206.

Next, left image input step S304 is performed. This corresponds to the process in which the left camera <NUM> captures an image in <FIG>, and through image processing S205, the left image is stored in image buffer step S206.

Next, parallax processing step S207 is performed. This corresponds to the process in which in <FIG>, the two images from the left and right cameras <NUM> and <NUM> are read out in image buffer processing step S206, parallax is calculated by implement collation between both images, and the parallax information obtained by the calculation is stored in the storage unit <NUM>. At this point, the image and the parallax information are stored in the storage unit <NUM>. In the processing flow <NUM>, the input processing to image buffer step S206 is performed following parallax processing step S207.

Processing information <NUM> until parallax processing step S207 of the processing flow <NUM> is transferred to the processing flow <NUM>, and the various objects recognition <NUM> is performed. Various objects recognition step S209 includes marker recognition step S209a, which will be described later.

The recognition process is performed in the various objects recognition step S209, vehicle control process S211 is performed, and the result is output to the in-vehicle network CAN <NUM>.

<FIG>, <FIG> are schematic explanatory views of a marker recognition function for performing marker recognition on an image stored in the image buffer <NUM>, and <FIG> shows an overall schematic flow of the marker recognition process according to marker recognition step S209a.

Marker recognition step S209a shown in <FIG> includes circle detection processing step S406 and identification processing step S407.

<FIG> is a diagram showing part of the image stored in image buffer step S206 through image processing step S205. <FIG> is a diagram showing the processing result of image processing the image shown in <FIG> for use in marker recognition step S209a, and is an image having brightness at a portion having a large change in shading in the vicinity in the image.

Further, <FIG> is a diagram showing a histogram obtained based on the change in shading of the image shown in <FIG>.

In marker recognition step S209a, first, circle detection processing step S406 (shown in <FIG>) in which a circle and the center of the circle are estimated from the image (image captured by the cameras <NUM> and <NUM>) shown in <FIG> is performed. In circle detection processing step S406, the center of the circle is obtained by the "center calculation". Circle detection processing step S406 is divided into a center estimation (step S408) and a radius estimation (step S409). The details of the radius estimation step S409 will be described later with reference to <FIG>.

First, in step S408, the center is estimated. A line segment <NUM> extending in the normal direction is drawn from each edge of the image shown in <FIG>, and it is estimated that a point where a certain number or more of intersections of the plurality of line segments <NUM> overlap each other is the center. In <FIG>, <NUM> is estimated to be the center. Next, in step S409, the radius of a circle <NUM> (first edge) is estimated based on the histogram shown in <FIG>. The identification process (identification (recognition) of the content of the marker) of the marker detected in identification processing step S407 is performed, and vehicle control processing step S211 is executed.

In the histogram of <FIG>, the horizontal axis represents the radius from the center <NUM> shown in <FIG>, and the vertical axis represents the number of edges within a predetermined width (within the width in the radial direction). The method of obtaining the histogram in <FIG> will be described.

Assume a circle whose radius gradually increases from the center <NUM> in <FIG>. When this circle and an edge (for example, the first edge <NUM> or the second edge <NUM>) overlap, the number of the overlapping edges is counted and is represented by the vertical axis in <FIG>.

When there is a circle like the first edge <NUM>, the number of edges <NUM> in the histogram shown in <FIG> has increased, and when there is no circle like <NUM> shown in <FIG>, the number of edges <NUM> in the histogram in <FIG> is reduced.

A radius at which the number of edges exceeds a predetermined number of edges threshold value <NUM> in the histogram shown in <FIG> is selected as a radius candidate.

The count of the number of edges having a predetermined radius or below from the center <NUM> may be omitted.

The calculation omission region may be set considering that the marker has a diameter of generally <NUM> or more and the process cannot be performed in identification processing step S407 when a marker at a very distant location is detected. By setting the calculation omission region, the calculation load of the marker recognition unit 209a can be reduced. That is, for a circle having a radius smaller than a predetermined width, it is possible to have a configuration in which the marker is not recognized based on the change in shading.

Due to improvement in the processing capacity of the marker recognition unit 209a, the calculation omission region may be reduced in order to recognize a marker very far away to improve safety.

<FIG> is a diagram showing a more detailed processing flow of the present invention, and shows the details of the radius estimation step S409 of the flow shown in <FIG>. The circle detection process (step S406) and the identification process (step S407) shown in <FIG> are executed by the arithmetic processing unit <NUM>.

In <FIG>, the random pattern separation is performed in radius estimation step S409. First, in histogram creation step S500, the histogram shown in <FIG> is created (creation of a histogram including the number of edges and the radii, each of which is the distance from the center of the circle to the edges).

Next, from the histogram created in step S500, a radius candidate is obtained by the predetermined number of edges threshold value <NUM>. The gradient of the histogram is calculated by histogram gradient calculation step S502 from the obtained radius candidate and the value of the histogram in the vicinity thereof. In random pattern separation step S503, the random pattern is separated by comparing the gradient obtained in histogram gradient calculation step S502 with the predetermined gradient threshold value.

This is based on the principle that a histogram indicating the marker and a histogram indicating a random pattern such as a plurality of tree branches have the different gradients of the histogram, so that the marker and the random pattern can be separated by classifying the gradient of the histogram according to the threshold value.

Then, in step S407, the identification process (marker content identification (recognition)) of the detected marker is performed.

<FIG>, <FIG> are explanatory diagrams in which the marker pattern and the random pattern are separated by the gradient of the histogram.

<FIG> is a diagram showing a histogram created from a captured image of a marker <NUM>. In <FIG>, the radius candidate estimated by the marker <NUM> is represented by <NUM>. <FIG> is a diagram showing the vicinity of the radius candidate <NUM> in the histogram shown in <FIG>.

A gradient <NUM> of the radius candidate <NUM> is calculated in histogram gradient calculation step S502. There are slopes on both sides of the apex of radius candidate <NUM> in the calculation of the gradient. The gradients of the slopes on both sides are calculated, and the larger gradient of them is regarded as the gradient <NUM> of the radius candidate <NUM>.

When the gradient <NUM> shown in <FIG> is greater than or equal to the gradient threshold value, the histogram is identified as a histogram indicating the marker. Since the example shown in <FIG> shows a histogram indicating the marker, the gradient <NUM> is greater than or equal to the gradient threshold value, and the histogram is identified as a histogram indicating the marker (step S503).

<FIG> is a diagram showing a histogram created from part <NUM> of a tree determined to be a circle in circle detection processing step S406 even though it is not a marker. In <FIG>, the radius candidate estimated by the tree branch <NUM> is represented by <NUM>. <FIG> is a diagram showing the vicinity of the radius candidate <NUM> in the histogram shown in <FIG>.

A gradient <NUM> of the radius candidate <NUM> is calculated in histogram gradient calculation step S502.

Since the example shown in <FIG> shows a histogram indicating tree branches, the gradient <NUM> is below the gradient threshold value, so that the example is identified as a random pattern, and is separated from the marker (step S503).

The gradient threshold value by which the marker and a random pattern such as tree branches are distinguished can be set at any value depending on the performance of the camera or the like. As an example, the gradient threshold value can be set to <NUM>/<NUM> = <NUM>.

The marker <NUM> and the random pattern <NUM> such as tree branches can be separated by comparing the gradient of the histogram with the predetermined gradient threshold value.

As mentioned above, according to an embodiment of the present invention, the circle detection process is performed on the image captured by the cameras (<NUM>, <NUM>), the histogram is created, a radius when the number of edges within the predetermined width exceeds the predetermined edge threshold value is regarded as a radius candidate, the gradient of the histogram is calculated from the radius candidate and the value of the histogram in the vicinity thereof, and a histogram whose calculated gradient is equal to or greater than the threshold gradient is identified as a marker.

Therefore, in the circle detection process, it is possible to provide a marker recognition method for a camera device and a marker recognition device capable of improving the marker recognition accuracy by excluding the random pattern such as tree branches, and suppressing incorrect determination of the random pattern as a marker.

In the circle detection process, radius estimation processing step S409 is performed by estimating the center of the circle portion (circle). The configuration may be such that a histogram is created from the radius and the number of edges on the basis of the change in shading of the angle image, the intensity image, or both of them within a predetermined width based on the position away from the estimated center by the radius, the gradient of the histogram is calculated, and the identification process is performed on the histogram whose calculated gradient is equal to or greater than the predetermined gradient threshold value to recognize the marker.

In addition, the configuration may be such that a radius at which the number of edges exceeds the predetermined number of edges threshold value in the histogram is regarded as a radius candidate, the gradient of the histogram is calculated from the radius candidate and the value of the histogram in the vicinity thereof to compare the calculated gradient with the gradient threshold value, and the random pattern is separated.

In the above-described embodiment, a histogram is created from the radius and the number of edges based on the change in shading of both the angle image and the intensity image.

Claim 1:
A computer-implemented marker identification method for a stereo camera device (<NUM>), the method comprising:
estimating, from an image captured by the stereo camera device (<NUM>), a circle and a center of the circle;
detecting, based on a change in shading of the image, or an intensity image, or the image and the intensity image, a number of edges within a predetermined width in a radial direction of the circle from the center and a radius which is a distance from the center to each of the edges to create a histogram including the number of edges and the radius;
selecting, as a radius candidate from the histogram, a radius at which the number of edges is equal to or greater than a predetermined number of edges threshold value,
calculating, a gradient of the histogram, from the selected radius candidate and a value of the histogram in a vicinity of the selected radius candidate, and
performing an identification process on a histogram in which the calculated gradient is equal to or greater than a predetermined gradient threshold value to identify a marker.