Patent Description:
In recent years, there have been developments in technology for automatically moving a mobile object such as a vehicle. In such technology, it is important to accurately detect a line drawn on a traveling path. It is necessary to be able to detect the color of this line since the meaning of the line differs depending on the color. For example, Patent Document <NUM> detects a line of a specific color by extracting an edge of change in color in an image. Patent Document <NUM> discloses an edge image generating unit which generates an edge image, into which edge points whose variation in luminance relative to a surrounding portion is equal to or more than a predetermined value have been extracted, from an image captured by a camera. A candidate image portion extracting unit extracts a candidate image portion, which is a candidate for an image portion of a lane mark, from the edge image. A lane mark recognizing unit detects a lane mark candidate on the basis of the candidate image portion and, in the case of having detected a plurality of lane mark candidates adjacent to each other, compares the luminance of a corresponding area in the captured image for each of the lane mark candidates and recognizes the lane mark candidate having the maximum luminance value as a lane mark.

Differently colored lines are used on traveling paths, and each colored line has its meaning. Therefore, it is preferable to accurately detect a line of a specific color.

An example of the problem to be solved by the present invention is to accurately detect a line of a specific color from an image including a traveling path.

The invention described in claim <NUM> is a line detection device.

The invention described in claim <NUM> is a line detection method executed by a computer.

The invention described in claim <NUM> is a program causing a computer to execute processing of line detection.

The invention described in claim <NUM> is a storage medium storing a program executable by a computer, the program causing the computer to execute processing of line detection. Further embodiments of the claimed invention are described in the dependent claims.

The objects described above, and other objects, features and advantages are further made apparent by suitable embodiments that will be described below and the following accompanying drawings.

Embodiments of the present invention will be described below by referring to the drawings. Moreover, in all the drawings, the same constituent elements are given the same reference numerals, and descriptions thereof will not be repeated.

<FIG> is a diagram showing a functional configuration of a line detection device <NUM> according to a first embodiment. <FIG> is a diagram showing a mobile object <NUM> mounted with the line detection device <NUM>. The line detection device <NUM> detects a line drawn on a traveling path on which the mobile object <NUM> travels and includes a dividing unit <NUM>, an estimated information generating unit <NUM>, and a specifying processing unit <NUM> (first specifying processing unit). The dividing unit <NUM> generates a plurality of first divided images by dividing an image including a traveling path on which a mobile object travels (hereinafter referred to as a processed image) along a first direction including a component of the extending direction of the traveling path. The estimated information generating unit <NUM> executes processing to select first candidate pixels with respect to each of the plurality of first divided images. The first candidate pixels are estimated as a portion of a first line drawn on the traveling path. The specifying processing unit <NUM> specifies the first line included in the processed image based on locations of a plurality of first candidate pixels in the processed image. Then, a first detection unit <NUM> outputs information showing the specified first line (first line information).

In addition, the line detection device <NUM> includes a distribution information generating unit <NUM>. Therefore, the line detection device <NUM> can be regarded as to include the distribution information generating unit <NUM>, the estimated information generating unit <NUM>, and the specifying processing unit <NUM>. The distribution information generating unit <NUM> generates first distribution information which shows a distribution of pixels satisfying a criterion from the processed image. The estimated information generating unit <NUM> selects the first candidate pixels using the first distribution information. The specifying processing unit <NUM> specifies the first line included in the processed image based on the locations of the first candidate pixels in the above-mentioned image.

The present embodiment will be explained in detail below.

The mobile object <NUM> is a vehicle, for example, an automobile, a motorcycle, or the like. In this case, one example of the traveling path is a road, and the first line and a later-described second line define a traffic lane. However, the mobile object <NUM> may be an airplane. In such a case, the traveling path is a runway.

As shown in <FIG>, the mobile object <NUM> includes an image pickup device <NUM> and a control device <NUM>. The image pickup device <NUM> generates a moving image including a traveling path by capturing the front of the mobile object <NUM>. Then, a plurality of frame images constituting this moving image are outputted to the line detection device <NUM>. The line detection device <NUM> detects a first line included in the traveling path for each frame image and generates first line information showing the detected first line. The first line information includes the location of the first line and the color of the first line. This processing to generate the first line information is performed with respect to each of the plurality of frame images. However, the processing to generate the first line information may be performed with respect only to a portion of the frame images constituting the moving image.

The first line information is outputted to the control device <NUM>. The control device <NUM> controls traveling of the mobile object <NUM>. In a case where the mobile object <NUM> is an automobile, the control device <NUM> is for autonomous driving. The level of autonomous driving performed by the control device <NUM> is, for example, equal to or greater than Level <NUM> but is not limited thereto. The control device <NUM> uses the first line information when controlling traveling of the mobile object <NUM>. Meanwhile, the control device <NUM> may generate information necessary when controlling the traveling of the mobile object <NUM> instead of directly controlling traveling of the mobile object <NUM>. In this case also, the control device <NUM> generates necessary information using the first line information. Information generated here is, for example, information to notify (for example, display) the possibility of a lane change, information to notify (for example, display) the need to temporarily stop, or the like. The information is displayed on, for example, a display device (for example, a display of a car navigation device inside the vehicle) visually observable by the operator of the mobile object <NUM>.

As shown in <FIG>, the line detection device <NUM> includes the dividing unit <NUM>, the distribution information generating unit <NUM>, the estimated information generating unit <NUM>, and the specifying processing unit <NUM>. Details of processing performed by the above will be described later using flowcharts.

Meanwhile, in the example shown in <FIG>, the line detection device <NUM> is mounted in the mobile object <NUM>. However, the line detection device <NUM> may be located outside the mobile object <NUM>. In this case, the line detection device <NUM> is connected to the image pickup device <NUM> and the control device <NUM> via a wireless communication circuit. Further, the control device <NUM> may be located outside the mobile object <NUM> or may be detachably mounted on the mobile object <NUM>. For example, in a case where the line detection device <NUM> is located at the exterior (an external server or the like) of the mobile object <NUM>, the line detection device <NUM> can specify a line drawn on a traveling path on which the mobile object <NUM> traveled (for example, a division line) and also update map information using information to specify the specified line by acquiring a moving image from the image pickup device <NUM> and analyzing frame images which constitute the moving image. In this case, the map information can be easily maintained.

<FIG> is a diagram showing one example of a hardware configuration of the line detection device <NUM>. The main configuration of the line detection device <NUM> is implemented using an integrated circuit. This integrated circuit includes a bus <NUM>, a processor <NUM>, a memory <NUM>, a storage device <NUM>, an input and output interface <NUM>, and a network interface <NUM>. The bus <NUM> is a data transmission path through which the processor <NUM>, the memory <NUM>, the storage device <NUM>, the input and output interface <NUM>, and the network interface <NUM> mutually transmit and receive data. However, the method of connecting the processor <NUM> or the like to each other is not limited to a bus connection. The processor <NUM> is an arithmetic processing unit implemented using a microprocessor or the like. The memory <NUM> is implemented by using a random access memory (RAM) or the like. The storage device <NUM> is implemented by using a read only memory (ROM), a flash memory, or the like.

The input and output interface <NUM> is for connecting the line detection device <NUM> to peripheral devices. In the present diagram, the image pickup device <NUM> and the control device <NUM> are connected to the input and output interface <NUM>.

The network interface <NUM> is for connecting the line detection device <NUM> to a communication network. This communication network is, for example, a controller area network (CAN). Meanwhile, the method by which the network interface <NUM> is connected to the communication network may be a wireless connection or a wired connection.

The storage device <NUM> stores a program module for implementing each functional element of the line detection device <NUM>. The processor <NUM> implements each function of the line detection device <NUM> by reading the program module into the memory <NUM> and executing it.

Meanwhile, the hardware configuration of the above-mentioned integrated circuit is not limited to the configuration in the present diagram. For example, the program module may be stored in the memory <NUM>. In this case, the integrated circuit need not include the storage device <NUM>.

<FIG> is a flowchart showing the processing performed by the line detection device <NUM>. <FIG> are diagrams to explain the processing shown in <FIG>. First, when the image pickup device <NUM> generates frame images which constitute the moving image, the line detection device <NUM> acquires the frame image as a processed image <NUM>. Then, the line detection device <NUM> performs the processing shown in <FIG> every time the line detection device <NUM> acquires a frame image.

First, as shown in <FIG>, the dividing unit <NUM> of the line detection device <NUM> generates a plurality of divided images <NUM> (first divided images) by dividing the processed image <NUM> along a first direction (S20 of <FIG>). The number of divided images <NUM> generated from one processed image <NUM> is, for example, equal to or greater than <NUM> and equal to or less than <NUM> but is not limited thereto. In a case where the image pickup device <NUM> captures the front of the mobile object <NUM>, the y axis direction includes a component of the extending direction of the traveling path (for example, a road). Therefore, in the example shown in <FIG>, the dividing unit <NUM> generates the divided images <NUM> by dividing the processed image <NUM> along the y axis direction. In this case, the load necessary for generating processing of the divided images <NUM> becomes small.

Thereafter, the distribution information generating unit <NUM> performs processing to generate first distribution information which shows the distribution of pixels satisfying the criterion with respect to each divided image <NUM> (S40 of <FIG>). The criterion used here is, for example, having a color belonging to a predetermined range in a color space. For example, in a case where the line detection device <NUM> detects a yellow line, the above-mentioned "predetermined range" is a range which is recognized as yellow. For example, the distribution information generating unit <NUM> converts a pixel belonging to the predetermined range into <NUM> and converts other pixels into <NUM> (binarization processing). In addition, the first distribution information shows a distribution of pixels in a second direction intersecting the above-mentioned first direction. For example, as shown in <FIG>, in a case where the first direction is the y axis direction, the second direction is, for example, the x axis direction. Further, as shown in <FIG>, the first distribution information is, for example, information showing a distribution of the number of pixels satisfying the criterion in the second direction.

Thereafter, the estimated information generating unit <NUM> performs processing to select the pixels estimated as a portion of the first line drawn on the traveling path (hereinafter referred to as first pixels) using the first distribution information generated by the distribution information generating unit <NUM> (S60 of <FIG>). This processing is performed with respect to each divided image <NUM>. For example, as shown in <FIG>, in a case where the distribution information shows a distribution of the number of pixels satisfying the criterion in the second direction, the estimated information generating unit <NUM> selects a first image using at least one of a standard deviation and variance in the first distribution information. For example, when the standard deviation is equal to or less than a reference value, or when the variance is equal to or less than the reference value, the estimated information generating unit <NUM> selects all of the first candidate pixels as the first pixels. In this case, the estimated information generating unit <NUM> determines that the first line is included in the divided image <NUM>. For example, as shown in <FIG>, all of the first candidate pixels are selected as the first pixels since the standard deviation or the variance is small. On the other hand, as shown in <FIG>, in a case where the standard deviation or the variance is great, no first pixel is selected from the first candidate pixels.

However, the estimated information generating unit <NUM> may select a portion of the first candidate pixels as the first pixels. In this case, as shown in <FIG>, the estimated information generating unit <NUM> selects, for example, pixels included in a region in which the number of pixels is equal to or greater than the reference value in the first distribution information as the first pixels.

Thereafter, the specifying processing unit <NUM> estimates the location of the first line included in the processed image <NUM> using the locations of the first pixels selected by the estimated information generating unit <NUM> in the processed image <NUM> (or locations in the divided image <NUM>) (S80 of <FIG>). For example, the specifying processing unit <NUM> estimates a mean value of the locations of the first pixels in the first direction as the location of the first line in the divided image <NUM>. Then, the specifying processing unit <NUM> estimates the first line included in the processed image <NUM> by connecting the locations of the first line in a plurality of divided images <NUM> or by performing regression analysis. In addition, when estimating the location of the first line in the divided image <NUM>, the most frequent value or a median may be used instead of the mean value of the locations of the first pixels in the first direction.

Each diagram of <FIG> is one example of S80 of <FIG>, and a diagram to explain a case where the first line is estimated by the specifying processing unit <NUM> using regression analysis. In this example, a regression line is used. <FIG> is a diagram in which the estimated location of the first line is plotted. The specifying processing unit <NUM> generates a regression line in the diagram and estimates that the generated regression line shows the first line. <FIG> is an example of a case where the first line estimated in the processed image <NUM> is applied. In this case, for example, the estimated first line exists in a lower portion of the processed image <NUM> (a portion in which the plots of <FIG> exist) and extends to the upper end of the upper portion the processed image <NUM>.

Meanwhile, as shown in <FIG>, the specifying processing unit <NUM> may specify a pixel overlapped with the regression line and a pixel continued with that pixel out of the first pixels as pixels constituting the first line. In other words, the specifying processing unit <NUM> may specify a lump overlapping the regression line out of lumps of the first pixels as pixels constituting the first line in each divided image <NUM>.

Further, the distribution information generating unit <NUM> can select a pixel having a luminance satisfying a predetermined criterion and can generate distribution information of the selected pixel (hereinafter referred to as second distribution information). This criterion may be, for example, equal to or greater than a lower limit value, less than an upper limit value, or equal to or greater than the lower limit value and equal to or less than the upper limit value. In this case, when the estimated information generating unit <NUM> and the specifying processing unit <NUM> use the second distribution information instead of the first distribution information, the line detection device <NUM> can detect a white line.

As described above, according to the present embodiment, edge detection processing is not included in the processing performed by the dividing unit <NUM>, the estimated information generating unit <NUM>, and the specifying processing unit <NUM>. Therefore, the necessary calculation amount for specifying the first line is reduced. A computing device having capabilities of calculating at high speed thus is unnecessary, and as a result, manufacturing costs of the line detection device <NUM> is reduced.

The distribution information generating unit <NUM> of the line detection device <NUM> generates first distribution information. The estimated information generating unit <NUM> selects first pixels (that is, pixels estimated to constitute the first line) using this first distribution information. Therefore, the calculation amount performed by the estimated information generating unit <NUM>, that is, the calculation amount when selecting the pixels estimated to constitute the first line, is reduced.

A line detection device <NUM> according to the present embodiment is the same as the line detection device <NUM> shown in the first embodiment except the processing performed by the dividing unit <NUM>.

<FIG> is a diagram to explain the processing performed by the dividing unit <NUM> in the present embodiment. In the present embodiment, the line detection device <NUM> generates divided images <NUM> only for a portion <NUM> of the processed image <NUM> and not generate divided images <NUM> for the entire processed image <NUM>. For example, the line detection device <NUM> takes out the portion <NUM> of the processed image <NUM> and generates divided images <NUM> by dividing the portion <NUM>. In the above manner, since the region to be processed by the distribution information generating unit <NUM>, the estimated information generating unit <NUM>, and the specifying processing unit <NUM> is limited to the portion <NUM> of the processed image <NUM>, the calculation amount performed by the line detection device <NUM> is even more reduced.

Meanwhile, the location of the portion <NUM> in the processed image <NUM> is set in advance. For example, in a case where the image pickup device <NUM> is mounted in the mobile object <NUM>, the road is highly possibly captured at the lower side of the processed image <NUM>. Therefore, it is preferable to set the portion <NUM> on the lower part of the mobile object <NUM>.

<FIG> is a diagram showing a functional configuration of a line detection device <NUM> according to a third embodiment. The line detection device <NUM> according to the present embodiment has the same configuration as that of the line detection device <NUM> according to the first or the second embodiment except that the line detection device <NUM> according to the present embodiment includes a data converting unit <NUM>.

The processed image <NUM> generated by the image pickup device <NUM> is shown by an RGB color space. The data converting unit <NUM> converts the processed image <NUM> into an image shown by a color space defined by indexes of hue, lightness (luminance), and chroma, for example, an image shown by an HLS color space (converted image). Meanwhile, instead of the HLS color space, an HSV color space or a Lab color space may also be used. In addition, the dividing unit <NUM>, the distribution information generating unit <NUM>, the estimated information generating unit <NUM>, and the specifying processing unit <NUM> perform processing using the converted processed image <NUM>.

Depending on the color of the first line, it is sometimes easier for the distribution information generating unit <NUM> to process (easier to perform binarization processing of) the color space defined by the indexes of the hue, lightness (luminance), and chroma, for example, an image shown by an HLS color space, than to process the image shown by an RGB color space. In such a case, the line detection device <NUM> according to the present embodiment can detect the first line with high accuracy compared to a case where the processed image <NUM> shown by an RGB color space is directly processed. In a case where the first line is an yellow line, this tendency becomes conspicuous.

Meanwhile, as shown in <FIG>, the data converting unit <NUM> may perform the above-mentioned data conversion processing with respect to the divided image <NUM>, and not to the processed image <NUM>.

<FIG> is a diagram showing a configuration of a line detection device <NUM> according to a fourth embodiment. The line detection device <NUM> according to the present embodiment is the same as the line detection device <NUM> according to any of the first to third embodiments except that the line detection device <NUM> according to the present embodiment includes a region setting unit <NUM>. <FIG> shows a case which is the same as the third embodiment.

In a case where the specifying processing unit <NUM> detects a first line in a first frame image, the region setting unit <NUM> reduces the region to be processed by the distribution information generating unit <NUM> in a frame image processed after (hereinafter referred to as a second frame image) the first frame image (for example, the next frame image). Specifically, the region setting unit <NUM> reduces the region to be processed by the distribution information generating unit <NUM> based on the location of the first line in the second direction (for example, the location of the x axis direction in <FIG>) detected in the first frame image. For example, the region setting unit <NUM> sets the location of the first line in the second direction in the first frame image to be at the center of the region to be processed. Further, the region setting unit <NUM> narrows the width of the region to be processed. The region setting unit <NUM> performs the above-mentioned processing on, for example, the divided image <NUM>.

Each diagram of <FIG> is to schematically explain the processing performed by the region setting unit <NUM>. As shown in <FIG>, in a case where the specifying processing unit <NUM> specifies a first line L1 in the first frame image (processed image 50a), the region setting unit <NUM> acquires information showing the location of the first line L1 (for example, information showing a line acquired by regression analysis) from the specifying processing unit <NUM>. Then, the region setting unit <NUM> sets a region <NUM> to be processed by the distribution information generating unit <NUM> with respect to a second frame image (processed image 50b) as shown in <FIG>. Thereafter, in a case where the specifying processing unit <NUM> can also specify the first line L1 with respect to the second frame image (processed image 50b), the region setting unit <NUM> sets the region <NUM> in a third frame image (processed image 50c) located after the second frame image (processed image 50b) to be narrower than the region <NUM> in the second frame image (processed image 50b).

In this manner, when the first line L1 continues to be detected, the region <NUM> gradually becomes narrower. However, there is a lower limit in the size of the region <NUM> (for example, the width). That is, the region setting unit <NUM> prevents the size of the region <NUM> (for example, the width) from becoming lower than the lower limit. It is preferable that the lower limit used here is set greater than the above-mentioned width corresponding to the reference value of the standard deviation to serve as a criterion for determination of whether the divided image <NUM> includes a portion of the line or not.

Further, in a case where the first line was not specified in any frame image (processed image <NUM>) after setting the region <NUM>, the region setting unit <NUM> expands the region <NUM> or cancels the setting of the region <NUM> in the frame image thereafter (for example, the next frame image).

Meanwhile, the region setting unit <NUM> may perform the above-mentioned setting of the region <NUM> with respect to the processed image <NUM> before being processed by the dividing unit <NUM> (may be the processed image <NUM> after being converted by the data converting unit <NUM>), or may perform the setting with respect to the first distribution information generated by the distribution information generating unit <NUM>.

According to the present embodiment, the region setting unit <NUM> narrows the region <NUM> to be processed by the distribution information generating unit <NUM>. At this time, the region setting unit <NUM> sets the region <NUM> based on the location of the first line L1 in the processed frame image. Therefore, it is possible to reduce the calculation amount of the line detection device <NUM> while maintaining detection accuracy of the line L1. Meanwhile, by narrowing the region <NUM> to be processed by the distribution information generating unit <NUM>, it is also possible to inhibit the influence of noise which is unnecessary for detecting the line L1. For example, as the noise, characters drawn using yellow lines on a road which is separate from the line L1, or a display object or the like drawn using yellow lines may be mentioned as one example.

<FIG> is a diagram showing a functional configuration of a line detection device <NUM> according to a fifth embodiment. The line detection device <NUM> according to the present embodiment includes a second detection unit <NUM> other than the first detection unit <NUM>. The first detection unit <NUM> is the same as any of the above-mentioned embodiments.

The second detection unit <NUM> detects a line drawn on a traveling path using luminance of pixels constituting the processed image <NUM>. Specifically, the second detection unit <NUM> selects pixels having a luminance that satisfies the criterion from the processed image <NUM> and detects a line (second line) using the selected pixels. One example of this process is binarization.

At this time, when the criterion of luminance is not appropriately set, there is a possibility that the second detection unit <NUM> detects the first line detected by the first detection unit <NUM> together with another line. For example, in a case where the first detection unit <NUM> aims at detecting a yellow line and the second detection unit <NUM> aims at detecting a white line, the second detection unit <NUM> may detect a yellow line together with a white line. Hence, in the present embodiment, the second detection unit <NUM> sets the criterion of luminance using the luminance of the pixels constituting the first line detected by the first detection unit <NUM>.

For example, in a case where the second detection unit <NUM> aims at detecting a white line, the above-mentioned criterion of luminance is a lower limit value. That is, the second detection unit <NUM> selects pixels having a luminance which is equal to or greater than a reference value. On the contrary, the second detection unit <NUM> may select pixels having a luminance which is equal to or less than the reference value. In this case, the second detection unit <NUM> can indirectly detect the aimed line by selecting pixels constituting a region other than the aimed line. Then, the second detection unit <NUM> sets the above-mentioned lower limit value based on a value acquired by statistically processing the luminance of the pixels constituting the first line detected by the first detection unit <NUM>. Here, an example of a statistically processed value is, for example, a mean value, the most frequent value, or the like. In addition, the second detection unit <NUM> sets, for example, a value which is a sum of a statistically processed value and a constant as the lower limit value.

Further, the second detection unit <NUM> may exclude pixels constituting the first line specified by the first detection unit <NUM> from the processed image <NUM>, and may detect a white line by selecting pixels having a luminance that is higher than a reference value from the excluded image. The lower limit value in this case is, for example, a fixed value, and a value higher than the luminance value of a common road surface is used.

<FIG> is a diagram showing one example of a functional configuration of the second detection unit <NUM>. In the example shown in the diagram, the second detection unit <NUM> includes a dividing unit <NUM>, a distribution information generating unit <NUM>, an estimated information generating unit <NUM>, and a specifying processing unit <NUM> (one example of a second specifying processing unit, a third specifying processing unit, or a white line detection processing unit). The processing performed by the dividing unit <NUM> is the same as the processing performed by the dividing unit <NUM>. In addition, the processing performed by the distribution information generating unit <NUM>, the estimated information generating unit <NUM>, and the specifying processing unit <NUM> is the same as the processing performed by the distribution information generating unit <NUM>, the estimated information generating unit <NUM>, and the specifying processing unit <NUM>, respectively, except that the selection criterion of pixels is luminance. Therefore, in the present embodiment, the first detection unit <NUM> can also function as the second detection unit <NUM>. In addition, the second detection unit <NUM> may include the data converting unit <NUM> before the dividing unit <NUM>.

The distribution information generating unit <NUM> can select pixels having a luminance satisfying the predetermined criterion and generate distribution information of the selected pixels (second distribution information). The distribution information generating unit <NUM> sets the criterion used in this process, as described above, using the luminance of the pixels constituting the first line detected by the first detection unit <NUM>.

Further, the distribution information generating unit <NUM> may exclude the pixels detected by the first detection unit <NUM> from the divided image <NUM> and may generate the second distribution information using the divided image <NUM> after exclusion of the pixels.

<FIG> is a flowchart showing one example of processing performed by the line detection device <NUM> according to the present embodiment. The line detection device <NUM> performs the processing shown in the present diagram with respect to each of the plurality of frame images. First, the first detection unit <NUM> performs detection processing of the first line (line of a first color). Thereafter, the second detection unit <NUM> performs detection processing of the second line (line of a second color) after receiving luminance information of the first line from the first detection unit <NUM>. Here, the luminance of the first line is lower than the luminance of the second line.

In a case where the second detection unit <NUM> performs the processing before the first detection unit <NUM> performs the processing, that is, in a case where the second line is detected before the first line, there is a possibility that the second detection unit <NUM> detects the first line together with the second line due to a reason such that the criterion of luminance used in detecting the second line is too low. In contrast to this, in the present embodiment, the second detection unit <NUM> performs the processing after the first detection unit <NUM> performed the processing. Therefore, there is low possibility of the first line being detected together with the second line. For example, as described above, when the second detection unit <NUM> performs the processing, the second detection unit may detect only a white line by excluding the pixels constituting the first line (yellow line) detected by the first detection unit <NUM>.

Meanwhile, details of the detection processing of the first line is as described in any of the above-mentioned embodiments. In addition, details of the detection processing of the second line is as described above.

Meanwhile, there may be a case where the first detection unit <NUM> fails to detect the first line. In this case, the distribution information generating unit <NUM> of the second detection unit <NUM> performs generating processing of the second distribution information using a predetermined criterion of luminance.

Further, the second detection unit <NUM> may perform the detecting process of the second line with respect to the entire processed image <NUM>, or may perform the detecting processing of the second line with respect to only a portion of the processed image <NUM>. In the latter case, the second detection unit <NUM> may determine a region in which the detecting processing of the second line is carried out with the first line as reference. In the above manner, the amount of calculation processing performed by the second detection unit <NUM> is reduced.

<FIG> is a diagram to explain one example of a region to be processed by the second detection unit <NUM>. As shown in the present diagram, a second line L2 (for example, a white line) may be drawn in parallel to a first line L1 (for example, a yellow line) on a traveling path (for example, a road). In such a case, the second detection unit <NUM> determines a range to be processed in a direction intersecting the line L1 based on the line L1, and detects the second line L2 located in the range. For example, in the lateral direction of the processed image <NUM> (x axis direction), the second detection unit <NUM> sets a first width W1 in a positive direction from the first line L1 as the center and sets a second width W2 in the negative direction from the center. Then, the range is set as a region <NUM>. Meanwhile, the first width W1 and the second width W2 may be equal to each other or different from each other. In addition, the width of the region <NUM> may be changed along the extending direction of the line L1. For example, in a case where the width of the region included in the processed image <NUM> becomes narrower toward the upper side of the processed image <NUM>, the width of the region <NUM> may become narrower toward the upper side of the processed image <NUM>.

As described above, according to the present embodiment, it is possible to detect the first line with accuracy and it is also possible to detect the second line with accuracy.

<FIG> is a diagram showing a functional configuration of a line detection device <NUM> according to a sixth embodiment. The line detection device <NUM> according to the present embodiment has the same configuration as that of the line detection device <NUM> according to the fifth embodiment except that the line detection device <NUM> according to the present embodiment includes a determination unit <NUM>.

As described above, the first detection unit <NUM> detects a first line included in the processed image <NUM> and the second detection unit <NUM> detects a second line included in the processed image <NUM>. The processed image <NUM> is each of frame images constituting a moving image. The determination unit <NUM> calculates the detection cycle of the first line using the processing results of the first detection unit <NUM> for each of the frame images and determines whether the first line is a dotted line or not. The determination unit <NUM> performs the same processing for the second line.

A case where the determination unit <NUM> processes a first line will be described below. Every time the first detection unit <NUM> processes a frame image (processed image <NUM>), the determination unit <NUM> specifies a first line included in the frame image. Then, the determination unit <NUM> determines the detection cycle of the first line by processing the transition between the frame images in the specified results.

For example, as described above, a case where the estimated information generating unit <NUM> of the first detection unit <NUM> performs processing to determine whether a first line is included or not in the divided images <NUM> with respect to each divided image <NUM> is considered. In this case, the determination unit <NUM> calculates the number of the divided images <NUM> determined to include the first line. Then, as shown in each diagram in <FIG>, the determination unit <NUM> determines whether the first line is a dotted line or not by using the transition of the number of the divided images <NUM>. Specifically, the determination unit <NUM> determines that the first line is a dotted line when an increase or a decrease in the number of the divided images <NUM> is repeated in a fixed period. Meanwhile, the determination unit <NUM> determines that the first line is a solid line in a case where a state in which the calculated number of the divided images <NUM> is equal to or greater than a reference number (for example, equal to or greater than <NUM>% of the number of the divided images <NUM> included in one frame image) continues in frame images of a fixed number or greater. Further, in a case where the increase or decrease in the number of the divided images <NUM> is irregularly repeated, the determination unit <NUM> determines that the first line is a solid line but is partly missing (or is blurred).

Meanwhile, <FIG> shows results of plotting the above-mentioned number of divided images <NUM> for each frame image and <FIG> shows results of plotting the transition of values acquired by averaging the number of the above divided images <NUM> over continuous plural number of (for example, <NUM>) frame images. In any case, since the increase or decrease of the number of the divided images <NUM> is repeated in a fixed cycle, the increase or decrease of the number of the divided images <NUM> can be an object of processing by the determination unit <NUM>. Meanwhile, <FIG> shows that the above-mentioned mean values of the divided images <NUM> are within a fixed range.

Meanwhile, the line detection device <NUM> may acquire information to specify the speed of the mobile object <NUM> when the image pickup device <NUM> generates the processed image <NUM>. For example, the line detection device <NUM> acquires information showing the time when the processed image <NUM> was generated with respect to each processed image <NUM> and acquires information showing the speed of the mobile object <NUM> according to time. In this case, the determination unit <NUM> can determine whether the first line is a dotted line or not by further using this speed.

Specifically, the determination unit <NUM> calculates the length of the first line using the above-mentioned detection cycle of the first line and the speed of the mobile object <NUM>. For example, as shown in <FIG>, a case where the portion <NUM> is set in the processed image <NUM> is considered. The determination unit <NUM> counts the pieces of the frame images from the frame image in which the number of the divided images <NUM> determined to include the first line is equal to or greater than a reference number to the frame image in which the number of the divided images <NUM> is equal to or less than the reference number, and calculates the length of the first line by multiplying the number of pieces of the frame images by the frame rate and the speed of the mobile object <NUM>. In addition, the determination unit <NUM> counts the pieces of the frame images from the frame image in which the number of the divided images <NUM> determined to include the first line is equal to or less than a reference number to the frame image in which the number of the divided images <NUM> is equal to or greater than the reference number, and calculates the interval between first lines by multiplying the number of pieces of the frame images by the frame rate and the speed of the mobile object <NUM>. Then, the determination unit <NUM> determines that the first line is a dotted line when variation in the calculated lengths of the first lines and variation in the interval between the first lines are equal to or less than a fixed value. Meanwhile, when the variation in the calculated lengths of the first lines is equal to or greater than a fixed value, the determination unit <NUM> determines that the first line is a solid line but is partly missing (or is blurred).

In a case where the determination unit <NUM> calculates at least one of the length of the first line and the interval between the first lines, whether the first line is a dotted line or not may be determined by using a degree of match between this length and a reference length. In this case, as shown in <FIG>, the determination unit <NUM> may acquire type information to specify the type of the traveling path and may determine the above-mentioned reference length by using the type information. The type information, for example, specifies which of a general road, a toll road, and an expressway is the type of the traveling path. In a case of the expressway, the reference length is long compared to a case of the general road. Meanwhile, the above-mentioned type information may directly specify the above-mentioned roads, or may be information showing the location of the mobile object <NUM> (for example, information showing latitude and longitude such as GPS information).

Determination results by the determination unit <NUM> are outputted to the control device <NUM> of the mobile object <NUM>. The control device <NUM> controls the traveling of the mobile object <NUM> (for example, the possibility of a lane change) using information showing the determination results, that is, whether the first line (or the second line) is a solid line or a dotted line. A specific example of the determination is determined based on, for example, traffic rules.

As described above, according to the present embodiment, the determination unit <NUM> can accurately determine whether the first line and the second line are dotted lines or not. Further, the control device <NUM> of the mobile object <NUM> allows advanced autonomous driving of the mobile object <NUM>, since the control device <NUM> controls the traveling of the mobile object <NUM> using the determination results.

Claim 1:
A line detection device (<NUM>) comprising:
a first processing unit (<NUM>) that extracts a first pixel located in a range comprising a predetermined color from an image (<NUM>) to detect a line (L1) of a first color comprised in the image using a distribution of the extracted first pixel in the image, wherein the image comprises a traveling path on which a mobile object (<NUM>) travels, and
a second processing unit (<NUM>) that extracts, after execution of processing by the first processing unit, a second pixel located in a luminance range comprising a predetermined luminance from the image and specifies a line (L2) of a second color which is comprised in the image and is different from the first color using a distribution of the extracted second pixel in the image,
wherein the second processing unit excludes the pixels constituting the line of the first color detected by the first detection unit from the image and performs the processing after the exclusion.