VEHICLE EXTERNAL ENVIRONMENT RECOGNITION APPARATUS

A vehicle external environment recognition apparatus includes at least one processor, and at least one memory coupled to the at least one processor. The at least one processor is configured to operate in cooperation with at least one program stored in the at least one memory to execute processing. The processing includes generating a distance image from luminance images, specifying, by using semantic segmentation, a floating matter class in the luminance images, and invalidating parallax associated with floating pixels that are included in the distance image and belong to the floating matter class.

BACKGROUND

The disclosure relates to a vehicle external environment recognition apparatus that is to be mounted on a vehicle to identify a specific object located in a direction in which the vehicle is headed.

Known techniques, such as those disclosed in Japanese Patent No. 3349060, have been developed to help vehicles to detect vehicles ahead of them with a view to enabling each vehicle to escape severe damage from a collision with a vehicle ahead of it or with a view to enabling each vehicle to keep a safe distance from a vehicle ahead of it by follow-up control.

SUMMARY

An aspect of the disclosure provides a vehicle external environment recognition apparatus. The vehicle external environment recognition apparatus includes at least one processor and at least one memory coupled to the at least one processor. The at least one processor is configured to operate in cooperation with at least one program stored in the at least one memory to execute processing. The processing includes generating a distance image from luminance images, specifying, by using semantic segmentation, a floating matter class in the luminance images, and invalidating parallax associated with floating pixels that are included in the distance image and belong to the floating matter class.

DETAILED DESCRIPTION

A vehicle occasionally encounters a water vapor mass floating over a road in a cold climate area or a high-altitude area. In some cases, a vehicle encounters a buildup of exhaust fumes emitted through an exhaust pipe of a vehicle ahead of it when the exhaust fumes stay over a road for a while before diffusing in the atmosphere. Floating matter, such as the water vapor mass or the buildup of exhaust fumes, is recognized as a three-dimensional object by the vehicle in receipt of images in which the floating matter is seen. The vehicle can misidentify the floating matter as a specific object (e.g., a vehicle or a pedestrian) ahead of it. If the vehicle misidentifies the floating matter as a specific object, stop control or deceleration control would be performed to avoid a collision between the vehicle and the floating matter. This impairs the riding comfort of the vehicle and can cause a collision between the vehicle and a vehicle behind it.

It is desirable to provide a vehicle external environment recognition apparatus that improves the accuracy of detecting floating matter.

Vehicle External Environment Recognition System100

FIG.1is a block diagram illustrating coupling between constituent elements of a vehicle external environment recognition system100. The vehicle external environment recognition system100includes two imaging apparatuses110, a vehicle external environment recognition apparatus120, and a vehicle control apparatus130.

The imaging apparatuses110each include an imaging device, such as a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS). The imaging apparatuses110are each capable of capturing an image of the external environment ahead of a vehicle1and generating a luminance image that includes information on luminance. For example, the luminance image is a color (RGB) image or a monochrome (gray-scale) image. The two imaging apparatuses110are arranged side by side substantially in a horizontal direction with a space therebetween at the front of the vehicle1in such a manner that their respective optical axes are substantially in parallel. The imaging apparatuses110each capture an image of a three-dimensional object in a detection region ahead of the vehicle1and generate a luminance image of the object consecutively. For example, the frame duration is set to ⅟60 seconds.

The vehicle external environment recognition apparatus120recognizes the environment outside the vehicle1on the basis of images such as luminance images acquired from the imaging apparatuses110and a distance image generated from two luminance images. The environment outside the vehicle1is hereinafter simply referred to as “external environment”. The vehicle external environment recognition apparatus120then controls the speed and the steering angle of the vehicle1on the basis of the recognition of the external environment and the traveling condition of the vehicle1. The vehicle external environment recognition apparatus120will be described in detail later.

The vehicle control apparatus130is configured as an electronic control unit (ECU). The vehicle control apparatus130accepts operation inputs performed on a steering wheel132, an accelerator pedal134, and a brake pedal136by the driver and controls a steering mechanism142, a drive mechanism144, and a control mechanism146on the basis of information generated by the vehicle external environment recognition apparatus120.

Vehicle External Environment Recognition Apparatus120

FIG.2is a functional block diagram schematically illustrating the functions of the vehicle external environment recognition apparatus120. Referring toFIG.2, the vehicle external environment recognition apparatus120includes an interface (I/F)150, a data storage152, and a central control unit154.

The I/F150enables exchange of information between each imaging apparatus110and the vehicle control apparatus130. The data storage152includes, for example, random-access memory (RAM) , flash memory, and a hard disk drive (HDD) and stores various kinds of information for use in processing carried out by functional modules that will be described below.

The central control unit154is configured as a semiconductor integrated circuit including, for example, a processor, ROM, and RAM. Programs are stored in the ROM. The RAM is a work area. The central control unit154controls the I/F150and the data storage152through a system bus156. The processor included in the central control unit154in the present embodiment operates in cooperation with the programs stored in the ROM to implement functional modules, such as an image acquiring module160, a distance image generating module162, a specific object tracking module164, a first class specifying module166, a second class specifying module168, a first disparity invalidating module170, a second disparity invalidating module172, an invalidation canceling module174, a three-dimensional object specifying module176, and a specific object determination module178. In this disclosure, “disparity” may rephrase “parallax”.

The central control unit154dictated by the functional modules determines the shapes of three-dimensional objects and their relative distances in luminance images and a distance image and then identifies a certain three-dimensional object as a specific object (e.g., a vehicle ahead of the vehicle1). The central control unit154thus enables the vehicle1to escape severe damage from a collision with a vehicle ahead of it or to keeping a safe distance from a vehicle ahead of it by follow-up control.

The three-dimensional object recognized on the basis of the luminance images and the distance images is not necessarily an object of interest in terms of collision avoidance. Although floating matter, such as water vapor or exhaust fumes staying over a road, may be seen in the luminance images, the vehicle1does not need to avoid a collision with the floating matter. That is, there is no problem with the vehicle1plowing its way into the floating matter. Examples of the floating matter include: water vapor that is in the form of a gas converted from liquid state; and smoke that is a mass made of minute solid or liquid particles and is a product of the combustion of a flammable substance.

If the central control unit154misidentifies the floating matter as a specific object, automatic emergency braking (AEB) would work in such a way as to subject the vehicle1to stop control or deceleration control. This impairs the riding comfort of the vehicle1and can cause a collision between the vehicle1and a vehicle behind it. The present embodiment is to improve the accuracy of detecting floating matter with a view to controlling the vehicle1properly.

The following describes an external environment recognition procedure, which is a distinctive feature of the present embodiment. The vehicle external environment recognition procedure includes: extracting a three-dimensional object located ahead of the vehicle1; and identifying a specific object (e.g., a vehicle ahead of the vehicle1) and/or floating matter (e.g., water vapor). Workings of the functional modules constituting the central control unit154will also be described in detail in relation to the vehicle external environment recognition procedure.

Vehicle External Environment Recognition Procedure

FIG.3is a flowchart of a vehicle external environment recognition procedure. The vehicle external environment recognition apparatus120executes the vehicle external environment recognition procedure periodically (upon lapse of a predetermined interrupt period). The vehicle external environment recognition procedure starts with acquisition of luminance images by the image acquiring module160(S200). The distance image generating module162generates a distance image from two luminance images (S202). With a three-dimensional object being identified as a specific object in the previous frame, the specific object tracking module164determines the position of the three-dimensional object in the current frame (S204). The first class specifying module166specifies or identifies, by using semantic segmentation, predetermined classes in the luminance images (S206). The second class specifying module168specifies, by using semantic segmentation, a floating matter class in the luminance images (S208). The first disparity invalidating module170invalidates the disparity associated with floating pixels in the distance image (S210). The floating pixels are pixels belonging to the floating matter class. The second disparity invalidating module172invalidates the disparity associated with neighboring pixels in the distance image (S212). The neighboring pixels are pixels located within a predetermined distance from the floating pixels. The invalidation canceling module174cancels the invalidation of disparity associated with the neighboring pixels when the neighboring pixels satisfy a predetermined cancellation condition (S214). The three-dimensional object specifying module176specifies a three-dimensional object on the basis of the luminance images and the distance image in which the disparity associated with some pixels is invalidated (S216). The specific object determination module178determines whether the three-dimensional object is a specific object (S218).

The vehicle external environment recognition procedure except for processing irrelevant to the features of the present embodiment will be described in detail below.

Image Acquiring Process S200

FIGS.4and5are explanatory drawings for describing luminance images.FIG.6is an explanatory drawing for describing a distance image. The image acquiring module160acquires images captured by the imaging apparatuses110, whose optical axes do not coincide with each other. For example, the image acquiring module160acquires two luminance images180. The luminance images180acquired by the image acquiring module160include a first luminance image180ainFIG.4and a second luminance image180binFIG.5. The first luminance image180ais captured by the imaging apparatus110that is closer to the right side of the vehicle1than to the left side of the vehicle1. The second luminance image180bis captured by the imaging apparatus110that is closer to the left side of the vehicle1than to the right side of the vehicle1.

Referring toFIGS.4and5, each three-dimensional object in the first luminance image180aand the corresponding object in the second luminance image180bare not in positional agreement with each other in the horizontal direction. This is due to the positional difference between the imaging apparatuses110, which capture images at their respective positions. The term “horizontal” herein refers to the right-and-left direction of the image area of each captured image, and the term “vertical” herein refers to the up-and-down direction of the image area of each captured image.

Distance Image Generating Process S202

The distance image generating module162generates a distance image from images acquired by the image acquiring module160. Referring toFIG.6, a distance image182is generated from the first luminance image180ainFIG.4and the second luminance image180binFIG.5such that distances can be determined with regard to objects of interest.

For example, the distance image generating module162employs pattern matching to derive disparity information including disparity and image positions. The term “image positions” herein refer to the positions of blocks in an image. A block is freely selected from the blocks constituting one of the luminance images, and the distance image generating module162searches the other luminance image for a block corresponding to the freely selected block. For example, a block is freely selected from the first luminance image180a, and the distance image generating module162then searches the second luminance image180bfor a block corresponding to the freely selected block. Each block may be a 4- by 4- pixel matrix with four horizontal rows and four vertical columns. Pixels in a block from which disparity information is derived are effective pixels having the derived disparity information. Pixels in a block from which no disparity information is derived do not have disparity information.

Examples of functions that may be used in pattern matching to assess the degree of agreement between blocks include: a similarity measure based on sum of absolute difference (SAD) in which differences in luminance are taken; a similarity measure based on sum of squared intensity difference (SSD) in which the differences are squared; and a similarity measure based on normalized cross correlation (NCC) in which the degree of similarity is determined for variances obtained by subtracting a mean luminance value from the luminance of each pixel.

In this way, the distance image generating module162derives disparity on a block-by-block basis. For example, all of the blocks seen in a detection region of 600 × 200 pixels undergo the disparity derivation. Each block is not necessarily a 4- by 4- pixel matrix and may include any desired number of pixels.

Although the distance image generating module162is capable of deriving disparity on a block-by-bock basis, that is, with a detection resolution of one block, the distance image generating module162is unable to identify a three-dimensional object to which the blocks belong. Thus, the disparity information is not derived for each three-dimensional object and is derived with the detection resolution in the detection region. For example, the disparity information is derived for each block on an individual basis. For convenience, the blocks from which disparity is derived are indicated by black dots inFIG.6.

The distance image generating module162derives positional information by using stereo vision, where the disparity information derived from the distance image182on a block-by-block basis is converted to a relative distance (z) to determine the position in three dimensions. The positional information includes a horizontal distance (x), a vertical distance (y), and the relative distance (z). The stereo vision is the process of determining the relative distance between a block of interest and each imaging apparatus110. The relative distance is determined by using triangulation on the basis of the disparity associated with the block. With regard to the distance image182, the terms “disparity” and “relative distance” may both refer to the distance in the depth direction.

The vehicle external environment recognition apparatus120recognizes the external environment on the basis of the luminance images180and the distance image182acquired as above and identifies a three-dimensional object located ahead of the vehicle1as a specific object (e.g., a vehicle ahead of the vehicle1).

Specific Object Tracking Process S204

The specific object tracking module164estimates the position of a three-dimensional object identified as a specific object in the previous frame by the specific object determination module178, which will be described later. The specific object tracking module164estimates the position of the three-dimensional object in the current frame on the basis of the position of the three-dimensional object in the distance image182and the speed. In a case where a three-dimensional object whose size, shape, and color arrangement are similar to those of the specific object is in or close to the estimated position, the specific object tracking module164identifies the three-dimensional object with the specific object identified in the previous frame. This enables follow-up control, where the specific object tracking module164tracks the specific object located ahead of the vehicle1. The specific object tracking module164may carry out the processing by using various known techniques such as machine learning, which will not be further elaborated here.

First Class Specifying Process S206

The first class specifying module166specifies, by using semantic segmentation, predetermined classes in the luminance images180.

Semantic segmentation is a deep learning algorithm that associates pixels in the luminance images180with labels, categories, or classes in accordance with what each pixel signifies as an image and on the basis of the degree of certainty that the pixel is relevant to the label, category, or class concerned.

Although the first luminance image180aand/or the second luminance image180bcan be partially unclear, semantic segmentation enables the vehicle external environment recognition apparatus120to recognize a cluster of pixels belonging to a certain characteristic class (e.g., a vehicle class representing vehicles) as an integral whole. Semantic segmentation may be implemented by using various known techniques and will not be fully dealt with in the following description, which will be given while focusing on the features involved in the present embodiment.

FIGS.7and8are explanatory drawings for describing the necessity of semantic segmentation. The first class specifying module166performs classification by applying semantic segmentation to the first luminance image180ainFIG.7, and as a result, a class image190inFIG.8is generated. For example, the first class specifying module166associates each pixel in the first luminance image180awith a class selected from the group consisting of a road surface class, a lane line class, a vehicle class, a sidewalk class, a fence class, a pole class, a traffic cone class, a roadside tree class, and a sky class.

Every pixel in the first luminance image180ais analyzed by the first class specifying module166, which determines the probability of the pixel belonging to these classes. The probability of a pixel belonging to a class of interest is hereinafter referred to as the degree of certainty. The first class specifying module166associates a pixel of interest in the first luminance image180awith the class endowed with the highest degree of certainty. A road surface192a, a lane line192b, and a vehicle192care illustrated in first luminance image180ainFIG.7. A road surface class194a, a lane line class194b, and a vehicle class194care illustrated in the class image190inFIG.8. In a case where a pixel of interest in the first luminance image180ais associated with the road surface192awith the highest degree of certainty, the pixel is classified as the road surface class194a. In a case where a pixel of interest in the first luminance image180ais associated with the lane line192bwith the highest degree of certainty, the pixel is classified as the lane line class194b. In a case where a pixel of interest in first luminance image180ais associated with the vehicle192cwith the highest degree of certainty, the pixel is classified as the vehicle class194c.

Referring toFIG.7, a section denoted by184ain the first luminance image180ais unclear due to raindrops. Semantic segmentation enables the first class specifying module166to overcome such an inconvenience; that is, a section denoted by184din the class image190is properly classified as the vehicle class194c, as illustrated inFIG.8. In this way, a vehicle ahead of the vehicle1is correctly identified by the first class specifying module166.

A mean value or a median value of the degrees of certainty that are determined with regard to pixels classified as the same class may be regarded as the degree of certainty for all the pixels belonging to the class. The first class specifying module166may perform classification by using the mean value or the median value.

Second Class Specifying Process S208

The second class specifying module168specifies, by using semantic segmentation, a certain class in each luminance image180. For example, the floating matter class representing floating matter that is a specific object is specified on the basis of the degree of certainty of being the floating matter. Both the first class specifying module166and the second class specifying module168use semantic segmentation. Unlike the first class specifying module166, the second class specifying module168specifies only the floating matter class.

FIG.9is an explanatory drawing for describing the floating matter class. The second class specifying module168differentiates the floating matter class from other classes; that is, the second class specifying module168determines whether a pixel of interest belongs to the floating matter class. In the example illustrated inFIG.9, the second class specifying module168specifies a floating matter class194d, which is crosshatched inFIG.9. The regions other than the crosshatched region are regarded as a class194e, which is different from the floating matter class194d. Unlike the classification inFIG.8, the classification performed by the second class specifying module168does not deal with many classes; that is, the second class specifying module168determines only whether a pixel of interest belongs to the floating matter class194d. The second class specifying module168is thus able to specify floating matter more accurately than the first class specifying module166does.

As described above, the first class specifying module166and the second class specifying module168operate independently of each other, and floating matter is identified by using the floating matter class194dspecified by the second class specifying module168. In a case where the floating matter class194dis one of the classes to be specified by the first class specifying module166and can be specified with a satisfactory degree of accuracy, the floating matter class194dspecified by the first class specifying module166may be adopted in the present embodiment. This eliminates the processing that is otherwise carried out by the second class specifying module168such that the processing load will be lightened.

First Disparity Invalidating Process S210

The first disparity invalidating module170invalidates the disparity associated with floating pixels in the distance image. The floating pixels are pixels that are classified as the floating matter class194dby the second class specifying module168. The disparity invalidation means that a valid value indicative of disparity associated with a pixel of interest is not regarded as disparity, for the purpose of specifying objects of interest in the vehicle external environment recognition procedure. The disparity in itself is not eliminated. The reason for this is that disparity associated with floating pixels will be used by the second disparity invalidating module172, which will be described later.

The disparity associated with pixels classified as the floating matter class194dby the second class specifying module168are invalidated. This is taken to mean that no three-dimensional object is located in the position corresponding to the pixels. Thus, the central control unit154is prevented from misidentifying floating matter (e.g., water vapor or exhaust fumes) as a specific object (e.g., a vehicle ahead of the vehicle1). This eliminates the possibility that the central control unit154will perform stop control or deceleration control to avoid a collision with the floating matter misidentified as a specific object. The riding comfort of the vehicle1is maintained accordingly. This also eliminates or reduces the possibility that a vehicle behind the vehicle1will come into collision with the vehicle1.

Second Disparity Invalidating Process S212

Floating matter does not have a uniform density, and the background can be seen through part of it. Thus, there is no guarantee that the floating matter will be classified as the floating matter class194d. For example, floating matter having a low density with the background visible therethrough can be classified as the class194e, which is different from the floating matter class194d. If the floating matter is classified as the class194e, which is different from the floating matter class194d, the region classified as the class194ewould be recognized as an object of interest in terms of collision avoidance, and the vehicle1would be subjected to stop control or deceleration control. As a workaround, the vehicle external environment recognition apparatus120recognizes pixels close to the floating pixels as floating matter.

The second disparity invalidating module172invalidates the disparity associated with neighboring pixels. The neighboring pixels are pixels within a predetermined distance from the floating pixels belonging to the floating matter class194d. The disparity in itself is not eliminated. The reason for this is that disparity associated with floating pixels will be used by the invalidation canceling module174, which will be described later.

FIGS.10and11are explanatory drawings for describing the processing that is to be carried out by the second disparity invalidating module172. Referring toFIG.10, pixels with disparity in the distance image182are plotted on the zx-plane, where z and x denote the relative distance and the horizontal distance, respectively. Each pixel is indicated by a white or black dot inFIG.10.

Assume that floating matter is seen in the distance image182. In the distance image182inFIG.10, disparity is found with regard to floating pixels196, which are indicated by black dots and belong to the floating matter class194d. The floating pixels196are concentrated in a certain region on the zx plane inFIG.10.

Some of the pixels around the floating pixels196on the zx-plane are not classified as the floating matter class194d. The pixels are close to only the floating pixels196and are not close to any other specific object (e.g., a vehicle). Thus, it is highly likely that the pixels correspond to floating matter. The second disparity invalidating module172determines that the pixels within a predetermined distance from the floating pixels196are neighboring pixels198, which are highly likely to correspond to floating matter. The disparity associated with the neighboring pixels198is invalidated by the second disparity invalidating module172as in the case of the disparity associated with the floating pixels196.

For example, the second disparity invalidating module172calculates Zmin and Zmax, as illustrated inFIG.10. Zmin and Zmax denote the minimum value and the maximum value, respectively, of the relative distance (z) between the vehicle1and the floating pixels196. Pixels that are located between points corresponding to the minimum value Zmin and the maximum value Zmax and that are not the floating pixels196are recognized as the neighboring pixels198by the second disparity invalidating module172, which then invalidates the disparity associated with the neighboring pixels198.

The processing above may be altered as will be described below. The second disparity invalidating module172calculates Zmin, Zmax, Xmin, and Xmax, as illustrated inFIG.11. Zmin and Zmax denote the minimum value and the maximum value, respectively, of the relative distance (z) between the vehicle1and the floating pixels196. Xmin and Xmax denote the left end and the right end, respectively, of a line segment corresponding to the horizontal distance (x) between the vehicle1and the floating pixels196. Pixels that are located between points corresponding to the minimum value Zmin and the maximum value Zmax and between points corresponding to the left end Xmin and the right end Xmax and that are not the floating pixels196are recognized as the neighboring pixels198by the second disparity invalidating module172, which then invalidates the disparity associated with the neighboring pixels198.

In some embodiments, pixels in the region in which the floating pixels196are located are recognized as the neighboring pixels198on the basis of each pixel’s position in three dimensions, that is, the position defined by the horizontal distance (x), the vertical distance (y), and the relative distance (z). In a case where the Euclidean distance between the pixel of interest and the floating pixels196(√(horizontal distance2+ vertical distance2+ relative distance2)) falls within a predetermined range, the pixel is recognized as the neighboring pixel198by the second disparity invalidating module172, which then invalidates the disparity associated with the pixel. Alternatively, pixels within a predetermined distance (e.g., 2 m) from the center of the region in which the floating pixels196are located may be recognized as the neighboring pixels198. Still alternatively, pixels within a predetermined distance (e.g., 2 m) from the mean value of all the floating pixels196, that is, from the center of gravity of the floating pixels196may be recognized as the neighboring pixels198.

Pixels regarded as floating matter include not only the floating pixels196belonging to the floating matter class194dbut also the neighboring pixels198that are close to the floating pixels196. This enables invalidation of disparity associated with the neighboring pixels198that ought to be recognized as floating matter. Thus, the vehicle external environment recognition apparatus120is prevented from misidentifying the floating matter (e.g., water vapor or exhaust fumes) as a specific object (e.g., a vehicle ahead of the vehicle1) .

For convenience, an example has been described in which the second disparity invalidating module172starts carrying out the processing after the completion of the processing carried out by the first disparity invalidating module170. In actuality, the first disparity invalidating module170extracts pixels in the detection region on a one-by-one basis. In a case where a pixel of interest belongs to the floating matter class194d, the first disparity invalidating module170recognizes the pixel as the floating pixel196and then invalidates the disparity associated with the pixel. The second disparity invalidating module172extracts, on a one-by-one basis, pixels within a predetermined neighboring range where the floating pixels196having undergone disparity invalidation. In a case where a pixel of interest is located within a predetermined distance from the floating pixels196and is not the floating pixel196, the second disparity invalidating module172recognizes the pixel as the neighboring pixel198and invalidates the disparity associated with the pixel. That is, the first disparity invalidating module170and the second disparity invalidating module172operate concurrently in such a manner that the second disparity invalidating module172carries out the processing in accordance with the result of processing carried out by the first disparity invalidating module170. The neighboring range is defined by a predetermined number of horizontal pixels and a predetermined number of vertical pixels in the luminance image180, where the center of the neighboring range corresponds to the floating pixels196having undergone disparity invalidation.

Simply invalidating the disparity associated with the pixels that are close to the floating pixels196can result in the following consequences.

The second disparity invalidating module172can possibly misidentify, as floating matter, a three-dimensional object that is close to the floating pixels196and is not the floating matter. In a case where another vehicle is located ahead of floating matter, the second disparity invalidating module172can possibly recognize, as floating matter, pixels corresponding to the vehicle.

In such a case, the second disparity invalidating module172invalidates disparity associated with pixels that ought to be recognized as a specific object. The vehicle external environment recognition apparatus120, which is supposed to enable the vehicle1to escape severe damage from a collision with the vehicle ahead of it or to keep a safe distance from a vehicle ahead of it by follow-up control, is not able to carry out such processing when encountering a specific object. As a workaround, the invalidation canceling module174strictly determines whether the neighboring pixels198correspond to any other specific object.

The invalidation canceling module174determines whether the neighboring pixels198having undergone the disparity invalidation performed by the second disparity invalidating module172each satisfy a predetermined cancellation condition. The invalidation canceling module174then cancels the invalidation of disparity associated with the neighboring pixels198that satisfy the predetermined condition. That is, the disparity associated with the pixels concerned are validated.

In a case where pixels recognized as the neighboring pixels198can possibly correspond to a specific object (i.e., a vehicle ahead of the vehicle1), the disparity associated with the neighboring pixels198are not invalidated. If the vehicle1misidentifies a vehicle ahead of it as floating matter, the vehicle1would come into collision with the vehicle ahead of it. Avoiding such situations deserves higher priority than preventing the vehicle1from misidentifying floating matter as a vehicle ahead of the vehicle1and from being erroneously subjected to stop control or deceleration control.

FIGS.12and13are explanatory drawings for describing the operation of the invalidation canceling module174. The second class specifying module168specifies the floating matter class194d, which is crosshatched inFIG.12. The first disparity invalidating module170specifies the floating pixels196belonging to the floating matter class194d. The second disparity invalidating module172specifies the neighboring pixels198within a predetermined distance from the floating pixels196. The neighboring pixels198are pixels close to the floating pixels196and are not the floating pixels196. The neighboring pixels198can possibly be pixels that are classified as the vehicle class194cby the first class specifying module166.

In a case where the pixels recognized as the neighboring pixels198by the second disparity invalidating module172are pixels classified as the vehicle class194cby the first class specifying module166, the invalidation canceling module174cancels the invalidation of disparity associated with the neighboring pixels198concerned.FIG.12illustrates an example in which the neighboring pixels198overlap the vehicle class194c. In this case, the invalidation canceling module174cancels the invalidation of disparity associated with the neighboring pixels198concerned.

As mentioned above, the specific object tracking module164estimates the position of a three-dimensional object identified as a vehicle in the previous frame. Referring toFIG.13, the specific object tracking module164estimates the position of the three-dimensional object in the current frame on the basis of the position of three-dimensional object in the distance image182and the speed. Once the specific object tracking module164estimates the position of a vehicle ahead of the vehicle1, the specific object tracking module164regards the vehicle as a region200, which has a rectangular shape defined by vertical lines passing through horizontal ends of the vehicle and horizontal lines passing through vertical ends of the vehicle. This enables the vehicle external environment recognition apparatus120to properly determine the size of the vehicle ahead of the vehicle1. Dealing with such a simple shape lightens the processing load.

Once the vehicle ahead of the vehicle1is regarded as the region200by the specific object tracking module164, the invalidation canceling module174cancels the invalidation of disparity associated with the pixels that are located in the region200and recognized as the neighboring pixels198by the second disparity invalidating module172.FIG.13illustrates an example in which the neighboring pixels198overlap the region200. With the vehicle ahead of the vehicle1being regarded as the region200, the invalidation canceling module174cancels the invalidation of disparity associated with the neighboring pixels198.

The invalidation canceling module174cancels the invalidation of disparity associated with the neighboring pixels198when (i) the neighboring pixels198are classified as the vehicle class194cby using semantic segmentation; and/or (ii) the neighboring pixels198correspond to a three-dimensional object in the current frame, with the three-dimensional object being identified as a vehicle in the previous frame.

The vehicle external environment recognition apparatus120is thus prevented from misidentifying a specific object (e.g., a vehicle ahead of the vehicle1) as floating matter and enables the vehicle1to escape severe damage from a collision with the specific object.

The three-dimensional object specifying module176specifies a road surface ahead of the vehicle1by using the luminance images180and the distance image182with the floating pixels196and the neighboring pixels198having undergone the disparity invalidation. The three-dimensional object specifying module176then specifies a three-dimensional object located vertically above the specified road surface. For example, the three-dimensional object specifying module176considers that blocks at a predetermined distance (e.g., 0.3 m) or more from the road surface in the vertical direction are likely to correspond to a three-dimensional object that juts out the road surface in the vertical direction. From among the blocks that are likely to correspond to a three-dimensional object located vertically above the road surface, blocks located at the same relative distance from the vehicle1are selected and combined into a group by the three-dimensional object specifying module176, which then specifies the group as a three-dimensional object.

Specific Object Determination Process S218

The specific object determination module178determines whether the three-dimensional object specified by the three-dimensional object specifying module176is a predetermined specific object (e.g., a vehicle ahead of the vehicle1, a pedestrian, or a building). The determination may be made by using various known techniques, which will not be further elaborated here.

The vehicle external environment recognition apparatus120and the vehicle external environment recognition procedure, which have been described above, improve the accuracy of detecting floating matter and prevent the vehicle1from misidentifying the floating matter as a specific object and from being erroneously subjected to stop control or deceleration control. The neighboring pixels198that are close to the floating pixels196are recognized as floating matter by the second disparity invalidating module172. This yields a further improvement in the accuracy of detecting floating matter. If there is a possibility that the neighboring pixels198do not correspond to floating matter, the invalidation canceling module174do not recognize the neighboring pixels198as floating matter. This enables the vehicle1to escape severe damage from a collision with a specific object.

This approach involves programs for causing a computer to operate as the vehicle external environment recognition apparatus120and a storage medium in which the programs are stored. Examples of the storage medium include computer-readable media, such as flexible discs, magneto-optical discs, read-only memory (ROM), compact discs (CDs), digital versatile discs (DVDs), and Blu-ray Discs (BDs). The programs herein refer to data processing means written in any desired language or described by using any desired method.

A preferred embodiment of the disclosure has been described with reference to the accompanying drawings. Needless to say, the disclosure is not limited to the embodiment. It is obvious that variations and modifications can be made by those skilled in the art without departing from the scope hereinafter claimed, and the variations and modifications also fall within the technical scope of the disclosure.

For example, an embodiment has been described in which the invalidation canceling module174cancels the invalidation of disparity associated with the neighboring pixels198when (i) the neighboring pixels198are classified as the vehicle class194cby using semantic segmentation; and/or (ii) the neighboring pixels198correspond to a three-dimensional object in the current frame, with the three-dimensional object being identified as a vehicle in the previous frame. In some embodiments, the invalidation canceling module174cancels the invalidation of disparity when the neighboring pixels198are recognized as an edge. This condition may be imposed in addition to or in place of the conditions (i) and (ii) mentioned above. The invalidation canceling module174determines the size of the edge by applying a filter in one direction (e.g., the horizontal direction or the vertical direction) or applying filters in both directions (e.g., the horizontal direction and the vertical direction) to the neighboring pixels198in the luminance images180. In a case where the size of the edge is greater than or equal to a threshold value, the invalidation canceling module174recognizes it as an edge of a specific object (e.g., a vehicle). The invalidation canceling module174cancels the invalidation of disparity associated with the neighboring pixels198that are recognized as an edge of the specific object.

Each filter may be an array with three rows and three columns. For example, the array {{-1, 0, 1}, {-2, 0, 2}, {-1, 0, 1)}} is used as a filter Dx in the horizontal direction, and the array {{-1, -2, -1}, {0, 0, 0}, {1, 2, 1}}} is used as a filter Dy in the vertical direction. A predetermined block with three rows and three columns is herein denoted by L. The size of an edge in the horizontal direction is determined by multiplying an edge of the block L by the filter Dx in the horizontal direction (Dx × L), and the size of an edge in the vertical direction is determined by multiplying the edge of the block L by the filter Dy in the vertical direction (Dy × L). In a case where the root-mean-square value (√(Lx2+ Ly2) ) that is the square root of the sum of squares of Lx and Ly is greater than or equal to a threshold value, the invalidation canceling module174cancels the invalidation of disparity associated with pixels included in the block L. Lx denotes the value obtained by multiplying the block L by the filter Dx in the horizontal direction, and Ly denotes the value obtained by multiplying the block L by the filter Dy in the vertical direction.

Detection of an edge in the horizontal direction and/or an edge in the vertical direction implies that the block L is less likely to correspond to floating matter, in which case the use of disparity associated with the block L is allowed. The edge detection may be performed by using various known techniques, which will not be further elaborated here.

The degree of certainty in prediction based on semantic segmentation may also be taken into consideration for cancellation of the invalidation of disparity. For example, it is determined whether the degree of certainty that the neighboring pixels198correspond to floating matter is less than a threshold value. The degree of certainty in prediction based on semantic segmentation is included in data that is input to softmax or sigmoid during arithmetic computations performed by the second class specifying module168or may be included in intermediate data obtained after softmax or sigmoid operations. For example, the invalidation canceling module174determines whether the degree of certainty about the neighboring pixels198in the luminance images180is less than or equal to a threshold value. The invalidation canceling module174cancels the invalidation of disparity associated with the neighboring pixels198for which it is determined that the degree of certainty is less than or equal to the threshold value.

When the degree of certainty that pixels of interest belong to the floating matter class194dis low, the pixels are less likely to correspond to floating matter, in which case the use of disparity associated with the pixels is allowed.

It is not required that the steps included in the vehicle external environment recognition procedure disclosed herein be performed chronologically in the order illustrated in the flowchart. The method may include parallel processing or subroutine processing.

The central control module154illustrated inFIG.2can be implemented by circuitry including at least one semiconductor integrated circuit such as at least one processor (e.g., a central processing unit (CPU)), at least one application specific integrated circuit (ASIC), and/or at least one field programmable gate array (FPGA). At least one processor can be configured, by reading instructions from at least one machine readable tangible medium, to perform all or a part of functions of the central control module154including the image acquiring module160, the distance image generating module162, the specific object tracking module164, the first class specifying module166, the second class specifying module168, the first disparity invalidating module170, the second disparity invalidating module172, the invalidation canceling module174, the three-dimensional object specifying module176, and the specific object determination module178. Such a medium may take many forms, including, but not limited to, any type of magnetic medium such as a hard disk, any type of optical medium such as a CD and a DVD, any type of semiconductor memory (i.e., semiconductor circuit) such as a volatile memory and a non-volatile memory. The volatile memory may include a DRAM and a SRAM, and the non-volatile memory may include a ROM and a NVRAM. The ASIC is an integrated circuit (IC) customized to perform, and the FPGA is an integrated circuit designed to be configured after manufacturing in order to perform, all or a part of the functions of the modules illustrated inFIG.2.