Patent Description:
Stereoscopic vision system use two, or more, cameras to capture pairs of images of the same scene. The stereoscopic images are processed by the vision system to extract <NUM> dimensional (3D) information, which may be used to determine information such as object size, distance etc. Stereoscopic vision systems typically detect corresponding features, such as a corner of an object, within each of the stereoscopic images and determine a disparity between the feature location in the two images. Based on the disparity information, 3D information can be extracted.

While the 3D information from stereoscopic vision systems may be useful, the feature extraction and feature matching between images required by the stereoscopic vision systems is computationally expensive and as such the use of such systems may be limited in applications.

The relevant state of the art is represented by <CIT> and <CIT>.

Accordingly the present teaching provides a method as detailed in claim <NUM>. Also provided is a system according to claim <NUM> and a non-transitory computer readable memory according to claim <NUM>.

Advantageous features are in the dependent claims.

A stereoscopic vision system rapidly discriminates between locations that are on the ground-plane and locations that are not on the ground-plane. The stereoscopic vision system uses an expected disparity for on-ground-plane locations to quickly generate a ground-plane mapping of the locations in the image that are on the ground-plane or off the ground-plane. The discrimination between on ground-plane locations and off ground-plane locations is based on a comparison between corresponding locations in the stereoscopic images. The corresponding locations are determined using the expected disparity. The stereoscopic vision system can discriminate between on and off ground-plane locations using simple comparisons which are computationally inexpensive and so can be performed rapidly.

The ground plane discrimination may be used in various applications, including for example, in autonomous, or semi-autonomous, vehicles. The stereoscopic vision system may be used to process stereoscopic images captured from forward-facing cameras in order to determine portions of the images that are on the ground plane, and as such may be considered as safe to drive on, subject to other requirements such as lane detection, pedestrian detection etc., and portions of the image that are not on the ground plane and as such may be regions of interest for further investigation. Since the stereoscopic vision system can rapidly discriminate the ground plane, the resulting ground-plane mapping can be used by other autonomous vehicle processes, such as route planning, obstacle avoidance, etc. Further, since the discrimination is computationally inexpensive, it allows compute resources to be dedicated to other processes such as traffic signal detection, road sign detection, as well as for redundancy.

The stereoscopic vision system and ground-plane discrimination will be described with reference to <FIG>, which depict schematically the concepts of the discrimination process. It will be appreciated that the depictions in <FIG> are simplified to highlight the concepts of the ground-plane discrimination. While the examples of <FIG> are simplified, it will be apparent that the same technique can be applied to real-world stereoscopic images.

<FIG> depicts stereoscopic images capturing a road. The stereoscopic pair <NUM> comprise a left image 102a captured by a first camera and a right image 102b captured by a second camera. The first and second cameras are assumed to be arranged on a vehicle and capturing a forward-facing view. In order to simplify the comparison between the stereoscopic pair <NUM>, it is desirable that the first camera and second camera have the same, or at least similar, configuration parameters, such as resolution, lens, focal length, etc. as well as being mounted parallel to each other, with the two cameras separated from each other by an appropriate amount, such as for example between <NUM> and <NUM> inches.

As depicted in <FIG> the left image 102a captures a view of a road 104a disappearing at the horizon <NUM> and the right image 102b captures a different view of the road 104b disappearing at the horizon, which is a result of the displacement between the mounting locations of the two cameras. As depicted in the left image 102a, a number of points along the road 108a, 110a, 112a, 114a are located respective distances A<NUM>-A<NUM> from a left edge of the image. Each of the points along the road 108a, 110a, 112a, 114a are spaced apart from each other vertically in the image, and may be spaced apart from each other horizontally, as depicted in image 102a, although the points along the road 108a, 110a, 112a, 114a need not be displaced horizontally from each other. Two points 108a and 108a' are depicted at the same vertical distance from the bottom of the image. As can be seen in the right image 102b, corresponding points 108b, 108b', 110b, 112b, 114b are located a different distance from the edge of the right image 102b than the left image 102a. The points are spaced the distance of the corresponding points in the left image, namely A<NUM>-A<NUM>, plus another amount, referred to as the disparity shift, B<NUM>-B<NUM> that is based on the vertical location of the point in the image. As can be seen, the points 108b, 110b, 112b, 114b which are all vertically offset from each other, are shifted from their corresponding points in the left image by different disparity shifts. Points 108b and 108b', which have the same vertical location in the image are shifted from their corresponding points in the left image by the same disparity shift, namely B<NUM>. From <FIG>, it can be seen that if a point in an image is on the ground, the corresponding point in the stereoscopic pair will be at a location equal to the first point's location plus a disparity shift that is determined from the point's vertical location in the image. The expected disparity shift <NUM> is depicted for the different vertical locations within the image.

<FIG> depicts stereoscopic images capturing a road and road sign. The stereoscopic pair <NUM> depicts the same road as shown in <FIG>; however, there is an additional vertical road sign, such as a speed limit sign. Turning to the left image 202a, it can be seen that two points on the road sign 204a, namely a point 206a where the sign meets the ground and a second point 208a where the sign is located above the ground are located respective distances C<NUM>, C<NUM> from the left side of the image 202a. In the right image 202b, the corresponding points 206b, 208b of the road sign 204b are located a distance equal to the corresponding points' 206a, 208b location, namely C<NUM>, C<NUM>, plus a disparity shift B<NUM>. In contrast to the road depicted in <FIG> in which points in the image vertically offset from each other are displaced from the corresponding points by different disparity shifts, the two corresponding points 206b, 208b of the road sign 204b, although vertically offset from each other in the image 202b, are displaced from the points 206a, 208a by the same disparity shift of B<NUM>. If the point 208a of the road sign 204a and the corresponding point 208b of the road sign 204b were on the ground in the real world, it would be expected that the points would be displaced from each other by a disparity shift of B<NUM>, as can be seen from the expected disparity shifts <NUM>. As can be seen, the expected disparity shifts <NUM> shrink towards the horizon. At and above the horizon, the expected disparity is <NUM>.

From <FIG> and <FIG> it can be seen that points in image that are on the ground in the real world will have a particular disparity shift between corresponding points that is dependent upon the vertical location of the points in the images. In contrast points in the image of objects that are off the ground in the real world will have the same vertical disparity for all of the points. It is noted that points of objects that are not perfectly vertical in the real-world may not have the same disparity shift between corresponding points, however the disparity will differ from the expected on-ground-plane disparity. The expected disparity shifts between corresponding points that are on the ground-plane can be used to quickly identify portions of the image that are on the ground plane or not on the ground plane.

<FIG> depicts schematically rows in the stereoscopic images. In discriminating between on-ground-plane and off-ground-plane locations within the stereoscopic images, the pair of images may be processed in rows with corresponding points between the images in the row having the same expected ground-plane disparity shift. <FIG> depicts the left and right images 202a, 202b segmented into a number of rows 304a, 306a, 308a, 310a, 312a, 314a, 304b, 306b, 308b, 310b, 312b, 314b. The segmentation of the images depicted in <FIG> is provided for explanatory purposes and need not be explicitly done when actually discriminating the ground plane in stereoscopic images. Further, the rows are depicted as being relatively large in the vertical dimension of the images for clarity of the figures. In practice, the height of the rows may be for example between <NUM> - <NUM> pixels.

<FIG> depicts schematically the processing of stereoscopic image rows. As depicted in <FIG> a number of sample locations L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM> in each of the rows 304a, 306a, 308a, 310a are extracted from the left image. The corresponding sample locations R<NUM>, R<NUM>, R<NUM>, R<NUM>, R<NUM>, R<NUM>, R<NUM>, R<NUM>, R<NUM>, R<NUM> in the rows 304b, 306b, 308b, 310b of the right image are determined by shifting the respective locations of the points L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM> by the expected disparity for the respective rows. As depicted for sample point L<NUM>in <FIG>, the points L<NUM>, L<NUM>, L<NUM>, in the bottom row 304a of the left image are shifted by the disparity shift amount B<NUM> associated with the row to the sample points R<NUM>, R<NUM>, R<NUM>. Similarly, sample points L<NUM>, L<NUM> in row 306a are shifted by the disparity shift amount B<NUM> of the row 306a to the sample points R<NUM>, R<NUM>. Similarly, points in rows 308a, 310a are shifted by respective disparity shift amounts B3, B4.

Although <FIG> depicts samples being extracted at multiple feature locations simultaneously, it is noted that these points are only shown for illustrative purposes. In actual processing, the sample for a row may be slid across the image without having to determine any features within the images.

Each of the sample points L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM> and corresponding sample points R<NUM>, R<NUM>, R<NUM>, R<NUM>, R<NUM>, R<NUM>, R<NUM>, R<NUM>, R<NUM>, R<NUM> are based on the pixel values in the sample area. The sample may be determined in various ways, including as an average of the pixel values, a weighted average, etc..

<FIG> depicts schematically discrimination of ground plane locations in the stereoscopic images. Each of the sample points L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM>, L<NUM> and corresponding sample points R<NUM>, R<NUM>, R<NUM>, R<NUM>, R<NUM>, R<NUM>, R<NUM>, R<NUM>, R<NUM>, R<NUM> from <FIG> are reproduced in <FIG>. As described above, if the sample points in the images correspond to objects on the ground plane in the real world, the sample points should be basically the same and as such, if a sample at a location X in the left image matches a sample at X plus the row's disparity shift in the right image, it can be assumed that the image locations correspond to an object on the ground in the real-world. It will be appreciated that while corresponding samples may be basically the same, they will not be identical since the images are captured from different locations. Accordingly, the match may be determined using a threshold value, that is it is determined if the left sample matches the right sample within the threshold amount. As depicted, the difference between two corresponding samples may be used as an indication of whether or not the sample location is on or off ground. As depicted in <FIG>, all of the differences of the samples are indicative of the location being on the ground-plane except for difference Diff<NUM> between sample L33 and corresponding sample R33 which is depicted as being off ground. The marked samples, provided by the differences between samples Diff<NUM>, Diff<NUM>, Diff13, Diff21, Diff22, Diff31, Diff32, Diff33, Diff41, Diff42, may be used in constructing a mapping of on and off ground plane locations.

<FIG> depicts schematically on and off ground plane locations within a stereoscopic images. The stereoscopic mapping <NUM> is depicted as a pair of mask images 602a, 602b with the locations determined to be on the ground plane set to white depicted by white portions 604a, 604b and the locations determined to be off the ground plane set to black depicted by white 606a, 606b. The white locations are considered as being on the ground, while the black locations are considered as being off the ground. The stereoscopic mapping <NUM> does not provide information about the off ground locations, other than that they are off the ground. However, the stereoscopic mapping can be used to further process the locations of possible off-ground objects in order to possibly identify and classify possible objects as well as information such as distance or range to the object. The mask images 602a, 602b may be used as a region of interest mask with the off-ground locations providing regions of interest for further processing. For example, the portions of the stereoscopic images corresponding to the black portions 606a, 606b in the masks may be extracted and processed further to identify possible objects in the region of interest. In, the road example depicted in <FIG>, this may include for example identify the object as a road sign, determining the type of road sign, namely that it is a speed limit sign and the associated information, namely that the speed limit is <NUM>/h. The object detection may be based on the particular application, which may dictate the expected scene, as well as the location within the image. For example, in an autonomous vehicle application, a right side of the image would be expected to have road sign information, a bottom portion of the image would be expected to have the road, and a top portion may include traffic lights or overhead signs. Depending upon the location of the regions of interest, different object detection/recognition algorithms that are tailored to the expected scenes may be applied.

<FIG> depicts schematically components of a stereoscopic vision system. The stereoscopic vision system <NUM> may be provided in a vehicle in order to provide road information to an autonomous vehicle control application that controls operation of the vehicle. The basis of the stereoscopic vision system comprises a processor <NUM> capable of executing instructions. An input/output (I/O) interface <NUM> may be used to connect to a pair of cameras, or image capture devices, 706a, 706b. The cameras 706a, 706b may be mounted in a known configuration relative to each other such that the cameras are displaced horizontally, or approximately horizontally from each other. Preferably, the characteristics of the cameras are the same and the cameras are genlocked to each other in order to capture images of a moving scene at the same time as each other. The vision system further comprises a non-transitory memory <NUM> that stores instructions which when executed cause the stereoscopic vision system implement the rapid ground-plane discrimination functionality <NUM> described above and set forth in further detail below. The vision system <NUM> may further comprise non-volatile (NV) storage <NUM> for the storage of data and instructions.

<FIG> depicts a method of discriminating a ground plane in stereoscopic images. The method <NUM> may be implemented by the vision system <NUM>. The method <NUM> receives stereoscopic images (<NUM>). The stereoscopic images comprise a first image and a second image that may be stored in the memory while processing. A first sample is obtained from a location in the first image (<NUM>). A disparity shift for the location is determined (<NUM>) and used to obtain a second sample from the second image (<NUM>). The disparity shift may be pre-calculated based on the camera's position. The disparity shift associated with different row heights within the images will depend upon where the cameras are pointing relative to the horizon. The disparity shift for each vertical location within the image can be determined taking into account the tilt of the vehicle or camera system. The determined disparity shift is used to obtain a second sample from a location corresponding to the first sample, but shifted by the determined disparity amount. The first and second samples may be obtained from the images in various ways. For example, the sample may be obtained by sampling values of one or more channel values of the images. The obtained sample may be for example an average of a 1x3 patch in the image. Alternatively, the obtained sample may be obtained as a running weighted average. For example, the samples may be obtained according to a weighted average: Avgweight = (<NUM> * Avgweight + Y)/<NUM>, where Y is the value of the image at the sampling location. Other techniques for sampling the location are possible. The obtained samples are compared to each other to determine if the match (<NUM>). The comparison may be made my determining if the difference between the obtained samples is larger than some threshold value, which may be for example <NUM>% of the possible range of values. If the samples match (yes at <NUM>), the locations in the stereoscopic images may be marked as on ground plane locations (<NUM>) and if the samples don't match (no at <NUM>) the locations may be marked as being off the ground plane (<NUM>). Although depicted as being marked as either on or off the ground plane, it may be possible to use additional classifications based on the comparison of the samples. For example, samples that match within <NUM>% may be marked as on the ground plane, samples that are a mismatch of greater than <NUM>% may be marked as off the ground plane and samples that match within <NUM>-<NUM>% may be marked as possibly on the ground plane. Once the location is marked, it is determined if processing should continue (<NUM>) and if it should (yes at <NUM>), the sample location is shifted (<NUM>). The location of the sample may be shifted horizontally, for example from left to right or right to left. Additionally the sample location may be shifted vertically, for example if a row of the images has been processed, the next row of the image may be processed. Additionally, rows in the image may be skipped, for example each <NUM> rows may be skipped in processing unless off ground plane locations are found, and then the skipped rows may be processed to locate the boundaries of the off ground locations. If there is no more processing (no at <NUM>) the method is done (<NUM>) and the marked locations can be used in other processing.

While the above method <NUM> has described the process for the ground plane discrimination, it will be appreciated that the method can be implemented in different ways in the vision system. One possible implementation is depicted in the following pseudo code listing. While more rows in image
While more samples in row:
extract left sample at X
extract right sample at X + row disparity
if left sample matches right sample within threshold:
mark sample location as on ground-plane
else
mark sample location as off ground-plane
shift X to next sample location
Done (while more samples in current row)
determine next row
reset X sample location
Done (while more rows).

Other modifications to the process are possible. For example, rather than obtaining a first sample, determining an expected disparity shift, and then obtaining a second sample as separate individual steps, the steps may be performed together could be combined according to: <MAT>.

The ground plane discrimination described above may be performed rapidly due to the simplicity of comparing one location in an image to another location in a second image. Further, the method may be massively parallelized since each row in the image is independent of other rows and as such the rows in an image may be computed in parallel. Further, by comparing portions of rows linearly, for example from left to right, the image portions for comparison may be easily pre-fetched from memory.

Various performed using an Intel i5, <NUM> processor and images of 1280x800 in the YUV422 space, in order to detect object locations. The rapid ground plane discrimination algorithm was able to process a frame and detect the location of an object in approximately <NUM> msec, without the use of a GPU. Similar results for the same images obtained using the OpenCV blob detection algorithm took approximately <NUM> msec.

<FIG> depicts a method of autonomous vehicle control using ground plane discrimination. In autonomous vehicle control, it is generally desirable to ensure that the immediate driving path is clear from obstacles, which will be objects that are not on the ground. Once the driving path is cleared, other processing may be performed such as object detection and classification. As depicted in process <NUM>, a possible vehicle path may be determined (<NUM>). The possible vehicle path may be determined using steering input information as well as path or route planning information. The possible driving path may be used to perform ground-plane discrimination of the possible vehicle path (<NUM>). The ground plane discrimination described above may be applied to the portions of the stereoscopic images that are within the possible vehicle path. The on ground plane locations may be passed to vehicle path determination functionality (<NUM>) that may use the information to plan the driving path for the vehicle. Off ground plane locations may be passed to object detection/classification functionality (<NUM>) that may attempt to determine what is off the ground plane, such as another vehicle, a pedestrian, etc. The results of the object detection/classification may also be passed to the vehicle path determination (<NUM>). It will be appreciated that in an autonomous vehicle, multiple redundant systems may be performing the same or similar determinations with the described vision system providing one set of data that can be combined with other sets. For example, the vision system information may be combined with LIDAR information as well as RADAR information. Once the vehicle path is determined, the information can be passed to vehicle control functionality (<NUM>). Once the driving path is cleared, additional portions of the images may be processed using the ground plane discrimination. For example, ground plane discrimination may be performed on possible traffic light areas (<NUM>), typically in a top portion of the images above the horizon where the expected disparity shift is <NUM>, and the off ground plane locations may be passed to traffic light detection functionality (<NUM>), and the results passed to the vehicle control functionality (<NUM>). The ground plane discrimination may also be applied to road sign locations (<NUM>), typically a right or left side of the image and the off ground plane locations passed to road sign detection functionality (<NUM>) that can detect road signs such as speed limits and pass the information to the vehicle control (<NUM>). The different locations in an image may obtain the samples from different channels. For example, in a YUV image, object detection may use only the Y channel, while traffic light detection may use a difference between the U and V channels.

Claim 1:
A method (<NUM>) of discriminating ground plane locations in stereoscopic images, the method implemented by a processor of a stereoscopic vision system, the method comprising:
receiving (<NUM>) stereoscopic images of a scene comprising a first image (102a) captured from a first camera (706a) and a second image (102b) captured from a second camera (706b), the first camera and the second camera arranged to capture the stereoscopic images, wherein the first image and second image capture different views of the scene that are shifted by a disparity shift as a result of a displacement between mounting locations of the first and second cameras;
obtaining (<NUM>) pixel values for a first sample point (L11-L42) from a first location in the first image;
determining (<NUM>), based on a vertical location of the first sample point (L11-L42) within the first image (102a), a shift amount corresponding to a value at the vertical location of the first sample point (L11-L42) of the disparity shift (B1-B4) for points in the first and second images that capture features of the scene that are on a ground plane;
obtaining (<NUM>) pixel values for a second corresponding sample point (R11-R42) from a second location in the second image (102b), wherein the second location in the second image (102b) is the first location shifted by the shift amount;
comparing (<NUM>) the pixel values for the first sample point (L11-L42) to the pixel values for the second corresponding sample point (R11-R42) ,
wherein the pixel values for the first sample point (L11-L42) in the first image (102a) and the second corresponding sample point (R11-R42) in the second image (102b) are determined as a weighted average of pixel values in a vicinity of the first sample point (L11-L42) and the second corresponding sample point (R11-R42) ; and
in response to comparing the pixel values for the first and second sample points:
marking (<NUM>) the first and second locations in the first and second images as corresponding to locations of the scene that are on the ground plane when the pixel values for the first and second sample points are the same within a similarity threshold; or
marking (<NUM>) the first and second locations in the first and second images as corresponding to locations of the scene that are off the ground plane when the pixel values for the first and second sample points are not the same within the similarity threshold.