System and method for detecting occluded objects based on image processing

The present invention is related to systems and methods for detecting an occluded object based on the shadow of the occluded object. In some examples, a vehicle of the present invention can capture one or more images while operating in an autonomous driving mode, and detecting shadow items within the captured image. In response to detecting a shadow item moving towards the direction of vehicle travel, the vehicle can reduce its speed to avoid a collision, should an occluded object enter the road. The shadow can be detected using image segmentation or a classifier trained using convolutional neural networks or another suitable algorithm, for example.

FIELD OF THE DISCLOSURE

This relates to an autonomous vehicle and, more particularly, to a system and method of an autonomous vehicle for detecting an occluded object based on the shadow of the occluded object.

BACKGROUND OF THE DISCLOSURE

Autonomous vehicles, including vehicles operating in a fully autonomous mode, a partially autonomous mode, or a driver assistance mode, can detect objects entering the vehicle's path of travel to avoid a collision. In some examples, however, a pedestrian, animal, or other object can suddenly enter the road, giving the vehicle little time to react. For example, the object can enter the road from behind a parked vehicle or other large object that conceals the object from one or more sensors (e.g., camera(s), radar, LiDAR, range sensors, ultrasonic sensors) of the autonomous vehicle. In these situations, the vehicle may have little time to reduce its speed or come to a complete stop to avoid a collision. It is an object of the present invention to use shadow images to assist with object avoidance during autonomous vehicular navigation.

SUMMARY OF THE DISCLOSURE

This relates to a system and method of an autonomous vehicle for detecting an occluded object based on the shadow of the occluded object. In some examples, the vehicle can operate in a shadow detection mode in based on the vehicle's location. In one embodiment, based on map or location data, the vehicle can determine it is currently in a pedestrian-heavy zone (e.g., parking lot, city, neighborhood, or school zone) and accordingly enter a shadow-detection mode of driving. While driving in the shadow detection mode, the vehicle can capture one or more images (e.g., still images or videos) with a camera, and identify one or more shadows of occluded objects moving towards the vehicle's direction of travel. The shadows can be detected using image segmentation and/or using a classifier trained using convolutional neural networks or a similar algorithm. In response to detecting a shadow moving towards the vehicle's path of travel, the vehicle can reduce its speed to allow more time to react, should an object enter the road, for example. In some examples, the shadow of the occluded object can be detected even when the occluded object itself may not be detected by the sensors (e.g., camera(s), LiDAR, radar, ultrasonic sensors, range sensors).

DETAILED DESCRIPTION

In the following description of examples, references are made to the accompanying drawings that form a part hereof, and in which it is shown by way of illustration specific examples that can be practiced. It is to be understood that other examples can be used and structural changes can be made without departing from the scope of the disclosed examples. Further, in the context of this disclosure, “autonomous driving” (or the like) can refer to autonomous driving, partially autonomous driving, and/or driver assistance systems.

FIG. 1illustrates an exemplary system block diagram of vehicle control system100according to examples of the disclosure. Vehicle control system100can perform any of the methods described below with reference toFIGS. 2-3. System100can be incorporated into a vehicle, such as a consumer automobile. Vehicle control system100can include one or more cameras106capable of capturing image data (e.g., video data) of the vehicle's surroundings, as will be described with reference toFIGS. 2-3. Vehicle control system100can also include one or more other sensors107(e.g., radar, ultrasonic, LIDAR, other range sensors, etc.) capable of detecting various characteristics of the vehicle's surroundings, and a location system, such as a Global Positioning System (GPS) receiver108, capable of determining the location of the vehicle. It should be noted that other types of location system can also be used, including cellar, WiFi, or other types of wireless-based location systems. Vehicle control system100includes an on-board computer110that is operatively coupled to the cameras106, sensors107and GPS receiver108, and that is capable of receiving the image data from the cameras and/or outputs from the sensors107and the GPS receiver108. The on-board computer110can also be capable of receiving map information105(e.g., via a wireless and/or interne connection at the vehicle). It is understood by ones of ordinary skill in the art that map data can be matched to location data in map-matching functions. In some examples, the vehicle can select an operation mode based on its location (e.g., a parking lot mode, an urban driving mode, a highway mode, or another location-based operation mode). In accordance with an embodiment of the present invention, in response to determining the vehicle is location is in a pedestrian heavy-zone where pedestrians, pets, or other objects may approach the vehicle's path of travel, the vehicle can enter a shadow detection mode as described below. Examples of pedestrian-heavy zones can include parking lots, school zones, neighborhoods, and cities. In accordance with one embodiment of the invention, the on-board computer110can be capable of operating in a fully or partially autonomous driving mode using camera(s)106and GPS receiver108, as described in this disclosure. In some examples, the on-board computer110includes storage112, memory116, and a processor114. Processor114can perform any of the methods described with reference toFIGS. 2-3. Additionally, storage112and/or memory116can store data and instructions for performing any of the methods described with reference toFIGS. 2-3. Storage112and/or memory116can be any non-transitory computer readable storage medium, such as a solid-state drive or a hard disk drive, among other possibilities. The vehicle control system100can also include a controller120capable of controlling one or more aspects of vehicle operation, such as controlling motion of the vehicle in a fully or partially autonomous driving mode.

In some examples, the vehicle control system100can be operatively coupled to (e.g., via controller120) one or more actuator systems130in the vehicle and one or more indicator systems140in the vehicle. The one or more actuator systems130can include, but are not limited to, a motor131or engine132, battery system133, transmission gearing134, suspension setup135, brakes136, steering system137and door system138. The vehicle control system100can control, via controller120, one or more of these actuator systems130during vehicle operation; for example, to open or close one or more of the doors of the vehicle using the door actuator system138, to control the vehicle during fully or partially autonomous driving operations using the motor131or engine132, battery system133, transmission gearing134, suspension setup135, brakes136and/or steering system137, etc. The one or more indicator systems140can include, but are not limited to, one or more speakers141in the vehicle (e.g., as part of an entertainment system in the vehicle), one or more lights142in the vehicle, one or more displays143in the vehicle (e.g., as part of a control or entertainment system in the vehicle) and one or more tactile actuators144in the vehicle (e.g., as part of a steering wheel or seat in the vehicle). The vehicle control system100can control, via controller120, one or more of these indicator systems140to provide indications to a driver of the vehicle of one or more aspects of the fully or partially autonomous driving mode, such as an indication that an occluded object has been detected based on detection of the occluded object's shadow.

FIG. 2Aillustrates an exemplary vehicle202detecting an occluded object206based on the shadow208of the occluded object according to examples of the disclosure. As an example, a vehicle202can be driving in a fully or partially autonomous mode including a shadow detection mode in a parking lot200or other pedestrian-heavy zone (e.g., a school zone, a neighborhood, a city). Parking lot200can include a plurality of parked vehicles204or other stationary objects that can block an occluded object206from vehicle202, for example. The occluded object can be a pedestrian or animal moving from a position between parked cars204towards the direction of vehicle202travel. In some examples, the occluded object206can be blocked from a camera (e.g., camera(s)106) or another sensor (e.g., sensor(s)107) of vehicle202. For example, the other sensors can include radar, LiDAR, or a range sensor and the occluded object206can essentially be shielded from these sensors by the parked cars204. Likewise, the occluded object206may not be visible in one or more images captured by the vehicle camera because it can be blocked by one or more parked cars204. Because the occluded object204itself may not be detectable by vehicle202, a dangerous situation can arise if the occluded object204moves into the path of vehicle travel with little time for vehicle202to react by slowing down or coming to a stop, which can cause a collision.

In some examples, although occluded object206may not be detectable by vehicle202, a shadow208of the occluded object can be visible to the vehicle's camera. Based on detecting the shadow208in one or more captured images and detecting (e.g., using onboard computer110) that the shadow is moving towards the direction of travel of the vehicle202, the vehicle can reduce its speed or stop to allow extra time to react should occluded object206enter the vehicle's intended path of travel, for example. In some examples, detecting movement of shadow208can cause the vehicle202to reduce its speed or stop when the occluded object206is moving towards the vehicle. However, if the shadow208is not moving, which can be indicative of a stationary object such as a fire hydrant or parked motorcycle, or is moving away from the direction of vehicle202travel, the vehicle may continue to drive without reducing its speed or stopping. In some examples, while operating in the shadow detection mode, the vehicle202can employ other techniques to detect an occluded object. For example, one or more cameras of the vehicle can capture an image of the occluded object206through a window of a parked car204or a radar can detect the occluded object if the radar waves bounce beneath the parked cars204. Other additional techniques of detecting occluded object206in conjunction with the shadow-detection mode are possible and multiple techniques can be used at once to increase the changes of detecting occluded object206.

FIG. 2Billustrates an exemplary method250of detecting an occluded object during a fully or partially autonomous driving mode of a vehicle according to examples of the disclosure. While driving in a fully or partially autonomous driving mode, the vehicle (e.g., vehicle202) can determine its location using GPS108and/or map information105, for example (step252of method250). In accordance with a determination that the vehicle is in a location where pedestrians or other hazards may enter the vehicle's direction of travel, the vehicle can enter a shadow-detection mode of driving and method250can proceed. For example, the vehicle can enter the shadow-detection mode and method250can proceed when the vehicle is in a “pedestrian-heavy zone” such as a parking lot, in a city, in a neighborhood, or in a school zone. While the vehicle is driving, a camera of the vehicle can capture one or more images of the vehicle's surroundings (step254of method250). One or more shadows on the ground in the one or more captured images can be detected (e.g., by onboard computer110) (step256of method250). Exemplary details for detecting one or more shadows will be described with reference toFIGS. 3A-3C. The vehicle can further determine if the detected shadows are moving towards the direction of vehicle travel (step258of method250). If the detected shadow is not moving towards the direction of vehicle travel (e.g., the shadow is stationary or moving away from the direction of vehicle travel), method250can start over at step252, for example. If the detected shadow is moving towards the direction of vehicle travel, the vehicle can reduce its speed or stop (step260of method250).

FIG. 3Aillustrates an exemplary image300captured by a camera of vehicle202including a shadow208of an occluded object206according to examples of the disclosure. Image300can be captured by one or more cameras (e.g., camera(s)106) of vehicle202and can further include parked cars204, shadows310of the parked cars, and a horizon312. The shadow208of the occluded object206can be identified using image segmentation, as described in further detail with reference toFIG. 3B, and/or using a learning method, as described in further detail with reference toFIG. 3C.

FIG. 3Billustrates an exemplary method350of identifying a shadow208of occluded object206using image segmentation according to examples of the disclosure. Method350can be performed during a shadow detection mode of the vehicle202in accordance with a determination that the vehicle202is in a pedestrian-heavy zone, for example. In some examples, vehicle202can identify one or more pixels of image300that capture the ground (step352of method350). For example, the vehicle202can identify pixels of image300that correspond to objects not on the ground, such as parked cars204and pixels above the horizon312(e.g., corresponding to the sky, buildings, or traffic lights). In some examples, vehicle202can detect parked cars204and any other objects using one or more of ultrasonic sensors, radar sensors, LiDAR sensors, and/or range sensors. The detected objects can be associated with one or more pixels of captured image300, for example. In some examples, the vehicle202can estimate the position of the horizon312based on a calibration procedure or a different horizon detection algorithm. Accordingly, by process of elimination, the ground pixels can be identified.

The vehicle202can further segment the ground pixels into regions based on brightness (step354of method350). For example, pixels proximate to one another having a darkness that is within a threshold difference of one another can form a segment. Variations in darkness in the image can be caused by discolorations of the ground, writing or lane markings on the ground, and/or shadows (e.g., shadow208or shadows310).

In some examples, the vehicle202can identify a difference in darkness (black level and/or contrast) of each region compared to the surrounding regions (step356of method350). For example, the shadow208of the occluded object206can have a first darkness and one or more regions surrounding it can have, on average, a second darkness, less than the first darkness by at least a threshold difference. The vehicle202can identify one or more “dark” regions surrounded by “light” regions as possibly corresponding to shadows.

Next, the vehicle202can determine whether the dark regions are moving (step358of method350). Detecting which dark regions are moving can eliminate dark regions corresponding to shadows of stationary objects (e.g., shadows310of parked cars204) and dark regions not corresponding to shadows (e.g., a puddle or another dark spot on the ground). In some examples, determining whether the dark regions are moving can be limited to detecting which dark regions are moving towards the path of vehicle202travel.

Optionally, vehicle202can compare the shape of the dark moving regions to one or more expected shadow shapes (step360of method350). In some examples, step360can include one or more steps of method370described below with reference toFIG. 3C. Vehicle202can store (e.g., within onboard computer110) one or more reference images corresponding to various shadows of people, pets, and other moving objects in a variety of lighting conditions to use for the comparison, for example.

In some examples, vehicle202can identify, using method350, one or more shadows208of occluded objects206that are moving towards the direction of vehicle travel (step362of method350). In response to detecting one or more shadows of occluded objects moving towards the direction of vehicle travel, the vehicle202can reduce its speed and/or come to a stop to allow more time to avoid the occluded object, should it enter the road.

It should be appreciated that in some embodiments a learning algorithm can be implemented such as a neural network (deep or shallow, which may employ a residual learning framework) and be applied instead of, or in conjunction with, another algorithm described herein to create additional modes or to improve the above-described modes and/or transitions between modes. Such learning algorithms may implement a feedforward neural network (e.g., a convolutional neural network) and/or a recurrent neural network, with structured learning, unstructured learning, and/or reinforcement learning. In some embodiments, backpropagation may be implemented (e.g., by implementing a supervised long short-term memory recurrent neural network, or a max-pooling convolutional neural network which may run on a graphics processing unit). Moreover, in some embodiments, unstructured learning methods may be used to improve structured learning methods. Moreover still, in some embodiments, resources such as energy and time may be saved by including spiking neurons in a neural network (e.g., neurons in a neural network that do not fire at each propagation cycle).

FIG. 3Cillustrates an exemplary method370of identifying a shadow208of occluded object206using a learning algorithm according to examples of the disclosure. In some examples, method370can be performed in addition or as an alternative to method350described above with reference toFIG. 3B. In some examples, one or more steps of method350can be combined with one or more steps of method370.

Vehicle202can collect example images to form a training data set (step372of method370). In some examples, the example images can be captured by one or more cameras (e.g., camera(s)106) of vehicle202. Additionally or alternatively, one or more example images can be uploaded to an onboard computer (e.g., onboard computer110) of vehicle202from a different camera. The images can include still images and/or videos captured in pedestrian-heavy zones such as parking lots, cities, school zones, neighborhoods, and other locations and scenarios where an occluded object may suddenly enter the path of vehicle travel.

In some examples, the example shadows of moving objects can be segmented in the example images (step374of method370). Step374can include segmenting the example images manually or using one or more steps of method350described above with reference toFIG. 3Bto automatically segment the images, for example. In some examples, vehicle202can store (e.g., using onboard computer110) the segmented example images.

Next, vehicle202can train a classifier to detect shadows of moving objects (e.g., such as shadow208of occluded object206) using the segmented example images (step376of method370). In some examples, vehicle202can train the classifier using a learning algorithm, such as a Convolutional Neural Network algorithm.

In some examples, steps372-376can be part of a vehicle setup procedure performed at a dealership or factory. Additionally or alternatively, steps372-376can be performed multiple times while the vehicle202is parked and/or while the vehicle202is in use. In some examples, vehicle202can use a wireless connection to receive one or more segmented or unsegmented example images from a server and/or another vehicle to train the classifier to identify shadows of occluded objects (e.g., shadow208of occluded object206). The classifier can be trained multiple times or on an ongoing basis as new example images become available to the vehicle202.

While the vehicle202is driving and capturing one or more still or video images (e.g., image300), the classifier can be applied to the images to identify moving shadows (step378of process370). For example, the classifier can associate one or more characteristics of the moving shadows in the training data set with a moving shadow and identify a moving shadow (step380of method370) in a captured image based on identifying one or more of the characteristics in the captured image.

In some examples, vehicle202can perform method350and/or370while operating in a shadow detection mode. One or more steps of method350and method370can be combined. In some examples, steps of method350and/or method370can be repeated, alternated, performed in any order, and/or skipped.

Thus, examples of the disclosure provide various ways a vehicle can detect an occluded object based on the shadow of the occluded object while driving in an autonomous driving me, allowing the vehicle to reduce its speed to avoid a collision should the object enter the vehicle's path of travel.

Therefore, according to the above, some examples of the disclosure are related to a vehicle comprising: one or more cameras; one or more actuator systems; and a processor operatively coupled to the one or more cameras and the one or more actuator systems, the processor configured to: identify a shadow in one or more images captured by the one or more cameras; determine whether the shadow is moving in a direction towards a direction of vehicle travel; and in accordance with a determination that the shadow is moving in a direction towards the direction of vehicle travel, reducing a speed of the vehicle using the one or more actuator systems. Additionally or alternatively, in some examples, the vehicle comprises a location system and a map interface, wherein the processor is operatively coupled to the location system and the map interface, and the processor is further configured to: identify a location of the vehicle based on one or more of the location system and the map interface; and based on a determination that the vehicle location is in a pedestrian heavy zone, enter a shadow detection mode, wherein the shadow detection mode causes the processor to identify the shadow and determine whether the shadow is moving. Additionally or alternatively, in some examples, the shadow is a shadow of an occluded object and the occluded object is not included in the one or images captured by the one or more cameras of the vehicle. Additionally or alternatively, in some examples, the processor is further configured to, in accordance with a determination that the shadow is stationary or moving in a direction away from the direction of vehicle travel, maintain the speed of the vehicle using the one or more actuator systems. Additionally or alternatively, in some examples, identifying the shadow in the one or more images comprises: segmenting a plurality of pixels of the one or more images into groups based on a darkness of each pixel, wherein pixels within each group have darknesses within a first threshold difference of each other; and identifying a plurality of dark pixels having a first darkness surrounded by a plurality of light pixels having a second darkness, the first darkness darker than the second darkness by at least a second threshold difference. Additionally or alternatively, in some examples, the vehicle further comprises one or more of a LiDAR sensor, an ultrasonic sensor, a radar sensor, and a range sensor, wherein identifying the shadow in the one or more images comprises: identifying a plurality of pixels of the one or more images illustrating an image of a ground based on data from the one or more of the LiDAR sensor, the ultrasonic sensor, the radar sensor, and the range sensor; and identifying the shadow within the pixels illustrating the image of the ground. Additionally or alternatively, in some examples, identifying the shadow in the one or more images comprises comparing the shadow to an expected shadow shape. Additionally or alternatively, in some examples, identifying the shadow in the one or more images comprises: collecting a plurality of example images; segmenting a plurality of example shadows in the plurality of example images; training a classifier using the plurality of example images; and applying the classifier to the one or more images.

Some examples of the disclosure are related to a method of operating a vehicle in an autonomous driving mode, the method comprising: capturing one or more images at one or more cameras of the vehicle; identifying a shadow in the one or more images; determining whether the shadow is moving in a direction towards a direction of vehicle travel; and in accordance with a determination that the shadow is moving in a direction towards the direction of vehicle travel, reducing a speed of the vehicle using one or more actuator systems of the vehicle. Additionally or alternatively, in some examples, the method further comprises identifying a location of the vehicle based on one or more of a location system and a map interface of the vehicle; and based on a determination that the vehicle location is in a pedestrian heavy zone, entering a shadow detection mode, wherein the shadow detection mode causes the processor to identify the shadow and determine whether the shadow is moving. Additionally or alternatively, in some examples, the shadow is a shadow of an occluded object and the occluded object is not included in the one or images captured by the one or more cameras of the vehicle. Additionally or alternatively, in some examples, the method further comprises, in accordance with a determination that the shadow is stationary or moving in a direction away from the direction of vehicle travel, maintaining the speed of the vehicle using the one or more actuator systems. Additionally or alternatively, in some examples, the method further comprises segmenting a plurality of pixels of the one or more images into groups based on a darkness of each pixel, wherein pixels within each group have darknesses within a first threshold difference of each other; and identifying a plurality of dark pixels having a first darkness surrounded by a plurality of light pixels having a second darkness, the first darkness darker than the second darkness by at least a second threshold difference. Additionally or alternatively, in some examples, the method further comprises identifying a plurality of pixels of the one or more images illustrating an image of a ground based on data from one or more of a LiDAR sensor, an ultrasonic sensor, a radar sensor, and a range sensor included in the vehicle; and identifying the shadow within the pixels illustrating the image of the ground. Additionally or alternatively, in some examples, the method further comprises comparing the shadow to an expected shadow shape. Additionally or alternatively, in some examples, the method further comprises collecting a plurality of example images; segmenting a plurality of example shadows in the plurality of example images; training a classifier using the plurality of example images; and applying the classifier to the one or more images.