System and method for tracking moving objects to avoid interference with vehicular door operations

A system that performs a method is disclosed. Image data from a first camera mounted on a vehicle is received, the vehicle including a first element configured to open into a first space external to the vehicle, and the image data indicative of motion of a first object during a first time period, the first object located outside of the first space. Whether the motion of the first object during the first time period indicates that the first object will interfere with operation of the first element during a second time period, after the first time period, is determined. In accordance with a determination that the motion of the first object indicates that the first object will interfere with the operation of the first element during the second time period, an action that avoids the interference is performed.

FIELD OF THE DISCLOSURE

This relates generally to object detection, and more particularly, to a system and method for tracking moving objects using one or more cameras on a vehicle to determine whether the objects will interfere with vehicle door operation.

BACKGROUND OF THE DISCLOSURE

Modern vehicles, especially automobiles, increasingly include various sensors for detecting and gathering information about the vehicles' surroundings. These sensors may include ultrasonic sensors for detecting the proximity of a vehicle to objects in the vehicle's surroundings, and may be used to provide information about the current locations of the objects with respect to the vehicle. However, some objects that may interfere with vehicle operation (e.g., the opening of vehicle doors) may be moving objects that, while not currently interfering with vehicle operation, may interfere with vehicle operation in the future. Current object detection techniques may not account for such future interference.

SUMMARY OF THE DISCLOSURE

Examples of the disclosure are directed to using one or more sensors (e.g., cameras) to detect the movements of objects in a vehicle's surroundings. In some examples, based on those movements, the vehicle can estimate trajectories for those objects, and determine whether those objects will interfere with the operation of vehicle doors in the future (e.g., the opening of the vehicle doors, the remaining open of the vehicle doors without colliding with an object, etc.). If future interference is determined, the vehicle can take appropriate action to avoid the interference, such as alerting a driver or a passenger of the potential interference by playing a sound through one or more speakers in the vehicle, causing one or more lights inside the vehicle to flash, displaying a visual indication on one or more display screens inside the vehicle, and/or providing vibration to the steering wheel of the vehicle or the driver's seat of the vehicle. In some examples, the vehicle can additionally or alternatively begin to automatically close the door(s) in danger of being obstructed (if the door(s) are already open), prevent the opening of the door(s) in danger of being obstructed (if the door(s) are not yet open), and/or provide an alert to the object (in case the object is a person or controlled by a person) in an effort to alert the person of the potential collision.

DETAILED DESCRIPTION

Some vehicles, such as automobiles, may include ultrasonic sensors for detecting the proximity of the vehicles to objects in the vehicles' surroundings. These sensors may be used to provide information about the current locations of the objects with respect to the vehicle. However, some objects that may interfere with vehicle operation (e.g., the opening of vehicle doors) may be moving objects that, while not currently interfering with vehicle operation, may interfere with vehicle operation in the future. Examples of the disclosure are directed to using one or more sensors (e.g., cameras) to detect the movements of objects in a vehicle's surroundings. In some examples, based on those movements, the vehicle can estimate trajectories for those objects, and determine whether those objects will interfere with the operation of vehicle doors in the future (e.g., the opening of the vehicle doors, the remaining open of the vehicle doors without colliding with an object, etc.). If future interference is determined, the vehicle can take appropriate action to avoid the interference, such as alerting a driver or a passenger of the potential interference by playing a sound through one or more speakers in the vehicle, causing one or more lights inside the vehicle to flash, displaying a visual indication on one or more display screens inside the vehicle, and/or providing vibration to the steering wheel of the vehicle or the driver's seat of the vehicle. In some examples, the vehicle can additionally or alternatively begin to automatically close the door(s) in danger of being hit (if the door(s) are already open), prevent the opening of the door(s) in danger of being hit (if the door(s) are not yet open), and/or provide an alert to the object (in case the object is a person or controlled by a person) in an effort to alert the person of the potential collision.

FIG. 1illustrates exemplary vehicle100having doors102according to examples of the disclosure. In some examples, doors102a,102b,102cand102d(referred to collectively as102) can be at least partially automated. For example, one or more of doors102can open automatically in response to a user input (e.g., button press) in vehicle100. Doors102can open into interaction spaces104external to vehicle100. Though doors102are described as being at least partially automated, the examples of the disclosure can additionally be applied to doors that are not automated at all or in the same way. Further, though vehicle100is described as having four doors, vehicles with fewer than or more than four doors are also within the scope of the disclosure.

In some examples, vehicle100can include one or more sensors (e.g., ultrasonic sensors, LIDAR sensors, cameras, etc., not illustrated) for detecting the presence of objects in an area proximate to the vehicle. This proximal area can be represented as area106inFIG. 1. In some examples, vehicle100can sense the presence of objects within area106when one or more of doors102are opened. If an object is within the interaction space(s)104of the door(s)102being opened, vehicle100can alert the driver, slow or stop the opening of the door(s), or a number of other actions to try to prevent the door(s) from colliding with the object.

However, some or all of the objects that may interfere with the opening of doors102may not be stationary objects, but rather moving objects. As such, it can be beneficial for vehicle100to track the movements of objects—not only their positions—and estimate their trajectories to determine whether their estimated trajectories will interfere with doors102in any way; for example, whether the objects will interfere with the opening of doors102, whether the objects will collide with doors102while they are open, etc. Further, it can be beneficial for vehicle100to be able to track the movements of objects, not only inside area106, but also outside of area106, because objects outside of area106can also interfere with doors102by eventually moving into interaction spaces104. For example, object108, despite being outside of area106, can be moving towards door102dand interaction space104dalong trajectory110, potentially interfering with the opening of door102d, or, if door102dis already open, potentially colliding with door102d. If vehicle100is able to track object108and its movement, and determine trajectory110, the vehicle can determine whether object108may interfere with the operation of doors102. The examples of the disclosure will be described in the context of doors, though it is understood that the examples of the disclosure can be implemented in the context of any element on vehicle102that can open into a space external to the vehicle, such as hoods, trunk lids, windows, gas cap access covers, side view mirrors, etc.

FIG. 2illustrates exemplary vehicle200having cameras220,224and230for tracking the movements of objects and determining object trajectories according to examples of the disclosure. Vehicle200can have doors202that can open into interaction spaces204, as described with reference toFIG. 1. Vehicle200can, though not necessarily, include one or more sensors for detecting objects within a proximity of the vehicle (e.g., within region106of vehicle100inFIG. 1), though they are not illustrated for simplicity.

To facilitate tracking the movements of objects and determining object trajectories, vehicle can include cameras220,224aand224b(referred to collectively as224) and230. Though four cameras are illustrated inFIG. 2, it is understood that vehicle200can include fewer or more than four cameras in implementing the examples of the disclosure. Further, though vehicle200is described as including cameras for the purpose of tracking the movements of objects and determining object trajectories, additional or alternative sensors can be used for this purpose; for example, ultrasonic sensors, LIDAR sensors, etc., can additionally or alternatively be used.

Camera220can be substantially forward-facing, and can have field of view222—thus, camera220can track objects substantially in front of vehicle200. Cameras224can be substantially rear-facing, and can have fields of view226and228that can, though not necessarily, at least partially include interaction spaces204into which doors202open—thus, cameras224can track objects substantially beside and behind vehicle200on both the left and the right. In some examples, in addition to being used to track objects, camera224acan be used as a replacement for a traditional left side-view mirror on vehicle200, and camera224bcan be used as a replacement for a traditional right side-view mirror on the vehicle. Camera230can be substantially rear-facing, and can have field of view232—thus, camera230can track objects substantially behind vehicle200. In some examples, in addition to being used to track objects, camera230can be used as a replacement for a traditional rear-view mirror on vehicle200.

Using one or more of cameras220,224and230, vehicle200can track the movements of objects, and determine whether the objects might interfere with the operation of one or more of doors202. For example, based on images captured by camera230, vehicle200can identify the current position of object208, and based on the current position and past positions of object208stored by the vehicle, determine estimated trajectory210of object208. This determination of trajectory210can be performed using any number of appropriate movement extrapolation techniques. Trajectory210can indicate that object208is moving towards, and will eventually move into, interaction space204dof door202d. As such, vehicle200can take appropriate action, as previously described, to avoid the interference of object208with the operation of door202d. For example, vehicle200can alert (e.g., an audio alert, a visual alert, a tactile alert, etc.) a driver or passenger of the vehicle of the potential collision, so that the driver or passenger can take action to rectify the situation. In some examples, the alert may include vehicle200playing a sound through one or more speakers in the vehicle, causing one or more lights inside the vehicle to flash, displaying a visual indication on one or more display screens (e.g., liquid crystal displays, organic light emitting diode displays, etc.) inside the vehicle and/or providing vibration to the steering wheel of the vehicle or the driver's seat of the vehicle. Additionally or alternatively, vehicle200can begin to automatically close the door(s) in danger of being hit (if the door(s) are already open), or can prevent the opening of the door(s) in danger of being hit (if the door(s) are not yet open). Vehicle200can additionally or alternatively provide an alert to the object (in case the object is a person or controlled by a person) in an effort to alert the person of the potential collision. For example, a lighting strip included in doors202(in some examples, only the door with which the object will interfere) be illuminated to make the doors more noticeable to those outside of vehicle200.

Similarly, based on images captured by camera224b, vehicle200can identify the current position of object212relative to the vehicle, and based on the current position and past positions of object212stored by the vehicle, determine estimated trajectory214of object212. Trajectory214can indicate that object212is not moving towards interaction spaces204cor204dof doors202cand202d, respectively (or any other interaction spaces of other doors of vehicle200). As such, vehicle200can determine that object212will likely not interfere with the opening of doors202cand202d(or any other doors of the vehicle), and can forgo providing an alert to that effect. In some examples, vehicle200can begin tracking the movements of one or more objects using a first camera, and can transition to tracking the movements of the one or more objects using a second camera, as appropriate (e.g., transitioning from using camera230to using camera224bto track the movement of object208as it moves from field of view232to field of view228towards interaction space204d). Exemplary techniques for tracking objects using a camera are described in BenAbdelkader, C., Burlina, P., & Davis, L. (2000), “Single camera multiplexing for multi-target tracking,” Multimedia Video-Based Surveillance Systems (pp. 130-142), Springer US, the contents of which is hereby incorporated by reference for all purposes.

It is understood that the object-door interference determinations of the disclosure can be performed before door(s)202are opened, while the door(s) are opening and/or while the door(s) are open. Thus, for example, while one or more of doors202are open, vehicle200can track the movement of objects using cameras220,224and/or230, and if the vehicle determines that an object will collide with one or more of the doors, the vehicle can take appropriate action to avoid such collision. For example, the vehicle can alert (e.g., with an audible, visual, tactile, etc., alert) a driver or passenger of vehicle200of the potential collision, so that the driver or passenger can take action to rectify the situation. Additionally or alternatively, vehicle200can stop/pause the operations of or begin to automatically close the door(s) in danger of being hit, and/or provide an alert to the object (in case the object is a person or controlled by a person) in an effort to alert the person of the potential collision. For example, a lighting strip included in doors202(in some examples, only the door with which the object will collide) can be illuminated to make the doors more noticeable to those outside of vehicle200. If one or more of doors202are opening when vehicle200determines that an object will interfere with the one or more doors, the vehicle can alert a driver or passenger of the vehicle of the potential interference, prevent the one or more doors from opening further, and/or automatically begin closing the one or more doors.

In each of the examples disclosed above, vehicle200can continuously track the movements of objects around the vehicle. As a result, if, after determining that an object will interfere with the operation of one or more of doors202, the object's movement changes such that its trajectory, determined by vehicle200, no longer indicates that it will interfere with the one or more doors, the vehicle can cancel and/or reverse any action that it took previously to avoid the interference. For example, if vehicle200had previously provided an alert to a driver or passenger of the vehicle, the vehicle can cease providing the alert. Similarly, if vehicle200had previously started closing one or more of doors202, the vehicle can stop closing the one or more doors and/or can begin automatically re-opening the one or more doors to their original positions. As another example, if vehicle200had previously prevented the opening of one or more of doors202, the vehicle can permit the opening of the one or more doors and/or begin automatically opening the one or more doors. Other cancellations or reversals of actions previously taken by vehicle200can analogously be executed.

FIG. 3illustrates exemplary process300for tracking objects to avoid interference with vehicle doors according to examples of the disclosure. Process300can be performed at any time, such as before the vehicle doors are opened, while the vehicle doors are opening and/or while the vehicle doors are open. At302, one or more images of the surroundings of a vehicle can be received. The one or more images can be captured by one or more cameras mounted on the vehicle (e.g., as described with reference toFIG. 2).

At step304, objects in the one or more images can be identified and tracked over time. For example, the vehicle can determine that one or more objects exist in the one or more images, and can track the positions of those objects over time (e.g., as described with reference toFIG. 2). In some examples, object tracking at304can be performed continuously, such that object tracking can continue even after steps306,308and310of process300(described below) are performed.

At step306, the vehicle can determine estimated trajectories for the objects in the one or more images (e.g., as described with reference toFIG. 2). These determinations can be performed using any appropriate trajectory determination techniques, such as various predictive filtering techniques. In some examples, estimated trajectory determinations at306can be performed continuously, such that estimated trajectory determinations can continue even after steps308and310of process300(described below) are performed, and such that estimated trajectories associated with objects can change as movements of the objects change.

At step308, the vehicle can determine whether the trajectories determined at306indicate potential object interference with the operation of one or more doors on the vehicle (e.g., as described with reference toFIG. 2). For example, the vehicle can determine whether the trajectory of an object in the one or more images intersects with the interaction space into which a vehicle door will open, is opening or has opened. If so, the vehicle can determine that the object will interfere with the operation (e.g., the opening or remaining open) of that door. In some examples, the vehicle may determine that an object will not interfere with the operation of a door, despite the estimated trajectory of that object intersecting with the door's interaction space, if the vehicle determines that the object will not reach the door's interaction space for longer than a threshold period of time (e.g., based on the speed of movement of the object). This threshold period of time can be the time it takes for the door to open fully if the door is currently closed, for example, and can be stored as a predetermined period of time in a memory in the vehicle. Alternatively, this threshold period of time can be the time it takes for the door to open fully, a person to enter or exit the vehicle through the door, and the door to close fully, if the door is currently closed, for example—this threshold period of time can also be stored as a predetermined period of time in a memory in the vehicle. In some examples, this threshold period of time can be the time it takes for the door to close fully if the door is currently open, for example. It is understood that any appropriate threshold period of time can be used in accordance with the examples of the disclosure. Further, in some examples, interference determinations at308can be performed continuously, such that interference determinations can continue even after step310of process300(described below) is performed, and such that interference determinations associated with objects can change as movements and/or trajectories of the objects change.

At step310, if the vehicle determines at308that one or more objects will interfere with the operation of one or more doors on the vehicle, then the vehicle can take appropriate action to attempt to avoid such interference, as described above with respect toFIG. 2. If the vehicle is set in an autonomous driving mode, at310, the vehicle can additionally or alternatively automatically move (e.g., by automatically disengaging the vehicle's brakes, engaging the vehicle's drive motor, and adjusting the steering angle to move at a calculated speed and direction) to a new location at which the one or more objects will no longer interfere with the operation of the vehicle's doors. In some examples, the vehicle can utilize dynamically obtained GPS data from a GPS service to automatically determine the new location to which to move, and/or data from other sensors on the vehicle to determine an available location and/or a driving path to that location. Additionally, in some examples, step310can include canceling or reversing an action previously performed by the vehicle due to changes in object movements or trajectories, as previously described.

FIG. 4illustrates an exemplary system block diagram of vehicle control system400according to examples of the disclosure. Vehicle control system400can perform any of the methods described with reference toFIGS. 2-3. System400can be incorporated into a vehicle, such as a consumer automobile. Other example vehicles that may incorporate the system400include, without limitation, airplanes, boats, or industrial automobiles. Vehicle control system400can include one or more cameras406capable of capturing image data (e.g., video data), as previously described with respect toFIGS. 2-3. Vehicle control system400can also include one or more other sensors407(e.g., radar, ultrasonic, etc.) capable of detecting objects in the vehicle's surroundings, and a Global Positioning System (GPS) receiver408capable of determining the location of the vehicle. Vehicle control system400can include an on-board computer410that is coupled to the cameras406, sensors407and GPS receiver408, and that is capable of receiving the image data from the camera and/or outputs from the sensors407and the GPS receiver408. The on-board computer410can be capable of tracking the movement of one or more objects and determining whether the one or more objects will interfere with the operation of one or more vehicle doors, as described in this disclosure. On-board computer410can include storage412, memory416, and a processor414. Processor414can perform any of the methods described with reference toFIGS. 2-3. Additionally, storage412and/or memory416can store data and instructions for performing any of the methods described with reference toFIGS. 2-3. Storage412and/or memory416can be any non-transitory computer readable storage medium, such as a solid-state drive or a hard disk drive, among other possibilities. The vehicle control system400can also include a controller420capable of controlling one or more aspects of vehicle operation, such as providing an alert to a driver or passenger, controlling the states of one or more vehicle doors, and moving the vehicle, based on the determinations of the on-board computer410.

In some examples, the vehicle control system400can be connected to (e.g., via controller420) one or more actuator systems430in the vehicle and one or more indicator systems440in the vehicle. The one or more actuator systems430can include, but are not limited to, a motor431or engine432, battery system433, transmission gearing434, suspension setup435, brakes436, steering system437and door system438. The vehicle control system400can control, via controller420, one or more of these actuator systems430during vehicle operation; for example, to open or close one or more of the doors of the vehicle using the door actuator system438, or to move the vehicle using the motor431or engine432, battery system433, transmission gearing434, suspension setup435, brakes436and/or steering system437. The one or more indicator systems440can include, but are not limited to, one or more speakers441in the vehicle (e.g., as part of an entertainment system in the vehicle), one or more lights442in the vehicle (e.g., built into the doors of the vehicle), one or more displays443in the vehicle (e.g., as part of a control or entertainment system in the vehicle) and one or more tactile actuators444in the vehicle (e.g., as part of a steering wheel or seat in the vehicle). The vehicle control system400can control, via controller420, one or more of these indicator systems440to provide indications to the driver of a potential object interfering with the operation of the vehicle doors. This can be done as described with reference toFIGS. 2-3, for example.

Thus, the examples of the disclosure provide various ways to prevent moving objects from interfering with vehicle door operation.

Therefore, according to the above, some examples of the disclosure are directed to a system comprising: one or more processors; and a memory including instructions, which when executed by the one or more processors, cause the one or more processors to perform a method comprising: receiving image data from a first camera mounted on a vehicle, the vehicle including a first element configured to open into a first space external to the vehicle, and the image data indicative of motion of a first object during a first time period, the first object located outside of the first space; determining whether the motion of the first object during the first time period indicates that the first object will interfere with operation of the first element during a second time period, after the first time period; and in accordance with a determination that the motion of the first object indicates that the first object will interfere with the operation of the first element during the second time period, performing an action that avoids the interference. Additionally or alternatively to one or more of the examples disclosed above, in some examples, the first element comprises a door configured to open into an interaction space external to the vehicle. Additionally or alternatively to one or more of the examples disclosed above, in some examples, during the first time period, the first element is open, interfering with the operation of the first element comprises colliding with the first element while the first element is open. Additionally or alternatively to one or more of the examples disclosed above, in some examples, during the first time period, the first element is closed, and interfering with the operation of the first element comprises preventing the first element from opening without colliding with the first object. Additionally or alternatively to one or more of the examples disclosed above, in some examples, determining whether the motion of the first object during the first time period indicates that the first object will interfere with the operation of the first element during the second time period comprises: determining an estimated trajectory of the first object during the first time period; in accordance with a determination that the estimated trajectory of the first object indicates that the first object will reach the first space during the second time period, determining that the first object will interfere with the operation of the first element; and in accordance with a determination that the estimated trajectory of the first object indicates that the first object will not reach the first space, determining that the first object will not interfere with the operation of the first element. Additionally or alternatively to one or more of the examples disclosed above, in some examples, determining whether the motion of the first object during the first time period indicates that the first object will interfere with the operation of the first element during the second time period comprises: in accordance with a determination that the second time period is greater than a predetermined threshold time after the first time period, forgoing determining that the first object will interfere with the operation of the first element. Additionally or alternatively to one or more of the examples disclosed above, in some examples, the first camera is additionally used as a side-view mirror replacement. Additionally or alternatively to one or more of the examples disclosed above, in some examples, the first camera is substantially forward-facing on the vehicle. Additionally or alternatively to one or more of the examples disclosed above, in some examples, the first camera is additionally used as a rear-view mirror replacement. Additionally or alternatively to one or more of the examples disclosed above, in some examples, the action comprises automatically closing the first element. Additionally or alternatively to one or more of the examples disclosed above, in some examples, the action comprises automatically preventing the opening of the first element. Additionally or alternatively to one or more of the examples disclosed above, in some examples, the action comprises automatically moving the vehicle to a new location, the new location based on the motion of the first object during the first time period, and data from a GPS receiver indicative of a current location of the vehicle. Additionally or alternatively to one or more of the examples disclosed above, in some examples, the action comprises providing an alert to a driver or a passenger of the vehicle. Additionally or alternatively to one or more of the examples disclosed above, in some examples, the action comprises providing an alert to the first object. Additionally or alternatively to one or more of the examples disclosed above, in some examples, the method further comprises: after the first time period, determining that the first object will no longer interfere with the operation of the first element; and in response to determining that the first object will no longer interfere with the operation of the first element, canceling the action. Additionally or alternatively to one or more of the examples disclosed above, in some examples, the method further comprises: after the first time period, determining that the first object will no longer interfere with the operation of the first element; and in response to determining that the first object will no longer interfere with the operation of the first element, reversing the action. Additionally or alternatively to one or more of the examples disclosed above, in some examples, the vehicle includes a second element, different from the first element, configured to open into a second space external to the vehicle, different from the first space, determining that the motion of the first object indicates that the first object will interfere with the operation of the first element comprises determining that the motion of the first object indicates that the first object will not interfere with the operation of the second element, and performing the action comprises performing the action with respect to the first element but not the second element. Additionally or alternatively to one or more of the examples disclosed above, in some examples, the method further comprises receiving image data indicative of motion of a second object during the first time period, the second object located outside of the second space, the vehicle includes a second element, different from the first element, configured to open into a second space external to the vehicle, different from the first space, and performing the action comprises, in accordance with a determination that the second object will interfere with operation of the second element during a third time period, after the first time period: performing a first action with respect to the first element that avoids the interference of the first object with the operation of the first element; and performing a second action, different from the first action, with respect to the second element that avoids the interference of the second object with the operation of the second element.

Some examples of the disclosure are directed to a non-transitory computer-readable medium including instructions, which when executed by one or more processors, cause the one or more processors to perform a method comprising: receiving image data from a first camera mounted on a vehicle, the vehicle including a first element configured to open into a first space external to the vehicle, and the image data indicative of motion of a first object during a first time period, the first object located outside of the first space; determining whether the motion of the first object during the first time period indicates that the first object will interfere with operation of the first element during a second time period, after the first time period; and in accordance with a determination that the motion of the first object indicates that the first object will interfere with the operation of the first element during the second time period, performing an action that avoids the interference.

Some examples of the disclosure are directed to a vehicle comprising: a first camera; one or more processors coupled to the first camera; a door actuator coupled to the one or more processors; a first element configured to open, using the door actuator, into a first space external to the vehicle; and a memory including instructions, which when executed by the one or more processors, cause the one or more processors to perform a method comprising: receiving image data from the first camera, the image data indicative of motion of a first object during a first time period, the first object located outside of the first space; determining whether the motion of the first object during the first time period indicates that the first object will interfere with operation of the first element during a second time period, after the first time period; and in accordance with a determination that the motion of the first object indicates that the first object will interfere with the operation of the first element during the second time period, performing an action that avoids the interference.