Surround view by drones

An apparatus includes a visual display to be viewed by a vehicle occupant. At least one drone includes a camera. A controller is configured to receive images from the camera on the at least one drone and generate an overhead view of the vehicle based on the images received from the at least one drone and display the overhead view on the visual display.

BACKGROUND

A vehicle can include a plurality of cameras that generate images of vehicle position that are combined together to present the driver with an overhead view of a position of the vehicle in relation to a surrounding environment. This type of visual display can assist the driver when maneuvering in tight constraints. In certain situations, the coverage provided by the cameras may be insufficient to fully understand the position of the vehicle in relation to the surrounding environment.

SUMMARY

In one exemplary embodiment, an apparatus includes a visual display to be viewed by a vehicle occupant. At least one drone includes a camera. A controller is configured to receive images from the camera on the at least one drone and generate an overhead view of the vehicle based on the images received from the at least one drone and display the overhead view on the visual display.

In a further embodiment of any of the above, the vehicle is coupled to a trailer and the overhead view shows both the vehicle and the trailer.

In a further embodiment of any of the above, the vehicle is provided with at least one camera to generate vehicle position data for the overhead view of the vehicle. The at least one drone is used to extend a visualization area to include position data for the trailer.

In a further embodiment of any of the above, the at least one drone is deployed from the vehicle, the trailer, or nearby infrastructure.

In a further embodiment of any of the above, the visual display displays images generated by at least one camera associated with the vehicle.

In a further embodiment of any of the above, the at least one camera includes a plurality of camera.

In a further embodiment of any of the above, the overhead view includes a 360 degree surround overhead view of the vehicle.

In a further embodiment of any of the above, the at least one drone is maneuvered by the vehicle occupant via HMI.

In another exemplary embodiment, a method of providing an overhead view of a vehicle includes (a) deploying at least one drone in response to the vehicle on a drivable surface encountering a visual obstruction relative to the vehicle, (b) generating an overhead view of the vehicle based at least on images from the at least one drone and (c) displaying the overhead view on a visual display.

In a further embodiment of any of the above, step (a) includes deploying the at least one drone from a docking station on at least one of the vehicle, a trailer coupled to the vehicle, or nearby infrastructure.

In a further embodiment of any of the above, step (b) includes generating the overhead view of the vehicle based on images from the at least one drone and images from at least one camera associated with the vehicle.

In a further embodiment of any of the above, the at least one drone extends a field of view surrounding the vehicle beyond a field of view of the at least one camera on the vehicle.

In a further embodiment of any of the above, the visual obstruction relative to vehicle includes reversing the vehicle along the drivable surface.

In a further embodiment of any of the above, the visual obstruction relative to the vehicle includes a structure adjacent the drivable surface.

In a further embodiment of any of the above, step (a) includes maneuvering the drone based on a position of the vehicle relative to the drivable surface.

In a further embodiment of any of the above, the position of the vehicle relative to the drivable surface includes the vehicle at an intersection of multiple drivable surfaces.

In a further embodiment of any of the above, the vehicle is coupled to a trailer.

In a further embodiment of any of the above, step (b) includes modifying the overhead view of the vehicle based on input from a vehicle occupant.

In a further embodiment of any of the above, the at least one drone is maneuvered via HMI to modify the overhead view of the vehicle.

In a further embodiment of any of the above, the overhead view includes a 360 degree surround exterior view of the vehicle.

DETAILED DESCRIPTION

The subject disclosure comprises a visualization system that provides for an improved overhead view of vehicle position for a vehicle occupant. In one example shown inFIG.1, a vehicle10is pulling a trailer12along a drivable surface14, such as a road, driveway, parking lot, or loading dock. The vehicle10includes a visual display16that is to be viewed by the vehicle occupant. The visual display16, among other things, is used to show an overhead view of the vehicle10in relation to a surrounding environment (seeFIG.2, for example). In the illustrated example, the overhead view is a 360 degree exterior surround view of the vehicle10. The visual display16can include one of an integrated display in the vehicle10, such as part of an entertainment system. Alternatively, the visual display could include an external display, or a personal mobile device, such as a cell phone, tablet, or laptop.

At least one drone18is used to communicate vehicle position data to the visual display16. The data may first be communicated to a system controller20, which analyzes the data and then communicates the data to the visual display16, through the use of a transmitter/receiver19on the drone18and a transmitter/receiver21on the controller20. In one example, the position data includes images captured by a camera17on the drone18. The position data from the drone18can be used to show both the position of the vehicle10and the trailer12in the overhead view.

In another example, the vehicle10is provided with one or more cameras22that send data to the controller20to generate the vehicle position for the overhead view of the vehicle10. In the illustrated example, the cameras22are associated with the vehicle10. The controller20combines the various images captured by the cameras22to generate the overhead view as known. The drone18can then be used as needed to enlarge or extend a visualization area to include position data for the trailer12. Additionally, the cameras17on the drone18can be used to provide images of areas that are outside a field of view of the cameras22on the vehicle10.

The drone18can be selectively or automatically deployed from the vehicle10, the trailer12, or a nearby infrastructure24such as a building, parking structure, loading dock, etc. In one example, the vehicle occupant maneuvers the drone18via Human Machine Interface (HMI), e.g. touch-screen, knob, etc.

As such, the system provides an improved overview view for a driver maneuvering a vehicle with or without a trailer by having a drone18fly over a scenario, e.g. vehicle and/or trailer, and displaying the scenario in a bird's eye perspective to the driver via surround view visualization. In one example, the surround view image provided by the drone18is by default controlled by the position of the vehicle10and/or trailer12. The H-IMI can then be selectively used to adjust the image by using zoom and shift commands.

Traditional surround view systems use image data coming from the cameras22on the vehicle10to stitch together a compiled image view through which the driver gets a displayed visualization of the vehicle with any surrounding obstacles from a bird's eye perspective. As discussed above, this helps the driver to visualize the surrounding vehicles in relation to the driver's vehicle and eases maneuvering of the vehicle in tight spaces and parking situations. In certain situations, the driver can choose where the viewpoint of the visualization is to be set.

The subject invention uses the footage from the drone18either to enlarge/extend the visualization in a situation where a trailer12is hitched to the vehicle10, or to provide the surround view visualization of the vehicle10itself. This is accomplished by the drone18hovering above the vehicle or by flying along the vehicle's path. The drone18can be selectively or automatically engaged from the vehicle10, the trailer12, or the infrastructure24. A drone-dock26can be mounted to the trailer12, vehicle10, or the infrastructure12to hold the drone18when the drone18is not deployed. The drone-dock26can include charging capability, communication interfaces, etc. as needed for the operation of the drone18.

This type of visualization system significantly improves driver maneuvering for vehicles10with trailers12. The drone18could also be utilized to capture critical scenarios like driveways where there are obstructions and drivers cannot see any lateral traffic or obstacles. Or, the drone18could be used in loading dock scenarios where drivers of a truck have to perfectly align the truck/trailer with a loading dock. The drone18could also automatically engage in such scenarios and the footage of the drone18could be merged with the surround view footage from the cameras22on the vehicle10.

In one example, the drone18is maneuvered by the vehicle occupant, e.g. driver, pinching and zooming and scrolling/dragging a surface of a touchscreen into certain directions on the surround view visualization. An algorithm translates this human machine input into a certain drone movement. Input can also be given by any other HMI input such as a3D input-knob (e.g. iDrive controller) for example.

FIG.3is a schematic view of the vehicle10, drone18, and visualization system incorporating the subject disclosure. In the illustrated example, the vehicle10is located at an intersection36of drivable surfaces14, such as a driveway entering onto a road. When the vehicle10is located on a driveway14with the intention of pulling out onto the road14with traffic traveling in multiple directions (see vehicles10B and10C), it is possible that a view of road14with multiple direction of traffic may be partially obstructed.

In this example, the view of the vehicle occupant and cameras22on the vehicle10is obstructed by a structure30, such as a building or a tree. The dashed lines32represents a field of view of the cameras22that is obstructed by the structure30. Therefore, when the drone18is deployed, an overhead surround view that is displayed on the visual display16can be expanded beyond the structure30and the trailer12as shown by dashed line34. Therefore, the overhead surround view generated at least in part from images from the drone18expands the field of view of the overhead surround view generated for the visual display16. This can warn the vehicle occupant of oncoming traffic, such as vehicle10B, or provide a larger view of the vehicle10and trailer12on the drivable surface14.

FIG.4illustrates an example method100of providing an overhead surround view of the vehicle10with or without the trailer12. The method100includes deploying at least one drone18when the vehicle10encountering a visual obstruction relative to the drivable surface14. (Step102). The visual obstruction can occur in a number of circumstances, such as reversing the vehicle10or a structure30that obstructs a field of view of the vehicle occupant or one of the cameras22on the vehicle10. When the drone18is deployed, the drone18departs from the drone-dock26on at least one of the vehicle10, the trailer12coupled to the vehicle10, or the infrastructure24.

The drone18can be maneuvered based on a number of criteria. In one example, the drone18is maneuvered based on a position of the vehicle10on the drivable surface14. The position of the vehicle10may include the vehicle10located at the intersection36of drivable surfaces14, such as exiting a narrow driveway onto a wider road, or at a loading dock.

The controller20can then generate an overhead view of the driveable surface14based on the images received from the drone18. (Step104). The overhead view of the driveable surface14based on the images from the drone18can provide a wider field of view than just the cameras22associated with the vehicle10or provide a view of the area surrounding the vehicle10that is obstructed from the cameras22. Additionally, the controller20can generate the overhead surround view based on both the images received from the drone18and the images received by the cameras22. Therefore, the images from the drone18can be used to supplement or extend the field of view captured by the images from the cameras22associated with the vehicle10or vice versa.

Furthermore, when the overhead surround view is displayed on the visual display16(Step106), the vehicle occupant can modify the overhead view further by interacting with the HMI. For example, the vehicle occupant could zoom in or out of the overhead view, change a direction of the camera17on the drone18to provide another view, and/or maneuver the drone18to provide another view.