Enhanced remote control of autonomous vehicles

Devices, systems, and methods for remote control of autonomous vehicles are disclosed herein. A method may include receiving, by a device, first data indicative of an autonomous vehicle in a parking area, and determining, based on the first data, a location of the autonomous vehicle. The method may include determining, based on a the location, first image data including a representation of an object. The method may include generating second image data based on the first data and the first image data, and presenting the second image data. The method may include receiving an input associated with controlling operation of the autonomous vehicle, and controlling, based on the input, the operation of the autonomous vehicle.

BACKGROUND

People increasingly are using autonomous vehicles. Some parking of autonomous vehicles may require human supervision. However, human operators may not be able to facilitate parking of autonomous vehicles.

DETAILED DESCRIPTION

Overview

People increasingly are using autonomous vehicles. To park autonomous vehicles, some laws, standards, and other governing regulations are being developed. In some autonomous vehicle parking systems, a human operator may be required to facilitate the autonomous vehicle parking. However, some methods for allowing human operators to remotely control the operation and parking of autonomous vehicles without being inside of the autonomous vehicles may suffer from connectivity delay (e.g., between devices used to control the autonomous vehicles), connectivity weakness, significant use of bandwidth (e.g., to stream images of vehicles, with more vehicles requiring additional bandwidth), and from an inability of a remote human operator to see from the perspective of a vehicle driver.

There is, therefore, a need for enhanced remote control of autonomous vehicles.

In one or more embodiments, autonomous vehicles may be directed to and from parking spaces in a parking area (e.g., parking lot, parking structure, or the like). The autonomous vehicles may be directed by a remote operator who may provide controls using one or more devices that result in corresponding commands sent to the autonomous vehicles to control speed, direction, and the like. The parking area may have cameras capable of detecting images of the parking areas at different locations. Because the locations of the cameras may be known, when images captured by the cameras indicate the presence of autonomous vehicles, the locations of the autonomous vehicles in or near the parking area may be determined (e.g., using one or more devices). Alternatively or in addition, the autonomous vehicles may provide their locations (e.g., using global positioning coordinates or the like) to one or more devices to identify vehicle locations in or near the parking area. The camera images also may be used to identify other vehicles, objects, people, or the like. Based on the camera images and vehicle location, one or more devices may generate a video feed in which the individual video frames represent a point of view that would be seen by a driver if there were a human driver of the autonomous vehicle. In this manner, a human operator representing a driver of the autonomous vehicle may see, in real time, other vehicles, objects, people, and the like near the autonomous vehicle, allowing the human operator to control the autonomous vehicle by directing the autonomous vehicle to a parking space.

In one or more embodiments, the image data of the camera images may be sent to one or more devices for image analysis. Using image analysis techniques, such as computer vision, machine learning, or the like, the one or more devices may identify objects near an autonomous vehicle. Alternatively or in addition, some objects may be known based on a location of a camera that captured an image of an autonomous vehicle. For example, entry/entrance structures, columns, signs, lights, and the like may be known objects of a parking area within a camera's field of view. Knowing the location of a vehicle, one or more devices may determine (e.g., using a map and/or other known object locations) some objects and their locations relative to the autonomous vehicle.

In one or more embodiments, using image and/or vehicle location data, one or more devices may generate a real time representation of a point of view of a driver of an autonomous vehicle if the autonomous vehicle had a driver (e.g., a point of view from the driver's seat). Images of the parking area may be provided to the one or more devices, or location data based on the images or vehicle information may be provided to the one or more devices (e.g., to reduce bandwidth and latency). Using known object locations based on a vehicle's location in a parking area, the one or more devices may generate images representing video frames. To reduce the amount of resources required for such image generation, some of the image data may be pre-generated. For example, images of vehicles, people, and/or objects may be pre-generated for vehicles, people, and/or objects known to be in a parking area. The pre-generated assets may be included in the image data used to represent real time video of the autonomous vehicle driving through the parking area. In this manner, rather than having to generate the exact representations of specific people, vehicles, or objects, which may be unnecessary for a human operator (e.g., the human operator may not need to know a nearby vehicle's color, make, model, etc., just that the vehicle is present at a certain distance from the autonomous vehicle being controlled), generic representations of people, vehicles, or objects may be included in the images. In this manner, the time needed to process and generate images may be reduced, allowing for real time video presentation to the human operator. In addition, bandwidth and other computer resources may be conserved by using the pre-generated assets.

In one or more embodiments, to account for a delay in wireless communications, one or more devices may consider the round trip time of communications between a human operator (e.g., remotely controlling an autonomous vehicle) and the time at which the autonomous vehicle performs a corresponding action. For example, it is not desirable for a human operator to provide a braking command to stop an autonomous vehicle only to have a delay between the command and the vehicle braking prevent the vehicle from braking too late. To account for the delay and the real time nature of an autonomous vehicle moving through a parking area, the scale of the real time video presented to the human operator may change to simulate when the autonomous vehicle is moving closer to or away from an object. For example, as the autonomous vehicle approaches another vehicle, a person, or an object, the presentation of the vehicle, a person, or an object to the human operator may show the vehicle, a person, or an object as becoming larger from frame to frame (or smaller from frame to frame when the vehicle, a person, or an object is moving further away from the autonomous vehicle). When multiple autonomous vehicles are being remotely controlled, the worst round trip delay time from among the autonomous vehicles may be applied to any of the autonomous vehicles when presenting the real time video to human operators (e.g., the round trip time may correspond to the rate at which the autonomous vehicle approaches or moves away from a vehicle, a person, or an object, and therefore may affect the size of the presentation of the vehicle, a person, or an object in the video). The speed of the autonomous vehicle may be provided by the autonomous vehicle or detected based on the location of the autonomous vehicle in the camera images captured at a frame rate (e.g., using the frame rate and locations of the autonomous vehicle in the images, a device may determine the distance divided by time, equaling the speed).

In one or more embodiments, directions to an open (unoccupied) parking space in the parking area may be presented to a human operator to allow the human operator to issue driving commands to the autonomous vehicle, causing the autonomous vehicle to drive to the open parking space. A device may maintain a list of parking spaces, whether they are occupied (e.g., based on image data and/or parking reservation/payment data, etc.), and the locations of the parking spaces in the parking area. Based on the location of an autonomous vehicle, a device may generate directions from the location of the autonomous vehicle to the parking space and may present the directions to the human operator (e.g., overlaying the real time video from the driver's seat perspective).

Illustrative Embodiments

Turning now to the drawings,FIG.1depicts an illustrative remote parking system100for autonomous vehicles in which techniques and structures for providing the systems and methods disclosed herein may be implemented.

Referring toFIG.1, the remote parking system may include a parking area102in which vehicles may park (e.g., a parking lot, a parking structure, etc.). Vehicles (e.g., vehicle104, vehicle106, vehicle108, etc.) may be autonomous vehicles. To park autonomous vehicles in the parking area102, a human operator109remotely may control operation of the autonomous vehicles (e.g., by controlling speed, steering, etc.). The parking area may include cameras (e.g., camera110, camera112, camera116, etc.) whose locations (and corresponding fields of view) may be known. Other objects in the parking area may be static/stationary, so their locations also may be known. For example, locations of the entrance gate120, the exit gate122, columns (e.g., column124), and the like (e.g., signs, trees, etc.) may have known locations (e.g., because their locations are fixed or are known at a given time of day). In this manner, the presence of known objects may identified in images of the parking area102simply by knowing the location of the camera that captured the images. The locations of some vehicles in the parking area102may be identified based on image analysis as explained further below, and also may be provided by the vehicles themselves. In addition, images of the parking area102may be captured not only by the cameras of the parking area102, but also by vehicle cameras (e.g., camera130of the vehicle104, camera132of the vehicle108, etc.). Other objects in the parking area102may not be known, but may be identified in images as explained further below. For example, vehicles (e.g., the vehicles104-108, vehicle140, vehicle142, vehicle144, vehicle146, etc.), in the parking area may be parked or moving, so controlling an autonomous vehicle in the parking area102may need to account for the presence of such objects. The remote parking system100may identify available and unavailable parking spaces (e.g., parking space150) by using image analysis and/or maintaining data for reserved and unreserved parking spaces.

Still referring toFIG.1, the remote parking system100may analyze images captured by the cameras of the parking area102. For example, using computer vision, machine learning, or other object identification techniques, devices of the remote parking system100(e.g., the cameras or one or more other devices152in or remote from the parking area102) may analyze images of the parking area provided by the cameras of the parking area102and/or the cameras of vehicles of the parking area102to identify objects and generate a real time representation of a driver's seat perspective from within an autonomous vehicle in the parking area102. As shown, an image160may represent an image captured by a camera of the parking area102(e.g., the camera110). When controlling the parking of the vehicle104, for example, the remote parking system100may analyze the image160and other images over time to identify the location of the vehicle104, the speed of the vehicle104(e.g., based on the vehicle's location across multiple images and the frame rate of the image capture), and the location of other objects in the parking area. The remote parking system100may determine which objects may be visible from the driver's seat of the vehicle104(e.g., objects in front of and to the side of the vehicle104), and how fast the vehicle104is approaching another object.

Still referring toFIG.1, using the image data of the image160, the remote parking system100may generate a representation (e.g., image data for respective frames of video) from the perspective of the driver's seat of the vehicle104to present to the human operator109. As shown, using a device172, the remote parking system100may present a real time representation174to the human operator109to allow the human operator109to simulate being in the vehicle104while driving the vehicle104. The real time representation174may include image data from the image160and/or pre-generated image assets. For example, the exact representation of the vehicle108(e.g., color, make, model, etc.) may not be necessary for the human operator109to see in order to operate the vehicle104. Instead, the presence of a vehicle in front of the vehicle104may suffice, so a pre-generated image of a vehicle176may be used to generate the real time representation174. In addition, because the locations of the image160and the location of the entrance gate120are known, the entrance gate120may be shown as a pre-generated representation178of the entrance gate120. In this manner, image processing may be less resource-intensive, reducing time, bandwidth, latency, and the like, so that the presentation of the real time representation174may be in real time. In addition, the remote parking system100may identify an open parking space, may generate directions to the open parking space from the known location of the vehicle104, and may present driving directions180using the real time representation174(e.g., using an overlay to allow the human operator109to see the driving directions180while operating the vehicle104).

In one or more embodiments, the data provided to the one or more devices202and/or to the device172may include the image160, or may include location data for the image160(e.g., the location of the camera that captured the image160). By providing the location data rather than the entire image, a live stream of video of the parking area102may not be necessary, thereby reducing bandwidth, latency, and processing resources. Instead, the one or more devices202and/or the device172may have access to the pre-generated assets and location data, and may use the pre-generated assets and location data to generate the real time representation174. For example, given the location of the image160and the vehicle104, the one or more devices202and/or the device172may identify known objects at the location. When the location data identifies that an object is detected (e.g., the vehicle108), the one or more devices202and/or the device172may use the pre-generated image of the vehicle176when generating the image data for the real time representation174.

In one or more embodiments, to account for a delay in wireless communications, the parking system100may consider the round trip time of communications between the human operator109(e.g., control inputs to the device172to control the speed, direction, etc. of the vehicle104) and the time at which the vehicle104performs a corresponding action (e.g., accelerates, brakes, turns, etc.). For example, it is not desirable for the human operator109to provide a braking command to stop the vehicle104only to have a delay between the command and the vehicle104braking prevent the vehicle104from braking too late. To account for the delay and the real time nature of an autonomous vehicle moving through a parking area, the scale of the real time representation174to the human operator109may change to simulate when the vehicle104is moving closer to or away from an object (e.g., as shown in more detail inFIGS.2and3). For example, as the vehicle104approaches another vehicle (e.g., the vehicle108), the real time representation174of the pre-generated image of the vehicle176may show the pre-generated image of the vehicle176as becoming larger from frame to frame (or smaller from frame to frame when the vehicle108is moving further away from the vehicle104). When multiple autonomous vehicles are being remotely controlled (e.g., the vehicle104-108), the worst round trip delay time from among the autonomous vehicles may be applied to any of the autonomous vehicles when presenting the real time representation174to the human operators109(or multiple human operators). The speed of the vehicle104may be provided by the vehicle104or detected based on the location of the vehicle104in the camera images captured at a frame rate (e.g., the frame rate indicative of the time, and when combined with a vehicle location, the vehicle speed may be determined).

FIG.2depicts an illustrative presentation200of remote parking of autonomous vehicles.

Referring toFIG.2, the real time representation174ofFIG.1is shown. Over time (e.g., multiple images including the image160ofFIG.1), the vehicle104ofFIG.1(e.g., the vehicle from whose point of view the real time representation174is being presented) may approach an object (e.g., the vehicle108ofFIG.1). To represent the speed at which the vehicle104approaches an object, and to account for the delay time between the human operator109ofFIG.1issuing a command to control operation of the vehicle104and the vehicle104performing an action corresponding to the command, the scale (e.g., size) with which the object is represented by the real time presentation174may be adjusted. For example, the faster the vehicle104approaches the vehicle108, the faster the scale of the pre-generated image of the vehicle176may be shown. InFIG.2, a real time representation274(e.g., representing a subsequent video frame after the real time representation174) may show a larger representation276of the vehicle108to indicate to the human operator109that the vehicle104is getting closer to the vehicle108. As shown inFIG.3, the scale of the pre-generated image of the vehicle176may be adjusted more quickly than the real time representation274ofFIG.2to show a larger representation of the vehicle108as the vehicle104gets closer to or more quickly approaches the vehicle108.

FIG.3depicts an illustrative presentation300of remote parking of autonomous vehicles.

Referring toFIG.3, the real time representation174ofFIG.1is shown. Over time (e.g., multiple images including the image160ofFIG.1), the vehicle104ofFIG.1(e.g., the vehicle from whose point of view the real time representation174is being presented) may approach an object (e.g., the vehicle108ofFIG.1). To represent the speed at which the vehicle104approaches an object, and to account for the delay time between the human operator109ofFIG.1issuing a command to control operation of the vehicle104and the vehicle104performing an action corresponding to the command, the scale (e.g., size) with which the object is represented by the real time presentation174may be adjusted. For example, the faster the vehicle104approaches the vehicle108, the faster the scale of the pre-generated image of the vehicle176may be shown. InFIG.3, a real time representation374(e.g., representing a subsequent video frame after the real time representation174) may show a larger representation376of the vehicle108to indicate to the human operator109that the vehicle104is getting closer to the vehicle108. As shown inFIG.3, the scale of the pre-generated image of the vehicle176may be adjusted more quickly than the real time representation274ofFIG.2to show a larger representation of the vehicle108as the vehicle104gets closer to or more quickly approaches the vehicle108. In this manner, the scale of the pre-generated image of the vehicle176may be based on the round trip delay time and/or the speed of the vehicle104.

FIG.4is a flowchart of an example method400for remote parking of autonomous vehicles.

At block402, a device (or system, e.g., the parking system100ofFIG.1, the one or more other devices152ofFIG.1, the device172ofFIG.1) may receive first data indicative of an autonomous vehicle in a parking area (e.g., the parking area102ofFIG.1). The first data may include image data (e.g., the image160ofFIG.1) and/or location data (e.g., a location of a camera that captured one or more images showing the autonomous vehicle, global positioning data provided by the autonomous vehicle, or the like). The first data may indicate that the autonomous vehicle is at a location in the parking area, and that other objects may be near the vehicle (e.g., based on known locations of objects at or near the autonomous vehicle's location and/or based on object detection using image analysis). When the first data include image data, the image data may be captured by cameras in the parking area, including vehicle cameras (e.g., the cameras103and132ofFIG.1).

At block404, the device may determine, based on the first data, a location of the autonomous vehicle. The first data may include one or more images, or may include location data for the one or more images (e.g., the location of the camera(s) that captured the one or more images). Based on the image data, the camera location, and/or vehicle location data (e.g., global positioning data), the device may determine the location of the vehicle within or near the parking area.

At block406, the device may determine, based on the location of the autonomous vehicle, first image data including a representation of an object. For example, given the location of the one or more images and the autonomous vehicle, the device may identify known objects at the location (e.g., based on a map or other location information). When the location data identifies that an object is detected (e.g., the vehicle108ofFIG.1), the device may use a pre-generated first image data of an object (e.g., the representation of the vehicle176ofFIG.1) when generating second image data.

At block408, the device may generate second image data based on the first data and the first image data. For example, using the first image data, the device may generate a representation (e.g., second image data for respective frames of video) from the perspective of the driver's seat of the autonomous vehicle to present to a human operator (e.g., the human operator109ofFIG.1). For example, the exact representation of the object may not be necessary for the human operator to see in order to operate the autonomous vehicle. Instead, the presence of an object near the autonomous vehicle may suffice, so a pre-generated image of an object may be used to generate the real time representation (e.g., the second image data). In addition, because the locations of the image and the location of one or more objects are known, the one or more objects may be shown as a pre-generated representation. In this manner, the first image data may not be present in the first data, but rather may represent pre-generated image assets that may be used to generate second image data that may not be exactly what is captured by any camera images of the parking area.

At block410, the device may present the second image data as a real time presentation of video (e.g., frames of video) such as is shown inFIG.1. The second image data may include a combination of image data captured by cameras of the parking area and the first image data, representing image data of generic or known objects near the autonomous vehicle and which a driver of the autonomous vehicle may see if sitting in the driver's seat of the autonomous vehicle. The scale of objects in any frame, or across multiple frames, may depend on the round trip time of data sent from the human operator to the autonomous vehicle, and/or based on the speed of the autonomous vehicle (e.g., to represent the rate at which the autonomous vehicle may approach another object). When multiple autonomous vehicles are in the parking area, the worst (e.g., longest) round trip time from among the autonomous vehicles may be used to generate the second image data (e.g., impacting the scale with which nearby objects are presented).

At block412, the device may receive an input associated with controlling operation of the autonomous vehicle. For example, the input may include a command to adjust speed, change direction, shift the vehicle into park, or the like. Because the input may be received from a remote operator, the second image data may allow the remote operator to control the autonomous vehicle as if driving the vehicle from the driver's seat. At block414, the device may control operation of the autonomous vehicle by sending the input to the autonomous vehicle. Accordingly, the second image data may be updated to show a real time representation of what a driver of the autonomous vehicle would see if sitting in the autonomous vehicle. For example, when the autonomous vehicle turns, the second image data may include representations of different objects in front of the autonomous vehicle during the turn, after the turn, etc.

The examples above are not meant to be limiting.

FIG.5is a block diagram illustrating an example of a computing device or computer system upon which any of one or more techniques (e.g., methods) may be performed, in accordance with one or more example embodiments of the present disclosure.

For example, the computing system500ofFIG.5may include or represent the parking system100ofFIG.1, the one or more other devices152ofFIG.1, the device172ofFIG.1. The computer system (system) includes one or more processors502-506. Processors502-506may include one or more internal levels of cache (not shown) and a bus controller (e.g., bus controller522) or bus interface (e.g., I/O interface520) unit to direct interaction with the processor bus512.

Processor bus512, also known as the host bus or the front side bus, may be used to couple the processors502-506and parking modules519(e.g., capable of performing the method400ofFIG.4) with the system interface524. System interface524may be connected to the processor bus512to interface other components of the system500with the processor bus512. For example, system interface524may include a memory controller518for interfacing a main memory516with the processor bus512. The main memory516typically includes one or more memory cards and a control circuit (not shown). System interface524may also include an input/output (I/O) interface520to interface one or more I/O bridges525or I/O devices530with the processor bus512. One or more I/O controllers and/or I/O devices may be connected with the I/O bus526, such as I/O controller528and I/O device530, as illustrated.

I/O device530may also include an input device (not shown), such as an alphanumeric input device, including alphanumeric and other keys for communicating information and/or command selections to the processors502-506, and/or the parking modules519. Another type of user input device includes cursor control, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to the processors502-506, and for controlling cursor movement on the display device.

System500may include a dynamic storage device, referred to as main memory516, or a random access memory (RAM) or other computer-readable devices coupled to the processor bus512for storing information and instructions to be executed by the processors502-506and/or the parking modules519. Main memory516also may be used for storing temporary variables or other intermediate information during execution of instructions by the processors502-506and/or the parking modules519. System500may include read-only memory (ROM) and/or other static storage device coupled to the processor bus512for storing static information and instructions for the processors502-506and/or the parking modules519. The system outlined inFIG.5is but one possible example of a computer system that may employ or be configured in accordance with aspects of the present disclosure.

According to one embodiment, the above techniques may be performed by computer system500in response to processor504executing one or more sequences of one or more instructions contained in main memory516. These instructions may be read into main memory516from another machine-readable medium, such as a storage device. Execution of the sequences of instructions contained in main memory516may cause processors502-506and/or the parking modules519to perform the process steps described herein. In alternative embodiments, circuitry may be used in place of or in combination with the software instructions. Thus, embodiments of the present disclosure may include both hardware and software components.

According to one embodiment, the processors502-506may represent machine learning models. For example, the processors502-506may allow for neural networking and/or other machine learning techniques used to operate the vehicle104ofFIG.1. For example, the processors502-506may include tensor processing units (TPUs) having artificial intelligence application-specific integrated circuits (ASICs), and may facilitate computer vision and other machine learning techniques for image analysis and generation.

In one or more embodiments, the computer system500may perform any of the steps of the processes described with respect toFIG.4.

A machine-readable medium includes any mechanism for storing or transmitting information in a form (e.g., software, processing application) readable by a machine (e.g., a computer). Such media may take the form of, but is not limited to, non-volatile media and volatile media and may include removable data storage media, non-removable data storage media, and/or external storage devices made available via a wired or wireless network architecture with such computer program products, including one or more database management products, web server products, application server products, and/or other additional software components. Examples of removable data storage media include Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc Read-Only Memory (DVD-ROM), magneto-optical disks, flash drives, and the like. Examples of non-removable data storage media include internal magnetic hard disks, solid state devices (SSDs), and the like. The one or more memory devices (not shown) may include volatile memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM), etc.) and/or non-volatile memory (e.g., read-only memory (ROM), flash memory, etc.).