Patent Description:
<CIT> discloses an autonomous vehicle which may include a stuck condition detection component and a communications component. The stuck-detection component may be configured to detect a condition in which the autonomous vehicle is impeded from navigating according to a first trajectory. The communications component may send an assistance signal to an assistance center and receive a response to the assistance signal. The assistance signal may include sensor information from the autonomous vehicle. The assistance center may include a communications component and a trajectory specification component. The communications component may receive the assistance signal and send a corresponding response. The trajectory specification component may specify a second trajectory for the autonomous vehicle and generate the corresponding response that includes a representation of the second trajectory. The second trajectory may be based on the first trajectory and ignore an object obstructing the first trajectory.

<CIT> discloses a method for controlling an at least partly automated vehicle, a vehicle computing system of the vehicle is used to generate an environmental model containing an entry for an object in an environment of the vehicle. The vehicle computing system is used to modify the entry for the object to generate a modified environmental model and to generate at least one proposed path for the vehicle based on the modified environmental model and transmit the at least one proposed path to a remote computing unit. The vehicle computing system is used to receive information concerning a selected path from the remote computing unit and to control the vehicle to drive along the selected path.

<CIT> discloses a method for autonomously operating a driverless vehicle along a path between a first geographic location and a destination which may include receiving communication signals from the driverless vehicle. The communication signals may include sensor data from the driverless vehicle and data indicating occurrence of an event associated with the path, and may also include data indicating that a confidence level associated with the path is less than a threshold confidence level due to the event. The method may also include determining, via a teleoperations system, a level of guidance to provide the driverless vehicle based on data associated with the communication signals, and transmitting teleoperations signals to the driverless vehicle. The teleoperations signals may include guidance to operate the driverless vehicle according to the determined level of guidance, so that a vehicle controller maneuvers the driverless vehicle to avoid, travel around, or pass through the event.

<CIT> discloses a system and method for remotely controlling the motion of a robot. In example implementations, the user is presented with a display having video from a camera on the robot augmented with abstract sensor data, and facilitates the user to draw motion path commands directly on the camera view. Analysis of the scene for obstacles is performed, so that the drawing of commands is interactive, preventing the user from drawing impossible paths. The path sketch is then transformed into the coordinate system of the robot and used to generate commands that will move the robot to the desired location. Sensor data is also used to augment the camera view, in particular for highlighting dangers and obstacles.

<CIT> discloses systems and methods to provide location specific assistance. A condition at a geographic location may be identified that is sensed in the environment of a vehicle and that a first autonomous vehicle control system is unable to, without location specific assistance, perceive, interpret and/or react to. A course at the geographic location that was previously determined by a second autonomous vehicle control system and/or followed by a person-driven vehicle when the condition was present at the geographic location may be found. The vehicle may be caused to follow the course previously determined by a second autonomous vehicle control system and/or followed by the person-driven vehicle.

Example embodiments described herein relate to user interface techniques for recommending remote assistance actions. Such techniques can help guide human operators when providing different forms of assistance to autonomous and semi-autonomous vehicles. Control of autonomous and semi-autonomous vehicles may therefore be improved.

The matter for protection is set out in the appended claims. In one aspect, an example method is provided. The method involves receiving, at a computing device, sensor data depicting a forward path for an autonomous vehicle. The computing device is positioned remotely from the autonomous vehicle. The method also involves displaying, by the computing device, a representation of the forward path based on the sensor data and augmenting the representation of the forward path to further depict one or more proposed trajectories available for the autonomous vehicle to perform. Each proposed trajectory conveys one or more maneuvers positioned relative to road boundaries in the forward path. The method involves receiving, at the computing device, a selection of a proposed trajectory from the one or more proposed trajectories available for the autonomous vehicle to perform, and providing, by the computing device and to the autonomous vehicle, navigation instructions based on the proposed trajectory.

In another aspect, an example system is provided. The system includes an autonomous vehicle and a computing device positioned at a remote location relative to the vehicle. The computing device is configured to receive sensor data depicting a forward path for an autonomous vehicle, display a representation of the forward path based on the sensor data, and augment the representation of the forward path to further depict one or more proposed trajectories available for the autonomous vehicle to perform. Each proposed trajectory conveys one or more maneuvers positioned relative to road boundaries in the forward path. The computing device is further configured to receive a selection of a proposed trajectory from the one or more proposed trajectories available for the autonomous vehicle to perform, and provide, to the autonomous vehicle, navigation instructions based on the proposed trajectory.

In yet another example, an example non-transitory computer readable medium having stored therein program instructions executable by a computing system comprising one or more processors to cause the computing system to perform operations is provided. The operations include receiving sensor data depicting a forward path for an autonomous vehicle, displaying a representation of the forward path based on the sensor data, and augmenting the representation of the forward path to further depict one or more proposed trajectories available for the autonomous vehicle to perform. Each proposed trajectory conveys one or more maneuvers positioned relative to road boundaries in the forward path. The operations also include receiving a selection of a proposed trajectory from the one or more proposed trajectories available for the autonomous vehicle to perform and providing, to the autonomous vehicle, navigation instructions based on the proposed trajectory.

In the following detailed description, reference is made to the accompanying figures, which form a part hereof. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, figures, and claims are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein.

As a vehicle navigates to a destination in an autonomous or semi-autonomous mode, the vehicle may encounter some situations that can interfere and potentially block its current trajectory. For instance, construction sites, poorly marked roadways, stranded vehicles, or obstructed views of signs or pathways can alter and sometimes limit a vehicle's ability to navigate its surrounding environment. In situations where vehicle systems fail to independently overcome obstacles encountered during navigation, remote assistance offers a solution that can help guide the vehicle through the situation. For instance, a vehicle may request assistance from a remote human operator, who can then provide route instructions that the vehicle can use to overcome an obstacle. In some cases, a remote operator can already be monitoring the navigation of a vehicle and provide assistance when the vehicle encounters a potential obstacle that requires assistance.

Example embodiments presented herein describe user interface techniques for recommending remote assistance actions, which can enable human operators to assist vehicles in a timely manner. By recommending remote assistance actions based on a vehicle's situation, a human operator may be guided by a computing device when trying to understand and provide assistance that can resolve the situation for the vehicle. For instance, the computing device used by a human operator can be configured to use vehicle context to identify and display a particular graphical user interface (GUI) template from available templates, limit and recommend specific selectable options, provide specific instructions, and/or perform other operations that can help accelerate the delivery of assistance to vehicles. Control of a vehicle may therefore be improved. As an example result, the human operator may be able to assist more vehicles by using the guidance provided based on the context of the vehicle's situation.

By way of an example, a computing device may receive sensor data depicting a forward path for an autonomous vehicle and subsequently display a representation of the forward path based on the sensor data. The sensor data can convey information about the surrounding environment, such as the position, heading, and velocities of nearby vehicles and other objects as well as boundary information. The computing device can display the representation of the vehicle's environment using the incoming sensor data received from the vehicle.

In some embodiments, the computing device is positioned remotely from the vehicle and used by a human operator to assist the vehicle. Through a wireless connection between the computing device and the vehicle, the human operator can monitor navigation of the vehicle to provide assistance when needed or can receive a request for assistance from the vehicle that initiates the communication session between the computing device and the vehicle. In other implementations, the computing device can be a personal computing device of a passenger, such as a smartphone or a wearable device.

The computing device can be configured to augment the representation of the vehicle's forward path to further depict one or more proposed trajectories available for the autonomous vehicle to perform. Each proposed trajectory may convey one or more maneuvers positioned relative to road boundaries in the forward path. Road boundaries can be based on the roads in the areas, nearby curbs, lane markers, traffic cones, painted markers, and/or other elements that serve to guide vehicles during navigation. The computing device may further receive a selection of a proposed trajectory from the one or more proposed trajectories available for the autonomous vehicle to perform and provide, by the computing device and to the autonomous vehicle, navigation instructions based on the proposed trajectory.

In some examples, the computing device initially receives a request for assistance from a vehicle operating in an autonomous mode or a semi-autonomous mode. The request for assistance may include location information and the sensor data from one or more vehicle sensors on the autonomous vehicle, such as cameras, radar, sonor, and/or lidar. For instance, the computing device may use a wireless communication connection to receive images (video) in near real-time from the vehicle, which can enable a human operator to view the vehicle's forward path and surrounding environment from the vehicle's perspective. In some examples, the computing device may convey a position of the vehicle relative to its surroundings from a bird eye's perspective based on sensor data obtained from the vehicle.

In addition, the computing device can receive predictive analytics from the vehicle, such as behavior predictions for objects near the vehicle. The behavior predictions can indicate trajectories that the vehicle systems predict that each object is likely to perform. In some cases, the behavior predictions can indicate a confidence level that an object will perform a trajectory, such as a percentage likelihood. For instance, a first behavior prediction assigned to a nearby vehicle may indicate that vehicle systems predict that the vehicle will continue its current navigation at the same velocity and heading with a first confidence level (e.g., a <NUM> percent likelihood) and a second behavior prediction assigned to another nearby vehicle may specify a maneuver that vehicle systems predict the vehicle to perform at a second confidence level (e.g., a <NUM> percent likelihood).

The computing device can augment the representation of the vehicle's forward path to further convey the predictive analytics received from the vehicle. This way, a human operator may be able to review, modify, suppress, and/or perform other actions related to the different behavior predictions associated with objects in the surrounding environment. For example, the computing device may display visual paths for objects based on behavior predictions associated with the objects. The computing device can also display confidence levels for the different behavior predictions associated with objects in the environment. As an example result, the computing device may receive modifications for one or multiple behavior predictions and update the proposed trajectories displayed via the GUI.

In some examples, the computing device can modify proposed trajectories for a vehicle to perform after receiving operator input that modified one or multiple behavior predictions for objects. The computing device can then display the modified proposed trajectories for the human operator to further review and select from.

In addition, user interface techniques may use colors and/or other visual cues to help convey contextual information to a human operator. For instance, the computing device can depict proposed trajectories available for the vehicle to perform in different colors according to a color scale. The color scale can represent differences in confidence levels associated with the proposed trajectories.

In some examples, a computing device can recommend a navigation strategy for review by a remote operator. The navigation strategy can be based on the geographic location and environment of the vehicle. For instance, the computing device may use past successful trajectories based on other vehicles navigation in the location to recommend a navigation strategy for the human operator to approve, reject, and/or modify. The computing device may use prior successes by the vehicle in that location, other vehicles, and/or vehicles driven by human drivers.

In some examples, the computing device may use a map comparison technique to detect obstacles. The obstacle detection can be used to recommend a navigation strategy for review by a human operator. The navigation strategy can be used by a vehicle operating in an autonomous or semi-autonomous mode. The map comparison may involve comparing an expected map of the location of a vehicle and sensor data depicting objects at the location from the perspective of the vehicle. The results of the map comparison can enable the computing device to identify differences, which may include objects or other obstacles that are causing an issue for the vehicle that is requesting assistance. As such, the computing device may recommend a trajectory or trajectories for review by a human operator. The human operator can select a trajectory to provide assistance to the vehicle.

Example techniques can allow for a remote operator to provide assistance to a vehicle with less latency, which can allow the vehicle to receive and execute operations based on the assistance before the vehicle even comes to a stop in some instances. In addition, these techniques can be useful for autonomous trucking and/or in specific situations, such as marking waypoints that adhere to different lane layouts that can arise within construction zones or other dynamic environments. In some embodiments, remote assistance may involve establishing a secure communication connection between a human operator and one or more vehicle systems or passengers traveling within a vehicle. The human operator may receive sensor data depicting the environment in near real-time and provide assistance to the vehicle (or passengers) immediately.

Example systems within the scope of the present disclosure will now be described in greater detail. An example system may be implemented in or may take the form of an automobile, but other example systems can be implemented in or take the form of other vehicles, such as cars, trucks, motorcycles, buses, boats, airplanes, helicopters, lawn mowers, earth movers, boats, snowmobiles, aircraft, recreational vehicles, amusement park vehicles, farm equipment, construction equipment, trams, golf carts, trains, trolleys, and robot devices. Other vehicles are possible as well.

Referring now to the figures, <FIG> is a functional block diagram illustrating vehicle <NUM>, which represents a vehicle capable of operating fully or partially in an autonomous mode. More specifically, vehicle <NUM> may operate in an autonomous mode without human interaction (or reduced human interaction) through receiving control instructions from a computing system (e.g., a vehicle control system). As part of operating in the autonomous mode, vehicle <NUM> may use sensors (e.g., sensor system <NUM>) to detect and possibly identify objects of the surrounding environment to enable safe navigation. In some implementations, vehicle <NUM> may also include subsystems that enable a driver (or a remote operator) to control operations of vehicle <NUM>.

As shown in <FIG>, vehicle <NUM> includes various subsystems, such as propulsion system <NUM>, sensor system <NUM>, control system <NUM>, one or more peripherals <NUM>, power supply <NUM>, computer system <NUM>, data storage <NUM>, and user interface <NUM>. The subsystems and components of vehicle <NUM> may be interconnected in various ways (e.g., wired or secure wireless connections). In other examples, vehicle <NUM> may include more or fewer subsystems. In addition, the functions of vehicle <NUM> described herein can be divided into additional functional or physical components, or combined into fewer functional or physical components within implementations.

Propulsion system <NUM> may include one or more components operable to provide powered motion for vehicle <NUM> and can include an engine/motor <NUM>, an energy source <NUM>, a transmission <NUM>, and wheels/tires <NUM>, among other possible components. For example, engine/motor <NUM> may be configured to convert energy source <NUM> into mechanical energy and can correspond to one or a combination of an internal combustion engine, one or more electric motors, steam engine, or Stirling engine, among other possible options. For instance, in some implementations, propulsion system <NUM> may include multiple types of engines and/or motors, such as a gasoline engine and an electric motor.

Energy source <NUM> represents a source of energy that may, in full or in part, power one or more systems of vehicle <NUM> (e.g., engine/motor <NUM>). For instance, energy source <NUM> can correspond to gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and/or other sources of electrical power. In some implementations, energy source <NUM> may include a combination of fuel tanks, batteries, capacitors, and/or flywheel.

Transmission <NUM> may transmit mechanical power from the engine/motor <NUM> to wheels/tires <NUM> and/or other possible systems of vehicle <NUM>. As such, transmission <NUM> may include a gearbox, a clutch, a differential, and a drive shaft, among other possible components. A drive shaft may include axles that connect to one or more wheels/tires <NUM>.

Wheels/tires <NUM> of vehicle <NUM> may have various configurations within example implementations. For instance, vehicle <NUM> may exist in a unicycle, bicycle/motorcycle, tricycle, or car/truck four-wheel format, among other possible configurations. As such, wheels/tires <NUM> may connect to vehicle <NUM> in various ways and can exist in different materials, such as metal and rubber.

Sensor system <NUM> can include various types of sensors, such as Global Positioning System (GPS) <NUM>, inertial measurement unit (IMU) <NUM>, one or more radar units <NUM>, laser rangefinder / LIDAR unit <NUM>, camera <NUM>, steering sensor <NUM>, and throttle/brake sensor <NUM>, among other possible sensors. In some implementations, sensor system <NUM> may also include sensors configured to monitor internal systems of the vehicle <NUM> (e.g., O<NUM> monitors, fuel gauge, engine oil temperature, condition of brakes).

GPS <NUM> may include a transceiver operable to provide information regarding the position of vehicle <NUM> with respect to the Earth. IMU <NUM> may have a configuration that uses one or more accelerometers and/or gyroscopes and may sense position and orientation changes of vehicle <NUM> based on inertial acceleration. For example, IMU <NUM> may detect a pitch and yaw of the vehicle <NUM> while vehicle <NUM> is stationary or in motion.

Radar unit <NUM> may represent one or more systems configured to use radio signals to sense objects (e.g., radar signals), including the speed and heading of the objects, within the local environment of vehicle <NUM>. As such, radar unit <NUM> may include one or more radar units equipped with one or more antennas configured to transmit and receive radar signals as discussed above. In some implementations, radar unit <NUM> may correspond to a mountable radar system configured to obtain measurements of the surrounding environment of vehicle <NUM>. For example, radar unit <NUM> can include one or more radar units configured to couple to the underbody of a vehicle.

Laser rangefinder / LIDAR <NUM> may include one or more laser sources, a laser scanner, and one or more detectors, among other system components, and may operate in a coherent mode (e.g., using heterodyne detection) or in an incoherent detection mode. Camera <NUM> may include one or more devices (e.g., still camera or video camera) configured to capture images of the environment of vehicle <NUM>.

Steering sensor <NUM> may sense a steering angle of vehicle <NUM>, which may involve measuring an angle of the steering wheel or measuring an electrical signal representative of the angle of the steering wheel. In some implementations, steering sensor <NUM> may measure an angle of the wheels of the vehicle <NUM>, such as detecting an angle of the wheels with respect to a forward axis of the vehicle <NUM>. Steering sensor <NUM> may also be configured to measure a combination (or a subset) of the angle of the steering wheel, electrical signal representing the angle of the steering wheel, and the angle of the wheels of vehicle <NUM>.

Throttle/brake sensor <NUM> may detect the position of either the throttle position or brake position of vehicle <NUM>. For instance, throttle/brake sensor <NUM> may measure the angle of both the gas pedal (throttle) and brake pedal or may measure an electrical signal that could represent, for instance, the angle of the gas pedal (throttle) and/or an angle of a brake pedal. Throttle/brake sensor <NUM> may also measure an angle of a throttle body of vehicle <NUM>, which may include part of the physical mechanism that provides modulation of energy source <NUM> to engine/motor <NUM> (e.g., a butterfly valve or carburetor). Additionally, throttle/brake sensor <NUM> may measure a pressure of one or more brake pads on a rotor of vehicle <NUM> or a combination (or a subset) of the angle of the gas pedal (throttle) and brake pedal, electrical signal representing the angle of the gas pedal (throttle) and brake pedal, the angle of the throttle body, and the pressure that at least one brake pad is applying to a rotor of vehicle <NUM>. In other embodiments, throttle/brake sensor <NUM> may be configured to measure a pressure applied to a pedal of the vehicle, such as a throttle or brake pedal.

Control system <NUM> may include components configured to assist in enabling navigation by vehicle <NUM>, such as steering unit <NUM>, throttle <NUM>, brake unit <NUM>, sensor fusion algorithm <NUM>, computer vision system <NUM>, navigation / pathing system <NUM>, and obstacle avoidance system <NUM>. More specifically, steering unit <NUM> may be operable to adjust the heading of vehicle <NUM>, and throttle <NUM> may control the operating speed of engine/motor <NUM> to control the acceleration of vehicle <NUM>. Brake unit <NUM> may decelerate vehicle <NUM>, which may involve using friction to decelerate wheels/tires <NUM>. In some implementations, brake unit <NUM> may convert kinetic energy of wheels/tires <NUM> to electric current for subsequent use by a system or systems of vehicle <NUM>.

Sensor fusion algorithm <NUM> may include a Kalman filter, Bayesian network, or other algorithms that can process data from sensor system <NUM>. In some implementations, sensor fusion algorithm <NUM> may provide assessments based on incoming sensor data, such as evaluations of individual objects and/or features, evaluations of a particular situation, and/or evaluations of potential impacts within a given situation.

Computer vision system <NUM> may include hardware and software operable to process and analyze images in an effort to determine objects, environmental objects (e.g., stop lights, road way boundaries, etc.), and obstacles. As such, computer vision system <NUM> may use object recognition, Structure from Motion (SFM), video tracking, and other algorithms used in computer vision, for instance, to recognize objects, map an environment, track objects, estimate the speed of objects, etc..

Navigation / pathing system <NUM> may determine a driving path for vehicle <NUM>, which may involve dynamically adjusting navigation during operation. As such, navigation / pathing system <NUM> may use data from sensor fusion algorithm <NUM>, GPS <NUM>, and maps, among other sources to navigate vehicle <NUM>. Obstacle avoidance system <NUM> may evaluate potential obstacles based on sensor data and cause systems of vehicle <NUM> to avoid or otherwise negotiate the potential obstacles.

As shown in <FIG>, vehicle <NUM> may also include peripherals <NUM>, such as wireless communication system <NUM>, touchscreen <NUM>, microphone <NUM>, and/or speaker <NUM>. Peripherals <NUM> may provide controls or other elements for a user to interact with user interface <NUM>. For example, touchscreen <NUM> may provide information to users of vehicle <NUM>. User interface <NUM> may also accept input from the user via touchscreen <NUM>. Peripherals <NUM> may also enable vehicle <NUM> to communicate with devices, such as other vehicle devices.

Wireless communication system <NUM> may securely and wirelessly communicate with one or more devices directly or via a communication network. For example, wireless communication system <NUM> could use <NUM> cellular communication, such as CDMA, EVDO, GSM/GPRS, or <NUM> cellular communication, such as WiMAX or LTE. Alternatively, wireless communication system <NUM> may communicate with a wireless local area network (WLAN) using WiFi or other possible connections. Wireless communication system <NUM> may also communicate directly with a device using an infrared link, Bluetooth, or ZigBee, for example. Other wireless protocols, such as various vehicular communication systems, are possible within the context of the disclosure. For example, wireless communication system <NUM> may include one or more dedicated short-range communications (DSRC) devices that could include public and/or private data communications between vehicles and/or roadside stations.

Vehicle <NUM> may include power supply <NUM> for powering components. Power supply <NUM> may include a rechargeable lithium-ion or lead-acid battery in some implementations. For instance, power supply <NUM> may include one or more batteries configured to provide electrical power. Vehicle <NUM> may also use other types of power supplies. In an example implementation, power supply <NUM> and energy source <NUM> may be integrated into a single energy source.

Vehicle <NUM> may also include computer system <NUM> to perform operations, such as operations described therein. As such, computer system <NUM> may include at least one processor <NUM> (which could include at least one microprocessor) operable to execute instructions <NUM> stored in a non-transitory computer readable medium, such as data storage <NUM>. In some implementations, computer system <NUM> may represent a plurality of computing devices that may serve to control individual components or subsystems of vehicle <NUM> in a distributed fashion.

In some implementations, data storage <NUM> may contain instructions <NUM> (e.g., program logic) executable by processor <NUM> to execute various functions of vehicle <NUM>, including those described above in connection with <FIG>. Data storage <NUM> may contain additional instructions as well, including instructions to transmit data to, receive data from, interact with, and/or control one or more of propulsion system <NUM>, sensor system <NUM>, control system <NUM>, and peripherals <NUM>.

In addition to instructions <NUM>, data storage <NUM> may store data such as roadway maps, path information, among other information. Such information may be used by vehicle <NUM> and computer system <NUM> during the operation of vehicle <NUM> in the autonomous, semi-autonomous, and/or manual modes.

Vehicle <NUM> may include user interface <NUM> for providing information to or receiving input from a user of vehicle <NUM>. User interface <NUM> may control or enable control of content and/or the layout of interactive images that could be displayed on touchscreen <NUM>. Further, user interface <NUM> could include one or more input/output devices within the set of peripherals <NUM>, such as wireless communication system <NUM>, touchscreen <NUM>, microphone <NUM>, and speaker <NUM>.

Computer system <NUM> may control the function of vehicle <NUM> based on inputs received from various subsystems (e.g., propulsion system <NUM>, sensor system <NUM>, and control system <NUM>), as well as from user interface <NUM>. For example, computer system <NUM> may utilize input from sensor system <NUM> in order to estimate the output produced by propulsion system <NUM> and control system <NUM>. Depending upon the embodiment, computer system <NUM> could be operable to monitor many aspects of vehicle <NUM> and its subsystems. In some embodiments, computer system <NUM> may disable some or all functions of the vehicle <NUM> based on signals received from sensor system <NUM>.

The components of vehicle <NUM> could be configured to work in an interconnected fashion with other components within or outside their respective systems. For instance, in an example embodiment, camera <NUM> could capture a plurality of images that could represent information about a state of an environment of vehicle <NUM> operating in an autonomous mode. The state of the environment could include parameters of the road on which the vehicle is operating. For example, computer vision system <NUM> may be able to recognize the slope (grade) or other features based on the plurality of images of a roadway. Additionally, the combination of GPS <NUM> and the features recognized by computer vision system <NUM> may be used with map data stored in data storage <NUM> to determine specific road parameters. Further, radar unit <NUM> may also provide information about the surroundings of the vehicle.

In other words, a combination of various sensors (which could be termed input-indication and output-indication sensors) and computer system <NUM> could interact to provide an indication of an input provided to control a vehicle or an indication of the surroundings of a vehicle.

In some embodiments, computer system <NUM> may make a determination about various objects based on data that is provided by systems other than the radio system. For example, vehicle <NUM> may have lasers or other optical sensors configured to sense objects in a field of view of the vehicle. Computer system <NUM> may use the outputs from the various sensors to determine information about objects in a field of view of the vehicle, and may determine distance and direction information to the various objects. Computer system <NUM> may also determine whether objects are desirable or undesirable based on the outputs from the various sensors. In addition, vehicle <NUM> may also include telematics control unit (TCU) <NUM>. TCU <NUM> may enable vehicle connectivity and internal passenger device connectivity through one or more wireless technologies.

Although <FIG> shows various components of vehicle <NUM>, i.e., wireless communication system <NUM>, computer system <NUM>, data storage <NUM>, and user interface <NUM>, as being integrated into the vehicle <NUM>, one or more of these components could be mounted or associated separately from vehicle <NUM>. For example, data storage <NUM> could, in part or in full, exist separate from vehicle <NUM>. Thus, vehicle <NUM> could be provided in the form of device elements that may be located separately or together. The device elements that make up vehicle <NUM> could be communicatively coupled together in a wired and/or wireless fashion.

<FIG>, <FIG>, and <FIG> illustrate different views of a physical configuration of vehicle <NUM>. The various views are included to depict example sensor positions <NUM>, <NUM>, <NUM>, <NUM>, <NUM> on vehicle <NUM>. In other examples, sensors can have different positions on vehicle <NUM>. Although vehicle <NUM> is depicted in <FIG> as a van, vehicle <NUM> can have other configurations within examples, such as a truck, a car, a semi-trailer truck, a motorcycle, a bus, a shuttle, a golf cart, an off-road vehicle, robotic device, or a farm vehicle, among other possible examples.

As discussed above, vehicle <NUM> may include sensors coupled at various exterior locations, such as sensor positions <NUM>-<NUM>. Vehicle sensors include one or more types of sensors with each sensor configured to capture information from the surrounding environment or perform other operations (e.g., communication links, obtain overall positioning information). For example, sensor positions <NUM>-<NUM> may serve as locations for any combination of one or more cameras, radars, LIDARs, range finders, radio devices (e.g., Bluetooth and/or <NUM>), and acoustic sensors, among other possible types of sensors.

When coupled at the example sensor positions <NUM>-<NUM> shown in <FIG>, various mechanical fasteners may be used, including permanent or non-permanent fasteners. For example, bolts, screws, clips, latches, rivets, anchors, and other types of fasteners may be used. In some examples, sensors may be coupled to the vehicle using adhesives. In further examples, sensors may be designed and built as part of the vehicle components (e.g., parts of the vehicle mirrors).

In some implementations, one or more sensors may be positioned at sensor positions <NUM>-<NUM> using movable mounts operable to adjust the orientation of one or more sensors. A movable mount may include a rotating platform that can rotate sensors so as to obtain information from multiple directions around vehicle <NUM>. For instance, a sensor located at sensor position <NUM> may use a movable mount that enables rotation and scanning within a particular range of angles and/or azimuths. As such, vehicle <NUM> may include mechanical structures that enable one or more sensors to be mounted on top the roof of vehicle <NUM>. Additionally, other mounting locations are possible within examples. In some situations, sensors coupled at these locations can provide data that can be used by a remote operator to provide assistance to vehicle <NUM>.

<FIG> is a simplified block diagram exemplifying computing device <NUM>, illustrating some of the components that could be included in a computing device arranged to operate in accordance with the embodiments herein. Computing device <NUM> could be a client device (e.g., a device actively operated by a user (e.g., a remote operator)), a server device (e.g., a device that provides computational services to client devices), or some other type of computational platform. In some embodiments, computing device <NUM> may be implemented as computer system <NUM>, which can be located on vehicle <NUM> and perform processing operations related to vehicle operations. For example, computing device <NUM> can be used to process sensor data received from sensor system <NUM>. Alternatively, computing device <NUM> can be located remotely from vehicle <NUM> and communicate via secure wireless communication. For example, computing device <NUM> may operate as a remotely positioned device that a remote human operator can use to communicate with one or more vehicles.

In the example embodiment shown in <FIG>, computing device <NUM> includes processing system <NUM>, memory <NUM>, input / output unit <NUM> and network interface <NUM>, all of which may be coupled by a system bus <NUM> or a similar mechanism. In some embodiments, computing device <NUM> may include other components and/or peripheral devices (e.g., detachable storage, sensors, and so on).

Processing system <NUM> may be one or more of any type of computer processing element, such as a central processing unit (CPU), a co-processor (e.g., a mathematics, graphics, or encryption co-processor), a digital signal processor (DSP), a network processor, and/or a form of integrated circuit or controller that performs processor operations. In some cases, processing system <NUM> may be one or more single-core processors. In other cases, processing system <NUM> may be one or more multi-core processors with multiple independent processing units. Processing system <NUM> may also include register memory for temporarily storing instructions being executed and related data, as well as cache memory for temporarily storing recently-used instructions and data.

Memory <NUM> may be any form of computer-usable memory, including but not limited to random access memory (RAM), read-only memory (ROM), and non-volatile memory. This may include flash memory, hard disk drives, solid state drives, rewritable compact discs (CDs), rewritable digital video discs (DVDs), and/or tape storage, as just a few examples.

Computing device <NUM> may include fixed memory as well as one or more removable memory units, the latter including but not limited to various types of secure digital (SD) cards. Thus, memory <NUM> can represent both main memory units, as well as long-term storage. Other types of memory may include biological memory.

Memory <NUM> may store program instructions and/or data on which program instructions may operate. By way of example, memory <NUM> may store these program instructions on a non-transitory, computer-readable medium, such that the instructions are executable by processing system <NUM> to carry out any of the methods, processes, or operations disclosed in this specification or the accompanying drawings.

As shown in <FIG>, memory <NUM> may include firmware 314A, kernel 314B, and/or applications 314C. Firmware 314A may be program code used to boot or otherwise initiate some or all of computing device <NUM>. Kernel 314B may be an operating system, including modules for memory management, scheduling and management of processes, input / output, and communication. Kernel 314B may also include device drivers that allow the operating system to communicate with the hardware modules (e.g., memory units, networking interfaces, ports, and busses), of computing device <NUM>. Applications 314C may be one or more user-space software programs, such as web browsers or email clients, as well as any software libraries used by these programs. In some examples, applications 314C may include one or more neural network applications and other deep learning-based applications. Memory <NUM> may also store data used by these and other programs and applications.

Input / output unit <NUM> may facilitate user and peripheral device interaction with computing device <NUM> and/or other computing systems. Input / output unit <NUM> may include one or more types of input devices, such as a keyboard, a mouse, one or more touch screens, sensors, biometric sensors, and so on. Similarly, input / output unit <NUM> may include one or more types of output devices, such as a screen, monitor, printer, speakers, and/or one or more light emitting diodes (LEDs). Additionally or alternatively, computing device <NUM> may communicate with other devices using a universal serial bus (USB) or high-definition multimedia interface (HDMI) port interface, for example. In some examples, input / output unit <NUM> can be configured to receive data from other devices. For instance, input / output unit <NUM> may receive sensor data from vehicle sensors.

As shown in <FIG>, input / output unit <NUM> includes GUI <NUM>, which can be configured to provide information to a remote operator or another user. GUI <NUM> may involve one or more display interfaces, or another type of mechanism for conveying information and receiving inputs. In some examples, the representation of GUI <NUM> may differ depending on a vehicle situation. For example, computing device <NUM> may provide GUI <NUM> in a particular format, such as a format with a single selectable option for a remote operator to select from.

Network interface <NUM> may take the form of one or more wireline interfaces, such as Ethernet (e.g., Fast Ethernet, Gigabit Ethernet, and so on). Network interface <NUM> may also support communication over one or more non-Ethernet media, such as coaxial cables or power lines, or over wide-area media, such as Synchronous Optical Networking (SONET) or digital subscriber line (DSL) technologies. Network interface <NUM> may additionally take the form of one or more wireless interfaces, such as IEEE <NUM> (Wifi), BLUETOOTH®, global positioning system (GPS), or a wide-area wireless interface. However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols may be used over network interface <NUM>. Furthermore, network interface <NUM> may comprise multiple physical interfaces. For instance, some embodiments of computing device <NUM> may include Ethernet, BLUETOOTH®, and Wifi interfaces. In some embodiments, network interface <NUM> may enable computing device <NUM> to connect with one or more vehicles to allow for remote assistance techniques presented herein.

In some embodiments, one or more instances of computing device <NUM> may be deployed to support a clustered architecture. The exact physical location, connectivity, and configuration of these computing devices may be unknown and/or unimportant to client devices. Accordingly, the computing devices may be referred to as "cloud-based" devices that may be housed at various remote data center locations. In addition, computing device <NUM> may enable the performance of embodiments described herein, including efficient assignment and processing of sensor data.

<FIG> is a system for wireless communication between computing devices and a vehicle, according to one or more example embodiments. System <NUM> may enable vehicles (e.g., vehicle <NUM>) to obtain remote assistance from human operators using computing devices positioned remotely from the vehicles (e.g., remote computing device <NUM>). Particularly, system <NUM> is shown with vehicle <NUM>, remote computing device <NUM>, and server <NUM> communicating wirelessly via network <NUM>. System <NUM> may include other components not shown within other embodiments, such as firewalls and multiple networks, among others.

Vehicle <NUM> may transport passengers or objects between locations, and may take the form of any one or more of the vehicles discussed above, including passenger vehicles, cargo shipping vehicles, farming and manufacturing vehicles, and dual-purpose vehicles. When operating in an autonomous mode (or semi-autonomous mode), vehicle <NUM> may navigate to pick up and drop off passengers (or cargo) between desired destinations. In some embodiments, vehicle <NUM> can operate as part of a fleet of vehicles, such as within a fleet of ride-share vehicles.

Remote computing device <NUM> may represent any type of device related to enabling remote assistance techniques, including but not limited to those described herein. Within examples, remote computing device <NUM> may represent any type of device configured to (i) receive information related to vehicle <NUM>, (ii) provide an interface (e.g., a GUI, physical input interfaces) through which a human operator can in turn perceive the information and input a response related to the information, and (iii) transmit the response to vehicle <NUM> or to other devices (e.g., storage at server <NUM>). As such, remote computing device <NUM> may take various forms, such as a workstation, a desktop computer, a laptop, a tablet, a mobile phone (e.g., a smart phone), a wearable device (e.g., a headset) and/or a server. In some examples, remote computing device <NUM> may include multiple computing devices operating together in a network configuration. In further embodiments, remote computing device <NUM> may resemble a vehicle simulation center with the remote operator positioned as the drive of the simulation center. In addition, remote computing device <NUM> may operate as a head mountable device that can simulate the perspective of vehicle <NUM>.

The position of remote computing device <NUM> relative to vehicle <NUM> can vary within examples. For instance, remote computing device <NUM> may have a remote position from vehicle <NUM>, such as operating inside a physical building. In another example, remote computing device <NUM> may be physically separate from vehicle <NUM>, but operate inside vehicle <NUM> to enable a passenger of vehicle <NUM> to act as the human operator. For instance, remote computing device <NUM> can be a touchscreen device accessible to a passenger of vehicle <NUM>. Operations described herein that are performed by remote computing device <NUM> may be additionally or alternatively performed by vehicle <NUM> (i.e., by any system(s) or subsystem(s) of vehicle <NUM>). In other words, vehicle <NUM> may be configured to provide a remote assistance mechanism with which a driver or passenger of the vehicle can interact.

Operations described herein can be performed by any of the components communicating via network <NUM>. For instance, remote computing device <NUM> may determine remote assist options for a human operator to review based on different levels of information provided by vehicle <NUM>. In some embodiments, vehicle <NUM> may determine potential navigation options for remote computing device <NUM> to display for a remote operator to review. Potential options could include routes, vehicle movements, and other navigation parameters for review by remote computing device <NUM> and/or a remote operator using remote computing device <NUM>.

In other embodiments, remote computing device <NUM> may analyze sensor data or other information from vehicle <NUM> to determine the situation and potential options for a remote operator to review. For instance, remote computing device <NUM> may determine a route and/or operations for vehicle <NUM> to execute using information from vehicle <NUM> and/or other external sources (e.g., server <NUM>). In some embodiments, remote computing device <NUM> may generate a GUI to display one or more selectable options for review by a remote operator.

Server <NUM> may be configured to wirelessly communicate with remote computing device <NUM> and vehicle <NUM> via network <NUM> (or perhaps directly with remote computing device <NUM> and/or vehicle <NUM>). As such, server <NUM> may represent any computing device configured to receive, store, determine, and/or send information relating to vehicle <NUM> and the remote assistance thereof. As such, server <NUM> may be configured to perform any operation(s), or portions of such operation(s), that is/are described herein as performed by remote computing system <NUM> and/or vehicle <NUM>. Some implementations of wireless communication related to remote assistance may utilize server <NUM>, while others may not.

Network <NUM> represents infrastructure that can enable wireless communication between computing devices, such as vehicle <NUM>, remote computing device <NUM>, and server <NUM>. For example, network <NUM> can correspond to a wireless communication network, such as the Internet or a cellular wireless communication network.

In some embodiments, remote assistance for vehicles can originate from a network of remote operators. For example, a vehicle may submit a request for assistance that is received at an entry point of the network. The entry point may connect the request with a remote operator that can provide assistance. The remote operator may be selected based on credentials associated with the remote operator that indicate that she or he is able to handle the type of assistance that is being requested and/or the operator's availability, among other potential parameters. The entry point may analyze information within the request to route requests for assistance accordingly. For example, the network of remote operators may be used to provide assistance to an entire fleet of autonomous vehicles.

<FIG> illustrates computing device <NUM> displaying GUI <NUM> for enabling delivery of remote assistance to a vehicle, according to one or more example embodiments. Computing device <NUM> can be implemented as computing device <NUM> shown in <FIG> and may enable wireless communication between human operators and vehicles.

In some cases, computing device <NUM> may receive a request for assistance from a vehicle that encountered a difficult situation and subsequently alert a human operator to provide assistance via various alert techniques, such as visual, audio, and/or tactile alerts. In other cases, a vehicle and computing device <NUM> may establish a wireless communication connection prior to the vehicle requiring assistance to resolve a situation. For instance, an operator may be tasked with monitoring vehicles through a particular situation (e.g., a construction site) and computing device <NUM> may automatically establish a connection with a vehicle in advance after detecting that the vehicle is approaching the situation. In some examples, computing device <NUM> may be used to enable a human operator to monitor a fleet of vehicles. For instance, a central system may route requests for assistance to available operators. In another example, the routing system can be a decentralized system that is supported via various nodes, such as computing device <NUM>.

In addition, computing device <NUM> may perform remote assist techniques in some examples, which can involve providing assistance to overcome various situations. For instance, computing device <NUM> may represent a powerful computing system that can perform simulations to check potential outcomes based on navigation information provided by a vehicle. As an example, a vehicle may provide video data and one or multiple proposed trajectories for the vehicle to perform, which can be analyzed and/or simulated by computing device <NUM> to generate an output. The output may indicate which trajectory the vehicle should perform in some instances. The computing device <NUM> may also notify a human operator when the output indicates that none of the proposed trajectories satisfy a success threshold. In some examples, computing device <NUM> can also generate proposed trajectories for a vehicle to perform based on sensor data representing the environment. For instance, computing device <NUM> can use sensor data from the vehicle to simulate different maneuvers until a particular trajectory satisfies a success threshold. In some cases, computing device <NUM> may submit a request for a human operator to review and approve the generated trajectory prior to sending the trajectory to the vehicle for subsequent performance.

In the example embodiment, computing device <NUM> is shown displaying GUI <NUM>, which includes environment representation <NUM>, road map <NUM>, and contextual information <NUM>. These visual elements are shown for illustration purposes and can be combined, further divided, replaced, and/or supplemented by other elements in other examples. For instance, GUI <NUM> may display only road map <NUM> in some implementations. In addition, the arrangement of the elements is for illustration purposes and can vary within implementations.

GUI <NUM> represents a system of interactive visual components for computer software, which can be used to display objects that convey information to a human operator and also represent actions that can be taken by the operator. For instance, computing device <NUM> may generate GUI <NUM> based on templates stored in memory and customized to a vehicle's given situation, which can enable an available remote operator to review and provide assistance. In some cases, GUI <NUM> can allow the remote operator to provide remote assistance that can be used to generate augmented route instructions that navigate vehicles with respect to an encountered obstacle or series of obstacles (e.g., a construction site). Computing device <NUM> may display GUI <NUM> on a display interface, such as a touch screen, external monitor, and/or a display interface associated with a head-mounted wearable computing device (e.g., augmented reality).

Computing device <NUM> may use GUI <NUM> to enable interaction between a human operator and vehicles. For instance, the human operator may provide inputs to computing device <NUM> via touch inputs, buttons or hardware inputs, motion, and/or vocal inputs. In some embodiments, computing device <NUM> may include a microphone that can receive vocal inputs and use speech recognition software to derive operations based on the vocal inputs from the operator. In addition, in some implementations, computing device <NUM> may resemble a vehicle emulator that can simulate the vehicle's perspective. The various elements (e.g., environment representation <NUM>, road map <NUM>, and contextual information <NUM>) shown in GUI <NUM> can be customized according to different settings enabled by computing device <NUM>.

Environment representation <NUM> is an object displayable via GUI <NUM> that can represent the current environment (or a recent environment) from one or more perspectives, such as the perspective of the vehicle or a simulated bird's-eye view. For instance, environment representation <NUM> may be images and/or videos captured by vehicle cameras. In other instances, sensor data from different types of sensors can be used to generate environment representation <NUM> displayed via GUI <NUM>. For instance, environment representation <NUM> may include data based on a point cloud developed using radar and/or LIDAR. As such, environment representation <NUM> can be updated in near-real time as the wireless communication between computing device <NUM> and a vehicle enables more information to be received and displayed.

In some cases, environment representation <NUM> can show the positions of obstacles or other environment elements in the vehicle's surrounding environment. For example, environment representation <NUM> may depict other vehicles, pedestrians, bicycles, traffic signals and signs, road barriers, buildings, and/or other features within the vehicle's environment. Computing device <NUM> may use visual indicators (e.g., arrows, boxes, and colors) to highlight aspects in environment representation <NUM>, such as the obstacles blocking the path of travel of the vehicle. For example, computing device <NUM> may use computer vision to detect and identify objects in the environment.

Computing device <NUM> may also display road map <NUM> via GUI <NUM> based on a location of the vehicle. Road map <NUM> may represent one or more maps of roads that depend on the current location and route of the vehicle. For instance, the vehicle may provide GPS measurements or another indication of the vehicle's location within the request for assistance or during subsequent communication between the vehicle and computing device <NUM>. By using the vehicle's location, computing device <NUM> can acquire road map <NUM> and further enhance the information included within environment representation <NUM> and/or other objects displayed via GUI <NUM>. For instance, road map <NUM> can be augmented to display obstacles detected in vehicle sensor data from the assistance requesting vehicle and/or other vehicles that captured measurements of that area. In some examples, computing device <NUM> can determine and display environment representation <NUM> as an elevated view of the vehicle and nearby surroundings estimated based on road map <NUM> and the sensor data from the vehicle. In some examples, GUI <NUM> may include both a sensor perspective of the vehicle's environment and the elevated view estimated based on one or both of the sensor data and/or road map <NUM>.

GUI <NUM> also includes contextual information <NUM>, which may convey additional information about the vehicle's situation. As shown in <FIG>, contextual information <NUM> includes vehicle information <NUM>, location information <NUM>, proposed trajectories <NUM>, and behavior predictions <NUM>. Vehicle information <NUM> may indicate a variety of information about the vehicle, such as the type of vehicle, the vehicle sensors on the vehicle, the quantity of the passengers, and target destination, etc. Location information <NUM> may represent information based on the current location of the vehicle, such as map data depicting the environment and lanes available for the vehicle to use. Contextual information <NUM> may also specify information related to the situation, such as how long has the vehicle been stranded and a reason proposed by the vehicle for the stranding.

Proposed trajectories <NUM> represent different navigation trajectories that vehicle systems may potentially cause a vehicle to perform and may originate from different sources within examples. For instance, computing device <NUM> may receive one or multiple proposed trajectories <NUM> from the vehicle that requested assistance. As the vehicle navigates, vehicle systems may use measurements of the surrounding environment to determine and perform a control strategy according to traffic rules and regulations. In some cases, the vehicle systems may fail to identify a trajectory that satisfies a confidence threshold required to perform the trajectory. As an example result, the vehicle may provide information representing proposed trajectories <NUM> to computing device <NUM> with a request for assistance, which can allow a human operator to review proposed trajectories <NUM>. In addition, computing device <NUM> can also filter proposed trajectories received from the vehicle prior to conveying proposed trajectories <NUM> to a human operator. For instance, computing device <NUM> can use a filter that removes proposed trajectories based on confidence levels and/or required maneuvers. In some examples, computing device <NUM> may generate one or more proposed trajectories <NUM> based on information received from a vehicle.

Behavior predictions <NUM> include information that represent potential actions by other objects located nearby the vehicle. Vehicle systems may predict the future states of nearby vehicles and other objects based on the current and past observations of the surrounding environment, which can enhance awareness of imminent hazards. As such, computing device <NUM> may receive behavior predictions <NUM> from the vehicle in some implementations. Computing device <NUM> may also generate behavior predictions for an object or objects based on sensor data received from the vehicle.

Computing device <NUM> may be configured to convey proposed trajectories <NUM> and behavior predictions <NUM> via GUI <NUM>. This way, a human operator may have the information available when providing assistance to a vehicle. In some examples, computing device <NUM> may augment environment representation <NUM> and/or road map <NUM> to further convey proposed trajectories <NUM>, behavior predictions <NUM> or both.

In some examples, computing device <NUM> may augment environment representation <NUM> to show visual trajectories for objects based on behavior predictions <NUM> and a visual trajectory for each proposed trajectory <NUM>. GUI <NUM> may also display instructions and other information that assists a human operator provide assistance to the vehicle.

<FIG> illustrates remote assistance situation <NUM>, which shows vehicle <NUM> encountering vehicle <NUM> stranded in the middle of the intersection and in path <NUM> being navigated by vehicle <NUM>. As such, situation <NUM> is shown to represent an example scenario where vehicle <NUM> may use remote assistance to overcome an obstacle (e.g., stranded vehicle <NUM>). Other example scenarios are possible.

In the example embodiment, situation <NUM> involves stranded vehicle <NUM> disrupting the current path <NUM> of vehicle <NUM>. In some cases, vehicle systems may alter path <NUM> to circumvent stranded vehicle <NUM> while also obeying traffic regulations (e.g., stopping at stop sign <NUM>) and avoiding vehicle <NUM>. In other cases, vehicle <NUM> may use remote assistance to obtain navigation instructions that overcomes situation <NUM>. For instance, vehicle <NUM> may communicate with a remote computing device, such as remote computing device <NUM> shown in <FIG> or computing device <NUM> shown in <FIG>.

<FIG> depicts GUI <NUM> displayable by a computing device to enable remote assist. In the example embodiment, GUI <NUM> shows the environment of vehicle <NUM> based on sensor data received from vehicle <NUM>. In particular, GUI <NUM> may use images and/or video received from vehicle <NUM> to convey the environment in near real-time, which shows vehicle <NUM> stranded in the middle of the intersection and in front of vehicle <NUM>. In addition, the environment also shows vehicle <NUM> traveling toward the intersection and nearby vehicle <NUM>.

GUI <NUM> further depicts additional information to assist a human operator understand the situation and provide assistance, including text instructions <NUM>, visual path A <NUM> with path A information <NUM>, and visual path B <NUM> with path B information <NUM>. As such, GUI <NUM> can enable interface techniques that allow a human operator to provide situational-aware assistance to vehicle <NUM>.

Visual path A <NUM> is shown as a trajectory that circumvents stranded vehicle <NUM> and realigns vehicle <NUM> according to the original path <NUM>. As shown, visual path A <NUM> involves avoiding vehicle <NUM> that is predicted to travel through the intersection according to behavior prediction A <NUM>. Vehicle <NUM> or the remote computing device may determine behavior prediction A <NUM> for vehicle <NUM> based on prior states of vehicle <NUM>.

In addition to visual path A <NUM>, GUI <NUM> also conveys path A information <NUM>, which provides additional contextual information that the human operator could consider when providing assistance. In the example embodiment, path A information <NUM> indicates that vehicle <NUM> determined a <NUM> percent confidence level associated with performing path A and that performance of path A would require an estimated <NUM> minutes of additional travel to reach the destination. In other examples, path A information <NUM> can indicate other information for the human operator to review, such as an identification of particular maneuvers that are outside than typical maneuvers performed during autonomous navigation (e.g., U-turns).

Visual path B <NUM> is shown as a different trajectory that turns on the other street and avoids stranded vehicle <NUM>. Because visual path B <NUM> would lead vehicle <NUM> down a different road that does not realign immediately with the original path <NUM>, visual path B <NUM> may be easier for vehicle <NUM> to perform. As an example result, vehicle path B information <NUM> indicates a higher confidence level at <NUM> percent, but a longer time to reach the vehicle's destination (<NUM> minutes).

GUI <NUM> further augments the environment representation with behavior prediction B <NUM> for vehicle <NUM>. For instance, vehicle <NUM> may request for assistance after incorrectly predicting behavior prediction B <NUM> for vehicle <NUM> when vehicle <NUM> is actually stranded. As such, a human operator may suppress or modify behavior prediction B <NUM> when providing assistance to vehicle <NUM>. Performing such an action may cause the computing device to generate and display new paths that depend on the action.

<FIG> is a flow chart of a method for vehicle occupancy confirmation, according to example implementations. Method <NUM> represents an example method that may include one or more operations, functions, or actions, as depicted by one or more of blocks <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>, each of which may be carried out by any of the systems, devices, and/or vehicles shown in <FIG>, among other possible systems. For instance, system <NUM> depicted in <FIG> may enable execution of method <NUM>.

Those skilled in the art will understand that the flowchart described herein illustrates functionality and operations of certain implementations of the present disclosure. In this regard, each block of the flowchart may represent a module, a segment, or a portion of program code, which includes one or more instructions executable by one or more processors for implementing specific logical functions or steps in the process. The program code may be stored on any type of computer readable medium, for example, such as a storage device including a disk or hard drive.

In addition, each block may represent circuitry that is wired to perform the specific logical functions in the process. Alternative implementations are included within the scope of the example implementations of the present application in which functions may be executed out of order from that shown or discussed, including substantially concurrent or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art.

At block <NUM>, method <NUM> involves receiving sensor data depicting a forward path for an autonomous vehicle. The computing device is positioned remotely from the autonomous vehicle. In some examples, the computing device may receive a request for assistance from the autonomous vehicle that includes location information and the sensor data from one or more vehicle sensors on the autonomous vehicle. For instance, the computing device can receive sensor data depicting the forward path from a vehicle camera system on the autonomous vehicle.

At block <NUM>, method <NUM> involves displaying, by the computing device, a representation of the forward path based on the sensor data.

At block <NUM>, method <NUM> involves augmenting the representation of the forward path to further depict one or more proposed trajectories available for the autonomous vehicle to perform. Each proposed trajectory may convey one or more maneuvers positioned relative to road boundaries in the forward path. In some instances, the road boundaries can be permanent (e.g., lane markers, curbs) and/or temporary (e.g., traffic cones). In some examples, the computing device may display proposed trajectories using a color scale that represents differences in confidence levels associated with the proposed trajectories.

In some examples, the computing device may determine one or more proposed trajectories based on navigation information received from the autonomous vehicle. The computing device may determine one or more proposed trajectories based on prior driver information that conveys one or more successful trajectories navigating around an obstacle in the forward path via one or more driver-controlled vehicles.

In some examples, the computing device may receive one or more proposed trajectories and corresponding confidence levels from the autonomous vehicle where a confidence level for a proposed trajectory depends on respective maneuvers associated with performing the proposed trajectory. The computing device may then filter the one or more proposed trajectories to remove any proposed trajectories having a given confidence level below a threshold confidence level and augment the representation of the forward path to further depict the one or more proposed trajectories based on filtering the one or more proposed trajectories.

In some examples, the computing device may perform a comparison between the one or more proposed trajectories and determine an additional proposed trajectory based on the comparison. The computing device may then augment the representation of the forward path to further depict the additional proposed trajectory.

At block <NUM>, method <NUM> involves receiving a selection of a proposed trajectory from the one or more proposed trajectories available for the autonomous vehicle to perform.

At block <NUM>, method <NUM> involves providing, by the computing device and to the autonomous vehicle, navigation instructions based on the proposed trajectory. The navigation instructions can cause the autonomous vehicle to follow the proposed trajectory.

In some examples, method <NUM> further involves receiving a plurality of behavior prediction trajectories and intents for one or more objects positioned relative to the forward path and augmenting the representation of the forward path to further convey respective behavior prediction trajectories and intents for the one or more objects. For instance, the computing device may display a first visual path for a first object based on a first behavior prediction trajectory and intent for the first object and display a second visual path for a second object based on a second behavior prediction trajectory and intent for the second object.

In some examples, the computing device may receive a first input that modifies the first visual path for the first object and subsequently provide navigation instructions further based on the first input that modifies the first visual path for the first object. For example, responsive to receiving the first input that modified the first visual path for the first object, the computing device may modify one or more proposed trajectories for the autonomous vehicle to perform and then display the representation of the forward path to further depict the one or more modified proposed trajectories available for the autonomous vehicle to perform. As such, the computing device may receive a given selection of a modified proposed trajectory available for the autonomous vehicle to perform.

In some examples, the computing device may receive a first behavior prediction trajectory for a first object positioned relative to the forward path and receive a second behavior prediction trajectory for a second object positioned relative to the forward path. The computing device may then augment the representation of the forward path to further convey the first behavior prediction trajectory for the first object and the second behavior prediction trajectory for the second object. For instance, the computing device may display a first visual path for the first object based on the first behavior prediction trajectory and a second visual path for the second object based on the second behavior prediction trajectory. The computing device may augment the representation to further depict a confidence level for each proposed trajectory available for the autonomous vehicle to perform.

In some cases, the computing device may receive a first input to suppress the first behavior prediction trajectory for the first object. Based on receiving the first input, the computing device may provide an indication to suppress the first behavior prediction trajectory for the first object to the vehicle and subsequently receive an updated proposed trajectory from the vehicle. The computing device may then augment the representation of the forward path to further depict the updated proposed trajectory.

In some examples, the computing device may receive the selection of the proposed trajectory from the one or more proposed trajectories available for the autonomous vehicle to perform and responsively perform a simulation that involves a virtual vehicle performing the proposed trajectory. The computing device may then provide the navigation instructions to the autonomous vehicle based on an outcome of the simulation exceeding a success threshold.

<FIG> is a schematic diagram of a computer program, according to an example implementation. In some implementations, the disclosed methods may be implemented as computer program instructions encoded on a non-transitory computer-readable storage media in a machine-readable format, or on other non-transitory media or articles of manufacture.

In the embodiment shown in <FIG>, computer program product <NUM> is provided using signal bearing medium <NUM>, which may include one or more programming instructions <NUM> that, when executed by one or more processors may provide functionality or portions of the functionality described above with respect to <FIG>.

Signal bearing medium <NUM> may encompass a non-transitory computer-readable medium <NUM>, such as, but not limited to, a hard disk drive, a Compact Disc (CD), a Digital Video Disk (DVD), a digital tape, memory, components to store remotely (e.g., on the cloud) etc. In some implementations, signal bearing medium <NUM> may encompass computer recordable medium <NUM>, such as, but not limited to, memory, read/write (R/W) CDs, R/W DVDs, etc..

In some implementations, signal bearing medium <NUM> may encompass communications medium <NUM>, such as, but not limited to, a digital and/or an analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communications link, a wireless communication link, etc.). Similarly, signal bearing medium <NUM> may correspond to a remote storage (e.g., a cloud). A computing system may share information with the cloud, including sending or receiving information. For example, the computing system may receive additional information from the cloud to augment information obtained from sensors or another entity. Thus, for example, signal bearing medium <NUM> may be conveyed by a wireless form of communications medium <NUM>.

One or more programming instructions <NUM> may be, for example, computer executable and/or logic implemented instructions. In some examples, a computing device such as computer system <NUM> shown in <FIG> or computing device <NUM> shown in <FIG> may be configured to provide various operations, functions, or actions in response to programming instructions <NUM> conveyed to the computer system by one or more of computer readable medium <NUM>, computer recordable medium <NUM>, and/or communications medium <NUM>. The non-transitory computer readable medium could also be distributed among multiple data storage elements and/or cloud (e.g., remotely), which could be remotely located from each other. Computing device that executes some or all of the stored instructions could be a vehicle. Alternatively, the computing device that executes some or all of the stored instructions could be another computing device, such as a server.

Claim 1:
A method (<NUM>) comprising:
receiving (<NUM>), at a computing device (<NUM>), sensor data depicting a forward path for an autonomous vehicle, wherein the computing device is positioned remotely from the autonomous vehicle;
displaying (<NUM>), by the computing device, a representation of the forward path based on the sensor data;
augmenting (<NUM>) the representation of the forward path to further depict one or more proposed trajectories available for the autonomous vehicle to perform, wherein each proposed trajectory conveys one or more maneuvers positioned relative to road boundaries in the forward path and is determined by the computing device;
receiving a plurality of behavior predictions for one or more objects positioned relative to the forward path;
augmenting the representation of the forward path to further convey respective behavior prediction trajectories for the one or more objects;
receiving (<NUM>), at the computing device, a selection of a proposed trajectory from the one or more proposed trajectories available for the autonomous vehicle to perform; and
providing (<NUM>), by the computing device and to the autonomous vehicle, navigation instructions based on the proposed trajectory.