Patent Description:
Autonomous vehicles rely on a variety of systems and methods for safely and efficiently navigating the world. At times, though, an autonomous vehicle may receive guidance from outside the system of the autonomous vehicle. For example, a teleoperator may provide a new route to the autonomous vehicle to re-route the autonomous vehicle past an accident that isn't reflected in a global map yet, or a passenger may identify a passerby as being an individual the autonomous vehicle should pick up. However, if the autonomous vehicle receives conflicting guidance, such as concurrent guidance from two different actors, the safety and efficiency of the autonomous vehicle may be compromised. <CIT> discloses a vehicle implementing one or more processing modules. These modules are configured to connect and interface with the various buses in the vehicle, where the various buses are connected with the various components of the vehicle to facilitate information transfer among the vehicle components. Each processing module is further modularized with the ability to add and replace other functional modules now or in the future. These functional modules can themselves act as distinct vehicle components. Each processing module may hand-off processing to other modules depending on its health, processing load, or by third-party control. <CIT> discloses that data including vehicle occupant authentication information is received. A request to control a vehicle component is received. A role for the vehicle occupant is identified. A determination is made whether the identified role includes a permission to control the vehicle component. The requested control of the vehicle component is performed if the identified role includes the permission to control the vehicle component. <CIT> discloses a system including a control computer that is programmed to perform an authentication based on an encryption key, upon being connected to a vehicle communication network. <CIT> discloses an apparatus and method for centralized control of a vehicle.

In the figures, the left-most digit(s) of a reference number identify the figure in which the reference number first appears.

As discussed above, if an autonomous vehicle receives conflicting guidance, such as concurrent guidance from two different actors, the safety and efficiency of the autonomous vehicle may be compromised. This application describes techniques that avoid conflicting guidance from multiple different actors. For instance, this application describes techniques for granting authority to an actor to provide guidance to an autonomous vehicle, transitioning the authority between actors, and tracking which actor has the authority. Guidance may comprise a command (e.g., one or more sets of instructions to be performed by the autonomous vehicle), a collaborative instruction (e.g., an instruction by which some information is provided to the autonomous vehicle and the autonomous vehicle is capable of using such information to control itself), a confirmation (e.g., one or more signals indicative that a proposed command determined by the autonomous vehicle should be executed), and/or ride control instructions. In some instances, the techniques may include limiting authority to a single actor at a time.

In some instances, an autonomous vehicle may track who has authority to provide guidance to the autonomous vehicle and the autonomous vehicle may exclude actors other than the actor that currently has authority ("engaged actor," herein) from providing guidance to the autonomous vehicle. In additional or alternate examples, another computing device, such as a cloud-based service, may track who has authority. In some instances, the engaged actor may be a human and/or an artificial intelligence (AI) component. Although it is contemplated that the actor may comprise more than one actor, only one actor may provide guidance to one component of the autonomous vehicle at a time.

If two actors have authority to provide guidance to the autonomous vehicle, the two actors are excluded from providing guidance to a same component of the autonomous vehicle but may provide guidance to two different components of the autonomous vehicle. To give an illustration, a first actor may have authority to provide guidance to a planner component of the autonomous vehicle (e.g., providing a proposed path to navigate a construction zone) and a second actor may have authority to provide internal temperature controls for passenger comfort. Regardless, as discussed herein, a first actor that has "authority" is permitted by the authority tracker to provide guidance to at least a component of the autonomous vehicle while a second actor (and/or any other actors) is excluded, by the authority tracker, from providing guidance to that same component, without the authority tracker first transitioning authority from the first actor to the second actor. For example, any guidance received from an excluded actor may be ignored and/or a request received from an excluded actor for the autonomous vehicle to conduct an operation may be denied. In some instances, an application running on a device associated with an actor will not populate a user interface with options to transmit guidance to the autonomous vehicle and/or the device may not transmit guidance to the autonomous vehicle until a state broadcast from the autonomous vehicle indicates that the actor has authority.

In some instances, the guidance may comprise data that is otherwise unavailable to the autonomous vehicle via one or more sensors of the autonomous vehicle (e.g., knowledge or directions of a passenger, such as, an instruction to pick up an additional passenger or an identification of the additional passenger from among a number of passersby), a confirmation of an output of a component of the autonomous vehicle, a collaborative instruction, or a command.

In some instances, the autonomous vehicle (and/or whatever device is tracking who has authority) may comprise an authority tracker that is configured with a policy that specifies one or more conditions in response to which authority may transition from one actor to another actor. For example, the policy may comprise a state machine that specifies at least one of a role (e.g., teleoperator, ride operator, supervisor, super user), a state, or an action for which authority may transition from one actor to another actor. For example, a teleoperator may provide guidance regarding motion of the autonomous vehicle and/or exterior emitter actions (e.g., turn on lights, make a sound via a speaker), a ride operation may provide guidance to different component(s) of the autonomous vehicle related to passenger interaction with the vehicle (e.g., open a door of the autonomous vehicle, identify a passenger associated with the autonomous vehicle, change an internal temperature of the autonomous vehicle), a supervisor may be associated with other actor(s), and/or a super user may be a special actor for closed-course testing of the autonomous vehicle and may comprise a test-driver, for example, that is override the policy. The authority tracker may store, in a memory, a state that identifies who currently has authority (the "engaged actor") to provide guidance to the autonomous vehicle, an identification of the autonomous vehicle and/or component of the autonomous vehicle over which the actor has authority, an identifier of a second actor that may take authority, and/or a status (e.g., success, failure, timeout, awaiting handoff, escalating). In some instances, the authority tracker may periodically transmit (publish), over a "state" channel to one or more devices subscribing to the state channel, the state currently stored in the memory.

The state may remain unmodified until the authority tracker authorizes a transition of authority. When the authority tracker authorizes a second actor to take authority (i.e., the authority transitions from a first actor to the second actor), the authority tracker may replace a first state stored in memory identifying a first actor as having authority with a second state identifying the second actor as having the authority.

In some instances, to receive requests for transitioning authority to a different actor, the authority tracker may subscribe to a "control" channel and may trigger an evaluation of the policy based at least in part on determining that a request sent over the control channel comprises an identifier of the autonomous vehicle (and/or a component thereof). In some instances, a request (e.g., a control message) received over the control channel may comprise at least one of an action, a target identifier (e.g., which autonomous vehicle and/or which component is targeted by the message), a source identifier (e.g., an identifier associated with an actor), and/or a second target identifier (e.g., when a first actor is seeking to transfer authority to a second actor).

An action may comprise, for example, a takeover message, a relinquish message, a handoff message, an escalate message, and/or a rejection message. The takeover message may indicate that the actor sending the takeover message is requesting a transition of authority from an engaged actor to the requesting actor. The relinquish message may indicate that an actor that currently has authority is giving up the authority, causing authority to revert to a former holder of authority (who may be identified by the second identifier), a default holder of authority (e.g. the autonomous vehicle), and/or to the autonomous vehicle. The handoff message may indicate that the engaged actor requests to transfer authority to a different actor. The escalation message may indicate that an engaged actor requests to transfer authority to a supervising actor.

In some instances, the authority tracker may be communicatively coupled to a directory service that publishes and/or serves user information such as identifiers (e.g., usernames) associated with actors authorized to take authority, a role associated with an actor, a host name (e.g., an identification of a computing device associated with an actor), a process name of a guidance application running on the computing device (e.g., a ride experience process, a collaborative instruction process, a driving envelope modification process, a route modification process, a city event notification process), and/or a hash thereof. In some instances, control messages may comprise user information of an actor that generated the control message and the authority tracker may verify that the actor is verified by the directory service before determining whether or not to grant authority to the actor.

The techniques described herein may improve the safety and efficiency of an autonomous vehicle. In instances where multiple actors attempt to provide guidance to the autonomous vehicle, the techniques discussed herein may reduce the time it takes for an autonomous vehicle to continue operating or to operate normally and may reduce computational cycles to determine which guidance to implement and/or to correct mistakes made by attempting to implement conflicting guidance.

<FIG> illustrates an example scenario <NUM> including an autonomous vehicle <NUM> and actor(s) <NUM>(<NUM>)-(x). Although the actor(s) <NUM>(<NUM>)-(x) are depicted as representations of humans, an actor <NUM>(x) may comprise a human, an AI component, and/or any other device. Although the discussion herein primarily uses a vehicle or an autonomous vehicle as an example, it is contemplated that the techniques discussed herein may be applied to any other scenario where two or more actors are available to supply guidance and/or instructions to a machine and/or where contemporaneous overlapping inputs may cause a degradation of machine operation.

In some instances, the autonomous vehicle <NUM> may be an autonomous vehicle configured to operate according to a Level <NUM> classification issued by the U. National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions for the entire trip, with the driver (or occupant) not being expected to control the vehicle at any time. However, in other examples, the autonomous vehicle <NUM> may be a fully or partially autonomous vehicle having any other level or classification. Moreover, in some instances, the guidance isolation techniques described herein may be usable by non-autonomous vehicles as well.

According to the techniques discussed herein, the autonomous vehicle <NUM> may receive sensor data from sensor(s) <NUM> of the autonomous vehicle <NUM>. For example, the sensor data may include a location signal (e.g., a GPS signal), an inertia signal (e.g., an accelerometer signal, a gyroscope signal, etc.), a magnetometer signal, a wheel encoder signal, a speedometer signal, a point cloud of accumulated LIDAR and/or RADAR points, an image (or images), an audio signal, and/or bariatric or other environmental signals, etc. The autonomous vehicle <NUM> may use sensor data received from such sensor(s) <NUM> to determine a trajectory for controlling motion of the autonomous vehicle <NUM> and/or a confidence level associated with the trajectory indicating a probability that the autonomous vehicle <NUM> is operating safely and/or efficiently.

In the course of operating according to the example scenario <NUM>, the autonomous vehicle <NUM> has come upon an obstruction <NUM> depicted as a fallen trash can. In some instances, based at least in part on determining that a confidence level associated with a trajectory for controlling the autonomous vehicle <NUM> does not meet a threshold confidence, the autonomous vehicle <NUM> may generate a teleoperation request. The autonomous vehicle <NUM> may transmit, via a network <NUM>, the teleoperation request to computing devices <NUM>(<NUM>)-(x) associated with the actor(s) <NUM>(<NUM>)-(x). This teleoperation request may be an indication that the autonomous vehicle <NUM> is seeking guidance. However, it is contemplated that the autonomous vehicle <NUM> does not need to seek guidance to receive guidance and/or for an actor <NUM>(x) to seek to take authority to provide guidance to the autonomous vehicle <NUM>. For example, a passenger may provide guidance, a nearby remote operator may provide guidance and/or direct control, a passerby may provide guidance, a teleoperator may provide guidance, etc. In instances where a passenger or passerby provides guidance to the autonomous vehicle <NUM>, the passenger or passerby may not be aware that, in providing guidance to the autonomous vehicle <NUM> via a computing device associated with the passenger or passerby, an application running on the computing device may generate a request to take authority, as discussed herein.

In some instances, a computing device <NUM>(x) may comprise a guidance application <NUM>. The guidance application <NUM> may be configured to generate and transmit, over a control channel, a control message and/or to receive state messages over a state channel. For example, the network <NUM> may be configured to route control messages from autonomous vehicle(s) and/or actor(s) that publish the control messages to subscribers of the control channel and/or route state messages from autonomous vehicle(s) that publish state messages to subscribers of the state channel. The guidance application <NUM>, when executed by the computing device <NUM>(x), may additionally, or alternatively, cause display of a user interface <NUM>. The user interface <NUM> may comprise selectable elements that, upon activation, may generate guidance comprising computer-executable instructions and/or logic. Another component of the user interface <NUM> may be selectable to cause the computing device <NUM>(x) to transmit the computer-executable instructions via the network <NUM> to the autonomous vehicle <NUM>.

In some instances, the guidance may comprise additional data (e.g., identification of an individual by a passenger/passerby, identification of an exigent circumstance of which the autonomous vehicle <NUM> may be unaware, e.g., an object lodged in a tire or in the undercarriage), a command, a collaborative instruction, and/or a confirmation. For example, the computer-executable instructions of the guidance may comprise:.

In the illustrated example, the actor <NUM>(x) may transmit, via the guidance application <NUM> a control message, over the control channel on network <NUM>, a request that may include a takeover message identifying the autonomous vehicle <NUM> and/or identifying the actor <NUM>(x). The autonomous vehicle <NUM> may include computing device(s) <NUM> that may run an authority tracker <NUM>. Although depicted in <FIG> as being in the autonomous vehicle <NUM> for illustrative purposes, such an authority tracker <NUM> may be remote therefrom (e.g., as a cloud based service).

The authority tracker <NUM> may track which actor currently has authority (i.e., the "engaged actor") to provide guidance to the autonomous vehicle <NUM> and/or the authority tracker <NUM> may determine, based at least in part on a policy, when to transition authority from one actor to another. For example, the authority tracker <NUM> may receive the request from the actor <NUM>(x) and, based at least in part on a policy, may determine to authorize the request. In some instances, the authority tracker <NUM> may track which actor has authority by storing a state <NUM> in a memory of the autonomous vehicle <NUM>. Upon determining to authorize the request, the authority tracker may modify the state <NUM> stored in memory to indicate (<NUM>) that actor <NUM>(x) has authority to provide guidance to the autonomous vehicle <NUM>.

Note that the autonomous vehicle <NUM> itself may be one of the actors and, in some instances, the autonomous vehicle <NUM> may be a default actor to which authority reverts and from which authority springs. For example, state <NUM> depicts actors <NUM>(<NUM>)-(x) and autonomous vehicle <NUM> as being potential actors that may have control and an indication (<NUM>) of the actor that actively possesses authority, which in this case is actor <NUM>(x), as discussed below.

Authority is exclusive (i.e., no other actors may simultaneously have authority to provide guidance to a same autonomous vehicle <NUM> and/or component of the autonomous vehicle <NUM>). However, when the authority tracker <NUM> grants authority to an actor other than the autonomous vehicle <NUM>, the authority granted may be limited to providing guidance to particular component(s) of the autonomous vehicle. In other words, the autonomous vehicle <NUM> may reserve ultimate control of the autonomous vehicle <NUM> by continuing to generate a trajectory to control motion of the autonomous vehicle <NUM> and/or other such safety critical features. In some instances, the autonomous vehicle <NUM> may use guidance received from an actor that has authority as part of a determination of instructions for controlling the autonomous vehicle <NUM> (e.g., the guidance may modify a driving envelope and/or a route, but the autonomous vehicle <NUM> reserves control over generating a trajectory subject to that driving envelope and/or new route).

However, in some instances, such as during test-driving of the autonomous vehicle during experimental driving on a closed course, an actor associated with a super user role may transmit a takeover message that allows the actor to control even trajectory generation, such as by reverting the autonomous vehicle <NUM> to a semi-autonomous mode.

Once actor <NUM>(x) has been granted authority by the authority tracker <NUM>, as indicated (and canonized) by modifying the state <NUM> identify actor <NUM>(x) as having authority, guidance received from actor <NUM>(x), now the engaged actor, may be used, least in part, by the autonomous vehicle <NUM> in determining instructions to control the autonomous vehicle <NUM>. In some instances, guidance received from actor <NUM>(x) may be received via the network <NUM> from computing device <NUM>(x) and/or guidance application <NUM>.

A policy stored by the authority tracker <NUM> may determine how authority transitions back to the autonomous vehicle <NUM> ("reverts") and/or transitions to another actor. In some instances, authority may revert to the autonomous vehicle based at least in part on receiving a relinquish message from the engaged actor, determining that a connection between the engaged actor and the autonomous vehicle <NUM> has timed out (e.g., the guidance application <NUM> may periodically transmit keepalive messages and, upon failing to receive a keepalive message within a specified time period, authority may revert to the autonomous vehicle <NUM>), and/or upon receiving a disengagement message from a super user (e.g., a test driver that is operating the autonomous vehicle <NUM> during a test on a closed course). In some examples, such a disengagement message may be auto generated based on, for example, the autonomous vehicle <NUM> surpassing some obstacle, generating a trajectory having some threshold confidence, or otherwise determining that it no longer needs to rely on external guidance. In some instances, authority may transition to another actor based at least in part on receiving a handoff message and/or an escalation message, and/or upon determining that a connection to the engaged actor has timed out.

<FIG> depicts a block diagram of an example system <NUM> for implementing the techniques described herein. In some instances, the example system <NUM> may include a computing device <NUM>. In the illustrated example, the computing device <NUM> comprises an autonomous vehicle; however, the computing device <NUM> may be any other type of computing device and/or semi-autonomous or non-autonomous vehicle. For example, computing device <NUM> may comprise autonomous vehicle <NUM>.

The computing device <NUM> may include sensor(s) <NUM>, emitter(s) <NUM>, network interface(s) <NUM>, and/or one or more drive modules <NUM>. In some instances, the sensor(s) <NUM> may include light detection and ranging (LIDAR) sensors, RADAR sensors, ultrasonic transducers, sonar sensors, location sensors (e.g., global positioning system (GPS), compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., red-green-blue (RGB), infrared (IR), intensity, depth, time of flight, etc.), microphones, wheel encoders, environment sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), etc. The sensor(s) <NUM> may include multiple instances of each of these or other types of sensors. For instance, the LIDAR sensors may include individual LIDAR sensors located at the corners, front, back, sides, and/or top of the computing device <NUM>. As another example, the camera sensors may include multiple cameras disposed at various locations about the exterior and/or interior of the computing device <NUM>. The sensor(s) <NUM> may provide input to the computing device <NUM>.

The vehicle <NUM> may also include emitter(s) <NUM> for emitting light and/or sound, as described above. The emitter(s) <NUM> in this example include interior audio and visual emitter(s) to communicate with passengers of the computing device <NUM>. By way of example and not limitation, interior emitter(s) may include speakers, lights, signs, display screens, touch screens, haptic emitter(s) (e.g., vibration and/or force feedback), mechanical actuators (e.g., seatbelt tensioners, seat positioners, headrest positioners, etc.), and the like. The emitter(s) <NUM> in this example also include exterior emitter(s). By way of example and not limitation, the exterior emitter(s) in this example include lights to signal a direction of travel or other indicator of vehicle action (e.g., indicator lights, signs, light arrays, etc.), and one or more audio emitter(s) (e.g., speakers, speaker arrays, horns, etc.) to audibly communicate with pedestrians or other nearby vehicles, one or more of which comprising acoustic beam steering technology.

The computing device <NUM> may also include network interface(s) <NUM> that enable communication between the computing device <NUM> and one or more other local or remote computing device(s). For instance, the network interface (s) <NUM> may facilitate communication with other local computing device(s) on the computing device <NUM> and/or the drive module(s) <NUM>. Also, the network interface (s) <NUM> may allow the vehicle to communicate with other nearby computing device(s) (e.g., other nearby vehicles, traffic signals, etc.). The network interface (s) <NUM> also enable the vehicle <NUM> to communicate with a computing device <NUM> associated with an actor and/or a computing device (directory device) <NUM> associated with a directory service.

The network interface(s) <NUM> may include physical and/or logical interfaces for connecting the computing device <NUM> to another computing device or a network, such as network(s) <NUM>. For example, the network interface(s) <NUM> may enable Wi-Fi-based communication such as via frequencies defined by the IEEE <NUM> standards, short range wireless frequencies such as Bluetooth®, cellular communication (e.g., <NUM>, <NUM>, <NUM>, <NUM> LTE, <NUM>, etc.) or any suitable wired or wireless communications protocol that enables the respective computing device to interface with the other computing device(s).

In some instances, the computing device <NUM> may include one or more drive modules <NUM>. In some instances, the computing device <NUM> may have a single drive module <NUM>. In some instances, the drive module(s) <NUM> may include one or more sensors to detect conditions of the drive module(s) <NUM> and/or the surroundings of the computing device <NUM>. By way of example and not limitation, the sensor(s) of the drive module(s) <NUM> may include one or more wheel encoders (e.g., rotary encoders) to sense rotation of the wheels of the drive modules, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) to measure orientation and acceleration of the drive module, cameras or other image sensors, ultrasonic sensors to acoustically detect objects in the surroundings of the drive module, LIDAR sensors, RADAR sensors, etc. Some sensors, such as the wheel encoders may be unique to the drive module(s) <NUM>. In some cases, the sensor(s) on the drive module(s) <NUM> may overlap or supplement corresponding systems of the computing device <NUM> (e.g., sensor(s) <NUM>).

The drive module(s) <NUM> may include many of the vehicle systems, including a high voltage battery, a motor to propel the vehicle, an inverter to convert direct current from the battery into alternating current for use by other vehicle systems, a steering system including a steering motor and steering rack (which may be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and/or pneumatic components, a stability control system for distributing brake forces to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head/tail lights to illuminate an exterior surrounding of the vehicle), and one or more other systems (e.g., cooling system, safety systems, onboard charging system, other electrical components such as a DC/DC converter, a high voltage junction, a high voltage cable, charging system, charge port, etc.). Additionally, the drive module(s) <NUM> may include a drive module controller which may receive and preprocess data from the sensor(s) and to control operation of the various vehicle systems. In some instances, the drive module controller may include one or more processors and memory communicatively coupled with the one or more processors. The memory may store one or more modules to perform various functionalities of the drive module(s) <NUM>. Furthermore, the drive module(s) <NUM> may also include one or more communication connection(s) that enable communication by the respective drive module with one or more other local or remote computing device(s).

The computing device <NUM> may include one or more processors <NUM> and memory <NUM> communicatively coupled with the one or more processors <NUM>. Computing device(s) <NUM> and/or <NUM> may also include processor(s) <NUM> and <NUM>, network interface(s) <NUM> and <NUM>, and/or memory <NUM> and <NUM>, respectively. The processor(s) <NUM>, <NUM>, and/or <NUM> may be any suitable processor capable of executing instructions to process data and perform operations as described herein. By way of example and not limitation, the processor(s) <NUM>, <NUM>, and/or <NUM> may comprise one or more central processing units (CPUs), graphics processing units (GPUs), integrated circuits (e.g., application-specific integrated circuits (ASICs), etc.), gate arrays (e.g., field-programmable gate arrays (FPGAs), etc.), and/or any other device or portion of a device that processes electronic data to transform that electronic data into other electronic data that may be stored in registers and/or memory.

Memory <NUM>, <NUM>, and/or <NUM> may be examples of non-transitory computer-readable media. The memory <NUM>, <NUM>, and/or <NUM> may store an operating system and one or more software applications, instructions, programs, and/or data to implement the methods described herein and the functions attributed to the various systems. In various implementations, the memory may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), nonvolatile/Flash-type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, of which those shown in the accompanying figures are merely examples that are related to the discussion herein.

In some instances, the memory <NUM> of the computing device <NUM> may store component(s) <NUM> such as, for example, a perception component, a localization component, a planning component, and/or similar components that, when executed by the one or more processors <NUM>, may perform various operations to operate the computing device <NUM>.

For example, the perception component may include functionality to perform object detection, segmentation, and/or classification. In some instances, the perception component may provide processed sensor data that indicates a presence of an entity that is proximate to the computing device <NUM> and/or a classification of the entity as an entity type (e.g., car, pedestrian, cyclist, animal, building, tree, road surface, curb, sidewalk, unknown, etc.). In additional and/or alternative instances, the perception component may provide processed sensor data that indicates one or more characteristics associated with a detected entity and/or the environment in which the entity is positioned/oriented. In some instances, characteristics associated with an entity may include, but are not limited to, an x-position/orientation (global position/orientation), a y-position/orientation (global position/orientation), a z-position/orientation (global position/orientation), an orientation, an entity type (e.g., a classification), a velocity of the entity, an extent of the entity (size), sensor signal quality and/or other characteristics, an indication of an amount of features present in a signal, a clarity score associated with a sensor signal, etc. Characteristics associated with the environment may include, but are not limited to, a presence of another entity in the environment, a state of another entity in the environment, a time of day, a day of a week, a season, a weather condition, an indication of darkness/light, etc. In some instances, guidance may specify an output of the perception component (e.g., the classification of an entity, a characteristic of the environment, an event happening in a region in which the computing device <NUM> is located).

In some instances, the localization component may include functionality to receive data from the sensor(s) <NUM> to determine a map and/or position/orientation of the computing device <NUM>. For example, the localization component may include and/or request/receive a map of an environment and may continuously determine a location of the autonomous vehicle within the map. In some instances, the localization component may utilize simultaneous localization and/or mapping (SLAM) and/or calibration, localization and/or mapping simultaneously (CLAMS), to receive image data, LIDAR data, RADAR data, IMU data, GPS data, wheel encoder data, and the like to accurately determine a location of the autonomous vehicle. In some instances, the localization component may provide data to various components of the computing device <NUM> to determine an initial position/orientation of an autonomous vehicle for generating a candidate trajectory, as discussed herein.

In general, the planning component may determine a path for the computing device <NUM> to follow to traverse through an environment. For example, the planning component may determine various routes and trajectories and various levels of detail. For example, the planning component may determine a route to travel from a first location (e.g., a current location) to a second location (e.g., a target location). For the purpose of this discussion, a route may be a sequence of waypoints for travelling between two locations. As non-limiting examples, waypoints include streets, intersections, coordinates (e.g., determined by the localization and/or mapping component and/or a GPS), etc. Further, the planning component may generate an instruction for guiding the computing device <NUM> along at least a portion of the route from the first location to the second location. In some instances, the planning component may determine how to guide the autonomous vehicle from a first waypoint in the sequence of waypoints to a second waypoint in the sequence of waypoints. In some instances, the instruction may be a trajectory, or a portion of a trajectory. In some instances, multiple trajectories may be substantially simultaneously generated (e.g., within technical tolerances) in accordance with a receding horizon technique. In some instances, and as discussed herein, the planning component may receive a map and/or position/orientation from the localization component and/or various data, such as identifications and/or classifications objects, events, and/or characteristics of the environment from the perception component.

In some instances, one or more of the component(s) <NUM> may be implemented in whole, or in part, as a neural network. As described herein, an exemplary neural network is a biologically inspired algorithm which passes input data through a series of connected layers to produce an output. Each layer in a neural network may also comprise another neural network, or may comprise any number of layers (whether convolutional or not). As may be understood in the context of this disclosure, a neural network may utilize machine learning, which may refer to a broad class of such algorithms in which an output is generated based on learned parameters.

Although discussed in the context of neural networks, any type of machine learning may be used consistent with this disclosure. For example, machine learning algorithms may include, but are not limited to, regression algorithms, instance-based algorithms, Bayesian algorithms, association rule learning algorithms, deep learning algorithms, etc. Additional or alternative examples of neural network architectures may include neural networks such as ResNet20, ResNet101, VGG, DenseNet, PointNet, and the like.

In some instances, the component(s) <NUM> may process sensor data, as described above, and may send their respective outputs, over the network(s) <NUM>, to one or more computing device(s) <NUM>. In some instances, the component(s) <NUM> may send their respective outputs to the one or more computing device(s) <NUM> at a particular frequency, after a lapse of a predetermined period of time, in near real-time, etc..

In some instances, the computing device <NUM> may send sensor data to one or more computing device(s) <NUM>, via the network(s) <NUM>. Computing device(s) <NUM> may represent computing device(s) <NUM>(<NUM>)-(x). In some instances, the vehicle <NUM> may send raw sensor data to the computing device(s) <NUM>. In other examples, the computing device <NUM> may send processed sensor data and/or representations of sensor data to the computing device(s) <NUM>.

In some instances, the memory <NUM> may further store one or more system controllers <NUM>, which may be configured to control steering, propulsion, braking, safety, emitter(s), communication, and other systems of the computing device <NUM>, such as by communicating instructions to the drive module(s) <NUM> and/or controlling corresponding systems of the drive module(s) <NUM> and/or other components of the computing device <NUM>.

In some instances, the memory <NUM> may additionally, or alternatively, store an authority tracker <NUM> that, when executed by the one or more processors <NUM> may cause the computing device <NUM> to perform various actions discussed herein. The authority tracker <NUM> may store state <NUM> in the memory <NUM> to track an actor that currently has authority to provide guidance to the computing device <NUM>. In some instances, the authority tracker <NUM> may determine to transition authority from a first actor to a second actor based at least in part on a policy <NUM> stored in memory <NUM>.

In some instances, the policy <NUM> may comprise a state machine. The state machine may specify one or more conditions and an individual condition may be associated with a transition. For example, if a condition is satisfied, such as by receiving a request from an actor that specifies data that satisfies that condition (e.g., the requesting actor is able to take control given the actor's role, the situation, etc.), the authority tracker <NUM> may transition authority from a first actor to the requesting actor by updating the state <NUM> to reflect the requesting actor as the engaged actor and transmitting a state message <NUM> to the computing device(s) <NUM>. Guidance subsequently received from the computing device(s) <NUM> associated with the engaged actor may be routed by the authority tracker <NUM> to the appropriate component(s) <NUM> identified by the state <NUM> and/or relevant to the guidance.

If a request doesn't satisfy the condition, the authority tracker <NUM> may maintain the current state <NUM> and, optionally, transmit the current state as the state message <NUM>, thereby indicating that no change in authority has occurred. Transmitting the current state (instead of transmitting an updated state) may also function as a rejection message, although the authority tracker <NUM> may additionally, or alternatively, transmit a rejection message. Any guidance received from a computing device <NUM> that does not correspond with an actor identified as being the engaged actor by the state <NUM> may be ignored by the authority tracker <NUM>. In other words, the state <NUM> may remain unmodified until the authority tracker <NUM> authorizes a transition of authority.

In some instances, a condition of the policy <NUM> may comprise at least one of an action, a role, or a state, though any other data is contemplated. For example, the policy <NUM> may specify that when the state <NUM> identifies the computing device <NUM> as having the authority, a condition of the policy <NUM> may require that, for an actor to take authority from the computing device <NUM>, a takeover message must be received and any role is suitable to take authority (e.g., any actor may take authority for any reason). However, the policy <NUM> may specify that when the state <NUM> identifies an actor other than the computing device <NUM> as having authority where the actor is not associated with a supervisor role, a condition of the policy <NUM> may require that, for a requesting actor to take authority from the engaged actor, a relinquish message, handover message, and/or escalation message must have been received from the engaged actor and/or the request must identify the requesting actor as being a supervisor. In some examples, when the vehicle holds the authority, actor(s) may be prevented from gaining authority at all and/or except for in narrowly-defined conditions (e.g., a safety-critical event is detected at the vehicle, the vehicle generates a teleoperations request). In some instances, the authority tracker <NUM> may verify an identity of a requesting actor before authorizing the actor to take the authority (e.g., via an open authorization (OAuth) exchange, by verifying a signed message using a public key, contacting a directory service, and the like). More examples of conditions that limit authority transitions are discussed below.

In some instances, the authority tracker <NUM> may transmit a state message <NUM>, via network interface(s) <NUM> over network(s) <NUM>, to the computing device(s) <NUM> periodically, based at least in part on receiving a control message <NUM>, based upon another event, etc. The state message <NUM> may include at least a portion of the state <NUM> stored at memory <NUM>.

In some instances, transmitting the state message <NUM> may comprise transmitting (broadcasting) the state message <NUM> over a state channel, which may comprise a publish-subscribe (pub/sub) service where computing device(s) <NUM> associated at least one actor is a subscriber. Similarly, the computing device(s) <NUM> may transmit a control message <NUM> over a control channel that comprises a pub/sub service where the computing device <NUM> may be a subscriber. Moreover, the computing device <NUM> and/or the computing device(s) <NUM> may subscribe to a user information channel to receive user information <NUM> from directory device(s) <NUM> associated with a directory service <NUM>.

In some instances, the state <NUM> and/or a state message <NUM> may comprise at least one of an identifier of an engaged actor, an identifier of the computing device <NUM> and/or a component thereof (e.g., a vehicle number, a vehicle number and an identification of one of the component(s) <NUM> that is being provided guidance); a status (e.g., success, failure, timeout, awaiting handoff, escalating); and/or a second identifier (e.g., a role targeted by a handoff/escalation, an actor targeted by a handoff/escalation). In some instances, "success" may indicate that the authority tracker <NUM> authorized an actor to take authority and "failed" may indicate that the authority tracker <NUM> denied an actor from taking authority.

In some instances, the control message <NUM> may comprise a request regarding the authority. For example, the control message <NUM> may comprise an action such as a takeover message, a relinquish message, a handoff message, an escalation message, and/or a rejection message. In some instances, the message, whichever it may be, may further comprise a target vehicle identifier (e.g., an identifier that distinguishes the computing device <NUM> from other vehicles), a source/identity identifier (e.g., a username associated an actor whose host machine sends the control message <NUM>), and/or a second identifier (e.g., to identify another actor and/or role, such as in a handoff and/or escalation message). In some instances, a source/identity identifier may comprise a username (e.g., a username that is unique across the system and that matches a directory record <NUM>), a host name (e.g., an identification of a specific computing device used by an actor, such as a media access control (MAC) address and/or an Internet protocol (IP) address), a process identifier and/or process name that identifies a guidance application <NUM> running on the computing device(s) <NUM>, and/or a hash of any portion of the above-described data. A unique identifier may be associated with each actor known to the system (and as recorded in the directory records <NUM>).

The takeover message may indicate that the actor sending the takeover message is requesting that the authority tracker <NUM> transition authority from an engaged actor to the requesting actor. In some instances, the takeover message may function as an acceptance to a request for teleoperator support, a handoff message, an escalation message, etc..

The relinquish message may indicate that an actor that currently has authority (the "engaged actor") is giving up the authority, causing authority to revert to a former holder of authority (who may be identified by the second identifier), a default holder of authority (e.g. the computing device <NUM>), to the computing device <NUM>, and/or to any currently waiting requesting actors.

The handoff message may indicate that the engaged actor requests to transfer authority to a different actor. In some instances, a handoff message may comprise second identifier of a specific actor to which the engaged actor requests to transfer authority or the second identifier may identify a role to which the engaged actor requests to transfer authority (e.g., any "peer" actors - actors having a same role as the engaged actor, any online actors). In some instances, the engaged actor may retain authority until a takeover message is received by the specific actor identified by the handoff message and/or an actor associated with a role identified by the handoff message (e.g., a takeover message is received by a peer actor). In some examples, requiring both a handoff and takeover ensures that the actor associated with the handoff is ready, willing, and able to provide the required guidance to the vehicle. In some instances, this process of requesting to transfer authority may result in a delay until the authority is actually transferred. In some instances, when the authority tracker receives a handoff message, the authority tracker may modify the state <NUM> to reflect the identifier of the recipient actor, an identification of the specified role, and/or "awaiting handoff.

The escalation message may indicate that an engaged actor requests to transfer authority to a supervising actor (e.g., an actor associated with a role of "supervisor" in the directory records <NUM>). Again, in some instances, this may comprise an identifier of a specific actor associated with a supervising role and/or this may comprise indicating a supervising role. In some instances, the engaged actor may retain authority until a takeover message is received by the specific supervising actor identified by the escalation message and/or an actor associated with a role identified by the escalation message (e.g., a supervisor role). In some instances, when the authority tracker receives an escalation message, the authority tracker may modify the state <NUM> to reflect the identifier of the recipient supervising actor, an identification of the specified role, and/or "escalating.

For example, an engaged actor may generate an escalation message when the engaged actor is unable to resolve an issue, the policy specifies that the engaged actor doesn't have the privileges to send a type of guidance to the vehicle and/or to view sensor data from the vehicle, and/or the engaged actor is not associated with a type of component of the vehicle (e.g., the engaged actor was handling a trajectory issue when a passenger ride issue arose, which the engaged actor may not have authorization and/or an application to handle).

In some instances, the guidance application <NUM> may cause the computing device(s) <NUM> to display a user interface for receiving input from an actor. Based on the input, the computing device(s) <NUM> may generate guidance comprising computer-executable instructions and/or logic, which may be transmitted via the network interface(s) <NUM> to the computing device <NUM>. In some instances, the user interface may comprise selectable elements that, upon input of an actor, generate and/or transmit a control message <NUM>. In some instances, the guidance application <NUM> may not populate the user interface with options to transmit guidance to the autonomous vehicle, may not present the selectable elements, and/or may not otherwise allow an actor to cause the computing device(s) <NUM> to generate guidance to transmit to the autonomous vehicle until a state message <NUM> broadcast from the computing device <NUM> indicates that the actor has authority.

In some instances, a state message <NUM> may comprise an identifier associated with the actor and/or a host machine identifier. Therefore, the computing device(s) <NUM> may receive a state message <NUM> via the network interface(s) <NUM> and guidance application <NUM> may determine that the host machine and/or the actor are identified in the state message <NUM>. In response, the guidance application <NUM> may populate the user interface with the selectable elements and/or otherwise allow generation and/or transmission of guidance from the computing device(s) <NUM> to the computing device <NUM>. In some instances, the guidance application <NUM> may retrieve a vehicle identifier from the state message <NUM> to populate control message(s) <NUM> and/or to associate with guidance transmitted to the computing device <NUM>.

In some instances, an actor may log into the computing device(s) <NUM> with a username recorded in the directory records <NUM> and, upon successfully logging in, the computing device(s) <NUM> may periodically transmit an "online" signal indicating that the actor is online to one or more other devices on the network(s) <NUM>. For example, where computing device(s) <NUM> comprise multiple devices associated with multiple actors, the computing device(s) <NUM> may transmit the online signal to the other devices. The computing device(s) <NUM> may additionally, or alternatively, store in memory <NUM> a record of active user(s) <NUM> based at least in part on online signals received from other devices.

In some instances, the computing device(s) <NUM> may store, in associated with the active user(s) <NUM> user information <NUM> received from the directory device(s) <NUM>. For example, a first actor may be operating a first computing device <NUM> and the first computing device <NUM> may receive a notification that a second actor is logged into a second computing device <NUM>. The first computing device <NUM> may access the user information <NUM> stored in memory <NUM>, request user information <NUM> from the directory device(s) <NUM>, and/or otherwise obtain the user information <NUM>. In some instances, the first computing device <NUM> may store an indication that the second actor is online in the record of active user(s) <NUM> and/or a role associated with the second actor (e.g., identified from the user information <NUM>).

In some instances, computing device(s) <NUM> may transmit additional signals to other computing device(s) <NUM> indicating activity at the computing device(s) <NUM>. For example, an online signal may additionally, or alternatively, indicate that the actor is online, but not an engaged actor (e.g., the actor has not received a notification for teleoperation support from a vehicle, the actor is not actively providing teleoperation support to a vehicle, the state channel does not include any state messages indicating that the actor has authority); online signal may additionally, or alternatively, indicate that the actor is online but is engaged (e.g., a state message indicates that the actor is the engaged actor for computing device <NUM>, the guidance application <NUM> makes this indication), etc. In some instances, the computing device(s) <NUM> may receive the signals and store indications associated with the signals in the record of active user(s) <NUM>.

The guidance application <NUM> may determine a subset of the active user(s) <NUM> (e.g., online users that are not occupied, online users that are associated with a role "supervisor") and the guidance application <NUM> may use this subset to populate a user interface and/or to determine a second device to which to transmit a request for teleoperation support received from the computing device <NUM>, a handoff message, and/or an escalation message. For example, an engaged actor may request to transition authority to a supervisor actor (e.g., an actor associated with the role "supervisor" in the directory records <NUM>). The guidance application <NUM> may cause the user interface to display a selectable element that, upon receiving input from an actor, causes the user interface to display an element that identifies supervisors that are online and/or that transmits an escalation message to the device(s) of one or more of the subset of the active user(s) <NUM> that are online and that are associated with the role "supervisor.

In some instances, the directory service <NUM> may comprise an open authorization (OAuth) application, a data distribution service (DDS) system, and/or an active directory server. The directory service <NUM> may provide user information <NUM> to the computing device <NUM> and/or computing device(s) <NUM> and/or may verify an identifier associated with a control message <NUM>. For example, upon receiving a control message <NUM>, the authority tracker <NUM> may transmit a verification request to the directory service <NUM> to ensure that an identifier associated with the control message <NUM> corresponds to an actor registered in the directory records <NUM> and/or that the identifier is associated with a role necessary for authority to transition to the actor (e.g., the control message <NUM> specifies an action, such as escalation, that requires that the actor be associated with a "supervisor" role). The directory service <NUM> my provide up-to-date information on role changes and/or authorized users, may track which users are online, and/or may store and push configuration data such as a keep-alive message interval, a timeout period, vehicle-specific settings, etc. In additional or alternate examples, the directory service <NUM> may associate a geographical area with an actor, such that the actor is only able to control vehicles in the geographical area, and/or Internal protocol (IP) address restrictions that only allow certain IP address(es) and/or IP address ranges and/or subnets to be verified by the directory service <NUM>. In some instances, a vehicle may be hard-coded to only respond to a range of IP addresses and/or actor(s) associated with a geographical area. The directory service <NUM> may issue public key/private key pairs and/or certificate(s) to an actor and/or to a vehicle for the vehicle to verify the identity of an actor. In some instances, the directory service <NUM> and/or the directory records <NUM> may be stored in memory <NUM>. For example, guidance application <NUM> may use directory service <NUM> to cryptographically sign the control message <NUM> with a certificate associated with an actor (which may be included in the user information <NUM>) and the authority tracker <NUM> may verify the signature (using the certificate received in the user information <NUM>) and/or associate the control message <NUM> with a username before determining whether to authorize an action specified by the control message <NUM>.

In some instances, aspects of some or all of the components discussed herein may include any models, algorithms, and/or machine learning algorithms.

It should be noted that while <FIG> is illustrated as a distributed system, in alternative examples, components of the computing device <NUM> may be associated with the computing device(s) <NUM> and/or <NUM> and/or components of the computing device(s) <NUM> and/or <NUM> may be associated with the computing device <NUM> and/or each other. That is, the computing device <NUM> may perform one or more of the functions associated with the computing device(s) <NUM> and/or <NUM>, and vice versa.

<FIG> illustrates a pictorial flow diagram of an example process <NUM> for granting authority to an actor, transitioning authority between actors, and/or tracking which actor has authority. Example process <NUM> may be performed by computing device(s) such as vehicle computing device(s) <NUM> and/or computing device(s) <NUM>.

At operation <NUM>, example process <NUM> may include storing a first state <NUM> identifying (<NUM>) a first actor (e.g., human "x" in the diagram and/or a computing device associated therewith) as having authority to provide guidance to an autonomous vehicle, according to any of the techniques discussed herein. In some instances, this may comprise storing the first state <NUM> in a memory at an autonomous vehicle. The autonomous vehicle may periodically and/or upon request provide at least part of the first state <NUM> as a state message broadcast over the state channel.

At operation <NUM>, example process <NUM> may include storing a policy specifying one or more conditions that, upon satisfaction of one or more conditions, may cause the authority to transition to another actor other than the first actor, according to any of the techniques discussed herein. <FIG> depicts a representation of an example policy <NUM>, which may comprise a state machine. The example policy <NUM> depicts states as circles, transitions as arrows, and conditions as hexagons. The bolded state of the policy indicates an example of a current state <NUM> associated with first state <NUM>. Example policy <NUM> is non-limiting and has been simplified for the sake of clarity. In some instances, one condition may be associated with one transition and there may be more than one transition associated with each state.

For example, in the example policy <NUM>, the current state <NUM> identifies the engaged actor as being an actor other than the vehicle and that the transition to this engaged actor was successful based on a previously received request from the engaged actor (i.e., identified as "actor, success" in the figure for simplicity). The current state <NUM> may transition to an "actor, awaiting handoff" state <NUM> if a handoff message <NUM> is received from the engaged actor, which may be a condition of that transition. Once in state <NUM>, state <NUM> may transition back to state <NUM> if a takeover message <NUM> is received from an actor identified in the handoff message <NUM>. Note that, if this were to happen, upon reverting to state <NUM>, the state would replace an identifier of the formerly engaged actor with an identifier of the newly engaged actor, which is not depicted for clarity.

Alternatively, the current state <NUM> may transition to an "actor, escalating" state <NUM> if an escalation message <NUM> is received from the engaged actor, and the state may transition from an "actor, escalating" state <NUM> to a "supervisor, success" state <NUM> if a takeover message is received from an actor associated with the role "supervisor" (<NUM>). As described in more detail below, an "actor, escalating" state <NUM> may be associated with a search for one or more supervisor(s) that are online and/or otherwise available (and/or that are associated with the engaged actor), based at least in part on data available at a director service. In some examples, the handoff message <NUM> may be associated with a search for actor(s) that have a same role and/or privileges as the engaged actor.

Alternatively, the current state may revert back to a default state specifying the vehicle as having the authority (<NUM>) if a relinquish message is received from the engaged actor (<NUM>). Note that, for the sake of clarity, policy <NUM> is merely an example policy and all possible connections are not shown. For example, when a supervisor relinquishes authority, it may revert to a previous actor or to the vehicle. Moreover, in some instances, a supervisor may take authority from actor(s) with which the supervisor is associated and authority may transition to that supervisor without a previous escalation message having been received. In other words, in some instances, a supervisor may be able to take authority from actor(s) that they supervise, at any time, when one of those actor(s) currently has authority. This may provide a backup for incapacitated, aberrant, or otherwise undesirable actor behavior.

In some instances, more permutations of states, transitions, and/or conditions may be included in the policy <NUM> and the actual states, transitions, and/or condition may vary in practice. A condition may specify at least one of a role, a state (e.g., an origin state and/or a destination state), an action specified by a control message (e.g., a takeover message, a handoff message), a verification policy (e.g., an identity of the requesting actor must be authenticated by the directory service), etc..

At operation <NUM>, example process <NUM> may include receiving a request to transfer the authority to a second actor, according to any of the techniques discussed herein. For example, the request may comprise a control message. <FIG> depicts a representation of an example control message <NUM> that includes an action comprising a handoff message, a target vehicle identifier, a source identifier, and a target identifier. This example control message <NUM> may convey that Actor "x" requests to transition to Actor "<NUM>" authority to provide guidance to "Vehicle <NUM>.

At operation <NUM>, example process <NUM> may include determining whether a condition of the policy is satisfied and/or whether to authorize or deny the request, according to any of the techniques discussed herein. Operation <NUM> may comprise determining a current state (e.g., retrieving the currently stored state from memory) and identifying, from the policy, one or more conditions associated with the state. In the depicted example, this may include determining that the current state is state <NUM> and that conditions <NUM>, <NUM>, and <NUM> are associated with the state (e.g., those are all the conditions associated with transitions away from the current state <NUM> given in the example so any transition allowed by example policy <NUM> will have to satisfy one of those states for authority to transition). Determining whether a condition is satisfied may additionally or alternatively include determining whether a role associated with the request is associated with a sufficient privilege to issue the request and/or to give guidance of a certain type. For example, a supervisor role may be associated with a privilege to cause a door of the vehicle to open even while the vehicle is moving (e.g., if an emergency event is detected at the vehicle).

At operation <NUM>, example process <NUM> may include determining to authorize the request based at least in part on the policy, according to any of the techniques discussed herein. In the depicted example, this may include determining that Actor <NUM> and/or Actor X are authenticated/verified by a directory service and/or that a handoff message is associated with a valid transition. Upon determining to authorize the request, operation <NUM> may further comprise storing a second state <NUM> identifying (<NUM>) the second actor as having authority. In some instances, the second state may replace the first state in the memory. Operation <NUM> may additionally, or alternatively, comprise transmitting the second state over a state channel instead of the first state.

Responsive to receiving authorization and/or the second state <NUM>, a guidance application associated with the actor may permit the second actor to provide guidance to the autonomous vehicle. Additionally, or alternatively, responsive to storing the second state <NUM>, the autonomous vehicle may generate a trajectory for controlling motion of the autonomous vehicle based at least in part on guidance received from the second actor.

At operation <NUM>, example process <NUM> may include determining to deny the request based at least in part on the policy, according to any of the techniques discussed herein. For example, in the illustrated example, this may include determining that Actor <NUM> and/or Actor x are not authenticated/verified by a directory service and/or that a current state is not associated with a handoff message (e.g., the current state doesn't allow handoffs therefrom). Based on determining to deny the request, the first state <NUM> may remain unmodified in the memory in which it is stored, identifying (<NUM>) Actor x as having authority.

<FIG>, <FIG>, and <FIG> illustrate block diagrams of example scenarios in which various authority transitions and/or authority isolations may take place, while <FIG>, <FIG>, and <FIG> illustrate example user interfaces that may be generated responsive to a computing device receiving a state message and/or that may cause generation and/or transmission of a control message. <FIG>, <FIG>, and <FIG> depict an autonomous vehicle <NUM> and actor(s) <NUM>(<NUM>)-(x) and computing devices <NUM>(<NUM>)-(x) associated therewith. Note that autonomous vehicle <NUM> may itself be an "actor" as well.

In some instances, a computing device (any one of <NUM>(<NUM>)-(x)) may transmit a control message over a control channel <NUM> and any subscriber to the control channel <NUM> may receive the control message, according to a pub/sub configuration although any other method of transmitting messages from an actor device to the autonomous vehicle <NUM> are contemplated. In some instances, the autonomous vehicle <NUM> and/or an authority tracker <NUM> of the autonomous vehicle <NUM> may be a subscriber to the control channel <NUM> (as illustrated by the arrow emanating from the control channel <NUM> to the authority tracker <NUM>). In some instances, the autonomous vehicle <NUM> may periodically, in response to a request received, and/or at any other time transmit a state message via a state channel <NUM> and any subscriber to the state channel <NUM> may receive the state message, according to a pub/sub configuration although any other method of transmitting messages from the autonomous vehicle <NUM> to computing device(s) <NUM>(<NUM>)-(x) are contemplated. In some instances, computing device(s) <NUM>(<NUM>)-(x) subscribe to the state channel <NUM> and thereby receive messages broadcasted over the state channel <NUM>.

<FIG> depicts an example scenario <NUM> in which actor <NUM>(<NUM>) transmits, over the control channel <NUM>, a takeover message <NUM>. In the example scenario <NUM>, a state <NUM> stored at the autonomous vehicle <NUM> identifies the autonomous vehicle <NUM> as having authority. In some instances, based at least in part on receiving the takeover message <NUM>, the authority tracker <NUM> may identify one or more conditions and transition(s) associated therewith in the policy based at least in part on the state <NUM>. In some instances, the policy may allow any actor to takeover when the vehicle is identified as having authority. In other instances, the policy may require that, for an actor to be granted authority, the autonomous vehicle <NUM> must have previously transmitted a request for guidance. In additional or alternate examples, the policy may specify event(s), role(s), etc. for which an actor may be authorized to take authority. For example, a supervisor role may take authority at any point, vehicle(s) may be assigned to specific actor(s) and/or supervisor(s) such that only those actor(s) and/or supervisor(s) may take over unless all of the assigned actor(s) and supervisor(s) are occupied, actor(s) may be prevented from taking authority unless the vehicle detects a safety critical event and/or generates a teleoperations request, etc..

<FIG> depicts a continuation of example scenario <NUM>. The authority tracker <NUM> may determine, based at least in part on the policy, to authorize (<NUM>) the takeover message. In some instances, the authority tracker <NUM> may determine that a condition associated with a transition in the policy has been satisfied by the takeover message <NUM>. The authority tracker <NUM> may then cause state <NUM> to be updated to identify actor <NUM>(<NUM>) as having authority to provide guidance to the autonomous vehicle <NUM>. This update may include replacing state <NUM> with state <NUM> in a memory of the autonomous vehicle <NUM>. Based at least in part on authorizing the takeover message <NUM>, the authority tracker <NUM> may cause a state message <NUM> to be broadcast over the state channel <NUM> that includes at least part of the information stored in state <NUM>. For example, state message <NUM> may identify actor <NUM>(<NUM>) as having authority to provide guidance to the autonomous vehicle <NUM>.

In some instances, based at least in part on receiving the state message <NUM> via the state channel <NUM>, the computing device <NUM>(<NUM>) may determine that the state message <NUM> identifies the actor <NUM>(<NUM>) and/or the computing device <NUM>(<NUM>) and, in response, may request and/or receive sensor data from the autonomous vehicle <NUM>. Upon receiving the state message <NUM>, the computing device <NUM>(<NUM>) may begin to transmit keepalive messages to the autonomous vehicle <NUM> and the autonomous vehicle <NUM> may maintain the state <NUM> until a control message is received over the control channel <NUM> that causes authority to transition to another actor (e.g., a handover message is received followed by a takeover message by another actor, an escalation message is received followed by a takeover message by a supervisor, a relinquish message is received from actor <NUM>(<NUM>)) or until the connection times out (e.g., the autonomous vehicle <NUM> determines that a keepalive message has not been received within a timeout period). If the connection times out, the authority tracker <NUM> may revert authority to a previous holder of authority and/or to a default holder of authority (e.g. the autonomous vehicle <NUM>). In some instances, based at least in part on receiving the state message <NUM> computing device <NUM>(<NUM>) may additionally, or alternatively, cause a user interface to be displayed by the computing device <NUM>(<NUM>) and/or to cause components of the user interface to transition from being unselectable (e.g., grayed-out, not displayed) to being selectable. In an additional or alternate example, based at least in part on receiving the state message <NUM>, the computing device <NUM>(<NUM>) may instantiate a process on the computing device <NUM>(<NUM>). This process may allow an actor to provide guidance to an autonomous vehicle and may cause a user interface to be displayed by a display associated with the computing device <NUM>(<NUM>).

<FIG> depicts an example user interface <NUM> that the computing device <NUM>(<NUM>) may generate and cause to be displayed, based at least in part on receiving the state message <NUM>. In some instances, the computing device <NUM>(<NUM>) may have caused the user interface <NUM> to display notification <NUM> based at least in part on receiving a request for guidance from the autonomous vehicle <NUM>. Notification <NUM> may have comprised two selectable elements, "Accept" and/or "Queue. " In some instances, input to the computing device <NUM>(<NUM>) corresponding to the selectable element "Accept" may have caused the computing device <NUM>(<NUM>) to transmit the takeover message <NUM> over the control channel <NUM>. Input corresponding to the selectable element "Queue" may cause the computing device <NUM>(<NUM>) to take no action regarding the request for guidance or may cause the computing device <NUM>(<NUM>) to transmit the request for guidance to another computing device associated with an actor that is online. However, notification <NUM> may be displayed contemporaneously and/or before sensor data is received from the vehicle <NUM> for causing display of at least part of the user interface <NUM>.

Actor <NUM>(<NUM>) may use the user interface <NUM> to provide input to the computing device <NUM>(<NUM>) that the computing device <NUM>(<NUM>) translates into guidance, which may comprise computer-executable instructions and/or logic comprising a command, collaborative instruction, and/or a confirmation that may be used as guidance by the autonomous vehicle <NUM>. For example, the example user interface <NUM> may comprise a birds-eye representation <NUM> of sensor data and/or guidance, a camera feed <NUM> (e.g., an image, a video), and/or a top-down representation <NUM> of the sensor data and/or guidance. In the depicted example user interface <NUM>, the guidance comprises a modified driving boundary <NUM>. In some examples, the actor may have provided input sufficient to create the shape of the modified driving boundary <NUM>. The modified driving boundary <NUM> is an example of a collaborative instruction and is not intended to limit the types of guidance that may be provided using user interface <NUM>. Examples of additional or alternate guidance and/or teleoperations operations may be found in <CIT>, <CIT>, <CIT>, and <CIT>.

Upon receiving input corresponding to guidance, the guidance application may cause a confirmation and/or some other selectable element (<NUM>) to be displayed via user interface <NUM>, the selection of which causes the guidance to be transmitted to the autonomous vehicle <NUM>. In some instances, the guidance application may append to the guidance an identifier associated with the actor <NUM>(<NUM>). The authority tracker <NUM> may transmit, to the appropriate component of the autonomous vehicle <NUM>, the guidance received from the computing device <NUM>(<NUM>) of the actor <NUM>(<NUM>) based at least in part on determining that the identifier corresponds to state <NUM> (e.g., the state <NUM> includes the identifier, thereby indicating that actor <NUM>(<NUM>) has authority to provide the guidance). For example, based at least in part on receiving instructions corresponding to the modified driving boundary <NUM>, the authority tracker <NUM> may transmit the instructions corresponding to the modified driving boundary <NUM> to a planner of the autonomous vehicle <NUM>. The planner may, in turn, generate a trajectory for the autonomous vehicle <NUM> that may cause the autonomous vehicle <NUM> to operate within the modified driving boundary <NUM>.

In some examples, elements available via the user interface <NUM> may be further based at least in part on a role associated with an actor. For example, a supervisor role may be associated with a user interface that displays sensor data of a vehicle, even when that vehicle is being controlled by another actor, but that may have at least or all guidance controls made unavailable. In an additional or alternate example, the user interface associated with a supervisor may make unavailable guidance controls associated with a role of the engaged actor (e.g., drive boundary modification, trajectory confirmation, route modification), but may make available guidance controls associated with the role of supervisor but not associated with the role of the engaged actor (e.g., open a door while moving, stop vehicle on highway).

In some instances, based at least in part on a record of active user(s), the computing device <NUM>(<NUM>) may additionally, or alternatively, cause the user interface <NUM> to display an indication of other actor(s) that are online (<NUM>) and/or supervisors (e.g., actors associated with the role supervisor) that are online (<NUM>). In some instances, the online actor(s) and/or supervisor(s) may be generally identified (e.g., by identifying a number of actor(s) and/or supervisor(s) that are online, as depicted in example user interface <NUM>) and/or one or more of the online actor(s) and/or supervisor(s) may be specifically indicated (e.g., by username, by a picture or other representation of the actor). Input received at the computing device <NUM>(<NUM>) that corresponds to <NUM> (whether selecting online users generally or selecting a specific user) may cause a handoff procedure to be initiated and/or input that corresponds to <NUM> (whether selecting online supervisors indicator generally, or selecting a specific supervisor) may cause an escalation procedure to be initiated.

<FIG> depicts an example scenario <NUM> in which state <NUM> is still stored in memory, indicating that actor <NUM>(<NUM>) has authority to provide guidance to the autonomous vehicle <NUM>. In example scenario <NUM>, actor <NUM>(<NUM>) transmits, over the control channel <NUM>, a relinquish message <NUM>.

<FIG> depicts a continuation of example scenario <NUM>. The authority tracker <NUM> may authorize <NUM> the relinquish message <NUM>, based at least in part on determining that the relinquish message <NUM> satisfies a condition of the policy. The authority tracker <NUM> may cause the authority to transition by reverting to a previous authority holder and/or to revert to a default authority holder (e.g., autonomous vehicle <NUM>). This may comprise storing a state <NUM> in a memory of the autonomous vehicle <NUM>. State <NUM> may represent state <NUM>, although state <NUM> may additionally indicate that the reversion was successful (e.g., "vehicle, success"). In some instances, the authority tracker <NUM> may transmit a state message <NUM> indicating the new authority holder (i.e., autonomous vehicle <NUM> in the depicted example) over the state channel <NUM>. Any subscribers to the state channel <NUM> may receive the state message <NUM> (e.g., computing device(s) <NUM>(<NUM>)-(x)). In some instances, upon receiving the state message <NUM>, computing device <NUM>(<NUM>) may terminate a process that allows actor <NUM>(<NUM>) to provide guidance and/or may cause component(s) of a user interface to be unavailable for selection.

In some instances, after transmitting the relinquish message <NUM> the computing device <NUM>(<NUM>) may keep a connection to the autonomous vehicle <NUM> open until the computing device <NUM>(<NUM>) receives a state message indicating that a transition of authority has successfully occurred, even if this was due to a timeout. In some instances, the actor <NUM>(<NUM>) may retain authority until the state message <NUM> is received by at least one of the computing device(s) <NUM>(<NUM>)-(x) and/or until a timeout occurs (and the authority tracker <NUM> thereby causes an authority transition due to the timeout).

In cases where the current state indicates that a supervisor has authority and a relinquish message has been received, the authority tracker <NUM> may determine to revert authority to a default authority holder and/or to a previous authority holder (e.g., an actor that escalated authority to the supervisor).

<FIG> depicts an example scenario <NUM> in which state <NUM> is still stored in memory, indicating that actor <NUM>(<NUM>) has authority to provide guidance to the autonomous vehicle <NUM>. In example scenario <NUM>, actor <NUM>(x) transmits, over the control channel <NUM>, a takeover message <NUM>.

<FIG> depicts a continuation of example scenario <NUM>. The authority tracker <NUM> may deny <NUM> the takeover message <NUM>, based at least in part on determining that the takeover message <NUM> does not satisfy a condition of the policy. For example, the authority tracker <NUM> may determine to deny the takeover message <NUM> based at least in part on reading state <NUM> and determining that actor <NUM>(<NUM>) currently has authority. Depending on the policy configuration, however, if the takeover message <NUM> identifies actor <NUM>(x) as a supervisor, according to some policy configurations, the authority tracker <NUM> may authorize the takeover message <NUM> and transition the state <NUM> to a state identifying actor <NUM>(x) as the authority holder. In other policy configurations, a takeover message is never authorized unless authority is being transitioned from a default authority holder or a message was received notifying the authority tracker <NUM> of a requested transition (e.g., a handover message, an escalation message).

In some instances, a supervisor may cause a takeover by transmitting from computing device <NUM>(x) a request to computing device <NUM>(<NUM>) that, upon acknowledgement and/or authorization of actor <NUM>(<NUM>) (e.g., such as by providing input corresponding to a notification of the request displayed via the user interface of the guidance application), the computing device <NUM>(<NUM>) may transmit an escalation message to the authority tracker <NUM> (over the control channel <NUM>) and/or an acknowledgement message back to the computing device <NUM>(x). The computing device <NUM>(x) may, responsive to receiving the acknowledgement message or after a defined time period has passed, transmit the takeover message <NUM>.

The authority tracker <NUM> may cause a state message <NUM> to be transmitted over the state channel <NUM> based at least in part on denying <NUM> the request and/or at a regular state transmission interval. The state message <NUM> may include at least part of state <NUM>, thereby continuing to identify actor <NUM>(<NUM>) as having authority. The state message <NUM> may be received by subscriber(s) to the state channel <NUM>, including computing device <NUM>(<NUM>) and/or computing device <NUM>(x).

In some instances, based at least in part on receiving state message <NUM>, computing device <NUM>(x) may cause selectable elements for providing guidance via a user interface to remain unavailable. In some instances, computing device <NUM>(x) may nonetheless cause sensor data received from the autonomous vehicle <NUM> to be displayed via the user interface, although in other cases, computing device <NUM>(x) may prevent this.

<FIG> depicts an example scenario <NUM> in which state <NUM> is still stored in memory, indicating that actor <NUM>(<NUM>) has authority to provide guidance to the autonomous vehicle <NUM>. In example scenario <NUM>, actor <NUM>(<NUM>) transmits, over the control channel <NUM>, a handover message and/or escalation message <NUM>. In some instances, a handover message and/or an escalation message may include both a source identifier (e.g., a username or other identifier of the originating actor) and a target identifier (e.g., an identifier of a specific actor and/or a role to which authority is requested to transition). The target identifier may include, for example, a username of a specific actor and/or a role.

In some instances, the computing device <NUM>(<NUM>) may transmit a handover/escalation notification <NUM> substantially simultaneously or before transmitting the handover message and/or escalation message <NUM> to computing device(s) associated with the target identifier.

In some instances, the computing device <NUM>(<NUM>) may generate and/or transmit handover message and/or escalation message <NUM> without receiving user input. For example, if input is not received at the computing device <NUM>(<NUM>) within a time period of having received a state message identifying the actor <NUM>(<NUM>) as having authority, the computing device <NUM>(<NUM>) may generate and/or transmit a handover message identifying, as a target, a peer of the actor <NUM>(<NUM>) (e.g., another actor having a same role as the actor <NUM>(<NUM>)), a same role as the actor <NUM>(<NUM>), etc., and/or an escalation message <NUM>.

In some instances, the computing device <NUM>(<NUM>) may auto-populate the target identifier with identifier(s) associated with one or more online actors (for handoffs) or supervisors (for escalations) and/or a predetermined number thereof.

<FIG> depicts an example user interface <NUM> for generating and/or transmitting a handoff message and/or escalation message. The example user interface <NUM> may comprise a selectable element <NUM> ("Handover") to initiate a handover. It is understood that this element may be substituted for an element that similarly initiates an escalation. Responsive to receiving input at the computing device <NUM>(<NUM>) that corresponds to selectable element <NUM>, the guidance application may identify at least a subset of actors that are online and, in the case of escalations, that are associated with the role of supervisor. In some instances, the computing device <NUM>(<NUM>) may cause the user interface <NUM> to display a list of selectable elements corresponding to peer(s) <NUM> (in the case of a handoff) and/or supervisor(s) <NUM> (in the case of an escalation) that correspond to the subset of actors identified by the guidance application. Input corresponding to one or more of the selectable elements <NUM> and/or <NUM> may cause the computing device <NUM>(<NUM>) to transmit the handover message and/or escalation message <NUM> comprising a target identifier that identifies the actor(s) selected. The computing device <NUM>(<NUM>) may additionally transmit handover/escalation notification <NUM> to computing device(s) associated with the actor(s) identified by the target identifier of the handover message and/or escalation message <NUM>.

In an additional or alternate example, responsive to receiving input at the computing device <NUM>(<NUM>) that corresponds to selectable element <NUM>, the guidance application may identify at least a subset of actors that are online and, in the case of escalations, that are associated with the role of supervisor, and the guidance application may automatically cause a handover message and/or escalation message <NUM> and/or handover/escalation notification <NUM> to be transmitted without further input of an actor. For example, the guidance application may auto-populate the target identifier of handover message and/or escalation message <NUM> with identifier(s) associated with one or more online actors (for handoffs) or supervisors (for escalations) and/or a predetermined number thereof. In some instances, the guidance application may identify supervisor(s) that are associated with actor <NUM>(<NUM>) and, if supervisor(s) associated with actor <NUM>(<NUM>) are not indicated as being available, the guidance application may expand a search for an available supervisors to supervisor(s) that are not associated with the actor <NUM>(<NUM>).

<FIG> depicts an example user interface <NUM> caused to be displayed by a computing device that receives handover/escalation notification <NUM>. The example user interface <NUM> may cause a notification <NUM> to be displayed over at least a portion of an existing display (e.g., the birds-eye view of a representation of a fleet of autonomous vehicles operating in a city). In some instances, the notification <NUM> may comprise selectable elements (e.g., indicated by "Accept" and "Queue" in the example user interface <NUM>). Input received at the computing device indicating an acceptance of the request to handover and/or escalate authority may cause the computing device to transmit a takeover message <NUM> over control channel <NUM> (depicted in <FIG>). Responsive to receiving the takeover message <NUM>, the authority tracker <NUM> may transition authority to an actor associated with the computing device. In response to transmitting the takeover message <NUM> and/or in response to receiving un updated state message, the computing device may instantiate a process and/or make components of a user interface available for selection. Input received at the computing device indicating a queue of the request may transmit the computing device to transmit a rejection message over the control channel <NUM> and/or may allow a user to select another actor to which to transmit the request and/or may transmit the request to additional actor(s). Responsive to receiving the takeover message <NUM>, the authority tracker <NUM> may transition the state to identify a "failure" and to continue to identify actor <NUM>(<NUM>) as having authority or, in some instances, revert authority to a default authority holder.

In examples where the target identifier identifies multiple actors, the authority tracker may transition authority to an actor associated with a takeover message that is received first and, if a rejection message is received, may maintain a "awaiting handoff" or "escalating" state until a transition occurs or until rejections are received from all the actors identified by the target identifier.

Claim 1:
One or more non-transitory computer-readable media storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
storing (<NUM>) a first state (<NUM>) identifying a computing device of an autonomous vehicle (<NUM>) as having exclusive authority to provide guidance to a component of the autonomous vehicle;
storing (<NUM>) a policy (<NUM>) specifying one or more conditions associated with transferring authority to an actor, the one or more conditions comprising a first condition requiring that a teleoperations request be previously received from the autonomous vehicle and at least one of an action, a role, or a state;
receiving (<NUM>), via a network interface and from a first device (<NUM>) associated with a first actor (<NUM>), a first request to transfer exclusive authority from the computing device of the autonomous vehicle to the first actor, the first request including at least one of an identifier specifying an identity or a first role of the first actor;
determining (<NUM>) to deny the first request based at least in part on determining that the autonomous vehicle had not transmitted the teleoperations request;
transmitting a teleoperations request;
receiving a second request to transfer authority to the first actor;
updating, based at least in part on determining that the teleoperations request was transmitted by the autonomous vehicle and that the identifier satisfies a condition specified by the policy stored in memory, the first state to a second state (<NUM>) identifying the first actor as having exclusive authority to provide guidance to the component of the autonomous vehicle;
transmitting, via the network interface and to at least the first device, the second state;
receiving a guidance from the first device via the network interface; and
controlling, based at least in part on the guidance received from the first device via the network interface and determining that the guidance is authorized by the second state, the component of the autonomous vehicle.