Patent Description:
An autonomous vehicle may navigate along designated routes or between waypoints. For example, when a control system receives a request from a user device to pick up the user at a location and provide transport to a destination location, the autonomous vehicle may receive, from the control system, instructions to navigate from the pickup location to the destination location. However, in some circumstances, progress of the autonomous vehicle navigating to the pickup location or the destination location may be stopped, such as by another vehicle. This may cause problems, such as delaying the autonomous vehicle or causing the autonomous vehicle to block the flow of traffic. <CIT> discloses systems and methods for remotely assisting an autonomous vehicle. The method includes: aggregating sensor data from the autonomous vehicle; identifying an assistance-desired scenario; generating an assistance request based on the sensor data; transmitting the assistance request to a remote assistance interface; and receiving and processing a response to the assistance request. The remote assistance interface includes a remote assistance interface that is used in generating the response to the assistance request. <CIT> discloses an autonomous vehicle which can implement a primary motion planner to continuously determine a first motion plan for the AV, and a secondary motion planner to continuously determine a backup motion plan for the AV. The secondary motion planner can comprise one or more cost metrics that act to diverge the backup motion plan from the first motion plan. A control system of the AV may then analyze a live sensor view generated by a sensor suite of the AV to operate acceleration, braking, and steering systems of the AV along sequential route trajectories selected between the first motion plan and the backup motion plan. <CIT> discloses an autonomous vehicle (AV) which can dynamically analyze sensor data from a sensor suite to autonomously operate acceleration, braking, and steering systems along a current route. In analyzing the sensor data, the AV can determine a teleassist state requiring remote human assistance, and determine a plurality of decision options to resolve the teleassist state. The AV may then generate a teleassistance data package corresponding to the plurality of decision options, and transmit the teleassistance data package to a remote teleassistance system to enable a human operator to select one of the plurality of decision options for execution by the AV.

In the figures, the leftmost digit(s) of a reference number identifies the figure in which the reference number first appears.

As discussed above, an autonomous vehicle may navigate along designation routes or between waypoints, such as by navigating from a pickup location to a destination location. However, in some circumstances, progress of the autonomous vehicle may be impeded. For a first example, the autonomous vehicle may store data representing a policy instructing the autonomous vehicle to yield to other vehicles that arrive at an intersection before the autonomous vehicle. If the autonomous vehicle is yielding to another vehicle at an intersection, where the other vehicle is not moving for one or more reasons (e.g., the other vehicle cannot navigate along a route until the autonomous vehicle moves due to an obstruction or narrow road), then the progress of the autonomous vehicle may be impeded. For a second example, the autonomous vehicle may set an interaction associated with another vehicle, where the interaction causes the autonomous vehicle to follow the other vehicle. If the other vehicle stops, such as by parking or breaking down, then the progress of the autonomous vehicle may be impeded.

It describes techniques for determining when to contact a teleoperator as well as techniques for navigating the autonomous vehicle using instructions that are received from the teleoperator. For example, when progress of the autonomous vehicle is impeded, the autonomous vehicle may determine an amount of time that the progress has been impeded. The autonomous vehicle may then determine whether the amount of time satisfies (e.g., is equal to or greater than) a threshold amount of time. The threshold amount of time may include, but is not limited to, three seconds, thirty seconds, one minute, three minutes, and/or any other period of time. If the autonomous vehicle determines that the amount of time does not satisfy (e.g., is below) the threshold amount of time, then the autonomous vehicle may determine not to contact the teleoperator. However, if the autonomous vehicle determines that the amount of time satisfies the threshold amount of time, then the autonomous vehicle may determine if the progress of the autonomous vehicle is impeded and/or the progress of the object is impeded due to an intervening condition.

In line with the invention, the autonomous vehicle determines if the progress of the autonomous vehicle is impeded because of traffic. In some instances, the autonomous vehicle may make the determination based at least in part on determining whether progress of another vehicle, in addition to the vehicle that the autonomous vehicle is yielding to, is also impeded. For instance, when the autonomous vehicle is navigating along a multilane road, the autonomous vehicle may determine if progress of another vehicle in another lane has also impeded. If the autonomous vehicle determines that the progress of another vehicle has also impeded, then the autonomous vehicle may determine that traffic has caused the progress of the autonomous vehicle to be impeded. As such, the autonomous vehicle may determine not to contact the teleoperator. However, if the autonomous vehicle determines that the progress of another vehicle has not been impeded and/or there is not another vehicle next to the autonomous vehicle, then the autonomous vehicle may determine that traffic has not caused the progress of the autonomous vehicle to be impeded. As such, the autonomous vehicle may determine to contact the teleoperator.

For a second example, the autonomous vehicle may determine if the progress of the autonomous vehicle is impeded because of a traffic light. In some instances, the autonomous vehicle may use sensor data to determine if a traffic light is causing the progress of the autonomous vehicle to be impeded. In line with the invention, the autonomous vehicle uses map data to determine if a traffic light is causing the progress of the autonomous vehicle to be impeded. In either instance, if the autonomous vehicle determines that a traffic light is causing the progress of the autonomous vehicle to be impeded, then the autonomous vehicle may determine not to contact the teleoperator. However, if the autonomous vehicle determines that a traffic light is not causing the progress of the autonomous vehicle to be impeded, then the autonomous vehicle may determine to contact the teleoperator.

When determining to contact the teleoperator, the autonomous vehicle may send sensor data to one or more computing devices associated with the teleoperator. The sensor data may include, but is not limited to, image data, light detection and ranging (lidar) data, radar data, and/or other types of sensor data representing the environment around the autonomous vehicle. In some instances, the autonomous vehicle may further send, to the one or more computing devices, data indicating which object(s) (e.g., vehicle(s), pedestrians, signs, construction equipment, etc.) are causing the progress of the autonomous vehicle to be impeded. For example, if the autonomous vehicle is yielding to another vehicle, the data may indicate that the autonomous vehicle is currently yielding to the other vehicle. In some instances, the autonomous vehicle may send, to the one or more computing devices, data indicating the amount of time that the progress of the autonomous vehicle has been impeded and/or the amount of time that the autonomous vehicle has been yielding to the object(s).

The one or more computing devices may receive the sensor data from the autonomous vehicle and, in response, display a user interface that includes content (e.g., image(s), video, etc.) represented by the sensor data. Additionally, the one or more computing devices may display graphical elements indicating which object(s) are causing the progress of the autonomous vehicle to be impeded. For example, the user interface may display a graphical element that indicates that the autonomous vehicle is yielding to an object. The teleoperator may then use the one or more computing devices to send instructions to the autonomous vehicle, where the instructions indicate how the autonomous vehicle is to proceed. In at least some examples, such instructions may comprise an instruction to disregard and/or alter a policy (e.g., disregard the yield instruction to a selected object in the environment).

For a first example, and if the autonomous vehicle is yielding to object(s) at an intersection, where the object(s) are not moving for one or more reasons, then the instructions may cause the autonomous vehicle to no longer yield to the object(s) and/or cause the autonomous vehicle to temporarily cease following (and/or alter) one or more policies. The one or more policies may include, but are not limited to, a first policy to yield to vehicles that arrive at an intersection before the autonomous vehicle, a second policy to travel in a road lane, and/or one or more additional policies. In some instances, temporarily ceasing following the one or more policies may include ceasing following the one or more policies for a period of time. The period of time may include, but is not limited to, five seconds, ten seconds, thirty seconds, one minute, and/or another period of time. Additionally, or alternatively, in some instances, temporarily ceasing following the one or more policies may include ceasing following the one or more policies until the autonomous vehicle navigates to a given location, navigates a given distance, navigates to a location at which the autonomous vehicle would no longer be yielding to the object(s), and/or the like. Regardless, using the instructions, the autonomous vehicle may determine to no longer yield to the object(s) and/or determine to temporarily cease following the one or more policies. As such, the autonomous vehicle may again begin navigating to a location.

For a second example, and if the autonomous vehicle is yielding to another vehicle for which the autonomous vehicle has a current interaction, such as "follow", then the instruction may cause the autonomous vehicle to change the interaction with the other vehicle. For instance, the instruction may cause the interaction to change from "follow" to "do not follow". Additionally, in some instances, the instruction may cause the autonomous vehicle to temporarily cease following one or more polices. For instance, the instructions may cause the autonomous vehicle to temporarily cease following the second policy, which includes traveling in a road lane, so that the autonomous vehicle can navigate around the other vehicle. Using the instructions, the autonomous vehicle may change the interaction and/or determine to temporarily cease following the one or more policies. As such, the autonomous vehicle may again begin navigating to a location.

In some instances, once the autonomous vehicle begins navigating after receiving the instructions, the autonomous vehicle may determine to once again yield to the object(s). For instance, the autonomous vehicle may determine to once again follow the one or more policies or change the interaction with the object(s) back to the original interaction setting. The autonomous vehicle may make the determination based at least in part on a movement of the object(s).

For a first example, the autonomous vehicle may determine that a velocity associated the object(s) has changed. For instance, the velocity may change from zero miles per hour to a velocity that is greater than zero miles per hour. Based at least in part on the change in the velocity, the autonomous vehicle may determine that the object(s) are starting to move. As such, the autonomous vehicle may determine to once again yield to the object(s) and/or change the interaction with the object(s) back to "follow". Additionally, in some instances, the autonomous vehicle may again contact the teleoperator.

For a second example, the object(s) may have originally been moving when the autonomous vehicle originally yielded to the object(s), however, the object(s) may have remained in a given area. For instance, the object(s) may include a construction vehicle (e.g., backhoe, paver, etc.) that is moving back-and-forth within the given area. In such an example, the instructions sent by the teleoperator may include a specified area around the object(s) for which the object(s) are expected to remain. If the object(s) move outside of the specified area, then the autonomous vehicle may determine to once again yield to the object(s). Additionally, the autonomous vehicle may once again contact the teleoperator.

While the examples described above include determining that the progress of the autonomous vehicle is impeded using policies and/or interactions, in other instances, the progress of the autonomous vehicle may be impeded based on one or more other factors. For a first example, the autonomous vehicle may determine that the progress is impeded based on identifying a specific type of object. For instance, the specific type of object may include a tractor and/or streetcleaner that navigates at a slow velocity. As such, the progress of the autonomous vehicle may be impeded when the specific type of object is located along the route of the autonomous vehicle. For a second example, the autonomous vehicle may determine that the progress is impeded based on identifying a specific type of scenario. The scenario may include, but is not limited to, another vehicle merging into the drive lane of the autonomous vehicle, another vehicle navigating at a velocity that is below a threshold velocity (which is described below), an accident that occurs along the route of the autonomous vehicle, and/or the like.

The examples above describe the autonomous vehicle as determining if "progress" of the autonomous vehicle has been impeded. In some instances, the autonomous vehicle may determine that the progress has been impeded based at least in part on determining that the autonomous vehicle has stopped at a location (e.g., the velocity of the autonomous vehicle is zero miles per hour). Additionally, or alternatively, in some instances, the autonomous vehicle may determine that the progress has been impeded based at least in part on determining that the autonomous vehicle is only navigating a give distance within a given amount of time. For example, the autonomous vehicle may determine that the progress has been impeded based at least in part on determining that the autonomous vehicle is navigating one foot per second, one foot per five seconds, one foot per minute, and/or the like. Additionally, or alternatively, in some instances, the autonomous vehicle may determine that the progress has been impeded based at least in part on determining that the current velocity of the autonomous vehicle is below a threshold velocity. The threshold velocity may include, but is not limited to, a half a mile per hour, a mile per hour, and/or any other velocity.

Additionally, the techniques may provide the teleoperator with contextual information of the situation at the autonomous vehicle so that the teleoperator may provide guidance to the autonomous vehicle. A teleoperations system may include one or more teleoperators, which may be human teleoperators, located at a teleoperations center. In some examples, one or more of the teleoperators may not be human, such as, for example, they may be computer systems leveraging artificial intelligence, machine learning, and/or other decision-making strategies. In some examples, the teleoperator may interact with one or more autonomous vehicles in a fleet of autonomous vehicles via a user interface that can include a teleoperator interface. The teleoperator interface may include one or more displays configured to provide the teleoperator with data related to operations of the autonomous vehicles. For example, the display(s) may be configured to show content related to sensor data received from the autonomous vehicles, content related to the road network, and/or additional content or information to facilitate providing assistance to the autonomous vehicles.

Additionally, or alternatively, the teleoperator interface may also include a teleoperator input device configured to allow the teleoperator to provide information to one or more of the autonomous vehicles, for example, in the form of teleoperation instructions providing guidance to the autonomous vehicles. The teleoperator input device may include one or more of a touch-sensitive screen, a stylus, a mouse, a dial, a keypad, a microphone, a touchscreen, and/or a gesture-input system configured to translate gestures performed by the teleoperator into input commands for the teleoperator interface. As explained in more detail herein, the teleoperations system may provide one or more of the autonomous vehicles with guidance to avoid, maneuver around, or pass through events, such as when progress of the autonomous vehicles has been impeded.

The techniques described herein can be implemented in a number of ways. Example implementations are provided below with reference to the following figures. Although discussed in the context of a vehicle, the methods, apparatuses, and systems described herein can be applied to a variety of systems.

<FIG> is an example environment <NUM> that includes autonomous vehicles <NUM>(<NUM>)-(<NUM>) determining when to contact a teleoperator, in accordance with embodiments of the invention. For example, the first autonomous vehicle <NUM>(<NUM>) may be navigating along a route <NUM>. While navigating, the first autonomous vehicle <NUM>(<NUM>) may encounter a vehicle <NUM> that is also navigating along the route <NUM>. As such, the first autonomous vehicle <NUM>(<NUM>) may use sensor data <NUM> generated by one or more sensors of the first autonomous vehicle <NUM>(<NUM>) to identify the vehicle <NUM>. Additionally, since the vehicle <NUM> is located in front of the first autonomous vehicle <NUM>(<NUM>), and in the same driving lane as the first autonomous vehicle <NUM>(<NUM>), the first autonomous vehicle <NUM>(<NUM>) may set an interaction with the vehicle <NUM> as "follow", which is represented by the interaction data <NUM>.

However, as the first autonomous vehicle <NUM>(<NUM>) is navigating along the route <NUM> and following the vehicle <NUM>, the vehicle <NUM> may stop for one or more reasons. For example, the vehicle <NUM> may breakdown, causing the vehicle <NUM> to at least partially obstruct the road, as illustrated in the example of <FIG>. As such, and since the first autonomous vehicle <NUM>(<NUM>) is following the vehicle <NUM>, the first autonomous vehicle <NUM>(<NUM>) may also stop at a location <NUM> along the route <NUM>. This may cause the first autonomous vehicle <NUM>(<NUM>) to determine that the progress of the first autonomous vehicle <NUM>(<NUM>) has been impeded.

While the progress is impeded, the first autonomous vehicle <NUM>(<NUM>) may determine that an amount of time at which the progress has been impeded satisfies a threshold amount of time. The first autonomous vehicle <NUM>(<NUM>) may further determine if the progress has been impeded due to an intervening condition (e.g., traffic, a traffic light, etc.). In the example of <FIG>, the first autonomous vehicle <NUM>(<NUM>) may determine that the progress has not been impeded due to an intervening condition. Since the amount of time satisfies the threshold amount of time, and since the progress has not been impeded due to an intervening condition, the first autonomous vehicle <NUM>(<NUM>) may determine to contact the teleoperator.

To contact the teleoperator, the first autonomous vehicle <NUM>(<NUM>) may send, to a teleoperator system <NUM> associated with the teleoperator, at least a portion of the sensor data <NUM>. For instance, the first autonomous vehicle <NUM>(<NUM>) may send at least sensor data <NUM> (e.g., image data) representing the vehicle <NUM> and/or any derived data therefrom (e.g., detections, classifications, control data, etc.) to the teleoperator system <NUM>. In some instances, the first autonomous vehicle <NUM>(<NUM>) may further send, to the teleoperator system <NUM>, indicator data <NUM> that indicates that the progress has been impeded due to the vehicle <NUM> and/or that the current interaction with the vehicle <NUM> includes "follow". The teleoperator may use the data received from the first autonomous vehicle <NUM>(<NUM>) to provide guidance to the first autonomous vehicle <NUM>(<NUM>), which is described in the example of <FIG>.

Additionally, in the example of <FIG>, the second autonomous vehicle <NUM>(<NUM>) may be navigating along a route <NUM>. While navigating, the second autonomous vehicle <NUM>(<NUM>) may encounter an intersection, which causes the second autonomous vehicle <NUM>(<NUM>) to stop at a location <NUM>. Additionally, the second autonomous vehicle <NUM>(<NUM>) may use sensor data <NUM> generated by one or more sensors of the second autonomous vehicle <NUM>(<NUM>) to identify that the vehicle <NUM> arrived at the intersection before the second autonomous vehicle <NUM>(<NUM>). As such, and based at least in part on a policy represented by the policy data <NUM>, the second autonomous vehicle <NUM>(<NUM>) may determine to yield to the vehicle <NUM> at the intersection.

However, the vehicle <NUM>, which is navigating along a route <NUM>, may be unable to advance along the route <NUM> until after the second autonomous vehicle <NUM>(<NUM>) navigates along the route <NUM>. This is because, based at least in part on a width <NUM> of the second autonomous vehicle <NUM>(<NUM>) and a width <NUM> of a drive envelope along the route <NUM>, the second autonomous vehicle <NUM>(<NUM>) is at least partially obstructing the route <NUM> of the vehicle <NUM>. As such, the vehicle <NUM> may continue to wait for the second autonomous vehicle <NUM>(<NUM>) to navigate along the route <NUM> before the vehicle <NUM> navigates along the route <NUM>. Since the second autonomous vehicle <NUM>(<NUM>) is yielding to the vehicle <NUM>, the progress of the second autonomous vehicle <NUM>(<NUM>) may be impeded at the location <NUM>.

While the progress is impeded, the second autonomous vehicle <NUM>(<NUM>) may determine that an amount of time at which the progress has been impeded satisfies a threshold amount of time. The second autonomous vehicle <NUM>(<NUM>) may further determine if the progress has been impeded due to an intervening condition (e.g., traffic, a traffic light, etc.). In the example of <FIG>, the second autonomous vehicle <NUM>(<NUM>) may determine that the progress has not been impeded due to an intervening condition. Since the amount of time satisfies the threshold amount of time, and since the progress is has not been impeded due to an intervening condition, the second autonomous vehicle <NUM>(<NUM>) may determine to contact the teleoperator.

To contact the teleoperator, the second autonomous vehicle <NUM>(<NUM>) may send, to the teleoperator system <NUM> associated with the teleoperator, at least a portion of the sensor data <NUM> (and/or any data derived therefrom). For instance, the second autonomous vehicle <NUM>(<NUM>) may send at least sensor data <NUM> (e.g., image data) representing the vehicle <NUM> to the teleoperator system <NUM>. In some instances, the second autonomous vehicle <NUM>(<NUM>) may further send, to the teleoperator system <NUM>, indicator data <NUM> that indicates that the progress has been impeded due to the vehicle <NUM> and/or that the second autonomous vehicle <NUM>(<NUM>) is currently yielding to the vehicle <NUM>. Still, in some instances, the second autonomous vehicle <NUM>(<NUM>) may send, to the teleoperator system <NUM>, the policy data <NUM> indicating the policy that is causing the second autonomous vehicle <NUM>(<NUM>) to yield to the vehicle <NUM>. The teleoperator may then use the data received from the second autonomous vehicle <NUM>(<NUM>) to provide guidance to the second autonomous vehicle <NUM>(<NUM>), which is described in the example of <FIG>.

<FIG> is the example environment <NUM> that now includes the autonomous vehicles <NUM>(<NUM>)-(<NUM>) receiving instructions from the teleoperator, in accordance with embodiments of the invention. For example, the teleoperator may interact with the autonomous vehicle <NUM>(<NUM>)-(<NUM>) via a user interface that can include a teleoperator interface. The teleoperator interface may include one or more displays configured to provide the teleoperator with data related to operation of the autonomous vehicles <NUM>(<NUM>)-(<NUM>). For example, the display(s) may be configured to show content (e.g., image(s)) related to the sensor data <NUM> received from the first autonomous vehicle <NUM>(<NUM>). Additionally, the display(s) may be configured to shown content (e.g., image(s)) related to the sensor data <NUM> received from the second autonomous vehicle <NUM>(<NUM>).

In some instances, the display(s) may further be configured to show content related to object(s) that are causing progresses of the autonomous vehicles <NUM>(<NUM>)-(<NUM>) to be impeded. For example, the display(s) may use the indicator data <NUM> in order to show a graphical element indicating that it is the vehicle <NUM> that is causing the progress of the first autonomous vehicle <NUM>(<NUM>) to be impeded. Additionally, the display(s) may use the indicator data <NUM> to show a graphical element indicating that it is the vehicle <NUM> that is causing the progress of the second autonomous vehicle <NUM>(<NUM>) to be impeded. As described herein, a graphical element may include, but is not limited to, a bounding box around an object, shading an object, text describing the object, and/or any other graphical element that may indicate information about the object.

The teleoperator interface may also include a teleoperator input device configured to allow the teleoperator to provide information to the autonomous vehicles <NUM>(<NUM>)-(<NUM>), for example, in the form of teleoperation instructions providing guidance to the autonomous vehicles <NUM>(<NUM>)-(<NUM>). For example, the teleoperator may provide instructions, represented by the instruction data <NUM>, for the first autonomous vehicle <NUM>(<NUM>). The instructions may indicate that the first autonomous vehicle <NUM>(<NUM>) is to change the interaction with the vehicle <NUM> from "follow" to "do not follow". In some instances, the instructions may further indicate that the first autonomous vehicle <NUM>(<NUM>) may temporarily cease following one or more policies (and/or temporarily alter such policies), such as a policy specifying that the first autonomous vehicle <NUM>(<NUM>) is to maintain a road lane, in order for the first autonomous vehicle <NUM>(<NUM>) to navigate around the vehicle <NUM>.

The first autonomous vehicle <NUM>(<NUM>) may receive the instruction data <NUM> from the teleoperator system <NUM>. Using the instruction data <NUM>, the first autonomous vehicle <NUM>(<NUM>) may update the interaction with the vehicle <NUM> to indicate "do not follow". Additionally, in some instances, the first autonomous vehicle <NUM>(<NUM>) may temporarily cease following (or otherwise alter) the one or more policies. As such, and based at least in part on the instructions, the first autonomous vehicle <NUM>(<NUM>) may determine a route <NUM> to get around the vehicle <NUM>. After getting around the vehicle <NUM>, the first autonomous vehicle <NUM>(<NUM>) may again navigate along the route <NUM> of the first autonomous vehicle <NUM>(<NUM>).

In some instances, while navigating along the route <NUM> to get around the vehicle <NUM>, the first autonomous vehicle <NUM>(<NUM>) may determine if the vehicle <NUM> begins moving. For example, the first autonomous vehicle <NUM>(<NUM>) may use the sensor data <NUM> to determine if a velocity for the vehicle <NUM> changes from zero miles per hour to a velocity that is greater than zero miles per hour. In some instances, if the first autonomous vehicle <NUM>(<NUM>) determines that the vehicle <NUM> begins moving, the first autonomous vehicle <NUM>(<NUM>) may stop navigating along the route <NUM>. Additionally, the first autonomous vehicle <NUM>(<NUM>) may again update the interaction with the vehicle <NUM> to include "follow," thereby returning to the vehicle's nominal operating behavior.

Additionally, or alternatively, in some instances, the instructions from the teleoperator system <NUM> may indicate an area <NUM> around the vehicle <NUM>. In such instances, while navigating along the route <NUM> to get around the vehicle <NUM>, the first autonomous vehicle <NUM>(<NUM>) may determine if the vehicle <NUM> moves outside of the area <NUM>. For example, the first autonomous vehicle <NUM>(<NUM>) may use the sensor data <NUM> to determine if at least a portion of the vehicle <NUM> moves to outside of the area <NUM>. If the first autonomous vehicle <NUM>(<NUM>) determines that the vehicle <NUM> moves outside of the area <NUM>, the first autonomous vehicle <NUM>(<NUM>) may stop navigating along the route <NUM>. Additionally, the first autonomous vehicle <NUM>(<NUM>) may again update the interaction with the vehicle <NUM> to include "follow".

Additionally, the teleoperator may provide instructions, represented by the instruction data <NUM>, for the second autonomous vehicle <NUM>(<NUM>). The instructions may indicate that the second autonomous vehicle <NUM>(<NUM>) is to cease yielding for the vehicle <NUM>. In some instances, the instructions may further indicate that the second autonomous vehicle <NUM>(<NUM>) may temporarily cease following (or otherwise alter) one or more policies, such as a policy specifying that the second autonomous vehicle <NUM>(<NUM>) is to yield to vehicles that arrive at intersections before the second autonomous vehicle <NUM>(<NUM>).

The second autonomous vehicle <NUM>(<NUM>) may receive the instruction data <NUM> from the teleoperator system <NUM>. Using the instruction data <NUM>, the second autonomous vehicle <NUM>(<NUM>) may determine to no longer yield for the vehicle <NUM> at the intersection. Additionally, in some instances, the second autonomous vehicle <NUM>(<NUM>) may temporarily cease following the one or more policies. As such, and based at least in part on the instructions, the second autonomous vehicle <NUM>(<NUM>) may begin navigating along the route <NUM> before the vehicle <NUM> begins navigating along the route <NUM> in order to provide the vehicle <NUM> with room to navigate along the route <NUM>. As shown in the example of <FIG>, after the second autonomous vehicle <NUM>(<NUM>) begins to navigate along the route <NUM>, the vehicle <NUM> may then begin to navigate along the route <NUM>.

While the example of <FIG> describes the progress of the first autonomous vehicle <NUM>(<NUM>) as being impeded based on the vehicle <NUM>, which the first autonomous vehicle <NUM>(<NUM>) is following stopping, in other examples, the progress of the first autonomous vehicle <NUM>(<NUM>) may stop for other reasons. For example, first autonomous vehicle <NUM>(<NUM>) may analyze the sensor data <NUM> and, based on the analysis, the first autonomous vehicle <NUM>(<NUM>) may determine that the vehicle <NUM> includes a specific type of object. Based on the determination, the first autonomous vehicle <NUM>(<NUM>) may determine that the progress is impeded. For a second example, the first autonomous vehicle <NUM>(<NUM>) may analyze the sensor data <NUM> and, based on the analysis, the first autonomous vehicle <NUM>(<NUM>) may determine that a velocity of the vehicle <NUM> is below a threshold velocity. Based on the determination, the first autonomous vehicle <NUM>(<NUM>) may determine that the progress is impeded.

<FIG> is an example environment <NUM> that includes an autonomous vehicle <NUM> determining not to contact a teleoperator based at least in part on the autonomous vehicle <NUM> being stuck in traffic, in accordance with embodiments of the invention. For instance, the autonomous vehicle <NUM> may be navigating along a route <NUM> on a road that includes more than one drive lane. In line with the invention, the autonomous vehicle <NUM> uses sensor data and map data to determine that the road includes more than two drive lanes. For a first example, the autonomous vehicle <NUM> may analyze the sensor data to identify marking on the road that indicate multiple drive lanes in the same direction. For a second examples, the autonomous vehicle <NUM> may analyze the sensor data to identify another vehicle navigating next to the autonomous vehicle <NUM> and in a same direction as the autonomous vehicle <NUM>. For a third example, the map data may indicate that the road includes multiple drive lanes.

Additionally, other vehicles <NUM>(<NUM>)-(<NUM>) may respectively be navigating along routes <NUM>(<NUM>)-(<NUM>). While navigating, the autonomous vehicle <NUM> and the vehicles <NUM>(<NUM>)-(<NUM>) may get stuck in traffic. For instance, the progress of the autonomous vehicle <NUM> may be impeded at a location along the route <NUM>. Based at least in part on the progress being impeded, the autonomous vehicle <NUM> may determine whether to contact the teleoperator.

For instance, the autonomous vehicle <NUM> may determine that the progress has been impeded for a threshold amount of time. The autonomous vehicle <NUM> may then determine if the progress is impeded due to traffic. In some instances, to determine if the progress has been impeded due to traffic, the autonomous vehicle <NUM> may determine that the progress for at least one other vehicle located near the autonomous vehicle <NUM> and/or located near the vehicle <NUM>(<NUM>) for which the autonomous vehicle <NUM> is following has also impeded.

For example, the autonomous vehicle <NUM> may be navigating in a first road lane <NUM>(<NUM>). The autonomous vehicle <NUM> may determine that the progress of the vehicle <NUM>(<NUM>) located in a second road lane <NUM>(<NUM>), the progress of the vehicle <NUM>(<NUM>) located in the second road lane <NUM>(<NUM>), the progress of the vehicle <NUM>(<NUM>) located in the second road lane <NUM>(<NUM>), the progress of the vehicle <NUM>(<NUM>) located in a third road lane <NUM>(<NUM>), the progress of the vehicle <NUM>(<NUM>) located in the third road lane <NUM>(<NUM>), and/or the progress of the vehicle <NUM>(<NUM>) located in the third road lane <NUM>(<NUM>) has also been impeded. Based at least in part on the determination(s), the autonomous vehicle <NUM> may determine that the progress is impeded due to traffic. As such, the autonomous vehicle <NUM> may determine not to contact the teleoperator even though the progress of the autonomous vehicle <NUM> has been impeded for the threshold amount of time.

<FIG> is an example environment <NUM> that includes an autonomous vehicle <NUM> determining not to contact a teleoperator based at least in part on the progress of the autonomous vehicle <NUM> being impeded due to a traffic light <NUM>, in accordance with embodiments of the invention. For instance, the autonomous vehicle <NUM> may be navigating along a route <NUM>. Additionally, other vehicles <NUM>(<NUM>)-(<NUM>) may respectively be navigating along routes <NUM>(<NUM>)-(<NUM>). While navigating, the progress of the autonomous vehicle <NUM> may be impeded based at least in part on the autonomous vehicle <NUM> yielding to the vehicle <NUM>(<NUM>), which also stopped. For instance, the vehicle <NUM>(<NUM>) may be stopped because the traffic light <NUM> is red. Based at least in part on the progress being impeded, the autonomous vehicle <NUM> may determine whether to contact the teleoperator.

For instance, the autonomous vehicle <NUM> may determine that the progress has been impeded for a threshold amount of time. The autonomous vehicle <NUM> may then determine if the progress is impeded due to the traffic light <NUM>. In some instances, to determine if the progress is stopped due to the traffic light <NUM>, the autonomous vehicle <NUM> may analyze sensor data generated by the autonomous vehicle <NUM>. Based at least in part on the analysis, the autonomous vehicle <NUM> may identify the traffic light <NUM> and determine that the traffic light <NUM> is currently red. As such, the autonomous vehicle <NUM> may determine that the progress is impeded due to the traffic light <NUM>. In some instances, since the progress is impeded due to the traffic light <NUM>, the autonomous vehicle <NUM> may determine not to contact the teleoperator even though the progress has been impeded for the threshold amount of time.

In a similar example, however, if such an environment <NUM> is devoid of traffic lights, the autonomous vehicle <NUM> may reason about how to proceed based on common practice or law dictating how to proceed through a four way stop (e.g., yield based on order of approach and/or to the vehicle to the right). In such an example, if the autonomous vehicle <NUM> has been yielding (according to the policy) to another vehicle (e.g., vehicle <NUM>(<NUM>)) for a period of time greater than or equal to a threshold amount of time, the autonomous vehicle <NUM> may send a signal to a teleoperator for guidance.

<FIG> depicts a block diagram of an example system <NUM> for implementing the techniques described herein, in accordance with embodiments of the invention. In at least one example, the system <NUM> can include a vehicle <NUM> (which may represent, and/or be similar to, the first autonomous vehicle <NUM>(<NUM>), the second autonomous vehicle <NUM>(<NUM>), the autonomous vehicle <NUM>, and/or the autonomous vehicle <NUM>). The vehicle <NUM> can include a vehicle computing device <NUM>, one or more sensor systems <NUM>, one or more emitters <NUM>, one or more communication connections <NUM>, at least one direct connection <NUM>, and one or more drive modules <NUM>.

The vehicle computing device <NUM> can include one or more processors <NUM> and a memory <NUM> communicatively coupled with the one or more processors <NUM>. In the illustrated example, the vehicle <NUM> is an autonomous vehicle. However, the vehicle <NUM> may be any other type of vehicle (e.g., a manually driven vehicle, a semi-autonomous vehicle, etc.), or any other system having at least an image capture device. In the illustrated example, the memory <NUM> of the vehicle computing device <NUM> stores a localization component <NUM>, a perception component <NUM>, a planning component <NUM>, a progress component <NUM>, an conditions component <NUM>, a teleoperator component <NUM>, one or more system controllers <NUM>, and one or more maps <NUM>. Though depicted in <FIG> as residing in the memory <NUM> for illustrative purposes, it is contemplated that the localization component <NUM>, the perception component <NUM>, the planning component <NUM>, the progress component <NUM>, the conditions component <NUM>, the teleoperator component <NUM>, the one or more system controllers <NUM>, and/or the one or more maps <NUM> can additionally, or alternatively, be accessible to the vehicle <NUM> (e.g., stored on, or otherwise accessible by, memory remote from the vehicle <NUM>).

In at least one example, the localization component <NUM> can include functionality to receive sensor data <NUM> (which may represent, and/or be similar to, the sensor data <NUM> and/or the sensor data <NUM>) from the sensor system(s) <NUM> and to determine a position and/or orientation of the vehicle <NUM> (e.g., one or more of an x-, y-, z-position, roll, pitch, or yaw). For example, the localization component <NUM> can include and/or request / receive a map of an environment and can continuously determine a location and/or orientation of the vehicle <NUM> within the map. In some instances, the localization component <NUM> can utilize SLAM (simultaneous localization and mapping), CLAMS (calibration, localization and mapping, simultaneously), relative SLAM, bundle adjustment, non-linear least squares optimization, or the like to receive image data, lidar data, radar data, IMU data, GPS data, wheel encoder data, and the like to accurately determine a location of the vehicle <NUM>. In some instances, the localization component <NUM> can provide data to various components of the vehicle <NUM> to determine an initial position of the vehicle <NUM> for generating a candidate trajectory, as discussed herein.

In some instances, the perception component <NUM> can include functionality to perform object detection, segmentation, and/or classification. In some instances, the perception component <NUM> can provide processed sensor data <NUM> that indicates a presence of an object that is proximate to the vehicle <NUM> and/or a classification of the object as an object type (e.g., car, pedestrian, cyclist, animal, building, tree, road surface, curb, sidewalk, unknown, etc.). In additional and/or alternative examples, the perception component <NUM> can provide processed sensor data <NUM> that indicates one or more characteristics associated with a detected object and/or the environment in which the object is positioned. In some instances, characteristics associated with an object can include, but are not limited to, an x-position (global position), a y-position (global position), a z-position (global position), an orientation (e.g., a roll, pitch, yaw), an object type (e.g., a classification), a velocity of the object, an acceleration of the object, an extent of the object (size), etc. Characteristics associated with the environment can include, but are not limited to, a presence of another object in the environment, a state of another object in the environment, a time of day, a day of a week, a season, a weather condition, an indication of darkness/light, etc..

In general, the planning component <NUM> can determine a path for the vehicle <NUM> to follow to traverse through an environment. For example, the planning component <NUM> can determine various routes and trajectories and various levels of detail. For example, the planning component <NUM> can determine a route to travel from a first location (e.g., a current location) to a second location (e.g., a target location). For the purpose of this discussion, a route can be a sequence of waypoints for travelling between two locations. As non-limiting examples, waypoints include streets, intersections, global positioning system (GPS) coordinates, etc. Further, the planning component <NUM> can generate an instruction for guiding the vehicle <NUM> along at least a portion of the route from the first location to the second location. In at least one example, the planning component <NUM> can determine how to guide the vehicle <NUM> from a first waypoint in the sequence of waypoints to a second waypoint in the sequence of waypoints. In some instances, the instruction can be a trajectory, or a portion of a trajectory. In some instances, multiple trajectories can be substantially simultaneously generated (e.g., within technical tolerances) in accordance with a receding horizon technique, wherein one of the multiple trajectories is selected for the vehicle <NUM> to navigate.

In at least one example, the planning component <NUM> can determine a pickup location associated with a location. As used herein, a pickup location can be a specific location (e.g., a parking space, a loading zone, a portion of a ground surface, etc.) within a threshold distance of a location (e.g., an address or location associated with a dispatch request) where the vehicle <NUM> can stop to pick up a passenger. In at least one example, the planning component <NUM> can determine a pickup location based at least in part on determining a user identity (e.g., determined via image recognition or received as an indication from a user device, as discussed herein). Arrival at a pickup location, arrival at a destination location, entry of the vehicle by a passenger, and receipt of a "start ride" command are additional examples of events that may be used for event-based data logging.

In some instances, the planning component <NUM> can further include functionality to determine when the vehicle <NUM> is to yield to object(s), such as other vehicle(s). For instance, the planning component <NUM> may use policy data <NUM> to determine when the vehicle <NUM> is to yield to the object(s). The policy data <NUM> may represent one or more policies that the vehicle <NUM> is to follow when navigating. The one or more policies may include, but are not limited to, a policy to yield to other vehicle(s) that arrive at an intersection before the vehicle <NUM>, a policy to travel in a road lane, a policy on how to change lanes, a policy on how to stop, a policy that the vehicle <NUM> should not tailgate, and/or the like. As such, the planning component <NUM> may determine that the vehicle <NUM> is to yield to another vehicle when the other vehicle arrives at an intersection before the vehicle <NUM>. Additionally, the planning component <NUM> may determine that the vehicle is to yield to two other vehicles when the two other vehicles arrive at an intersection before the vehicle <NUM>.

In some instances, the planning component <NUM> can further include functionality to determine an interaction between the vehicle <NUM> and object(s), such as vehicle(s). For instance, the planning component <NUM> may use interaction data <NUM> to determine the interaction between the vehicle <NUM> and the object(s). The interaction data <NUM> may represent one or more interactions, such as, but not limited to, an interaction to follow object(s) that are located along a trajectory of the vehicle <NUM>, an interaction not to follow object(s) that are not located along a trajectory of the vehicle <NUM>, an interaction to yield to object(s) for which the vehicle <NUM> is following, and/or the like. In some instances, following may involve varying longitudinal controls of the vehicle <NUM> (e.g., velocity and/or acceleration) to maintain a minimum distance between the vehicle <NUM> and the object, wherein the minimum distance may be based, at least in part, on a relative distance and/or velocity between the vehicle <NUM> and the object. As such, the planning component <NUM> may determine that the vehicle <NUM> is to follow another vehicle when the other vehicle is located in the trajectory of the vehicle <NUM>.

In some instances, the planning component <NUM> can further include functionality to generate indicator data <NUM>, where the indicator data <NUM> represents object(s) that the vehicle <NUM> is currently yielding to and/or object(s) that the vehicle <NUM> is following. In some instances, the planning component <NUM> may generate the indicator data <NUM> each time the perception component determines that the vehicle <NUM> is yielding to object(s) and/or each time the planning component <NUM> determines that the vehicle <NUM> is following object(s). In other instances, the planning component <NUM> may generate the indicator data <NUM> when the vehicle <NUM> determines to contact the teleoperator.

In some instances, the planning component <NUM> includes functionality to use instruction data <NUM> sent by the teleoperator system <NUM> in order to determine how to navigate the vehicle <NUM>. For a first example, the instruction data <NUM> may represent an instruction to cease yielding to object(s). Based at least in part on the instruction data <NUM>, the planning component <NUM> may cause the vehicle <NUM> to cease yielding to the object(s) and/or determine a route for navigating the vehicle <NUM>. For a second example, the instruction data <NUM> may represent an instruction to temporarily cease following one or more policies. Based at least in part on the instruction data <NUM>, the planning component <NUM> may cause the vehicle <NUM> to temporarily cease following the one or more policies and/or determine a route for navigating the vehicle <NUM>. For a third example, the instruction data <NUM> may represent an instruction to change an interaction with object(s) from "follow" to "no longer follow". Based at least in part on the instruction data <NUM>, the planning component <NUM> may cause the vehicle <NUM> to change the interaction with the object(s) and/or determine a route for navigating around the object(s).

The progress component <NUM> can include functionality to determine when the progress of the vehicle <NUM> has been impeded. In some instances, the progress component <NUM> may determine that the progress of the vehicle <NUM> has been impeded when the vehicle <NUM> is not moving (e.g., a velocity of the vehicle <NUM> is zero miles per hour). Additionally, or alternatively, in some instances, the progress component <NUM> may determine that the progress of the vehicle <NUM> has been impeded when the vehicle <NUM> only navigates a give distance within a given amount of time. Additionally, or alternatively, in some instances, the progress component <NUM> may determine that the progress of the vehicle <NUM> has been impeded when the current velocity of the vehicle <NUM> is below a threshold velocity.

The progress component <NUM> can include functionality to determine when the progress of the vehicle <NUM> is impeded by an object. In some instances, the progress component <NUM> may determine that the progress of the vehicle <NUM> may be impeded when the vehicle <NUM> identifies a specific type of object located within the environment. In some instances, the progress component <NUM> may determine that the progress of the vehicle <NUM> may be impeded when the vehicle <NUM> identifies a specific scenario occurring within the environment. The scenario may include, but is not limited to, another vehicle merging into the drive lane of the vehicle <NUM>, another vehicle navigating at a velocity that is below a threshold velocity (which is described below), an accident that occurs along the route of the vehicle <NUM>, and/or the like.

In some instances, the progress component <NUM> can further include functionality to determine when the progress of the vehicle <NUM> has been impeded for a threshold amount of time, which may be represented by threshold data <NUM>. Additionally, or alternatively, in some instances, the progress component <NUM> can include functionality to determine when the vehicle <NUM> is yielding to object(s) for a threshold amount of time. As described herein, the threshold amount of time may include, but is not limited to, three seconds, thirty seconds, one minute, three minutes, and/or any other period of time.

The conditions component <NUM> can include functionality to determine when one or more intervening conditions are causing the progress of the vehicle <NUM> to be impeded. As described herein, an intervening condition may include, but is not limited to, the vehicle <NUM> being stuck in traffic, the vehicle <NUM> being impeded because of a traffic light, and/or the like.

The teleoperator component <NUM> can include functionality to determine when to contact the teleoperator. In some instances, the teleoperator component <NUM> determines to contact the teleoperator when the progress of the vehicle <NUM> has been impeded for the threshold amount of time. In some instances, the teleoperator component <NUM> determines to contact the teleoperator when the progress of the vehicle <NUM> has been impeded for the threshold amount of time and an intervening condition that causes the progress of the vehicle <NUM> to be impeded has not occurred. Still, in some instances, the teleoperator component <NUM> determines to contact the teleoperator when the progress of the vehicle has been impeded by a specific type of object or specific scenario. In either instance, after determining to contact the teleoperator, the teleoperator component <NUM> can include functionality to cause the vehicle <NUM> to send at least a portion of the sensor data <NUM> to the teleoperator system <NUM>. Additionally, in some instances, the teleoperator component <NUM> can include functionality to cause the vehicle <NUM> to send the indicator data <NUM> to the teleoperator system <NUM>.

In at least one example, the vehicle computing device <NUM> can include one or more system controllers <NUM>, which can be configured to control steering, propulsion, braking, safety, emitters, communication, and other systems of the vehicle <NUM>. These system controller(s) <NUM> can communicate with and/or control corresponding systems of the drive module(s) <NUM> and/or other components of the vehicle <NUM>.

The memory <NUM> can further include one or more maps <NUM> that can be used by the vehicle <NUM> to navigate within the environment. For the purpose of this discussion, a map can be any number of data structures modeled in two dimensions, three dimensions, or N-dimensions that are capable of providing information about an environment, such as, but not limited to, topologies (such as intersections), streets, mountain ranges, roads, terrain, and the environment in general. In some instances, a map can include, but is not limited to: texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV/HSL color information), and the like), intensity information (e.g., lidar information, radar information, and the like); spatial information (e.g., image data projected onto a mesh, individual "surfels" (e.g., polygons associated with individual color and/or intensity)), reflectivity information (e.g., specularity information, retroreflectivity information, BRDF information, BSSRDF information, and the like). In one example, a map can include a three-dimensional mesh of the environment. In some instances, the map can be stored in a tiled format, such that individual tiles of the map represent a discrete portion of an environment and can be loaded into working memory as needed. In at least one example, the one or more maps <NUM> can include at least one map (e.g., images and/or a mesh). In some example, the vehicle <NUM> can be controlled based at least in part on the map(s) <NUM>. That is, the map(s) <NUM> can be used in connection with the localization component <NUM>, the perception component <NUM>, and/or the planning component <NUM> to determine a location of the vehicle <NUM>, identify entities in an environment, and/or generate routes and/or trajectories to navigate within an environment.

In some instances, aspects of some or all of the components discussed herein can include any models, algorithms, and/or machine learning algorithms. For example, in some instances, the components in the memory <NUM> can be implemented as a neural network. As described herein, an exemplary neural network is a biologically inspired algorithm which passes input data through a series of connected layers to produce an output. Each layer in a neural network can also comprise another neural network, or can comprise any number of layers (whether convolutional or not). As can be understood in the context of this disclosure, a neural network can utilize machine learning, which can refer to a broad class of such algorithms in which an output is generated based at least in part on learned parameters.

Although discussed in the context of neural networks, any type of machine learning can be used consistent with this disclosure. For example, machine learning algorithms can include, but are not limited to, regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), locally estimated scatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least-angle regression (LARS)), decisions tree algorithms (e.g., classification and regression tree (CART), iterative dichotomiser <NUM> (ID2), Chi-squared automatic interaction detection (CHAID), decision stump, conditional decision trees), Bayesian algorithms (e.g., naive Bayes, Gaussian naive Bayes, multinomial naive Bayes, average one-dependence estimators (AODE), Bayesian belief network (BNN), Bayesian networks), clustering algorithms (e.g., k-means, k-medians, expectation maximization (EM), hierarchical clustering), association rule learning algorithms (e.g., perceptron, back-propagation, hopfield network, Radial Basis Function Network (RBFN)), deep learning algorithms (e.g., Deep Boltzmann Machine (DBM), Deep Belief Networks (DBN), Convolutional Neural Network (CNN), Stacked Auto-Encoders), Dimensionality Reduction Algorithms (e.g., Principal Component Analysis (PCA), Principal Component Regression (PCR), Partial Least Squares Regression (PLSR), Sammon Mapping, Multidimensional Scaling (MDS), Projection Pursuit, Linear Discriminant Analysis (LDA), Mixture Discriminant Analysis (MDA), Quadratic Discriminant Analysis (QDA), Flexible Discriminant Analysis (FDA)), Ensemble Algorithms (e.g., Boosting, Bootstrapped Aggregation (Bagging), AdaBoost, Stacked Generalization (blending), Gradient Boosting Machines (GBM), Gradient Boosted Regression Trees (GBRT), Random Forest), SVM (support vector machine), supervised learning, unsupervised learning, semi-supervised learning, etc..

Additional examples of architectures include neural networks such as ResNet70, ResNet101, VGG, DenseNet, PointNet, and the like.

As discussed above, in at least one example, the sensor system(s) <NUM> can include lidar sensors, radar sensors, ultrasonic transducers, sonar sensors, location sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB, IR, intensity, depth, time of flight, etc.), microphones, wheel encoders, environment sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), etc. The sensor system(s) <NUM> can include multiple instances of each of these or other types of sensors. For instance, the lidar sensors can include individual lidar sensors located at the corners, front, back, sides, and/or top of the vehicle <NUM>. As another example, the camera sensors can include multiple cameras disposed at various locations about the exterior and/or interior of the vehicle <NUM>. The sensor system(s) <NUM> can provide input to the vehicle computing device <NUM>. Additionally or alternatively, the sensor system(s) <NUM> can send the sensor data <NUM>, via the one or more network(s) <NUM>, to a control system <NUM> at a particular frequency, after a lapse of a predetermined period of time, upon occurrence of one or more conditions, in near real-time, etc..

The vehicle <NUM> can also include one or more emitters <NUM> for emitting light and/or sound, as described above. The emitter(s) <NUM> in this example include interior audio and visual emitters to communicate with passengers of the vehicle <NUM>. By way of example and not limitation, interior emitters can include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and/or force feedback), mechanical actuators (e.g., seatbelt tensioners, seat positioners, headrest positioners, etc.), and the like. The emitter(s) <NUM> in this example also include exterior emitters. By way of example and not limitation, the exterior emitters in this example include lights to signal a direction of travel or other indicator of vehicle action (e.g., indicator lights, signs, light arrays, etc.), and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) to audibly communicate with pedestrians or other nearby vehicles, one or more of which comprising acoustic beam steering technology.

The vehicle <NUM> can also include one or more communication connection(s) <NUM> that enable communication between the vehicle <NUM> and one or more other local or remote computing device(s). For instance, the communication connection(s) <NUM> can facilitate communication with other local computing device(s) on the vehicle <NUM> and/or the drive module(s) <NUM>. Also, the communication connection(s) <NUM> can allow the vehicle <NUM> to communicate with other nearby computing device(s) (e.g., other nearby vehicles, traffic signals, etc.). The communications connection(s) <NUM> also enable the vehicle <NUM> to communicate with the remote teleoperations computing devices (e.g., the teleoperator system <NUM>) or other remote services.

The communications connection(s) <NUM> can include physical and/or logical interfaces for connecting the vehicle computing device <NUM> to another computing device or a network, such as network(s) <NUM>. For example, the communications connection(s) <NUM> can enable Wi-Fi-based communication such as via frequencies defined by the IEEE <NUM> standards, short range wireless frequencies such as Bluetooth®, cellular communication (e.g., <NUM>, <NUM>, <NUM>, <NUM> LTE, <NUM>, etc.) or any suitable wired or wireless communications protocol that enables the respective computing device to interface with the other computing device(s).

In at least one example, the vehicle <NUM> can include one or more drive modules <NUM>. In some instances, the vehicle <NUM> can have a single drive module <NUM>. In at least one example, if the vehicle <NUM> has multiple drive modules <NUM>, individual drive modules <NUM> can be positioned on opposite ends of the vehicle <NUM> (e.g., the front and the rear, etc.). In at least one example, the drive module(s) <NUM> can include one or more sensor systems to detect conditions of the drive module(s) <NUM> and/or the surroundings of the vehicle <NUM>. By way of example and not limitation, the sensor system(s) <NUM> can include one or more wheel encoders (e.g., rotary encoders) to sense rotation of the wheels of the drive modules, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) to measure orientation and acceleration of the drive module, cameras or other image sensors, ultrasonic sensors to acoustically detect entities in the surroundings of the drive module, lidar sensors, radar sensors, etc. Some sensors, such as the wheel encoders can be unique to the drive module(s) <NUM>. In some cases, the sensor system(s) <NUM> on the drive module(s) <NUM> can overlap or supplement corresponding systems of the vehicle <NUM> (e.g., sensor system(s) <NUM>).

The drive module(s) <NUM> can include many of the vehicle systems, including a high voltage battery, a motor to propel the vehicle <NUM>, an inverter to convert direct current from the battery into alternating current for use by other vehicle systems, a steering system including a steering motor and steering rack (which can be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and/or pneumatic components, a stability control system for distributing brake forces to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head/tail lights to illuminate an exterior surrounding of the vehicle), and one or more other systems (e.g., cooling system, safety systems, onboard charging system, other electrical components such as a DC/DC converter, a high voltage junction, a high voltage cable, charging system, charge port, etc.). Additionally, the drive module(s) <NUM> can include a drive module controller which can receive and preprocess the sensor data <NUM> from the sensor system(s) <NUM> and to control operation of the various vehicle systems. In some instances, the drive module controller can include one or more processors and memory communicatively coupled with the one or more processors. The memory can store one or more modules to perform various functionalities of the drive module(s) <NUM>. Furthermore, the drive module(s) <NUM> also include one or more communication connection(s) that enable communication by the respective drive module with one or more other local or remote computing device(s).

In at least one example, the direct connection <NUM> can provide a physical interface to couple the one or more drive module(s) <NUM> with the body of the vehicle <NUM>. For example, the direct connection <NUM> can allow the transfer of energy, fluids, air, data, etc. between the drive module(s) <NUM> and the vehicle <NUM>. In some instances, the direct connection <NUM> can further releasably secure the drive module(s) <NUM> to the body of the vehicle <NUM>.

As further illustrated in <FIG>, the control system <NUM> can include processor(s) <NUM>, communication connection(s) <NUM>, and memory <NUM>. The processor(s) <NUM> of the vehicle <NUM> and/or the processor(s) <NUM> of the control system <NUM> (and/or other processor(s) described herein) can be any suitable processor capable of executing instructions to process data and perform operations as described herein. By way of example and not limitation, the processor(s) <NUM> and <NUM> can comprise one or more Central Processing Units (CPUs), Graphics Processing Units (GPUs), or any other device or portion of a device that processes electronic data to transform that electronic data into other electronic data that can be stored in registers and/or memory. In some instances, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices can also be considered processors in so far as they are configured to implement encoded instructions.

The memory <NUM> and the memory <NUM> (and/or other memory described herein) are examples of non-transitory computer-readable media. The memory <NUM> and the memory <NUM> can store an operating system and one or more software applications, instructions, programs, and/or data to implement the methods described herein and the functions attributed to the various systems. In various implementations, the memory can be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), nonvolatile/Flash-type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein can include many other logical, programmatic, and physical components, of which those shown in the accompanying figures are merely examples that are related to the discussion herein.

It should be noted that while <FIG> is illustrated as a distributed system, in alternative examples, components of the vehicle <NUM> can be associated with the control system <NUM> and/or components of the control system <NUM> can be associated with the vehicle <NUM>. That is, the vehicle <NUM> can perform one or more of the functions associated with the control system <NUM>, and vice versa.

<FIG> shows an example architecture <NUM> including a fleet of vehicles <NUM> and the teleoperator system <NUM>. The example vehicle fleet <NUM> includes one or more vehicles (e.g., the vehicle <NUM>), at least some which are communicatively coupled to the teleoperator system <NUM> via the communication connection(s) of the vehicles. A teleoperations receiver <NUM> and a teleoperations transmitter <NUM> associated with the teleoperator system <NUM> may be communicatively coupled to the respective communication connection(s) of the vehicles. For example, the vehicle <NUM> may send communication signals via the communication connection(s) <NUM>, which are received by the teleoperations receiver <NUM>. In some examples, the communication signals may include, for example, the sensor data <NUM>, the indicator data <NUM>, operation state data, and/or any data and/or output from one or more components of the vehicle <NUM>. In some examples, the sensor data <NUM> may include raw sensor data and/or processed sensor data, such as a subset of operation state data such as a representation of the sensor data (e.g. a bounding box). In some examples, the communication signals may include a subset of the operation state data discussed above and/or any other derived data (planning controls, etc.). In some examples, the communication signals from the vehicle <NUM> may include a request for assistance to the teleoperator system <NUM>.

As shown in <FIG>, a situational awareness engine ("SAE") <NUM> may obtain the communication signals from the vehicle <NUM> and relay at least a subset of them to a teleoperations interface <NUM>. The SAE <NUM> may additionally or alternatively obtain signals generated via the teleoperations interface <NUM> and relay them to a teleoperations network <NUM> and/or transmit at least a subset of them to one or more vehicles of the vehicle fleet <NUM> via the teleoperations transmitter <NUM>. The SAE <NUM> may additionally or alternatively obtain signals from the teleoperations network <NUM> and relay at least a subset of them to the teleoperations interface <NUM> and/or the teleoperations transmitter <NUM>. In some examples, the teleoperations interface <NUM> may directly communicate with the vehicle fleet <NUM>.

In some examples, the SAE <NUM> can be implemented on a device that is separate from a device that includes the teleoperations interface <NUM>. For example, the SAE <NUM> can include a gateway device and an application programming interface ("API") or similar interface. In some examples, the SAE <NUM> includes an application interface and/or a model, such as, for example, a FSM, an ANN, and/or a DAG. In some examples, the SAE <NUM> is configured to determine a presentation configuration of data received from one or more elements discussed herein (e.g., the vehicle <NUM>, the vehicle fleet <NUM>, and/or other teleoperations interfaces) and/or input options to present to the teleoperator to provide guidance to one or more vehicles (e.g., an option to select a displayed button that confirms a trajectory determined by the vehicle).

In some examples, the teleoperations receiver <NUM> may be communicatively coupled to the teleoperations interface <NUM> via the SAE <NUM>, and in some examples, a teleoperator <NUM> may be able to access the sensor data <NUM>, the indicator data <NUM>, the operation state data, and/or any other data in the communication signals received from the vehicle <NUM> via the teleoperations interface <NUM>. In some examples, the teleoperator <NUM> may be able to selectively access the sensor data <NUM>, the indicator data <NUM>, the operation state data, and/or other data via an input device, and view the selected data via one or more displays. In some examples, such selective accessing can include transmitting a request for data from the vehicle <NUM> via the teleoperations transmitter <NUM>. In some examples, the SAE <NUM> may present a subset or representation of the data to the teleoperator <NUM> via the teleoperations interface <NUM>. As a non-limiting example, the SAE <NUM> may create simplistic pictorial representations, bounding boxes, arrows indicating a bearing and velocity of objects, icons representing objects, colorization of the sensor data, or other representations of the data which may simplify interpretation by a teleoperator <NUM>.

In the example shown, the teleoperator system <NUM> also includes the teleoperations network <NUM> configured to provide communication between two or more of the teleoperations interfaces <NUM> and the respective teleoperators <NUM>, and/or communication with teleoperations data <NUM>. For example, the teleoperator system <NUM> may include a plurality of teleoperations interfaces <NUM> and respective teleoperators <NUM>, and the teleoperators <NUM> may communicate with one another via the teleoperations network <NUM> to facilitate and/or coordinate the guidance provided to the vehicle fleet <NUM>. In some examples, there may be a teleoperator <NUM> assigned to each vehicle from the vehicle fleet <NUM>, and in some examples, a teleoperator <NUM> may be assigned to more than a single vehicle of the vehicle fleet <NUM>. In some examples, more than one teleoperator <NUM> may be assigned to a single vehicle. In some examples, teleoperators <NUM> may not be assigned to specific vehicles of the vehicle fleet <NUM>, but may instead provide guidance to vehicles that have encountered certain types of events and/or to vehicles based at least in part on, for example, a level of urgency associated with the vehicle's encounter with the event. In some examples, data associated with an event and/or the guidance provided by a teleoperator <NUM> may be stored by the teleoperator system <NUM>, for example, in storage for the teleoperations data <NUM>, and/or accessed by other teleoperators <NUM>.

In some examples, the teleoperations data <NUM> may be accessible by the teleoperators <NUM>, for example, via the teleoperations interface <NUM>, for use in providing guidance to the vehicles <NUM>. For example, the teleoperations data <NUM> may include global and/or local map data related to the road network, events associated with the road network, and/or travel conditions associated with the road network due to, for example, traffic volume, weather conditions, construction zones, and/or special events. In some examples, the teleoperations data <NUM> may include data associated with one more of the vehicles of the vehicle fleet <NUM>, such as, for example, maintenance and service information, and/or operational history including, for example, event history associated with the vehicle <NUM>, route histories, occupancy histories, and other types of data associated with the vehicle <NUM>.

In some examples, a teleoperator <NUM> and/or a teleoperations interface <NUM> can be associated with credentials <NUM>. For example, to activate a session at the teleoperations interface <NUM>, the teleoperator system <NUM> may require the teleoperator <NUM> to authenticate using the credentials <NUM>. In some examples, the credentials <NUM> may be inherent to the particular teleoperations interface <NUM>. In some examples, requests for assistance with a particular permutation of operation state data (e.g., certain events) and/or a particular teleoperation option may require resolution by a teleoperations interface <NUM> having elevated credentials <NUM> (e.g., credentials with greater permissions). For example, if a teleoperator <NUM> selects an action at a teleoperations interface <NUM> that would affect the entire vehicle fleet <NUM> instead of just one vehicle, the action could be transmitted to a second teleoperations interface <NUM> that has elevated credentials associated therewith for confirmation, modification, and/or rejection. A request and/or operation state data associated with an elevated level of credentials can be used to determine a teleoperations interface <NUM> to which to relay the request. In some examples, the SAE <NUM> can relay a request and/or operation state data associated with an elevated level of credentials to multiple teleoperations interfaces <NUM> instead of a single teleoperations interface <NUM>.

<FIG> is a first example user interface <NUM> showing the vehicle <NUM> yielding for object(s), in accordance with embodiments of the invention. As shown, the user interface <NUM> includes four portions <NUM>(<NUM>)-(<NUM>) illustrating information about the environment for which the vehicle <NUM> is located. For instance, the first portion <NUM>(<NUM>) of the user interface <NUM> includes a first video rendered as if taken from above the vehicle <NUM>, the second portion <NUM>(<NUM>) of the user interface <NUM> includes a second video rendered as if taken from a side of the vehicle <NUM>, the third portion <NUM>(<NUM>) of the user interface <NUM> includes a third video captured by a first camera of the vehicle <NUM>, and the fourth portion <NUM>(<NUM>) of the user interface <NUM> includes a fourth video captured by a second camera of the vehicle <NUM>. While the example of <FIG> illustrates the user interface <NUM> as including four portions <NUM>(<NUM>)-(<NUM>), in other examples, the user interface <NUM> may include any number of portions.

As shown, the portions <NUM>(<NUM>)-(<NUM>) of the user interface <NUM> include representations of the vehicle <NUM>. In some instances, one or more of the portions <NUM>(<NUM>)-(<NUM>) of the user interface <NUM> may include graphical element(s) indicating the vehicle <NUM>. The graphical element(s) may include, but are not limited to, a bounding box around the vehicle <NUM>, a shading of the vehicle <NUM>, text describing the vehicle <NUM>, and/or any other type graphical element that may indicate the vehicle <NUM>. This way, the teleoperator <NUM> can easily identify where the vehicle <NUM> is located within the environment.

Additionally, the portions <NUM>(<NUM>)-(<NUM>) of the user interface <NUM> include representations of the object <NUM> for which the vehicle <NUM> is yielding, which includes a crane parked at least partially within the intersection in the example of <FIG>. In some instances, one or more of the portions <NUM>(<NUM>)-(<NUM>) of the user interface <NUM> may include graphical element(s) indicating the object <NUM>. The graphical element(s) may include, but are not limited to, a bounding box around the object <NUM>, a shading of the object <NUM>, text describing the object <NUM>, and/or any other type graphical element that may indicate the object <NUM>. This way, the teleoperator <NUM> can easily identify the object <NUM> for which the vehicle <NUM> is yielding.

In some instances, the teleoperator <NUM> can select the object <NUM> using the user interface <NUM>. The teleoperator <NUM> can then use the user interface <NUM> to provide instructions for how the vehicle <NUM> is to proceed. For example, the teleoperator <NUM> can use the user interface <NUM> to instruct the vehicle <NUM> to no longer yield to the object <NUM>. This is because, in the example of <FIG>, the object <NUM> parked at least partially within the intersection and as such, is blocking the route of the vehicle <NUM>. Therefore, the vehicle <NUM> may have to cease yielding to the object <NUM> in order to navigate around the object <NUM>.

The vehicle <NUM> may receive the instructions from the teleoperator system <NUM> and proceed according to the instructions. For example, the vehicle <NUM> may determine to cease yielding to the object <NUM> and/or determine to temporarily cease following one or more policies with regard to the object <NUM>. The vehicle <NUM> may then determine how to proceed navigating based on the one or more policies with regard to other objects located within the environment. For example, if the vehicle <NUM> is not yielding for another object, the vehicle <NUM> may then select a route such that the vehicle <NUM> navigates around the object <NUM>. The vehicle <NUM> may then begin to navigate along the selected route. However, if the vehicle <NUM> is still yielding to another object, then the vehicle <NUM> may continue to yield for the other object before proceeding.

For example, in some instances, the vehicle <NUM> may be yielding to more than one object. For example, the vehicle <NUM> may further be yielding for a second object <NUM> that is located within the intersection. In such an example, the user interface <NUM> may include a graphical element indicating the second object <NUM>. In some instances, the graphical element indicating the object <NUM> may be different than the graphical element indicating the second object <NUM>. For example, the graphical element indicating the object <NUM> may tell the teleoperator <NUM> that the vehicle <NUM> is currently yielding for the object <NUM>, and the graphical element indicating the second object <NUM> may tell the teleoperator <NUM> that the vehicle <NUM> will later be yielding for the second object <NUM> (e.g., if the second object <NUM> is still located within the intersection).

<FIG> is a first example user interface <NUM> showing the vehicle <NUM> yielding for object(s), in accordance with embodiments of the invention. As shown, the user interface <NUM> includes four portions <NUM>(<NUM>)-(<NUM>) illustrating information about the environment for which the vehicle <NUM> is located. For instance, the first portion <NUM>(<NUM>) of the user interface <NUM> includes a first video from above the vehicle <NUM>, the second portion <NUM>(<NUM>) of the user interface <NUM> includes a second video from a side of the vehicle <NUM>, the third portion <NUM>(<NUM>) of the user interface <NUM> includes a third video captured by a first camera of the vehicle <NUM>, and the fourth portion <NUM>(<NUM>) of the user interface <NUM> includes a fourth video captured by a second camera of the vehicle <NUM>. While the example of <FIG> illustrates the user interface <NUM> as including four portions <NUM>(<NUM>)-(<NUM>), in other examples, the user interface <NUM> may include any number of portions.

Additionally, the portions <NUM>(<NUM>)-(<NUM>) of the user interface <NUM> include representations of the object <NUM> for which the vehicle <NUM> is yielding, which includes a moving truck trying to navigate past the vehicle <NUM> in the example of <FIG>. In some instances, one or more of the portions <NUM>(<NUM>)-(<NUM>) of the user interface <NUM> may include graphical element(s) indicating the object <NUM>. The graphical element(s) may include, but are not limited to, a bounding box around the object <NUM>, a shading of the object <NUM>, text describing the object <NUM>, and/or any other type graphical element that may indicate the object <NUM>. This way, the teleoperator <NUM> can easily identify the object <NUM> for which the vehicle <NUM> is yielding.

In the example of <FIG>, the vehicle <NUM> may be yielding to the object <NUM>. However, the object <NUM>, which is attempting to navigate in the lane next to the vehicle <NUM>, may be unable to navigate since an object <NUM> is blocking at least a portion of the lane for which the object <NUM> is using. As such, the object <NUM> may be stopped in the intersection and waiting for the vehicle <NUM> to move in order for the object <NUM> to navigate around the object <NUM>. Additionally, the vehicle <NUM> may be stopped and yielding for the object <NUM> such that the vehicle <NUM> will not proceed until the object <NUM> navigates through the intersection.

As such, and in some instances, the teleoperator <NUM> can select the object <NUM> using the user interface <NUM>. The teleoperator <NUM> can then use the user interface <NUM> to provide instructions for how the vehicle <NUM> is to proceed. For example, the teleoperator <NUM> can use the user interface <NUM> to instruct the vehicle <NUM> to no longer yield to the object <NUM>. This way, the vehicle <NUM> can begin to navigate even when the object <NUM> is still located within the intersection. Once the vehicle <NUM> begins to navigate, the object <NUM> will then also begin to navigate around the object <NUM>.

<FIG> illustrate example processes in accordance with embodiments of the invention. These processes are illustrated as logical flow graphs, each operation of which represents a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and/or in parallel to implement the processes.

<FIG> depicts an example process <NUM> for determining whether to contact a teleoperator, in accordance with embodiments of the present invention. At operation <NUM>, the process <NUM> may include causing an autonomous vehicle to navigate along a route. For instance, the vehicle <NUM> may receive instructions from the control system <NUM>, where the instructions cause the vehicle <NUM> to navigate along the route from a first location to a second location. Based at least in part on the instructions, the vehicle <NUM> may begin navigating along the route from the first location to the second location.

At operations <NUM>, the process <NUM> may include determining that progress of the autonomous vehicle is impeded at a location along the route. For instance, the vehicle <NUM> may determine that the progress of the vehicle <NUM> is impeded at the location along the route. In some instances, the vehicle <NUM> may make the determination based on determining that the vehicle is yielding to object(s) for a threshold period of time. In some instances, the vehicle <NUM> may make the determination based on determining that a specific scenario is occurring around the vehicle <NUM>. Still, in some instances, the vehicle <NUM> may make the determination based on determining that a specific type of object is located near the vehicle <NUM>.

At operation <NUM>, the process <NUM> may include determining an amount of time that the progress of the autonomous vehicle has been impeded. For instance, the vehicle <NUM> may determine the amount of time at which the progress of the vehicle <NUM> has been impeded.

At operations <NUM>, the process <NUM> may include determining if the amount of time satisfies a threshold. For instance, the vehicle <NUM> may determine if the amount of time satisfies (e.g., is equal to or greater than) the threshold. If, at operation <NUM>, the amount of time does not satisfy the threshold, then at operation <NUM>, the process <NUM> may include determining not to contact a remote operator. For instance, if the vehicle <NUM> determines that the amount of time does not satisfy (e.g., if below) the threshold, then the vehicle <NUM> may determine not to contact the teleoperator <NUM>. Additionally, the vehicle <NUM> may repeat operations <NUM>-<NUM>.

However, if at operation <NUM>, the amount of time satisfies the threshold, then at operation <NUM>, the process <NUM> may include determining if an intervening condition has occurred. For instance, the vehicle <NUM> may determine if an intervening condition is causing the progress to be impeded. In some instances, the vehicle <NUM> may determine that the intervening condition has occurred when the vehicle <NUM> is stuck in traffic. In some instances, the vehicle <NUM> may determine that the intervening condition has occurred when the vehicle <NUM> is stopped at a traffic light.

If, at operation <NUM>, the intervening condition has occurred, then the process <NUM> may determine not to contact the remote operator. For instance, if the vehicle <NUM> determines that the intervening condition has occurred, then the vehicle <NUM> may determine not to contact the teleoperator <NUM>.

However, if at operation <NUM>, the interviewing condition has not occurred, then at operation <NUM>, the process <NUM> may include determining to contact the remote operator. For instance, if the vehicle <NUM> determines that the intervening condition has not occurred, then the vehicle <NUM> may determine to contact the teleoperator <NUM>. Additionally, the vehicle <NUM> may send sensor data <NUM> and/or indicator data <NUM> to the teleoperator system <NUM>. The teleoperator <NUM> may then use the sensor data <NUM> and/or indicator data <NUM> to determine how the vehicle <NUM> should proceed.

At operation <NUM>, the process <NUM> may include receiving, from a system, an instruction associated with navigating. For instance, the vehicle <NUM> may receive instruction data <NUM> from the teleoperator system <NUM>, where the instruction data <NUM> represents an instruction associated with navigating the vehicle <NUM>. For a first example, the instruction data <NUM> may include an instruction to temporarily cease following the policy associated with yielding to objects that arrive at intersections before the vehicle <NUM>. For a second example, the instruction data <NUM> may include an instruction to change an interaction with the object from "follow" to "do not follow". In either example, the vehicle <NUM> may cease yielding to the object.

At operation <NUM>, the process <NUM> may include causing the autonomous vehicle to navigate. For instance, based at least in part on the instruction data <NUM>, the vehicle <NUM> may begin navigating. In some instances, the vehicle <NUM> begins navigating since the vehicle <NUM> is no longer yielding to object(s). In some instances, the vehicle <NUM> begins navigating since the vehicle is no longer following object(s). Still, in some instances, the vehicle <NUM> begins navigating along the original route of the vehicle.

<FIG> depicts an example process <NUM> for navigating an autonomous vehicle using instructions from a teleoperator, in accordance with embodiments of the present invention. At operation <NUM>, the process <NUM> may include generating sensor data using one or more sensors. For instance, the vehicle <NUM> may use the sensor system(s) <NUM> to generate the sensor data <NUM>. The vehicle <NUM> may generate the sensor data <NUM> while navigating around an environment.

At operations <NUM>, the process <NUM> may include determining that progress of the autonomous vehicle is impeded. For instance, the vehicle <NUM> may determine that the progress of the vehicle <NUM> is impeded. In some instances, the vehicle <NUM> may make the determination based on determining that the vehicle is yielding to object(s) for a threshold period of time. In some instances, the vehicle <NUM> may make the determination based on determining that a specific scenario is occurring around the vehicle <NUM>. Still, in some instances, the vehicle <NUM> may make the determination based on determining that a specific type of object is located near the vehicle <NUM>.

At operation <NUM>, the process <NUM> may include sending at least a portion of the sensor data to a teleoperator system (and/or data derived therefrom). For instance, the vehicle <NUM> may send the at least the portion of the sensor data <NUM> to the teleoperator system <NUM>. The at least the portion of the sensor data <NUM> may represent at least the object causing the progress to be impeded. In some instances, the vehicle <NUM> sends the at least the portion of the sensor data <NUM> based at least in part on determining that progress of the vehicle <NUM> has been impeded for a threshold amount of time.

At operation <NUM>, the process <NUM> may include sending, to the teleoperator system, an indication that the progress of the autonomous vehicle is impeded to an object. For instance, the vehicle <NUM> may send indicator data <NUM> to the teleoperator system <NUM>, where the indicator data <NUM> indicates that the progress of the vehicle <NUM> is impeded to the object. In some instances, the indicator data <NUM> may further indicate the amount of time that the vehicle <NUM> has been impeded to the object.

At operation <NUM>, the process <NUM> may include receiving, from the teleoperator system, an instruction associated with navigating. For instance, the vehicle <NUM> may receive instruction data <NUM> from the teleoperator system <NUM>, where the instruction data <NUM> represents an instruction associated with navigating the vehicle <NUM>. For a first example, the instruction data <NUM> may include an instruction to temporarily cease following the policy associated with yielding to objects that arrive at intersections before the vehicle <NUM>. For a second example, the instruction data <NUM> may include an instruction to change an interaction with the object from "follow" to "do not follow". In either example, the vehicle <NUM> may cease yielding to the object.

At operation <NUM>, the process <NUM> may include causing the autonomous vehicle to navigate to a second location. For instance, based at least in part on the instruction data <NUM>, the vehicle <NUM> may begin navigating to the second location. In some instances, the vehicle <NUM> begins navigating since the vehicle <NUM> is no longer yielding to object(s). In some instances, the vehicle <NUM> begins navigating since the vehicle is no longer following object(s). Still, in some instances, the vehicle <NUM> begins navigating along the original route of the vehicle.

At operation <NUM>, the process <NUM> may include determining if an intervening condition occurs while navigating to the second location. For instance, the vehicle <NUM> may use the sensor data <NUM> to determine if the intervening condition occurs after beginning to navigate to the second location. In some instances, the intervening condition may include the object beginning to move. In some instances, the intervening condition may include the object moving outside of a designated area associated with the object.

If, at operation <NUM>, the intervening condition occurs, then at operation <NUM>, the process <NUM> may include causing the autonomous vehicle to cease navigating to the second location. For instance, if the vehicle <NUM> determines that the intervening condition occurs, then the vehicle <NUM> may cease navigating to the second location. In some instances, the vehicle <NUM> may stop. In some instances, the vehicle <NUM> may change an interaction with the object from "do not follow" to "follow".

However, if at operation <NUM>, the intervening condition does not occur, then at operation <NUM>, the process <NUM> may include causing the autonomous vehicle to continue navigating to the second location. For instance, if the vehicle <NUM> determines that the intervening condition did not occur, then the vehicle <NUM> will continue to navigate to the second location.

While one or more examples of the techniques described herein have been described, various alterations, additions, permutations and equivalents are possible within the scope of the appended claims.

Claim 1:
An autonomous vehicle (<NUM>) comprising:
one or more network components;
one or more sensors;
one or more processors; and
memory storing instructions that, when executed by the one or more processors, configure the autonomous vehicle to perform operations comprising:
causing the autonomous vehicle to navigate along a route (<NUM>);
receiving sensor data generated by the one or more sensors;
determining, based at least in part on the sensor data (<NUM>), that a progress of the autonomous vehicle is impeded by an object (<NUM>);
determining, based at least in part on the progress being impeded by the object and map data, whether to contact a remote operator;
receiving, from one or more computing devices associated with the remote operator, an instruction associated with navigating by the object comprising information configured to alter a policy associated with the object; and
causing, based at least in part on the instruction, the autonomous vehicle to navigate by the object.