Patent Description:
Planning systems utilize information associated with objects (e.g., dynamic agents) in an environment to determine actions relative to those objects. For example, some existing systems control a vehicle to slow down and/or stop upon determining that an object is likely to enter a drive path of the vehicle. However, conventional models may over- or under-estimate the object behavior, which may result in abrupt acceleration or deceleration, making travel unpleasant for the rider. For example, conventional systems may not consider probabilities associated with multiple objects when determining an action or reaction while travelling in an environment, thereby leading to less than optimal efficiency and/or passenger comfort and satisfaction. <CIT> discloses an automatic braking system for a vehicle that includes an electronic brake system capable of applying wheel brakes to decelerate the vehicle and a controller. The controller includes instructions for detecting an object proximate to a vehicle and determining a collision confidence value based upon the probability of collision with the object. The controller also includes instructions for determining a desired velocity profile of the vehicle that provides for deceleration of the vehicle. <CIT> discloses a driving assistant adapted for active control of a vehicle based on predictions of a behavior of a detected object. The driving assistant may accept a first prediction of a behavior associated with the detected object from a first prediction subsystem and a second prediction from a second prediction subsystem, determine a control signal based on a combination of the first prediction and the second prediction, and initiate active control of the vehicle based on the control signal. <CIT> discloses a method for controlling speed and/or distance in motor vehicles having distance-related longitudinal control systems, wherein during clear driving, a clear drive acceleration value is set to reach a preset speed, and while following a detected target object, a following acceleration value is set to maintain a preset distance from the target object, and wherein if the target object is lost during a following drive operation, a transition acceleration value is set. The transition acceleration value is set for the duration of a time interval, which is dependent on the speed at the time of the target object loss. <CIT> discloses a method for controlling speed and/or distance in motor vehicle having distance-related longitudinal systems in which a congestion likelihood value is determined from data associated with a current traffic situation and an acceleration setpoint is determined based on the congestion likelihood. <CIT> discloses a method for determining a desired acceleration value and/or desired delay value for reaching a given speed and/or distance relative to an ahead-driving target object based on data of the target object and a pre-object driving in front of the target object.

Techniques described herein are directed to responsive control of a vehicle, such as an autonomous vehicle, in an environment. For example, in implementations described herein, a control (such as an acceleration, steering angle, torques, etc.) of a vehicle along a planned path may be determined by considering a plurality of different objects (e.g., dynamic obstacles, including but not limited to pedestrians, animals, cyclists, trucks, motorcycles, other vehicles, or the like) in an environment of the vehicle and the predicted actions of those objects. For example, the vehicle may be controlled at an acceleration that is a weighted average of a plurality of discrete reactions to each of a plurality of objects.

A vehicle according to aspects of this invention includes a sensor, one or more processors, and memory storing executable instructions. The instructions, when executed by the one or more processors, configure the vehicle to: determine a planned path of the vehicle through an environment, determine a first acceleration of the vehicle the planned path; receive sensor data from the sensor, the sensor data comprising information about an object in the environment; determine, based at least in part on the sensor data, that a probability that the object will impact travel of the vehicle along the planned path meets or exceeds a threshold probability; determine a second acceleration of the vehicle along the planned path, the second acceleration being an acceleration of the vehicle responsive to the object impacting travel of the vehicle along the planned path; determine, as a function of at least the first acceleration and the second acceleration, a third acceleration of the vehicle along the planned path, the third acceleration being different from the first acceleration and the second acceleration; and control the vehicle to accelerate along the planned path in accordance with the third acceleration.

Aspects of this invention also relate to a method comprising: determining a planned path of a vehicle through an environment, determining a first acceleration of the vehicle along the planned path; identifying, based at least in part on sensor data acquired from a sensor associated with the vehicle, an object in the environment; determining a second acceleration of the vehicle, different from the first acceleration, along the planned path based at least in part on the object; determining a probability that the object will impact the planned path of the vehicle; determining, based at least on the first acceleration, the second acceleration, and the probability, a third acceleration of the vehicle along the planned path, the third acceleration being different from the first acceleration and from the second acceleration; and controlling the vehicle to accelerate along the planned path in accordance with the third acceleration.

In some examples, the computing device of the autonomous vehicle may also determine a weighting factor for each of the determined accelerations. For instance, the first acceleration may have a fixed weighting factor, e.g., because that acceleration may be the default or preferred acceleration. Weighting factors for each of the second accelerations may be based on the probability that each event will occur. Continuing with the example of the detected pedestrian that may cross the road, a weighting factor associated with the second acceleration responsive to that pedestrian could be a function of the probability of the pedestrian crossing the road.

In some implementations, the control acceleration for the autonomous vehicle may be determined as a weighted average of each of the determined accelerations, using the weighting factors. Thus, for example, each of the objects may be considered in determining the control acceleration. Moreover, because the weighting factors may be functions of event probabilities, the objects may be relatively weighted.

The systems and techniques described herein are distinct from previous systems, in that previous systems merely selected a control acceleration based on a likelihood of an event occurring. For example, in previous systems, if the likelihood that the example pedestrian would cross in front of the autonomous vehicle exceeded a threshold, the vehicle would use the second acceleration responsive to that event as the control acceleration. Alternatively, if the likelihood of that pedestrian crossing was lower than the threshold, the pedestrian (and responsive acceleration) were ignored. Thus, unlike conventional systems, aspects of this disclosure may be inclusive of more events when determining a control acceleration. This inclusion may result in a smoother ride for passengers and increased safety for the autonomous vehicle and/or objects in the environment.

Techniques described herein are directed to leveraging sensor and perception data to enable a vehicle, such as an autonomous vehicle, to increase a level of safety associated with navigating through an environment while circumventing objects in the environment, in addition to providing a pleasant riding experience. Techniques described herein can utilize semantic and/or probabilistic information to determine a vehicle acceleration (and/or other controls) along a drive path such that the vehicle can travel relative to those objects in a more efficient manner and/or with increased comfort to passengers than with existing navigation techniques. For example, techniques described herein may provide a technological improvement over existing prediction and navigation technology. In addition to improving the accuracy with which sensor data can be used to determine the vehicle acceleration, techniques described herein can provide a smoother ride and improve safety outcomes by, for example, more accurately determining a safe speed at which the vehicle may operate to reach an intended destination. Techniques described herein may also reduce component wear and energy requirements associated with the vehicle. For example, existing techniques may more harshly brake and/or accelerate, putting additional and unnecessary stress on vehicle components.

<FIG> provide additional details associated with techniques described herein.

<FIG> is a schematic diagram illustrating an example implementation of responsive control of a vehicle through an environment as described herein. More specifically, <FIG> illustrates an example environment <NUM> in which a vehicle <NUM> is operating. In the illustrated example, the vehicle <NUM> is driving in the environment <NUM>, although in other examples the vehicle <NUM> may be stationary and/or parked in the environment <NUM>. One or more objects, or agents, also are in the environment <NUM>. For instance, <FIG> illustrates two additional vehicles 104a, 104b (collectively, the "additional vehicles <NUM>") and a pedestrian <NUM>. Although not illustrated, any number and/or type of objects, including static objects, e.g., road signs, parked vehicles, fire hydrants, buildings, curbs, or the like, and/or dynamic objects, e.g., pedestrians, animals, cyclists, trucks, motorcycles, other vehicles, or the like, can additionally or alternatively be present in the environment <NUM>.

The vehicle <NUM> can be an autonomous vehicle configured to operate according to a Level <NUM> classification issued by the U. National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions for the entire trip, with the driver (or occupant) not being expected to control the vehicle at any time. In such an example, since the vehicle <NUM> can be configured to control all functions from start to stop, including all parking functions, it can be unoccupied. This is merely an example, and the systems and methods described herein can be incorporated into any ground-borne, airborne, or waterborne vehicle, including those ranging from vehicles that need to be manually controlled by a driver at all times, to those that are partially or fully autonomously controlled. Additional details associated with the vehicle <NUM> are described below.

In the example of <FIG>, the vehicle <NUM> can be associated with one or more sensor systems <NUM>. The sensor system(s) <NUM> can generate sensor data <NUM>, which can be utilized by one or more vehicle computing devices <NUM> associated with the vehicle <NUM> to recognize the one or more objects, e.g., the additional vehicles <NUM> and/or the pedestrian <NUM>. The sensor system(s) <NUM> can include, but is/are not limited to, light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, ultrasonic transducers, sound navigation and ranging (SONAR) sensors, location sensors (e.g., global positioning system (GPS), compass, etc.), inertial sensors (e.g., inertial measurement units, accelerometers, magnetometers, gyroscopes, etc.), time of flight sensors, cameras (e.g., RGB, IR, intensity, depth, etc.), wheel encoders, microphones, environment sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), etc..

In at least one example, the vehicle computing device(s) <NUM> can include a perception system, which can perform object detection, segmentation, and/or classification based at least in part on the sensor data <NUM> received from the sensor system(s) <NUM>. For instance, the perception system can detect the additional vehicles <NUM> and/or the pedestrian <NUM> in the environment <NUM> based on the sensor data <NUM> generated by the sensor system(s) <NUM>. Additionally, the perception system can determine an extent of the additional vehicles <NUM> and/or the pedestrian <NUM> (e.g., height, weight, length, etc.), a pose of the additional vehicles <NUM> and/or the pedestrian <NUM> (e.g., x-coordinate, y-coordinate, z-coordinate, pitch, roll, yaw), and/or additional attributes of the additional vehicle <NUM> and/or the pedestrian <NUM>. The sensor system(s) <NUM> can continuously generate the sensor data <NUM> (e.g., in near-real time), which can be utilized by the perception system (and/or other systems of the vehicle computing device(s) <NUM>).

The vehicle computing device(s) <NUM> can also include an acceleration determination system <NUM>, which, as described further herein, can determine a control acceleration at which the vehicle <NUM> is to proceed. The control acceleration may be an acceleration for the vehicle <NUM> at which the vehicle <NUM> can travel safely through the environment <NUM>, e.g., relative to the additional vehicles <NUM>, the pedestrian <NUM>, and/or other objects. In some implementations, the control acceleration may be an acceleration along a travel path <NUM>, which travel path <NUM> may be determined by the vehicle computing device(s) <NUM> as a path along which the vehicle <NUM> may travel through the environment <NUM>. In some instances, and as further described herein, the planner system can determine the travel path based at least in part on the sensor data <NUM> received from the sensor system(s) <NUM> and/or any determinations made by a perception system. For instance, the planning system can determine the travel path <NUM> as a route to navigate safely through the environment relative to the additional vehicles <NUM> and the pedestrian <NUM>, as identified by the perception system.

More specifically, <FIG> illustrates a scenario in which the vehicle <NUM> is travelling through the environment <NUM>, generally in the direction of arrow <NUM>. The vehicle <NUM> is travelling on a road <NUM> having a first lane <NUM> and a second lane <NUM>. The vehicle <NUM> is in the first lane <NUM>, behind the first additional vehicle 104a, and, in the example, may be travelling relatively the same speed as the first additional vehicle 104a. For example, the vehicle <NUM> may be controlled to maintain a relatively constant distance from the rear of the first additional vehicle 104a. For instance, the vehicle computing device(s) <NUM> may receive sensor data <NUM> indicative of a position and/or movement of the first additional vehicle 104a and generate controls for maintaining the predetermined distance. In examples of this disclosure, at least a portion of the controls may include an acceleration of the vehicle <NUM>, e.g., the control acceleration determined by the acceleration determination system <NUM>. Thus, if the sensor data <NUM> indicates that the distance between the vehicle <NUM> and the first additional vehicle 104a is decreasing, the vehicle computing device(s) <NUM> may generate a control to decelerate.

In the example illustrated in <FIG>, in addition to using information about the first additional vehicle 104a, the acceleration determination system <NUM> may also include information about additional objects in the environment, including but not limited to the second additional vehicle 104b and/or the pedestrian <NUM>, to determine the control acceleration for the vehicle <NUM>. In some implementations, the acceleration determination system <NUM> may also consider object information <NUM>, which may include information about semantic classifications and/or probabilistic models. In implementations described herein, the vehicle computing device(s) <NUM>, using the sensor data <NUM> and/or the agent information <NUM>, may determine information about objects in the environment <NUM>, which may include the first additional vehicle 104a, the second additional vehicle 104b, the pedestrian <NUM>, and/or additional objects. For example, and as will be described further herein, the vehicle computing device(s) <NUM>, using the acceleration determination system <NUM>, may consider information about each of the objects, including the object information <NUM>, to determine a probability that each object will enter the drive path <NUM> and determine the control acceleration for the vehicle <NUM> based at least in part on those probabilities.

With further reference to <FIG>, the second additional vehicle 104b may be travelling in the second lane <NUM>, and may take a number of actions. For example, the second additional vehicle 104b could proceed along a first vehicle trajectory 130a, which would maintain the second additional vehicle 104b in the second lane <NUM>. The second additional vehicle 104b may follow the first vehicle trajectory 130a to pass the first additional vehicle or to be passed by the vehicle <NUM>, for instance. In other examples, the second additional vehicle 104b may proceed along a second vehicle trajectory 130b, along which the second additional vehicle may change from the second lane <NUM> to the first lane <NUM>. Along this second vehicle trajectory 130b, the second additional vehicle 104b could attempt to merge into the first lane <NUM>, e.g., by merging behind the first additional vehicle 104a and in front of the vehicle <NUM>. Similarly, the pedestrian <NUM> may take any of a number of actions. For instance, the pedestrian <NUM> may proceed along a first pedestrian trajectory 132a, e.g., generally along the side of the road, or may proceed along a second pedestrian trajectory 132b, e.g., along which the pedestrian <NUM> would veer into the first lane <NUM>. As may be appreciated, should the vehicle computing device(s) <NUM> determine that the second additional vehicle 104b is to proceed along the first vehicle trajectory 130a and that the pedestrian <NUM> is to proceed along the first pedestrian trajectory 132a, the vehicle <NUM> may be controlled by ignoring the first additional vehicle 104a and the pedestrian <NUM>, e.g., to maintain vehicle <NUM> at the predetermined distance from the first additional vehicle 104a as discussed above. Alternatively, should the vehicle computing device(s) <NUM> determine that the second additional vehicle 104b is likely to proceed along the second vehicle trajectory 130b and/or that the pedestrian <NUM> is likely to proceed along the second pedestrian trajectory 132b, e.g., into the travel path <NUM>, the vehicle <NUM> may be controlled to decelerate, e.g., to allow the second additional vehicle 104b into the first lane <NUM> and/or to continue to proceed with caution relative to the pedestrian <NUM>.

As described further herein, the vehicle computing device(s) <NUM> may be configured to determine probabilities associated with the second additional vehicle 104b travelling along the first vehicle trajectory 130a or the second vehicle trajectory 130b and probabilities associated with the pedestrian <NUM> travelling along the first pedestrian trajectory 132a or the second pedestrian trajectory 132b. Based on these probabilities, the acceleration determination system <NUM> may determine an acceleration for the vehicle <NUM> that takes into account the first additional vehicle 104a, the second additional vehicle 104b, and the pedestrian <NUM>. In instances in which additional objects are detected and identified in the environment <NUM>, those objects may also be considered when determining the control acceleration. Thus, the vehicle <NUM> is controlled to accelerate along the travel path in accordance with information about a plurality of objects in the environment. In some implementations, semantic information about the objects may be used to determine the trajectories (e.g., the trajectories 130a, 130b, 132a, 132b), the likelihoods that the respective objects will follow those trajectories, the speed and/or acceleration at which the vehicles are likely to follow those trajectories, or the like. Thus, in <FIG>, the pedestrian <NUM> may be associated with a first classification and the additional vehicles <NUM> may be associated with one or more second classifications. Moreover, vehicles within the second classification(s) may be further classified as static, as in the case of a parked vehicle, or dynamic, as in the case of the additional vehicles 104a, 104b. In further examples, probabilistic information about the objects may also or alternatively be used to determine the likelihood that the additional vehicles <NUM> and/or the pedestrian follows the trajectories. For example, there may be some uncertainty associated with the actions of the moving pedestrian <NUM> and/or the moving additional vehicles 104a, 104b, e.g., because the pedestrian <NUM> or the drivers of the additional vehicles 104a, 104b may change course. Probabilistic information may also be used to account for sensor inaccuracies, e.g., associated with the sensor system(s) <NUM>.

In the example of <FIG>, the first and second vehicle trajectories 130a, 130b and the first and second pedestrian trajectories 132a, 132b are shown for illustration only. The second additional vehicle 104b and/or the pedestrian <NUM> may proceed along any number of trajectories. In practice, the vehicle computing device(s) <NUM>, e.g., using a perception system, a planning system, and/or the acceleration determination system <NUM>, may consider any number of trajectories for objects in the environment <NUM>. In some implementations, the vehicle computing device(s) <NUM> may determine a single probability for each object, e.g., a probability that the object will interfere with travel of the vehicle <NUM>, and may weight accelerations associated with navigating relative to each of the objects using the probabilities. For example, the probability may be the probability associated with the one trajectory or action (of multiple trajectories or actions) having the highest probability of impacting travel of the vehicle <NUM>. In some conventional examples, if it was determined that a probability that the second additional vehicle 104b was going to switch lanes, e.g., by following the second vehicle trajectory 130b, exceeded a threshold probability, the vehicle would decelerate at a predetermined rate, e.g., to maintain a safe distance from the merging second additional vehicle 104b allowing it to merge naturally and safely. Alternatively, in these conventional systems, if the probability that the second additional vehicle 104b was going to switch lanes was determined to be below the threshold probability, the vehicle computing device(s) <NUM> would disregard the second additional vehicle 104b, and generate controls to maintain the distance from the first additional vehicle 104a. Thus, conventional systems which instituted a binary approach (e.g., brake to allow the second additional vehicle 104b to merge or disregard the second additional vehicle 104b entirely and accelerate to follow the first additional vehicle 104a) often resulted in abrupt acceleration and deceleration, particularly when the second additional vehicle 104b is drifting in the second lane <NUM> or making similar unorthodox maneuvers. In contrast, in implementations detailed further herein, a probability that each object is likely to impact or enter the travel path <NUM> may be used to weight a responsive acceleration, and the weighted accelerations may be averaged to determine an acceleration for controlling the vehicle <NUM>.

<FIG> illustrates a single example of navigating an environment using acceleration determination techniques. Other examples also are contemplated. For example, although <FIG> illustrates two possible events, e.g., a potentially-merging vehicle and a nearby pedestrian, additional events also are contemplated. For example, a vehicle entering the road <NUM>, e.g., from a driveway or an intersecting street, zipper merges on highways, and the like may be considered according to techniques described herein. Any object or event that could impact the travel of the vehicle <NUM> along the travel path <NUM> may be used to consider the control acceleration. Several other examples are discussed further herein.

<FIG> is a block diagram illustrating an example system <NUM> for implementing responsive control of a vehicle through an environment as described herein. In at least one example, the system <NUM> can include a vehicle <NUM>, which can be the same as the vehicle <NUM> described above with reference to <FIG>. The vehicle <NUM> can include one or more vehicle computing devices <NUM>, one or more sensor systems <NUM> (which may be the sensor system(s) <NUM>), one or more emitters <NUM>, one or more communication connections <NUM>, at least one direct connection <NUM>, and one or more drive modules <NUM>. In at least one example, the vehicle computing device(s) <NUM> can correspond to the vehicle computing device(s) <NUM> described above with reference to <FIG>.

The vehicle computing device(s) <NUM> can include processor(s) <NUM> and memory <NUM> communicatively coupled with the processor(s) <NUM>. In the illustrated example, the vehicle <NUM> is an autonomous vehicle; however, the vehicle <NUM> could be any other type of vehicle. In the illustrated example, the memory <NUM> of the vehicle computing device(s) <NUM> stores a localization system <NUM>, a perception system <NUM>, a prediction system <NUM>, a planning system <NUM>, an acceleration determination system <NUM>, and one or more system controllers <NUM>. Although these systems and components are illustrated, and will be described below, as separate components for ease of understanding, functionality of the various systems and controllers may be attributed differently than discussed. By way of non-limiting example, functionality attributed to the perception system <NUM> may be carried out by the localization system <NUM> and/or the prediction system <NUM>. Fewer or more systems and components may be utilized to perform the various functionalities described herein. Furthermore, though depicted in <FIG> as residing in memory <NUM> for illustrative purposes, it is contemplated that the localization system <NUM>, the perception component <NUM>, the prediction system <NUM>, the planning system <NUM>, the acceleration determination component <NUM>, and/or the one or more system controllers <NUM> can additionally, or alternatively, be accessible to the vehicle <NUM> (e.g., stored on, or otherwise accessible by, memory remote from the vehicle <NUM>).

As also illustrated in <FIG>, the memory <NUM> can include a map storage <NUM> and/or object information storage <NUM>. The map storage <NUM> may store one or more maps, and the object information storage <NUM> may store the object information <NUM>. A map can be any number of data structures modeled in two dimensions or three dimensions that are capable of providing information about an environment, such as, but not limited to, topologies (such as intersections), streets, mountain ranges, roads, terrain, and the environment in general. As discussed further herein, the object information <NUM> may be any information about objects, including, without limitation, information that is useful for interpreting sensor data to classify or otherwise understand objects in the environment. By way of non-limiting example, the object information <NUM> may include information used to identify, classify, and/or otherwise understand an object in the environment <NUM>.

In at least one example, the localization system <NUM> can include functionality to receive data from the sensor system(s) <NUM> to determine a position and/or orientation of the vehicle <NUM> (e.g., one or more of an x-, y-, z-position, roll, pitch, or yaw). For example, the localization system <NUM> can include and/or request/receive a map of an environment (e.g., from the map storage <NUM>) and can continuously determine a location and/or orientation of the vehicle <NUM> within the map. In some instances, the localization system <NUM> can utilize SLAM (simultaneous localization and mapping), CLAMS (calibration, localization and mapping, simultaneously), relative SLAM, bundle adjustment, non-linear least squares optimization, differential dynamic programming, or the like to receive image data, LIDAR data, radar data, IMU data, GPS data, wheel encoder data, and the like to accurately determine a location of the autonomous vehicle. In some instances, the localization system <NUM> can provide data to various components of the vehicle <NUM> to determine an initial position of an autonomous vehicle for generating a candidate trajectory for travelling in the environment.

In some instances, the perception system <NUM> can include functionality to perform object detection, segmentation, and/or classification. In some examples, the perception system <NUM> can provide processed sensor data that indicates a presence of an object that is proximate to the vehicle <NUM>, such as the additional vehicles 104a, 104b and/or the pedestrian <NUM>. The perception system <NUM> may also determine a classification of the entity as an entity type (e.g., car, pedestrian, cyclist, animal, building, tree, road surface, curb, sidewalk, unknown, etc.). For instance, the perception system <NUM> may compare sensor data to the object information <NUM> in the object information database <NUM> to determine the classification. In additional and/or alternative examples, the perception system <NUM> can provide processed sensor data that indicates one or more characteristics associated with a detected object and/or the environment in which the object is positioned. In some examples, characteristics associated with an object can include, but are not limited to, an x-position (global and/or local position), a y-position (global and/or local position), a z-position (global and/or local position), an orientation (e.g., a roll, pitch, yaw), an object type (e.g., a classification), a velocity of the object, an acceleration of the object, an extent of the object (size), etc. Characteristics associated with the environment can include, but are not limited to, a presence of another object in the environment, a state of another object in the environment, a time of day, a day of a week, a season, a weather condition, an indication of darkness/light, etc..

The prediction system <NUM> can access sensor data from the sensor system(s) <NUM>, map data from the map storage <NUM>, and, in some examples, perception data output from the perception system <NUM> (e.g., processed sensor data). In at least one example, the prediction system <NUM> can determine features associated with the object based at least in part on the sensor data, the map data, and/or the perception data. As described above, features can include an extent of an object (e.g., height, weight, length, etc.), a pose of an object (e.g., x-coordinate, y-coordinate, z-coordinate, pitch, roll, yaw), a velocity of an object, an acceleration of an object, and a direction of travel of an object (e.g., a heading). Moreover, the prediction system <NUM> may be configured to determine a distance between an object and a proximate driving lane, a width of a current driving lane, proximity to a crosswalk, semantic feature(s), interactive feature(s), etc..

The prediction system <NUM> can also analyze features of objects to predict future actions of the objects. For instance, the prediction system <NUM> can predict lane changes, decelerations, accelerations, turns, changes of direction, or the like. The prediction system <NUM> can send prediction data to the acceleration determination system <NUM> so that the acceleration determination system <NUM> can, using the prediction data, determine an appropriate acceleration for controlling the vehicle <NUM> along a travel path, such as the travel path <NUM>. For instance, the prediction system <NUM> can generate prediction data indicative of a likelihood that an object will enter the travel path <NUM> and/or otherwise interfere with a current path and/or trajectory of the vehicle <NUM>. In some examples, the prediction data may be a numerical probability, e.g., from <NUM> to <NUM> that the object will enter the travel path <NUM> and information about the acceleration and/or speed of the object. In some examples, the prediction system <NUM> may provide an indication (e.g., an audio and/or visual alert, e.g. via one or more of the emitter(s) <NUM>) to the object and/or to a passenger (or driver when the vehicle <NUM> is not fully autonomous) when it is determined that the object may affect travel.

In general, the planning system <NUM> can determine a path, e.g., the travel path <NUM>, along which the vehicle <NUM> can traverse through an environment. For example, the planning system <NUM> can determine various routes and trajectories and various levels of detail. The planning system <NUM> can determine a route to travel from a first location (e.g., a current location) to a second location (e.g., a target location). For the purpose of this discussion, a route can be a sequence of waypoints for travelling between two locations. As non-limiting examples, waypoints include streets, intersections, global positioning system (GPS) coordinates, etc. Further, the planning system <NUM> can generate an instruction for guiding the autonomous vehicle along at least a portion of the route from the first location to the second location. In at least one example, the planning system <NUM> can determine how to guide the autonomous vehicle from a first waypoint in the sequence of waypoints to a second waypoint in the sequence of waypoints. In some examples, the instruction can be a traj ectory, or a portion of a trajectory. In some examples, multiple trajectories can be substantially simultaneously generated (e.g., within technical tolerances) in accordance with a receding horizon technique, wherein one of the multiple trajectories is selected for the vehicle <NUM> to navigate. Thus, in example implementations described herein, the planning system <NUM> may generate trajectories along which the vehicle can navigate, with the trajectories being along the travel path <NUM>.

The acceleration determination system <NUM> is configured to determine an acceleration at which the vehicle <NUM> is to be controlled along a travel path, such as the travel path <NUM>. Although illustrated as a separate block in the memory <NUM>, in some examples and implementations, the acceleration determination system <NUM> may be a part of the planning system <NUM>. The acceleration determination system <NUM> can access sensor data from the sensor system(s) <NUM>, map data from the map storage <NUM>, object information from the object information store <NUM>, and/or outputs from one or more of the localization system <NUM>, the perception system <NUM>, and/or the prediction system <NUM> (e.g., processed data). By way of non-limiting example, the acceleration determination system <NUM> may access (e.g., retrieve or receive) a probability that each object in the environment will impact travel of the vehicle <NUM>. For instance, and as discussed above, the prediction system <NUM> may determine for each object a probability, e.g., a numerical value from <NUM> to <NUM>, that the object will enter the travel path <NUM>, and provide those probabilities to the acceleration determination system <NUM>. The acceleration determination system <NUM> may use these probabilities to determine an acceleration at which to control the vehicle <NUM>. For instance, and with reference to <FIG>, the acceleration determination system <NUM> may receive (or retrieve) a first probability that the second additional vehicle 104b will change lanes, e.g., enter the first lane <NUM> in front of the vehicle <NUM>, and a second probability that the pedestrian <NUM> will enter the road <NUM>. In at least some examples, such a prediction system <NUM> may also assign a unique probability for each potential action an object may take. As a non-limiting example, such probabilities may comprise a first probability an object will change lanes and a second probability the object will continue along its current course.

The acceleration determination system <NUM> may determine which objects to include in, and/or which objects to exclude from, its acceleration determination. For example, the acceleration determination system <NUM> may exclude or ignore information about objects having a probability of interfering with travel of the vehicle below a threshold value. Considering <FIG>, if the sensor data <NUM> indicates that the second additional vehicle 104b has a turn signal indicating a right turn, and there is an upcoming intersection with an available right turn, the probability of the second additional vehicle continuing along an interfering trajectory, like the second vehicle trajectory 130b, may be relatively low. In this example case, then, the predictions system <NUM> may determine that the second additional vehicle has a <NUM> probability of travelling into the travel path <NUM>. Because the probability is relatively low, the acceleration determination system <NUM> may ignore the second additional vehicle 104b altogether in this example (although in other examples, the second additional vehicle 104b may have a low weighting factor, as discussed further herein). Although not limiting, the acceleration determination system <NUM> may exclude information about objects (and/or thresholds associated with potential actions of such objects) having a probability of less than about. <NUM>, although higher or lower thresholds may be used. Moreover, although this example describes the acceleration determination system <NUM> as identifying which objects to include in the acceleration determination, in other implementations the acceleration determination system <NUM> may only receive (or retrieve) information about objects that meet or exceed the threshold probability.

The acceleration determination system <NUM> may also determine accelerations associated with each of the objects. Using <FIG> as an example, the acceleration determination system <NUM> may determine a first acceleration (e.g., which may be a deceleration) at which the vehicle <NUM> would be controlled in the event the second additional vehicle 104b enters the first lane <NUM>. The acceleration determination system <NUM> may also determine a second acceleration at which the vehicle <NUM> would be controlled in the event the pedestrian <NUM> walks onto the road <NUM>. The first and second accelerations may be determined based on a number of factors, including but not limited to semantic information about the objects, probabilistic information about the objects, a relative distance of the vehicle <NUM> to the object, a velocity of the object relative to the vehicle <NUM>, a classification of the objects, models associated with objects, including predetermined models and modeling conducted in near-real time. In such an example, the acceleration determination system <NUM> may output a unique acceleration associated with each detected object (and/or those having an associated probability above a threshold) to safely interact with the detected object. As a non-limiting example, the acceleration determination system <NUM> may output a first acceleration associated with the vehicle 104a, a second acceleration associated with the second vehicle 104b, and a third acceleration associated with the pedestrian <NUM>.

In addition to determining the accelerations, the acceleration determination system <NUM> may also determine weighting factors for each of the accelerations. The weighting factors may be based on the probabilities that the object will enter the vehicle path <NUM>, described above. For instance, the probability that the second additional vehicle 104b enters the first lane <NUM> may be higher than the probability that the pedestrian <NUM> walks into the road <NUM>, and, correspondingly, a weighting factor associated with the first acceleration may be greater than a weighting factor associated with the second acceleration. For example, the weighting factor may be a function, e.g., a linear function, an exponential function or other higher-order function, etc., of the probability. In other examples, the weighting factors may be based at least in part on a type or classification of an object. For instance, other vehicles in the road may have lower or higher weighting factors than do pedestrians, animals, and/or other object types. Moreover, the weighting factors may be determined based at least in part on a confidence associated with the object. In some implementations, the acceleration determination system <NUM> may also determine a weighting factor associated with an acceleration at which the vehicle <NUM> would proceed in the absence of any objects. In the example of <FIG>, such an acceleration may be the acceleration for following the first additional vehicle 104a. In other examples, the acceleration may be an acceleration necessary to achieve and/or maintain a desired speed, e.g., a speed limit.

Using information about the objects, acceleration(s) for navigating relative to those and/or other objects, and/or weighting factors associated with the probabilities of events, the acceleration determination system <NUM> can determine an acceleration at which to control the vehicle <NUM>, <NUM>, e.g., a control acceleration. For example, the acceleration determination system <NUM> may determine the acceleration as a weighted average of the acceleration of all objects considered. Thus, in the example of <FIG> and continuing the discussion above, the acceleration determination system <NUM> may calculate a weighted average of the first acceleration (e.g., the acceleration responsive to the second additional vehicle 104b entering the first lane <NUM>), the second acceleration (e.g., the acceleration responsive to the pedestrian <NUM> walking closer to the road <NUM>), and the acceleration for following the first additional vehicle 104a. This weighted-average approach is different from conventional systems, in which one of the accelerations may be chosen, e.g., because it is associated with a highest probability, and applied to control the vehicle <NUM>, which can result in abrupt acceleration and braking.

In some examples, the acceleration determination system <NUM> may determine the acceleration by optimizing Equation (<NUM>): <MAT> Equation (<NUM>) generally represents an optimization for determining a control acceleration ax*(t) (e.g., the acceleration at which the vehicle will be controlled in the environment in the longitudinal, x, direction at a time, t). The control acceleration determined by optimizing Equation (<NUM>) may provide the minimum deviation from each of the candidate accelerations (e.g., for each of n active subgoals, which may be objects or events, such as the pedestrian <NUM> or the cut-in event associated with the pedestrian <NUM> or the vehicle 104b or the cut-in event associated with the vehicle 104b). In Equation (<NUM>), the summation term generally includes the term Rs, which represents the weighting factor for the respective event (e.g., maintaining the distance from the first additional vehicle 104a, the second additional vehicle 104b changing lanes, the pedestrian <NUM> entering the road <NUM>, or the like) and as, which represents the acceleration for the event. Equation (<NUM>) uses L2 norms of the difference between the accelerations, e.g., ax(t) and as. When the L2 norm is used, and absent any constraints, the optimization of the Equation (<NUM>) may be a weighted average of each of the acceleration values, e.g., weighted by the weighting factors. Although Equation (<NUM>) is illustrated as using L2 norms, other functions may also be used. For example, L1 norms or L-infinity norms could be used, exponentials, polynomials, and the like. In still other examples, the optimization could include a hard constraint, e.g., a rule that the acceleration has to be greater than or less than some predetermined acceleration.

Equation (<NUM>) may also include the term Rt(ax(t)-ax*(t-<NUM>))<NUM> , which may be a smoothing term weighted by factor Rt that takes into account an acceleration in a previous time step (a subgoal in which deviation from a previous control acceleration are penalized, promoting smooth control). In implementations described herein, the summation term may consider all events that are above a threshold probability. As set forth, Equation (<NUM>) may also require that the control acceleration be between certain minimum and maximum accelerations. For instance, the equation may require that acceleration falls within a range between a minimum acceleration and a maximum acceleration. As a brief example, in a scenario in which the vehicle <NUM> may be stopping for a stop sign, amax may be the maximum acceleration that ensures the vehicle <NUM> stops in time. However, the techniques may choose a smaller ax (e.g., to a minimum acceleration, amin) if other events are present. As each event is considered in the equation (<NUM>), the techniques described herein consider the overlap between the events (e.g., the amin and amax values) and may default to the most conservative case. Thus, for each event, ax_min=min(ax_min,ax_event_min) and ax_max= min(ax_max,axevent_max). Equation (<NUM>) may require that the final ax*(t) is within this bound. For example, the use of this range may reduce computational load compared to placing explicit constraints on the optimization problem, which may require an iterative solver.

Because of the changing nature of the environment, e.g., because the vehicle and dynamic objects are moving, new objects may enter the environment, or the like, the acceleration for controlling the vehicle may be performed substantially continuously, e.g., at fixed iterations. For example, Equation (<NUM>) may be solved in near real time, e.g., at intervals of from about. <NUM> seconds to about <NUM> seconds. With each iteration, addition, fewer, and/or different objects and/or events may be considered, e.g., as the probability that those objects/events will impact travel of the vehicle <NUM> changes. Moreover, each iterative acceleration determination may be based on current information, e.g., a current state of the environment, and/or on a future prediction of the environment <NUM>. For example, controls for the vehicle <NUM> may be based on a forward-in-time-looking decision horizon. In some implementations, the vehicle computing device(s) <NUM>, e.g., implementing the prediction system <NUM>, the planning system <NUM>, and/or the acceleration determination system <NUM> may consider discrete times in a decision horizon, which may be less than <NUM> second to several seconds or more. At each discrete time step, the vehicle computing device(s) <NUM> may determine a reaction for the vehicle based on information about static objects in the environment, e.g., based on an assumption that such objects will remain static, and predictions of dynamic entities, e.g., determined by the prediction system <NUM>. Thus, at each time step, the vehicle computing device(s) <NUM>, e.g., the planning system <NUM>, may evaluate a position of the vehicle, positions of other dynamic objects, and positions of the static objects. A reaction (e.g., the acceleration for traversing along the drive path <NUM>) at each of these time steps may then be considered by the acceleration determination system <NUM> to determine the control acceleration for the vehicle <NUM>. Additional examples of the functioning of the acceleration determination system <NUM> are described further herein, including in connection with <FIG>, below.

While the acceleration determination system <NUM> is illustrated as being separate from the planning system <NUM>, one or more of the functionalities of the acceleration determination system <NUM> may be carried out by the planning system <NUM>. In some embodiments, the acceleration determination system <NUM> may be a part of the planning system <NUM>.

In at least one example, the localization system <NUM>, the perception system <NUM>, the prediction system <NUM>, the planning system <NUM>, and/or the acceleration determination system <NUM> can process sensor data, as described above, and can send their respective outputs over network(s) <NUM>, to computing device(s) <NUM>. In at least one example, the localization system <NUM>, the perception system <NUM>, the prediction system <NUM>, and/or the planning system <NUM> can send their respective outputs to the computing device(s) <NUM> at a particular frequency, after a lapse of a predetermined period of time, in near real-time, etc..

In at least one example, the vehicle computing device(s) <NUM> can include one or more system controllers <NUM>, which can be configured to control steering, propulsion, braking, safety, emitters, communication, and other systems of the vehicle <NUM>. These system controller(s) <NUM> can communicate with and/or control corresponding systems of the drive module(s) <NUM> and/or other components of the vehicle <NUM>. For example, the system controllers <NUM> may cause the vehicle to traverse along a drive path determined by the planning system <NUM>, e.g., at an acceleration determined by the acceleration determination system <NUM>.

In at least one example, the sensor system(s) <NUM> can include LIDAR sensors, RADAR sensors, ultrasonic transducers, SONAR sensors, location sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units, accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB, UV, IR, intensity, depth, etc.), microphones, wheel encoders, environment sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), etc. The sensor system(s) <NUM> can include multiple instances of each of these or other types of sensors. For instance, the LIDAR sensors can include individual LIDAR sensors located at the corners, front, back, sides, and/or top of the vehicle <NUM>. As another example, the camera sensors can include multiple cameras disposed at various locations about the exterior and/or interior of the vehicle <NUM>. The sensor system(s) <NUM> can provide input to the vehicle computing device(s) <NUM>. Additionally and/or alternatively, the sensor system(s) <NUM> can send sensor data, via the network(s) <NUM>, to the computing device(s) <NUM> at a particular frequency, after a lapse of a predetermined period of time, in near real-time, etc..

The vehicle <NUM> can also include one or more emitters <NUM> for emitting light and/or sound. The emitter(s) <NUM> in this example can include interior audio and visual emitters to communicate with passengers of the vehicle <NUM>. By way of example and not limitation, interior emitters can include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and/or force feedback), mechanical actuators (e.g., seatbelt tensioners, seat positioners, headrest positioners, etc.), and the like. The emitter(s) <NUM> in this example also include exterior emitters. By way of example and not limitation, the exterior emitters can include light emitters (e.g., indicator lights, signs, light arrays, etc.) to visually communicate with pedestrians, other drivers, other nearby vehicles, etc., one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) to audibly communicate with pedestrians, other drivers, other nearby vehicles, etc., or the like. In at least one example, the emitter(s) <NUM> can be disposed at various locations about the exterior and/or interior of the vehicle <NUM>.

The vehicle <NUM> can also include communication connection(s) <NUM> that enable communication between the vehicle <NUM> and other local or remote computing device(s). For instance, the communication connection(s) <NUM> can facilitate communication with other local computing device(s) on the vehicle <NUM> and/or the drive module(s) <NUM>. Also, the communication connection(s) <NUM> can allow the vehicle to communicate with other nearby computing device(s) (e.g., other nearby vehicles, traffic signals, etc.). The communications connection(s) <NUM> also enable the vehicle <NUM> to communicate with a remote teleoperations computing device or other remote services.

The communications connection(s) <NUM> can include physical and/or logical interfaces for connecting the vehicle computing device(s) <NUM> to another computing device or a network, such as network(s) <NUM>. For example, the communications connection(s) <NUM> can enable Wi-Fi-based communication such as via frequencies defined by the IEEE <NUM> standards, short range wireless frequencies such as BLUETOOTH®, or any suitable wired or wireless communications protocol that enables the respective computing device to interface with the other computing device(s).

In at least one example, the vehicle <NUM> can include drive module(s) <NUM>. In some examples, the vehicle <NUM> can have a single drive module <NUM>. In at least one example, if the vehicle <NUM> has multiple drive modules <NUM>, individual drive modules <NUM> can be positioned on opposite ends of the vehicle <NUM> (e.g., the front and the rear, etc.). In at least one example, the drive module(s) <NUM> can include sensor system(s) to detect conditions of the drive module(s) <NUM> and/or the surroundings of the vehicle <NUM>. By way of example and not limitation, the sensor system(s) <NUM> can include wheel encoder(s) (e.g., rotary encoders) to sense rotation of the wheels of the drive module, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) to measure position and acceleration of the drive module, cameras or other image sensors, ultrasonic sensors to acoustically detect objects in the surroundings of the drive module, LIDAR sensors, RADAR sensors, etc. Some sensors, such as the wheel encoder(s) can be unique to the drive module(s) <NUM>. In some cases, the sensor system(s) on the drive module(s) <NUM> can overlap or supplement corresponding systems of the vehicle <NUM> (e.g., the sensor system(s) <NUM>).

The drive module(s) <NUM> can include many of the vehicle systems, including a high voltage battery, a motor to propel the vehicle <NUM>, an inverter to convert direct current from the battery into alternating current for use by other vehicle systems, a steering system including a steering motor and steering rack (which can be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and/or pneumatic components, a stability control system for distributing brake forces to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head/tail lights to illuminate an exterior surrounding of the vehicle), and one or more other systems (e.g., cooling system, safety systems, onboard charging system, other electrical components such as a DC/DC converter, a high voltage junction, a high voltage cable, charging system, charge port, etc.). Additionally, the drive module(s) <NUM> can include a drive module controller which can receive and preprocess data from the sensor system(s) and to control operation of the various vehicle systems. In some examples, the drive module controller can include processor(s) and memory communicatively coupled with the processor(s). The memory can store one or more modules to perform various functionalities of the drive module(s) <NUM>. Furthermore, the drive module(s) <NUM> also include communication connection(s) that enable communication by the respective drive module with other local or remote computing device(s).

As described above, the vehicle <NUM> can send sensor data to the computing device(s) <NUM>, via the network(s) <NUM>. In some examples, the vehicle <NUM> can send raw sensor data to the computing device(s) <NUM>. In other examples, the vehicle <NUM> can send processed sensor data and/or representations of sensor data to the computing device(s) <NUM> (e.g., data output from the localization system <NUM>, the perception system <NUM>, the prediction system <NUM>, and/or the planning system <NUM>). In some examples, the vehicle <NUM> can send sensor data to the computing device(s) <NUM> at a particular frequency, after a lapse of a predetermined period of time, in near real-time, etc..

The computing device(s) <NUM> can receive the sensor data (raw or processed) from the vehicle <NUM> and/or one or more other vehicles and/or data collection devices, and can determine a drive envelope based on the sensor data and other information. In at least one example, the computing device(s) <NUM> can include processor(s) <NUM> and memory <NUM> communicatively coupled with the processor(s) <NUM>. In the illustrated example, the memory <NUM> of the computing device(s) <NUM> stores an acceleration determination component <NUM> and agent information storage <NUM>, for example. In at least one example, the object information storage <NUM> can correspond to the object information storage <NUM> and/or the object information storage <NUM>.

The acceleration determination component <NUM> may correspond to the acceleration determination system <NUM> described above. For example, the acceleration determination component <NUM> may process data to determine an acceleration for the vehicle <NUM> to traverse along a drive path. For example, the acceleration may be determined at the computing device(s) <NUM> and transferred back to the vehicle <NUM>, e.g., via the networks <NUM>. Moreover, the acceleration determination component <NUM> can perform one or more operations as described above and ascribed to the localization system <NUM>, the perception system <NUM>, the prediction system <NUM>, and/or the planning system <NUM>.

The processor(s) <NUM> of the vehicle <NUM> and the processor(s) <NUM> of the computing device(s) <NUM> can be any suitable processor capable of executing instructions to process data and perform operations as described herein. By way of example and not limitation, the processor(s) <NUM> and <NUM> can comprise one or more Central Processing Units (CPUs), Graphics Processing Units (GPUs), or any other device or portion of a device that processes electronic data to transform that electronic data into other electronic data that can be stored in registers and/or memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices can also be considered processors in so far as they are configured to implement encoded instructions.

Memory <NUM> and <NUM> are examples of non-transitory computer-readable media. Memory <NUM> and <NUM> can store an operating system and one or more software applications, instructions, programs, and/or data to implement the methods described herein and the functions attributed to the various systems. In various implementations, the memory can be implemented using any suitable memory technology, such as static random-access memory (SRAM), synchronous dynamic RAM (SDRAM), nonvolatile/Flash-type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein can include many other logical, programmatic, and physical components, of which those shown in the accompanying figures are merely examples that are related to the discussion herein.

While <FIG> is illustrated as a distributed system, in alternative examples, components of the vehicle <NUM> can be associated with the computing device(s) <NUM> and/or components of the computing device(s) <NUM> can be associated with the vehicle <NUM>. That is, the vehicle <NUM> can perform one or more of the functions associated with the computing device(s) <NUM>, and vice versa. Moreover, although various systems and components are illustrated as being discrete systems, the illustrations are examples only, and more or fewer discrete systems may perform the various functions described herein.

<FIG> is a flowchart showing an example method involving acceleration determination as described herein. Specifically, <FIG> illustrates a method <NUM> in which objects in the environment are used to determine an acceleration of a vehicle along a drive path. The method illustrated in <FIG> is described with reference to the vehicles <NUM> and/or <NUM> shown in <FIG> and <FIG> for convenience and ease of understanding. However, the method illustrated in <FIG> is not limited to being performed using the vehicles <NUM>, <NUM>, and can be implemented using any of the other vehicles described in this application, as well as vehicles other than those described herein. Moreover, the vehicles <NUM>, <NUM> described herein are not limited to performing the method illustrated in <FIG>.

The method <NUM> is illustrated as a collection of blocks in a logical flow graph, which represents a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the blocks represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by processor(s), perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order and/or in parallel to implement the processes. In some embodiments, one or more blocks of the process can be omitted entirely. Moreover, the method <NUM> can be combined in whole or in part with other methods.

<FIG> is a flowchart illustrating an example method <NUM> for determining an acceleration for traversing in an environment along a planned path, as described herein.

At <NUM>, the method <NUM> can include determining a planned path through an environment. For example, and as described above, a vehicle <NUM> may include or be associated with one or more computing devices <NUM> that include the planning system <NUM>. The planning system <NUM> may receive information from one or more sensor systems <NUM>, the localization system <NUM>, the perception system <NUM>, the prediction system <NUM>, the maps <NUM>, and/or other sources to determine the drive path <NUM> along which the vehicle traverses through the environment <NUM>.

At <NUM>, the method <NUM> can include determining a first acceleration along the planned path. In the example illustrated in <FIG> and discussed above, the vehicle computing device <NUM> may determine that the vehicle <NUM> should maintain a predetermined distance behind the first additional vehicle 104a. Sensor data <NUM> from the sensor system(s) <NUM> may be used to determine a pose, a position, a velocity, an acceleration, and/or other information about the first additional vehicle 104a, and the acceleration determination system <NUM> may determine an acceleration required to maintain the predetermined distance. For example, when the first additional vehicle 104a accelerates away from the vehicle <NUM>, the acceleration determination system <NUM> may generate a command to accelerate along with the first additional vehicle 104a. Similarly, when the first additional vehicle 104a brakes, the acceleration determination system <NUM> may generate a command to decelerate the vehicle <NUM>. Although this example contemplates maintaining a predetermined distance between the vehicle <NUM> and the first additional vehicle 104a, modifications are contemplated. By way of non-limiting example, instead of a single predetermined distance between the vehicle <NUM> and the first additional vehicle 104a, the distance-to-be-maintained may vary based on the velocity of the first additional vehicle 104a. For instance, the vehicle <NUM> may be controlled to maintain a greater distance behind the first additional vehicle 104a at relatively higher speeds and a smaller distance at relatively slower speeds. In other implementations, the acceleration of the vehicle may be determined independent of the first additional vehicle 104a. For example, when the first additional vehicle 104a is not present or is exceeding a threshold speed, the vehicle <NUM> may be controlled to travel at the speed limit or some other speed along the travel path <NUM>.

At <NUM>, the method <NUM> can include receiving sensor data indicating an object in the environment. As described above, the vehicle <NUM> can include sensor system(s) <NUM> capable of generating sensor data about objects, including the additional vehicles <NUM> and the pedestrian <NUM>. In at least one example, the sensor system(s) <NUM> can include LIDAR sensors, RADAR sensors, ultrasonic transducers, SONAR sensors, location sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units, accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB, IR, intensity, depth, etc.), microphones, wheel encoders, environment sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), etc. The sensor system(s) <NUM> can provide input to the vehicle computing device(s) <NUM>, and one or more systems of the vehicle computing device(s) <NUM> can utilize the input. For instance, the vehicle computing device(s) <NUM> can receive the input for processing by the localization system <NUM>, the perception system <NUM>, the prediction system <NUM>, the planning system <NUM>, and/or the acceleration determination system <NUM>. As described above, in at least one example, the vehicle <NUM> can send sensor data to the computing device(s) <NUM>, via the network(s) <NUM>. In some examples, the vehicle <NUM> can send raw sensor data to the computing device(s) <NUM>. In other examples, the vehicle <NUM> can send processed sensor data and/or representations of sensor data to the computing device(s) <NUM> (e.g., data output from the localization system <NUM>, the perception system <NUM>, the prediction system <NUM>, and/or the planning system <NUM>). In some examples, the vehicle <NUM> can send sensor data to the computing device(s) <NUM> at a particular frequency, after a lapse of a predetermined period of time, in near real-time, etc. Based on the sensor data, the vehicle computing devices <NUM> may, e.g., via image processing, vision detection, classification and/or segmentation, LIDAR detection, segmentation, and/or classification, and/or other processes, recognize the objects. In addition to the example objects discussed herein, techniques may also determine additional details about the environment and/or features in the environment.

At <NUM>, the method <NUM> can include determining, for each object, a probability that the object will impact the planned path. As discussed above, the prediction system <NUM> may use sensor data generated by the sensor system(s) <NUM>, object information <NUM>, and/or additional information to determine a likelihood that each object will affect travel of the vehicle <NUM>. The probability may be a numerical probability, e.g., between <NUM> and <NUM>, which may be the highest probability from among multiple probabilities that the object will impact the planned path. For instance, each object or object may take any number of actions. However, only a subset of all those actions will actually impact travel of the vehicle <NUM>. Techniques described herein may identify the action from within that subset of actions having the highest probability of impacting travel, and use that probability at <NUM>. In some examples, the probability may be determined based at least in part on information about each of the objects, which information may include physical features associated with the objects, including but not limited to an extent of an object (e.g., height, weight, length, etc.), a pose of the object (e.g., x-coordinate, y-coordinate, z-coordinate, pitch, roll, yaw), a velocity of the object, an acceleration of the object, a direction of travel of the object (e.g., heading), a semantic classification, etc., as described herein. Additionally or alternatively, such features can include an orientation relative to a sensor (e.g., which direction is a pedestrian facing), distance to a map feature (e.g., a start of a crosswalk, an end of a crosswalk, a side of a crosswalk, a start of a junction, an end of a junction, a side of a junction, etc.), semantic features (e.g., crosswalk sign, traffic lights, etc.), feasibility of performing an intent (e.g., cross-ability of a crosswalk, etc.), etc. In some examples, data from one or more of the localization system, the planner system, the prediction system, and/or the perception system may be used at block <NUM> to determine the features. The features may also include a predicted trajectory of the object(s) and/or course of action. In at least one example, the vehicle may analyze the sensor data and/or the map data to determine an event associated with an object (e.g., a cut-in event, a crosswalk event, etc.). The prediction system <NUM> may then determine a probability of the event happening. In the context of <FIG>, <NUM> may include determining a probability that the first additional vehicle 104a will change lanes and determining a probability that the pedestrian will enter the road <NUM>.

At <NUM>, the method <NUM> can include determining, for each object, whether the probability is equal to or above a threshold probability. In some implementations, the vehicle computing device(s) <NUM> may determine that certain identified objects should be considered when determining an acceleration for the vehicle <NUM> and/or potential actions taken by the objects, whereas other objects should be ignored. For example, some objects may be of such a low likelihood of impacting travel that they may not be considered by the acceleration determination system <NUM> to determine the control acceleration. Using the example of <FIG>, if the second additional vehicle 104b was positioned substantially next to the vehicle <NUM>, instead of ahead of the vehicle <NUM>, the vehicle computing device(s) <NUM>, <NUM> may determine that there is a relatively low (i.e., below a threshold) probability that the second additional vehicle 104b will change lanes. In this instance, the vehicle computing device(s) <NUM>, <NUM> may not consider the second additional vehicle 104b when determining an acceleration, despite identifying the second additional vehicle 104b. Similarly, other objects that are far away from the vehicle <NUM>, are behind the vehicle <NUM>, that are not expected to move, e.g., parked cars, buildings, or the like, may have a sufficiently low probability of moving that they may not be considered when determining the acceleration. In some examples, such objects may be included. As a non-limiting example, the system may account for associated probabilities of parked cars moving into a drive lane. In examples, the threshold probability at which objects are considered may be any threshold, e.g., a non-zero threshold. In some examples, the threshold may be between about. <NUM> and about. <NUM> when probabilities range from <NUM> to <NUM>.

If it is determined at <NUM> that the probability that the object will enter the planned path is below the threshold, at <NUM> the method <NUM> can include disregarding the object in determining control acceleration. For example, because the object (and/or a predicted action of the object) is below the threshold probability, the object is effectively ignored for purposes of determining an acceleration at which to proceed. In this manner, although an object is identified as being in the environment <NUM>, it may not be necessary to control the vehicle <NUM>, e.g. accelerate or decelerate, in response to that object. As noted above, the sensor system(s) <NUM> may sense information about a large number of objects in the environment, but only certain of those objects may be considered likely to impact acceleration of the vehicle <NUM> as it traverses the environment. For example, objects may have a relatively low probability of impacting travel of the vehicle <NUM>, and thus may be may be excluded, based on their position relative to the vehicle <NUM>, based on their velocity relative to that of the vehicle <NUM>, or the like. As will be appreciated, each object.

Alternatively, if it is determined at block <NUM> that the probability that the object will impact travel along the planned path meets or exceeds the threshold, the method <NUM> may proceed to block <NUM>. At <NUM>, the method <NUM> may include determining a second acceleration for reacting to the object. In some examples, the second acceleration may be an acceleration at which the vehicle <NUM> would be controlled in the event that there is an extreme likelihood, e.g., a virtual certainty, that an object will enter the travel path <NUM> of the vehicle <NUM>. Thus, and again referencing the example of <FIG>, the second acceleration may be an acceleration (which may be a deceleration or a zero-acceleration) at which the vehicle <NUM> would be controlled if the second additional vehicle 104b changes lanes from the second lane <NUM> to the first lane and/or the pedestrian <NUM> walks into the road <NUM>. As described above, in some conventional systems, the vehicle may be controlled in accordance with the second acceleration upon a determination that the probability that the object, e.g., the second additional vehicle 104b and/or the pedestrian <NUM>, is going to enter the path of travel <NUM> is above a threshold.

At <NUM>, the method <NUM> can include determining a first weighting factor for the first acceleration and a second weighting factor for the second acceleration. For example, the vehicle computing device(s) <NUM>, e.g., executing the acceleration determination system <NUM>, may determine the first weighting factor as a predetermined, static weighting factor, and the second weighting factor as a function of the probability determined at <NUM>. By way of non-limiting example, the first weighting factor may be a weighting factor of <NUM>, whereas the second weighting factor may vary from <NUM> to <NUM>, depending upon the probability of the object impacting travel, e.g., entering the drive path <NUM>. For instance, the weighting factor may be linearly proportional to the probability. As described herein, other relationships between the probability and the weighting factor, and other values and scales for the weighting factors may also be used, and are contemplated by this disclosure. The weighting factors may vary based on the application, and based on a desired relation of the objects and accelerations. Moreover, the weighting factors may vary based on the type of the object. For instance, a pedestrian may be weighted differently than a vehicle, which may be weighted differently than other objects and/or events.

At <NUM>, the method <NUM> can include determining a third acceleration based on the first weighting factor, the first acceleration, the second weighting factor, and the second acceleration. For instance, as described herein, the acceleration determination system <NUM> may perform an optimization using the first weighting factor, the first acceleration, the second weighting factor, and the second acceleration. The optimization may determine the third acceleration as a minimum deviation from each of the candidate accelerations, e.g., the first acceleration and the second acceleration, while accounting for the weighting factors. The Equation (<NUM>) described above uses L2 norms and, accordingly, the third acceleration may be a weighted average of the first and second accelerations. While only a single second acceleration is used in the example of the method <NUM>, it will be appreciated that additional objects may also be considered in determining the control acceleration.

At <NUM>, the method <NUM> can include controlling the vehicle according to the third acceleration. As described above, the vehicle computing device(s) <NUM> can include one or more system controllers <NUM>, which can be configured to control steering, propulsion, braking, safety, emitters, communication, and other systems of the vehicle <NUM>. These system controller(s) <NUM> can communicate with and/or control corresponding systems of the drive module(s) <NUM> and/or other components of the vehicle <NUM>. In at least one example, the system controller(s) <NUM> can receive the acceleration as a component of a trajectory and can communicate with and/or control corresponding systems of the drive module(s) <NUM> and/or other components of the vehicle <NUM> such to cause the vehicle <NUM> to navigate along the trajectory.

An example of the method <NUM> now will be described with reference to <FIG>. Specifically, at <NUM>, vehicle computing device(s) <NUM>, e.g., using a planning system, may determine the vehicle path <NUM> through the environment <NUM>. At <NUM>, the vehicle computing device(s) <NUM>, e.g., using the planning system and/or the acceleration determination system <NUM>, may determine a first acceleration, which may be an acceleration for maintaining a predetermined distance from the first additional vehicle 104a in the first lane <NUM>. In this example, the acceleration may be <NUM>/s<NUM>. At <NUM>, the vehicle computing device(s) may receive the sensor data <NUM>, which indicates the presence of at least the second additional vehicle 104b and the pedestrian <NUM>. The sensor data <NUM> may also include information about the vehicle 104b and/or the pedestrian, including, velocity, acceleration, pose, an object type, an object classification, and/or other information. At <NUM>, the vehicle computing device(s) <NUM>, e.g., using a prediction system, may also determine a first probability that the second additional vehicle 104b will enter the vehicle path <NUM> and a second probability that the pedestrian will enter the vehicle path <NUM>. For example, the first probability may be <NUM> and the second probability may be <NUM>.

At <NUM>, the vehicle computing device(s) <NUM> may determine that the second probability is below a threshold, e.g., <NUM>, and thus, at <NUM>, may disregard the pedestrian in determining the acceleration. Also at <NUM>, the vehicle computing device(s) <NUM> may determine that the first probability (associated with the second additional vehicle 104b) exceeds the <NUM> threshold. Thus, at <NUM>, the vehicle computing device(s) <NUM> may determine a second acceleration for reacting to the second additional vehicle 104b. The second acceleration may be an acceleration at which the vehicle <NUM> would proceed if the second additional vehicle 104b is, in fact, executing a lane change into the first lane <NUM>. In this example, the second acceleration may be <NUM>/s<NUM>. Then, at <NUM>, the vehicle computing device(s) <NUM> may determine a first weighting factor for the first acceleration and a second weighting factor for the second acceleration. The first weighting factor may be fixed, e.g., the integer <NUM> in this example, and the second weighting factor may be based on the first probability calculated at <NUM>. In this example, the second weighting factor may be <NUM>. At <NUM>, the vehicle computing device(s) <NUM>, e.g., using the acceleration determination system <NUM>, may determine an acceleration for controlling the vehicle <NUM> based on the first acceleration, the first weighting factor, the second acceleration, and the second weighting factor. For example, the acceleration may be determined as an optimization, such as an optimization of the equation (<NUM>), discussed above. In some examples, and because the optimization may use an L2 loss function, the control acceleration may be <NUM>/s<NUM>, which is the weighted average of the two accelerations (e.g., (<NUM>/s<NUM>*<NUM> + <NUM>/s<NUM>*<NUM>)/<NUM> = <NUM>/s<NUM>). At <NUM>, the vehicle <NUM> is controlled to accelerate at <NUM>/s<NUM>.

In a conventional system, the <NUM> probability associated with the second additional vehicle 104b changing lanes may have resulted in the vehicle <NUM> being controlled to immediately proceed at the second acceleration (<NUM>/s<NUM>), instead of accelerating behind the accelerating first additional vehicle 104a. However, if at a next time interval, the second probability has decreased below some threshold, e.g.,. <NUM>, the vehicle <NUM> may accelerate quickly (even more quickly than the <NUM>/s<NUM> first acceleration) to return to a predetermined distance behind the first additional vehicle 104a. In contrast, according to techniques described herein, a similar change in probability associated with the second additional vehicle 104b would not result in the drastic changes in acceleration plaguing conventional systems.

As will be appreciated, as the vehicle moves through the environment, objects in the environment are ever-changing. Accordingly, the method <NUM> may be carried out iteratively. For example, status of objects in the environment, including probabilities that those objects will enter the path of travel of the vehicle <NUM>, may be determined in near real-time, e.g., at intervals of from about. <NUM> seconds to about <NUM> seconds. Moreover, as objects change positions, velocities, attitudes, and the like, relative to the vehicle, objects previously excluded from the acceleration determination may be included and objects previously included may be excluded, e.g., as probabilities associated with those objects change.

Although not illustrated in <FIG>, in some implementations techniques described herein may determine that an event is so likely to occur that the vehicle will react to that event. For example, at <NUM> in <FIG> it is determined whether a probability that an object will impact a planned path meets or exceeds a threshold. In some implementations, that threshold may be a first threshold and another block may include determining whether the probability meets or exceeds a second, higher probability. In such an example, if the probability meets or exceeds the second probability, techniques described herein may determine that it is very likely that the object will impact the planned path, and thus determine to control the vehicle according to the second acceleration for reacting to the object. Accordingly, when an object is very likely to impact travel along the planned path, the vehicle may react only to that object, e.g., without an optimization step. In some examples, the second threshold may be. <NUM> or some other probability value.

The various techniques described herein can be implemented in the context of computer-executable instructions or software, such as program modules, that are stored in computer-readable storage and executed by the processor(s) of one or more computers or other devices such as those illustrated in the figures. Generally, program modules include routines, programs, objects, components, data structures, etc., and define operating logic for performing particular tasks or implement particular abstract data types.

Other architectures can be used to implement the described functionality, and are intended to be within the scope of this disclosure. Furthermore, although specific distributions of responsibilities are defined above for purposes of discussion, the various functions and responsibilities might be distributed and divided in different ways, depending on circumstances.

Similarly, software can be stored and distributed in various ways and using different means, and the particular software storage and execution configurations described above can be varied in many different ways. Thus, software implementing the techniques described above can be distributed on various types of computer-readable media, not limited to the forms of memory that are specifically described.

Claim 1:
A vehicle comprising:
a sensor;
one or more processors; and
memory storing processor-executable instructions that, when executed by the one or more processors, configure the vehicle to:
determine a planned path of the vehicle through an environment;
determine a first acceleration of the vehicle along the planned path;
receive sensor data from the sensor, the sensor data comprising information about an object in the environment;
determine, based at least in part on the sensor data, that a probability that the object will impact travel of the vehicle along the planned path meets or exceeds a threshold probability;
determine a second acceleration of the vehicle along the planned path, the second acceleration being an acceleration of the vehicle responsive to the object impacting travel of the vehicle along the planned path;
determine, as a function of at least the first acceleration and the second acceleration, a third acceleration of the vehicle along the planned path, the third acceleration being different from the first acceleration and the second acceleration; and
control the vehicle to accelerate along the planned path in accordance with the third acceleration.