Patent Description:
Advances in mobile machine automation are proceeding at a rapid pace as more companies become involved in development of automation solutions. Generally, mobile machine automation can require highly accurate sensing of the environment in which the mobile machine is operating (e.g., the path being followed by the mobile machine, other mobile machines, stationary objects, obstructions in the pathway, etc.). The control mechanisms for the mobile machine are also required to be highly accurate and resistant to failure when components within the mobile machine fail, retaining control of the mobile machine and continuing operation or bringing the mobile machine to a stop.

Ideally, the mobile machine departs from a location and arrives at a destination location uneventfully, in some cases transporting payloads to a destination without any noticeable signs of unexpected events or behaviors. To support this goal, high performance computer systems are often employed in the automation controller to sense the environment, plan a trajectory, and implement the trajectory by controlling the mobile machine. If a failure occurs, the automation controller can stop the mobile machine. However, determining that a failure has occurred in the system is complex and difficult to implement. If the automation controller is too conservative and detects failure more often than failure actually occurs, it may exhibit unexpected behavior and payloads may be delayed or disrupted. On the other hand, if the automation controller is too aggressive and does not detect failure when failure actually occurs, the mobile machine may be unable to complete its mission. <CIT> is a document of the prior art, which discloses computing stop trajectories for use if an adverse event occurs.

The following detailed description refers to the accompanying drawings, which are now briefly described.

While embodiments described in this disclosure may be susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. The headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description.

<FIG> is a block diagram of one embodiment of an automation system <NUM>. In the illustrated embodiment, the system <NUM> may include a plurality of sensors <NUM>, one or more first computers <NUM>, one or more second computers <NUM>, and a plurality of actuators <NUM>. The sensors <NUM> are coupled to the computers <NUM>, which are coupled to the computers <NUM>. The computers <NUM> are coupled to the actuators <NUM>.

In some embodiments, the computers <NUM> receive a destination (e.g., from a user input device, a storage device, a network interface, etc.). In some embodiments, the automation system <NUM> is to direct a mobile machine using the actuators <NUM> from its current location to the destination. For example, global position system (GPS) or other geo-location data such as triangulation from cell towers or the like may be used to determine the current location, and map information may be used with various navigation algorithms to determine a route from the current location to the destination. In an embodiment, current environmental conditions (e.g., traffic conditions, weather conditions, potential obstacles in the route, etc.) may be considered in determining the route as well, attempting to minimize travel time.

More generally, the automation system <NUM> may be configured, in some embodiments, to control the motion of the mobile machine through space. The mobile machine may be any machine capable of spatial movement, even if the space in which the machine is capable of moving is restricted. For example, the mobile machine may be a robot that performs a task within a confined space. The robot may be fixed to the floor, but may have appendages (e.g., "arms") that automatically perform (under the control of the automation system <NUM>) a defined task such as assembly, welding, painting, and the like. The robot may be fully mobile, and may be designed to move payloads from location to location in a warehouse, factory, or the like. The robot may be configured to perform a task that includes motion (e.g., a vacuum cleaner that automatically vacuums a floor, in which the payload may be the refuse collected by the vacuum cleaner; an automated lawn mower that cuts a lawn; etc.). Mobile machines may include any type of land, water, air, or space-based vehicles such as cars powered by internal combustion gasoline engines, diesel engines, hybrid engines, full electric engines, etc. The mobile machine may include pickup trucks and larger trucks that transport goods (e.g., "semis"). The mobile machine may include buses and other mass transportation vehicles. The mobile machine may include motorcycles. Water-based mobile machines may include boats, ships, submarines, sailboats, etc. Air-based mobile machines may include aircraft (e.g., airplanes, drones, blimps, helicopters, etc.). Space-based vehicles may include rockets, capsules, satellites, space stations, etc..

Mobile machines may move along a pathway under control of the automation system <NUM>. The pathway may be any permitted route from the current location to a destination, and may depend on the character of the mobile machine. For example, the pathway may be a land-based pathway (e.g., a roadway, a sidewalk, a pathway defined within a building, a path that covers a prescribed area, etc.), a water-based pathway (e.g., a channel, shipping lanes, a riverbed, etc.), an airspace-based pathway (e.g., the direction of movement of a robotic arm, air travel routes, etc.), or space-based pathways (e.g., orbits, launch flight paths, etc.). Pathways may include one or more paths in which the mobile machine may travel (e.g., the lanes of a roadway, multiple shipping lanes, parallel paths on a warehouse floor, flight paths in the air, orbital paths, etc.).

The automation system <NUM> may be configured to initiate motion in the mobile machine, sensing the pathway as well as various other objects in the area around the mobile machine using the sensors <NUM> (communicated as "sensor data" to the computers <NUM> in <FIG>). The objects may include other mobile machines, pedestrians, animals, fixed objects such as buildings, signs, trees, other large flora, etc., other moving objects, furniture and other indoor furnishings for inside pathways, etc. The automation system <NUM> may be configured to plan a destination trajectory <NUM> which guides the mobile machine toward the destination, avoiding other objects (e.g., obstructions or obstacles) detected by the sensors <NUM>. The destination trajectory may not be the entire route, but may rather be the trajectory planned for the near term but generally along the route to the destination. The destination trajectory may be planned for several seconds or minutes into the future, and may be updated periodically based an additional sensor data from the sensors <NUM> and the progress of the mobile machine towards the destination. The destination trajectory is sometimes also referred to as the "long term trajectory.

In addition to the destination trajectory <NUM>, the automation system <NUM> may determine a plurality of stop trajectories as the mobile machine moves along the destination trajectory. Each stop trajectory may correspond to a point along the destination trajectory, and may guide the mobile machine from the current point to a stopping point in the event that there is a failure in the system and the destination trajectory cannot continue to be updated and/or new stop trajectories cannot be generated. For example, the sensors <NUM> may fail in whole or in part, leaving the automation system <NUM> with less information regarding nearby objects. The computers <NUM> may fail in whole or in part, preventing the processing of sensor data to produce new stop trajectories and updated destination trajectories. Communications mechanisms between the sensors <NUM> and the computers <NUM>, or between the computers <NUM> and the computers <NUM>, may fail. In some embodiments, failures in the mobile machine itself may be detected and cause a stop. Failures may be temporary (e.g., occluded sensors, crashes of the computer systems <NUM>, noise on the interfaces between the components, etc.) or more permanent hardware failures. A "stop" may generally refer to halting mobile machine motion without colliding with other objects (at least as most recently detected via the sensors <NUM>) or otherwise damaging the mobile machine or endangering the mobile machine's payloads, if any. The stop may be planned to be smooth, decelerating at a reasonable pace that does not agitate or damage the payloads and moving to the stopping point without any rapid maneuvers such as swerves or the like, in an embodiment. If the location permits, the stopping point may be planned to be off the active pathway (e.g., in a stop zone) so as not to impede the movement of other mobile machines that may be on the active pathway. Stopping in the active pathway may be used if the situation does not permit stopping in a non-travel area such as a stop zone.

The computers <NUM> may be configured to transmit the stop trajectories to the computers <NUM>, which may be configured to process the trajectories into actuator commands for the actuators <NUM>. Since the destination trajectory is not provided, the computers <NUM> may be simplified and need not make decisions about when the destination trajectory is no longer useable (e.g., failure has occurred) and switching to the stop trajectory is needed. The computers <NUM> may simply follow the most recently provided stop trajectory. If the stop trajectories are provided at the expected time intervals (e.g., on the order of <NUM> millisecond intervals, or <NUM> millisecond intervals, or any time interval specified by the system), the difference between the stop trajectories and the destination trajectory within the interval may be minimal (e.g., less than a predefined threshold) and thus the destination trajectory may be followed as long as no failure occurs. If failure does occur, the computers <NUM> do not receive updated stop trajectories and may follow the most recently provided trajectory to bring the mobile machine to a stop.

As illustrated in <FIG>, there may be three phases to the processing of sensor data into control for the mobile machine: Sense (reference numeral <NUM>), plan (reference numeral <NUM>), and act (reference numeral <NUM>). The computers <NUM> may handle the sense and plan phases <NUM> and <NUM>, and the computers <NUM> may handle the act phase <NUM>. The computers <NUM> may receive sensor data from the sensors <NUM> and may process the data to generate a model of the surroundings and the mobile machine itself. The surroundings (e.g., location and identification of objects, location of the pathway, etc.) may be described by a "world estimate" provided to the plan phase. Additionally, motion estimates for objects in motion (and/or for the mobile machine itself) may be generated so that the plan phase <NUM> may adjust for the movement of objects as the mobile machine moves forward. The sensor data provided by the sensors <NUM> may be raw sensor data, or may be processed locally at the sensors <NUM> to produce a list of objects and relative locations to the mobile machine, or any combination thereof. For example, in one embodiment, the sensors may include cameras, radio detection and ranging (radar) sensors, and light detection and ranging (lidar) sensors. The radar and lidar sensor data may be preprocessed to identify objects, while the camera data may be provided raw. Other embodiments may implement different sensors and different combinations of preprocessed and/or raw sensor data.

In the plan phase <NUM>, the computers <NUM> may use the world estimate, motion estimates, and destination information to update the destination trajectory <NUM> and to generate another stop trajectory. The destination trajectory <NUM> may be kept local to the computers <NUM>, while the stop trajectories may be transmitted to the computers <NUM>. In an embodiment, the destination trajectory <NUM> may not be explicitly generated and stored, but rather may be a factor in determining the stop trajectories.

In the act phase <NUM>, the computers <NUM> may process the current (most recently received) stop trajectory to produce actuator commands to control actuators in the mobile machine. For example, acceleration, deceleration, and guidance (e.g., steering) actuators may be provided.

In an embodiment, the computers <NUM> may be one or more high performance computers that implement significant processing capacity to perform the sense and plan phases <NUM> and <NUM>. For example, computationally intensive neural networks may be used as part of the sense and/or plan phases <NUM> and <NUM>, and the computers <NUM> may include neural network accelerators designed to provide high performance processing of the neural networks. A certain amount of redundancy may be implemented to provide for fail degraded or fail operational operation in the phases, in an embodiment.

The computers <NUM> may be somewhat lower performance, but environmentally-hardened computers that implement a different level of redundancy. The computers <NUM> may be fail operational or partially fail operational in various embodiments.

Any combination of sensors may be placed at various locations in the mobile machine and may be configured to monitor different fields of view around the mobile machine. For example, sensors such as camera sensors (cameras), radar sensors, lidar sensors, etc. may be included. The camera may be any sort of sensor that captures a visible light image of the field of view. The camera output may be a set of pixels which indicate the color/intensity of light at that position within the frame (or picture) captured by the camera. Other types of cameras may capture other wavelengths of light (e.g., infrared cameras). The camera sensor may be a passive sensor if the sensed wavelengths is/are prevalent in the environment and reflected by objects in the environment (e.g., visible light) or are actively emitted by the objects. A given camera sensor may be an active sensor if the camera actively emits the light and observes any reflected light (e.g., infrared light).

A radar sensor may be an active sensor that emits electromagnetic waves in the radio spectrum (radio waves) and/or microwave spectrum, and observes the reflection of the radio waves/microwaves to detect objects that reflect radio waves. Radar may be used to detect the range of an object (e.g., a position and distance), velocity of the object, etc. A lidar sensor may also be an active sensor that emits electromagnetic waves having wavelengths in the light spectrum (light waves) and observing the reflections of the emitted waves. For example, lidar sensors may emit infrared wave pulses from lasers and detect reflected pulses. Other lidar sensors may use lasers that emit other wavelengths of light such as ultraviolet, visible light, near infrared, etc. Like radar, the lidar sensor may be used to detect range, velocity, etc. Additional sensors beyond those described above may be used (e.g., ultrasonic, etc.) in addition to and/or as alternatives to the above-mentioned sensors.

In accordance with this description, in one embodiment, an automation system may comprise one or more first computers configured to periodically generate a trajectory for a mobile machine to bring the mobile machine to a stop based on sensor data up to a point in time at which the trajectory is generated. The sensor data is received from a plurality of sensors on the mobile machine that sense the surroundings of the mobile machine. One or more second computers are coupled to the one or more first computers and are configured to control a plurality of actuators in the mobile machine. The one or more second computers are configured to periodically receive the trajectory from the one or more first computers, and to control the plurality of actuators to cause the mobile machine to follow a current instance of the trajectory. The one or more second computers are configured to replace the current instance of the trajectory with a subsequent instance of the trajectory received from the one or more first computers, and wherein the one or more second computers are configured to control the plurality of actuators to follow the subsequent version of the trajectory received from the one or more first computers.

As mentioned above, stop trajectories may be generated at various points in time, at a regular interval. For example, a first stop trajectory may correspond to a time t<NUM>. Similarly, a second stop trajectory may correspond to a time t<NUM>; a third stop trajectory may correspond to a time t<NUM>; etc. The time between t<NUM> and t<NUM> may be approximately equal to the time between t<NUM> and t<NUM>, etc. Each stop trajectory may remain approximately parallel to the destination trajectory for an initial segment up until the next point in time at which a stop trajectory is generated. Thus, the first stop trajectory may parallel the destination trajectory from time t<NUM> until time t<NUM>; the second stop trajectory may parallel the destination trajectory <NUM> from time t<NUM> to time t<NUM>, and the third stop trajectory may parallel the destination trajectory <NUM> from time t<NUM> to time t<NUM>. Thus, the times t<NUM>, t<NUM>, t<NUM>, t<NUM>, etc. may correspond to the periodicity at which the stop trajectories are to be generated. That is, the difference in time between two consecutive times may be the period for generating the stop trajectories (and updating the destination trajectory as well to reflect newly sensed objects or changes in the objects that are in motion).

The initial segment may be generated in any fashion. For example, the initial segment may be selected from the destination trajectory <NUM> and the remaining segment may deviate from the destination trajectory to the stop point. Alternatively, the plan phase <NUM> may be configured to generate the initial segment of the stop trajectories so that they do not deviate from the destination trajectory <NUM> by more than a threshold amount until after the next stop trajectory should be generated.

The stop trajectory generated at a given point in time may bring the mobile machine to a halt at a location selected by the plan phase <NUM>. Generally, the stop location or stop point may be determined based on one or more stop criteria. The stop criteria may include any desired rules that have a high likelihood of maneuvering to a stop point without colliding with any other objects, causing damage, etc. For example, various traffic conventions may be in place for the mobile machine movement in the presence of other mobile machines (e.g., roadway conventions, waterway conventions, airpath conventions, etc.), and the conventions may be used in identifying a stop point and the stop trajectory to reach the stop point. For example, the plan phase <NUM> may attempt to select a stop location that is least exposed to other traffic. The stop location may be the location that the mobile machine may reach without colliding with other objects and that is least obstructive to continuing movement in the pathway. Thus, if there is a stop zone to a side of the pathway, the stop point may be the stop zone. A stop zone may be any location outside of the pathway at which stopping is permissible or desirable in the overall environment. In a land-based environment, for example, a stop zone may include a shoulder of a roadway, a driveway, a designated location in a warehouse or other building, a charging port or other fueling point for the mobile machine, etc. In a water-based environment, a stop zone may include a dock, a mooring buoy, a river bank, a lake shore, a sea shore, an ocean shore, a channel boundary, etc. In air-based environment, a stop zone may include a landing zone, a helipad, an empty field, etc. If there is no stop zone, the stop point may be in the current path. If there are multiple paths, a convention may be used for selecting a stop point (e.g., the rightmost path, followed by leftmost path, followed by in-path, where right and left are based on the direction of travel).

Accordingly, in an embodiment, the automation system (e.g., the one or more first computers) may be configured to generate a stop trajectory based on the path in which the mobile machine is traveling (of one or more possible paths, for a given pathway), presence of one or more other mobile machines in the one or more paths, and/or presence of a stop zone bordering the one or more paths. For example, the one or more first computers may be configured to generate the trajectory to a point on a stop zone bordering a pathway on which the mobile machine is traveling based on existence of an unobstructed path between the mobile machine and the stop zone. In another example, the one or more first computers are configured to generate the trajectory to stop in a current path of travel for the mobile machine based on one or more obstructions between the mobile machine and the stop zone. In yet another example, the one or more first computers are configured to generate the trajectory to stop in a current path of travel for the mobile machine based on an absence of the stop zone. In this context, the stop zone may be the stop zone on the right side of the mobile machine. In some cases, a stop zone on the left side of the mobile machine may be used if it is present and there is an unobstructed path the to the left stop zone (e.g., from a left path of travel).

In an embodiment, an automation system may comprise one or more first computers configured to generate a plurality of trajectories at a plurality of points in time. A given trajectory of the plurality of trajectories includes a first segment and a second segment delineated by a difference between respective points in time of the plurality of points in time. The first segment may be derived from a destination trajectory that leads a mobile machine toward a destination, and the second segment may be derived from a stop trajectory that leads the mobile machine to a stop based on sensor data from a plurality of sensors up to a given point in time at which the given trajectory is generated. One or more second computers are coupled to the one or more first computers, wherein the one or more second computers may be configured to receive the plurality of trajectories and, at given time, are configured to control a plurality of actuators of the mobile machine to follow a most recently received trajectory of the plurality of trajectories.

<FIG> is flowchart illustrating one embodiment of certain operations of the act phase <NUM>. While the blocks are shown in a particular order for ease of understanding, other orders may be used. The act phase <NUM> may comprise a plurality of instructions which, when executed by the computers <NUM>, may cause operations including the operations described in <FIG>.

The act phase <NUM> may include generating actuator commands based on the current stop trajectory (block <NUM>). That is, the commands may cause the actuators to move the mobile machine along the stop trajectory. The act phase may continue generating commands based on the current stop trajectory until a new stop trajectory is received from the plan phase <NUM> (decision block <NUM>, "yes" leg) and the newly received trajectory is authenticated and validated (decision block <NUM>, "yes" leg), at which time the act phase <NUM> may replace the current stop trajectory with the new stop trajectory (block <NUM>). The new stop trajectory thus becomes the current stop trajectory, and actuator commands may be generated based on the new stop trajectory (block <NUM>). Authentication and validation may be performed to ensure that the trajectory was indeed generated by the plan phase <NUM> (and not, e.g., by a nefarious intrusion into the system) and to verify that the trajectory has not been changed. That is, the data describing the new stop trajectory may be cryptographically signed and may include error detection data to allow for detection of a change to the data (e.g., due to noise or other incorrect operation in the transmission of the trajectory to the act phase <NUM>). A received stop trajectory that fails authentication and/or validation may be treated as if the trajectory were not received, in an embodiment. There may be error logging in the system, in an embodiment, and the failure (and perhaps a copy of the data) may be logged in the error log, in an embodiment.

<FIG> is a flowchart illustrating one embodiment of certain operations of the plan phase <NUM>. While the blocks are shown in a particular order for ease of understanding, other orders may be used. The plan phase <NUM> may comprise a plurality of instructions which, when executed by the computers <NUM>, may cause operations including the operations described in <FIG>.

The operation illustrated in <FIG> may occur during one time interval (e.g., the difference between times t<NUM> and t<NUM>, times t<NUM> and t<NUM>, etc. as discussed above). The plan phase may process sensor data received during the interval (and in some cases, sensor data from preceding time intervals) to update the destination trajectory (block <NUM>). For example, newly identified objects in the surroundings may change the destination trajectory, and/or the destination trajectory may be extended to replace the portion that has been traversed with an additional planned portion farther ahead.

The plan phase <NUM> may also generate a new stop trajectory. An initial segment of the stop trajectory (e.g., the distance covered up to the next time interval, which depends on the speed of the mobile machine in addition to the size of the time interval) may be derived from the corresponding segment of the destination trajectory (block <NUM>). For example, the initial segment of the stop trajectory may be equal to the corresponding segment of the destination trajectory. Alternatively, the initial segment of the stop trajectory may vary from the corresponding segment of the destination trajectory by no more than a threshold amount that leads to a lack of agitation of payloads (e.g., to avoid damage to fragile payloads, to prevent shifting or movement of payloads, to avoid jarring live payloads in a fashion that alarms the live payload, etc.) Thus, the initial segment of the trajectory may deviate from the destination trajectory by less than a predetermined amount for a period of time that is based on the periodicity of the instances of the trajectory.

The remaining segment of the stop trajectory may be generated by considering the various objects and other information in the sensor data. In an embodiment, if there is an occlusion on the destination trajectory (decision block <NUM>, "yes" leg), the stop trajectory may be generated only in the visible area (block <NUM>). For example, the mobile machine may be approaching an intersection of two pathways at which the destination trajectory makes a turn. The intersection maybe the intersection in a roadway, for example, or may be an intersection of two hallways in a building, or the like. The sensors may not be able to sense around the corner until the mobile machine is quite close to the corner (e.g., if there are buildings, large trees, walls, etc. on the side of the pathway). In such situations, the stop trajectory may extend beyond the intersection down the current pathway, for example, instead of attempting the turn.

Within the visible area, the plan phase <NUM> may determine if there is an unimpeded path to the rightmost stopping point (e.g., a stop zone, or the right path if there is no stop zone) (decision block <NUM>). If so (decision block <NUM>, "yes" leg), the plan phase <NUM> may select the move to rightmost option (block <NUM>). If not (decision block <NUM>, "no" leg), the plan phase <NUM> may determine if the mobile machine is in the leftmost path and there is a left stop zone (decision block <NUM>). In some embodiments, the mobile machine may not be in the leftmost path but may be within N paths of the leftmost path and may use the left stop zone if there is an unimpeded path to the left stop zone. If there is an unimpeded path to the left stop zone (decision block <NUM>, "yes" leg), the plan phase <NUM> may select the move to leftmost trajectory (block <NUM>). If both the left stop zone and right stop zone cases, the plan phase <NUM> may also consider if there are any obstacles in the stop zone itself. If there is no unimpeded path the left stop zone, or there is no left stop zone, or the mobile machine is not in the leftmost path or N leftmost paths) (decision block <NUM>, "no" leg), the plan phase <NUM> may select the stop in path trajectory (block <NUM>). The remaining segment of the stop trajectory may be generated based on the selected trajectory (move to rightmost, move to leftmost, or stop in path) (block <NUM>). While the rightmost is favored over the leftmost in this example, other examples may favor the leftmost over the rightmost.

The stop trajectory may include a path to follow to the selected stopping point, and may also generally include a deceleration from the current speed to stopped over the path. The deceleration may be capped at a maximum rate, so that the stop does not agitate the payload, if any. Additionally, the stop may be required to occur within a maximum amount of time (e.g., <NUM>-<NUM> seconds, although the time may higher or lower in various embodiments). Since the stop trajectory is followed to completion when a failure occurs and new stop trajectories are not being provided, the traversal of the stop trajectory may be performed without additional sensor data input (e.g., the automation system may be "blind" to changes in the surroundings). Accordingly, limiting the length of the stop trajectory may reduce the likelihood that an object is introduced into the path described by the stop trajectory.

In an embodiment, the automation system <NUM> may be further enhanced with an automatic exception stop (ES) trajectory that may modify the stop trajectory in the event that an object is introduced into the path after a failure has occurred (a "late reveal" object). The exception stop may be activated if detected while traversing the stop trajectory because of a failure in providing new stop trajectories. The late reveal object may be a previously undetected object that appears in the most recently received instance (or current instance) of the stop trajectory, for example. The late reveal object may be an object that is in motion and that unexpectedly moves into the stop trajectory, or that moves into the field of view of the sensors <NUM> on the mobile machine as a result of its motion.

<FIG> is a block diagram of one embodiment of an automation system <NUM> that includes exception stop features. The automation system <NUM> in <FIG> may include the sensors <NUM>, the computers <NUM>, the computers <NUM>, and the actuators <NUM> similar to the embodiment of <FIG> (with the computers <NUM> implementing the sense and plan phases <NUM> and <NUM> and storing the destination trajectory <NUM>, and the computers <NUM> implementing the act phase <NUM>). Additionally, the automation system <NUM> may include one or more computers <NUM>. The computers <NUM> may be coupled to a subset of the sensors <NUM> and/or to additional sensors <NUM>, and may be coupled to the computers <NUM> and <NUM>. More particularly, the computers <NUM> may be configured to receive the stop trajectories from the computers <NUM>, and may be configured to provide an exception stop indication to the computers <NUM>.

In an embodiment, the computers <NUM> may be configured to detect that one or more instances of the stop trajectory have not been received for at least N iterations of the period at which the subsequent instance should have been received. If N iterations occur in which a stop trajectory is not received, the computers <NUM> may be configured to enable the exception stop indication from the computers <NUM> (that is, the computers <NUM> may be a different source than the computers <NUM> for trajectory data). If stop trajectories are being successfully received, the computers <NUM> may be configured to disable the exception stop indication, in an embodiment.

In the context of "N iterations," "N" may be a positive integer, and may be set to any desired number of iterations in an embodiment. For example, N may be <NUM> or <NUM>, or even a higher number or <NUM>, in various embodiments. Generally, it may be desirable for N to be greater than <NUM> to eliminate intermittent errors such as noise-induced trajectory losses and the like, but a relatively small integer to provide rapid response to a sustained loss of trajectories from the computers <NUM>.

The exception stop indication may include an assertion that an exception stop condition has been detected, as well as a description of the exception stop trajectory. In an embodiment, the exception stop trajectory may follow the most recent stop trajectory but may modify one or more aspects of the trajectory. For example, the exception stop trajectory may terminate the original stop trajectory early (e.g., the exception stop trajectory may include a more rapid deceleration than the original stop trajectory). The exception stop trajectory may modify the original stop trajectory with an evasive maneuver to avoid the late reveal object. In other embodiments, the exception stop trajectory may be independent of the most recent stop trajectory and may be provided in a similar format to the stop trajectories.

The computers <NUM> may receive sensor data from a subset of the sensors <NUM> and/or from one or more ES sensors <NUM>. Any combination of the subset of the sensors <NUM> and the ES sensors <NUM> may be used. That is, embodiments that employ only a subset of the sensors <NUM>, only the ES sensors <NUM>, and a combination of the subset of the sensors <NUM> and the ES sensors <NUM> are contemplated. The sensors <NUM> may supply the ES sensor data using a different connection to the computer <NUM> than the connection to the computers <NUM> (to protect against failure in the path of the sensor data). In an embodiment, the sensors <NUM> that provide the ES sensor data may provide a lower bandwidth stream of sensor data (e.g., the data may be lower resolution than the data provided by the corresponding sensor to the computers <NUM>). Accordingly, the exception stop may be a coarser decision-making process than the planning of the destination trajectories and stop trajectories by the computers <NUM>. In some embodiments, the exception stop may be subject to more "false positive" identifications of obstacles in the path. However, since the exception stop functionality is provided to avoid collisions when a failure in the main system (e.g., computers <NUM> or one or more of the sensors <NUM>), some number of false positives may be acceptable.

The computers <NUM> may implement a sense phase <NUM> and a plan phase <NUM> to detect late reveal objects/obstacles in the stop trajectories and to generate the exception stop indication for the computers <NUM>. The sense phase <NUM> may process the receive ES sensor data in a manner similar to the sense phase <NUM>, and may provide a world estimate to the ES plan phase <NUM> (or simply a set of objects) and the ES plan phase <NUM> may plan the ES stop trajectory based on the sensed objects.

<FIG> is a flowchart illustrating certain operation of one embodiment of the act phase <NUM> with exception stop functionality. While the blocks are shown in a particular order for ease of understanding, other orders may be used. The act phase <NUM> may comprise a plurality of instructions which, when executed by the computers <NUM>, may cause operations including the operations described in <FIG>.

Similar to the embodiment of <FIG>, the act phase <NUM> may generate actuator commands based on the current trajectory (block <NUM>), and may replace the current trajectory with a new stop trajectory if received, authenticated, and validated (decision blocks <NUM> and <NUM>, "yes" legs and block <NUM>). In the embodiment of <FIG>, however, the current trajectory may either be a most-recently received stop trajectory or an exception stop trajectory.

If a new stop trajectory is not received at the expected point in time (decision block <NUM>, "no" leg) or the trajectory is received but does not authenticate or validate correctly (decision block <NUM>, "no" leg), the act phase <NUM> may record that the expected stop trajectory was not received and may determine if the most recent N expected stop trajectories have not been received (decision block <NUM>). If N consecutive stop trajectories have not been received (decision block <NUM>, "yes" leg), the act phase <NUM> may enable the exception stop functionality (block <NUM>). For example, the act phase <NUM> may begin monitoring, or "listening" to the exception stop indication from the computers <NUM>. If the exception stop indication is received (decision block <NUM>, "yes" leg), the act phase <NUM> may replace the current stop trajectory with the exception stop trajectory (or modify the current stop trajectory as specified by the exception stop indication) (block <NUM>). The act phase <NUM> may begin generating actuator commands based on the replaced/modified trajectory (block <NUM>). If the ES stop indication is not received (decision block <NUM>, "no" leg), the act phase <NUM> may continue following the current trajectory (block <NUM>). Similarly, if the number of consecutive stop trajectories that were not received has not reached N (decision block <NUM>, "no" leg), the act phase <NUM> may continue generating actuator commands based on the current trajectory (block <NUM>).

As discussed previously, a wide variety of mobile machines and corresponding pathways are contemplated, from land-based to water-based to air-based and even space-based. One embodiment that may be particularly familiar is an automobile or other vehicle operated on a roadway. Further examples with regard to this embodiment are presented below with regard to <FIG>.

<FIG> is a block diagram of one embodiment of a mobile machine M1 traveling along a roadway, and a mobile machine M2 in front of M1. The roadway includes two lanes, both with a direction of travel from left to right as illustrated in <FIG>. For a portion of the roadway illustrated in <FIG>, there is a stop zone (e.g., a shoulder, a parking area, a driveway, etc.) on the right side of the pathway (where right is viewed from the direction of travel) (reference numeral <NUM>). The stop zone ends toward the middle of <FIG>, and thus there is no stop zone for the remainder of the roadway as illustrated (reference numeral <NUM>).

The position of the mobile machine M1 corresponds to a time t<NUM>, as illustrated in <FIG>. The destination trajectory is illustrated as a dotted line <NUM> in <FIG>. As illustrated, the mobile machine M1 continues in the right lane until it begins to overtake the mobile machine M2 and passes the mobile machine by changing to the left lane.

Also illustrated in <FIG> are various stop trajectories 36A, 36B, and 36C. The stop trajectory 36A corresponds to the time t<NUM>. Similarly, the stop trajectory 36B corresponds to a time t<NUM> and the stop trajectory 36C corresponds to the time t<NUM>. Each stop trajectory may remain approximately parallel to the destination trajectory <NUM> for an initial segment up until the next point in time at which a stop trajectory is generated. Thus, the stop trajectory 36A parallels the destination trajectory <NUM> until time t<NUM>, the stop trajectory 36B parallels the destination trajectory <NUM> from t<NUM> to t<NUM>, and the stop trajectory 36C parallels the destination trajectory <NUM> from t<NUM> to t<NUM>. Thus, the times t<NUM>, t<NUM>, t<NUM>, t<NUM>, etc. may correspond to the periodicity at which the stop trajectories are to be generated. That is, the difference in time between two consecutive times as illustrated in <FIG> may be the period for generating the stop trajectories (and updating the destination trajectory <NUM> as well to reflect newly sensed objects or changes in the objects that are in motion).

The initial segment may be generated in any fashion. For example, the initial segment may be selected from the destination trajectory <NUM> and the remaining segment may deviate from the destination trajectory <NUM> to the stop point. Alternatively, the plan phase <NUM> may be configured to generate the initial segment of the stop trajectories so that they do not deviate from the destination trajectory <NUM> by more than a threshold amount until after the next stop trajectory should be generated. The threshold amount may be based on a human payload's perception: that is, if there are no failures in the system, the mobile machine should be perceived by the payload as smoothly tracking along the destination trajectory <NUM>.

The stop trajectory 36A-36D generated at a given point in time may bring the mobile machine smoothly to a halt at a location selected by the plan phase <NUM>. The plan phase <NUM> may attempt to select a stop location that is least exposed to other traffic. That is, the stop location may be the location that the mobile machine may reach without colliding with other objects and that is least obstructive to continuing traffic in the roadway. Thus, if there is a stop zone to the right of the roadway, the stop point may be the stop zone. The stop zone is the stop point for trajectories 36A and 36B in <FIG>. If there is no stop zone, the stop point may be in the right lane of travel, such as the trajectory 36C shown in <FIG>. The trajectory may move the mobile machine as far to the right as possible in the lane, as shown in <FIG>, or may stop in the lane directly.

In other cases, it may not be possible to move to the right. For example, at the points tn-<NUM> and tn in <FIG>, the existence of the mobile machine M2 in the right path may make it imprudent to attempt to move to the right path. The stop trajectory 36D may thus stop in the left path as shown in <FIG>.

Accordingly, the automation system (e.g., the one or more first computers) may be configured to generate a stop trajectory based on the lane in which the mobile machine is traveling (of one or more possible lanes, for a given roadway), presence of one or more other mobile machines in the one or more lanes, and presence of a stop zone bordering the one or more lanes. For example, the one or more first computers may be configured to generate the trajectory to a point on a stop zone bordering a roadway on which the mobile machine is traveling based on existence of an unobstructed path between the mobile machine and the stop zone. In another example, the one or more first computers are configured to generate the trajectory to stop in a current lane of travel for the mobile machine based on one or more obstructions between the mobile machine and the stop zone. In yet another example, the one or more first computers are configured to generate the trajectory to stop in a current lane of travel for the mobile machine based on an absence of the stop zone. In this context, the stop zone may be the stop zone on the right side of the mobile machine. In some cases, a stop zone on the left side of the mobile machine may be used if it is present and there is an unobstructed path to the left stop zone (e.g., from a left lane of travel).

<FIG> are block diagrams illustrating stop trajectory generation for various scenarios. In each figure, three lanes of roadway are shown with traffic traveling from left to right in the figure. Thus, the right side of the roadway (based on the direction of travel of the mobile machine) is on the bottom of each figure and the left side is on top. There is a stop zone on the left and right side of the roadway for a portion of the roadway (reference numerals <NUM> and <NUM>, respectively), and no stop zone for another portion of the roadway (reference numerals <NUM> and <NUM>, respectively). <FIG> illustrates stop trajectories that may be generated in light traffic conditions, and <FIG> illustrates stop trajectories that may be generated in heavy traffic.

Generally, the stop trajectories that move the mobile machine to the right side of the roadway may be preferable to other trajectories, in this embodiment. The right side of the roadway is where the slowest moving traffic is expected to travel (and there may be a stop zone as well). For example, moving to the right to allow faster moving traffic to overtake a mobile machine to the left is a traffic law that mobile machines are expected to follow in many locations.

As shown in <FIG>, if there is an unimpeded path to the right stop zone <NUM>, the stop trajectory may be generated to bring the mobile machine to a stop in the right stop zone <NUM>. For example, stop trajectories 58A-58C may be generated from mobile machines in any of the lanes to the right stop zone <NUM>. In the case where there is no stop zone <NUM> on the right side (or there are obstructions in the right stop zone <NUM>), the stop trajectories may move the mobile machine to the rightmost lane such as trajectories 58D-58F. Still further, in an embodiment, the stop trajectories may move the mobile machine to a lane that is more to the right, if the rightmost lane is occupied or there is not an unimpeded path to the rightmost lane. <FIG> is intended to show light traffic conditions (e.g., when there are few or no mobile machines in close proximity to a given mobile machine having the illustrated stop trajectories). Thus, for example, in <FIG>, the trajectory 58A may be generated if the mobile machines M1 and M2 are not present or are farther ahead of or behind the mobile machine M3. The three mobile machines are shown to illustrate the trajectories from any lane to the right stop zone or lane.

In <FIG>, various stop trajectories <NUM>, <NUM>, 58J, and <NUM> are illustrated for heavier traffic conditions. In heaver traffic, mobile machines traveling in other than the rightmost lane may not have an unobstructed path to the right (e.g., mobile machine M5 is blocked by mobile machines M1 and M6, mobile machine M4 is blocked by mobile machines M3 and M5, mobile machine M11 is blocked by mobile machine M8 and M9, and mobile machine M10 is blocked by mobile machines M7 and M11). Mobile machines in the rightmost lane may stop in the stop zone <NUM> or in the rightmost lane if there is no stop zone (or there are obstacles in the stop zone <NUM>).

Mobile machines in the left lane of travel may generate a stop trajectory <NUM> to the left stop zone <NUM>, in an embodiment, or may stop in the leftmost lane (stop trajectory 58J) if there is no left stop zone <NUM> or the left stop zone <NUM> has obstacles. Mobile machines in middle lanes (not leftmost or rightmost) may stop in lane in crowded conditions (trajectories <NUM> and <NUM>). In some embodiments, more than one leftmost lane may attempt to move to the left in heavy traffic conditions if there is an unobstructed path to the left.

This, in order of preference for one embodiment, the stop trajectory may be generated to the right stop zone <NUM>, to the left stop zone <NUM> from the leftmost lane or lanes (assuming there are no lanes of oncoming traffic between the leftmost lanes and the left stop zone) or to stop in lane (rightmost preferable, or in the current lane of travel).

It is noted that generation of stop trajectories related to <FIG>, <FIG> are described in accordance with traffic conventions of the United States of America. However, exception stop trajectories can be determined in accordance with any suitable traffic conventions (e.g., traffic conventions of the United Kingdom, etc.). It is further noted that the trajectories and other features illustrated in <FIG>, <FIG> are not necessarily to scale.

<FIG> is a block diagram illustrating various fields of view that sensors for the exception stop functionality may have. Two sensors S2 and S1 are illustrated on a mobile machine M1. The field of view of the sensor S1 is illustrated via the solid-lined shape <NUM> and the field of view of the sensor S2 is illustrated by a dash-lined shape <NUM>. At the top of <FIG>, the area of overlap of the two sensors (illustrated by dash-lined shape <NUM>, which uses smaller dashes than the shape <NUM>) may represent a high confidence ES object detection. That is, in the area illustrated by shape <NUM>, an object may be detected by both sensors S1 and S2, and thus there is high confidence that the object is correctly sensed. At the bottom of <FIG>, the shapes <NUM> and <NUM> are shown again but another area is illustrated via dash-lined shape <NUM> (again using smaller dashes than the shape <NUM>). This area represented by the shape <NUM> may be a mid-level of confidence for ES object detection, since it includes some area that is covered by the field of view of only one sensor S1 or S2. However, the additional area of which objects are sensed may allow earlier detection of a late reveal object/obstacle that is about to enter the stop trajectory.

Combinations of the high confidence area <NUM> and the mid confidence area <NUM> may be used to signal an exception stop to the act phase <NUM>. Alternatively, one of the areas <NUM> or <NUM> may be used.

Turning next to <FIG>, a block diagram of one embodiment of a computer accessible storage medium <NUM> is shown. Generally speaking, a computer accessible storage medium may include any storage media accessible by a computer during use to provide instructions and/or data to the computer. For example, a computer accessible storage medium may include storage media such as magnetic or optical media, e.g., disk (fixed or removable), tape, CD-ROM, DVD-ROM, CD-R, CD-RW, DVD-R, DVD-RW, or Blu-Ray. Storage media may further include volatile or non-volatile memory media such as RAM (e.g., synchronous dynamic RAM (SDRAM), Rambus DRAM (RDRAM), static RAM (SRAM), etc.), ROM, or Flash memory. The storage media may be physically included within the computer to which the storage media provides instructions/data. Alternatively, the storage media may be connected to the computer. For example, the storage media may be connected to the computer over a network or wireless link, such as network attached storage. The storage media may be connected through a peripheral interface such as the Universal Serial Bus (USB). Generally, the computer accessible storage medium <NUM> may store data in a non-transitory manner, where non-transitory in this context may refer to not transmitting the instructions/data on a signal. For example, a non-transitory computer accessible storage medium may be volatile (and may lose the stored instructions/data in response to a power down) or non-volatile.

The computer accessible storage medium <NUM> in <FIG> may store automation code <NUM>. The automation code <NUM> may include instructions which, when executed by a computer or computers <NUM>, <NUM>, or <NUM>, implement the operation described for the various code above, particularly with regard to <FIG>, <FIG>, and <FIG>. A carrier medium may include computer accessible storage media as well as transmission media such as wired or wireless transmission.

Claim 1:
An automation system (<NUM>) comprising:
one or more first computers (<NUM>) configured to:
periodically generate a stop trajectory for a mobile machine to bring the mobile machine to a stop based on sensor data up to a point in time at which the stop trajectory is generated, wherein the stop trajectory (36A) includes an initial segment that parallels a destination trajectory (<NUM>) until a next point in time at which another instance of the stop trajectory is generated, wherein the sensor data is received from a plurality of sensors (<NUM>) on the mobile machine that sense surroundings of the mobile machine; and
one or more second computers (<NUM>) configured to:
periodically receive the stop trajectory from the one or more first computers; control a plurality of actuators (<NUM>) in the mobile machine to cause the mobile machine to follow a current instance of the stop trajectory replace the current instance of the stop trajectory with a subsequent instance of the stop trajectory received from the one or more first computers; and
control the plurality of actuators to follow the subsequent instance of the stop trajectory received from the one or more first computers.