Patent Description:
Robotics and autonomous mobile platforms such as Autonomous Mobile Robots (AMRs) and/or Autonomous Guided Vehicles (AGVs) are key components in factories, warehouses, hospitals, and other industrial and/or commercial environments. Autonomous mobile platforms implement perception and manipulation jointly to accomplish a given task by navigating an environment. AMRs may communicate and coordinating with one other and/or with a central controller. Robotics and autonomous mobile platforms may create hazardous environments for humans and require extensive safety monitoring or isolation of humans from active autonomous environments. Publication <CIT> relates to a map information update system based on information acquired using multiple robots. Publication <CIT> relates to providing personalized patient care using robots. Publication <NPL> presents a human-centered method for indoor service robots to provide people with physical assistance and active guidance while traveling through congested and narrow spaces.

The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate the present disclosure and, together with the description, and further serve to explain the principles of the disclosure and to enable a person skilled in the pertinent art to make and use the techniques discussed herein.

The present disclosure will be described with reference to the accompanying drawings. The drawing in which an element first appears is typically indicated by the leftmost digit(s) in the corresponding reference number.

The following detailed description refers to the accompanying drawings that show, by way of illustration, exemplary details in which the disclosure may be practiced. However, it will be apparent to those skilled in the art that the various designs, including structures, systems, and methods, may be practiced without these specific details. The description and representation herein are the common means used by those experienced or skilled in the art to most effectively convey the substance of their work to others skilled in the art. In other instances, well-known methods, procedures, components, and circuitry have not been described in detail to avoid unnecessarily obscuring the disclosure.

Robotics and autonomous environments require safety monitoring and/or isolation of humans from active autonomous environments to reduce or prevent potential harm to humans entering the environment and surrounding area of the robot within the environment. To reduce or prevent robot-human accidents, conventional systems have addressed shared environments and separate environment arrangements. In shared environments, the robots must be aware of humans in their surrounding and if there is a risk of collision, reduce speed or even stop operations. In addition, workers may be trained to respect robots working areas. As a consequence, the costs of the robots increase as there is the need for additional sensors for perception of humans, usually with special safety requirements that increase costs, and for additional safety functionality. Also untrained workers are generally not allowed to enter the working areas, or must be accompanied by a trained worker. In separate working area arrangements, the robot environment is separated from human environments to prevent robot-human interactions. While these arrangements may reduce the number of sensors and monitoring of the autonomous environment, all are generally required to be shut down when a human enters the autonomous environment.

Large warehouses or other sites are increasing working in full autonomy configurations without the presence of humans. In such cases, the number of sensors used to monitor the environment for humans to ensure human safety in the environment are reduced or the sensors are eliminating completely to reduce costs. However, these configurations require that the entire site or at least portions of the site need to be shut down when humans are present (e.g. for maintenance work), which reduces the efficiency of robots and the overall production of the facility.

The present disclosure provides an advantageous solution to address human interactions with autonomous agents (e.g. robots) in partial and full autonomous environments using one or more Safety Guide Robots (SGRs). <FIG> illustrates an exemplary environment <NUM> utilizing autonomous mobile robots (AMRs) <NUM> and the one or more Safety Guide Robots (SGRs) <NUM>, in accordance with the disclosure. The SGR <NUM> is configured to accompany and guide one or more humans <NUM> safely through the environment <NUM> to a given destination. The SGR <NUM> may be equipped with one or more sensors to monitor the human(s) <NUM>, localize itself in the environment <NUM>, and communicate with the human(s) <NUM> and/or one or more other devices in the environment, such as one or more other SGRs <NUM> and/or AMR(s) <NUM>. For example, upon a human entering the environment <NUM> (e.g. the facility with active AMRs), including a working area of an AMR <NUM>, the SGR <NUM> may guide and/or restrict the movement of the human within the environment <NUM> and/or to enforce a speed reduction of the AMR(s) <NUM> an/or force one or more AMRs <NUM> to stop operating. The SGR <NUM> advantageously ensures safety of the human while reducing the impact to the continued operation of the AMRs <NUM> in the environment, including being able to finely tailor the control of the AMRs <NUM> that may encounter the human.

With continued reference to <FIG>, the environment <NUM> supports any suitable number of AMRs <NUM>, with three AMRs <NUM>-<NUM> being shown for ease of explanation, and any number of SGRs <NUM>, with one SGR <NUM> being shown for easy of explanation. The environment <NUM> may include one or more sensors <NUM> configured to monitor the locations and activities of the SGRs <NUM>, AMRs <NUM>, and/or humans <NUM> within the environment <NUM>. The sensors <NUM> may include, for example, radar, LIDAR, optical sensors, infrared sensors, cameras, or other sensors as would be understood by one or ordinary skill in the art. The sensors may communicate information (sensor data) with the computing device <NUM> (via access point(s) <NUM>). Although not shown in <FIG> for purposes of brevity, the sensor(s) <NUM> may additionally communicate with one another and/or with one or more of the AMRs <NUM> and/or SGRs <NUM>.

The environment <NUM> may be any suitable type of environment that uses the AMRs <NUM>, such as a factory, warehouse, hospital, office building, etc. The AMRs <NUM> may have any suitable type of design and function to communicate with other components of a network infrastructure as further disused below. The AMRs <NUM> may operate autonomously or semi-autonomously and be configured as mobile robots that move within the environment <NUM> to complete specific tasks. One or more of the AMRs <NUM> may alternatively be configured as a stationary robots having moveable components (e.g. moveable arms) to complete localized tasks.

The AMRs <NUM> may include any suitable number and/or type of sensors to enable sensing of their surroundings and the identification of feedback regarding the environment <NUM>. The AMRs <NUM> may further be configured with any suitable number and/or type of wireless radio components to facilitate the transmission and/or reception of data. For example, the AMRs <NUM> may transmit data indicative of current tasks being executed, location, orientation, velocity, trajectory, heading, etc. within the environment <NUM> (via transceiver <NUM> as shown in <FIG>). As another example, the AMRs <NUM> may receive commands and/or planned path information from the computing device <NUM>, which each AMR <NUM> may execute to navigate to a specific location within the environment <NUM>. Although not shown in <FIG> for purposes of brevity, the AMRs <NUM> may additionally communicate with one another to determine information (e.g. current tasks being executed, location, orientation, velocity, trajectory, heading, etc.) with respect to the other AMRs <NUM>, as well as other information such as sensor data generated by other AMRs <NUM>.

The SGR(s) <NUM> may include any suitable number and/or type of sensors to enable sensing of their surroundings and the identification of feedback regarding the environment <NUM>. The SGR(s) <NUM> may further be configured with any suitable number and/or type of wireless radio components to facilitate the transmission and/or reception of data. For example, the SGR(s) <NUM> may transmit data indicative of current tasks being executed, location, orientation, velocity, trajectory, heading, etc. within the environment <NUM> (via transceiver <NUM> as shown in <FIG>). As another example, the SGR(s) <NUM> may receive commands and/or planned path information from the computing device <NUM>, which each SGR(s) <NUM> may execute to navigate to a specific location within the environment <NUM>. Although not shown in <FIG> for purposes of brevity, the SGR(s) <NUM> may additionally communicate with one or more other SGRs <NUM> another to determine information (e.g. current tasks being executed, location, orientation, velocity, trajectory, heading, etc.) with respect to the other SGR(s) <NUM>, as well as other information such as sensor data generated by other SGR(s) <NUM>. The SGR(s) <NUM> may additionally include input/output (I/O) interfaces for communicating with the human(s) <NUM> and/or one or more transceivers for communicating with other devices in the environment, such as one or more other SGRs <NUM> and/or AMR(s) <NUM>. The I/O interfaces may include microphone(s), speaker(s), display(s), image projector(s), light(s), laser(s), or other interfaces as would be understood by one of ordinary skill in the arts.

Although the disclosure includes examples of the environment <NUM> being a factory or warehouse that supports AMRs <NUM> operating within such an environment, this is by way of example and not limitation. The teachings of the disclosure may implemented in accordance with any suitable type of environment and/or type of mobile agent. For instance, the environment <NUM> may be outdoors and be identified with a region such as a roadway that is utilized by autonomous vehicles. Thus, the teachings of the disclosure are applicable to AMRs as well as other types of autonomous agents that may operate in any suitable type of environment based upon any suitable application or desired function.

The AMRs <NUM> and SGRs <NUM> operate within the environment <NUM> by communicating with the various components of the supporting network infrastructure. The network infrastructure may include any suitable number and/or type of components to support communications with the AMRs <NUM> and SGRs <NUM>. For example, the network infrastructure may include any suitable combination of wired and/or wireless networking components that operate in accordance with any suitable number and/or type of communication protocols. For instance, the network infrastructure may include interconnections using wired links such as Ethernet or optical links, as well as wireless links such as Wi-Fi (e.g. <NUM> protocols) and cellular links (e.g. 3GPP standard protocols, LTE, <NUM>, etc.). The network infrastructure may be, for example, an access network, an edge network, a mobile edge computing (MEC) network, etc. In the example shown in <FIG>, the network infrastructure includes one or more cloud servers <NUM> that enable a connection to the Internet, which may be implemented as any suitable number and/or type of cloud computing devices. The network infrastructure may additionally include a computing device <NUM>, which may be implemented as any suitable number and/or type of computing device such as a server. The computing device <NUM> may be implemented as an Edge server and/or Edge computing device.

According to the disclosure, the computing device <NUM> may communicate with the one or more cloud servers <NUM> via one or more links <NUM>, which may represent an aggregation of any suitable number and/or type of wired and/or wireless links as well as other network infrastructure components that are not shown in <FIG> for purposes of brevity. For instance, the link <NUM> may represent additional cellular network towers (e.g. one or more base stations, eNode Bs, relays, macrocells, femtocells, etc.). According to the disclosure, the network infrastructure may further include one or more access points (APs) <NUM>. The APs <NUM> which may be implemented in accordance with any suitable number and/or type of AP configured to facilitate communications in accordance with any suitable type of communication protocols. The APs <NUM> may be configured to support communications in accordance with any suitable number and/or type of communication protocols, such as an Institute of Electrical and Electronics Engineers (IEEE) <NUM> Working Group Standards. Alternatively, the APs <NUM> may operate in accordance with other types of communication standards other than the <NUM> Working Group, such as cellular based standards (e.g. "private cellular networks) or other local wireless network systems, for instance. Additionally or alternatively, the AMRs <NUM> and/or SGR(s) <NUM> may communicate directly with the computing device <NUM> or other suitable components of the network infrastructure without the need to use the APs <NUM>. Additionally or alternatively, one or more of AMRs <NUM> may communicate directly with one or more other AMRs <NUM> and/or with the SGRs <NUM>. Similarly, one or more of SGRs <NUM> may communicate directly with one or more other SGRs <NUM> and/or with one or more of the AMRs <NUM>.

In the environment <NUM> as shown in <FIG>, the computing device <NUM> is configured to communicate with each of the AMRs <NUM> and/or SGRs <NUM> to receive data from the AMRs <NUM> and/or SGRs <NUM>, respectively, and to transmit data to the AMRs <NUM> and/or SGRs <NUM>, respectively. This functionality may be additionally or alternatively be performed by other network infrastructure components that are capable of communicating directly or indirectly with the AMRs <NUM> and/or SGRs <NUM>, such as the one or more cloud servers <NUM>, for instance. However, the local nature of the computing device <NUM> may provide additional advantages in that the communication between the computing device <NUM> and the AMRs <NUM> and/or SGRs <NUM> may occur with reduced network latency. Thus, according to the disclosure, the computing device <NUM> is used as the primary example when describing this functionality, although it is understood that this is by way of example and not limitation.

The computing device <NUM> may thus receive sensor data from each for the AMRs <NUM> and/or SGRs <NUM> via the APs <NUM> and use the respective sensor data, together with other information about the environment <NUM> that is already known (e.g. data regarding the size and location of static objects in the environment <NUM>), to generate a shared environment model that represents the environment <NUM>. This shared environment model may be represented as a navigation grid having cells of any suitable size and/or shape, with each cell having specific properties with respect to the type of object contained (or not contained) in the cell, whether an object in the cell is static or moving, etc., which enables the environment model to accurately depict the nature of the environment <NUM>. As an example, grid cells may be squares of predetermined sizes (e.g. <NUM>) based upon a desired granularity for a particular environment and accompanying application. The environment model may thus be dynamically updated by the SGRs <NUM> and/or AMRs <NUM> directly and/or via the computing device <NUM> on a cell-by-cell basis as new sensor data is received from the SGRs <NUM> and/or AMRs <NUM>. The updates to the shared environment model thus reflect any recent changes in the environment <NUM> such as the position and orientation of each of the SGRs <NUM> and/or AMRs <NUM> and other obstacles that may change in a dynamic manner within the environment <NUM> (e.g. people, forklifts, machinery, etc.). The shared environment model may additionally or alternatively be updated based upon data received from other sensors <NUM> or devices within the environment <NUM>, such as stationary cameras for example, which may enable a more accurate depiction of the positions of the SGRs <NUM> and/or AMRs <NUM> without relying of SGR and/or AMR communications.

Each SGR <NUM> (and/or AMR <NUM> when configured as mobile robots) executes a path planning algorithm and uses the shared environment model at a particular time (e.g. the most recently constructed) to calculate navigational paths for each SGRs <NUM> and/or AMR <NUM>. These navigational paths include sets of intermediate points ("waypoints") or nodes that define an SGR and/or AMR trajectory within the environment <NUM> between a starting point (e.g. its current location in the environment <NUM>) to a destination (goal point) within the environment <NUM>. That is, the waypoints indicate to the SGRs <NUM> and/or AMRs <NUM> how to execute a respective planned navigational path to proceed to each of the intermediate points at a specific time until a destination is reached.

The computing device (controller) <NUM> may alternatively or additionally (potentially in collaboration with one or more of the SGRs <NUM> and/or AMRs <NUM>) calculate navigational paths for one or more of the SGRs <NUM> and/or AMRs <NUM>. Alternatively or additionally, the cloud server(s) <NUM> may be configured to calculate navigational paths for one or more of the SGRs <NUM> and/or AMRs <NUM>, which may then be transmitted to the SGRs <NUM> and/or AMRs <NUM>, respectively. It should be appreciated that any combination of the SGRs <NUM>, AMRs <NUM>, computing device <NUM>, and cloud server(s) <NUM> may calculate the navigational paths. The SGRs <NUM>, AMRs <NUM>, computing device <NUM>, and/or cloud server(s) <NUM> may include processing circuitry that is configured to perform the respective functions of the SGRs <NUM>, AMRs <NUM>, computing device <NUM>, and/or cloud server(s) <NUM>, respectively. One or more of these devices may further be implemented with machine-learning capabilities.

Information dynamically discovered by the SGRs <NUM> and/or AMRs <NUM> may be, for instance, a result of each SGRs <NUM> and/or AMR <NUM> locally processing its respective sensor data. The updated shared environment model may be maintained by the central controller (computing device <NUM>) and shared with each of the SGRs <NUM> and/or AMRs <NUM> as well being used for planning tasks. Thus, at any given point in time, the SGRs <NUM> and/or AMRs <NUM> may be attempting to determine which cells to add to a particular route (e.g. a planned path) or move to so that the assigned tasks of the SGRs <NUM> and/or the assigned tasks of the AMRs <NUM> may be accomplished in the most efficient manner. In other words, because of the dynamic nature of the environment <NUM>, each SGRs <NUM> and/or AMR <NUM> may calculate its own respective navigation path in a continuous and iterative manner using iterative updates that are provided to the shared environment model. Thus, the shared environment model may be stored in the computing device <NUM> and/or locally in a memory associated with or otherwise accessed by each one of the SGRs <NUM> and/or AMRs <NUM>. Additionally or alternatively, the shared environment model may be stored in any other suitable components of the network infrastructure or devices connected thereto. In any event, the SGRs <NUM> and/or AMRs <NUM> may iteratively receive or otherwise access the shared environment model, including the most recent updates, to perform navigation path planning functions as discussed herein. The shared environment model may thus be updated as new sensor data is received by the central controller (computing device <NUM>) and processed, and/or processed locally by the SGRs <NUM> and/or AMRs <NUM>, and be performed in a periodic manner or in accordance with any suitable schedule.

With reference to <FIG>, the SGRs <NUM> may implement a suite of onboard sensors <NUM> to generate sensor data indicative of the location, position, velocity, heading orientation, etc. of the SGRs <NUM> within the environment <NUM> and/or the location, position, velocity, heading orientation, etc. of one or more humans <NUM> guided by the respective SGR <NUM>. These sensors <NUM> may be implemented as any suitable number and/or type that are generally known and/or used for autonomous navigation and environmental monitoring. Examples of such sensors may include radar, LIDAR, optical sensors, cameras, compasses, gyroscopes, positioning systems for localization, accelerometers, etc. Thus, the sensor data may indicate the presence of and/or range to various objects near each SGR <NUM>. Each SGRs <NUM> may additionally process this sensor data to identify obstacles, track the movement of the human(s) <NUM> to be guided, and/or other relevant information within the environment <NUM> that will impact the shared environment model. The SGRs <NUM> may then use the shared environment model to iteratively calculate respective navigation paths, as further discussed herein. According to the disclosure, the SGR <NUM> may be provided with an original path to the desired location in which the human <NUM> intends to or is expected to traverse. Additionally or alternatively, the SGR <NUM> has knowledge of the intended task to be performed by the human <NUM>. The SGR <NUM> may receive this path and/or task information from the controller <NUM> and/or such information may be preloaded in the memory <NUM>. When the SGR <NUM> is aware of the intended task to be performed but is unaware of the intended path (or is only partially aware of the intended path), the processing circuitry <NUM> is configured to plan an appropriate path to the location for the intended task.

The SGR <NUM> may be configured to operate in a guidance mode where the SGR <NUM> guides a complying human <NUM> to the planned destination. If the human <NUM> diverts from the planned path, the SGR <NUM> may operate in a tracking mode where the SGR <NUM> tracks and follows the human <NUM> if the human <NUM> diverts form the planned route. While tracking, the SGR <NUM> may provide continued notifications of safe regions on the diverted path, warning of hazardous regions along the diverted path, and/or to instruct AMRs <NUM> to stop operations or reduce operational speed/movement that the human <NUM> encounters or approaches along the diverted path.

The SGRs <NUM><NUM> may also any suitable number and/or type of hardware and software configuration to facilitate autonomous navigation functions within the environment <NUM>, including known configurations. For example, each SGR <NUM> may implement a controller that may comprise one or more processors or processing circuitry <NUM>, which may execute software that is installed on a local memory <NUM> to perform various autonomous navigation-related functions.

The SGR <NUM> may use onboard sensors <NUM> to perform pose estimation and/or to identify e.g. a position, orientation, velocity, direction, and/or location of the SGR <NUM> within the environment <NUM> as the SGR <NUM> moves along a particular planned path. The processing circuitry <NUM> can execute a path planning algorithm stored in memory <NUM> to execute path planning for navigation-related functions (e.g. SLAM, octomap generation, multi-robot path planning, etc.) of the SGR <NUM>.

<FIG> illustrates a block diagram of an exemplary autonomous agent, in accordance with the disclosure. The autonomous agent <NUM> as shown and described with respect to <FIG> may be identified with the SGR <NUM> as shown in <FIG> and discussed herein, for instance. The autonomous agent <NUM> may include processing circuitry <NUM>, one or more sensors <NUM>, a transceiver <NUM>, input/output (I/O) interface <NUM>, drive <NUM>, and a memory <NUM>. The components shown in <FIG> are provided for ease of explanation, and the autonomous agent <NUM> may implement additional, less, or alternative components as those shown in <FIG>. The operation of the autonomous agent <NUM> is described below with additional references to <FIG>. <FIG> illustrates processing operations of the autonomous agent <NUM> (e.g. by the processing circuitry <NUM>). <FIG> illustrates an example of the environment model according to the disclosure. <FIG> illustrate safe (green) and unsafe (red) regions in the environment model.

The sensors <NUM> may be implemented as any suitable number and/or type of sensors that may be used for autonomous navigation and environmental monitoring, including tracking of the human <NUM> that is being guided by the autonomous agent <NUM>. Examples of such sensors may include radar, LIDAR, optical sensors, cameras, compasses, gyroscopes, positioning systems for localization, accelerometers, etc..

The processing circuitry <NUM> may be configured as any suitable number and/or type of computer processors, which may function to control the autonomous agent <NUM> and/or other components of the autonomous agent <NUM>. The processing circuitry <NUM> may be identified with one or more processors (or suitable portions thereof) implemented by the autonomous agent <NUM>.

The processing circuitry <NUM> may be configured to carry out instructions to perform arithmetical, logical, and/or input/output (I/O) operations, and/or to control the operation of one or more components of autonomous agent <NUM> to perform various functions associated with the disclosure as described herein. For example, the processing circuitry <NUM> may include one or more microprocessor cores, memory registers, buffers, clocks, etc., and may generate electronic control signals associated with the components of the autonomous agent <NUM> to control and/or modify the operation of these components. For example, the processing circuitry <NUM> may control functions associated with the sensors <NUM>, the transceiver <NUM>, I/O interface <NUM>, drive <NUM>, and/or the memory <NUM>. The processing circuitry <NUM> may additionally perform various operations to control the movement, speed, and/or tasks executed by the autonomous agent <NUM>, which may be based upon global and/or local path planning algorithms, as discussed herein.

The processing circuitry <NUM> may process an environment model generated by the controller <NUM> and received by the SGR <NUM> to perform autonomous navigation. This environment model may be represented as a navigation grid having cells of any suitable size and/or shape, with each cell having specific properties with respect to the type of object contained (or not contained) in the cell, whether an object in the cell is static or moving, etc., which enables the environment model to accurately depict the nature of the environment <NUM>.

The processing circuitry <NUM> may use sensor data from one or more of its sensors <NUM>, sensor data and/or other data from one or more of the AMRs <NUM>, data from the controller <NUM>, and/or other data as would be understood by one of ordinary skill in the art (e.g. commands or other data from an operator of the system) to dynamically update the environment model to iteratively calculate the navigation path of the SGR <NUM>. For example, the processing circuitry <NUM> may process the sensor data to identify obstacles, track the movement of the human(s) <NUM> to be guided, and/or other relevant information within the environment <NUM> that will impact the shared environment model. The SGRs <NUM> may then use the shared environment model to iteratively calculate respective navigation paths. According to the disclosure, the processing circuitry <NUM> may execute a path planning algorithm (e.g. stored in memory <NUM>) to perform autonomous navigation using the environment model and sensor data from one or more of its sensors <NUM>, sensor data and/or other data from one or more of the AMRs <NUM>, data from the controller <NUM>, and/or other data as would be understood by one of ordinary skill in the art. The algorithm may be performed iteratively via the processing circuitry <NUM> to dynamically update the environment model. The updated environment model may be provided to one or more other SGRs <NUM>, one or more of the AMRs <NUM>, and/or to the controller <NUM>.

The controller <NUM> may alternatively, or in conjunction with the SGR <NUM>, be configured to dynamically update the environment model based on sensor and/or other data from the SGR <NUM>, sensor and/or other data from one or more AMRs <NUM>, and/or other data as would be understood by one of ordinary skill in the art, and then provide the updated environment model to the SGRs <NUM> and/or AMRs <NUM>.

The transceiver <NUM> may be implemented as any suitable number and/or type of components configured to transmit and/or receive data packets and/or wireless signals in accordance with any suitable number and/or type of communication protocols. The transceiver <NUM> may include any suitable type of components to facilitate this functionality, including components associated with known transceiver, transmitter, and/or receiver operation, configurations, and implementations. Although depicted in <FIG> as a transceiver, the transceiver <NUM> may include any suitable number of transmitters, receivers, or combinations of these that may be integrated into a single transceiver or as multiple transceivers or transceiver modules. For example, the transceiver <NUM> may include components typically identified with an RF front end and include, for example, antennas, ports, power amplifiers (PAs), RF filters, mixers, local oscillators (LOs), low noise amplifiers (LNAs), upconverters, downconverters, channel tuners, etc. The transceiver <NUM> may also include analog-to-digital converters (ADCs), digital to analog converters, intermediate frequency (IF) amplifiers and/or filters, modulators, demodulators, baseband processors, and/or other communication circuitry as would be understood by one of ordinary skill in the art.

I/O interface <NUM> may be implemented as any suitable number and/or type of components configured to communicate with the human(s) <NUM>. The I/O interface <NUM> may include microphone(s), speaker(s), display(s), image projector(s), light(s), laser(s), and/or other interfaces as would be understood by one of ordinary skill in the arts. The I/O interface <NUM> may be configured to provide instructions to the human <NUM> to guide the human <NUM> along a designated path (<FIG> and <FIG>), such as by projecting path guides (e.g. colored path markers, arrows, or the like) on the ground to convey the appropriate areas in which the human <NUM> is to traverse. As shown in <FIG>, the SGR <NUM> has identified a safe path (green) with an arrow to guide the human <NUM> on the suggested path determined by the SGR <NUM> and/or controller <NUM>. The green path indicator and/or the arrow may be projected onto the ground to notify the human <NUM> of the path and/or the SGR <NUM> may communicate with the human <NUM> using one or more other I/O interfaces <NUM> (e.g. audio commands via a speaker).

The drive <NUM> may be implemented as any suitable number and/or type of components configured to drive the autonomous agent <NUM>, such as a motor or other driving mechanism. The processing circuitry <NUM> may be configured to control the drive <NUM> to move the autonomous agent <NUM> in a desired direction and at a desired velocity.

The memory <NUM> stores data and/or instructions such that, when the instructions are executed by the processing circuitry <NUM>, cause the autonomous agent <NUM> to perform various functions as described herein. The memory <NUM> may be implemented as any well-known volatile and/or non-volatile memory. The memory <NUM> may be implemented as a non-transitory computer readable medium storing one or more executable instructions such as, for example, logic, algorithms, code, etc. The instructions, logic, code, etc., stored in the memory <NUM> may enable the features disclosed herein to be functionally realized. For example, the memory <NUM> may include the path planning algorithm. For hardware implementations, the memory <NUM> may include instructions and/or code to facilitate control and/or monitor the operation of such hardware components. The disclosure may include the processing circuitry <NUM> executing the instructions stored in the memory <NUM> in conjunction with one or more hardware components to perform the various functions described herein.

With reference to <FIG>, the AMRs <NUM> may implement a suite of onboard sensors <NUM> to generate sensor data indicative of the location, position, velocity, heading orientation, etc. of the AMR <NUM> within the environment <NUM>. These sensors <NUM> may be implemented as any suitable number and/or type that are generally known and/or used for autonomous navigation and environmental monitoring. Examples of such sensors may include radar, LIDAR, optical sensors, cameras, compasses, gyroscopes, positioning systems for localization, accelerometers, etc. Thus, the sensor data may indicate the presence of and/or range to various objects near each AMR <NUM>. Each AMR <NUM> may additionally process this sensor data to identify obstacles or other relevant information within the environment <NUM> that will impact the shared environment model. The AMRs <NUM> may then use the shared environment model to iteratively calculate respective navigation paths, as further discussed herein. The AMRs <NUM> may also any suitable number and/or type of hardware and software configuration to facilitate autonomous navigation functions within the environment <NUM>, including known configurations. For example, each AMR <NUM> may implement a controller that may comprise one or more processors or processing circuitry <NUM>, which may execute software that is installed on a local memory <NUM> to perform various autonomous navigation-related functions.

The AMR <NUM> may use onboard sensors <NUM> to perform pose estimation and/or to identify e.g. a position, orientation, velocity, direction, and/or location of the AMR <NUM> within the environment <NUM> as the AMR <NUM> moves along a particular planned path. The processing circuitry <NUM> can execute a path planning algorithm stored in memory <NUM> to execute path planning and sampling functionalities for navigation-related functions (e.g. SLAM, octomap generation, multi-robot path planning, etc.) of the AMR <NUM>.

<FIG> illustrates a block diagram of an exemplary autonomous agent <NUM>, in accordance with the disclosure. The autonomous agent <NUM> as shown and described with respect to <FIG> may be identified with one or more of the AMRs <NUM> as shown in <FIG> and discussed herein, and is similar to the autonomous agent <NUM> shown in <FIG>. The autonomous agent <NUM> may include processing circuitry <NUM>, one or more sensors <NUM>, a transceiver <NUM>, and a memory <NUM>. The autonomous agent <NUM> may additionally include input/output (I/O) interface <NUM> and/or drive <NUM> (e.g. when the agent <NUM> is a mobile agent). The components shown in <FIG> are provided for ease of explanation, and the autonomous agent <NUM> may implement additional, less, or alternative components as those shown in <FIG>.

The processing circuitry <NUM> may be configured as any suitable number and/or type of computer processors, which may function to control the autonomous agent <NUM> and/or other components of the autonomous agent <NUM>. The processing circuitry <NUM> may be identified with one or more processors (or suitable portions thereof) implemented by the autonomous agent <NUM>. The processing circuitry <NUM> may be configured to carry out instructions to perform arithmetical, logical, and/or input/output (I/O) operations, and/or to control the operation of one or more components of autonomous agent <NUM> to perform various functions associated with the disclosure as described herein. For example, the processing circuitry <NUM> may include one or more microprocessor cores, memory registers, buffers, clocks, etc., and may generate electronic control signals associated with the components of the autonomous agent <NUM> to control and/or modify the operation of these components. For example, the processing circuitry <NUM> may control functions associated with the sensors <NUM>, the transceiver <NUM>, interface <NUM>, drive <NUM>, and/or the memory <NUM>. The processing circuitry <NUM> may additionally perform various operations to control the movement, speed, and/or tasks executed by the autonomous agent <NUM>, which may be based upon global and/or local path planning algorithms, as discussed herein.

The sensors <NUM> may be implemented as any suitable number and/or type of sensors that may be used for autonomous navigation and environmental monitoring. Examples of such sensors may include radar, LIDAR, optical sensors, cameras, compasses, gyroscopes, positioning systems for localization, accelerometers, etc..

I/O interface <NUM> may be implemented as any suitable number and/or type of components configured to communicate with the human(s) <NUM>. The I/O interface <NUM> may include microphone(s), speaker(s), display(s), image projector(s), light(s), laser(s), and/or other interfaces as would be understood by one of ordinary skill in the arts.

The memory <NUM> stores data and/or instructions such that, when the instructions are executed by the processing circuitry <NUM>, cause the autonomous agent <NUM> to perform various functions as described herein. The memory <NUM> may be implemented as any well-known volatile and/or non-volatile memory. The memory <NUM> may be implemented as a non-transitory computer readable medium storing one or more executable instructions such as, for example, logic, algorithms, code, etc. The instructions, logic, code, etc., stored in the memory <NUM> may enable the features disclosed herein to be functionally realized. For hardware implementations, the modules shown in <FIG> associated with the memory <NUM> may include instructions and/or code to facilitate control and/or monitor the operation of such hardware components.

<FIG> illustrates a block diagram of an exemplary computing device <NUM>, in accordance with the disclosure. The computing device <NUM> as shown and described with respect to <FIG> may be identified with the computing device <NUM> and/or server <NUM> as shown in <FIG> and discussed herein, for instance. The computing device <NUM> may include processing circuitry <NUM>, one or more sensors <NUM>, a transceiver <NUM>, and a memory <NUM>. In some examples, the computer device <NUM> is configured to interact with one or more external sensors (e.g. sensor <NUM>) as an alternative or in addition to including internal sensors <NUM>. The components shown in <FIG> are provided for ease of explanation, and the computing device <NUM> may implement additional, less, or alternative components as those shown in <FIG>.

The processing circuitry <NUM> may be configured as any suitable number and/or type of computer processors, which may function to control the computing device <NUM> and/or other components of the computing device <NUM>. The processing circuitry <NUM> may be identified with one or more processors (or suitable portions thereof) implemented by the computing device <NUM>.

The processing circuitry <NUM> may be configured to carry out instructions to perform arithmetical, logical, and/or input/output (I/O) operations, and/or to control the operation of one or more components of computing device <NUM> to perform various functions as described herein. For example, the processing circuitry <NUM> may include one or more microprocessor cores, memory registers, buffers, clocks, etc., and may generate electronic control signals associated with the components of the computing device <NUM> to control and/or modify the operation of these components. For example, the processing circuitry <NUM> may control functions associated with the sensors <NUM>, the transceiver <NUM>, and/or the memory <NUM>.

The sensors <NUM> may be implemented as any suitable number and/or type of sensors that may be used for autonomous navigation and environmental monitoring. Examples of such sensors may include radar, LIDAR, optical sensors, cameras, compasses, gyroscopes, positioning systems for localization, accelerometers, etc. In some examples, the computing device <NUM> is additionally or alternatively configured to communicate with one or more external sensors similar to sensors <NUM> (e.g. sensor <NUM> in <FIG>).

The memory <NUM> stores data and/or instructions such that, when the instructions are executed by the processing circuitry <NUM>, cause the computing device <NUM> to perform various functions as described herein. The memory <NUM> may be implemented as any well-known volatile and/or non-volatile memory. The memory <NUM> may be implemented as a non-transitory computer readable medium storing one or more executable instructions such as, for example, logic, algorithms, code, etc. The instructions, logic, code, etc., stored in the memory <NUM> are represented by the various modules as shown in <FIG>, which may enable the features described herein to be functionally realized. For example, the memory <NUM> may include the ITS module <NUM> representing the ITS algorithm. The ITS algorithm may further include a sampler configured to perform the sampling operations and a path planner configured to perform the path planning operations. For hardware implementations, the modules shown in <FIG> associated with the memory <NUM> may include instructions and/or code to facilitate control and/or monitor the operation of such hardware components. In other words, the modules shown in <FIG> are provided for ease of explanation regarding the functional association between hardware and software components. Thus, the disclosure includes the processing circuitry <NUM> executing the instructions stored in these respective modules in conjunction with one or more hardware components to perform the various functions described herein.

<FIG> illustrates a SGR system and operational flow according to the present disclosure. The SGR system may be configured to create a comprehensive environment model around the current position of the SGR <NUM> (environment monitoring <NUM>). As described herein, any combination of the controller <NUM>, server <NUM>, SGRs <NUM> and AMRs <NUM> may be configured to generate and adapt the environmental model. The model may be adapted based on sensor and/or other data as described herein. The environment monitoring <NUM> by the SGR <NUM> may include monitoring position and/or motion of the human <NUM> and/or the position and/or motion of the AMRs <NUM> and/or other static and/or dynamic devices within the environment <NUM>.

The environment model may be created and/or adapted based on the sensor data generated by one or more sensors <NUM> of the SGR <NUM>, sensor data and/or other data (e.g. internal state information) from one or more of the AMRs <NUM> (robot communication <NUM>), data from the controller <NUM>, and/or other data as would be understood by one of ordinary skill in the art.

Based on this environment model, the SGR <NUM>, controller <NUM>, and/or server <NUM> assess and predict human behavior (Human Monitoring <NUM>). Using the results from both the environment monitoring <NUM> and the human monitoring <NUM>, the SGR <NUM>, controller <NUM>, and/or server <NUM> perform a safety assessment <NUM>. The safety assessment <NUM> determines a safe path as described herein for the SGR <NUM> to travel and guide the human <NUM> as well as determines one or more AMRs <NUM> that should be controlled (e.g. instructed to stop or modify operations) along the path.

The SGR system also provides communication with the human <NUM> (Human Communication <NUM>) to provide the human <NUM> with a guided path and/or other instructions.

As will be appreciated the computation tasks of the SGR system may be performed by any combination of the SGR <NUM>, controller <NUM>, server <NUM>, and one or more AMRs <NUM>. Sensor and/or other data from the SGRs <NUM> and/or AMRs <NUM> may be provided to the controller <NUM> and/or server <NUM>, and vice versa. In configurations where computations are performed by the controller <NUM> and/or server <NUM>, sensor and/or other data from the SGR <NUM> and/or AMRs <NUM> is provided to the controller <NUM> and/or server <NUM>, and the controller <NUM> and/or server <NUM> perform path planning and generation and/or adaptation of the environment model, which is then provided to the SGRs <NUM> and/or AMRs <NUM>. With the environment model, the SGR <NUM> communicates with the human <NUM> (e.g. guide path, audio and/or video assistance) to guide the human <NUM> along a safe path identified in the path planning processing.

SGR system may be configured to create a comprehensive environment model around the current position of the SGR <NUM> (environment monitoring <NUM>). <FIG> illustrates an environment model according to the disclosure. The environment model may be generated and/or adapted by the controller <NUM>, server <NUM>, SGRs <NUM> and AMRs <NUM> based on sensor and/or other data as described herein. For example, the environment model may be based on sensor data from one or more sensors <NUM> (e.g. radar, LIDAR, camera) of the SGR <NUM> and/or sensor data and/or other data (e.g. internal state information) from one or more of the AMRs <NUM>. The SGR system (e.g. SGR <NUM>) is configured to detect and/or classify objects in the environment <NUM> based on the sensor data, and/or to perform self-localization of the SGR <NUM> based on the sensor data (e.g. GPS, wireless communication geo-location, LIDAR, radar, etc.). The generated environment model may include the position and motion of all AMRs <NUM>, the SGR <NUM>, the human <NUM>, and/or other static and dynamic objects within the environment. As illustrated in <FIG>, the model includes locations of the AMRs <NUM> and determine paths through the environment <NUM>.

The SGR system leverages the environment model to perform monitoring of the human(s) <NUM> within the environment <NUM>. In performing human monitoring operations <NUM>, the SGR system can predict the motion of the human <NUM> for a predetermined time period (e.g. serval seconds) into the future. Based the predicted movement of the human <NUM>, the SGR system is configured to plan and optimize the path of the SGR <NUM>. The planned path may additionally be based on the predetermined path provided to the system and/or determined from a provided task to be performed by the human <NUM>. The SGR <NUM> may then use the planned path to operate in a guiding mode to guide the human <NUM> safely within the environment <NUM>. As is shown in <FIG> and <FIG>, the SGR <NUM> is configured to highlight or otherwise convey unsafe areas within the environment <NUM> to the human <NUM>. For example, the SGR <NUM> may provide audio/visual messages to the human <NUM> (via I/O interface <NUM>) and/or a projected path indicator (e.g. laser projected markers onto the floor of the environment.

The conveyance of information to the human <NUM> improves as the SGR <NUM> improves in location relative to the movement of the human <NUM>. Therefore, the SGR <NUM> is configured to adapt its movement based on predictions of the movements of the human <NUM>. According to the disclosure, the SGR <NUM> may be configured to compare the current location and motion of the human <NUM> to the originally planned path (e.g. from entrance to the working location). If the SGR <NUM> and/or other components (e.g. controller <NUM> and/or server <NUM>) detects a deviation from the original planned path (e.g. because the human <NUM> ignores guidance and/or commands from the SGR <NUM>), the SGR <NUM> may being to operate in a tracking (following) mode. In the tracking mode, the SGR tracks and follows the human <NUM> as the human <NUM> moves through the environment to ensure safety of the human <NUM>. For example, the SGR <NUM> and/or other components (e.g. controller <NUM> and/or server <NUM>) may adaptively indicate safe path information to the human <NUM> and/or slow down or shutdown AMRs <NUM> the SGR <NUM> predicts may be encountered by the human <NUM> while deviating from the original path. The SGR <NUM> and/or other components (e.g. controller <NUM> and/or server <NUM>) may determine the new path that will be taken by the human <NUM> during a deviation by predicting future movements of the human <NUM>. According to the disclosure, when the SGR <NUM> and/or other components (e.g. controller <NUM> and/or server <NUM>) detect that the human <NUM> is deviating from the original planned path, the SGR <NUM> may position itself so as to block the human <NUM> from moving in a particular direction to force or encourage the human <NUM> back onto the originally planned path. In this example, the SGR <NUM> may additionally or alternative warn the human <NUM> that they are deviating from the planned path via the I/O interface <NUM> (e.g. audio or visional warning). The SGR may additionally or alternatively restrain and/or contain the human <NUM> so as to prevent the human <NUM> from making an undesired movement.

According to the disclosure, the SGR system may detect possible anomalies/deviations associated with human <NUM> in addition or as an alternative to deviations from the originally planned path. For example, the SGR system may detect that the human <NUM> is performing a different, unplanned task, and/or is accessing restricted information and/or locations (e.g. reading terminal screens that should not be read, accessing restricted machinery, etc.).

In detecting anomalies in the human activity, the SGR <NUM> may be configured to perform behavior anomaly detection operations. The anomaly detection may include determining a behavior certainty score (k) reflecting a degree of certainty the SGR system assigns to the behavior of the human <NUM> within the environment <NUM> based on the original planned behavior (e.g. predicted behavior based on planned task, etc.). In this example, a score of <NUM> corresponds to perfect/complete fulfillment, while a score of <NUM> corresponds to situations where evidence suggests that the human behavior is no longer in alignment to the planned behavior. The behavior certainty score may be determined based on the following equation:<MAT>.

Using the environment model as well as the predicted path(s) for the human <NUM>, the SGR system may perform a safety assessment to evaluate the risk associated with each AMR <NUM> (and/or other objects) within the environment <NUM>. According to the disclosure, the SGR <NUM> and/or other components (e.g. controller <NUM> and/or server <NUM>) are configured to determine a collision probability (as a function of distance between the human <NUM> and the AMRs <NUM> and the certainty score), the exposure/duration in which the human <NUM> would be within a dangerous or potentially hazardous distance to the AMR(s) <NUM> (e.g. determining if the human <NUM> is within a hazardous distance threshold to the AMR(s) <NUM>), and the potential severity of harm. According to the disclosure, the risk estimation may be performed for several operating modes of one or more AMRs <NUM>, including, for example, standing (shut down) operation, reduced speed operation, and normal operating speed. In this example, the SGR system may determine which of the AMRs <NUM> can continue or resume operating at normal operational parameters (e.g. normal speed) due to low risk, which robots should operate at a reduced speed, and which robots should be stopped (shut down completely). The SGR <NUM> and/or other components (e.g. controller <NUM> and/or server <NUM>) may send this semantic information as control commands to the appropriate AMRs <NUM> to control the AMRs <NUM> accordingly. According to the disclosure, the environment model may be updated with the information on safe/unsafe regions as depicted in <FIG> and the SGR <NUM> may provide corresponding notifications to the human <NUM> (e.g. audio and/or visual notifications and warnings) during the Human Communication <NUM> processing.

According to the disclosure, the SGR <NUM> and/or other components (e.g. controller <NUM> and/or server <NUM>) may generate control commands for the SGR <NUM> based on the path to the desired location within the environment <NUM>, the predicted motion of the human <NUM>, the certainty score, the environment model, and/or the information on the safe/unsafe regions in the environment <NUM>. Further, based on the operating mode of the SGR <NUM> (e.g. guide mode vs. tracking mode), the SGR <NUM> and/or other components (e.g. controller <NUM> and/or server <NUM>) may generate different commands for the same environment model and robot constellations. For example, if the human <NUM> does not follow the originally planned path, and the SGR <NUM> operates in the tracking mode, the control commands may cause the SGR <NUM> to move/cut off the human <NUM> to redirect the human <NUM> back to the originally planned path. Similarly, if the certainty score is high and the SGR <NUM> is operating in the guiding mode, the velocity of the SGR <NUM> may be reduced to cause the human <NUM> to traverse the environment <NUM> at a corresponding slower speed.

A further advantage of the SGR system according to the disclosure is that the SGR system may leverage its knowledge of the tasks of the various AMRs <NUM> to adjust the movement of the human <NUM> within the environment <NUM>. For example, because the SGR <NUM> is aware of the tasks performed by the AMRs <NUM>, the SGR <NUM> may slow down its movement to cause the human <NUM> to reduce their speed as the human <NUM> and SGR <NUM> approach a particular AMR <NUM> that may be currently performing a task that will be completed in a short period of time. The reduction in the speed of the SGR <NUM> will provide the additional time for the AMR <NUM> to complete its task (thereby rendering the AMR <NUM> safe for the human <NUM> to approach) instead of having to slow or halt the operation of the AMR <NUM> to provide safe passage by the human <NUM>. That is, the omniscient nature of the system provides that the system has knowledge of AMR tasks, SGR position and planned movement, human position and predicted movement, the planned path, and planned human task, such that the system is configured to determine the optimal and best economical path and task schedule.

The human communication <NUM> operations provide for the SGR <NUM> to communicate with the human <NUM>. In operation, the SGR <NUM> is configured to provide audio and visual communications (e.g. warnings, directions), such as audible warning, warning displayed on a display of the SGR <NUM>, laser generated pointers or guides projected on the ground, light projected onto regions of the floor, and/or projected warning signals into the air, to make the human <NUM> aware of safe and unsafe areas (e.g. green vs. red areas in <FIG>). Moreover, the SGR <NUM> can highlight the path to the desired destination using these interfaces <NUM>. Audio communications may include audio messages to request, for example, that the human <NUM> walk slower, turn left or right, or the like. Furthermore, if a deviation from the originally planned path is detected, a warning message (e.g. audio and/or visual) can be generated, as well as visual warning signals displayed on a display screen to the human <NUM>.

According to the disclosure, the SGR may additionally be configured to provide first-level support given the knowledge of the anticipated task to be performed by the human <NUM>. For example, the SGR <NUM> may provide support on-site, such as providing an initial check of the situation and/or even use attempting to fix an issue using a manipulator arm of the SGR <NUM>.

According to the disclosure, the SGR <NUM> may additionally or alternatively be configured to inspect the environment <NUM> (e.g. using one or more of its sensors <NUM>) to predict required maintenance. For example, the SGR <NUM> could traverse the environment <NUM> to monitor behavior of the AMRs <NUM> to observe the motion and actions of the AMRs <NUM> to detect possible anomalies. Such inspection operations may be requested based on analysis of detected behavior by, for example the controller and/or server <NUM> of the AMRs <NUM>, performed periodically, and/or performed while the SGR <NUM> is performing guidance operations through the environment <NUM> with a human <NUM>.

The designs of the disclosure may be implemented in hardware (e.g., circuits), firmware, software, or any combination thereof. Designs may also be implemented as instructions stored on a machine-readable medium, which may be read and executed by one or more processors. A machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). A machine-readable medium may include read only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices; electrical, optical, acoustical or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.), and others. Further, firmware, software, routines, instructions may be described herein as performing certain actions. However, it should be appreciated that such descriptions are merely for convenience and that such actions in fact results from computing devices, processors, controllers, or other devices executing the firmware, software, routines, instructions, etc. Further, any of the implementation variations may be carried out by a general purpose computer.

The term "data" as used herein may be understood to include information in any suitable analog or digital form, e.g., provided as a file, a portion of a file, a set of files, a signal or stream, a portion of a signal or stream, a set of signals or streams, and the like. Further, the term "data" may also be used to mean a reference to information, e.g., in form of a pointer. The term "data", however, is not limited to the aforementioned data types and may take various forms and represent any information as understood in the art.

The terms "processor," "processing circuitry," or "controller" as used herein may be understood as any kind of technological entity that allows handling of data. The data may be handled according to one or more specific functions executed by the processor, processing circuitry, or controller. Further, processing circuitry, a processor, or a controller as used herein may be understood as any kind of circuit, e.g., any kind of analog or digital circuit. Processing circuitry, a processor, or a controller may thus be or include an analog circuit, digital circuit, mixed-signal circuit, logic circuit, processor, microprocessor, Central Processing Unit (CPU), Graphics Processing Unit (GPU), Digital Signal Processor (DSP), Field Programmable Gate Array (FPGA), integrated circuit, Application Specific Integrated Circuit (ASIC), etc., or any combination thereof. Any other kind of implementation of the respective functions, which will be described below in further detail, may also be understood as processing circuitry, a processor, controller, or logic circuit. It is understood that any two (or more) of the processors, controllers, logic circuits, or processing circuitries detailed herein may be realized as a single entity with equivalent functionality or the like, and conversely that any single processor, controller, logic circuit, or processing circuitry detailed herein may be realized as two (or more) separate entities with equivalent functionality or the like.

As used herein, "memory" is understood as a computer-readable medium in which data or information can be stored for retrieval. References to "memory" included herein may thus be understood as referring to volatile or non-volatile memory, including random access memory (RAM), read-only memory (ROM), flash memory, solid-state storage, magnetic tape, hard disk drive, optical drive, among others, or any combination thereof. Registers, shift registers, processor registers, data buffers, among others, are also embraced herein by the term memory. The term "software" refers to any type of executable instruction, including firmware.

In one or more of the implementations described herein, processing circuitry can include memory that stores data and/or instructions. The memory can be any well-known volatile and/or non-volatile memory, including read-only memory (ROM), random access memory (RAM), flash memory, a magnetic storage media, an optical disc, erasable programmable read only memory (EPROM), and programmable read only memory (PROM). The memory can be non-removable, removable, or a combination of both.

Unless explicitly specified, the term "transmit" encompasses both direct (point-to-point) and indirect transmission (via one or more intermediary points). Similarly, the term "receive" encompasses both direct and indirect reception. Furthermore, the terms "transmit," "receive," "communicate," and other similar terms encompass both physical transmission (e.g., the transmission of radio signals) and logical transmission (e.g., the transmission of digital data over a logical software-level connection). Processing circuitry, a processor, or a controller may transmit or receive data over a software-level connection with another processor, controller, or processing circuitry in the form of radio signals, where the physical transmission and reception is handled by radio-layer components such as RF transceivers and antennas, and the logical transmission and reception over the software-level connection is performed by the processors or controllers. The term "communicate" encompasses one or both of transmitting and receiving, i.e., unidirectional or bidirectional communication in one or both of the incoming and outgoing directions. The term "calculate" encompasses both 'direct' calculations via a mathematical expression/formula/relationship and 'indirect' calculations via lookup or hash tables and other array indexing or searching operations.

An "agent" may be understood to include any type of driven object. An agent may be a driven object with a combustion engine, a reaction engine, an electrically driven object, a hybrid driven object, or a combination thereof. An agent may be or may include a moving robot, a personal transporter, a drone, and the like.

The term "autonomous agent" may describe an agent that implements all or substantially all navigational changes, at least during some (significant) part (spatial or temporal, e.g., in certain areas, or when ambient conditions are fair, or on highways, or above or below a certain speed) of some drives. Sometimes an "autonomous agent" is distinguished from a "partially autonomous agent" or a "semi-autonomous agent" to indicate that the agent is capable of implementing some (but not all) navigational changes, possibly at certain times, under certain conditions, or in certain areas. A navigational change may describe or include a change in one or more of steering, braking, or acceleration/deceleration of the agent. An agent may be described as autonomous even in case the agent is not fully automatic (fully operational with driver or without driver input). Autonomous agents may include those agents that can operate under driver control during certain time periods and without driver control during other time periods. Autonomous agents may also include agents that control only some implementations of agent navigation, such as steering (e.g., to maintain an agent course between agent lane constraints) or some steering operations under certain circumstances (but not under all circumstances), but may leave other implementations of agent navigation to the driver (e.g., braking or braking under certain circumstances). Autonomous agents may also include agents that share the control of one or more implementations of agent navigation under certain circumstances (e.g., hands-on, such as responsive to a driver input) and agents that control one or more implementations of agent navigation under certain circumstances (e.g., hands-off, such as independent of driver input). Autonomous agents may also include agents that control one or more implementations of agent navigation under certain circumstances, such as under certain environmental conditions (e.g., spatial areas, roadway conditions). In some implementations, autonomous agents may handle some or all implementations of braking, speed control, velocity control, and/or steering of the agent. An autonomous agent may include those agents that can operate without a driver. The level of autonomy of an agent may be described or determined by the Society of Automotive Engineers (SAE) level of the agent (as defined by the SAE in SAE J3016 <NUM>: Taxonomy and definitions for terms related to driving automation systems for on road motor vehicles) or by other relevant professional organizations. The SAE level may have a value ranging from a minimum level, e.g. level <NUM> (illustratively, substantially no driving automation), to a maximum level, e.g. level <NUM> (illustratively, full driving automation).

The systems and methods of the disclosure may utilize one or more machine learning models to perform corresponding functions of the agent (or other functions described herein). The term "model" as, for example, used herein may be understood as any kind of algorithm, which provides output data from input data (e.g., any kind of algorithm generating or calculating output data from input data). A machine learning model may be executed by a computing system to progressively improve performance of a specific task. According to the disclosure, parameters of a machine learning model may be adjusted during a training phase based on training data. A trained machine learning model may then be used during an inference phase to make predictions or decisions based on input data.

The machine learning models described herein may take any suitable form or utilize any suitable techniques. For example, any of the machine learning models may utilize supervised learning, semi-supervised learning, unsupervised learning, or reinforcement learning techniques.

In supervised learning, the model may be built using a training set of data that contains both the inputs and corresponding desired outputs. Each training instance may include one or more inputs and a desired output. Training may include iterating through training instances and using an objective function to teach the model to predict the output for new inputs. In semi-supervised learning, a portion of the inputs in the training set may be missing the desired outputs.

In unsupervised learning, the model may be built from a set of data which contains only inputs and no desired outputs. The unsupervised model may be used to find structure in the data (e.g., grouping or clustering of data points) by discovering patterns in the data. Techniques that may be implemented in an unsupervised learning model include, e.g., self-organizing maps, nearestneighbor mapping, k-means clustering, and singular value decomposition.

Reinforcement learning models may be given positive or negative feedback to improve accuracy. A reinforcement learning model may attempt to maximize one or more objectives/rewards. Techniques that may be implemented in a reinforcement learning model may include, e.g., Q-learning, temporal difference (TD), and deep adversarial networks.

The systems and methods of the disclosure may utilize one or more classification models. In a classification model, the outputs may be restricted to a limited set of values (e.g., one or more classes). The classification model may output a class for an input set of one or more input values. An input set may include road condition data, event data, sensor data, such as image data, radar data, LIDAR data and the like, and/or other data as would be understood by one of ordinary skill in the art. A classification model as described herein may, for example, classify certain driving conditions and/or environmental conditions, such as weather conditions, road conditions, and the like. References herein to classification models may contemplate a model that implements, e.g., any one or more of the following techniques: linear classifiers (e.g., logistic regression or naive Bayes classifier), support vector machines, decision trees, boosted trees, random forest, neural networks, or nearest neighbor.

One or more regression models may be used. A regression model may output a numerical value from a continuous range based on an input set of one or more values. References herein to regression models may contemplate a model that implements, e.g., any one or more of the following techniques (or other suitable techniques): linear regression, decision trees, random forest, or neural networks.

A machine learning model described herein may be or may include a neural network. The neural network may be any kind of neural network, such as a convolutional neural network, an autoencoder network, a variational autoencoder network, a sparse autoencoder network, a recurrent neural network, a deconvolutional network, a generative adversarial network, a forward-thinking neural network, a sum-product neural network, and the like. The neural network may include any number of layers. The training of the neural network (e.g., adapting the layers of the neural network) may use or may be based on any kind of training principle, such as backpropagation (e.g., using the backpropagation algorithm).

Claim 1:
A controller (<NUM>) for an autonomous agent (<NUM>), the controller comprising:
interface means for receiving sensor data; and
processing means configured for:
determining, based on the received sensor data: a position of the autonomous agent (<NUM>), a position of a human (<NUM>) being guided by the autonomous agent (<NUM>) in an environment, and operational information of one or more other autonomous agents (<NUM>, <NUM>, <NUM>);
characterized in that the processing means is further configured for:
estimating a risk of harm to the human (<NUM>) based on a collision probability of the one or more other autonomous agents (<NUM>, <NUM>, <NUM>) with the human (<NUM>); wherein
the collision probability is determined based on a distance of the human (<NUM>) to the one or more other autonomous agents (<NUM>, <NUM>, <NUM>) and a behavior certainty score; and
the behavior certainty score is determined based on a current movement of the human (<NUM>) and a planned path of the autonomous agent (<NUM>) through the environment; and
controlling a movement of the autonomous agent (<NUM>) based on the determined position of the autonomous agent (<NUM>), the determined position of the human (<NUM>) being guided by the autonomous agent (<NUM>), and the operational information of one or more other autonomous agents (<NUM>, <NUM>, <NUM>) and based on the estimated risk of harm to the human (<NUM>).