Patent Description:
Reinforcement learning models are an attractive field of study in interactive environments in which an agent is trained to accomplish a goal or maximize a result in connection with a certain process. For instance, using reinforcement learning, an agent can be trained to achieve a higher score in a video game via exchange of states and rewards. Robotics and related devices are under-utilized in the field of reinforcement learning due to issues of reliability and poor learning performance with robots. Some reinforcement learning models have proven to be somewhat effective in computer simulations, which are non-real-world environments that do not involve control of physical robotics and devices. However, when applied to real-world environments in which physical robotics and devices operate in real-time, these simulated reinforcement learning models are ineffective and characterized by poor performance. The deficiencies of reinforcement learning models operating in real-time in physical real-world environments are attributed to a variety of factors, including slow rate of data collection, partial observability, noisy sensors, safety, and frailty of the physical devices involved.

Apart from the foregoing deficiencies, a significant issue reinforcement learning systems encounter is that delays or time offsets between events in the system can inhibit reinforcement learning by an agent. For example, variabilities and time delays associated with receipt of sensorimotor packets by a reinforcement learning system and receipt of action commands by a robotic system external to the reinforcement learning system significantly deteriorate the learning performance of the reinforcement learning system. In real-world reinforcement learning environments, the delay in receipt of various information received by the reinforcement learning agent may disassociate causation between a reward or state and a corresponding action. In short, such delays can add uncertainty as to how actions affect subsequent observations and may also affect the responsiveness of the robotic system to action commands and stimulus. As a result, the performance real-world reinforcement learning systems involving robotics has thus far been inferior to the theoretical performance of reinforcement learning simulations involving simulated response of robotics and devices.

<CIT> relates to a method for operating a robot involving generating combined actions out of a set of original actions stored in an action library and storing the combined actions in the actions library. A reinforcement learning algorithm is applied to the set of actions stored in the action library to learn a control policy making use of the original actions and the combined actions. The robot is operated on the basis of the resulting action library.

<NPL>" relates to an approach for performing concurrent activities in Markov decision processes (MDPs) based on the coarticulation framework.

Briefly stated, embodiments disclosed herein are directed to systems and methods for implementing a reinforcement learning architecture that interacts with an external real-world environment having a plurality of physical devices operating therein.

The invention is as defined in claim <NUM>, to a method, and in claim <NUM>, to a computer system, of the appended set of claims.

<FIG> shows a diagram of a reinforcement learning process <NUM> according to one or more embodiments. The reinforcement learning process <NUM> is a model for reinforcement machine learning in which a reinforcement learning agent <NUM> learns via interaction with an environment <NUM>. In particular, the reinforcement learning agent <NUM> determines a state <NUM> representing a state of the environment <NUM> at a time t and determines a reward <NUM> associated with the last transition or change that the agent <NUM> caused in the environment <NUM>. The state <NUM> may represent characteristics of the environment <NUM>, such as a position, operating state, speed, direction, or orientation of an entity or object in the environment <NUM>. The reward <NUM> represents whether the last transition or change in the environment <NUM> was successful with respect to a defined goal. The reward <NUM> may further represent a degree to which the transition or change was successful or unsuccessful. The state <NUM> and the reward <NUM> may each comprise a set of scalar values each representative of an aspect of the environment <NUM>.

The agent <NUM> adapts a reinforcement learning model for behavior by interacting with the environment <NUM> in discrete time steps to maximize the cumulative reward received by the agent <NUM> in association with performing actions in connection with the environment <NUM>. The agent <NUM> processes the state <NUM> and reward <NUM> received for the time t and determines an action <NUM> to be performed based on the state <NUM> and the reward <NUM>. The agent <NUM> then performs or causes performance of the action <NUM> involving the environment <NUM>, such as by controlling a device operating in a physical space to perform a defined operation. Thereafter, an entity operating in the environment <NUM> determines a reward <NUM> and a state <NUM> for a time t+<NUM> resulting from the action <NUM>. The agent <NUM> adapts its reinforcement learning model in response to success or failure indicated by the reward <NUM> and the new state <NUM> of the environment <NUM> and/or agent <NUM>. According to the reinforcement learning process <NUM>, the behavior of the agent <NUM> can be adapted by trial and error/success to optimize reward received, which may be associated with a goal to be achieved.

In the context of robotics or operation of a collection of devices in real-time and in a physical real-world environment, developing the reinforcement learning model of the reinforcement learning agent <NUM> has proven to be a difficult challenge. For instance, controlling robotic devices operating in a real-work environment is difficult due to the variability of time associated with steps in the process. Time delays between a reinforcement learning agent decision and sensor readings obtained may vary between the devices associated with the sensors. Processing sensory and state data may take different amounts of time for different devices in the real-world system and, as a result, the accuracy of the decision made by the reinforcement learning agent may not reflect the actual state of the real-world system. Another issue is that the reinforcement learning agent may sequentially obtain sensory data from devices operating in the real-world environment and sequentially process the sensory data to determine the actions to be performed, which can lead to disconnect between the observed state of the real-world environment and the actual state of the real-world environment. For example, a reinforcement learning agent may cause a first device to perform a first action in response to a state observed for the first device at a first time, and then cause a second device to perform a second action in response to a state observed for the second device at a second time. However, a problem may arise in which the first action changes the state of the second device, so the second action performed may not have the desired or intended effect. Worse yet, the reinforcement learning agent may receive feedback indicating that the second action was undesirable and change its policy in response when, in actuality, the second action may have been effective if performed by the second device in connection with the first action.

The present disclosure provides an operational framework in which learning and performance characteristics of real-world reinforcement learning tasks performed by the reinforcement learning agent <NUM> are similar to or approach the learning characteristics and performance of a simulated reinforcement learning algorithm. The technologies disclosed herein provide numerous benefits to a reinforcement learning system operating to achieve a defined objective in a real-world real-time external system via control of a plurality of devices. One feature of the reinforcement learning system disclosed herein is that time delays between sensory readings and actions initiated by a reinforcement learning agent are reduced by collecting and generating joint vectors based on a plurality of data instances. The time delay is also reduced by causing the reinforcement learning agent to operate in a suspended state for a defined period of time before transitioning to an active state in which a policy is applied. The reinforcement learning system of the present disclosure also improves consistency of time intervals between times at which the reinforcement learning agent makes decisions regarding the actions to be performed by the devices operating in the real-time environment. A further feature of the present disclosure is to provide a framework that is adaptable to operate different types and different configurations of devices to facilitate reinforcement learning. For instance, the framework provided herein is adaptable to add new types of devices or change the current configuration of devices operating in a real-world environment to achieve a desired objective according to a reinforcement learning process.

The following description, along with the accompanying drawings, sets forth certain specific details in order to provide a thorough understanding of various disclosed embodiments. However, one skilled in the relevant art will recognize that the disclosed embodiments may be practiced in various combinations, without one or more of these specific details, or with other methods, components, devices, materials, etc. In other instances, well-known structures or components that are associated with the environment of the present disclosure, including but not limited to the communication systems and networks and the environment, have not been shown or described in order to avoid unnecessarily obscuring descriptions of the embodiments. Additionally, the various embodiments may be methods, systems, media, or devices. Accordingly, the various embodiments may be entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects.

Throughout the specification, claims, and drawings, the following terms take the meaning explicitly associated herein, unless the context clearly dictates otherwise. The term "herein" refers to the specification, claims, and drawings associated with the current application. The phrases "in one embodiment," "in another embodiment," "in various embodiments," "in some embodiments," "in other embodiments," and other variations thereof refer to one or more features, structures, functions, limitations, or characteristics of the present disclosure, and are not limited to the same or different embodiments unless the context clearly dictates otherwise. As used herein, the term "or" is an inclusive "or" operator, and is equivalent to the phrases "A or B, or both" or "A or B or C, or any combination thereof," and lists with additional elements are similarly treated. The term "based on" is not exclusive and allows for being based on additional features, functions, aspects, or limitations not described, unless the context clearly dictates otherwise. In addition, throughout the specification, the meaning of "a," "an," and "the" include singular and plural references.

References to the term "set" (e.g., "a set of items"), as used herein, unless otherwise noted or contradicted by context, is to be construed as a nonempty collection comprising one or more members or instances.

References to the term "subset" (e.g., "a subset of the set of items"), as used herein, unless otherwise noted or contradicted by context, is to be construed as a nonempty collection comprising one or more members or instances of a set or plurality of members or instances.

<FIG> shows a diagram in which a computer system <NUM> communicates in real-time with external devices operating in an environment <NUM>. The computer system <NUM> includes one or more processors and memory storing a set of instructions that, as a result of execution by the one or more processors, cause the computer system <NUM> to perform operations described herein. The set of instructions, as result of execution by the one or more processors, cause the computer system <NUM> to invoke and run a reinforcement learning agent of a reinforcement learning system, as described with respect to <FIG> and elsewhere herein. The computer system <NUM>, in particular, operates at least in part according to a reinforcement learning architecture <NUM> that includes one or more policies π defining behavior of the computer system <NUM> with respect to the environment <NUM>. The one or more policies π of the reinforcement learning architecture <NUM> dictate actions to be performed based on a state and/or a reward observed in connection with operation of the devices in the environment <NUM> and which may correspond to associations or rules regarding relationships between stimulus (state, reward) and response (action). The reinforcement learning architecture <NUM> may include value functions for determining a long-term value return of a current state under a particular policy π. The reinforcement learning architecture <NUM> may include a model that is representative of the environment <NUM>.

The computer system <NUM> is separate from the devices in the environment <NUM> such that the computer system <NUM> is not included as a component of any of the devices in the environment <NUM>. The computer system <NUM> communicates with devices in the environment <NUM> over one or more communication networks <NUM>. The one or more networks <NUM> may include wired and/or wireless networks for facilitating communications between the computer system <NUM> and devices in the environment <NUM>. The one or more networks <NUM> may include service provider networks providing communication interconnectivity among a plurality of entities in one or more regions, public networks (e.g., the internet, the world wide web), private networks, wide area networks, local area networks, satellite networks, mesh networks, intermediate networks connecting separate networks, local area networks, satellite networks, and/or combinations thereof, by way of non-limiting example. As one particular non-limiting example, the one or more networks <NUM> may be a local area network to which the computer system <NUM> and devices in the environment <NUM>, and over which the computer system <NUM> and the devices may communicate via wired or wireless connections. Although not shown, the environment <NUM> may include a router for directing wireless communications between the computer system <NUM> and the devices in the environment <NUM>. Communications between a device in the environment <NUM> and the computer system <NUM> may have a time delay that may contribute to the aforementioned difficulties associated with reinforcement learning in real-time and in a real-world setting.

In some embodiments, the computer system <NUM> may be implemented as a cloud-computing distributed environment in which different processes are executed on separate devices, computers, or servers distributed among two or more networks or locations that are remotely located to the environment <NUM>. For instance, a first process of the reinforcement learning architecture <NUM> may be executed on a first computer located at a first geographical location and a second process of the reinforcement learning architecture <NUM> may be executed on a second computer located at a second geographical location. The first computer and the second computer may be different computers in the same facility, or may be computers that are geographically distributed from each other.

Examples of devices in the environment <NUM> include a robot <NUM> for performing tasks in the physical space of the environment <NUM> and a sensor <NUM> for detecting conditions in the physical space of the environment <NUM>. The robot <NUM> includes a set of controllable devices (e.g., robotic arm, imaging sensors, robotic grippers, locomotion devices, actuators), one or more controllers configured to control the set of controllable devices, a communication interface including one or more communication devices (e.g., Wi-Fi communication adapter, Zigbee communication adapter, Bluetooth communication adapters, wired network adapter, universal serial bus port), a body, one or more processors, and memory storing instructions for controlling operation of the robot <NUM>. The robot <NUM> may receive, from the computer system <NUM> via the communication device, instructions that, as a result of execution, cause the robot <NUM> to interact in and with environment <NUM>. For example, the robot <NUM> could receive executable instructions that cause the robot <NUM> to pick-up one or more items <NUM> and <NUM> and place the items <NUM> and <NUM> on a scale <NUM> and then in box <NUM>. Devices of the robot <NUM> may include a sensor that detects a state of the device, such as a position, orientation, speed, or acceleration of the device.

The sensor <NUM> shown in the environment <NUM> is a camera that captures images or video of the environment <NUM>. The sensor <NUM> includes a communication interface (e.g., wireless network adapter, video output port) via which the sensor <NUM> transmits the images or video captured to the computer system <NUM> over the network <NUM>. The sensor <NUM> may receive communication signals from the computer system <NUM> over the network <NUM> that cause the sensor <NUM> to perform various operations. The sensor <NUM> may receive instructions that cause the sensor <NUM> to adjust its operating mode, such as by adjusting settings (e.g., focus, frame rate, resolution). The sensor <NUM> may have various devices associated therewith that can be controlled responsive to communications from the computer system <NUM> - for example, a set of motors may be attached to the sensor <NUM> for selectively adjusting an orientation of the sensor <NUM> and a light source coupled to the sensor <NUM> may be selectively operable to illuminate areas in the environment <NUM>. Non-limiting examples of other sensors that may be included in the environment <NUM> include time-of-flight sensors, temperature sensors, microphones, laser range finder (e.g., Lidar), radar, speed sensors, force sensors (e.g., scale), pressure sensors, gyroscopes, electromagnetics sensors (e.g., Hall effect sensor), strain gauges, and proximity sensors.

The environment <NUM> may include devices other than the robot <NUM>, which are controllable by the computer system <NUM> and which may provide information to the computer system <NUM> regarding the state of the device at a given time. Such devices include robotic arms, motors, conveyor belts and/or wheels, hoppers, actuators, sorting devices, switches, valves, and the like.

For a time t in the environment <NUM>, signals may be transmitted over the network <NUM> indicating a state <NUM> associated with one or more robots <NUM> or devices operating in the environment <NUM>, or detected by one or more sensors <NUM> that observe the environment <NUM>. Each of the devices <NUM> operating in the environment <NUM> may provide an observation that partially describes the state <NUM> of the environment <NUM>. Signals may also be transmitted indicating a set of rewards <NUM> associated with one or more robots <NUM>, sensors <NUM>, and devices in the environment <NUM> for the time t. The state <NUM> is sensory information indicating the state of the corresponding device in the environment <NUM>. For a robotic arm, the state <NUM> may indicate information regarding position, orientation, movement, etc., of the arm. For the sensor <NUM>, the state <NUM> may be a measurement for the time t, such as an image captured of the environment <NUM> at the time t. The reward <NUM> is an alphanumeric scalar value corresponding to the last action or transition experienced by the robots <NUM>, sensors <NUM>, and the devices in the environment <NUM>. The reward <NUM>, in at least some embodiments, is calculated or determined by a process executing on the computer system <NUM>, such as a process of the reinforcement learning architecture <NUM>. The reward <NUM> may indicate, at least in part, a desirability of a current state of the corresponding device, sensor, robot, etc., in the environment <NUM> relative to a defined objective. For a state of the robot <NUM> that desirable, for example, the reward <NUM> may be a positive integer having a value proportional to a degree of desirability of the state of the robot <NUM> relative to one or more goals. By contrast, the reward <NUM> may be a negative integer having a value proportional to a degree of undesirability of the state of the robot <NUM> relative to the one or more goals.

The computer system <NUM> processes the set of state(s) <NUM> for the time t and generates a set of actions <NUM> to be performed by the robot <NUM>, the sensor <NUM>, and/or the devices in the environment <NUM>. The set of actions <NUM> are transmitted over the network <NUM> to the corresponding robots <NUM>, sensors <NUM>, and devices that are to perform the actions. The set of actions <NUM> may be selected from a defined plurality of actions specified in memory of the computer system <NUM>. For a robotic arm, an action of the set of actions <NUM> may cause a gripper of the robotic arm to move to a particular position and orientation. The computer system <NUM> processes the set of reward(s) <NUM> in connection with a reinforcement learning process. Other actions may include movement of gears to move the robot to a certain location in the environment <NUM>. Each of the actions <NUM> may include parameters that specify a target device, robot, or sensor; the action to be performed; and parameters indicating characteristics of the action (e.g., position, speed, direction).

<FIG> shows a view of the robot <NUM> in the environment <NUM> according to one or more embodiments. The robot <NUM> comprises a variety of devices that each perform one or more functions or operations. The robot <NUM> shown, for instance, includes a camera <NUM> that captures images of the environment <NUM>, a pair of robotic arms <NUM> for interacting with objects and features in the environment <NUM>, and a set of motors <NUM> for providing motive power to move and orient the robot <NUM> within the environment <NUM>.

Each of the devices may be configured to provide sensory information regarding a state of the respective device and/or regarding the environment <NUM>. The camera <NUM> may provide sensory information in the form of images depicting a state of the environment <NUM> from the perspective of the robot <NUM>. The robotic arm <NUM> may include a gripping portion <NUM> and a plurality of segments 312a through 312c that independently move to position and orient the gripping portion <NUM>. Each of the segments 312a through 312c and the gripping portion <NUM> include servomotors that provide sensory information regarding a position, orientation, state, etc., associated with the corresponding portions. The gripping portion <NUM> may include sensors (e.g., capacitive, strain gauge, resistive) configured to provide sensory information indicating whether the gripping portion <NUM> is gripping an object or feature and a pressure applied to the object or feature. The robotic arm <NUM> may include other sensors that provide sensory information regarding the object or feature. The set of motors <NUM> may be stepper motors or servomotors that provide sensory information regarding a rotational or linear position of each respective motor.

Each of the devices may be configured to receive actuation information for controlling operation of the respective device. The camera <NUM>, for example, may receive actuation information for adjusting settings thereof. The camera <NUM> may also have associated therewith one or more motors that receive actuation information that controls a position and/or an orientation of the camera <NUM>, such as by adjusting a rotational position of a head <NUM> of the robot <NUM>. The robotic arm <NUM> receives actuation information that controls the segments 312a through 312c and operation of the gripper portion <NUM>. The set of motors <NUM> receive actuation information that causes application of motive force to move or orient a body <NUM> of the robot <NUM> in the environment <NUM>.

In a real-world setting in which the robot <NUM> operates in real-time and communicates with the computer system <NUM> over the network <NUM>, each of the motors have different operational characteristics. As one exemplary configuration of the robot <NUM>, the camera <NUM> streams images at a rate of <NUM> Hertz (Hz), a motor of the robotic arm <NUM> streams sensory information packets at <NUM> and is actuated at a rate of <NUM>, and a motor of the set of motors <NUM> is actuated at a rate of <NUM> and streams sensory information packets at a rate of <NUM>. Previously, it was difficult to implement an architecture in which the reinforcement learning agent of the computer system <NUM> could receive the sensory information from the robot <NUM> and control the motors at a particular timescale due to the differences in performance characteristics between the devices. The reinforcement learning architecture <NUM> provides a uniform interface and framework in which multiple processes operate independently and in parallel to perform data processing, task-specific computation, and agent-specific computation, as described in further detail herein.

<FIG> shows a schematic diagram <NUM> of the computer system <NUM> and devices operating in the environment <NUM> according to one or more embodiments. As discussed herein, robots may take any of a wide variety of forms. <FIG> schematically shows parts of robot <NUM>. Robot <NUM> includes at least one body, and a control subsystem <NUM> that includes at least one processor <NUM>, at least one non-transitory tangible computer- and processor-readable data storage <NUM>, and at least one bus <NUM> to which the at least one processor <NUM> and the at least one non-transitory tangible computer- or processor-readable data storage <NUM> are communicatively coupled.

The at least one processor <NUM> may be any logic processing unit, such as one or more microprocessors, central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), programmable gate arrays (PGAs), programmed logic units (PLUs), and the like. At least one processor <NUM> may be referred to herein by the singular, but may be two or more processors.

Robot <NUM> may include a communications subsystem <NUM> communicatively coupled to (e.g., in communication with) the bus(es) <NUM> and provides bi-directional communication with other systems (e.g., systems external to the robot <NUM>) via a network or non-network communication channel, such as the network(s) <NUM> described herein. The communications subsystem <NUM> may include one or more buffers. The communications subsystem <NUM> receives and sends data for the robot <NUM>, such as sensory information and actuation information.

The communications subsystem <NUM> may be any circuitry effecting bidirectional communication of processor-readable data, and processor-executable instructions, for instance radios (e.g., radio or microwave frequency transmitters, receivers, transceivers), communications ports and/or associated controllers. Suitable communication protocols include FTP, HTTP, Web Services, SOAP with XML, WI-FI compliant, BLUETOOTH compliant, cellular (e.g., GSM, CDMA), and the like.

Robot <NUM> includes an input subsystem <NUM>. In any of the implementations, the input subsystem <NUM> can include one or more sensors that measure conditions or states of robot <NUM>, and/or conditions in the environment in which the robot <NUM> operates. Such sensors include cameras or other imaging devices (e.g., responsive in visible and/or nonvisible ranges of the electromagnetic spectrum including for instance infrared and ultraviolet), radars, sonars, touch sensors, pressure sensors, load cells, microphones, meteorological sensors, chemical sensors, or the like. Such sensors include internal sensors, pressure sensors, load cells, strain gauges, vibration sensors, microphones, ammeter, voltmeter, or the like. In some implementations, the input subsystem <NUM> includes receivers to receive position and/or orientation information. For example, a global position system (GPS) receiver to receive GPS data, two more time signals for the control subsystem <NUM> to create a position measurement based on data in the signals, such as, time of flight, signal strength, or other data to effect (e.g., make) a position measurement. Also for example, one or more accelerometers can provide inertial or directional data in one, two, or three axes. In some implementations, the input subsystem <NUM> includes receivers to receive information that represents posture. For example, one or more accelerometers or one or more inertial measurement units can provide inertial or directional data in one, two, or three axes to the control subsystem <NUM> to create a position and orientation measurements. The control subsystem <NUM> may receive joint angle data from the input subsystem <NUM> or the manipulation subsystem described herein.

Robot <NUM> includes an output subsystem <NUM> comprising output devices, such as, speakers, lights, and displays. The input subsystem <NUM> and output subsystem <NUM>, are communicatively coupled to the processor(s) <NUM> via the bus(es) <NUM>.

Robot <NUM> includes a propulsion or motion subsystem <NUM> comprising motors, actuators, drivetrain, wheels, tracks, treads, and the like to propel or move the robot <NUM> within a physical space and interact with it. The propulsion or motion subsystem <NUM> comprises of one or more motors, solenoids or other actuators, and associated hardware (e.g., drivetrain, wheel(s), treads), to propel robot <NUM> in a physical space. For example, the propulsion or motion subsystem <NUM> may include a drive train and wheels, or may include legs independently operable via electric motors. Propulsion or motion subsystem <NUM> may move the body <NUM> of the robot <NUM> within the environment <NUM> as a result of motive force applied by the set of motors <NUM>.

Robot <NUM> includes a manipulation subsystem <NUM>, for example comprising one or more arms, end-effectors, associated motors, solenoids, other actuators, gears, linkages, drive-belts, and the like coupled and operable to cause the arm(s) and/or end-effector(s) to move within a range of motions. For example, the manipulation subsystem <NUM> causes actuation of the robotic arm <NUM> or other device for interacting with objects or features in the environment <NUM>. The manipulation subsystem <NUM> is communicatively coupled to the processor(s) <NUM> via the bus(es) <NUM>, which communications can be bi-directional or uni-directional.

Components in robot <NUM> may be varied, combined, split, omitted, or the like. For example, robot <NUM> could include a pair of cameras (e.g., stereo pair) or a plurality of microphones. Robot <NUM> may include one, two, or three robotic arms or manipulators associated with the manipulation subsystem <NUM>. In some implementations, the bus(es) <NUM> include a plurality of different types of buses (e.g., data buses, instruction buses, power buses) included in the at least one body <NUM>. For example, robot <NUM> may include a modular computing architecture where computational resources devices are distributed over the components of robot <NUM>. In some implementations, a robot (e.g., robot <NUM>), could have a processor in a left arm and data storage in its thorax. In some implementations, computational resources are located in the interstitial spaces between structural or mechanical components of the robot <NUM>. A data storage device could be in a leg and a separate data storage device in another limb or appendage. In some implementations, the computational resources distributed over the body include redundant computational resources.

The at least one data storage <NUM> includes at least one non-transitory or tangible storage device. The at least one data storage <NUM> can include two or more distinct non-transitory storage devices. The data storage <NUM> can, for example, include one or more a volatile storage devices, for instance random access memory (RAM), and/or one or more non-volatile storage devices, for instance read only memory (ROM), Flash memory, magnetic hard disk (HDD), optical disk, solid state disk (SSD), and the like. A person of skill in the art will appreciate storage may be implemented in a variety of non-transitory structures, for instance a read only memory (ROM), random access memory (RAM), a hard disk drive (HDD), a network drive, flash memory, digital versatile disk (DVD), any other forms of computer- and processor- readable memory or storage medium, and/or a combination thereof. Storage can be read only or read-write as needed. Further, volatile storage and non-volatile storage may be conflated, for example, caching, using solid-state devices as hard drives, in-memory data processing, and the like.

The at least one data storage <NUM> includes or stores processor-executable instructions and/or processor-readable data <NUM> associated with the operation of robot <NUM> or other devices. Here, processor-executable instructions and/or processor-readable data may be abbreviated to processor-executable instructions and/or data.

The execution of the processor-executable instructions and/or data <NUM> cause the at least one processor <NUM> to carry out various methods and actions, for example via the motion subsystem <NUM> or the manipulation subsystem <NUM>. The processor(s) <NUM> and/or control subsystem <NUM> can cause robot <NUM> to carry out various methods and actions including receiving, transforming, and presenting information; moving in the environment <NUM>; manipulating items; and acquiring data from sensors. Processor-executable instructions and/or data <NUM> can, for example, include a basic input/output system (BIOS) <NUM>, an operating system <NUM>, drivers <NUM>, communication instructions and data <NUM>, input instructions and data <NUM>, output instructions and data <NUM>, motion instructions and data <NUM>, and executive instructions and data <NUM>.

Exemplary operating systems <NUM> include ANDROID™, LINUX®, and WINDOWS®. The drivers <NUM> include processor-executable instructions and/or data that allow control subsystem <NUM> to control circuitry of robot <NUM>. The processor-executable communication instructions and/or data <NUM> include processor-executable instructions and data to implement communications between robot <NUM> and an operator interface, terminal, a computer, or the like. The processor-executable input instructions and/or data <NUM> guide robot <NUM> to process input from sensors in input subsystem <NUM>. The processor-executable input instructions and/or data <NUM> implement, in part, the methods described herein. The processor-executable output instructions and/or data <NUM> guide robot <NUM> to provide information that represents, or produce control signal that transforms, information for display. The processor-executable motion instructions and/or data <NUM>, when executed, cause the robot <NUM> to move in a physical space and/or manipulate one or more items. The processor-executable motion instructions and/or data <NUM>, when executed, may guide the robot <NUM> in moving within its environment via components in propulsion or motion subsystem <NUM> and/or manipulation subsystem <NUM>. The processor-executable executive instructions and/or data <NUM>, when executed, guide the robot <NUM> the instant application or task for devices and sensors in the environment <NUM>. The processor-executable executive instructions and/or data <NUM>, when executed, guide the robot <NUM> in reasoning, problem solving, planning tasks, performing tasks, and the like.

The computer system <NUM> includes one or more processors <NUM>, memory <NUM>, and a communication interface <NUM>. The memory <NUM> is computer-readable non-transitory data storage that stores a set of computer program instructions that the one or more processors <NUM> may execute to implement one or more embodiments of the present disclosure. The memory <NUM> generally includes RAM, ROM and/or other persistent or non-transitory computer-readable storage media, such as magnetic hard drives, solid state drives, optical drives, and the like. The memory <NUM> may store an operating system comprising computer program instructions useable by the one or more processors <NUM> in the general administration and operation of the computer system <NUM>. The memory <NUM> also stores instructions that, as a result of execution, cause the one or more processors <NUM> to implement the reinforcement learning architecture <NUM> described herein.

The communication interface <NUM> includes one or more communication devices for transmitting communications and receiving communications via the network <NUM>. The one or more communication devices of the communication interface may include wired communication devices and/or wireless communication devices. Non-limiting examples of wireless communication devices include radio frequency communication adapters (e.g., Zigbee adapters, Bluetooth adapters, Wi-Fi adapters) using corresponding communication protocols, satellite communication transceivers, free-space optical communication devices, cellular network transceivers, and the like. Non-limiting examples of wired communication devices include serial communication interfaces (e.g., RS-<NUM>, Universal Serial Bus, IEEE <NUM>), parallel communication interfaces, Ethernet interfaces, coaxial interfaces, optical fiber interfaces, and power-line communication interfaces.

<FIG> shows a computational environment of the computer system <NUM> in which the reinforcement learning architecture <NUM> is implemented according to one or more embodiments. The reinforcement learning architecture <NUM> includes a reinforcement learning agent <NUM>, a task manager <NUM> that communicates with the reinforcement learning agent <NUM>, and a plurality of device communicators 506a, 506b,. , 506N (collectively referred to as communicators <NUM>) that communicates with the task manager <NUM> and that communicates with a plurality of devices 508a, 508b,. , 508N operating in the environment <NUM>. The reinforcement learning agent <NUM> is a process running on the one or more processors <NUM> of the computer system <NUM> as a result of execution of the set of instructions stored in the memory <NUM>. The task manager <NUM> is a process running independently of and in parallel with the process of the reinforcement learning agent <NUM> on the one or more processors <NUM> of the computer system <NUM> as a result of execution of the set of instructions stored in the memory <NUM>.

The device communicators <NUM> are a set of processes running independently of and in parallel with the reinforcement learning agent <NUM> and the task manager <NUM> on the one or more processors <NUM> of the computer system <NUM> as a result of execution of the set of instructions stored in the memory <NUM>. In some embodiments, each of the device communicators <NUM> is a process running independently of and in parallel with the other device communicator processes. For instance, the device communicator 506a corresponds to a first process and the device communicator 506b is a second process running independently of and in parallel with the first process. In some embodiments, the device communicators <NUM> may be implemented collectively as a single process or a plurality of processes fewer in number than the number of device communicators <NUM>. In some situations, the computational resources (e.g., number of processing cycles) associated with a group of device communicators <NUM> may be relatively low, and so grouping two or more device communicators <NUM> into a single process may be beneficial.

A first process and a second process are considered as running in parallel with each other if the first process and the second process are being executed by different cores of a processor of the computer system <NUM> or by different processors of the computer system <NUM>. Thus, the computational performance of a first core or processor associated with execution of the first process does not affect the computational performance of a second core or processor associated with execution of the second process.

Each of the communicators <NUM> is configured to communicate with a controller of a single one of the devices <NUM> in the environment <NUM>. For instance, the communicator 506a communicates with a controller of the device 508a, the communicator 506b communicates with a controller of the device 508b, and so on. The number of communicators <NUM> instantiated on the computer system <NUM> may correspond to the number of devices <NUM> that a user has configured the reinforcement learning architecture <NUM> to control. However, the number of communicators <NUM> may be more than the number of controllable devices operating in the environment <NUM> - for example, some of the devices <NUM> may be devices from which sensory data is read (e.g., camera, temperature sensor) and that do not receive actuation commands for operational control of the device. The reinforcement learning architecture <NUM> includes a user interface with which a user can interact to configure the communicators <NUM> to communicate with the devices <NUM>. The user may interact with the user interface, for example, to cause the computer system <NUM> to detect discoverable devices (e.g., the devices <NUM>), determine the set of communicators <NUM> to be instantiated, and associate each of the set of communicators with a corresponding device of the discoverable devices. Each of the communicators <NUM> is configured to communicate with a corresponding device of the devices <NUM> via the communication interface <NUM> described herein.

Each of the devices 508a, 508b,. , 508N may send sensory data 510a, 510b,. , 510N regarding a state of the respective device, such as position, speed, orientation, etc. In some embodiments, the devices <NUM> may continuously send the sensory data <NUM> as a data stream comprising a sequence of discrete encoded signals or packets. For instance, one or more of the communicators <NUM> may send a command to corresponding devices <NUM> instructing the controllers to make the sensory data <NUM> available in a streaming manner. In some embodiments, one or more of the devices <NUM> send the sensory data <NUM> in response to receiving a request from a communicator <NUM> to provide the sensory information <NUM> - for instance, in response to a communicator sending a read command to a corresponding controller of a device. The read command may cause the communicator <NUM> to wait for a packet arrival period PA having a length of time t<NUM>. The communicators <NUM> respectively receive the sensory data 510a, 510b,. , 510N from corresponding devices 508a, 508b,. , 508N operating in the environment <NUM>. The sensory data <NUM> for sensor devices is a scalar value having an alphanumeric format; however, in some embodiments, the sensory data <NUM> may be an array of scalar values for some types of sensor data, such as image data.

The reinforcement learning architecture <NUM> includes a plurality of sensor buffers 512a, 512b,. , 512N (collectively referred to as sensor buffers <NUM>) each respectively associated with or coupled to a corresponding communicator of the communicators 506a, 506b,. In response to receiving the sensory data <NUM>, the communicators 506a, 506b,. , 506N store the sensory data 510a, 510b,. , 510N in the associated sensor buffer of the plurality of sensor buffers 512a, 512b,. Each of the sensor buffers <NUM> is a data structure in the memory <NUM> configured to store sensory data <NUM>. The sensor buffers <NUM> may be, for instance, circular buffers having a defined size and in which the sensory data <NUM> is sequentially written to memory locations in the buffer and, subsequent to data being written to an end memory location of the buffer, data at a starting memory location of the circular buffer is overwritten with new data. Each of the sensor buffers <NUM> may have a size and/or configuration defined based on the device <NUM> with which the sensor buffer <NUM> is associated. For instance, a sensor buffer <NUM> for storing image data from a camera may be assigned a significantly larger amount of memory than a sensor buffer for storing data from a servomotor. Each of the sensor buffers 512a, 512b,. , 512N may store samples of sensory data <NUM> obtained by a corresponding device <NUM> for a plurality of times.

The task manager <NUM> obtains the sensory data <NUM> stored in the sensor buffers <NUM> and processes the sensory data <NUM>. In particular, the task manager <NUM> reads the next unread sensory data 510a, 510b,. , 510N from each of the sensory buffers 512a, 512b,. , 512N and generates a joint state vector <NUM> based on the sensory data 510a, 510b,. , 510N read. The joint state vector <NUM> may include observation data and reward data to be used in a reinforcement learning algorithm implemented by the reinforcement learning agent <NUM>. The joint state vector <NUM> may be organized as an array of elements that sequentially correspond to the devices 508a, 508b,. For instance, a first element of the joint state vector <NUM> may correspond to the device 508a, a second element of the joint state vector <NUM> may correspond to the device 508b, and so forth.

The computer system <NUM> includes a defined objective, such as a task or goal to be achieved, in the environment <NUM>. The task manager <NUM> evaluates states of the environment <NUM>, e.g., as observed in view of the state data <NUM>, and actions performed in the environment <NUM>, e.g., in view of actuation commands <NUM> provided by the reinforcement learning agent <NUM>, and generates the joint state vector <NUM> based on a result of the evaluation. The joint state vector <NUM> may include a plurality of values that correspond to observations of the environment <NUM> (e.g., device readings) and values that correspond to rewards associated with one or more states in the environment <NUM>. The defined objective may correspond to a set of defined conditions stored in the memory <NUM> that the task manager <NUM> accesses to assess states and actions. The joint state vector <NUM> may include reward information that is representative of a change in the state of the environment <NUM> as a result of a set of preceding action commands with respect to the defined objective. For instance, a positive reward may indicate that the last actions provided by the reinforcement learning agent <NUM> progressed the state of the environment <NUM> closer to achieving the defined objective whereas a negative reward may indicate that the last actions provided by the reinforcement learning agent <NUM> regressed the state of the environment <NUM> farther from achieving the defined objective. The task manager <NUM> may consider a long-term value of the current state of the environment <NUM> or estimated return of the last actions when generating the joint state vector <NUM>, such as by adjusting the reward to account for a more favorable long-term result of the state of the environment <NUM>.

The task manager <NUM> stores the joint state vector <NUM> in a state buffer <NUM> in the memory <NUM>. The reinforcement learning agent <NUM> obtains the joint state vector <NUM> from the state buffer <NUM> and determines, according to a policy π defined in the reinforcement learning agent <NUM>, a set of actions to be performed by the devices 508a, 508b,. , 508N based on perceived states of the environment <NUM>, as indicated by the joint state vector <NUM>.

The policy π is a set of instructions stored in the memory <NUM> that cause the process of the reinforcement learning agent <NUM> to generate an action in response to observation data and/or reward data in the joint state vector <NUM>. The policy π maps states of the environment <NUM>, such as states of the devices <NUM>, to actions to be performed in response to the detected states. The policy π is defined by a user based on tasks, goals, or desired end states to be achieved in the environment <NUM>. The policy π may have associated therewith values V or action-values Q indicating a long-term return or value for achieving the task based on a state of the environment <NUM>. The reinforcement learning agent <NUM> may also perform a learning-update process in which learning parameters (e.g., weights, biases) of the reinforcement learning model are adjusted to improve the ability of the reinforcement learning agent <NUM> to achieve the desired task, goal, end state, etc., in the environment <NUM>. The policy π may include a probability distribution indicating probabilities of a next state and/or reward that would be achieved in response to performing a particular action in response to a given state. For example, the probability distribution may indicate the probabilities associated with each of a discrete set of defined actions of causing a robotic arm to progress to a more desirable state relative to a goal of picking up an object.

The reinforcement learning agent <NUM> generates, according to the policy π, a joint action vector <NUM> that includes actuation commands indicating actions to be performed by the corresponding devices <NUM> in the next time period. For instance, the joint action vector <NUM> indicates a first action to be performed by the device 508a, a second action to be performed by the device 508b, and so on. The reinforcement learning agent <NUM> stores the joint action vector <NUM> in an actuation buffer <NUM>. As a result of generating and storing the joint action vector <NUM>, the reinforcement learning agent <NUM> returns to the suspended state for the period of time Ps. The action buffer <NUM> is a region allocated in the memory <NUM> for storing the joint action vector <NUM>. The action buffer <NUM> is a shared region in the memory that is shared between the task manager <NUM> and the reinforcement learning agent <NUM>, and which is inaccessible to other processes executing in the computer system <NUM>.

The period of time Ps may be determined by the reinforcement learning agent <NUM> based on a step time S for the reinforcement learning agent <NUM>. The step time S may be a defined period of time (e.g., by a user) that includes the time period Ps in which the reinforcement learning agent <NUM> operates in a suspended state, and a time period PA in which the reinforcement learning agent <NUM> operates in the active state, such that the following Equation <NUM> is satisfied: <MAT>.

A step time S of <NUM>, for example, may be defined for the reinforcement learning agent <NUM> by a user. The reinforcement learning agent may transition from the suspended state to the active state and obtain a joint state vector <NUM> from the state buffer <NUM> in the active state. An amount of time that it takes for the joint state vector <NUM> may vary depending on various factors associated with the joint state vector <NUM>, such as the number of observations comprising the joint state vector <NUM>. In a first example iteration of the reinforcement learning architecture <NUM>, it may take the reinforcement learning agent <NUM> to process the joint state vector <NUM> and store the joint action vector <NUM> generated in the action buffer <NUM>. Thus, the reinforcement learning agent <NUM> determines that the active time period PA is <NUM> and that the suspended time period Ps is <NUM> based on Equation <NUM>. The reinforcement learning agent <NUM> transitions from the active state to the suspended state for the remaining <NUM> of the step time S. In a second example iteration after the first example iteration, the reinforcement learning agent <NUM> may spend a greater amount of time in the active state due, e.g., to a higher overall CPU load sufficient to process another joint state vector <NUM>. In which case, the reinforcement learning agent <NUM> operates in the active state for a shorter time period than the first example iteration to ensure that the step times S for each iteration is approximately equal to the desired step time S of <NUM>.

The task manager <NUM> obtains the joint action vector <NUM> from the action buffer <NUM> and parses the joint action vector <NUM> into actuation commands 522a, 522b,. , 522N indicating actions that are to be performed by the devices 508a, 508b,. Each instance of the actuation commands <NUM> may have a format particular to the corresponding device <NUM> to which the instance of actuation commands <NUM> is to be transmitted. Individual instances or elements of the actuation commands <NUM> may be a value specifying both magnitude and direction - for instance, a positive floating point value of the actuation commands <NUM> may cause a device to move in a certain direction by an amount corresponding to the value.

The reinforcement learning architecture <NUM> includes a plurality of actuator buffers 524a, 524b,. , 524N that each correspond to one of the device communicators 506a, 506b,. Each of the actuator buffers 524a, 524b,. , 524N is a data structure in the memory <NUM> each configured to store an instance or element of data of the one or more actuation commands 522a, 522b,. Each of the actuator buffers 524a, 524b,. , 524N is, for example, a circular buffer having a defined size and in which the actuation commands <NUM> is sequentially written to memory locations in the buffer and, subsequent to data being written to an end memory location of the buffer, data at a starting memory location of the circular buffer is overwritten with new data. Each of the action buffers <NUM> may have a size and/or configuration defined based on the device <NUM> with which the action buffer <NUM> is associated. For instance, a first buffer of the action buffers <NUM> for storing data for operating a robotic arm may have a different size or configuration than a second buffer of the action buffers <NUM> for operating a conveyor belt.

The task manager <NUM> loads each data element or instance of the actuation commands 522a, 522b,. , 522N parsed from the joint action vector <NUM> into a corresponding buffer of the action buffers <NUM>. The task manager <NUM>, as an example, may parse and store a first element from the joint action vector <NUM> into the action buffer 524a, may parse and store a second element from the joint action vector <NUM> into the action buffer <NUM>, etc., until all elements of the joint action vector <NUM> are stored in a corresponding action buffer <NUM>. In some embodiments, the task manager <NUM> may store the actuation commands 522a, 522b,. , 522N in the actuator buffers 524a, 524b,. 524N sequentially in the order in which the actuation commands are parsed. In some embodiments, the task manager <NUM> may parse the entire joint action vector <NUM> and then store the actuation commands <NUM> obtained therefrom to the actuator buffers <NUM>. In some implementations, the joint action vector <NUM> may not include an action element for each of the devices <NUM> - for example, action elements may not be provided for inactive devices (e.g., cameras, measurement sensors) that do not move or interact on the environment <NUM>. In some embodiments, a null or zero value may be assigned for action elements corresponding to inactive devices in the environment <NUM>.

Thereafter, each of the device communicators 506a, 506b,. , 506N obtains the actuation commands 522a, 522b,. , 522N stored in a corresponding one of the actuator buffers 524a, 524b,. The communicators <NUM> then transmit the actuation commands <NUM> to the associated device <NUM>, thereby causing the associated device <NUM> to perform according to the actuation commands provided.

<FIG> shows a computational environment of the computer system <NUM> in which a communicator process <NUM> of the reinforcement learning architecture <NUM> operates to communicate with a device <NUM> according to one or more embodiments. The communicator process <NUM> is a multi-threaded process corresponding to the device communicator <NUM> of <FIG> and elsewhere herein. The communicator process <NUM> is assigned to the device <NUM> and dedicated to receiving sensory data from the device <NUM> and transmitting actuation commands to the device <NUM>. As described above, the reinforcement learning architecture <NUM> may include a communicator process <NUM> for each device <NUM> to be controlled by the reinforcement learning architecture <NUM> in the environment <NUM>.

The communicator process <NUM> executes independently of the reinforcement learning agent <NUM> and the task manager <NUM>. The communicator process <NUM> is a multi-threaded process executing in the computer system <NUM> that includes a read thread <NUM> and a write thread <NUM>. The read thread <NUM> and the write thread <NUM> may execute in different cycles of the communicator process <NUM>. For instance, operations may be performed by a processor of the computer system <NUM> according to instructions of the read thread <NUM> for a first set of cycles and operations may be performed by the processor of the computer system <NUM> for a second set of cycles different than the first set.

The read thread <NUM> is dedicated to reading and storing sensory data from the device <NUM>. In particular, while the reinforcement learning architecture <NUM> is running, the read thread <NUM> reads sensory data <NUM> transmitted by the device <NUM> over the network <NUM>, and stores the sensory data <NUM> in a sensor buffer <NUM>, which may be a circular buffer dedicated to the particular communicator process <NUM>. The sensory data <NUM> may be received as a data packet or a sequence of data packets. Storing the sensory data <NUM> in the circular buffer may cause a write pointer associated with the circular buffer to be updated. In some embodiments, the read thread <NUM> may evaluate whether the sensory data <NUM> provided by the device <NUM> is valid before storing the sensory data <NUM> in the sensor buffer <NUM>.

The write thread <NUM> is dedicated to obtaining actuator commands provided by the task manager <NUM> and transmitting the actuator commands to the device <NUM>. While the reinforcement learning architecture <NUM> is running, the write thread <NUM> reads an actuation command <NUM> from an actuator buffer <NUM> and causes the actuation command <NUM> to be transmitted to the device <NUM> over the network <NUM>. The actuation buffer <NUM> may be a circular buffer, and reading the actuation command <NUM> from the actuation buffer <NUM> may cause a read pointer associated with the circular buffer to be updated. The actuation buffer <NUM> may wait for the actuation buffer <NUM> to be updated before reading the actuation command <NUM> in some instances - e.g., if the write pointer of the actuation buffer <NUM> matches the read pointer, the write thread <NUM> may wait until the write pointer is updated before reading the actuation command <NUM> therefrom.

<FIG> shows a computational environment of the computer system <NUM> in which a reinforcement learning agent <NUM> of the reinforcement learning architecture <NUM> interacts with a task manager <NUM> to control operations of devices operating in the environment <NUM> according to a reinforcement learning policy π. The reinforcement learning agent <NUM> and the task manager <NUM> are respective embodiments of the reinforcement learning agent <NUM> and the task manager <NUM> described herein. The reinforcement learning agent <NUM> is a first process of the reinforcement learning architecture <NUM> running on a first processor of the computer system <NUM> and the task manager <NUM> is a second process of the reinforcement learning architecture <NUM> running on a second processor of the computer system <NUM>. The reinforcement learning agent <NUM> process is independent of the process of the task manager <NUM>.

The task manager <NUM> obtains a plurality of sensory data <NUM> from a plurality of sensor buffers <NUM> each associated with a corresponding communicator process. The task manager <NUM> may obtain the plurality of sensory data <NUM> collectively as a unit - for instance, by executing read operations to read the plurality of sensory data <NUM> from the sensor buffers <NUM> in parallel. The task manager <NUM> may track the status of the sensor buffers <NUM> and read sensory data from one or more of the sensor buffers <NUM> in response to detecting an update to a sensor buffer (e.g., change in a write pointer).

The task manager <NUM> generates a joint state vector <NUM> based on the plurality of sensory data <NUM> obtained from the plurality of sensor buffers <NUM>. Generating the joint state vector <NUM> may include generating observation data and may include generating reward data. The observation data may correspond to an observed state of the environment <NUM>, such as the relationship of the devices operating in the environment <NUM> to achieving a defined objective (e.g., picking up an object and placing it in a box). The reward data may correspond to a contribution of the preceding actions commanded by the reinforcement learning agent <NUM> in relation to achieving the defined objective - for example, an amount of progress made toward or away from the defined objective as a result of performance of previous action commands.

The task manager <NUM> stores the joint state vector <NUM> in a state buffer <NUM>. The state buffer <NUM> is a region allocated in the memory <NUM> for storing the joint state vector <NUM>. The size of the state buffer <NUM> may be determined based on characteristics of the sensory data <NUM> provided by the devices <NUM> operating in the environment <NUM>. The state buffer <NUM> is a shared region in the memory that is shared between the task manager <NUM> and the reinforcement learning agent <NUM>, and which is inaccessible to other processes executing in the computer system <NUM>. The task manager <NUM> may update the joint state vector <NUM> as a result of detecting an update or change in sensory data <NUM> stored in one or more of the sensor buffers <NUM>. The task manager <NUM> may calculate a new joint state vector <NUM> for each change detected or update a portion of the joint state vector <NUM>.

The reinforcement learning agent <NUM> obtains the joint state vector <NUM> from the state buffer <NUM> and invokes a policy π to determine a response based on the joint state vector <NUM>. The reinforcement learning agent <NUM> transitions between a suspended state in which the joint state vector <NUM> is not read from the state buffer <NUM> and processed, and an active state in which the reinforcement learning agent <NUM> obtains and processes the joint state vector <NUM>.

In the active state, the reinforcement learning agent <NUM> determines, according to the policy π, a set of actions to be performed by the devices 508a, 508b,. , 508N based on perceived states of the environment <NUM>, as represented by the joint state vector <NUM>. The reinforcement learning agent <NUM> operates in the suspended state for a step or period of time Ps, then obtains and processes a single joint state vector <NUM>, and then returns to the suspended state for the period of time Ps. The period of time Ps is a defined operating parameter of the reinforcement learning agent <NUM> and which may be selectively adjustable by a user.

In the active state, the reinforcement learning agent <NUM> generates a joint action vector <NUM> based on the joint state vector <NUM> and stores the joint action vector <NUM> in an action buffer <NUM>. The joint action vector <NUM> may be organized as an array of elements that sequentially correspond to actions to be performed by corresponding devices 508a, 508b,. , 508N, as described with respect to the joint state vector <NUM>. For instance, a first element of the joint action vector <NUM> may correspond to the device 508a, a second element of the joint action vector <NUM> may correspond to the device 508b, and so forth. The reinforcement learning agent <NUM> may transition back and forth between the suspended state and the active state while the reinforcement learning architecture <NUM> is running.

The joint action vector <NUM> includes actuation commands that cause corresponding devices <NUM> operating in the environment <NUM> to perform the actions specified therein. For instance, the joint action vector <NUM> may indicate a first action to be performed by the device 508a, a second action to be performed by the device 508b, and so on, when received by the devices. As a result of generating and storing the joint action vector <NUM>, the reinforcement learning agent <NUM> returns to the suspended state for the period of time Ps. The action buffer <NUM> is a region allocated in the memory <NUM> for storing the joint action vector <NUM>. The action buffer <NUM> is a shared region in the memory that is shared between the task manager <NUM> and the reinforcement learning agent <NUM>, and which is inaccessible to other processes executing in the computer system <NUM>.

The task manager <NUM> reads the joint action vector <NUM> from the action buffer <NUM>. The task manager <NUM> may monitor the action buffer <NUM> and wait until the action buffer <NUM> is populated or updated with the joint action vector <NUM> in response to storing the joint state vector <NUM> in the state buffer <NUM>. The task manager <NUM> parses the joint action vector <NUM> into a plurality of actuation commands <NUM> and distributes each of the actuation commands <NUM> parsed to an actuation buffer of a plurality of action buffers <NUM>. Each of the action buffers <NUM> is associated with a communicator, such as the communication process <NUM>.

<FIG> shows a method of operation of a device communicator of the reinforcement learning architecture <NUM>, such as the device communicators <NUM> or the communicator process <NUM>, according to one or more embodiments. As described elsewhere herein, the device communicator is a process running on a processor of the computer system <NUM> and is a process different than processes of the task manager and the reinforcement learning agent. Each device communicators may perform the method <NUM> independently of and in parallel with other device communicators of the reinforcement learning architecture <NUM>. The method <NUM> includes establishing <NUM> communications with device operating in the environment <NUM> over the network <NUM>. Establishing <NUM> communications may include establishing a communication channel according to one or more defined communication protocols. For instance, the device communicator may cause the communication interface <NUM> to establish a Wi-Fi connection with the corresponding device or "pair" with the corresponding device according to a Bluetooth protocol. In connection with establishing <NUM> communications with the device, the device communicator may send a command to the device causing it to transition to a data streaming mode in which the device streams sensory data, e.g., in a continuous sequence of data packets.

Thereafter, the method <NUM> includes obtaining <NUM> sensory data from the device indicating a measured state of device by a sensor associated with the device. In some implementations, the device communicator may invoke a read operation to obtain a data packet or a set of data packets of sensory data from the data stream. In some implementations, the device communicator may cause the communication interface <NUM> to send a request to the corresponding device to provide sensory data. The method <NUM> then includes loading <NUM> the sensory data obtained in <NUM> into a sensor buffer, as described with respect to <FIG>, <FIG>, and <FIG>, such as an element in a circular buffer. The device communicator may verify whether the sensory data is valid (e.g., not corrupt, readable) as a condition precedent to loading <NUM> the sensory data into a sensor buffer. Each of the device communicators is configured to independently load, in <NUM>, sensory data into a corresponding sensor buffer associated therewith. Obtaining <NUM> and loading <NUM> may comprise a cycle performed by the read thread <NUM> during which the write thread <NUM> is blocked or prevented from performing operations.

Then, the device communicator obtains <NUM> an actuation command from an actuator buffer. For instance, the write thread <NUM> of the device communicator may track the status of the actuator buffer associated with the device communicator. In response to detecting a change, the write thread <NUM> may read the actuation command from the actuator buffer. The write thread <NUM> may perform a read operation from the actuator buffer that a read pointer as a result of detecting that the read pointer of the actuator buffer does not match the write pointer. The actuation command obtained in <NUM> may be a command in response to the sensor data loaded in <NUM>. Responsive to successfully obtaining <NUM> an actuation command from the actuator buffer, the method includes transmitting <NUM> the actuation command to the device associated with the device communicator. In particular, the device communicator causes the communication interface <NUM> to transmit the actuation command over the network <NUM> to cause the device to perform the action specified in the actuation command in the environment <NUM>. Obtaining <NUM> and transmitting <NUM> may comprise a cycle performed by the write thread <NUM> during which the read thread <NUM> is blocked or prevented from performing operations.

The operations of the method <NUM> may be performed in a different order than shown in <FIG>. For instance, obtaining <NUM> and transmitting <NUM> may be performed before obtaining <NUM> and loading <NUM>. The method <NUM> may be an iterative process in which portions of the method are repeatedly performed. The method <NUM> may return to obtain <NUM> sensory data from a device operating in the environment <NUM> subsequent to transmitting <NUM> the action data.

<FIG> shows a method of operation of a task manager of the reinforcement learning architecture <NUM>, such as the task manager <NUM> or the task manager <NUM>, according to one or more embodiments. As described herein, the task manager is a process running on a processor of the computer system <NUM> and is a process that is different than the processes of the device communicator and the reinforcement learning agent. The method <NUM> includes obtaining <NUM> sensor data from the plurality of sensor buffers. For instance, the task manager may read a single instance of sensor data from each of the plurality of sensor buffers.

The method <NUM> includes generating <NUM> a joint state vector representative of the sensor data obtained in <NUM>. Generating <NUM> the joint state vector may include calculating observation data representative of a state of the environment <NUM> relative to a defined objective of the task manager and reward data representative of progress toward or away from the defined objective as a result of performing a preceding set of actions. In some previously-implemented solutions, a reinforcement learning agent would sequentially obtain sensory data and process the data to determine what actions to take, which can cause a disconnect between action and effect in the feedback loop of reinforcement learning due to time delays, thereby inhibiting the ability of the reinforcement learning agent to effectively and efficiently learn to achieve the defined objective.

Generating <NUM> the joint state vector by the task manager has numerous benefits over these previous approaches. For example, because the communicators operate independently and in parallel with each other, the sensory data stored is effectively correlated with a more precise period of time in contrast to the sequential collection of information in previous implementations. Additionally, generating <NUM> a joint state vector helps to distribute computational resources of the reinforcement learning agent into a separate process, thereby decreasing computational resources used by the reinforcement learning agent to determine a set of actions to be performed. This procedure also improves the learning progress of the reinforcement learning agent by reducing latency in the system and improving the temporal coherence of the sensor data.

The method <NUM> includes loading <NUM> the joint state vector into a state buffer of the reinforcement learning architecture <NUM>, such as the state buffer <NUM> or the state buffer <NUM>. The method <NUM> further includes obtaining <NUM> a joint action vector from an action buffer of the reinforcement learning architecture <NUM>, such as the action buffer <NUM> or the action buffer <NUM>. Obtaining <NUM> the joint action vector is in response to loading <NUM> the joint state vector. For instance, subsequent to loading <NUM>, the task manager waits until the action buffer is updated with a new joint action vector, at which point the task manager reads the joint action vector from the action buffer.

The method <NUM> further involves parsing <NUM> the joint action vector into actuation commands to be transmitted to the devices operating in the environment <NUM>. Parsing <NUM> the joint action vector may include checking boundaries and constraints of the environment <NUM> and evaluating a risk associated with execution of the actuation command by the corresponding device in the environment <NUM>. For example, the task manager may determine whether performance of the actuation command would result in a collision; exceed defined safety boundaries of the environment <NUM>; pose a risk to persons, animals, property, etc., in the environment <NUM>, etc. The boundaries and constraints of the environment <NUM> may be a defined set of rules or physical boundaries associated with the environment <NUM> that performance of the actuation command should not violate. If the risk associated with performance of the actuation command would violate or exceed a defined boundary or constraint, the task manager may elect not to send the actuation command(s). Evaluating the risk of performance of the actuation command is a non-trivial procedure that, when executed asynchronously by the task manager, facilitates reduction of further delays or events that could be adverse to achievement of the defined objective in an efficient manner and according to the reinforcement learning model.

The method <NUM> also includes loading <NUM> the actuation commands into the plurality of actuator buffers. The task manager, in some embodiments, may parse <NUM> each actuation command from the joint action vector and then load <NUM> the actuation commands into the actuator buffers. In some embodiments, the task manager may load <NUM> each actuation command into a corresponding actuation buffer after it is parsed <NUM> from the joint action vector and before parsing <NUM> the next actuation command of the joint action vector.

The operations of the method <NUM> may be performed in a different order than shown in <FIG>. For instance, obtaining <NUM>, parsing <NUM>, and loading <NUM> may be performed before obtaining <NUM>, generating <NUM>, and loading <NUM>. The method <NUM> may be an iterative process in which portions of the method are repeatedly performed. For example, the method <NUM> may return to obtain <NUM> sensory data from the plurality of sensor buffers subsequent to loading the actuation commands in <NUM>.

<FIG> shows a method <NUM> of operating a reinforcement learning agent of the reinforcement learning architecture <NUM>, such as the reinforcement learning agent <NUM> or the reinforcement learning agent <NUM>. As described herein, the reinforcement learning agent is a process running on a processor of the computer system <NUM> and is a process that is different than the processes of the device communicator and the task manager.

The method <NUM> includes obtaining <NUM> a joint state vector from a state buffer, which was loaded by the task manager in <NUM>. Obtaining <NUM> may be performed by the reinforcement learning agent after transitioning from the suspended state to the active state after a defined period of time Ps in the suspended state. Then, the method <NUM> involves processing <NUM> observation information and/or reward information included in the joint state vector obtained in <NUM>. The method <NUM> further includes determining <NUM> a set of actions to be performed by the devices operating in the environment <NUM> based on a policy π of the reinforcement learning agent. For instance, the reinforcement learning agent may execute a function corresponding to the policy π using the observation information as an input thereto. An output of the policy π may be the joint action vector that is representative of the actions to be performed in the environment <NUM> to achieve a next state. The reinforcement learning agent generates <NUM> a joint action vector based on the set of actions determined in <NUM> and loads <NUM> the joint action vector into the action buffer. The reinforcement learning agent may be unaware of the specific devices operating in the environment <NUM> and so the joint action vector generated in <NUM> may indicate a state change of the environment <NUM> to be performed to progress the state of the environment <NUM> closer to the defined objective.

The method <NUM> includes updating <NUM> a learning model or planning model of the reinforcement learning agent based on the joint state vector and a previous joint action vector generated that caused a state change in the environment <NUM> corresponding to the joint state vector. Updating the learning model may include updating the policy π based on the reward information and value information optionally associated therewith. Updating the policy π is part of the reinforcement learning process by which the reinforcement learning agent improves its efficiency and effectiveness to perform the defined objective. Updating <NUM> may include updating weights or biases of the policy π, a value function, a Q-value function, or other functions of the reinforcement learning agent.

The method <NUM> includes transitioning <NUM> the reinforcement learning agent from the active state into the suspended state for a defined period of time Ps. The defined period of time Ps may be a user defined amount of time, such as <NUM> milliseconds, during which the reinforcement learning agent does not obtain <NUM> or process <NUM> a new joint state vector from the state buffer. The reinforcement learning agent performs certain operations while in the suspended state. For instance, updating <NUM> is performed when the reinforcement learning agent is in the suspended state for the period of time Ps. As a result of loading <NUM> the joint action vector into the action buffer, the reinforcement learning agent transitions <NUM> to the suspended state before or without updating <NUM> the learning or planning model. Updating the learning or planning model <NUM> during the suspended state improves the efficiency of the reinforcement learning agent. The period of time Ps should be selected, in such instances, to provide sufficient time for the reinforcement learning agent to complete the updating <NUM>.

The method <NUM> is an iterative process which then returns to obtain <NUM> the joint state vector from the state buffer subsequent to expiration of the period of time Ps.

Claim 1:
A method (<NUM>, <NUM>, <NUM>), comprising:
establishing (<NUM>), via one or more communication interfaces, communications in real-time between a computer system (<NUM>) and a system comprising a plurality of devices operating in a real-world environment (<NUM>);
obtaining (<NUM>), by a set of first processes executing on the computer system, a plurality of state data respectively indicating states of the plurality of devices;
loading (<NUM>), by the set of first processes, individual state data of the plurality of state data into a corresponding buffer of a plurality of first buffers;
generating (<NUM>), by a second process executing on the computer system, a joint state vector based on the plurality of state data stored in the plurality of first buffers;
transitioning, by a third process executing on the computer system, from a suspended state, in which the third process operates for a predefined period of time during which the third process does not obtain or process a new joint state vector from a second buffer, to an active state;
generating (<NUM>), by the third process during the active state, a joint action vector based on the joint state vector according to a defined policy of the third process, wherein the defined policy is a set of instructions stored in a memory (<NUM>) that causes the third process to generate an action in response to observation data and/or reward data in the joint state vector (<NUM>) and is defined by a user based on tasks, goals or desired end states to be achieved in the real-world environment (<NUM>);
transitioning (<NUM>), by the third process in response to generating the joint action vector, back to the suspended state;
updating, by the third process, the defined policy based on the joint state vector during a defined period of time in which the third process is in the suspended state;
parsing (<NUM>), by the second process, the joint action vector into a plurality of actuation commands respectively indicating operations for each of the plurality of devices to perform; and
causing, by each of the set of first processes, the one or more communication interfaces to transmit (<NUM>) respective actuation commands of the plurality of actuation commands to a corresponding device of the plurality of devices.