Patent Publication Number: US-11657266-B2

Title: Cooperative multi-goal, multi-agent, multi-stage reinforcement learning

Description:
BACKGROUND 
     Efficient exploration in reinforcement learning may be difficult in a multi-agent setting, where the state space and joint action space scale exponentially with the number of agents. Relying on random exploration to learn both individual task completion and cooperative behavior concurrently may be highly inefficient. Agents who have not yet learned to accomplish local goals may rarely encounter the region of state space where cooperation is needed, rendering any exploration actions useless for learning cooperative behavior. On the other extreme, exploratory actions taken in situations that require coordination may lead to failure, and the resulting penalties may cause agents to avoid the coordination problem altogether and fail to learn their individual tasks. 
     BRIEF DESCRIPTION 
     According to one aspect, a method for cooperative multi-goal, multi-agent, multi-stage (CM3) reinforcement learning may include training a first agent based on a first policy gradient and training a first critic based on a first loss function to learn one or more features in a single-agent environment using a Markov decision process. The first agent may be associated with a first agent neural network and the first critic may be associated with a first critic neural network. The method may include training a number of N agents based on the first policy gradient and a second policy gradient and training a second critic based on the first loss function and a second loss function to learn cooperation between the N agents in a multi-agent environment using a Markov game to instantiate a second agent neural network. Each of the N agents may be instantiated with the first agent neural network in a pre-trained fashion. The method may include generating a cooperative multi-goal, multi-agent, multi-stage network policy based on the first agent neural network and the second agent neural network. 
     The first critic may be a decentralized critic and the second critic may be a centralized critic. Training of the first agent in the single-agent environment may occur prior to training the N agents in the multi-agent environment. The method may include training the number of N agents based on a combined policy gradient derived from the first policy gradient and the second policy gradient. 
     The second agent neural network may be associated with an o others  parameter for each of the N agents indicative of a local observation of each of the corresponding N agents. The o others  parameter may be indicative of a velocity of the first agent, a number of lanes or sub-lanes between the first agent and one of the N agents, a distance from the first agent to a goal position, or a vehicle type associated with the first agent. The o others  parameter may be indicative of a vehicle occupancy status associated with one of the N agents, a relative velocity of one of the N agents relative to the first agent, or a vehicle type associated with one of the N agents. The number of N agents may include the first agent. Training of the first agent and training of the number of N agents may include generating one or more actions including a no-operation action, an acceleration action, a deceleration action, a shift left one sub-lane action, and a shift right one sub-lane action. 
     According to one aspect, a system for cooperative multi-goal, multi-agent, multi-stage (CM3) reinforcement learning may include a processor, a memory, and a simulator implemented via the processor and memory. The simulator may perform training a first agent based on a first policy gradient and training a first critic based on a first loss function to learn one or more features in a single-agent environment using a Markov decision process. The first agent may be associated with a first agent neural network and the first critic may be associated with a first critic neural network. The simulator may perform training a number of N agents based on the first policy gradient and a second policy gradient and training a second critic based on the first loss function and a second loss function to learn cooperation between the N agents in a multi-agent environment using a Markov game to instantiate a second agent neural network. Each of the N agents may be instantiated with the first agent neural network in a pre-trained fashion. The simulator may perform generating a cooperative multi-goal, multi-agent, multi-stage network policy based on the first agent neural network and the second agent neural network. 
     The first critic may be a decentralized critic and the second critic may be a centralized critic. The simulator may train the first agent in the single-agent environment prior to training the N agents in the multi-agent environment. The second agent neural network may be associated with an o others  parameter for each of the N agents indicative of a local observation of each of the corresponding N agents. 
     According to one aspect, a cooperative multi-goal, multi-agent, multi-stage (CM3) reinforcement learning based autonomous vehicle may be a vehicle including a storage device and a controller. The storage device may store a cooperative multi-goal, multi-agent, multi-stage network policy. The controller may operate the autonomous vehicle in an autonomous fashion based on the cooperative multi-goal, multi-agent, multi-stage network policy. The cooperative multi-goal, multi-agent, multi-stage network policy may be generated based on a first agent neural network and a second agent neural network. A first agent may be trained based on a first policy gradient and a first critic trained based on a first loss function to learn one or more features in a single-agent environment using a Markov decision process. The first agent may be associated with the first agent neural network and the first critic may be associated with a first critic neural network. A number of N agents may be trained based on the first policy gradient and a second policy gradient and a second critic trained based on the first loss function and a second loss function to learn cooperation between the N agents in a multi-agent environment using a Markov game to instantiate the second agent neural network Each of the N agents may be instantiated with the first agent neural network in a pre-trained fashion. 
     The second agent neural network may be associated with an o others  parameter for each of the N agents indicative of a local observation of each of the corresponding N agents. The o others  parameter may be indicative of a velocity of the first agent, a number of lanes or sub-lanes between the first agent and one of the N agents, a distance from the first agent to a goal position, or a vehicle type associated with the first agent. The o others  parameter may be indicative of a vehicle occupancy status associated with one of the N agents, a relative velocity of one of the N agents relative to the first agent, or a vehicle type associated with one of the N agents. The number of N agents may include the first agent. Training the first agent and training the number of N agents may include generating one or more actions including a no-operation action, an acceleration action, a deceleration action, a shift left one sub-lane action, and a shift right one sub-lane action. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    is an exemplary component diagram of a system for cooperative multi-goal, multi-agent, multi-stage (CM3) reinforcement learning, according to one aspect. 
         FIG.  2    is an exemplary component diagram of a simulator for the system for cooperative multi-goal, multi-agent, multi-stage reinforcement learning of  FIG.  1   . 
         FIG.  3    is an exemplary flow diagram of a method for cooperative multi-goal, multi-agent, multi-stage reinforcement learning, according to one aspect. 
         FIG.  4    is an exemplary data flow diagram in association with a method for cooperative multi-goal, multi-agent, multi-stage reinforcement learning, according to one aspect. 
         FIGS.  5 A- 5 B  is an exemplary flow diagram of a method for cooperative multi-goal, multi-agent, multi-stage reinforcement learning, according to one aspect. 
         FIG.  6    is an exemplary simulation scenario in association with cooperative multi-goal, multi-agent, multi-stage reinforcement learning, according to one aspect. 
         FIGS.  7 - 8    are exemplary performance results in association with cooperative multi-goal, multi-agent, multi-stage reinforcement learning, according to one aspect. 
         FIG.  9    is an exemplary simulation scenario in association with cooperative multi-goal, multi-agent, multi-stage reinforcement learning, according to one aspect. 
         FIG.  10    is an exemplary reward versus episode graph in association with cooperative multi-goal, multi-agent, multi-stage reinforcement learning, according to one aspect. 
         FIG.  11    is an illustration of an example computer-readable medium or computer-readable device including processor-executable instructions configured to embody one or more of the provisions set forth herein, according to one aspect. 
         FIG.  12    is an illustration of an example computing environment where one or more of the provisions set forth herein are implemented, according to one aspect. 
     
    
    
     DETAILED DESCRIPTION 
     The following includes definitions of selected terms employed herein. The definitions include various examples and/or forms of components that fall within the scope of a term and that may be used for implementation. The examples are not intended to be limiting. Further, one having ordinary skill in the art will appreciate that the components discussed herein, may be combined, omitted or organized with other components or organized into different architectures. 
     A “processor”, as used herein, processes signals and performs general computing and arithmetic functions. Signals processed by the processor may include digital signals, data signals, computer instructions, processor instructions, messages, a bit, a bit stream, or other means that may be received, transmitted, and/or detected. Generally, the processor may be a variety of various processors including multiple single and multicore processors and co-processors and other multiple single and multicore processor and co-processor architectures. The processor may include various modules to execute various functions. 
     A “memory”, as used herein, may include volatile memory and/or non-volatile memory. Non-volatile memory may include, for example, ROM (read only memory), PROM (programmable read only memory), EPROM (erasable PROM), and EEPROM (electrically erasable PROM). Volatile memory may include, for example, RAM (random access memory), synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), and direct RAM bus RAM (DRRAM). The memory may store an operating system that controls or allocates resources of a computing device. 
     A “disk” or “drive”, as used herein, may be a magnetic disk drive, a solid state disk drive, a floppy disk drive, a tape drive, a Zip drive, a flash memory card, and/or a memory stick. Furthermore, the disk may be a CD-ROM (compact disk ROM), a CD recordable drive (CD-R drive), a CD rewritable drive (CD-RW drive), and/or a digital video ROM drive (DVD-ROM). The disk may store an operating system that controls or allocates resources of a computing device. 
     A “bus”, as used herein, refers to an interconnected architecture that is operably connected to other computer components inside a computer or between computers. The bus may transfer data between the computer components. The bus may be a memory bus, a memory controller, a peripheral bus, an external bus, a crossbar switch, and/or a local bus, among others. The bus may also be a vehicle bus that interconnects components inside a vehicle using protocols such as Media Oriented Systems Transport (MOST), Controller Area network (CAN), Local Interconnect Network (LIN), among others. 
     A “database”, as used herein, may refer to a table, a set of tables, and a set of data stores (e.g., disks) and/or methods for accessing and/or manipulating those data stores. 
     An “operable connection”, or a connection by which entities are “operably connected”, is one in which signals, physical communications, and/or logical communications may be sent and/or received. An operable connection may include a wireless interface, a physical interface, a data interface, and/or an electrical interface. 
     A “computer communication”, as used herein, refers to a communication between two or more computing devices (e.g., computer, personal digital assistant, cellular telephone, network device) and may be, for example, a network transfer, a file transfer, an applet transfer, an email, a hypertext transfer protocol (HTTP) transfer, and so on. A computer communication may occur across, for example, a wireless system (e.g., IEEE 802.11), an Ethernet system (e.g., IEEE 802.3), a token ring system (e.g., IEEE 802.5), a local area network (LAN), a wide area network (WAN), a point-to-point system, a circuit switching system, a packet switching system, among others. 
     A “vehicle”, as used herein, may refer to any moving vehicle that is capable of carrying one or more human occupants and is powered by any form of energy. The term “vehicle” includes cars, trucks, vans, minivans, SUVs, motorcycles, scooters, boats, personal watercraft, and aircraft. In some scenarios, a motor vehicle includes one or more engines. Further, the term “vehicle” may refer to an electric vehicle (EV) that is powered entirely or partially by one or more electric motors powered by an electric battery. The EV may include battery electric vehicles (BEV) and plug-in hybrid electric vehicles (PHEV). Additionally, the term “vehicle” may refer to an autonomous vehicle and/or self-driving vehicle powered by any form of energy. The autonomous vehicle may or may not carry one or more human occupants. 
     An “agent”, as used herein, may refer to a “vehicle”, such as a vehicle within a simulation or a simulated vehicle. Similarly, “actor” as used herein, may be used interchangeably with “agent”. Additionally, “setting” as used herein, may be used interchangeably with “environment”. A, “feature” as used herein, may include a goal. 
     A “vehicle system”, as used herein, may be any automatic or manual systems that may be used to enhance the vehicle, driving, and/or safety. Exemplary vehicle systems include an autonomous driving system, an electronic stability control system, an anti-lock brake system, a brake assist system, an automatic brake prefill system, a low speed follow system, a cruise control system, a collision warning system, a collision mitigation braking system, an auto cruise control system, a lane departure warning system, a blind spot indicator system, a lane keep assist system, a navigation system, a transmission system, brake pedal systems, an electronic power steering system, visual devices (e.g., camera systems, proximity sensor systems), a climate control system, an electronic pretensioning system, a monitoring system, a passenger detection system, a vehicle suspension system, a vehicle seat configuration system, a vehicle cabin lighting system, an audio system, a sensory system, among others. 
     The aspects discussed herein may be described and implemented in the context of non-transitory computer-readable storage medium storing computer-executable instructions. Non-transitory computer-readable storage media include computer storage media and communication media. For example, flash memory drives, digital versatile discs (DVDs), compact discs (CDs), floppy disks, and tape cassettes. Non-transitory computer-readable storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, modules, or other data. 
     As used herein, “CM3” may refer to the use of a method for cooperative multi-goal, multi-agent, multi-stage reinforcement learning or a system for cooperative multi-goal, multi-agent, multi-stage reinforcement learning, including but not limited to the system of  FIG.  1   , the method of  FIG.  3   , the method of  FIGS.  5 A- 5 B , etc. 
       FIG.  1    is an exemplary component diagram of a system  100  for cooperative multi-goal, multi-agent, multi-stage (CM3) reinforcement learning, according to one aspect. The system  100  for CM3 reinforcement learning may include a processor  102 , a memory  104 , a bus  106 , and a simulator  108 . The simulator  108  may be implemented via the processor  102  and the memory  104 . The simulator  108  may simulate or perform simulation associated with one or more agents  122 , taking one or more actions  124 , within a simulation environment  126 , where one or more critics  128  interpret or evaluate one or more of the actions  124  taken by one or more of the agents  122  to determine one or more rewards  132  and one or more states  134  resulting from the actions taken. 
     The simulator  108  or the processor  102  may generate a CM3 policy network  140 , which may be stored on the memory  104  of the system  100  for CM3 reinforcement learning. The system may further include a communication interface  150  which enables the CM3 policy network  140  to be transmitted to other devices, such as a server  160 , which may include a CM3 database  162 . In this way, the CM3 policy network  140  generated by the system  100  for CM3 reinforcement learning may be stored on the CM3 database  162  of the server  160 . 
     The server may then propagate the CM3 policy network  140  to one or more vehicles, such as a first vehicle  170  and a second vehicle  180 . The first vehicle may be equipped with a vehicle communication interface  172 , a storage device  174 , a controller  176 , and one or more vehicle systems  178 . The storage device may store the CM3 policy network  140  from the server, and the controller may operate the first vehicle in an autonomous fashion based on the CM3 policy network  140 . Similarly, the second vehicle  180  may be equipped with a vehicle communication interface  182 , a storage device  184 , a controller  186 , and one or more vehicle systems  188 , and may operate similarly to the first vehicle. 
     In any event, when the CM3 policy network  140  is stored on the storage device of the vehicle, this enables the controller to autonomously drive the vehicle around based on the CM3 policy network  140 , and to make autonomous driving decisions according to the CM3 reinforcement learning which occurred within the simulator  108  because the CM3 policy network  140  may be indicative of one or more of the policies or decisions which should be made based on the training or the simulation. For example, the CM3 network policy may receive an input of an observation associated with the first autonomous vehicle or the second autonomous vehicle (e.g., a vehicle state or an environment state) and output a suggested action. 
       FIG.  2    is an exemplary component diagram of the simulator  108  for the system  100  for cooperative multi-goal, multi-agent, multi-stage reinforcement learning of  FIG.  1   . In  FIG.  2   , the simulator  108  of the system  100  for CM3 reinforcement learning of  FIG.  1    may be seen. Here, the agent may take the action in the environment. This may be interpreted, by the critic, as the reward or penalty and a representation of the state, which may be then fed back into the agent. The agent may interact with the environment by taking the action at a discrete time step. At each time step, the agent may receive an observation which may include the reward. The agent may select one action from a set of available actions, which results in a new state and a new reward for a subsequent time step. The goal of the agent is generally to collect the greatest amount of rewards possible. 
     Stage One/Local View/Single-Agent Environment 
     According to one aspect, the simulator  108  may perform stage one training including training a first agent based on a first policy gradient and a training first critic based on a first loss function to learn one or more features in a single-agent environment using a Markov decision process (MDP). During the first stage or stage one, the simulator  108  may train the agent to achieve one or more individual tasks or to achieve different goals in a single-agent setting or environment (i.e., where there are no other agents within the environment). This may be a static environment. The single-agent setting or environment provided by the simulator  108  in stage one enables the first agent to learn or be trained on the one or more features in an expedient fashion. In other words, by reducing the number of agents in the first stage, the first agent may be trained in a faster, more efficient manner. Explained yet another way, agents that achieve local objectives in the absence of other agents may more reliably produce the desired state configurations for learning cooperative behavior. 
     The simulator  108  may train the first agent by enabling the first agent to select an action from a set of one or more actions including a no-operation action, an acceleration action, a deceleration action, a shift left one sub-lane action, and a shift right one sub-lane action. 
     Examples of one or more of the features which may be learned in stage one by the first agent may include learning rules and regulations of the roadway, how to drive on a roadway within the speed limit or maintain speed, how to maintain the vehicle within the lane, how to change lanes or change sub-lane positions, how to exit on an exit ramp, how to enter an on-ramp, etc. 
     In this way, the first policy network has a decentralized critic which may learn different features in the single-agent setting or environment. Because the first critic may be the decentralized critic, the first stage or stage one may be associated with a local view or individualized rewards, individualized learning, etc. for the first agent. 
     Stage Two/Global View/Multi-Agent Environment 
     According to one aspect, the simulator  108  may perform stage two training including training a number of N agents based on the first policy gradient and a second policy gradient and training a second critic based on the first loss function and a second loss function to learn a second set of features, such as cooperation, between the N agents in a multi-agent environment using a Markov game to instantiate a second agent neural network. During the stage two training, each one of the N agents may be instantiated with the pre-trained first agent neural network. In this way, by instantiating each one of the N agents with prior knowledge from the pre-trained first agent neural network, curriculum gain may be implemented via the system  100  for CM3 reinforcement learning. In this regard, the simulator  108  may train the first agent in the single-agent environment may occur prior to training the N agents in the multi-agent environment. The simulator  108  may train the number of N agents based on a combined policy gradient derived from the first policy gradient and the second policy gradient. The number of N agents may include the first agent. 
     For the Markov game, each agent may thus learn to accomplish one or more goals or features within a finite set, cooperate with other agents for collective success of goals between or among the N number of agents, and act independently with limited local information during the simulation. The simulator  108  may formulate the environments (e.g., the single-agent and the multi-agent environments) as episodic environments, where each agent may be assigned randomly-sampled goals during each training episode. In any event, the simulator  108  may setup the environments as a multi-goal Markov game, and review the actor-critic approach to centralized training of decentralized policies. 
     With respect to the Markov game, the simulator  108  may define a multi-goal Markov game as a tuple  S, {O n }, {A n }, P, R, G, N, γ , with N agents labeled by n=[1 . . . N]. Each agent may have one goal g n  ϵ G during each episode. At each time step t, the configuration of agents may be completely specified by a state s t  ϵ S, while each agent n receives a partial observation o t   n  ϵ O n  and chooses an action a t   n  ϵ A n . The simulator  108  may set the environment to a next state s t+1  due to the joint action a t :={a t   1  . . . a t   N }, according to a transition probability P(S t+1 |s t , a): S×A 1 × . . . ×A N ×S→[0, 1]. Each agent may receive a reward R: S×A n ×G→  and the learning task is to find stochastic policies π n (a n |o n , g n ): O n ×G×A n →[0,1], which condition only on local observations and goals, to maximize    s     t˜p     π,a     t˜π   [Σ t=0   T γ t Σ n=1   N R(s t , a t   n , g n )] over horizon T, where is y a discount factor. As used herein, a −n  and g −n  may denote the collection of all agents&#39; actions and goals, respectively, except that of agent n; g may denote the collection of all agents&#39; goals; and (π(a|o, g):=Π n π n (a n |o n , g n ) may denote the joint policy.    π [.] may represent for    s˜p     π     ,a˜π[.] , where ρ π  is defined as the discounted stationary state distribution under π. 
     In stage two, the simulator  108  may incorporate the learnings of other undertrained individuals or agents by observing the number of N agents using a centralized critic. In other words, the second critic may be the centralized critic. The simulator  108  may command the centralized critic to provide local and/or joint rewards. In this way, centralized learning centralized critic of decentralized policies may be provided. Because the second critic may be the centralized critic, the second stage or stage two may be associated with a global view or cooperation type learning for all of the N agents. The global view may be indicative of a flow of traffic for the N number of agents as a whole, while the local view may be indicative of movement, flow, or efficiency of merely the first agent. The global view and/or the local view may be taken from a birds-eye-view or perspective of the environment. Two views of the learning objective, corresponding to agents acting toward their own goal and corresponding to cooperation for the success of other agents may thus be provided by the two stages of simulation by the simulator  108 . These two stages may be incorporated into a two-stage (e.g., or additional stages, such as three, four, etc.) curriculum bridged by neural network construction. 
     Explained yet again, in stage two, cooperation is learned by the number of N agents during simulation to achieve different features in a multi-agent setting or environment based on a double critic configuration. For example, the first agent may be trained in the first stage and the first, decentralized critic may facilitate learning of multiple features in the single-agent setting or environment. In the second stage, or stage two, the number of N agents may be instantiated into the Markov game, and equipped with the pre-trained π 1  (e.g., the previously trained parameters from the stage one first agent) and Vπ (e.g., the previously trained parameters from the stage one first, decentralized critic) and the new centralized critic Q (e.g., the second critic). 
     During the second stage or stage two, the simulator  108  may train the N number of agents to achieve one or more cooperative tasks or to achieve different goals in the multi-agent setting or environment. This may be a dynamic environment. The multi-agent setting or environment provided by the simulator  108  in stage two may enable the N agents to learn to interact with one another, while enabling each one of the N agents to ultimately achieve their goals of reaching their target locations, respectively. 
     The simulator  108  may train the N number of agents by enabling each one of the N agents to select an action from a set of one or more actions including a no-operation action, an acceleration action, a deceleration action, a shift left one sub-lane action, and a shift right one sub-lane action, similarly to the first agent. 
     Examples of one or more of the features which may be learned in stage two by the N number of agents may include learning to avoid collisions with other agents, optimizing times to target locations. However, because the learning which occurred in stage one (e.g., learning how to drive in the static environment) is carried over, or instantiated into the simulation of stage two, learning may be achieved at a faster rate than if no division of stage one and stage two were provided by the simulator  108 . According to one aspect, the one or more features learned in the single-agent environment of stage one may be different than the features learned in the multi-agent environment of stage two. In this way, the simulator  108  may instantiate the features learned from the first stage to the N agents of the second stage to facilitate curriculum learning. 
     As previously discussed, in stage two, the simulator  108  may incorporate the learnings of other undertrained individuals or agents by observing the number of N agents using the centralized critic. Because each of the N agents may be instantiated with the teachings (e.g., network policies) from the first stage or stage one, each one of the N agents may be pre-trained on how to drive around in the static environment. Further, because there may be the number of N agents, each one of the N agents may be placed in different locations within the multi-agent environment, and be exposed to different scenarios. By having the simulator  108  and critic observe the number of N agents, learning for different scenarios may occur in parallel. Stated another way, the learning which may occur in stage two may be achieved by sharing data learned by a first agent with a second agent and vice versa (e.g., sharing data learned by the second agent with the first agent). 
     For example, if a first agent is trying to make a left lane change and is positioned in front of a second agent trying to make a right lane change, each one of the agents is being exposed to a different scenario, and network policies may be updated for both, thereby enhancing the efficiency of the learning, greatly reducing the amount of training time associated with training a CM3 system. In this example, the first agent and the second agent have different goals (e.g., a first goal of performing the left lane change and a second goal of performing the right lane change, perhaps to gain access to an exit ramp). It should be noted that cooperation learning may be implemented by the simulator  108 . For example, if the first agent refused to allow the second agent to perform the right lane change and if the second agent refused to allow the first agent to perform the left lane change, a stalemate scenario may occur, benefitting neither agent. 
     The second agent neural network may be associated with an o others  parameter for each of the N agents indicative of a local observation of each of the corresponding N agents. According to one aspect, the o others  parameter may be indicative of a velocity of the first agent, a number of lanes or sub-lanes between the first agent and one of the N agents, a distance from the first agent to a goal position, or a vehicle type associated with the first agent. The o others  parameter may be indicative of a vehicle occupancy status associated with one of the N agents, a relative velocity of one of the N agents relative to the first agent, or a vehicle type associated with one of the N agents. 
     Although, a third stage or stage three is not described herein, it is contemplated that in the third stage, each one of the N agents may be instantiated, by the simulator  108 , with the pre-trained first agent neural network and the pre-trained second agent neural network and additional learning may occur, thereby enabling curriculum learning to be provided. 
     Policy Generation/Centralized Learning of Decentralized Policies 
     The processor  102  or the simulator  108  may generate a CM3 network policy based on the first agent neural network and the second agent neural network. The simulator  108  may bridge the two stages (e.g., stage one and stage two) by modular augmentation of neural network policy and value functions. The CM3 network policy may be indicative of data which may be utilized to direct the controller of the autonomous vehicle(s) of  FIG.  1    to operation in an autonomous fashion. For example, the CM3 network policy may receive an input of an observation associated with the first autonomous vehicle or the second autonomous vehicle (e.g., a vehicle state or an environment state) and output a suggested action, which may include the no-operation action, the acceleration action, the deceleration action, the shift left one sub-lane action, and the shift right one sub-lane action, similarly to the actions used during simulation and provided by the simulator  108 . 
     The CM3 network policy may be indicative of a combined view which is representative of the local view and the global view associated with the first stage and the second stage. According to one aspect, the simulator  108  may optimize the CM3 network policy based on the local view and the global view. In this way, a two-stage curriculum, including a single-agent stage for learning to accomplish individual tasks, followed by a multi-agent stage for learning to cooperate in the presence of other agents may be provided. The simulator  108  may bridge these two stages (e.g., the first stage and the second stage) by modular augmentation of neural network policy and value functions from the first stage and the second stage, respectively. In other words, the simulator  108  may modularly augment the neural network policy associated with the first stage or stage one with the neural network policy associated with the second stage or stage two. Similarly, the simulator  108  may modularly augment one or more value functions associated with the first stage or stage one with one or more value functions associated with the second stage or stage two. 
     As previously discussed, CM3 may implement an actor-critic or agent-critic framework to this curriculum by formulating local and global views of the policy gradient and learning, via a double critic, including a decentralized value function (e.g., associated with the first, decentralized critic) and a centralized action-value function (e.g., associated with the second, centralized critic). The simulator  108  may thus simulate both a single-agent setting and a multi-agent setting where each agent learns to accomplish goals or features within a finite set and cooperate with other agents with potentially different goals or features. In this regard, the CM3 framework of the method for cooperative multi-goal, multi-agent, multi-stage reinforcement learning or the system  100  for cooperative multi-goal, multi-agent, multi-stage reinforcement learning enables a multi-agent curriculum to provide one or more benefits or advantages. 
     For example, multi-agent exploration has previously been a difficult problem to tackle. By training a stage one network policy to achieve different goals in a single-agent setting, exploratory actions useful for finding cooperative solutions may be achieved after agents are setup to reliably generate conflict by acting toward individual goals learned in the single-agent setting. 
     As another example, a neural network construction for multi-agent reinforcement learning (MARL) may be provided. Specifically, stage one training for the agent or the actor and critic neural networks may be simplified by setting the associated input space to a part (e.g., this may be pre-defined, pre-programmed, set manually, etc. to a first set of goals or features) that is sufficient for achieving individual goals in the single-agent environment. In the stage two neural network construction, the architecture may be augmented, adjusted, or modified for further learning in the full multi-agent environment or based on the stage one training (e.g., stage one variables, parameters, neural networks, etc.). This enables decomposition of an agent&#39;s observation into a representation of the agent&#39;s own state (e.g., distance to a target location), and a representation of other agents. 
     Additionally, stage one and stage two are combined. In other words, a decentralized policy is trained using a double critic, including a decentralized value function for learning local objectives and a centralized action-value function for learning cooperation, thereby enabling local objectives or goals to be considered while also considering cooperation between agents by showing two equivalent views of the policy gradient and implementing the new actor-critic or agent-critic adaptation. 
     The CM3 framework (e.g., associated with the CM3 system and CM3 method) combines these features within a single curriculum including a simplified policy network with a decentralized critic to learn to achieve multiple goals in stage one, while stage two augments the policy&#39;s observation space to represent other agents and learns multi-agent cooperation using the double critic implementation. According to one aspect, parameter-sharing may be provided among one or more to all of the agents by the simulator  108 . 
     A centralized critic receiving full state-action information may speed up learning of decentralized actors or agents (e.g., policies) that receive only local information, and only the actors may be retained for execution after training. According to one aspect, in a single-agent setting, policy π (with parameter θ may maximize the objective J(π):=   n Σ t γ t R(s t )] by ascending the gradient:
 
∇ θ   J (π)=   π [∇ θ log π( a|s )( Q   π ( s,a )− b ( s ))]  (1)
 
     In equation (1), Q π (s, a):=   s     t˜p     π     ,at˜π   [Σ t γ t R(s t , a t )|s o =s, a o =a)] is the action-value function, and b(s) is any state-dependent baseline. 
     According to one aspect, a counterfactual baseline may be defined as:
 
 b ( s,a   −n ):=Σ â     n   π n ( â   n   |o   n ) Q ( s ,( a   −n   , â   n ))  (2)
 
     The counterfactual baseline may address the issue of multi-agent credit assignment: Q(s, (a −n , a n ))−b(s, a −n ) may represent the contribution of an agent&#39;s chosen action and versus the average of all possible counterfactual actions â n , keeping other agents&#39; actions fixed. The simulator  108  may employ parameter-sharing among all agents, meaning that all agents execute the same policy but may behave differently according to their individual observations. 
     The policy gradient may be defined as:
 
∇ θ   J (π)=   π [Σ n ∇ 0  log π n ( a   n   |o   n )( Q ( s,a )− b ( s,a   −n ]  (3)
 
     The CM3 framework described herein may be based on the counterfactual baseline (e.g., equation (3)) that accounts for individual objectives and mixtures of local and joint rewards. 
       FIG.  3    is an exemplary flow diagram of a method  300  for cooperative multi-goal, multi-agent, multi-stage reinforcement learning, according to one aspect. The method  300  may include training  302  a first agent and a first critic to learn features in a single-agent environment, training  304  a number of N agents to learn cooperation between agents in a multi-agent environment, instantiating N agents with a pre-trained first agent neural network, and generating  306  a cooperative multi-goal, multi-agent, multi-stage (CM3) network policy based on the first agent neural network and the second agent neural network. Therefore, the acts of training  302  and training  304  including an incremental, multi-stage learning process to build the network policy of  306 . 
     According to one aspect, the training  302  of the first agent in the single-agent environment may include training the first agent based on a first policy gradient and training a first critic based on a first loss function to learn one or more features in the single-agent environment using a Markov decision process. The first agent may be associated with a first agent neural network and the first critic may be associated with a first critic neural network. The first critic may be a decentralized critic. 
     According to one aspect, the training  304  of the N number of agents in the multi-agent environment may include training the number of N agents based on the first policy gradient and a second policy gradient and a second critic trained based on the first loss function and a second loss function to learn cooperation between the N agents in a multi-agent environment using a Markov game to instantiate a second agent neural network. Each of the N agents may be instantiated with the first agent neural network in a pre-trained fashion. The second critic may be a centralized critic. In this way, a double critic method may be provided. Further, training the first agent in the single-agent environment may occur prior to training the N agents in the multi-agent environment, thereby enhancing the efficiency of the CM3 reinforcement learning, as will be described herein. 
       FIG.  4    is an exemplary data flow diagram in association with a method for cooperative multi-goal, multi-agent, multi-stage reinforcement learning, according to one aspect. In stage one of  FIG.  4   , a pair of reduced networks V 1  and π 1  may learn to achieve multiple goals in a single-agent environment, using policy gradient ∇J local . A new policy network  7  may be constructed from the trained π 1 , a new module π 2 , and construction for V. In the full multi-agent environment of stage two of  FIG.  4   , these larger π and V may be instantiated for each of the N agents (with full parameter-sharing), along with a new centralized critic Q, and trained using the interpolated policy gradient. 
     A joint policy π(a|o, g) with implied parameter θ shared by all individual π n (a n |o n , g n ) may be defined to maximize the objective:
 
 J (π):=   π [Σ t γ t Σ n=1   N   R ( s   t   ,a   t   n   ,g   n )]  (4)
 
     The objective may be viewed in two ways (e.g., local and global), leading to the actor-critic pair of decentralized and centralized critics. 
     For the local view (e.g., first view), the simulator  108  may learn a decentralized critic from these rewards to provide a policy gradient for agents to achieve local goals without explicit regard to the joint success of all agents. Objectives J n (π):=   π [Σ t γ t R(s t , a t   n , g n )] may be defined to correspond to individual goals g n . J local  may be maximized by ascending the gradient:
 
∇ θ   J   local (π)=Σ n=1   N ∇ θ   J   n (π)=   π [Σ n=1   N  log π( a|o,g ) Q   n   π ( s,a )]≈   π [∇ o Σ n=1   N  log π( a|o,g )( R ( s   t   ,a   t   n   ,g   n )− V   π ( o   t   ,g   n ))]  (5)
 
     Each Q n   π (s, a):=   π [Σ t γ t R(s t , a t   n , g n )|s 0 =s, a 0 =a)] is the state-action value corresponding to the individual reward R(s, a n , g n ). 
     In the second line of equation (5), the following approximations may be made: 
     1) for scalability, approximate all Q n   π (s, a) by a single Q π (s, a, g n ) with an additional input goal g n , instead of using n different function approximators; 
     2) without changing the expectation, replace Q π (s, a, g n ) with the advantage function A π (s, a, g n ):=Q π (s, a, g n )−Q π (s, a, g n ), and use the TD error δ t   n :=R(s t , a t   n , g n )+γV π (s t+1 , g n )−V π (s t , g n ) as an unbiased estimate of the advantage value; 
     3) calculated the decentralized critic V(o t , g n ) by making a final approximation of s t  with o t . 
     Parameterized by θ V , the critic is updated by minimizing the loss:
 
 (θ V )=   s     t     ,a   t   n   ,s   t+1 [( R ( s   t   ,a   t   n   ,g   n )+γ V ( o   t+1   ,g   n ;θ′ v )− V ( o   t   ,g   n ; θ V )) 2 ]  (6)
 
     where θ′ V  are parameters of a target network that slowly updates towards the main θ V . 
     For the global view (e.g., second view), a joint reward R g (s, a, g):=Σ n=1   N R(s, a n , g n ) may be defined. The joint reward may be used to learn a centralized critic that encourages each agent to contribute to other agents&#39; success. J global  may be defined as:
 
∇ θ   J   global (π)=   π [Σ n ∇ θ log π n ( a   n   |o   n   ,g   n )( Q   π ( s,a,g )− b ( s,a   −n   ,g ))]  (7)
 
     Q π (s, a, g)=   π [Σ t γ t R g (s t , a t , g)|s 0 =s, a 0 =a)] is the centralized critic and b(s, a −n , g) may be the generalized counterfactual baseline with multiple goals:
 
 b ( s,a   −n   ,g ):=Σ â     n   π n ( â   n   |o   n   ,g   n ) Q   π ( s ,( a   −n   ,â   n ), g )  (8)
 
     Parameterized by θ Q , the centralized critic is updated by minimizing the loss:
 
 (θ Q )=   s     t     a     t     s     t+1   [ R   9 ( s   t   ,a   t   ,g )+λ Q ( s   t+1   ,a′   t+1   , g; θ′   Q )| a′   t+1   π−Q ( s   t   ,a   t   ,g; θ   Q )) 2 ]  (9)
 
     In equation (9), θ′ Q  and π′ represent slowly-updating target Q and target policy networks, respectively. 
     For the combined view (e.g., third view), interpolation between both views using ∝ϵ (0,1) may be performed by the simulator  108  to determine the extent that the joint reward affects each agent&#39;s policy. 
     The overall policy gradient is defined as:
 
∇ θ   J (π):=α∇ θ   J   local (π)+(1−α)∇ θ   J   global (π)  (10)
 
     The policy gradient may be viewed as a weighted-sum scalarization of a two-objective optimization problem, whose Pareto optima may be found by optimizing along equation (10) using convexity assumptions. 
     In stage one, an actor π and decentralized critic V π  may be trained to learn multiple goals in a single-agent setting. This stage may be based on the single-agent Markov decision process (MDP). The simulator  108  may train an actor π(a n |o n , g n ) and critic V π (o n , g n ) according to policy gradient ∇J local  equation (5) and loss  (θ V ) equation (6), respectively. A goal may be uniformly sampled from G in each training episode, for the agent to learn all goals over the course of training. The simulator  108  may use deep neural networks for function approximation, and the input to the actor and critic networks may include the agent&#39;s observation vector o n  and a vector g n  representing the goal for a particular episode. 
     The simulator  108  may make the simple observation that multi-agent environments typically permit a decomposition of the agent&#39;s observation space into O n =O self   n  ∪ O others   n , where o self  ϵO self   n  contains information about the agent&#39;s own state (e.g. position) while p others  ϵO others   n  is the agent&#39;s local observation of surrounding agents, and that the ability to process o others  is unnecessary in stage one. In this way, the simulator  108  may reduce the size of the input space of π and V to be set to (O self   n , g n ), thereby reducing the number of trainable parameters in stage one and enhancing training speed. The simulator  108  may train these reduced actor and critic networks until convergence, label them as π 1  and V 1 , respectively. 
     In stage two, the simulator  108  may instantiate all agents into the full Markov game, equipped with the pre-trained π and V π  and a new centralized critic Q, and conduct further training for cooperative behavior. The Markov game may be instantiated with N number of agents. The simulator  108  may retain the previously-trained π 1  parameters, instantiate a new neural network π 2  for agents to process the o others  parameter of their local observation, and introduce hidden connections from the output of π 2  to a selected layer of π 1 . Specifically, h i   1  ϵ   m     i    may define the hidden activations of layer i≤L with m i  units in an L-layer neural network representation of π 1 , connected to layer i−1 via h i   1 =f(W i   1 h i−1   1 ) with W i   1  ϵ    m     i     x m     i−1    and activation function f. stage two introduces a K-layer neural network π 2  (o others ) with output layer k ϵ   m     K    chooses a particular layer i* of π 1 , and augments the hidden activations h i*   1  to be:
 
 h   i*   1   =f ( W   i*   1   ,h   i*−1   1   +W   1:2   h   K   2 (11), with  W   1:2 ϵ   m     i     *X m     K    
 
     An equivalent or similar augmentation may be made to critic V 1  using a new neural network V 2 ( o   others ). The simulator  108  may instantiate the centralized critic Q π (s; a; g), which was not required and therefore absent during stage one, and train π; V; Q using the combined gradient ∇J equation (10), loss  (θ v ) equation (6) and loss L(θ Q ) equation (9), respectively. Similarly to stage one, the simulator  108  may assign goals to agents by sampling (e.g., randomly) from a distribution over G during each training episode. The distribution can be constructed to ensure sufficient training on difficult goal combinations that require cooperation, along with easier combinations for maintaining agents&#39; ability to act toward their goal. 
     This two-stage construction of actor-critic networks with curriculum learning improves learning speed compared to direct training on the full multi-agent environment. In this way, CM3 framework based systems and methods learn significantly faster and find more successful policies than even strong baselines. The two-stage curriculum and the decentralized critic facilitate this success, while the global view of policy gradient gives a noticeable advantage in finding a cooperative solution. 
     Further hidden layers i&lt;i* that were pre-trained for processing (o self   n , g n ) in stage one retain the ability to process task information, while the new module learns the effect of surrounding agents. Higher layers i≥i* that can generate goal-directed actions in the single-agent setting of stage one are fine-tuned by the combined gradient to generate cooperative actions for joint success of all agents. 
       FIGS.  5 A- 5 B  are an exemplary flow diagram of a method for cooperative multi-goal, multi-agent, multi-stage reinforcement learning, according to one aspect. According to one aspect, the method of  FIGS.  5 A- 5 B  illustrates pseudo-code for implementing a CM3 framework. 
       FIG.  6    is an exemplary simulation scenario in association with cooperative multi-goal, multi-agent, multi-stage reinforcement learning, according to one aspect. In  FIG.  6   , one segment of a large road network in a traffic simulator  108  is illustrated. The simulator  108  may include l initial lanes starting at horizontal position x i , two of which encounter a merge point, and l−1 goal lanes at terminal position x f . In each episode, N agents are emitted at x i , on randomly selected initial lanes, and each agent n is associated with a randomly-selected goal lane g n  that it should learn to reach at position x f . Agents may receive observations with a limited field of view, choose actions from a discrete action space, and receive rewards according to both terminal and instantaneous criteria (e.g. reached goal, exceeded speed limit). The simulator  108  may define the following environments: 
     E1: a single agent on an otherwise empty road learns to reach any goal lane from any initial lane. This is used for stage one of CM3, which trains initial networks π 1  and V 1  with objective J local . 
     E2: N=4 agents are randomly initialized: with probability 0.8, initial and goal lanes are set so that a double-merge occurs; with probability 0.2, initial and goal lanes are uniformly sampled. The full stage two architecture of CM3 is trained in E2. 
     E2+: used to test generalization, with the traffic simulator-controlled vehicles emitted with probability 0.5/sec. 
     According to one aspect, the simulator  108  may train stage one in E1, followed by training of stage two on E2. According to another aspect, competitor methods may be trained directly in E2 (e.g., where vehicles or agents may compete rather than cooperate). 
     The simulator  108  may be setup to simulate a straight road of a total length, including one or more main lanes and one merge lane. Vehicles on the merge lane may be able to merge onto main lanes within a merge window, and the merge lane may end at a predetermined length. The lanes may have a width, and vehicles may be aligned along any of four sub-lanes within a lane, with a lateral spacing. The speed limit may be set to a predetermined speed. In E2+, the traffic simulator-controlled passenger cars and trucks (e.g., type trailer) that behave according to the Krauβ car-following model may be emitted on to main lanes with a predetermined probability per second. Simulation time resolution may be set to a simulation step time. 
     According to one aspect, in E1, the single agent&#39;s initial lane and goal lane may be sampled randomly from uniform distributions over the number of start and end lanes. In E2, with a probability 0:2, all agents&#39; initial and goal lanes may be sampled independently from uniform distributions over the number of start and end lanes; with probability 0:8, agents [1,2,3,4] were initialized with initial lanes [2,2,3,3] and goal lanes [4,4,0,0]. Departure times may be drawn from normal distributions with mean [0; 2; 0; 2]s and standard deviation 0.2 s. 
     Local Observation 
     According to one aspect, each agent vehicle&#39;s local observation on may include two parameters. The first parameter relating to self, used in all training stages, may be a vector including agent&#39;s speed normalized, normalized number of sub-lanes between agent&#39;s current sub-lane and center sub-lane of goal lane, normalized longitudinal distance to goal position, binary indicator for being on the merge lane, normalized distance to the next segment along the road (using predefined segment boundaries). 
     According to another aspect, the second parameter may be a discretized observation grid centered on the agent, with four channels including: binary indicator of vehicle occupancy, normalized relative speed between other vehicle and agent, binary indicator of vehicle type being a passenger car, binary indicator of vehicle type being a truck. 
     Global State 
     The global state vectors may be the concatenation o self   1 , . . . , o self   N  of all agents&#39; observation parameter o self   n . 
     Goals 
     Each goal vector g n  may be a one-hot vector of length, indicating the goal lane at which agent n should arrive once it reaches a predetermined position x. Goals may be randomly sampled for all agents during each episode. 
     Actions 
     All agents have the same discrete action space, including five options: no-operation action (e.g., maintain current speed and lane), accelerate, decelerate, and shift one sub-lane to the left, shift one sub-lane to the right. Each agent&#39;s action a n  may be represented as a one-hot vector of a predetermined length. 
     Individual Rewards 
     According to one aspect, the reward R(s t , a t   n , g n ); gn) for agent n with goal g n  may be given according to the conditions: 
     −10 for a collision (followed by termination of episode) 
     −10 for time-out (exceed 120 simulation steps during episode) 
     +10(1−Δ) for reaching the end of the road and having a normalized sub-lane difference of from the center of the goal lane 
     −5 for entering the merge lane from another lane during 200 m&lt;x&lt;400 m 
     −0.5 for being in the merge lane during 200 m&lt;x&lt;400 m 
     −0.1 if current speed exceeds 35.7 m/s 
     Shared Global Reward 
     According to one aspect, a shared global reward Rg(s t ; a t ; g) may be determined by: 
     −10 if any collision occurred; and 
     the average of all individual rewards of agents who reached the end of the road at time t. 
     Architecture 
     According to one aspect, the policy network π 1  during stage one feeds each of the inputs o self  and g n  to one fully-connected layer with 32 units. The concatenation may be fully-connected to a layer h* π1  with 64 units, and fully-connected to a softmax output layer with 5 units, each corresponding to one discrete action. In stage two, the input observation grid o others  may be processed by a convolutional layer with 4 filters of size 5×3 and stride 1×1, flattened and fully-connected to a layer with 64 units, then fully-connected to the layer h* π     1    of π 1 . ReLU nonlinearity may be used for all hidden layers. Action probabilities are computed by lower-bounding the softmax outputs via Pr(a n =i)=(1−ϵ)softmax(i)+ϵ/|A|, where ϵ is a decaying exploration parameter and |A|=5. 
     The decentralized critic V 1 , during stage one, may feed each of the inputs o self  and g n  to one fully-connected layer with 32 units. The concatenation may be fully-connected to the output linear layer hV 1 _ with a single unit. In stage two, the input observation grid o others  may be processed by a convolutional layer  13  with 4 filters of size 5×3 and stride 1×1, flattened and fully-connected to a layer with 32 units, then fully-connected to the output layer h* V     1    of V 1 . ReLU nonlinearity may be used for all hidden layers. 
     The centralized critic Q(s, a, g) may receive input (s, a −n , g n , g −n , n), which may be connected to two fully connected layers with 128 units and ReLU activation, and fully-connected to a linear output layer with 5 units. The value of each output node i may be interpreted as the action-value Q(s, a −n , a n ≤i, g) for agent n taking action i and all other agents taking action a −n . The agent label vector n is a one-hot indicator vector, used as input to differentiate between evaluations of the Q-function for different agents. 
     Double replay buffers B 1  and B 2  may be used as a heuristic to improve training stability for all algorithms on stage two. Instead of storing each environment transition immediately, an additional episode buffer may be used to store all transitions encountered during each episode. At the end of each episode, the cumulative reward of all agents may be compared against a threshold (e.g.,  32 ), to determine whether the transitions in the episode buffer should be stored into B 1  or B 2 . For training, half of the minibatch is sampled from each of B 1  and B 2 . 
       FIGS.  7 - 8    are exemplary performance results in association with cooperative multi-goal, multi-agent, multi-stage reinforcement learning, according to one aspect. As seen in  FIG.  7   , CM3 converged to higher performance more than 45 k episodes earlier than other techniques in E2. In  FIG.  8   , it may be seen that a policy learned by a CM3 framework based system or method generalizes better to E2+. 
       FIG.  9    is an exemplary simulation scenario in association with cooperative multi-goal, multi-agent, multi-stage reinforcement learning, according to one aspect. In  FIG.  9   , an initial lane configuration associated with E2 requires agents to perform a double-merge or a double-lane change to reach their goal lanes. 
       FIG.  10    is an exemplary reward versus episode graph in the single-agent environment E1, according to one aspect. 
     Still another aspect involves a computer-readable medium including processor-executable instructions configured to implement one aspect of the techniques presented herein. An aspect of a computer-readable medium or a computer-readable device devised in these ways is illustrated in  FIG.  11   , wherein an implementation  1100  includes a computer-readable medium  1108 , such as a CD-R, DVD-R, flash drive, a platter of a hard disk drive, etc., on which is encoded computer-readable data  1106 . This encoded computer-readable data  1106 , such as binary data including a plurality of zero&#39;s and one&#39;s as shown in  1106 , in turn includes a set of processor-executable computer instructions  1104  configured to operate according to one or more of the principles set forth herein. In this implementation  1100 , the processor-executable computer instructions  1104  may be configured to perform a method  1102 , such as the method  300  of  FIG.  3    or the method of  FIGS.  5 A- 5 B . In another aspect, the processor-executable computer instructions  1104  may be configured to implement a system, such as the system  100  of  FIG.  1   . Many such computer-readable media may be devised by those of ordinary skill in the art that are configured to operate in accordance with the techniques presented herein. 
     As used in this application, the terms “component”, “module,” “system”, “interface”, and the like are generally intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to being, a process running on a processor, a processing unit, an object, an executable, a thread of execution, a program, or a computer. By way of illustration, both an application running on a controller and the controller may be a component. One or more components residing within a process or thread of execution and a component may be localized on one computer or distributed between two or more computers. 
     Further, the claimed subject matter is implemented as a method, apparatus, or article of manufacture using standard programming or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computer to implement the disclosed subject matter. The term “article of manufacture” as used herein is intended to encompass a computer program accessible from any computer-readable device, carrier, or media. Of course, many modifications may be made to this configuration without departing from the scope or spirit of the claimed subject matter. 
       FIG.  12    and the following discussion provide a description of a suitable computing environment to implement aspects of one or more of the provisions set forth herein. The operating environment of  FIG.  12    is merely one example of a suitable operating environment and is not intended to suggest any limitation as to the scope of use or functionality of the operating environment. Example computing devices include, but are not limited to, personal computers, server computers, hand-held or laptop devices, mobile devices, such as mobile phones, Personal Digital Assistants (PDAs), media players, and the like, multiprocessor systems, consumer electronics, mini computers, mainframe computers, distributed computing environments that include any of the above systems or devices, etc. 
     Generally, aspects are described in the general context of “computer readable instructions” being executed by one or more computing devices. Computer readable instructions may be distributed via computer readable media as will be discussed below. Computer readable instructions may be implemented as program modules, such as functions, objects, Application Programming Interfaces (APIs), data structures, and the like, that perform one or more tasks or implement one or more abstract data types. Typically, the functionality of the computer readable instructions are combined or distributed as desired in various environments. 
       FIG.  12    illustrates a system  1200  including a computing device  1212  configured to implement one aspect provided herein. In one configuration, the computing device  1212  includes at least one processing unit  1216  and memory  1218 . Depending on the exact configuration and type of computing device, memory  1218  may be volatile, such as RAM, non-volatile, such as ROM, flash memory, etc., or a combination of the two. This configuration is illustrated in  FIG.  12    by dashed line  1214 . 
     In other aspects, the computing device  1212  includes additional features or functionality. For example, the computing device  1212  may include additional storage such as removable storage or non-removable storage, including, but not limited to, magnetic storage, optical storage, etc. Such additional storage is illustrated in  FIG.  12    by storage  1220 . In one aspect, computer readable instructions to implement one aspect provided herein are in storage  1220 . Storage  1220  may store other computer readable instructions to implement an operating system, an application program, etc. Computer readable instructions may be loaded in memory  1218  for execution by processing unit  1216 , for example. 
     The term “computer readable media” as used herein includes computer storage media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions or other data. Memory  1218  and storage  1220  are examples of computer storage media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which may be used to store the desired information and which may be accessed by the computing device  1212 . Any such computer storage media is part of the computing device  1212 . 
     The term “computer readable media” includes communication media. Communication media typically embodies computer readable instructions or other data in a “modulated data signal” such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” includes a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. 
     The computing device  1212  includes input device(s)  1224  such as keyboard, mouse, pen, voice input device, touch input device, infrared cameras, video input devices, or any other input device. Output device(s)  1222  such as one or more displays, speakers, printers, or any other output device may be included with the computing device  1212 . Input device(s)  1224  and output device(s)  1222  may be connected to the computing device  1212  via a wired connection, wireless connection, or any combination thereof. In one aspect, an input device or an output device from another computing device may be used as input device(s)  1224  or output device(s)  1222  for the computing device  1212 . The computing device  1212  may include communication connection(s)  1226  to facilitate communications with one or more other devices  1230 , such as through network  1228 , for example. 
     Although the subject matter has been described in language specific to structural features or methodological acts, it is to be understood that the subject matter of the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example aspects. 
     Various operations of aspects are provided herein. The order in which one or more or all of the operations are described should not be construed as to imply that these operations are necessarily order dependent. Alternative ordering will be appreciated based on this description. Further, not all operations may necessarily be present in each aspect provided herein. 
     As used in this application, “or” is intended to mean an inclusive “or” rather than an exclusive “or”. Further, an inclusive “or” may include any combination thereof (e.g., A, B, or any combination thereof). In addition, “a” and “an” as used in this application are generally construed to mean “one or more” unless specified otherwise or clear from context to be directed to a singular form. Additionally, at least one of A and B and/or the like generally means A or B or both A and B. Further, to the extent that “includes”, “having”, “has”, “with”, or variants thereof are used in either the detailed description or the claims, such terms are intended to be inclusive in a manner similar to the term “comprising”. 
     Further, unless specified otherwise, “first”, “second”, or the like are not intended to imply a temporal aspect, a spatial aspect, an ordering, etc. Rather, such terms are merely used as identifiers, names, etc. for features, elements, items, etc. For example, a first channel and a second channel generally correspond to channel A and channel B or two different or two identical channels or the same channel. Additionally, “comprising”, “comprises”, “including”, “includes”, or the like generally means comprising or including, but not limited to. 
     It will be appreciated that various of the above-disclosed and other features and functions, or alternatives or varieties thereof, may be desirably combined into many other different systems or applications. Also that various presently unforeseen or unanticipated alternatives, modifications, variations or improvements therein may be subsequently made by those skilled in the art which are also intended to be encompassed by the following claims.