Patent Publication Number: US-2022230080-A1

Title: System and method for utilizing a recursive reasoning graph in multi-agent reinforcement learning

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application claims priority to U.S. Provisional Application Ser. No. 63/139,690 filed on Jan. 20, 2021, which is expressly incorporated herein by reference. 
    
    
     BACKGROUND 
     Many real-world scenarios involve interactions between multiple agents with limited information exchange. Multi-robot navigation and autonomous driving applications such as highway merging, four-way stops, and lane changing are examples of situations where interaction is required between multiple mobile agents. For example, two mobile agents may be attempting to make maneuvers that may cross each other&#39;s paths. Modeling interactions of various agents may be difficult as continuous leaning is necessary. In scenarios where complex interactions may occur between numerous agents adequate machine based understanding of multiple agent reasoning to properly model such interactions has not been successfully achieved. 
     BRIEF DESCRIPTION 
     According to one aspect, a computer-implemented method for utilizing a recursive reasoning graph in multi-agent reinforcement learning that includes receiving data associated with an ego agent and a target agent that are traveling within a multi-agent environment. The computer-implemented method also includes utilizing a multi-agent central actor-critic framework to analyze the data associated with the ego agent and the target agent. The computer-implemented method additionally includes performing level-k recursive reasoning based on the multi-agent actor-critic framework to calculate higher level recursion actions of the ego agent and the target agent. An output of the level-k recursive reasoning is used to learn an agent action policy that is associated with the ego agent and an agent action policy that is associated with the target agent. The computer-implemented method further includes controlling at least one of: the ego agent and the target agent to operate within the multi-agent environment based on at least one of: the agent action policy that is associated with the ego agent and the agent action policy that is associated with the target agent. 
     According to another aspect, a system for utilizing a recursive reasoning graph in multi-agent reinforcement learning that includes a memory storing instructions when executed by a processor cause the processor to receive data associated with an ego agent and a target agent that are traveling within a multi-agent environment. The instructions also cause the processor to utilize a multi-agent central actor-critic framework to analyze the data associated with the ego agent and the target agent. The instructions additionally cause the processor to perform level-k recursive reasoning based on the multi-agent actor-critic framework to calculate higher level recursion actions of the ego agent and the target agent. An output of the level-k recursive reasoning is used to learn an agent action policy that is associated with the ego agent and an agent action policy that is associated with the target agent. The instructions further cause the processor to control at least one of: the ego agent and the target agent to operate within the multi-agent environment based on at least one of: the agent action policy that is associated with the ego agent and the agent action policy that is associated with the target agent. 
     According to yet another aspect, a non-transitory computer readable storage medium storing instructions that when executed by a computer, which includes a processor perform a method that includes receiving data associated with an ego agent and a target agent that are traveling within a multi-agent environment. The method also includes utilizing a multi-agent central actor-critic framework to analyze the data associated with the ego agent and the target agent. The method additionally includes performing level-k recursive reasoning based on the multi-agent actor-critic framework to calculate higher level recursion actions of the ego agent and the target agent. An output of the level-k recursive reasoning is used to learn an agent action policy that is associated with the ego agent and an agent action policy that is associated with the target agent. The method further includes controlling at least one of: the ego agent and the target agent to operate within the multi-agent environment based on at least one of: the agent action policy that is associated with the ego agent and the agent action policy that is associated with the target agent. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The novel features believed to be characteristic of the disclosure are set forth in the appended claims. In the descriptions that follow, like parts are marked throughout the specification and drawings with the same numerals, respectively. The drawing figures are not necessarily drawn to scale and certain figures can be shown in exaggerated or generalized form in the interest of clarity and conciseness. The disclosure itself, however, as well as a preferred mode of use, further objects and advances thereof, will be best understood by reference to the following detailed description of illustrative embodiments when read in conjunction with the accompanying drawings, wherein: 
         FIG. 1  is a schematic view of an exemplary system for utilizing a recursive reasoning graph in multi-agent reinforcement learning according to an exemplary embodiment of the present disclosure; 
         FIG. 2  is an illustrative example of a multi-agent environment according to an exemplary embodiment of the present disclosure; 
         FIG. 3  is a process flow diagram of a method for receiving data associated with the multi-agent environment and processing a simulated multi-agent environment model according to an exemplary embodiment of the present disclosure; 
         FIG. 4  is an illustrative example of the simulated multi-agent environment model which virtually represents the multi-agent environment according to an exemplary embodiment of the present disclosure; 
         FIG. 5  is a process flow diagram of a method for learning agent action policies that are to be executed to control an ego agent and/or a target agent to operate within the multi-agent environment according to an exemplary embodiment of the present disclosure; 
         FIG. 6  is an illustrative example of the recursive reasoning graph in a three-agent stochastic game according to an exemplary embodiment of the present disclosure; and 
         FIG. 7  is a process flow diagram of a method for utilizing a recursive reasoning graph in multi-agent reinforcement learning according to an exemplary embodiment of the present disclosure. 
     
    
    
     DETAILED DESCRIPTION 
     The following includes definitions of selected terms employed herein. The definitions include various examples and/or forms of components that fall within the scope of a term and that may be used for implementation. The examples are not intended to be limiting. 
     A “bus”, as used herein, refers to an interconnected architecture that is operably connected to other computer components inside a computer or between computers. The bus may transfer data between the computer components. The bus may be a memory bus, a memory controller, a peripheral bus, an external bus, a crossbar switch, and/or a local bus, among others. The bus may also be a vehicle bus that interconnects components inside a vehicle using protocols such as Media Oriented Systems Transport (MOST), Controller Area network (CAN), Local Interconnect Network (LIN), among others. 
     “Computer communication”, as used herein, refers to a communication between two or more computing devices (e.g., computer, personal digital assistant, cellular telephone, network device) and may be, for example, a network transfer, a file transfer, an applet transfer, an email, a hypertext transfer protocol (HTTP) transfer, and so on. A computer communication may occur across, for example, a wireless system (e.g., IEEE 802.11), an Ethernet system (e.g., IEEE 802.3), a token ring system (e.g., IEEE 802.5), a local area network (LAN), a wide area network (WAN), a point-to-point system, a circuit switching system, a packet switching system, among others. 
     A “disk”, as used herein may be, for example, a magnetic disk drive, a solid state disk drive, a floppy disk drive, a tape drive, a Zip drive, a flash memory card, and/or a memory stick. Furthermore, the disk may be a CD-ROM (compact disk ROM), a CD recordable drive (CD-R drive), a CD rewritable drive (CD-RW drive), and/or a digital video ROM drive (DVD ROM). The disk may store an operating system that controls or allocates resources of a computing device. 
     A “memory”, as used herein may include volatile memory and/or non-volatile memory. Non-volatile memory may include, for example, ROM (read only memory), PROM (programmable read only memory), EPROM (erasable PROM), and EEPROM (electrically erasable PROM). Volatile memory may include, for example, RAM (random access memory), synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), and direct RAM bus RAM (DRRAM). The memory may store an operating system that controls or allocates resources of a computing device. 
     A “module”, as used herein, includes, but is not limited to, non-transitory computer readable medium that stores instructions, instructions in execution on a machine, hardware, firmware, software in execution on a machine, and/or combinations of each to perform a function(s) or an action(s), and/or to cause a function or action from another module, method, and/or system. A module may also include logic, a software controlled microprocessor, a discrete logic circuit, an analog circuit, a digital circuit, a programmed logic device, a memory device containing executing instructions, logic gates, a combination of gates, and/or other circuit components. Multiple modules may be combined into one module and single modules may be distributed among multiple modules. 
     An “operable connection”, or a connection by which entities are “operably connected”, is one in which signals, physical communications, and/or logical communications may be sent and/or received. An operable connection may include a wireless interface, a physical interface, a data interface and/or an electrical interface. 
     A “processor”, as used herein, processes signals and performs general computing and arithmetic functions. Signals processed by the processor may include digital signals, data signals, computer instructions, processor instructions, messages, a bit, a bit stream, or other means that may be received, transmitted and/or detected. Generally, the processor may be a variety of various processors including multiple single and multicore processors and co-processors and other multiple single and multicore processor and co-processor architectures. The processor may include various modules to execute various functions. 
     A “vehicle”, as used herein, refers to any moving vehicle that is capable of carrying one or more human occupants and is powered by any form of energy. The term “vehicle” includes, but is not limited to: cars, trucks, vans, minivans, SUVs, motorcycles, scooters, boats, go-karts, amusement ride cars, rail transport, personal watercraft, and aircraft. In some cases, a motor vehicle includes one or more engines. Further, the term “vehicle” may refer to an electric vehicle (EV) that is capable of carrying one or more human occupants and is powered entirely or partially by one or more electric motors powered by an electric battery. The EV may include battery electric vehicles (BEV) and plug-in hybrid electric vehicles (PHEV). The term “vehicle” may also refer to an autonomous vehicle and/or self-driving vehicle powered by any form of energy. The autonomous vehicle may or may not carry one or more human occupants. Further, the term “vehicle” may include vehicles that are automated or non-automated with pre-determined paths or free-moving vehicles. 
     A “value” and “level”, as used herein may include, but is not limited to, a numerical or other kind of value or level such as a percentage, a non-numerical value, a discrete state, a discrete value, a continuous value, among others. The term “value of X” or “level of X” as used throughout this detailed description and in the claims refers to any numerical or other kind of value for distinguishing between two or more states of X. For example, in some cases, the value or level of X may be given as a percentage between 0% and 100%. In other cases, the value or level of X could be a value in the range between 1 and 10. In still other cases, the value or level of X may not be a numerical value, but could be associated with a given discrete state, such as “not X”, “slightly x”, “x”, “very x” and “extremely x”. 
     I. System Overview 
     Referring now to the drawings, wherein the showings are for purposes of illustrating one or more exemplary embodiments and not for purposes of limiting same,  FIG. 1  is a schematic view of an exemplary system  100  utilizing a recursive reasoning graph in multi-agent reinforcement learning according to an exemplary embodiment of the present disclosure. The components of the system  100 , as well as the components of other systems, hardware architectures, and software architectures discussed herein, may be combined, omitted, or organized into different architectures for various embodiments. 
     Generally, the system  100  includes an ego agent  102  and one or more target agents  104 . For purposes of simplicity, this disclosure will describe the embodiments of the system  100  with respect to a single ego agent  102  and a single target agent  104 . However, it is appreciated that the system  100  may include more than one ego agent  102  and more than one target agent  104  and that the embodiments and processes discussed herein may be utilized in an environment that includes one or more ego agents  102  and one or more target agents  104 . 
     As shown in the illustrative example of  FIG. 2 , the ego agent  102  and the target agent  104  may be traveling in a multi-agent environment  200 . In one or more configurations, the ego agent  102  and/or the target agent  104  may include, but may not be limited to, an automobile, a robot, a forklift, a bicycle, an airplane, a construction crane, and the like that may be traveling within one or more types of multi-agent environments. In one embodiment, the multi-agent environment  200  may include, but may not be limited to areas that are evaluated to provide navigable pathways for the ego agent  102  and/or the target agent  104  that are traveling on the roadway  202 , as shown in the illustrative example of  FIG. 2 . 
     In additional embodiments, one or more multi-agent environments may include, but may not be limited to, additional types of roadways such as a narrow street or tunnel and/or a pathway that may exist within a confined location such as a factory floor, a construction site, or an airport taxiway. For purposes of simplicity, the exemplary embodiments and examples discussed herein will mainly be described with reference to the multi-agent environment  200  that includes the roadway, as shown in the illustrative example of  FIG. 2 . However, it is appreciated that the multi-agent environment  200  may include the additional types of roadways, discussed above. 
     As shown in  FIG. 2 , the ego agent  102  and the target agent  104  may be traveling within adjacent lanes of a roadway  202  of the multi-agent environment  200 . The ego agent  102  and the target agent  104  may be traveling in respective directions and in locations that are within a particular distance of one another. As shown, the ego agent  102  may be traveling on the roadway  202  to reach a goal  204  (e.g., way point, destination) while the target agent  104  may be traveling on the roadway  202  to reach a goal  206  (e.g., way point, destination). In some instances, as shown, a path  208  of the ego agent  102  may potentially cross a path  210  of the target agent  104  as each of the agents  102 ,  104  are attempting to reach their respective goals  204 ,  206 . 
     With reference  FIG. 1  and  FIG. 2 , in an exemplary embodiment, the system  100  may include a multi-agent recursive reasoning reinforcement learning application (multi-agent application)  106  that may be configured to complete multi-agent reinforcement learning for simultaneously learning policies for multiple agents interacting amongst one another. As discussed below, one or more policies may be executed to autonomously control the ego agent  102  and/or the target agent  104  to reach their respective goals  204 ,  206  while taking one another into account. As discussed below, the multi-agent application  106  may be configured to utilize a neural network  108  and may execute instructions to utilize a recursive reasoning model in a centralized-training-decentralized execution framework to facilitate maneuvers within the multi-agent environment  200  with respect to the agents  102 ,  104 . 
     The multi-agent application  106  may execute a recursive reasoning graph as a graph structure that may be utilized in a simulation of the multi-agent environment  200  to mimic a recursive reasoning procedure under the centralized-training-decentralized-execution framework for multi-agent reinforcement learning. The multi-agent application  106  may thereby be configured to utilize a multi-agent central actor critic model (shown in  FIG. 6 ) to learn one or more agent action policies that account for the actions of the ego agent  102  and the target agent  104  that are traveling within the multi-agent environment  200 . 
     The ego agent  102  and the target agent  104  are evaluated as the central actors and are treated as nodes to build the recursive reasoning graph to efficiently calculate higher level recursive actions of interacting agents. As central actors, the ego agent  102  and the target agent  104  are analyzed as self-interested agents that are attempting to reach their respective goals  204 ,  206  in a most efficient manner. The multi-agent central actor critic model includes one or more iterations of Markov Games where one or more critics evaluate one or more actions (output of actor models) taken by a simulated ego agent and a simulated target agent to determine one or more rewards and one or more states related to a goal-specific reward function. 
     The recursive reasoning graph structure enables the multi-agent application  106  to model the relationship between the ego agent  102  and the target agent  104  and may explicitly consider their responses as central actors. The recursive actions of each agent  102 ,  104  are efficiently sampled and shared through message passing in the graph. Accordingly, the recursive reasoning graph executed by the multi-agent application  106  explicitly models the recursive reasoning process of the agents  102 ,  104  in general-sum games. As discussed below, the multi-agent application  106  may be configured to augment the existing centralized-training-decentralized-execution algorithms with centralized actors and graph-like message passing to efficiently train learning agents in a simulated environment (shown in  FIG. 4 ) that represent the agents  102 ,  104  of the multi-agent environment  200 . 
     Stated differently, the multi-agent application  106  utilizes the multi-agent central actor critic model to perform centralized training, then decentralized execution. The multi-agent application  106  accordingly evaluates the positions, locations, and paths of all agents  102 ,  104 . As discussed, the multi-agent application  106  may evaluate the actions of the ego agent  102  from the perspective of the target agent  104  and may evaluate the actions of the target agent  104  from the perspective of the ego agent  102 . Accordingly, the Markov Game simulation completed by the multi-agent application  106  may take account the actions of both the ego agent  102  and the target agent  104  as central actors. 
     In one or more embodiments, the multi-agent application  106  may be configured to learn multiple interactive policies for multiple agents, including, but not limited to, the ego agent  102  and the target agent  104  that are traveling within the multi-agent environment  200 . In particular, the output of the graph are utilized to thereby learn respective agent action policies that may be utilized to enable the ego agent  102  and/or the target agent  104  to be autonomously operated to reach their respective goals  204 ,  206  while accounting for one another within the multi-agent environment  200 . 
     This functionality allows the multi-agent application  106  to acquire knowledge regarding the actions of all of the agents  102 ,  104  simultaneously rather than to optimize different agent actions separately. Additionally, this functionality allows the multi-agent application  106  to learn multiple agent action policies associated with each of the ego agent  102  and the target agent  104 . As discussed below, the multi-agent application  106  may incorporate actions of the ego agent  102  to learn the agent action policy associated with the target agent  104 . Additionally, the multi-agent application  106  may incorporate actions of the target agent  104  to learn the agent action policy associated with the ego agent  102 . 
     This may be accomplished by the use of level-k reasoning to assume an opposing agent&#39;s reasoning into the respective agent&#39;s reasoning to reach their respective goal. For example, the level-k reasoning allows the multi-agent application  106  to assume the reasoning of the target agent  104  with respect to following the path  208  of reaching its goal  204  when learning the ego agent&#39;s agent action policy. Similarly, the level-K reasoning allows the multi-agent application  106  to assume the reasoning of the ego agent  102  with respect to following the path  210  of reaching its goal  206  when learning the target agent&#39;s agent action policy. The level-k reasoning may incorporate a plurality of levels (e.g., level-zero reasoning, level-one reasoning, level-two reasoning, level-k reasoning) to thereby provide action determinations regarding opposing agent&#39;s actions in each agent  102 ,  104  reaching their respective goal. 
     As discussed below, upon learning the agent action policy associated with the ego agent  102  and the agent action policy associated with the target agent  104 , the multi-agent application  106  may be configured to train the neural network  108  with the respective agent action policies. As discussed below, the multi-agent application  106  may communicate with the neural network  108  to receive the respective agent action policies associated with the ego agent  102  and/or the target agent  104  to be executed to control autonomous operation (e.g., driving) of the ego agent  102  and/or the target agent  104  to thereby follow particular paths at a particular speed, acceleration rate, steering angle, deceleration rate, and the like while maneuvering within the multi-agent environment  200  to reach the respective goals  204 ,  206  without any conflict amongst one another. Accordingly, the agent action policies may be utilized by the multi-agent application  106  to autonomously control the ego agent  102  and/or the target agent  104  within the multi-agent environment  200  and/or similar multi-agent environments (e.g., real-world environments) that include similar driving scenarios to safely and efficiently navigate to their respective goals  204 ,  206 . 
     As discussed below, the multi-agent application  106  may determine a virtual simulated model of the multi-agent environment  200  in which the ego agent  102  and the target agent  104  and the respective goals  204 ,  206  are virtually represented at each discrete time step. The simulated model may be determined based on image data and/or LiDAR data that may be provided to the multi-agent application  106  by one or more components of the ego agent  102  and/or the target agent  104 . For example, the simulated model may include lanes  202   a ,  202   d  on which the ego agent  102  and the target agent  104  are traveling in addition to lanes  202   b ,  202   c  that fall between the lanes  202   a ,  202   d  on which the ego agent  102  and the target agent  104  are traveling. As discussed below, the simulated model includes respective observations and respective goals that may be inputted into the multi-central actor critic model and used to complete level-k recursive reasoning to learn the agent action policies respectively associated with the ego agent  102  and the target agent  104 . 
     With continued reference to  FIG. 1 , the ego agent  102  and the target agent  104  may include respective electronic control devices (ECUs)  110   a ,  110   b . The ECUs  110   a ,  110   b  may execute one or more applications, operating systems, vehicle system and subsystem executable instructions, among others. In one or more embodiments, the ECUs  110   a ,  110   b  may include a respective microprocessor, one or more application-specific integrated circuit(s) (ASIC), or other similar devices. The ECUs  110   a ,  110   b  may also include respective internal processing memory, an interface circuit, and bus lines for transferring data, sending commands, and communicating with the plurality of components of the ego agent  102  and/or the target agent  104 . 
     The ECUs  110   a ,  110   b  may also include a respective communication device (not shown) for sending data internally to components of the respective agents  102 ,  104  and communicating with externally hosted computing systems (e.g., external to the agents  102 ,  104 ). Generally, the ECUs  110   a ,  110   b  communicate with respective storage units  114   a ,  114   b  to execute the one or more applications, operating systems, vehicle systems and subsystem user interfaces, and the like that are stored within the respective storage units  114   a ,  114   b.    
     In some embodiments, the storage units  114   a ,  114   b  may respectively store the respectively learned agent-action polices associated with the ego agent  102  and/or the target agent  104 . Accordingly, the storage units  114   a ,  114   b  may be accessed by the multi-agent application  106  to store the respective agent-action polices learned by the application  106  to be followed by the respective agents  102 ,  104 . In some embodiments, the storage units  114   a ,  114   b  may be accessed by the application  106  to retrieve the respective agent-action polices to autonomously control the operation of the ego agent  102  and/or the target agent  104  to account for the presence of one another (e.g., other agents) within the multi-agent environment  200 . 
     In an exemplary embodiment, the ECUs  110   a ,  110   b  may be configured to operably control the plurality of components of the respective agents  102 ,  104 . The ECUs  110   a ,  110   b  may additionally provide one or more commands to one or more control units (not shown) of the agents  102 ,  104  including, but not limited to a respective engine control unit, a respective braking control unit, a respective transmission control unit, a respective steering control unit, and the like to control the ego agent  102  and/or target agent  104  to be autonomously operated. 
     With continued reference to  FIG. 1 , the respective storage units  114   a ,  114   b  of the ego agent  102  and the target agent  104  may be configured to store one or more executable files associated with one or more operating systems, applications, associated operating system data, application data, vehicle system and subsystem user interface data, and the like that are executed by the respective ECUs  110   a ,  110   b . In one or more embodiments, the storage units  114   a ,  114   b  may be accessed by the multi-agent application  106  to store data, for example, one or more images, videos, one or more sets of image coordinates, one or more sets of LiDAR coordinates (e.g., LiDAR coordinates associated with the position of an object), one or more sets of locational coordinates (e.g., GPS/DGPS coordinates) and/or vehicle dynamic data associated respectively with the ego agent  102  and the target agent  104 . 
     The ECUs  110   a ,  110   b  may be additionally configured to operably control respective camera systems  116   a ,  116   b  of the ego agent  102  and the target agent  104 . The camera systems  116   a ,  116   b  may include one or more cameras that are positioned at one or more exterior portions of the respective agents  102 ,  104 . The camera(s) of the camera systems  116   a ,  116   b  may be positioned in a direction to capture the surrounding environment of the respective agents  102 ,  104  that include predetermined areas located around (front/sides/behind) the respective agents  102 ,  104  of the multi-agent environment  200 . 
     In one or more configurations, the one or more cameras of the respective camera systems  116   a ,  116   b  may be disposed at external front, rear, and/or side portions of the respective agents  102 ,  104  including, but not limited to different portions of the bumpers, lighting units, fenders/body panels, and/or windshields. The one or more cameras may be positioned on a respective planar sweep pedestal (not shown) that allows the one or more cameras to be oscillated to capture images of the surrounding environments of the respective agents  102 ,  104 . 
     With respect to the ego agent  102 , the multi-agent application  106  may receive image data associated with untrimmed images/video of the surrounding environment of the ego agent  102  from the camera system  116   a  and may execute image logic to analyze the image data and determine ego agent image based observations associated with the multi-agent environment  200 , one or more target agents  104  that may be located within the multi-agent environment  200 , one or lanes  202   a - 202   d  (pathways) within the multi-agent environment  200 , and/or one or more objects (not shown) that may be located within the multi-agent environment  200 . 
     With respect to the target agent  104 , the multi-agent application  106  may receive image data associated with untrimmed images/video of the surrounding environment of the target agent  104  from the camera system  116   b  and may execute image logic to analyze the image data and determine target agent image based observations associated with the multi-agent environment  200 , the ego agent  102  that may be located within the multi-agent environment  200 , one or lanes  202   a - 202   d  (pathways) within the multi-agent environment  299 , and/or one or more objects (not shown) that may be located within the multi-agent environment  200 . 
     In one or more embodiments, the ECUs  110   a ,  110   b  may also be operably connected to respective vehicle laser projection systems  118   a ,  118   b  that may include one or more respective LiDAR transceivers (not shown). The one or more respective LiDAR transceivers of the respective vehicle laser projection systems  118   a ,  118   b  may be disposed at respective external front, rear, and/or side portions of the respective agents  102 ,  104 , including, but not limited to different portions of bumpers, body panels, fenders, lighting units, and/or windshields. 
     The one or more respective LiDAR transceivers may include one or more planar sweep lasers that include may be configured to oscillate and emit one or more laser beams of ultraviolet, visible, or near infrared light toward the surrounding environment of the respective agents  102 ,  104 . The vehicle laser projection systems  118   a ,  118   b  may be configured to receive one or more reflected laser waves based on the one or more laser beams emitted by the LiDAR transceivers. The one or more reflected laser waves may be reflected off of one or more boundaries  212   a ,  212   b  (e.g., guardrails) of the multi-agent environment  200 , and/or one or more objects (e.g., other agents, cones, pedestrians, etc.) that may be located within the multi-agent environment  200 . 
     In an exemplary embodiment, the vehicle laser projection systems  118   a ,  118   b  may be configured to output LiDAR data associated to one or more reflected laser waves. With respect to the ego agent  102 , the multi-agent application  106  may receive LiDAR data communicated by the vehicle laser projection system  118   a  and may execute LiDAR logic to analyze the LiDAR data and determine ego agent LiDAR based observations associated with the multi-agent environment  200 , and more specifically the lane  202   a  on which the ego agent  102  is traveling, additional lanes  202   b - 202   d  included within the multi-agent environment  200 , one or more target agents  104  that may be located within the multi-agent environment  200 , one or more boundaries  212   a ,  212   b  of the multi-agent environment  200 , and/or one or more objects that may be located within the multi-agent environment  200 . 
     With respect to the target agent  104 , the multi-agent application  106  may receive LiDAR data communicated by the vehicle laser projection system  118   b  and may execute LiDAR logic to analyze the LiDAR data and determine target agent LiDAR based observations associated with the multi-agent environment  200 , and more specifically the lane  202   d  on which the target agent  104  is traveling, additional lanes  202   a - 202   c  included within the multi-agent environment  200 , the ego agent  102  that may be located within the multi-agent environment  200 , one or more boundaries  212   a ,  212   b  of the multi-agent environment  200 , and/or one or more objects that may be located within the multi-agent environment  200 . 
     In one or more embodiments, the ego agent  102  and the target agent  104  may additionally include respective communication units  120   a ,  120   b  that may be operably controlled by the respective ECUs  110   a ,  110   b  of the respective agents  102 ,  104 . The communication units  120   a ,  120   b  may each be operably connected to one or more transceivers (not shown) of the respective agents  102 ,  104 . The communication units  120   a ,  120   b  may be configured to communicate through an internet cloud  122  through one or more wireless communication signals that may include, but may not be limited to Bluetooth® signals, Wi-Fi signals, ZigBee signals, Wi-Max signals, and the like. In some embodiments, the communication unit  120   a  of the ego agent  102  may be configured to communicate via vehicle-to-vehicle (V2V) with the communication unit  120   b  of the target agent  104  to exchange information about the positon, speed, steering angles, acceleration rates, deceleration rates, and the like of the agents  102 ,  104  traveling within the multi-agent environment  200 . 
     In one embodiment, the communication units  120   a ,  120   b  may be configured to connect to the internet cloud  122  to send and receive communication signals to and from an externally hosted server infrastructure (external server)  124 . The external server  124  may host the neural network  108  and may execute the multi-agent application  106  to utilize processing power to learn one or more respective agent action policies and to thereby train the neural network  108  with the one or more respective agent action policies associated with the ego agent  102  and/or the target agent  104 . 
     In particular, the neural network  108  may be trained at one or more time steps based on learning of one or more agent action policies that are associated with the ego agent  102  and/or the target agent  104  that are traveling within the multi-agent environment  200 . The training of the neural network  108  may allow the agents  102 ,  104  to receive data pertaining to real-time or similar multi-agent scenarios (e.g., ego agent  102  and target agent  104  located with respect to one another) that may occur within a multi-agent environment  200  to ensure that policies are learned that may be utilized by the ego agent  102  and/or the target agent  104  to simultaneously achieve respective goals  204 ,  206  in a non-conflicting manner while accounting for one another within the multi-agent environment  200 . 
     In an exemplary embodiment, components of the external server  124  including the neural network  108  may be operably controlled by a processor  126 . The processor  126  may be configured to operably control the neural network  108  to utilize machine learning/deep learning to provide artificial intelligence capabilities that may be utilized to build a multi-agent machine learning dataset  112 . 
     With continued reference to the external server  124 , the processor  126  may be operably connected to a memory  130 . The memory  130  may store one or more operating systems, applications, associated operating system data, application data, executable data, and the like. In one embodiment, the processor  126  may be configured to process information derived from one or more agent action polices associated with the ego agent  102  and/or the target agent  104  learned by the multi-agent application  106  at one or more time steps that may be utilized to train the neural network  108  by updating the multi-agent machine learning dataset  112  stored on the memory  130 . 
     In one or more embodiments, the multi-agent machine learning dataset  112  may be configured as a dataset that includes one or more fields associated with each of the ego agent  102  and the target agent  104  with travel pathway geo-location information associated with one or more perspective pathways and vehicle dynamics data associated with particular speeds, acceleration rates, steering angles, deceleration rates, and the like that may be determined to be utilized by the ego agent  102  and/or the target agent  104  to reach the respective goals  204 ,  206  based on the learned agent action policies respectively associated with the ego agent  102  and/or the target agent  104 . 
     In one embodiment, the processor  126  of the external server  124  may additionally be configured to communicate with a communication unit  128 . The communication unit  128  may be configured to communicate through the internet cloud  122  through one or more wireless communication signals that may include, but may not be limited to Bluetooth® signals, Wi-Fi signals, ZigBee signals, Wi-Max signals, and the like. In one embodiment, the communication unit  128  may be configured to connect to the internet cloud  122  to send and receive communication signals to and from the ego agent  102  and/or the target agent  104 . In particular, the external server  124  may receive image data and LiDAR data that may be communicated by the ego agent  102  and/or the target agent  104  based on the utilization of one or more of the camera systems  116   a ,  116   b  and the vehicle laser projection systems  118   a ,  118   b . As discussed below, such data may be utilized to determine simulated multi-agent environment that pertains to the multi-agent environment  200  (real-world) and is used with respect to multi-agent recursive reasoning reinforcement learning executed by the multi-agent application  106 . 
     II. THE MULTI-AGENT RECURSIVE REASONING REINFORCEMENT LEARNING APPLICATION, RELATED METHODS, AND ILLUSTRATIVE POLICY RESULTS EXAMPLES 
     The components of the multi-agent application  106  will now be described according to an exemplary embodiment and with reference to  FIG. 1 . In an exemplary embodiment, the multi-agent application  106  may be stored on the memory  130  and executed by the processor  126  of the external server  124 . In another embodiment, the multi-agent application  106  may be stored on the storage unit  114   a  of the ego agent  102  and may be executed by the ECU  110   a  of the ego agent  102 . In some embodiments, in addition to be stored and executed by the external server  124  and/or by the ego agent  102 , the application  106  may also be executed by the ECU  110   b  of the target agent  104 . 
     The general functionality of the multi-agent application  106  will now be discussed. In an exemplary embodiment, the multi-agent application  106  may include a simulation module  132 , a policy learning module  134 , a neural network training module  136 , and an agent control module  138 . However, it is to be appreciated that the multi-agent application  106  may include one or more additional modules and/or sub-modules that are included in addition to the modules  132 - 138 . Methods and examples describing process steps that are executed by the modules  132 - 138  of the multi-agent application  106  will now be described in more detail. 
       FIG. 3  is a process flow diagram of a method  300  for receiving data associated with the multi-agent environment  200  in which the ego agent  102  and the target agent  104  are traveling and processing a simulated multi-agent environment model (simulated model) which virtually represents the multi-agent environment  200  according to an exemplary embodiment of the present disclosure.  FIG. 3  will be described with reference to the components of  FIG. 1 ,  FIG. 2 , and  FIG. 4 , though it is to be appreciated that the method of  FIG. 3  may be used with other systems/components. 
     As discussed above, the simulated model may be determined by the application  106  as a virtual representation (e.g., virtual model) of the multi-agent environment  200  to be utilized within the multi-agent central actor critic model. In particular, the simulated model may be determined by the application  106  as a virtual world model of the multi-agent environment  200  that is utilized when executing one or more iterations of Markov games to learn the agent action policy associated with the ego agent  102  and/or the agent action policy associated with the target agent  104 . 
     In an exemplary embodiment, the method  300  may begin at block  302 , wherein the method  300  may include receiving image data. In one embodiment, the simulation module  132  may communicate with the camera system  116   a  of the ego agent  102  and/or the camera system  116   b  of the target agent  104  to collect untrimmed images/video of the surrounding environment of the agents  102 ,  104 . The untrimmed images/video may include a 360 degree external views of the surrounding environments of the agents  102 ,  104  that includes the multi-agent environment  200 . 
     With reference to the illustrative example of  FIG. 2 , from the perspective of the ego agent  102 , such views may include observations of the ego agent  102  that include the target agent  104 , the goal  204  of the ego agent  102 , lanes  202   a - 202   d  included within the multi-agent environment  200 , and boundaries  212   a ,  212   b  of the multi-agent environment  200 . Additionally, from the perspective of the target agent  104 , such views may include observations of the target agent  104  that include the ego agent  102 , the goal  206  of the target agent  104 , lanes  202   a - 202   d  included within the multi-agent environment  200 , and boundaries  212   a ,  212   b  of the multi-agent environment  200 . In one embodiment, the simulation module  132  may package and store the image data received from the camera system  116   a  and/or the image data received from the camera system  116   b  on the memory  130  of the external server  124  to be further evaluated by the simulation module  132 . 
     The method  300  may proceed to block  304 , wherein the method  300  may include receiving LiDAR data. In an exemplary embodiment, the simulation module  132  may communicate with the vehicle laser projection system  118   a  of the ego agent  102  and/or the vehicle laser projection system  118   b  of the target agent  104  to collect LiDAR data that includes LiDAR based observations from the ego agent  102  and/or the target agent  104 . The LiDAR based observations may indicate the location, range, and positions of the one or more objects off which the reflected laser waves were reflected with respect to a location/position of the respective agents  102 ,  104 . 
     With reference again to  FIG. 2 , from the perspective of the ego agent  102 , the simulation module  132  may communicate with the vehicle laser projection system  118   a  of the ego agent  102  to collect ego agent LiDAR based observations that classifies sets of LiDAR coordinates that are associated with the target agent  104 , the goal  204  of the ego agent  102  and boundaries  212   a ,  212   b  of the multi-agent environment  200 . Additionally, from the perspective of the target agent  104 , the simulation module  132  may communicate with the vehicle laser projection system  118   b  of the target agent  104  to collect target agent LiDAR based observations that classifies sets of LiDAR coordinates that are associated with the ego agent  102 , the goal  206  of the ego agent  102  and boundaries  212   a ,  212   b  of the multi-agent environment  200 . In one embodiment, the simulation module  132  may package and store the ego agent LiDAR based observations received from the vehicle laser projection system  118   a  and/or the target agent LiDAR based observations received from the vehicle laser projection system  118   b  on the memory  130  of the external server  124  to be further evaluated by the simulation module  132 . 
     The method  300  may proceed to block  306 , wherein the method  300  may include fusing the image data and LiDAR data. In an exemplary embodiment, the simulation module  132  may communicate with the neural network  108  to provide artificial intelligence capabilities to conduct multimodal fusion of the image data received from the camera system  116   a  of the ego agent  102  and/or the camera system  116   b  of the target agent  104  with the LiDAR data received from the vehicle laser projection system  118   a  of the ego agent  102  and/or the vehicle laser projection system  118   b  of the target agent  104 . The simulation module  132  may aggregate the image data and the LiDAR data into fused environmental data that is associated with the multi-agent environment  200  to be evaluated further by the simulation module  132 . 
     As an illustrative example, the simulation module  132  may communicate with the neural network  108  to provide artificial intelligence capabilities to utilize one or more machine learning/deep learning fusion processes to aggregate the image data received from the camera system  116   a  of the ego agent  102  and the image data received from the camera system  116   b  of the target agent  104  into aggregated image data. Accordingly, the ego agent image based observations of the multi-agent environment  200  may be aggregated with the target agent image based observations of the multi-agent environment  200 . 
     The simulation module  132  may also utilize the neural network  108  to provide artificial intelligence capabilities to utilize one or more machine learning/deep learning fusion processes to aggregate the LiDAR data received from the vehicle laser projection system  118   a  of the ego agent  102  and the LiDAR data received from the vehicle laser projection system  118   a  of the target agent  104  into aggregated LiDAR data. Accordingly, the ego agent LiDAR based observations of the multi-agent environment  200  may be aggregated with the target agent LiDAR based observations of the multi-agent environment  200 . The simulation module  132  may additionally employ the neural network  108  to provide artificial intelligence capabilities to utilize one or more machine learning/deep learning fusion processes to aggregate the aggregated image data and the aggregated LiDAR data into fused environmental data. 
     The method  300  may proceed to block  308 , wherein the method  300  may include evaluating the fused environmental data associated with the multi-agent environment  200  and determining a simulated multi-agent environment model. In an exemplary embodiment, the simulation module  132  may communicate with the neural network  108  to utilize one or more machine learning/deep learning fusion processes to evaluate the fused environmental data to determine one or more sets of environmental coordinates that are based on the aggregated observations of the ego agent  102  and the target agent  104 . 
     The one or more sets of environmental coordinates may include positional coordinates (e.g., x, y grid world coordinates) that represent the ego agent  102 , the target agent  104 , the boundaries of the multi-agent environment  200 , respective goals  204 ,  206  associated with the ego agent  102  and the target agent  104  (defined based on the source of the image data and/or the LiDAR data), and lanes on which the ego agent  102  and the target agent  104  may travel within the multi-agent environment  200  to be utilized to process the simulated environment. 
     The one or more sets of environmental coordinates may thereby define a simulated model (e.g., virtual grid world) that is representative of the multi-agent environment  200  that includes the ego agent  102  and the target agent  104  and may be utilized to execute one or more iterations of Markov games to learn the single agent policies and multi-agent policies associated with the ego agent  102  and the target agent  104 . As discussed below, the simulated model includes a virtual ego agent that represents the ego agent  102  and a virtual target agent that represents the target agent  104  along with virtual markers that may represent respective goals  204 ,  204 , lanes  202   a - 202   d  on a roadway of the multi-agent environment  200 , and the boundaries  212   a ,  212   b  of the multi-agent environment  200 . 
     In an exemplary embodiment, upon determining the simulated model (at block  308  of the method  300 ), the simulation module  132  may communicate data pertaining to the simulated model to the policy learning module  134 . The policy learning module  134  may thereby utilize the simulated model to execute one or more iterations of stochastic games to learn the respective agent action policies associated with the ego agent  102  and the target agent  104 . 
       FIG. 4  includes an illustrative example of the simulated model  400  which virtually represents the multi-agent environment  200  according to an exemplary embodiment of the present disclosure. The simulated model  400  may be processed by the simulation module  132  of the multi-agent application  106  based on the execution of the method  300 , as discussed above. In one embodiment, the simulated model  400  may include a simulated virtual model of the ego agent  102  that is provided as a virtual ego agent  102   a  that is presented in a respective location of a simulated model that replicates the real-world surrounding environment of the ego agent  102  within the multi-agent environment  200 . The simulated model  400  may also include a virtual model of the target agent  104  that is provided as a virtual target agent  104   a  that is presented in a respective location of a simulated model  400  that replicates the real-world location of the target agent  104  within the multi-agent environment  200 . 
     As shown in  FIG. 4 , the respective goals  204 ,  206  of the ego agent  102  and the target agent  104  may also be virtually represented within the simulated model  400  as respective virtual goals  204   a ,  206   a . In one or more embodiments, the simulated model  400  may be utilized during one or more executions of the stochastic games with respect to the virtual ego agent  102   a  representing the ego agent  102  and the virtual target agent  104   a  representing the target agent  104  to learn one or more agent action policies that are respectively associated with the ego agent  102  and/or the target agent  104 . 
     In some embodiments, the simulated model  400  may also include vehicle dynamic data points (not shown) that may be interpreted by the multi-agent application  106 . The vehicle dynamic data points may be represented as a vector with real values parameters that are respectively associated with the virtual ego agent  102   a  and the virtual target agent  104   a . With respect to the virtual ego agent  102   a , the real value parameters may correspond to the speed of the virtual ego agent  102   a , the steering angle of the virtual ego agent  102   a , the acceleration rate of the virtual ego agent  102   a , the deceleration rate of the virtual ego agent  102   a , and the like. Similarly, with respect to the virtual target agent  104   a , the real value parameters may correspond to the speed of the virtual target agent  104   a , the steering angle of the virtual target agent  104   a , the acceleration rate of the virtual target agent  104   a , the deceleration rate of the virtual target agent  104   a , and the like. In one embodiment, these real value parameters may be adjusted for the ego agent  102  and/or the target agent  104  based on the training of the neural network  108  to thereby allow the ego agent  102  and the target agent  104  to reach their respective goals  204 ,  206  without any conflict amongst one another. 
       FIG. 5  is a process flow diagram of a method  500  for learning agent action policies that are to be executed to control the ego agent  102  and/or the target agent  104  to operate within the multi-agent environment  200  according to an exemplary embodiment of the present disclosure.  FIG. 5  will be described with reference to the components of  FIG. 1 ,  FIG. 2 , and  FIG. 4 , though it is to be appreciated that the method of  FIG. 5  may be used with other systems/components. The method  500  may begin at block  502 , wherein the method  500  may include receiving data associated with the simulated model  400 . 
     In an exemplary embodiment, the simulation module  132  of the multi-agent application  106  may communicate data associated with the simulated model  400  to the policy learning module  134 . The policy learning module  134  may evaluate the data and may determine observations associated with the multi-agent environment  200  from the perspective of the ego agent  102  and the target agent  104 . 
     In particular, the policy learning module  134  may evaluate the data associated with the simulated model  400  and may determine the goal  206  of the ego agent  102 , a lane  202   a  on which the ego agent  102  is traveling, additional lanes  202   b - 202   d  of the roadway on which the ego agent  102  is traveling, boundaries of the multi-agent environment  200 , and the like. Additionally, the policy learning module  134  may evaluate the data associated with the simulated model  400  and may determine the goal  204  of the target agent  104 , a lane  202   d  on which the target agent  104  is traveling, additional lanes  202   a - 202   c  of the roadway on which the target agent  104  is traveling, boundaries of the multi-agent environment  200 , and the like. 
     The policy learning module  134  may utilize such data to perform one or more executions of the Markov Games with respect to the virtual ego agent  102   a  representing the ego agent  102  and the virtual target agent  104   a  representing the target agent  104  to learn the agent action policies that are associated with the ego agent  102  and/or the target agent  104 . Accordingly, the simulated model  400  may be utilized to simulate one or more potential actions that may be performed by the virtual ego agent  102   a  and/or the virtual target agent  104   a  to independently reach their respective virtual goals  204   a ,  206   a  irrespective of one another. These independent actions may be evaluated using the multi-agent central actor critic model and level-k recursive reasoning to learn the respective agent action policies associated with the ego agent  102  and the target agent  104 . 
     The method  500  may proceed to block  504 , wherein the method  500  may include augmenting each agent  102 ,  104  with a central actor component to model its conditional response. In an exemplary embodiment, upon evaluating the data associated with the simulated model  400  and determining the virtual goals  204   a  of the virtual ego agent  102   a , the lanes  202   s - 202   d  of the roadway on which the virtual ego agent  102   a  and the virtual target agent  104   a  are traveling, boundaries of the multi-agent environment  200 , and the like, the policy learning module  134  may model the relationship between the virtual ego agent  102   a  and the virtual target agent  104   a  and consider their responses with auxiliary central actors. 
     In an exemplary embodiment, using the simulated model  400 , the Markov Game may be specified by (S,{A i } i=1   n ,T,{r i } i=1   n ,Reject,s 0 ), where n is the number of agents  102   a ,  104   a ; S is the state space containing the state for all agents  102   a ,  104   a ; A″ represent the action space for agent i (where agent i is the virtual ego agent  102   a  when determining the reward for the virtual ego agent  102   a , where agent i is the virtual target agent  104   a  when determining the reward for the virtual target agent  104   a ); T:S×Π i=1   n A i ×S→R represents the transition probability conditioned on the current state as well as the actions of all agents  102   a ,  104   a ; r i : S×Π i=1   n A i ×S→R represents the reward for agent i; and s 0 :S→R represents the initial state distribution of all agents  102   a ,  104   a.    
     In one embodiment, the learning objective for executing of the Markov Games is to get a set of polices {A i } i=1   n , where for each agent i, A i :S→A i  maps the state to its action. However, since the policy learning module  134  may determine an inherent conflict with respect to each self-interested agent that is attempting to reach their respective virtual goals  204   a ,  206   a  in a most efficient manner, the policy learning module  134  may utilize a Nash Equilibrium where all agents act in best response to each other&#39;s current strategy. In other words, by utilizing the Nash Equilibrium, the virtual ego agent  102   a  and the virtual target agent  104   a  may not perform better by unilaterally changing their own strategy. 
     Using the actor-critic framework, the policy learning module  134  may train the critic, Qθ (s, a), to estimate the return value of the state-action pair (s, a) with the loss J Q     ë   E s,α˜D [(Qθ (s,α)−{circumflex over (Q)})], where   is the replay buffer storing the exploration experiences and {circumflex over (Q)} is an empirical estimate of the return value. An actor, πφ(s), is trained to maximize the return value with the loss JA = [−Qθ(s, a)]. Additional terms like the policy entropy could also be added to Jπφ to improve the training. In one embodiment, the policy learning module  134  trains a central critic, Q ë   i (s,a i ,a −i ), for each agent i, to estimate the return value of the state and the joint—action; i.e., J Q     ë       i   =E s,a     i     ,a     −i     ˜D [(Qθ (s, a i , a −i )−{circumflex over (Q)})]. Each actor, π , is then trained to minimize the loss    [(Qθ(s, a i , a −i )]. The policy learning module  134  uses an equivalent term Q . (s, a, a − ) for each agent i. 
     Using centralized-training-decentralized execution, the centralized critic is defined as: Q i (s, a i , a −i ) and the decentralized actor is defined as: π i (a i |s). The training may be defined as: Q i (s, a i , a −i )←r+γV(s′); π i (a i |s)←argmax a     i   Q i (s, a i , a −i ). The centralized critic is thereby utilized to determine how good a particular action is to select an optimum set of actions for the virtual ego agent  102   a  and the virtual target agent  104   a.    
     Referring again to the method  500  of  FIG. 5 , the method  500  may proceed to block  506 , wherein the method  500  may include processing a recursive reasoning graph using level-k reasoning. In an exemplary, after completion of action selection using the multi-agent actor-critic model, the policy learning module  134  may utilize recursive reasoning as a process of reasoning about the other agent&#39;s reasoning process during decision making. For example, the policy learning module  134  may utilize recursive reasoning as a process of reasoning used by the virtual ego agent  102   a  about the target agent&#39;s reasoning process during decision making. This functionality may allow the virtual ego agent  102   a  to consider the potential change in strategy of the virtual target agent  104   a  instead of treating the virtual target agent  104   a  as a fixed agent. Similarly, the policy learning module  134  may utilize recursive reasoning as a process of reasoning used by the virtual target agent  104   b  about the virtual ego agent&#39;s reasoning process during decision making to allow the virtual ego agent  102   a  to consider the potential change in strategy of the virtual ego agent  102   a  instead of treating the virtual ego agent  102   a  as a fixed agent. 
     Using the level-k reasoning, the policy learning module  134  may complete various levels of reasoning that base the virtual ego agent&#39;s operational decisions on the virtual target agent&#39;s operational decisions and the virtual target agent&#39;s operational decisions on the virtual ego agent&#39;s operational decisions. In other words, the policy learning module  134  utilizes the level-k reasoning model to accomplish recursive reasoning. Using the level-k reasoning at level 0, all agents (the virtual ego agent  102   a  and the virtual target agent  104   a ) choose their respective actions based on base policies, A (0) . The policy learning module  134  executes the level-k reasoning such that at each level k, each agent chooses the best policy by assuming the others follow the level k−1 policy. For example, level 1, level 2, level k the virtual ego agent  102   a  may choose the best policy by assuming that the virtual target agent  104   a  follows the virtual ego agent&#39;s level k−1 policy. 
     In multi-agent RL, a natural level-0 policy is the agent&#39;s current policy, i.e. A i,(0) =A i . Given the actions of other agents at level k−1: a −i,(k-1) , the best level-k action for agent i should be 
         a   i,(k) =argmax  Q   i ( s,a   i   ,a   i,(k-1) )  (1)
 
     where Q .  is the estimated return of agent i. This formulation holds for general-sum games. π c,ψ   i (s, a −i ) to approximate argmax a     i    Q θ   i (s, a i , a −i ) by minimizing the loss: 
         J   π     c,ψ       i     =E   s,a     −i     ˜D,a   −i ˜π c,ψ   i [− Q   0   i ( s,a   i   ,a   −i )]  (2)s
 
     A central actor πci (s, a −i ) is introduced which learns the best response for i given state s and the other agents&#39; actions a − . 
     In an illustrative example, at level zero, the virtual ego agent  102   a  may treat the virtual target agent  104   a  as an obstacle based on its base policy A (0) . 
       Level-0:  a   i,(0) =π i ( a   i   |s )
 
     The policy learning module  134  may thereby execute level one reasoning to allow the virtual ego agent  102   a  to take into account the actions of the virtual target agent  104   a  (if the virtual ego agent  102   a  merges to reach its virtual goal  206   a ) such that the virtual target agent  104   a  may brake to allow the virtual ego agent  102   a  to merge or the virtual target agent  104   a  may itself simultaneously merge to reach its virtual goal  204   a . The policy learning module  134  may thereby execute level level-k reasoning (e.g., level two reasoning) to determine the best action that the virtual ego agent  102   a  should take based on the prior level (e.g., level one) action of the virtual target agent  104   a  as this is known based on the virtual ego agent  102   a  and the virtual target agent  104   a  both being central actors. Accordingly, Level-k: a i,(k) =argmax a     i   Q i (s, a i , a −i,(k-1) ). 
     Based on message passing between the central actors, level-k reasoning ensures that at each level k after level zero, each agent&#39;s policy takes into account the other agent&#39;s actions in determining the best action to be taken to reach the respective virtual goals  204   a ,  206   a  of the virtual ego agent  102   a  and the virtual target agent  104   a . In one embodiment, the policy learning module  134  may calculate a −i ′ k using the message passing process in a recursive reasoning graph R2G:  =( , ε). A node set  =A c   1 , A c   n  contains the central actor node for each agent  102 ,  104 , and the edge set ε contains edges between all interacting agents  102 ,  104 . Accordingly, the node set contains the central actor node for the virtual ego agent  102   a  and the central actor node for the virtual target agent  104   a  and the edge set contains edges between the virtual ego agent  102   a  and the virtual target agent  104   a.    
     An undirected fully-connected graph may be used by assuming all agents are interacting with each other. The messages in the edges are the sampled actions a i ′ k from the central actors. The initial level-0 actions are sampled from the individual policies: 
         a   i′ 0˜ A   i ( s )  (3) At each level k≥1, there is:
 
         a   i′   k˜A   c   i ( s,AGG   j∈N   ia   j′   k− 1)  (4)
 
     where AGG is the aggregation function and   is the node neighborhood function. Since concatenation may be used for AGG, AGG j∈Λ     r    i a j′ k−1 is interchangeable with a −i  in fully—connected graphs. 
     Accordingly, as discussed above, at level k, each central actor node takes the input message of a i ′ k−1 and outputs its best response k. Therefore, one complete message passing through the graph gives one-level up in the recursion. The entire calculation is differentiable with respect to a i ′ 0 through re-parameterization. Thus, it provides both the value and gradient information of the higher level responses. The computation complexity scales linearly with the agent number n and the recursion level k. 
       FIG. 6  provides an illustrative example of the recursive reasoning graph in a three-agent stochastic game. As shown, the actions of the central actors  602  a n , k may be evaluated by critics  604  to output policies  606  for each of the central actors  602  that include an optimum set of actions for the virtual ego agent  102   a  and the virtual target agent  104   a . Accordingly, level-k recursive reasoning uses the message passing process in a recursive reasoning graph in which a node set of the recursive reasoning graph contains a central actor node for each agent and an edge set contains edges between the ego agent and the target agent. Each actor node takes an input message based on a level-k policy and outputs a response based on a prior level action of an opposing agent. 
     In one configuration, each component of the recursive reasoning graph is trained such that for the central critic Q ë   i , a soft Bellman residual is adopted: 
         j   Q     ë       x     =E   D [( Q   ë   i ( s,a   1   ,a   −i )−( r   i ( s,a   i   ,a   −i )+Reject{circumflex over ( )}( s ′)))]  (5)
 
     where the next state value V(s′)Reject is estimated by: 
         {circumflex over (V)} ( s ′)= E   a     i   0˜ [ Q   ë   i  Reject( s′,a   i ,0 a   −i   ,k )−Reject ii  log  ( a   i ,0| s ′)].  (6)
 
     where Reject i  is the temperature variable trained and Q ë Reject is the delayed updated version of the critic network. The individual policy   is trained to minimize the KLdivergence with respect to the corresponding energy-based distribution represented by Q ë   i  using  a   −i,(k) : 
       π = E   a   i 0˜ [Reject i  log( ( a   i 0| s )) Q   ë   i ( s,a   i ,0 a   −i   ,k )]  (7)
 
     The central actor   is trained by the loss given in equation 2 above. The output of the recursive reasoning graph will be the agent action policy for each of the virtual ego agent  102   a  and the virtual target agent  104   a . In summary, agent action polices output based on the recursive reasoning graph may be shown as: Policy  , ∀i∈1, n. 
     In an exemplary embodiment, the policy learning module  134  may be configured to communicate data associated with agent action policy learned for the virtual ego agent  102   a  and the agent action policy learned for the virtual target agent  104   a  to the neural network training module  136  of the multi-agent application  106 . Upon receiving the data associated with the agent action polices learned by the policy learning module  134 , the neural network training module  136  may access the multi-agent machine learning dataset  112  and populate one or more fields associated with each of the ego agent  102  and/or the target agent  104  with respective agent action policies associated with the ego agent  102  and/or the target agent  104  for multiple respective time steps based on multiple executions of the recursive reasoning graph using level-k reasoning. Accordingly, the neural network  108  may be trained with multiple agent action policies for multiple time steps that may be utilized to autonomously control the ego agent  102  and/or the target agent  104  to reach their respective goals  204 ,  206  without any conflict amongst one another within the multi-agent environment  200 . 
     Referring again to the method  500  of  FIG. 5 , upon training an agent action policy to account for higher level recursion actions on interaction agents based on utilization of the recursive reasoning graph, the method  500  may proceed to block  508 , wherein the method  500  may include analyzing the multi-agent machine learning dataset  112  and implementing the agent action policy to operate the ego agent  102  and/or the target agent  104 . In an exemplary embodiment, the agent control module  138  may access the multi-agent machine learning dataset  112  and may analyze the dataset to retrieve an agent action policy that is associated with the ego agent  102  and/or an agent action policy that is associated with the target agent  104  at a particular time step. 
     The method  500  may proceed to block  510 , wherein the method  500  may include autonomously controlling the ego agent  102  and/or the target agent  104  based on the respective agent action policy. In an exemplary embodiment, upon retrieving the agent action policy that is associated with the ego agent  102  and/or an agent action policy that is associated with the target agent  104  at a particular time step, the agent control module  138  may communicate respective data associated with the agent action policy associated with the ego agent  102  and/or the agent action policy associated with the target agent  104  to the ECU  110   a  of the ego agent  102  and/or the ECU  110   b  of the target agent  104  to autonomously control the respective agent(s)  102 ,  104  based on the respective associated agent action policies. 
     In an exemplary embodiment, the agent control module  138  may analyze the agent action policy associated with the ego agent  102  and/or the target agent  104  and may thereby communicate with the ECU  110   a  of the ego agent  102  and/or the ECU  110   b  of the target agent  104  to control the ego agent  102  and/or the target agent  104  to be autonomously (or semi-autonomously) operated (e.g., driven) within the multi-agent environment  200  according to the respective multi-agent policy. The ECU(s)  110   a ,  110   b  may communicate with one or more of the respective systems/control units (not shown) to thereby control the ego agent  102  and/or the target agent  104  to thereby follow particular pathways at a respective speed(s), acceleration rate(s), steering angle(s), deceleration rate(s), and the like while maneuvering within the multi-agent environment  200  to reach the respective goals  204 ,  206  without any conflict. 
     As an illustrative example, as the ego agent  102  and target agent  104  must cross paths to reach their respective goals  204 ,  206 , as shown in  FIG. 2 , based on the autonomous control of the agents  102 ,  104  (based on the execution of respectively associated agent action policies that are based on the level-k recursive reasoning), the agents  102 ,  104  are able to successfully interact without any conflict. Accordingly, based on the associated agent action policies learned by the multi-agent application  106  and trained to the neural network  108  to be implemented, the target agent  104  may be autonomously controlled to decelerate at a particular deceleration rate to allow the ego agent  102  to merge towards its goal  206 . The target agent  104  may subsequently accelerate to efficiently reach its goal  204  after the ego agent  102  passes a potential point of overlap between the agents  102 ,  104 . It is appreciated that alternate types of autonomous controls may be followed based on alternate agent action policies that are based on the level-k recursive reasoning. 
       FIG. 7  is a process flow diagram of a method  700  for utilizing a recursive reasoning graph in multi-agent reinforcement learning according to an exemplary embodiment of the present disclosure.  FIG. 7  will be described with reference to the components of  FIG. 1 , though it is to be appreciated that the method of  FIG. 7  may be used with other systems/components. The method  700  may begin at block  702 , wherein the method  700  may include receiving data associated with an ego agent  102  and a target agent  104  that are traveling within a multi-agent environment  200 . 
     The method  700  may proceed to block  704 , wherein the method  700  may include utilizing a multi-agent actor-critic framework to analyze the data associated with the ego agent  102  and the target agent  104 . The method  700  may proceed to block  706 , wherein the method  700  may include performing level-k recursive reasoning based on the multi-agent actor-critic framework to calculate higher level recursion actions of the ego agent  102  and the target agent  104 . 
     In one embodiment, an output of the level-k recursive reasoning is used to learn an agent action policy that is associated with the ego agent  102  and an agent action policy that is associated with the target agent  104 . The method  700  may proceed to block  708 , wherein the method  700  may include controlling at least one of: the ego agent  102  and the target agent  104  to operate within the multi-agent environment  200  based on at least one of: the agent action policy that is associated with the ego agent  102  and the agent action policy that is associated with the target agent  104 . 
     It should be apparent from the foregoing description that various exemplary embodiments of the invention may be implemented in hardware. Furthermore, various exemplary embodiments may be implemented as instructions stored on a non-transitory machine-readable storage medium, such as a volatile or non-volatile memory, which may be read and executed by at least one processor to perform the operations described in detail herein. A machine-readable storage medium may include any mechanism for storing information in a form readable by a machine, such as a personal or laptop computer, a server, or other computing device. Thus, a non-transitory machine-readable storage medium excludes transitory signals but may include both volatile and non-volatile memories, including but not limited to read-only memory (ROM), random-access memory (RAM), magnetic disk storage media, optical storage media, flash-memory devices, and similar storage media. 
     It should be appreciated by those skilled in the art that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the invention. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in machine readable media and so executed by a computer or processor, whether or not such computer or processor is explicitly shown. 
     It will be appreciated that various implementations of the above-disclosed and other features and functions, or alternatives or varieties thereof, may be desirably combined into many other different systems or applications. Also that various presently unforeseen or unanticipated alternatives, modifications, variations or improvements therein may be subsequently made by those skilled in the art which are also intended to be encompassed by the following claims.