Relational Forward Models for Multi-Agent Learning

The behavioral dynamics of multi-agent systems have a rich and orderly structure , which can be leveraged to understand these systems , and to improve how artificial agents learn to operate in them . Here we introduce Relational Forward Models ( RFM ) for multi-agent learning , networks that can learn to make accurate predictions of agents ’ future behavior in multi-agent environments . Because these models operate on the discrete entities and relations present in the environment , they produce interpretable intermediate representations which offer insights into what drives agents ’ behavior , and what events mediate the intensity and valence of social interactions . Furthermore , we show that embedding RFM modules inside agents results in faster learning systems compared to non-augmented baselines . As more and more of the autonomous systems we develop and interact with become multi-agent in nature , developing richer analysis tools for characterizing how and why agents make decisions is increasingly necessary . Moreover , developing artificial agents that quickly and safely learn to coordinate with one another , and with humans in shared environments , is crucial . 1 INTRODUCTION . The study of multi-agent systems has received considerable attention in recent years and some of the most advanced autonomous systems in the world today are multi-agent in nature ( e.g . assembly lines and warehouse management systems ) . In particular , research in multi-agent reinforcement learning ( MARL ) , where multiple learning agents perceive and act in a shared environment , has produced impressive results ( Jaderberg et al. , 2018 ; Pachocki et al. , 2018 ; Leibo et al. , 2017 ; Hughes et al. , 2018 ; Peysakhovich & Lerer , 2017a ; Lerer & Peysakhovich , 2017 ; Bansal et al. , 2017 ; Lanctot et al. , 2017 ) . One of the outstanding challenges in this domain is how to foster coordinated behavior among learning agents . In hand-engineered multi-agent systems ( e.g . assembly lines ) , it is possible to obtain coordination by design , where expert engineers carefully orchestrate each agent ’ s behavior and role in the system . This , however , rules out situations where either humans or artificial learning agents are present in the environment . In learning-based systems , there have been some successes by introducing a centralized controller ( D ’ Andrea , 2012 ; Foerster et al. , 2016 ; 2017 ; Hong et al. , 2017 ; Lowe et al. , 2017 ) . However , these can not scale to large number of agents or to mixed human-robot ensembles . There is thus an increasing focus on multi-agent systems that learn how to coordinate on their own ( Jaderberg et al. , 2018 ; Pachocki et al. , 2018 ; Perolat et al. , 2017 ) . Alongside the challenges of learning coordinated behaviors , there are also the challenges of measuring them . In learning-based systems , the analysis tools currently available to researchers focus on the functioning of each single agent , and are ill-equipped to characterize systems of diverse agents as a whole . Moreover , there has been little development of tools for measuring the contextual interdependence of agents ’ behaviors in complex environment , which will be valuable for identifying the conditions under which agents are successfully coordinating . Here we address these two challenges by developing Relational Forward Models ( RFM ) for multiagent systems . We build on recent advances in neural networks that effectively perform relational reasoning with graph networks ( GN ) ( Battaglia et al. , 2018 ) to construct models that learn to predict the forward dynamics of multi-agent systems . First , we show that our models can surpass previous top methods on this task ( Kipf et al. , 2018 ; Hoshen , 2017 ) . Perhaps more importantly , they produce intermediate representations that support the social analysis of multi-agent systems : we use our models to propose a new way to characterize what drives each agent ’ s behavior , track when agents influence each other , and identify which factors in the environment mediate the presence and valence of social interactions . Finally , we embed our models inside agents and use them to augment the host agent ’ s observations with predictions of others ’ behavior . Our results show that this leads to agents that learn to coordinate with one another faster than non-augmented baselines . 1.1 RELATED WORK . Relational reasoning has received considerable attention in recent years and researchers have developed deep learning models that operate on graphs , rather than vectors or images , and structure their computations accordingly . These methods have been successfully applied to learning the forward dynamics of systems comprised of multiple entities and a rich relational structure , like physics simulation , multi-object scenes , visual question answering and motion-capture data ( Scarselli et al. , 2009 ; Battaglia et al. , 2016 ; Raposo et al. , 2017 ; Santoro et al. , 2017 ; Gilmer et al. , 2017 ; Watters et al. , 2017 ; Kipf et al. , 2018 ; Zambaldi et al. , 2018 ) . Recently , this class of methods have been shown to successfully predict the forward dynamics of multi-agent systems , like basketball or soccer games , and to some extent , to provide insights into the relational and social structures present in the data ( Hoshen , 2017 ; Kipf et al. , 2018 ; Zhan et al. , 2018 ; Zheng et al. , 2017 ) . With the recent renaissance of deep-learning methods in general , and of deep reinforcement learning ( RL ) in particular , considerable attention has been devoted to developing analysis tools that allow researchers to understand what drives agents ’ behavior , what are the most common failure modes and provide insights into the inner workings of learning systems ( Zeiler & Fergus , 2013 ; Yosinski et al. , 2015 ; Olah et al. , 2017 ; Morcos et al. , 2018 ) . In particular , Rabinowitz et al . ( 2018 ) embed entire behavioral trajectories of single RL agents as points in an unstructured , high-dimensional space , relying on the inherent structure of the data to yield interpretable representations of whole behavioral motifs . Finally , coordination in multi-agent system has been a topic of major interest as of late and some of the most advanced MARL systems rely on the emergence of coordination among teammates to complete the task at hand ( Jaderberg et al. , 2018 ; Pachocki et al. , 2018 ) . Despite these recent successes , coordination is still considered a hard problem and several attempts have been made to promote the emergence of coordination by relaxing some assumptions ( Foerster et al. , 2016 ; 2017 ; Sukhbaatar et al. , 2016 ; Raileanu et al. , 2018 ; He et al. , 2016a ) . Here we show that , by embedding RFM modules in RL agents , they can learn to coordinate with one another faster than baseline agents , analogous to imagination-augmented agents in single-agent RL settings ( Hamrick et al. , 2017 ; Pascanu et al. , 2017 ; Weber et al. , 2017 ) . 2 RELATIONAL ANALYSIS OF MARL SYSTEMS . 2.1 METHODS . Our RFM is based on graph networks ( GN ) ( Battaglia et al. , 2018 ) , and is trained by supervised learning to predict the dynamics of multi-agent systems . Our model takes as input a semantic description of the state of the environment , and outputs either an action prediction for each agent , or a prediction of the cumulative reward each agent will receive until the end of the episode . We show that our model performs well at these tasks , and crucially , produces interpretable intermediate representations that are useful both as analysis tools , and as inputs to artificial agents who can exploit these predictions to improve their decision-making . GN encoder Graph GRU GN decoder Gtout < latexit sha1_base64= '' Fm4Iz64LvWn082kf2+Dq6sd1g9I= '' > AAAB+3icbVDLSsNAFJ34rPUV69LNYBFclUQEXbgouNBlBfuANobJdNIOnWTCzI1YQn7FjQtF3Poj7vwbJ20W2npg4HDOvdwzJ0gE1+A439bK6tr6xmZlq7q9s7u3bx/UOlqmirI2lUKqXkA0EzxmbeAgWC9RjESBYN1gcl343UemNJfxPUwT5kVkFPOQUwJG8u3azQP42SAiMFZRJlPIc9+uOw1nBrxM3JLUUYmWb38NhpKmEYuBCqJ133US8DKigFPB8uog1SwhdEJGrG9oTCKmvWyWPccnRhniUCrzYsAz9fdGRiKtp1FgJouQetErxP+8fgrhpZfxOEmBxXR+KEwFBomLIvCQK0ZBTA0hVHGTFdMxUYSCqatqSnAXv7xMOmcN12m4d+f15lVZRwUdoWN0ilx0gZroFrVQG1H0hJ7RK3qzcuvFerc+5qMrVrlziP7A+vwB7zuU/g== < /latexit > < latexit sha1_base64= '' Fm4Iz64LvWn082kf2+Dq6sd1g9I= '' > AAAB+3icbVDLSsNAFJ34rPUV69LNYBFclUQEXbgouNBlBfuANobJdNIOnWTCzI1YQn7FjQtF3Poj7vwbJ20W2npg4HDOvdwzJ0gE1+A439bK6tr6xmZlq7q9s7u3bx/UOlqmirI2lUKqXkA0EzxmbeAgWC9RjESBYN1gcl343UemNJfxPUwT5kVkFPOQUwJG8u3azQP42SAiMFZRJlPIc9+uOw1nBrxM3JLUUYmWb38NhpKmEYuBCqJ133US8DKigFPB8uog1SwhdEJGrG9oTCKmvWyWPccnRhniUCrzYsAz9fdGRiKtp1FgJouQetErxP+8fgrhpZfxOEmBxXR+KEwFBomLIvCQK0ZBTA0hVHGTFdMxUYSCqatqSnAXv7xMOmcN12m4d+f15lVZRwUdoWN0ilx0gZroFrVQG1H0hJ7RK3qzcuvFerc+5qMrVrlziP7A+vwB7zuU/g== < /latexit > < latexit sha1_base64= '' Fm4Iz64LvWn082kf2+Dq6sd1g9I= '' > AAAB+3icbVDLSsNAFJ34rPUV69LNYBFclUQEXbgouNBlBfuANobJdNIOnWTCzI1YQn7FjQtF3Poj7vwbJ20W2npg4HDOvdwzJ0gE1+A439bK6tr6xmZlq7q9s7u3bx/UOlqmirI2lUKqXkA0EzxmbeAgWC9RjESBYN1gcl343UemNJfxPUwT5kVkFPOQUwJG8u3azQP42SAiMFZRJlPIc9+uOw1nBrxM3JLUUYmWb38NhpKmEYuBCqJ133US8DKigFPB8uog1SwhdEJGrG9oTCKmvWyWPccnRhniUCrzYsAz9fdGRiKtp1FgJouQetErxP+8fgrhpZfxOEmBxXR+KEwFBomLIvCQK0ZBTA0hVHGTFdMxUYSCqatqSnAXv7xMOmcN12m4d+f15lVZRwUdoWN0ilx0gZroFrVQG1H0hJ7RK3qzcuvFerc+5qMrVrlziP7A+vwB7zuU/g== < /latexit > < latexit sha1_base64= '' Fm4Iz64LvWn082kf2+Dq6sd1g9I= '' > AAAB+3icbVDLSsNAFJ34rPUV69LNYBFclUQEXbgouNBlBfuANobJdNIOnWTCzI1YQn7FjQtF3Poj7vwbJ20W2npg4HDOvdwzJ0gE1+A439bK6tr6xmZlq7q9s7u3bx/UOlqmirI2lUKqXkA0EzxmbeAgWC9RjESBYN1gcl343UemNJfxPUwT5kVkFPOQUwJG8u3azQP42SAiMFZRJlPIc9+uOw1nBrxM3JLUUYmWb38NhpKmEYuBCqJ133US8DKigFPB8uog1SwhdEJGrG9oTCKmvWyWPccnRhniUCrzYsAz9fdGRiKtp1FgJouQetErxP+8fgrhpZfxOEmBxXR+KEwFBomLIvCQK0ZBTA0hVHGTFdMxUYSCqatqSnAXv7xMOmcN12m4d+f15lVZRwUdoWN0ilx0gZroFrVQG1H0hJ7RK3qzcuvFerc+5qMrVrlziP7A+vwB7zuU/g== < /latexit > Gt 1hid < latexit sha1_base64= '' TTYaLKTJF7FnWkmBKBRepTsjYHc= '' > AAAB/3icbVDLSsNAFJ34rPUVFdy4GSyCG0sigi5cFFzosoJ9QBvDZDJph85MwsxEKDELf8WNC0Xc+hvu/BsnbRbaemDgcM693DMnSBhV2nG+rYXFpeWV1cpadX1jc2vb3tltqziVmLRwzGLZDZAijArS0lQz0k0kQTxgpBOMrgq/80CkorG40+OEeBwNBI0oRtpIvr1/fZ/pEzf3sz5Heih5NqRhnvt2zak7E8B54pakBko0ffurH8Y45URozJBSPddJtJchqSlmJK/2U0UShEdoQHqGCsSJ8rJJ/hweGSWEUSzNExpO1N8bGeJKjXlgJouQatYrxP+8XqqjCy+jIkk1EXh6KEoZ1DEsyoAhlQRrNjYEYUlNVoiHSCKsTWVVU4I7++V50j6tu07dvT2rNS7LOirgAByCY+CCc9AAN6AJWgCDR/AMXsGb9WS9WO/Wx3R0wSp39sAfWJ8/b2WWWQ== < /latexit > < latexit sha1_base64= '' TTYaLKTJF7FnWkmBKBRepTsjYHc= '' > AAAB/3icbVDLSsNAFJ34rPUVFdy4GSyCG0sigi5cFFzosoJ9QBvDZDJph85MwsxEKDELf8WNC0Xc+hvu/BsnbRbaemDgcM693DMnSBhV2nG+rYXFpeWV1cpadX1jc2vb3tltqziVmLRwzGLZDZAijArS0lQz0k0kQTxgpBOMrgq/80CkorG40+OEeBwNBI0oRtpIvr1/fZ/pEzf3sz5Heih5NqRhnvt2zak7E8B54pakBko0ffurH8Y45URozJBSPddJtJchqSlmJK/2U0UShEdoQHqGCsSJ8rJJ/hweGSWEUSzNExpO1N8bGeJKjXlgJouQatYrxP+8XqqjCy+jIkk1EXh6KEoZ1DEsyoAhlQRrNjYEYUlNVoiHSCKsTWVVU4I7++V50j6tu07dvT2rNS7LOirgAByCY+CCc9AAN6AJWgCDR/AMXsGb9WS9WO/Wx3R0wSp39sAfWJ8/b2WWWQ== < /latexit > < latexit sha1_base64= '' TTYaLKTJF7FnWkmBKBRepTsjYHc= '' > AAAB/3icbVDLSsNAFJ34rPUVFdy4GSyCG0sigi5cFFzosoJ9QBvDZDJph85MwsxEKDELf8WNC0Xc+hvu/BsnbRbaemDgcM693DMnSBhV2nG+rYXFpeWV1cpadX1jc2vb3tltqziVmLRwzGLZDZAijArS0lQz0k0kQTxgpBOMrgq/80CkorG40+OEeBwNBI0oRtpIvr1/fZ/pEzf3sz5Heih5NqRhnvt2zak7E8B54pakBko0ffurH8Y45URozJBSPddJtJchqSlmJK/2U0UShEdoQHqGCsSJ8rJJ/hweGSWEUSzNExpO1N8bGeJKjXlgJouQatYrxP+8XqqjCy+jIkk1EXh6KEoZ1DEsyoAhlQRrNjYEYUlNVoiHSCKsTWVVU4I7++V50j6tu07dvT2rNS7LOirgAByCY+CCc9AAN6AJWgCDR/AMXsGb9WS9WO/Wx3R0wSp39sAfWJ8/b2WWWQ== < /latexit > < latexit sha1_base64= '' TTYaLKTJF7FnWkmBKBRepTsjYHc= '' > AAAB/3icbVDLSsNAFJ34rPUVFdy4GSyCG0sigi5cFFzosoJ9QBvDZDJph85MwsxEKDELf8WNC0Xc+hvu/BsnbRbaemDgcM693DMnSBhV2nG+rYXFpeWV1cpadX1jc2vb3tltqziVmLRwzGLZDZAijArS0lQz0k0kQTxgpBOMrgq/80CkorG40+OEeBwNBI0oRtpIvr1/fZ/pEzf3sz5Heih5NqRhnvt2zak7E8B54pakBko0ffurH8Y45URozJBSPddJtJchqSlmJK/2U0UShEdoQHqGCsSJ8rJJ/hweGSWEUSzNExpO1N8bGeJKjXlgJouQatYrxP+8XqqjCy+jIkk1EXh6KEoZ1DEsyoAhlQRrNjYEYUlNVoiHSCKsTWVVU4I7++V50j6tu07dvT2rNS7LOirgAByCY+CCc9AAN6AJWgCDR/AMXsGb9WS9WO/Wx3R0wSp39sAfWJ8/b2WWWQ== < /latexit > Gthid < latexit sha1_base64= '' N3DONKaVTWiaOJQfnkEeGTR4488= '' > AAAB+3icbVDLSsNAFL3xWesr1qWbYBFclUQEXbgouNBlBfuANobJZNoOnZmEmYlYQn7FjQtF3Poj7vwbJ20W2npg4HDOvdwzJ0wYVdp1v62V1bX1jc3KVnV7Z3dv3z6odVScSkzaOGax7IVIEUYFaWuqGeklkiAeMtINJ9eF330kUtFY3OtpQnyORoIOKUbaSIFdu3nQQTbgSI8lz8Y0yvPArrsNdwZnmXglqUOJVmB/DaIYp5wIjRlSqu+5ifYzJDXFjOTVQapIgvAEjUjfUIE4UX42y547J0aJnGEszRPamam/NzLElZry0EwWIdWiV4j/ef1UDy/9jIok1UTg+aFhyhwdO0URTkQlwZpNDUFYUpPVwWMkEdamrqopwVv88jLpnDU8t+HdndebV2UdFTiCYzgFDy6gCbfQgjZgeIJneIU3K7derHfrYz66YpU7h/AH1ucPuc+U2w== < /latexit > < latexit sha1_base64= '' N3DONKaVTWiaOJQfnkEeGTR4488= '' > AAAB+3icbVDLSsNAFL3xWesr1qWbYBFclUQEXbgouNBlBfuANobJZNoOnZmEmYlYQn7FjQtF3Poj7vwbJ20W2npg4HDOvdwzJ0wYVdp1v62V1bX1jc3KVnV7Z3dv3z6odVScSkzaOGax7IVIEUYFaWuqGeklkiAeMtINJ9eF330kUtFY3OtpQnyORoIOKUbaSIFdu3nQQTbgSI8lz8Y0yvPArrsNdwZnmXglqUOJVmB/DaIYp5wIjRlSqu+5ifYzJDXFjOTVQapIgvAEjUjfUIE4UX42y547J0aJnGEszRPamam/NzLElZry0EwWIdWiV4j/ef1UDy/9jIok1UTg+aFhyhwdO0URTkQlwZpNDUFYUpPVwWMkEdamrqopwVv88jLpnDU8t+HdndebV2UdFTiCYzgFDy6gCbfQgjZgeIJneIU3K7derHfrYz66YpU7h/AH1ucPuc+U2w== < /latexit > < latexit sha1_base64= '' N3DONKaVTWiaOJQfnkEeGTR4488= '' > AAAB+3icbVDLSsNAFL3xWesr1qWbYBFclUQEXbgouNBlBfuANobJZNoOnZmEmYlYQn7FjQtF3Poj7vwbJ20W2npg4HDOvdwzJ0wYVdp1v62V1bX1jc3KVnV7Z3dv3z6odVScSkzaOGax7IVIEUYFaWuqGeklkiAeMtINJ9eF330kUtFY3OtpQnyORoIOKUbaSIFdu3nQQTbgSI8lz8Y0yvPArrsNdwZnmXglqUOJVmB/DaIYp5wIjRlSqu+5ifYzJDXFjOTVQapIgvAEjUjfUIE4UX42y547J0aJnGEszRPamam/NzLElZry0EwWIdWiV4j/ef1UDy/9jIok1UTg+aFhyhwdO0URTkQlwZpNDUFYUpPVwWMkEdamrqopwVv88jLpnDU8t+HdndebV2UdFTiCYzgFDy6gCbfQgjZgeIJneIU3K7derHfrYz66YpU7h/AH1ucPuc+U2w== < /latexit > < latexit sha1_base64= '' N3DONKaVTWiaOJQfnkEeGTR4488= '' > AAAB+3icbVDLSsNAFL3xWesr1qWbYBFclUQEXbgouNBlBfuANobJZNoOnZmEmYlYQn7FjQtF3Poj7vwbJ20W2npg4HDOvdwzJ0wYVdp1v62V1bX1jc3KVnV7Z3dv3z6odVScSkzaOGax7IVIEUYFaWuqGeklkiAeMtINJ9eF330kUtFY3OtpQnyORoIOKUbaSIFdu3nQQTbgSI8lz8Y0yvPArrsNdwZnmXglqUOJVmB/DaIYp5wIjRlSqu+5ifYzJDXFjOTVQapIgvAEjUjfUIE4UX42y547J0aJnGEszRPamam/NzLElZry0EwWIdWiV4j/ef1UDy/9jIok1UTg+aFhyhwdO0URTkQlwZpNDUFYUpPVwWMkEdamrqopwVv88jLpnDU8t+HdndebV2UdFTiCYzgFDy6gCbfQgjZgeIJneIU3K7derHfrYz66YpU7h/AH1ucPuc+U2w== < /latexit > Gtin < latexit sha1_base64= '' poANnJCpZrUCRM6QZRfO85b+M60= '' > AAAB/HicbVDLSsNAFJ34rPUV7dJNsAiuSiKCLlwUXOiygn1AG8NkOmmHzkzCzI1QQvwVNy4UceuHuPNvnLRZaOuBgcM59zLnnjDhTIPrflsrq2vrG5uVrer2zu7evn1w2NFxqghtk5jHqhdiTTmTtA0MOO0limIRctoNJ9eF332kSrNY3sM0ob7AI8kiRjAYKbBrNw8Z5EE2EBjGSmRM5nlg192GO4OzTLyS1FGJVmB/DYYxSQWVQDjWuu+5CfgZVsAIp3l1kGqaYDLBI9o3VGJBtZ/NwufOiVGGThQr8yQ4M/X3RoaF1lMRmskio170CvE/r59CdOmbg5IUqCTzj6KUOxA7RRPOkClKgE8NwUQxk9UhY6wwAdNX1ZTgLZ68TDpnDc9teHfn9eZVWUcFHaFjdIo8dIGa6Ba1UBsRNEXP6BW9WU/Wi/VufcxHV6xyp4b+wPr8AdHQlX8= < /latexit > < latexit sha1_base64= '' poANnJCpZrUCRM6QZRfO85b+M60= '' > AAAB/HicbVDLSsNAFJ34rPUV7dJNsAiuSiKCLlwUXOiygn1AG8NkOmmHzkzCzI1QQvwVNy4UceuHuPNvnLRZaOuBgcM59zLnnjDhTIPrflsrq2vrG5uVrer2zu7evn1w2NFxqghtk5jHqhdiTTmTtA0MOO0limIRctoNJ9eF332kSrNY3sM0ob7AI8kiRjAYKbBrNw8Z5EE2EBjGSmRM5nlg192GO4OzTLyS1FGJVmB/DYYxSQWVQDjWuu+5CfgZVsAIp3l1kGqaYDLBI9o3VGJBtZ/NwufOiVGGThQr8yQ4M/X3RoaF1lMRmskio170CvE/r59CdOmbg5IUqCTzj6KUOxA7RRPOkClKgE8NwUQxk9UhY6wwAdNX1ZTgLZ68TDpnDc9teHfn9eZVWUcFHaFjdIo8dIGa6Ba1UBsRNEXP6BW9WU/Wi/VufcxHV6xyp4b+wPr8AdHQlX8= < /latexit > < latexit sha1_base64= '' poANnJCpZrUCRM6QZRfO85b+M60= '' > AAAB/HicbVDLSsNAFJ34rPUV7dJNsAiuSiKCLlwUXOiygn1AG8NkOmmHzkzCzI1QQvwVNy4UceuHuPNvnLRZaOuBgcM59zLnnjDhTIPrflsrq2vrG5uVrer2zu7evn1w2NFxqghtk5jHqhdiTTmTtA0MOO0limIRctoNJ9eF332kSrNY3sM0ob7AI8kiRjAYKbBrNw8Z5EE2EBjGSmRM5nlg192GO4OzTLyS1FGJVmB/DYYxSQWVQDjWuu+5CfgZVsAIp3l1kGqaYDLBI9o3VGJBtZ/NwufOiVGGThQr8yQ4M/X3RoaF1lMRmskio170CvE/r59CdOmbg5IUqCTzj6KUOxA7RRPOkClKgE8NwUQxk9UhY6wwAdNX1ZTgLZ68TDpnDc9teHfn9eZVWUcFHaFjdIo8dIGa6Ba1UBsRNEXP6BW9WU/Wi/VufcxHV6xyp4b+wPr8AdHQlX8= < /latexit > < latexit sha1_base64= '' poANnJCpZrUCRM6QZRfO85b+M60= '' > AAAB/HicbVDLSsNAFJ34rPUV7dJNsAiuSiKCLlwUXOiygn1AG8NkOmmHzkzCzI1QQvwVNy4UceuHuPNvnLRZaOuBgcM59zLnnjDhTIPrflsrq2vrG5uVrer2zu7evn1w2NFxqghtk5jHqhdiTTmTtA0MOO0limIRctoNJ9eF332kSrNY3sM0ob7AI8kiRjAYKbBrNw8Z5EE2EBjGSmRM5nlg192GO4OzTLyS1FGJVmB/DYYxSQWVQDjWuu+5CfgZVsAIp3l1kGqaYDLBI9o3VGJBtZ/NwufOiVGGThQr8yQ4M/X3RoaF1lMRmskio170CvE/r59CdOmbg5IUqCTzj6KUOxA7RRPOkClKgE8NwUQxk9UhY6wwAdNX1ZTgLZ68TDpnDc9teHfn9eZVWUcFHaFjdIo8dIGa6Ba1UBsRNEXP6BW9WU/Wi/VufcxHV6xyp4b+wPr8AdHQlX8= < /latexit > ( a ) ( b ) Agent RFM 횷 Teammate Action Prediction Ego-centric Observations Action Reward Environment Environment Graph Teammate ActionStop Gradient Augmented Observations + ( c ) Figure 1 : ( a ) The RFM module stacks a GN Encoder , a Graph GRU and a GN Decoder to obtain a relational reasoning module that holds state information across time steps . ( b ) Example of an environment graph representation . Edges connect agents ( magenta and orange ) to all entities and are color-coded according to the identity of the receiver . ( c ) RFM-augmented agents , the output of the the RFM module is appended to the original observation input to the policy network . The on-board RFM module is trained with full supervision and alongside the policy network . 2.1.1 RELATIONAL FORWARD MODELS AND BASELINES ARCHITECTURES . A GN is a neural network that operates on graphs . The input to a GN is a directed graph , ( u , V , E ) , where u ∈ Rdu is a graph-level attribute vector ( e.g . the score in a football game ) , V = { vi } i=1 : nv is a set of vertices ( e.g . the players ) with attributes vi ∈ Rdv ( e.g the players ’ ( x , y ) coordinates on the pitch ) , and E = { ( ek , rk , sk ) } k=1 : ne is a set of directed edges which connect sender vertex , vsk to receiver vertex vrk and have attribute ek ∈ Rde ( e.g . same team or opponent ) . The output of a GN is also a graph , with the same connectivity structure as the input graph ( i.e. , same number of vertices and edges , as well as same sender and receiver for each edge ) , but updated global , vertex , and edge attributes . See Fig . 1b for an example graph . The sequence of computations in a GN proceed by updating the edge attributes , followed by the vertex attributes , and finally the global attributes . These computations are implemented via three “ update ” functions ( the φs ) and three “ aggregation ” functions ( the ρs ) , e′k = φ e ( ek , vrk , vsk , u ) , v′i = φ v ( ē′i , vi , u ) , u′ = φu ( ē′ , v̄′ , u ) , ē′i = ρ e→v ( E′i ) , v̄′ = ρv→u ( V ′ ) , ē′ = ρe→u ( E′ ) ( 1 ) where E′i = { ( e′k , rk , sk ) } rk=i . The edges are updated by φe , as a function of the sender vertex , receiver vertex , edge , and global attributes . We term the updated edge attribute the “ message ” , e′k . Next , each vertex , i , is updated by aggregating the e′k messages for which i = rk , and computing the updated vertex attribute , v′i , as a function ( φ v ) of these aggregated messages , as well as the current vertex and global attributes . Finally , the global attributes are updated , by φu , as a function of the current global attribute and all aggregated e′k and v ′ i attributes . Since a GN takes as input a graph and outputs a graph , GN blocks can be composed to form more complex , and powerful , architectures . These architecture can also be made recurrent in time by introducing a state graph , and using recurrent neural networks ( RNNs ) as the φ functions . GN-based architectures can be optimized with respect to some objective function by gradient descent ( using backpropagation through time for recurrent implementations ) . Here we focus on supervised learning using datasets of input-output pairs . See ( Battaglia et al. , 2018 ) for further details . We construct our RFM architecture by arranging three GN blocks as in Fig . 1a . We selected this specific architecture to allow our model to perform relational reasoning steps both on the raw input data , before time recurrence is included , and then again on the output of our time recurrent block . This allows the recurrent block to construct memories of the relations between entities and not simply of their current state . Architecture details are as follows : input graphs Gtin go through a GN encoder block , a basic GN module whose φv , φe and φu are three separate 64-unit MLPs , with 1 hidden layer , and ReLU activations and whose ρ functions are summations . The output of the GN encoder block is used , in conjunction with a state graph Gt−1hid , in a “ GraphGRU ” , where each φ function is a Gated Recurrent Unit ( GRU ) ( Cho et al. , 2014 ) with a hidden state size of 32 for each of vertices , edges and globals . The GraphGRU ’ s output is then copied into a state graph and an output graph . The state graph is used in the following time step , while the output graph is passed through a GN decoder block . This last block ’ s structure has an identical to the GN encoder ’ s , and outputs the model ’ s predictions ( e.g . the actions of each agent ) . We compared the prediction performance of our RFM module to two state-of-the-art relational reasoning baselines : Neural Relational Inference networks ( Kipf et al. , 2018 ) and Vertex Attention Interaction Networks ( Hoshen , 2017 ) . These architectures are similar to our RFM module . In particular , NRI models operate on graph structured data and , with the exception that the graph connectivity map is not given , but rather estimated from trajectories using an auto-encoder architecture , they are identical to our model . VAIN networks are essentially single feed-forward GN blocks where the φe and ρe→v functions take particular and restricted form : φe ( ek , vrk , vsk , u ) = e ‖a ( vrk ) −a ( vsk ) ‖ 2 and ρe→v ( E′i ) = vi ∑ sk e′k , with a ( · ) a learnable function . We also compared our full RFM against ablated variants , which allowed us to measure the importance of the relational reasoning component , and of time recurrence . In particular we considered a Feedforward model , which had no GraphGRU block , and a No-relation model , which was a full fledged RFM module but operated on graphs with only self-connections ( i.e. , edges where si = ri ) . Finally , we included a vector-based MLP + LSTM model among our baselines , so as to highlight the advantage of using graphs over vector based modules . This last model operated on the concatenation of the vertex attributes and had a standard Encoder MLP ( 64-units ) , LSTM ( 32-hidden units ) , Decoder MLP ( 2 hidden layers , 32-units each ) architecture . We matched all models for capacity ( with the exception of NRI which has about 3x more parameters than other models because of its autoencoder connectivity map estimator ) . Models were within 3 % of each other in terms of number of parameters ( as reported by the TensorFlow checkpoint loader ) .

Summary: This paper used graph neural networks to do relational reasoning of multi-agent systems to predict the actions and returns of MARL agents that they call Relational Forward Modeling. They used RFM to analyze and assess the coordination between agents in three different multi-agent environments. They then constructed an RFM-aumented RL agent and showed improved training speeds over non relational reasoning baseline methods.