IA-MARL: Imputation Assisted Multi-Agent Reinforcement Learning for Missing Training Data

1 INTRODUCTION . Reinforcement learning ( RL ) solves many challenging problems including the game playing ( Mnih et al. , 2015 ) and the robot control ( Levine et al. , 2016 ) , which focus on the single-agent RL environment , modeled as the Markov decision process ( Sutton and Barto , 2011 ) . However , there exist many real-world problems that involve interaction among multiple agents such as multi-robot control ( Hüttenrauch et al. , 2019 ) and multiplayer games ( Silver et al. , 2017 ; Bard et al. , 2020 ) . Hence , the multi-agent reinforcement learning ( MARL ) that operates in multi-agent domain has been introduced and now becomes one of the most active and challenging RL research areas . In MARL , the decentralized approach has been used to train each agent based on its trajectory ( Tan , 1993 ) . However , it often shows unstable and low performance due to non-stationary environment and partially observable information ( Tan , 1993 ; Foerster et al. , 2017 ) that inherits from the decentralization . Specifically , as the agents evolve their policies independently , the environment becomes non-stationary , which unstabilizes training at each agent . In addition , the agent may not observe the information of other agents , which causes low performance in the cooperative or competitive environment ( Lowe et al. , 2017 ) . Recently , the centralized training with decentralized execution ( CTDE ) framework has been introduced for MARL ( Oliehoek et al. , 2008 ; Foerster et al. , 2018 ) . This can alleviate the non-stationary environment and the partially observable information problems ( Lowe et al. , 2017 ) and encourages coordination among agents ( Foerster et al. , 2018 ) . In the execution of CTDE , each agent takes an action based on its observation , while the training of the agents is performed at a centralized server after collecting observations , actions , and rewards of all agents . In existing works , those data from all agents are assumed to be available at the centralized server , which may not be always true in reality . The data from distributed agents can be unavailable due to practical reasons including the communication failure , hardware limit , and security attacks ( Lakshminarayan et al. , 1999 ; Twala , 2009 ) . For instance , in wireless sensor applications of MARL such as vehicle tracking ( Liang et al. , 2020 ) and environmental monitoring ( Li et al. , 2020 ) , sensors ( i.e. , agents ) transmit their sensed information to a receiver ( i.e. , centralized server ) for training . The training data , transmitted from sensors , can be missed when the communication is unstable . In addition , even when the training data successfully arrives at the centralized server , certain data can be removed from the training dataset due to security attacks such as false data injection ( Yan et al. , 2016 ) and unauthorized data modification ( Ferretti et al. , 2014 ) . As one can readily imagine , this missing training data can cause a serious problem in MARL as the training can not be performed . One possible solution on this missing data problem is to use only the training data that contains data from all agents without missing . However , in this case , the number of training data can dramatically decrease as the number of agents increases or the data missing happens more often , which delays the training . Another solution can be the use of imputation for replacing the missing training data in the MARL . However , the training data of this case can be different from the original data , which potentially degrades the performance . Therefore , as discussed above , the missing data problem should be carefully considered for bringing the MARL to the next level for wider range of applications . Despite of it , to the best of our knowledge , the missing data problem in MARL has not been taken into account in existing works . In this paper , we propose an imputation assisted multi-agent reinforcement learning ( IA-MARL ) with considering the missing training data problem , where the training data of each agent , consists of observation , action , and reward , can be randomly missed with certain probability . The proposed IA-MARL consists of two steps : 1 ) the imputation of missing training data and 2 ) the mask-based update of the networks . Specifically , for the imputation of the missing training data , we use a generative adversarial imputation network ( GAIN ) to impute the data from all agents , and we form the data for the training of agents . We then perform the mask-based update , which trains the value function and the policy of each agent by selectively using the training data of the corresponding agent , not missed over the consecutive times . In the experimental results , we show the IA-MARL outperforms a decentralized approach and also can achieve the performance of MARL with all training data without missing . We then also show the performance of IA-MARL for different missing probabilities , the number of agents , and the number of pre-training episodes for GAIN . From the ablation study , we also verify the importance of the mask-based update as well as the imputation accuracy in multi-agent environments when the training data can be missed . 2 RELATED WORK . Independent Q-learning has been proposed as a decentralized approach to train each agent using its own data independently ( Tan , 1993 ) . The independent Q-learning is used for the tabular environment ( Littman , 1994 ) , and deep learning-based approaches are presented in Tampuu et al . ( 2017 ) ; Gupta et al . ( 2017 ) . Independent Q-learning , however , suffers from the non-stationary environment and partially observable information problems . CTDE is one of the solutions for those problems . In CTDE , the data from all agents are used for the centralized training , while the execution at each agent only requires its observation ( Oliehoek et al. , 2008 ) . For instance , the centralized server trains the value function of each agent using observations , actions , and rewards of all agents , while each agent takes an action based on its observation ( Lowe et al. , 2017 ) . Recently , MARL algorithms that adopt CTDE framework have been presented . For instance , Lowe et al . ( 2017 ) proposes the multi-agent deep deterministic policy gradient ( MADDPG ) that extends deep deterministic policy gradient ( DDPG ) for the continuous control of multiple agents . For the credit assignment problem , the counterfactual baseline ( Foerster et al. , 2018 ) and the value function factorization that determines the contribution of each agent are used ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Son et al. , 2019 ) . To improve the performance of MARL , the soft value function and the multi-head attention are used in Iqbal and Sha ( 2019 ) , and the communication between agents that provides additional information to each agent is introduced in Foerster et al . ( 2016 ) ; Mordatch and Abbeel ( 2018 ) while the limited communication channel between agents during the execution is considered in Kim et al . ( 2019 ) to address the real world communication channel constraints in MARL . However , none of the prior work considers the missing training data problem . Imputation is the research area that replaces missing data , which has been used for many applications including medical data and image concealment ( Rubin , 2004 ) . For the imputation , some techniques such as multivariate imputation by chained equations ( MICE ) ( Buuren and Groothuis-Oudshoorn , 2010 ) , matrix completion ( Mazumder et al. , 2010 ) , and MissForest ( Stekhoven and Bühlmann , 2012 ) are proposed . However , when the imputation is used for the data that have large data space ( e.g. , the data obtained from agents in CTDE ) , the techniques with insufficient expressive power might result in low performance . For this case , some imputation techniques that have more expressiveness by adopting the deep neural networks can be more suitable such as the multiple imputation using denoising autoencoders ( MIDA ) ( Gondara and Wang , 2018 ) , the bidirectional recurrent imputation for time series ( BRITS ) ( Cao et al. , 2018 ) , and the generative adversarial imputation network ( GAIN ) ( Yoon et al. , 2018 ) . 3 BACKGROUND . We consider a decentralized partially observable Markov decision process , defined by a tuple ( S , A , P , r , Ω , O , γ , n ) , where S , A , and Ω are set of states , actions , and observations , respectively . r and γ are the reward and the discount factor , respectively , and n is the number of agents . We use s ∈ S , a ∈ A , and o ∈ Ω for a state , an action , and an observation , respectively . We use subscript i for the corresponding agent and t for the time , e.g. , oi , t is the observation of agent i at time t. We use bold symbols to denote observations , actions , and rewards of all agents , e.g. , at = ( a1 , t , · · · , an , t ) . Here , P ( st+1|st , at ) and O ( ot|st ) are the transition probability and the conditional observation probability , respectively . 3.1 DDPG AND MADDPG . The objective of the agent in the environment is to maximize the cumulative reward Rt =∑T t′=t γ t′−trt′ . For this , we use the actor-critic method . The expected cumulative reward for given action and state is Q ( st , at ) = E [ Rt|s = st , a = at ] , which is called as an action-value function or value function . Using the Bellman equation , the value function can be rewritten as Q ( st , at ) = Ert , st+1 , at+1 [ rt + γQ ( st+1 , at+1 ) ] . When the parameter θ is used for the value function approximation , the value function Q can be learned by minimizing the loss L ( θ ) , given as L ( θ ) = E [ ( Qθ ( st , at|θ ) − y ) 2 ] , y = rt + γQθ ( st+1 , at+1 ) . ( 1 ) DDPG is the widely-used choices for the policy update . The policy parameterized by φ takes state s as an input and outputs deterministic action a = µφ ( s ) in DDPG , where the gradient of φ is given as ∇φJ ( φ ) = E [ ∇φµφ ( at|st ) ∇atQθ ( st , at ) |at=µφ ( st ) ] . ( 2 ) MADDPG is an algorithm that uses DDPG in the CTDE framework ( Lowe et al. , 2017 ) . The value function in MADDPG takes observations and actions of all agents as an input . Meanwhile , the policy of each agent takes its observation as an input since the agent can access to its observation only . In CTDE , the loss for the value function and the gradient for the policy are given as L ( θi ) = E [ ( Qθi ( ot , at ) − y ) 2 ] , y = ri , t + γQθi ( ot+1 , at+1 ) , ( 3 ) ∇φiJ ( φi ) = E [ ∇φiµφi ( ai , t|oi , t ) ∇ai , tQθi ( ot , at ) |ai , t=µφi ( oi , t ) ] . ( 4 ) Here , each agent has a different value function and a policy , parameterized respectively as θi and φi .

Summary: The submission proposes a cooperative MARL problem settings in which the observation-action-reward tuples generated during training are unavailable with some non-zero probability. The submission suggests addressing this problem setting by first imputing the missing training data and what they call a mask based update. The submission presents experiments for multi-agent particle environments.