IA-MARL: Imputation Assisted Multi-Agent Reinforcement Learning for Missing Training Data

1 INTRODUCTION . Reinforcement learning ( RL ) solves many challenging problems including the game playing ( Mnih et al. , 2015 ) and the robot control ( Levine et al. , 2016 ) , which focus on the single-agent RL environment , modeled as the Markov decision process ( Sutton and Barto , 2011 ) . However , there exist many real-world problems that involve interaction among multiple agents such as multi-robot control ( Hüttenrauch et al. , 2019 ) and multiplayer games ( Silver et al. , 2017 ; Bard et al. , 2020 ) . Hence , the multi-agent reinforcement learning ( MARL ) that operates in multi-agent domain has been introduced and now becomes one of the most active and challenging RL research areas . In MARL , the decentralized approach has been used to train each agent based on its trajectory ( Tan , 1993 ) . However , it often shows unstable and low performance due to non-stationary environment and partially observable information ( Tan , 1993 ; Foerster et al. , 2017 ) that inherits from the decentralization . Specifically , as the agents evolve their policies independently , the environment becomes non-stationary , which unstabilizes training at each agent . In addition , the agent may not observe the information of other agents , which causes low performance in the cooperative or competitive environment ( Lowe et al. , 2017 ) . Recently , the centralized training with decentralized execution ( CTDE ) framework has been introduced for MARL ( Oliehoek et al. , 2008 ; Foerster et al. , 2018 ) . This can alleviate the non-stationary environment and the partially observable information problems ( Lowe et al. , 2017 ) and encourages coordination among agents ( Foerster et al. , 2018 ) . In the execution of CTDE , each agent takes an action based on its observation , while the training of the agents is performed at a centralized server after collecting observations , actions , and rewards of all agents . In existing works , those data from all agents are assumed to be available at the centralized server , which may not be always true in reality . The data from distributed agents can be unavailable due to practical reasons including the communication failure , hardware limit , and security attacks ( Lakshminarayan et al. , 1999 ; Twala , 2009 ) . For instance , in wireless sensor applications of MARL such as vehicle tracking ( Liang et al. , 2020 ) and environmental monitoring ( Li et al. , 2020 ) , sensors ( i.e. , agents ) transmit their sensed information to a receiver ( i.e. , centralized server ) for training . The training data , transmitted from sensors , can be missed when the communication is unstable . In addition , even when the training data successfully arrives at the centralized server , certain data can be removed from the training dataset due to security attacks such as false data injection ( Yan et al. , 2016 ) and unauthorized data modification ( Ferretti et al. , 2014 ) . As one can readily imagine , this missing training data can cause a serious problem in MARL as the training can not be performed . One possible solution on this missing data problem is to use only the training data that contains data from all agents without missing . However , in this case , the number of training data can dramatically decrease as the number of agents increases or the data missing happens more often , which delays the training . Another solution can be the use of imputation for replacing the missing training data in the MARL . However , the training data of this case can be different from the original data , which potentially degrades the performance . Therefore , as discussed above , the missing data problem should be carefully considered for bringing the MARL to the next level for wider range of applications . Despite of it , to the best of our knowledge , the missing data problem in MARL has not been taken into account in existing works . In this paper , we propose an imputation assisted multi-agent reinforcement learning ( IA-MARL ) with considering the missing training data problem , where the training data of each agent , consists of observation , action , and reward , can be randomly missed with certain probability . The proposed IA-MARL consists of two steps : 1 ) the imputation of missing training data and 2 ) the mask-based update of the networks . Specifically , for the imputation of the missing training data , we use a generative adversarial imputation network ( GAIN ) to impute the data from all agents , and we form the data for the training of agents . We then perform the mask-based update , which trains the value function and the policy of each agent by selectively using the training data of the corresponding agent , not missed over the consecutive times . In the experimental results , we show the IA-MARL outperforms a decentralized approach and also can achieve the performance of MARL with all training data without missing . We then also show the performance of IA-MARL for different missing probabilities , the number of agents , and the number of pre-training episodes for GAIN . From the ablation study , we also verify the importance of the mask-based update as well as the imputation accuracy in multi-agent environments when the training data can be missed . 2 RELATED WORK . Independent Q-learning has been proposed as a decentralized approach to train each agent using its own data independently ( Tan , 1993 ) . The independent Q-learning is used for the tabular environment ( Littman , 1994 ) , and deep learning-based approaches are presented in Tampuu et al . ( 2017 ) ; Gupta et al . ( 2017 ) . Independent Q-learning , however , suffers from the non-stationary environment and partially observable information problems . CTDE is one of the solutions for those problems . In CTDE , the data from all agents are used for the centralized training , while the execution at each agent only requires its observation ( Oliehoek et al. , 2008 ) . For instance , the centralized server trains the value function of each agent using observations , actions , and rewards of all agents , while each agent takes an action based on its observation ( Lowe et al. , 2017 ) . Recently , MARL algorithms that adopt CTDE framework have been presented . For instance , Lowe et al . ( 2017 ) proposes the multi-agent deep deterministic policy gradient ( MADDPG ) that extends deep deterministic policy gradient ( DDPG ) for the continuous control of multiple agents . For the credit assignment problem , the counterfactual baseline ( Foerster et al. , 2018 ) and the value function factorization that determines the contribution of each agent are used ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ; Son et al. , 2019 ) . To improve the performance of MARL , the soft value function and the multi-head attention are used in Iqbal and Sha ( 2019 ) , and the communication between agents that provides additional information to each agent is introduced in Foerster et al . ( 2016 ) ; Mordatch and Abbeel ( 2018 ) while the limited communication channel between agents during the execution is considered in Kim et al . ( 2019 ) to address the real world communication channel constraints in MARL . However , none of the prior work considers the missing training data problem . Imputation is the research area that replaces missing data , which has been used for many applications including medical data and image concealment ( Rubin , 2004 ) . For the imputation , some techniques such as multivariate imputation by chained equations ( MICE ) ( Buuren and Groothuis-Oudshoorn , 2010 ) , matrix completion ( Mazumder et al. , 2010 ) , and MissForest ( Stekhoven and Bühlmann , 2012 ) are proposed . However , when the imputation is used for the data that have large data space ( e.g. , the data obtained from agents in CTDE ) , the techniques with insufficient expressive power might result in low performance . For this case , some imputation techniques that have more expressiveness by adopting the deep neural networks can be more suitable such as the multiple imputation using denoising autoencoders ( MIDA ) ( Gondara and Wang , 2018 ) , the bidirectional recurrent imputation for time series ( BRITS ) ( Cao et al. , 2018 ) , and the generative adversarial imputation network ( GAIN ) ( Yoon et al. , 2018 ) . 3 BACKGROUND . We consider a decentralized partially observable Markov decision process , defined by a tuple ( S , A , P , r , Ω , O , γ , n ) , where S , A , and Ω are set of states , actions , and observations , respectively . r and γ are the reward and the discount factor , respectively , and n is the number of agents . We use s ∈ S , a ∈ A , and o ∈ Ω for a state , an action , and an observation , respectively . We use subscript i for the corresponding agent and t for the time , e.g. , oi , t is the observation of agent i at time t. We use bold symbols to denote observations , actions , and rewards of all agents , e.g. , at = ( a1 , t , · · · , an , t ) . Here , P ( st+1|st , at ) and O ( ot|st ) are the transition probability and the conditional observation probability , respectively . 3.1 DDPG AND MADDPG . The objective of the agent in the environment is to maximize the cumulative reward Rt =∑T t′=t γ t′−trt′ . For this , we use the actor-critic method . The expected cumulative reward for given action and state is Q ( st , at ) = E [ Rt|s = st , a = at ] , which is called as an action-value function or value function . Using the Bellman equation , the value function can be rewritten as Q ( st , at ) = Ert , st+1 , at+1 [ rt + γQ ( st+1 , at+1 ) ] . When the parameter θ is used for the value function approximation , the value function Q can be learned by minimizing the loss L ( θ ) , given as L ( θ ) = E [ ( Qθ ( st , at|θ ) − y ) 2 ] , y = rt + γQθ ( st+1 , at+1 ) . ( 1 ) DDPG is the widely-used choices for the policy update . The policy parameterized by φ takes state s as an input and outputs deterministic action a = µφ ( s ) in DDPG , where the gradient of φ is given as ∇φJ ( φ ) = E [ ∇φµφ ( at|st ) ∇atQθ ( st , at ) |at=µφ ( st ) ] . ( 2 ) MADDPG is an algorithm that uses DDPG in the CTDE framework ( Lowe et al. , 2017 ) . The value function in MADDPG takes observations and actions of all agents as an input . Meanwhile , the policy of each agent takes its observation as an input since the agent can access to its observation only . In CTDE , the loss for the value function and the gradient for the policy are given as L ( θi ) = E [ ( Qθi ( ot , at ) − y ) 2 ] , y = ri , t + γQθi ( ot+1 , at+1 ) , ( 3 ) ∇φiJ ( φi ) = E [ ∇φiµφi ( ai , t|oi , t ) ∇ai , tQθi ( ot , at ) |ai , t=µφi ( oi , t ) ] . ( 4 ) Here , each agent has a different value function and a policy , parameterized respectively as θi and φi .

Summary: Authors present a novel and effective method leveraging a generative adversarial approach to a specific MARL problem with regard to missing training data. The approach presented treats the missing data as targets of imputation, loosely similar to that of inpainting problems in computer vision, where missing pixels are "filled in". Results show that IA-MARL outperforms one baseline (MADDPG) that has not been originally devised for missing training data.