Maximizing Ensemble Diversity in Deep Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) agents trained with high capacity function approximators such a deep neural networks have shown to solve complex sequential decision-making problems , including the board games of Chess , GO and Shogi ( Silver et al. , 2016 ; 2017 ; 2018 ) , achieving super-human performance in video games ( Mnih et al. , 2015 ; Vinyals et al. , 2019 ) and solving robotic manipulation tasks ( Liu et al. , 2021 ) . Despite achieving these tremendous goals , modern deep reinforcement learning ( DRL ) algorithms have plethora of limitations . For example , it is well-known that DRL algorithms are sample-inefficient and require stupendous amount of environment interactions to learn an optimal policy ( Łukasz Kaiser et al. , 2020 ) . Additional problems encountered and exacerbates during training a DRL agent includes the overestimation bias that occurs while estimating the target values for Q-learning ( Fujimoto et al. , 2018 ; Lan et al. , 2020 ; Hado van Hasselt et al. , 2016 ) , error propagation during Bellman backup ( Kumar et al. , 2019 ) and trade-off between exploration and exploitation ( Chen et al. , 2017 ) . Recently , the use of ensemble has been a popular choice to address the above mentioned issues . These methods combine multiple neural networks to model the value functions or ( and ) the policy ( Osband et al. , 2016 ; Chen et al. , 2017 ; Lan et al. , 2020 ; Lee et al. , 2020 ) . For example , TD3 ( Fujimoto et al. , 2018 ) used two critics to address the overestimation bias problem in continuous control problems while MaxminDQN ( Lan et al. , 2020 ) provided a mechanism to use the cardinality of the ensemble to use as a knob to tune between over and under estimation in deep Q-learning . Similarly , Bootstrapped DQN ( Osband et al. , 2016 ; Chen et al. , 2017 ) used ensemble for effective exploration . ∗Partial work done while being a Ph.D student at University of Central Florida The primary insight of this paper is that the performance of ensemble based methods is contingent on maintaining sufficient diversity between the neural networks of the ensemble . If the neural networks in the ensembles converge to a common representation ( we will show that this is the case in many scenarios ) , the performance of these approaches significantly degrades . We note that even with different representations , the Q-values will still converge towards a shared optimum , but they are statistically less likely to follow the same learning trajectory elsewhere . In this paper , we propose Maximize Ensemble Diversity in Reinforcement Learning ( MED-RL ) , a set of regularization methods inspired from the economics and consensus optimization to improve diversity and to prevent the collapse of the representations in the ensemble-based deep reinforcement learning methods by encouraging inequality between the networks during training . The objective of the regularizers is solely to keep the representations different , while still allowing the models to converge to the optimal Q-value . The motivation for the regularizers came from topic of income distribution in economic theory that provides a rich source of mathematical formulations that measure inequality . While in economics , high inequality is seen as a negative , in our case we used the inequality metrics to encourage diversity between the neural networks . To summarize , our contributions are following : 1 . We empirically show that high representation similarity between neural network based Q-functions leads to degradation in performance in ensemble based Q-learning methods . 2 . To mitigate this , we propose five regularizers based on inequality measures from economics theory and consensus optimization that maximize diversity between the neural networks in ensemble based reinforcement learning methods . 3 . We integrated MED-RL in TD3 ( Fujimoto et al. , 2018 ) , SAC ( Haarnoja et al. , 2018 ) and REDQ ( Chen et al. , 2021 ) for continuous control tasks and in MaxminDQN ( Lan et al. , 2020 ) and EnsembleDQN ( Anschel et al. , 2017 ) for discrete control tasks and evaluated on six Mujoco environments and six Atari games . Our results show that MED-RL augmented algorithms outperform their un-regularized counterparts significantly and in some cases achieved more than 300 % in performance gains and are up to 75 % more sample-efficient . 4 . We also show that MED-RL augmented SAC is more sample-efficient than REDQ , an ensemble based method specifically designed for sample-efficiency , and can achieve similar performance to REDQ up to 50 times faster on wall-clock time . 2 RELATED WORK . Ensembles in Deep RL : Use of an ensemble of neural networks in Deep RL has been studied in several recent studies for different purposes . In ( Fujimoto et al. , 2018 ; Anschel et al. , 2017 ; Lan et al. , 2020 ) have used an ensemble to address the overestimation bias in deep Q-learning based methods for both continuous and discrete control tasks . Similarly , Bootstrapped DQN and extensions ( Osband et al. , 2016 ; Chen et al. , 2017 ) have leveraged ensemble of neural networks for efficient exploration . The problem of error propagation in Bellman backup was addressed in ( Kumar et al. , 2019 ) using an ensemble of neural networks . Sample efficiency , a notorious problem in RL has taken advantage from an ensemble ( Chen et al. , 2021 ) . Recently , SUNRISE ( Lee et al. , 2020 ) proposed a unified framework for ensemble-based deep reinforcement learning . Diversity in Ensembles : Diversity in neural network ensembles has been studied years before the resurgence of deep learning ( Brown , 2004 ) . Even though diversity is an important topic in neural networks , most of the studies in this topic revolve around addressing problems in supervised learning settings . More recently there has been a number of studies that have diversity in ensembles to measure and improve model uncertainty . Jain et al . ( 2020 ) have proposed a diversity regularizer to improve the uncertainty estimates in out-of-data distribution . Lee et al . ( 2015 ) have used Multiple choice Learning to learn diverse Convolutional Neural Networks for image recognition . Regularization in Reinforcement Learning : Regularization in reinforcement learning has been used to perform effective exploration and learning generalized policies . For instance , ( Grau-Moya et al. , 2019 ) uses mutual-information regularization to optimize a prior action distribution for better performance and exploration , ( Cheng et al. , 2019 ) regularizes the policy π ( a|s ) using a control prior , ( Galashov et al. , 2019 ) uses temporal difference error regularization to reduce variance in Generalized Advantage Estimation ( Schulman et al. , 2016 ) . Generalization in reinforcement learning refers to the performance of the policy on different environment compared to the training environment . For example , ( Farebrother et al. , 2018 ) studied the effect ofL2 norm on DQN on generalization , ( Tobin et al. , 2017 ) studied generalization between simulations vs. the real world , ( Pattanaik et al. , 2018 ) studied parameter variations and ( Zhang et al. , 2018 ) studied the effect of different random seeds in environment generation . Diversity in Reinforcement Learning : Diversity in reinforcement learning is active area of research . ( Pacchiano et al. , 2020 ) uses Determinantal Point Processes to promote behavioral diversity , Lupu et al . ( 2021 ) have used policy diversity to improve zero-shot coordination in multi-agent setting . ( Tang et al. , 2021 ) uses reward randomization for discovering diverse strategic policies in complex multi-agent games . In ( Li et al. , 2021 ) proposed CDS that uses information-theoretical objective to maximize the mutual information between agents ’ identities and trajectories and encourage diversity . More recently ( An et al. , 2021 ) have used diversified Q-ensembles to address overestimation in offline reinforcement learning . Representation Similarity : Measuring similarity between the representations learned by different neural networks is an active area of research . For instance , ( Raghu et al. , 2017 ) used Canonical Correlation Analysis ( CCA ) to measure the representation similarity . CCA find two basis matrices such that when original matrices are projected on these bases , the correlation is maximized . ( Raghu et al. , 2017 ; Mroueh et al. , 2015 ) used truncated singular value decomposition on the activations to make it robust for perturbations . Other work such as ( Li et al. , 2015 ) and ( Wang et al. , 2018 ) studied the correlation between the neurons in the neural networks . 3 BACKGROUND . Reinforcement learning : We consider an agent as a Markov Decision Process ( MDP ) defined as a five element tuple ( S , A , P , r , γ ) , where S is the state space , A is the action space , P : S ×A×S → [ 0 , 1 ] are the state-action transition probabilities , r : S × A × S → R is the reward mapping and γ → [ 0 , 1 ] is the discount factor . At each time step t the agent observes the state of the environment st ∈ S and selects an action at ∈ A . The effect of the action triggers a transition to a new state st+1 ∈ S according to the transition probabilities P , while the agent receives a scalar reward Rt = r ( st , at , st+1 ) . The goal of the agent is to learn a policy π that maximizes the expectation of the discounted sum of future rewards . Representation Similarity Measure : LetX ∈ Rn×p1 denote a matrix of activations of p1 neurons for n examples and Y ∈ Rn×p2 denote a matrix of activations of p2 neurons for the same n examples . Furthermore , we consider Kij = k ( xi , xj ) and Lij = l ( yi , yj ) where k and l are two kernels . Centered Kernel Alignment ( CKA ) ( Kornblith et al. , 2019 ; Cortes et al. , 2012 ; Cristianini et al. , 2002 ) is a method for comparing representations of neural networks , and identifying correspondences between layers , not only in the same network but also on different neural network architectures . CKA is a normalized form of Hilbert-Schmidt Independence Criterion ( HSIC ) ( Gretton et al. , 2005 ) . Formally , CKA is defined as : CKA ( K , L ) = HSIC ( K , L ) √ HSIC ( K , K ) · HSIC ( L , L ) HSIC is a test statistic for determining whether two sets of variables are independent . The empirical estimator of HSIC is defined as : HSIC ( K , L ) = 1 ( n− 1 ) 2 tr ( KHLH ) where H is the centering matrix Hn = In − 1 n 11T . 4 MAXIMIZE ENSEMBLE DIVERSITY IN REINFORCEMENT LEARNING . In this section , we propose MED-RL : Maximize Ensemble Diversity in Reinforcement Learning , a set of regularizers inspired from the Economics and consensus optimization to improve diversity and to prevent the collapse of the representations in the ensemble-based deep reinforcement learning methods by encouraging inequality between the networks during training . This section is organized as follows : 1 . We empirically show that high representation similarity between neural network based Q-functions leads to degradation in performance in ensemble based Q-learning methods . 2. we present the Economics theory and consensus optimization inspired regularizers with their mathematical formulation .

Summary: The paper considers a problem of ensemble-based deep RL methods that ensembles of critic networks converge to the same point in the representation space. To address it, the paper proposes a regularization technique that forces representations of a critic network to be dissimilar from those of other critic networks in the ensemble. It is empirically shown that this regularization technique improves the sample efficiency and asymptotic performance of baseline algorithms.