Preventing Value Function Collapse in Ensemble  Q-Learning by Maximizing Representation Diversity

1 INTRODUCTION . Q-learning ( Watkins , 1989 ) and its deep learning based successors inaugurated by DQN ( Mnih et al. , 2015 ) are model-free , value function based reinforcement learning algorithms . Their popularity stems from their intuitive , easy-to-implement update rule derived from the Bellman equation . At each time step , the agent updates its Q-value towards the expectation of the current reward plus the value corresponding to the maximal action in the next state . This state-action value represents the maximum sum of reward the agent believes it could obtain from the current state by taking the current action . Unfortunately ( Thrun & Schwartz , 1993 ; van Hasselt , 2010 ) have shown that this simple rule suffers from overestimation bias : due to the maximization operator in the update rule , positive and negative errors do not cancel each other out , but positive errors accumulate . The overestimation bias is particularly problematic under function approximation and have contributed towards learning sub-optimal policies ( Thrun & Schwartz , 1993 ; Szita & Lőrincz , 2008 ; Strehl et al. , 2009 ) . A possible solution is to introduce underestimation bias in the estimation of the Q-value . Double Q-learning ( van Hasselt , 2010 ) maintains two independent state-action value estimators ( Q-functions ) . The state-action value of estimator one is calculated by adding observed reward and maximal stateaction value from the other estimator . Double DQN ( Hado van Hasselt et al. , 2016 ) applied this idea using neural networks , and was shown to provide better performance than DQN . More recent actor-critic type deep RL algorithms such as TD3 ( Fujimoto et al. , 2018 ) and SAC ( Haarnoja et al. , 2018 ) also use two Q function estimators ( in combination with other techniques ) . Other approaches such as EnsembleDQN ( Anschel et al. , 2017 ) and MaxminDQN ( Lan et al. , 2020 ) maintain ensembles of Q-functions to estimate an unbiased Q-function . EnsembleDQN estimates the state-action values by adding the current observed reward and the maximal state-action value from the average of Q-functions from the ensemble . MaxminDQN creates a proxy Q-function by selecting the minimum Q-value for each action from all the Q-functions and using the maximal state-action value from the proxy Q-function to estimate an unbiased Q-function . Both EnsembleDQN and MaxminDQN have been shown to perform better than Double DQN . The primary insight of this paper is that the performance of ensemble based methods is contingent on maintaining sufficient diversity in the representation space between the Q-functions in the ensembles . If the Q-functions in the ensembles converge to a common representation ( we will show that this is the case in many scenarios ) , the performance of these approaches significantly degrades . In this paper we propose to use cross-learner regularizers to prevent the collapse of the representation space in ensemble-based Q-learning methods . Intuitively , these representations capture an inductive bias towards more diverse representations . We have investigated five different regularizers . The mathematical formulation of four of the regularizers correspond to inequality measures borrowed from economics theory . While in economics , high inequality is seen as a negative , in this case we use the metrics to encourage inequality between the representations . The fifth regularizer is inspired from consensus optimization . There is a separate line of reinforcement learning literature where ensembles are used to address several different issues ( Chen et al. , 2017 ; Chua et al. , 2018 ; Kurutach et al. , 2018 ; Lee et al. , 2020 ; Osband et al. , 2016 ) such as exploration and error propagation but we limit our solution to algorithms addressing the overestimation bias problem only . To summarize , our contributions are following : 1 . We show that high representation similarity between neural network based Q-functions leads to decline in performance in ensemble based Q-learning methods . 2 . To mitigate this , we propose five regularizers based on inequality measures from economics theory and consensus optimization that maximize representation diversity between Q-functions in ensemble based Q-learning methods . 3 . We show that applying the proposed regularizers to the MaxminDQN and EnsembleDQN methods can lead to significant improvement in performance over a variety of benchmarks . 2 BACKGROUND . Reinforcement learning considers an agent as a Markov Decision Process ( MDP ) defined as a five element tuple ( S , A , P , r , γ ) , where S is the state space , A is the action space , P : S×A×S → [ 0 , 1 ] are the state-action transition probabilities , r : S × A × S → R is the reward mapping and γ → [ 0 , 1 ] is the discount factor . At each time step t the agent observes the state of the environment st ∈ S and selects an action at ∈ A . The effect of the action triggers a transition to a new state st+1 ∈ S according to the transition probabilities P , while the agent receives a scalar reward Rt = r ( st , at , st+1 ) . The goal of the agent is to learn a policy π that maximizes the expectation of the discounted sum of future rewards . One way to implicitly learn the policy π is the Q-learning algorithm that estimates the expected sum of rewards of state st if we take the action at by solving the Bellman equation Q∗ ( st , at ) = E [ Rt + max a′∈A Q∗ ( st+1 , a ′ ) ] The implicit policy π can extracted by acting greedily with respect to the optimal Q-function : arg max a∈A Q∗ ( s , a ) . One possible way to estimate the optimal Q-value is by iteratively updating it for sampled states st and action at using Q∗ ( st , at ) ← Q∗ ( st , at ) + α ( Yt −Q∗ ( st , at ) ) where Yt = Rt + max a′∈A Q∗ ( st+1 , a ′ ) where α is the step size and Yt is called the target value . While this algorithm had been initially studied in the context of a tabular representation of Q for discrete states and actions , in many practical applications the Q value is approximated by a learned function . Since the emergence of deep learning , the preferred approximation technique is based on a deep neural network . DQN ( Mnih et al. , 2015 ) , had demonstrated super-human performance in Atari Games , but required a very large number of training iterations . From this baseline , subsequent algorithms improved both the learning speed and achievable performance , with one of the main means for this being techniques to reduce the overestimation bias of the Q-function . EnsembleDQN ( Anschel et al. , 2017 ) uses an ensemble of N neural networks to estimate state-action values and uses their average to reduce both overestimation bias and estimation variance . Formally , the target value for EnsembleDQN is calculated using QE ( · ) = 1 N N∑ i=1 Qi ( · ) Y Et = Rt + max a′∈A QE ( st+1 , a ′ ) ( 1 ) More recent , MaxminDQN ( Lan et al. , 2020 ) addresses the overestimation bias using order statistics , using the ensemble size N as a hyperparameter to tune between underestimating and overestimating bias . The target value for MaxminDQN is calculated using QM ( · , · ) = min i=1 , ... , N Qi ( · , · ) YMt = Rt + max a′∈A QM ( st+1 , a ′ ) ( 2 ) 3 RELATED WORK . Techniques to Address Overestimation Bias in RL : Addressing overestimation bias is a long standing research topic not only in reinforcement learning but other fields of science such as economics and statistics . It is commonly known as max-operator bias in statistics ( D ’ Eramo et al. , 2017 ) and as the winner ’ s curse in economics ( Thaler , 2012 ; Smith & Winkler , 2006 ) . To address this , ( van Hasselt , 2010 ) proposed Double Q-learning , subsequently adapted to a neural network based function approximators as Double DQN ( Hado van Hasselt et al. , 2016 ) . Alternatively , ( Zhang et al. , 2017 ; Lv et al. , 2019 ) proposed weighted estimators of Double Q-learning and ( Lee et al. , 2013 ) introduced a bias correction term . Other approaches to address the overestimation are based on averaging and ensembling . Techniques include averaging Q-values from previous N versions of the Q-network ( Anschel et al. , 2017 ) , taking linear combinations of min and max over the pool of Q-values ( Kumar et al. , 2019 ) , or using a random mixture from the pool ( Agarwal et al. , 2019 ) . Regularization in Reinforcement Learning : Regularization in reinforcement learning has been used to perform effective exploration and learning generalized policies . For instance , ( Grau-Moya et al. , 2019 ) uses mutual-information regularization to optimize a prior action distribution for better performance and exploration , ( Cheng et al. , 2019 ) regularizes the policy π ( a|s ) using a control prior , ( Galashov et al. , 2019 ) uses temporal difference error regularization to reduce variance in Generalized Advantage Estimation ( Schulman et al. , 2016 ) . Generalization in reinforcement learning refers to the performance of the policy on different environment compared to the training environment . For example , ( Farebrother et al. , 2018 ) studied the effect ofL2 norm on DQN on generalization , ( Tobin et al. , 2017 ) studied generalization between simulations vs. the real world , ( Pattanaik et al. , 2018 ) studied parameter variations and ( Zhang et al. , 2018 ) studied the effect of different random seeds in environment generation . Representation Similarity : Measuring similarity between the representations learned by different neural networks is an active area of research . For instance , ( Raghu et al. , 2017 ) used Canonical Correlation Analysis ( CCA ) to measure the representation similarity . CCA find two basis matrices such that when original matrices are projected on these bases , the correlation is maximized . ( Raghu et al. , 2017 ; Mroueh et al. , 2015 ) used truncated singular value decomposition on the activations to make it robust for perturbations . Other work such as ( Li et al. , 2015 ) and ( Wang et al. , 2018 ) studied the correlation between the neurons in the neural networks . 4 MAXIMIZING REPRESENTATION DIVERSITY IN ENSEMBLE-BASED DEEP Q-LEARNING . The work described in this paper is based on the conjecture that while ensemble-based deep Qlearning approaches aim to reduce the overestimation bias , this only works to the degree that the neural networks in the ensemble use diverse representations . If during training , these networks collapse to closely related representations , the learning performance decreases . From this idea , we propose to use regularization techniques to maximize representation diversity between the networks of the ensemble . 4.1 REPRESENTATION SIMILARITY MEASURE . Let X ∈ Rn×p1 denote a matrix of activations of p1 neurons for n examples and Y ∈ Rn×p2 denote a matrix of activations of p2 neurons for the same n examples . Furthermore , we consider Kij = k ( xi , xj ) and Lij = l ( yi , yj ) where k and l are two kernels . Centered Kernel Alignment ( CKA ) ( Kornblith et al. , 2019 ; Cortes et al. , 2012 ; Cristianini et al. , 2002 ) is a method for comparing representations of neural networks , and identifying correspondences between layers , not only in the same network but also on different neural network architectures . CKA is a normalized form of Hilbert-Schmidt Independence Criterion ( HSIC ) ( Gretton et al. , 2005 ) . Formally , CKA is defined as : CKA ( K , L ) = HSIC ( K , L ) √ HSIC ( K , K ) · HSIC ( L , L ) HSIC is a test statistic for determining whether two sets of variables are independent . The empirical estimator of HSIC is defined as : HSIC ( K , L ) = 1 ( n− 1 ) 2 tr ( KHLH ) where H is the centering matrix Hn = In − 1 n 11T .

Summary: This paper proposes methods to induce diversity in the networks of ensemble-based Q-Learning methods. This is achieved my maximizing a variety of measures of inequality based on the L2 parameter norms of individual networks in an ensemble. This is motivated by the benefit of having diversity in the learned features, which itself is motivated by observations on the CKA of some ensembleDQN networks.