Deep Randomized Least Squares Value Iteration

1 INTRODUCTION . In Reinforcement Learning ( RL ) , an agent seeks to maximize the cumulative rewards obtained from interactions with an unknown environment ( Sutton et al. , 1998 ) . Since the agent can learn only by its interactions with the environment , it faces the exploration-exploitation dilemma : Should it take actions that will maximize the rewards based on its current knowledge or instead take actions to potentially improve its knowledge in the hope of achieving better future performance . Thus , to find the optimal policy the agent needs to use an appropriate exploration strategy . Classic RL algorithms were designed to face problems in the tabular settings where a table containing a value for each state-action pair can be stored in the computer ’ s memory . For more general settings , where generalization is required , a common practice is to use hand-designed state representation ( or state-action ) , upon which a function approximation can be learned to represent the value for each state and action . RL algorithms based on linear function approximation have demonstrated stability , data efficiency and enjoys convergence guarantees under mild assumptions ( Tsitsiklis & Van Roy , 1997 ; Lagoudakis & Parr , 2003 ) . They require that the desired learned function , e.g . Qfunction , will be a linear combination of the state representation . This is , of course , a hard constraint as the representation is hand-designed , where the designer often does not know how the optimal value-function will look like . Furthermore , hand-designed representation is environment-specific and requires re-designing for every new environment . The DQN algorithm ( Mnih et al. , 2015 ) has changed RL . Using Deep Neural Networks ( DNN ) as function approximators , the DQN algorithm enabled the learning of policies directly from raw highdimensional data and led to unprecedented achievements over a wide variety of domains ( Mnih et al. , 2015 ) . Over the years , many improvements to DQN were presented , suggesting more fitting network architectures ( Wang et al. , 2015 ) , reducing overestimation ( Van Hasselt et al. , 2016 ; Anschel et al. , 2017 ) or improving its data efficiency ( Schaul et al. , 2015 ) . Despite its great success , DQN uses the overly simple -greedy strategy for exploration . This strategy is one of the simplest exploration strategies that currently exist . The agent takes random action with probability and takes the optimal action according to its current belief with probability 1− . This strategy is commonly used despite its simplicity and proven inefficiency ( Osband et al. , 2016 ) . The main shortcoming of -greedy and similar strategies derives from the fact that they do not use observed data to improve exploration . To explore , it takes a completely random action , regardless of the experience obtained by the agent . Thompson Sampling ( TS ) ( Thompson , 1933 ) , is one of the oldest heuristics to address the ’ exploration/exploitation ’ trade-off in sequential decision-making problems . Its variations were proposed in RL ( Wyatt , 1998 ; Strens , 2000 ) and various bandits settings ( Chapelle & Li , 2011 ; Scott , 2010 ) . For Multi-Armed Bandit ( MAB ) problems , TS is very effective both in theory ( Agrawal & Goyal , 2012 ; 2013 ) and practice ( Chapelle & Li , 2011 ) . Intuitively , TS randomly takes actions according to the probability it believes to be optimal . In practice , a prior distribution is assumed over the model ’ s parameters p ( w ) , and a posterior distribution p ( w|D ) is computed using the Bayes theorem , where D is the observed data . TS acts by sampling models from the posterior distribution , and plays the best action according to these samples . Randomized Least Squares Value Iteration ( Osband et al. , 2016 ) is an RL algorithm which uses linear function approximation and is inspired by Thompson Sampling . It explores by sampling plausible Q-functions from uncertainty sets and selecting the action that optimizes the sampled models . This algorithm was proven to be efficient in tabular settings , with a bound on the expected regret that match the worst-case lower bound up to logarithmic factors . More importantly , it demonstrates efficiency even when generalization is required . Alas , as it assumes a linearly parametrized value function on a hand-designed state representation , the success of this algorithm crucially depends on the quality of the given state representation . In this paper , we present a new DRL algorithm that combines the exploration mechanism of RLSVI with the representation learning mechanism of DQN ; we call it the Deep Randomized Least Squares Value Iteration ( DRLSVI ) algorithm . We use standard DQN to learn state representation and explores by using the last layer ’ s activations of DQN as state representation for RLSVI . To compensate for the constantly changing representation and the finite memory of DQN , we use a likelihood matching mechanism , which allows the transfer of information held by an old representation regarding past experience . We evaluate our method on a toy-problem – the Augmented Chain environment – for a qualitative evaluation of our method on a small MDP with a known optimal value function . Then , we compare our algorithm to the DQN and Rainbow algorithms on several Atari benchmarks . We show that it outperforms DQN both in learning speed and performance . 2 RELATED WORK . Thompson Sampling in Multi-Armed Bandit problems : Thompson Sampling ( TS ) ( Thompson , 1933 ) , is one of the oldest heuristics to address the ’ exploration/exploitation ’ trade-off in sequential decision-making problems . Chapelle & Li ( 2011 ) sparked much of the interest in Thompson Sampling in recent years . They rewrote the TS algorithm for Bernoulli bandit and showed impressive empirical results on synthetic and real data sets that demonstrate the effectiveness of the TS algorithm . Their results demonstrate why TS might be a better alternative to balance between exploration and exploitation in sequential decision-making problems than other popular alternatives like the Upper Confidence Bound algorithm ( Auer et al. , 2002 ) . Agrawal & Goyal ( 2013 ) suggested a Thompson Sampling algorithm for the linear contextual bandit problem and supplied a high-probability regret bound for it . They use Bayesian Linear Regression ( BLR ) with Gaussian likelihood and Gaussian prior to design their version of Thompson Sampling algorithm . Riquelme et al . ( 2018 ) suggested performing a BLR on top of the representation of the last layer of a neural network . The predicted value vi for each action ai is given by vi = βTi zx , where zx is the output of the last hidden layer of the network for context x . While linear methods directly try to regress values v on x , they independently trained a DNN to learn a representation z , and then used a BLR to regress v on z , obtaining uncertainty estimates on the β ’ s , and making decisions accordingly via Thompson Sampling . Moreover , the network is only being used to find good representation – z . Since training the network and updating the BLR can be done independently , they train the network for a fixed number of iterations , then , perform a forward pass on all the training data to obtain the new zx , which is then fed to the BLR . This procedure of evaluating the new representation for all the observed data is very costly , moreover , it requires infinite memory which obviously does not scale . Zahavy & Mannor ( 2019 ) suggested matching the likelihood of the reward under old and new representation to avoid catastrophic forgetting when using such an algorithm with finite memory . Thompson Sampling in RL : In the Reinforcement Learning settings , Strens ( 2000 ) suggested a method named ” Posterior Sampling for Reinforcement Learning ” ( PSRL ) which is an application of Thompson Sampling to Model-Based Reinforcement Learning . PSRL estimates the posterior distribution over MDPs . Each episode , the algorithm samples MDP from it and finds the optimal policy for this sampled MDP by dynamic programming . Recent work ( Osband et al. , 2013 ; Osband & Van Roy , 2017 ) have shown a theoretical analysis of PSRL that guarantees strong expected performance over a wide range of environments . The main problem with PSRL , like all modelbased approaches , is that it may be applied to relatively small environments . The Randomized Least Squares Value Iteration ( RLSVI ) algorithm is an application of Thompson Sampling to Model-Free Reinforcement Learning . It explores by sampling plausible Q-functions from uncertainty sets and selecting the action that optimizes the sampled models . Thompson Sampling in DRL : Various approaches have been suggested to extend the idea behind RLSVI to DRL . Bootstrapped DQN ( Osband et al. , 2017 ) uses an ensemble of Q-networks , each trained with slightly different data samples . To explore , Bootstrapped DQN randomly samples one of the networks and acts greedy with respect to it . Recently , Osband et al . ( 2018 ) extended this idea by supplying each member of the ensemble with a different prior . Fortunato et al . ( 2017 ) and Plappert et al . ( 2017 ) investigate a similar idea and propose to adaptively perturb the parameterspace , which can also be thought of as tracking approximate posterior over the network ’ s parameters . O ’ Donoghue et al . ( 2017 ) proposed TS in combination with uncertainty Bellman equation , which connects the uncertainty at any time-step to the expected uncertainties at subsequent timesteps . Recently and most similar to our work , Azizzadenesheli et al . ( 2018 ) experimented with a Deep Learning extension to RLSVI . They changed the network architecture to exclude the last layer weights , optimized the hyper parameters and used double-DQN . In contrary , we don ’ t change anything in the DQN agent . We use the representation learned by DQN to perform RLSVI , however , the network structure , loss and hyper-parameters are the same . Additionally , differently from our method , they don ’ t compensate for the changing representation and solve BLR problem with the same arbitrary prior every time . 3 PRELIMINARIES . We consider the standard RL settings ( Sutton et al. , 1998 ) , in which an environment with discrete time steps is modeled by a Markov Decision Process ( MDP ) . An MDP is a tuple < S , A , P , R , γ > , where S is a state space , A a finite action space , P : S × A −→ ∆ ( S ) , is a transition kernel , and R : S × A −→ R a reward function . At each step the agent receives an observation st ∈ S which represents the current physical state of the system , takes an action at ∈ A which is applied to the environment , receives a scalar reward rt = r ( st , at ) , and observes a new state st+1 which the environment transitions to . As mentioned above , the agent seeks an optimal policy π∗ : S −→ ∆ ( A ) , mapping an environment state to probabilities over the agent ’ s executable actions . γ ∈ ( 0 , 1 ) is the discount factor – a scalar representing the trade-off between immediate and delayed reward . A brief survey of the DQN algorithm can be found in Appendix 1 .

Summary: The paper proposes to extend the popular linear-control algorithm, RLSVI, to utilize learned representations. This is done by adapting a work from bandit literature that utilizes BLR with representations that are learned via a DNN. The proposed solution is then compared to DQN with fixed epsilon as the exploration strategy in a chain MDP, and to the Rainbow agent and DQN in 5 selected Atari games to show sample-efficiency improvements.