Frequency-based Search-control in Dyna

1 INTRODUCTION . Model-based reinforcement learning ( MBRL ) ( Lin , 1992 ; Sutton , 1991b ; Daw , 2012 ; Sutton & Barto , 2018 ) methods have successfully been applied to many benchmark domains ( Gu et al. , 2016 ; Ha , David and Schmidhuber , Jürgen , 2018 ; Kaiser et al. , 2020 ) . The Dyna architecture , introduced by Sutton ( 1991a ) , is one of the classical MBRL architectures , which integrates model-free and model-based policy updates in an online RL setting ( Algorithm 2 in Appendix A.3 ) . At each time step , a Dyna agent uses the real experience to learn a model and to perform model-free policy update , and during the planning stage , simulated experiences are acquired from the model to further improve the policy . A closely related method in model-free learning setting is experience replay ( ER ) ( Lin , 1992 ; Adam et al. , 2012 ) , which utilizes a buffer to store experiences . An agent using the ER buffer randomly samples the recorded experiences at each time step to update the policy . Though ER can be thought of as a simplified form of MBRL ( van Seijen & Sutton , 2015 ) , a model provides more flexibility in acquiring simulated experiences . A crucial aspect of the Dyna architecture is the search-control mechanism . It is the mechanism for selecting states or state-action pairs to query the model in order to generate simulated experiences ( cf . Section 8.2 of Sutton & Barto 2018 ) . We call the corresponding data structure for storing those states or state-action pairs the search-control queue . Search-control is of vital importance in Dyna , as it can significantly affect the model-based agent ’ s sample efficiency . A simple approach to searchcontrol is to sample visited states or state-action pairs , i.e. , use the initial state-action pairs stored in the ER buffer as the search-control queue . This approach , however , does not lead to an agent that outperforms a model-free agent that uses ER . To see this , consider a deterministic environment , and assume that we have the exact model . If we simply sample visited state-action pairs for searchcontrol , the next-state and reward would be the same as those in the ER buffer . In practice , we have ∗Equal contribution . model errors too , which causes some performance deterioration ( Talvitie , 2014 ; 2017 ) . Without an elegant search-control mechanism , we are not likely to benefit from the flexibility given by a model . Several search-control mechanisms have already been explored . Prioritized sweeping ( Moore & Atkeson , 1993 ) is one such method that is designed to speed up the value iteration process : the simulated transitions are updated based on the absolute temporal difference error . It has been adopted to continuous domains with function approximation too ( Sutton et al. , 2008 ; Pan et al. , 2018 ; Corneil et al. , 2018 ) . Gu et al . ( 2016 ) utilizes local linear models to generate optimal trajectories through iLQR ( Li & Todorov , 2004 ) . Pan et al . ( 2019 ) suggest a method to generate states for the searchcontrol queue by hill climbing on the value function estimate . This paper proposes an alternative perspective to design search-control strategy : we can sample more frequently from the state space where the value function is more difficult to estimate . We first review some basic background in MBRL ( Section 2 ) . Afterwards , we review some concepts in signal processing and conduct experiments in the supervised learning setting to show that a high frequency function is more difficult to approximate ( Section 3 ) . In order to quantify the difficulty of estimation , we borrow a crucial idea from the signal processing literature : a signal with higher frequency terms requires more samples for accurate reconstruction . We then propose a method to locally measure the frequency of a point in a function ’ s domain and provide a theoretical justification for our method ( Theorem 1 in Section 3.2 ) . We use the hill climbing approach of Pan et al . ( 2019 ) to adapt our method to design a search-control mechanism for the Dyna architecture ( Section 4 ) . We conduct experiments on benchmark and challenging domains to illustrate the properties and utilities of our method ( Section 5 ) . 2 BACKGROUND . Reinforcement learning ( RL ) problems are typically formulated as Markov Decision Processes ( MDPs ) ( Sutton & Barto , 2018 ; Szepesvári , 2010 ) . An MDP ( S , A , P , R , γ ) is determined by state space S , action space A , transition function P , reward function R : S × A × S → R , and discount factor γ ∈ [ 0 , 1 ] . At each step t , an agent observes a state st ∈ S , and takes an action at ∈ A . The environment receives at , and transits to the next state st+1 ∼ P ( ·|st , at ) . The agent receives a reward scalar rt+1 = R ( st , at , st+1 ) . The agent maintains a policy π : S × A → [ 0 , 1 ] that determines the probability of choosing an action at a given state . For a given state-action pair ( s , a ) , the action-value function of policy π is defined as Qπ ( s , a ) = E [ Gt|St = s , At = a ; At+1 : ∞ ∼ π ] where Gt def = ∑∞ t=0 γ tR ( st , at , st+1 ) is the return of s0 , a0 , s1 , a1 , ... following the policy π and transition P. Value-based RL methods learn the action-value function ( Watkins & Dayan , 1992 ) , and act greedily w.r.t . the action-value function . Policy-based RL methods perform gradient update of parameters to learn policies with high expected rewards ( Sutton et al. , 1999 ) . Both value and policy-based RL methods can be easily adopted in the Dyna framework . Model-based RL . A model is a mapping that takes a state-action pair as its input and outputs some projection of the future state . A model can be local ( Tassa et al. , 2012 ; Gu et al. , 2016 ) or global ( Ha , David and Schmidhuber , Jürgen , 2018 ; Pan et al. , 2018 ) , deterministic ( Sutton et al. , 2008 ) or stochastic ( Deisenroth & Rasmussen , 2011 ; Ha , David and Schmidhuber , Jürgen , 2018 ) , feature-to-feature ( Corneil et al. , 2018 ; Ha , David and Schmidhuber , Jürgen , 2018 ) or observationto-observation ( Gu et al. , 2016 ; Pan et al. , 2018 ; Kaiser et al. , 2020 ) , one-step ( Gu et al. , 2016 ; Pan et al. , 2018 ) , or multi-step ( Sorg & Singh , 2010 ; Oh et al. , 2017 ) , or decision-aware ( Joseph et al. , 2013 ; Farahmand et al. , 2017 ; Silver et al. , 2017 ) . Modelling the environment dynamics through a reproducing kernel Hilbert space ( RKHS ) embedding has been also studied ( Grunewalder et al. , 2012 ) , where the Bellman operator is approximated in an RKHS . The model we consider in this work is a one-step environment dynamics model , which takes a state-action pair as its input and returns the next-state and reward . Our proposed search-control approach , however , can be naturally used for different types of models . The most relevant work to ours is hill climbing Dyna ( Pan et al. , 2019 ) . Pan et al . ( 2019 ) proposes a search-control mechanism based on hill climbing on the value estimates ( see Algorithm 3 in Appendix A.3 ) . We briefly review the key steps of their algorithm , which is called ( Hill Climbing ) HCDyna , as it helps to understand ours . HC-Dyna maintains an ER buffer . At each step , a state is randomly sampled from the ER buffer and is used as the initial state to perform hill climbing ( i.e . gradient ascent ) on the learned value function . The states along the trajectory are stored in the search-control queue.1 During the planning stage , states are sampled from the search-control queue and are paired with their corresponding on-policy actions ( i.e. , actions selected by the current Q network at the sampled states ) . Afterwards , the model is queried for each of the state-action pairs to get the next-state and reward . These simulated transitions are then mixed with samples from the ER buffer , which are observed by the agent during its interaction with the real environment , to train the value function estimator , e.g. , a deep neural network . The heuristic idea behind the search-control mechanism of HC-Dyna is that the magnitude of the value function provides useful information for guiding where to query the model . This heuristic can intuitively be understood by noticing that an RL agent tends to move towards high-value regions of the state space ; by performing gradient ascent on the ( estimated ) value function , we provide the agent with more samples from regions where it may move towards in the future . Even if the estimated value function is incorrect and the samples are indeed from the low-value regions of the state space , these extra samples lead to the fast correction of the estimated value in those regions . Nevertheless , the magnitude of the value function is only one source of extra information from which we can design a search-control mechanism . This work suggests a different perspective : we should sample more from the regions of the state space where learning the value function is more difficult . 3 UNDERSTANDING THE DIFFICULTY OF FUNCTION APPROXIMATION . In a regular regression setting , we illustrate that high frequency regions of a function is difficult to approximate . We show that by assigning more training data to those regions , the learning performance considerably improves . To make this insight practically useful , we employ the sum of gradient and hessian norms of a function as a measure of the local frequency of a function . We establish a theoretical connection between our proposed criterion and the local frequency of a function . This would be the foundation of our frequency-based search-control method in Section 4 . 3.1 WHAT TYPE OF FUNCTION IS DIFFICULT TO APPROXIMATE ? . Consider the standard regression problem with the mean square loss . Given a training set D = { ( xi , yi ) } i=1 : n , our goal is to learn an unknown target function f∗ ( x ) = E [ Y |X = x ] by empirical risk minimization . Formally , we aim to solve f = arg min f∈H 1 n n∑ i=1 ( f ( xi ) − yi ) 2 , where H is some hypothesis space . Suppose that we can choose the distributions of samples { xi } . How should we select them in order to improve the quality of the learned function ? One intuitive heuristic is that if we know the regions in the domain of f∗ that are more difficult to approximate , we can assign more training data there in order to help the learning process . The important question is how to quantify the difficulty of approximating a function . We borrow an idea from the field of signal processing to suggest a method . The Nyquist-Shannon sampling theorem in signal processing states that given a band-limited function ( or signal ) f : R → R with the highest frequency ( in the Fourier domain ) of ωbandwidth , we can perfectly reconstruct it based on regular samples ( in the time domain ) obtained at the sampling rate of 2ωbandwidth ( Zayed , 1993 ) .2 Therefore , if the Fourier transform of a function has high frequency terms , more samples are required to reconstruct it accurately . We note that the sampling theory has been applied in the sample complexity analysis of machine learning algorithms ( Smale & Zhou , 2004 ; 2005 ; Jiang , 2019 ) . Although the problem setting in machine learning is somewhat different from this result in signal processing , it still provides a high-level intuition for us : regions with more high frequency signals require more learning data . 1According to the original paper , natural gradient is used to guarantee a certain level of coordinate invariance property , so it can handle state variables with vastly different numerical scales . 2Sampling rate refers to number of samples per second used to reconstruct continuous signals . To make this high-level intuition concrete , we consider the following function : fsin ( x ) = { sin ( 8πx ) x ∈ [ −2 , 0 ) , sin ( πx ) x ∈ [ 0 , 2 ] . ( 1 ) It is easy to check that the regions [ −2 , 0 ) and [ 0 , 2 ] contain signals with frequency ratio 8 : 1 . Based on the intuition from the sampling theorem , the [ −2 , 0 ) interval requires more training data than the [ 0 , 2 ] interval . Given the same amount of training data , and the same learning algorithm , we would expect that assigning more fraction of the training data on [ −2 , 0 ) to perform better than distributing them uniformly or assigning more samples to the [ 0 , 2 ] interval . An illustrative experiment . To empirically verify the intuition , we conduct a simple regression task , with fsin as the target function . The training set D = { ( xi , yi ) } i=1 : n is generated by sampling x ∈ [ −2 , 2 ] , and adding Gaussian noise N ( 0 , σ2 ) on Eq . ( 1 ) , where the standard deviation is set to be σ = 0.1 . We present the ` 2 regression learning curves of training datasets with different biased sampling ratios pb ∈ { 60 % , 70 % , 80 % } , as shown in Fig . 1 ( a ) - ( c ) . We observe that biased training data sampling ratios towards high frequency region clearly speeds up learning . This is consistent with the intuitive insight and suggests that our heuristic to assign more data to high frequency regions leads to faster learning .

Summary: This paper proposes a new way to select states from which do do transitions in dyna algorithm (which trains policy from model experience as if it was a real experience). It proposes to look for states where frequency of value function as a function of a real valued state is large, because these are the states where the function is harder to approximate. The paper also shows that such frequency is large where the gradient of the function is large in magnitude which allows for finding such states in practice. In more detail, similar to previous algorithms, this algorithm keeps both an experience replay buffer as well as another buffer of states (search-control queue) and uses a hill climbing strategy to find states with both higher frequency and higher value. The paper tests the algorithm on toy domains - the mountain car and a maze with doors.