Frequency-based Search-control in Dyna
Model-based reinforcement learning has been empirically demonstrated as a successful strategy to improve sample efficiency.Particularly, Dyna architecture, as an elegant model-based architecture integrating learning and planning, provides huge flexibility of using a model.One of the most important components in Dyna is called search-control, which refers to the process of generating state or state-action pairs from which we query the model to acquire simulated experiences.Search-control is critical to improve learning efficiency.In this work, we propose a simple and novel search-control strategy by searching high frequency region on value function.Our main intuition is built on Shannon sampling theorem from signal processing, which indicates that a high frequency signal requires more samples to reconstruct.We empirically show that a high frequency function is more difficult to approximate.This suggests a search-control strategy: we should use states in high frequency region of the value function to query the model to acquire more samples.We develop a simple strategy to locally measure the frequency of a function by gradient norm, and provide theoretical justification for this approach.We then apply our strategy to search-control in Dyna, and conduct experiments to show its property and effectiveness on benchmark domains.
Acquire states from high frequency region for search-control in Dyna.The authors propose to do sampling in the high-frequency domain to increase the sample efficiencyThis paper proposes a new way to select states from which do do transitions in dyna algorithm.