We introduce Implicit Policy, a general class of expressive policies that can flexibly represent complex action distributions in reinforcement learning, with efficient algorithms to compute entropy regularized policy gradients. We empirically show that, despite its simplicity in implementation, entropy regularization combined with a rich policy class can attain desirable properties displayed under maximum entropy reinforcement learning framework, such as robustness and multi-modality.
There is a number of prior works that discuss different policy architectures. The most common policy for continuous control is unimodal Gaussian @cite_30 @cite_25 @cite_32 . @cite_6 discusses mixtures of Gaussian, which can represent multi-modal policies but it is necessary to specify the number of modes in advance. @cite_24 also represents a policy using implicit model, but the policy is trained to sample from the soft Q function instead of being trained directly. Recently, we find @cite_9 also uses Normalizing Flows to represent policies, but their focus is learning an hierarchy and involves layers of pre-training. Contrary to early works, we propose to represent flexible policies using implicit models Normalizing Flows and efficient algorithms to train the policy end-to-end.
@cite_30 @cite_9 @cite_32 @cite_6 @cite_24 @cite_25
2949608212 2796206912 2964043796 2781726626 2949561945 2736601468
Policy search methods can allow robots to learn control policies for a wide range of tasks, but practical applications of policy search often require hand-engineered components for perception, state estimation, and low-level control. In this paper, we aim to answer the following question: does training the perception and control systems jointly end-to-end provide better performance than training each component separately? To this end, we develop a method that can be used to learn policies that map raw image observations directly to torques at the robot's motors. The policies are represented by deep convolutional neural networks (CNNs) with 92,000 parameters, and are trained using a guided policy search method, which transforms policy search into supervised learning, with supervision provided by a simple trajectory-centric reinforcement learning method. We evaluate our method on a range of real-world manipulation tasks that require close coordination between vision and control, such as screwing a cap onto a bottle, and present simulated comparisons to a range of prior policy search methods. Policy search methods can allow robots to learn control policies for a wide range of tasks, but practical applications of policy search often require hand-engineered components for perception, state estimation, and low-level control. In this paper, we aim to answer the following question: does training the perception and control systems jointly end-to-end provide better performance than training each component separately? To this end, we develop a method that can be used to learn policies that map raw image observations directly to torques at the robot's motors. The policies are represented by deep convolutional neural networks (CNNs) with 92,000 parameters, and are trained using a partially observed guided policy search method, which transforms policy search into supervised learning, with supervision provided by a simple trajectory-centric reinforcement learning method. We evaluate our method on a range of real-world manipulation tasks that require close coordination between vision and control, such as screwing a cap onto a bottle, and present simulated comparisons to a range of prior policy search methods. Policy learning for partially observed control tasks requires policies that have the ability to store information from past observations. In this paper, we present a method for learning policies with memory for high-dimensional, continuous systems. Our approach does not assume a known state representation and does not attempt to explicitly model the belief over the unobserved state. Instead, we directly learn a policy that can read and write from an internal continuous-valued memory. This type of policy can be interpreted as a type of recurrent neural network (RNN). However, our approach avoids many of the common problems that plague RNNs, such as the vanishing and exploding gradient issues, by instead representing the memory as state variables. The policy is then optimized by using a guided policy search algorithm that alternates between optimizing trajectories through state space (including both physical and memory states), and training a policy with supervised learning to match these trajectories. We evaluate our method on tasks involving continuous control in manipulation and navigation settings, and show that our method can learn complex policies that successfully complete a range of tasks that require memory. Previous research indicates that during the past 20 years, the highest-quality work has been published in an increasingly diverse and larger group of journals. In this article, we examine whether this diversification has also affected the handful of elite journals that are traditionally considered to be the best. We examine citation patterns during the past 40 years of seven long-standing traditionally elite journals and six journals that have been increasing in importance during the past 20 years. To be among the top 5 or 1 cited papers, papers now need about twice as many citations as they did 40 years ago. Since the late 1980s and early 1990s, elite journals have been publishing a decreasing proportion of these top-cited papers. This also applies to the two journals that are typically considered as the top venues and often used as bibliometric indicators of "excellence": Science and Nature. On the other hand, several new and established journals are publishing an increasing proportion of the most-cited papers. These changes bring new challenges and opportunities for all parties. Journals can enact policies to increase or maintain their relative position in the journal hierarchy. Researchers now have the option to publish in more diverse venues knowing that their work can still reach the same audiences. Finally, evaluators and administrators need to know that although there will always be a certain prestige associated with publishing in "elite" journals, journal hierarchies are in constant flux.