###### Abstract  
Scaling end-to-end learning to control robots with vision inputs is a challenging problem in the field of deep reinforcement learning (DRL). While achieving remarkable success in complex sequential tasks, vision-based DRL remains extremely data-inefficient, especially when dealing with high-dimensional pixels inputs. Many recent studies have tried to leverage state representation learning (SRL) to break through such a barrier. Some of them could even help the agent learn from pixels as efficiently as from states. Reproducing existing work, accurately judging the improvements offered by novel methods, and applying these approaches to new tasks are vital for sustaining this progress. However, the demands of these three aspects are seldom straightforward. Without significant criteria and tighter standardization of experimental reporting, it is difficult to determine whether improvements over the previous methods are meaningful. For this reason, we conducted ablation studies on hyperparameters, embedding network architecture, embedded dimension, regularization methods, sample quality and SRL methods to compare and analyze their effects on representation learning and reinforcement learning systematically. Three evaluation metrics are summarized, including five baseline algorithms (including both value-based and policy-based methods) and eight tasks are adopted to avoid the particularity of each experiment setting. We highlight the variability in reported methods and suggest guidelines to make future results in SRL more reproducible and stable based on a wide number of experimental analyses. We aim to spur discussion about how to assure continued progress in the field by minimizing wasted effort stemming from results that are non-reproducible and easily misinterpreted. deep reinforcement learning; representation learning; unsupervised learning; feature embedding; end-to-end learning +
Footnote †: journal: Medical  
observable Markov Decision Processes (MDPs). However, this assumption is unworkable in real-world robotics due to factors including sensor sensitivity limitations and sensor noise and the lack of knowledge about whether the observation design is complete or not. Particularly in the case of learning directly from raw images, partial observability problem arises, which is defined as a partially observable MDP (POMDP), in which the agent can only sees a portion of the world state. Furthermore, there are numerous extrinsic factors (e.g., hyperparameters or codebases) and intrinsic elements (e.g., effects of random seeds or environment properties) that might affect the performance of DRL [9]. As a result, vision-based DRL typically suffers from slow learning speeds and frequently requires an excessive amount of training time and data to attain desired performance, making it unsuitable to real-world situations where data collection is difficult and expensive. For this reason, most of the present DRL algorithms are still limited to solving tasks with state spaces of relatively low dimensionality (less than 10 intrinsic dimensions for value-function based methods and less than 100 for policy gradient methods [10]). In addition, when the ground-truth state information cannot be easily accessed, it is inevitable to extract the embedded states from the original observations to learn desirable policies from DRL algorithms. To overcome this issue, state representation learning (SRL) [11] was proposed which aims at learning to extract the state information which can be used for the policy optimization. As shown in Figure 1. This paper studies the influence factors (listed on the bottom left corner of this figure) of SRL, and their effects on different kinds of RL methods (listed on the bottom left corner of this figure). In the setting of SRL, instead of getting a low-dimensional coordinate state \(s_{t}\in S\) at time \(t\), the agent receives rather a high-dimensional observation \(0_{t}\in O\), which is a rendering of potentially incomplete view of the corresponding state \(s_{t}\) of the environment [12].