Learning Representations for Unsupervised Skill Discovery
Creators: name: Adeniji, Ademiloluwa
Publisher: name: Stanford Digital Repository
Year: 2024
Subjects: unsupervised learning
skill discovery
representation learning
reinforcement learning

description: Intelligent creatures leverage vast stores of prior knowledge when attempting to complete a new task. However, most current reinforcement learning (RL) agents learn each new task from scratch, possibly collecting new data and learning to a new objective with no prior knowledge. This represents a massive roadblock in the way of integration of RL in many real-time applications such as robotic control.
					
When completing object manipulation tasks, humans do not relearn low-level motion primitives such as reaching or grasping, rather they perceive the objects in their environment and execute high-level plans to manipulate them as needed. Acquiring these low-level skill primitives without an explicit reward signal is an open-area of research for autonomous agents. Since skills should ideally comprise a complete set of behaviors for solving arbitrary tasks in the environment, we contend that learned skills should have three properties: 1. be diverse, 2. represent a set of building blocks for reaching all possible configurations in the space, 3. each represent a consistent temporally extended behavior.
					
To our knowledge, existing approaches to skill discovery do not fulfill all three of these criteria. It is common for skills to collapse into indistinguishable modes of behavior, fail to displace meaningfully from the agent’s initialized position, or execute two or three inconsistent behaviors over some temporal horizon. A number of properties of unsupervised skill discovery algorithms can explain these shortcomings, including non-stationarity of intrinsic reward and overfitting in adversarial training reminiscent of mode-collapse in Generative Adversarial Networks. However, the most troubling of pathologies is that skill discovery algorithms operate on the full state of the agent. Here, the state representation is far too entangled for learned behavioral diversity to be meaningful, let alone produce helpful skills. 
					
In this work, we first provide a high-level background on the major technical elements underlying the discussion. We provide a broader generalization of skills and then propose a multitask learning framework for skill acquisition in a relatively assumption-free context. We experimentally examine the merits and limitations of this approach in order to motivate the notion of skills. With this motivation established, we conduct a more detailed investigation into the pathologies of state-of-the-art skill discovery methods and introduce several algorithmic approaches to address them with offline demonstration data. We then identify several practical limitations of these approaches and advance a methodology for leveraging demonstrations indirectly by representation learning.
					
In our core method, we propose to leverage offline expert demonstrations of distinct behaviors to learn a latent representation of the agent state that admits a clearer notion of behavioral diversity to bias the emergence of more useful skills per our criteria. Learning this representation from a few diverse demonstrations presents three main advantages. For one, it eliminates noise from high-dimensional proprioceptive data to simplify the diversity optimization problem of the skill learning algorithm. Secondly, it allows for the acquisition of an arbitrary number of new, useful skills. Thirdly, the kinds of skills learned can be customized because the representation learned will correspond to the kinds of demonstrations provided. We aim to validate that the combination of useful motion primitives guided by a few offline demonstrations comprise a powerful library of motor capabilities for learning a diverse set of challenging downstream tasks. We present several algorithmic approaches and regularizations for embedding these representations and present visual justifications for their utility. We proceed to integrate this algorithm fully with the skill discovery framework and demonstrate our contributions in the context of more realistic robot manipulation environments.