Package: simpleoptions
SimpleOptionsThis Python package aims to provide a simple framework for implementing and using options in Hierarchical Reinforcement Learning (HRL) projects.Key classes:BaseOption: An abstract class representing an option with an initiation set, option policy, and termination condition.BaseEnvironment: An abstract base class representing an agent's environment. The environment specification is based on the OpenAI GymEnvspecifciation, but does not implement it directly. It supports both primitive actions and options, as well as functionality for constructing State-Transition Graphs (STGs) out-of-the-box using NetworkX.OptionAgent: A class representing an HRL agent, which can interact with its environment and has access to a number of options. It includes implementations of Macro-Q Learning and Intra-Option learning, with many customisable features.This code was written with tabular, graph-based HRL methods in mind. It's less of a plug-and-play repository, and is intended to be used to as a basic framework for developing your ownBaseOptionandEnvironmentimplementations.How to InstallThe easiest way to install this package is to simply runpip install simpleoptions.If you want to install from source, download the repository and, in the root directory, run the commandpip install .How to Use This CodeBelow, you will find a step-by-step guide introducing the intended workflow for using this code.Step 1: Implement an EnvironmentThe first step to using this framework involves defining an environment for your agents to interact with. This can be done by implementing the methods specified in theBaseEnvironmentclass. If you have previously worked with OpenAI Gym, much of this will be familiar to you, although there are a few additional methods on top of the usualstepandresetthat you'll need to implement.Step 2: Define Your OptionsYou must now define/discover options for your agent to use when interacting with its environment. How you go about this is up to you — this framework allows you to train agents using options, not discover them. An ever-growing number of option discovery methods can be found in the hierarchical reinforcement learning literature.To define an option, you need to subclassBaseOptionand implement the following methods:initiation- a method that takes a state as its input, and returns whether the option can be invoked in that state.termination- a method that takes a state as its input, and returns the probability that the option terminates in that state.policy- a method that takes a state as its input, and returns the action (either a primitive action or another option) that this option would select in this state.This minimal framework gives you a lot of flexibility in defining your options. For example, yourpolicymethod could make use of a simple dictionary mapping states to actions, it could be based on some learned action-value function, or any other function of the state.As an example, consider an example option that takes an agent to a sub-goal state from any of the nearest 50 states.initiationwould returnTruefor the nearest 50 states to the subgoal, andFalseotherwise.terminationwould return0.0for states in the initiation set, and1.0otherwise.policywoudl return the primitive action that takes the agent one step along the shortest path to the subgoal state.Finally, we also include aPrimitiveOptionthat can be used to represent the primitive actions made available by a given environment.Step 5: Running an Agent Which Uses OptionsOur framework also includes an implementation of Macro-Q Learning and Intra-Option Learning, which can be used to train an agent in an environment with a given set of options.Once you have an environment and a set of options, you can instantiate theOptionsAgentclass and use itsrun_agentmethod to train it.Example ImplementationAn complete end-to-end example implementation of a simple environment and set of options can be foundhere.Additional EnvironmentsA number of environment implementations based on ourBaseEnvironmentclass cna be foundhere.