Package: imitation
Imitation Learning Baseline ImplementationsThis project aims to provide clean implementations of imitation and reward learning algorithms.
Currently, we have implementations of the algorithms below. 'Discrete' and 'Continous' stands for whether the algorithm supports discrete or continuous action/state spaces respectively.Algorithm (+ link to paper)API DocsDiscreteContinuousBehavioral Cloningalgorithms.bc✅✅DAggeralgorithms.dagger✅✅Density-Based Reward Modelingalgorithms.density✅✅Maximum Causal Entropy Inverse Reinforcement Learningalgorithms.mce_irl✅❌Adversarial Inverse Reinforcement Learningalgoritms.airl✅✅Generative Adversarial Imitation Learningalgorithms.gail✅✅Deep RL from Human Preferencesalgorithms.preference_comparisons✅✅Soft Q Imitation Learningalgorithms.sqil✅❌You can findthe documentation here.InstallationPrerequisitesPython 3.8+(Optional) OpenGL (to render Gym environments)(Optional) FFmpeg (to encode videos of renders)(Optional) MuJoCo (follow instructions to installmujoco_py v1.5 here)Installing PyPI releaseInstalling the PyPI release is the standard way to useimitation, and the recommended way for most users.pip install imitationInstall from sourceIf you like, you can installimitationfrom source tocontribute to the projector access the very last features before a stable release. You can do this by cloning the GitHub repository and running the installer directly. First run:git clone http://github.com/HumanCompatibleAI/imitation && cd imitation.For development mode, then run:pip install -e ".[dev]"This will runsetup.pyin development mode, and install the additional dependencies required for development. For regular use, run insteadpip install .Additional extras are available depending on your needs. Namely,testsfor running the test suite,docsfor building the documentation,parallelfor parallelizing the training, andatarifor including atari environments. Thedevextra already installs thetests,docs, andataridependencies automatically, andtestsinstalls theataridependencies.For macOS users, some packages are required to run experiments (see./experiments/README.mdfor details). First, install Homebrew if not available (seeHomebrew). Then, run:brew install coreutils gnu-getopt parallelCLI QuickstartWe provide several CLI scripts as a front-end to the algorithms implemented inimitation. These useSacredfor configuration and replicability.Fromexamples/quickstart.sh:# Train PPO agent on pendulum and collect expert demonstrations. Tensorboard logs saved in quickstart/rl/python-mimitation.scripts.train_rlwithpendulumenvironment.fastpolicy_evaluation.fastrl.fastfastlogging.log_dir=quickstart/rl/# Train GAIL from demonstrations. Tensorboard logs saved in output/ (default log directory).python-mimitation.scripts.train_adversarialgailwithpendulumenvironment.fastdemonstrations.fastpolicy_evaluation.fastrl.fastfastdemonstrations.path=quickstart/rl/rollouts/final.npzdemonstrations.source=local# Train AIRL from demonstrations. Tensorboard logs saved in output/ (default log directory).python-mimitation.scripts.train_adversarialairlwithpendulumenvironment.fastdemonstrations.fastpolicy_evaluation.fastrl.fastfastdemonstrations.path=quickstart/rl/rollouts/final.npzdemonstrations.source=localTips:Remove the "fast" options from the commands above to allow training run to completion.python -m imitation.scripts.train_rl print_configwill list Sacred script options. These configuration options are documented in each script's docstrings.For more information on how to configure Sacred CLI options, see theSacred docs.Python Interface QuickstartSeeexamples/quickstart.pyfor an example script that loads CartPole-v1 demonstrations and trains BC, GAIL, and AIRL models on that data.Density reward baselineWe also implement a density-based reward baseline. You can find anexample notebook here.Citations (BibTeX)@misc{gleave2022imitation,
  author = {Gleave, Adam and Taufeeque, Mohammad and Rocamonde, Juan and Jenner, Erik and Wang, Steven H. and Toyer, Sam and Ernestus, Maximilian and Belrose, Nora and Emmons, Scott and Russell, Stuart},
  title = {imitation: Clean Imitation Learning Implementations},
  year = {2022},
  howPublished = {arXiv:2211.11972v1 [cs.LG]},
  archivePrefix = {arXiv},
  eprint = {2211.11972},
  primaryClass = {cs.LG},
  url = {https://arxiv.org/abs/2211.11972},
}ContributingSeeContributing to imitationfor more information.