Package: rl-algo-impls
rl-algo-implsImplementations of reinforcement learning algorithms.WandB benchmark reportsBasic, PyBullet, and Atari games
(v0.0.9)v0.0.8v0.0.4procgen
(starpilot, hard)andprocgen (easy)Gridnet MicroRTSMicroRTS SelfplayLux AI Season 2 TrainingHuggingface modelsPrerequisites: Weights & Biases (WandB)Training and benchmarking assumes you have a Weights & Biases project to upload runs to.
By default training goes to a rl-algo-impls project while benchmarks go to
rl-algo-impls-benchmarks. During training and benchmarking runs, videos of the best
models and the model weights are uploaded to WandB.Before doing anything below, you'll need to create a wandb account and runwandb login.Setup and UsageLambda Labs instance for benchmarkingBenchmark runs are uploaded to WandB, which can be made into reports (for
example). So far I've found Lambda
Labs A10 instances to be a good balance of performance (14 hours to train PPO in 14
environments [5 basic gym, 4 PyBullet, CarRacing-v0, and 4 Atari] across 3 seeds) vs
cost ($0.60/hr).git clone https://github.com/sgoodfriend/rl-algo-impls.git
cd rl-algo-impls
# git checkout BRANCH_NAME if running on non-main branch
bash ./scripts/setup.sh
wandb login
bash ./scripts/benchmark.sh [-a {"ppo"}] [-e ENVS] [-j {6}] [-p {rl-algo-impls-benchmarks}] [-s {"1 2 3"}]Benchmarking runs are by default upload to a rl-algo-impls-benchmarks project. Runs upload
videos of the running best model and the weights of the best and last model.
Benchmarking runs are tagged with a shorted commit hash (i.e.,benchmark_5598ebc) and
hostname (i.e.,host_192-9-145-26)Publishing models to HuggingfacePublishing benchmarks to Huggingface requires logging into Huggingface with a
write-capable API token:git config --global credential.helper store
huggingface-cli login
# For example: python benchmark_publish.py --wandb-tags host_192-9-147-166 benchmark_1d4094f --wandb-report-url https://api.wandb.ai/links/sgoodfriend/099h4lvj
# --virtual-display likely must be specified if running on a remote machine.
python benchmark_publish.py --wandb-tags HOST_TAG COMMIT_TAG --wandb-report-url WANDB_REPORT_URL [--virtual-display]Hyperparameter tuning with OptunaHyperparameter tuning can be done with thetuning/tuning.shscript, which runs
multiple processes of optimize.py. Start by doing all the setup meant for training
before runningtuning/tuning.sh:# Setup similar to training above
wandb login
bash scripts/tuning.sh -a ALGO -e ENV -j N_JOBS -s NUM_SEEDSGoogle Colab Pro+3 notebooks in the colab directory are setup to be used with Google Colab:colab_benchmark.ipynb:
Even with a Google Colab Pro+ subscription you'd need to only run parts of the
benchmark. The file recommends 4 splits (basic+pybullet, carcarcing, atari1, atari2)
because it would otherwise exceed the 24-hour session limit. This mostly comes from
being unable to get pool_size above 1 because of WandB errors.colab_train.ipynb:
Train models while being able to specify the env, seeds, and algo. By default training
runs are uploaded to the rl-algo-impls project.colab_enjoy.ipynb:
Download models from WandB and evaluate them. Training is likely to be more
interesting given videos are uploaded.macOSInstallationMy local development has been on an M1 Mac. These instructions might not be complete,
but these are the approximate setup and usage I've been using:Install libraries with homebrewbrew install swig
brew install --cask xquartzDownload and install Miniconda for arm64curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh
sh Miniconda3-latest-MacOSX-arm64.shCreate a conda environment from this repo'senvironment.ymlconda env create -f environment.yml -n rl_algo_impls
conda activate rl_algo_implsInstall other dependencies with poetrypoetry installUsageTraining, benchmarking, and watching the agents playing the environments can be done
locally:python train.py [-h] [--algo {ppo}] [--env ENV [ENV ...]] [--seed [SEED ...]] [--wandb-project-name WANDB_PROJECT_NAME] [--wandb-tags [WANDB_TAGS ...]] [--pool-size POOL_SIZE] [-virtual-display]train.py by default uploads to the rl-algo-impls WandB project. Training creates videos
of the running best model, which will cause popups. Creating the first video requires a
display, so you shouldn't shutoff the display until the video of the initial model is
created (1-5 minutes depending on environment). The --virtual-display flag should allow
headless mode, but that hasn't been reliable on macOS.python enjoy.py [-h] [--algo {ppo}] [--env ENV] [--seed SEED] [--render RENDER] [--best BEST] [--n_episodes N_EPISODES] [--deterministic-eval DETERMINISTIC_EVAL] [--no-print-returns]
# OR
python enjoy.py [--wandb-run-path WANDB_RUN_PATH]The first enjoy.py where you specify algo, env, and seed loads a model you locally
trained with those parameters and renders the agent playing the environment.The second enjoy.py downloads the model and hyperparameters from a WandB run. An
example run path issgoodfriend/rl-algo-impls-benchmarks/09gea50gHyperparametersThese are specified in yaml files in the hyperparams directory by game (atariis a
special case for all Atari games).procgen Setupprocgen envs use gym3, which don't expose a straightforward way to set seed to allow for
repeatable runs.openai/procgendoesn't support Apple Silicon, butpatch
instructions exist. The changes to the
repo are for now in a fork since the openai/procgen project is in maintenance mode:brew install wget cmake glow qt5
git clone https://github.com/sgoodfriend/procgen.git
cd procgen
pip install -e .
python -c "from procgen import ProcgenGym3Env; ProcgenGym3Env(num=1, env_name='coinrun')"
python -m procgen.interactiveamd64 Linux machines (e.g., Lambda Labs and Google Colab) should install procgen withpython -m pip install '.[procgen]'gym-microrts Setuppython -m pip install -e '.[microrts]'Requires Java SDK to also be installed.