Package: qwop-gym
QWOP GymA Gym environment for Bennet Foddy's game calledQWOP.Give it a tryand see why it's such a
good candidate for Reinforcement Learning :)You should also check thisvideofor a demo.FeaturesA call to.step()advances exactly N game frames (configurable)Option to disable WebGL rendering for improved performanceSatisfies the Markov property *State extraction for a slim observation of 60 bytesReal-time visualization of various game stats (optional)Additional in-game controls for easier debugging* given the state includes the steps since last hard reset, see♻️ ResettingGetting startedInstallPython3.10 or higherInstall a chrome-based web browser (Google Chrome, Brave, Chromium, etc.)Downloadchromedriver116.0 or higherInstall theqwop-gympackage and patch QWOP.min.js from your terminal:pipinstallqwop-gym# Fetch & patch QWOP source codecurl-sLhttps://www.foddy.net/QWOP.min.js|qwop-gympatchCreate an instance in your code:importqwop_gymenv=gym.make("QWOP-v1",browser="/browser/path",driver="/driver/path")Theqwop-gymtoolTheqwop-gymexecutable is a handy command-line tool which makes it easy to
play, record and replay episodes, train agents and more.Firstly, perform the initial setup:qwop-gym bootstrapPlay the game (use Q, W, O, P keys):qwop-gymplayExplore the other available commands:$qwop-gym-h
usage:qwop-gym[options]<action>

options:-h,--helpshowthishelpmessageandexit-cFILEconfigfile,defaultstoconfig/<action>.yml

action:playplayQWOP,optionallyrecordingactionsreplayreplayrecordedgameactionstrain_bctrainusingBehavioralCloning(BC)train_gailtrainusingGenerativeAdversarialImitationLearning(GAIL)train_airltrainusingAdversarialInverseReinforcementLearning(AIRL)train_ppotrainusingProximalPolicyOptimization(PPO)train_dqntrainusingDeepQNetwork(DQN)train_qrdqntrainusingQuantileRegressionDQN(QRDQN)spectatewatchatrainedmodelplayQWOP,optionallyrecordingactionsbenchmarkevaluatetheactions/sachievablewiththisenvbootstrapperforminitialsetuppatchapplypatchtooriginalQWOP.min.jscodehelpprintthishelpmessage

examples:qwop-gymplayqwop-gym-cconfig/record.ymlplayFor example, to train a PPO agent, editconfig/ppo.ymland run:pythonqwop-gymtrain_ppo[!WARNING]
Although no rendering occurs during training, the browser window must remain
open as the game is actually running at very high speeds behind the curtains.Visualize tensorboard graphs:tensorboard--logdirdata/Configuremodel_fileinconfig/spectate.ymland watch your trained agent play the game:pythonqwop-gymspectateImitation[!NOTE]
Imitation learning is powered by theimitationlibrary, which
depends on the deprecatedgymlibrary which makes it incompatible with
QwopEnv. This can be resolved as soon asimitationintroduces support forgymnasium. As a workaround, you can checkout theqwop-gymproject
locally and use thegym-compatbranch instead.# In this branch, QwopEnv works with the deprecated `gym` librarygitcheckoutgym-compat# Note that python-3.10 is required, see notes in requirements.txtpipinstall-rrequirements.txt# Patch the game again as this branch works with different pathscurl-sLhttps://www.foddy.net/QWOP.min.js|python-msrc.game.patcherFor imitation learning, first record some of your own games:pythonqwop-gym.pyplay-cconfig/record.ymlTrain an imitator viaBehavioral Cloning:pythonqwop-gym.pytrain_bcW&B sweepsIf you are a fan ofW&B, you can
use the provided configs inconfig/wandb/and create your own sweeps.wandbis a rather bulky dependency and is not installed by default. Install
it withpip install wandbbefore proceeding with the below examples.# create a new W&B sweepwandbsweepconfig/wandb/qrdqn.yml# start a new W&B agentwandbagent<username>/qwop/<sweep>You can check out my W&B public QWOP projecthere.
There you can find pre-trained model artifacts (zip files) of some
well-performing agents, as well as see how they compare to each other. Thisyoutube videoshowcases some of
them.Developer documentationInfo about the Gym env can be foundhereDetails about the QWOP game can be foundhereSimilar projectshttps://github.com/Wesleyliao/QWOP-RLhttps://github.com/drakesvoboda/RL-QWOPhttps://github.com/juanto121/qwop-aihttps://github.com/ShawnHymel/qwop-aiIn comparison, qwop-gym offers several key features:the env isperformant- perfect for on-policy algorithms as observations
can be collected at great speeds (more than 2000 observations/sec on an Apple
M2 CPU - orders of magnitute faster than the other QWOP RL envs).the env satisfies theMarkov property- there are no race conditions and
randomness can be removed if desired, so recorded episodes are 100% replayablethe env has asimple reward modeland compared to other QWOP envs, it is
less biased, eg. no special logic for stuff likeknee bending,low torso height,vertical movement, etc.the env allows all possible key combinations (15), other QWOP envs usually
allow only the "useful" 8 key combinations.great results (fast, human-like running) achieved by RL agents trained
entirely through self-play, without pre-recorded expert demonstrationsqwop-gym already contains scripts for training with 6 different algorithms
and adding more to the list is simple - this makes it suitable for exploring
and/or benchmarking a variety of RL algorithms.qwop-gym uses reliable open-source implementations of RL algorithms in
contrast to many other projects using "roll-your-own" implementations.QWOP's original JS source code is barely modified: 99% of all extra
functionality is designed as a plugin, bundled separately and only a "diff"
of QWOP.min.js is published here (in respect to Benett Foddy's kind request
to refrain from publishing the QWOP source code as part of isnotopen-source).CaveatsThe below list highlights some areas in which the project could use some
improvements:the OS may put some pretty rough restrictions on the web browser's rendering
as soon as it's put in the background (on OS X at least). Ideally, the browser
should run in a headless mode, but I couldn't find a headless browser that can
support WebGL.gymis deprecated since October 2022, but theimitationlibrary still
does not officially supportgymnasium. As soon as that is addressed, there
will no longer be required to use the specialgym-compatbranch here for
imitation learning.wandbuses a monkey-patch for collecting tensorboard logs which does not
work well with GAIL/AIRL/BC (and possibly other algos fromimitation). As a
result, graphs in wandb have weird names. This is mostly an issue withwandband/orimitationlibraries, however there could be a way to work around this
here.firefox browser and geckodriver are not supported as an alternative
browser/driver pair, but adding support for them should be fairly easyContributingHere is a simple guide to follow if you want to contribute to this project:Find an existing issue to work on or submit a new issue which you're also
going to fix. Make sure to notify that you're working on a fix for the issue
you picked.Branch out from latestmain.Make sure you have formatted your code with theblackformatter.Commit and push your changes in your branch.Submit a PR.