Package: universe
Universeis a software
platform for measuring and training an AI’s general intelligence
across the world’s supply of games, websites and other
applications. This is theuniverseopen-source library, which
provides a simpleGyminterface to each Universe environment.Universe allows anyone to train and evaluate AI agents on an extremely
wide range of real-time, complex environments.Universe makes it possible for any existing program to become an
OpenAI Gym environment, without needing special access to the
program’s internals, source code, or APIs. It does this by packaging
the program into a Docker container, and presenting the AI with the
same interface a human uses: sending keyboard and mouse events, and
receiving screen pixels. Our initial release contains over 1,000
environments in which an AI agent can take actions and gather
observations.Additionally, some environments include a reward signal sent to the
agent, to guide reinforcement learning. We’ve included a few hundred
environments with reward signals. These environments also include
automated start menu clickthroughs, allowing your agent to skip to the
interesting part of the environment.We’d like the community’shelpto grow the number of available environments, including integrating
increasingly large and complex games.The following classes of tasks are packaged inside of
publicly-available Docker containers, and can be run today with no
work on your part:Atari and CartPole environments over VNC:gym-core.Pong-v3,gym-core.CartPole-v0, etc.Flashgames over VNC:flashgames.DuskDrive-v0, etc.Browser tasks (“World of Bits”) over VNC:wob.mini.TicTacToe-v0, etc.We’ve scoped out integrations for many other games, including
completing a high-quality GTA V integration (thanks toCraig Quiterand NVIDIA), but these aren’t included in today’s release.Contents of this documentGetting startedInstallationSystem overviewRun your first agentTestingAdditional documentationGetting helpWhat’s next?ChangelogGetting startedInstallationSupported systemsWe currently support Linux and OSX running Python 2.7 or 3.5.We recommend setting up aconda environmentbefore getting started, to keep all your Universe-related packages in the same place.Install UniverseTo get started, first installuniverse:gitclonehttps://github.com/openai/universe.gitcduniversepipinstall-e.If this errors out, you may be missing some required packages. Here’s
the list of required packages we know about so far (please let us know
if you had to install any others).On Ubuntu 16.04:pipinstallnumpysudoapt-getinstallgolanglibjpeg-turbo8-devmakeOn Ubuntu 14.04:sudoadd-apt-repositoryppa:ubuntu-lxc/lxd-stable# for newer golangsudoapt-getupdatesudoapt-getinstallgolanglibjpeg-turbo8-devmakeOn OSX:You might need to install Command Line Tools by running:xcode-select--installOrnumpy,libjpeg-turboandincrementalpackages:pipinstallnumpyincrementalbrewinstallgolanglibjpeg-turboInstall DockerThe majority of the environments in Universe run inside Docker
containers, so you will need toinstall Docker(on OSX, we
recommendDocker for Mac). You should be able to
rundocker psand get something like this:$dockerpsCONTAINERIDIMAGECOMMANDCREATEDSTATUSPORTSNAMESNotes on installationWhen installinguniverse, you may seewarningmessages.  These lines occur when installing numpy and are normal.You’ll need ago versionof at least 1.5. Ubuntu 14.04 has an older Go, so you’ll need toupgradeyour Go installation.We run Python 3.5 internally, so the Python 3.5 variants will be much more thoroughly performance tested. Please let us know if you see any issues on 2.7.While we don’t officially support Windows, we expect our code to be very close to working there. We’d be happy to take pull requests that take our Windows compatibility to 100%.System overviewA Universeenvironmentis similar to any other Gym environment:
the agent submits actions and receives observations using thestep()method.Internally, a Universe environment consists of two pieces: aclientand aremote:Theclientis aVNCEnvinstance which lives in the same process as the agent. It performs
functions like receiving the agent’s actions, proxying them to theremote, queuing up rewards for the agent, and maintaining a
local view of the current episode state.Theremoteis the running environment dynamics, usually a
program running inside of a Docker container. It can run anywhere –
locally, on a remote server, or in the cloud. (We have a separate
page describing how to manageremotes.)The client and the remote communicate with one another using theVNCremote desktop system, as well as over an auxiliary WebSocket
channel for reward, diagnostic, and control messages. (For more
information on client-remote communication, see the separate page on
theUniverse internal communication protocols.)The code in this repository corresponds to theclientside of the
Universe environments. Additionally, you can freely access the Docker
images for theremotes. We’ll release the source repositories for
the remotes in the future, along with tools to enable users to
integrate new environments. Please sign up for ourbetaif you’d like early access.Run your first agentNow that you’ve installed theuniverselibrary, you should make
sure it actually works. You can paste the example below into yourpythonREPL. (You may need to press enter an extra time to make
sure thewhileloop is executing.)importgymimportuniverse# register the universe environmentsenv=gym.make('flashgames.DuskDrive-v0')env.configure(remotes=1)# automatically creates a local docker containerobservation_n=env.reset()whileTrue:action_n=[[('KeyEvent','ArrowUp',True)]forobinobservation_n]# your agent hereobservation_n,reward_n,done_n,info=env.step(action_n)env.render()The example will instantiate a client in your Python process,
automatically pull thequay.io/openai/universe.flashgamesimage,
and will start that image as the remote. (In ourremotesdocumentation page, we explain other ways you can run
remotes.)It will take a few minutes for the image to pull the first time. After that,
if all goes well, a window like the one below will soon pop up. Your
agent, which is just pressing the up arrow repeatedly, is now
playing a Flash racing game calledDusk Drive. Your agent
is programmatically controlling a VNC client, connected to a VNC
server running inside of a Docker container in the cloud, rendering a
headless Chrome with Flash enabled:You can even connect your own VNC client to the environment, either
just to observe or to interfere with your agent. Ourflashgamesandgym-coreimages conveniently bundle a browser-based VNC
client, which can be accessed athttp://localhost:15900/viewer/?password=openai. If you’re on Mac,
connecting to a VNC server is as easy as running:openvnc://localhost:5900.(If using docker-machine, you’ll need to replace “localhost” with the
IP address of your Docker daemon, and useopenaias the password.)Breaking down the exampleSo we managed to run an agent, what did all the code actually
mean? We’ll go line-by-line through the example.First, we import thegymlibrary,
which is the base on which Universe is built. We also importuniverse, whichregistersall the Universe environments.importgymimportuniverse# register the universe environmentsNext, we create the environment instance. Behind the scenes,gymlooks up theregistrationforflashgames.DuskDrive-v0, and instantiates aVNCEnvobject which has beenwrappedto add a few useful diagnostics and utilities. TheVNCEnvobject
is theclientpart of the environment, and it is not yet connected
to aremote.env=gym.make('flashgames.DuskDrive-v0')The call toconfigure()connects the client to a remote
environment server. When called withconfigure(remotes=1),
Universe will automatically create a Docker image running locally on
your computer. The local client connects to the remote using VNC.
(More information on client-remote communication can be found in the
page onuniverse internal communication protocols. More on configuring remotes is atremotes.)env.configure(remotes=1)When starting a new environment, you callenv.reset(). Universe
environments run in real-time, rather than stepping synchronously
with the agent’s actions, soresetis asynchronous and returns
immediately. Since the environment will not have waited to finish
connecting to the VNC server before returning, the initial observations
fromresetwill beNoneto indicate that there is
not yet a valid observation.Similarly, the environment keeps running in the background even
if the agent does not callenv.step().  This means that an agent
that successfully learns from a Universe environment cannot take
“thinking breaks”:  it must keep sending actions to the environment at all times.Additionally, Universe introduces thevectorizedGym
API. Rather than controlling a single environment at a time, the agent
can control a fixed-size vector ofnenvironments, each with its
own remote. The return value fromresetis therefore avectorof observations. For more information, see the separate page onenvironment semantics)observation_n=env.reset()At eachstep()call, the agent submits a vector of actions; one for
each environment instance it is controlling. Each VNC action is a
list of events; above, each action is the single event “press theArrowUpkey”. The agent could press and release the key in one
action by instead submitting[('KeyEvent','ArrowUp', True), ('KeyEvent', 'ArrowUp', False)]for each observation.In fact, the agent could largely have the same effect by just
submitting('KeyEvent', 'ArrowUp', True)once and then callingenv.step([[]for ob in observation_n])thereafter, without ever
releasing the key using('KeyEvent', 'ArrowUp', False). The
browser running inside the remote would continue to statefully
represent the arrow key as being pressed. Sending other unrelated
keypresses would not disrupt the up arrow keypress; only explicitly
releasing the key would cancel it.  There’s one slight subtlety:
when the episode resets, the browser will reset, and will forget
about the keypress; you’d need to submit a newArrowUpat the
start of each episode.action_n=[[('KeyEvent','ArrowUp',True)]forobinobservation_n]After we submit the action to the environment and render one frame,step()returns a list ofobservations, a list ofrewards, a
list of“done” booleansindicating whether the episode has ended,
and then finally aninfo dictionaryof the form{'n':[{},...]}, in which you can access the info for environmentiasinfo['n'][i].Each environment’sinfomessage contains useful diagnostic
information, including latency data, client and remote timings,
VNC update counts, and reward message counts.observation_n,reward_n,done_n,info=env.step(action_n)env.render()We callstepin what looks like a busy loop. In reality, there
is aThrottlewrapper on the client which defaults to a target frame rate of 60fps, or one
frame every 16.7ms. If you call it more frequently than that,stepwillsleepwith any leftover time.TestingWe are usingpytestfor tests. You can run them via:pytestRunpytest--helpfor useful options, such aspytest-s(disables output capture) orpytest-k<expression>(runs only specific tests).Additional documentationMore documentation not covered in this README can be found in thedoc folderof this repository.Getting helpIf you encounter a problem that is not addressed in this README page
or in theextra docs, then try our wiki page ofsolutions
to common problems-
and add to it if your solution isn’t there!You can also search through theissueson this repository and ourdiscussion boardto see if another user has posted
about the same problem or to ask for help from the community.If you still can’t solve your problem after trying all of the above
steps, please post an issue on this repository.What’s next?Get started training RL algorithms! You can try out theUniverse Starter Agent, an implementation of theA3C algorithmthat can solve several VNC environments.For more information on how to manage remotes, see the separate documentation page onremotes.Sign up for abetato get early access to upcoming Universe releases, such as tools to integrate new Universe environments or a dataset of recorded human demonstrations.Changelog2017-02-08: The old location for wrappers.SafeActionSpace has been moved to wrappers.experimental.SafeActionSpace. SoftmaxClickMouse has also been moved to wrappers.experimental.SoftmaxClickMouse2017-01-08: The wrappers.SafeActionSpace has been moved to wrappers.experimental.SafeActionSpace. The old location will remain with a deprecation warning until 2017-02-08.2016-12-27: BACKWARDS INCOMPATIBILITY: The gym monitor is now a
wrapper. Rather than starting monitoring asenv.monitor.start(directory), envs are now wrapped as follows:env = wrappers.Monitor(env, directory). This change is on master
and will be released with 0.21.0.