Package: palaestrai
AboutpalaestrAI is a distributed framework to train and test all kinds of
autonomous agents. It provides interfaces to any environment, be it
OpenAI Gym or co-simulation environments via mosaik. palaestrAI can
train and test any kind of autonomous agent in these environments:
From Deep Reinforcement Learning (DRL) algorithms over model-based to
simple rule-based agents, all can train and test with or against
each other in a shared environment.In short, palaestrAI can……train and test one or more agent of any algorithm…place the agents on one or several environments at once,
depending on the agents’ algorithm…provides facilities to define and reproducibly run experimentspalaestrAI is the core framework of a whole ecosystem:hARL provides implementations of several DRL algorithms and
interfaces to existing DRL libraries.arsenAI provides all facilities needed for proper design
of experiments.palaestrai-mosaik is a interface to the mosaik co-simulation
softwarepalaestrai-environments provides a number of simple,
easy to use environments for playing with palaestrAIDocumentation can be found athttp://docs.palaestr.ai/Use CasespalaestrAI is the framework for the Adversarial Resilience Learning
(ARL) reference implementation. The ARL core concept consists of two
agents, attacker and defender agents, working an a common model of a
cyber-phyiscal system (CPS). The attacker’s goal is to de-stabilize the CPS,
whereas the defender works to keep the system in a stable and operational
state. Both agents do not perceive their opponent’s actions directly, but only
the state of the CPS itself. This imples that none of the agents knows whether
anything they perceive through their sensors is the result of the dynamics of
the CPS itself or of another agent’s action.  Also, none of the agents has an
internal model of the CPS. Attacker and defender alike have to explore the CPS
given their sensors and actuators independently and adapt to it. ARL is, in
that sense, suitable to a reinforcement learning approach.  Combined with the
fact the both agents do not simply learn the CPS, but also its respective
opponent, ARL implements system-of-systems deep reinforcement learning.InstallationpalaestrAI is mainly written in Python, with the usual third-party library
here and there. It provides asetup.pyfile just as any well-behaved Python
program. Use:pip install palaestraior for a fullstack installation with all subpackages from the framework
includingpalaestrai-arsenai,palaestrai-agents(harl),palaestrai-environments,palaestrai-mosaikas well as all packages frommidas-mosaikandpysimmods, run:pip install palaestrai[full]or for development, clone the palaestrAI repository fromhttps://gitlab.com/arl2/palaestrai/-/tree/developmentand run:pip install -e .[dev](With zsh, you need to escape the parenthesis like this:pip install -e .\[dev\])palaestraicomes with shell completion definitions for bash, zsh, and
fish, courtesy of click. They are installed in/etc/bash_completion.d,/etc/zsh_completion.d, and/etc/fish_completion.d, respectively. The/etcdirectory is relative to your installation root. I.e., if you
install palaestrai in your virtualenv directory~/palaestrai/venv, then
the completion files will be installed in~/palaestrai/venv/etc/{bash,zsh,fish}_completion.d. (Note thatpip install-edoes not install data files, but in that case you have the files
in the main directory of your git repo clone.) You will have to source the
appropriate file to enable shell completion. To do so automatically, add the
approporiate stanza to your~/.bashrc,~/.zshrc, or~/.fishrc,
respectively. E.g.,echo ‘. /etc/zsh_completion.d/palaestrai_completion.zsh’ >> ~/.zshrcAfter installation, you can start the dummy experiment run with:palaestrai experiment-start tests/fixtures/dummy_run.ymlHowever, to use all palaestrAI functionalities, some additional setup is
currently required. First, you need to install Docker. You find docker in
your preferred distributions
package manager or you can use the following commands to start a convience
script:curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.shAfterwards, you should add yourself to the Docker group:sudo usermod -aG docker $USERTry if your Docker installation works by typing:docker images(or any other valid docker command). If that’s not the case, the easiest way
is to restart your system (or use some magic commands to start the docker
daemon). The next step is to start a Docker container for the store:docker run -d --name timescaledb -p 5432:5432 -e POSTGRES_PASSWORD=password timescale/timescaledb:1.7.4-pg12You only need to run this command once. If you want to start the container
after a system reboot, the following command is sufficient:docker start timescaledbNext, you need to setup the database. Type the following three commands:docker exec -it timescaledb psql -U postgres
CREATE DATABASE Palaestrai;
exitNow, you need to add astore_uriin yourPalaestrai-runtime.conf.yaml.
If you followed this guide step-by-step, the following store_uri should do:store_uri: postgresql://postgres:password@172.17.0.1:5432/PalaestraiFinally, you can enable some loggers by changing their value from ERROR to
DEBUG or keep palaestrAI silent.UsageAfter installing (and, probably, setting log levels in the palaestrai-runtime.conf.yaml),
type:palaestrai database-createto create the data store.Everything that steers palaestrAI is defined throughexperiment runfiles.
They define which agents, which algorithms, and which environments to use.
You can then either start palaestrAI standalone from the command line as:palaestrai experiment-start RUNFILEE.g.,:palaestrai experiment-start my_run.ymlOr you use palaestrAI from your Jupyter notebook:import palaestrai
palaestrai.execute('my_run.yml')DevelopmentHandling a BugIf you find any kind of bug, please create an issue in GitLab:A prose description of the bug: what did you intend to do, what happened
instead?The error message, if there is any.The command line parameters and configurationYour version of Python you are using, and the version of all modules
(pip freezegives you that).ContributingThe typical work flow is as such:File a bug/feature/support request in the issue trackerCreate a feature branch to work on your issue. Name itbug-<num>-<shortname>for bugs,feature-<num>-<shortname>for new
features, etc.Provide a unit test for the bug/feature you have been working on.Fix the bug/work on the feature.Runblack-l79 ./src/palaestrai ./teststo auto-format the codeRuntoxand clean up all errors. (Runtox-efull-dockerto also run system tests using docker and docker-compose)Request a merge. The merge will happen after a code review;
work-in-progress code gets first merged intodevelopmentOnce the current development branch has ripened enough, it is merged tomaster. The master branch must contain code that is stable. New
releases are only tagged on master branch commits.Coding StyleHave a look at our architecture document and diagrams indoc/architecture.rst.We adhere toPEP8or black with line length of 79.Try hard to find fitting names for new modules and subdirectories. If you are
importing your own module asimport X as YandYis
differing semantically fromX, it might be the right time to change
the name ofX. Please refrain from abbreviated names if it is not
absolutely clear (in two years from now) what the abbreviation signifies.
Specifically, use short variable names in functions, if you like to, but stick
to commonly known and accepted abbreviations, such asfor i in listor
similar things. Avoid confusion with function names from the Python base
library, e.g., do not useexpas a shorthand for “experiment” (cf.math.exp(x, y).Supply docstrings for every class and public function. Otherwise, when you
find yourself writing comments, consider writing better, self-explaining code
instead. When adding “TODO” or “FIXME” comments, make sure somebody else can
understand and begin to work on them. Use type hinting wherever possible.Functions should not span more than one screen length.Copyright & AuthorsAll source code, except where otherwise mentioned, is Copyright (C) 2018, 2019
OFFIS e.V. Contributing authors are listed in order of their appeareance in
the file AUTHORS.The dynamic loader used in the command-line utility relies more or less
verbatim on code from the Python project. See the filedoc/python-license.The code inpalaestrai.typescomes from the OpenAIGymproject.  See the filedoc/gym-license.md.Related RepositoriesSee here for a list of palaestrAI-related repositories, for example interfaces to other software or simple agent/environment implementations:https://gitlab.com/arl2