Package: deepbots
Deepbots is a simple framework which is used as "middleware" between the free
and open-sourceCyberbotics' Webotsrobot simulator
and Reinforcement Learning algorithms. When it comes to Reinforcement Learning
theOpenAI gymenvironment has been established as
the most used interface between the actual application and the RL algorithm.
Deepbots is a framework which follows the OpenAI gym environment interface
logic in order to be used by Webots applications.InstallationPrerequisitesInstall WebotsWindowsLinuxmacOSInstall Python version 3.X(please
refer toUsing Pythonto select the proper Python version for your system)Follow theUsing Pythonguide provided by WebotsWebots provides a basic code editor, but if you want to usePyCharmas your IDE refer tousing PyCharm IDEprovided by WebotsYou will probably also need a backend library to implement the neural networks,
such asPyTorchorTensorFlow. Deepbots interfaces with RL agents
using the OpenAI gym logic, so it can work with any backend library you choose
to implement the agent with and any agent that already works with gym.Install deepbotsDeepbots can be installed through the package installerpiprunning the following command:pip install deepbotsOfficial resourcesOnthe deepbots-tutorials repositoryyou can find the official tutorials for deepbotsOnthe deepworlds repositoryyou
can find examples of deepbots being used.Feel free to contribute your
own!CitationConference paper (AIAI2020):https://link.springer.com/chapter/10.1007/978-3-030-49186-4_6@InProceedings{10.1007/978-3-030-49186-4_6,author="Kirtas, M.and Tsampazis, K.and Passalis, N.and Tefas, A.",title="Deepbots: A Webots-Based Deep Reinforcement Learning Framework for Robotics",booktitle="Artificial Intelligence Applications and Innovations",year="2020",publisher="Springer International Publishing",address="Cham",pages="64--75",isbn="978-3-030-49186-4"}How it worksFirst of all let's set up a simple glossary:World: Webots uses a tree structure to represent the different entities in
the scene. The World is the root entity which contains all the
entities/nodes. For example, the world contains the Supervisor and Robot
entities as well as other objects which might be included in the scene.Supervisor: The Supervisor is an entity which has access to all other
entities of the world, while having no physical presence in it. For example,
the Supervisor knows the exact position of all the entities of the world and
can manipulate them. Additionally, the Supervisor has the Supervisor
Controller as one of its child nodes.Supervisor Controller: The Supervisor Controller is a python script which
is responsible for the Supervisor. For example, in the Supervisor Controller
script the distance between two entities in the world can be calculated.Robot: The Robot is an entity that represents a robot in the world. It
might have sensors and other active components, like motors, etc. as child
entities. Also, one of its children is the Robot Controller. For example,epuckandTIAGoare robots.Robot Controller: The Robot Controller is a python script which is
responsible for the Robot's movement and sensors. With the Robot Controller
it is possible to observe the world and act accordingly.Environment: The Environment is the interface as described by the OpenAI
gym. The Environment interface has the following methods:get_observations(): Return the observations of the robot. For example,
metrics from sensors, a camera image etc.step(action): Each timestep, the agent chooses an action, and the
environment returns the observation, the reward and the state of the
problem (done or not).get_reward(action): The reward the agent receives as a result of their
action.is_done(): Whether it’s time to reset the environment. Most (but not all)
tasks are divided up into well-defined episodes, and done being True
indicates the episode has terminated. For example, if a robot has the task
to reach a goal, then the done condition might happen when the robot
"touches" the goal.reset(): Used to reset the world to the initial state.In order to set up a task in Deepbots it is necessary to understand the
intention of the OpenAI gym environment. According to the OpenAI gym
documentation, the framework follows the classic “agent-environment loop”.
"Each timestep, the agent chooses anaction, and the environment returns anobservationand areward. The process gets started by callingreset(),
which returns an initialobservation."Deepbots follows this exact agent-environment loop with the only difference
being that the agent, which is responsible to choose an action, runs on the
Supervisor and the observations are acquired by the robot. The goal of the
deepbots framework is to hide this communication from the user, especially from
those who are familiar with the OpenAI gym environment. More specifically,SupervisorEnvis the interface which is used by the Reinforcement Learning
algorithms and follows the OpenAI Gym environment logic. The Deepbots framework
provides different levels of abstraction according to the user's needs.
Moreover, a goal of the framework is to provide different wrappers for a wide
range of robots.Deepbots also provides a default implementation of thereset()method,
leveraging Webots' built-in simulation reset functions, removing the need for
the user to implement reset procedures for simpler use-cases. It is always
possible to override this method and implement any custom reset procedure, as
needed.Emitter - receiver schemeCurrently, the communication between theSupervisorand theRobotis
achieved via anemitterand areceiver. Separating theSupervisorfrom
theRobot, deepbots can fit a variety of use-cases, e.g. multipleRobotscollecting experience and aSupervisorcontrolling them with a single agent.
The way Webots implementsemitter/receivercommunication requires messages
to be packed and unpacked, which introduces an overhead that becomes
prohibiting in use-cases where the observations are high-dimensional or long,
such as camera images. Deepbots provides another partially abstract class that
combines theSupervisorand theRobotinto one controller and circumvents
that issue, while being less flexible, which is discussedlater.On one hand, theemitteris an entity which is provided by Webots, that
broadcasts messages to the world. On the other hand, thereceiveris an
entity that is used to receive messages from theWorld. Consequently, the
agent-environment loop is transformed accordingly. Firstly, theRobotuses
its sensors to retrieve the observation from theWorldand in turn uses theemittercomponent to broadcast this observation. Secondly, theSupervisorreceives the observation via thereceivercomponent and in turn, the agent
uses it to choose an action. It should be noted that the observation the agent
uses might be extended from theSupervisor. For example, a model might use
LiDAR sensors installed on theRobot, but also the Euclidean distance between
theRobotand an object. As it is expected, theRobotdoes not know the
Euclidean distance, only theSupervisorcan calculate it, because it has
access to all entities in theWorld.You can follow theemitter-receiver scheme tutorialto get started and work your way up from there.Combined Robot-Supervisor schemeAs mentioned earlier, in use-cases where the observation transmitted between
theRobotand theSupervisoris high-dimensional or long, e.g. high
resolution images taken from a camera, a significant overhead is introduced.
This is circumvented by inheriting and implementing the partially abstractRobotSupervisorthat combines theRobot controllerand theSupervisor Controllerinto one, forgoing allemitter/receivercommunication. This new controller runs on theRobot, but requiresSupervisorprivileges and is limited to oneRobot, oneSupervisor.You can follow therobot-supervisor scheme tutorialto get started and work your way up from there. We recommended this
tutorial to get started with deepbots.Abstraction LevelsThe deepbots framework has been created mostly for educational purposes. The
aim of the framework is to enable people to use Reinforcement Learning in
Webots. More specifically, we can consider deepbots as a wrapper of Webots
exposing an OpenAI gym style interface. For this reason there are multiple
levels of abstraction. For example, a user can choose if they want to use CSVemitter/receiveror if they want to make an implementation from scratch. In
the top level of the abstraction hierarchy is theSupervisorEnvwhich is the
OpenAI gym interface. Below that level there are partially implemented classes
with common functionality. These implementations aim to hide the communication
between theSupervisorand theRobot, as described in the two different
schemes ealier. Similarly, in theemitter/receiverscheme theRobotalso
has different abstraction levels. According to their needs, users can choose
either to process the messages received from theSupervisorthemselves or use
the existing implementations.AcknowledgmentsThis project has received funding from the European Union's Horizon 2020
research and innovation programme under grant agreement No 871449 (OpenDR).
This publication reflects the authors’ views only. The European Commission is
not responsible for any use that may be made of the information it contains.Contributors ✨Thanks goes to these wonderful people
(emoji key):Manos Kirtas💻Kostas Tsampazis💻Jiun Kai Yang💻MentalGear🤔Dreamtale🐛Nikolaos Kokkinis-Ntrenis💻📖🤔This project follows theall-contributorsspecification. Contributions of any kind welcome!Special thanks toPapanikolaou Evangeliafor designing project's logo!