Package: enlil
EnlilEnlil is a solution to ease and speed up the containerization of entire robotic infrastructures withROS.
Enlil does so by automatically generating all files required byDocker Composein order to both containerize and simultaneously orchestrate multipleROSworkspaces.
All the files generated by Enlil are ready-to-use.It is named after the sumerian God of air andcloud.Table of ContentsEnlilTable of ContentsFeaturesTechnologiesInstallationManual InstallationPip InstallationCLI UsageGeneration PipelineTemplate VariablesSyntax DocumentationAreasRobotsImagesPackagesGlobalsProject StructureArticlesLicenseFeaturesConvertsYAMLfiles into complete and ready-to-useDockerfileandDocker Composefiles.Generates containerized ROS packages on-the-fly.Support for simultaneous usage of multiple ROS distributions and versions.Supports usage of private code bases.Full support of existingDocker Composesyntax.Completely open-source.TechnologiesEnlil is completely written inPython 3.
It depends solely on the following external modules:Docopt- to generate with ease a command line interface.Jinja2- to generate all output files.Pip3- package-management system for module installation.InstallationManual InstallationInstall the dependencies running the following command:gitclonehttps://gitlab.inesctec.pt/pedro.m.melo/enlilcdenlil
pip3install-rrequirements.txtIn case of manual installtion it is recommended to create analiasfor Enlil for ease of usage with:echo'alias enlil="python3 <PATH_TO_ENLIL>/enlil.py"'>>~/.bashrc
.~/.bashrcYou may now be able to run Enlil with:enlil<ARGUMENTS>Pip InstallationYou can install Enlil directly throughpip:pip3installenlilYou may now be able to run Enlil with:enlil<ARGUMENTS>CLI UsageEnlil has a command line interface, that expects the following arguments:CLI ARGUMENTDESCRIPTIONREQUIREDconfiguration_filethe inputYAMLconfiguration fileYesoutput_filepath and name for finalDocker ComposefileNoIf not specified, the CLI argumentoutput_filecreates a filedocker_compose.ymlon the same folder where Enlil is execited.For more information about usage the following command can be executed:enlil-hEnlil must be provided anYAMLconfiguration file (CLI argumentconfiguration_file) as shown:---areas:-id:IILAB-fastenrobots:-fridayrobots:-id:fridayros:melodicimages:-ros-developmentpackages:-rosbridge-universal_robotvars:-UR_VERSION:10-ROSBRIDGE_PORT:9091images:-id:ros-developmentimage:rosstdin_open:truetty:truevolumes:-./ros_workspace:/ros_workspacepackages:-id:universal_robotpath:./packages/git:-https://github.com/ros-industrial/universal_robot:{{ROBOT_ROS_DISTRO}}-develcommand:"roslaunchur_gazebour{{UR_VERSION}}.launchgui:=false"-id:rosbridgepath:./packages/apt:-ros-{{ROBOT_ROS_DISTRO}}-rosbridge-server-ros-{{ROBOT_ROS_DISTRO}}-tfports:-"{{ROSBRIDGE_PORT}}:{{ROSBRIDGE_PORT}}"command:"roslaunchrosbridge_serverrosbridge_websocket.launchport:={{ROSBRIDGE_PORT}}"globals:-id:webvizimage:cruise/webvizports:-"8081:8080"Generation PipelineEnlil divides input data intofive distinct typesofentities:Area: eacharearepresents a physical space where multiple robots may work simultaneously.Robot: eachrobotrepresents a robotic agent runningROSand performing some action inside a given area.Image: eachimagerepresentsan existingDockerimage that is part of theROSstack running inside the robot.Package: eachpackagerepresents, in oposition toimages, aROSnode that is stillto be containerized. The requiredDockerfilefor suchROSnodes are generated alongside the finalDocker Composefile.Global (Entity): eachglobalconsists of an existingDockerimage that is relevant to the infrastructure and that may or may not interact with theROSnodes.The pipeline adopted by Enlil to generate the finalDocker Composefile andDockerfileis show in the next picture.The pipeline is composed of the following stages:Parsing Stage (I): responsible for the parsing of the input file and initialization of typedentities.Decoding Stage (II): responsible for the decoding of template variables, as explained in detail in the next section.Converting Stage (III): responsible for transforming the decoded data into a more useful format for rendering.Rendering Stage (IV): responsible for generating the finalDocker Composefile and all requiredDockerfiles.Template VariablesEnlil emphasizes reusage of code having an inner templating mechanism.All textual information placed in between curly braces is interpreted by Enlil to be a template variable. Similar to variables in a programming language, template variables work as placeholders for useful information. They are a powerful tool to hide complexity allowing developers to describe more complex scenarios with less effort by reutilizing a samepackageorimagedeclaration multiple times within multiple robots.There are two types of template variables:the variablesdefined by developersat the configuration file level. These variables may also be used be developers.the variablesset automaticallyby Enlil and that are used internally.DEFAULT TEMPLATE VARIABLELEVELDESCRIPTION{{ AREA_ID }}Areathearea's unique identifier{{ ROBOT_ID }}Robottherobot's unique identifier{{ ROBOT_ROS_DISTRO }}RobottheROSdistribution associated with therobot{{ ROBOT_ROS_METADATA }}Roboteither the port number for theROS master node(ROSdistributions) or the value of environment variableROS_DOMAIN_ID(ROS2distributions)As shown in the next section, developers can define their own template variables developers by means of the fieldvars.The decoding of template variables into real information is performed at the second stage of the presented pipeline.
The decoding stage takes advantage of the hierarchy existent between the different types of entities. Starting with all entities of typeareathe proposed solution analyzes all data about therobotswithin a single area. Then, it individually analyzes the data of eachimagewithin each robot. The same is done for everypackagewithin the robot. This way, information can be passed from entities of typeareato entities of typerobot, and from these to entities of typesimageandpackage.As a suggestion, developers may combine Enlil's template variables with Docker Compose's template variables ( enclosed in between ${...} ) to obtain an even higher degree of code reusage.Syntax DocumentationEnlil expects the inputYAMLconfiguration file to be divided into the following five sections:areas,robots,images,packagesandglobals. Each of these sections contains data about the respective typedentitiesand is also expected to follow a specific syntax, as explained in detail in this section. Enlil returns a proper error anytime a section does not comply with the said syntax.Always keep in mind thatthe syntax adopted by Enlil is fully compliant with the syntax employed byDocker Compose.Enlil will only take into consideration the fields specified below. Therefore, any extra fields declared within sectionsimagesandpackageswill be passed to the finalDocker Composefile. This feature allows you to to declare, for example,Dockervolumes andDockernetworks just as you would normally while using regularDockertools. In spite of not being used directly by Enlil, all information declared within the said sections goes through the entire generation pipeline and may take advantages of the already explained template variables.AreasThis section is mandatory for all input files!FIELDDESCRIPTIONREQUIREDidthearea's unique identifieryesrobotsa list containing  the unique identifiers of all the robots operating within theareayesImportant considerations:Each area has a private network associated to it, to which all its robots are connected and may use to communicate.All areas are required to have at least one robot.RobotsThis section is mandatory for all input files!FIELDDESCRIPTIONREQUIREDidtherobot's unique identifieryesimagesa list containing the unique identifiers of all the existingDockerimages to be run within the robotyes (but only ifpackagesis not declared)packagesa list containing the unique identifiers of all theDockercontainer to be generated on-the-fly and run within the robotyes (but only ifimagesis not declared)roscorresponds to the ROS distribution to be run inside the robotyesvarsa list of template variables associated with the robotnoImportant considerations:All robots must be declared within an area or otherwise they will not be considered by Enlil.Robots from the same area can communicate with themselves (using their area's network) while robots from different areas can't communicate between themselves at all.At least one of the fieldsimagesandpackagesmust be declared.Fieldrosfollows the format<DISTRO:MASTER>whereDISTROis mandatory and corresponds to the name of the desiredROSdistribution andMASTERis optional and corresponds the the networking port in which theROS master nodewill be listening for communications. In case aROS2distribution is chosen then the value ofMASTERcorresponds to the desired value for environment variableROS_DOMAIN_ID. IfMASTERis ommited, then the default value of11311is chosen forROSdistributions and the default value of42is chosen forROS2distributions.Entries invarsfollow the format<KEY:VALUE>whereKEYis an unique identifier for the template variablewithin the robotandVALUEis the value for the variabel itself.At the moment onlyROSdistributionsmelodic,noeticandfoxywere tested.ImagesFIELDDESCRIPTIONREQUIREDidtheimage's unique identifieryesimagetheDockerimage to be usedyesImportant considerations:Fieldimagefollows the format<IMAGE:TAG>whereIMAGEis mandatory and corresponds to the name of the desiredDockerimage to use whileTAGspecifies the desired version of the specifiedDockerimage. Contrarily to theDockerstandard, where the default value for an image tag islatest, the default value used by Enlil always matches theROSdistribution of the robot where the entity is declared.PackagesFIELDDESCRIPTIONREQUIREDaptlist of required software modules to be installed via the inbuilt package manageryes (but only if neithergitorrosinstallare declared)commandthe command to be run after theDockercontainer is initializedyesfieldslist of complete file paths required inside the final containernogitlist ofURLs for requiredgitrepositoriesyes (but only if neitheraptorrosinstallare declared)idthepackage's unique identifieryespaththe output path for thepackage'sDockerfileyesrosinstallpath to a.rosinstallfile listing code dependencies to be installedyes (but only if neithergitoraptare declared)sshlist of complete file paths required to install private code dependenciesnoImportant considerations:The generatedDockerfilecan be used freely to build and deploy customizedDockerimages ofROSnodes.Whenever the fieldsgitorrosinstallare declared the creation of the finalDockercontainer is done in two stages, allowing for the secure usage of private code bases using existingSSH deploy keys. During the first stage the contents of the host folderssh/are copied into the container before the downloading of any declared repository. Only the downloaded repositories are then passed to the final stage, making access to theSSH deploy keysimpossible through the final container orDockerimage.At least one of the fieldsapt,gitandrosinstallmust be declared.The baseDockerimages used areros:melodic,ros:noeticandros:foxy, all of which come with theaptpackage manager already installed. Checkhttps://hub.docker.com/_/rosfor more information.GlobalsImportant considerations:All declared globals are placed by Enlil in the finalDocker Composefileas isand without any changes.Use this section to declare everything you would otherwise declare in a normalDocker Composefile.The default values for globals match the ones used byDockerandDocker Compose. For instance the defaulttagfor anDockerimage is againlatest.Since they are passed unaltered to the finalDocker Composeglobals should not use template variables.Project StructureThe source code for this project is rganized as follows:docs: contains all the information regarding documentation, both textual and images.examples: contains exampleYAMLconfiguration files that serve as example for developers trying to learn how to use Enlil.pipeline: contains the source code for the presented file generation pipeline:/decoder: contains the source code responsible for stages II and III of the presented pipeline./loader: contains the source code responsible for stages I of the presented pipeline./entities: contains the implementations for each type of entity./renderizer: contains the source code responsible for stages IV of the presented pipeline./templates: contains the templates used to generated the finalDocker Composefile and necessaryDockerfiles.tests: contains the unit tests for the application.enlil.py: application entry point.ArticlesEnlil is featured at theINDINconference (2021 edition).More information about the submitted article will be placed soon.LicenseEnlil is itself free, open-source and is deployed with MIT license.