Package: corl
The following Site/Repository is currently under construction. We are still porting items and updating instructions for github site/CICD.Autonomy Capability Team (ACT3) Home Pagehttps://www.afrl.af.mil/ACT3/The Air Force Research Laboratory’s (AFRL) Autonomy Capability Team (ACT3) is an AI Special Operations organization whose mission is to Operationalize AI at Scale for the Air Force. Commissioned by the AFRL Commander, ACT3 leverages an innovative ‘start-up’ business model as an alternative approach to the traditional AFRL Technical Directorate R&D model by combining the blue sky vision of an academic institution; the flexibility of an AI startup; and the discipline of a production development company. ACT3 integrates the world’s best under one roof. The goal of the ACT3 business model is to define the shortest path to successful transition of solutions using AFRL’s internal expertise and collaborations with the best academic and commercial AI researchers in the world. Successful implementation may mean new technology or new implementation of existing technology.ACT3 RL CoreCore act3 reinforcement learning library- The Core Reinforcement Learning library is intended to enable scalable deep reinforcement learning experimentation in a manner extensible to new simulations and new ways for the learning agents to interact with them. The hope is that this makes RL research easier by removing lock-in to particular simulations.The work is released under the follow APRS approval.DateRelease NumberDescription2022-05-20AFRL-2022-2455Initial release ofACT3 CoRL2023-03-02APRS-RYZ-2023-01-00006Second release ofACT3 CoRLRelated Publications:https://breakingdefense.com/2023/01/inside-the-special-f-16-the-air-force-is-using-to-test-out-ai/https://www.wpafb.af.mil/News/Article-Display/Article/3244878/afrl-aftc-collaborate-on-future-technology-via-weeklong-autonomy-summit/https://aerospaceamerica.aiaa.org/year-in-review/demonstrating-and-testing-artificial-intelligence-applications-in-aerospace/Documentationhttps://act3-ace.github.io/CoRL/Framework Overview - Hyper configurable environment enabling rapid exploration and integration pathwaysA framework for developing highly-configurable environments and agentsDevelop core components in pythonConfigure experiments/agents in json/ymlProvides tooling to help validate configuration files and give useful feedback when files are misconfiguredDesigned with integration in mindDramatically reduce development time to put trained agents into an integration or using a different simulationCan work with any training frameworkCurrently limited to Ray/RLLIB due to multi-agent requirementEnvironment pre-written, users implement pluginsSimulatorPlatforms & Platform PartsGluesRewardsDonesValidators -Configuration guarantees for enabling validation of user configuration going into the major componentsAll major CoRL python components have a validatorValidators are python dataclasses implemented through the pydantic libraryValidators check and validate user configuration arguments going into the major componentsIf a component successfully initializes, the validators guarantee the developer that the data listed in the validator is available to themIf a component doesn’t initialize, a nice helpful error message is automatically produced by pydanticAdds a pseudo static typing to python classesEpisode Parameter Provider (EPP) -Domain Randomization & Curriculum Learning at Environment, Platform, and Agent based on trainingAn important tool for RL environments is the ability to randomize as much as possibleStarting conditions / goal location / etc.This leads to more general agents who are more robust to noise when solving a taskAnother tool sometimes used in RL is curriculum learning (CL)Starting from an easier problem and gradually making the environment match the required specifications can significantly speed up trainingCoRL Agents and the environment all have an epp, which provides simulator or user defined parameters to be used during a specific episodeSimulator classes know what parameters they expect to setup an episodeConfiguration parameters to the various functors can all be provided from an EPPAn EPP can also update parameters over the course of trainingMake a goal parameter harder based on the agents win rateOpen the environment up to wider bounds once the agent initially starts to learnSimulator Class -Extensible interface for transitioning between Dubins and other simulator backendsResponsible for setting up the world for a agents to manipulateSetting up and configuring the simulationCreating the simulation platformsPlacing those platforms in the worldResponsible for knowing how to advance the simulation when requestedThe simulation returns a simulation state when reset or advanced that rewards or done conditions can useThis state contains at least both the time and the list of simulation platformsResponsible for saving any information about the current training episodeSaving video/logsSimulator Platforms + parts -Extensible base interface for parts to be added to planforms with an integration focus.Simulation platforms represent some object that can be manipulated in the simulationCar/plane/robot/etc.Have a config file to allow modes of configurationEach platform has a set of parts attached to itParts take simulation specific code and wrap it in an interface that allows agents to read from and write to themParts do nothing unless a user configures a connection between the agent and a part using a glue (to be explained)Parts could include things such as a throttle, a game button, a steering wheel, etc.Parts are registered to a simulator using a stringSensor_Throttle,Controller_Throttle, etc.Glues -Connecting layers to allow exposing observable state to rewards, termination/goal criteria, and agentsA stateful functorResponsible for producing actions and observations for the agentMay directly read/write to parts or other gluesGlues reading/writing to each other is called “wrapping”Glues implement the composable and reusable behavior useful for developersCommon glues turn any sensor part into an obs and apply actions to any controller partWrapper glues can implement behaviors such as framestacking, delta actionsMay not directly read from the simulation, only interface through partsRewards, Dones (Goal & Termination) -Composable functors common interface for sharing rewards and termination criteria in a stateful mannerComposable state functorsRewards generate the current step reward for the agentDones evaluate if the episode should stop on the current timestepThese done’s can be triggered for either success or failureBoth Done and Reward Functors can view the entire simulation state to reward agentsDone conditions typically add to the state when they trigger to signify what type of Done they areWIN/LOSE/DRAWRewards are processed after Done conditions during anupdate, so rewards can read these labelsThere can be an arbitrary number of reward or done functors for an agentAgent + Experiment ClassAgent ClassResponsible for holding all of the Done/Reward/Glue functors for a given agentCan be many agent classes per platformWhen one agent class on a platform reaches a done, all on that platform doDifferent subclasses may process information in different ways or do different thingsExperiment ClassResponsible for setting up an experiment and running itConfigures and creates the environmentCreates and configures the agent classesUse of this class allows for any arbitrary RL training framework to be used as the backend for trainingCoRL Integration and Simulator SwappingIn CoRL all simulation specific components must be registered and retrieved from a plug-in libraryAs long as a specific simulator has all of the parts registered to it that an agent needs, CoRL can swap the simulator and parts out from under an agent seamlesslyAs long as the parts for the two simulators have the same properties (in terms of sensed value bounds or controller inputs) there is no difference to the agent between the two and the regular environment can be used for integrationBesides integration this also allows for cross simulation evaluation or training of an agent to be resumed in another simulatorBenifitsCoRL helps make RL environment development significantly easierCoRL provides hyper configurable environments/agents and experimentsInstead of a new file every time a new observation is added, now just add a few lines of configMakes it possible to reuse glues/dones/rewards between different tasks if they are generalProvides tools to use both domain randomization and curriculum learning through EPPAn integration first focus means that integrating agents to the real world or different simulators is significantly easierInstallInstall the source - Miniconda - local host:Miniconda Install Instruction# Create a virtual environment to install/run codecondacreate-nCoRLpython==3.10.4# Activate the virtual environmentcondaactivateCoRL# install poetrypipinstallpoetry# Install the CoRL dependenciespoetryinstall# Pre-commit setuppre-commitinstallHow to install pip packageBuildHow to build the wheel fileThe following project supports building python packages viaPoetry.# Create a virtual environment to install/run codecondacreate-nCoRLpython==3.10.4# Activate the virtual environmentcondaactivateCoRL# install poetrypipinstallpoetry# Build the CoRL packagepoetrybuildHow to build the documentations - LocalThe follow project is setup to useMKDOCSwhich is a fast, simple and downright gorgeous static site generator that's geared towards building project documentation. Documentation source files are written in Markdown, and configured with a single YAML configuration file.To build the documentation:mkdocs buildTo serve the documentation:mkdocs serveHow to build the Docker containersThe following project support development via Docker containers in VSCode. This is not strictly required but does provide the mode conveniet way to get started.Note:fuller documentation is available in the documentation folder or online docs.Setup the user env file:in code directory run the following script  -->./scripts/setup_env_docker.shBuild the Docker containers using compose:run the following commandGITHUB -->docker-compose -f docker-compose-github.yml buildACT3 -->docker-compose -f docker-compose.yml buildDocker-Compose UtilitiesTest UTILITIES with DOCKERdocs:run the following commandGITHUB -->docker-compose -f docker-compose-github.yml up docsACT3 -->docker-compose -f docker-compose.yml up docspre-commit:run the following commandGITHUB -->docker-compose -f docker-compose-github.yml up pre-commitACT3 -->docker-compose -f docker-compose.yml up pre-commitpytest:run the following commandGITHUB -->docker-compose -f docker-compose-github.yml up pytestACT3 -->docker-compose -f docker-compose.yml up pytestPOETRY UTILITIES with DOCKERGenerate Poetry Lock File:run the following commandGITHUB -->docker-compose -f docker-compose-github.yml up poetry-lockACT3 -->docker-compose -f docker-compose.yml up poetry-lockGenerate Poetry Update Lock File:run the following commandGITHUB -->docker-compose -f docker-compose-github.yml up poetry-updateACT3 -->docker-compose -f docker-compose.yml up poetry-updateGenerate Poetry Create Dist:run the following commandGITHUB -->docker-compose -f docker-compose-github.yml up poetry-build-distACT3 -->docker-compose -f docker-compose.yml up poetry-build-distRunning base examplespython-mcorl.train_rl--cfgconfig/experiments/cartpole_v1.ymlInitial ContributorsInitial contributors include scientists and engineers associated with theAir Force Research Laboratory (AFRL),Autonomy Capability Team 3 (ACT3), and theAerospace Systems Directorate (RQ):Autonomous Air Combat Operations (AACO) TeamTerry Wilson(PM/PI)Karl Salva (System Integration)James Patrick (Modeling & Simulation)Benjamin Heiner (AI Behavior Training Lead)Training TeamCameron Long (ML Training)Steve Fierro (ML Training / System Integration)Brian Stieber (ML Training)Joshua Blackburn (ML Training / System Integration)Madison Blake (ML Infrastructure and Evaluation)Safe AutonomyKerianne HobbsJohn McCarrollUmberto RavaioliJamie CunninghamKyle DunlapNate HamiltonFundemental ResearchJared CulbertsonHamilton ClouseJustin MerrickIan CannonIan LeongVardaan GangalDesignation IndicatorControlled by: Air Force Research Laboratory (AFRL)Controlled by: AFRL Autonomy Capability Team (ACT3)LDC/Distribution Statement: DIST-APOCs:terry.wilson.11@us.af.mil(AFRL ACT3)bejamin.heiner@us.af.mil(AFRL ACT3)kerianne.hobbs@us.af.mil(AFRL ACT3)Notices and WarningsCiting CORLIf you use CORL in your work, please use the following bibtex@inproceedings{title={{CoRL}: Environment Creation and Management Focused on System Integration},author={Justin D. Merrick, Benjamin K. Heiner, Cameron Long, Brian Stieber, Steve Fierro, Vardaan Gangal, Madison Blake, Joshua Blackburn},year={2023},url={https://arxiv.org/abs/2303.02182}}White Paper ->CoRL: Environment Creation and Management Focused on System IntegrationAuthors ->Justin D. Merrick, Benjamin K. Heiner, Cameron Long, Brian Stieber, Steve Fierro, Vardaan Gangal, Madison Blake, Joshua BlackburnAbstract ->Existing reinforcement learning environment libraries use monolithic environment classes, provide shallow methods for altering agent observation and action spaces, and/or are tied to a specific simulation environment. The Core Reinforcement Learning library (CoRL) is a modular, composable, and hyper-configurable environment creation tool. It allows minute control over agent observations, rewards, and done conditions through the use of easy-to-read configuration files, pydantic validators, and a functor design pattern. Using integration pathways allows agents to be quickly implemented in new simulation environments, encourages rapid exploration, and enables transition of knowledge from low-fidelity to high-fidelity simulations. Natively multi-agent design and integration with Ray/RLLib (Liang et al., 2018) at release allow for easy scalability of agent complexity and computing power. The code is publicly released and available at this https URL.