Package: actorch
Welcome toactorch, a deep reinforcement learning framework for fast prototyping based onPyTorch. The following algorithms have been implemented so far:REINFORCEAdvantage Actor-Critic (A2C)Actor-Critic Kronecker-Factored Trust Region (ACKTR)Trust Region Policy Optimization (TRPO)Proximal Policy Optimization (PPO)Advantage-Weighted Regression (AWR)Deep Deterministic Policy Gradient (DDPG)Distributional Deep Deterministic Policy Gradient (D3PG)Twin Delayed Deep Deterministic Policy Gradient (TD3)Soft Actor-Critic (SAC)💡 Key featuresSupport forOpenAI GymnasiumenvironmentsSupport forcustom observation/action spacesSupport forcustom multimodal input multimodal output modelsSupport forrecurrent models(e.g. RNNs, LSTMs, GRUs, etc.)Support forcustom policy/value distributionsSupport forcustom preprocessing/postprocessing pipelinesSupport forcustom exploration strategiesSupport fornormalizing flowsBatched environments (both for training and evaluation)Batchedtrajectory replayBatched anddistributional value estimation(e.g. batched and distributionalRetraceandV-trace)Data parallel and distributed data parallelmulti-GPU training and evaluationAutomaticmixed precision trainingIntegration withRay Tunefor experiment execution andhyperparameter tuningat any scaleEffortless experiment definition throughPython-based configuration filesBuilt-invisualization toolto plot performance metricsModularobject-orienteddesignDetailedAPI documentation🛠️️ InstallationFor Windows, make sure the latestVisual C++ runtimeis installed.Using PipFirst of all, installPython 3.6 or later. Open a terminal and run:pipinstallactorchUsing Conda virtual environmentClone or download and extract the repository, navigate to<path-to-repository>/binand run
the installation script (install.shfor Linux/macOS,install.batfor Windows).actorchand its dependencies (pinned to a specific version) will be installed in
aCondavirtual environment namedactorch-env.NOTE: you can directly useactorch-envand theactorchpackage in the local project
directory for development (seeFor development).Using Docker (Linux/macOS only)First of all, installDockerandNVIDIA Container Runtime.
Clone or download and extract the repository, navigate to<path-to-repository>, open a
terminal and run:dockerbuild-t<desired-image-name>.# Build imagedockerrun-it--runtime=nvidia<desired-image-name># Run container from imageactorchand its dependencies (pinned to a specific version) will be installed in the
specified Docker image.NOTE: you can directly use theactorchpackage in the local project directory inside
a Docker container run from the specified Docker image for development (seeFor development).From sourceFirst of all, installPython 3.6 or later.
Clone or download and extract the repository, navigate to<path-to-repository>, open a
terminal and run:pipinstall.For developmentFirst of all, installPython 3.6 or laterandGit.
Clone or download and extract the repository, navigate to<path-to-repository>, open a
terminal and run:pipinstall-e.[all]pre-commitinstall-fThis will install the package in editable mode (any change to the package in the local
project directory will automatically reflect on the environment-wide package installed
in thesite-packagesdirectory of your environment) along with its development, test
and optional dependencies.
Additionally, it installs agit commit hook.
Each time you commit, unit tests, static type checkers, code formatters and linters are
run automatically. Runpre-commit run --all-filesto check that the hook was successfully
installed. For more details, seepre-commit's documentation.▶️ QuickstartIn this example we will solve theOpenAI GymnasiumenvironmentCartPole-v1usingREINFORCE.
Copy the following configuration in a file namedREINFORCE_CartPole-v1.py(with the
same indentation):importgymnasiumasgymfromtorch.optimimportAdamfromactorchimport*experiment_params=ExperimentParams(run_or_experiment=REINFORCE,stop={"training_iteration":50},resources_per_trial={"cpu":1,"gpu":0},checkpoint_freq=10,checkpoint_at_end=True,log_to_file=True,export_formats=["checkpoint","model"],config=REINFORCE.Config(train_env_builder=lambda**config:ParallelBatchedEnv(lambda**kwargs:gym.make("CartPole-v1",**kwargs),config,num_workers=2,),train_num_episodes_per_iter=5,eval_freq=10,eval_env_config={"render_mode":None},eval_num_episodes_per_iter=10,policy_network_model_builder=FCNet,policy_network_model_config={"torso_fc_configs":[{"out_features":64,"bias":True}],},policy_network_optimizer_builder=Adam,policy_network_optimizer_config={"lr":1e-1},discount=0.99,entropy_coeff=0.001,max_grad_l2_norm=0.5,seed=0,enable_amp=False,enable_reproducibility=True,log_sys_usage=True,suppress_warnings=True,),)Open a terminal in the directory where you saved the configuration file and run
(if you installedactorchin a virtual environment, you first need to activate
it, e.g.conda activate actorch-envif you installedactorchusingConda):pipinstallgymnasium[classic_control]# Install dependencies for CartPole-v1actorchrunREINFORCE_CartPole-v1.py# Run experimentNOTE: training artifacts (e.g. checkpoints, metrics, etc.) are saved in nested subdirectories.
This might cause issues on Windows, since the maximum path length is 260 characters. In that case,
move the configuration file (or setlocal_dir) to an upper level directory (e.g.Desktop),
shorten the configuration file name, and/or shorten the algorithm name
(e.g.DistributedDataParallelREINFORCE.rename("DDPR")).Wait for a few minutes until the training ends. The mean cumulative reward over
the last 100 episodes should exceed 475, which means that the environment was
successfully solved. You can now plot the performance metrics saved in the auto-generatedTensorBoard(or CSV) log files usingPlotly(orMatplotlib):pipinstallactorch[vistool]# Install dependencies for VisToolcdexperiments/REINFORCE_CartPole-v1/<auto-generated-experiment-name>
actorchvistoolplotlytensorboardYou can find the generated plots inplots.Congratulations, you ran your first experiment!Seeexamplesfor additional configuration file examples.HINT: since a configuration file is a regular Python script, you can use all the
features of the language (e.g. inheritance).🔗 Useful linksIntroduction to deep reinforcement learningHyperparameter tuning with Ray TuneandOptuna integrationLogging with Ray TuneMonitoring jobs with Ray DashboardSetting up a cluster with Ray Cluster@ Citation@misc{DellaLibera2022ACTorch,
  author = {Luca Della Libera},
  title = {{ACTorch}: a Deep Reinforcement Learning Framework for Fast Prototyping},
  year = {2022},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/lucadellalib/actorch}},
}📧 Contactluca.dellalib@gmail.com