Package: rrt-ml
RRT-MLRapidly exploring random trees with machine learning - learned sampling distributions, local reinforcement learning controller and learned supervised distance function for car-like mobile robotsTable of ContentsAbout The ProjectGetting StartedPrerequisitesInstallationUsageInterfaceConfiguring an ExperimentLicenseAcknowledgmentsAbout The ProjectThis project unites optimal rapidly exploring random trees (RRT*) with the following machine learning techniques:Learned distribution of robot configurations (Itcher et. al., 2017)Reinforcement learning agent trained with DDPG (Lillicrap et. al., 2015) and MEGA (Pitis et. al., 2020) as a local controllerOrdinal supervised learning (Shi et. al. 2021) of a distance metric induced by the agentThe experiments are conducted in PyBullet with a car-like mobile robot in a narrow passage type of scenario.
The code allows the training and testing of each machine learning modules individually, but also in the context of the broader RRT* approach.Getting StartedFollow the instructions below to install the package.PrerequisitesYou need to installPyTorchandPyBullet.TorchThe easiest way is to use conda.If you have a CUDA-enabled GPU:conda install pytorch torchvision torchaudio pytorch-cuda=11.6 -c pytorch -c nvidiaIf you want to use CPU only:conda install pytorch torchvision torchaudio cpuonly -c pytorchPyBulletInstalling PyBullet with pip requires build tools. I recommend using conda:conda install -c conda-forge pybulletInstallationInstall the package with pip:pip install rrt-ml(back to top)UsageYou can run experiments with different parameters for each module. Run from the command line or from any python file.InterfaceRun experiments from the command line:rrt-ml (--rl | --sl | --rrt) (--train | --test) [--hyper] [--config CONFIG]The first option controls which algorithm will run:--rl: reinforcement learning agent as a local controller--sl: "sample learner" to learn sampling distributions for optimal motion planning with RRT*--rrt: optimal rapidly-exploring random treeThe second option controls how should it run:--train: train a machine learning model (RL or SL) or grow the RRT* tree--test: generate various results (only after training)The third option,--hyper, determines whether a search for hyperparameters should be made. If the second option is--train, then each possible config of the grid search is trained. If the second option is--test, then a bunch of results will be produced in order to compare different models (only after training).The fourth option specify the name of the config. Below there are instructions to create a config file or directly run it, without using the command line.Configuring an ExperimentCreate a python file anywhere. Create aMasterConfigobject and set a name to it:from rrt_ml.utilities.configs import MasterConfig


cfg = MasterConfig()

cfg.general.config_name_or_prefix = "MyExperimentConfigName"Your IDE should auto-completecfgand show all nested attributes. You can change various settings for all algorithms individually:# Change reinforcement learning agent config
cfg.rl.general.gamma = 0.98
cfg.rl.actor.lr = 0.01
cfg.rl.critic.lr = 0.01

# Change sample learner (beta-cvae) config
cfg.sl.loss.beta = 5
cfg.sl.train.batch_size = 128

# Change RRT* config
cfg.rrt.general.seed = 1
cfg.rrt.sample.goal_prob = 0.05
cfg.rrt.names.rl = 'best'  # Use config name 'best' as controller for RRT
cfg.rrt.names.sl = 'best'  # Use config name 'best' as sample generatorYou can also set a grid search over any setting that exists inside therl|sl|rrtattributes above. Below an example on how to setup a grid search over the hyper-parameters for the RL agent:cfg.hyperparams.rl.general.gamma = [0.95, 0.96, 0.97, 0.98, 0.99]
cfg.hyperparams.rl.actor.lr = [0.1, 0.01, 0.001]
cfg.hyperparams.rl.net.activ = ['ReLU', 'GeLU']Now you can save the config to run later from the command line:cfg.save()Training an RL agent with this example config from the command line:rrt-ml --rl --train --config=MyExperimentConfigNameInstead of saving and the running from the command line, you can simply run it from the python file you are editing, by adding this:cfg.run(algorithm_to_run='rl', train_or_test='train', hyperparam_search_or_test=False)The parameters of therunmethod are equal to the ones from the command line.(back to top)LicenseDistributed under the MIT License. SeeLICENSE.txtfor more information.(back to top)AcknowledgmentsPyBulletPyTorchModular RLPython RoboticsCoral-PyTorchTorchEnsembleOptuna(back to top)