Package: tril
TRILTransformers Reinforcement and Imitation Learning LibraryTRILis a modular library for Reinforcment Learning (RL) and Imitation Learning (IL) algorithm development with transformers. We directly build on top oftransformers,accelerate, andpeftlibraries by 🤗 Hugging Face. That way TRIL is able to support open-sourced pretrained models, distributed computing, as well as parameter efficient training. Note we currently support most decoder and encoder-decoder architectures availble intransformers.Supported Algorithms:Behavior Cloning (i.e. Supervised Fine Tuning)Proximal Policy Optimization (PPO) (https://arxiv.org/abs/1707.06347)Generative Adversarial Imitation Learning (GAIL) (https://arxiv.org/abs/1606.03476)PPO++ (https://arxiv.org/pdf/2306.11816)AggreVaTeD (https://arxiv.org/pdf/2306.11816)Locally Optimal Learning to Search (LOLS) (https://arxiv.org/pdf/2306.11816)Direct and Differentiable Locally Optimal Learning to Search (D2LOLS) (https://arxiv.org/pdf/2306.11816)Supported Tasks:IMDB Positive Sentiment (https://arxiv.org/abs/2210.01241)CommonGen: Common Sense Generation (https://arxiv.org/abs/1911.03705)TL;DR Summarization (https://arxiv.org/pdf/2203.02155.pdf)Planned Algorithms:Direct Preference Optimization (DPO) (https://arxiv.org/pdf/2305.18290.pdf)Statistical Rejection Sampling Optimization (RSO) (https://arxiv.org/pdf/2309.06657.pdf)Phasic Policy Gradient (PPG) (https://arxiv.org/abs/2009.04416)Pairwise Proximal Policy Optimization (P3O) (https://arxiv.org/pdf/2310.00212.pdf)Advantage-Induced Policy Alignment (APA) (https://arxiv.org/pdf/2306.02231.pdf)Advantage-Leftover Lunch RL (A-LoL) (https://arxiv.org/abs/2305.14718)Planned Tasks:Helpfulness and Harmfullness (https://arxiv.org/pdf/2204.05862.pdf)InstallationTo installtrildo:pip install trilFor the run scripts and the example scripts for usage please see the respository.To setup a development environment we usecondafor version control. To install TRIL, please follow these stepsconda create -n tril python=3.10
conda activate tril
pip install -e .Optionally, forcaption_metricssuch as CiDER-D and SPICE, please install these additional dependencies.# Spacy model install
python -m spacy download en_core_web_sm

# CoreNLP library install
cd src/tril/metrics/caption_metrics/spice && bash get_stanford_models.shExample ScriptsIn theexamplesdirectory, there are example scripts to run TRIL algorithms onIMDBpositive sentiment generation using pytorchFully Sharded Data Parallel (FSDP)andTL;DRsummarization usingdeepspeed. The name of each script is of the format,<task>_<alg>.yaml. Run each experiment like the following:./examples/<task>/<script>Within each script the command isaccelerate --config <accelerate config> [accelerate args] main.py task=<task config> alg=<alg config> [hydra CLI config specification]Please see theacceleratelaunch tutorialfor how to launch jobs withaccelerate. We provide examples of differentaccelerateconfigs in theaccelerate_cfgsdirectoy. For more details on Hydra CLI and config usage please see thistutorial.Usage ExampleHere is a minimal example of running PPO with TRIL:importhydrafromaccelerateimportAcceleratorfromtrilimporttril_runfromtril.loggingimportTrackerfromtril.algorithmsimportPPO@hydra.main(version_base=None,config_path="cfgs",config_name="config")# Hydra Decorator for Config@tril_run# TRIL decorator for hydra config processingdefrun_ppo(cfg):# Initialize accelerator for distributed computingaccelerator=Accelerator()# Grab experiment save directory from Hydrasave_path=hydra.core.hydra_config.HydraConfig.get().runtime.output_dir# Instantiate TRIL logger for WandB and CLI logging/savingtracker=Tracker(save_path,OmegaConf.to_container(cfg,resolve=True),cfg.project_name,cfg.experiment_name,cfg.entity_name,cfg.log_to_wandb,log_level=logging.INFO,is_main_process=accelerator.is_main_process,)# Instantiate Algorithmppo=PPO(cfg,accelerator,tracker)# Start learn to train LLMppo.learn()if__name__=='__main__':run_ppo()TRILalso provides anAlgorithmRegistryto instantiate algorithms. Please see ourmain.pyto see how our scripts instantiate the algorithms. The list of available algorithms can be seen by the configs incfgs/task.Current Task/Algorithm Support MatrixAlgorithmIMDBCommonGenTL;DRPPO✅✅✅PPO++✅✅✅AggreVaTeD✅✅✅LOLS✅✅✅D2LOLS✅✅✅BC✅✅✅GAIL✅Code StructureThe directory structure of the configs, run script, and TRIL components looks like this.├── cfgs                    <- Hydra configs
│   ├── alg                 <- Algorithm configs (e.g. PPO)
│   ├── task                <- Task configs (e.g. TL;DR summarization)
│   ├── logging             <- Logging configs (e.g. WandB)
│   │
│   └── config.yaml         <- Main config for training
│
├── accelerate_cfgs         <- Accelerate configs
│
├── main.py                 <- TRIL main function
│
├── tril                    <- TRIL src
│   ├── algorithms          <- Algorithm implementations
│   ├── buffers             <- Data Buffer (e.g. OnlineBuffer, PromptBuffer)
│   ├── metrics             <- Evaluation Metrics
│   ├── policies            <- Language Model Policies (e.g. Actor, ActorCritic)
│   ├── rewards             <- Reward Functions
│   ├── tasks               <- Supported Tasks
│   ├── utils               <- Helper functions for TRIL
│   │
│   ├── agent.py            <- Agent contains all torch.nn Modules (i.e. Policy and Reward)
│   ├── base_algorithm.py   <- Algorithm abstract class
│   ├── base_metric.py      <- Metric abstract class
│   ├── base_reward.py      <- Reward abstract class
│   ├── base_task.py        <- Task abstract class
│   └── logging.py          <- TRIL LoggerIn each directory's__init__.py, there is a registry to register all supportedalgorithms,metrics,rewards, andtasks. When extendingTRIL, please add the respective addition to one of these registries.LoggingTRIL support Weights and Biases logging. Please enter yourwandbdetails such asentity_nameandproject_nameintocfgs/logging/wandb.yaml. If you would not like to log towandb, please setlog_to_wandb=False.By default, we save training and evaluation information inoutputs/<experiment_name>/<datetime>You can defineexperiment_nameincfgs/config.yamlor through Hydra CLI,main.py experiment_name=<name>.Example WandB ReportsHere is an example WandB Report of how the logging would look like when running multiple different algorithmsCommonGen Report.TL;DR PPO Report.Citing TRILIf you use TRIL in your publication, please cite it by using the following BibTeX entry.@misc{TRIL,title={TRIL: Transformers Reinforcement and Imitation Learning Library},author={Jonathan D Chang and Kiante Brantley and Rajkumar Ramamurthy and Dipendra Misra and Wen Sun},howpublished={\url{https://github.com/Cornell-RL/tril}},year={2023}}Here is the citation of the accompanying paper for many of the supported algorithms.@misc{chang2023learning,title={Learning to Generate Better Than Your LLM},author={Jonathan D. Chang and Kiante Brantley and Rajkumar Ramamurthy and Dipendra Misra and Wen Sun},year={2023},eprint={2306.11816},archivePrefix={arXiv},primaryClass={cs.LG}}AcknowledgementsWe would like to acknowledgeRL4LMs,TRL, andTRLxfor being inspirations for this library.