Package: rl4co
Documentation|Getting Started|Usage|Contributing|Paper|Join UsRL4CO has been accepted as an oral presentation at theNeurIPS 2023 GLFrontiers Workshop! 🎉An extensive Reinforcement Learning (RL) for Combinatorial Optimization (CO) benchmark. Our goal is to provide a unified framework for RL-based CO algorithms, and to facilitate reproducible research in this field, decoupling the science from the engineering.RL4CO is built upon:TorchRL: official PyTorch framework for RL algorithms and vectorized environments on GPUsTensorDict: a library to easily handle heterogeneous data such as states, actions and rewardsPyTorch Lightning: a lightweight PyTorch wrapper for high-performance AI researchHydra: a framework for elegantly configuring complex applicationsWe provide several utilities and modularization. For autoregressive policies, we modularize reusable components such asenvironment embeddingsthat can easily be swapped tosolve new problemsGetting startedRL4CO is now available for installation onpip!pipinstallrl4coTo get started, we recommend checking out ourquickstart notebookor theminimalistic examplebelow.Install from sourceThis command installs the bleeding edgemainversion, useful for staying up-to-date with the latest developments - for instance, if a bug has been fixed since the last official release but a new release hasn’t been rolled out yet:pipinstall-Ugit+https://github.com/ai4co/rl4co.gitLocal install and developmentIf you want to develop RL4CO we recommend you to install it locally withpipin editable mode:gitclonehttps://github.com/ai4co/rl4co&&cdrl4co
pipinstall-e.We recommend using a virtual environment such ascondato installrl4colocally.UsageTrain model with default configuration (AM on TSP environment):pythonrun.pyChange experimentTrain model with chosen experiment configuration fromconfigs/experiment/(e.g. tsp/am, and environment with 42 cities)pythonrun.pyexperiment=routing/amenv.num_loc=42Disable loggingpythonrun.pyexperiment=routing/amlogger=none'~callbacks.learning_rate_monitor'Note that~is used to disable a callback that would need a logger.Create a sweep over hyperparameters (-m for multirun)pythonrun.py-mexperiment=routing/amtrain.optimizer.lr=1e-3,1e-4,1e-5Minimalistic ExampleHere is a minimalistic example training the Attention Model with greedy rollout baseline on TSP in less than 30 lines of code:fromrl4co.envsimportTSPEnvfromrl4co.modelsimportAttentionModelfromrl4co.utilsimportRL4COTrainer# Environment, Model, and Lightning Moduleenv=TSPEnv(num_loc=20)model=AttentionModel(env,baseline="rollout",train_data_size=100_000,test_data_size=10_000,optimizer_kwargs={'lr':1e-4})# Trainertrainer=RL4COTrainer(max_epochs=3)# Fit the modeltrainer.fit(model)# Test the modeltrainer.test(model)Other examples can be found on thedocumentation!TestingRun tests withpytestfrom the root directory:pytesttestsKnown BugsBugs installing PyTorch Geometric (PyG)InstallingPyGviaCondaseems to update Torch itself. We have found that this update introduces some bugs withtorchrl. At this moment, we recommend installingPyGwithPip:pipinstalltorch_geometricContributingHave a suggestion, request, or found a bug? Feel free toopen an issueorsubmit a pull request.
If you would like to contribute, please check out our contribution guidelineshere. We welcome and look forward to all contributions to RL4CO!We are also onSlackif you have any questions or would like to discuss RL4CO with us. We are open to collaborations and would love to hear from you 🚀ContributorsCitationIf you find RL4CO valuable for your research or applied projects:@inproceedings{berto2023rl4co,title={{RL}4{CO}: a Unified Reinforcement Learning for Combinatorial Optimization Library},author={Federico Berto and Chuanbo Hua and Junyoung Park and Minsu Kim and Hyeonah Kim and Jiwoo Son and Haeyeon Kim and Joungho Kim and Jinkyoo Park},booktitle={NeurIPS 2023 Workshop: New Frontiers in Graph Learning},year={2023},url={https://openreview.net/forum?id=YXSJxi8dOV},note={\url{https://github.com/ai4co/rl4co}}}Join usWe invite you to join our AI4CO community, an open research group in Artificial Intelligence (AI) for Combinatorial Optimization (CO)!