NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment
Authors: name: Gerald Shen; name: Zhilin Wang; name: Olivier Delalleau; name: Jiaqi Zeng; name: Yi Dong; name: Daniel Egert; name: Shengyang Sun; name: Jimmy Zhang; name: Sahil Jain; name: Ali Taghibakhshi; name: Markel Sanz Ausin; name: Ashwath Aithal; name: Oleksii Kuchaiev
Published: 2024-05-02 17:13:40

Abstract: Aligning Large Language Models (LLMs) with human values and preferences is
essential for making them helpful and safe. However, building efficient tools
to perform alignment can be challenging, especially for the largest and most
competent LLMs which often contain tens or hundreds of billions of parameters.
We create NeMo-Aligner, a toolkit for model alignment that can efficiently
scale to using hundreds of GPUs for training. NeMo-Aligner comes with highly
optimized and scalable implementations for major paradigms of model alignment
such as: Reinforcement Learning from Human Feedback (RLHF), Direct Preference
Optimization (DPO), SteerLM, and Self-Play Fine-Tuning (SPIN). Additionally,
our toolkit supports running most of the alignment techniques in a Parameter
Efficient Fine-Tuning (PEFT) setting. NeMo-Aligner is designed for
extensibility, allowing support for other alignment techniques with minimal
effort. It is open-sourced with Apache 2.0 License and we invite community
contributions at https://github.com/NVIDIA/NeMo-Aligner