Package: fairseq
Fairseq(-py) is a sequence modeling toolkit that allows researchers and
developers to train custom models for translation, summarization, language
modeling and other text generation tasks.We provide reference implementations of various sequence modeling papers:List of implemented papersConvolutional Neural Networks (CNN)Language Modeling with Gated Convolutional Networks (Dauphin et al., 2017)Convolutional Sequence to Sequence Learning (Gehring et al., 2017)Classical Structured Prediction Losses for Sequence to Sequence Learning (Edunov et al., 2018)Hierarchical Neural Story Generation (Fan et al., 2018)wav2vec: Unsupervised Pre-training for Speech Recognition (Schneider et al., 2019)LightConv and DynamicConv modelsPay Less Attention with Lightweight and Dynamic Convolutions (Wu et al., 2019)Long Short-Term Memory (LSTM) networksEffective Approaches to Attention-based Neural Machine Translation (Luong et al., 2015)Transformer (self-attention) networksAttention Is All You Need (Vaswani et al., 2017)Scaling Neural Machine Translation (Ott et al., 2018)Understanding Back-Translation at Scale (Edunov et al., 2018)Adaptive Input Representations for Neural Language Modeling (Baevski and Auli, 2018)Lexically constrained decoding with dynamic beam allocation (Post & Vilar, 2018)Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context (Dai et al., 2019)Adaptive Attention Span in Transformers (Sukhbaatar et al., 2019)Mixture Models for Diverse Machine Translation: Tricks of the Trade (Shen et al., 2019)RoBERTa: A Robustly Optimized BERT Pretraining Approach (Liu et al., 2019)Facebook FAIR's WMT19 News Translation Task Submission (Ng et al., 2019)Jointly Learning to Align and Translate with Transformer Models (Garg et al., 2019)Multilingual Denoising Pre-training for Neural Machine Translation (Liu et at., 2020)Neural Machine Translation with Byte-Level Subwords (Wang et al., 2020)Unsupervised Quality Estimation for Neural Machine Translation (Fomicheva et al., 2020)wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (Baevski et al., 2020)Generating Medical Reports from Patient-Doctor Conversations Using Sequence-to-Sequence Models (Enarvi et al., 2020)Linformer: Self-Attention with Linear Complexity (Wang et al., 2020)Cross-lingual Retrieval for Iterative Self-Supervised Training (Tran et al., 2020)Deep Transformers with Latent Depth (Li et al., 2020)Unsupervised Cross-lingual Representation Learning for Speech Recognition (Conneau et al., 2020)Self-training and Pre-training are Complementary for Speech Recognition (Xu et al., 2020)Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training (Hsu, et al., 2021)Unsupervised Speech Recognition (Baevski, et al., 2021)Simple and Effective Zero-shot Cross-lingual Phoneme Recognition (Xu et al., 2021)VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (Xu et. al., 2021)VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding (Xu et. al., 2021)NormFormer: Improved Transformer Pretraining with Extra Normalization (Shleifer et. al, 2021)Non-autoregressive TransformersNon-Autoregressive Neural Machine Translation (Gu et al., 2017)Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement (Lee et al. 2018)Insertion Transformer: Flexible Sequence Generation via Insertion Operations (Stern et al. 2019)Mask-Predict: Parallel Decoding of Conditional Masked Language Models (Ghazvininejad et al., 2019)Levenshtein Transformer (Gu et al., 2019)FinetuningBetter Fine-Tuning by Reducing Representational Collapse (Aghajanyan et al. 2020)What's New:June 2022Released code for wav2vec-U 2.0 from Towards End-to-end Unsupervised Speech Recognition (Liu, et al., 2022)May 2022Integration with xFormersDecember 2021Released Direct speech-to-speech translation codeOctober 2021Released VideoCLIP and VLM modelsOctober 2021Released multilingual finetuned XLSR-53 modelSeptember 2021masterbranch renamed tomain.July 2021Released DrNMT codeJuly 2021Released Robust wav2vec 2.0 modelJune 2021Released XLMR-XL and XLMR-XXL modelsMay 2021Released Unsupervised Speech Recognition codeMarch 2021Added full parameter and optimizer state sharding + CPU offloadingFebruary 2021Added LASER training codeDecember 2020:Added Adaptive Attention Span codeDecember 2020:GottBERT model and code releasedNovember 2020: Adopted theHydraconfiguration frameworksee documentation explaining how to use it for new and existing projectsNovember 2020:fairseq 0.10.0 releasedOctober 2020:Added R3F/R4F (Better Fine-Tuning) codeOctober 2020:Deep Transformer with Latent Depth code releasedOctober 2020:Added CRISS models and codePrevious updatesSeptember 2020:Added Linformer codeSeptember 2020:Added pointer-generator networksAugust 2020:Added lexically constrained decodingAugust 2020:wav2vec2 models and code releasedJuly 2020:Unsupervised Quality Estimation code releasedMay 2020:Follow fairseq on TwitterApril 2020:Monotonic Multihead Attention code releasedApril 2020:Quant-Noise code releasedApril 2020:Initial model parallel support and 11B parameters unidirectional LM releasedMarch 2020:Byte-level BPE code releasedFebruary 2020:mBART model and code releasedFebruary 2020:Added tutorial for back-translationDecember 2019:fairseq 0.9.0 releasedNovember 2019:VizSeq released (a visual analysis toolkit for evaluating fairseq models)November 2019:CamemBERT model and code releasedNovember 2019:BART model and code releasedNovember 2019:XLM-R models and code releasedSeptember 2019:Nonautoregressive translation code releasedAugust 2019:WMT'19 models releasedJuly 2019: fairseq relicensed under MIT licenseJuly 2019:RoBERTa models and code releasedJune 2019:wav2vec models and code releasedFeatures:multi-GPU training on one machine or across multiple machines (data and model parallel)fast generation on both CPU and GPU with multiple search algorithms implemented:beam searchDiverse Beam Search (Vijayakumar et al., 2016)sampling (unconstrained, top-k and top-p/nucleus)lexically constrained decoding(Post & Vilar, 2018)gradient accumulationenables training with large mini-batches even on a single GPUmixed precision training(trains faster with less GPU memory onNVIDIA tensor cores)extensible: easily register new models, criterions, tasks, optimizers and learning rate schedulersflexible configurationbased onHydraallowing a combination of code, command-line and file based configurationfull parameter and optimizer state shardingoffloading parameters to CPUWe also providepre-trained models for translation and language modelingwith a convenienttorch.hubinterface:en2de=torch.hub.load('pytorch/fairseq','transformer.wmt19.en-de.single_model')en2de.translate('Hello world',beam=5)# 'Hallo Welt'See the PyTorch Hub tutorials fortranslationandRoBERTafor more examples.Requirements and InstallationPyTorchversion >= 1.5.0Python version >= 3.6For training new models, you'll also need an NVIDIA GPU andNCCLTo install fairseqand develop locally:gitclonehttps://github.com/pytorch/fairseqcdfairseq
pipinstall--editable./# on MacOS:# CFLAGS="-stdlib=libc++" pip install --editable ./# to install the latest stable release (0.10.x)# pip install fairseqFor faster traininginstall NVIDIA'sapexlibrary:gitclonehttps://github.com/NVIDIA/apexcdapex
pipinstall-v--no-cache-dir--global-option="--cpp_ext"--global-option="--cuda_ext"\--global-option="--deprecated_fused_adam"--global-option="--xentropy"\--global-option="--fast_multihead_attn"./For large datasetsinstallPyArrow:pip install pyarrowIf you use Docker make sure to increase the shared memory size either with--ipc=hostor--shm-sizeas command line options tonvidia-docker run.Getting StartedThefull documentationcontains instructions
for getting started, training new models and extending fairseq with new model
types and tasks.Pre-trained models and examplesWe provide pre-trained models and pre-processed, binarized test sets for several tasks listed below,
as well as example training and evaluation commands.Translation: convolutional and transformer models are availableLanguage Modeling: convolutional and transformer models are availableWe also have more detailed READMEs to reproduce results from specific papers:XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (Babu et al., 2021)Cross-lingual Retrieval for Iterative Self-Supervised Training (Tran et al., 2020)wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (Baevski et al., 2020)Unsupervised Quality Estimation for Neural Machine Translation (Fomicheva et al., 2020)Training with Quantization Noise for Extreme Model Compression ({Fan*, Stock*} et al., 2020)Neural Machine Translation with Byte-Level Subwords (Wang et al., 2020)Multilingual Denoising Pre-training for Neural Machine Translation (Liu et at., 2020)Reducing Transformer Depth on Demand with Structured Dropout (Fan et al., 2019)Jointly Learning to Align and Translate with Transformer Models (Garg et al., 2019)Levenshtein Transformer (Gu et al., 2019)Facebook FAIR's WMT19 News Translation Task Submission (Ng et al., 2019)RoBERTa: A Robustly Optimized BERT Pretraining Approach (Liu et al., 2019)wav2vec: Unsupervised Pre-training for Speech Recognition (Schneider et al., 2019)Mixture Models for Diverse Machine Translation: Tricks of the Trade (Shen et al., 2019)Pay Less Attention with Lightweight and Dynamic Convolutions (Wu et al., 2019)Understanding Back-Translation at Scale (Edunov et al., 2018)Classical Structured Prediction Losses for Sequence to Sequence Learning (Edunov et al., 2018)Hierarchical Neural Story Generation (Fan et al., 2018)Scaling Neural Machine Translation (Ott et al., 2018)Convolutional Sequence to Sequence Learning (Gehring et al., 2017)Language Modeling with Gated Convolutional Networks (Dauphin et al., 2017)Join the fairseq communityTwitter:https://twitter.com/fairseqFacebook page:https://www.facebook.com/groups/fairseq.usersGoogle group:https://groups.google.com/forum/#!forum/fairseq-usersLicensefairseq(-py) is MIT-licensed.
The license applies to the pre-trained models as well.CitationPlease cite as:@inproceedings{ott2019fairseq,title={fairseq: A Fast, Extensible Toolkit for Sequence Modeling},author={Myle Ott and Sergey Edunov and Alexei Baevski and Angela Fan and Sam Gross and Nathan Ng and David Grangier and Michael Auli},booktitle={Proceedings of NAACL-HLT 2019: Demonstrations},year={2019},}