Package: lightseq
LightSeq: A High Performance Library for Sequence Processing and Generation[2021/06/18]:tada: :tada: :tada:LightSeq supports fast training for models in the Transformer family now,
please check outherefor details.LightSeq is a high performance training and inference library for sequence processing and generation implemented
in CUDA.
It enables highly efficient computation of modern NLP models such asBERT,GPT,Transformer, etc.
It is therefore best useful forMachine Translation,Text Generation,Dialog,Language
Modelling,Sentiment Analysis, and other related tasks with sequence data.The library is built on top of CUDA official
library(cuBLAS,Thrust,CUB) and
custom kernel functions which are specially fused and optimized for Transformer model family. In
addition to model components, the inference library also provide easy-to deploy model management and serving backend based onTensorRT Inference
Server.
With LightSeq, one can easily develop modified Transformer architecture with little additional code.Features>>> TrainingThe following is a support matrix of LightSeqtraininglibrary compared withDeepSpeed.>>> InferenceThe following is a support matrix of LightSeqinferencelibrary compared withTurboTransformersandFasterTransformer.Performance>>> TrainingHere we present the experimental results on WMT14 English to German translation task based on Transformer-big models. We train Transformer models of different sizes on eight NVIDIA Tesla V100/NVIDIA Tesla A100 GPUs with data parallel and fp16 mixed precision.FairseqwithApexis choosed as our baseline.We compute speedup on different batch size using the WPS (real words per second) metric.More results is availablehere>>> InferenceHere we present the experimental results on neural machine translation based on Transformer-base models using beam search methods.
We choose Tensorflow andFasterTransformeras a comparison.
The implementation fromtensor2tensorwas used as the benchmark of Tensorflow.More results is availablehere.Quick StartComplete user guide is availablehere.InstallationYou can install LightSeq from PyPI:$pipinstalllightseqLightSeq installation from PyPI only supports Python 3.6 to 3.8 on Linux for now. Consider compiling from source if you have other environments:$PATH=/usr/local/hdf5/:$PATHENABLE_FP32=0ENABLE_DEBUG=0pipinstall-e$PROJECT_DIRDetailed building introduction is availablehere.Fast training from FairseqYou can experience lightning fast training by running following commands,
Firstly install these requirements.$pipinstalllightseqfairseqsacremosesThen you can train a translation task on wmt14 en2de dataset by running the following script$shexamples/training/fairseq/ls_fairseq_wmt14en2de.shTo compare lightseq with fairseq, delete the arguments withls_prefix to using the original fairseq implementationMore usage is availablehere.Fast inference from HuggingFace bartWe provide an end2end bart-base example to see how fast Lightseq is compared to HuggingFace. First you should install these requirements.$pipinstalltorchtensorflowtransformerslightseq
$cdexamples/inference/pythonthen you can check the performance by simply running following commands.hf_bart_export.pyis used to transform pytorch weights to LightSeq protobuffer.$pythonexport/huggingface/hf_bart_export.py
$pythontest/ls_bart.pyMore usage is availablehere.Fast deploy inference serverWe provide a docker image which contains tritonserver and lightseq's dynamic link library, and you can deploy a inference server by simply replace the model file with your own model file.$sudodockerpullhexisyztem/tritonserver_lightseq:22.01-1More usage is availablehere.Cite UsIf you use LightSeq in your research, please cite the following paper.@InProceedings{wang2021lightseq,
    title = "{L}ight{S}eq: A High Performance Inference Library for Transformers",
    author = "Wang, Xiaohui and Xiong, Ying and Wei, Yang and Wang, Mingxuan and Li, Lei",
    booktitle = "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers (NAACL-HLT)",
    month = jun,
    year = "2021",
    publisher = "Association for Computational Linguistics",
    pages = "113--120",
}

@article{wang2021lightseq2,
  title={LightSeq2: Accelerated Training for Transformer-based Models on GPUs},
  author={Wang, Xiaohui and Xiong, Ying and Qian, Xian and Wei, Yang and Li, Lei and Wang, Mingxuan},
  journal={arXiv preprint arXiv:2110.05722},
  year={2021}
}ContactAny questions or suggestions, please feel free to contact us atwangxiaohui.neo@bytedance.com,xiongying.taka@bytedance.com,qian.xian@bytedance.com,weiyang.god@bytedance.com,wangmingxuan.89@bytedance.com,lilei@cs.ucsb.eduHiringThe LightSeq team is hiring Interns/FTEs with backgrounds in deep learning system/natural language processing/computer vision/speech.
We are based in Beijing and Shanghai. If you are interested, please send your resume towangxiaohui.neo@bytedance.com.