Package: nm-transformers
English|简体中文|繁體中文|한국어|Español|日本語|हिन्दी|РусскийState-of-the-art Machine Learning for JAX, PyTorch and TensorFlow🤗 Transformers provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio.These models can be applied on:📝 Text, for tasks like text classification, information extraction, question answering, summarization, translation, text generation, in over 100 languages.🖼️ Images, for tasks like image classification, object detection, and segmentation.🗣️ Audio, for tasks like speech recognition and audio classification.Transformer models can also perform tasks onseveral modalities combined, such as table question answering, optical character recognition, information extraction from scanned documents, video classification, and visual question answering.🤗 Transformers provides APIs to quickly download and use those pretrained models on a given text, fine-tune them on your own datasets and then share them with the community on ourmodel hub. At the same time, each python module defining an architecture is fully standalone and can be modified to enable quick research experiments.🤗 Transformers is backed by the three most popular deep learning libraries —Jax,PyTorchandTensorFlow— with a seamless integration between them. It's straightforward to train your models with one before loading them for inference with the other.Online demosYou can test most of our models directly on their pages from themodel hub. We also offerprivate model hosting, versioning, & an inference APIfor public and private models.Here are a few examples:In Natural Language Processing:Masked word completion with BERTName Entity Recognition with ElectraText generation with GPT-2Natural Language Inference with RoBERTaSummarization with BARTQuestion answering with DistilBERTTranslation with T5In Computer Vision:Image classification with ViTObject Detection with DETRSemantic Segmentation with SegFormerPanoptic Segmentation with MaskFormerDepth Estimation with DPTVideo Classification with VideoMAEUniversal Segmentation with OneFormerIn Audio:Automatic Speech Recognition with Wav2Vec2Keyword Spotting with Wav2Vec2Audio Classification with Audio Spectrogram TransformerIn Multimodal tasks:Table Question Answering with TAPASVisual Question Answering with ViLTZero-shot Image Classification with CLIPDocument Question Answering with LayoutLMZero-shot Video Classification with X-CLIP100 projects using TransformersTransformers is more than a toolkit to use pretrained models: it's a community of projects built around it and the
Hugging Face Hub. We want Transformers to enable developers, researchers, students, professors, engineers, and anyone
else to build their dream projects.In order to celebrate the 100,000 stars of transformers, we have decided to put the spotlight on the
community, and we have created theawesome-transformerspage which lists 100
incredible projects built in the vicinity of transformers.If you own or use a project that you believe should be part of the list, please open a PR to add it!If you are looking for custom support from the Hugging Face teamQuick tourTo immediately use a model on a given input (text, image, audio, ...), we provide thepipelineAPI. Pipelines group together a pretrained model with the preprocessing that was used during that model's training. Here is how to quickly use a pipeline to classify positive versus negative texts:>>>fromtransformersimportpipeline# Allocate a pipeline for sentiment-analysis>>>classifier=pipeline('sentiment-analysis')>>>classifier('We are very happy to introduce pipeline to the transformers repository.')[{'label':'POSITIVE','score':0.9996980428695679}]The second line of code downloads and caches the pretrained model used by the pipeline, while the third evaluates it on the given text. Here the answer is "positive" with a confidence of 99.97%.Many tasks have a pre-trainedpipelineready to go, in NLP but also in computer vision and speech. For example, we can easily extract detected objects in an image:>>>importrequests>>>fromPILimportImage>>>fromtransformersimportpipeline# Download an image with cute cats>>>url="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco_sample.png">>>image_data=requests.get(url,stream=True).raw>>>image=Image.open(image_data)# Allocate a pipeline for object detection>>>object_detector=pipeline('object-detection')>>>object_detector(image)[{'score':0.9982201457023621,'label':'remote','box':{'xmin':40,'ymin':70,'xmax':175,'ymax':117}},{'score':0.9960021376609802,'label':'remote','box':{'xmin':333,'ymin':72,'xmax':368,'ymax':187}},{'score':0.9954745173454285,'label':'couch','box':{'xmin':0,'ymin':1,'xmax':639,'ymax':473}},{'score':0.9988006353378296,'label':'cat','box':{'xmin':13,'ymin':52,'xmax':314,'ymax':470}},{'score':0.9986783862113953,'label':'cat','box':{'xmin':345,'ymin':23,'xmax':640,'ymax':368}}]Here we get a list of objects detected in the image, with a box surrounding the object and a confidence score. Here is the original image on the left, with the predictions displayed on the right:You can learn more about the tasks supported by thepipelineAPI inthis tutorial.In addition topipeline, to download and use any of the pretrained models on your given task, all it takes is three lines of code. Here is the PyTorch version:>>>fromtransformersimportAutoTokenizer,AutoModel>>>tokenizer=AutoTokenizer.from_pretrained("bert-base-uncased")>>>model=AutoModel.from_pretrained("bert-base-uncased")>>>inputs=tokenizer("Hello world!",return_tensors="pt")>>>outputs=model(**inputs)And here is the equivalent code for TensorFlow:>>>fromtransformersimportAutoTokenizer,TFAutoModel>>>tokenizer=AutoTokenizer.from_pretrained("bert-base-uncased")>>>model=TFAutoModel.from_pretrained("bert-base-uncased")>>>inputs=tokenizer("Hello world!",return_tensors="tf")>>>outputs=model(**inputs)The tokenizer is responsible for all the preprocessing the pretrained model expects, and can be called directly on a single string (as in the above examples) or a list. It will output a dictionary that you can use in downstream code or simply directly pass to your model using the ** argument unpacking operator.The model itself is a regularPytorchnn.Moduleor aTensorFlowtf.keras.Model(depending on your backend) which you can use as usual.This tutorialexplains how to integrate such a model into a classic PyTorch or TensorFlow training loop, or how to use ourTrainerAPI to quickly fine-tune on a new dataset.Why should I use transformers?Easy-to-use state-of-the-art models:High performance on natural language understanding & generation, computer vision, and audio tasks.Low barrier to entry for educators and practitioners.Few user-facing abstractions with just three classes to learn.A unified API for using all our pretrained models.Lower compute costs, smaller carbon footprint:Researchers can share trained models instead of always retraining.Practitioners can reduce compute time and production costs.Dozens of architectures with over 60,000 pretrained models across all modalities.Choose the right framework for every part of a model's lifetime:Train state-of-the-art models in 3 lines of code.Move a single model between TF2.0/PyTorch/JAX frameworks at will.Seamlessly pick the right framework for training, evaluation and production.Easily customize a model or an example to your needs:We provide examples for each architecture to reproduce the results published by its original authors.Model internals are exposed as consistently as possible.Model files can be used independently of the library for quick experiments.Why shouldn't I use transformers?This library is not a modular toolbox of building blocks for neural nets. The code in the model files is not refactored with additional abstractions on purpose, so that researchers can quickly iterate on each of the models without diving into additional abstractions/files.The training API is not intended to work on any model but is optimized to work with the models provided by the library. For generic machine learning loops, you should use another library (possibly,Accelerate).While we strive to present as many use cases as possible, the scripts in ourexamples folderare just that: examples. It is expected that they won't work out-of-the box on your specific problem and that you will be required to change a few lines of code to adapt them to your needs.InstallationWith pipThis repository is tested on Python 3.8+, Flax 0.4.1+, PyTorch 1.10+ and TensorFlow 2.6+.You should install 🤗 Transformers in avirtual environment. If you're unfamiliar with Python virtual environments, check out theuser guide.First, create a virtual environment with the version of Python you're going to use and activate it.Then, you will need to install at least one of Flax, PyTorch or TensorFlow.
Please refer toTensorFlow installation page,PyTorch installation pageand/orFlaxandJaxinstallation pages regarding the specific installation command for your platform.When one of those backends has been installed, 🤗 Transformers can be installed using pip as follows:pipinstalltransformersIf you'd like to play with the examples or need the bleeding edge of the code and can't wait for a new release, you mustinstall the library from source.With condaSince Transformers version v4.0.0, we now have a conda channel:huggingface.🤗 Transformers can be installed using conda as follows:condainstall-chuggingfacetransformersFollow the installation pages of Flax, PyTorch or TensorFlow to see how to install them with conda.NOTE:On Windows, you may be prompted to activate Developer Mode in order to benefit from caching. If this is not an option for you, please let us know inthis issue.Model architecturesAll the model checkpointsprovided by 🤗 Transformers are seamlessly integrated from the huggingface.comodel hubwhere they are uploaded directly byusersandorganizations.Current number of checkpoints:🤗 Transformers currently provides the following architectures (seeherefor a high-level summary of each them):ALBERT(from Google Research and the Toyota Technological Institute at Chicago) released with the paperALBERT: A Lite BERT for Self-supervised Learning of Language Representations, by Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut.ALIGN(from Google Research) released with the paperScaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisionby Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung, Zhen Li, Tom Duerig.AltCLIP(from BAAI) released with the paperAltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilitiesby Chen, Zhongzhi and Liu, Guang and Zhang, Bo-Wen and Ye, Fulong and Yang, Qinghong and Wu, Ledell.Audio Spectrogram Transformer(from MIT) released with the paperAST: Audio Spectrogram Transformerby Yuan Gong, Yu-An Chung, James Glass.Autoformer(from Tsinghua University) released with the paperAutoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingby Haixu Wu, Jiehui Xu, Jianmin Wang, Mingsheng Long.Bark(from Suno) released in the repositorysuno-ai/barkby Suno AI team.BART(from Facebook) released with the paperBART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensionby Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov and Luke Zettlemoyer.BARThez(from École polytechnique) released with the paperBARThez: a Skilled Pretrained French Sequence-to-Sequence Modelby Moussa Kamal Eddine, Antoine J.-P. Tixier, Michalis Vazirgiannis.BARTpho(from VinAI Research) released with the paperBARTpho: Pre-trained Sequence-to-Sequence Models for Vietnameseby Nguyen Luong Tran, Duong Minh Le and Dat Quoc Nguyen.BEiT(from Microsoft) released with the paperBEiT: BERT Pre-Training of Image Transformersby Hangbo Bao, Li Dong, Furu Wei.BERT(from Google) released with the paperBERT: Pre-training of Deep Bidirectional Transformers for Language Understandingby Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova.BERT For Sequence Generation(from Google) released with the paperLeveraging Pre-trained Checkpoints for Sequence Generation Tasksby Sascha Rothe, Shashi Narayan, Aliaksei Severyn.BERTweet(from VinAI Research) released with the paperBERTweet: A pre-trained language model for English Tweetsby Dat Quoc Nguyen, Thanh Vu and Anh Tuan Nguyen.BigBird-Pegasus(from Google Research) released with the paperBig Bird: Transformers for Longer Sequencesby Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, Amr Ahmed.BigBird-RoBERTa(from Google Research) released with the paperBig Bird: Transformers for Longer Sequencesby Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, Amr Ahmed.BioGpt(from Microsoft Research AI4Science) released with the paperBioGPT: generative pre-trained transformer for biomedical text generation and miningby Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon and Tie-Yan Liu.BiT(from Google AI) released with the paperBig Transfer (BiT): General Visual Representation Learningby Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, Neil Houlsby.Blenderbot(from Facebook) released with the paperRecipes for building an open-domain chatbotby Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, Jason Weston.BlenderbotSmall(from Facebook) released with the paperRecipes for building an open-domain chatbotby Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, Jason Weston.BLIP(from Salesforce) released with the paperBLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationby Junnan Li, Dongxu Li, Caiming Xiong, Steven Hoi.BLIP-2(from Salesforce) released with the paperBLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelsby Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi.BLOOM(from BigScience workshop) released by theBigScience Workshop.BORT(from Alexa) released with the paperOptimal Subarchitecture Extraction For BERTby Adrian de Wynter and Daniel J. Perry.BridgeTower(from Harbin Institute of Technology/Microsoft Research Asia/Intel Labs) released with the paperBridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningby Xiao Xu, Chenfei Wu, Shachar Rosenman, Vasudev Lal, Wanxiang Che, Nan Duan.BROS(from NAVER CLOVA) released with the paperBROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documentsby Teakgyu Hong, Donghyun Kim, Mingi Ji, Wonseok Hwang, Daehyun Nam, Sungrae Park.ByT5(from Google Research) released with the paperByT5: Towards a token-free future with pre-trained byte-to-byte modelsby Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel.CamemBERT(from Inria/Facebook/Sorbonne) released with the paperCamemBERT: a Tasty French Language Modelby Louis Martin*, Benjamin Muller*, Pedro Javier Ortiz Suárez*, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah and Benoît Sagot.CANINE(from Google Research) released with the paperCANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationby Jonathan H. Clark, Dan Garrette, Iulia Turc, John Wieting.Chinese-CLIP(from OFA-Sys) released with the paperChinese CLIP: Contrastive Vision-Language Pretraining in Chineseby An Yang, Junshu Pan, Junyang Lin, Rui Men, Yichang Zhang, Jingren Zhou, Chang Zhou.CLAP(from LAION-AI) released with the paperLarge-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationby Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, Shlomo Dubnov.CLIP(from OpenAI) released with the paperLearning Transferable Visual Models From Natural Language Supervisionby Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever.CLIPSeg(from University of Göttingen) released with the paperImage Segmentation Using Text and Image Promptsby Timo Lüddecke and Alexander Ecker.CodeGen(from Salesforce) released with the paperA Conversational Paradigm for Program Synthesisby Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, Caiming Xiong.CodeLlama(from MetaAI) released with the paperCode Llama: Open Foundation Models for Codeby Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, Gabriel Synnaeve.Conditional DETR(from Microsoft Research Asia) released with the paperConditional DETR for Fast Training Convergenceby Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, Jingdong Wang.ConvBERT(from YituTech) released with the paperConvBERT: Improving BERT with Span-based Dynamic Convolutionby Zihang Jiang, Weihao Yu, Daquan Zhou, Yunpeng Chen, Jiashi Feng, Shuicheng Yan.ConvNeXT(from Facebook AI) released with the paperA ConvNet for the 2020sby Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, Saining Xie.ConvNeXTV2(from Facebook AI) released with the paperConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencodersby Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, Saining Xie.CPM(from Tsinghua University) released with the paperCPM: A Large-scale Generative Chinese Pre-trained Language Modelby Zhengyan Zhang, Xu Han, Hao Zhou, Pei Ke, Yuxian Gu, Deming Ye, Yujia Qin, Yusheng Su, Haozhe Ji, Jian Guan, Fanchao Qi, Xiaozhi Wang, Yanan Zheng, Guoyang Zeng, Huanqi Cao, Shengqi Chen, Daixuan Li, Zhenbo Sun, Zhiyuan Liu, Minlie Huang, Wentao Han, Jie Tang, Juanzi Li, Xiaoyan Zhu, Maosong Sun.CPM-Ant(from OpenBMB) released by theOpenBMB.CTRL(from Salesforce) released with the paperCTRL: A Conditional Transformer Language Model for Controllable Generationby Nitish Shirish Keskar*, Bryan McCann*, Lav R. Varshney, Caiming Xiong and Richard Socher.CvT(from Microsoft) released with the paperCvT: Introducing Convolutions to Vision Transformersby Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, Lei Zhang.Data2Vec(from Facebook) released with the paperData2Vec:  A General Framework for Self-supervised Learning in Speech, Vision and Languageby Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, Michael Auli.DeBERTa(from Microsoft) released with the paperDeBERTa: Decoding-enhanced BERT with Disentangled Attentionby Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen.DeBERTa-v2(from Microsoft) released with the paperDeBERTa: Decoding-enhanced BERT with Disentangled Attentionby Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen.Decision Transformer(from Berkeley/Facebook/Google) released with the paperDecision Transformer: Reinforcement Learning via Sequence Modelingby Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, Igor Mordatch.Deformable DETR(from SenseTime Research) released with the paperDeformable DETR: Deformable Transformers for End-to-End Object Detectionby Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, Jifeng Dai.DeiT(from Facebook) released with the paperTraining data-efficient image transformers & distillation through attentionby Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, Hervé Jégou.DePlot(from Google AI) released with the paperDePlot: One-shot visual language reasoning by plot-to-table translationby Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, Yasemin Altun.DETA(from The University of Texas at Austin) released with the paperNMS Strikes Backby Jeffrey Ouyang-Zhang, Jang Hyun Cho, Xingyi Zhou, Philipp Krähenbühl.DETR(from Facebook) released with the paperEnd-to-End Object Detection with Transformersby Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko.DialoGPT(from Microsoft Research) released with the paperDialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationby Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, Bill Dolan.DiNAT(from SHI Labs) released with the paperDilated Neighborhood Attention Transformerby Ali Hassani and Humphrey Shi.DINOv2(from Meta AI) released with the paperDINOv2: Learning Robust Visual Features without Supervisionby Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, Piotr Bojanowski.DistilBERT(from HuggingFace), released together with the paperDistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighterby Victor Sanh, Lysandre Debut and Thomas Wolf. The same method has been applied to compress GPT2 intoDistilGPT2, RoBERTa intoDistilRoBERTa, Multilingual BERT intoDistilmBERTand a German version of DistilBERT.DiT(from Microsoft Research) released with the paperDiT: Self-supervised Pre-training for Document Image Transformerby Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, Furu Wei.Donut(from NAVER), released together with the paperOCR-free Document Understanding Transformerby Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeongyeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, Seunghyun Park.DPR(from Facebook) released with the paperDense Passage Retrieval for Open-Domain Question Answeringby Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.DPT(from Intel Labs) released with the paperVision Transformers for Dense Predictionby René Ranftl, Alexey Bochkovskiy, Vladlen Koltun.EfficientFormer(from Snap Research) released with the paperEfficientFormer: Vision Transformers at MobileNetSpeedby Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, Jian Ren.EfficientNet(from Google Brain) released with the paperEfficientNet: Rethinking Model Scaling for Convolutional Neural Networksby Mingxing Tan, Quoc V. Le.ELECTRA(from Google Research/Stanford University) released with the paperELECTRA: Pre-training text encoders as discriminators rather than generatorsby Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning.EnCodec(from Meta AI) released with the paperHigh Fidelity Neural Audio Compressionby Alexandre Défossez, Jade Copet, Gabriel Synnaeve, Yossi Adi.EncoderDecoder(from Google Research) released with the paperLeveraging Pre-trained Checkpoints for Sequence Generation Tasksby Sascha Rothe, Shashi Narayan, Aliaksei Severyn.ERNIE(from Baidu) released with the paperERNIE: Enhanced Representation through Knowledge Integrationby Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, Hua Wu.ErnieM(from Baidu) released with the paperERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corporaby Xuan Ouyang, Shuohuan Wang, Chao Pang, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang.ESM(from Meta AI) are transformer protein language models.ESM-1bwas released with the paperBiological structure and function emerge from scaling unsupervised learning to 250 million protein sequencesby Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus.ESM-1vwas released with the paperLanguage models enable zero-shot prediction of the effects of mutations on protein functionby Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu and Alexander Rives.ESM-2 and ESMFoldwere released with the paperLanguage models of protein sequences at the scale of evolution enable accurate structure predictionby Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, Alexander Rives.Falcon(from Technology Innovation Institute) by Almazrouei, Ebtesam and Alobeidli, Hamza and Alshamsi, Abdulaziz and Cappelli, Alessandro and Cojocaru, Ruxandra and Debbah, Merouane and Goffinet, Etienne and Heslow, Daniel and Launay, Julien and Malartic, Quentin and Noune, Badreddine and Pannier, Baptiste and Penedo, Guilherme.FLAN-T5(from Google AI) released in the repositorygoogle-research/t5xby Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason WeiFLAN-UL2(from Google AI) released in the repositorygoogle-research/t5xby Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason WeiFlauBERT(from CNRS) released with the paperFlauBERT: Unsupervised Language Model Pre-training for Frenchby Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoît Crabbé, Laurent Besacier, Didier Schwab.FLAVA(from Facebook AI) released with the paperFLAVA: A Foundational Language And Vision Alignment Modelby Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela.FNet(from Google Research) released with the paperFNet: Mixing Tokens with Fourier Transformsby James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon.FocalNet(from Microsoft Research) released with the paperFocal Modulation Networksby Jianwei Yang, Chunyuan Li, Xiyang Dai, Lu Yuan, Jianfeng Gao.Funnel Transformer(from CMU/Google Brain) released with the paperFunnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingby Zihang Dai, Guokun Lai, Yiming Yang, Quoc V. Le.GIT(from Microsoft Research) released with the paperGIT: A Generative Image-to-text Transformer for Vision and Languageby Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, Lijuan Wang.GLPN(from KAIST) released with the paperGlobal-Local Path Networks for Monocular Depth Estimation with Vertical CutDepthby Doyeon Kim, Woonghyun Ga, Pyungwhan Ahn, Donggyu Joo, Sehwan Chun, Junmo Kim.GPT(from OpenAI) released with the paperImproving Language Understanding by Generative Pre-Trainingby Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever.GPT Neo(from EleutherAI) released in the repositoryEleutherAI/gpt-neoby Sid Black, Stella Biderman, Leo Gao, Phil Wang and Connor Leahy.GPT NeoX(from EleutherAI) released with the paperGPT-NeoX-20B: An Open-Source Autoregressive Language Modelby Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, Samuel WeinbachGPT NeoX Japanese(from ABEJA) released by Shinya Otani, Takayoshi Makabe, Anuj Arora, and Kyo Hattori.GPT-2(from OpenAI) released with the paperLanguage Models are Unsupervised Multitask Learnersby Alec Radford*, Jeffrey Wu*, Rewon Child, David Luan, Dario Amodei** and Ilya Sutskever**.GPT-J(from EleutherAI) released in the repositorykingoflolz/mesh-transformer-jaxby Ben Wang and Aran Komatsuzaki.GPT-Sw3(from AI-Sweden) released with the paperLessons Learned from GPT-SW3: Building the First Large-Scale Generative Language Model for Swedishby Ariel Ekgren, Amaru Cuba Gyllensten, Evangelia Gogoulou, Alice Heiman, Severine Verlinden, Joey Öhman, Fredrik Carlsson, Magnus Sahlgren.GPTBigCode(from BigCode) released with the paperSantaCoder: don't reach for the stars!by Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, Logesh Kumar Umapathi, Carolyn Jane Anderson, Yangtian Zi, Joel Lamy Poirier, Hailey Schoelkopf, Sergey Troshin, Dmitry Abulkhanov, Manuel Romero, Michael Lappert, Francesco De Toni, Bernardo García del Río, Qian Liu, Shamik Bose, Urvashi Bhattacharyya, Terry Yue Zhuo, Ian Yu, Paulo Villegas, Marco Zocca, Sourab Mangrulkar, David Lansky, Huu Nguyen, Danish Contractor, Luis Villa, Jia Li, Dzmitry Bahdanau, Yacine Jernite, Sean Hughes, Daniel Fried, Arjun Guha, Harm de Vries, Leandro von Werra.GPTSAN-japanesereleased in the repositorytanreinama/GPTSANby Toshiyuki Sakamoto(tanreinama).Graphormer(from Microsoft) released with the paperDo Transformers Really Perform Bad for Graph Representation?by Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, Tie-Yan Liu.GroupViT(from UCSD, NVIDIA) released with the paperGroupViT: Semantic Segmentation Emerges from Text Supervisionby Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, Xiaolong Wang.HerBERT(from Allegro.pl, AGH University of Science and Technology) released with the paperKLEJ: Comprehensive Benchmark for Polish Language Understandingby Piotr Rybak, Robert Mroczkowski, Janusz Tracz, Ireneusz Gawlik.Hubert(from Facebook) released with the paperHuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitsby Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, Abdelrahman Mohamed.I-BERT(from Berkeley) released with the paperI-BERT: Integer-only BERT Quantizationby Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W. Mahoney, Kurt Keutzer.IDEFICS(from HuggingFace) released with the paperOBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentsby Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, Victor Sanh.ImageGPT(from OpenAI) released with the paperGenerative Pretraining from Pixelsby Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, Ilya Sutskever.Informer(from Beihang University, UC Berkeley, Rutgers University, SEDD Company) released with the paperInformer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingby Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang.InstructBLIP(from Salesforce) released with the paperInstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningby Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, Steven Hoi.Jukebox(from OpenAI) released with the paperJukebox: A Generative Model for Musicby Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, Ilya Sutskever.LayoutLM(from Microsoft Research Asia) released with the paperLayoutLM: Pre-training of Text and Layout for Document Image Understandingby Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou.LayoutLMv2(from Microsoft Research Asia) released with the paperLayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingby Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou.LayoutLMv3(from Microsoft Research Asia) released with the paperLayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingby Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, Furu Wei.LayoutXLM(from Microsoft Research Asia) released with the paperLayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understandingby Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei.LED(from AllenAI) released with the paperLongformer: The Long-Document Transformerby Iz Beltagy, Matthew E. Peters, Arman Cohan.LeViT(from Meta AI) released with the paperLeViT: A Vision Transformer in ConvNet's Clothing for Faster Inferenceby Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, Matthijs Douze.LiLT(from South China University of Technology) released with the paperLiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understandingby Jiapeng Wang, Lianwen Jin, Kai Ding.LLaMA(from The FAIR team of Meta AI) released with the paperLLaMA: Open and Efficient Foundation Language Modelsby Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample.Llama2(from The FAIR team of Meta AI) released with the paperLlama2: Open Foundation and Fine-Tuned Chat Modelsby Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushka rMishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing EllenTan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, Thomas Scialom.Longformer(from AllenAI) released with the paperLongformer: The Long-Document Transformerby Iz Beltagy, Matthew E. Peters, Arman Cohan.LongT5(from Google AI) released with the paperLongT5: Efficient Text-To-Text Transformer for Long Sequencesby Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, Yinfei Yang.LUKE(from Studio Ousia) released with the paperLUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentionby Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, Yuji Matsumoto.LXMERT(from UNC Chapel Hill) released with the paperLXMERT: Learning Cross-Modality Encoder Representations from Transformers for Open-Domain Question Answeringby Hao Tan and Mohit Bansal.M-CTC-T(from Facebook) released with the paperPseudo-Labeling For Massively Multilingual Speech Recognitionby Loren Lugosch, Tatiana Likhomanenko, Gabriel Synnaeve, and Ronan Collobert.M2M100(from Facebook) released with the paperBeyond English-Centric Multilingual Machine Translationby Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, Armand Joulin.MarianMTMachine translation models trained usingOPUSdata by Jörg Tiedemann. TheMarian Frameworkis being developed by the Microsoft Translator Team.MarkupLM(from Microsoft Research Asia) released with the paperMarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understandingby Junlong Li, Yiheng Xu, Lei Cui, Furu Wei.Mask2Former(from FAIR and UIUC) released with the paperMasked-attention Mask Transformer for Universal Image Segmentationby Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, Rohit Girdhar.MaskFormer(from Meta and UIUC) released with the paperPer-Pixel Classification is Not All You Need for Semantic Segmentationby Bowen Cheng, Alexander G. Schwing, Alexander Kirillov.MatCha(from Google AI) released with the paperMatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derenderingby Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, Julian Martin Eisenschlos.mBART(from Facebook) released with the paperMultilingual Denoising Pre-training for Neural Machine Translationby Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer.mBART-50(from Facebook) released with the paperMultilingual Translation with Extensible Multilingual Pretraining and Finetuningby Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, Angela Fan.MEGA(from Meta/USC/CMU/SJTU) released with the paperMega: Moving Average Equipped Gated Attentionby Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer.Megatron-BERT(from NVIDIA) released with the paperMegatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismby Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper and Bryan Catanzaro.Megatron-GPT2(from NVIDIA) released with the paperMegatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismby Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper and Bryan Catanzaro.MGP-STR(from Alibaba Research) released with the paperMulti-Granularity Prediction for Scene Text Recognitionby Peng Wang, Cheng Da, and Cong Yao.Mistral(from Mistral AI) by TheMistral AIteam: Albert Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, William El Sayed.mLUKE(from Studio Ousia) released with the papermLUKE: The Power of Entity Representations in Multilingual Pretrained Language Modelsby Ryokan Ri, Ikuya Yamada, and Yoshimasa Tsuruoka.MMS(from Facebook) released with the paperScaling Speech Technology to 1,000+ Languagesby Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, Michael Auli.MobileBERT(from CMU/Google Brain) released with the paperMobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicesby Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou.MobileNetV1(from Google Inc.) released with the paperMobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationsby Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, Hartwig Adam.MobileNetV2(from Google Inc.) released with the paperMobileNetV2: Inverted Residuals and Linear Bottlenecksby Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen.MobileViT(from Apple) released with the paperMobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformerby Sachin Mehta and Mohammad Rastegari.MobileViTV2(from Apple) released with the paperSeparable Self-attention for Mobile Vision Transformersby Sachin Mehta and Mohammad Rastegari.MPNet(from Microsoft Research) released with the paperMPNet: Masked and Permuted Pre-training for Language Understandingby Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, Tie-Yan Liu.MPT(from MosaiML) released with the repositoryllm-foundryby the MosaicML NLP Team.MRA(from the University of Wisconsin - Madison) released with the paperMulti Resolution Analysis (MRA) for Approximate Self-Attentionby Zhanpeng Zeng, Sourav Pal, Jeffery Kline, Glenn M Fung, Vikas Singh.MT5(from Google AI) released with the papermT5: A massively multilingual pre-trained text-to-text transformerby Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel.MusicGen(from Meta) released with the paperSimple and Controllable Music Generationby Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi and Alexandre Défossez.MVP(from RUC AI Box) released with the paperMVP: Multi-task Supervised Pre-training for Natural Language Generationby Tianyi Tang, Junyi Li, Wayne Xin Zhao and Ji-Rong Wen.NAT(from SHI Labs) released with the paperNeighborhood Attention Transformerby Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi.Nezha(from Huawei Noah’s Ark Lab) released with the paperNEZHA: Neural Contextualized Representation for Chinese Language Understandingby Junqiu Wei, Xiaozhe Ren, Xiaoguang Li, Wenyong Huang, Yi Liao, Yasheng Wang, Jiashu Lin, Xin Jiang, Xiao Chen and Qun Liu.NLLB(from Meta) released with the paperNo Language Left Behind: Scaling Human-Centered Machine Translationby the NLLB team.NLLB-MOE(from Meta) released with the paperNo Language Left Behind: Scaling Human-Centered Machine Translationby the NLLB team.Nougat(from Meta AI) released with the paperNougat: Neural Optical Understanding for Academic Documentsby Lukas Blecher, Guillem Cucurull, Thomas Scialom, Robert Stojnic.Nyströmformer(from the University of Wisconsin - Madison) released with the paperNyströmformer: A Nyström-Based Algorithm for Approximating Self-Attentionby Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, Vikas Singh.OneFormer(from SHI Labs) released with the paperOneFormer: One Transformer to Rule Universal Image Segmentationby Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, Humphrey Shi.OpenLlama(froms-JoL) released inOpen-Llama.OPT(from Meta AI) released with the paperOPT: Open Pre-trained Transformer Language Modelsby Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen et al.OWL-ViT(from Google AI) released with the paperSimple Open-Vocabulary Object Detection with Vision Transformersby Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby.Pegasus(from Google) released with the paperPEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationby Jingqing Zhang, Yao Zhao, Mohammad Saleh and Peter J. Liu.PEGASUS-X(from Google) released with the paperInvestigating Efficiently Extending Transformers for Long Input Summarizationby Jason Phang, Yao Zhao, and Peter J. Liu.Perceiver IO(from Deepmind) released with the paperPerceiver IO: A General Architecture for Structured Inputs & Outputsby Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, João Carreira.Persimmon(from ADEPT) released in ablog postby Erich Elsen, Augustus Odena, Maxwell Nye, Sağnak Taşırlar, Tri Dao, Curtis Hawthorne, Deepak Moparthi, Arushi Somani.PhoBERT(from VinAI Research) released with the paperPhoBERT: Pre-trained language models for Vietnameseby Dat Quoc Nguyen and Anh Tuan Nguyen.Pix2Struct(from Google) released with the paperPix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingby Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, Kristina Toutanova.PLBart(from UCLA NLP) released with the paperUnified Pre-training for Program Understanding and Generationby Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, Kai-Wei Chang.PoolFormer(from Sea AI Labs) released with the paperMetaFormer is Actually What You Need for Visionby Yu, Weihao and Luo, Mi and Zhou, Pan and Si, Chenyang and Zhou, Yichen and Wang, Xinchao and Feng, Jiashi and Yan, Shuicheng.Pop2Pianoreleased with the paperPop2Piano : Pop Audio-based Piano Cover Generationby Jongho Choi and Kyogu Lee.ProphetNet(from Microsoft Research) released with the paperProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingby Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang and Ming Zhou.PVT(from Nanjing University, The University of Hong Kong etc.) released with the paperPyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionsby Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao.QDQBert(from NVIDIA) released with the paperInteger Quantization for Deep Learning Inference: Principles and Empirical Evaluationby Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev and Paulius Micikevicius.RAG(from Facebook) released with the paperRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasksby Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela.REALM(from Google Research) released with the paperREALM: Retrieval-Augmented Language Model Pre-Trainingby Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat and Ming-Wei Chang.Reformer(from Google Research) released with the paperReformer: The Efficient Transformerby Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya.RegNet(from META Platforms) released with the paperDesigning Network Design Spaceby Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, Piotr Dollár.RemBERT(from Google Research) released with the paperRethinking embedding coupling in pre-trained language modelsby Hyung Won Chung, Thibault Févry, Henry Tsai, M. Johnson, Sebastian Ruder.ResNet(from Microsoft Research) released with the paperDeep Residual Learning for Image Recognitionby Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun.RoBERTa(from Facebook), released together with the paperRoBERTa: A Robustly Optimized BERT Pretraining Approachby Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov.RoBERTa-PreLayerNorm(from Facebook) released with the paperfairseq: A Fast, Extensible Toolkit for Sequence Modelingby Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, Michael Auli.RoCBert(from WeChatAI) released with the paperRoCBert: Robust Chinese Bert with Multimodal Contrastive Pretrainingby HuiSu, WeiweiShi, XiaoyuShen, XiaoZhou, TuoJi, JiaruiFang, JieZhou.RoFormer(from ZhuiyiTechnology), released together with the paperRoFormer: Enhanced Transformer with Rotary Position Embeddingby Jianlin Su and Yu Lu and Shengfeng Pan and Bo Wen and Yunfeng Liu.RWKV(from Bo Peng), released onthis repoby Bo Peng.SegFormer(from NVIDIA) released with the paperSegFormer: Simple and Efficient Design for Semantic Segmentation with Transformersby Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, Ping Luo.Segment Anything(from Meta AI) released with the paperSegment Anythingby Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alex Berg, Wan-Yen Lo, Piotr Dollar, Ross Girshick.SEW(from ASAPP) released with the paperPerformance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitionby Felix Wu, Kwangyoun Kim, Jing Pan, Kyu Han, Kilian Q. Weinberger, Yoav Artzi.SEW-D(from ASAPP) released with the paperPerformance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitionby Felix Wu, Kwangyoun Kim, Jing Pan, Kyu Han, Kilian Q. Weinberger, Yoav Artzi.SpeechT5(from Microsoft Research) released with the paperSpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingby Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei.SpeechToTextTransformer(from Facebook), released together with the paperfairseq S2T: Fast Speech-to-Text Modeling with fairseqby Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, Juan Pino.SpeechToTextTransformer2(from Facebook), released together with the paperLarge-Scale Self- and Semi-Supervised Learning for Speech Translationby Changhan Wang, Anne Wu, Juan Pino, Alexei Baevski, Michael Auli, Alexis Conneau.Splinter(from Tel Aviv University), released together with the paperFew-Shot Question Answering by Pretraining Span Selectionby Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, Omer Levy.SqueezeBERT(from Berkeley) released with the paperSqueezeBERT: What can computer vision teach NLP about efficient neural networks?by Forrest N. Iandola, Albert E. Shaw, Ravi Krishna, and Kurt W. Keutzer.SwiftFormer(from MBZUAI) released with the paperSwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applicationsby Abdelrahman Shaker, Muhammad Maaz, Hanoona Rasheed, Salman Khan, Ming-Hsuan Yang, Fahad Shahbaz Khan.Swin Transformer(from Microsoft) released with the paperSwin Transformer: Hierarchical Vision Transformer using Shifted Windowsby Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo.Swin Transformer V2(from Microsoft) released with the paperSwin Transformer V2: Scaling Up Capacity and Resolutionby Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, Baining Guo.Swin2SR(from University of Würzburg) released with the paperSwin2SR: SwinV2 Transformer for Compressed Image Super-Resolution and Restorationby Marcos V. Conde, Ui-Jin Choi, Maxime Burchi, Radu Timofte.SwitchTransformers(from Google) released with the paperSwitch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsityby William Fedus, Barret Zoph, Noam Shazeer.T5(from Google AI) released with the paperExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformerby Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu.T5v1.1(from Google AI) released in the repositorygoogle-research/text-to-text-transfer-transformerby Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu.Table Transformer(from Microsoft Research) released with the paperPubTables-1M: Towards Comprehensive Table Extraction From Unstructured Documentsby Brandon Smock, Rohith Pesala, Robin Abraham.TAPAS(from Google AI) released with the paperTAPAS: Weakly Supervised Table Parsing via Pre-trainingby Jonathan Herzig, Paweł Krzysztof Nowak, Thomas Müller, Francesco Piccinno and Julian Martin Eisenschlos.TAPEX(from Microsoft Research) released with the paperTAPEX: Table Pre-training via Learning a Neural SQL Executorby Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, Jian-Guang Lou.Time Series Transformer(from HuggingFace).TimeSformer(from Facebook) released with the paperIs Space-Time Attention All You Need for Video Understanding?by Gedas Bertasius, Heng Wang, Lorenzo Torresani.Trajectory Transformer(from the University of California at Berkeley) released with the paperOffline Reinforcement Learning as One Big Sequence Modeling Problemby Michael Janner, Qiyang Li, Sergey LevineTransformer-XL(from Google/CMU) released with the paperTransformer-XL: Attentive Language Models Beyond a Fixed-Length Contextby Zihang Dai*, Zhilin Yang*, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov.TrOCR(from Microsoft), released together with the paperTrOCR: Transformer-based Optical Character Recognition with Pre-trained Modelsby Minghao Li, Tengchao Lv, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei.TVLT(from UNC Chapel Hill) released with the paperTVLT: Textless Vision-Language Transformerby Zineng Tang, Jaemin Cho, Yixin Nie, Mohit Bansal.UL2(from Google Research) released with the paperUnifying Language Learning Paradigmsby Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, Donald MetzlerUMT5(from Google Research) released with the paperUniMax: Fairer and More Effective Language Sampling for Large-Scale Multilingual Pretrainingby Hyung Won Chung, Xavier Garcia, Adam Roberts, Yi Tay, Orhan Firat, Sharan Narang, Noah Constant.UniSpeech(from Microsoft Research) released with the paperUniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Databy Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael Zeng, Xuedong Huang.UniSpeechSat(from Microsoft Research) released with the paperUNISPEECH-SAT: UNIVERSAL SPEECH REPRESENTATION LEARNING WITH SPEAKER AWARE PRE-TRAININGby Sanyuan Chen, Yu Wu, Chengyi Wang, Zhengyang Chen, Zhuo Chen, Shujie Liu, Jian Wu, Yao Qian, Furu Wei, Jinyu Li, Xiangzhan Yu.UPerNet(from Peking University) released with the paperUnified Perceptual Parsing for Scene Understandingby Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, Jian Sun.VAN(from Tsinghua University and Nankai University) released with the paperVisual Attention Networkby Meng-Hao Guo, Cheng-Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, Shi-Min Hu.VideoMAE(from Multimedia Computing Group, Nanjing University) released with the paperVideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingby Zhan Tong, Yibing Song, Jue Wang, Limin Wang.ViLT(from NAVER AI Lab/Kakao Enterprise/Kakao Brain) released with the paperViLT: Vision-and-Language Transformer Without Convolution or Region Supervisionby Wonjae Kim, Bokyung Son, Ildoo Kim.Vision Transformer (ViT)(from Google AI) released with the paperAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scaleby Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby.VisualBERT(from UCLA NLP) released with the paperVisualBERT: A Simple and Performant Baseline for Vision and Languageby Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, Kai-Wei Chang.ViT Hybrid(from Google AI) released with the paperAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scaleby Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby.VitDet(from Meta AI) released with the paperExploring Plain Vision Transformer Backbones for Object Detectionby Yanghao Li, Hanzi Mao, Ross Girshick, Kaiming He.ViTMAE(from Meta AI) released with the paperMasked Autoencoders Are Scalable Vision Learnersby Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick.ViTMatte(from HUST-VL) rreleased with the paperViTMatte: Boosting Image Matting with Pretrained Plain Vision Transformersby Jingfeng Yao, Xinggang Wang, Shusheng Yang, Baoyuan Wang.ViTMSN(from Meta AI) released with the paperMasked Siamese Networks for Label-Efficient Learningby Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, Nicolas Ballas.VITS(from Kakao Enterprise) released with the paperConditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechby Jaehyeon Kim, Jungil Kong, Juhee Son.ViViT(from Google Research) released with the paperViViT: A Video Vision Transformerby Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, Cordelia Schmid.Wav2Vec2(from Facebook AI) released with the paperwav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationsby Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael Auli.Wav2Vec2-Conformer(from Facebook AI) released with the paperFAIRSEQ S2T: Fast Speech-to-Text Modeling with FAIRSEQby Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Sravya Popuri, Dmytro Okhonko, Juan Pino.Wav2Vec2Phoneme(from Facebook AI) released with the paperSimple and Effective Zero-shot Cross-lingual Phoneme Recognitionby Qiantong Xu, Alexei Baevski, Michael Auli.WavLM(from Microsoft Research) released with the paperWavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingby Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Furu Wei.Whisper(from OpenAI) released with the paperRobust Speech Recognition via Large-Scale Weak Supervisionby Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever.X-CLIP(from Microsoft Research) released with the paperExpanding Language-Image Pretrained Models for General Video Recognitionby Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, Haibin Ling.X-MOD(from Meta AI) released with the paperLifting the Curse of Multilinguality by Pre-training Modular Transformersby Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, Mikel Artetxe.XGLM(From Facebook AI) released with the paperFew-shot Learning with Multilingual Language Modelsby Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li.XLM(from Facebook) released together with the paperCross-lingual Language Model Pretrainingby Guillaume Lample and Alexis Conneau.XLM-ProphetNet(from Microsoft Research) released with the paperProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingby Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang and Ming Zhou.XLM-RoBERTa(from Facebook AI), released together with the paperUnsupervised Cross-lingual Representation Learning at Scaleby Alexis Conneau*, Kartikay Khandelwal*, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer and Veselin Stoyanov.XLM-RoBERTa-XL(from Facebook AI), released together with the paperLarger-Scale Transformers for Multilingual Masked Language Modelingby Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, Alexis Conneau.XLM-V(from Meta AI) released with the paperXLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelsby Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, Madian Khabsa.XLNet(from Google/CMU) released with the paper​XLNet: Generalized Autoregressive Pretraining for Language Understandingby Zhilin Yang*, Zihang Dai*, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le.XLS-R(from Facebook AI) released with the paperXLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scaleby Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, Michael Auli.XLSR-Wav2Vec2(from Facebook AI) released with the paperUnsupervised Cross-Lingual Representation Learning For Speech Recognitionby Alexis Conneau, Alexei Baevski, Ronan Collobert, Abdelrahman Mohamed, Michael Auli.YOLOS(from Huazhong University of Science & Technology) released with the paperYou Only Look at One Sequence: Rethinking Transformer in Vision through Object Detectionby Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, Wenyu Liu.YOSO(from the University of Wisconsin - Madison) released with the paperYou Only Sample (Almost) Once: Linear Cost Self-Attention Via Bernoulli Samplingby Zhanpeng Zeng, Yunyang Xiong, Sathya N. Ravi, Shailesh Acharya, Glenn Fung, Vikas Singh.Want to contribute a new model? We have added adetailed guide and templatesto guide you in the process of adding a new model. You can find them in thetemplatesfolder of the repository. Be sure to check thecontributing guidelinesand contact the maintainers or open an issue to collect feedbacks before starting your PR.To check if each model has an implementation in Flax, PyTorch or TensorFlow, or has an associated tokenizer backed by the 🤗 Tokenizers library, refer tothis table.These implementations have been tested on several datasets (see the example scripts) and should match the performance of the original implementations. You can find more details on performance in the Examples section of thedocumentation.Learn moreSectionDescriptionDocumentationFull API documentation and tutorialsTask summaryTasks supported by 🤗 TransformersPreprocessing tutorialUsing theTokenizerclass to prepare data for the modelsTraining and fine-tuningUsing the models provided by 🤗 Transformers in a PyTorch/TensorFlow training loop and theTrainerAPIQuick tour: Fine-tuning/usage scriptsExample scripts for fine-tuning models on a wide range of tasksModel sharing and uploadingUpload and share your fine-tuned models with the communityCitationWe now have apaperyou can cite for the 🤗 Transformers library:@inproceedings{wolf-etal-2020-transformers,title="Transformers: State-of-the-Art Natural Language Processing",author="Thomas Wolf and Lysandre Debut and Victor Sanh and Julien Chaumond and Clement Delangue and Anthony Moi and Pierric Cistac and Tim Rault and Rémi Louf and Morgan Funtowicz and Joe Davison and Sam Shleifer and Patrick von Platen and Clara Ma and Yacine Jernite and Julien Plu and Canwen Xu and Teven Le Scao and Sylvain Gugger and Mariama Drame and Quentin Lhoest and Alexander M. Rush",booktitle="Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",month=oct,year="2020",address="Online",publisher="Association for Computational Linguistics",url="https://www.aclweb.org/anthology/2020.emnlp-demos.6",pages="38--45"}