Package: jshbtf0302
State-of-the-art Natural Language Processing for PyTorch and TensorFlow 2.0🤗 Transformers provides thousands of pretrained models to perform tasks on texts such as classification, information extraction, question answering, summarization, translation, text generation, etc in 100+ languages. Its aim is to make cutting-edge NLP easier to use for everyone.🤗 Transformers provides APIs to quickly download and use those pretrained models on a given text, fine-tune them on your own datasets then share them with the community on ourmodel hub. At the same time, each python module defining an architecture can be used as a standalone and modified to enable quick research experiments.🤗 Transformers is backed by the two most popular deep learning libraries,PyTorchandTensorFlow, with a seamless integration between them, allowing you to train your models with one then load it for inference with the other.Online demosYou can test most of our models directly on their pages from themodel hub. We also offerprivate model hosting, versioning, & an inference APIto use those models.Here are a few examples:Masked word completion with BERTName Entity Recognition with ElectraText generation with GPT-2Natural Language Inference with RoBERTaSummarization with BARTQuestion answering with DistilBERTTranslation with T5Write With Transformer, built by the Hugging Face team, is the official demo of this repo’s text generation capabilities.Quick tourTo immediately use a model on a given text, we provide thepipelineAPI. Pipelines group together a pretrained model with the preprocessing that was used during that model training. Here is how to quickly use a pipeline to classify positive versus negative texts>>>fromtransformersimportpipeline# Allocate a pipeline for sentiment-analysis>>>classifier=pipeline('sentiment-analysis')>>>classifier('We are very happy to include pipeline into the transformers repository.')[{'label':'POSITIVE','score':0.9978193640708923}]The second line of code downloads and caches the pretrained model used by the pipeline, the third line evaluates it on the given text. Here the answer is "positive" with a confidence of 99.8%.This is another example of pipeline used for that can extract question answers from some context:>>>fromtransformersimportpipeline# Allocate a pipeline for question-answering>>>question_answerer=pipeline('question-answering')>>>question_answerer({...'question':'What is the name of the repository ?',...'context':'Pipeline have been included in the huggingface/transformers repository'...}){'score':0.5135612454720828,'start':35,'end':59,'answer':'huggingface/transformers'}On top of the answer, the pretrained model used here returned its confidence score, along with the start position and its end position in the tokenized sentence. You can learn more about the tasks supported by thepipelineAPI inthis tutorial.To download and use any of the pretrained models on your given task, you just need to use those three lines of codes (PyTorch version):>>>fromtransformersimportAutoTokenizer,AutoModel>>>tokenizer=AutoTokenizer.from_pretrained("bert-base-uncased")>>>model=AutoModel.from_pretrained("bert-base-uncased")>>>inputs=tokenizer("Hello world!",return_tensors="pt")>>>outputs=model(**inputs)or for TensorFlow:>>>fromtransformersimportAutoTokenizer,TFAutoModel>>>tokenizer=AutoTokenizer.from_pretrained("bert-base-uncased")>>>model=TFAutoModel.from_pretrained("bert-base-uncased")>>>inputs=tokenizer("Hello world!",return_tensors="tf")>>>outputs=model(**inputs)The tokenizer is responsible for all the preprocessing the pretrained model expects, and can be called directly on one (or list) of texts (as we can see on the fourth line of both code examples). It will output a dictionary you can directly pass to your model (which is done on the fifth line).The model itself is a regularPytorchnn.Moduleor aTensorFlowtf.keras.Model(depending on your backend) which you can use normally. For instance,this tutorialexplains how to integrate such a model in classic PyTorch or TensorFlow training loop, or how to use ourTrainerAPI to quickly fine-tune the on a new dataset.Why should I use transformers?Easy-to-use state-of-the-art models:High performance on NLU and NLG tasks.Low barrier to entry for educators and practitioners.Few user-facing abstractions with just three classes to learn.A unified API for using all our pretrained models.Lower compute costs, smaller carbon footprint:Researchers can share trained models instead of always retraining.Practitioners can reduce compute time and production costs.Dozens of architectures with over 2,000 pretrained models, some in more than 100 languages.Choose the right framework for every part of a model's lifetime:Train state-of-the-art models in 3 lines of code.Move a single model between TF2.0/PyTorch frameworks at will.Seamlessly pick the right framework for training, evaluation, production.Easily customize a model or an example to your needs:Examples for each architecture to reproduce the results by the official authors of said architecture.Expose the models internal as consistently as possible.Model files can be used independently of the library for quick experiments.Why shouldn't I use transformers?This library is not a modular toolbox of building blocks for neural nets. The code in the model files is not refactored with additional abstractions on purpose, so that researchers can quickly iterate on each of the models without diving in additional abstractions/files.The training API is not intended to work on any model but is optimized to work with the models provided by the library. For generic machine learning loops, you should use another library.While we strive to present as many use cases as possible, the scripts in ourexamples folderare just that: examples. It is expected that they won't work out-of-the box on your specific problem and that you will be required to change a few lines of code to adapt them to your needs.InstallationWith pipThis repository is tested on Python 3.6+, PyTorch 1.0.0+ (PyTorch 1.3.1+ forexamples) and TensorFlow 2.0.You should install 🤗 Transformers in avirtual environment. If you're unfamiliar with Python virtual environments, check out theuser guide.First, create a virtual environment with the version of Python you're going to use and activate it.Then, you will need to install at least one of TensorFlow 2.0, PyTorch or Flax.
Please refer toTensorFlow installation page,PyTorch installation pageregarding the specific install command for your platform and/orFlax installation page.When TensorFlow 2.0 and/or PyTorch has been installed, 🤗 Transformers can be installed using pip as follows:pipinstalltransformersIf you'd like to play with the examples or need the bleeding edge of the code and can't wait for a new release, you mustinstall the library from source.With condaSince Transformers version v4.0.0, we now have a conda channel:huggingface.🤗 Transformers can be installed using conda as follows:condainstall-chuggingfacetransformersFollow the installation pages of TensorFlow, PyTorch or Flax to see how to install them with conda.Models architecturesAll the model checkpointsprovided by 🤗 Transformers are seamlessly integrated from the huggingface.comodel hubwhere they are uploaded directly byusersandorganizations.Current number of checkpoints:🤗 Transformers currently provides the following architectures (seeherefor a high-level summary of each them):ALBERT(from Google Research and the Toyota Technological Institute at Chicago) released with the paperALBERT: A Lite BERT for Self-supervised Learning of Language Representations, by Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut.BART(from Facebook) released with the paperBART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensionby Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov and Luke Zettlemoyer.BARThez(from École polytechnique) released with the paperBARThez: a Skilled Pretrained French Sequence-to-Sequence Modelby Moussa Kamal Eddine, Antoine J.-P. Tixier, Michalis Vazirgiannis.BERT(from Google) released with the paperBERT: Pre-training of Deep Bidirectional Transformers for Language Understandingby Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova.BERT For Sequence Generation(from Google) released with the paperLeveraging Pre-trained Checkpoints for Sequence Generation Tasksby Sascha Rothe, Shashi Narayan, Aliaksei Severyn.Blenderbot(from Facebook) released with the paperRecipes for building an open-domain chatbotby Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, Jason Weston.BlenderbotSmall(from Facebook) released with the paperRecipes for building an open-domain chatbotby Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, Jason Weston.BORT(from Alexa) released with the paperOptimal Subarchitecture Extraction For BERTby Adrian de Wynter and Daniel J. Perry.CamemBERT(from Inria/Facebook/Sorbonne) released with the paperCamemBERT: a Tasty French Language Modelby Louis Martin*, Benjamin Muller*, Pedro Javier Ortiz Suárez*, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah and Benoît Sagot.ConvBERT(from YituTech) released with the paperConvBERT: Improving BERT with Span-based Dynamic Convolutionby Zihang Jiang, Weihao Yu, Daquan Zhou, Yunpeng Chen, Jiashi Feng, Shuicheng Yan.CTRL(from Salesforce) released with the paperCTRL: A Conditional Transformer Language Model for Controllable Generationby Nitish Shirish Keskar*, Bryan McCann*, Lav R. Varshney, Caiming Xiong and Richard Socher.DeBERTa(from Microsoft) released with the paperDeBERTa: Decoding-enhanced BERT with Disentangled Attentionby Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen.DeBERTa-v2(from Microsoft) released with the paperDeBERTa: Decoding-enhanced BERT with Disentangled Attentionby Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen.DialoGPT(from Microsoft Research) released with the paperDialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationby Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, Bill Dolan.DistilBERT(from HuggingFace), released together with the paperDistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighterby Victor Sanh, Lysandre Debut and Thomas Wolf. The same method has been applied to compress GPT2 intoDistilGPT2, RoBERTa intoDistilRoBERTa, Multilingual BERT intoDistilmBERTand a German version of DistilBERT.DPR(from Facebook) released with the paperDense Passage Retrieval
for Open-Domain Question Answeringby Vladimir Karpukhin, Barlas Oğuz, Sewon
Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.ELECTRA(from Google Research/Stanford University) released with the paperELECTRA: Pre-training text encoders as discriminators rather than generatorsby Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning.FlauBERT(from CNRS) released with the paperFlauBERT: Unsupervised Language Model Pre-training for Frenchby Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoît Crabbé, Laurent Besacier, Didier Schwab.Funnel Transformer(from CMU/Google Brain) released with the paperFunnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingby Zihang Dai, Guokun Lai, Yiming Yang, Quoc V. Le.GPT(from OpenAI) released with the paperImproving Language Understanding by Generative Pre-Trainingby Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever.GPT-2(from OpenAI) released with the paperLanguage Models are Unsupervised Multitask Learnersby Alec Radford*, Jeffrey Wu*, Rewon Child, David Luan, Dario Amodei** and Ilya Sutskever**.I-BERT(from Berkeley) released with the paperI-BERT: Integer-only BERT Quantizationby Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W. Mahoney, Kurt KeutzerLayoutLM(from Microsoft Research Asia) released with the paperLayoutLM: Pre-training of Text and Layout for Document Image Understandingby Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou.LED(from AllenAI) released with the paperLongformer: The Long-Document Transformerby Iz Beltagy, Matthew E. Peters, Arman Cohan.Longformer(from AllenAI) released with the paperLongformer: The Long-Document Transformerby Iz Beltagy, Matthew E. Peters, Arman Cohan.LXMERT(from UNC Chapel Hill) released with the paperLXMERT: Learning Cross-Modality Encoder Representations from Transformers for Open-Domain Question Answeringby Hao Tan and Mohit Bansal.MarianMTMachine translation models trained usingOPUSdata by Jörg Tiedemann. TheMarian Frameworkis being developed by the Microsoft Translator Team.MBart(from Facebook) released with the paperMultilingual Denoising Pre-training for Neural Machine Translationby Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer.MBart-50(from Facebook) released with the paperMultilingual Translation with Extensible Multilingual Pretraining and Finetuningby Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, Angela Fan.MPNet(from Microsoft Research) released with the paperMPNet: Masked and Permuted Pre-training for Language Understandingby Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, Tie-Yan Liu.MT5(from Google AI) released with the papermT5: A massively multilingual pre-trained text-to-text transformerby Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel.Pegasus(from Google) released with the paperPEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization> by Jingqing Zhang, Yao Zhao, Mohammad Saleh and Peter J. Liu.ProphetNet(from Microsoft Research) released with the paperProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingby Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang and Ming Zhou.Reformer(from Google Research) released with the paperReformer: The Efficient Transformerby Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya.RoBERTa(from Facebook), released together with the paper aRobustly Optimized BERT Pretraining Approachby Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov.SqueezeBertreleased with the paperSqueezeBERT: What can computer vision teach NLP about efficient neural networks?by Forrest N. Iandola, Albert E. Shaw, Ravi Krishna, and Kurt W. Keutzer.T5(from Google AI) released with the paperExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformerby Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu.TAPAS(from Google AI) released with the paperTAPAS: Weakly Supervised Table Parsing via Pre-trainingby Jonathan Herzig, Paweł Krzysztof Nowak, Thomas Müller, Francesco Piccinno and Julian Martin Eisenschlos.Transformer-XL(from Google/CMU) released with the paperTransformer-XL: Attentive Language Models Beyond a Fixed-Length Contextby Zihang Dai*, Zhilin Yang*, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov.Wav2Vec2(from Facebook AI) released with the paperwav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationsby Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael Auli.XLM(from Facebook) released together with the paperCross-lingual Language Model Pretrainingby Guillaume Lample and Alexis Conneau.XLM-ProphetNet(from Microsoft Research) released with the paperProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingby Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang and Ming Zhou.XLM-RoBERTa(from Facebook AI), released together with the paperUnsupervised Cross-lingual Representation Learning at Scaleby Alexis Conneau*, Kartikay Khandelwal*, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer and Veselin Stoyanov.XLNet(from Google/CMU) released with the paper​XLNet: Generalized Autoregressive Pretraining for Language Understandingby Zhilin Yang*, Zihang Dai*, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le.Want to contribute a new model? We have added adetailed guide and templatesto guide you in the process of adding a new model. You can find them in thetemplatesfolder of the repository. Be sure to check thecontributing guidelinesand contact the maintainers or open an issue to collect feedbacks before starting your PR.To check if each model has an implementation in PyTorch/TensorFlow/Flax or has an associated tokenizer backed by the 🤗 Tokenizers library, refer tothis tableThese implementations have been tested on several datasets (see the example scripts) and should match the performances of the original implementations. You can find more details on the performances in the Examples section of thedocumentation.Learn moreSectionDescriptionDocumentationFull API documentation and tutorialsTask summaryTasks supported by 🤗 TransformersPreprocessing tutorialUsing theTokenizerclass to prepare data for the modelsTraining and fine-tuningUsing the models provided by 🤗 Transformers in a PyTorch/TensorFlow training loop and theTrainerAPIQuick tour: Fine-tuning/usage scriptsExample scripts for fine-tuning models on a wide range of tasksModel sharing and uploadingUpload and share your fine-tuned models with the communityMigrationMigrate to 🤗 Transformers frompytorch-transformersorpytorch-pretrained-bertCitationWe now have apaperyou can cite for the 🤗 Transformers library:@inproceedings{wolf-etal-2020-transformers,title="Transformers: State-of-the-Art Natural Language Processing",author="Thomas Wolf and Lysandre Debut and Victor Sanh and Julien Chaumond and Clement Delangue and Anthony Moi and Pierric Cistac and Tim Rault and Rémi Louf and Morgan Funtowicz and Joe Davison and Sam Shleifer and Patrick von Platen and Clara Ma and Yacine Jernite and Julien Plu and Canwen Xu and Teven Le Scao and Sylvain Gugger and Mariama Drame and Quentin Lhoest and Alexander M. Rush",booktitle="Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",month=oct,year="2020",address="Online",publisher="Association for Computational Linguistics",url="https://www.aclweb.org/anthology/2020.emnlp-demos.6",pages="38--45"}