Package: transformers-phobert
State-of-the-art Natural Language Processing for PyTorch and TensorFlow 2.0🤗 Transformers (formerly known aspytorch-transformersandpytorch-pretrained-bert) provides state-of-the-art general-purpose architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet, T5, CTRL...) for Natural Language Understanding (NLU) and Natural Language Generation (NLG) with over thousands of pretrained models in 100+ languages and deep interoperability between PyTorch & TensorFlow 2.0.Recent contributorsFeaturesHigh performance on NLU and NLG tasksLow barrier to entry for educators and practitionersState-of-the-art NLP for everyoneDeep learning researchersHands-on practitionersAI/ML/NLP teachers and educatorsLower compute costs, smaller carbon footprintResearchers can share trained models instead of always retrainingPractitioners can reduce compute time and production costsDozens of architectures with over 1,000 pretrained models, some in more than 100 languagesChoose the right framework for every part of a model's lifetimeTrain state-of-the-art models in 3 lines of codeDeep interoperability between TensorFlow 2.0 and PyTorch modelsMove a single model between TF2.0/PyTorch frameworks at willSeamlessly pick the right framework for training, evaluation, productionSectionDescriptionInstallationHow to install the packageModel architecturesArchitectures (with pretrained weights)Online demoExperimenting with this repo’s text generation capabilitiesQuick tour: UsageTokenizers & models usage: Bert and GPT-2Quick tour: TF 2.0 and PyTorchTrain a TF 2.0 model in 10 lines of code, load it in PyTorchQuick tour: pipelinesUsing Pipelines: Wrapper around tokenizer and models to use finetuned modelsQuick tour: Fine-tuning/usage scriptsUsing provided scripts: GLUE, SQuAD and Text generationQuick tour: Share your modelsUpload and share your fine-tuned models with the communityMigrating from pytorch-transformers to transformersMigrating your code from pytorch-transformers to transformersMigrating from pytorch-pretrained-bert to pytorch-transformersMigrating your code from pytorch-pretrained-bert to transformersDocumentationFull API documentation and moreInstallationThis repo is tested on Python 3.6+, PyTorch 1.0.0+ (PyTorch 1.3.1+ for examples) and TensorFlow 2.0.You should install 🤗 Transformers in avirtual environment. If you're unfamiliar with Python virtual environments, check out theuser guide.Create a virtual environment with the version of Python you're going to use and activate it.Now, if you want to use 🤗 Transformers, you can install it with pip. If you'd like to play with the examples, you must install it from source.With pipFirst you need to install one of, or both, TensorFlow 2.0 and PyTorch.
Please refer toTensorFlow installation pageand/orPyTorch installation pageregarding the specific install command for your platform.When TensorFlow 2.0 and/or PyTorch has been installed, 🤗 Transformers can be installed using pip as follows:pipinstalltransformers-phobertFrom sourceHere also, you first need to install one of, or both, TensorFlow 2.0 and PyTorch.
Please refer toTensorFlow installation pageand/orPyTorch installation pageregarding the specific install command for your platform.When TensorFlow 2.0 and/or PyTorch has been installed, you can install from source by cloning the repository and running:gitclonehttps://github.com/huggingface/transformerscdtransformers
pipinstall.When you update the repository, you should upgrade the transformers installation and its dependencies as follows:gitpull
pipinstall--upgrade.Run the examplesExamples are included in the repository but are not shipped with the library.Therefore, in order to run the latest versions of the examples, you need to install from source, as described above.Look at theREADMEfor how to run examples.TestsA series of tests are included for the library and for some example scripts. Library tests can be found in thetests folderand examples tests in theexamples folder.Depending on which framework is installed (TensorFlow 2.0 and/or PyTorch), the irrelevant tests will be skipped. Ensure that both frameworks are installed if you want to execute all tests.Here's the easiest way to run tests for the library:pipinstall-e".[testing]"maketestand for the examples:pipinstall-e".[testing]"pipinstall-rexamples/requirements.txt
maketest-examplesFor details, refer to thecontributing guide.Do you want to run a Transformer model on a mobile device?You should check out ourswift-coreml-transformersrepo.It contains a set of tools to convert PyTorch or TensorFlow 2.0 trained Transformer models (currently containsGPT-2,DistilGPT-2,BERT, andDistilBERT) to CoreML models that run on iOS devices.At some point in the future, you'll be able to seamlessly move from pre-training or fine-tuning models to productizing them in CoreML, or prototype a model or an app in CoreML then research its hyperparameters or architecture from TensorFlow 2.0 and/or PyTorch. Super exciting!Model architectures🤗 Transformers currently provides the following NLU/NLG architectures:BERT(from Google) released with the paperBERT: Pre-training of Deep Bidirectional Transformers for Language Understandingby Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova.GPT(from OpenAI) released with the paperImproving Language Understanding by Generative Pre-Trainingby Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever.GPT-2(from OpenAI) released with the paperLanguage Models are Unsupervised Multitask Learnersby Alec Radford*, Jeffrey Wu*, Rewon Child, David Luan, Dario Amodei** and Ilya Sutskever**.Transformer-XL(from Google/CMU) released with the paperTransformer-XL: Attentive Language Models Beyond a Fixed-Length Contextby Zihang Dai*, Zhilin Yang*, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov.XLNet(from Google/CMU) released with the paper​XLNet: Generalized Autoregressive Pretraining for Language Understandingby Zhilin Yang*, Zihang Dai*, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le.XLM(from Facebook) released together with the paperCross-lingual Language Model Pretrainingby Guillaume Lample and Alexis Conneau.RoBERTa(from Facebook), released together with the paper aRobustly Optimized BERT Pretraining Approachby Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov.DistilBERT(from HuggingFace), released together with the paperDistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighterby Victor Sanh, Lysandre Debut and Thomas Wolf. The same method has been applied to compress GPT2 intoDistilGPT2, RoBERTa intoDistilRoBERTa, Multilingual BERT intoDistilmBERTand a German version of DistilBERT.CTRL(from Salesforce) released with the paperCTRL: A Conditional Transformer Language Model for Controllable Generationby Nitish Shirish Keskar*, Bryan McCann*, Lav R. Varshney, Caiming Xiong and Richard Socher.CamemBERT(from Inria/Facebook/Sorbonne) released with the paperCamemBERT: a Tasty French Language Modelby Louis Martin*, Benjamin Muller*, Pedro Javier Ortiz Suárez*, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah and Benoît Sagot.ALBERT(from Google Research and the Toyota Technological Institute at Chicago) released with the paperALBERT: A Lite BERT for Self-supervised Learning of Language Representations, by Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut.T5(from Google AI) released with the paperExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformerby Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu.XLM-RoBERTa(from Facebook AI), released together with the paperUnsupervised Cross-lingual Representation Learning at Scaleby Alexis Conneau*, Kartikay Khandelwal*, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer and Veselin Stoyanov.MMBT(from Facebook), released together with the paper aSupervised Multimodal Bitransformers for Classifying Images and Textby Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Davide Testuggine.FlauBERT(from CNRS) released with the paperFlauBERT: Unsupervised Language Model Pre-training for Frenchby Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoît Crabbé, Laurent Besacier, Didier Schwab.BART(from Facebook) released with the paperBART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensionby Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov and Luke Zettlemoyer.ELECTRA(from Google Research/Stanford University) released with the paperELECTRA: Pre-training text encoders as discriminators rather than generatorsby Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning.DialoGPT(from Microsoft Research) released with the paperDialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationby Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, Bill Dolan.Reformer(from Google Research) released with the paperReformer: The Efficient Transformerby Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya.MarianMTMachine translation models trained usingOPUSdata by Jörg Tiedemann. TheMarian Frameworkis being developed by the Microsoft Translator Team.Longformer(from AllenAI) released with the paperLongformer: The Long-Document Transformerby Iz Beltagy, Matthew E. Peters, Arman Cohan.DPR(from Facebook) released with the paperDense Passage Retrieval
for Open-Domain Question Answeringby Vladimir Karpukhin, Barlas Oğuz, Sewon
Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.Pegasus(from Google) released with the paperPEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization> by Jingqing Zhang, Yao Zhao, Mohammad Saleh and Peter J. Liu.MBart(from Facebook) released with the paperMultilingual Denoising Pre-training for Neural Machine Translationby Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer.LXMERT(from UNC Chapel Hill) released with the paperLXMERT: Learning Cross-Modality Encoder Representations from Transformers for Open-Domain Question Answeringby Hao Tan and Mohit Bansal.BERTweet(from VinAI Research) released with the paperBERTweet: A pre-trained language model for English Tweetsby Dat Quoc Nguyen, Thanh Vu and Anh Tuan Nguyen.PhoBERT(from VinAI Research) released with the paperPhoBERT: Pre-trained language models for Vietnameseby Dat Quoc Nguyen and Anh Tuan Nguyen.Other community models, contributed by thecommunity.Want to contribute a new model? We have added adetailed guide and templatesto guide you in the process of adding a new model. You can find them in thetemplatesfolder of the repository. Be sure to check thecontributing guidelinesand contact the maintainers or open an issue to collect feedbacks before starting your PR.These implementations have been tested on several datasets (see the example scripts) and should match the performances of the original implementations (e.g. ~93 F1 on SQuAD for BERT Whole-Word-Masking, ~88 F1 on RocStories for OpenAI GPT, ~18.3 perplexity on WikiText 103 for Transformer-XL, ~0.916 Pearson R coefficient on STS-B for XLNet). You can find more details on the performances in the Examples section of thedocumentation.Online demoYou can test our inference API on most model pages from the model hub:https://huggingface.co/modelsFor example:Masked word completion with BERTNER with ElectraText generation with GPT-2NLI with RoBERTaSummarization with BARTQuestion answering with DistilBERTTranslation with T5Write With Transformer, built by the Hugging Face team at transformer.huggingface.co, is the official demo of this repo’s text generation capabilities.Quick tourLet's do a very quick overview of the model architectures in 🤗 Transformers. Detailed examples for each model architecture (Bert, GPT, GPT-2, Transformer-XL, XLNet and XLM) can be found in thefull documentation.importtorchfromtransformersimport*# Transformers has a unified API# for 10 transformer architectures and 30 pretrained weights.#          Model          | Tokenizer          | Pretrained weights shortcutMODELS=[(BertModel,BertTokenizer,'bert-base-uncased'),(OpenAIGPTModel,OpenAIGPTTokenizer,'openai-gpt'),(GPT2Model,GPT2Tokenizer,'gpt2'),(CTRLModel,CTRLTokenizer,'ctrl'),(TransfoXLModel,TransfoXLTokenizer,'transfo-xl-wt103'),(XLNetModel,XLNetTokenizer,'xlnet-base-cased'),(XLMModel,XLMTokenizer,'xlm-mlm-enfr-1024'),(DistilBertModel,DistilBertTokenizer,'distilbert-base-cased'),(RobertaModel,RobertaTokenizer,'roberta-base'),(XLMRobertaModel,XLMRobertaTokenizer,'xlm-roberta-base'),]# To use TensorFlow 2.0 versions of the models, simply prefix the class names with 'TF', e.g. `TFRobertaModel` is the TF 2.0 counterpart of the PyTorch model `RobertaModel`# Let's encode some text in a sequence of hidden-states using each model:formodel_class,tokenizer_class,pretrained_weightsinMODELS:# Load pretrained model/tokenizertokenizer=tokenizer_class.from_pretrained(pretrained_weights)model=model_class.from_pretrained(pretrained_weights)# Encode textinput_ids=torch.tensor([tokenizer.encode("Here is some text to encode",add_special_tokens=True)])# Add special tokens takes care of adding [CLS], [SEP], 
... tokens in the right way for each model.withtorch.no_grad():last_hidden_states=model(input_ids)[0]# Models outputs are now tuples# Each architecture is provided with several class for fine-tuning on down-stream tasks, e.g.BERT_MODEL_CLASSES=[BertModel,BertForPreTraining,BertForMaskedLM,BertForNextSentencePrediction,BertForSequenceClassification,BertForTokenClassification,BertForQuestionAnswering]# All the classes for an architecture can be initiated from pretrained weights for this architecture# Note that additional weights added for fine-tuning are only initialized# and need to be trained on the down-stream taskpretrained_weights='bert-base-uncased'tokenizer=BertTokenizer.from_pretrained(pretrained_weights)formodel_classinBERT_MODEL_CLASSES:# Load pretrained model/tokenizermodel=model_class.from_pretrained(pretrained_weights)# Models can return full list of hidden-states & attentions weights at each layermodel=model_class.from_pretrained(pretrained_weights,output_hidden_states=True,output_attentions=True)input_ids=torch.tensor([tokenizer.encode("Let's see all hidden-states and attentions on this text")])all_hidden_states,all_attentions=model(input_ids)[-2:]# Models are compatible with Torchscriptmodel=model_class.from_pretrained(pretrained_weights,torchscript=True)traced_model=torch.jit.trace(model,(input_ids,))# Simple serialization for models and tokenizersmodel.save_pretrained('./directory/to/save/')# savemodel=model_class.from_pretrained('./directory/to/save/')# re-loadtokenizer.save_pretrained('./directory/to/save/')# savetokenizer=BertTokenizer.from_pretrained('./directory/to/save/')# re-load# SOTA examples for GLUE, SQUAD, text generation...Quick tour TF 2.0 training and PyTorch interoperabilityLet's do a quick example of how a TensorFlow 2.0 model can be trained in 12 lines of code with 🤗 Transformers and then loaded in PyTorch for fast inspection/tests.importtensorflowastfimporttensorflow_datasetsfromtransformersimport*# Load dataset, tokenizer, model from pretrained model/vocabularytokenizer=BertTokenizer.from_pretrained('bert-base-cased')model=TFBertForSequenceClassification.from_pretrained('bert-base-cased')data=tensorflow_datasets.load('glue/mrpc')# Prepare dataset for GLUE as a tf.data.Dataset instancetrain_dataset=glue_convert_examples_to_features(data['train'],tokenizer,max_length=128,task='mrpc')valid_dataset=glue_convert_examples_to_features(data['validation'],tokenizer,max_length=128,task='mrpc')train_dataset=train_dataset.shuffle(100).batch(32).repeat(2)valid_dataset=valid_dataset.batch(64)# Prepare training: Compile tf.keras model with optimizer, loss and learning rate scheduleoptimizer=tf.keras.optimizers.Adam(learning_rate=3e-5,epsilon=1e-08,clipnorm=1.0)loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)metric=tf.keras.metrics.SparseCategoricalAccuracy('accuracy')model.compile(optimizer=optimizer,loss=loss,metrics=[metric])# Train and evaluate using tf.keras.Model.fit()history=model.fit(train_dataset,epochs=2,steps_per_epoch=115,validation_data=valid_dataset,validation_steps=7)# Load the TensorFlow model in PyTorch for inspectionmodel.save_pretrained('./save/')pytorch_model=BertForSequenceClassification.from_pretrained('./save/',from_tf=True)# Quickly test a few predictions - MRPC is a paraphrasing task, let's see if our model learned the tasksentence_0="This research was consistent with his findings."sentence_1="His findings were compatible with this research."sentence_2="His findings were not compatible with this research."inputs_1=tokenizer(sentence_0,sentence_1,add_special_tokens=True,return_tensors='pt')inputs_2=tokenizer(sentence_0,sentence_2,add_special_tokens=True,return_tensors='pt')pred_1=pytorch_model(inputs_1['input_ids'],token_type_ids=inputs_1['token_type_ids'])[0].argmax().item()pred_2=pytorch_model(inputs_2['input_ids'],token_type_ids=inputs_2['token_type_ids'])[0].argmax().item()print("sentence_1 is","a paraphrase"ifpred_1else"not a paraphrase","of sentence_0")print("sentence_2 is","a paraphrase"ifpred_2else"not a paraphrase","of sentence_0")Quick tour of the fine-tuning/usage scriptsImportantBefore running the fine-tuning scripts, please read theinstructionson how to
setup your environment to run the examples.The library comprises several example scripts with SOTA performances for NLU and NLG tasks:run_glue.py: an example fine-tuning sequence classification models on nine different GLUE tasks (sequence-level classification)run_squad.py: an example fine-tuning question answering models on the question answering dataset SQuAD 2.0 (token-level classification)run_ner.py: an example fine-tuning token classification models on named entity recognition (token-level classification)run_generation.py: an example using GPT, GPT-2, CTRL, Transformer-XL and XLNet for conditional language generationother model-specific examples (see the documentation).Here are three quick usage examples for these scripts:run_glue.py: Fine-tuning on GLUE tasks for sequence classificationTheGeneral Language Understanding Evaluation (GLUE) benchmarkis a collection of nine sentence- or sentence-pair language understanding tasks for evaluating and analyzing natural language understanding systems.Before running any of these GLUE tasks you should download theGLUE databy runningthis scriptand unpack it to some directory$GLUE_DIR.You should also install the additional packages required by the examples:pipinstall-r./examples/requirements.txtexportGLUE_DIR=/path/to/glueexportTASK_NAME=MRPC

python./examples/text-classification/run_glue.py\--model_name_or_pathbert-base-uncased\--task_name$TASK_NAME\--do_train\--do_eval\--data_dir$GLUE_DIR/$TASK_NAME\--max_seq_length128\--per_device_eval_batch_size=8\--per_device_train_batch_size=8\--learning_rate2e-5\--num_train_epochs3.0\--output_dir/tmp/$TASK_NAME/where task name can be one of CoLA, SST-2, MRPC, STS-B, QQP, MNLI, QNLI, RTE, WNLI.The dev set results will be present within the text file 'eval_results.txt' in the specified output_dir. In case of MNLI, since there are two separate dev sets, matched and mismatched, there will be a separate output folder called '/tmp/MNLI-MM/' in addition to '/tmp/MNLI/'.Fine-tuning XLNet model on the STS-B regression taskThis example code fine-tunes XLNet on the STS-B corpus using parallel training on a server with 4 V100 GPUs.
Parallel training is a simple way to use several GPUs (but is slower and less flexible than distributed training, see below).exportGLUE_DIR=/path/to/glue

python./examples/text-classification/run_glue.py\--model_name_or_pathxlnet-large-cased\--do_train\--do_eval\--task_name=sts-b\--data_dir=${GLUE_DIR}/STS-B\--output_dir=./proc_data/sts-b-110\--max_seq_length=128\--per_device_eval_batch_size=8\--per_device_train_batch_size=8\--gradient_accumulation_steps=1\--max_steps=1200\--model_name=xlnet-large-cased\--overwrite_output_dir\--overwrite_cache\--warmup_steps=120On this machine we thus have a batch size of 32, please increasegradient_accumulation_stepsto reach the same batch size if you have a smaller machine. These hyper-parameters should result in a Pearson correlation coefficient of+0.917on the development set.Fine-tuning Bert model on the MRPC classification taskThis example code fine-tunes the Bert Whole Word Masking model on the Microsoft Research Paraphrase Corpus (MRPC) corpus using distributed training on 8 V100 GPUs to reach a F1 > 92.python-mtorch.distributed.launch--nproc_per_node8./examples/text-classification/run_glue.py\--model_name_or_pathbert-large-uncased-whole-word-masking\--task_nameMRPC\--do_train\--do_eval\--data_dir$GLUE_DIR/MRPC/\--max_seq_length128\--per_device_eval_batch_size=8\--per_device_train_batch_size=8\--learning_rate2e-5\--num_train_epochs3.0\--output_dir/tmp/mrpc_output/\--overwrite_output_dir\--overwrite_cache\Training with these hyper-parameters gave us the following results:acc=0.8823529411764706acc_and_f1=0.901702786377709eval_loss=0.3418912578906332f1=0.9210526315789473global_step=174loss=0.07231863956341798run_squad.py: Fine-tuning on SQuAD for question-answeringThis example code fine-tunes BERT on the SQuAD dataset using distributed training on 8 V100 GPUs and Bert Whole Word Masking uncased model to reach a F1 > 93 on SQuAD:python-mtorch.distributed.launch--nproc_per_node=8./examples/question-answering/run_squad.py\--model_typebert\--model_name_or_pathbert-large-uncased-whole-word-masking\--do_train\--do_eval\--train_file$SQUAD_DIR/train-v1.1.json\--predict_file$SQUAD_DIR/dev-v1.1.json\--learning_rate3e-5\--num_train_epochs2\--max_seq_length384\--doc_stride128\--output_dir../models/wwm_uncased_finetuned_squad/\--per_device_eval_batch_size=3\--per_device_train_batch_size=3\Training with these hyper-parameters gave us the following results:python$SQUAD_DIR/evaluate-v1.1.py$SQUAD_DIR/dev-v1.1.json../models/wwm_uncased_finetuned_squad/predictions.json{"exact_match":86.91579943235573,"f1":93.1532499015869}This is the model provided asbert-large-uncased-whole-word-masking-finetuned-squad.run_generation.py: Text generation with GPT, GPT-2, CTRL, Transformer-XL and XLNetA conditional generation script is also included to generate text from a prompt.
The generation script includes thetricksproposed by Aman Rusia to get high-quality generation with memory models like Transformer-XL and XLNet (include a predefined text to make short inputs longer).Here is how to run the script with the small version of OpenAI GPT-2 model:python./examples/text-generation/run_generation.py\--model_type=gpt2\--length=20\--model_name_or_path=gpt2\and from the Salesforce CTRL model:python./examples/text-generation/run_generation.py\--model_type=ctrl\--length=20\--model_name_or_path=ctrl\--temperature=0\--repetition_penalty=1.2\Quick tour of model sharingStarting withv2.2.2, you can now upload and share your fine-tuned models with the community, using theCLIthat's built-in to the library.First, create an account onhttps://huggingface.co/join. Optionally, join an existing organization or create a new one. Then:transformers-clilogin# log in using the same credentials as on huggingface.coUpload your model:transformers-cliupload./path/to/pretrained_model/# ^^ Upload folder containing weights/tokenizer/config# saved via `.save_pretrained()`transformers-cliupload./config.json[--filenamefolder/foobar.json]# ^^ Upload a single file# (you can optionally override its filename, which can be nested inside a folder)If you want your model to be namespaced by your organization name rather than your username, add the following flag to any command:--organizationorganization_nameYour model will then be accessible through its identifier, a concatenation of your username (or organization name) and the folder name above:"username/pretrained_model"# or if an org:"organization_name/pretrained_model"Please add a README.md model cardto the repo undermodel_cards/with: model description, training params (dataset, preprocessing, hardware used, hyperparameters), evaluation results, intended uses & limitations, etc.Your model now has a page on huggingface.co/models 🔥Anyone can load it from code:tokenizer=AutoTokenizer.from_pretrained("namespace/pretrained_model")model=AutoModel.from_pretrained("namespace/pretrained_model")List all your files on S3:transformers-clis3lsYou can also delete unneeded files:transformers-clis3rm…Quick tour of pipelinesNew in versionv2.3:Pipelineare high-level objects which automatically handle tokenization, running your data through a transformers model
and outputting the result in a structured object.You can createPipelineobjects for the following down-stream tasks:feature-extraction: Generates a tensor representation for the input sequencener: Generates named entity mapping for each word in the input sequence.sentiment-analysis: Gives the polarity (positive / negative) of the whole input sequence.text-classification: Initialize aTextClassificationPipelinedirectly, or seesentiment-analysisfor an example.question-answering: Provided some context and a question refering to the context, it will extract the answer to the question in the context.fill-mask: Takes an input sequence containing a masked token (e.g.<mask>) and return list of most probable filled sequences, with their probabilities.summarizationtranslation_xx_to_yy>>>fromtransformersimportpipeline# Allocate a pipeline for sentiment-analysis>>>nlp=pipeline('sentiment-analysis')>>>nlp('We are very happy to include pipeline into the transformers repository.')[{'label':'POSITIVE','score':0.9978193640708923}]# Allocate a pipeline for question-answering>>>nlp=pipeline('question-answering')>>>nlp({...'question':'What is the name of the repository ?',...'context':'Pipeline have been included in the huggingface/transformers repository'...}){'score':0.5135612454720828,'start':35,'end':59,'answer':'huggingface/transformers'}Migrating from pytorch-transformers to transformersHere is a quick summary of what you should take care of when migrating frompytorch-transformerstotransformers.Positional order of some models' keywords inputs (attention_mask,token_type_ids...) changedTo be able to use Torchscript (see #1010, #1204 and #1195) the specific order of some modelskeywords inputs(attention_mask,token_type_ids...) has been changed.If you used to call the models with keyword names for keyword arguments, e.g.model(inputs_ids, attention_mask=attention_mask, token_type_ids=token_type_ids), this should not cause any change.If you used to call the models with positional inputs for keyword arguments, e.g.model(inputs_ids, attention_mask, token_type_ids), you may have to double check the exact order of input arguments.Migrating from pytorch-pretrained-bert to transformersHere is a quick summary of what you should take care of when migrating frompytorch-pretrained-berttotransformers.Models always outputtuplesThe main breaking change when migrating frompytorch-pretrained-berttotransformersis that every model's forward method always outputs atuplewith various elements depending on the model and the configuration parameters.The exact content of the tuples for each model is detailed in the models' docstrings and thedocumentation.In pretty much every case, you will be fine by taking the first element of the output as the output you previously used inpytorch-pretrained-bert.Here is apytorch-pretrained-berttotransformersconversion example for aBertForSequenceClassificationclassification model:# Let's load our modelmodel=BertForSequenceClassification.from_pretrained('bert-base-uncased')# If you used to have this line in pytorch-pretrained-bert:loss=model(input_ids,labels=labels)# Now just use this line in transformers to extract the loss from the output tuple:outputs=model(input_ids,labels=labels)loss=outputs[0]# In transformers you can also have access to the logits:loss,logits=outputs[:2]# And even the attention weights if you configure the model to output them (and other outputs too, see the docstrings and documentation)model=BertForSequenceClassification.from_pretrained('bert-base-uncased',output_attentions=True)outputs=model(input_ids,labels=labels)loss,logits,attentions=outputsUsing hidden statesBy enabling the configuration optionoutput_hidden_states, it was possible to retrieve the last hidden states of the encoder. Inpytorch-transformersas well astransformersthe return value has changed slightly:all_hidden_statesnow also includes the hidden state of the embeddings in addition to those of the encoding layers. This allows users to easily access the embeddings final state.SerializationBreaking change in thefrom_pretrained()method:Models are now set in evaluation mode by default when instantiated with thefrom_pretrained()method. To train them, don't forget to set them back in training mode (model.train()) to activate the dropout modules.The additional*inputand**kwargsarguments supplied to thefrom_pretrained()method used to be directly passed to the underlying model's class__init__()method. They are now used to update the model configuration attribute instead, which can break derived model classes built based on the previousBertForSequenceClassificationexamples. We are working on a way to mitigate this breaking change in#866by forwarding the model's__init__()method (i) the provided positional arguments and (ii) the keyword arguments which do not match any configuration class attributes.Also, while not a breaking change, the serialization methods have been standardized and you probably should switch to the new methodsave_pretrained(save_directory)if you were using any other serialization method before.Here is an example:### Let's load a model and tokenizermodel=BertForSequenceClassification.from_pretrained('bert-base-uncased')tokenizer=BertTokenizer.from_pretrained('bert-base-uncased')### Do some stuff to our model and tokenizer# Ex: add new tokens to the vocabulary and embeddings of our modeltokenizer.add_tokens(['[SPECIAL_TOKEN_1]','[SPECIAL_TOKEN_2]'])model.resize_token_embeddings(len(tokenizer))# Train our modeltrain(model)### Now let's save our model and tokenizer to a directorymodel.save_pretrained('./my_saved_model_directory/')tokenizer.save_pretrained('./my_saved_model_directory/')### Reload the model and the tokenizermodel=BertForSequenceClassification.from_pretrained('./my_saved_model_directory/')tokenizer=BertTokenizer.from_pretrained('./my_saved_model_directory/')Optimizers: BertAdam & OpenAIAdam are now AdamW, schedules are standard PyTorch schedulesThe two optimizers previously included,BertAdamandOpenAIAdam, have been replaced by a singleAdamWoptimizer which has a few differences:it only implements weights decay correction,schedules are now externals (see below),gradient clipping is now also external (see below).The new optimizerAdamWmatches PyTorchAdamoptimizer API and let you use standard PyTorch or apex methods for the schedule and clipping.The schedules are now standardPyTorch learning rate schedulersand not part of the optimizer anymore.Here is a conversion examples fromBertAdamwith a linear warmup and decay schedule toAdamWand the same schedule:# Parameters:lr=1e-3max_grad_norm=1.0num_training_steps=1000num_warmup_steps=100warmup_proportion=float(num_warmup_steps)/float(num_training_steps)# 0.1### Previously BertAdam optimizer was instantiated like this:optimizer=BertAdam(model.parameters(),lr=lr,schedule='warmup_linear',warmup=warmup_proportion,t_total=num_training_steps)### and used like this:forbatchintrain_data:loss=model(batch)loss.backward()optimizer.step()### In Transformers, optimizer and schedules are splitted and instantiated like this:optimizer=AdamW(model.parameters(),lr=lr,correct_bias=False)# To reproduce BertAdam specific behavior set correct_bias=Falsescheduler=get_linear_schedule_with_warmup(optimizer,num_warmup_steps=num_warmup_steps,num_training_steps=num_training_steps)# PyTorch scheduler### and used like this:forbatchintrain_data:model.train()loss=model(batch)loss.backward()torch.nn.utils.clip_grad_norm_(model.parameters(),max_grad_norm)# Gradient clipping is not in AdamW anymore (so you can use amp without issue)optimizer.step()scheduler.step()optimizer.zero_grad()CitationWe now have apaperyou can cite for the 🤗 Transformers library:@article{Wolf2019HuggingFacesTS,title={HuggingFace's Transformers: State-of-the-art Natural Language Processing},author={Thomas Wolf and Lysandre Debut and Victor Sanh and Julien Chaumond and Clement Delangue and Anthony Moi and Pierric Cistac and Tim Rault and Rémi Louf and Morgan Funtowicz and Joe Davison and Sam Shleifer and Patrick von Platen and Clara Ma and Yacine Jernite and Julien Plu and Canwen Xu and Teven Le Scao and Sylvain Gugger and Mariama Drame and Quentin Lhoest and Alexander M. Rush},journal={ArXiv},year={2019},volume={abs/1910.03771}}