Package: pytorch-pretrained-bert
PyTorch Pretrained BERT: The Big & Extending Repository of pretrained TransformersThis repository contains op-for-op PyTorch reimplementations, pre-trained models and fine-tuning examples for:Google's BERT model,OpenAI's GPT model,Google/CMU's Transformer-XL model, andOpenAI's GPT-2 model.These implementations have been tested on several datasets (see the examples) and should match the performances of the associated TensorFlow implementations (e.g. ~91 F1 on SQuAD for BERT, ~88 F1 on RocStories for OpenAI GPT and ~18.3 perplexity on WikiText 103 for the Transformer-XL). You can find more details in theExamplessection below.Here are some information on these models:BERTwas released together with the paperBERT: Pre-training of Deep Bidirectional Transformers for Language Understandingby Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova.
This PyTorch implementation of BERT is provided withGoogle's pre-trained models, examples, notebooks and a command-line interface to load any pre-trained TensorFlow checkpoint for BERT is also provided.OpenAI GPTwas released together with the paperImproving Language Understanding by Generative Pre-Trainingby Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever.
This PyTorch implementation of OpenAI GPT is an adaptation of thePyTorch implementation by HuggingFaceand is provided withOpenAI's pre-trained modeland a command-line interface that was used to convert the pre-trained NumPy checkpoint in PyTorch.Google/CMU's Transformer-XLwas released together with the paperTransformer-XL: Attentive Language Models Beyond a Fixed-Length Contextby Zihang Dai*, Zhilin Yang*, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov.
This PyTorch implementation of Transformer-XL is an adaptation of the originalPyTorch implementationwhich has been slightly modified to match the performances of the TensorFlow implementation and allow to re-use the pretrained weights. A command-line interface is provided to convert TensorFlow checkpoints in PyTorch models.OpenAI GPT-2was released together with the paperLanguage Models are Unsupervised Multitask Learnersby Alec Radford*, Jeffrey Wu*, Rewon Child, David Luan, Dario Amodei** and Ilya Sutskever**.
This PyTorch implementation of OpenAI GPT-2 is an adaptation of theOpenAI's implementationand is provided withOpenAI's pre-trained modeland a command-line interface that was used to convert the TensorFlow checkpoint in PyTorch.ContentSectionDescriptionInstallationHow to install the packageOverviewOverview of the packageUsageQuickstart examplesDocDetailed documentationExamplesDetailed examples on how to fine-tune BertNotebooksIntroduction on the provided Jupyter NotebooksTPUNotes on TPU support and pretraining scriptsCommand-line interfaceConvert a TensorFlow checkpoint in a PyTorch dumpInstallationThis repo was tested on Python 2.7 and 3.5+ (examples are tested only on python 3.5+) and PyTorch 0.4.1/1.0.0With pipPyTorch pretrained bert can be installed by pip as follows:pipinstallpytorch-pretrained-bertIf you want to reproduce the original tokenization process of theOpenAI GPTpaper, you will need to installftfy(limit to version 4.4.3 if you are using Python 2) andSpaCy:pipinstallspacyftfy==4.4.3
python-mspacydownloadenIf you don't installftfyandSpaCy, theOpenAI GPTtokenizer will default to tokenize using BERT'sBasicTokenizerfollowed by Byte-Pair Encoding (which should be fine for most usage, don't worry).From sourceClone the repository and run:pipinstall[--editable].Here also, if you want to reproduce the original tokenization process of theOpenAI GPTmodel, you will need to installftfy(limit to version 4.4.3 if you are using Python 2) andSpaCy:pipinstallspacyftfy==4.4.3
python-mspacydownloadenAgain, if you don't installftfyandSpaCy, theOpenAI GPTtokenizer will default to tokenize using BERT'sBasicTokenizerfollowed by Byte-Pair Encoding (which should be fine for most usage).A series of tests is included in thetests folderand can be run usingpytest(install pytest if needed:pip install pytest).You can run the tests with the command:python-mpytest-svtests/OverviewThis package comprises the following classes that can be imported in Python and are detailed in theDocsection of this readme:EightBertPyTorch models (torch.nn.Module) with pre-trained weights (in themodeling.pyfile):BertModel- raw BERT Transformer model (fully pre-trained),BertForMaskedLM- BERT Transformer with the pre-trained masked language modeling head on top (fully pre-trained),BertForNextSentencePrediction- BERT Transformer with the pre-trained next sentence prediction classifier on top  (fully pre-trained),BertForPreTraining- BERT Transformer with masked language modeling head and next sentence prediction classifier on top (fully pre-trained),BertForSequenceClassification- BERT Transformer with a sequence classification head on top (BERT Transformer ispre-trained, the sequence classification headis only initialized and has to be trained),BertForMultipleChoice- BERT Transformer with a multiple choice head on top (used for task like Swag) (BERT Transformer ispre-trained, the multiple choice classification headis only initialized and has to be trained),BertForTokenClassification- BERT Transformer with a token classification head on top (BERT Transformer ispre-trained, the token classification headis only initialized and has to be trained),BertForQuestionAnswering- BERT Transformer with a token classification head on top (BERT Transformer ispre-trained, the token classification headis only initialized and has to be trained).ThreeOpenAI GPTPyTorch models (torch.nn.Module) with pre-trained weights (in themodeling_openai.pyfile):OpenAIGPTModel- raw OpenAI GPT Transformer model (fully pre-trained),OpenAIGPTLMHeadModel- OpenAI GPT Transformer with the tied language modeling head on top (fully pre-trained),OpenAIGPTDoubleHeadsModel- OpenAI GPT Transformer with the tied language modeling head and a multiple choice classification head on top (OpenAI GPT Transformer ispre-trained, the multiple choice classification headis only initialized and has to be trained),TwoTransformer-XLPyTorch models (torch.nn.Module) with pre-trained weights (in themodeling_transfo_xl.pyfile):TransfoXLModel- Transformer-XL model which outputs the last hidden state and memory cells (fully pre-trained),TransfoXLLMHeadModel- Transformer-XL with the tied adaptive softmax head on top for language modeling which outputs the logits/loss and memory cells (fully pre-trained),ThreeOpenAI GPT-2PyTorch models (torch.nn.Module) with pre-trained weights (in themodeling_gpt2.pyfile):GPT2Model- raw OpenAI GPT-2 Transformer model (fully pre-trained),GPT2LMHeadModel- OpenAI GPT-2 Transformer with the tied language modeling head on top (fully pre-trained),GPT2DoubleHeadsModel- OpenAI GPT-2 Transformer with the tied language modeling head and a multiple choice classification head on top (OpenAI GPT-2 Transformer ispre-trained, the multiple choice classification headis only initialized and has to be trained),Tokenizers forBERT(using word-piece) (in thetokenization.pyfile):BasicTokenizer- basic tokenization (punctuation splitting, lower casing, etc.),WordpieceTokenizer- WordPiece tokenization,BertTokenizer- perform end-to-end tokenization, i.e. basic tokenization followed by WordPiece tokenization.Tokenizer forOpenAI GPT(using Byte-Pair-Encoding) (in thetokenization_openai.pyfile):OpenAIGPTTokenizer- perform Byte-Pair-Encoding (BPE) tokenization.Tokenizer forTransformer-XL(word tokens ordered by frequency for adaptive softmax) (in thetokenization_transfo_xl.pyfile):OpenAIGPTTokenizer- perform word tokenization and can order words by frequency in a corpus for use in an adaptive softmax.Tokenizer forOpenAI GPT-2(using byte-level Byte-Pair-Encoding) (in thetokenization_gpt2.pyfile):GPT2Tokenizer- perform byte-level Byte-Pair-Encoding (BPE) tokenization.Optimizer forBERT(in theoptimization.pyfile):BertAdam- Bert version of Adam algorithm with weight decay fix, warmup and linear decay of the learning rate.Optimizer forOpenAI GPT(in theoptimization_openai.pyfile):OpenAIAdam- OpenAI GPT version of Adam algorithm with weight decay fix, warmup and linear decay of the learning rate.Configuration classes for BERT, OpenAI GPT and Transformer-XL (in the respectivemodeling.py,modeling_openai.py,modeling_transfo_xl.pyfiles):BertConfig- Configuration class to store the configuration of aBertModelwith utilities to read and write from JSON configuration files.OpenAIGPTConfig- Configuration class to store the configuration of aOpenAIGPTModelwith utilities to read and write from JSON configuration files.GPT2Config- Configuration class to store the configuration of aGPT2Modelwith utilities to read and write from JSON configuration files.TransfoXLConfig- Configuration class to store the configuration of aTransfoXLModelwith utilities to read and write from JSON configuration files.The repository further comprises:Five examples on how to useBERT(in theexamplesfolder):extract_features.py- Show how to extract hidden states from an instance ofBertModel,run_classifier.py- Show how to fine-tune an instance ofBertForSequenceClassificationon GLUE's MRPC task,run_squad.py- Show how to fine-tune an instance ofBertForQuestionAnsweringon SQuAD v1.0 and SQuAD v2.0 tasks.run_swag.py- Show how to fine-tune an instance ofBertForMultipleChoiceon Swag task.simple_lm_finetuning.py- Show how to fine-tune an instance ofBertForPretrainingon a target text corpus.One example on how to useOpenAI GPT(in theexamplesfolder):run_openai_gpt.py- Show how to fine-tune an instance ofOpenGPTDoubleHeadsModelon the RocStories task.One example on how to useTransformer-XL(in theexamplesfolder):run_transfo_xl.py- Show how to load and evaluate a pre-trained model ofTransfoXLLMHeadModelon WikiText 103.One example on how to useOpenAI GPT-2in the unconditional and interactive mode (in theexamplesfolder):run_gpt2.py- Show how to use OpenAI GPT-2 an instance ofGPT2LMHeadModelto generate text (same as the original OpenAI GPT-2 examples).These examples are detailed in theExamplessection of this readme.Three notebooks that were used to check that the TensorFlow and PyTorch models behave identically (in thenotebooksfolder):Comparing-TF-and-PT-models.ipynb- Compare the hidden states predicted byBertModel,Comparing-TF-and-PT-models-SQuAD.ipynb- Compare the spans predicted byBertForQuestionAnsweringinstances,Comparing-TF-and-PT-models-MLM-NSP.ipynb- Compare the predictions of theBertForPretraininginstances.These notebooks are detailed in theNotebookssection of this readme.A command-line interface to convert TensorFlow checkpoints (BERT, Transformer-XL) or NumPy checkpoint (OpenAI) in a PyTorch save of the associated PyTorch model:This CLI is detailed in theCommand-line interfacesection of this readme.UsageBERTHere is a quick-start example usingBertTokenizer,BertModelandBertForMaskedLMclass with Google AI's pre-trainedBert base uncasedmodel. See thedoc sectionbelow for all the details on these classes.First let's prepare a tokenized input withBertTokenizerimporttorchfrompytorch_pretrained_bertimportBertTokenizer,BertModel,BertForMaskedLM# OPTIONAL: if you want to have more information on what's happening, activate the logger as followsimportlogginglogging.basicConfig(level=logging.INFO)# Load pre-trained model tokenizer (vocabulary)tokenizer=BertTokenizer.from_pretrained('bert-base-uncased')# Tokenized inputtext="[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"tokenized_text=tokenizer.tokenize(text)# Mask a token that we will try to predict back with `BertForMaskedLM`masked_index=8tokenized_text[masked_index]='[MASK]'asserttokenized_text==['[CLS]','who','was','jim','henson','?','[SEP]','jim','[MASK]','was','a','puppet','##eer','[SEP]']# Convert token to vocabulary indicesindexed_tokens=tokenizer.convert_tokens_to_ids(tokenized_text)# Define sentence A and B indices associated to 1st and 2nd sentences (see paper)segments_ids=[0,0,0,0,0,0,0,1,1,1,1,1,1,1]# Convert inputs to PyTorch tensorstokens_tensor=torch.tensor([indexed_tokens])segments_tensors=torch.tensor([segments_ids])Let's see how to useBertModelto get hidden states# Load pre-trained model (weights)model=BertModel.from_pretrained('bert-base-uncased')model.eval()# If you have a GPU, put everything on cudatokens_tensor=tokens_tensor.to('cuda')segments_tensors=segments_tensors.to('cuda')model.to('cuda')# Predict hidden states features for each layerwithtorch.no_grad():encoded_layers,_=model(tokens_tensor,segments_tensors)# We have a hidden states for each of the 12 layers in model bert-base-uncasedassertlen(encoded_layers)==12And how to useBertForMaskedLM# Load pre-trained model (weights)model=BertForMaskedLM.from_pretrained('bert-base-uncased')model.eval()# If you have a GPU, put everything on cudatokens_tensor=tokens_tensor.to('cuda')segments_tensors=segments_tensors.to('cuda')model.to('cuda')# Predict all tokenswithtorch.no_grad():predictions=model(tokens_tensor,segments_tensors)# confirm we were able to predict 'henson'predicted_index=torch.argmax(predictions[0,masked_index]).item()predicted_token=tokenizer.convert_ids_to_tokens([predicted_index])[0]assertpredicted_token=='henson'OpenAI GPTHere is a quick-start example usingOpenAIGPTTokenizer,OpenAIGPTModelandOpenAIGPTLMHeadModelclass with OpenAI's pre-trained  model. See thedoc sectionbelow for all the details on these classes.First let's prepare a tokenized input withOpenAIGPTTokenizerimporttorchfrompytorch_pretrained_bertimportOpenAIGPTTokenizer,OpenAIGPTModel,OpenAIGPTLMHeadModel# OPTIONAL: if you want to have more information on what's happening, activate the logger as followsimportlogginglogging.basicConfig(level=logging.INFO)# Load pre-trained model tokenizer (vocabulary)tokenizer=OpenAIGPTTokenizer.from_pretrained('openai-gpt')# Tokenized inputtext="Who was Jim Henson ? Jim Henson was a puppeteer"tokenized_text=tokenizer.tokenize(text)# Convert token to vocabulary indicesindexed_tokens=tokenizer.convert_tokens_to_ids(tokenized_text)# Convert inputs to PyTorch tensorstokens_tensor=torch.tensor([indexed_tokens])Let's see how to useOpenAIGPTModelto get hidden states# Load pre-trained model (weights)model=OpenAIGPTModel.from_pretrained('openai-gpt')model.eval()# If you have a GPU, put everything on cudatokens_tensor=tokens_tensor.to('cuda')model.to('cuda')# Predict hidden states features for each layerwithtorch.no_grad():hidden_states=model(tokens_tensor)And how to useOpenAIGPTLMHeadModel# Load pre-trained model (weights)model=OpenAIGPTLMHeadModel.from_pretrained('openai-gpt')model.eval()# If you have a GPU, put everything on cudatokens_tensor=tokens_tensor.to('cuda')model.to('cuda')# Predict all tokenswithtorch.no_grad():predictions=model(tokens_tensor)# get the predicted last tokenpredicted_index=torch.argmax(predictions[0,-1,:]).item()predicted_token=tokenizer.convert_ids_to_tokens([predicted_index])[0]assertpredicted_token=='.</w>'Transformer-XLHere is a quick-start example usingTransfoXLTokenizer,TransfoXLModelandTransfoXLModelLMHeadModelclass with the Transformer-XL model pre-trained on WikiText-103. See thedoc sectionbelow for all the details on these classes.First let's prepare a tokenized input withTransfoXLTokenizerimporttorchfrompytorch_pretrained_bertimportTransfoXLTokenizer,TransfoXLModel,TransfoXLLMHeadModel# OPTIONAL: if you want to have more information on what's happening, activate the logger as followsimportlogginglogging.basicConfig(level=logging.INFO)# Load pre-trained model tokenizer (vocabulary from wikitext 103)tokenizer=TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')# Tokenized inputtext_1="Who was Jim Henson ?"text_2="Jim Henson was a puppeteer"tokenized_text_1=tokenizer.tokenize(text_1)tokenized_text_2=tokenizer.tokenize(text_2)# Convert token to vocabulary indicesindexed_tokens_1=tokenizer.convert_tokens_to_ids(tokenized_text_1)indexed_tokens_2=tokenizer.convert_tokens_to_ids(tokenized_text_2)# Convert inputs to PyTorch tensorstokens_tensor_1=torch.tensor([indexed_tokens_1])tokens_tensor_2=torch.tensor([indexed_tokens_2])Let's see how to useTransfoXLModelto get hidden states# Load pre-trained model (weights)model=TransfoXLModel.from_pretrained('transfo-xl-wt103')model.eval()# If you have a GPU, put everything on cudatokens_tensor_1=tokens_tensor_1.to('cuda')tokens_tensor_2=tokens_tensor_2.to('cuda')model.to('cuda')withtorch.no_grad():# Predict hidden states features for each layerhidden_states_1,mems_1=model(tokens_tensor_1)# We can re-use the memory cells in a subsequent call to attend a longer contexthidden_states_2,mems_2=model(tokens_tensor_2,mems=mems_1)And how to useTransfoXLLMHeadModel# Load pre-trained model (weights)model=TransfoXLLMHeadModel.from_pretrained('transfo-xl-wt103')model.eval()# If you have a GPU, put everything on cudatokens_tensor_1=tokens_tensor_1.to('cuda')tokens_tensor_2=tokens_tensor_2.to('cuda')model.to('cuda')withtorch.no_grad():# Predict all tokenspredictions_1,mems_1=model(tokens_tensor_1)# We can re-use the memory cells in a subsequent call to attend a longer contextpredictions_2,mems_2=model(tokens_tensor_2,mems=mems_1)# get the predicted last tokenpredicted_index=torch.argmax(predictions_2[0,-1,:]).item()predicted_token=tokenizer.convert_ids_to_tokens([predicted_index])[0]assertpredicted_token=='who'OpenAI GPT-2Here is a quick-start example usingGPT2Tokenizer,GPT2ModelandGPT2LMHeadModelclass with OpenAI's pre-trained  model. See thedoc sectionbelow for all the details on these classes.First let's prepare a tokenized input withGPT2Tokenizerimporttorchfrompytorch_pretrained_bertimportGPT2Tokenizer,GPT2Model,GPT2LMHeadModel# OPTIONAL: if you want to have more information on what's happening, activate the logger as followsimportlogginglogging.basicConfig(level=logging.INFO)# Load pre-trained model tokenizer (vocabulary)tokenizer=GPT2Tokenizer.from_pretrained('gpt2')# Encode some inputstext_1="Who was Jim Henson ?"text_2="Jim Henson was a puppeteer"indexed_tokens_1=tokenizer.encode(text_1)indexed_tokens_2=tokenizer.encode(text_2)# Convert inputs to PyTorch tensorstokens_tensor_1=torch.tensor([indexed_tokens_1])tokens_tensor_2=torch.tensor([indexed_tokens_2])Let's see how to useGPT2Modelto get hidden states# Load pre-trained model (weights)model=GPT2Model.from_pretrained('gpt2')model.eval()# If you have a GPU, put everything on cudatokens_tensor_1=tokens_tensor_1.to('cuda')tokens_tensor_2=tokens_tensor_2.to('cuda')model.to('cuda')# Predict hidden states features for each layerwithtorch.no_grad():hidden_states_1,past=model(tokens_tensor_1)# past can be used to reuse precomputed hidden state in a subsequent predictions# (see beam-search examples in the run_gpt2.py example).hidden_states_2,past=model(tokens_tensor_2,past=past)And how to useGPT2LMHeadModel# Load pre-trained model (weights)model=GPT2LMHeadModel.from_pretrained('gpt2')model.eval()# If you have a GPU, put everything on cudatokens_tensor_1=tokens_tensor_1.to('cuda')tokens_tensor_2=tokens_tensor_2.to('cuda')model.to('cuda')# Predict all tokenswithtorch.no_grad():predictions_1,past=model(tokens_tensor_1)# past can be used to reuse precomputed hidden state in a subsequent predictions# (see beam-search examples in the run_gpt2.py example).predictions_2,past=model(tokens_tensor_2,past=past)# get the predicted last tokenpredicted_index=torch.argmax(predictions_2[0,-1,:]).item()predicted_token=tokenizer.decode([predicted_index])DocHere is a detailed documentation of the classes in the package and how to use them:Sub-sectionDescriptionLoading pre-trained weightsHow to load Google AI/OpenAI's pre-trained weight or a PyTorch saved instanceSerialization best-practicesHow to save and reload a fine-tuned modelConfigurationsAPI of the configuration classes for BERT, GPT, GPT-2 and Transformer-XLModelsAPI of the PyTorch model classes for BERT, GPT, GPT-2 and Transformer-XLTokenizersAPI of the tokenizers class for BERT, GPT, GPT-2 and Transformer-XLOptimizersAPI of the optimizersLoading Google AI or OpenAI pre-trained weights or PyTorch dumpTo load one of Google AI's, OpenAI's pre-trained models or a PyTorch saved model (an instance ofBertForPreTrainingsaved withtorch.save()), the PyTorch model classes and the tokenizer can be instantiated asmodel=BERT_CLASS.from_pretrained(PRE_TRAINED_MODEL_NAME_OR_PATH,cache_dir=None)whereBERT_CLASSis either a tokenizer to load the vocabulary (BertTokenizerorOpenAIGPTTokenizerclasses) or one of the eight BERT or three OpenAI GPT PyTorch model classes (to load the pre-trained weights):BertModel,BertForMaskedLM,BertForNextSentencePrediction,BertForPreTraining,BertForSequenceClassification,BertForTokenClassification,BertForMultipleChoice,BertForQuestionAnswering,OpenAIGPTModel,OpenAIGPTLMHeadModelorOpenAIGPTDoubleHeadsModel, andPRE_TRAINED_MODEL_NAME_OR_PATHis either:the shortcut name of a Google AI's or OpenAI's pre-trained model selected in the list:bert-base-uncased: 12-layer, 768-hidden, 12-heads, 110M parametersbert-large-uncased: 24-layer, 1024-hidden, 16-heads, 340M parametersbert-base-cased: 12-layer, 768-hidden, 12-heads , 110M parametersbert-large-cased: 24-layer, 1024-hidden, 16-heads, 340M parametersbert-base-multilingual-uncased: (Orig, not recommended) 102 languages, 12-layer, 768-hidden, 12-heads, 110M parametersbert-base-multilingual-cased:(New, recommended)104 languages, 12-layer, 768-hidden, 12-heads, 110M parametersbert-base-chinese: Chinese Simplified and Traditional, 12-layer, 768-hidden, 12-heads, 110M parametersopenai-gpt: OpenAI English model, 12-layer, 768-hidden, 12-heads, 110M parameterstransfo-xl-wt103: Transformer-XL English model trained on wikitext-103, 18-layer, 1024-hidden, 16-heads, 257M parametersgpt2: OpenAI GPT-2 English model, 12-layer, 768-hidden, 12-heads, 117M parametersa path or url to a pretrained model archive containing:bert_config.jsonoropenai_gpt_config.jsona configuration file for the model, andpytorch_model.bina PyTorch dump of a pre-trained instance ofBertForPreTraining,OpenAIGPTModel,TransfoXLModel,GPT2LMHeadModel(saved with the usualtorch.save())IfPRE_TRAINED_MODEL_NAME_OR_PATHis a shortcut name, the pre-trained weights will be downloaded from AWS S3 (see the linkshere) and stored in a cache folder to avoid future download (the cache folder can be found at~/.pytorch_pretrained_bert/).cache_dircan be an optional path to a specific directory to download and cache the pre-trained model weights. This option is useful in particular when you are using distributed training: to avoid concurrent access to the same weights you can set for examplecache_dir='./pretrained_model_{}'.format(args.local_rank)(see the section on distributed training for more information).Uncasedmeans that the text has been lowercased before WordPiece tokenization, e.g.,John Smithbecomesjohn smith. The Uncased model also strips out any accent markers.Casedmeans that the true case and accent markers are preserved. Typically, the Uncased model is better unless you know that case information is important for your task (e.g., Named Entity Recognition or Part-of-Speech tagging). For information about the Multilingual and Chinese model, see theMultilingual READMEor the original TensorFlow repository.When using anuncased model, make sure to pass--do_lower_caseto the example training scripts (or passdo_lower_case=Trueto FullTokenizer if you're using your own script and loading the tokenizer your-self.).Examples:# BERTtokenizer=BertTokenizer.from_pretrained('bert-base-uncased',do_lower_case=True,do_basic_tokenize=True)model=BertForSequenceClassification.from_pretrained('bert-base-uncased')# OpenAI GPTtokenizer=OpenAIGPTTokenizer.from_pretrained('openai-gpt')model=OpenAIGPTModel.from_pretrained('openai-gpt')# Transformer-XLtokenizer=TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')model=TransfoXLModel.from_pretrained('transfo-xl-wt103')# OpenAI GPT-2tokenizer=GPT2Tokenizer.from_pretrained('gpt2')model=GPT2Model.from_pretrained('gpt2')Serialization best-practicesThis section explain how you can save and re-load a fine-tuned model (BERT, GPT, GPT-2 and Transformer-XL).
There are three types of files you need to save to be able to reload a fine-tuned model:the model it-self which should be saved following PyTorch serializationbest practices,the configuration file of the model which is saved as a JSON file, andthe vocabulary (and the merges for the BPE-based models GPT and GPT-2).Here is the recommended way of saving the model, configuration and vocabulary to anoutput_dirdirectory and reloading the model and tokenizer afterwards:frompytorch_pretrained_bertimportWEIGHTS_NAME,CONFIG_NAMEoutput_dir="./models/"# Step 1: Save a model, configuration and vocabulary that you have fine-tuned# If we have a distributed model, save only the encapsulated model# (it was wrapped in PyTorch DistributedDataParallel or DataParallel)model_to_save=model.moduleifhasattr(model,'module')elsemodel# If we save using the predefined names, we can load using `from_pretrained`output_model_file=os.path.join(output_dir,WEIGHTS_NAME)output_config_file=os.path.join(output_dir,CONFIG_NAME)torch.save(model_to_save.state_dict(),output_model_file)model_to_save.config.to_json_file(output_config_file)tokenizer.save_vocabulary(output_dir)# Step 2: Re-load the saved model and vocabulary# Example for a Bert modelmodel=BertForQuestionAnswering.from_pretrained(output_dir)tokenizer=BertTokenizer.from_pretrained(output_dir,do_lower_case=args.do_lower_case)# Add specific options if needed# Example for a GPT modelmodel=OpenAIGPTDoubleHeadsModel.from_pretrained(output_dir)tokenizer=OpenAIGPTTokenizer.from_pretrained(output_dir)Here is another way you can save and reload the model if you want to use specific paths for each type of files:output_model_file="./models/my_own_model_file.bin"output_config_file="./models/my_own_config_file.bin"output_vocab_file="./models/my_own_vocab_file.bin"# Step 1: Save a model, configuration and vocabulary that you have fine-tuned# If we have a distributed model, save only the encapsulated model# (it was wrapped in PyTorch DistributedDataParallel or DataParallel)model_to_save=model.moduleifhasattr(model,'module')elsemodeltorch.save(model_to_save.state_dict(),output_model_file)model_to_save.config.to_json_file(output_config_file)tokenizer.save_vocabulary(output_vocab_file)# Step 2: Re-load the saved model and vocabulary# We didn't save using the predefined WEIGHTS_NAME, CONFIG_NAME names, we cannot load using `from_pretrained`.# Here is how to do it in this situation:# Example for a Bert modelconfig=BertConfig.from_json_file(output_config_file)model=BertForQuestionAnswering(config)state_dict=torch.load(output_model_file)model.load_state_dict(state_dict)tokenizer=BertTokenizer(output_vocab_file,do_lower_case=args.do_lower_case)# Example for a GPT modelconfig=OpenAIGPTConfig.from_json_file(output_config_file)model=OpenAIGPTDoubleHeadsModel(config)state_dict=torch.load(output_model_file)model.load_state_dict(state_dict)tokenizer=OpenAIGPTTokenizer(output_vocab_file)ConfigurationsModels (BERT, GPT, GPT-2 and Transformer-XL) are defined and build from configuration classes which containes the parameters of the models (number of layers, dimensionalities...) and a few utilities to read and write from JSON configuration files. The respective configuration classes are:BertConfigforBertModeland BERT classes instances.OpenAIGPTConfigforOpenAIGPTModeland OpenAI GPT classes instances.GPT2ConfigforGPT2Modeland OpenAI GPT-2 classes instances.TransfoXLConfigforTransfoXLModeland Transformer-XL classes instances.These configuration classes contains a few utilities to load and save configurations:from_dict(cls, json_object): A class method to construct a configuration from a Python dictionary of parameters. Returns an instance of the configuration class.from_json_file(cls, json_file): A class method to construct a configuration from a json file of parameters. Returns an instance of the configuration class.to_dict(): Serializes an instance to a Python dictionary. Returns a dictionary.to_json_string(): Serializes an instance to a JSON string. Returns a string.to_json_file(json_file_path): Save an instance to a json file.Models1.BertModelBertModelis the basic BERT Transformer model with a layer of summed token, position and sequence embeddings followed by a series of identical self-attention blocks (12 for BERT-base, 24 for BERT-large).The inputs and output areidentical to the TensorFlow model inputs and outputs.We detail them here. This model takes asinputs:modeling.pyinput_ids: a torch.LongTensor of shape [batch_size, sequence_length] with the word token indices in the vocabulary (see the tokens preprocessing logic in the scriptsextract_features.py,run_classifier.pyandrun_squad.py), andtoken_type_ids: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token types indices selected in [0, 1]. Type 0 corresponds to asentence Aand type 1 corresponds to asentence Btoken (see BERT paper for more details).attention_mask: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [0, 1]. It's a mask to be used if some input sequence lengths are smaller than the max input sequence length of the current batch. It's the mask that we typically use for attention when a batch has varying length sentences.output_all_encoded_layers: boolean which controls the content of theencoded_layersoutput as described below. Default:True.This modeloutputsa tuple composed of:encoded_layers: controled by the value of theoutput_encoded_layersargument:output_all_encoded_layers=True: outputs a list of the encoded-hidden-states at the end of each attention block (i.e. 12 full sequences for BERT-base, 24 for BERT-large), each encoded-hidden-state is a torch.FloatTensor of size [batch_size, sequence_length, hidden_size],output_all_encoded_layers=False: outputs only the encoded-hidden-states corresponding to the last attention block, i.e. a single torch.FloatTensor of size [batch_size, sequence_length, hidden_size],pooled_output: a torch.FloatTensor of size [batch_size, hidden_size] which is the output of a classifier pretrained on top of the hidden state associated to the first character of the input (CLF) to train on the Next-Sentence task (see BERT's paper).An example on how to use this class is given in theextract_features.pyscript which can be used to extract the hidden states of the model for a given input.2.BertForPreTrainingBertForPreTrainingincludes theBertModelTransformer followed by the two pre-training heads:the masked language modeling head, andthe next sentence classification head.Inputscomprises the inputs of theBertModelclass plus two optional labels:masked_lm_labels: masked language modeling labels: torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [-1, 0, ..., vocab_size]. All labels set to -1 are ignored (masked), the loss is only computed for the labels set in [0, ..., vocab_size]next_sentence_label: next sentence classification loss: torch.LongTensor of shape [batch_size] with indices selected in [0, 1]. 0 => next sentence is the continuation, 1 => next sentence is a random sentence.Outputs:ifmasked_lm_labelsandnext_sentence_labelare notNone: Outputs the total_loss which is the sum of the masked language modeling loss and the next sentence classification loss.ifmasked_lm_labelsornext_sentence_labelisNone: Outputs a tuple comprisingthe masked language modeling logits, andthe next sentence classification logits.An example on how to use this class is given in therun_lm_finetuning.pyscript which can be used to fine-tune the BERT language model on your specific different text corpus. This should improve model performance, if the language style is different from the original BERT training corpus (Wiki + BookCorpus).3.BertForMaskedLMBertForMaskedLMincludes theBertModelTransformer followed by the (possibly) pre-trained  masked language modeling head.Inputscomprises the inputs of theBertModelclass plus optional label:masked_lm_labels: masked language modeling labels: torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [-1, 0, ..., vocab_size]. All labels set to -1 are ignored (masked), the loss is only computed for the labels set in [0, ..., vocab_size]Outputs:ifmasked_lm_labelsis notNone: Outputs the masked language modeling loss.ifmasked_lm_labelsisNone: Outputs the masked language modeling logits.4.BertForNextSentencePredictionBertForNextSentencePredictionincludes theBertModelTransformer followed by the next sentence classification head.Inputscomprises the inputs of theBertModelclass plus an optional label:next_sentence_label: next sentence classification loss: torch.LongTensor of shape [batch_size] with indices selected in [0, 1]. 0 => next sentence is the continuation, 1 => next sentence is a random sentence.Outputs:ifnext_sentence_labelis notNone: Outputs the next sentence classification loss.ifnext_sentence_labelisNone: Outputs the next sentence classification logits.5.BertForSequenceClassificationBertForSequenceClassificationis a fine-tuning model that includesBertModeland a sequence-level (sequence or pair of sequences) classifier on top of theBertModel.The sequence-level classifier is a linear layer that takes as input the last hidden state of the first character in the input sequence (see Figures 3a and 3b in the BERT paper).An example on how to use this class is given in therun_classifier.pyscript which can be used to fine-tune a single sequence (or pair of sequence) classifier using BERT, for example for the MRPC task.6.BertForMultipleChoiceBertForMultipleChoiceis a fine-tuning model that includesBertModeland a linear layer on top of theBertModel.The linear layer outputs a single value for each choice of a multiple choice problem, then all the outputs corresponding to an instance are passed through a softmax to get the model choice.This implementation is largely inspired by the work of OpenAI inImproving Language Understanding by Generative Pre-Trainingand the answer of Jacob Devlin in the followingissue.An example on how to use this class is given in therun_swag.pyscript which can be used to fine-tune a multiple choice classifier using BERT, for example for the Swag task.7.BertForTokenClassificationBertForTokenClassificationis a fine-tuning model that includesBertModeland a token-level classifier on top of theBertModel.The token-level classifier is a linear layer that takes as input the last hidden state of the sequence.8.BertForQuestionAnsweringBertForQuestionAnsweringis a fine-tuning model that includesBertModelwith a token-level classifiers on top of the full sequence of last hidden states.The token-level classifier takes as input the full sequence of the last hidden state and compute several (e.g. two) scores for each tokens that can for example respectively be the score that a given token is astart_spanand aend_spantoken (see Figures 3c and 3d in the BERT paper).An example on how to use this class is given in therun_squad.pyscript which can be used to fine-tune a token classifier using BERT, for example for the SQuAD task.9.OpenAIGPTModelOpenAIGPTModelis the basic OpenAI GPT Transformer model with a layer of summed token and position embeddings followed by a series of 12 identical self-attention blocks.OpenAI GPT use a single embedding matrix to store the word and special embeddings.
Special tokens embeddings are additional tokens that are not pre-trained:[SEP],[CLS]...
Special tokens need to be trained during the fine-tuning if you use them.
The number of special embeddings can be controled using theset_num_special_tokens(num_special_tokens)function.The embeddings are ordered as follow in the token embeddings matrice:[0,----------------------...->wordembeddingsconfig.vocab_size-1,______________________config.vocab_size,...->specialembeddingsconfig.vocab_size+config.n_special-1]______________________where total_tokens_embeddings can be obtained as config.total_tokens_embeddings and is:total_tokens_embeddings = config.vocab_size + config.n_specialYou should use the associate indices to index the embeddings.The inputs and output areidentical to the TensorFlow model inputs and outputs.We detail them here. This model takes asinputs:modeling_openai.pyinput_ids: a torch.LongTensor of shape [batch_size, sequence_length] (or more generally [d_1, ..., d_n, sequence_length] were d_1 ... d_n are arbitrary dimensions) with the word BPE token indices selected in the range [0, total_tokens_embeddings[position_ids: an optional torch.LongTensor with the same shape as input_ids
with the position indices (selected in the range [0, config.n_positions - 1[.token_type_ids: an optional torch.LongTensor with the same shape as input_ids
You can use it to add a third type of embedding to each input token in the sequence
(the previous two being the word and position embeddings). The input, position and token_type embeddings are summed inside the Transformer before the first self-attention block.This modeloutputs:hidden_states: the encoded-hidden-states at the top of the model as a torch.FloatTensor of size [batch_size, sequence_length, hidden_size] (or more generally [d_1, ..., d_n, hidden_size] were d_1 ... d_n are the dimension of input_ids)10.OpenAIGPTLMHeadModelOpenAIGPTLMHeadModelincludes theOpenAIGPTModelTransformer followed by a language modeling head with weights tied to the input embeddings (no additional parameters).Inputsare the same as the inputs of theOpenAIGPTModelclass plus optional labels:lm_labels: optional language modeling labels: torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [-1, 0, ..., vocab_size]. All labels set to -1 are ignored (masked), the loss is only computed for the labels set in [0, ..., vocab_size].Outputs:iflm_labelsis notNone:
Outputs the language modeling loss.else:
Outputslm_logits: the language modeling logits as a torch.FloatTensor of size [batch_size, sequence_length, total_tokens_embeddings] (or more generally [d_1, ..., d_n, total_tokens_embeddings] were d_1 ... d_n are the dimension of input_ids)11.OpenAIGPTDoubleHeadsModelOpenAIGPTDoubleHeadsModelincludes theOpenAIGPTModelTransformer followed by two heads:a language modeling head with weights tied to the input embeddings (no additional parameters) and:a multiple choice classifier (linear layer that take as input a hidden state in a sequence to compute a score, see details in paper).Inputsare the same as the inputs of theOpenAIGPTModelclass plus a classification mask and two optional labels:multiple_choice_token_ids: a torch.LongTensor of shape [batch_size, num_choices] with the index of the token whose hidden state should be used as input for the multiple choice classifier (usually the [CLS] token for each choice).lm_labels: optional language modeling labels: torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [-1, 0, ..., vocab_size]. All labels set to -1 are ignored (masked), the loss is only computed for the labels set in [0, ..., vocab_size].multiple_choice_labels: optional multiple choice labels: torch.LongTensor of shape [batch_size] with indices selected in [0, ..., num_choices].Outputs:iflm_labelsandmultiple_choice_labelsare notNone:
Outputs a tuple of losses with the language modeling loss and the multiple choice loss.else Outputs a tuple with:lm_logits: the language modeling logits as a torch.FloatTensor of size [batch_size, num_choices, sequence_length, total_tokens_embeddings]multiple_choice_logits: the multiple choice logits as a torch.FloatTensor of size [batch_size, num_choices]12.TransfoXLModelThe Transformer-XL model is described in "Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context".Transformer XL use a relative positioning with sinusiodal patterns and adaptive softmax inputs which means that:you don't need to specify positioning embeddings indicesthe tokens in the vocabulary have to be sorted to decreasing frequency.This model takes asinputs:modeling_transfo_xl.pyinput_ids: a torch.LongTensor of shape [batch_size, sequence_length] with the token indices selected in the range [0, self.config.n_token[mems: an optional memory of hidden states from previous forward passes as a list (num layers) of hidden states at the entry of each layer. Each hidden states has shape [self.config.mem_len, bsz, self.config.d_model]. Note that the first two dimensions are transposed inmemswith regards toinput_ids.This modeloutputsa tuple of (last_hidden_state, new_mems)last_hidden_state: the encoded-hidden-states at the top of the model as a torch.FloatTensor of size [batch_size, sequence_length, self.config.d_model]new_mems: list (num layers) of updated mem states at the entry of each layer each mem state is a torch.FloatTensor of size [self.config.mem_len, batch_size, self.config.d_model]. Note that the first two dimensions are transposed inmemswith regards toinput_ids.Extracting a list of the hidden states at each layer of the Transformer-XL fromlast_hidden_stateandnew_mems:Thenew_memscontain all the hidden states PLUS the output of the embeddings (new_mems[0]).new_mems[-1]is the output of the hidden state of the layer below the last layer andlast_hidden_stateis the output of the last layer (i.E. the input of the softmax when we have a language modeling head on top).There are two differences between the shapes ofnew_memsandlast_hidden_state:new_memshave transposed first dimensions and are longer (of sizeself.config.mem_len). Here is how to extract the full list of hidden states from the model output:hidden_states,mems=model(tokens_tensor)seq_length=hidden_states.size(1)lower_hidden_states=list(t[-seq_length:,...].transpose(0,1)fortinmems)all_hidden_states=lower_hidden_states+[hidden_states]13.TransfoXLLMHeadModelTransfoXLLMHeadModelincludes theTransfoXLModelTransformer followed by an (adaptive) softmax head with weights tied to the input embeddings.Inputsare the same as the inputs of theTransfoXLModelclass plus optional labels:target: an optional torch.LongTensor of shape [batch_size, sequence_length] with the target token indices selected in the range [0, self.config.n_token[Outputsa tuple of (last_hidden_state, new_mems)softmax_output: output of the (adaptive) softmax:if target is None: log probabilities of tokens, shape [batch_size, sequence_length, n_tokens]else: Negative log likelihood of target tokens with shape [batch_size, sequence_length]new_mems: list (num layers) of updated mem states at the entry of each layer each mem state is a torch.FloatTensor of size [self.config.mem_len, batch_size, self.config.d_model]. Note that the first two dimensions are transposed inmemswith regards toinput_ids.14.GPT2ModelGPT2Modelis the OpenAI GPT-2 Transformer model with a layer of summed token and position embeddings followed by a series of 12 identical self-attention blocks.The inputs and output areidentical to the TensorFlow model inputs and outputs.We detail them here. This model takes asinputs:modeling_gpt2.pyinput_ids: a torch.LongTensor of shape [batch_size, sequence_length] (or more generally [d_1, ..., d_n, sequence_length] were d_1 ... d_n are arbitrary dimensions) with the word BPE token indices selected in the range [0, vocab_size[position_ids: an optional torch.LongTensor with the same shape as input_ids
with the position indices (selected in the range [0, config.n_positions - 1[.token_type_ids: an optional torch.LongTensor with the same shape as input_ids
You can use it to add a third type of embedding to each input token in the sequence
(the previous two being the word and position embeddings). The input, position and token_type embeddings are summed inside the Transformer before the first self-attention block.past: an optional list of torch.LongTensor that contains pre-computed hidden-states (key and values in the attention blocks) to speed up sequential decoding (this is thepresentsoutput of the model, cf. below).This modeloutputs:hidden_states: the encoded-hidden-states at the top of the model as a torch.FloatTensor of size [batch_size, sequence_length, hidden_size] (or more generally [d_1, ..., d_n, hidden_size] were d_1 ... d_n are the dimension of input_ids)presents: a list of pre-computed hidden-states (key and values in each attention blocks) as a torch.FloatTensors. They can be reused to speed up sequential decoding (see therun_gpt2.pyexample).15.GPT2LMHeadModelGPT2LMHeadModelincludes theGPT2ModelTransformer followed by a language modeling head with weights tied to the input embeddings (no additional parameters).Inputsare the same as the inputs of theGPT2Modelclass plus optional labels:lm_labels: optional language modeling labels: torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [-1, 0, ..., vocab_size]. All labels set to -1 are ignored (masked), the loss is only computed for the labels set in [0, ..., vocab_size].Outputs:iflm_labelsis notNone:
Outputs the language modeling loss.else: a tuple oflm_logits: the language modeling logits as a torch.FloatTensor of size [batch_size, sequence_length, total_tokens_embeddings] (or more generally [d_1, ..., d_n, total_tokens_embeddings] were d_1 ... d_n are the dimension of input_ids)presents: a list of pre-computed hidden-states (key and values in each attention blocks) as a torch.FloatTensors. They can be reused to speed up sequential decoding (see therun_gpt2.pyexample).16.GPT2DoubleHeadsModelGPT2DoubleHeadsModelincludes theGPT2ModelTransformer followed by two heads:a language modeling head with weights tied to the input embeddings (no additional parameters) and:a multiple choice classifier (linear layer that take as input a hidden state in a sequence to compute a score, see details in paper).Inputsare the same as the inputs of theGPT2Modelclass plus a classification mask and two optional labels:multiple_choice_token_ids: a torch.LongTensor of shape [batch_size, num_choices] with the index of the token whose hidden state should be used as input for the multiple choice classifier (usually the [CLS] token for each choice).lm_labels: optional language modeling labels: torch.LongTensor of shape [batch_size, sequence_length] with indices selected in [-1, 0, ..., vocab_size]. All labels set to -1 are ignored (masked), the loss is only computed for the labels set in [0, ..., vocab_size].multiple_choice_labels: optional multiple choice labels: torch.LongTensor of shape [batch_size] with indices selected in [0, ..., num_choices].Outputs:iflm_labelsandmultiple_choice_labelsare notNone:
Outputs a tuple of losses with the language modeling loss and the multiple choice loss.else Outputs a tuple with:lm_logits: the language modeling logits as a torch.FloatTensor of size [batch_size, num_choices, sequence_length, total_tokens_embeddings]multiple_choice_logits: the multiple choice logits as a torch.FloatTensor of size [batch_size, num_choices]presents: a list of pre-computed hidden-states (key and values in each attention blocks) as a torch.FloatTensors. They can be reused to speed up sequential decoding (see therun_gpt2.pyexample).TokenizersBertTokenizerBertTokenizerperform end-to-end tokenization, i.e. basic tokenization followed by WordPiece tokenization.This class has five arguments:vocab_file: path to a vocabulary file.do_lower_case: convert text to lower-case while tokenizing.Default = True.max_len: max length to filter the input of the Transformer. Default to pre-trained value for the model ifNone.Default = Nonedo_basic_tokenize: Do basic tokenization before wordpice tokenization. Set to false if text is pre-tokenized.Default = True.never_split: a list of tokens that should not be splitted during tokenization.Default =["[UNK]", "[SEP]", "[PAD]", "[CLS]", "[MASK]"]and three methods:tokenize(text): convert astrin a list ofstrtokens by (1) performing basic tokenization and (2) WordPiece tokenization.convert_tokens_to_ids(tokens): convert a list ofstrtokens in a list ofintindices in the vocabulary.convert_ids_to_tokens(tokens): convert a list ofintindices in a list ofstrtokens in the vocabulary.save_vocabulary(directory_path): save the vocabulary file todirectory_path. Return the path to the saved vocabulary file:vocab_file_path. The vocabulary can be reloaded withBertTokenizer.from_pretrained('vocab_file_path')orBertTokenizer.from_pretrained('directory_path').Please refer to the doc strings and code intokenization.pyfor the details of theBasicTokenizerandWordpieceTokenizerclasses. In general it is recommended to useBertTokenizerunless you know what you are doing.OpenAIGPTTokenizerOpenAIGPTTokenizerperform Byte-Pair-Encoding (BPE) tokenization.This class has four arguments:vocab_file: path to a vocabulary file.merges_file: path to a file containing the BPE merges.max_len: max length to filter the input of the Transformer. Default to pre-trained value for the model ifNone.Default = Nonespecial_tokens: a list of tokens to add to the vocabulary for fine-tuning. If SpaCy is not installed and BERT'sBasicTokenizeris used as the pre-BPE tokenizer, these tokens are not split.Default= Noneand five methods:tokenize(text): convert astrin a list ofstrtokens by performing BPE tokenization.convert_tokens_to_ids(tokens): convert a list ofstrtokens in a list ofintindices in the vocabulary.convert_ids_to_tokens(tokens): convert a list ofintindices in a list ofstrtokens in the vocabulary.set_special_tokens(self, special_tokens): update the list of special tokens (see above arguments)encode(text): convert astrin a list ofinttokens by performing BPE encoding.decode(ids, skip_special_tokens=False, clean_up_tokenization_spaces=False): decode a list ofintindices in a string and do some post-processing if needed: (i) remove special tokens from the output and (ii) clean up tokenization spaces.save_vocabulary(directory_path): save the vocabulary, merge and special tokens files todirectory_path. Return the path to the three files:vocab_file_path,merge_file_path,special_tokens_file_path. The vocabulary can be reloaded withOpenAIGPTTokenizer.from_pretrained('directory_path').Please refer to the doc strings and code intokenization_openai.pyfor the details of theOpenAIGPTTokenizer.TransfoXLTokenizerTransfoXLTokenizerperform word tokenization. This tokenizer can be used for adaptive softmax and has utilities for counting tokens in a corpus to create a vocabulary ordered by toekn frequency (for adaptive softmax). See the adaptive softmax paper (Efficient softmax approximation for GPUs) for more details.The API is similar to the API ofBertTokenizer(see above).Please refer to the doc strings and code intokenization_transfo_xl.pyfor the details of these additional methods inTransfoXLTokenizer.GPT2TokenizerGPT2Tokenizerperform byte-level Byte-Pair-Encoding (BPE) tokenization.This class has three arguments:vocab_file: path to a vocabulary file.merges_file: path to a file containing the BPE merges.errors: How to handle unicode decoding errors.Default =replaceand two methods:tokenize(text): convert astrin a list ofstrtokens by performing byte-level BPE.convert_tokens_to_ids(tokens): convert a list ofstrtokens in a list ofintindices in the vocabulary.convert_ids_to_tokens(tokens): convert a list ofintindices in a list ofstrtokens in the vocabulary.set_special_tokens(self, special_tokens): update the list of special tokens (see above arguments)encode(text): convert astrin a list ofinttokens by performing byte-level BPE.decode(tokens): convert back a list ofinttokens in astr.save_vocabulary(directory_path): save the vocabulary, merge and special tokens files todirectory_path. Return the path to the three files:vocab_file_path,merge_file_path,special_tokens_file_path. The vocabulary can be reloaded withOpenAIGPTTokenizer.from_pretrained('directory_path').Please refer totokenization_gpt2.pyfor more details on theGPT2Tokenizer.OptimizersBertAdamBertAdamis atorch.optimizeradapted to be closer to the optimizer used in the TensorFlow implementation of Bert. The differences with PyTorch Adam optimizer are the following:BertAdam implements weight decay fix,BertAdam doesn't compensate for bias as in the regular Adam optimizer.The optimizer accepts the following arguments:lr: learning ratewarmup: portion oft_totalfor the warmup,-1means no warmup. Default :-1t_total: total number of training steps for the learning
rate schedule,-1means constant learning rate. Default :-1schedule: schedule to use for the warmup (see above).
Can be'warmup_linear','warmup_constant','warmup_cosine','none',Noneor a_LRScheduleobject (see below).
IfNoneor'none', learning rate is always kept constant.
Default :'warmup_linear'b1: Adams b1. Default :0.9b2: Adams b2. Default :0.999e: Adams epsilon. Default :1e-6weight_decay:Weight decay. Default :0.01max_grad_norm: Maximum norm for the gradients (-1means no clipping). Default :1.0OpenAIAdamOpenAIAdamis similar toBertAdam.
The differences withBertAdamis thatOpenAIAdamcompensate for bias as in the regular Adam optimizer.OpenAIAdamaccepts the same arguments asBertAdam.Learning Rate SchedulesThe.optimizationmodule also provides additional schedules in the form of schedule objects that inherit from_LRSchedule.
All_LRSchedulesubclasses acceptwarmupandt_totalarguments at construction.
When an_LRScheduleobject is passed intoBertAdamorOpenAIAdam,
thewarmupandt_totalarguments on the optimizer are ignored and the ones in the_LRScheduleobject are used.
An overview of the implemented schedules:ConstantLR: always returns learning rate 1.WarmupConstantSchedule: Linearly increases learning rate from 0 to 1 overwarmupfraction of training steps.
Keeps learning rate equal to 1. after warmup.WarmupLinearSchedule: Linearly increases learning rate from 0 to 1 overwarmupfraction of training steps.
Linearly decreases learning rate from 1. to 0. over remaining1 - warmupsteps.WarmupCosineSchedule: Linearly increases learning rate from 0 to 1 overwarmupfraction of training steps.
Decreases learning rate from 1. to 0. over remaining1 - warmupsteps following a cosine curve.
Ifcycles(default=0.5) is different from default, learning rate follows cosine function after warmup.WarmupCosineWithHardRestartsSchedule: Linearly increases learning rate from 0 to 1 overwarmupfraction of training steps.
Ifcycles(default=1.) is different from default, learning rate followscyclestimes a cosine decaying learning rate (with hard restarts).WarmupCosineWithWarmupRestartsSchedule: All training progress is divided incycles(default=1.) parts of equal length.
Every part follows a schedule with the firstwarmupfraction of the training steps linearly increasing from 0. to 1.,
followed by a learning rate decreasing from 1. to 0. following a cosine curve.
Note that the total number of all warmup steps over all cycles together is equal towarmup*cyclesExamplesSub-sectionDescriptionTraining large models: introduction, tools and examplesHow to use gradient-accumulation, multi-gpu training, distributed training, optimize on CPU and 16-bits training to train Bert modelsFine-tuning with BERT: running the examplesRunning the examples in./examples:extract_classif.py,run_classifier.py,run_squad.pyandrun_lm_finetuning.pyFine-tuning with OpenAI GPT, Transformer-XL and GPT-2Running the examples in./examples:run_openai_gpt.py,run_transfo_xl.pyandrun_gpt2.pyFine-tuning BERT-large on GPUsHow to fine tuneBERT largeTraining large models: introduction, tools and examplesBERT-base and BERT-large are respectively 110M and 340M parameters models and it can be difficult to fine-tune them on a single GPU with the recommended batch size for good performance (in most case a batch size of 32).To help with fine-tuning these models, we have included several techniques that you can activate in the fine-tuning scriptsrun_classifier.pyandrun_squad.py: gradient-accumulation, multi-gpu training, distributed training and 16-bits training . For more details on how to use these techniques you can readthe tips on training large batches in PyTorchthat I published earlier this month.Here is how to use these techniques in our scripts:Gradient Accumulation: Gradient accumulation can be used by supplying a integer greater than 1 to the--gradient_accumulation_stepsargument. The batch at each step will be divided by this integer and gradient will be accumulated overgradient_accumulation_stepssteps.Multi-GPU: Multi-GPU is automatically activated when several GPUs are detected and the batches are splitted over the GPUs.Distributed training: Distributed training can be activated by supplying an integer greater or equal to 0 to the--local_rankargument (see below).16-bits training: 16-bits training, also called mixed-precision training, can reduce the memory requirement of your model on the GPU by using half-precision training, basically allowing to double the batch size. If you have a recent GPU (starting from NVIDIA Volta architecture) you should see no decrease in speed. A good introduction to Mixed precision training can be foundhereand a full documentation ishere. In our scripts, this option can be activated by setting the--fp16flag and you can play with loss scaling using the--loss_scaleflag (see the previously linked documentation for details on loss scaling). The loss scale can be zero in which case the scale is dynamically adjusted or a positive power of two in which case the scaling is static.To use 16-bits training and distributed training, you need to install NVIDIA's apex extensionas detailed here. You will find more information regarding the internals ofapexand how to useapexinthe doc and the associated repository. The results of the tests performed on pytorch-BERT by the NVIDIA team (and my trials at reproducing them) can be consulted inthe relevant PR of the present repository.Note: To useDistributed Training, you will need to run one training script on each of your machines. This can be done for example by running the following command on each server (seethe above mentioned blog postfor more details):python-mtorch.distributed.launch--nproc_per_node=4--nnodes=2--node_rank=$THIS_MACHINE_INDEX--master_addr="192.168.1.1"--master_port=1234run_classifier.py(--arg1--arg2--arg3andallotherargumentsoftherun_classifierscript)Where$THIS_MACHINE_INDEXis an sequential index assigned to each of your machine (0, 1, 2...) and the machine with rank 0 has an IP address192.168.1.1and an open port1234.Fine-tuning with BERT: running the examplesWe showcase several fine-tuning examples based on (and extended from)the original implementation:asequence-level classifieron nine different GLUE tasks,atoken-level classifieron the question answering dataset SQuAD, andasequence-level multiple-choice classifieron the SWAG classification corpus.aBERT language modelon another target corpusGLUE results on dev setWe get the following results on the dev set of GLUE benchmark with an uncased BERT base
model. All experiments were run on a P100 GPU with a batch size of 32.TaskMetricResultCoLAMatthew's corr.57.29SST-2accuracy93.00MRPCF1/accuracy88.85/83.82STS-BPearson/Spearman corr.89.70/89.37QQPaccuracy/F190.72/87.41MNLImatched acc./mismatched acc.83.95/84.39QNLIaccuracy89.04RTEaccuracy61.01WNLIaccuracy53.52Some of these results are significantly different from the ones reported on the test set
of GLUE benchmark on the website. For QQP and WNLI, please refer toFAQ #12on the webite.Before running anyone of these GLUE tasks you should download theGLUE databy runningthis scriptand unpack it to some directory$GLUE_DIR.exportGLUE_DIR=/path/to/glueexportTASK_NAME=MRPC

pythonrun_classifier.py\--task_name$TASK_NAME\--do_train\--do_eval\--do_lower_case\--data_dir$GLUE_DIR/$TASK_NAME\--bert_modelbert-base-uncased\--max_seq_length128\--train_batch_size32\--learning_rate2e-5\--num_train_epochs3.0\--output_dir/tmp/$TASK_NAME/where task name can be one of CoLA, SST-2, MRPC, STS-B, QQP, MNLI, QNLI, RTE, WNLI.The dev set results will be present within the text file 'eval_results.txt' in the specified output_dir. In case of MNLI, since there are two separate dev sets, matched and mismatched, there will be a separate output folder called '/tmp/MNLI-MM/' in addition to '/tmp/MNLI/'.The code has not been tested with half-precision training with apex on any GLUE task apart from MRPC, MNLI, CoLA, SST-2. The following section provides details on how to run half-precision training with MRPC. With that being said, there shouldn't be any issues in running half-precision training with the remaining GLUE tasks as well, since the data processor for each task inherits from the base class DataProcessor.MRPCThis example code fine-tunes BERT on the Microsoft Research Paraphrase
Corpus (MRPC) corpus and runs in less than 10 minutes on a single K-80 and in 27 seconds (!) on single tesla V100 16GB with apex installed.Before running this example you should download theGLUE databy runningthis scriptand unpack it to some directory$GLUE_DIR.exportGLUE_DIR=/path/to/glue

pythonrun_classifier.py\--task_nameMRPC\--do_train\--do_eval\--do_lower_case\--data_dir$GLUE_DIR/MRPC/\--bert_modelbert-base-uncased\--max_seq_length128\--train_batch_size32\--learning_rate2e-5\--num_train_epochs3.0\--output_dir/tmp/mrpc_output/Our test ran on a few seeds withthe original implementation hyper-parametersgave evaluation results between 84% and 88%.Fast run with apex and 16 bit precision: fine-tuning on MRPC in 27 seconds!First install apex as indicatedhere.
Then runexportGLUE_DIR=/path/to/glue

pythonrun_classifier.py\--task_nameMRPC\--do_train\--do_eval\--do_lower_case\--data_dir$GLUE_DIR/MRPC/\--bert_modelbert-base-uncased\--max_seq_length128\--train_batch_size32\--learning_rate2e-5\--num_train_epochs3.0\--output_dir/tmp/mrpc_output/\--fp16SQuADThis example code fine-tunes BERT on the SQuAD dataset. It runs in 24 min (with BERT-base) or 68 min (with BERT-large) on a single tesla V100 16GB.The data for SQuAD can be downloaded with the following links and should be saved in a$SQUAD_DIRdirectory.train-v1.1.jsondev-v1.1.jsonevaluate-v1.1.pyexportSQUAD_DIR=/path/to/SQUAD

pythonrun_squad.py\--bert_modelbert-base-uncased\--do_train\--do_predict\--do_lower_case\--train_file$SQUAD_DIR/train-v1.1.json\--predict_file$SQUAD_DIR/dev-v1.1.json\--train_batch_size12\--learning_rate3e-5\--num_train_epochs2.0\--max_seq_length384\--doc_stride128\--output_dir/tmp/debug_squad/Training with the previous hyper-parameters gave us the following results:{"f1":88.52381567990474,"exact_match":81.22043519394512}SWAGThe data for SWAG can be downloaded by cloning the followingrepositoryexportSWAG_DIR=/path/to/SWAG

pythonrun_swag.py\--bert_modelbert-base-uncased\--do_train\--do_lower_case\--do_eval\--data_dir$SWAG_DIR/data\--train_batch_size16\--learning_rate2e-5\--num_train_epochs3.0\--max_seq_length80\--output_dir/tmp/swag_output/\--gradient_accumulation_steps4Training with the previous hyper-parameters on a single GPU gave us the following results:eval_accuracy = 0.8062081375587323
eval_loss = 0.5966546792367169
global_step = 13788
loss = 0.06423990014260186LM Fine-tuningThe data should be a text file in the same format assample_text.txt(one sentence per line, docs separated by empty line).
You can download anexemplary training corpusgenerated from wikipedia articles and splitted into ~500k sentences with spaCy.
Training one epoch on this corpus takes about 1:20h on 4 x NVIDIA Tesla P100 withtrain_batch_size=200andmax_seq_length=128:Thank to the work of @Rocketknight1 and @tholor there are nowseveral scriptsthat can be used to fine-tune BERT using the pretraining objective (combination of masked-language modeling and next sentence prediction loss). These scripts are detailed in theREADMEof theexamples/lm_finetuning/folder.OpenAI GPT, Transformer-XL and GPT-2: running the examplesWe provide three examples of scripts for OpenAI GPT, Transformer-XL and OpenAI GPT-2 based on (and extended from) the respective original implementations:fine-tuning OpenAI GPT on the ROCStories datasetevaluating Transformer-XL on Wikitext 103unconditional and conditional generation from a pre-trained OpenAI GPT-2 modelFine-tuning OpenAI GPT on the RocStories datasetThis example code fine-tunes OpenAI GPT on the RocStories dataset.Before running this example you should download theRocStories datasetand unpack it to some directory$ROC_STORIES_DIR.exportROC_STORIES_DIR=/path/to/RocStories

pythonrun_openai_gpt.py\--model_nameopenai-gpt\--do_train\--do_eval\--train_dataset$ROC_STORIES_DIR/cloze_test_val__spring2016\-\cloze_test_ALL_val.csv\--eval_dataset$ROC_STORIES_DIR/cloze_test_test__spring2016\-\cloze_test_ALL_test.csv\--output_dir../log\--train_batch_size16\This command runs in about 10 min on a single K-80 an gives an evaluation accuracy of about 87.7% (the authors report a median accuracy with the TensorFlow code of 85.8% and the OpenAI GPT paper reports a best single run accuracy of 86.5%).Evaluating the pre-trained Transformer-XL on the WikiText 103 datasetThis example code evaluate the pre-trained Transformer-XL on the WikiText 103 dataset.
This command will download a pre-processed version of the WikiText 103 dataset in which the vocabulary has been computed.pythonrun_transfo_xl.py--work_dir../logThis command runs in about 1 min on a V100 and gives an evaluation perplexity of 18.22 on WikiText-103 (the authors report a perplexity of about 18.3 on this dataset with the TensorFlow code).Unconditional and conditional generation from OpenAI's GPT-2 modelThis example code is identical to the original unconditional and conditional generation codes.Conditional generation:pythonrun_gpt2.pyUnconditional generation:pythonrun_gpt2.py--unconditionalThe same option as in the original scripts are provided, please refere to the code of the example and the original repository of OpenAI.Fine-tuning BERT-large on GPUsThe options we list above allow to fine-tune BERT-large rather easily on GPU(s) instead of the TPU used by the original implementation.For example, fine-tuning BERT-large on SQuAD can be done on a server with 4 k-80 (these are pretty old now) in 18 hours. Our results are similar to the TensorFlow implementation results (actually slightly higher):{"exact_match":84.56953642384106,"f1":91.04028647786927}To get these results we used a combination of:multi-GPU training (automatically activated on a multi-GPU server),2 steps of gradient accumulation andperform the optimization step on CPU to store Adam's averages in RAM.Here is the full list of hyper-parameters for this run:exportSQUAD_DIR=/path/to/SQUAD

python./run_squad.py\--bert_modelbert-large-uncased\--do_train\--do_predict\--do_lower_case\--train_file$SQUAD_DIR/train-v1.1.json\--predict_file$SQUAD_DIR/dev-v1.1.json\--learning_rate3e-5\--num_train_epochs2\--max_seq_length384\--doc_stride128\--output_dir/tmp/debug_squad/\--train_batch_size24\--gradient_accumulation_steps2If you have a recent GPU (starting from NVIDIA Volta series), you should try16-bit fine-tuning(FP16).Here is an example of hyper-parameters for a FP16 run we tried:exportSQUAD_DIR=/path/to/SQUAD

python./run_squad.py\--bert_modelbert-large-uncased\--do_train\--do_predict\--do_lower_case\--train_file$SQUAD_DIR/train-v1.1.json\--predict_file$SQUAD_DIR/dev-v1.1.json\--learning_rate3e-5\--num_train_epochs2\--max_seq_length384\--doc_stride128\--output_dir/tmp/debug_squad/\--train_batch_size24\--fp16\--loss_scale128The results were similar to the above FP32 results (actually slightly higher):{"exact_match":84.65468306527909,"f1":91.238669287002}NotebooksWe includethree Jupyter Notebooksthat can be used to check that the predictions of the PyTorch model are identical to the predictions of the original TensorFlow model.The first NoteBook (Comparing-TF-and-PT-models.ipynb) extracts the hidden states of a full sequence on each layers of the TensorFlow and the PyTorch models and computes the standard deviation between them. In the given example, we get a standard deviation of 1.5e-7 to 9e-7 on the various hidden state of the models.The second NoteBook (Comparing-TF-and-PT-models-SQuAD.ipynb) compares the loss computed by the TensorFlow and the PyTorch models for identical initialization of the fine-tuning layer of theBertForQuestionAnsweringand computes the standard deviation between them. In the given example, we get a standard deviation of 2.5e-7 between the models.The third NoteBook (Comparing-TF-and-PT-models-MLM-NSP.ipynb) compares the predictions computed by the TensorFlow and the PyTorch models for masked token language modeling using the pre-trained masked language modeling model.Please follow the instructions given in the notebooks to run and modify them.Command-line interfaceA command-line interface is provided to convert a TensorFlow checkpoint in a PyTorch dump of theBertForPreTrainingclass  (for BERT) or NumPy checkpoint in a PyTorch dump of theOpenAIGPTModelclass  (for OpenAI GPT).BERTYou can convert any TensorFlow checkpoint for BERT (in particularthe pre-trained models released by Google) in a PyTorch save file by using theconvert_tf_checkpoint_to_pytorch.pyscript.This CLI takes as input a TensorFlow checkpoint (three files starting withbert_model.ckpt) and the associated configuration file (bert_config.json), and creates a PyTorch model for this configuration, loads the weights from the TensorFlow checkpoint in the PyTorch model and saves the resulting model in a standard PyTorch save file that can be imported usingtorch.load()(see examples inextract_features.py,run_classifier.pyandrun_squad.py).You only need to run this conversion scriptonceto get a PyTorch model. You can then disregard the TensorFlow checkpoint (the three files starting withbert_model.ckpt) but be sure to keep the configuration file (bert_config.json) and the vocabulary file (vocab.txt) as these are needed for the PyTorch model too.To run this specific conversion script you will need to have TensorFlow and PyTorch installed (pip install tensorflow). The rest of the repository only requires PyTorch.Here is an example of the conversion process for a pre-trainedBERT-Base Uncasedmodel:exportBERT_BASE_DIR=/path/to/bert/uncased_L-12_H-768_A-12

pytorch_pretrained_bertconvert_tf_checkpoint_to_pytorch\$BERT_BASE_DIR/bert_model.ckpt\$BERT_BASE_DIR/bert_config.json\$BERT_BASE_DIR/pytorch_model.binYou can download Google's pre-trained models for the conversionhere.OpenAI GPTHere is an example of the conversion process for a pre-trained OpenAI GPT model, assuming that your NumPy checkpoint save as the same format than OpenAI pretrained model (seehere)exportOPENAI_GPT_CHECKPOINT_FOLDER_PATH=/path/to/openai/pretrained/numpy/weights

pytorch_pretrained_bertconvert_openai_checkpoint\$OPENAI_GPT_CHECKPOINT_FOLDER_PATH\$PYTORCH_DUMP_OUTPUT\[OPENAI_GPT_CONFIG]Transformer-XLHere is an example of the conversion process for a pre-trained Transformer-XL model (seehere)exportTRANSFO_XL_CHECKPOINT_FOLDER_PATH=/path/to/transfo/xl/checkpoint

pytorch_pretrained_bertconvert_transfo_xl_checkpoint\$TRANSFO_XL_CHECKPOINT_FOLDER_PATH\$PYTORCH_DUMP_OUTPUT\[TRANSFO_XL_CONFIG]GPT-2Here is an example of the conversion process for a pre-trained OpenAI's GPT-2 model.exportGPT2_DIR=/path/to/gpt2/checkpoint

pytorch_pretrained_bertconvert_gpt2_checkpoint\$GPT2_DIR/model.ckpt\$PYTORCH_DUMP_OUTPUT\[GPT2_CONFIG]TPUTPU support and pretraining scriptsTPU are not supported by the current stable release of PyTorch (0.4.1). However, the next version of PyTorch (v1.0) should support training on TPU and is expected to be released soon (see the recentofficial announcement).We will add TPU support when this next release is published.The original TensorFlow code further comprises two scripts for pre-training BERT:create_pretraining_data.pyandrun_pretraining.py.Since, pre-training BERT is a particularly expensive operation that basically requires one or several TPUs to be completed in a reasonable amout of time (see detailshere) we have decided to wait for the inclusion of TPU support in PyTorch to convert these pre-training scripts.