Customizing GenerationMixin to output attentions

Question:
Hi all,
I’m using a Pegasus model (or really BartForConditionalGeneration since almost everything is inherited) and I’m interested in the attention outputs of various encoder and decoder blocks throughout the model.
Following the documentation, simply tokenizing an input context and running
model(**input_tokens, output_attentions = True)
allows me to dissect the attentions of each token in the input sequence in every layer (the dimensions being (batch_size, num_heads, seq_length, seq_length) for each layer). This is good.
Now I want to see the attentions that lead to the predictions of each token returned from model.generate(). Since I am on master and build from source, I edit the method _generate_beam_search in the GenerationMixin class in transformers/generation_utils.py to also pass the arg output_attentions = True. Here is the edited snippet:
decoder_attentions = [] # added this line

        while cur_len < max_length:
            model_inputs = self.prepare_inputs_for_generation(
                input_ids, past=past, attention_mask=attention_mask, use_cache=use_cache, **model_kwargs
            )
            outputs = self(**model_inputs, return_dict=True, output_attentions = True)  # edited this line
            decoder_attentions.append(outputs['decoder_attentions']) # added this line
            next_token_logits = outputs.logits[:, -1, :]  # (batch_size * num_beams, vocab_size)

As you can see, I try to get the decoder attentions for every iteration, append it, and later I pass up the list and return it from the main generate() method.
The problem is these attentions are no longer of the shape (batch_size, num_heads, seq_length, seq_length). Specifically, I would expect the sequence lengths to be at least the input context sequence length, but they are much shorter, and do not even match the length of the final sequence prediction.
I feel like I have some misunderstanding about how things are working. I know encoder_outputs and some hidden states are precomputed, but I don’t know if they are affecting this.
Can anyone help me understand what is going on here? Is there a way to see the influence of specific input context tokens in decoder block attention heads on predicted tokens ?

Context:
I think this just returning the attentions from the deocder since the encoder hidden states are only computed once in the generate function. So the returned attentions at every iteration is attention over the tokens generated so far.
pinging @sshleifer


Question:
Hello!
I am trying to find resources/code samples to retrain the DistilGPT2 model with text I have preprocessed myself, but could not find any. Most of the documentation relates to DistilBert and it’s uses.
Furthermore, I also have trained a gpt2-simple (tensorflow based) model. If there is any way to distil the same, it will help me too!
Thanks for your help.

Context:
Hi @abhilashpal, you can find distillation code here 27. The same script that produces distillbert can be used for GPT-2, it’s not documented though.
you should be able to use this command after processing your dataset
python train.py \
    --student_type gpt2 \
    --student_config training_configs/distilgpt2.json \
    --teacher_type gpt2 \
    --teacher_name gpt2 \ # or your own teacher model
    --alpha_ce 5.0  --alpha_cos 1.0 --alpha_clm 0.5 \
    --freeze_pos_embs \
    --dump_path serialization_dir/my_first_training \
    --data_file data/binarized_text.bert-base-uncased.pickle \ # your data path
    --token_counts data/token_counts.bert-base-uncased.pickle \ # your own pickle file path
    --force # overwrites the `dump_path` if it already exists.

pinging @julien-c for more info


Question:
Hello,
Sorry if I duplicate the question. I made some brief search in the forum, but did not really found the
answer.
So it was decided to make some fine-tuning of longformer using dataset which consists of 3000 pairs. Length of each input is up to 4096 tokens. After some simple computations understood that it is needed around 24Gb HBM on GPU to run BS=1. I do not have such GPU and I looked on my old 2-socket 20-core Xeon with 64gb of ram.
I installed pytorch optimized by mkldnn for Intel processors… and you know after running I realized that fine-tuning on 3000 pairs will take around 100 hours. 100 hours, Carl! Either this Xeon is too old (only AVX supported) or mkl-dnn does not optimize bert-like pytorch models.
Anyway I’m looking into renting some GPU server. And finally I’m coming to my questions.
Assuming that I need 24gb of memory for 1 batch, then can I take server with 2 GPU with 16 gb each? Do you know if pytorch + cuda can split into 2 GPUs even for batch size = 1 w/o degradation?
Or I need to look for single Nvidia V100 with 32gb of HBM to solve this problem?
Anybody tried already longformer and can share some performance results with details of used HW?
Thanks!!!

Context: I faced the same problem. As of now, I used a 32 GB GPU (p3dn.24xlarge ec2 instance). Also reduced the number of tokens and the batch size. At present Longformer doesn’t support multiple GPUs. We can shard the training data and train it iteratively by saving the model (still exploring this). We can explore checkpoint feature in pytorch as well (https://pytorch.org/tutorials/recipes/recipes/saving_and_loading_a_general_checkpoint.html 5). Let me know if you have already found an efficient way to do this.


Question:
Please make me clear difference between checkpoint and saving the weights of the model,
which one can I use to load later?
Also I could not find my checkpoints (may be overwrite option at my end), so the same can done via these line of code

trainer.save_model("/content/drive//results/distillbert/trainer")
tokenizer.save_pretrained("/content/drive/results/distillbert/tokenizer")

Context:
I think a “checkpoint” is what we call a partial save during training.
To take a checkpoint during training, you can save the model’s state_dict, which is a list of the current values of all the parameters that have been updated during this training run.
Note that this doesn’t save the non-variable parameters, and it doesn’t save the weights in any frozen layers.
To reload the model to that checkpoint state, you first of all have to load a complete model with the right configuration.  You can do this either by initializing randomly with the config file, or by loading a suitable pre-trained model.   Then you update that complete model with the saved state_dict weights.
If you want to continue the training from the same point, you also need information about the scheduler and the optimizer.  This can be saved and applied using the optimizer’s state_dict.
I haven’t any examples of using save_model or save_pretrained, but here’s an example of saving a model and optimizer during training.

filedt = datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
torch.save(model.state_dict(),’/content/drive/My Drive/ftregmod-’ + filedt)
torch.save(optimizer.state_dict(),’/content/drive/My Drive/ftregopt-’ + filedt)

and then to reload and continue training:

READFROMNAMEMODEL = ‘/content/drive/My Drive/ftregmod-20200911-014657’   ####
READFROMNAMEOPT = ‘/content/drive/My Drive/ftregopt-20200911-014657’   ####

model = BertForSequenceClassification.from_pretrained(‘bert-base-uncased’,
num_labels=NCLASSES,
output_attentions=True)

model.load_state_dict(torch.load(READFROMNAMEMODEL), strict=False)

optimizer = AdamW(model.parameters(),
lr = LEARNRATE, #  default is 5e-5
eps = 1e-8 #  default is 1e-8.
)

optimizer.load_state_dict(torch.load(READFROMNAMEOPT))


Dataset for fake news detection, fine tune or pre-train

Question:
Is there any dataset for fake news (different from sentiment analysis) detection? I have one NELA-GT but then I would need to pre-train that from scratch?
Any methods, am I on the correct page ? https://huggingface.co/transformers/training.html 6
I want to use BERT model,
thanks

Context:
You could try to get baseline with fine-tuning before going for pre-training and then make decision based on the results.
This thread has nice pointer for pre-training

    Pre-Train BERT (from scratch) Research

    BERT has been trained on MLM and NSP objective. I wanted to train BERT with/without NSP objective (with NSP in case suggested approach is different). I haven’t performed pre-training in full sense before. Can you please share how to obtain the data (crawl and tokenization details which were used) on which BERT was trained on ?. Since it takes a lot of time, I am looking for well tested code that can yield the BERT with/without NSP in one go. Any suggestions will be helpful.
I know about some pr…


Question:
Lets say I have mix of words and numbers,which represent for example prices or sizes of objects. The relation/magnitude between the numbers is very important. With normal tokenization it would split those large/less common numbers into smaller pieces (few digits) and concatenate them. Is this not very inefficient and suboptimal, since the transformer has to learn how digits/numbers work and would assign  every digit/combination its own learned vector. Would it make sense to manually convert these numbers and pass in its own vector?
Example:
DATA (Laying carpet 150sqft | price 400$)-> LABEL (okay)
DATA (Carpet type 133 installation with an area of 70m2  | price 8000$)-> LABEL ( Not okay)

Context:
If your task is closely related to those numbers, and all examples follow the same pattern, perhaps you should just use the price and surface as raw features in your model, along with the text. You could concatenate the price and surface values to the Transformer embedding for the text at the last hidden-layer before classification. I am not sure if that is what you meant by

 bone:

Would it make sense to manually convert these numbers and pass in its own vector?

Maybe also normalize the surface values to one standard.
I don’t know if there are better ways to deal with values at tokenizing, but this would be my suggestion.


Question: What is the difference between [PYTORCH-TRANSFORMERS](https://pytorch.org/hub/huggingface_pytorch-transformers/) and transformers, when running RobertA on PYTORCH-TRANSFORMERS, it worked fine. please guide.

Context: pytorch_transformers was older name/version. It later got renamed to transformers after adding support for TF. Also Roberta works in transformers. What’s not working for you ?


How to use transformer attention model when the input is features

Question:
I am totally new to this NLP and transformer and attention.
I was playing with models sentence-transformers and want to explore more, but now I got stuck.
I have an input of BxKx768 which is my embedded features.
Is there a way to give them to a transformer(which has attention model ) and get the output of size BxM where M can be any number?
I learned to do it when input is a sentence, but have no idea how to do it when input is features.
I guess what I am asking is how to give my input to a transform model and have my output.
Apologies in advance if it is a bad question.
If that is easy, are there different models to try, like e.g. in resnet we have resnet 18,50,etc, do have the same thing here?

Context:
The models in the HF library are focused on NLP, hence they have extra stuff related to language, such as an embeddings layer, positional and token types information as well as other model specific features.
If you want to build your own model using Transformer layers then perhaps you should look at https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html 4.
Regarding the last question, resnet18/50… are pre-trained models with different model sizes, see this  post. With Transformers you can also change the size of the network by specifying different number of attention heads or layers. For instance bert-base model has 12-layers in depth and each has 12 attention heads, while bert-large is 24 layers deep and uses 16 attention heads, that’s why the embeddings they produce are different in size, 768 vs 1024, which has to do with the number of attention heads in this case. If you want to understand more I recommend this post. So I think the parallelism would be bert-base being a Resnet-18 and bert-large a Resnet-50.
You could use pretrained Language Modelling models listed here for language related tasks as you would with pre-trained Resnets for computer vision tasks, although there are more task-specific trained models you can explore here.
So as you can see there’s a lot happening, so welcome to the NLP world


Predictions for sequenceclassification task

Question:
I am working on SequenceClassification task and dont know how to see the predictions
here is my code

from transformers import DistilBertTokenizer, TFDistilBertForMaskedLM
import tensorflow as tf
import torch
loaded_model = DistilBertForSequenceClassification.from_pretrained("/content/results/distillbert/model1",return_dict=True)
tokenizer = DistilBertTokenizer.from_pretrained(‘distilbert-base-cased’)
inputs = tokenizer(“Hello, this news is Not good for learning”, return_tensors=“pt”)
labels = torch.tensor([1]).unsqueeze(0)  # Batch size 1
outputs = loaded_model(**inputs, labels=labels)
loss = outputs.loss
logits = outputs.logits

Context:
Checkout this docs 5.
First, you won’t need to pass labels for predictions and for prediction put the code under torch.no_grad() to avoid calculating gradients.
The returned logits have shape [batch_size, num_classes], you can then apply argmax to get the index highest scored class.
classes = ["NEGATIVE", "POSITIVE"]
enc = tokenizer("positive text", return_tensors="pt")
with torch.no_grad():
  logits = model(**enc, return_dict=True).logits

pred_class_idx = torch.argmax(logits).item()
classes[pred_class_idx]

you can also apply softmax on logits to get the probabilities.
and better yet, just use pipeline as shown in the docs above, which does all of this for you.


Is MBart tensorflow2.0 model ready for use?

Question: I am looking for MBart pretrained model, but it seems like there is only a pytorch version. I am wondering whether there is a tf2.0 version for this model. If not, can I convert this pytorch version to tf2.0 version?

Context:
Not at the moment, MBart inherits from Bart and it’s only available with torch.
pinging @sshleifer


Question:
Hi
I am new to the usage of huggingface, before that I used BERT but using original code and just used sequence classification for sentiment analysis, now I am puzzled that Which task is more suitable for the purpose of detection tasks e.g., fake news
(i) MaskedLM
(ii) Sequence classification
thanks

Context:
Hi @shainaraza
Fake news detection is classification, so  Sequence classification is suitable.
masked language modelling (MLM) is used for pre-training and is not meant to be used for direct downstream tasks. For downstream tasks we take pre-trained model and add a task specific head on top of it.
These two docs should help

Summary of the models 5
Summary of the tasks 8


Training a regression model using Roberta (SMILES to CCS) Cheminformatics

Question:
Using SMILES string to predict a float
I’ve been learning how to use this library over the past few weeks and getting stuck into it. I don’t have a lot of experience with NNs but I have some understanding. I want to use Roberta to build a regression model which would predict the CCS (collisional cross section) area of a molecule given it’s formula in a SMILES string (which is a string representation of the molecule).
I’ve run into an index out of range in self within the model and I’m wondering if anyone has any suggestions. I’m assuming it’s a problem with the forward method in my model or my Dataset. I am currently still unable to train the model at all.
Any suggestions would be amazing, Thank you.
a smiles string looks like this:
COC1=CC=CC=C1CNCCC2=CC(=C(C=C2OC)Cl)OC
The error
raceback (most recent call last):
  File "c:/Users/ktzd064/Documents/Python/CCS_Prediction/CCSround2.py", line 165, in <module>
    trainer.train()
  File "C:\ProgramData\Anaconda3\envs\tf-transformer\lib\site-packages\transformers\trainer.py", line 707, in train
    tr_loss += self.training_step(model, inputs)
  File "C:\ProgramData\Anaconda3\envs\tf-transformer\lib\site-packages\transformers\trainer.py", line 994, in training_step
    outputs = model(**inputs)
  File "C:\ProgramData\Anaconda3\envs\tf-transformer\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "c:/Users/ktzd064/Documents/Python/CCS_Prediction/CCSround2.py", line 21, in forward
    out, _ = self.bert(input_ids, token_type_ids, attention_mask)
  File "C:\ProgramData\Anaconda3\envs\tf-transformer\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\tf-transformer\lib\site-packages\transformers\modeling_roberta.py", line 479, in forward
    return_dict=return_dict,
  File "C:\ProgramData\Anaconda3\envs\tf-transformer\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\tf-transformer\lib\site-packages\transformers\modeling_bert.py", line 825, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "C:\ProgramData\Anaconda3\envs\tf-transformer\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\tf-transformer\lib\site-packages\transformers\modeling_roberta.py", line 82, in forward
    input_ids, token_type_ids=token_type_ids, position_ids=position_ids, inputs_embeds=inputs_embeds
  File "C:\ProgramData\Anaconda3\envs\tf-transformer\lib\site-packages\transformers\modeling_bert.py", line 209, in forward
    token_type_embeddings = self.token_type_embeddings(token_type_ids)
  File "C:\ProgramData\Anaconda3\envs\tf-transformer\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\tf-transformer\lib\site-packages\torch\nn\modules\sparse.py", line 126, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "C:\ProgramData\Anaconda3\envs\tf-transformer\lib\site-packages\torch\nn\functional.py", line 1814, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

The model
class regression_model(nn.Module):
    def __init__(self):
        super(regression_model, self).__init__()
        self.bert = RobertaForSequenceClassification(config=config)
        self.drop = nn.Dropout(p=0.3)
        self.out = nn.Linear(self.bert.config.hidden_size, 1)

    def forward(self, input_ids, attention_mask, targets, token_type_ids,):
        out, _ = self.bert(input_ids, token_type_ids, attention_mask)
        out = self.dropout(out)
        loss = nn.MSELoss()
        output = loss(out, targets)
        return output

The Dataset
class smiles_dataset(Dataset):
    def __init__(self, smiles, targets, tokenizer, max_len):
        self.smiles = smiles
        self.targets = targets
        self.tokenizer = tokenizer
        self.max_len = max_len

    def __len__(self):
        return len(self.smiles)

    def __getitem__(self, index):
        smiles = str(self.smiles[index])
        targets = float(self.targets[index])

        encoding = self.tokenizer.encode(
            smiles,
        )

        return {
            'SMILES': smiles,
            'input_ids': encoding.ids,
            'attention_mask': encoding.attention_mask,
            'targets': torch.tensor(targets, dtype=torch.float32),
            'token_type_ids': encoding.type_ids,
        }

Tokenizer
I have trained the tokenizer on the SMILES dataset I am using
tokenizer = ByteLevelBPETokenizer()
tokenizer.train('SMILES.txt', vocab_size=800, min_frequency=1, special_tokens=["<s>",
    "<PAD>",
    "<MASK>",])

tokenizer.save_model("CCSround2")

from tokenizers.implementations import ByteLevelBPETokenizer
from tokenizers.processors import BertProcessing

tokenizer = ByteLevelBPETokenizer(
    "./CCSround2/vocab.json",
    "./CCSround2/merges.txt",
)

tokenizer._tokenizer.post_processor = BertProcessing(
    ("<PAD>", tokenizer.token_to_id("<PAD>")),
    ("<MASK>", tokenizer.token_to_id("<MASK>")),
)

tokenizer.enable_padding(length=300)
tokenizer.save

The training config
train, test = train_test_split(SMILESandCCS, test_size=0.2)

train_dataset = smiles_dataset(train['SMILES'].values, train['CCS'].values, tokenizer, 300)

test_dataset = smiles_dataset(test['SMILES'].values, test['CCS'].values, tokenizer, 300)

from transformers import RobertaConfig, RobertaTokenizerFast, RobertaForSequenceClassification

config = RobertaConfig(
    vocab_size=800,
    max_positional_embeddings=224,
    num_attention_heads=12,
    num_hidden_layers=6,
    type_vocab_size=1,
)

tokenizer = RobertaTokenizerFast.from_pretrained('./CCSround2', max_len=300)

from transformers import DataCollatorForLanguageModeling

model = regression_model()

training_args = TrainingArguments(
     output_dir = '/results',
     num_train_epochs = 3,
     per_device_train_batch_size=16,
     per_device_eval_batch_size=64,
     weight_decay=0.01,
     logging_dir='/logs',
)

trainer = Trainer(
    model = model,
    args = training_args,
    train_dataset = train_dataset,
)

trainer.train()

Context: Hi, are you sure your ByteLevelBPETokenizer that you trained and saved is compatible with the RobertaTokenizerFast  that you are loading it into?


Failed to train bart-cnn from bart-base using my own code

Question:
Update: problem solved. caused by a bug in model saving function
There’re not much difference on Rouge score before and after training.
The generated summarization of same article is almost same.
Need some suggestion.
I used original setting in fairseq 2.

label_smoothed_cross_entropy,

label-smoothing 0.1,

attention-dropout 0.1,

weight-decay 0.01,

lr-scheduler polynomial_decay,

adam-betas “(0.9, 0.999)”

adam-eps   1e-08,

LR=3e-05

some difference:

batch-size 3,
gradient_accumulation_steps 16,
adamW,
fp32,
clip-norm 1.0,

trained for 7 hours on 2 tesla P4.
Loss (my fault made it hard to read):
image1475×463 36.5 KB
Model seems not converge .
Rouge score before training:

“rouge1”: 37.325248080026086,
“rouge2”: 18.03751262341448,
“rougeL”: 25.82438158757282

Rouge score after training:

“rouge1”: 37.818,
“rouge2”: 17.246,
“rougeL”: 23.7038

Update: problem solved. caused by a bug in model saving function

Context: AFAIK there isn’t bart-base checkpoint trained for summarization, how did you get those metrics without training ?


Facebook/bart-large-cnn has a low rouge score on cnn_dailymail

Question:
I test on test split in the tensorflow_dataset 4. Use python library rouge to output rouge score.
The score is quite low, compared to the score reported in the paper 17

{‘rouge-1’: {‘f’: 0.38628074837405213,
‘p’: 0.38253581551915466,
‘r’: 0.4136606028772784},
‘rouge-2’: {‘f’: 0.1810805831229415,
‘p’: 0.17948749808930747,
‘r’: 0.193921872080545},
‘rouge-l’: {‘f’: 0.3747852342130126,
‘p’: 0.37128779953880464,
‘r’: 0.3958861147871471}}

the score reported in the paper:
BART 44.16 21.28 40.90 (R1, R2, RL)
parameter of generate function:

num_beams=4,
length_penalty=2.0,
max_length=256,
min_length=10,
no_repeat_ngram_size=3

Context:
Hi @LiuYangyang,
can you post the parameters you used for the generate  function. (assuming you used generate)


Question:
Hi I try to reproduce the result related to BART and the result is not comparable to the claimed performance. I tried sshleifer/distilbart-cnn-12-6 and facebook/bart-large-cnn and met the same problem.
My generation process is modified based on the released summarization pipeline.
python run_eval.py sshleifer/distilbart-cnn-12-6 $DATA_DIR/test.source $OUTPUT_FILE \
    --reference_path $DATA_DIR/test.target \
    --task summarization \
    --device cuda \
    --fp16 \
    --bs 32

My performance without post-processing:
1 ROUGE-1 Average_R: 0.48286 (95%-conf.int. 0.48036 - 0.48554)
1 ROUGE-1 Average_P: 0.33581 (95%-conf.int. 0.33356 - 0.33802)
1 ROUGE-1 Average_F: 0.38536 (95%-conf.int. 0.38338 - 0.38737)
---------------------------------------------
1 ROUGE-2 Average_R: 0.20405 (95%-conf.int. 0.20148 - 0.20648)
1 ROUGE-2 Average_P: 0.14260 (95%-conf.int. 0.14067 - 0.14449)
1 ROUGE-2 Average_F: 0.16314 (95%-conf.int. 0.16108 - 0.16517)
---------------------------------------------
1 ROUGE-L Average_R: 0.40419 (95%-conf.int. 0.40174 - 0.40665)
1 ROUGE-L Average_P: 0.28191 (95%-conf.int. 0.27984 - 0.28396)
1 ROUGE-L Average_F: 0.32309 (95%-conf.int. 0.32111 - 0.32509)

My performance with post-posting (from ProphetNet):
1 ROUGE-1 Average_R: 0.49758 (95%-conf.int. 0.49505 - 0.50028)
1 ROUGE-1 Average_P: 0.35663 (95%-conf.int. 0.35421 - 0.35889)
1 ROUGE-1 Average_F: 0.40406 (95%-conf.int. 0.40200 - 0.40607)
---------------------------------------------
1 ROUGE-2 Average_R: 0.21882 (95%-conf.int. 0.21622 - 0.22125)
1 ROUGE-2 Average_P: 0.15750 (95%-conf.int. 0.15543 - 0.15947)
1 ROUGE-2 Average_F: 0.17794 (95%-conf.int. 0.17576 - 0.17998)
---------------------------------------------
1 ROUGE-L Average_R: 0.41627 (95%-conf.int. 0.41375 - 0.41881)
1 ROUGE-L Average_P: 0.29928 (95%-conf.int. 0.29712 - 0.30132)
1 ROUGE-L Average_F: 0.33860 (95%-conf.int. 0.33658 - 0.34056)

The expected performance for sshleifer/distilbart-cnn-12-6 is ?/21.26/30.59 and I can only achieve 40.41/17.79/33.86. So is the trick related to the post-processing, or how can I achieve the expected performance?
Thank you!

Context:
For anyone may see this post, the problem is solved by using larger batch size. Above result is using batch size 32 on GeForce 2080 Ti, and I change to Tesla V100 today with batch size 128. The result is pretty closed to expected:
1 ROUGE-1 Average_R: 0.53399 (95%-conf.int. 0.53146 - 0.53669)
1 ROUGE-1 Average_P: 0.39205 (95%-conf.int. 0.38963 - 0.39451)
1 ROUGE-1 Average_F: 0.44179 (95%-conf.int. 0.43972 - 0.44408)
---------------------------------------------
1 ROUGE-2 Average_R: 0.25584 (95%-conf.int. 0.25312 - 0.25867)
1 ROUGE-2 Average_P: 0.18821 (95%-conf.int. 0.18605 - 0.19056)
1 ROUGE-2 Average_F: 0.21172 (95%-conf.int. 0.20940 - 0.21420)
---------------------------------------------
1 ROUGE-L Average_R: 0.44976 (95%-conf.int. 0.44719 - 0.45244)
1 ROUGE-L Average_P: 0.33090 (95%-conf.int. 0.32864 - 0.33330)
1 ROUGE-L Average_F: 0.37251 (95%-conf.int. 0.37040 - 0.37465)


Question:
Hello,
I want to share my dataset, I’m following the instructions here:

      huggingface.co

Sharing your dataset — datasets 1.0.2 documentation 2

I copied thins command in order to add the metadata:
python datasets-cli test datasets/ --save_infos --all_configs
but it gives me a syntax error each time
[Errno 2] No such file or directory: ‘sample_data/’ /content/sample_data File “dataset-sharing.py”, line 6 python datasets-cli test datasets/sample_data --save_infos --all_configs ^ SyntaxError: invalid syntax
What is the correct way to specify the dataset folder?
thanks

Context:
Hi,
In the link you shared, under “Adding tests and metadata to the dataset”, it says:
“In the rest of this section, you should make sure that you run all of the commands  from the root  of your local  datasets  repository.”
Since you are getting ‘No such file or directory’, are you running from the root?


Question:
There are many examples showing how to train T5 in PyTorch, e.g.: https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb 74 , but none so far for Tensorflow. Many people have been asking on transformers: https://github.com/huggingface/transformers/issues/3626 8 .
Does anybody have a good notebook showing how to train T5 in Tensorflow?
Otherwise, I will try to translate @valhalla 's great notebook: https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb 74 to Tensorflow.

Context:
Thanks very much Patrick @patrickvonplaten!
In Suraj Patil’s notebook, he employed Pytorch Trainer to train T5.
At first, I didn’t know that we can use Trainer with Seq2Seq problems (according to “The Big Table of Tasks”  https://huggingface.co/transformers/examples.html 29 which stated that Trainer does not yet support Translation / Summarization )
I will try to use TFTrainer for TF2 on Seq2Seq problems. If that doesn’t work, I think I will try to write custom loop in TF2.


Speed expectations for production BERT models on CPU vs GPU?

Question:
Hi! I’m working with an ML model produced by a researcher. I’m trying to set it up to run economically in production on large volumes of text. I know a lot about production engineering, but next to nothing about ML. I’m getting some results that are surprising to me, and I’m hoping for pointers to explanations and advice.
Right now I have the key code broken out into 3 methods, so it’s easy to profile.
def _tokenize(self, text):
    return self.tokenizer.encode_plus(text, add_special_tokens=True, return_tensors='pt').to(self.device)

def _run_model(self, model_input):
    return self.model(model_input['input_ids'], token_type_ids=model_input['token_type_ids'])[0]

def _extract_results(self, logits):
    return logits[0][0].item(), logits[0][1].item()

If I run this using my laptop CPU, I get numbers that make sense to me. For 1459 items, those three methods take 196.4 seconds, or about 135 ms per item. About 2.9 seconds is _tokenize, and the rest is _run_model.
When I switch over to my laptop GPU, I get numbers that mystify me. The same data takes 131.8 seconds. 2.5 seconds to tokenize, and running the model takes 20.3 seconds. But extracting the result takes 108.8 seconds!
The _extract_results method costs the same whether I extract one logit or both. The first one that I extract is slow, whether that’s [0] or [1]. The second one is effectively free.
From nvidia-smi, I can see that the GPU is really being used, and my process is using ~850MB of the 2 GB of GPU RAM. So that seems fine. And if it matters, this is a GeForce 940MX on a Thinkpad t470.
Do these numbers make sense to more experienced hands? I was expecting the GPU runs to be much faster, but if I actually want to get the results, it’s only a little faster.
Thanks!

Context:
(probably too late to be useful…)
Is the GPU being used for all of the sections?  (Maybe your researcher only put the middle section on GPU.)
Does the run-model step actually train and update the model, or does it just calculate answers for the data you entered?


Question:
Hello,
How pre-trained are obtained?
In the library there is always some pre-trained model downloaded from the server, but how is it originally trained? Is it also trained using transformers library?

Context:
In most if not all built-in cases (e.g. bert-base-cased), the original paper implementations are ported to the transformers architecture. (you can have a look at conversion scripts, e.g. https://github.com/huggingface/transformers/blob/77cd0e13d2d09f60d2f6d8fb8b08f493d7ca51fe/src/transformers/convert_pytorch_checkpoint_to_tf2.py 2, https://github.com/huggingface/transformers/blob/d155b38d6ea70fef3dec2e1f678269e713672bb7/src/transformers/commands/convert.py)
User models (e.g. username/mymodel-uncased) may have been trained in other ways or ported to the transformers architecture manually, with custom scripts, or they are trained by using the transformers library directly.


How to convert Tokenizer to TokenizerFast?

Question:
I’m trying to run the question answering pipeline on chunks of of a long document. The DistilBertTokenizerFast has good features to support this. It can break a document down into overlapping chunks and it can give you the offsets mapping so that you can match up labels with tokens. But in the pipeline they use by default the DistilBertTokenizer, which has neither of these features. Is there a better way than below to find out what model is being used in the pipeline and construct a “fast” tokenizer to match?
from transformers import pipeline
from transformers import DistilBertTokenizerFast

nlp = pipeline("question-answering")
regular_tokenizer = nlp.tokenizer
# look up which model is being used on github
# https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines.py
# currently the model is:
model_name = "distilbert-base-cased-distilled-squad"
fast_tokenizer = DistilBertTokenizerFast.from_pretrained(model_name)

Context:
You can pass tokenizer to the pipeline
pipeline("", tokenizer=your_tok)


Model results differ after creating pipeline with same model

Question:
Im using a pre-trained tokeniser and a fine-tuned model.
With the same model, when I create pipeline, my results differ both the time.
tokenizer = AutoTokenizer.from_pretrained("huggingface/CodeBERTa-small-v1")
mymodelclf  = TextClassificationPipeline(model=model, tokenizer = tokenizer,return_all_scores = True )
print (mymodelclf( negative_samples ) )

The same code is run twice, but 2 different results are obtained. Is this expected?
image2000×482 109 KB

Context: Are you sure your model is in evaluation mode?


Question: So I trained a new gpt2 model using the language modeling script and it spits out a pytorch.bin file that I assume is the model. How do I go about converting it to TF 2.0? I see there is a convert script but I am not sure if I can just run that to make it TF compatible?

Context:
You should create your model like this:
from transformers import TFGPT2Model
model = TFGPT2Model.from_pretrained("path_to_dir", from_pt=True)

where path_to_dir should be replaced with the path to the directory where your pytorch model is (what you set in  GPT2Model.save_pretrained()).
Then you can use your TF model and save it with the save_pretrained method.


Question:
Hello everyone,
I am rewriting some old code to use the new tokenizer syntax and the Trainer class but I believe I am missing something.
This is how I am building the training dataset to be passed to the Trainer constructor:
    encoded_texts = tokenizer(texts, padding = True, truncation = True, return_tensors = 'pt')
    labels = torch.tensor(labels)
    dataset = TensorDataset(encoded_texts['input_ids'], encoded_texts['attention_mask'], labels)

Can you please help me understand what I am doing wrong/missing? When I run trainer.train() I get the following error:
vars() argument must have __dict__ attribute
Thanks in advance!

Context:
I’ll give you the full picture.
The workflow:
You create an instance of GlueDataset(data_args, tokenizer). Then you pass it to Trainer(...) class. In trainer, you also pass in default_data_collator. The reason is that GlueDataset return InputExample which is HF specific and cannot be used by Pytorch dataloader directly. So the default_data_collator takes in List[InputExamples] and returns a dict. This dict is then used by the dataloader.
So in trainer, if you pass default_data_collator with TensorDataset, it won’t work directly (That’s why you’re getting the error). This error is raised when dataloader will pass the batch to default_data_collator. I’d suggest using the default Pytorch collate_fn with your TensorDataset, it would work just fine.
One more additional thing:
Make sure the dataloader returns the dict with same key values forward method expects.
Inside _training_step, you’ll pass inputs to the function, and then after the inputs are passed kept on gpu, the function does:
output = model(**inputs)
In this case, the keyword arguments have to match. In case, they don’t,  you can inherit from Trainer and redefine your own method.
I hope this answers your question.


How to split a dataset into train, test, and validation?

Question:
I am having difficulties trying to figure out how I can split my dataset into train, test, and validation. I’ve been going through the documentation here:

      github.com

    huggingface/datasets/blob/37d4840a39eeff5d472beb890c8f850dc7723bb8/datasets/wikihow/wikihow.py 10
# coding=utf-8
# Copyright 2020 The TensorFlow Datasets Authors and the HuggingFace Datasets Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Lint as: python3
"""WikiHow Datasets."""

from __future__ import absolute_import, division, print_function

  This file has been truncated. show original

and the template here:

      github.com

    huggingface/datasets/blob/master/templates/new_dataset_script.py#L63 5

        Args:
            data_size: the size of the training set we want to us (xs, s, m, l, xl)
            **kwargs: keyword arguments forwarded to super.
        """
        self.data_size = data_size

class NewDataset(datasets.GeneratorBasedBuilder):
    """TODO: Short description of my dataset."""

    VERSION = datasets.Version("1.1.0")

    # This is an example of a dataset with multiple configurations.
    # If you don't want/need to define several sub-sets in your dataset,
    # just remove the BUILDER_CONFIG_CLASS and the BUILDER_CONFIGS attributes.
    BUILDER_CONFIG_CLASS = NewDatasetConfig
    BUILDER_CONFIGS = [
        NewDatasetConfig(name="my_dataset_" + size, description="A small dataset", data_size=size) for size in ["small", "medium", "large"]
    ]

but it hasn’t become any clearer.
this is the error I keep getting:
TypeError: ‘NoneType’ object is not callable
and this is the code I’m using:
def _split_generators(self, dl_manager):
    """Returns SplitGenerators."""
    dl_path = dl_manager.download_and_extract(_URLS)
    titles = {k: set() for k in dl_path}
    for k, path in dl_path.items():
        with open(path, encoding="utf-8") as f:
            for line in f:
                titles[k].add(line.strip())

    path_to_manual_file = os.path.join(
        os.path.abspath(os.path.expanduser(dl_manager.manual_dir)), self.config.filename
    )

    if not os.path.exists(path_to_manual_file):
        raise FileNotFoundError(
            "{} does not exist. Make sure you insert a manual dir via `datasets.load_dataset('wikihow', data_dir=...)` that includes a file name {}. Manual download instructions: {})".format(
                path_to_manual_file, self.config.filename, self.manual_download_instructions
            )
        )
    return [
        datasets.SplitGenerator(
            name=datasets.Split.TRAIN,
            gen_kwargs={
                "path": path_to_manual_file,
                "title_set": titles["train"],
            },
        ),
        datasets.SplitGenerator(
            name=datasets.Split.VALIDATION,
            gen_kwargs={
                "path": path_to_manual_file,
                "title_set": titles["validation"],
            },
        ),
        datasets.SplitGenerator(
            name=datasets.Split.TEST,
            gen_kwargs={
                "path": path_to_manual_file,
                "title_set": titles["test"],
            },
        ),

Context:
I think it’s answered here
How to split main dataset into train, dev, test as DatasetDict 959


Recommended Hardware for NER Pipeline Model

Question:
Trying to get started and run the pretrained NER pipeline model (https://huggingface.co/transformers/task_summary.html#named-entity-recognition 4) on about 10 million instances of text. Would you recommend using a CPU or GPU?
Previously, I used spaCy and broke my data up into batch sizes of 1k then ran those batches on a C5 instance on AWS. The AWS EC2 instance was compute optimized and had 96 cores. I’m able to run any instance on AWS. Thanks!

Context: if you have 10M examples and have access to GPU then definitely use GPU. If you want fast inference on CPU for ner pipeline then you could try onnx_transformers 8, which provides same API as pipeline  but leverages onnx for accelerated inference.


Run_ner.py slower on multi-GPU than single GPU

Question:
Am i missing something? Why run_ner.py in 3.0.2 is that slower when running on 2 GPUs vs a single GPU ?
(1) 7 minutes for fp16, 1 GPU
(2) 13 minutes for fp16, 2 GPUs
(3) 11 minutes for fp16, python -m torch.distributed.launch --nproc_per_node 2 run_ner.py
(1)
07/13/2020 13:21:49 - INFO - transformers.training_args -   PyTorch: setting up devices
07/13/2020 13:21:50 - WARNING - main -   Process rank: -1, device: cuda:0, n_gpu: 1, distributed training: False, 16-bits training: True
07/13/2020 13:21:50 - INFO - main -   Training/evaluation parameters TrainingArguments(output_dir=’/opt/ml/model/output’, overwrite_output_dir=False, do_train=True, do_eval=True, do_predict=True, evaluate_du
ring_training=False, per_device_train_batch_size=6, per_device_eval_batch_size=8, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, learning_rate=5e-05, weight_decay=0.0
, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=4, max_steps=-1, warmup_steps=0, logging_dir=’/opt/ml/model/log’, logging_first_step=False, logging_steps=500, save_steps=750, save_total_limit=None, no_
cuda=False, seed=1, fp16=True, fp16_opt_level=‘O1’, local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=1000, past_index=-1)
07/13/2020 13:21:50 - INFO - transformers.configuration_utils -   loading configuration file /opt/program/models/bert-base-multilingual-cased/config.json
…
Defaults for this optimization level are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
Processing user overrides (additional kwargs that are not None)…
After processing overrides, optimization options are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
07/13/2020 13:21:58 - INFO - transformers.trainer -   ***** Running training *****
07/13/2020 13:21:58 - INFO - transformers.trainer -     Num examples = 7239
07/13/2020 13:21:58 - INFO - transformers.trainer -     Num Epochs = 4
07/13/2020 13:21:58 - INFO - transformers.trainer -     Instantaneous batch size per device = 6
07/13/2020 13:21:58 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 6
07/13/2020 13:21:58 - INFO - transformers.trainer -     Gradient Accumulation steps = 1
07/13/2020 13:21:58 - INFO - transformers.trainer -     Total optimization steps = 4828
07/13/2020 13:21:58 - INFO - transformers.trainer -     Starting fine-tuning.
…
Epoch: 100%|██████████| 4/4 [07:13<00:00, 108.33s/it]
07/13/2020 13:29:12 - INFO - transformers.trainer -
Training completed. Do not forget to share your model on huggingface.co/models =)
(2)
07/13/2020 15:21:33 - INFO - transformers.training_args -   PyTorch: setting up devices
07/13/2020 15:21:33 - WARNING - main -   Process rank: -1, device: cuda:0, n_gpu: 2, distributed training: False, 16-bits training: True
07/13/2020 15:21:33 - INFO - main -   Training/evaluation parameters TrainingArguments(output_dir=’/opt/ml/model/output’, overwrite_output_dir=False, do_train=True, do_eval=True, do_predict=True, evaluate_du
ring_training=False, per_device_train_batch_size=6, per_device_eval_batch_size=8, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, learning_rate=5e-05, weight_decay=0.0
, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=4, max_steps=-1, warmup_steps=0, logging_dir=’/opt/ml/model/log’, logging_first_step=False, logging_steps=500, save_steps=750, save_total_limit=None, no_
cuda=False, seed=1, fp16=True, fp16_opt_level=‘O1’, local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=1000, past_index=-1)
07/13/2020 15:21:33 - INFO - transformers.configuration_utils -   loading configuration file /opt/program/models/bert-base-multilingual-cased/config.json
…
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
Processing user overrides (additional kwargs that are not None)…
After processing overrides, optimization options are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
07/13/2020 15:21:57 - INFO - transformers.trainer -   ***** Running training *****
07/13/2020 15:21:57 - INFO - transformers.trainer -     Num examples = 7239
07/13/2020 15:21:57 - INFO - transformers.trainer -     Num Epochs = 4
07/13/2020 15:21:57 - INFO - transformers.trainer -     Instantaneous batch size per device = 6
07/13/2020 15:21:57 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 12
07/13/2020 15:21:57 - INFO - transformers.trainer -     Gradient Accumulation steps = 1
07/13/2020 15:21:57 - INFO - transformers.trainer -     Total optimization steps = 2416
07/13/2020 15:21:57 - INFO - transformers.trainer -     Starting fine-tuning.
…
Epoch: 100%|██████████| 4/4 [13:16<00:00, 199.08s/it]
07/13/2020 15:35:14 - INFO - transformers.trainer -
Training completed. Do not forget to share your model…
(3)
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

07/13/2020 15:47:16 - INFO - transformers.training_args -   PyTorch: setting up devices
07/13/2020 15:47:16 - WARNING - main -   Process rank: 1, device: cuda:1, n_gpu: 1, distributed training: True, 16-bits training: True
07/13/2020 15:47:16 - WARNING - main -   Process rank: 0, device: cuda:0, n_gpu: 1, distributed training: True, 16-bits training: True
07/13/2020 15:47:16 - INFO - main -   Training/evaluation parameters TrainingArguments(output_dir=’/opt/ml/model/output’, overwrite_output_dir=False, do_train=True, do_eval=True, do_predict=True, evaluate_du
ring_training=False, per_device_train_batch_size=6, per_device_eval_batch_size=8, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, learning_rate=5e-05, weight_decay=0.0
, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=4.0, max_steps=-1, warmup_steps=0, logging_dir=’/opt/ml/model/log’, logging_first_step=False, logging_steps=500, save_steps=750, save_total_limit=None, n
o_cuda=False, seed=1, fp16=True, fp16_opt_level=‘O1’, local_rank=0, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=1000, past_index=-1)
07/13/2020 15:47:16 - INFO - transformers.configuration_utils -   loading configuration file /opt/program/models/bert-base-multilingual-cased/config.json
…
Defaults for this optimization level are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
Processing user overrides (additional kwargs that are not None)…
After processing overrides, optimization options are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
07/13/2020 15:47:41 - WARNING - transformers.trainer -   You are instantiating a Trainer but Tensorboard is not installed. You should consider installing it.
07/13/2020 15:47:41 - INFO - transformers.trainer -   ***** Running training *****
07/13/2020 15:47:41 - INFO - transformers.trainer -     Num examples = 7239
07/13/2020 15:47:41 - INFO - transformers.trainer -     Num Epochs = 4
07/13/2020 15:47:41 - INFO - transformers.trainer -     Instantaneous batch size per device = 6
07/13/2020 15:47:41 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 12
07/13/2020 15:47:41 - INFO - transformers.trainer -     Gradient Accumulation steps = 1
07/13/2020 15:47:41 - INFO - transformers.trainer -     Total optimization steps = 2416
07/13/2020 15:47:41 - INFO - transformers.trainer -     Starting fine-tuning.
…
Epoch: 100%|██████████| 4/4 [11:13<00:00, 168.32s/it]
07/13/2020 15:58:54 - INFO - transformers.trainer -
Training completed. Do not forget to share your model…

Context: It might be that since local_rank =-1 for the 2nd setting, only one gpu was still being used.


Cannot download tensorflow model of cahya/bert-base-indonesian-522M

Question:
I was going to download this 3 model, and then I was going to save it later to be used with bert-serving. Since bert-serving only supports tensorflow model, I need to download the tensorflow one and not the PyTorch. The PyTorch model downloads just fine, but the I cannot download the tensorflow model. I used this code to download:
from transformers import BertTokenizer, TFBertModel

model_name='cahya/bert-base-indonesian-522M'
model = TFBertModel.from_pretrained(model_name)

Here’s what I got when running the code on Ubuntu 16.04, python3.5, transformers==2.5.1,
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/username/.local/lib/python3.5/site-packages/transformers/modeling_tf_utils.py", line 346, in from_pretrained
    assert os.path.isfile(resolved_archive_file), "Error retrieving file {}".format(resolved_archive_file)
  File "/usr/lib/python3.5/genericpath.py", line 30, in isfile
    st = os.stat(path)
TypeError: stat: can't specify None for path argument

And here’s what I got when running it on Windows 10, python 3.6.5, transformers 3.1.0:
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
C:\ProgramData\Anaconda3\lib\site-packages\transformers\modeling_tf_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    579                 if resolved_archive_file is None:
--> 580                     raise EnvironmentError
    581             except EnvironmentError:

OSError:

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-3-c2f14f761f05> in <module>()
      3 model_name='cahya/bert-base-indonesian-522M'
      4 tokenizer = BertTokenizer.from_pretrained(model_name)
----> 5 model = TFBertModel.from_pretrained(model_name)

C:\ProgramData\Anaconda3\lib\site-packages\transformers\modeling_tf_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    585                     f"- or '{pretrained_model_name_or_path}' is the correct path to a directory containing a file named one of {TF2_WEIGHTS_NAME}, {WEIGHTS_NAME}.\n\n"
    586                 )
--> 587                 raise EnvironmentError(msg)
    588             if resolved_archive_file == archive_file:
    589                 logger.info("loading weights file {}".format(archive_file))

OSError: Can't load weights for 'cahya/bert-base-indonesian-522M'. Make sure that:

- 'cahya/bert-base-indonesian-522M' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'cahya/bert-base-indonesian-522M' is the correct path to a directory containing a file named one of tf_model.h5, pytorch_model.bin.

This also happens with other cahya/ models. This page 3 says that you can use the tensorflow model. However, based on the error, it seems like the file does not exist over there?
I tried downloading other pretrained model like bert-base-uncased etc. and they download just fine. This issue only happens with cahya/ models.
Am I missing something? or should I report this issue to a github issue?

Context: @cahya do you have any idea what might cause this?


Question:
Hi,
So I’m currently trying to train a model in tensorflow on SMILES which is a bit of chemical information which tells you the molecular formula of a given molecule. I thought a transformer would work well because of the importance of context of each character of the string within the SMILES.
I am currently having an issue running the model this is the error I get:
C:\ProgramData\Anaconda3\envs\tf-transformer\lib\site-packages\transformers\trainer_tf.py:653 distributed_training_steps  *
    self.args.strategy.run(self.apply_gradients, batch)
C:\ProgramData\Anaconda3\envs\tf-transformer\lib\site-packages\transformers\trainer_tf.py:618 apply_gradients  *
    gradients = self.training_step(features, labels)
C:\ProgramData\Anaconda3\envs\tf-transformer\lib\site-packages\transformers\trainer_tf.py:601 training_step  *
    per_example_loss, _ = self.run_model(features, labels, True)
C:\ProgramData\Anaconda3\envs\tf-transformer\lib\site-packages\transformers\trainer_tf.py:682 run_model  *
    outputs = self.model(features, labels=labels, training=training)[:2]
C:\ProgramData\Anaconda3\envs\tf-transformer\lib\site-packages\transformers\modeling_tf_bert.py:1127 call  *
    outputs = self.bert(
C:\ProgramData\Anaconda3\envs\tf-transformer\lib\site-packages\transformers\modeling_tf_bert.py:615 call  *
    embedding_output = self.embeddings(input_ids, position_ids, token_type_ids, inputs_embeds, training=training)
C:\ProgramData\Anaconda3\envs\tf-transformer\lib\site-packages\transformers\modeling_tf_bert.py:191 call  *
    return self._embedding(input_ids, position_ids, token_type_ids, inputs_embeds, training=training)
C:\ProgramData\Anaconda3\envs\tf-transformer\lib\site-packages\transformers\modeling_tf_bert.py:206 _embedding  *
    seq_length = input_shape[1]

IndexError: list index out of range

This part is caused by running the trainer.train()
I have checked to make sure there are no zero values in my data and the dataset I have just contains the SMILES (String) which is a feature and then the target which is the CCS (Collisional Cross Section) (float)
This is my code:
  dataFrameData = pd.DataFrame(numpyArrayOfData, columns=['CAS', 'CCS', 'Compound', 'Adducts', 'Mass', 'SMILES']).iloc[1:]
# splitting the data into testing and training
train, test = train_test_split(dataFrameData, test_size=0.2)

# getting the CCS as the target from the data
targetTrain = train.pop('CCS').astype(float)
targetTest = test.pop('CCS').astype(float)
# getting the SMILES as first feature from data
dfStrippedTrain = train[['SMILES']].copy()
dfStrippedTest = test[['SMILES']].copy()

# compile these two into a test and training data set and then use enumerate so there is indexing
dataSetTrain = tf.data.Dataset.from_tensor_slices((dfStrippedTrain.values, targetTrain.values)).enumerate()

dataSetTest = tf.data.Dataset.from_tensor_slices((dfStrippedTest.values, targetTest.values)).enumerate()

model = TFBertForSequenceClassification.from_pretrained("bert-large-cased")

training_args = TFTrainingArguments(
     output_dir = '/results',
     num_train_epochs = 3,
     per_device_train_batch_size=16,
     per_device_eval_batch_size=64,
     weight_decay=0.01,
     logging_dir='/logs'
)

trainer = TFTrainer(
     model = model,
     args = training_args,
     train_dataset = dataSetTrain,
     eval_dataset = dataSetTest
)

trainer.train()
trainer.evaluate()

Any help you can give me would be greatly appreciated. Thank you in advance.

Context:
Hi,
what does your SMILES data string look like?   Have you considered tokenizing it?


Tokenizer decoding using BERT, RoBERTa, XLNet, GPT2

Question:
I’ve been using  BERT and am fairly familiar with it at this point. I’m now trying out RoBERTa, XLNet, and GPT2. When I try to do basic tokenizer encoding and decoding, I’m getting unexpected output.
Here is an example of using BERT for tokenization and decoding:
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
result = tokenizer(text='the needs of the many', text_pair='outweigh the needs of the few')
input_ids = result['input_ids']
print(input_ids)
print(tokenizer.decode(input_ids))
print(tokenizer.convert_ids_to_tokens(input_ids))

The output is expected:
[101, 1996, 3791, 1997, 1996, 2116, 102, 2041, 27204, 2232, 1996, 3791, 1997, 1996, 2261, 102]
[CLS] the needs of the many [SEP] outweigh the needs of the few [SEP]
['[CLS]', 'the', 'needs', 'of', 'the', 'many', '[SEP]', 'out', '##weig', '##h', 'the', 'needs', 'of', 'the', 'few', '[SEP]']

I understand the special tokens like [CLS] and the wordpiece tokens like ##weig.
However, when I try other models, I get crazy output.
RoBERTa
tokenizer = AutoTokenizer.from_pretrained('roberta-base')
result = tokenizer(text='the needs of the many', text_pair='outweigh the needs of the few')
input_ids = result['input_ids']
print(input_ids)
print(tokenizer.decode(input_ids))
print(tokenizer.convert_ids_to_tokens(input_ids))

Output:
[0, 627, 782, 9, 5, 171, 2, 2, 995, 1694, 8774, 5, 782, 9, 5, 367, 2]
<s>the needs of the many</s></s>outweigh the needs of the few</s>
['<s>', 'the', 'Ġneeds', 'Ġof', 'Ġthe', 'Ġmany', '</s>', '</s>', 'out', 'we', 'igh', 'Ġthe', 'Ġneeds', 'Ġof', 'Ġthe', 'Ġfew', '</s>']

Where are those Ġ characters coming from?
XLNet
tokenizer = AutoTokenizer.from_pretrained('xlnet-base-cased')
result = tokenizer(text='the needs of the many', text_pair='outweigh the needs of the few')
input_ids = result['input_ids']
print(input_ids)
print(tokenizer.decode(input_ids))
print(tokenizer.convert_ids_to_tokens(input_ids))

Output:
/usr/local/lib/python3.6/dist-packages/transformers/configuration_xlnet.py:211: FutureWarning: This config doesn't use attention memories, a core feature of XLNet. Consider setting `men_len` to a non-zero value, for example `xlnet = XLNetLMHeadModel.from_pretrained('xlnet-base-cased'', mem_len=1024)`, for accurate training performance as well as an order of magnitude faster inference. Starting from version 3.5.0, the default parameter will be 1024, following the implementation in https://arxiv.org/abs/1906.08237
  FutureWarning,
[18, 794, 20, 18, 142, 4, 23837, 18, 794, 20, 18, 274, 4, 3]
the needs of the many<sep> outweigh the needs of the few<sep><cls>
['▁the', '▁needs', '▁of', '▁the', '▁many', '<sep>', '▁outweigh', '▁the', '▁needs', '▁of', '▁the', '▁few', '<sep>', '<cls>']

Why are there underscore ▁ characters?
GPT2
tokenizer = AutoTokenizer.from_pretrained('gpt2')
result = tokenizer(text='the needs of the many', text_pair='outweigh the needs of the few')
input_ids = result['input_ids']
print(input_ids)
print(tokenizer.decode(input_ids))
print(tokenizer.convert_ids_to_tokens(input_ids))

Output:
[1169, 2476, 286, 262, 867, 448, 732, 394, 262, 2476, 286, 262, 1178]
the needs of the manyoutweigh the needs of the few
['the', 'Ġneeds', 'Ġof', 'Ġthe', 'Ġmany', 'out', 'we', 'igh', 'Ġthe', 'Ġneeds', 'Ġof', 'Ġthe', 'Ġfew']

Again, where are those Ġ characters coming from?
I understand there are different subword tokenization schemes used by each. I also have the original research papers. Can someone please explain how the Huggingface Transformers implementation is producing these different outputs?

Context:
Each tokenizer has its own way of representing pieces of the same word, because the model expact them in different ways. For instance Bert model expects BPE tokenization because it was pretrained this way. GPT-2 and RoBERTa expect byet-level BPE (because they were pretrained this way) which results in outputs with those Ġ (which basically represent space) whereas XLNet expects sentencepiece tokenized texts, which uses ▁ to represent the space character.
You can find a high-level summary of the tokenizers and which one is used for each model in this doc page 30.


Why positional embeddings are implemented as just simple embeddings?

Question:
Hello!
I can’t figure out why the positional embeddings are implemented as just the vanilla Embedding layer in both PyTorch 26 and Tensorflow 25. Based on my current understanding, positional embeddings should be implemented as non-trainable sin/cos or axial positional encodings (from reformer).
Can anyone please enlighten me with this? Thank you so much!

Context:
Hi @miguelvictor ! Both are valid strategies: iirc the original Transformers paper had sinusoidal embeddings with a fixed rate, but BERT learned a full vector for each of the 512 expected positions.
Currently, the Transformers library has sinusoidal embeddings in the TransfoXL model, check it out!


Question: This is a more open-ended question. Suppose you’ve trained a model and after looking at the loss curve you see it hasn’t learned (the loss curve is flat). What sort of things would you begin to investigate to understand what might be the cause? Additionally, what sort of things might you log apriori to help you debug a model that didn’t learn (in the event that it happened)?

Context:
First things first: don’t wait until the model is done learning; use a tool like tensorboard or weights and biases, or simply print your loss after every step. Just so you can jump in quickly.
The first thing that your model should be capable of is overfitting on a tiny dataset (e.g. 1—5 samples). Trying this out should be relatively simple. If it doesn’t do so, you know something is wrong.
Good things to check: learning rate too high or too low? Did you accidentally freeze the model and your model doesn’t have trainable parameters? Are your shapes in your forward pass correct? Did you clear the gradients after optimising? And so on.


Question:
https://huggingface.co/exbert/?model=bert-base-uncased&modelKind=bidirectional&sentence=The%20girl%20ran%20to%20a%20local%20pub%20to%20escape%20the%20din%20of%20her%20city.&layer=0&heads=..0,1,2,3,4,5,6,7,8,9,10,11&threshold=0.7&tokenInd=null&tokenSide=null&maskInds= 5…&hideClsSep=true
Is it possible to run this in Google Colab? There are some changes I’d like to make, but I’m really not sure how to do it.

Context:
Hi zanderbush,
I don’t know whether it is possible to run ExBert on Colab, but it is definitely possible to run BertViz on Colab.  (I use it.)    Do you need the extra functions of ExBert, or would BertViz be enough?
I would expect it to be possible to install ExBert on Colab, though you might have memory problems if you want to do the Corpus part.
I don’t think bert-base-uncased is one of the available options in ExBert, so you might have to use bert-base-cased.


How to train a language model from scratch when my dataset is bigger than RAM?

Question:
I intend to train a language model from scratch. I have been following the tutorial on Hugging Face’s blog 108. However, when I tried to use their google Colab’s code, I had a memory exceeded error. Their Google Colab’s code uses the LineByLineTextDataset. After digging through the source code, it turns out that LineByLineTextDatasets loads the entire dataset eagerly. No wonder why I had a Memory Exceeded error. My dataset is larger than my RAM capacity.
The article hints that if my dataset is bigger than my capacity, I could “opt to load and tokenize examples on the fly, rather than as a preprocessing step.” However, I’m not certain how to achieve this. I will be very grateful if others can points me to the right direction, especially if I can still use the transformers.Trainer.

Context: Do you want to use TensorFlow or PyTorch for the training?


Question:
Hi, I’m currently training a T5 model using the HF Trainer and looking through the TrainingArguments documentation as well as example scripts.
At the moment I am trying to understand the use of the the two optional arguments within TrainingArguments ‘do_train’ and ‘do_eval’. Some examples notebooks don’t specify these, and therefore default these to false.
From experimenting, you can run trainer.evaluate() without ‘do_eval’, and I can’t figure out what ‘do_train’ would do if it defaults to False
Thanks,

Context: Those arguments are only used by the scripts, not the Trainer itself.


Does training tokenizer and adding new token to model when training BART on custom dataset improve performance?

Question:
I have a dataset composed of 300,000 articles. Is it wise to train tokenizer on my own dataset and add new token to the model then train BART?
Is my dataset large enough to pre-train the embedding of new tokens added?

Context:
If the domain of your dataset isn’t very specialized from the pre-training datatset then training tokenizer won’t help much. Also note that if you train a new tokenizer then you’ll need to do the pre-training again, and when you train your tokenizer from scratch you don’t need to add the new tokens any more since you already trained tokenizer.  Hope this makes sense.

Is my dataset large enough

IMO this is highly subjective question and depends on the quality of the data, the task etc.
@sshleifer might have some advice  for this


Question:
I have a corpus of about 15,000 documents, with a total of about 8gb of text, which I want to use as the source-material for a text generator.
With that much data, would it make sense to use a pre-trained model (like gpt2-large), and then fine-tune it on my corpus? Or would it make more sense to train a new language model from scratch using my data? What would be the trade-offs between those two options?
I know the answer is probably complex, but I’m interested in understanding what considerations to take into account, especially how those two options effect my budget for running the training process in the cloud.

Context:
Definitely finetune IMO. Will be much faster and better results. Would you rather teach a third grader how to predict the next word on your dataset or a newborn?
One argument for from scratch would be  more control. It is less likely your from scratch model says something racist or otherwise wrong if it is trained on just your (presumably friendly) data.


Question:
Hello,
I am a newbie in using transformers and never used it before.
I want to know, what’s the difference between T5Model and T5forConditionalGeneration? Where are they used?

Context:
Hi @ashiishkarhade
T5Model contains the encoder (stack of encoder layers) and decoder (stack of decoder layers)  without any task specific heads. It returns the raw hidden states of the decoder as output.
T5ForConditionalGeneration also contains the encoder and decoder and adds an additional linear layer (lm_head) which takes the final hidden states of decoder and generates the next token.
For fine-tuning the model for seq2seq generation you should use  T5ForConditionalGeneration, if you want to add some different task specific head then  you can T5Model.
And almost all library models have this structure, a base model which returns raw hidden states and additional models with task specific heads(ForSequenceClassification, ForQuestionAnswering etc) on top of the base model.


Language-modeling script “killed” when fine-tuning gpt2-medium

Question:
Hello! I’m just getting started with the huggingface libraries for text-generation.
I cloned the transformers repo from github, and then I was able to successfully run fine-tuning on the GPT-2 small model, on my macbook, using this command…
python3 examples/language-modeling/run_language_modeling.py \
    --output_dir=/my/data/gpt2-small-finetune \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train \
    --train_data_file=/path/to/my_corpus_text.txt

For now, I’m using a very small corpus… with a total of around 150kb of text. The training process took about 30 minutes (which I assume was using the CPU rather than GPU). Once it finished running, I was able to successfully use the fine-tuned model to generate text, like this:
python3 examples/text-generation/run_generation.py \
    --model_type=gpt2 \
    --model_name_or_path=/my/data/gpt2-small-finetune \
    --length=100 \
    --prompt="Once upon a time, there was a "

Next, I tried running the same exact process, with the same training corpus, but with a new output directory and with model_name_or_path set to gpt2-medium…
python3 examples/language-modeling/run_language_modeling.py \
    --output_dir=/my/data/gpt2-medium-finetune \
    --model_type=gpt2 \
    --model_name_or_path=gpt2-medium \
    --do_train \
    --train_data_file=/path/to/my_corpus_text.txt

The process starts up and emits some logs, which look normal…
09/13/2020 17:31:51 - WARNING - __main__ -   Process rank: -1, device: cpu, n_gpu: 0, distributed training: False, 16-bits training: False
09/13/2020 17:31:51 - INFO - __main__ -   Training/evaluation parameters TrainingArguments(output_dir='/my/data/gpt2-small-finetune', overwrite_output_dir=False, do_train=True, do_eval=False, do_predict=False, evaluate_during_training=False, prediction_loss_only=False, per_device_train_batch_size=8, per_device_eval_batch_size=8, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, learning_rate=5e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=3.0, max_steps=-1, warmup_steps=0, logging_dir='runs/Sep13_17-31-51_APTI5214-MBP', logging_first_step=False, logging_steps=500, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=1000, past_index=-1, run_name=None, disable_tqdm=False, remove_unused_columns=True)
/usr/local/lib/python3.8/site-packages/transformers/modeling_auto.py:777: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.
  warnings.warn(
/usr/local/lib/python3.8/site-packages/transformers/tokenization_utils_base.py:1319: FutureWarning: The `max_len` attribute has been deprecated and will be removed in a future version, use `model_max_length` instead.
  warnings.warn(
09/13/2020 17:32:03 - INFO - filelock -   Lock 5925535648 acquired on /path/to/cached_lm_GPT2Tokenizer_1024_my_corpus_text.txt.lock
09/13/2020 17:32:03 - INFO - filelock -   Lock 5925535648 released on /path/to/cached_lm_GPT2Tokenizer_1024_my_corpus_text.txt.lock
/usr/local/lib/python3.8/site-packages/transformers/trainer.py:249: FutureWarning: Passing `prediction_loss_only` as a keyword argument is deprecated and won't be possible in a future version. Use `args.prediction_loss_only` instead.

Then after about two minutes, the process exits with the message
Killed: 9

So now I’m stuck… Does anybody know what might be wrong? This is a 2019 MacBook Pro with 16 GB of RAM and plenty of free space on the SSD. I’m an experienced Java developer, but I’m a python novice, so I might be missing something critical about the environment.

Context: @lysandre  might know this


Question:
In Bart for the summarization task, the input length is 1024 (1024 token), what does this input represents: (for example i have a document with s1, s2, s500) does this mean that we feed a sequence as sentence per sentence or the whole document must be as input as 1024 token ( all sentences must fit with truncation ) if this true doesn’t  it cause information loss ?
And if it’s sequence per sequence, let’s say 20 sentences at a time of 500, will the output at the top encoder change each time?
To be honest I’m having a difficult time imagining how the encoder is processing the document.

Context:
Hi @Hildweig
For BART max input length  is 1024 tokens. You can think of a token as as a word for simplicity (words can be split in multiple tokens as well). It’s not sentences.
Here document means a seq with max 1024 tokens. Processing longer sequences than that is still a topic of ongoing research.
And I would recommend to read the original Transformers paper (Attention is all you need ) to get an idea about how a sequence is processed by the encoder.  Or the illustrated transformers 3


Finetuning a specific task when pretrained model isn’t trained on that specific task? Using the task model vs using the base model

Question:
I want to fine tune a RobertaForSequenceClassification task on microsoft/codebert-base model. This microsoft/codebert-base model hasn’t been trained for  Sequence-Classification task.
Can I load this pre-trained model inside  a SequenceClassification function and fine tune it on my dataset?
model = RobertaForSequenceClassification.from_pretrained( "microsoft/codebert-base" )

Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at microsoft/codebert-base and are newly initialized: ['classifier.dense.weight', 'classifier.out_proj.weight', 'classifier.out_proj.bias', 'classifier.dense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

While loading I get this message which is expected as the model isn’t trained on the task and thus would not have the weights.
Can I proceed with fine-tuning this RobertaForSequenceClassification model  or would I need to define my own classifier layer on top of RobertaModel and train that?

Context:
Hi @mayanksatnalika
Yes, you can load a pre-trained base model for SequenceClassification.
RobertaForSequenceClassification  adds the classification head itself, you won’t need to do that manually. So you can fine-tune it for classification.


Current best practice for final linear classifier layer(s)?

Question:
I’m using BERT to perform text classification (sentiment analysis or NLI). I pass a 768-D vector through linear layers to get to a final N-way softmax. I was wondering what is the current best practice for the final block of linear layers?
I see in the implementation of BertForSequenceClassification that the 768-D pooled output is passed through a Dropout and a Linear layer 8.
pooled_output = outputs[1]

pooled_output = self.dropout(pooled_output)
logits = self.classifier(pooled_output)

Is this the current best practice? What about adding more linear layers, dropout, relu, batchnorm, etc.?
I am using this classifier architecture:
pooled_output = outputs[1] # 768-D
# --- start block ---
Linear (out=1000-D)
ReLU
BatchNorm
Dropout (0.25)
# --- end block ---
Linear (out=N) # Final N-way softmax

I can repeat the classifier block as many times as I want with any intermediate dimensionality. I’m worried that my knowledge of using ReLU, batchnorm, and dropout may be outdated.
Any help would be appreciated.

Context: There is already one hidden layer between the final hidden state and the pooled output you see, so the one in SequenceClassificationHead is the second one. Usually for classification head, two hidden layers are sufficient (talking about vision as well as text here), but you can certainly try more and see if you get better results.


How to get weights indicating the importance of each words in a sentence corresponding to the label

Question:
I am currently working on sentiment analyses using bert’s pooled output or classification task model in pytorch. I would want to see the importance weight of each word in a sentence which is leading me to the sentiment.
More formally, I want a  batch_size x max_length  size tensor, where each value of the matrix is the weight indicating the importance of that word on the  ith  row.
I’ve seen many suggesting to look at the 11th layer of bert but I am unable to find anything like that. Any help regarding this matter, please?

Context:
Isn’t there anybody who could help me out with this? I badly need guidance on it.
Thank you in advance.


Question:
I am using this api endpoint of gpt2 from https://huggingface.co/gpt2 3
this is given curl request,
" curl -X POST
-H “Authorization: Bearer api_xxxxxxxxxxxxxxxxxxxxxxx”
-H “Content-Type: application/json”
-d ‘“My name is Mariama, my favorite”’
https://api-inference.huggingface.co/models/gpt2 2 "
The problem is this request only specifies start of the output text. “my name is marima, my favorite” in this case.
But when using transformers you can specify max lenght of output too with
>>> generator("Hello, I'm a language model,", max_length=30, num_return_sequences=5)

is it achievable with inference api ? Probably something like below request.
" curl -X POST
-H “Authorization: Bearer api_xxxxxxxxxxxxxxxxxxxxxxx”
-H “Content-Type: application/json”
-d ‘“My name is Mariama, my favorite”’
-d “max_length = 30”
htt/api-inference.huggingface.co/models/gpt2 "

Context:
hi hexapoda,
I have no idea what your post is asking about, but the link doesn’t seem to work (I get “method not allowed” when I try to use it).
It might be good to include more detail too.


Speeding up zero shot classification [Solved]

Question:
Hi,
I was wondering if there was a way to speed up zero shot classification as outlined here 28 if I was to use pytorch directly.
For example I’m guessing this default method tokenises and pads to length 512 whereas most of my text is < 50 words. I’ve had some experience in using BertWordPieceTokenizer. So I’m guessing it would also be faster to tokenize everything in one go and send it to a pytorch model directly, rather than one by one which is what I’m guessing is happening here?
Would really appreciate even a starting point if such a thing is possible.

Context: By default it actually pads to the length of the longest sequence in the batch, so that part is efficient. The thing to keep in mind with this method is that each sequence/label pair has to be fed through the model together. So if you are running with a large # of candidate labels, that’s going to be your bottleneck. The other thing is that the default model, bart-large-mnli, is pretty big. Theoretically, the pipeline can be used with any model that has been trained on NLI 23, which you can specify with the model parameter when you create the pipeline. So you could try out some smaller models, but you probably won’t get anything to work as well as Bart or Roberta in terms of accuracy.


Question:
Hi all,
I am new to NLP. I would like to understand how model dbmdz/electra-large-discriminator-finetuned-conll03-english was fine tuned on Conll03-english dataset. Is there any available code that I can take a look? I understand it was trained by @stefan-it.
Thank you,
Sergul

Context: Hi @sergulaydore you can find the ner training code here 66


Question:
Hello everyone, I would like to ask for help with the following:
I want to fine-tuning of a language model for text generation, for this I will use run_lm_fine_tuning.py
I would like to know what is the optimal way to train the model. Can it be trained with a pure csv file, or should the file have some kind of pre-processing?
Or is it necessary to train it with an instance of the Dataset class ?
Thanks for the help

Context:
Hi @kintaro,
run_lm_fine_tuning.py is now renamed to run_language_modeling.py , you can fine it here.
The data file can be in one of the two formats

Line by line: which means each example is on it’s own line seperated by \n. Set the --line_by_line command line argument
Or just a plain text file from which examples will be sampled .

This notebook 4 also walks through how a language model can be trained.


BertForMaskedLM on a fine-tuned base model

Question:
Hello,
Is there a way for me to fine-tune the base bert/roberta architecture on a task like sequence classification, and then use the fine-tuned model as a base model for MLM predictions? I tried this by copying the state dict over from the sequence classification task into the MLM architecture, but that did not work at all. Seems like the weights that I swap from the sequence prediction task do not play well with the MLM objective.
Here is a code snippet -
#Load the fine-tuned ‘roberta-base’ model into RobertaForMaskedLM
roberta_mlm_model = RobertaForMaskedLM.from_pretrained(MODEL_FILE)
#load the default model
default_model = RobertaForMaskedLM.from_pretrained(‘roberta-base’)
#swap the weights for the head
roberta_mlm_model.lm_head.load_state_dict(default_model.lm_head.state_dict())
Can someone tell me if I am thinking in the right direction here?
Nikhil

Context: This will be problematic because the heads are not compatible. In other words, you can fine tune the model and use the weights from one model in the other, but you still have the issue that the heads are different and cannot be mapped. So after fine tuning for sequenence classification, saving model, and loading that model in a MLM version of that architecture  the LMHead will not have pretrained weights.


Question: I have a 1GB raw text dataset in a niche domain. I want to train an ELECTRA model but I couldn’t find any tutorial/examples to do so. Can anyone help me? I tried using the simpletransformers package, but it has memory issues at the moment and after a few epochs my colab session crashes.

Context: pinging @lysandre


Question: I am looking to fine-tune a BART-large model for a summarization task and I am creating a dataset to tune on.  How should I structure this dataset?  Should it have a column of text blocks and another column with associated summaries?  Or, will simply providing the raw text (the text blocks) without summaries suffice? Thanks!

Context:
Hi @Buckeyes2019
You can fine the seq2seq fine-tuning scripts here. The readme explains how the data should formatted and saved.

      GitHub

huggingface/transformers 15

🤗Transformers: State-of-the-art Natural Language Processing for Pytorch and TensorFlow 2.0. - huggingface/transformers


Question:
I fine tuned the T5 model and want to load the checkpoint. I have done this successfully for a lot of times but I got an error for one model:

INFO:transformers.tokenization_utils_base:loading file https://s3.amazonaws.com/models.huggingface.co/bert/t5-spiece.model 1 from cache at /home/t-miahu/.cache/torch/transformers/68f1b8dbca4350743bb54b8c4169fd38cbabaad564f85a9239337a8d0342af9f.9995af32582a1a7062cb3173c118cb7b4636fa03feb967340f20fc37406f021f
Traceback (most recent call last):
File “eval_checkpoint_WEnoLem_PI_fromPI.py”, line 39, in
model = T5FineTuner.load_from_checkpoint(PATH)
File “/home/t-miahu/anaconda3/envs/T5env/lib/python3.6/site-packages/pytorch_lightning/core/lightning.py”, line 1514, in load_from_checkpoint
checkpoint = torch.load(checkpoint_path, map_location=lambda storage, loc: storage)
File “/home/t-miahu/anaconda3/envs/T5env/lib/python3.6/site-packages/torch/serialization.py”, line 527, in load
with _open_zipfile_reader(f) as opened_zipfile:
File “/home/t-miahu/anaconda3/envs/T5env/lib/python3.6/site-packages/torch/serialization.py”, line 224, in init
super(_open_zipfile_reader, self).init(torch.C.PyTorchFileReader(name_or_buffer))
RuntimeError: version <= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at /pytorch/caffe2/serialize/inline_container.cc:132, please report a bug to PyTorch. Attempted to read a PyTorch file with version 3, but the maximum supported version for reading is 2. Your PyTorch installation may be too old. (init at /pytorch/caffe2/serialize/inline_container.cc:132)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7f47cc697193 in /home/t-miahu/anaconda3/envs/T5env/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: caffe2::serialize::PyTorchStreamReader::init() + 0x1f5b (0x7f472cc2b9eb in /home/t-miahu/anaconda3/envs/T5env/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #2: caffe2::serialize::PyTorchStreamReader::PyTorchStreamReader(std::string const&) + 0x64 (0x7f472cc2cc04 in /home/t-miahu/anaconda3/envs/T5env/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #3:  + 0x6c53a6 (0x7f47cd5623a6 in /home/t-miahu/anaconda3/envs/T5env/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #4:  + 0x2961c4 (0x7f47cd1331c4 in /home/t-miahu/anaconda3/envs/T5env/lib/python3.6/site-packages/torch/lib/libtorch_python.so)

frame #39: __libc_start_main + 0xe7 (0x7f47d1d25b97 in /lib/x86_64-linux-gnu/libc.so.6)

It seems the version has some problem, but why I did not have this error before? Any idea why I have no problem with other checkpoints but have error for this one?

Context: I have the same issue… Did you solve it??


Question:
Hi, just quickly getting started with GPT2.
from https://huggingface.co/gpt2 2 :
from transformers import GPT2Tokenizer, GPT2Model
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2Model.from_pretrained('gpt2')
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)

is said to yield the features of the text.
Upon inspecting the output, it is an irregularly shaped tuple with nested tensors. Looking at the source code for GPT2Model, this is supposed to represent the hidden state. I can guess what some of these dimensions represent, for example the 768 dimension is obviously the word embedding, but in general I can’t find any documentation about interpreting the information in output
I also tried adding:
output = model(**encoded_input, output_attentions = True)
but I do not know how to interpret the dimensions of this either.
I am told to “See attentions under returned tensors for more detail.” in the docstring at https://huggingface.co/transformers/_modules/transformers/modeling_gpt2.html#GPT2Model 2
But I cannot find what this is referring to. Can someone help me interpret the dimensions of these nested tuples?

Context:
Please refer to GPT2 docs 16. It will give a detailed description of what GPT2Model is supposed to return.
It returns (in order of output):

last_hidden_state  : (batch_size, sequence_length, hidden_size)

past:  (2, batch_size, num_heads, sequence_length, embed_size_per_head)

hidden_states
attentions


Does run_generation.py has a “no_repeat_ngram_size” attribute?

Question:
Good day,
I’m generating text with run_generation.py. I would like to know if there is any way to pass the attribute no_repeat_ngram_size. I have seen that attribute with model. generate but not in run_generation.py.

Context:
hi @kintaro,
run_generation.py  is used for auto-regressive generation which uses sampling.
no_repeat_ngram_size is used for beam search so run_generation.py  doesn’t have that attribute since it only uses sampling for generation


Is there any difference in the tokenized output if I load the tokenizer from a different pretrained model

Question:
So let’s say if I do
GPT2TokenizerFast.from_pretrained('gpt2-medium') vs GPT2TokenizerFast.from_pretrained('distilgpt2')
Is there actually any differences in their tokenized output?

Context: In that particular case, I don’t think so, but there are definitely cases where tokenizers from the same model type but different pretrained configurations are different. bert-base-uncased vs bert-base-cased would be one clear example.


Learning rate, LR scheduler and optimiser choice for fine-tuning GPT2

Question: I know the best choice is different depending on the actual dataset that we are fine-tuning on but I am just curious to know what combinations of learning rate, LR scheduler and optimiser have you guys found to be a good combination to train with in general? I am currently using AdamW, CosineAnnealingWarmRestarts, with a learning rate going from 0.002 to 0.0001, restarting at the end of each epoch.

Context: You can refer to TrainingArguments to look at the defaults. Link 362. They usually work well.


Gradient overflow when fine-tune t5 on CNN/DM dataset

Question:
I was trying to fine-tune t5 on CNN/DM dataset for summarization task.
I use the data based on README file in examples/seq2seq:
wget https://s3.amazonaws.com/datasets.huggingface.co/summarization/cnn_dm.tgz
tar -xzvf cnn_dm.tgz

I also successfully fine-tuned sshleifer/distilbart-cnn-12-6 on this dataset. But when I try to do it using t5-base, I receive the following error:
Epoch 1:   0%|                                                                                | 2/37154 [00:07<40:46:19,  3.95s/it, loss=nan, v_num=13]Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0
Epoch 1:   0%|                                                                                | 3/37154 [00:08<27:57:13,  2.71s/it, loss=nan, v_num=13]Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0
Epoch 1:   0%|                                                                                | 4/37154 [00:08<21:32:17,  2.09s/it, loss=nan, v_num=13]Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2048.0
Epoch 1:   0%|                                                                                | 5/37154 [00:08<17:41:05,  1.71s/it, loss=nan, v_num=13]Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1024.0
Epoch 1:   0%|                                                                                | 6/37154 [00:08<15:15:58,  1.48s/it, loss=nan, v_num=13]Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 512.0
Epoch 1:   0%|                                                                                | 7/37154 [00:09<13:24:44,  1.30s/it, loss=nan, v_num=13]Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 256.0
Epoch 1:   0%|                                                                                | 8/37154 [00:09<12:01:25,  1.17s/it, loss=nan, v_num=13]Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 128.0
Epoch 1:   0%|                                                                                | 9/37154 [00:09<10:56:27,  1.06s/it, loss=nan, v_num=13]Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 64.0
Epoch 1:   0%|                                                                                | 10/37154 [00:09<10:04:29,  1.02it/s, loss=nan, v_num=13]Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32.0
<...>
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.0
Epoch 1:   3%|███▋                                                                                | 1091/37154 [04:13<2:19:28,  4.31it/s, loss=nan, v_num=13]Traceback (most recent call last):
  File "finetune.py", line 409, in <module>
    main(args)
  File "finetune.py", line 383, in main
    logger=logger,
  File "/data/User/v3/bart/lightning_base.py", line 303, in generic_train
    trainer.fit(model)
  File "/data/User/v3/bart/venv/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1003, in fit
    results = self.single_gpu_train(model)
  File "/data/User/v3/bart/venv/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_parts.py", line 186, in single_gpu_train
    results = self.run_pretrain_routine(model)
  File "/data/User/v3/bart/venv/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1213, in run_pretrain_routine
    self.train()
  File "/data/User/v3/bart/venv/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 370, in train
    self.run_training_epoch()
  File "/data/User/v3/bart/venv/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 452, in run_training_epoch
    batch_output = self.run_training_batch(batch, batch_idx)
  File "/data/User/v3/bart/venv/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 632, in run_training_batch
    self.hiddens
  File "/data/User/v3/bart/venv/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 822, in optimizer_closure
    error = context.__exit__(a, b, c)
  File "/root/anaconda3/lib/python3.7/contextlib.py", line 119, in __exit__
    next(self.gen)
  File "/data/User/v3/bart/venv/lib/python3.7/site-packages/apex/amp/handle.py", line 123, in scale_loss
    optimizer._post_amp_backward(loss_scaler)
  File "/data/User/v3/bart/venv/lib/python3.7/site-packages/apex/amp/_process_optimizer.py", line 190, in post_backward_with_master_weights
    models_are_masters=False)
  File "/data/User/v3/bart/venv/lib/python3.7/site-packages/apex/amp/scaler.py", line 117, in unscale
    1./scale)
ZeroDivisionError: float division by zero

My code for fine-tuning is modified based on examples/seq2seq:
./finetune.sh \
    --data_dir $DATA_DIR \
    --train_batch_size=8 \
    --eval_batch_size=8 \
    --output_dir=$OUTPUT_DIR \
    --num_train_epochs 5 \
    --model_name_or_path t5-base

Can anyone provide some suggestions? Thank you!

Context: Looks like you are using fp16, currently there are few bugs with fp16 for T5 and I think those are not fixed, so try turning off fp16. Pinging @sshleifer for more info


What can cause model.generate (BART) output to be gibberish after fine-tuning?

Question:
I’m trying to fine-tune BART for paraphrasing, it’s my first time fine-tuning a model so I think I’m doing something wrong…
The input data is just pairs of sentences. I managed to get the training working, at least apparently. Here’s the code:
model = BartForConditionalGeneration.from_pretrained('facebook/bart-base')
model.train()
from transformers import AdamW
optimizer = AdamW(model.parameters(), lr=1e-5)
for i_epoch in tqdm(range(3), desc="Epoch"):
  print("Epoch #", i_epoch)
  for i_batch, data in enumerate(tqdm(train_dataloader, desc="Training batches")):
    outputs = model(data['train_ids'],
                    attention_mask=data['train_att'],
                    decoder_input_ids=data['val_ids'],
                    decoder_attention_mask=data['val_att'],
                    labels=data['val_ids']
                    )
    loss = outputs[0]
    if i_batch % 10 == 0:
      print("Batch", i_batch, " loss =", loss)
    loss.backward()
    optimizer.step()

I guess the first question is if this looks correct? The data from the dataloader is populated with the input_ids and attention_mask results from tokenizer() on the training and validation sets. I wanted to write out the training myself instead of using Trainer for learning purposes.
Looking at the loss alone, this kinda seems to work? Here’s the output:
Epoch # 0
Training batches: 100%
90/90 [3:55:25<00:00, 156.96s/it]
Batch 0  loss = tensor(14.6109, grad_fn=<NllLossBackward>)
Batch 10  loss = tensor(11.6407, grad_fn=<NllLossBackward>)
Batch 20  loss = tensor(10.6421, grad_fn=<NllLossBackward>)
Batch 30  loss = tensor(9.4968, grad_fn=<NllLossBackward>)
Batch 40  loss = tensor(8.4232, grad_fn=<NllLossBackward>)
Batch 50  loss = tensor(6.9087, grad_fn=<NllLossBackward>)
Batch 60  loss = tensor(5.8986, grad_fn=<NllLossBackward>)
Batch 70  loss = tensor(5.3631, grad_fn=<NllLossBackward>)
Batch 80  loss = tensor(4.9015, grad_fn=<NllLossBackward>)

Epoch # 1
Training batches: 100%
90/90 [1:57:47<00:00, 78.53s/it]
Batch 0  loss = tensor(4.5883, grad_fn=<NllLossBackward>)
Batch 10  loss = tensor(4.1895, grad_fn=<NllLossBackward>)
Batch 20  loss = tensor(3.7548, grad_fn=<NllLossBackward>)
Batch 30  loss = tensor(3.4811, grad_fn=<NllLossBackward>)
Batch 40  loss = tensor(3.2216, grad_fn=<NllLossBackward>)
Batch 50  loss = tensor(2.9044, grad_fn=<NllLossBackward>)
Batch 60  loss = tensor(2.3631, grad_fn=<NllLossBackward>)
Batch 70  loss = tensor(2.1639, grad_fn=<NllLossBackward>)
Batch 80  loss = tensor(1.8803, grad_fn=<NllLossBackward>)

Epoch # 2
Training batches: 100%
90/90 [1:57:34<00:00, 78.39s/it]
Batch 0  loss = tensor(1.6901, grad_fn=<NllLossBackward>)
Batch 10  loss = tensor(1.7226, grad_fn=<NllLossBackward>)
Batch 20  loss = tensor(1.2894, grad_fn=<NllLossBackward>)
Batch 30  loss = tensor(0.9937, grad_fn=<NllLossBackward>)
Batch 40  loss = tensor(0.9841, grad_fn=<NllLossBackward>)
Batch 50  loss = tensor(0.9459, grad_fn=<NllLossBackward>)
Batch 60  loss = tensor(0.6868, grad_fn=<NllLossBackward>)
Batch 70  loss = tensor(0.6640, grad_fn=<NllLossBackward>)
Batch 80  loss = tensor(0.5731, grad_fn=<NllLossBackward>)

The loss is consistently going down over the course of the training, which I thought should be a basic sign that something is happening.
But, my generate code generates repetitive gibberish instead! Here’s the generation code:
test_input = "Yesterday they went to the park, and today they will go to the store."
test_inputs = tokenizer([test_input], return_tensors='pt')
summary_ids = model.generate(
    test_inputs['input_ids'],
    num_beams=12,
    temperature=1.0,
    num_return_sequences=10,
    repetition_penalty=1.0,
    do_sample=False
  )
for res in [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summary_ids]:
  print(res)

The output is this:
For for for for for for for for for for for for for for for for for
Yesterday,,,,,,,,,,,,,,,,
The the the the the the the the the the the the the the the the the
The thethe the the the the the the the the the the the the the the
On on on on on on on on on on on on on on on on on
For for for for for for for for for for forfor for for for for for
The thethethe the the the the the the the the the the the the the
Yesterday Yesterday Yesterday Yesterday Yesterday,,,,,,,,,,,,
For for for for for for for for for for for for For for for for for
The the the the the the the the the the store store store store store store store

(In comparison, before fine-tuning, the output for this code is the same as the input)
Granted, this is fine-tuned on a small toy dataset of 10k sentence pairs mainly to test if the code runs. But seeing the output be repeated tokens like this makes me hesitant to try a bigger dataset. Surely 10k is enough data that the output should at least not be complete junk? So I’m wondering now if I did something simple wrong, or if it’s actually a data problem and I just need to train on a fuller dataset (1m+ examples?) to see results.

Context:
Do you need to zero your gradients for BART?
(I’ve not used Bart, but in training Bert I need to use model.zero_grad before passing each batch of data to the model).
Does your data look similar to the data Bart was originally trained on?   If it is totally different then your model could get worse before it gets better.   What are you hoping it will learn from your new data?


How does NER model learns from the way it is processed during training?

Question:
Hi guys, I am just getting started with NER using transformers. But I am getting some issues while understanding how the NER model is trained and how it is expected to perform during inference.
During training, tokenization and labeling is done in such a way that for each word the label is assigned to the first token only and -100 (ignore_index) to all other tokens of that word. But during inference, it is expected to predict the same label for all the tokens.
Training: https://github.com/huggingface/transformers/blob/6b4c617666fd26646d44d54f0c45dfe1332b12ca/examples/token-classification/utils_ner.py#L110-L117 7
Inference: https://huggingface.co/transformers/usage.html#named-entity-recognition
Just trying to understand how does the model learns this way. Thanks
Great ecosystem BTW

Context: ping @stefan-it @vblagoje