Question: I came to know Hugging Face use optimized onnx models for inference on cpu. I tried to do something like that using keras VGGNet16 pretrained model using keras-onnx package (see this github issue 5) but couldn’t see any performance benefits. Can I know how exactly Hugging Face is optimizing models under the hood?

Context:
These two blog posts might help

      Medium – 17 Jun 20

Accelerate your NLP pipelines using Hugging Face Transformers and ONNX Runtime 25

This post was written by Morgan Funtowicz from Hugging Face and Tianlei Wu from Microsoft

  Reading time: 6 min read

      Medium – 31 Aug 20

Faster and smaller quantized NLP with Hugging Face and ONNX Runtime 37

Popular Hugging Face Transformer models (BERT, GPT-2, etc) can be shrunk and accelerated with ONNX Runtime quantization without retraining.

  Reading time: 6 min read


T5: Tips for finetuning on crossword clues (clue => answer)

Question:
As a baseline for a research project, I am trying to finetune T5 on a large crossword clue set (130,000 clues), where the source is a Clue, and the target is an Answer.

I am using T5ForConditionalGeneration and the finetune.py script (examples/seq2seq). I started with T5-small.
My source/target files have one pair per line (<Clue>\n, <Answer>\n).
I started with from_pretrained(t5-small) for both model and tokenizer.
I didn’t add any tokens to the vocabulary.

The initial run gave me only gibberish (long strings of entirely non-English outputs), so I am trying an even simpler task: Can T5 learn to select the first word of the input sentence. I.e., I’ve modified inputs and outputs to be something like
source: This is a clue with some normal language
target: This
Where again each entry is on its own line.
I observed (under the same training regime as above, with T5-small) that, after 300 epochs, the model gives outputs that look like
<first word> <long string of gibberish>
I wonder if anyone has some ideas:

Is there any issue with having only one word targets? I.e., Should I be using a different loss function than the default? At epoch 2, my loss was already down to 0.001. Rouge scores were around 1.5.
I did not change the task name. The finetune.py script seems to default to adding task name summarization. Maybe I should remove or change the task name?
How would T5 small vs T5 base or large change the results?
I did not add separator tokens, but I think I am not required to given that the example (e.g. finetune_bart_tiny) does not add separator tokens
My inputs and outputs generally do not have punctuation (i.e. the clues don’t end in a period, and the answers don’t end in a period). I wonder if this would help?
I read the through https://discuss.huggingface.co/t/t5-finetuning-tips/684 1 but I’m not sure if those tweaks will change the results here. I’m mostly just slightly adapting the given finetune_t5.sh script.
What’s a reasonable number of epochs of finetuning (using a 60% split of 130,000, so roughly 80k training examples) before I should expect the model to learn to output the first word?

I also have an implementation question:
Is there a way to get the finetune.py script to print validation results at every epoch so that I can see how the model is learning (qualitatively) over time?

Context: I filed this bug 5 for the gibberish outputs I am observing


RAG Example and Word-Level contributions

Question:
When RAG was presented, it did so along this very nice post:

      ai.facebook.com

Retrieval Augmented Generation: Streamlining the creation of intelligent... 27

Teaching computers to understand how humans write and speak, known as natural language processing or NLP, is one of the oldest challenges in AI research. There has been…

I was wondering if there is a way to obtain the same information shown in the graphs when using HF RAG implementation. That is, the documents weights, as well as the Word-level contribution as referred in the article, or the RAG-Token document posterior as in the paper.
I am aware the document weights can be obtained when doing a forward pass, however these are not obtainable when using the generate() method, which I think would be a “nice have”. I guess now they can be obtained with an extra forward pass before generating, or just tweaking the generate method locally to return them.
However I am not sure how they obtain the posterior for each document. I am guessing it has to do with an average value from the tokens coming from each of the documents (so one would need to “split” the last hidden layer into the document chunks?). Does anyone know better how these could be obtained at generation in order to obtain similar figures as in the article?
Thanks

Context:
There’s a demo 86 by the awesome @yjernite that shows that you can get the per-examples and per-word contributions.
There’s currently a PR to open source the code of the demo here 33 where you can check the code.
Not sure we can get the RAG-Token posterior easily though


What is the license of /nlptown/bert-base-multilingual-uncased-sentiment?

Question:
Hi,
I would like to use this multilingual sentiment model (nice work by the way) in the backend of a Dataiku DSS plugin.
What is the open-source license of this model?
Cheers,
Alex Combessie

Context:
Probably best to ask the authors. I don’t think they’re active on this forum.
https://mobile.twitter.com/nlptown
https://mobile.twitter.com/yvespeirsman 2


Out of Memory on very small custom transformer

Question:
Hi everyone,
I am creating a custom transformer to classify protein sequences. These sequences range from 20 to 60000 in length in the form “ANTGGTANGT…”.
Once tokenized with a custom BPE tokenizer of vocab 1000, the longest input_ids sequence is 12000.
For the transformer I have tried Roberta and Longformer.
I got out of memory with both, even with very small parameters and splitting long sequences. For example:
Config:
config = LongformerConfig(
    vocab_size=1000,
    max_position_embeddings=5000,
    num_attention_heads=8,
    num_hidden_layers=3,
    type_vocab_size=1,
    hidden_size=64,
    intermediate_size=128
)

Model:
Net(
  (roberta): LongformerModel(
    (embeddings): RobertaEmbeddings(
      (word_embeddings): Embedding(1000, 64, padding_idx=1)
      (position_embeddings): Embedding(12000, 64, padding_idx=1)
      (token_type_embeddings): Embedding(1, 64)
      (LayerNorm): LayerNorm((64,), eps=1e-12, elementwise_affine=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )
    (encoder): LongformerEncoder(
      (layer): ModuleList(
        (0): LongformerLayer(
          (attention): LongformerAttention(
            (self): LongformerSelfAttention(
              (query): Linear(in_features=64, out_features=64, bias=True)
              (key): Linear(in_features=64, out_features=64, bias=True)
              (value): Linear(in_features=64, out_features=64, bias=True)
              (query_global): Linear(in_features=64, out_features=64, bias=True)
              (key_global): Linear(in_features=64, out_features=64, bias=True)
              (value_global): Linear(in_features=64, out_features=64, bias=True)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=64, out_features=64, bias=True)
              (LayerNorm): LayerNorm((64,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=64, out_features=128, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=128, out_features=64, bias=True)
            (LayerNorm): LayerNorm((64,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (1): LongformerLayer(
          (attention): LongformerAttention(
            (self): LongformerSelfAttention(
              (query): Linear(in_features=64, out_features=64, bias=True)
              (key): Linear(in_features=64, out_features=64, bias=True)
              (value): Linear(in_features=64, out_features=64, bias=True)
              (query_global): Linear(in_features=64, out_features=64, bias=True)
              (key_global): Linear(in_features=64, out_features=64, bias=True)
              (value_global): Linear(in_features=64, out_features=64, bias=True)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=64, out_features=64, bias=True)
              (LayerNorm): LayerNorm((64,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=64, out_features=128, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=128, out_features=64, bias=True)
            (LayerNorm): LayerNorm((64,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (2): LongformerLayer(
          (attention): LongformerAttention(
            (self): LongformerSelfAttention(
              (query): Linear(in_features=64, out_features=64, bias=True)
              (key): Linear(in_features=64, out_features=64, bias=True)
              (value): Linear(in_features=64, out_features=64, bias=True)
              (query_global): Linear(in_features=64, out_features=64, bias=True)
              (key_global): Linear(in_features=64, out_features=64, bias=True)
              (value_global): Linear(in_features=64, out_features=64, bias=True)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=64, out_features=64, bias=True)
              (LayerNorm): LayerNorm((64,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=64, out_features=128, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=128, out_features=64, bias=True)
            (LayerNorm): LayerNorm((64,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
      )
    )
    (pooler): BertPooler(
      (dense): Linear(in_features=64, out_features=64, bias=True)
      (activation): Tanh()
    )
  )
  (cls): Linear(in_features=64, out_features=1314, bias=True)
)

The out of memory error I do not get it when instanciating the model and moving it to a cuda device, I got the error always at the start of training, even with a batch size of 1.
I got 8GB of cuda memory, and previously I have been able to train pretrained bert-base, roberta-base… and more models before.
Is there something I am doing wrong? Do I need to change other parameters of the transformer architecture?
Thank you very much

Context: 5000 max_positions_embedings is still too much for a 8GB GPU, you could try using fp16


Question:
For the SpanBert model fine-tuned on TACRED dataset (ie. https://huggingface.co/mrm8488/spanbert-base-finetuned-tacred 8) the subject and object entities (ie. PERSON, ORGANIZATION, etc) are substituted by an unused token at the original code :

replace the subject and object entities by their NER
tags such as “[CLS][SUBJ-PER] was born in
[OBJ-LOC] , Michigan, . . . ”

as described in their paper:  https://arxiv.org/pdf/2004.14855.pdf 2
And can also be found in their code: https://github.com/facebookresearch/SpanBERT/blob/master/code/run_tacred.py
in lines 134 to 139:
    def get_special_token(w):
        if w not in special_tokens:
            special_tokens[w] = "[unused%d]" % (len(special_tokens) + 1)
        return special_tokens[w]

    ...

    SUBJECT_START = get_special_token("SUBJ_START")
    SUBJECT_END = get_special_token("SUBJ_END")
    OBJECT_START = get_special_token("OBJ_START")
    OBJECT_END = get_special_token("OBJ_END")
    SUBJECT_NER = get_special_token("SUBJ=%s" % example.ner1)
    OBJECT_NER = get_special_token("OBJ=%s" % example.ner2)

The issue is that to use the pre-trained models one has to substitute those tokens before tokenizing, but there is no way to obtain the originally used ones without the original data (which is not freely available).  Does anyone have access to the TACRED dataset and is able to obtain these tokens (or the special_tokens dict) using the original code and share it? And maybe add it somewhere in the repo so it can be easily accessed.
Thanks!

Context: Tagging model author @mrm8488


What’s the license of joeddav/xlm-roberta-large-xnli?

Question:
Hi again,
I would like to use this multilingual model for zero-shot classification. Super useful work btw!
Before I can embed it in my Dataiku DSS plugin, I need to check the license. Could you confirm what type of open source license this model is released with?
Cheers,
Alex Combessie

Context:
Hey, glad you’ve found it useful! The set of models it’s fine-tuned from (xlm-roberta 3) list MIT as the license, so I’ve added the same to xlm-roberta-large-xnli 6.
Also here 6 is the license for the XNLI dataset which it was fine-tuned on for reference.


Question:
I just saw that Facebook AI released a blog post about RAG 2  ( Retrieval Augmented Generation: Streamlining the creation of intelligent natural language processing models) and that it is already incorporated in the HuggingFace API 3.
I looked quickly, and I couldn’t see how to use a custom dataset with it. It seems like it will only pull down indexed datasets from HuggingFace’s AWS storage 3. I’m wondering if anyone can show me how to

Create an indexed dataset. I’m assuming this is just a big collection of embeddings that have been made by running documents through a model and taking the output embedding. I’m wondering which model(s) can be used, how many dimensions the embeddings are expected to be, and how to format all of these vectors.
Use that custom dataset with HF Rag models.

Context:
Indeed, it’s actually very simple to do with datasets and somehow explained on this page: https://huggingface.co/docs/datasets/faiss_and_ea.html 33
We will add an example script on this.


Why does RoBERTa behave differently if I provide a corpus that contains special tokens?

Question:
Hello all,
Recently I’ve been working on training from scratch a RoBERTa model starting from the code in this tutorial 1.
I am working with a specific corpus I prepared according to my own format
<s> ID 10 <i> COUNTRY  USA <i> CAPITAL Washington DC </s>
and I noticed that one of the parameters that can be passed to the tokenizer.encoder_plus function is add_special_tokens.
If add_special_tokens=True, the result of the encoding of the sentence
<s> ID 10 <i> COUNTRY  USA <i> CAPITAL Washington DC </s>
becomes
<s> <s> ID 10 <i> COUNTRY  USA <i> CAPITAL Washington DC </s> </s>
and the special_tokens_mask is 1 0 0 … 0 1
When I tried add_special_tokens=False on the same sentence
<s> ID 10 <i> COUNTRY  USA <i> CAPITAL Washington DC </s>
the result of the encoding was correct:
<s> ID 10 <i> COUNTRY  USA <i> CAPITAL Washington DC </s>
However, the special_tokens_mask  remains 0 0 … 0 0.
After testing both versions, the result I got from the first was very good, while the second failed.
This raises a few issues that I wasn’t able to solve on my own:

How can I access the special_tokens_mask to correct it to what it should be?
Where does RoBERTa make use of that mask, if it does?
Is there a method for setting the mask to something I want? e.g. the mask for <s> ID 10 <i> COUNTRY  USA </s> should be 1 0 0 1 0 0 1 if <s>, </s> and <i> should all be treated as special tokens.
If RoBERTa is not the correct model to do this, what model should I go for?

Thanks a lot!

Context: This actually feels more like a bug than a problem on your end. I suspect that the tokenization between the two is identical (i.e. <s> gets the same ID in both cases), but that means that the special_tokens_mask should also be the same. Best to wait for others who might be more certain.


Question:
Hi all,
I have been looking for a model to run a NER task in French. I see there are Camembert and RobertA models for token classification but these models are not fine-tuned for any NER tasks. Any suggestions on this? If there is not any model, is there any French dataset tagged for NER?
Thank you,
Sergul

Context:
I just asked Pedro (https://github.com/pjox 2), maybe he knows some good NER datasets for French that are publicly available for fine-tuning.
If someone could give me access to FTB dataset (see CamemBERT paper), I could fine-tune a model + upload it to the model hub


Does the default weight_decay of 0.0 in transformers.AdamW make sense?

Question:
Hi,
I have a question regarding the AdamW optimizer default weight_decay value.
In the Docs 3 we can clearly see that the AdamW optimizer sets the default weight decay to 0.0.
Given that the whole purpose of AdamW is to decouple the weight decay regularization, is my understanding that the results anyone can get with AdamW and Adam if both are used with  weight_decay=0.0  (this is, without weight decay) should be exactly the same.
Therefore, shouldn’t make more sense to have the default weight decay for AdamW > 0?
Thank you so much!

Context:
Even though I agree about the default value (it should probably be 0.01 as in the PyTorch 3 implementation), this probably should not be changed without warning because it breaks backwards compatibility. It was also implemented in transformers before it was available in PyTorch itself.
I guess it is implemented in this way, because most of the time you decide in the initialization which parameters you want to decay and which ones shouldn’t be decayed, such as here:

      github.com

    huggingface/transformers/blob/a75c64d80c76c3dc71f735d9197a4a601847e0cd/examples/contrib/run_openai_gpt.py#L230-L237 14

optimizer_grouped_parameters = [
    {
        "params": [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
        "weight_decay": args.weight_decay,
    },
    {"params": [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], "weight_decay": 0.0},
]
optimizer = AdamW(optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon)


Tips for Choosing Model: Automatic Labelling of Articles

Question:
I am looking for an advice in choosing the right model to perform

Automatic generation of labels in an Article
Any models which does “title” generation for articles can also be helpful

** Preferably I need some light-weight model to run it in my laptop **
Any suggestions are welcome!

Context: You could try to use Seq2Seq models for this (BART, T5 etc) where input seq would be the article and output seq the title. But both these models are large, T5 has a small variant t5-small which is around 230MB.


BertModel.forward() output caveat removed?

Question:
In Transformers 3.0.2 and prior, there used to be a small caveat accompanying the description of 3.0.2 BertModel.forward() 1:

Returns
pooler_output (torch.FloatTensor: of shape (batch_size, hidden_size)):
Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during pre-training.
This output is usually not a good summary of the semantic content of the input, you’re often better with averaging or pooling the sequence of hidden-states for the whole input sequence.

However, the caveat was removed in 3.1.0 (current master). The description of 3.1.0 BertModel.forward() now just says:

Returns
pooler_output (torch.FloatTensor of shape (batch_size, hidden_size)) - Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.

Is there any deeply meaningful reason why that line was removed? Is there a new important secret to that [CLS] token that you’re not telling us?

Context: That caveat was removed because this output is actually what is used for classification in the Sequence Classification model and experiments showed it was given the same kinds of results as the mean/average.


How to use Huggingface model for continuous values directly?

Question:
Hi, I have a dataset which contains continuous values [ batch_size, features ]
Features look like this :
[0.49221584, -0.021571456, -0.0920076, -0.14408934, -0.62306774]
I want to apply transformer model on these values and pass it to the final layer, something like this
batch_data ==> Transformer ==> output_layer ==> classification
Currently, I am using hand-coded multi-head attention and norm with the feed-forward network to pass these values to the transformer block.
I went through huggingface models, but all the models accept tokens and sequences, Is there any way/hack How I can use hugging face transformer models on direct continuous values?
Looking for TensorFlow/Keras quick template to start with continuous values, that’d be helpful.

Context:
I think @patrickvonplaten’s suggestion in https://github.com/huggingface/transformers/issues/6608 27 to use input_embeds would be the way to go here.
Or you could also try simply bucketizing your continuous values into buckets and embedding them, like tokens.


Question:
sentence-transformers were recently added in the model hub. I wanted to know the following if :

all these models were trained on NLI (which dataset ? MNLI)
for how many epochs ?
If I want to use it for inference on NLI task, do I need to train an additional linear layer ? So freeze the AutoModel, train a nn.Linear for 1 epoch on the mean output and then perform inference. Can you please confirm this usage ?

tfmr = AutoModel.from_pretrained("sentence-transformers/bert-base-nli-mean-tokens")

class Model(nn.Module):
    def __init__(self, model):
        super(model, self).__init__()
        self.encoder = model
        for param in self.encoder.parameters():
            param.requires_grad = False
        self.classifier = nn.Linear(768, 3)
        self.criterion = nn.CrossEntropy()

    def forward(self, inputs): # x will come from dataloader using default_data_collator

        labels = inputs.pop('labels')
        model_output = self.encoder(**inputs)
        sentence_embeddings = mean_pooling(model_output,  inputs['attention_mask'])
        logits = self.classifier(sentence_embeddings)
        loss = self.criterion(logits, labels)
        return loss, logits

model = Model(tfmr)

Context: I trained it on MNLI with the model frozen, it gets around 46% accuracy after epoch 1. Seems like it has not been trained on MNLI. Can anyone confirm ?


Question:
Have the romance language models for opus-mt  been removed? I tried it-en and ROMACE-en, but it seems they are not online:
{
“error”: “Model name ‘Helsinki-NLP/opus-mt-ROMANCE-en’ was not found in tokenizers model name list (). We assumed ‘Helsinki-NLP/opus-mt-ROMANCE-en’ was a path or url to a directory containing vocabulary files named [‘source.spm’, ‘target.spm’, ‘vocab.json’, ‘tokenizer_config.json’], but couldn’t find such vocabulary files at this path or url.”
}

Context:
Not that I know of, but try en-roa/roa-en.
It works for me locally.
Note that the inference API is out of disk space until tomorrow.


How can I use class_weights when training?

Question:
I have an unbalanced dataset. When training I want to pass class_weights so the update for rare classes is highen than for large classes. How is this possible in HF with PyTorch?
Thanks
Philip

Context:
Answering my own question:
Subclass Trainer and override the compute_loss method (see example here 423).


Generation Probabilities: How to compute probabilities of output scores for GPT2

Question:
Now that it is possible to return the logits generated at each step, one might wonder how to compute the probabilities for each generated sequence accordingly.
The following code snippet showcases how to do so for generation with do_sample=True for GPT2:
import torch
from transformers import AutoModelForCausalLM
from transformers import AutoTokenizer

gpt2 = AutoModelForCausalLM.from_pretrained("gpt2", return_dict_in_generate=True)
tokenizer = AutoTokenizer.from_pretrained("gpt2")

input_ids = tokenizer("Today is a nice day", return_tensors="pt").input_ids

generated_outputs = gpt2.generate(input_ids, do_sample=True, num_return_sequences=3, output_scores=True)

# only use id's that were generated
# gen_sequences has shape [3, 15]
gen_sequences = generated_outputs.sequences[:, input_ids.shape[-1]:]

# let's stack the logits generated at each step to a tensor and transform
# logits to probs
probs = torch.stack(generated_outputs.scores, dim=1).softmax(-1)  # -> shape [3, 15, vocab_size]

# now we need to collect the probability of the generated token
# we need to add a dummy dim in the end to make gather work
gen_probs = torch.gather(probs, 2, gen_sequences[:, :, None]).squeeze(-1)

# now we can do all kinds of things with the probs

# 1) the probs that exactly those sequences are generated again
# those are normally going to be very small
unique_prob_per_sequence = gen_probs.prod(-1)

# 2) normalize the probs over the three sequences
normed_gen_probs = gen_probs / gen_probs.sum(0)
assert normed_gen_probs[:, 0].sum() == 1.0, "probs should be normalized"

# 3) compare normalized probs to each other like in 1)
unique_normed_prob_per_sequence = normed_gen_probs.prod(-1)

Context: Can I use this to generate sequences only over a probability threshold?


Question:
Hi, wanted to use huggingface in production, but I know that some models don’t allow their use in production because of the license (example: GPL).
Can I use all of the huggingface models while respecting the Apache license on the huggingface Github?
Or should I check the main repo for that model to check the license to make sure I can use it?
Thanks!

Context: To add to the question above, what happens if a model doesn’t have a license on the repo or tag? Does it inherit the Apache 2.0 Transformer’s license?


How to create Wav2Vec2 With Language model

Question:
Now that language model boosted decoding is possible for Wav2Vec2 (https://twitter.com/PatrickPlaten/status/1468999507488788480 11 and patrickvonplaten/wav2vec2-large-xlsr-53-spanish-with-lm · Hugging Face 8
it’s important to know How can one create a Wav2Vec2 + LM repo?.
Let’s explain (hopefully this is simpler in the future):

Install kenlm:
The best guide to build kenlm is this github gist here: kenlm/BUILDING at master · kpu/kenlm · GitHub 13 IMO

Create an ngram model:
This is explained quite well here: GitHub - kpu/kenlm: KenLM: Faster and Smaller Language Model Queries 11
I wrote a short python script that allows to quickly create a ngram from a text corpus of common voice:

#!/usr/bin/env python3
from datasets import load_dataset
import os
import argparse

parser = argparse.ArgumentParser()
parser.add_argument(
    "--language", default="polish", type=str, required=True, help="Language to run comparison on. Choose one of 'polish', 'portuguese', 'spanish' or add more to this script."
)
parser.add_argument(
    "--path_to_ngram", type=str, required=True, help="Path to kenLM ngram"
)
args = parser.parse_args()

ds = load_dataset("multilingual_librispeech", f"{args.language}", split="train")

with open("text.txt", "w") as f:
    f.write(" ".join(ds["text"]))

os.system(f"./kenlm/build/bin/lmplz -o 5 <text.txt > {args.path_to_ngram}")

## VERY IMPORTANT!!!:
# After the language model is created, one should open the file. one should add a `</s>`
# The file should have a structure which looks more or less as follows:

# \data\
# ngram 1=86586
# ngram 2=546387
# ngram 3=796581
# ngram 4=843999
# ngram 5=850874

# \1-grams:
# -5.7532206      <unk>   0
# 0       <s>     -0.06677356
# -3.4645514      drugi   -0.2088903
# ...

# Now it is very important also add a </s> token to the n-gram
# so that it can be correctly loaded. You can simple copy the line:

# 0       <s>     -0.06677356

# and change <s> to </s>. When doing this you should also inclease `ngram` by 1.
# The new ngram should look as follows:

# \data\
# ngram 1=86587
# ngram 2=546387
# ngram 3=796581
# ngram 4=843999
# ngram 5=850874

# \1-grams:
# -5.7532206      <unk>   0
# 0       <s>     -0.06677356
# 0       </s>     -0.06677356
# -3.4645514      drugi   -0.2088903
# ...

# Now the ngram can be correctly used with `pyctcdecode`

See: Wav2Vec2_PyCTCDecode/create_ngram.py at main · patrickvonplaten/Wav2Vec2_PyCTCDecode · GitHub 14
Feel free to copy those lines of code. Multi-Lingual librispeech is already a very clean text corpus. You might want to pre-process other text corpora to not include any punctuation etc…
As an example this step created the Spanish 5-gram here: kensho/5gram-spanish-kenLM · Hugging Face 8

Now we should load the language model in a PyCTCBeamSearchDecoder as this is the format we need. Here one should be very careful to choose exactly the same vocabulary as the Wav2Vec2’s tokenizer vocab.

At first we should pick a fine-tuned Wav2Vec2 model that we would like to add a language model to.
Let’s choose: jonatasgrosman/wav2vec2-large-xlsr-53-spanish · Hugging Face 2
Now we instantiate a BeamSearchDecoder and save it to a folder wav2vec2_with_lm.
E.g. you can run this code:
from transformers import AutoTokenizer
from pyctcdecode import build_ctcdecoder

tokenizer = AutoTokenizer.from_pretrained("jonatasgrosman/wav2vec2-large-xlsr-53-spanish")

vocab_dict = tokenizer.get_vocab()
sorted_dict = {k: v for k, v in sorted(vocab_dict.items(), key=lambda item: item[1])}

decoder = build_ctcdecoder(
    list(sorted_dict.keys()),
    args.path_to_ngram,
)

decoder.save_to_dir("wav2vec2_with_lm")

Now we should have saved the following files in wav2vec2_with_lm:
- language_model
  - attrs.json
  - kenLM.arpa
  - unigrams.txt
- alphabet.json

That’s it! Now all you need to do is to upload this to your Wav2Vec2 model so that the directory structure looks as follows:

      huggingface.co

patrickvonplaten/wav2vec2-large-xlsr-53-spanish-with-lm at main 19

Now, all you need to change is your decoding to:

      huggingface.co

patrickvonplaten/wav2vec2-large-xlsr-53-spanish-with-lm · Hugging Face 14

Context:
This is cool!
Can you make the inference widget use the language model?


Fine Tuning BERT model on custom dataset

Question:
I am trying to fine-tune the BERT model on a custom dataset for ner task with PyTorch. I did NOT get the EXACT dataset accepted by transformers example training scripts.
The conll_2003 dataset comes in .txt files that are NOT accepted by training scripts. None of the CSV works well. Can I get a dataset that I can download and use locally without referring to the hub datasets?
Why do datasets are NOT working when we use locally?

Context:
Hi,
The scripts are meant as examples, you can easily tweak them to make it work with your local dataset.
For instance, you can load a HuggingFace Dataset with your local data (as explained in the docs 1), which could be CSV, JSON, txt, Parquet, etc.
Just make sure that you prepare your data in IOB format (as this is required for the token classification models).
If you can provide me with a small portion, I can illustrate how to make a HuggingFace Dataset with it, and make it work with the script.


Question:
I am looking at the example for torchscripting BERT-like models here: Exporting 🤗 Transformers Models. I have a basic question about the dummy inputs being passed for tracing which don’t make obvious sense to me.
The input passed is a list containing token_ids and segment_ids (or token_type_ids) which torchscript will unpack. Now, BertModel.forward() expects input_ids and attention_mask as the first and second arguments respectively. So, how why is segment_ids being passed as the second argument for both tracing and later on for inference with the loaded torchscripted model? Does it somehow work because of the flag torchscript=True that’s passed when instantiating the model? If so, how does it work?

Context: cc’ing @lewtun here


Question:
Hi @patrickvonplaten and all other wav2vec LM users,
thanks for the new LM addition. When using the following code (also without GPU), the CPU will run out of memory. Looks like something should be freed up but isn’t. This is not happening without the LM, so I guess this has something to do with pyctcdecode. Any ideas or am I missing a step?
This code will trigger the OOM on a GPU. Maybe set the range higher if needed, but memory usage should increase anyways.
from datasets import load_dataset
from transformers import Wav2Vec2ProcessorWithLM, Wav2Vec2ForCTC
import torch

dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")
processor = Wav2Vec2ProcessorWithLM.from_pretrained("patrickvonplaten/wav2vec2-base-100h-with-lm")
model = Wav2Vec2ForCTC.from_pretrained("patrickvonplaten/wav2vec2-base-100h-with-lm").to("cuda")
print("* * * * * * loaded models")

for i in range(200):
    audio_sample = dataset[i]
    print(" * * * * Sample: ", i)
    inputs = processor(audio_sample["audio"]["array"], sampling_rate=audio_sample["audio"]["sampling_rate"], return_tensors="pt").to("cuda")
    with torch.no_grad():
      logits = model(**inputs).logits
    transcription = processor.batch_decode(logits.cpu().numpy()).text
    print(transcription[0].lower())

And a CPU version, will just run slower OOM
from datasets import load_dataset
from transformers import Wav2Vec2ProcessorWithLM, Wav2Vec2ForCTC
import torch

dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")
processor = Wav2Vec2ProcessorWithLM.from_pretrained("patrickvonplaten/wav2vec2-base-100h-with-lm")
model = Wav2Vec2ForCTC.from_pretrained("patrickvonplaten/wav2vec2-base-100h-with-lm")
print("* * * * * * loaded models")

for i in range(200):
    audio_sample = dataset[i]
    print(" * * * * Sample: ", i)
    inputs = processor(audio_sample["audio"]["array"], sampling_rate=audio_sample["audio"]["sampling_rate"], return_tensors="pt")
    with torch.no_grad():
      logits = model(**inputs).logits
    transcription = processor.batch_decode(logits.numpy()).text
    print(transcription[0].lower())

Context: And solved on Github, a missing pool.close() in the processor code.


M2M model finetuning on multiple language pairs

Question:
Hi all, can anyone please help in suggesting how to finetune m2m100 on more than one pair? I am able to finetune for one lang pair using the below script:
CUDA_VISIBLE_DEVICES=0,1,2,3,6 python -m torch.distributed.run --nproc_per_node=5 run_translation.py --model_name_or_path=m2m100_418M_new_token --do_train --do_eval --source_lang ja --target_lang en --fp16=True --evaluation_strategy epoch --output_dir bigfrall --per_device_train_batch_size=48 --per_device_eval_batch_size=48 --overwrite_output_dir --forced_bos_token “en” --train_file orig_manga/orig/train_exp_frame_50k.json --validation_file orig_manga/orig/valid_exp_frame_50k.json --tokenizer_name tokenizer_new_token --num_train_epochs 50 --save_total_limit=5 --save_strategy=epoch --load_best_model_at_end=True --predict_with_generate
But, now I want to finetune it on ja-en and ja-zh pairs. How to pass these both languages?

Context: I’m also curious about this. @nikhiljais - did you ever work this out?


CUDA out memory only when performing hyperparameter search

Question:
I  am working with a GTX3070, which only has 8GB of GPU RAM. When I am running using trainer.train(), I run fine with a maximum batch size of 7 (6 if running in Jupiter notebook). However, when I attempt to run in a hyperparameter search with ray, I get CUDA out of memory every single time.
I am wondering why this could be the case.
Here is my code. Sorry if it’s a little long. It’s based off the following Jupiter notebooks:

notebooks/text_classification.ipynb at master · huggingface/notebooks · GitHub
Transformers-Tutorials/Fine_tune_TrOCR_on_IAM_Handwriting_Database_using_Seq2SeqTrainer.ipynb at master · NielsRogge/Transformers-Tutorials · GitHub

def model_init():
        if args.pretrained_checkpoint:
            model = VisionEncoderDecoderModel.from_pretrained(
                args.pretrained_checkpoint
            )
        else:
            model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained(
                args.encoder_checkpoint, args.decoder_checkpoint
            )

        # set special tokens used for creating the decoder_input_ids from the labels
        model.config.decoder_start_token_id = tokenizer.cls_token_id
        model.config.pad_token_id = tokenizer.pad_token_id
        # make sure vocab size is set correctly
        model.config.vocab_size = model.config.decoder.vocab_size

        # set beam search parameters
        model.config.eos_token_id = tokenizer.sep_token_id
        model.config.max_length = 64
        model.config.early_stopping = True
        model.config.no_repeat_ngram_size = 3
        model.config.length_penalty = 2.0
        model.config.num_beams = 4
        return model

    def compute_metrics(pred, verbose=args.verbose_inference) -> Dict[str, float]:
        labels_ids = pred.label_ids
        pred_ids = pred.predictions

        pred_str = tokenizer.batch_decode(pred_ids, skip_special_tokens=True)
        labels_ids[labels_ids == -100] = tokenizer.pad_token_id
        label_str = tokenizer.batch_decode(labels_ids, skip_special_tokens=True)
        if verbose:
            print(pred_str)

        # TODO: The following package from datasets load_metric
        # cer = cer_metric.compute(predictions=pred_str, references=label_str)
        cer = sum([editdistance.eval(a, b) for a, b in zip(pred_str, label_str)]) / sum(
            [len(b) for b in label_str]
        )

        return {"cer": cer}

    training_args = Seq2SeqTrainingArguments(
        num_train_epochs=100,
        predict_with_generate=True,
        evaluation_strategy="epoch",
        per_device_train_batch_size=3,  # 7 max for py, 6 max for ipynb
        per_device_eval_batch_size=3,
        fp16=True,  # set to false if turning off gpu
        output_dir=args.logging_dir,
        save_strategy="epoch",
        save_total_limit=10,
        logging_steps=1000,
        learning_rate=1e-4,
        load_best_model_at_end=True,
        report_to="wandb",
    )

    # instantiate trainer
    trainer = Seq2SeqTrainer(
        model_init=model_init,
        tokenizer=feature_extractor,
        args=training_args,
        compute_metrics=compute_metrics,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        data_collator=default_data_collator,
    )

    def hp_space(trial) -> Dict[str, float]:
        # backend for ray
        return {
            "learning_rate": tune.loguniform(1e-6, 1e-4),
            "num_train_epochs": tune.choice(list(range(1, 6))),
            "seed": tune.uniform(1, 40),
            "per_device_train_batch_size": 1,
        }

    if args.hyperparameter_search:
        trainer.hyperparameter_search(
            hp_space=hp_space,
            backend="ray",
            n_trials=10,
            # search_alg=HyperOptSearch(metric="objective", mode="max"),
            # scheduler=ASHAScheduler(metric="loss", mode="min"),
            # fail_fast=True,
            max_failures=-1,
            name="testing_run_hellobro",
        )
    else:
        trainer.train()

Context: I encountered the same problem so +1 (post only for more attention)


How to force bos_token_id for each example individually in MBart?

Question: Say I have a batch of examples with fields of input_ids of size m*n and bos_token_id of size n. Is there a way that I could specify the bos_token_id for each example during the evaluation step when using generate?

Context: I’m also curious about this. @mralexis  - did you ever work this out? It seems like a similar question was also asked here: M2M model finetuning on multiple language pairs which also had no reply.


Question:
Hi community,
I have tried the convert_graph_to_onnx.py script to convert one transformer model from PyTorch to ONNX format. I have a few questions :

I have installed onnxrutime-gpu. Does the model generated with the script will be functionning only with GPU or will it work also with CPU onnx rutile? So, do I have to generate one onnx model per device?

Does the ONNX model dependant of the hardware it bas been generated from or do I have to generate the ONNX model on the target hardware where will be run the inference ?

Are the outputs of the ONNX model identical wherever hardware the inference is run on? So, can I use the embeddings generated from the ONNX model but from different hardware platforms?

How can I apply quantization on ONNX model for both CPU and GPU devices ? It seems that the --quantize flag IS deprecated, and I can’t manage to apply dynamical quantize on my ONNX model.

Thanks!

Context:
Hi @Matthieu,
Here’s a few tentative answers to your questions (I’m somewhat new to using ONNX):

If you are not running optimisations like dynamic quantisation, then the resulting ONNX model should work on both CPU and GPU. You can see in the source code here 12 that convert_graph_to_onnx.py has a convert function that relies on e.g. the native ONNX module in PyTorch. ONNX Runtime is only used in the optimize function here 6.
Similar to question 1, if you are not applying any optimisations then my understanding is that the resulting model should be hardware-independent (this is meant to be the whole benefit of having a universal format like ONNX :))
Interesting question. I don’t know the answer, but my naive guess is that it depends on which hardware accelerator you’re using (e.g. ONNX Runtime vs something else).
As far as I know, quantisation for GPU is not supported (see issue here 13), but it definitely is for CPU. What kind of trouble are you running into? One thing you can try is using ONNX Runtime directly with an exported ONNX model as follows:

from onnxruntime.quantization import quantize_dynamic, QuantType
model_input = "onnx/model.onnx"
model_output = "onnx/model.quant.onnx"
quantize_dynamic(model_input, model_output, weight_type=QuantType.QInt8)

There’s also some useful notebooks on the ONNX Runtime repo that I found useful: onnxruntime/PyTorch_Bert-Squad_OnnxRuntime_GPU.ipynb at master · microsoft/onnxruntime · GitHub 25


Question:
Our goal is to train Bert for NER using a json dataset much like Conll2007 but with additional features:

Distance from top of document (top)
Distance from left edge of document (left)
Note that I don’t want to do this with embeddings because I think using these values in training will get much better results.  It seems like it should be easy to add the features but nothing seems to work. so this link 2 is not what I am looking for.

I created a new object for BertForTokenClassification with a new forward().  The new forward() has 4 additional parameters (pos, chunk, top, left) and I also added these to the outputs call:
class AeepBertForTokenClassification(BertForTokenClassification):
    def forward(
        self,
        input_ids=None,
        attention_mask=None,
        token_type_ids=None,
        position_ids=None,
        pos=None,
        chunk=None,
        top=None,
        left=None,
        head_mask=None,
        inputs_embeds=None,
        labels=None,
        output_attentions=None,
        output_hidden_states=None,
        return_dict=None,
    ):
        r"""
        labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`):
            Labels for computing the token classification loss. Indices should be in ``[0, ..., config.num_labels -
            1]``.
        """
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        outputs = self.bert(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            pos=pos,
            chunk=chunk,
            top=top,
            left=left,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,

        )

        sequence_output = outputs[0]

        sequence_output = self.dropout(sequence_output)
        logits = self.classifier(sequence_output)

        loss = None
        if labels is not None:
            loss_fct = CrossEntropyLoss()
            # Only keep active parts of the loss
            if attention_mask is not None:
                active_loss = attention_mask.view(-1) == 1
                active_logits = logits.view(-1, self.num_labels)
                active_labels = torch.where(
                    active_loss, labels.view(-1), torch.tensor(loss_fct.ignore_index).type_as(labels)
                )
                loss = loss_fct(active_logits, active_labels)
            else:
                loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))

        if not return_dict:
            output = (logits,) + outputs[2:]
            return ((loss,) + output) if loss is not None else output

        return TokenClassifierOutput(
            loss=loss,
            logits=logits,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )

This results in the error message:
Running tokenizer on prediction dataset: 100%|██████████| 26/26 [00:03<00:00,  6.96ba/s]
[INFO|trainer.py:540] 2021-11-04 09:00:04,306 >> The following columns in the training set  don't have a corresponding argument in `AeepBertForTokenClassification.forward` and have been ignored: pos_tags, id, tokens, ner_tags, chunk_tags.
[INFO|trainer.py:1196] 2021-11-04 09:00:04,334 >> ***** Running training *****
[INFO|trainer.py:1197] 2021-11-04 09:00:04,334 >>   Num examples = 1287
[INFO|trainer.py:1198] 2021-11-04 09:00:04,334 >>   Num Epochs = 3
[INFO|trainer.py:1199] 2021-11-04 09:00:04,334 >>   Instantaneous batch size per device = 8
[INFO|trainer.py:1200] 2021-11-04 09:00:04,334 >>   Total train batch size (w. parallel, distributed & accumulation) = 8
[INFO|trainer.py:1201] 2021-11-04 09:00:04,334 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:1202] 2021-11-04 09:00:04,334 >>   Total optimization steps = 483
  0%|          | 0/483 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd.py", line 1483, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/cccc/PycharmProjects/aegis-ml/aegis-ml-ner-trainer/mainSingleThread2.py", line 702, in <module>
    main()
  File "/Users/cccc/PycharmProjects/aegis-ml/aegis-ml-ner-trainer/mainSingleThread2.py", line 610, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/Users/cccc/PycharmProjects/aegis-ml/venv-source/lib/python3.8/site-packages/transformers/trainer.py", line 1316, in train
    tr_loss_step = self.training_step(model, inputs)
  File "/Users/cccc/PycharmProjects/aegis-ml/venv-source/lib/python3.8/site-packages/transformers/trainer.py", line 1849, in training_step
    loss = self.compute_loss(model, inputs)
  File "/Users/cccc/PycharmProjects/aegis-ml/venv-source/lib/python3.8/site-packages/transformers/trainer.py", line 1881, in compute_loss
    outputs = model(**inputs)
  File "/Users/cccc/PycharmProjects/aegis-ml/venv-source/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/cccc/PycharmProjects/aegis-ml/aegis-ml-ner-trainer/mainSingleThread2.py", line 87, in forward
    outputs = self.bert(
  File "/Users/cccc/PycharmProjects/aegis-ml/venv-source/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'pos'
python-BaseException
[ERROR|tokenization_utils_base.py:954] 2021-11-04 10:20:38,330 >> Using bos_token, but it is not set yet.
[ERROR|tokenization_utils_base.py:964] 2021-11-04 10:20:38,338 >> Using eos_token, but it is not set yet.

Self.bert() seems to be a pyTorch module object (module.py) which, of course, does not know that we have added features (pos, etc) but it seems to expect only  kwargs and not expect specific parameters.  So I don’t know what I need to do to pass these new parameters on through forward().  I can’t tell for sure what exactly it is calling when I trace it.
Can someone help me with adding these features?

Context:
@g3casey I also read your comments in this thread 1. I don’t have a solution for this and believe that it is a very important issue to consider for Huggingface experts. Adding several extra features (in my case, semantic features) should be feasible more easily by just adjusting some parameters maybe. And specially for token classification tasks.
@nielsr @sgugger


Question:
Hello there
I am trying to run the script run_mlm.py to perform the training of a BERT model. Basically, the idea is to start from an already existing italian BERT model, perform a second training on a specific topic (biomedical texts), and later fine-tune it on a QuestionAnswering downstream task.
I was able to run the run_mlm.py script, both with and without –line_by_line parameter. I have a couple of questions, if you could kindly answer or point me to somewhere in the documentation:

The run with –line_by_line took like 10x longer that the one without, why? I have full access to the complete dataset, and I can organize it as I want, so which is the best format?

Is there a way to feed the model more files, if my corpus is split on several of them?

Does this script train the model for the NSP task as well?

If I evaluate the model, I get the perplexity score. Is there a way to get accuracy for the NSP task? (I think that accuracy does not make sense for MLM, right?)

Many thanks for your patience

Context:
Ok so, regarding point 2 (load more files at the same time), I tried to tweak the code a bit (line 280 of the run_mlm.py original script).
ORIGINAL CODE:
    else:
        data_files = {}
        if data_args.train_file is not None:
            data_files["train"] = data_args.train_file
            extension = data_args.train_file.split(".")[-1]
        if data_args.validation_file is not None:
            data_files["validation"] = data_args.validation_file
            extension = data_args.validation_file.split(".")[-1]
        if extension == "txt":
            extension = "text"
        raw_datasets = load_dataset(extension, data_files=data_files, cache_dir=model_args.cache_dir)

MY EDITS:
    else:
        data_files = {}
        if data_args.train_file is not None:
            files_full_path = []
            for filepath in os.listdir(data_args.train_file):
                files_full_path.append(data_args.train_file+filepath)
            data_train_from_file = []
            for el in files_full_path:
                data_train_from_file.append(el)
            data_files["train"] = data_train_from_file
            extension = files_full_path[0].split(".")[-1]
        if data_args.validation_file is not None:
            data_files["validation"] = data_args.validation_file
            extension = data_args.validation_file.split(".")[-1]
        if extension == "txt":
            extension = "text"
        raw_datasets = load_dataset(extension, data_files=data_files, cache_dir=model_args.cache_dir)

Do you think this approach can make sense? I am trying the code right now and it seems to work. My concern is that when I will load the entire dataset (several GBs of text data), the training will crash/be extremely long (several weeks).
Thanks everyone


Inference of finetuned wav2vec2-xls-r-300m model using the ASR pipeline does not remove special tokens

Question:
I have finetuned a wav2vec2-xls-r-300m model on Hindi language using the mozilla-foundation/common_voice_7_0 dataset. But when I infer the model using the ASR pipeline special tokens are not skipped. And even in the Huggingface demo special characters are retained.
Model: shivam/xls-r-hindi (shivam/xls-r-hindi · Hugging Face 1)
Another example of same issue in different model: Harveenchadha/vakyansh-wav2vec2-hindi-him-4200 (Harveenchadha/vakyansh-wav2vec2-hindi-him-4200 · Hugging Face 1)
The tokenizer of wav2vec2-xls-r-300m is “Wav2Vec2CTCTokenizer” and in the asr pipeline if “CTC” is present in tokenizer class name then skip_special_tokens is set to False (transformers/automatic_speech_recognition.py at v4.15.0 · huggingface/transformers · GitHub 1), because of this special tokens are included in the output when using asr pipeline.

Screenshot from 2022-01-21 11-18-50751×90 13.4 KB

Screenshot from 2022-01-21 17-14-32660×516 34.8 KB

Context:
We need to not skip special tokens for CTC (wav2vec2 in particular) because of the [PAD] token.
HELLO can only be transcribed because the CTC tokens are H, E, L, PAD, L, L, L, O, O for instance.
It seems here that maybe <s> got confused with <pad> and hence is not properly skipped during decoding. Could that be it ? Also I am unsure to know how it would behave on the unicode scripts you are using (I hope if we remove <s> by properly using the <pad> token everything will work magically, but I can’t be sure.


Finetuning T5 large for paraphrasing multiple time with the same parameters and data gives different results

Question:
I use this script to fine-tune T5 large for paraphrasing (Google Colab 2).
And i do fine-tuning 2 times from scratch, using the same data and the same parameters and expected that the results are the same. However the results are quite different.
Is there any way to set for example seed, so I can get the same result if everything else is the same.
I am not sure if this means that the results should be the same?
def set_seed(seed):
    random.seed(seed)
    np.random.seed(seed)
 torch.manual_seed(seed)

set_seed(42)

Context:
Hi @zokica , are you using colab pro for finetuninig t5-large model?, In my case I am getting RAM issues.
Is there a way to finetune it on free colab instance?


AttributeError: ‘str’ object has no attribute ‘item’ - Bert Fine tuning

Question:
Hi, I am working on a intent classification problem, so I am fine tuning bert for it,
here is my code:-
import random
import numpy as np

# This training code is based on the `run_glue.py` script here:
# https://github.com/huggingface/transformers/blob/5bfcd0485ece086ebcbed2d008813037968a9e58/examples/run_glue.py#L128

# Set the seed value all over the place to make this reproducible.
seed_val = 42

random.seed(seed_val)
np.random.seed(seed_val)
torch.manual_seed(seed_val)
torch.cuda.manual_seed_all(seed_val)

# We'll store a number of quantities such as training and validation loss,
# validation accuracy, and timings.
training_stats = []

# Measure the total training time for the whole run.
total_t0 = time.time()

# For each epoch...
for epoch_i in range(0, epochs):

    # ========================================
    #               Training
    # ========================================

    # Perform one full pass over the training set.

    print("")
    print('======== Epoch {:} / {:} ========'.format(epoch_i + 1, epochs))
    print('Training...')

    # Measure how long the training epoch takes.
    t0 = time.time()

    # Reset the total loss for this epoch.
    total_train_loss = 0

    # Put the model into training mode. Don't be mislead--the call to
    # `train` just changes the *mode*, it doesn't *perform* the training.
    # `dropout` and `batchnorm` layers behave differently during training
    # vs. test (source: https://stackoverflow.com/questions/51433378/what-does-model-train-do-in-pytorch)
    model.train()

    # For each batch of training data...
    for step, batch in enumerate(train_dataloader):

        # Progress update every 40 batches.
        if step % 40 == 0 and not step == 0:
            # Calculate elapsed time in minutes.
            elapsed = format_time(time.time() - t0)

            # Report progress.
            print('  Batch {:>5,}  of  {:>5,}.    Elapsed: {:}.'.format(step, len(train_dataloader), elapsed))

        # Unpack this training batch from our dataloader.
        #
        # As we unpack the batch, we'll also copy each tensor to the GPU using the
        # `to` method.
        #
        # `batch` contains three pytorch tensors:
        #   [0]: input ids
        #   [1]: attention masks
        #   [2]: labels
        b_input_ids = batch[0].to(device)
        b_input_mask = batch[1].to(device)
        b_labels = batch[2].to(device)

        # Always clear any previously calculated gradients before performing a
        # backward pass. PyTorch doesn't do this automatically because
        # accumulating the gradients is "convenient while training RNNs".
        # (source: https://stackoverflow.com/questions/48001598/why-do-we-need-to-call-zero-grad-in-pytorch)
        model.zero_grad()

        # Perform a forward pass (evaluate the model on this training batch).
        # The documentation for this `model` function is here:
        # https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification
        # It returns different numbers of parameters depending on what arguments
        # arge given and what flags are set. For our useage here, it returns
        # the loss (because we provided labels) and the "logits"--the model
        # outputs prior to activation.
        loss, logits = model(b_input_ids,
                             token_type_ids=None,
                             attention_mask=b_input_mask,
                             labels=b_labels)

        # Accumulate the training loss over all of the batches so that we can
        # calculate the average loss at the end. `loss` is a Tensor containing a
        # single value; the `.item()` function just returns the Python value
        # from the tensor.
        print(loss)
        total_train_loss += loss.item()

        # Perform a backward pass to calculate the gradients.
        loss.backward()

        # Clip the norm of the gradients to 1.0.
        # This is to help prevent the "exploding gradients" problem.
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

        # Update parameters and take a step using the computed gradient.
        # The optimizer dictates the "update rule"--how the parameters are
        # modified based on their gradients, the learning rate, etc.
        optimizer.step()

        # Update the learning rate.
        scheduler.step()

    # Calculate the average loss over all of the batches.
    avg_train_loss = total_train_loss / len(train_dataloader)

    # Measure how long this epoch took.
    training_time = format_time(time.time() - t0)

    print("")
    print("  Average training loss: {0:.2f}".format(avg_train_loss))
    print("  Training epcoh took: {:}".format(training_time))

    # ========================================
    #               Validation
    # ========================================
    # After the completion of each training epoch, measure our performance on
    # our validation set.

    print("")
    print("Running Validation...")

    t0 = time.time()

    # Put the model in evaluation mode--the dropout layers behave differently
    # during evaluation.
    model.eval()

    # Tracking variables
    total_eval_accuracy = 0
    total_eval_loss = 0
    nb_eval_steps = 0

    # Evaluate data for one epoch
    for batch in validation_dataloader:

        # Unpack this training batch from our dataloader.
        #
        # As we unpack the batch, we'll also copy each tensor to the GPU using
        # the `to` method.
        #
        # `batch` contains three pytorch tensors:
        #   [0]: input ids
        #   [1]: attention masks
        #   [2]: labels
        b_input_ids = batch[0].to(device)
        b_input_mask = batch[1].to(device)
        b_labels = batch[2].to(device)

        # Tell pytorch not to bother with constructing the compute graph during
        # the forward pass, since this is only needed for backprop (training).
        with torch.no_grad():

            # Forward pass, calculate logit predictions.
            # token_type_ids is the same as the "segment ids", which
            # differentiates sentence 1 and 2 in 2-sentence tasks.
            # The documentation for this `model` function is here:
            # https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification
            # Get the "logits" output by the model. The "logits" are the output
            # values prior to applying an activation function like the softmax.
            (loss, logits) = model(b_input_ids,
                                   token_type_ids=None,
                                   attention_mask=b_input_mask,
                                   labels=b_labels)

        # Accumulate the validation loss.
        total_eval_loss += loss.item()

        # Move logits and labels to CPU
        logits = logits.detach().cpu().numpy()
        label_ids = b_labels.to('cpu').numpy()

        # Calculate the accuracy for this batch of test sentences, and
        # accumulate it over all batches.
        total_eval_accuracy += flat_accuracy(logits, label_ids)

    # Report the final accuracy for this validation run.
    avg_val_accuracy = total_eval_accuracy / len(validation_dataloader)
    print("  Accuracy: {0:.2f}".format(avg_val_accuracy))

    # Calculate the average loss over all of the batches.
    avg_val_loss = total_eval_loss / len(validation_dataloader)

    # Measure how long the validation run took.
    validation_time = format_time(time.time() - t0)

    print("  Validation Loss: {0:.2f}".format(avg_val_loss))
    print("  Validation took: {:}".format(validation_time))

    # Record all statistics from this epoch.
    training_stats.append(
        {
            'epoch': epoch_i + 1,
            'Training Loss': avg_train_loss,
            'Valid. Loss': avg_val_loss,
            'Valid. Accur.': avg_val_accuracy,
            'Training Time': training_time,
            'Validation Time': validation_time
        }
    )

print("")
print("Training complete!")

print("Total training took {:} (h:mm:ss)".format(format_time(time.time()-total_t0)))

I am getting a error:-
AttributeError                            Traceback (most recent call last)
<ipython-input-27-a6f23d2754c8> in <module>()
     92         # from the tensor.
     93         print(loss)
---> 94         total_train_loss += loss.item()
     95
     96         # Perform a backward pass to calculate the gradients.

AttributeError: 'str' object has no attribute 'item'

Context: Here is the collab:- Google Colab


Does starting training from a previous checkpoint reset the learning rate?

Question:
Hi,
I want to start training a new model by loading a previous model I trained, I want to know what happens to the learning rate in this case – does it start at the learning rate I set, or does it start from the prev learning rate of the checkpoint?

Context:
Hi!
As far as I have experienced, it continues from the last learning rate number saved in the checkpoint from where you resumed.
It also starts from the last epoch number.


Question:
We have just fixed the T5 fp16 issue for some of the T5 models!
(Announcing it here, since lots of  users were facing  this issue and T5 is one most widely used model in the library)
TL;DR:
Previously, there was an issue when using T5 models in fp16; it was producing nan loss and logits. Now on the master, this issue is fixed for the following T5 models and versions. Now you should be able to train and use these models for inference in fp16  and see a decent speed-up!

T5v1 : t5-small , t5-base , t5-large

T5v1_1 : google/t5-v1_1-small , google/t5-v1_1-base

MT5 : google/mt5-small , google/mt5-base

For those of you who are interested, here’s a description of what was causing nan loss and how it is fixed.
t5-small was the only T5 model that was working in fp16. The rest of the models produce nan  loss/logits.
for all the models and versions (v1, v1.1, mT5), at some point, we get inf  values in hidden_states after applying the final linear layer (wo) in T5DenseReluDense and T5DenseGatedGeluDense.

      github.com

    huggingface/transformers/blob/02e05fb0a532e572b56ba75dad6ba3db625bbdeb/src/transformers/models/t5/modeling_t5.py#L248-L278 29

class T5DenseReluDense(nn.Module):    def __init__(self, config):        super().__init__()        self.wi = nn.Linear(config.d_model, config.d_ff, bias=False)        self.wo = nn.Linear(config.d_ff, config.d_model, bias=False)        self.dropout = nn.Dropout(config.dropout_rate)    def forward(self, hidden_states):        hidden_states = self.wi(hidden_states)        hidden_states = F.relu(hidden_states)        hidden_states = self.dropout(hidden_states)        hidden_states = self.wo(hidden_states)        return hidden_statesclass T5DenseGatedGeluDense(nn.Module):    def __init__(self, config):        super().__init__()        self.wi_0 = nn.Linear(config.d_model, config.d_ff, bias=False)        self.wi_1 = nn.Linear(config.d_model, config.d_ff, bias=False)

  This file has been truncated. show original

which results in nan values in T5LayerNorm.
Also for t5-large, t5-v1_1-base, t5-v1_1-large, there are inf values in the output of T5LayerSelfAttention and T5LayerCrossAttention, specifically where we add the attn output with the hidden_states

      github.com

    huggingface/transformers/blob/02e05fb0a532e572b56ba75dad6ba3db625bbdeb/src/transformers/models/t5/modeling_t5.py#L548 10

        normed_hidden_states = self.layer_norm(hidden_states)        attention_output = self.SelfAttention(            normed_hidden_states,            mask=attention_mask,            position_bias=position_bias,            head_mask=head_mask,            past_key_value=past_key_value,            use_cache=use_cache,            output_attentions=output_attentions,        )        hidden_states = hidden_states + self.dropout(attention_output[0])        outputs = (hidden_states,) + attention_output[1:]  # add attentions if we output them        return outputsclass T5LayerCrossAttention(nn.Module):    def __init__(self, config):        super().__init__()        self.EncDecAttention = T5Attention(config, has_relative_attention_bias=False)        self.layer_norm = T5LayerNorm(config.d_model, eps=config.layer_norm_epsilon)        self.dropout = nn.Dropout(config.dropout_rate)

      github.com

    huggingface/transformers/blob/02e05fb0a532e572b56ba75dad6ba3db625bbdeb/src/transformers/models/t5/modeling_t5.py#L584 7

            normed_hidden_states,            mask=attention_mask,            key_value_states=key_value_states,            position_bias=position_bias,            head_mask=head_mask,            past_key_value=past_key_value,            use_cache=use_cache,            query_length=query_length,            output_attentions=output_attentions,        )        layer_output = hidden_states + self.dropout(attention_output[0])        outputs = (layer_output,) + attention_output[1:]  # add attentions if we output them        return outputsclass T5Block(nn.Module):    def __init__(self, config, has_relative_attention_bias=False):        super().__init__()        self.is_decoder = config.is_decoder        self.layer = nn.ModuleList()        self.layer.append(T5LayerSelfAttention(config, has_relative_attention_bias=has_relative_attention_bias))

This happens during both training and inference, to reproduce
fix
To avoid inf values we could clamp the hidden_states to the max values for the current data type if there are inf in it. i.e
if torch.isinf(hidden_states).any():
    clamp_value = torch.finfo(hidden_states.dtype).max - 1000
    hidden_states = torch.clamp(hidden_states, min=-clamp_value, max=clamp_value)

we need to add this after self attn, cross-attn, and the feed-forward layer which is where the inf values occur. This works for both apex and amp
To verify this fix, trained t5-base, t5-v1_1-base and t5-v1_1-small on cnn/dm for 10k steps (1.11 epochs)
Here’s the training command, to run this  navigate to examples/seq2seq dir, follow the instructions in the readme to download cnn_dm and dataset, and then run the following command
export M=google/t5-v1_1-base
export OUT_DIR=t5-v1_1-base-cnn-fp16
export DATA_DIR=cnn_dm

python finetune_trainer.py \
    --model_name_or_path $M \
    --data_dir $DATA_DIR \
    --output_dir $OUT_DIR --overwrite_output_dir \
    --max_steps=10000 \
    --gradient_accumulation_steps=8 \
    --learning_rate=1e-4 \
    --per_device_train_batch_size=4 \
    --n_val 500 \
    --max_target_length=56 --val_max_target_length=128 \
    --fp16 --fp16_backend apex \
    --do_train --do_eval --evaluation_strategy steps \
    --logging_steps=100 --logging_first_step --eval_steps=2500 --save_steps=2500 --save_total_limit=2 \
    --sortish_sampler \

for evaluation
python run_eval.py \
    t5-v1_1-base-cnn-fp16  cnn_dm/test.source hypothesis.txt \
    --reference_path cnn_dm/test.target \
    --score_path metrics.json \
    --device cuda:0 \
    --prefix summarize: \
    --bs 16 \
    --fp16 \

and got the following metrics (ROUGE2)

for t5-base: 19.2804
for t5-v1.1-base: 18.4316
(note that the score for t5-base is more because it’s already pre-trained on cnn/dm)

To compare this, evaluated the pre-trained t5-base in both fp32 and fp16, which gave the following results

fp16: 18.3681

fp32: 18.394

So the results are close enough.
To verify the fix for  t5-large, evaluated the pre-trained t5-large in fp32 and fp16 (use the same command above to evaluate t5-large)  and got the following results

fp16: 19.2734

fp32: 19.2342

Surprisingly, rouge2 is slightly better in fp16.
So with the above fix, the following model types now work in fp16 (opt level 01), and give descent speed-up

T5v1: t5-small, t5-base, t5-large

T5v1_1: google/t5-v1_1-small, google/t5-v1_1-base

MT5: google/mt5-small, google/mt5-base

One interesting observation,
For inference, the  t5-base fine-tuned with fp16 and evaluated in fp32 is faster (~1.31x) than pre-trained t5-base evaluated in fp16.  See this colab 49

Context:
Nice fix!
The speed discrepancy might be because of different length generations.


Can trainer.predict() return multiple generations for each sample?

Question:
I am looking for a similar feature as in model.generate() which takes a parameter num_return_sequences. It decides how many generations should be returned for each sample. It is especially useful when using beam search and analyzing the effect of beam search on the metrics.
Trainer.predict() does not seem to support this feature. Is there a workaround? I can use model.generate() but then it was very slow last time because I have to create a for loop iterating over batches whereas trainer.predict automatically handles the data loading separating

Context: Hi @berkayberabi! I have a similar problem. Did you manage to solve this issue?


Question:
Hey there everyone, I haven’t used this amazing API for a while. The code below is able to pool the [CLS] output embedding from DistilBERT?
distilbert = DistilBertModel.from_pretrained('distilbert-base-uncased', return_dict=True)

# more code ...

# input_ids & attention_mask from a batch
outputs = distilbert(input_ids, attention_mask=attention_mask)
seq_embeddings = outputs[0] # DistilBERT output embeddings
cls_embedding = seq_embeddings[:,0,:] # <--- [CLS] output embedding

I’ve taken this idea from the following DPR code:

      github.com

    huggingface/transformers/blob/master/src/transformers/models/dpr/modeling_dpr.py#L204

          ) -> Union[BaseModelOutputWithPooling, Tuple[Tensor, ...]]:
              outputs = self.bert_model(
                  input_ids=input_ids,
                  attention_mask=attention_mask,
                  token_type_ids=token_type_ids,
                  inputs_embeds=inputs_embeds,
                  output_attentions=output_attentions,
                  output_hidden_states=output_hidden_states,
                  return_dict=return_dict,
              )
              sequence_output, pooled_output = outputs[:2]
              pooled_output = sequence_output[:, 0, :]
              if self.projection_dim > 0:
                  pooled_output = self.encode_proj(pooled_output)

    if not return_dict:
                  return (sequence_output, pooled_output) + outputs[2:]

    return BaseModelOutputWithPooling(
                  last_hidden_state=sequence_output,
                  pooler_output=pooled_output,

Context: did you get a solution for your question, please


Fixing the random seed in the Trainer does not produce the same results across runs

Question:
Hi folks,
I’ve noticed that fixing the seed in the Trainer does not produce the same results across multiple training runs.
For example, suppose we fix the seed in the TrainingArguments and instantiate a model and trainer as follows:
batch_size = 16
model_checkpoint = "distilbert-base-uncased"
model = AutoModelForSequenceClassification.from_pretrained(model_checkpoint)

args = TrainingArguments(
    "test-glue",
    evaluation_strategy = "epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    num_train_epochs=3,
    seed=123
)

trainer = Trainer(
    model,
    args,
    train_dataset=train_ds,
    eval_dataset=valid_ds,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics
)

If I fine-tune this on COLA, I get the following results
Epoch	Training Loss	Validation Loss	Matthews Correlation	Runtime	Samples Per Second
1	0.517900	0.467562	0.455635	0.740500	1408.496000
2	0.335300	0.500026	0.490934	0.686200	1519.946000
3	0.232300	0.618692	0.493626	0.693100	1504.833000

Now suppose we re-instantiate the model and trainer, and fine-tune:
model = AutoModelForSequenceClassification.from_pretrained(model_checkpoint)
trainer = Trainer(
    model,
    args,
    train_dataset=train_ds,
    eval_dataset=valid_ds,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics
)
trainer.train()

I would expect to get exactly the same results, but instead find small differences in the outputs:
Epoch	Training Loss	Validation Loss	Matthews Correlation	Runtime	Samples Per Second
1	0.519000	0.473168	0.458883	0.702900	1483.911000
2	0.335800	0.502644	0.486363	0.701400	1487.070000
3	0.229200	0.616023	0.497130	0.710800	1467.258000

My best guess is that the DataLoader is the source of the difference since it uses a RandomSampler. However, I would have thought that fixing the seed would also fix that.
Although these differences are tiny, having reproducible runs really helps stay sane during debugging and I’m wondering whether anyone here knows how to fix this?
For context, I’m working in Jupyter notebooks and here is a small Colab notebook from which I produced the above numbers: https://colab.research.google.com/drive/15nv40o81JfKwubFBVOjZkqk8bRauXig8?usp=sharing 7

Context: For full reproducibility you need to instantiate your model inside the Trainer by using the model_init argument (or setting a seed before instantiating your model). You have random weights in your model head and those are different in your two runs in the code you show.


Question:
Hey guys,
I am currently using the Trainer 4 in order to train my DistilBertForSequenceClassification.
My problem: I want to stepwise print/save the loss and accuracy of my training set by using the Trainer. Is there a way to do so?
What I did so far: I have adjusted compute_metrics. But this function is only carried out on my evaluation set. I need the same for my training set. I also tried out the TrainerCallback 2. But I can’t access the current predictions of the model by using the predefined callbacks.
Another idea would be to customize the Trainer using a custom train function. But firstly I want to ask you whether there is an easier way to do so.
Thank you in advance!

Context: There is no way to do this directly in the Trainer, it’s just not built that way (because evaluation is often pretty slow). You should twek the code in your own subclass of Trainer to add a self.evaluate(self.train_dataset) at the appropriate line and then handle the logging.


Question:
Hi,
I am following this fantastic notebook to fine-tune a multi classifier.
Context:

I am using my own dataset.
Dataset is a CSV file with two values, text and label.
Labels are all numbers.
I have 7 labels.
When loading the pre-trained model, I am assigning num_labels=7.

from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased",num_labels=7)

When training, I am receiving this error:
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)
   2844     if size_average is not None or reduce is not None:
   2845         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2846     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
   2847
   2848

IndexError: Target 7 is out of bounds.

I have tried changing the number of labels to 2 and 5 and that didn’t solve the issue. Still getting out of bounds error.
Training arguments:
from transformers import TrainingArguments, Trainer

training_args = TrainingArguments(
    output_dir="./results",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=5,
    weight_decay=0.01,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokinized_jobs["train"],
    eval_dataset=tokinized_jobs["test"],
    tokenizer=tokenizer,
    data_collator=data_collator,
)

trainer.train()

and here is how tokenized data look like
DatasetDict({
    train: Dataset({
        features: ['attention_mask', 'input_ids', 'label', 'text', 'token_type_ids'],
        num_rows: 1598
    })
    test: Dataset({
        features: ['attention_mask', 'input_ids', 'label', 'text', 'token_type_ids'],
        num_rows: 400
    })
})

Sample:
{
'attention_mask': [1, 1, 1, 1, 1, 1, 1],
 'input_ids': [101, 1015, 1011, 2095, 3325, 6871, 102],
 'label': 2,
 'text': '1-year experience preferred',
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0]
}

I tried it on Colab with GPU and TPU.
Any idea what is the issue?

Context:
I found the solution. It was an indexing issue with my labels.
My labels were starting from 1 to 8, I changed them to 0…7 and that fixed the issue for me.
Credit to this answer on Stackoverflow.
Hope this will help someone in the future.


How to add a custom argument to TrainingArguments?

Question:
I’m using my own loss function with the Trainer. I need to pass a custom criterion I wrote that will be used in the loss function to compute the loss. I have the following setup:
from transformers import Trainer, TrainingArguments

class MyTrainer(Trainer):
    def compute_loss(self, model, inputs, return_outputs=False):
        # I compute the loss here and I need my `criterion`
        return loss

training_args = TrainingArguments(# the arguments...
)

# model = my model...

trainer = MyTrainer(model=model,
                    args=training_args,
                    # rest of the arguments...
)

I wonder if there is any way I can pass my custom criterion object to the Trainer either through the Trainer or TrainingArguments? Or, what is the best way to use my criterion without changing the Trainer?

Context:
If I understand your scenario correctly you are creating your own child class, using the Trainer class as parent class, is that right? If so, you should be able to add any arguments you want to the child class, since it is yours to amend, no?
Just like in this example: How do I add arguments to a subclass in Python 3 - Stack Overflow 3
Let me know if this is helpful at all or if I complete misunderstood your question
Cheers
Heiko


Question:
I set the early stopping callback in my trainer as follows:
trainer = MyTrainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        compute_metrics=compute_metrics,
        callbacks=[EarlyStoppingCallback(3, 0.0)]
    )

the values for this callback in the TrainingArguments are as follows:
load_best_model_at_end=True,
metric_for_best_model=eval_loss,
greater_is_better=False

What I expect is that the training will continue as long as the eval_loss metric continues to drop. While the training will stop only when the eval_loss has not dropped for more than 3 epochs and the best model will be loaded.
During the training I get these values for the eval_loss:
epoch1: 'eval_loss': 0.8832499384880066
epoch2: 'eval_loss': 0.6109879612922668
epoch3: 'eval_loss': 0.52149897813797
epoch4: 'eval_loss': 0.48024266958236694

therefore, as it always drops, I would expect the training to continue. Instead the training stopped after 4 epochs and during the evaluation it uploaded the model related to the first epoch, where the eval_loss had the greatest value, as you can see in the following:
01/26/2021 11:08:57 - INFO - __main__ -  ***** Eval results *****
01/26/2021 11:08:57 - INFO - __main__ -    eval_loss = 0.8832499384880066

Am I wrong to set some parameters?
Thanks!
EDIT: to clarify, I also printed the TrainerState values at the end of the training:
log_history=[
{'eval_loss': 0.837020993232727, 'eval_accuracy_score': 0.8039973127309372, 'eval_precision': 0.7904381747255738, 'eval_recall': 0.7808047316067748, 'eval_f1': 0.7855919213776935, 'eval_runtime': 8.375, 'eval_samples_per_second': 67.343, 'epoch': 1.0, 'step': 411}, {'loss': 1.5377, 'learning_rate': 4.6958980235865466e-05, 'epoch': 1.22, 'step': 500},
{'eval_loss': 0.6051444411277771, 'eval_accuracy_score': 0.8406953308700034, 'eval_precision': 0.8297104717236403, 'eval_recall': 0.8243570212384622, 'eval_f1': 0.8270250831610176, 'eval_runtime': 8.3919, 'eval_samples_per_second': 67.208, 'epoch': 2.0, 'step': 822}, {'loss': 0.6285, 'learning_rate': 4.3917595505563304e-05, 'epoch': 2.43, 'step': 1000},
{'eval_loss': 0.5184187889099121, 'eval_accuracy_score': 0.856567013772254, 'eval_precision': 0.8464932024849194, 'eval_recall': 0.8425486154673358, 'eval_f1': 0.8445163028833199, 'eval_runtime': 8.4159, 'eval_samples_per_second': 67.016, 'epoch': 3.0, 'step': 1233}, {'loss': 0.4561, 'learning_rate': 4.087621077526113e-05, 'epoch': 3.65, 'step': 1500},
{'eval_loss': 0.46523478627204895, 'eval_accuracy_score': 0.868743701713134, 'eval_precision': 0.8599369085173502, 'eval_recall': 0.8550049287570571, 'eval_f1': 0.8574638267277793, 'eval_runtime': 8.3682, 'eval_samples_per_second': 67.398, 'epoch': 4.0, 'step': 1644}, {'train_runtime': 1783.4323, 'train_samples_per_second': 4.609, 'epoch': 4.0, 'step': 1644}
],
best_metric=0.837020993232727

as you can also see from here, the best_metric is the value of the val_loss of the first epoch and not the lowest among the epochs it has done (which are still few because the value is always decreasing and therefore the training should not even stop …).

Context: I’m trying to reproduce your issue, but on my side, the best_metric is correct and decreasing. Could you check you are using the latest version of Transformers and post the way you are creating your TrainingArguments?


How much data were used to pre-train facebook/wav2vec2-base

Question:
The Original problem is here 1
I changed the pre-training model facebook/wav2vec2-base-100k-voxpopuli from to facebook/wav2vec2-base 1. It works. Everything is ok. However [facebook/wav2vec2-base-100k-voxpopuli] was pretrained on 100k hours data, I don’t know how much data were used to pretrain facebook/wav2vec2-base。 Maybe there is something wrong in facebook/wav2vec2-base-100k-voxpopuli?

Context:
Hey @zzuczy,
We could indeed have explained it better on the model card. As you can see in the official paper: https://arxiv.org/pdf/2006.11477.pdf the model was pretrained on 960h hours of Librispeech


Question:
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

above code raise error requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/api/bert/models/bert 9
the whole call stack is
Traceback (most recent call last):
  File "key_gen_search.py", line 20, in <module>
    from reader import dataset_str
  File "/home/jack/prjs/TSMH/key_gen/reader.py", line 9, in <module>
    from utils import keyword_pos2sta_vec
  File "/home/jack/prjs/TSMH/key_gen/utils.py", line 18, in <module>
    bert_scorer = BERT_Scorer(config.bert_path)
  File "../bert/bert_scorer.py", line 11, in __init__
    self.tokenizer = BertTokenizer.from_pretrained(pretrained)
  File "/home/jack/.local/share/virtualenvs/key_gen-eKA6qkca/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1649, in from_pretrained
    fast_tokenizer_file = get_fast_tokenizer_file(
  File "/home/jack/.local/share/virtualenvs/key_gen-eKA6qkca/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 3425, in get_fast_tokenizer_file
    all_files = get_list_of_files(
  File "/home/jack/.local/share/virtualenvs/key_gen-eKA6qkca/lib/python3.8/site-packages/transformers/file_utils.py", line 1730, in get_list_of_files
    model_info = HfApi(endpoint=HUGGINGFACE_CO_RESOLVE_ENDPOINT).model_info(
  File "/home/jack/.local/share/virtualenvs/key_gen-eKA6qkca/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 503, in model_info
    r.raise_for_status()
  File "/home/jack/.local/share/virtualenvs/key_gen-eKA6qkca/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/api/bert/models/bert

Context: @jackliusr  Did you ever resolve this issue? Facing same error


Question: Is there any codebase in huggingface that could be used to pretrain T5 model? Looking into the examples dir in the repo there is nothing mentioned about T5. Thanks!

Context: Still need help on this…


How do I change the classification head of a model?

Question:
Hello,
I like to change the number of labels that a trained model has. I am loading a model that was trained on 17 classes and I like adapt this model to my own task. Now if I simply change the number of labels like this:
model_checkpoint ="vblagoje/bert-english-uncased-finetuned-pos"
model = AutoModelForTokenClassification.from_pretrained(model_checkpoint,num_labels=2)

I get an error saying:
RuntimeError: Error(s) in loading state_dict for BertForTokenClassification:
        size mismatch for classifier.weight: copying a param with shape torch.Size([17, 768]) from checkpoint, the shape in current model is torch.Size([2, 768]).
        size mismatch for classifier.bias: copying a param with shape torch.Size([17]) from checkpoint, the shape in current model is torch.Size([2]).

My question is: How do I replace the classification head?
Thanks a lot

Context:
The reason is: you are trying to use mode, which has already pretrained on a particular classification task. You have to remove the last part ( classification head) of the model.
This is actually a kind of design fault too. In practice
( BERT base uncased + Classification ) = new Model .
is your model. Now, if you want to reuse them on a different tasks, either use BERT base uncased or extract that part from new Model.


Question:
Hi there,
I was searching for code references, blogs and any sort of resources that can help me in fine-tuning XLS-R on multiple languages all at once using huggingface.
All help is highly appreciated
@patrickvonplaten

Context:
We don’t have much content for this at the moment sadly! It does sound like a very cool project though. I think the closest thing that I can think of that might help is this paper:
[2109.11680] Simple and Effective Zero-shot Cross-lingual Phoneme Recognition 1 (we have 2 of the models merged to the Hub)


Trainer option to disable saving DeepSpeed checkpoints

Question:
I’d like to ask for opinions about adding a Trainer configuration option to disable saving of DeepSpeed checkpoints (potentially only keeping the model weights).
Context: I’m finetuning gpt-j-6b for basic translation phrases on consumer hardware (128GB System RAM and Nvidia GPU with 24GB RAM). I use the DeepSpeed Zero optimizer, stages 2 and 3 so 99% of my system memory is fully allocated (I also have a huge swap file on an NVMe).
Arguments for adding this flag:

The DeepSpeed checkpoints are huge (60GB+) and take a long time to save. Because my system RAM is used by the DeepSpeed optimizer, I can’t use local storage (no system ram available for buffers) so I have to transfer over the network to a NAS.
The DeepSpeed save_checkpoint code (as of v0.5.8) is very hungry for system RAM and may even have some memory leaks as after 10+ attempts I’ve never been able to save more than three checkpoints without the linux kernel deciding to OOM the finetuning python process. Out of frustration, I’ve modified the trainer code by commenting out all the calls to deepspeed.save_checkpoint(output_dir) and surprise: I’ve been able to finetune and have saved the model weights 15+ times without getting OOMed.

The default state for this option would obviously be to save everything. I read the previous topic on the subject: Disable checkpointing in Trainer 1 which is nice but does not exactly cover my use case.

Context:
Well, if you start from weights-only you waste training resources as your optimizer will take time to get to a point where you stopped at. So when you resume training you typically want to resume the optimizer states and not start from scratch. That’s the whole point of saving intermediary checkpoints. Shuffling data shouldn’t make any difference to wanting ongoing optim states.
You don’t need to change any DS code, you just need to set:
stage3_gather_fp16_weights_on_model_save=false
in ds_config file and it won’t gather and save the fp16 weights. And you can then extract the perfect fp32 weights at the end of your training using zero_to_fp32.py script.
Of course do a very short training and try using zero_to_fp32.py script and ensure that you know that this is what you want.
So the proposal is this:

don’t gather any zero3 weights and use additional CPU memory to build a state_dict and just dump the intermediary states to disk really fast (should be ~0 CPU RAM overhead)
resume training from checkpoint until you have finished your training - i.e. repeat this step as many times as you need to
extract the final fp32 weights with zero_to_fp32.py

If something is unclear please don’t hesitate to ask for further clarifications, @mihai


How should I handle pre/post-processing with slow tokenizers for tasks like NER and question answering?

Question:
How should folks using slow tokenizers perform pre/post processing tasks for tasks like question answering and token classification … both of which, at least from the course, appear heavily dependent on the fast-tokenizer only methods word_ids() and sequence_ids().
Also, I’m curious to know why the slow tokenizers don’t have word_ids and sequence_ids methods … and if there is a way we can get at, or build, the equivalent of them for slow tokenizers?
Thanks much!

Context: There are no easy model-agnostic way to tackle those tasks for slow tokenizers, so you should really use a fast one for those tasks.


Question:
Hi everyone,
I want to fine-tune the AutoModelWithLMHead model from this repository, which is a German GPT-2 model.
I have prepocessed a bunch of text passages for the fine-tuning, but when beginning training, I receive the following error (copied with a little context):
File "GPT\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "GPT\lib\site-packages\transformers\models\gpt2\modeling_gpt2.py", line 774, in forward
    raise ValueError("You have to specify either input_ids or inputs_embeds")
ValueError: You have to specify either input_ids or inputs_embeds

It’s asking for either input ids or embeddings, which I thought I provided by instantiating the trainer. Here’s my code for the preparation of the model:
# Load data
with open("Fine-Tuning Dataset/train.txt", "r", encoding="utf-8") as train_file:
    train_data = train_file.read().split("--")

with open("Fine-Tuning Dataset/test.txt", "r", encoding="utf-8") as test_file:
    test_data = test_file.read().split("--")

# Load pre-trained tokenizer and prepare input
tokenizer = AutoTokenizer.from_pretrained('dbmdz/german-gpt2')

tokenizer.pad_token = tokenizer.eos_token
train_input = tokenizer(train_data, padding="longest")
test_input = tokenizer(test_data, padding="longest")

# Define model

model = AutoModelWithLMHead.from_pretrained("dbmdz/german-gpt2")
training_args = TrainingArguments("test_trainer")

# Evaluation

metric = load_metric("accuracy")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = numpy.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

# Train
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_input,
    eval_dataset=test_input,
    compute_metrics=compute_metrics,
)
trainer.train()
trainer.evaluate()

Does anyone know the cause for this? Any help is gladly appreciated! Thank you.

Context: It looks like your train_input is not a dataset containing the "input_ids", as expected by the model. Look at train_input[0] for instance to see which keys it contains.


Do trainer.save_model saves the best model?

Question: I have set load_best_model_at_end to True for the Trainer class. And I want to save the best model in a specified directory. Does the method save_model of Trainer saves the best model or the last model in the specified directory?

Context: save_model itself does what it say on the can: saves the model, good, bad, best it does not matter. It’s the rotate checkpoints method that will keep the best model from being deleted.


Question:
Hi folks,
When I am running a lot of quick and dirty experiments, I do not want / need to save any models as I’m usually relying on the metrics from the Trainer 3 to guide my next decision.
One thing that slows down my iteration speed is the fact that the Trainer will save a checkpoint after some number of steps, defined by the save_steps parameter in TrainingArguments 11.
To disable checkpointing, what I currently do is set save_steps to some large number, but my question is whether there is a more elegant way to do this? For example, is there a Trainer argument I can set that will disable checkpointing altogether?
Thanks!

Context:
There is none for now. We could definitely add a save_strategy like  the evaluation_strategy that could take the values no/steps/epoch.
If you want to tackle this in a PR that would be a welcome contribution!


How to use only one bert to do generation task with ‘past_key_values’ mechanism？

Question:
I really like the rich text generation APIs of this project, especially the 'past_key_values' mechanism, which makes the generation process efficiently. I use UniLM, sadly it's not implemented in huggingface, and I'm eager to implement UniLM with 'past_key_values' mechanism, but has encountered a lot of difficulties.
The structure of UniLM is virtually the same as Bert, except the mask type of attention, so first I tried 'BertForMaskedLM', but it's forward function does't support 'past_key_values'. Then I tried 'BertModel', but the shape of past_key_values it returns are strange, so at the next step of decoding, input the past_key_values into the generate function causes:

--> 930  past_key_values_length = past_key_values[0][0].shape[2] if past_key_values is not None else 0
IndexError: tuple index out of range

what's the simplest way to implement UniLM, and used the rich APIs for text generation especially the 'past_key_values' mechanism? Please help me, thank you very much!

Context: Any updates？


Question:
For people interested in tools for logging and comparing different models and training runs in general, Weights & Biases 158 is directly integrated with  Transformers.
You just need to have wandb installed and logged in.
It automatically logs losses, metrics, learning rate, computer ressources, etc.
image1176×718 56.5 KB
Here is another cool example 90 where I ran a sweep to fine-tune GPT-2 on tweets.
image1281×957 678 KB
Finally you can use your runs to create cool reports. See for example my huggingtweets report 52.
See documentation 158 for more details or this colab 122.
At the moment it is integrated with Trainer and TFTrainer.
If you use Pytorch Lightning, you can use WandbLogger. See Pytorch Lightning documentation 25.
Let me know if you have any questions or ideas to make it better!

Context:
@boris I have a few questions for the HF Transformers integration:

It looks like wandb is charting the loss, learning rate, and epoch for a given run of Trainer.train(). Are there other things that would be useful to have charted for a finetuning run?

It also looks like wandb is using the logging_steps value in TrainerArguments. Is this right?

Is it preferred to set wandb behavior through the environment variables or in the finetuning script directly?


Precision vs recall when using transformer models?

Question:
Hi,
I am  using a fine-tuned bert model for text classification and, of course, I am getting very high accuracy scores. However, this masks a more nuanced reality when I look at the precision/recall metrics. See, for instance:
              precision    recall  f1-score   support

           0      0.962     0.976     0.969     75532
           1      0.516     0.395     0.448      4820

    accuracy                          0.941     80352
   macro avg      0.739     0.686     0.708     80352
weighted avg      0.935     0.941     0.938     80352

My question is: for a given transformers architecture (say Bert) do we know what we can do to improve either precision or recall (I know there is a tradeoff between the two). For instance is there evidence that oversampling the most infrequent class could improve precision? What can be done here, beyond having more training data?
Thanks!

Context: You can use class weights in your loss function if one label is more probable than the other or if your dataset is imbalanced. There’s plenty of tutorials on this so it shouldn’t be too hard to find.