Issue in the Documentation of transformers for BiET

Question:
Hello,
I have been reading the documentation of Biet model here 2. In the section pooler output this is what is written

Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for the BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining

After going through the code (here)[transformers/modeling_beit.py at master · huggingface/transformers · GitHub], the pooler output is actually the mean of all hidden states and not a linear projection on the CLS token.
Is it possible to update the documentation as it creates confusion while going through it?
Note: I thought of raising the issue in the GitHub repo but couldn’t find how to do it incase of documentation.

Context:
Hi,
Thanks for reporting. Indeed, BeitModel currently returns an output of type BaseModelOutputWithPooling. This is a generic class that automatically generates the documentation for the model, defined here 1. However, in this case, it might be better to define a custom BeitModelOutput, that better describes the outputs of the model.
Do you mind opening a PR for this? This would mean defining a new dataclass within modeling_beit.py.
Otherwise, I’ll do it


Question:
I am trying to run run_translation.py with mt5-large and DeepSpeed enabled. I use ds_config_zero3.json as the config file. However, when I try to run this, I get the following error:

ValueError: fp16 is enabled but the following parameters have dtype that is not fp16: lm_head.weight

Is there some config setting I’m missing that could help resolve this issue?

Context:
Hey did you figure out to resolve this? I’d be interested to learn what you did.
I ran the ASR example here, and it ran fine, but I noticed it had fp16 set to false. If I try to save memory by passing --fp16 at the command line or manually invoking fp16=True when calling TrainingArguments, I get the same error you report.


Importance of padding for tokens and same size inputs for transformers

Question:
We usually pad our inputs/tokens to the transformers to have the same size (e.g. 512 in bert)
I have a general question regarding padding to have a fix size (in your case 512).
would transformers be able to learn if inputs are not pad to fix sizes?
lets say I have batch size of B, I can fix the size of the data to be the same as the one with the largest sample in each batch, in this case in each iteration the batch will have different sizes.
But I was wondering if it will actually work for transformers? and if not what is the reason and why?
I really appreciate it if someone that know the answer or tried it before can help me here.
Thanks
S

Context:
Hey @seyeeet
I have found a tutorial where the input pipeline is just the way you have described to be. Here the inputs are padded to the biggest sequence in the batch, where each batch has different padding lengths.
Tutorial link: भाषा की समझ के लिए ट्रांसफार्मर मॉडल  |  Text  |  TensorFlow 2
Hope this helps


Question:
Hi all,
I’m using the run_mlm.py 1 script.
My evaluation set is “too large” (i.e., takes too long to run through the entire evaluation set, every n steps of training); so I was hoping to sub-sample from the evaluation set during training.
Is there a simple way to extend or use the  Trainer with custom logic for sub-sampling examples from the provided eval_dataset? In the worst case, I can manually specify a subset of the eval set to be fed into the Trainer, but I was hoping to do a random subsample for each in-training evaluation so that I don’t overfit to one sub-sample of the evaluation set.
Thanks!

Context: Why do you not give a smaller evaluation dataset? You can then run trainer.evaluate(full_eval_dataset) to evaluate on the full validation dataset.


Why does ignore_mismatched_sizes increase the number of TfAlbertMainLayer parameters?

Question:
If I load  the model from pretrained without much in the way of configs, I get about 11 million parameters in the albert main layer. If I load it but change the problem type and set it to ignore mismatched sizes, the main layer has 222 million parameters. This seems strange to me, I thought changing the problem type would only effect the classifier?
model = TFAlbertForSequenceClassification.from_pretrained('albert-base-v2', config=AlbertConfig(problem_type="single_label_classification"), ignore_mismatched_sizes=True)

Context:
Hi! The problem here is that your config object has default layer numbers and sizes that are totally different from the ones in albert-base-v2. If you’d like to train a sequence classification model on top of Albert, you can just do:
model = TFAlbertForSequenceClassification.from_pretrained('albert-base-v2', num_labels=2)

Alternatively, if you want to use a config object, you should initialize it from albert-base-v2 like this:
model = TFAlbertForSequenceClassification.from_pretrained('albert-base-v2', config=AlbertConfig.from_pretrained('albert-base-v2', problem_type="single_label_classification"))


Question:
Is it possible to include sample weights in the compute_metrics function used by Trainer?
For simpler cases, I can just encode the gold labels for some samples as -1 and give them a weight of 0 when computing the metrics, but in other cases it might be trickier.

Context: I don’t think it’s possible. For elaborate evaluations you can pick on your predictions after using trainer.predict then do something more sophisticated on them.


How to fine-tune BERT model for NER if forward method doesn’t have “labels” argument

Question:
I want to fine-tune a german basic BERT model on the task of NER. Therefore I want to load a model like this. But when I load it like this
model = AutoModel.from_pretrained("dbmdz/bert-base-german-cased", num_labels=len(unique_tags))
and then give it the training data like this
outputs = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)
I get this error:
TypeError: forward() got an unexpected keyword argument 'labels'
When I check which kind of class AutoModel uses, it is BertForMaskedLM, which makes sense. So the forward method of this class doesn’t take labels. But how can I then fine-tune it for this specific task?
I don’t want to use a model from the hub for token classification or NER because I have my special kind of labels I want to use for specific tokens. Therefore something like this isn’t an option I guess.
Thank you very much!

Context: Instead of AutoModel you’ll probably just need AutoModelForTokenClassification. You can also have a look at the existing examples with NER: transformers/examples/pytorch/token-classification at master · huggingface/transformers · GitHub 2


Why TFBlenderbot SmallModel and TFBlenderbot SmallForConditionalGeneration are the same trainable_variables?

Question:
i downloaded model from facebook / blenderbot_small-90M , and loaded it with BlenderbotSmallTokenizer.from_pretrained() and BlenderbotSmallForConditionalGeneration.from_pretrained() respectively. When i looked into trainable_variables using:
for v in model.trainable_variables():
	print(v)

i found they were equal, but doc says there is a a language modeling head in TFBlenderbotSmallForConditionalGeneration, how can i get the weights of the head?

Context:
if last head dense in TFBlenderbotSmallForConditionalGeneration uses the kernel which weights shared with embeddings, how about the bias?

Checking the PyTorch implementation, it seems that the language modeling head doesn’t use a bias, as seen here.

how can i get all the variables of the model include trainable and untrainable variables?

In PyTorch, you can get all parameters of a model as follows:
for name, param in model.named_parameters():
     print(name, param.shape)

cc’ing @Rocketknight1 for how to do this in Tensorflow.


Why save_steps should be a round multiple of eval_steps when load_best_model_at_end=True?

Question:
I am confused about these three arguments, as explained here 1 by @sgugger save_steps doesn’t care about the best model, so if I set eval_steps=100 and save_steps=200, every 200 steps, there is a checkpoint (200, 400, 600, …) but every 100 steps we have an evaluation of our model (100, 200, 300, …). Now, if the evaluation in 300 is the best, it will not be saved and is lost.
But if we set load_best_model_at_end=True and keep the eval_steps=100, save_steps=200, eval_steps will override the save_steps because it will save a checkpoint every 100 steps so it could load the best model at the end.
Here is the question: If all I said is true, why when load_best_model_at_end=True is set, save_steps should be a round multiple of eval_steps? It doesn’t make sense because when load_best_model_at_end is True, the model doesn’t care about save_steps and saves every eval_steps.

Context: You are entirely right! It’s a bug in the Trainer which should be fixed by this PR 3.


Trainer using Checkpoint makes TPU crash

Question:
Hi,
I am running a Bert from scratch on google cloud, and it is working.
But when I am doing the training using a saved Checkpoint with “model_name_or_path” it makes the TPU crash with SIGKILL error (memory issue I guess).
I don’t understand this behavior, since if I rerun it without the checkpoint it works with no problems.
Running with checkpoint works only if I use just 1 core (num_cores=1) which is not convenient for me (takes a much larger time).
Anyone have an idea to help me?
Thanks

Context: You might need more RAM to be able to resume from a checkpoint. The core of the issue is that the optimizer state is loaded on each TPU before being transferred to the XLA device (it can’t be directly loaded on the XLA device sadly) but since you have 8 processes loading it, it’s loaded 8 times on CPU.


Question:
I able to train on a word level, after that i test with fill-mask pipeline and get below warning
I get this error :
Some weights of BertModel were not initialized from the model checkpoint at ./output_model and are newly initialized: ['bert.pooler.dense.weight', 'bert.pooler.dense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

tokenizer = PreTrainedTokenizerFast(tokenizer_file="./my-tokenizer.json")
model = BertForMaskedLM(config=BertConfig(vovab_size=1000000))
< after training>

fill_mask = pipeline(
    "fill-mask",
    model="./output_model",
    tokenizer=tokenizer
) # output warning above

Context:
Hi,
I also get the same warning, while using AutoModelForMaskedLM on a fill-mask pipeline. Even though I finetuned it with AutoModelForMaskedLM…
Having randomly initialized layers should not be good for using the model. Is there any help to solve it?


How to convert model output logits into string sentences during training to check what the model is outputting?

Question:
I’m training GPT-2, constructed in the following manner:
                configuration = GPT2Config(
                    output_hidden_states=True,
                )
                architecture = AutoModelForCausalLM.from_config(
                    configuration)

When I pass an input sequence into the model, like so, I can access its output logits:
            # Output type: BaseModelOutputWithPastAndCrossAttentions
            architecture_output = self.architecture(
                input_ids=intractn.task_input_ids,
                attention_mask=intractn.task_attention_masks)

How does one convert the logits into string sentences? I discovered the .generate() method, but this seems to generate new outputs.
I suppose I could convert the logits to a distribution, sample, convert to token ids and then use tokenizer.decode() but that seems too manual. What’s the right way to do this in HuggingFace?

Context:
RylanSchaeffer:

 # shape: (batch size, sequence length, hidden dimension e.g. 768)

It seems to me that these are the outputs of the base model. To get token predictions, you need the output of the LMHead (often a linear projection of hidden_dim → vocab_size). If you have the logits of shape bs, seqlen, vocab_size, you can simply do a softmax on that last dimension, select top1, and decode. This is not as much manual work as you may expect, but you do need the outputs of the LMHead.
As you said, you’d typically generate with generate, so I am not sure whether I understand your use case.


No loss being logged, when running MLM script (Colab)

Question:
When using the run_MLM script and pairing with XLA, I am seeing that despite logging to files I still don’t get a step-by-step output of the metrics.
%%bash
python xla_spawn.py --num_cores=8 ./run_mlm.py --output_dir="./results" \
    --model_type="big_bird" \
    --config_name="./config" \
    --tokenizer_name="./tokenizer" \
    --train_file="./dataset.txt" \
    --validation_file="./val.txt" \
    --line_by_line="True" \
    --max_seq_length="16000" \
    --weight_decay="0.01" \
    --per_device_train_batch_size="1" \
    --per_device_eval_batch_size="1" \
    --learning_rate="3e-4" \
    --tpu_num_cores='8' \
    --warmup_steps="1000" \
    --overwrite_output_dir \
    --pad_to_max_length \
    --num_train_epochs="1" \
    --adam_beta1="0.9" \
    --adam_beta2="0.98" \
    --do_train \
    --do_eval \
    --logging_steps="10" \
    --evaluation_strategy="steps" \
    --eval_accumulation_steps='10' \
    --report_to="tensorboard" \
    --logging_dir='./logs' \
    --save_strategy="epoch" \
    --load_best_model_at_end='True' \
    --metric_for_best_model='accuracy' \
    --skip_memory_metrics='False'  \
    --gradient_accumulation_steps='500' \
    --use_fast_tokenizer='True' \
    --log_level='info' \
    --logging_first_step='True' \
    1> >(tee -a stdout.log) \
    2> >(tee -a stderr.log >&2)

As you can see, I am logging out stderr and stdout to files but I can see that it doesn’t log any step - only the end-of-epoch ones when training is finished. Using TensorBoard also doesn’t help when loss isn’t being logged anyways  which is quite weird.
I have adjusted logging_steps but that doesn’t seem to help. I am quite confused - Trainer is supposed to ouput loss to the Cell output too, but that doesn’t happen either.
Does anyone know how I can log the metrics for ‘n’ steps?

Context:
Basically, despite providing the logging_steps argument, it doesn’t apparently override the default which I presume to be set to epoch - same with evaluation strategy which also runs during epochs instead of the no. of steps provided.
This is what the script receives on its side:-
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.98,
adam_epsilon=1e-08,
dataloader_drop_last=False,
dataloader_num_workers=0,
dataloader_pin_memory=True,
ddp_find_unused_parameters=None,
debug=[],
deepspeed=None,
disable_tqdm=False,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=10,
eval_steps=10,
evaluation_strategy=IntervalStrategy.STEPS,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
gradient_accumulation_steps=500,
greater_is_better=True,
group_by_length=False,
ignore_data_skip=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=0.0003,
length_column_name=length,
load_best_model_at_end=True,
local_rank=-1,
log_level=20,
log_level_replica=-1,
log_on_each_node=True,
logging_dir=./logs,
logging_first_step=True,
logging_steps=10,
logging_strategy=IntervalStrategy.STEPS,
lr_scheduler_type=SchedulerType.LINEAR,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=accuracy,
mp_parameters=,
no_cuda=False,
num_train_epochs=1.0,
output_dir=./results,
overwrite_output_dir=True,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=1,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=results,
push_to_hub_organization=None,
push_to_hub_token=None,
remove_unused_columns=True,
report_to=['tensorboard'],
resume_from_checkpoint=None,
run_name=./results,
save_on_each_node=False,
save_steps=500,
save_strategy=IntervalStrategy.EPOCH,
save_total_limit=None,
seed=42,
sharded_ddp=[],
skip_memory_metrics=False,
tpu_metrics_debug=False,
tpu_num_cores=8,
use_legacy_prediction_loop=False,
warmup_ratio=0.0,
warmup_steps=1000,
weight_decay=0.01,
)

Which seem to be true to my provided flags, but just not being acted upon.
Will dig more in the script to see what might be the issue.


Question:
Hi all,
I was reading through the encoder decoder 23 transformers and saw how loss was generated. But I’m just wondering how it is internally generated?
Is it something like the following: Suppose I have the following pair: ("How are you?", "I am doing great"). In this case, is it calculating the cross entropy loss for the four output tokens and then averaging them?

Context:
padding tokens in the labels should be replaced by -100 so the cross_entriopy loss ignores the pad tokens when computing the loss.
and the loss is actually computed like this
shifted_prediction_scores = prediction_scores[:, :-1, :].contiguous() # prediction_scores is logits
labels = labels[:, 1:].contiguous()
loss_fct = CrossEntropyLoss()
lm_loss = loss_fct(shifted_prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))


How DeepSpeed interacts with Trainer optimizer

Question:
Say we would like to use the Transformers+DeepSpeed integration to fine-tune a relatively large model. That model is too big to fit both the parameters and the full optimizer states in GPU memory at once, so instead we want to freeze most of the parameters and fine-tune a subset of them, or alternatively to tune an adapter that wraps the model. That way we avoid needing to store Adam buffers for the frozen ones. Additionally, we want to use DeepSpeed for ZeRO-Offload.
In order to do that, I believe we need to manually pass our optimizer to Trainer (otherwise Trainer will create an optimizer for all of the parameters, which we would like to avoid). But it looks like DeepSpeed itself has certain optimizer configurations. If we pass a custom optimizer to Trainer along with the DeepSpeed config, will it appropriately train only the subset of parameters specified in that optimizer, or will it end up creating another on the backend that tries to optimize the whole model?

Context:
Might’ve answered my own question, looking at this piece of documentation: transformers.deepspeed — transformers 4.12.0.dev0 documentation 3
If you pass in an optimizer to Trainer and do not include an “optimizer” in the DeepSpeed configuration, it looks like it will use that passed in optimizer.


Next sentence prediction on custom model

Question:
I’m trying to use a BERT-based model (jeniya/BERTOverflow · Hugging Face 4) to do Next Sentence Prediction. This is essentially a BERT model that has been pretrained on StackOverflow data.
Now, to pretrain it, they should have obviously used the Next Sentence Prediction task. But when I do a AutoModelForNextSentencePrediction.from_pretrained("jeniya/BERTOverflow"), I get a warning message saying:
Some weights of BertForNextSentencePrediction were not initialized from the model checkpoint at jeniya/BERTOverflow and are newly initialized: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

Now, I get that the message is telling me that the NSP head does not come with this model and so has been initialized randomly. My question is, if they have published pre-trained a BERT model on some custom data, shouldn’t they also have used an NSP head for their pretraining objective? If so, where did that head go? Did they just throw it away?
If so, how would I go about getting this custom model to work for the task of NSP? Should I pre-train the whole goddamn thing again, but this time not throw away the NSP head? Or can I simply do something like use AutoModel, and extract the [CLS] token representation, and put a MLP on top of that and train it with a few examples to do NSP? The former would be infeasible given the compute requirements. I feel like the latter is just wrong. Am I missing something?
Any help would be greatly appreciated! Thank you!

Context:
Hey there @msamogh
I am facing a similar problem as yours: have you discovered something since the time you created this thread?
Also, if you know it, does this mean that models with architecture “BertForMaskedLM” have been trained ONLY on MLM, and not on NSP, and so I have to do that again?


Decoding the predicted output array in distilbertbase uncased model for NER

Question:
Hi
I am working on extracting legal entities (date) from a corpus of agreements. In the training set, I have tokenized the agreement text, date labels are tagged in IOB convetion and fed to the distilbertbase uncased model. Defined training arguements, data collator and compute matrix methods. After the training is completed, I input the preprocessed prediction dataset which has only agreement text and I need to predict the date label in it.
I am getting a tensor array as output and not aware of how to extract the date labels from this array
Can I get support on this issue ?

Context:
In case of NER, one typically uses an xxxForTokenClassification model (which adds a linear layer on top of the base Transformer model). The logits of such models are typically of shape (batch_size, seq_len, num_labels). Let’s take an existing, fine-tuned BertForTokenClassification model from the hub and perform inference on a new, unseen text:
from transformers import AutoTokenizer, BertForTokenClassification

model_name = "dslim/bert-base-NER"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = BertForTokenClassification.from_pretrained(model_name)

Let’s prepare a new text for the model:
text = "Obama was the president of the United States and he was born in Hawai."

encoding = tokenizer(text, return_tensors="pt")

We now forward it through the model:
# forward pass
outputs = model(**encoding)

We now take the logits from the outputs, which are the scores that the model gives for each of the classes. In case of token classification, the logits are of shape (batch_size, seq_len, num_labels). Let’s check the shape:
logits = outputs.logits
print(logits.shape)

This prints torch.Size([1, 18, 9]). The batch size is 1 as we only have a single sentence, we have a sequence length of 18 tokens, and the number of labels is 9. So apparently this model classifies each token to belong to 1 of 9 possible labels. We can get the predictions by performing an argmax on the last dimension (i.e., the labels dimension), as follows:
predicted_label_classes = logits.argmax(-1)
print(predicted_label_classes)

This prints:
tensor([[0, 3, 0, 0, 0, 0, 0, 7, 8, 0, 0, 0, 0, 0, 7, 7, 0, 0]])
Let’s now convert each predicted class index to the corresponding label name using the id2label mapping of the model’s configuration, as follows:
predicted_labels = [model.config.id2label[id] for id in predicted_label_classes.squeeze().tolist()]
print(predicted_labels)

This prints:
['O', 'B-PER', 'O', 'O', 'O', 'O', 'O', 'B-LOC', 'I-LOC', 'O', 'O', 'O', 'O', 'O', 'B-LOC', 'B-LOC', 'O', 'O']
To see the correspondence between the tokens and the predicted labels, let’s print them side-by-side:
for id, label in zip(encoding.input_ids.squeeze().tolist(), predicted_labels):
  print(tokenizer.decode([id]), label)

This prints:
[CLS] O
Obama B-PER
was O
the O
president O
of O
the O
United B-LOC
States I-LOC
and O
he O
was O
born O
in O
Ha B-LOC
##wai B-LOC
. O
[SEP] O

So now we can clearly see the predictions. However, as the model uses subword tokenization, we would like to convert those to word-level predictions.
Here, a number of aggregation strategies apply. One strategy is to just select the prediction for the first token of each word, another strategy is to average the predictions for all tokens of a word, another strategy is to take the biggest logit for all tokens of a word, etc. This depends on how the model was fine-tuned.
The pipeline API 1 of HuggingFace supports various aggregation strategies, and abstracts away all of what I did above + grouping the entities for the user. You can call it as follows:
from transformers import pipeline

nlp = pipeline("ner", model=model, tokenizer=tokenizer, aggregation_strategy="first")

nlp(text)

This prints:
[{'end': 5,
  'entity_group': 'PER',
  'score': 0.998693,
  'start': 0,
  'word': 'Obama'},
 {'end': 44,
  'entity_group': 'LOC',
  'score': 0.9994223,
  'start': 31,
  'word': 'United States'},
 {'end': 69,
  'entity_group': 'LOC',
  'score': 0.9988722,
  'start': 64,
  'word': 'Hawai'}]

Under the hood, it will use the offsets_mapping (which is only supported by fast tokenizers) to know to which word each token belongs. You can check the source code here 3.


Question:
Hello
I cannot seem to find this information anywhere, but perhaps because the search terms are quite general.
I am wondering how the input data format has to look for language modeling. I am particularly interested in CLM but I’d also like to know for MLM.
My intuition tells me that one should:

split dataset into sentences

? do linguistic tokenization (split by whitespace) ?

insert a “begin of sentence” and “end of sentence” tokens
merge the sentences again
chunk the text up in blocks of max_seq_len length for the model (you could even do a sliding window of max_seq_len size)
so you just have a text file and every paragraph is one (potentially huge) line

This way, the model should be able to generalize better as it learns arbitrary start and ending positions for sentences. That being said, I do not know whether the position embeddings have a negative impact on this process as they are not “correct” any more. (For chunked sentences, the first word may not be the first word of the sentence.)
Looking at the LM examples 1, it seems that such steps are not taken automatically, except for chunking. So is my intuition wrong? If so, how exactly should a given text file look?
As a side question: it is not clear to me from the example run_clm script what exactly is meant with " We drop the small remainder". If a text file is given with potentially very long lines (one line per paragraph), does that mean that everything exceeding the block size in that line is discarded? If so, there must be a better way to organise your text file than I illustrated above.
EDIT: I found this useful documentation 4 by HF concerning a strided sliding window, which is exactly what I intended above. It seems that this is not implemented in the examples, however. I wonder why.

Context: hey @BramVanroy you should check the new TF notebooks about that. i think they explain very clearly what are the steps involved. notebooks/language_modeling-tf.ipynb at new_tf_notebooks · huggingface/notebooks · GitHub 5


Question:
Hi there,
I am wondering, what would be the optimal solution to also report and log perplexity during the training loop via the Trainer API. How would the corresponding compute_metrics function look like. So far I tried without success since I am not 100% sure how the EvalPrediction output would look like.
Thanks in advance
Simon

Context: This brings me to an adjacent question: How would a compute_metrics function look like that can also report the relative change perplexity respectively train_loss? I would be super grateful if anyone could provide a little guidance!


Question:
Hi all!
I want to train T5 in a new language from scratch an I think the best way to do this is through the unsupervised denoising task. I’ve found that there is no function in huggingface to create the train data (masked data) as the T5 documentation indicates 25 so I’ve tried to create by my own.
According with the T5 original paper, if you have two consecutive tokens to masking you must mask them using only one sentinel token so I need a function that searches the consecutive tokens (“rachas” in my language) in the random indices choosed. Here you have my code:
def racha_detection(lista):
    # It returns a list of lists where each sub-list contains the consecutive tokens in the list
    rachas = []
    racha = []
    for i, element in enumerate(lista):
        if (i<len(lista)-1) and (lista[i+1] == element+1):
            racha.append(element)
        else:
            if len(racha)>0:
                rachas.append(racha + [element])
            else:# (i!=len(lista)-1):
                rachas.append([element])
            racha = []
    return rachas

def masking(tokenized_sentence, rachas):
    # Function to mask a tokenized_sentence (token ids) following the rachas described in rachas
    # Only one sentinel_token per racha
    sent_token_id = 0
    enmascared = tokenized_sentence.copy()
    for racha in rachas:
         sent_token = f'<extra_id_{sent_token_id}>'
         sent_id = tokenizer.encode(sent_token)[0]
        for i, idx in enumerate(racha):
            if i==0:
                enmascared[idx] = sent_id
            else:
                enmascared[idx] = -100
        sent_token_id += 1

    enmascared = [t for t in enmascared if t!=-100]

    return enmascared

def add_noise(sentence, tokenizer, percent=0.15):
    # Function that takes a sentence, tokenizer and a noise percentage and returns
    # the masked input_ids and masked target_ids accordling with the T5 paper and HuggingFace docs
    # To see the process working uncomment all the prints ;)
    tokenized_sentence = tokenizer.encode(sentence)
    #print('PRE-MASKED:')
    #print('INPUT: {}'.format(tokenizer.convert_ids_to_tokens(tokenized_sentence)))

    idxs_2_mask = sorted(random.sample(range(len(tokenized_sentence)),
                                       int(len(tokenized_sentence)*percent)))
    rachas = racha_detection(idxs_2_mask)
    enmascared_input = masking(tokenized_sentence, rachas)
    #print('RACHAS INPUT: {}'.format(rachas))
    idxs_2_mask = [idx for idx in range(len(tokenized_sentence)) if idx not in idxs_2_mask]
    rachas = racha_detection(idxs_2_mask)
    enmascared_target = masking(tokenized_sentence, rachas)
    #print('RACHAS TARGET: {}'.format(rachas))

    #print('POST-MASKED:')
    #print('INPUT: {}'.format(tokenizer.convert_ids_to_tokens(enmascared_input)))
    #print('TARGET: {}'.format(tokenizer.convert_ids_to_tokens(enmascared_target)))

    return enmascared_input, enmascared_target

I dont know if it is correct but it generates sequences like the sequences in the examples   What do you think?

Context: @amlarraz - how did you get on with this? Did it work ok with T5 training? Any tips gratefully appreciated!


Question:
Hi,
after the model training I’ve got saved training_args.bin. However, loading them with torch.load(...) as suggested in How to load training_args 1 results in an error:

Снимок экрана 2021-10-08 в 13.55.192038×876 161 KB

Am I doing something wrong or is it a bug?
Thanks!

Context: A mistake on my side, though may be relevant to somebody. I overwrote the class TrainingArguments to add my own one, and the file with the new class (training_arguments.py in my case) must be present in the same directory from which you are uploading the arguments. Closing this.


Making sense of duplicate arguments in Huggingface’s hyperparameter search work flow

Question:
In trying to use the Trainer’s hyperparameter_search to run trials of my experiments, I found that there are some arguments that seemingly mean the same thing and I am trying to understand which take precedence.or if any of them can be ignored. I think there should be some documentation on this issue as it’s easy to get confused.
The reason for this duplicity comes from the fact that there are the arguments to hyperparameter_search itself, the arguments to TrainingArguments, as well as the additional arguments that can be passed to the library doing the hyperparameter tuning (such as ray tune) and they are not necessarily mutually exclusive (at least in the way that I interpret them).
Here is a list of issues regarding parameter confusion that I am having:

Checkpointing arguments. TrainingArguments and RayTune define their own checkpointing parameters. TrainingArguments has “metric_for_best_model”,“save_steps”, “save_total_limit” and ray tune has “checkpoint_score_attr”, “checkpoint_freq”, “keep_checkpoints_num”

TrainingArgument’s load_best_model_at_end. This is more so a concern that I have in regards to issue #1. I am using WandB to log my results and for each trial, I want it to save the best performing model to its “artifacts” folder but can I have confidence that “load_best_model_at_end” will do anything if ray-tune handles checkpointing on its own side of things?

Metric Optimization Parameters. hyperparameter_search has “compute_objective” and “direction”. TrainingArguments has metric_for_best_model and greater_is_better. Then ray-tune has metric and mode. I’m assuming the arguments to hyperparameter_search simply pass those arguments down to the corresponding ones of ray-tune (or whatever tuning library is being used). I think that just to give the user confidence in what is going on maybe a warning saying that passing those additional ray tune parameters is redundant?

@sgugger maybe you can clear up some of this confusion? I know that there is also this post in regards to the hyperparameter_search function. But felt it better to make a new one as this is a bit of a loaded question.

Context:
I am also concerned with the following arguments. Still, I can probably clear up some of the questions for you.

From what I’ve inspected from the logs, load_best_model_at_end argument is ignored during the hyperparameter search.
Since load_best_model_at_end is ignored, I believe all the corresponding args (e.g. metric_for_best_model) are ignored as well - at least reproducing the results while changing this parameter, I get the same values.

Hope @sgugger can clarify.


How release notes are created in Transformers repo

Question:
Hi,
The release notes in Transformers repository are amazing. Fully detailed, well written, organized. It feels like nothing is missing, PR are linked, contributors are mentioned.
I’m trying to do the same for my repositories, but I’m facing some issues :

I’m doing everything manually. It means sometimes I forget a PR, or a contributor.

I do everything at once : when I make a release, I filter all the PR made since the last release, and summarize them in the release note. It’s time consuming, and I feel a continuous approach would be better.

So I’m wondering how do you guys handle Release notes at HuggingFace ? Do you have a specific process ? Specific tools ?

Context:
Hi @colanim!
Thank you for the kind words! Release notes are an important part of the project, as mentioning each contributor’s contribution is essential: the project would not exist without these contributions.
In the past, we were doing everything manually, but, as you have seen, it is error-prone in a domain where errors are not acceptable. In order to mitigate this, we built a tool that scans all commits merged since the previous version, and formats them correctly. We’re left with a big list of contributions that can copy/paste in the release notes. I’d happily share the tool with you but it has quite a few flaws that make it unfit for general usage.
Then comes a second important part, where we do it like you do: we do everything at once, filtering PRs and grouping them in categories. This is indeed time-consuming, but it is important work. It serves as documentation, and as a guide for potential breaking changes. It is not unusual to spend between 1 and 2 hours on release notes, reordering commits in categories.
I hope that helps!


EvalPrediction has an unequal number of label_ids and predictions

Question:
The EvalPrediction object received in Trainers compute_metrics function returns an unequal number of label_ids and predictions.
My compute_metrics function given to the Trainer looks like this:

The shapes I print:

And the error message I get:

image807×55 8.67 KB

Does anyone of you have a solution to this?

Context: What does your data look like? I’m afraid there might have been some labels that were a list concatenated with single labels or something like that.


How to use Transformer XL for sequence classification?

Question:
I cannot understand how to fine-tune ‘Transformer XL’ for sequence classification.
I am getting this error

RuntimeError: stack expects each tensor to be equal size, but got [1] at entry 0 and [8] at entry 2

and I understand that it is due to my sequences having varying lengths but I am not sure how this is intended to be remedied for this specific model. I have created a simple reproduceable toy example with the intention of understanding how this model is intended to be used:
!pip install transformers==4.10.0
!pip install datasets==1.9.0

from transformers import AutoTokenizer
from transformers import TransfoXLForSequenceClassification
from transformers import TrainingArguments, Trainer
import torch

class newDatasetSimple(torch.utils.data.Dataset):

    def __init__(self, encodings, labels):
        self.encodings = encodings
        self.labels = labels

    def __getitem__(self, idx):
        item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
        item['labels'] = torch.tensor(self.labels[idx])
        # print(item)
        return item

    def __len__(self):
        return len(self.labels)

tokenizer = AutoTokenizer.from_pretrained('transfo-xl-wt103')
model = TransfoXLForSequenceClassification.from_pretrained('transfo-xl-wt103')

texts = ['This is a sentence', 'Here is another', 'Short sentence', 'I ran', 'This is the longest sentence of the bunch', 'What?', 'Who?', 'Test.', 'Hey', 'A', 'The', 'So', 'yes', 'cool', 'beans', 'In the flesh']
labels = [0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1]
encodings = tokenizer(list(texts))

ds = newDatasetSimple(encodings, labels)

training_args = TrainingArguments(output_dir='.',
                                  num_train_epochs=50,
                                  per_device_train_batch_size=16,
                                  warmup_steps=0,
                                  logging_steps=1,
                                  learning_rate=1e-5)

trainer = Trainer(model=model,
                  args=training_args,
                  train_dataset=ds)

trainer.train()

Could someone show me what changes need to be made in the code to get the model to train properly?
Thank you for reading!

Context:
Can you post the full error message?
As noted here 3, TransformerXL is the only model in the library that is not supported by the Trainer (you would need to overwrite it).


How to create a config.json after saving a model

Question:
Hi, I am trying to convert my model to onnx format with the help of this notebook 1
I got error , since config.json does not exist.
My model is a custom model with extra layers, similar to this, 1
Now how can I create a config.json file for this?

Context:
You need to subclass it to have the save_pretrained methods available. So instead of
class Mean_Pooling_Model(nn.Module):

use
from transformers.modeling_utils import PreTrainedModel
class Mean_Pooling_Model(PreTrainedModel):

It will add extra functionality on top of nn.Module.

      github.com

    huggingface/transformers/blob/bcc3f7b6560c1ed427f051107c7755956a27a9f2/src/transformers/modeling_utils.py#L415 1

                      exclude_embeddings (:obj:`bool`, `optional`, defaults to :obj:`True`):
                          Whether or not to count embedding and softmax operations.

        Returns:
                      :obj:`int`: The number of floating-point operations.
                  """

        return 6 * self.estimate_tokens(input_dict) * self.num_parameters(exclude_embeddings=exclude_embeddings)

class PreTrainedModel(nn.Module, ModuleUtilsMixin, GenerationMixin, PushToHubMixin):
              r"""
              Base class for all models.

    :class:`~transformers.PreTrainedModel` takes care of storing the configuration of the models and handles methods
              for loading, downloading and saving models as well as a few methods common to all models to:

        * resize the input embeddings,
                  * prune heads in the self-attention heads.

    Class attributes (overridden by derived classes):


Specify Loss for Trainer / TrainingArguments

Question: I’d like to fine-tune for a regression task rather than a classification task. How do I change the default loss in either TrainingArguments or Trainer()?

Context:
You can overwrite the compute_loss method of the Trainer, like so:
from torch import nn
from transformers import Trainer

class RegressionTrainer(Trainer):
    def compute_loss(self, model, inputs, return_outputs=False):
        labels = inputs.get("labels")
        outputs = model(**inputs)
        logits = outputs.get('logits')
        loss_fct = MSELoss()
        loss = loss_fct(logits.squeeze(), labels.squeeze())
        return (loss, outputs) if return_outputs else loss

However, several models in the library 3 have an attribute of their config called problem_type, which you can set to “regression”. In that case, you shouldn’t overwrite anything, and you can just use the default loss of the model.


Fine-Tuning results suggest some underlying implementation error?

Question:
I’m trying to find-tune BERT on a regression task. My target values are approximately uniformly distributed over 0 to 100. I train using MSE
The training loss and validation loss appear to go down:

But when I look at the output predictions, they’re all nearly the same value:
13.34873,13.34946,13.34548,13.34980,13.34415,13.35009,13.35031,13.35068,13.35060,13.34515,13.34916,13.34391,13.32421,13.33146,13.29470,13.34953,13.35133,13.34735,13.34369,13.34804,13.35447,13.34434,13.35356,13.34438,13.35195,13.35314,13.34806,13.33857,13.34869,13.34059,13.35074,13.34365,13.35027,13.34974,13.35198,13.34209,13.34324,13.35140,13.35044,13.34025,13.34005,13.35257,13.30577,13.34795,13.33279,13.34773,13.33482,13.35300,13.34842,13.33357,13.34200,13.35000

This suggests to me that I’m doing something incorrectly. My code is below. Could someone please help me?

    training_args = TrainingArguments(
        output_dir=results_dir,  # output directory
        num_train_epochs=10,  # total number of training epochs
        per_device_train_batch_size=16,  # batch size per device during training
        per_device_eval_batch_size=64,  # batch size for evaluation
        warmup_steps=500,  # number of warmup steps for learning rate scheduler
        weight_decay=0.01,  # strength of weight decay
        logging_dir=results_dir,  # directory for storing logs
        logging_steps=10,
        report_to='wandb',
        do_eval=True,
        evaluation_strategy="steps",
        eval_steps=10,
    )

    class RegressionTrainer(Trainer):

        def compute_loss(self,
                         model,
                         inputs,
                         return_outputs=False):
            labels = inputs.get("labels")
            outputs = model(**inputs)
            logits = outputs.get('logits')
            loss = torch.mean(torch.square(logits.squeeze() - labels.squeeze()))
            return (loss, outputs) if return_outputs else loss

    pytorch_model_save_path = os.path.join(results_dir, 'pytorch_model.bin')
    if os.path.isfile(pytorch_model_save_path):  # If model was already fine-tuned
        # yes, pass the whole results dir; see https://github.com/huggingface/transformers/issues/1620
        model = DistilBertForSequenceClassification.from_pretrained(
            results_dir,
            num_labels=1)
    else:  # If model needs to be fine-tuned
        # Set output dimension to 1 to perform regression
        model = DistilBertForSequenceClassification.from_pretrained(
            "distilbert-base-uncased",
            num_labels=1)

    trainer = RegressionTrainer(
        model=model,  # the instantiated 🤗 Transformers model to be trained
        args=training_args,  # training arguments, defined above
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        # compute_metrics=compute_eval_metrics,
    )

    if force_train or not os.path.isfile(pytorch_model_save_path):
        trainer.train()
        trainer.save_model(output_dir=results_dir)

    all_prediction_output = trainer.predict(all_dataset)

Context:
Turns out there was no error! Two things:

The learning rate was small and the validation loss was being evaluated very frequently, which explains why the validation loss was so smooth.

I needed to run 50 training epochs to see a real difference. That seems odd, but so be it


Subclassing a pretrained model for a new objective

Question:
I would like to use a pretrained model as an encoder for a new task. It is essentially multiple sequence classification objectives, like in the ...ForSequenceClassification models, but with an output layer for each subtask.
I could just create wrappers around the encoder, but I’d like to subclass from PreTrainedModel to better integrate with the Trainer class. How exactly should I do? Do I need to create a config class as well? I will at least need to supply an extra list or dict to the config telling how many classes each subtask has.
Thanks!

Context: You can definitely subclass PretrainedConfig for your custom config and PreTrainedModel for your custom model, then access all the methods of the library.


What is cause and solution to Trainer error: cuda RuntimeError 711?

Question:
I’m trying to fine-tune DistilBert using Trainer, following the HuggingFace tutorial:

      huggingface.co

Fine-tuning with custom datasets

  This tutorial will take you through several examples of using 🤗 Transformers models with your own datasets. The guide shows one of many valid workflows for u...

When I try running the example, I get the following error:
RuntimeError: cuda runtime error (711) : peer mapping resources exhausted at /pytorch/aten/src/THC/THCGeneral.cpp:139
What does this error mean and how do I fix it?

Context:
RylanSchaeffer:

RuntimeError: cuda runtime error (711) : peer mapping resources exhausted

Can you run your code on CPU, to get a more informative error message?


Saving eval loss for every evaluation/saved checkpoint with Trainer

Question:
I am running the trainer with
--do_eval  --report_to none --evaluation_strategy epoch --num_train_epochs 10

so I am expecting the trainer to evaluate after every epoch and saving those results in either eval_results.json or all_results.json. But those files unfortunately only contain the results of the last evaluation.
It is useful that checkpoints are saved, but I cannot seem to find the evaluation results for each checkpoint. That would be very useful to select the optimal checkpoint.

Context: There will be in the log_history field of the trainer_state, which is also saved in the same folder.


Using BERT and RoBERTa for (causal?) language modeling

Question:
Hi all!
I’m hoping to use pretrained BERT/RoBERTa for language modeling, i.e. scoring the likelihood of sentences. There have been quite a few blog posts/issues on this, but no obvious consensus yet. I attempted an implementation with BertLMHeadModel and RobertaForCausalLM here 4, which had weird issues like:

RoBERTa has super large perplexity values, and
BERT cannot correctly compare the relative perplexity of simple sentences.
(Please see more details in the Github issue above.)

@gugarosa kindly suggests that I shouldn’t evaluate pretrained BERT/RoBERTa directly, but should train them with causal LM objective beforehand. However, given the size of their pretraining data, it’s unlikely to retrain them myself. Is there any existing checkpoints that I can use directly? Or, would you recommend any other models (e.g. BertForMaskedLM?) or evaluation metrics (other than perplexity) instead?
Thanks in advance!

Context:
Continuing our discussion on Github…
You are definitely correct when saying that it might be unfeasible to train from scratch as they initially did, specially due to the size of your data.
On the other hand, imagine that you have a pre-trained BERT/RoBERTa model and you attach the LMHead on top of it. You could freeze the pre-trained parameters from the initial pre-trained BERT or even attach a small learning rate to this part of architecture, while you fine-tune the LMHead with a more aggressive rate using your own data and a causal language modeling (CLM) objective. The idea behind this would be to attempt to adapt the pre-trained BERT and start understanding how to model a CLM task, directly on your data and without losing some features that it may have already learned from the previous training. Nonetheless, it just an initial thought that I had and do not know how it will work in “real-world”, as my experience is just based on directly working with autoregressive models, such as GPT and Transformer-XL.
Regarding some pre-trained models for language generation / CLM, there are few that I could found by tagging text-generation and bert: Models - Hugging Face 4. However, I can not assure whether they were trained with masked LM or CLM, as there were no model cards with descriptions.
Regarding the evaluation metric, it is sure a challenge to define an appropriate metric or even just relying on the loss/perplexity. The problem with loss and perplexity is that they might mislead us when comparing models with close values because it strictly relies on the conditional probability of a token being generated given the previous tokens, so essentially we are trying to match the information according to a given target, whereas that given target might be valid if employed with some variations.
For example:
A sample in the test set “Hello, how are you” might give different perplexity when comparing to a generated prompt like “Hello, how you doing” and “Hello, how it is going”, even though they might have similar meaning, semantically speaking.
I have seen some works that attempt to employ a exact match or even a partial match metric, trying to correlate the n-grams between a generated text and a reference (test sample), in the same way as BLEU, METEOR and ROUGE would be applied to a machine translation task. A qualitative assessment is also pretty interesting, specially if the model is going to be deployed into a real-world application or something like. Unfortunately, we are still lacking some advancements on how to turn grammar, syntax and semantics into more proper quantitative metrics, but that might be changed in the near future… at least I hope so!


No skipping steps after loading from checkpoint

Question:
Hey! I am trying to continue training by loading a checkpoint. But for some reason, it always starts from scratch. Probably I am just missing something.
training_arguments = Seq2SeqTrainingArguments(
            predict_with_generate=True,
            evaluation_strategy='steps',
            per_device_train_batch_size=training_config['per_device_train_batch_size'],
            per_device_eval_batch_size=training_config['per_device_eval_batch_size'],
            fp16=True,
            output_dir=training_output_path,
            overwrite_output_dir=True,
            logging_steps=training_config['logging_steps'],
            save_steps=training_config['save_steps'],
            eval_steps=training_config['eval_steps'],
            warmup_steps=training_config['warmup_steps'],
            metric_for_best_model='eval_loss',
            greater_is_better=False)

trainer = Seq2SeqTrainer(
            model=model,
            tokenizer=tokenizer,
            args=training_arguments,
            compute_metrics=compute_metrics,
            train_dataset=train_ds,
            eval_dataset=eval_ds,
        )

Here are the logs:
loading weights file .../models/checkpoint-2000/pytorch_model.bin
All model checkpoint weights were used when initializing EncoderDecoderModel.
***** Running training *****
  Num examples = 222862
  Num Epochs = 3
  Instantaneous batch size per device = 8
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 1
  Total optimization steps = 83574

I am missing some like:
Continuing training from checkpoint, will skip to saved global_step
Continuing training from epoch 0
Continuing training from global step 48000
Continuing training from 0 non-embedding floating-point operations
Will skip the first 48000 steps in the first epoch

Which I found here: Load from checkpoint not skipping steps - Transformers - Hugging Face Forums 20
Maybe somebody can help me? Thank you in advance!

Context: With overwrite_output_dir=True you reset the output dir of your Trainer, which deletes the checkpoints. If you remove that option, it should resume from the lastest checkpoint.


Sentence Embeddings From Fine-Tuned BERTForSequenceClassification

Question:
Hey everyone,
I have a binary classification task for a set of documents, and I’d like to visualize these documents from their embeddings. I’ve previously used the sentence-transformers library to do this, but I wanted to see if it was possible to improve these embeddings by fine-tuning my own BERT model to the particular task rather than just using a pre-trained model. I read through some guides 2 and discussions online, and it seems like I should be able to use the embedding for the CLS token from the last hidden state layer as a sentence embedding. However, when I pull those values from the hidden_states of the fine-tuned BERTForSequenceClassification model, every embedding is the same.
This is the code I’m using to fine-tune the pre-trained model:
model = BertForSequenceClassification.from_pretrained("bert-base-cased",
                                                      num_labels=1,
                                                      output_attentions=False,
                                                      output_hidden_states=True)
model.to(device)
model.train()
optim = AdamW(model.parameters(), lr=5e-5)

for epoch in range(3):
    for batch in dataloader_train:
        optim.zero_grad()
        input_ids = batch[0].to(device)
        attention_mask = batch[1].to(device)
        labels = batch[2].to(device)
        outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
        loss = outputs[0]
        loss.backward()
        optim.step()

model.eval()

And this is the code I’m using to pull the embeddings:
def embeddings(model, dataloader_val):
    model.cuda()
    embeddings = np.zeros((0, 768))

    for batch in dataloader_val:

        batch = tuple(b.to(device) for b in batch)

        inputs = {'input_ids':      batch[0],
                  'attention_mask': batch[1],
                 }

        with torch.no_grad():
            outputs = model(**inputs)
        embeddings = np.concatenate((embeddings, outputs[1][0][:,0,:].cpu().numpy()), axis=0)

    return embeddings

Any ideas or thoughts on why the embeddings for all of the CLS tokens would be the same?

Context: One way around this problem that I was thinking of was to train via the sequence classification task and then load the trained model as a normal BertModel and used the normal pooler_output. Has anyone tried something like this?


Correct numeric labels for classification?

Question:
Hello,
This is a simple question but better safe than sorry! My understanding is that the transformers class of models (for text classification) can only deal with integer labels as classes.
So it’s up to the user to provide a mapping between labels and scores. In the usual example one could have 0  = negative, 1 = neutral, 2 = positive.
Here is the basic question: do the numeric scores necessarily need to be integers from 0 to N (the number of classes) or I can use any other numbers of my liking?
Thanks!

Context:
yes, I can confirm the labels have to be integers starting at zero. I still wonder what is the mathematical reason for that? Any ideas @nielsr by any chance?
Thanks!


How to Pretrain XLSR wav2vec on my unlabeled speech data

Question:
I want to update the XLSR wav2vec2 weights via unlabeled training data(.wav audios) of my domain. Or you can say that I want to pretrain it in that way it can get exposed to my data before I start it to fine-tune it on label data.
This is the model from hugging face
model = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-large-xlsr-53",
attention_dropout=0.1,
hidden_dropout=0.1,
feat_proj_dropout=0.0,
mask_time_prob=0.05,
layerdrop=0.1,
gradient_checkpointing=True,
ctc_loss_reduction="mean",
pad_token_id=processor.tokenizer.pad_token_id,
vocab_size=len(processor.tokenizer)

Please do let me know how can I just use some code to expose XLSR to my unlabeled data as well.
Also when I try to Train the XLSR model on my unlabeled data without any validation data and evaluation measure, it gives me this error
KeyError: ‘loss’
here is my code

from transformers import Trainer
from transformers import TrainingArguments

training_args = TrainingArguments(
  output_dir="/content/drive/MyDrive/wav2vec2-large-xlsr",
  group_by_length=True,
  per_device_train_batch_size=16,
  gradient_accumulation_steps=2,
  evaluation_strategy="steps",
  num_train_epochs=30,
  fp16=True,
  save_steps=200,
  eval_steps=200,
  logging_steps=200,
  learning_rate=3e-4,
  warmup_steps=300,
  save_total_limit=3,
  do_train=True,
)
trainer = Trainer(
    model=model,
    data_collator=data_collator,
    args=training_args,
    train_dataset=data,
    tokenizer=processor.feature_extractor,
)
trainer.train()

Thanks in advance

Context: @patrickvonplaten I hope you can help me with this.


Question:
Hi There,
I am looking for pre-trained model I could add to my project pipeline for phoneme recognition. I can see that wav2vec2.0 can be pretrained to do so. But is there any existing pertained model that I can use?
Thanks!

Context:
Hello Manisa
Have you managed to find the pretrained model for phonemes recognition?..  I’m also looking for one at the moment!..


Question:
seq2seq decoding is inherently slow and using onnx is one obvious  solution to speed it up. The onnxt5 62  package already provides one way to use onnx for t5.
But if we export the complete T5 model to onnx, then we can’t use the past_key_values for decoding since for the first decoding step past_key_values will be None  and onnx doesn’t accept None input. Without past_key_values onnx won’t give any speed-up over torch for beam search.
One other solution is to export the encoder and lm_head to onnx and keep the decoder in torch, this way the decoder can use the past_key_values.
I’ve written a proof-of-concept script which does exactly this and also makes it compatible with the generate method

      gist.github.com

    https://gist.github.com/patil-suraj/09244978af5f7598dd30fb9b4f54fe29 178
onnx_t5.py
import inspect
import logging
import os
from pathlib import Path

import torch
from psutil import cpu_count
from transformers import T5Config, T5ForConditionalGeneration, T5Tokenizer
from transformers.generation_utils import GenerationMixin
from transformers.modeling_outputs import BaseModelOutputWithPast, Seq2SeqLMOutput
This file has been truncated. show original
requirements.txt
--find-links https://download.pytorch.org/whl/torch_stable.html
torch==1.6.0+cpu
transformers>=3.1.0
onnxruntime>=1.4.0
onnxruntime-tools>=1.4.2
psutil

With this you can
enc = tokenizer("translate English to French: This is cool!", return_tensors="pt")
onnx_model = OnnxT5(model_name_or_path="t5-small", onnx_path="onnx_models")
tokens = onnx_model.generate(**enc, num_beams=2, use_cache=True) # same HF's generate method
tokenizer.batch_decode(tokens)

In my experiments this gave ~1.4-1.6x speed-up with beam search.
The first time you call OnnxT5 it’ll load the model from the hub, export it to onnx as described above and save the exported graphs at onnx_path. So loading will be  slower the first time.
Now to gain further speed-up we could distill the model and use less decoder layers.
onnx + distillation should give even more speed-up with minimal drop in accuracy.
@sshleifer has just published awesome seq2seq distillation paper 31 which can be used to distill T5 model  as well.
I’ll be sharing T5 distillation results soon!
now this is a very hacky solution, so feel free to suggest feedback and improvements or any other method that can help speed things up
cc. @abel, @patrickvonplaten , @sshleifer

Context:
Here’s a self-contained notebook 78 and small benchmark for summarization and translation tasks.
summ_benchmark862×682 8.7 KB
translation_benchmark862×682 8.97 KB


Question:
While training a LayoutLM V2 model with a QA head we noticed that the evaluation loop stops using the GPU and will take hours to complete a single loop.

Any ideas what could be happening here?

Context: The evaluation for question-answering is pretty long as the post-processing (going from the predictions of the models to spans of texts in the contexts) is not on the GPU and is rather long. If it’s for an evaluation during training, you should use a smaller validation set.


What should I do if I want to use model from DeepSpeed

Question:
I am trying the language model pre-trained by using run_mlm.py. I want to add the mixture of expert (MoE) integrated with the original Bert-base model. Specifically, I reuse the MoELayer implemented by DeepSpeed and add it to the BertForMaskedLM. From the document of DeepSpeed, I find the training of DeepSpeed requires to call some functions like this:
model_engine, optimizer, _, _ = deepspeed.initialize(args=cmd_args,
                                                     model=net,
                                                     model_parameters=net.parameters())

However, I just want to reuse the MoE model implemented by DeepSpeed and maintain the training behaviors of Huggingface. Currently, I ignore calling this function and directly pass the language model (with DeepSpeed model) into the Trainer(). Although it runs successfully, my question is that does this incurs some potential dangers?

Context:
Hi ezio98.
I can’t answer your question, but I’m a bit confused.   From what I have read about the MoE layer, the point of it is to facilitate the use (Mixture) of many different models (Experts) concurrently, but you say you are using the MoE layer on top of a single BertForMaskedLM model.
What are you hoping the MoE layer will do for you?   Does it have some other advantages?


Why does translation quality go down after fine-tuning only one epoch?

Question:
This might be better suited to #beginners, but it is definitely #transformers-specific…  I’m new to  and wanted to try model adaptation of one of the Helsinki-NLP MT models using fine-tuning.
I’ve created a DatasetDict with train, dev and test, and managed to load the pre-trained model and run a trainer for one epoch.  My corpus is very small (<1k segments), so my expectation is that it would have little impact on the baseline model.
However, when I use the locally-saved config to translate, the results look more like the output of a model just starting its training.  Is this expected, or am I doing something terribly wrong?
Thanks!

Context: Actually, the real problem is even more embarrassing…  Even without freezing layers, the output shouldn’t have looked the way it did.  It turns out that I was using .from_config() instead of .from_pretrained() in trying to load my fine-tuned model.  Once I fixed that, the results were much more in line with my expectations.


Difference betweeen DistilBertTokenizerFast and DistilBertTokenizer?

Question:
Hi everyone,
I’m trying to understand what is the difference between the DistilBertTokenizerFast 18 and the DistilBertTokenizer 3.
From the documentation it looks like that the DistilBertTokenizerFast “Construct a “fast” DistilBERT tokenizer (backed by HuggingFace’s tokenizers library).” whereas the DistilBertTokenizer “Construct a DistilBERT tokenizer.”, but I don’t understand what that means.
This is what I found from the documentation: Tokenizer — transformers 4.5.0.dev0 documentation 8
To the best of my understanding the difference between a “fast” and a “non-fast” tokenizer is computation speed but there is no functional difference between them. Please correct me if I’m not the right direction.
Any help will be greatly appreciated.
Thank you,
Ayala

Context:
@ayalaall I also have the same thought as you.
I also want to know about this. Are they the same in terms of function?


Is T5 expected to ignore padding tokens in `decoder_input_ids` when `decoder_attention_mask` is not provided

Question:
I’m currently trying to train T5ForConditionalGeneration model for seq2seq task, and I was wondering if we can expect T5 to internally ignore (e.g. by generating attention mask) padding tokens in decoder_input_ids if we don’t explicitly provide decoder_attention_mask?
I noticed from the code that T5 simply create attention mask of all 1s if decoder_attention_mask is not provided, so it seems we’re attending to padding tokens. I also ran a sanity check to see if providing decoder_attention_mask had any meaningful difference for the logits and saw that it does matter.
So I’m wondering if this is by design, because it doesn’t seem to make sense to attend to padding tokens for batched passes.
Below is sanity check that I ran (I know decoder_input_ids is supposed to be different from input_ids normally, but figured it’s not important for this particular issue).
import torch
import transformers

model = transformers.T5ForConditionalGeneration.from_pretrained("t5-base")
tokenizer = transformers.T5Tokenizer.from_pretrained("t5-base")
model.cuda()
model.eval()

texts = ["This is a test input.", "This is a test input to test T5 padding scheme."]
input_ids = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")
input_ids.to("cuda")

with torch.inference_mode():
    # Shift decoder input ids to the right
    decoder_input_ids = model._shift_right(input_ids.input_ids)

    # Manually give correct attention mask
    with_attn_mask_logits = model(
        input_ids=input_ids.input_ids,
        attention_mask=input_ids.attention_mask,
        decoder_input_ids=decoder_input_ids,
        decoder_attention_mask=torch.cat((
            torch.tensor([[1], [1]], device="cuda"),
            input_ids.attention_mask[:, :-1]), dim=1
        )
    ).logits

    # Give attention mask of all 1 explicitly
    all_1_attn_mask_logits = model(
        input_ids=input_ids.input_ids,
        attention_mask=input_ids.attention_mask,
        decoder_input_ids=decoder_input_ids,
        decoder_attention_mask=torch.ones(decoder_input_ids.shape, device="cuda"),
    ).logits

    # Do give attention mask at all.
    no_attn_mask_logits = model(
        input_ids=input_ids.input_ids,
        attention_mask=input_ids.attention_mask,
        decoder_input_ids=decoder_input_ids,
    ).logits

    print(torch.all(torch.isclose(with_attn_mask_logits, no_attn_mask_logits)).item())  # False
    print(torch.equal(with_attn_mask_logits, no_attn_mask_logits))  # False

    print(torch.all(torch.isclose(all_1_attn_mask_logits, no_attn_mask_logits)).item())  # True
    print(torch.equal(all_1_attn_mask_logits, no_attn_mask_logits))  # True

Context:
Good catch. I think that in the decoder, one attends to all tokens (including padding), but as one needs to set the labels of the padding tokens to -100, they are not taken into account by the loss function.
But it’s weird indeed, as in the encoder one uses the attention mask to ignore padding tokens when calculating the attention scores.
cc @patrickvonplaten @valhalla


Usind a fine-tuned sentence completion model in a Masked LM task

Question:
Is it possible to use a BERT model that has been fine-tuned already (e.g. SQUAD-tuned BERT) on a masked LM task? I suspect that the sentence-completion model that is added on top of BERT is fundamentally incompatible with a masked LM task, but I’d like to know for a fact.
I’ve attempted to do this, using:
from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("deepset/bert-base-cased-squad2")
model = AutoModelForMaskedLM.from_pretrained("deepset/bert-base-cased-squad2")

but the results are very bad, so maybe I’m missing a step.
Thanks!

Context:
Hi,
that’s not really possible, unless the model has a language modeling head that has been trained. If that’s not the case, it will load a randomly initialized language modeling head, which gives random predictions.
The "deepset/bert-base-cased-squad2" checkpoint has a fine-tuned question-answering head, but not a trained language modeling head.


How to use BART as an encoder and a decoder separately for summarization?

Question: Hi! I am a new comer to huggingface. I wanted to modify the encoder outputs from BART’s encoder and apply some operations on them, and then generate tokens from the decoder step by step. But I am not able to find any BARTEncoder model in huggingface, neither a BARTDecoder, so how should I go about it?

Context:
You might need to modify the encoder (and/or) the decoder in this:
https://huggingface.co/transformers/_modules/transformers/models/bart/modeling_bart.html#BartModel 2
Copy-paste the entire file, save it locally as ‘modifiedbart.py’, and call it instead of the default “BARTModel” in transformers.


Question:
I need to make a feature extractor for a project, so I am able to translate a given financial statement (text) into a vector that can be used as features in my main problem. I am currently doing revenue forecasting. I use historical fundamentals data in addition to stock prices in order to predict revenue growth for next quarter (regression problem). In addition I use text data (financial statements) where I want to use BERT in order to get new features for my regression model. That is, the vector from the BERT feature extraction will later be combined with several other values (fundamentals and stock price data) for the final prediction (next quarter revenue growth) in e.g. a random forest or XGBoost model.
I want to try both a FINE-TUNED FinBERT model and a PRE-TRAINED FinBERT MODEL and compare. But how do I fine-tune the FinBERT model on my dataset (regression problem) and then use that new FinBERT model to do the feature extraction?

Context:
You can easily get a feature vector for a given piece of text as follows:
from transformers import BertTokenizer, BertModel

tokenizer = BertTokenizer("ProsusAI/finbert")
model = BertModel.from_pretrained("ProsusAI/finbert")

text = "hello world"
encoding = tokenizer(text, return_tensors="pt")

# forward pass
outputs = model(**encoding)

# get feature vector
feature_vector = outputs.last_hidden_state[:,0,:]

Here I’m taking the final hidden state of the [CLS] token, which serves as a good representation of an entire piece of text. This is a vector of size (768,) for BERT-base-sized models.
I’m not sure what you mean by fine-tuning a BERT model on your dataset. You can fine-tune BERT on a regression problem, where the inputs are text and the outputs are floats. Is this what you want to do?


What is the classification head doing exactly?

Question:
Hello,
When using a transformer model for text classification, one usually loads a model and then uses AutoModelForSequenceClassification to train the classifier over the N classes in the data.

My question: which model is actually used for classification? Is it a logistic model (with uses as input the CLS representation)?

In the case of several classes (say bad, neutral, good) the usual methodology in machine learning is to train several one-vs-all classifiers and then predict the label with most votes. Is this what is happening under the hood with huggingface?

Thanks!

Context: @nielsr I would be curious to have your take on this, if you have a few moments. Your comments have been incredibly useful so far. Thanks!


Uploading files larger than 5GB to model hub

Question:
I want to upload ctrl to model hub. I have followed the instructions from the documentation and it seems that they are applicable for smaller models (<5GB). Issues have been raised here 3 and here 3 but it still seems unresolved.
I followed the one shared by @julien-c (aws s3 cp …) but got the error Unable to locate credentials indicating that they are for HF staff.
I am getting the error could not push some refs as 5GB limit is crossed. Can you please suggest a workaround if that exists ?
@patrickvonplaten

Context:
Did you install git-lfs? git lfs install
You also have to install our custom transfer agent for files >5GB: transformers-cli lfs-enable-largefiles
Let me know if this helps!


Model weights warning while loading any model from HuggingFace models

Question:
Hi ,
I am trying to load the model from huggingface hub and it throws me some initialization warning. It is happening while trying to load any kind of model.
I am not able understand why this is happening. Can anyone please help me out here?

image1816×226 30 KB

Context:
This warns you that some of the model weights have not been found in the checkpoint, and that there were therefore randomly initialized. This is because you are loading a model trained for masked language modeling (bert-base-cased) in a model for sequence classification, the classification head is randomly initialized.
The warnings ends by telling you that your model needs to be trained because of that random initialization.


Question:
Hi everyone,
I’m converting a finetuned bert model from huggingface to onnx (following this post 1).
Transformers version: 4.10.2
When I run this on a terminal
python -m transformers.onnx --model=/path/to/checkpoint output=/tmp
I get this error;
 line 143, in validate_model_outputs
    from onnxruntime import InferenceSession, SessionOptions
ModuleNotFoundError: No module named 'onnxruntime

’
I tried installing onnxruntime-tools but I still get the same error.
Any ideas?

Context: The package to install was onnxruntime


Question:
Hello there!
I have a quite specific corpus and I wanted to try an approach where I do everything from scratch (well, almost!). Specifically, I was thinking about:

training a language model from scratch using my own corpus and creating my own tokenizer. Hugginface provides two colab for that (see Google Colab 4 for the LM)

Then, simply loading the language model created at 1. and fine-tuning it for text classification using the usual imports

from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained("my_language_model")
#do some classification
#save final model

Does that make sense? Are these the right conceptual steps?
Thanks!

Context:
Yes, that are the right steps: first pre-training, than fine-tuning a head. However, note that you need a relatively big corpus in order for pre-training to be effective. If you really have some in-domain data that’s different from the corpora that were used to pre-train models like BERT and RoBERTa, then it might be useful to do it.
Examples are BioBERT (pre-trained on biomedical language), SciBERT (pre-trained on scientific text), etc.


What to use for the target input in the decoder for autoregressive usage

Question:
I want to use a transformer (encoder and decoder) and use it for the seq 2 seq modeling.
I like to use it as autoregressive model during the inference time.
I know that I should use the targets to the decoder during the training, but then the issue is that when I do it my model overfits and performed well only if I have good targets as the input of the decoder.
I was wondering if there is a trick or rule on how should I select the target for the decoder?
my second question is if the target inputs to the decoder should be exactly similar to the target outputs of the decoder or it is kinda an art to choose the right target input for the decoder?

Context:
Encoder-decoder (seq2seq2) models like T5, BART and PEGASUS are trained using what is called “teacher forcing”, this just means supervised learning, i.e. the model needs to produce the target sentence given the source text.
Normally, if your dataset is diverse enough, it will also perform well at inference time, when using model.generate(). Using decoding techniques such as beam search and top-k sampling, you can get good results, even on unseen inputs.


Customizing the ordering of training samples

Question:
I am using BERT to classify text that (in many cases) greatly exceeds 512 tokens. What I am doing is splititng the text into segments of 512 tokens and using those as the training samples instead. However, the segments pertaining to the same sequence will of course be quite similar to one another and therefore some bias towards longer sequences is introduced. And assuming that the huggingface library randomly samples from the training set, segments from the longer sequences are more likely to be chosen.
What I want to do is override the ordering such that a segment from each sequence is used before sampling another segment from a sequence that has already been used. For shorter sequences, it is okay if the same segment needs to be sampled multiple times.
For example if I have three sequences, each composed of three segments, say: [[‘a’, ‘b’, ‘c’], [‘d’, ‘e’, ‘f’], [‘g’, ‘h’, ‘i’]], then I would want a potential training order to be something like:
‘b’, ‘f’, ‘i’, ‘a’, ‘d’, ‘h’, ‘c’, ‘e’, ‘g’
How can I accomplish such a task?

Context:
Hi,
What you can do is the same as what is explained in this tutorial: using a sliding window approach. This means that you create multiple training examples for a given text, by sliding a window (with some overlap) across the text. You can then label each training example with the label of the text. In this way, you have multiple training examples.
You just need to add an additional return_overflowing_tokens=True when calling the tokenizer.
Next, you can create a standard PyTorch dataloader, setting shuffle=True. This will automatically randomize all training examples, either coming from the same text or not.


Question:
Right now im doing this:
inputs = torch.tensor([tokenizer.encode(“The Manhattan bridge”)])
traced_script_module = torch.jit.trace(model,  inputs )
And got this error:
Tracer cannot infer type of CausalLMOutputWithPast(loss=None, logits=tensor([[[ -7.3835,  -6.2460,  -8.1929,  ...

Context: cc @valhalla


Question:
Hey everyone!
I’m currently using gbert from huggingface to do sentence similarity.
The dataset is nearly 3M
The encoding part is taking too long.
for sentence in list(data_dict.values()):
    tokens = {'input_ids': [], 'attention_mask': []}
    new_tokens = tokenizer.encode_plus(sentence, max_length=512,
                                       truncation=True, padding='max_length',
                                       return_tensors='pt',
                                       return_attention_mask=True)
    tokens['input_ids'].append(new_tokens['input_ids'][0])
    tokens['attention_mask'].append(new_tokens['attention_mask'][0])

    # reformat list of tensors into single tensor
    tokens['input_ids'] = torch.stack(tokens['input_ids'])
    tokens['attention_mask'] = torch.stack(tokens['attention_mask'])

    outputs = model(**tokens) # takes too long
    embeddings = outputs[0]

Can someone advise me how to speed up this process? Is it possible to run
outputs = model(**tokens)
on GPU? Would converting the model into an onnx help?
Thank you!

Context:
Hi,
Looking at your code, you can already make it faster in two ways: by (1) batching the sentences and (2) by using a GPU, indeed.
Deep learning models are always trained in batches of examples, hence you can also use them at inference time on batches. The tokenizer also supports preparing several examples at a time.
Here’s a code example:
from transformers import BertTokenizer, BertForSequenceClassification
import torch

model_name = "nlptown/bert-base-multilingual-uncased-sentiment"
tokenizer = BertTokenizer.from_pretrained(model_name)
model = BertForSequenceClassification.from_pretrained(model_name)

sentences = ["I like this movie a lot", "This movie is super bad"]
encoding = tokenizer(sentences, max_length=512, truncation=True, padding='max_length', return_tensors='pt')

# set device to GPU
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# put model on GPU (is done in-place)
model.to(device)

# put data on GPU
for k,v in encoding.items():
     encoding[k] = v.to(device)

# forward pass
outputs = model(**encoding)

# get predicted class indices
predicted_class_indices = outputs.logits.argmax(-1).tolist()
# turn into actual class names
predicted_classes = [model.config.id2label[label] for label in predicted_class_indices]
print(predicted_classes)

If you want to speed it up even more, then you can indeed look at converting your trained model to ONNX.


Training loss is not decreasing using TFBertModel

Question:
I have used the TFBertModel and AutoModel from the transformer library for training a two-class classification task and the training loss is not decreasing.
bert = TFBertModel.from_pretrained('bert-base-uncased')
input_ids = tf.keras.layers.Input(shape=(SEQ_LEN,), name='input_ids', dtype='int32')
mask = tf.keras.layers.Input(shape=(SEQ_LEN,), name='attention_mask', dtype='int32')
embeddings = bert(input_ids, attention_mask=mask)[1]
X = tf.keras.layers.Dropout(0.1)(embeddings)
X = tf.keras.layers.Dense(128, activation='relu')(X)
y = tf.keras.layers.Dense(1, activation='sigmoid', name='outputs')(X)
bert_model = tf.keras.Model(inputs=[input_ids, mask], outputs=y)
bert_model.compile(optimizer='adam',               loss='binary_crossentropy',               metrics=['accuracy'])
But when I use the TFBertForSequenceClassification model the model converges fast and the training loss reaches zero.
bert_model = TFBertForSequenceClassification.from_pretrained('bert-base-uncased',num_labels=2)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
optimizer = tf.keras.optimizers.Adam(learning_rate=2e-5,epsilon=1e-08)
bert_model.compile(loss=loss, optimizer=optimizer, metrics=[metric])
I want to use the sequence output of BERT and hence I need to load the model with TFBertModel or something similar which returns the outputs of BERT.
@Rocketknight1

Context:
Your code in the first block looks like it would work. I suspect the problem is one of two things:

The dropout may prevent the model from fully converging.
The default learning rate for Adam is 1e-3, which is much too high for training Transformer models. Try learning rates in the range 1e-5 to 1e-4. If training loss is still not decreasing even with a lower learning rate and no dropout then let me know and I’ll investigate further.


Can we use Gradient Checkpointing and Gradient Accumulation at Once?

Question:
Hi I’m trying to train large batch size for my model,
So can I use Gradient Checkpointing and Gradient Accumulation at once?
I’m not sure that gradient would safely added when checkpointing is done
P.S : would it be okay to use multi-GPU + Gradient Checkpointing + Gradient Accumulation at Once?

Context: Yes, those two techniques can be used together, and also with distributed training as well.


Any simple functionality to use multiple metrics together?

Question:
Some tasks like question generation requires multiple metrics (BLEU, METEOR, ROUGE).
It would be quite helpful if there is a function such as load_metric(['bleu', 'meteor', 'rouge'])

Context:
hey @ad26kr, thanks for the suggestion!
one complication i see is that some metrics like glue are paired a subtask:
metric = load_metric('glue', sub_task)

so it’s not clear what should happen if someone passes something like load_metric(["bleu", "glue"]) in your proposal. nevertheless, i invite you to open a feature request on the datasets library where it can be discussed in more detail: Issues · huggingface/datasets · GitHub 11


What to do for non-finite warning in `clip_grad_norm`?

Question:
I started to see this warning for a language model training
FutureWarning: Non-finite norm encountered in torch.nn.utils.clip_grad_norm_; continuing anyway. Note that the default behavior will change in a future release to error out if a non-finite total norm is encountered. At that point, setting error_if_nonfinite=false will be required to retain the old behavior.

Is this an indicator that my model is not working well? And if so, is there any recommendation on what to change? Thanks!

Context: Any insights on this?


Which Bert model should we use for this problem. Next Word prediction using LM? Or Keyword Extraction problem?

Question:
Hi,
I am trying to solve a text based problem. For a given text, say product description from amazon, I want to extract the brand name, type of the product, units and colour if mentioned in text.
eg:
input : Kore PVC 20-50 Kg Home Gym Set with One Plain + One Curl and One Pair Dumbbell Rods with Gym Accessories and PVC Dumbbells
output :
Kore Gym Set 50Kg
#In the output brand is Kore and type is Gym Set and Units mentioned are 50Kg
Any references to the similar problems or little bit of explanation guide on which direction should i move would be a great help.
Thanks in advance.

Context:
Maybe you can view this as a NER-like problem. That is if all the brand/type/units can directly be identified (marked) in the text. For this a BERT-model is well suited.
It depends a bit on how much labeled training data you have, and how it is organised.
Another alternative could be to view it as a summarisation problem. I would then first consider using a seq-to-seq model, like BART or T5.


Question about Gradient Accumulation step in Trainer

Question:
I can see that gradient accumulation step helps to increase batch size.
and also I can understand if the model has Batch Norm layer, gradient accumulation will not guarantee the exact same performance as the model that we trained in a large batch size (not using accumulation)
but, most of models in Transformers are based on transformer architecture which utilizes layer normalization,
so does that mean Can I guarantee that the trained model would give same metric performance in both ways? (e.g  batch size 64 with 4 batch per device * 4 gpus * 4 accumulation step  == batch size 64 with 16 batch per device * 4 gpus )
In short,
My question is for transformer models which use layer normalization, will give same model performance between train batch size in once and using gradient accumulation steps

Context: Yes, layer normalization does track statistics, so you will get the exact same thing with 4 batch size * 4 gradient accumulation or 16 batch size (and that would not be the case with neural nets using BatchNorm indeed).


Question: Hello! Can I extract a similar word from the model? Also, is it possible to assess the similarity between two words, not sentences, but only words? I’ll hope you help.

Context: Unless you want to compare words in a specific context, context-sensitive models are not the best way to do what you want. Better to look for word2vec 26 or similar then.


Attention type ‘block_sparse’ is not possible if sequence_length: 458 <= num global tokens:

Question:
I am using the pre-trained google/bigbird-pegasus-large-arxiv model.
But I receive the following update during the forward pass.
Attention type 'block_sparse' is not possible if sequence_length: 458 <= num global tokens: 2 * config.block_size + min. num sliding tokens: 3 * config.block_size + config.num_random_blocks * config.block_size + additional buffer: config.num_random_blocks * config.block_size = 704 with config.block_size = 64, config.num_random_blocks = 3.Changing attention type to 'original_full'...

I understand the update and I am aware of benefit of time and memory it saves while using block_sparse than original_full.
So, how should I go about selecting the suitable block_size and  num_random_blocks when I know that there is a lot of variation in the sequence length of my inputs?

Context: I think one workaround is to pad your sequence length to a fixed number. i.e., >= 512/1024/2048 etc


Question:
Hi, I was thinking of adding cross attention between a visual transformer and a bert model. Was wondering if there was a way that I could do this using the HF library.
What I was thinking was if somewhere in the HF Bert model API if I had access to where it took in the queries, keys, and values, I could subclass the BERT submodule and add cross attention instead of just having self attention. I’m visualizing something very much like this code snippet from the annotated transformer paper 1. Specifically the DecodeLayer class where there is self_attn as well an additional src_attn which I would need to add in.
I am also aware that I would need to copy the weights for everything but the src_attn module. Just need some mechanism to do so. Fingers crossed there is some place in the HF API that I can do exactly that.
Happy to do this myself if someone can point where in the HF library I should be looking at to see where it uses queries, keys and values arguments.

Context: bump. Sorry guys just wondering if anyone had any ideas about this.


Question:
I’m working on the CLEF dataset for a research project, which contains ~480 users’ posts, each user has a varied range of posts (text) starting from 10 to 2000, and each user has an associated binary label 0/1 (whether the user is depressed).
So, basically, I’ve to take all the posts of a user and classify the label of the user.
I’m thinking about how to go about solving this problem with HF transformers (BERT or something maybe), if anyone has any suggestions or pointers (notebook, etc.), please share, would be really helpful. Thanks
also would like to cc: @lewtun and @sgugger

Context:
Hey @rasel277144, if I understand correctly you’d like to classify whether whether a user is “depressed” based on their posts?
In this case, you could concatenate all the user posts and treat it as a standard classification problem and Sylvain has created a nice tutorial for this task here.
Having said that, you will probably run into limitations with the maximum context size of models like BERT (typically just a few paragraphs), so you might want to see if models like BigBird 1 or LongFormer can help as their context size is 8x that of BERT. If that’s still not sufficient, you might want to adapt some of the suggestions in this thread 2 to text classification (e.g. you could create an embedding for each user post, average the embeddings, and then use those embeddings for a simple logistic regression classifier)
PS I put “depressed” in quotes because I assume this is not a phenomenon we can hope to capture accurately from written text alone. I also suggest treading very carefully in this domain as there’s plenty of public examples 1 where using NLP to diagnose patient well-being leads to bad outcomes.


Question:
In a lot of cases, ONNX optimized models offer much better performance benefits as compared to using PyTorch models.
This performance boost coupled with the pipelines offered by HuggingFace are a really great combo for delivering a great experience both in terms of inference speed and model performance.
Right now, it’s possible to use ONNX models with a little bit of modification to the pipeline.py code. On my tests on the QuestionAnsweringPipeline with SQuADv2 dataset, I see performance improvements of 1.5-2x with models like  bert-large-uncased-whole-word-masking-finetuned-squad  4 and  distilbert-base-cased-distilled-squad  6
I’m wondering if this is something that the devs/community considers worthwhile for supporting directly in the transformers repo. I’m doing some work on this for my own project, but it’s mostly a hacky impl. at this point.
If there’s greater interest in this then I could try to integrate ONNX support for inference more fully in the code base.
Would love to hear everyone’s thoughts.
Thanks

Context: I know @mfuntowicz is working on this so he may have more input.