How to get cross-attention values of T5?

Question:
Hello,
I am trying to finetune the T5 model and need to get cross attention scores as well as self-attention scores. However when I set output_attentions=True, the model only returns self-attention values.
Any idea how to get cross-attention values such as 6 elements with B,8,Tx,Ty ? (num_heads=8, num_layers=6)
I am doing forward call on t5 model
t5 = T5ForConditionalGeneration.from_pretrained(“t5-small”)
outputs = t5(input_ids=input_ids, labels=output_ids, use_cache=False, output_attentions=True, output_hidden_states=True)
output returns 7 elements
outputs[0]: loss
outputs[1]: input_embeddings
outputs[2]: decoder_hiddens :  B,Ty,H
outputs[3]: decoder_self_attentions     # B,8,Ty,Ty
outputs[4]: encoder_last_hidden states ?
outputs[5]: encoder_hiddens?
outputs[6]: encoder_self_attentions # B,Tx, Tx
Thanks a lot!

Context:
Hello,
I have been trying to get the cross-attention weights as well for the MarianMT model.
I found out inside the source code that attention weights from the encoder-decoder layer wasn’t being included in the attentions being returned.
I’m basing my assumptions from looking at the DecoderLayer class here 7
If you do find a workaround , please share it on this thread !
Thanks!


Further Pretrain Basic BERT for sequence classification

Question: Has anyone successfully used Hugging Face to further pretrain Baisc BERT model and apply it to sequence classification task ? I tried to find any relevant posts/documents on Hugging Face website but failed to find any useful information. The usual way to further pretrain BERT is to use original google BERT implementation. I want to stick with Huggingface and see if there is a way to work around with this further pretraining task. If anyone could offer some help or guidances I sincerely appreciate it.

Context: You seem to be looking for the term finetuning.  You should be able to find a lot of information if you search for that term. Here is a good one to get you started: https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_multi_label_classification.ipynb 52


SavedModel export for DistilBERT is failing

Question:
Hi.
I used the pre-trained weights of the DistilBERT model in order to train a text classification model with the SST-2 dataset. Now, when I am trying to export the trained model to SavedModel format, it fails. Here’s 1 the full error trace.
Here’s the notebook 2 for the training code. I use the following lines of code in order to export the trained model to SavedModel -
!mkdir DistilBERT
model.save("DistilBERT")

Context:
Hi @sayakpaul,
Where are you running this notebook, example, in Google Colab?


Question:
I am fine tuning custom dataset using this link https://huggingface.co/transformers/master/custom_datasets.html 14
it is saving too many checkpoints that make my disk full, any suggestions?
thanks

Context: If it’s the part using Trainer, you can use the argument save_total_limit (in TrainingArguments) to limit the number of checkpoints kept.


Looking for tool class to do predictions

Question:
Hi,
I am looking for a tool class to do predictions with BERT models. Like the Trainer class just for prediction time. There is much more to do than just doing logits = model(input_ids). When you have large input lists of text you have to do this in batches or you get an OOM from your GPU and there is some more advanced stuff like smart batching (sorting the sentences by token length and batch short sentences with other short sentences to have short batches with minimal padding).
Is there something like this?
Thanks
Philip

Context:
Hi Philip,
maybe the HF Pipelines implementation could help:

      github.com

    huggingface/transformers/blob/master/notebooks/03-pipelines.ipynb 2
{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.6"

  This file has been truncated. show original

It should also support e.g. multiple sentences (List[str]) as inputs and you could use of course own trained and fine-tuned models. Detailed implementation can be found here:

      github.com

    huggingface/transformers/blob/master/src/transformers/pipelines.py 3
# coding=utf-8
# Copyright 2018 The HuggingFace Inc. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import csv
import json
import os
import pickle

  This file has been truncated. show original

To use it for text classification/sentiment analysis:

nlp_sentence_classif = pipeline('sentiment-analysis')
nlp_sentence_classif('Such a nice weather outside !')

and it also outputs the score:
[{'label': 'POSITIVE', 'score': 0.9997656}]

I like using Pipelines especially for testing NER models


Change default models text-classification text

Question:
Is there any way to change then default text for models from “I like you. I love you” to something else ?
I want to change it to something like that because my model is based on protein sequences not text:

      huggingface.co

Rostlab/prot_bert_bfd_membrane · Hugging Face

Context: Yes! Have a look at this PR 4


Performance with new NVIDIA RTX 30 series

Question:
Hi there,
I just got my new RTX 3090 and spent the whole weekend on compiling PyTorch 1.8 (latest master) with the latest CUDA 11.1 (that should support the new 30 series properly).
I did some performance comparisons against a 2080 TI for token classification and question answering and want to share the results
For token classification I just measured the iterations per second for fine-tuning multi-lingual BERT on GermEval 2014 dataset. With the 2080 TI, I achieved 4 iterations per second, with new RTX 2090 6.8 iterations per second. Using fp16 the 2080 TI achieved 7.7 iterations per second and 8.4 iterations per second with the RTX 3090 (and “native” support of half precision).
I also measured the time that is needed to fine-tune a model on SQuAD. With the 2080 TI it took 38:57 minutes for one epoch and 22:25 minutes on the RTX 3090.
I’m planing to do more benchmarks with other examples. I don’t have any TITAN RTX card, but if someone from the community could provide some benchmarks (also for other cards!!), it would be awesome

Context:
I have a Titan RTX card with some spare runtime.  I’d be interested in running some SQuAD 2.0 fine-tuning comparisons to your RTX 3090.
My modelcard for  'ahotrod/electra_large_discriminator_squad2_512`  5 includes the fine-tuning script file, Tensorboard loss rate image, nvidia-smi image for memory usage, and other specs for the 4-1/2 hr fine-tuning.  See complete details in “List all files in model”  of the modelcard.
deepset-ai/haystack ( https://github.com/deepset-ai/haystack 13 ) has convenient reader speed and accuracy benchmarks for benchmarking RTX3090 vs Titan RTX, https://github.com/deepset-ai/haystack/issues/441#issuecomment-701748635 10 using reader.eval_on_file("dev-v2.0.json")


Question:
A couple of questions/thoughts related to the distillation of the text-generation models:

I see that Distill-T5 is not functional.
https://github.com/huggingface/transformers/blob/master/examples/seq2seq/distillation.py#L299-L301 84
Wondering if there is any concrete technical barrier that prevents us from extending the same distillation that worked for BART to T5.

I wonder if it is more feasible to distill from T5 to BART. Any thoughts?

Tagging @sshleifer since I think you might know more on this topics.

Context:
For which task?

t5 distillation is very feasible, I just got excited about bart/pegasus since it performed the best in my summarization experiments.  There is no feasability issue.

It is much less feasible to distill from t5 -> bart than to distill from a large finetuned t5 checkpoint to a smaller one.


What is default value for number of labels in BertForSequenceClassification?

Question:
Hi,
I try to rebuild the TFBertForSequenceClassification model from the plain TFBertModel.
TFBertForSequenceClassification uses a dropout and a dense layer on top of the main BERT model. The number of neurons is config.num_labels.
I wonder, where is this values because it seems it has no default value.
And finetuning TFBertForSequenceClassification like here:

      huggingface.co

Training and fine-tuning — transformers 3.3.0 documentation 10

does not handle a value for num_labels.
So where does the 2 neurons come from in the last layer (by default)?
Look here:

      huggingface.co

transformers.modeling_tf_bert — transformers 3.3.0 documentation 1

there is
self.num_labels = config.num_labels
but config.num_labels has no default value?
See

Context:
No it has no default value because it is supposed to be passed at init either because:

the user is loading a pretrained model and its config has some num_labels

the user is passes num_labels=n when using a generic pretrained model and it udpates the config.


Question:
Hello,
I have a few questions about the BertModelLMHeadModel:

Is BertModelLMHeadModel  used to conduct the regular language modeling (next token prediction), as it is the case for the GPT2LMHeadModel?

For GPT2LMHeadModel, I can just specify labels = input_ids for convenience. I just specify the labels in this way for the BertModelLMHeadModel as well?

Thanks,

Context:
Hi @h56cho
do you mean the BertLMHeadModel ?
If yes then, it’s intended to be used with the EncoderDecoder model which allows you to use pre-trained encoder for as both encoder and decoder for seq2seq tasks.  It’s not intended for language modeling.
While you can use that class as a standalone decoder by passing is_decoder=True  to config it might not give you good results as it’s trained as an encoder.


Question:
Hi All,
Just want to know, is there any way to batch decode variable length sentences.
For example [S1, S2] , where S1 has 5 words abd S2 has 10 words . Can we decode it using GPT2 , BERT etc?

Context:
Hi @s4sarath
What do you mean by decoding, as in decoding the generated tokens by GPT2  or making predictions on a batch of sequences ?


Question:
[TL;DR]
Please go through the questions and any help would be appreciated.
I am trying to train GPT2 model from scratch.
I am not sure if I am doing right and I have got a few questions.  Here is my current implementation.
Tokenizer:
Question1:
Am I training the tokenizer right way? Should I use all of training text files to train tokenizers?
from pathlib import Path

from tokenizers import ByteLevelBPETokenizer

paths = [str(x) for x in Path(".").glob("**/*.txt")]

# Initialize a tokenizer
tokenizer = ByteLevelBPETokenizer()

# Customize training
tokenizer.train(files=paths, vocab_size=52_000, min_frequency=2, special_tokens=[
    "<s>",
    "<pad>",
    "</s>",
    "<unk>",
    "<mask>",
])

!mkdir German
tokenizer.save_model("German")

The above code gives me : [‘German/vocab.json’, ‘German/merges.txt’]
Initializing the GPT2 tokenizer
Question2: Is this the right way to do it?
from transformers import GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained("./German",additional_special_tokens=["<s>","<pad>","</s>","<unk>","<mask>"],pad_token='<pad>' ,max_len=512)

Initlizing the GPT2 model:
Question3: Am I initilizing GPT2 Language model properly?
from transformers import GPT2Model, GPT2Config,GPT2LMHeadModel

# Initializing a GPT2 configuration
configuration = GPT2Config(vocab_size=52_000)
model = GPT2LMHeadModel(config=configuration)

Dataset:
My dataset is just an example text file containing 1 million German Dataset:
dataset1646×292 79.8 KB
Question4: Now What kind of dataset do you suggest I use to train this model?
Here is my logic for Dataset loading and training.
from transformers import TextDataset

dataset = TextDataset(
    tokenizer=tokenizer,
    file_path="./deu-de_web-public_2019_1M-sentences.txt",
    block_size=128,
)

from transformers import DataCollatorForLanguageModeling

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=False,
)

from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="./output",
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_gpu_train_batch_size=64,
    save_steps=10_000,
    save_total_limit=2,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
    prediction_loss_only=True,
)

trainer.train()

Question5: Is this the way to train the model? Is there any specific data format the model expects the training data to be?
Question6: HOw can we use multiple GPUs?

Context: Hey, @vikasRajashekar did this method worked? I am trying this on Hindi using the same approach


Is the multiple-choice head for the pre-trained `LongformerForMultipleChoice` model pre-trained as well?

Question:
Hello,
Is the multiple-choice head for the pre-trained LongformerForMultipleChoice model pre-trained as well?
I am asking because like for the pre-trained GPT2DoubleHeadsModel, the main body of the model is trained but its multiple-choice head is not.
Is the multiple-choice head for the pre-trained LongformerForMultipleChoice model untrained, just like the case for the GPT2DoubleHeadsModel?
Thank you,

Context:
Hi @h56cho
Yes , multiple-choice head is randomly initialized, you need to train these heads according to your  tasks.


Trainer doesn’t show the loss at each step

Question:
Hi,
is there a way to display/print the loss (or metrics if you are evaluating) at each step (or n steps) or every time you log? I don’t see any option for that. This is very important cause’ it is the only way to tell if the model is learning or not. I thought “debug” was going to work but it seems to be deprecated. I am training in a jupyter notebook by the way.
Also, what about an early stopping option?

Context: The loss and metrics are printed every logging_steps (there was w bug recently fixed, so you might need to update your install to an installation from source). As for early stopping, there is a PR under review with it, so it should come soon.


Calculating accuracy during fine-tuning the BERTForMaskedLM

Question:
Hello,
While fine-tuning, we can only see loss and perplexity which is useful.
Is it also possible to see the accuracy of the model and also the tensorboard when using the “run_language_modeling.py” script? It would be really helpful if anyone could explain how the “loss” is calculated for BERTForMaskedLM task (as there are no labels provided while fine-tuning).

Context: To replicate the original training loss from the paper it should be calculated as “The training loss is the sum of the mean masked LM likelihood and the mean next sentence prediction likelihood.” You have more details in the BERT paper.


How do I do multi Class (multi head) classification?

Question:
Hi,
I want to use BERT models to do multi Class (multi head) classification. I have text and want to do a binary classification for churn and one binary classification for sentiment.
Is that possible “out of the box”? Or do I have to develop my own “BertForSequenceClassification” Class?
Thanks
Philip

Context: This is all in the loss function, so you can definitely use BertForSequenceClassification with two labels, then use the proper loss function (probably BCEWithLogitsLoss).


Question:
@valhalla
Questions about Seq2SeqTrainer

does it know how to do multigpu sortish sampler?
does it know how to sync metrics in a multigpu setting?
is TPU faster than GPU  ?

Experiments

Would be interested to know how  finetune bart-large on xsum performs, for example, esp. on TPU .
Would be cool to make a blogpost showing some sort of optuna/ray search for summarization or translation finetuning. Would also be cool to know the best hparams.
Good hparams to check are learning rate, gradient_accumulation_steps, label_smoothing, dropout.

Context:
Yes, it uses DistributedSortishSampler for multi-gpu, it’s handled here 9

Yes, metrics are handled by Trainer and  logged in the global  main process. pinging @sgugger for confirmation. TPU  training is multi-device, so for example you see TPU logs here for distilbart-cnn-12-3.

(Note: as generate   crashes/hangs on TPU we don’t calculate BLEU, ROUGE metrics on TPU)

Yes,  I’ve observed 3-4x speed boost on TPU v3-8 compared to single V100

distilbart-cnn-12-3, 1 epoch

on TPU V3-8: 25.5 min
on V100: ~102 min

marin-enro-6-3 , 1 epoch

on TPU v3-8: ~5min
on V100 : 21 min

All experiments are logged in this wandb project 6 if you want to compare yourself .

Coming soon


BertForSequenceClassification only seems to have linear activation at the end - is this a bug?

Question:
Hi,
the BertForSequenceClassification class seems to have a linear activation at the end of the head.
See here: https://github.com/huggingface/transformers/blob/3323146e904a1092def4e8527de9d2a7479c1c14/src/transformers/modeling_bert.py#L1351 39
IMO for a binary classification it should have a sigmoid function at the end and for a one of multiple classes classification there should be a softmax at the end. How does this come?
Thanks
Philip

Context:
Hi @PhilipMay
This is not a bug, At L1351 the pooled output is passed through classification head (linear layer) to get the logits
CrossEntropyLoss does not require softmax , it calculates loss using logits
BertForSequenceClassification returns logits, then you can the apply softmax on the returned logits to get the class scores.


Preventing every dropout in the GPT2DoubleHeadsModel

Question:
Hello,
If I execute the lines below, would they prevent every dropout in the GPT2DoubleHeadsModel? Or do I need to change extra settings in order to completely prevent the dropout in my  GPT2DoubleHeadsModel? Thank you,
    # disable dropouts for the model_gpt2_double_heads.
    model_gpt2_double_heads.config.resid_pdrop = 0.
    model_gpt2_double_heads.config.embd_pdrop = 0.
    model_gpt2_double_heads.config.attn_pdrop = 0.
    model_gpt2_double_heads.config.summary_first_dropout = 0.

Context:
You should do that when instantiating your model:
GPT2DoubleHeadsModel.from_pretrainec(checkpoint, resid_pdrop=0, embd_pdrop=0., ...)

for instance


Does using FP16 help accelerate generation? (HuggingFace BART)

Question:
I also this question in StackOverflow, but couldn’t get a response yet (https://stackoverflow.com/questions/64102036/does-using-fp16-help-accelerate-generation-huggingface-bart 32).
I follow the guide below to use FP16 in PyTorch.

      pytorch.org

PyTorch 18

An open source deep learning platform that provides a seamless path from research prototyping to production deployment.

Basically, I’m using BART in HuggingFace for generation

During the training phase, I’m able to get 2x speedup and less GPU memory consumption

But.

I found out there is no speedup when I call model.generate under torch.cuda.amp.autocast().

with torch.cuda.amp.autocast():
   model.generate(...)

When I save the model by:

model.save_pretrained("model_folder")

the size does not decrease to half. But I have to call model.half() before saving in order to make the model half size.
Thus, my questions:

Is the issue in 1. expected or there should be something I did wrong?
Is the operation I did in 2. proper?

Context:
To use the model for inference in fp16 you should call model.half() after loading it.
Note that calling half puts all models weights in fp16, but in mixed precision training some parts are still kept in fp32 for stability (like softmax layers), so it might be a better idea to use amp in 01 opt mode instead of calling half.
I’m not sure if torch.cuda.amp.autocast()  helps in inference, better to ask this on pytorch forum


Finetuning GPT2 using Multiple GPU and Trainer

Question:
I’m finetuning GPT2 on my corpus for text generation. I am also using the Trainer class to handle the training. I have multiple gpu available to me. As I understand from the documentation and forum, if I wanted to utilze these multiple gpu for training in Trainer, I would set the no_cuda parameter to False (which it is by default). Is there anything else that needs to be done in order to utilize these gpu in Trainer for training?
Thanks in advance!

Context: It starts training on multiple GPU’s if available. You can control which GPU’s to use using CUDA_VISIBLE_DEVICES environment variable i.e if CUDA_VISIBLE_DEVICES=1,2 then it’ll use the 1 and 2 cuda devices. Pinging @sgugger for more info.


TypeError when loading a BERT model using TFAutoModel

Question:
When I load the bert-base-german-cased pytorch model using the following command lines:

from transformers.modeling_tf_auto import TFAutoModel
bert_model = TFAutoModel.from_pretrained(‘bert-base-german-cased’, from_pt=True, config=self.bert_config)

or the tensorflow checkpoint:

TFAutoModel.from_pretrained(‘path_to_bert_base_german_cased_tensorflow_checkpoint’, config=self.bert_config)

I get the following error:
   Traceback (most recent call last):
  File "train.py", line 23, in <module>
    model = util.get_model(config)
  File "/home/asi/PycharmProjects/coreference_resolution/util.py", line 21, in get_model
    return independent.CorefModel(config)
  File "/home/asi/PycharmProjects/coreference_resolution/independent.py", line 59, in __init__
    self.predictions, self.loss = self.get_predictions_and_loss(*self.input_tensors)
  File "/home/asi/PycharmProjects/coreference_resolution/independent.py", line 263, in get_predictions_and_loss
    bert_model = TFAutoModel.from_pretrained('bert-base-german-cased', from_pt=True, config=self.bert_config)
  File "/home/asi/anaconda3/envs/coref_resol/lib/python3.6/site-packages/transformers/modeling_tf_auto.py", line 423, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "/home/asi/anaconda3/envs/coref_resol/lib/python3.6/site-packages/transformers/modeling_tf_utils.py", line 482, in from_pretrained
    return load_pytorch_checkpoint_in_tf2_model(model, resolved_archive_file, allow_missing_keys=True)
  File "/home/asi/anaconda3/envs/coref_resol/lib/python3.6/site-packages/transformers/modeling_tf_pytorch_utils.py", line 93, in load_pytorch_checkpoint_in_tf2_model
    tf_model, pt_state_dict, tf_inputs=tf_inputs, allow_missing_keys=allow_missing_keys
  File "/home/asi/anaconda3/envs/coref_resol/lib/python3.6/site-packages/transformers/modeling_tf_pytorch_utils.py", line 125, in load_pytorch_weights_in_tf2_model
    tf_model(tf_inputs, training=False)  # Make sure model is built
  File "/home/asi/anaconda3/envs/coref_resol/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 854, in __call__
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "/home/asi/anaconda3/envs/coref_resol/lib/python3.6/site-packages/tensorflow_core/python/autograph/impl/api.py", line 237, in wrapper
    raise e.ag_error_metadata.to_exception(e)
TypeError: in converted code:
    relative to /home/asi/anaconda3/envs/coref_resol/lib/python3.6/site-packages:

    transformers/modeling_tf_bert.py:739 call  *
        outputs = self.bert(inputs, **kwargs)
    tensorflow_core/python/keras/engine/base_layer.py:854 __call__
        outputs = call_fn(cast_inputs, *args, **kwargs)
    transformers/modeling_tf_bert.py:606 call  *
        embedding_output = self.embeddings([input_ids, position_ids, token_type_ids, inputs_embeds], training=training)
    tensorflow_core/python/keras/engine/base_layer.py:894 __call__
        self._maybe_build(inputs)
    tensorflow_core/python/keras/engine/base_layer.py:2146 _maybe_build
        self.build(input_shapes)
    transformers/modeling_tf_bert.py:146 build
        initializer=get_initializer(self.initializer_range),
    tensorflow_core/python/keras/engine/base_layer.py:529 add_weight
        aggregation=aggregation)
    tensorflow_core/python/training/tracking/base.py:712 _add_variable_with_custom_getter
        **kwargs_for_getter)
    tensorflow_core/python/keras/engine/base_layer_utils.py:139 make_variable
        shape=variable_shape if variable_shape else None)
    tensorflow_core/python/ops/variables.py:258 __call__
        return cls._variable_v1_call(*args, **kwargs)
    tensorflow_core/python/ops/variables.py:219 _variable_v1_call
        shape=shape)
    tensorflow_core/python/ops/variables.py:197 <lambda>
        previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
    tensorflow_core/python/ops/variable_scope.py:2503 default_variable_creator
        shape=shape)
    tensorflow_core/python/ops/variables.py:262 __call__
        return super(VariableMetaclass, cls).__call__(*args, **kwargs)
    tensorflow_core/python/ops/resource_variable_ops.py:1406 __init__
        distribute_strategy=distribute_strategy)
    tensorflow_core/python/ops/resource_variable_ops.py:1537 _init_from_args
        initial_value() if init_from_fn else initial_value,
    tensorflow_core/python/keras/engine/base_layer_utils.py:119 <lambda>
        init_val = lambda: initializer(shape, dtype=dtype)
    tensorflow_core/python/ops/init_ops.py:369 __call__
        shape, self.mean, self.stddev, dtype, seed=self.seed)
    tensorflow_core/python/ops/random_ops.py:171 truncated_normal
        mean_tensor = ops.convert_to_tensor(mean, dtype=dtype, name="mean")
    tensorflow_core/python/framework/ops.py:1184 convert_to_tensor
        return convert_to_tensor_v2(value, dtype, preferred_dtype, name)
    tensorflow_core/python/framework/ops.py:1242 convert_to_tensor_v2
        as_ref=False)
    tensorflow_core/python/framework/ops.py:1297 internal_convert_to_tensor
        ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
    tensorflow_core/python/framework/tensor_conversion_registry.py:52 _default_conversion_function
        return constant_op.constant(value, dtype, name=name)
    tensorflow_core/python/framework/constant_op.py:227 constant
        allow_broadcast=True)
    tensorflow_core/python/framework/constant_op.py:265 _constant_impl
        allow_broadcast=allow_broadcast))
    tensorflow_core/python/framework/tensor_util.py:449 make_tensor_proto
        _AssertCompatible(values, dtype)
    tensorflow_core/python/framework/tensor_util.py:331 _AssertCompatible
        (dtype.name, repr(mismatch), type(mismatch).__name__))

    TypeError: Expected int32, got 0.0 of type 'float' instead.

Environment info is as follows:
transformers: 3.0.2
tensorflow and tensorflow-gpu: 1.15.2
platform: ubuntu 16.04
CUDA: V10.0.130
python: 3.6.10
Similar problem discussed in the following link when importing TFDistilBert model: https://github.com/huggingface/transformers/issues/2641 12 but did not solve my issue.
Any clues how to fix this issue?
Thanks

Context: I had the same problem - Dana


Question: I’m using a text generation pipeline to generate text after some prompt. My chatbot requires me to have a prompt of say 64 tokens and to generate a maximum length of 32 tokens. If I set max_length=32 it tells me “The context has 64 number of tokens, but max_length is only 32.” If I set max_length=32+64, it generates 96 tokens on top of my context, so in total 64+96=160 tokens. How can I generate only 32 tokens on top of my prompt?

Context:
length = len(prompt)
max_length = 32 + length


[new model] FSMT has been released + 9 models ported

Question:
FSMT (=FairSeq Machine Translation) architecture has been ported from fairseq wmt19 1.
9 trained models have been ported and made available to everybody:

4 wmt19 facebook models 5 (Sergei Edunov, et al)

3 wmt16 and 2 wmt19 allenai models 3 (Jungo Kasai, et al)

You will find all the necessary details, including sample code and bleu scores and the scoring code on the model pages.
At the moment only the translation part has been thoroughly tested and eval’ed with sacrebleu, other parts (training/finetuning/etc.) may or may not work, so if you encounter any problems please file an issue 1 and tag @stas00 to it.
It appears to be giving the best score for en <=> ru and en <=> de for the current transformers models we have (scores acquired using sacrebleu against the wmt19 dataset). Metrics on v100 using SortishSampler + --fp16 --bs 64:
screenshot_9762×182 19.4 KB
All the credits for the high BLEU scores go to the facebook team who did the original architecture design and massive pretraining.
Our scores are a tad below fairseq’s scores, since transformers at the moment doesn’t support model ensemble. So we use the best performing checkpoint.
I’m also working on an article of how the porting process went, so it might help others to do similar work. I will link to it once it’s done.
I want to thank @sshleifer for your incredible support, time and mentorship in this difficult process. Your caring help and encouragement were invaluable to me, Sam! I especially appreciated how every so often you were gently suggesting that I get back to work on this difficult project and not get distracted with multiple minor easy improvements I wanted to make for the transformers repo. Thank you.
Conversion Instructions
This should work for fairseq models with moses+bpe tokenizers:
export save_dir='cool_new_fr-en'
python src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py \
    --fsmt_checkpoint_path path_to_fairseq_model.pt \
    --pytorch_dump_folder_path $save_dir
cp README.md $save_dir/  # model card
transformers-cli upload $save_dir

For examples please see these scripts that provide conversion for full sets of models:

allenai-wmt19 1
allenai-wmt16 1
facebook-wmt19 1

Context: Great work stas   You finished a hard project with great test coverage!


Question: Is it possible to use EncoderDecoderModel with encoder decoder parameter sharing as used in https://arxiv.org/abs/1907.12461 2 (ROBERTASHARE, BERTSHARE)? Model still achieves state-of-art results while it’s more affordable to train it. Or maybe there are some plans to implement this?

Context:
Yes ! This can be done using
roberta2roberta = EncoderDecoderModel.from_encoder_decoder_pretrained("roberta-base", "roberta-base", tie_encoder_decoder=True)

Here’s an example model 14


Chatbot using Reformer model. Not able to find “token_type_ids” like BERT

Question:
I want to use Reformer model as Chatbot.
So in order to train it along with the input_ids and lm_labels. I am looking for token_type_ids to encode the context, speaker1 and speaker2 information.
But from the documentation of Reformer
*class* `transformers.` `ReformerForMaskedLM`
`forward` ( *input_ids=None* ,  *position_ids=None* ,  *attention_mask=None* ,  *head_mask=None* ,  *inputs_embeds=None* ,  *num_hashes=None* ,  *labels=None* ,  *output_hidden_states=None* ,  *output_attentions=None* ,  *return_dict=None* )

I do not see “token_type_ids”. How do I fine-tune this model to work as a chatbot?
Also, How do I train this model from scratch in some other language for example German.

Context:
there are no token_type_ids for reformer, you could some special delimiter tokens to separate context and speakers.
This article 25 could help


Question:
I am using the Question and Answering model and the run time is about 14 seconds per data point, my project of ~7000 text data samples took around 26 hours.
What is the best run time I can expect? Will this improve in the near future?

Context:
Do you think you can provide some more information ?
Are you using GPU, pipeline, which  model are you using etc


How does T5 create the correct decoder_input_ids?

Question:
huggingface.co

T5 — transformers 3.1.0 documentation 3

I have been reading the documentation for the T5 model and in the training section, for both unsupervised denoising and supervised training, the comment states the model is able to create the correct decoder_input_ids. However, I’m not sure how the model is able to do. The documentation tells me that if the decoder_input_ids is None (which it is by default) it takes the values of input_ids. Does this mean the start token is the first token of the input sequence?
The description also mentions a start-sequence token. But I can’t seem to find where this is generated or what the token is.

Context:
The T5PreTrainedModel._shift_right method here 29 takes labels and prepares decoder_input_ids
Basically, all it does is,
takes labels -> add pad token at the beginning -> remove the eos -> decoder_input_ids


Fairseq Roberta to Transformers: torch.nn.modules.module.ModuleAttributeError: ‘RobertaModel’ object has no attribute ‘decoder’

Question:
I trained a roberta model with fairseq and I am trying to convert it with the class convert_roberta_original_pytorch_checkpoint_to_pytorch 5 and its method : convert_roberta_checkpoint_to_pytorch.
I provide the checkpoints of the fairseq traning, the checkpoint_best.pt is the model.pt. The dict.txt is also given.
However when running
convert_roberta_original_pytorch_checkpoint_to_pytorch.convert_roberta_checkpoint_to_pytorch(’/path/to/checkpoints’,‘savind_directory’,False)
I get "torch.nn.modules.module.ModuleAttributeError: ‘RobertaModel’ object has no attribute ‘decoder’ "
The error is raised when meeting this line of code:
roberta_sent_encoder = roberta.model.decoder.sentence_encoder
What does this mean ? What is missing ? How can get this attribute from my trained fairseq model

Context: FWIW, it looks like it has been fixed here 130.


Use of “input_ids,token_type_ids and lm_labels” in BERT Language model

Question:
I am trying to figure out what is happening when we try to train/fine-tune Lanuguage model like BERT.
Here is the sample code:
model = BertForMaskedLM.from_pretrained("bert-base-german-cased")
model.config.is_decoder = True

lm_loss, prediction_scores, *_ = model(input_ids = input_ids, token_type_ids= token_type_ids, lm_labels = lm_labels)

What happens under the hood?? now the tokeized
input_ids example: [4,5,6,7,8,9,10,11]
token_type_ids:[0,0,0,0,1,1,1,1]
lm_labels:[-1,-1,-1,-1,8,9,10,11]
1.Lets say I pass the above to the model. Does it mean that thee loss is calculated only for last 4 values of the input_ids?
2.Does it mean network tries to learn that first 4 elements of input_ids are part of one sentence and other 4 is part of other sentence and it learns to predict the series?
I have kept the example as simple as possible so that it is easy to answer in simple way.

Context:
Hi @vikasRajashekar,
I assume what you said lm_labels is labels, and -1 is -100. (see docs here 5)

 vikasRajashekar:

1.Lets say I pass the above to the model. Does it mean that thee loss is calculated only for last 4 values of the input_ids?

Yes

 vikasRajashekar:

2.Does it mean network tries to learn that first 4 elements of input_ids are part of one sentence and other 4 is part of other sentence and it learns to predict the series?

The model tries to learn to predict the last 4 tokens from the context. The context is all input tokens, includes the last four tokens, even the last four tokens are masked or replaced tokens, they contribute correct position information to context. Any way, all input tokens will be used to predict the last four tokens.


Question:
I am no expert but have learned a few things
check your yaml front matter with this tool
https://nodeca.github.io/js-yaml/
if it says “expected a single document in the stream, but found more”
Your yaml is valid, otherwise your yaml is invalid.
like this is valid
Valid:
---
language:
- en
- zh
tags:
- translation
license: apache-2.0
---
### My fancy model

No spaces before tags (like “-zh”) will make this invalid.
More to come as I learn more!
Also the website takes 3 minutes for new tags to be reflected.

Context:
What tags can you add?
---
language: "ISO 639-1 code for your language, or `multilingual`"
thumbnail: "url to a thumbnail used in social sharing"
tags:
- array
- of
- tags
license: "any valid license identifier"
datasets:
- array of dataset identifiers
metrics:
- array of metric identifiers
---


Is_pretokenized argument for tokenizer doesn’t work?

Question:
Or maybe I’m misunderstanding how it’s supposed to be used. I’m tokenizing my whole text and then dividing the tokenized text into chunks of size 126 or less (all but the last chunk are of size 126). Then I send the list of chunks into the tokenizer:
batch_encoding = tokenizer(text_chunks, add_special_tokens=True, padding='max_length', max_length=max_seq_len, is_pretokenized=True, return_attention_mask=True)

What I want it to do is simply add the special tokens [CLS], [SEP], and [PAD]  (if needed), and then turn the wordpieces into input_ids. Instead it does another tokenization with the wordpieces:

[‘ka’, ‘##vana’, ‘##ugh’]

is turned into

[‘ka’, ‘#’, ‘#’, ‘van’, ‘##a’, ‘#’, ‘#’, ‘u’, ‘##gh’].

Context:
Hi there! The argument is_pretokenized is for when your inputs have been pre-tokenized, that is, split into words (since the tokenizers of transformers are all subwords tokenizers). So you should pass your tokenizer ["I", "am", "talking", "about", "Kavanaugh"] with is_pretokenized=True for instance, not ["I", "am", "talk", "##ing", "about", "Ka", "##vana", "##ugh"].
If you just want to add the special tokens, the method you want to call is tokenizer.prepare_for_model I believe.
I could make something clearer in the docs but you’re far from being the first user that got confused, so I think we are going to rename this argument to something less confusing.


Pytorch NLP model doesn’t use GPU when making inference

Question:
There is NLP model trained on Pytorch to be run in Jetson Xavier. I installed Jetson stats to monitor usage of CPU and GPU. When I run the Python script, only CPU cores work on-load, GPU bar does not increase. I have searched on Google about that with keywords of " How to check if pytorch is using the GPU?" and checked results on stackoverflow.com 21 etc. According to their advices to someone else facing similar issue, cuda is available and there is cuda device in my Jetson Xavier. However, I don’t understand why GPU bar does not change, CPU core bars go to the ends.
I don’t want to use CPU, it takes so long to compute. In my opinion, it uses CPU, not GPU. How can I be sure and if it uses CPU, how can I change it to GPU?
Note:  Model is taken from huggingface transformers library. I have tried to use cuda() method on the model. (model.cuda()) In this scenario, GPU is used but I can not get an output from model and raises exception.

Context:
model.cuda() or model.to("cuda") puts the model on GPU, you also need to put the inputs on GPU when model is on GPU, may be that’s why you are getting the exception.
Could post your code snippet and the exception, so we can take a look ?


Using Pegasus Model for Transfer Learning is generating garbage summaries

Question:
Hi ,
I am using the PegasusForConditionalGeneration from transformers library for transfer learning and generate summaries of chats based on samsum dataset. But, the output generated is repetitive and not related at all to the context.
Also, getting loss as nan. Can someone help me regarding what went wrong?
Please note that the same code works perfectly fine for T5ForConditionalGeneration.
Is there any difference in the implementation method?
Should I try with other Loss Metrics or Optimizer?

Context: Could you post the command you are using for fine-tuning ?


CircleCI+Github Actions: Which tests run where and when

Question:
@stas
9 CircleCi jobs that run every time a pull request gets new commits. You know these well
There are 3 jobs run by github actions here:

torch hub integration - checks whether torch hub integration works. Julien is in charge if that breaks. It doesn’t break very often.

self-hosted/push: runs fast tests on GPU only on commits on master. It only runs if a commit on master has updated the code in one of the following folders: src, tests``, .github` (to prevent running on added model cards, notebooks, etc.) (you can see what it does in .github/self-push.yaml)

self-hosted/scheduled.yaml: (`.github/self-scheduled.yaml’) does
RUN_SLOW=1 USE_CUDA=1 pytest tests/
RUN_SLOW=1 USE_CUDA=1 pytest examples/

on the same self-hosted AWS v100 machine.

Context: Thanks a lot, @sshleifer -  I was able to see where the results are - that’s great!


Explicitly set number of training steps using Trainer

Question:
I’m using Trainer to handle finetuning a GPT2 model. I see in TrainingArguments there is a max_steps that overrides num_train_epochs.
For a batch size of 32, is setting max_steps=1000000 the equivalent of setting num_train_epochs=31250?
Also, what happens if I have a batch size of 6 but want to set max_steps=1000000? Does Trainer stop training after the nearest divisible whole number or does it change the batch size at the end?
Thanks in advance!!

Context: The number of steps is the number of update steps. It’s not the number of training examples to be seen.


Question:
Some useful resources:
Search ISO 639 code tables to see if a multilingual model is what you think:
https://iso639-3.sil.org/code_tables/639/data?title=aav&field_iso639_cd_st_mmbrshp_639_1_tid=All&name_3=&field_iso639_element_scope_tid=All&field_iso639_language_type_tid=All&items_per_page=200 17
Otherwise feel free to ask something like “Do you have english to german?” Yes, en-de!

Context:
This gist lists all newly added three letter codes and the consituent languages (often other three letter codes) https://gist.github.com/sshleifer/e79fbbabe0fab3da519fd39edffee4d2 12
If you know your lang’s ISO-639-3 code, you can cmd-f for it in that file to see which models will support it.


[Proposal] Copy Pasting modeling_bart.py

Question:
We are considering copy pasting the contents of modeling_bart.py into modeling_pegasus.py, modeling_mbart.py and  modeling_marian.py.  and wanted to see what the community thinks.
Currently, here is modeling_pegasus.py:

      github.com

    huggingface/transformers/blob/master/src/transformers/modeling_pegasus.py 3
# coding=utf-8
# Copyright 2020 Google and The HuggingFace Inc. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""PyTorch Pegasus model, ported from https://github.com/google-research/pegasus"""

from .configuration_pegasus import PegasusConfig
from .file_utils import add_start_docstrings
from .modeling_bart import BART_START_DOCSTRING, BartForConditionalGeneration

  This file has been truncated. show original

marian and mbart are similar. They do nothing and inherit the complete implementation from bart. They are only their own classes so that AutoConfig/ AutoTokenizer work properly.
Arguments for copy paste

if you want to know how a model works you can read it in one file.
This is how the rest of the repo works and there is no good reason to introduce a stylistic inconsistency here.
we can delete 6 configuration parameters, that would become always True/always False.

        # params needed for mbart
        self.scale_embedding = scale_embedding  # scale factor will be sqrt(d_model) if True
        self.normalize_embedding = normalize_embedding  # True for mbart, False otherwise
        self.normalize_before = normalize_before  # combo of fairseq's encoder_ and decoder_normalize_before
        self.add_final_layer_norm = add_final_layer_norm

        # Params introduced for Marian
        self.add_bias_logits = add_bias_logits
        self.static_position_embeddings = static_position_embeddings

we could also delete 6 corresponding if statements
it might be easier to make one off fixes or modify one model without unintended consequences.
in the survey, 65% of respondents said they wanted more “Self-contained” code. This is what users want!

Arguments against copy paste

we have to add and maintain about 4,500 lines of code. Could slow us down on adding new features/models.
this might make the code harder to maintain. Some bugs/features that require 1 change will require 4 changes.
the implementations could drift, making it more difficult for the same finetuning code to work with multiple models.
Integration tests enforce that you can’t have big unintended consequences when you are refactoring, so the “unintended side effects” point is moot.
It is actually harder to understand what is going on in pegasus/mbart/marian if the code is duplicated. Seeing that pegasus state dict is identical to bart makes reading the code easier.  “It is annoying to have to read so many similar implementations and have no easy way to see the delta.”
Do not repeat yourself/rule of 3 are best practice and there is no good reason for this lib to deviate.
In the survey, people said they wanted better examples/more models faster, having to maintain more code would distract the team from those objectives.
Maintaining equivalent test coverage after copy paste would require adding 111 tests!

Would love to hear other people’s opinions as there is some disagreement within the team.
You can vote here 11 and comment below.
I am probably missing alternative solutions, pros and cons. I would love to hear!

Context:
I really like keeping all four models combined into one file (as it is the case right now). I believe this is much better than copy-pasting. In addition to the software engineering advantages that are mentioned above about maintainability (which are not small things with a library that’s growing so fast like yours), I want to add two points from personal experience,

Easier to understand and compare models: Here’s a recent use case; I have a working model that uses BART and I wanted to try Pegasus. Running the same training code results in NaNs with fp16. With a quick diff of the two configs, I can find out the model design changes that might be the reason for the fp16 issues. Now to compare with T5 (implemented in a separate file), it is many hours of code-reading to find the differences (compared with 1 minute to diff the configs).

Easier to try new ideas: I was recently experimenting with adding Longformer self-attention to various models on the HF/transformers repo. Once I have it implemented for BART, I get it for free for Pegasus and the other models sharing the same file. On the other hand, you can check the Longformer Github repo for the number of issues requesting adding longformer self-attention to BERT, ALBERT, ELECTRA, … etc. Each one requires its own implementation, and each one is a ton more work. Many similar use cases exist, experimenting with a new activation function, a new position embedding, a new type of layer normalization … etc.

I also want to comment on some of the arguments for the copy-paste mentioned above:

delete 6 configuration parameters and delete 6 corresponding if statements

I think the benefits mentioned above outweigh the gains from deleting these lines. Also, remember that removing these lines comes at the expense of a massive amount of added code.

if you want to know how a model works you can read it in one file.

It is still one file of code, but with an accompanying config file. If the additional config file is undesirable, maybe it can be clarified in code via comments so that the user doesn’t even need the config. Something like:
if config.position_embedding == `learned`: # BART, mBART
   self.embedding = LearnedPositionalEmbedding()
elif config.position_embedding == `sinusoidal`:: # Pegasus, Marina, T5
   self.embedding = SinusoidalPositionalEmbedding()

The annoying code design that makes reading model code difficult is when you introduce higher-level abstractions and move them to their own files. Having a separate file for self-attention, another file for the embedding layer, and a third for objective functions means you keep moving from one abstraction to another and jumping between files, which makes it challenging to grasp the model details. This is not the case here.

in the survey, 65% of respondents said they wanted more “Self-contained” code. This is what users want!

This code design satisfies the self-containment requirement. The few if-else statements with code for other models won’t be a deal-breaker. As mentioned above, I think the design that the users were voting against is something like fairseq and allennlp that have lots of layers of abstractions and lots of magic that happens behind the scene. I think the current design strikes a nice balance between fairseq/allennlp as one extreme, and absolute self-containment with tons of duplicate code on the other extreme.


Question:
I’ve tried to train a Reformer model with a toy task as described in the original paper 1: a sequence of 1024 tokens, such that the first half is the same as the second.
I tried replicating the configurations given by the authors (one LSH layer, 256 hidden units, 4 attention heads), and it still took me some tuning with the learning rate until I got around 99.4% accuracy on the dev set with 4 hashing rounds, still slightly below their 99.9%.
But worse is the very unstable loss curve. It falls, then bounces back and actually tends to increase. Here is the training loss plot:

I compared with a deterministic setting, with a single local (no LSH) attention layer and context length to the size of the inputs (1024). It was much more stable, accuracy went to 1.0 and stayed there.
So, even with a random component, I find it hard to understand why the Reformer loss should be so unstable. Some experiments with enwik8 yielded even worse results.
Do you think there might be something wrong in the LSH implementation or this is just a super sensitive model?

Context:
Can you share a colab link for this? It would be easier to debug.
Some additional questions:

Do you use FP16?
How did you tuned the learning rate?
Do you use gradient clipping?
What is the batch size?


Generating NER data sample for RoBERTa model

Question:
I have a trained RoBERTa model with a Byte Level BPE Encoding algorithm, which I want to benchmark on a custom NER dataset.
Each sample is as followed:
Text: John is playing football
Labels: B-PER O O O
The text could be run through the tokenizer to generate subword tokens. However, the total count of the subwords tokens may be different from the text tokens, and I don’t know how to align the labels accordingly

Context: cc @stefan-it


What I know and don’t know about sequence to sequence batching

Question:
The old code in examples/seq2seq/finetune.py:
decoder_input_ids = labels[:, 1:]

deletes a token if there is no bos at the start of the tensor.
The new code
decoder_input_ids = shift_tokens_right(labels)

Doesn’t delete tokens.
helps metrics of fine-tuned mbart and pegasus (don’t use BOS)
does not change metrics of finetuned models that use bos.

This makes sense. Deleting the first word of every tgt example makes finetuning worse.
What I don’t understand:
bart’s shift_tokens_right wraps eos token around to the 0th position of decoder_input_ids.
This means that models are finetuned with the equivalent of decoder_start_token_id=eos_token_id.
However, when it comes time to evaluate mbart/pegasus/marian, having decoder_start_token_id=pad_token_id produces better metrics than decoder_start_token_id=eos_token_id. For the bart variants, decoder_start_token_id=eos_token_id works best.
Additionally, switch to the t5 shift_tokens_right functionality, which puts decoder_start_token_id at position 0 for finetuning doesn’t improve metrics at all.
Does this make sense? What am I missing?

Context:
@sshleifer
Isn’t it necessary to add a start token to the decoder sequence (be it anything eos, bos, pad or whatever else)  when training generative models, because the labels need to be shifted  to calculate loss, without shifting the labels the models will only learn to produce the last token it has seen.
Is this right ?
And I think MBart and PEGASUS don’t have bos  and may have been pre-trained by using pad token (like T5) as the first token which could explain this behavior. I couldn’t find this info in PEGASUS repo. What was the decoder_start_token_id when pre-training PEGASUS and MBart ?


Question:
I am trying to generate summaries using t5-small with a maximum target length of 30. My original inputs are german PDF invoices. I run OCR and concatenate the words to create input text. My outputs should be the invoice numbers. However even after 3 days on a V100 I get exactly 200 token long summaries (since epoch 1 or 2 out of 300) and garbage results. Summaries look like someone shuffled the original words a little but they do contain the invoice number somewhere near to the start.
What might cause it to stick to 200 generated tokens?

Context:
Hi, are you fine-tuning the model or just generating summaries ?
If you’re just looking to generate short summaries then bart-large-xsum https://huggingface.co/facebook/bart-large-xsum 39 should give you better results.


Merging bert-base-uncased models after trainer but before predict

Question:
Hi everyone,
I created and trained two bert-base-uncased models, using the run_ner.py 4 script from the huggingface transformers examples, to predict one the PoS-tags and one the DEPREL-tags (both are attributes of the CoNLL-U Format).
I trained the two models separately on the same dataset to which I made some changes: in the first model I made it predict the labels corresponding to the PoS-tags and in the second model I made it predict the labels corresponding to the DEPREL-tags.
Once the training is finished, I loaded the weights (AutoModelForTokenClassification.from_pretrained), the configurations (AutoConfig.from_pretrained) and the tokenizers (AutoTokenizer.from_pretrained) of the two models using the functions made available by the library.
By doing this I get two different trainers, like the following:
trainer_pos_tag = Trainer(
        model=model_pos_tag,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        compute_metrics=compute_metrics,
    )

trainer_deprel_tag = Trainer(
        model=model_deprel_tag,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        compute_metrics=compute_metrics,
    )

with their respective configurations and tokenizers.
What I would like to achieve is to use these two models to predict both the PoS-tag and the DEPREL-tag at the same time (so I think it is necessary to be able to have a single model that predicts both labels, right?).
Now my problem is that I necessarily need to merge these two models (merge their weights (does that make sense?) or something like that) before doing the prediction (trainer.predict(test_dataset) ) operation.
How can I do?
Do you have any suggestions?
P.s. The important thing is that the model resulting from the union/merge of these two models always has a  Trainer  type.
Thanks in advance!
The question is also on StackOverflow with the same title (I can’t put more than 2 links in the post because I’m a new user).

Context: I think you would get better results averaging the predictions rather than merging the weights. You can do this by calling trainer_xxx.predict(...) and then average the results.


T5 weights auto conversion from TF to PyTorch

Question:
Hi Great community, I am wondering: is there a script that help automatically convert TF weights (*.h5) to PyTorch weights (.bin) and from PyTorch to TF for T5?
My goal is to use T5ForConditionalGeneration and TFT5ForConditionalGeneration interchangeably, i.e., I fine-tune a T5 model first in TF and then want to do inference with it in PyTorch.

Context:
Here you go my friend -

      github.com

    huggingface/transformers/blob/master/src/transformers/convert_t5_original_tf_checkpoint_to_pytorch.py 11
# coding=utf-8
# Copyright 2018 The T5 authors and HuggingFace Inc. team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Convert T5 checkpoint."""

import argparse

import torch

  This file has been truncated. show original


Issues with translating inputs containing repeated phrases

Question:
While porting fairseq wmt19 7 I discovered it doesn’t handle correctly inputs containing repeated phrases - it truncates big parts of the sentence.
So, obviously, the ported model with the same pre-trained weights has the same issues.
I wonder how transformers translation models handle such inputs.
Here is a script demonstrating the issue:
import torch

text = [
    """This is good. This is bad. This is good. This is bad. What do you think?""",
    """Here's a little song I wrote. You might want to sing it note for note. Don't worry, be happy. In every life we have some trouble. But when you worry you make it double. Don't worry, be happy. Don't worry, be happy now."""
]

src, tgt = "en", "ru"
fs_mname = f"transformer.wmt19.{src}-{tgt}"
checkpoint_file = 'model1.pt'
#checkpoint_file = 'model1.pt:model2.pt:model3.pt:model4.pt'
fs_model = torch.hub.load('pytorch/fairseq', fs_mname, checkpoint_file=checkpoint_file, tokenizer='moses', bpe='fastbpe')

for s in text:
    print(s)
    fs_input_ids = fs_model.encode(s)
    fs_outputs = fs_model.generate(fs_input_ids, beam=5, verbose=True)
    for i, output in enumerate(fs_outputs):
        fs_decoded = fs_model.decode(output['tokens'])
        print(f"{i}: {fs_decoded}")
    print("--------")

For the first, contrived, input we get:
This is good. This is bad. This is good. This is bad. What do you think?
0: Это хорошо. Это плохо. Это хорошо. Это плохо.
1: Это хорошо. Это плохо. Это хорошо. Это плохо. Что вы думаете?
2: Это хорошо, это плохо, это хорошо, это плохо.
3: Это хорошо. Это плохо. Это хорошо. Это плохо. Как вы думаете?
4: Это хорошо. Это плохо. Это хорошо.

So hypos 1 and 3 are correct, the rest have the last 1-2 sentences chopped off. Yet, it chose the truncated version as the best.
For the second input we get:
Original:
(1) Here's a little song I wrote.
(2) You might want to sing it note for note.
(3) Don't worry, be happy.
(4) In every life we have some trouble.
(5) But when you worry you make it double.
(6) Don't worry, be happy.
(7) Don't worry, be happy now.

Hypo 0:
(1)
(2)
(3) Не волнуйтесь, будьте счастливы.
(4) В каждой жизни у нас есть некоторые проблемы.
(5) Но когда вы волнуетесь, вы делаете это дважды.
(6) Не волнуйтесь, будьте счастливы.
(7) Не волнуйтесь, будьте счастливы сейчас.

Hypo 1:
(1)
(2)
(3) Не волнуйтесь, будьте счастливы.
(4) В каждой жизни у нас есть некоторые проблемы.
(5) Но когда вы волнуетесь, вы делаете это дважды.
(6) Не волнуйтесь, будьте счастливы.
(7) Не беспокойтесь, будьте счастливы сейчас.

Hypo 2:
(1) Вот маленькая песня, которую я написал.
(2) Вы, возможно, захотите спеть ее на заметку.
(3) Не волнуйтесь, будьте счастливы.
(4)
(5)
(6)
(7)

Hypo 3:
(1) Вот маленькая песня, которую я написал.
(2) Вы могли бы захотеть спеть ее на заметку.
(3) Не волнуйтесь, будьте счастливы.
(4)
(5)
(6)
(7)

Hypo 4:
(1)
(2)
(3) Не волнуйтесь, будьте счастливы.
(4) В каждой жизни у нас есть некоторые проблемы.
(5) Но когда вы волнуетесь, вы делаете это дважды.
(6) Не волнуйтесь, будьте счастливы.
(7)

I rewrapped and annotated it to show what’s missing, so you don’t need to understand Russian to see that there is a problem. the empty “slots” are the corresponding missing sentences in the translation’s output.
Here we have hypos 0, 1, 4 chopping off the first 2 sentences. and 2,3 chopping off the last 3 sentences.
If use their en-de model, the first input it has the same problem - truncating the last sentence. With the second input the first result is perfect, but the same truncation is observed in the following hypos.
p.s. it behaves the same with the model ensemble (fairseq uses 4 checkpoints in their wmt19 submission).

Context:
Hmm, found another odd ball:
“NOOOOOOOOOOO ANOTHER BLACK PERSON,” wrote Mrs Plasco-Flaxman to her husband in a text.
fairseq translates this as:
"НОООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООО
"НОООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООООО
(this is an entry from wmt19 test dataset)


NLI 2-sentence classification with GPT2, XLNet, etc.?

Question:
I’m doing research on NLI with 2-sentence classification. I have already successfully used BERT and its BertForSequenceClassification class to feed in two sentences in the form of an input string [CLS] sent1 [SEP] sent2 [SEP] and then perform classification.
I’d like to do the same with the other models available, such as GPT2, XLNet, and RoBERTa. However, I can’t seem to find any example code that takes two sentences for those models. Do they use the same type of input string as BERT? Can someone point me to the relevant webpages for more information?
Thank you for any help.

Context: Surely there must be someone who’s done NLI with one of these other non-BERT models?


Question:
I downloaded comet_ml, but when I try to use the Trainer() function, I get the following error:
Traceback (most recent call last):
File “”, line 1, in
File “/Users/aviaronkraft/anaconda3/lib/python3.8/site-packages/transformers/trainer.py”, line 266, in init
self.setup_comet()
File “/Users/aviaronkraft/anaconda3/lib/python3.8/site-packages/transformers/trainer.py”, line 504, in setup_comet
experiment = comet_ml.Experiment(**args)
File “/Users/aviaronkraft/anaconda3/lib/python3.8/site-packages/comet_ml/init.py”, line 202, in init
raise ValueError(
ValueError: Comet.ml requires an API key. Please provide as the first argument to Experiment(api_key) or as an environment variable named COMET_API_KEY
I found a Github thread on this error where someone suggested that the initialization function for the Experiment class needs to have api_key set to None as a default, but I checked and that’s already the case. I haven’t been able to find any other suggestions online. Does anyone know what to do here?

Context: This looks like a problem coming from comet. If you uninstall it it’ll go away, and you should probably ask for help on their side if you want to use it.


How to get the index of the masked token after passing the sentence to the model

Question:
Assuming that I am using one of the BERT instances for a language modeling task
from transformers import BertForMaskedLM, BertTokenizerFast
import torch
model = BertForMaskedLM.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')

now let’s say that I have the next sentence
text = "Jeremy Bentham was the founder of modern utilitarianism"
inputs  =  tokenizer(text,  return_tensors="pt", add_special_tokens = True, truncation=True,  max_length=64)

If I say to mask a random token from some part of the sentence, for this example let’s put an arbitrary example of the 7th token
inputs['input_ids'][0][7] = tokenizer.mask_token_id

and then pass this to the model as
outputs = model(inputs)
scores = outputs[0]

If I want the embedding (vector) of only the masked token, how should I access it?

Context: Your scores have a shape [batch_size, seq_len, vocab_size] so scores[0][7] (the indices where you masked) should have the predictions for the masked token.


What is the purpose of ‘use_cache’ in decoder?

Question:
I am using BartForConditionalGeneration for text summarization.
The ‘use_cache’ option is True by default when pre-training the Bart.
Purpose: The purpose of ‘use_cache’ option seems speed-up decoding.
(https://github.com/huggingface/transformers/blob/d822ab636b6a14ed50f7bca0797c1de42c19de61/src/transformers/modeling_bart.py#L120-L122 29)
Implementation: When ‘use_cache’ = True, the decoder use only the last time steps of input_ids & positional embedding.
(https://github.com/huggingface/transformers/blob/d822ab636b6a14ed50f7bca0797c1de42c19de61/src/transformers/modeling_bart.py#L551-L553 17)
Can anyone explain how above purpose & implementation are related?
Thank you

Context:
Hi @lifelongeek!
The cache is only used for generation, not for training.
Say you have M input tokens and want to generate N out put tokens.
Without cache, the model computes the M hidden states for the input, then generates a first output token. Then, it computes the hidden state for the first generated token, and generates a second one. Then, it computes the hidden state for the first two generated tokens to generate the third one, and so on an so forth.
However, since the output side is auto-regressive, an output token hidden state remains the same once computed for every further generation step, so recomputing it every time we want to generate a new token seems wasteful.
With the cache, the model saves the hidden state once it has been computed, and only computes the one for the most recently generated output token at each time step, re-using the saved ones for hidden tokens. This reduces the generation complexity from O(n^3) to O(n^2) for a transformer model.
Hope that helps, let me know if you have any further questions!


What is the input vector size for a BERT and Transformer-XL?

Question:
Does anyone know what size vectors the BERT and Transformer-XL models take and output?
For example, I know that bert-large is 24-layer, 1024-hidden, 16-heads per block, 340M parameters. (bert-base is 12 heads per block) does that mean it takes a vector size of [24,1024,16]? or am I miss understanding?
Any help is much appreciated

Context: No the inputs are usually a tensor of ints of size batch_size x sequence_length (with integers between 0 and the vocabulary size of the model -1). You can find all expected shapes and output shapes in the documentation, for instance here 99 for BERT.


Question:
from transformers import BertPreTrainedModel, RobertaModel
import torch

class RobertaForMD(BertPreTrainedModel): # Metaphor Detection, modified from BertForTokenClassification
    def __init__(self, config):
        super().__init__(config)
        self.num_labels = config.num_labels

        self.bert = RobertaModel(config)
        self.dropout = torch.nn.Dropout(config.hidden_dropout_prob)
        self.classifier = torch.nn.Linear(config.hidden_size, self.config.num_labels)
        # self.loss = torch.nn.BCEWithLogitsLoss(pos_weight=torch.tensor([3], dtype=torch.float32))
        self.loss = torch.nn.BCEWithLogitsLoss()

        self.init_weights()

    def forward(
        self,
        input_ids=None,
        attention_mask=None,
        token_type_ids=None,
        position_ids=None,
        head_mask=None,
        inputs_embeds=None,
        labels=None,
        word_posi=None
    ):
        outputs = self.bert(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
        )

        last_hidden_state = outputs[0]
        last_hidden_state = self.dropout(last_hidden_state)
        batch_size = input_ids.shape[0]

        word_state = torch.empty((0, last_hidden_state.shape[2]), dtype=torch.float32).cuda()
        for i in range(batch_size):
            word_state = torch.cat((word_state, last_hidden_state[i][word_posi[i]].unsqueeze(0)))

        logits = self.classifier(word_state)

        outputs = (logits,) + outputs[2:]  # add hidden states and attention if they are here

        if labels is not None:
            loss = self.loss(logits.view(-1), labels.to(torch.float32))
            outputs = (loss,) + outputs

        return outputs  # (loss), logits, (hidden_states), (attentions)

I am calling this using
model = RobertaForMD.from_pretrained(model_name, num_labels=1)
Error
File “main.py”, line 276, in main
model = RobertaForMD.from_pretrained(model_name, num_labels=1)
File “/nas/home/tuhinc/miniconda3/lib/python3.7/site-packages/transformers/modeling_utils.py”, line 438, in from_pretrained
**kwargs,
File “/nas/home/tuhinc/miniconda3/lib/python3.7/site-packages/transformers/configuration_utils.py”, line 199, in from_pretrained
config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
File “/nas/home/tuhinc/miniconda3/lib/python3.7/site-packages/transformers/configuration_utils.py”, line 269, in get_config_dict
raise EnvironmentError(msg)
OSError: Can’t load ‘roberta-large’. Make sure that:

‘roberta-large’ is a correct model identifier listed on ‘https://huggingface.co/models’

or ‘roberta-large’ is the correct path to a directory containing a ‘config.json’ file

Context:
tuhinnlp:

config

Not sure how you mean , any code or link which does that ?


Issues with Trainer class on custom dataset

Question:
I am following along with the Sequence Classification on IMDB Dataset example here:
https://huggingface.co/transformers/master/custom_datasets.html?utm_campaign=Hugging%2BFace&utm_medium=web&utm_source=Hugging_Face_1 16
However, I am using a custom dataset. Rather than iterate through using read_imdb_split I load my data from a csv, get the value of the sequences and labels, then convert them to a list to pass in to the tokenizer() method. From there I convert them to datasets using the subclassed dataset object as shown in the documentation.
Next, I create TrainingArguments and  Trainer as shown and my issue then arises when calling trainer.train(). I receive the following error stack trace:

AttributeError                            Traceback (most recent call last)
 in
----> 1 trainer.train()
~\anaconda3\lib\site-packages\transformers\trainer.py in train(self, model_path)
512                 self._past = None
513
–> 514             for step, inputs in enumerate(epoch_iterator):
515
516                 # Skip past any already trained steps if resuming training
~\anaconda3\lib\site-packages\tqdm\notebook.py in iter(self, *args, **kwargs)
215     def iter(self, *args, **kwargs):
216         try:
–> 217             for obj in super(tqdm_notebook, self).iter(*args, **kwargs):
218                 # return super(tqdm…) will not catch exception
219                 yield obj
~\anaconda3\lib\site-packages\tqdm\std.py in iter(self)
1127
1128         try:
-> 1129             for obj in iterable:
1130                 yield obj
1131                 # Update and possibly print the progressbar.
~\anaconda3\lib\site-packages\torch\utils\data\dataloader.py in next(self)
343
344     def next(self):
–> 345         data = self._next_data()
346         self._num_yielded += 1
347         if self._dataset_kind == _DatasetKind.Iterable and \
~\anaconda3\lib\site-packages\torch\utils\data\dataloader.py in _next_data(self)
383     def _next_data(self):
384         index = self._next_index()  # may raise StopIteration
–> 385         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
386         if self._pin_memory:
387             data = _utils.pin_memory.pin_memory(data)
~\anaconda3\lib\site-packages\torch\utils\data_utils\fetch.py in fetch(self, possibly_batched_index)
42     def fetch(self, possibly_batched_index):
43         if self.auto_collation:
—> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
45         else:
46             data = self.dataset[possibly_batched_index]
~\anaconda3\lib\site-packages\torch\utils\data_utils\fetch.py in (.0)
42     def fetch(self, possibly_batched_index):
43         if self.auto_collation:
—> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
45         else:
46             data = self.dataset[possibly_batched_index]
 in getitem(self, idx)
5
6     def getitem(self, idx):
----> 7         item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
8         item[‘labels’] = torch.tensor(self.labels[idx])
9         return item
AttributeError: ‘list’ object has no attribute ‘items’

I figure this is a result of me loading my data differently and converting them to lists, but I pass the datasets, not lists, to the trainer so I am unclear what is causing the error.

Context: The problem is when indexing into your custom dataset, not Trainer. If you want help with that, you’ll have to share your code on how it’s built. To debug your problem, try to see if you can do train_dataset[0] (or any other index).


Question:
@stas was about to write this to you but decided to send it here!
I have done everything with pip install awscli but that may require more privs.  (you probably still want awscli to run ls)
Each model should be its own subdir under stas/ with the full complement of files:
something like
aws s3 ls s3://models.huggingface.co/bert/Helsinki-NLP/opus-mt-ru-en/
2020-08-21 10:42:49       1148 README.md
2020-08-18 20:43:55       1133 config.json
2020-04-29 09:40:09  306991893 pytorch_model.bin
2020-05-27 11:23:35  563074700 rust_model.ot
2020-04-29 09:40:21    1080169 source.spm
2020-04-29 09:40:21     802781 target.spm
2020-08-18 20:43:55         42 tokenizer_config.json
2020-08-18 20:43:55    2601758 vocab.json

If you are missing files, you can sometimes get them with
working_model.save_pretrained('en_de') # will mkdir en_de
tokenizer.save_pretrained('en_de')

That’s all that comes to mind at the moment, but lmk if you need anything!

Context:
another thing that bothered me earlier that transformers-cli handles
you have to duplicate all tokenizer files across every model subdir. S3 cant do symlinks.


Question:
I am trying to upload a model trained on question answering; it is approx 440mb but the transformers-cli terminal is stuck on “Uploading… This might take a while if files are large” since about 20 minutes.
Is this slow uploading speed normal (currently my network has an uploading speed of 20Mb/s) or is there something strange going on?
I appreciate any help,
thanks

Context: Not sure if there is a problem on the S3, @julien-c or @mfuntowicz ?


How much memory is needed for mbart-large-cc25?

Question: I tried to finetune without freezing but a 16GB V100 ran out of memory with batch size 1, max source length 512.

Context:
I am trying to run the mbart En-Ro example and I get CUDA OOM even with max_len 64, n_train 5000 and bs 1 (using Google colab with 15GB gpu P8).
Anyone managed to run it with under 16 GB?


Tricks to control/surpress logging output

Question:
I think since the logger PR, I have started getting much more logging output.
What is a reasonable level for a training script is ERROR too aggressive? @lysandre ?
from transformers.utils.logging import get_logger, ERROR
logger = get_logger('seq2seq/finetune')
logger.setLevel(ERROR)

Example of logging outputs:
Model name 'google/pegasus-xsum' not found in model shortcut name list (google/reformer-crime-and-punishment). Assuming 'google/pegasus-xsum' is a path, a model identifier, or url to a directory containing tokenizer files.
loading file https://s3.amazonaws.com/models.huggingface.co/bert/google/pegasus-xsum/spiece.model from cache at /home/shleifer/.cache/torch/transformers/3848e191fec9935f1369d13fe14c5d785a9ff32b812559047361f135a53aaf13.efce77b8dcd2c57b109b0d10170fcdcd53f23c21286974d4f66706536758ab6e
loading file https://s3.amazonaws.com/models.huggingface.co/bert/google/pegasus-xsum/added_tokens.json from cache at None
loading file https://s3.amazonaws.com/models.huggingface.co/bert/google/pegasus-xsum/special_tokens_map.json from cache at /home/shleifer/.cache/torch/transformers/bc045ff6a50eaed77dd55c18cb2baf14d21b5288b0fb06615ade9cb08983bd38.d142dfa55f201f5033fe9ee40eb8fe1ca965dcb0f38b175386020492986d507f
loading file https://s3.amazonaws.com/models.huggingface.co/bert/google/pegasus-xsum/tokenizer_config.json from cache at /home/shleifer/.cache/torch/transformers/1cc20bfd3eea66838b0cfe76e83bfd96b26ef1ab1ba0e8119f4850fa781a5710.c7d99a644bcf983aa23a0081b95cf8be1bc67c66b6afc27a266363ca1d8e463e
loading file https://s3.amazonaws.com/models.huggingface.co/bert/google/pegasus-xsum/tokenizer.json from cache at None
loading weights file https://cdn.huggingface.co/google/pegasus-large/pytorch_model.bin from cache at /home/shleifer/.cache/torch/transformers/e95d8a42105877a796b5f878b2ccebfb3ed36c1ae87be48d0a2656afff150431.b6d2710a7db33f51964e675a76e102fa14b361b2471d45ef76b544bcb730ef46
All model checkpoint weights were used when initializing PegasusForConditionalGeneration.

Context: related: @stas solution to  https://github.com/huggingface/transformers/issues/3050 52


Saving standard BertModel english and BertModel multilingual have drastically different sizes?

Question:
When I do
bert_config = BertConfig.from_pretrained(‘bert-base-multilingual-cased’)
model = BertModel(bert_config)
torch.save(model.state_dict(), “temp.p”)
print(‘Size (MB):’, os.path.getsize(“temp.p”)/1e6)
and I save this model my model size is around 711 MB. But if I do the same for ‘bert-base-uncased’ or ‘bert-base-cased’ which is actually expected due to 110M parameters. Is this expected/ anyone know what I might be potentially doing wrong.

Context:
Hey @mkumar10,
The bert-base-multilingual-cased’ model  has large vocab_size  (119547), so the embedding matrix is bigger than standard English BERT model. Which is why it takes more memory.


Containerizing transformers with Docker and FastAPI

Question:
Hi everyone!
I’ve been working on putting GPU accelerated transformer inference into production using Docker. I thought it would be helpful for me to share how I did it (link to gist 113):
<script src="https://gist.github.com/lmwilkin/359ef8ada2eb1766d049719e9fc7053a.js"></script>

You’ll need to have installed nvidia-docker 27
I used FastAPI to set up a basic rest interface. The Dockerfile would be a good place to add an environmental variable for something like model name too if you want to set that dynamically.
My plan is to try write a container that can swap out models at request which I’ll share if I can get working.
Hope this helps!

Context: Hey @lmwilkin, great! Thank you for sharing this.


Gradual Unfreezing support for Fine tuning models

Question: Does huggingface library has support for gradual unfreezing? As thats one of the key strategy to fine tune model effectively for down stream task.

Context:
You will have to do that manually. The Trainer won’t do it automatically for you. To freeze the pytorch model, you can do something like this:
model = AutoModel.from_pretrained(model_id)
for param in model.parameters():
    param.requires_grad = False

In this way, you’d have more granular control over which layers you want to freeze. For your use case, you’d want to train just the linear layer initially. You can do this by freezing just the encoder
for param in model.encoder.parameters():
    param.requires_grad = False

You’d have to check if encoder is an attribute of all models to be fully sure. At later stages, you can turn the bool to True, to train the encoder.
I hope this gives you an idea.


Question:
Hello. I’m trying to train a RoBERTa model on a 97GB corpus of text.
Should I tokenize the text on-the-fly, or should I precompute them to reduce CPU load? (Let’s say I have 200-300GB of RAM)
I did tried to tokenize the text on-the-fly, but due to some company shared resource restriction (shared CPU and RAM), the training speed is pretty slow. After some iterations, it froze for a litte bit and then continue. This reduce the normal training speed for 4-5 times.
Were I have to precompute the data, is there any way to efficiently work with them? I made an experiment where I tokenized the text and dump to a bunch of pickle files (total size 9GB), but when I tried to load them to memory to feed through the trainer, it cost me about 80-90GB of RAM, which is unresonably high.
Thank you.

Context: What I did to in this case is to split the data in multiple chunks of size batch_size. And then use DataLoader getitem(id=chunk_file_id) to process a chunk file at a time and feed it to the model. I can write some dummy code for clarification.


Questions about Pegasus for Summarization

Question:
Three questions:  (1) does anyone have good examples of the google/pegasus models being used for summarization using the Automodal and Autotokenizer classes?
(2) Using the google/pegasus models, how do you set the minimum and/or maximum length for the generated summary?
(3) Is there a token maximum input for Pegasus (as there is with T-5 (512) and BART (1024)?
Thanks!

Context: Hey @Buckeyes2019, there’s new thread for discussing all Pegasus related questions here 95, mind asking your question on that thread


Question:
Notebooks are now automatically created from the tutorials in the documentation of transformers. you can find them all here 132 or click on the brand new “Open in colab” button on the doc pages that have one.
Plan is to add more tutorials and the corresponding notebooks very soon. I’ll focus on integration with nlp datasets and the Trainer in the next few weeks, but if you have ideas of tutorials you’d like to see, please let me know here.

Context: Awesome!  Thanks @sgugger


Question:
I am working with GPT-2 and I was looking at the LM head and how it performs the forward pass when labels are provided: https://huggingface.co/transformers/_modules/transformers/modeling_gpt2.html#GPT2LMHeadModel 57
It looks like the logits are shifted right (last value is ignored) and the labels are shifted left (first value is ignored).
Why are the logits and labels shifted in different directions?

Context: The logits are not shifted, just the last value is ignored. The labels are shifted inside the model as described in the docs ("Note that the labels are shifted inside the model, i.e. you can set labels = input_ids") because we want to avoid any processing on them and just set them equal to the inputs. That way the batch creation can be as easy as possible. The downside is that you don’t compute the loss on the last character of the sentence, but we found it’s acceptable since the sequence length is 512.


Why does the median cross entropy loss change when I change the random seed?

Question:
Hello,
I’ve noticed that, when I use the pre-trained GPT2DoubleHeadsModel to process multiple choice questions, the median of the cross entropy loss generated for the same set of multiple choice questions change when I change the type of my random seed (NOTE: I changed my random seed before loading the pre-trained BTP GPT-2 tokenizer and loading the pre-trained  GPT2DoubleHeadsModel … I also did my_gpt2_model.eval() before evaluating the loss to prevent dropout).
Why does this occur? I thought the parameters of both the pre-trained model and the tokenizer are fixed, so to me, the cross entropy loss should be the same regardless of the type of random seed?
For more information, below are my code:
# for our main experiment, we use G1G2, G4G5, G7G8, G10G12 files
def fill_MC_loss_tensor( ...):

    for m in range(num_mc_questions):

        # make an empty list to store the mc_loss
        mc_loss_list = []

        # Turn on the evaluation mode
        best_model_gpt2DoubleHeadsModel.eval()

        # for each layer j = 1,...,12, extract the hidden states at the layer j
        input_hidden_state = best_model_gpt2DoubleHeadsModel(input_ids,
                       token_type_ids = token_type_ids,
                       attention_mask = attention_mask)[3][0][:,:,:].detach()

        for j in range(nlayer):

            # Turn on the evaluation mode
            layer_hidden_state =
best_model_gpt2DoubleHeadsModel.transformer.h[j](input_hidden_state)

            # feed the hidden states from each layer directly into the multiple-choice head
            mc_logits =
best_model_gpt2DoubleHeadsModel.multiple_choice_head(layer_hidden_state[0]).squeeze(-1).detach()

            del layer_hidden_state
            gc.collect()

            # define the loss function
            loss_fct = CrossEntropyLoss()
            # calculate the mc_loss
            mc_loss = loss_fct(mc_logits.view(-1, mc_logits.size(-1)),
                                     mc_labels.view(-1))
            # store the mc_loss in a list
            mc_loss_list = mc_loss_list + [mc_loss.tolist()]

            del mc_logits
            gc.collect()

        mc_loss_tensor[m,:] = torch.tensor(mc_loss_list)
        print('m={}'.format(m))

    return mc_loss_tensor

# main function for analysis
def main_function(...):

    # set initial seed
    seed(125)

    num_iter = 200

    # define mc_loss_tensor_num_iter
    mc_loss_tensor_num_iter = torch.zeros(num_iter, int(num_mc_questions),
                                               nlayer)
    mc_loss_tensor_num_iter[mc_loss_tensor_num_iter == 0] = nan

    for i in range(num_iter):

        # change seed at each iteration
        s = randint(1,999999)
        seed(s)

        # import the pre-trained HuggingFace GPT2Tokenizer
        gpt2_tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

        # make a dictionary of special tokens
        special_tokens_dict = {'pad_token': '<pad>'}

        # add the special tokens to the tokenizer
        gpt2_tokenizer.add_special_tokens(special_tokens_dict)
        assert gpt2_tokenizer.pad_token == '<pad>'

        # get the encoding for the special tokens
        pub2_pad_token_id = gpt2_tokenizer.convert_tokens_to_ids('<pad>')
        pub2_eos_token_id = gpt2_tokenizer.convert_tokens_to_ids(gpt2_tokenizer.eos_token)

        # sanity check
        len(gpt2_tokenizer) # note: original size of the tokenizer is 50257 + <pad> = 50258

        # get the pre-trained HuggingFace GPT2DoubleHeadsModel and
        # resize the token embeddings after adding the special token
        best_model_gpt2DoubleHeadsModel = GPT2DoubleHeadsModel.from_pretrained('gpt2',
                                               output_hidden_states = True)
        best_model_gpt2DoubleHeadsModel.resize_token_embeddings(len(gpt2_tokenizer))

        #######
        # make an empty tensor to store mc loss
        mc_loss_tensor = torch.zeros(num_mc_questions, nlayer).float()
        mc_loss_tensor[mc_loss_tensor == 0] = nan

        mc_loss_tensor = fill_MC_loss_tensor(...)

        if torch.isnan(mc_loss_tensor).any().tolist():
            sys.exit('nan found in mc_loss_tensor')

        mc_loss_tensor_num_iter[i,:,:] = mc_loss_tensor

        print('i={}'.format(i))

    return mc_loss_tensor_num_iter

# for each of the 200 iteration, the computed median
# (median over all questions)
# cross entropy loss are different,
# for the same layer.
>>> main_function(...)

Thank you,

Context:
How do you set your seed? There’s a few things that you should set to a fixed value to be sure. I always use the following, which has worked consistently.
    def set_seed(seed: Optional[int]):
        """ Set all seeds to make results reproducible (deterministic mode).
             When seed is None, disables deterministic mode. """
        if seed is not None:
            torch.manual_seed(seed)
            torch.cuda.manual_seed_all(seed)
            torch.backends.cudnn.deterministic = True
            torch.backends.cudnn.benchmark = False
            np.random.seed(seed)
            random.seed(seed)
            os.environ['PYTHONHASHSEED'] = str(seed)

But even then, I don’th think variance is to be expected in inference mode. Not sure what is happening here.


Question:
My company has a 6-gpu server. However, I only want to train the model on a specific number of gpus (maybe 1 or 2).
How can I achieve that?

Context: What are you using for training ? Trainer or the pytorch-lightning ?


How to evaluate all model ckpts when using `run_language_modeling` with trainer?

Question:
github.com

    huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py#L272 5

    # For convenience, we also re-save the tokenizer to the same directory,
    # so that you can share your model easily on huggingface.co/models =)
    if trainer.is_world_master():
        tokenizer.save_pretrained(training_args.output_dir)

# Evaluation
results = {}
if training_args.do_eval:
    logger.info("*** Evaluate ***")

    eval_output = trainer.evaluate()

    perplexity = math.exp(eval_output["eval_loss"])
    result = {"perplexity": perplexity}

    output_eval_file = os.path.join(training_args.output_dir, "eval_results_lm.txt")
    if trainer.is_world_master():
        with open(output_eval_file, "w") as writer:
            logger.info("***** Eval results *****")
            for key in sorted(result.keys()):
                logger.info("  %s = %s", key, str(result[key]))

Here it seems the code only evaluates the last ckpt, while the common practice is to evaluate all ckpts and select the best one. How could we achieve that with trainer? Thanks!

Context: This code evaluates the model you pass to it. I think you mean that the train method does not save the best checkpoint at the root at the end of training, only the last checkpoint.


@sgugger Progress Update Aug 4 -> Aug 19

Question:
Following @sshleifer example, here is what I worked on the past few weeks and plan to work in the near future.
).
Model outputs:
Finished cleaning up all model outputs and made sure PyTorch and TensorFlow have the same API.
Documentation:

Work on cleaning up and updating docstrings and documentation of the main classes: config, tokenizer, models and pipelines.
Automatic conversion of the tutorials into notebooks and added the “open in colab” button.

Trainer:

A bit of clean-up to make sure Trainer and TFTrainer have the same API.
Exposed the customization points when the user wants to subclass and override.
Initial work to add hyperparameter search (see #6576 1).
Initial work to have an easy bridge between  nlp and Trainer (see #6449 1).

Repository consistency:
People love the fact each model file is self-contained and the code is not refactored since they can then quickly experiment, but it can be hard to maintain! Added a script that checks all models are tested (by the common tests) and documented.
Funnel Transformer:
Paper 5 - Initial work to understand the implementation and port it to  Transformers. PyTorch version is almost done.
Plans:

Continue the work on Trainer with hyperparameter search and  nlp interface.
Work on refactoring all examples to use Trainer and  nlp
Finish porting Funnel Transformer.

Context: Very excited about Optuna integration and Funnel Transformer! Also absolutely love your code reviews and suggestions !


What is rust_model.ot and how can I use it?

Question:
Lots of models have rust weights available for them in the file rust_model.ot and there’s even a rust tag on the hub 8. Is there any rust project coming up from HF ? Also is there any we way I can use these weights to do inference from rust ?
cc. @julien-c

Context: I see on Twitter 29 that you figured it out, but those weights are for https://github.com/guillaume-be/rust-bert 131 which is a great project – I think the author is planning to communicate about it in a couple of weeks


@sshleifer Progress Update Aug 4 -> Aug 19

Question:
In the spirit of openness (and inspired by sylvain’s google doc) I’m posting my standup/progress update openly instead of on google docs.
Big thank yous to @sgugger for 35 code reviews, @julien-c  for moonlanding help support with marian, and @valhalla and @stas  for help in so many areas over the past few weeks.
Successes/Done:

#6340 4 12 new pegasus checkpoints - SOTA summarization on many new datasets.

#6342 1 357 new MARIAN checkpoints, 48 upgrades = our 1st experiment with model versioning.
conversion code uses very rough heuristics to check whether we have an identical model. (across naming schemes), and either converts, upgrades, or ignores based on BLEU score posted in README.MD. This also added very involved, automated model-carding.

#6526 2 Partially undo adjust_logits_during_generation hack: metrics improvements across the board. We now are pretty close to metric parity.
Successfully distilled mbart english to romanian. It works on MT! see from_pretrained(sshleifer/distilbart-enro-12-6) or 12-4.
Lots of DM onboarding/QA with stas. He is porting fairseq’s SOTA translator now, after many smaller helpful bug fixes.
Lots of maintenance while others were on vacation.
Secret support mission going well.

Struggles:

I am close (4 common tests failing, integration test passing) on tfbart, could use some help. I think I need to do the huge signature like TFT5ForConditionalGeneration. After this, tfpegasus, tfmbart etc. will be easier.
Our Seq2Seq dataloader is slower than fairseq’s because it doesn’t use dynamic batch size.
external contributors cannot get TPU+seq2seq working, I may take a pass.
After huge amount of effort, many 24h experiments, our Seq2Seq finetune metrics are still slightly below fairseq, (0.2 ROUGE xsum, 1 BLEU wmt_en_ro). I suspect this is the dataloader.
Slow GPU CI has been broken for 10 days.
Haven’t carved out time for research in a while.

Future:

get marian, pegasus tweeted and “released”. Support as needed.
tfbart
CARVE OUT RESEARCH TIME: freezing idea.
seq2seq dataloader - dynamic batch size, cleanup to always use prepare_seq2seq_batch

seq2seq finetune onTPU (through Trainer !)

Context:
Already sounds like great progress! I’m very interested in the seq2seq stuff these days, and I miss customizability in onmt a bit  - I did some efforts, but everything is very interconnected (data and models)) so customization is not very straightforward - so having great seq2seq in transformers would be helpful for me.
About SOTA on summarization; are you planning to jot down your findings in a paper or blogpost? Might be an interesting read!


Question:
Hi all, newbie here! So I have understood that transformers stand out a lot for seq2seq tasks since they are much faster to train and are more powerful in their comprehension abilities.
However, I had a particular use-case where I want to train a model from scratch. Basically, it involves a dataset of ciphers and the model will have to decode the plaintext from the encrypted value.
Now The cipher is pretty advanced, so the relationship would obviously not be something very simple. The length of the output sequence will be variable. So can anyone point me to a model that preferably uses transformers and that is pretty good for finding out complex relations between the sequences? It does not have to be compulsorily from the HF transformers library. Any model that you all think is particularly good for this purpose will be considered. Also, please also give an explanation of why you recommend that model so that I can research further about it.
Cheers!
Neel Gupta

Context: how long are the sequences? 10+ words, 100+ words, 1000+ words?


How to load a google’s bert ckpt using tf2

Question:
Is there any way to load a google released ckpt in https://github.com/google-research/bert 5 (because some settings are not provided in hf transformers) using tf 2+ ?
I’ve found docs on loading those models and converting them to pytorch models in here:
https://huggingface.co/transformers/converting_tensorflow_models.html 10
However, it only works with pytorch.

Context:
Have you looked at using the keras-bert library (instead of using huggingface transformers library)?

      GitHub

CyberZHG/keras-bert 12

Implementation of BERT that could load official pre-trained models for feature extraction and prediction - CyberZHG/keras-bert

keras-bert works with TF2.   I used it to load a bert model from the tensorflow hub into colab.   I copied it onto my google drive first, and loaded from that copy, but I believe it is also possible to load directly using
!wget -q https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip 2
!unzip -o uncased_L-12_H-768_A-12.zip
So far as I can tell, models loaded with keras-bert are not compatible with transformers.  I tried to get my fine-tuned model into transformers format (because I wanted to use Jesse Vig’s visualisations), but I never got it to work.
I believe there is something about the way the transformers code is written that means you can’t use non-transformers models in place of transformers models, or at least not without some serious poking of the python.   However, I could be mistaken.


Question: I may be wrong, but this issue seems to have begun following the release of Transformers v3.0. I apologize if I am being annoying, considering you are doing an immense favor for us in making the demo available.

Context:
Hello @zanderbush! Why do you say it’s broken? Do you think the text generation is off?
The completions trigger correctly:
image2042×1096 93.3 KB


How to do selective masking in Language modeling

Question:
Hi Huggingfacers
I have a number of questions regarding finetuning a language model:

How to mask a selective portion of a given input sentence instead of masking randomly.
For example, if I am using ALBERT as a model, and I am aiming to do a different kind of loss function than the standard MLM loss  for the masked tokens, how to access the model output of the masked tokens

Context: I think the answer is similar another post: Selective masking in Language modeling 19


Question: Can anyone provide a link to a visual equation walkthrough for the MLM loss, or better how would it be implemented in torch.

Context: This thread 131 might help you


Language pair with multiple models on the model hub?

Question:
I’m interested in comparing different models that can translate between the same source and target language. Even better would be multiple models for multiple source/target pairs. When I search for models on huggingface.co/models, I see many models like this one: Helsinki-NLP/opus-mt-en-fr.
However, I don’t really see any other models available for translation. Are there any other translation models available, for example, for English to French, English to Chinese, or English to German translation? Thanks.

Context: T5 can do english to french and english to german