Question:
As a follow-up from my previous question 5, I am trying to fine-tune a model, but I am getting an error: IndexError: tuple index out of range.
I am trying to classify individual sentences with a binary classification.  I am using transformers version 4.2.1 and datasets version 1.2.1
The dataset(s) are .csv files with two columns: “sentence” and “label”.  The following is the code that led to the error - if anyone can help identify my error, please let me know
import numpy as np
from transformers import TrainingArguments, Trainer
from transformers import BertTokenizer, BertForSequenceClassification
from datasets import load_dataset, load_metric

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)

dataset = load_dataset('csv', data_files={'train': "train_data.csv",
                                          'test':  "test_data.csv"})

metric = load_metric('f1', 'accuracy')

encoded_dataset = dataset.map(lambda x: tokenizer(x['sentence'], padding=True, truncation=True), batched=True,load_from_cache_file=False)

batch_size = 16
args = TrainingArguments(
    "test_20210201_1200",
    evaluation_strategy = "epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    num_train_epochs=5,
    weight_decay=0.01,
    seed=18,
    label_names='label',
    load_best_model_at_end=True,
    metric_for_best_model='f1',
)

def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    predictions = np.argmax(predictions, axis=1)
    return metric.compute(predictions=predictions, references=labels)

trainer = Trainer(
    model,
    args,
    train_dataset=encoded_dataset['train'],
    eval_dataset=encoded_dataset['test'],
    tokenizer=tokenizer,
    compute_metrics=compute_metrics
)

All of that runs with no problem.  However, I get the following error next:
trainer.train()

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-2-3435b262f1ae> in <module>
----> 1 trainer.train()

/usr/local/bin/miniconda3/envs/tfhub/lib/python3.8/site-packages/transformers/trainer.py in train(self, model_path, trial)
    933
    934             self.control = self.callback_handler.on_epoch_end(self.args, self.state, self.control)
--> 935             self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
    936
    937             if self.args.tpu_metrics_debug or self.args.debug:

/usr/local/bin/miniconda3/envs/tfhub/lib/python3.8/site-packages/transformers/trainer.py in _maybe_log_save_evaluate(self, tr_loss, model, trial, epoch)
   1002         metrics = None
   1003         if self.control.should_evaluate:
-> 1004             metrics = self.evaluate()
   1005             self._report_to_hp_search(trial, epoch, metrics)
   1006

/usr/local/bin/miniconda3/envs/tfhub/lib/python3.8/site-packages/transformers/trainer.py in evaluate(self, eval_dataset, ignore_keys, metric_key_prefix)
   1440         start_time = time.time()
   1441
-> 1442         output = self.prediction_loop(
   1443             eval_dataloader,
   1444             description="Evaluation",

/usr/local/bin/miniconda3/envs/tfhub/lib/python3.8/site-packages/transformers/trainer.py in prediction_loop(self, dataloader, description, prediction_loss_only, ignore_keys, metric_key_prefix)
   1569                 losses_host = losses if losses_host is None else torch.cat((losses_host, losses), dim=0)
   1570             if logits is not None:
-> 1571                 preds_host = logits if preds_host is None else nested_concat(preds_host, logits, padding_index=-100)
   1572             if labels is not None:
   1573                 labels_host = labels if labels_host is None else nested_concat(labels_host, labels, padding_index=-100)

/usr/local/bin/miniconda3/envs/tfhub/lib/python3.8/site-packages/transformers/trainer_pt_utils.py in nested_concat(tensors, new_tensors, padding_index)
     83     ), f"Expected `tensors` and `new_tensors` to have the same type but found {type(tensors)} and {type(new_tensors)}."
     84     if isinstance(tensors, (list, tuple)):
---> 85         return type(tensors)(nested_concat(t, n, padding_index=padding_index) for t, n in zip(tensors, new_tensors))
     86     elif isinstance(tensors, torch.Tensor):
     87         return torch_pad_and_concatenate(tensors, new_tensors, padding_index=padding_index)

/usr/local/bin/miniconda3/envs/tfhub/lib/python3.8/site-packages/transformers/trainer_pt_utils.py in <genexpr>(.0)
     83     ), f"Expected `tensors` and `new_tensors` to have the same type but found {type(tensors)} and {type(new_tensors)}."
     84     if isinstance(tensors, (list, tuple)):
---> 85         return type(tensors)(nested_concat(t, n, padding_index=padding_index) for t, n in zip(tensors, new_tensors))
     86     elif isinstance(tensors, torch.Tensor):
     87         return torch_pad_and_concatenate(tensors, new_tensors, padding_index=padding_index)

/usr/local/bin/miniconda3/envs/tfhub/lib/python3.8/site-packages/transformers/trainer_pt_utils.py in nested_concat(tensors, new_tensors, padding_index)
     85         return type(tensors)(nested_concat(t, n, padding_index=padding_index) for t, n in zip(tensors, new_tensors))
     86     elif isinstance(tensors, torch.Tensor):
---> 87         return torch_pad_and_concatenate(tensors, new_tensors, padding_index=padding_index)
     88     elif isinstance(tensors, np.ndarray):
     89         return numpy_pad_and_concatenate(tensors, new_tensors, padding_index=padding_index)

/usr/local/bin/miniconda3/envs/tfhub/lib/python3.8/site-packages/transformers/trainer_pt_utils.py in torch_pad_and_concatenate(tensor1, tensor2, padding_index)
     46 def torch_pad_and_concatenate(tensor1, tensor2, padding_index=-100):
     47     """Concatenates `tensor1` and `tensor2` on first axis, applying padding on the second if necessary."""
---> 48     if len(tensor1.shape) == 1 or tensor1.shape[1] == tensor2.shape[1]:
     49         return torch.cat((tensor1, tensor2), dim=0)
     50

IndexError: tuple index out of range

Thanks in advance!

Context:
Hi @AlanFeder, judging by the stack trace my first guess is that the problem comes from a conflict between padding in the dataset.map operation vs padding on-the-fly in the Trainer.
As described in the Trainer docs 7, when you pass the tokenizer to the Trainer it will be used as follows:

The tokenizer used to preprocess the data. If provided, will be used to automatically pad the inputs the maximum length when batching inputs, and it will be saved along the model to make it easier to rerun an interrupted training or reuse the fine-tuned model.

So it seems that in your code, you’re doing padding twice: once in dataset.map and then again during training.
Can you remove the padding=True argument from your tokenization step and see if that works?


Question:
Hey guys,
I’ve managed to create a quantize version of gpt2 using onnxruntime but i don’t seem to be able to run it for some reason. anyone has a tutorial for it? also how does the “generate” method of the model will work with that any ideas?

Context:
Hi @yanagar25 when you say you cannot run the quantized version, what kind of error are you running into?
Here’s a notebook that explains how to export a pretrained model to the ONNX format: transformers/04-onnx-export.ipynb at master · huggingface/transformers · GitHub 83
You can also find more details here: Exporting transformers models — transformers 4.2.0 documentation 28
I don’t see an obvious reason why the generate method should not work after quantization, so as with most things in deep learning the best advice is to just try and see if it does


Training a domain-specific roberta from roberta-base

Question:
Hey there,
I apologize in advance if the question below is simple but I’m new to transformers and I want to make sure I get things right before wasting my GPU time training the “wrong” model.
The goal: I want to train a domain-specific roberta model, building on the pre-trained roberta model, therefore starting from roberta-base’s weights rather than from scratch.
The issue: I first followed your tutorial 20 , before realizing that the weights  were not initialized on roberta-base’s before training.
My question: What are the correct steps to train a domain-specific model on-top of roberta-base?

Train a ByteLevelBPETokenizer on my data

tokenizer = ByteLevelBPETokenizer()
tokenizer.train(files=paths, vocab_size=50_000, min_frequency=2,
special_tokens=["<s>", “<pad>”, “</s>”, “<unk>”, “<mask>”])
tokenizer.save_model(“mymodel”)’

and use it to preprocess my data, FYI I have 700,000 sentences stored in a txt file, one sentence per line.

from transformers import LineByLineTextDataset
training_dataset = LineByLineTextDataset(tokenizer=tokenizer,    file_path=“data/training.txt”, block_size=128,)
evalutation_dataset = LineByLineTextDataset(tokenizer=tokenizer,     file_path=“data/testing.txt”, block_size=128,)

Get the roberta config

from transformers import RobertaConfig, RobertaForMaskedLM
config = RobertaConfig( vocab_size=52_000, max_position_embeddings=514,
num_attention_heads=12, num_hidden_layers=12, type_vocab_size=1,)
config.save_model(“mymodel”)
model = RobertaForMaskedLM(config=config)

Or should I instead use:

from transformers import RobertaForMaskedLM
model= RobertaForMaskedLM.from_pretrained(‘roberta-base’)

Get data Collaor

from transformers import DataCollatorForLanguageModeling
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=True, mlm_probability=0.15)

Set up training arguments and train

from transformers import Trainer, TrainingArguments, EvalPrediction

What I currently have:

training_args = TrainingArguments(
output_dir="./mymodel",
evaluation_strategy=“steps”,
prediction_loss_only=True,
per_device_train_batch_size= 32,
per_device_eval_batch_size=32,
eval_accumulation_steps = 200,
weight_decay=0.01,
adam_epsilon=1e-6,
max_steps=200000,
warmup_steps=1,
save_steps=200,
save_total_limit=5,
eval_steps= 100,
)
trainer = Trainer(
model=model,
args=training_args,
data_collator=data_collator,
train_dataset=training_dataset,
eval_dataset=evalutation_dataset,
)
trainer.train()

Should I instead use:

python run_mlm.py
–model_name_or_path roberta-base
–train_file path_to_train_file
–validation_file path_to_validation_file
–do_train
–do_eval
–output_dir /tmp/test-mlm

Thanks in advance for any help you can provide!

Context:
Hi aberquand,
I don’t think you can use the pre-trained weights with your domain-specific vocabulary.
[I am not an expert, and I’ve only used BERT not RoBERTA, and I didn’t use the Trainer, so I could be wrong.]
If I understand it correctly, the way the weights learn is dependent on the particular vocabulary.
I suggest you use the pre-trained vocabulary as well as the pre-trained weights.
How different is your vocabulary from the original RoBERTA vocabulary?   I would expect 700000 sentences would be enough to do fine-tuning, but probably not enough to train from scratch.
Did you know, if you use Google Colaboratory you can get a limited amount of GPU-time for free.  It generally maxes out after about 7 hours each day.  Colab uses Jupyter notebooks.
For more information on tokenizer vocabulary, I recommend Chris McCormick’s blogs, eg https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/ 5


Question:
Hi,
I am new to Transformers/NLP.
I am trying to use Transformers for text classification.  If I am not classifying to one of the pre-made GLUE benchmarks (and using my own use-case classes & texts), do I have to “fine-tune” the model?  If I have 35k texts, and 2 labels (imbalanced – 98% vs 2%) can I just use the AutoModelforSequenceClassification to throw a softmax on the end of the transformer, and train that softmax?  Or do I fine-tune the whole thing, using this tutorial 9?
Thanks!  I am excited about better understanding the field and more effectively using the library.

Context:
Hi @AlanFeder,
I’m not familiar with imbalanced datasets, but if I were you, I would try using examples/text-classification/run_glue.py 10.
In this example, instead of assigning GLUE benchmarks task names, we can use our own train_file, validation_file, and test_file.
python run_glue.py \
  --model_name_or_path bert-base-cased \
  --train_file train_file_name \
  --validation_file validation_file_name \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir /path/to/output/

You can use CSV or JSON.
datasets.load_dataset is used in the script, so the document of datasets.load_dataset for local files 1 may help you.
I remember that columns of the following format were easier to handle, but even if the names or order of the columns are different, it may work if the code is rewritten appropriately.
(Please see transformers/run_glue.py at 5ed5a54684ef059fa4c9710858b8e03c61295914 · huggingface/transformers · GitHub 3 for the detail.)
sentence1, sentence2, label

or
sentence, label

Again, I’m not familiar with imbalanced datasets, so I don’t think I’ve answered your question “do I have to fine-tune?” Sorry.
I am just one of the users who is learning about this library, so I hope you can get more useful advice from someone who knows more than I do.


Issue with Transformer notebook’s Getting Started Tokenizers

Question:
In Documentation, I went to Transformer Notebooks and clicked on the collab for Getting Started Tokenizer.  I executed each cell and when I got to the cell where:
from tokenizers.trainers import BpeTrainer
trainer = BpeTrainer(vocab_size=25000, show_progress=True, initial_alphabet=ByteLevel.alphabet())
tokenizer.train(trainer, [“big.txt”])
print(“Trained vocab size: {}”.format(tokenizer.get_vocab_size()))
I ran the cell and got this:
TypeError: Can’t convert <tokenizers.trainers.BpeTrainer object at 0x7f8641325570> to Sequence
I am assuming these cells should work so something changed with the software but not updated in the notebook. I am trying to learn transformers on my own so where can I go to learn if Hugging Face Doc is not up to date? Any help will be appreciated.

Context:
Hi @krwin, this indeed seems to be a bug in the notebook 5, where the order of the arguments for tokenizer.train() in this cell
from tokenizers.trainers import BpeTrainer

# We initialize our trainer, giving him the details about the vocabulary we want to generate
trainer = BpeTrainer(vocab_size=25000, show_progress=True, initial_alphabet=ByteLevel.alphabet())
tokenizer.train(trainer, ["big.txt"])

print("Trained vocab size: {}".format(tokenizer.get_vocab_size()))

is back-to-front (see the docs 6). To fix the problem you can just specify the arguments explicitly:
tokenizer.train(trainer=trainer, files=["big.txt"])

cc: @anthony


Question:
I am attempting to use the following code snippet to load my own custom csv file:
from datasets import load_dataset
dataset = load_dataset(‘csv’, data_files=[‘my_file_1.csv’, ‘my_file_2.csv’])
I am replacing [‘my_file_1.csv’, ‘my_file_2.csv’] with one file name ‘my_file_1.csv’. However, I am getting the following error:   TypeError : expected string or bytes-like object’
Any suggestions?

Context: Hi ! Can you post the full stack trace please ?


BartTokenizer with vocab.json and merge.txt which were created by ByteLevelBPETokenizer encode <s> into 3 tokens

Question:
Hi, I want to create vocab.json and merge.txt and use them with BartTokenizer.
But somehow tokenizer encode <s> into [32, 87, 34] which was originally [0].
Could you show me how to create vocab.json and merge.txt correctly.
or my way of loading vocab.json and merge.txt may be wrong.
Anyway here is what I did.
# in this notebook we'll only get one of the files (the Oscar one) for the sake of simplicity and performance
# !wget -c https://cdn-datasets.huggingface.co/EsperBERTo/data/oscar.eo.txt

# import
from pathlib import Path
from tokenizers import ByteLevelBPETokenizer

paths = [str(x) for x in Path(".").glob("**/*.txt")]

# Initialize a tokenizer
tokenizer = ByteLevelBPETokenizer()

# Customize training
tokenizer.train(files=paths, vocab_size=52_000, min_frequency=2, special_tokens=[
    "<s>",
    "<pad>",
    "</s>",
    "<unk>",
    "<mask>",
])

# check a sentence.

input1 = "Mi estas Julien."
tokenizer.encode("Mi estas Julien.").tokens
Output: ['Mi', 'Ġestas', 'ĠJuli', 'en', '.'] < looks good.

tokenizer.encode("Mi estas Julien.").ids
Output: [958, 316, 14540, 276, 18] < looks good

# check <s> and </s>
tokenizer.encode("<s>").ids, tokenizer.encode("</s>").ids
Output: ([0], [2]) < looks good

# save vocab and merge
!mkdir output
tokenizer.save_model("output","test")

# now let's load vocab.json and merge.txt

# import BartTokenizer
from transformers import BartTokenizer

tokenizer = BartTokenizer(
    vocab_file="output/test-vocab.json",
    merges_file="output/test-merges.txt",
    bos_token="<s>",
    eos_token="</s>",
    sep_token="</s>",
    cls_token="<s>",
    unk_token="<unk>",
    pad_token="<pad>",
    mask_token="<mask>",
)

input1 = "Mi estas Julien."

encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[  958,   316, 14540,   276,    18]])  < looks good

input1 = "<s>Mi estas Julien.</s>"

encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[   32,    87,    34,   958,   316, 14540,   276,    18,   918,    87,  34]]) < ?

# <s> is now [32, 87, 34] ???
input1 = "<s>"

encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[32, 87, 34]]) < ???

It seems encoding and decoding is working but only special_tokens is not working.
Could you give me hint to fix this problem?
My ultimate goal is to train Bart model with my language.
Or is it okay that tokenizer encode <s> into 3 tokens?
Or can I modify vocab.json and merge.txt manually to let BartTokenizer encode <s> into [0] ?
Thanks in advance.

Context:
[UPDATED] I got a workaround.
It seems like initializing BartTokenizer from vocab.json and merge.txt cause the problem.
Even when I initialize BartTokenizer with vocab.json and merge.txt form Roberta’s pre-trained ones, same problem happened.
Here’s my codes.
# import BartTokenizer
from transformers import BartTokenizer

tokenizer = BartTokenizer(
    vocab_file="roberta/vocab.json",
    merges_file="roberta/merges.txt",
    bos_token="<s>",
    eos_token="</s>",
    sep_token="</s>",
    cls_token="<s>",
    unk_token="<unk>",
    pad_token="<pad>",
    mask_token="<mask>",
)

vocab.json and merge.txt was downloaded from below.
https://huggingface.co/roberta-base/resolve/main/vocab.json
https://huggingface.co/roberta-base/resolve/main/merges.txt

input1 = "This is a pen."
encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[ 713,   16,   10, 7670,    4]])

input1 = "<s>This is a pen.</s>"
encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[41552,    29, 15698,   713,    16,    10,  7670, 49803,    29, 15698]]) < ???

input1 = "<s>"
encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[41552,    29, 15698]]) < ???

I got similar problem even with from_pretrained.
tokenizer = BartTokenizer.from_pretrained('facebook/bart-base', add_prefix_space=True)

input1 = "This is a pen."
encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[ 152,   16,   10, 7670,    4]])

input1 = "<s> This is a pen.</s>"
encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[1437,    0,  152,   16,   10, 7670,    4,    2]]) < ???

encoded = tokenizer("<s>", add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[1437,    0]]) <<< ???

But when I use AutoTokenizer, it works fine.
from transformers import AutoTokenizer

# tokenizer
tokenizer = AutoTokenizer.from_pretrained(
    "facebook/bart-base",
)

input1 = "This is a pen."
encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[ 713,   16,   10, 7670,    4]])

input1 = "<s>This is a pen.</s>"
encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[   0,  713,   16,   10, 7670,    4,    2]])

encoded = tokenizer("<s>", add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[0]])  < looks good

And I found a workaround.
Saving pre-trained tokenizer model first and replacing vocab.json and merge.txt with the files created by ByteLevelBPETokenizer works.
# save tokenizer model.
tokenizer.save_pretrained("./saved_model")

# replace vocab.json and merge.txt

# load tokenizer model
tokenizer = AutoTokenizer.from_pretrained('./saved_model/')

input1 = "Mi estas Julien."
encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[  958,   316, 14540,   276,    18]])

input1 = "<s>Mi estas Julien.</s>"
encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[    0,   958,   316, 14540,   276,    18,     2]])

input1 = "<s>"
encoded = tokenizer(input1, add_special_tokens=False, return_tensors="pt").input_ids
Output: tensor([[0]]) < looks good!


Why `from_pretrained` method still works when model config is mismatched?

Question:
Hi,
This might be a silly question. But I try to config a customized Bart, and use from_pretrained method to load weights. And I expect some errors raised, as my config neither completely matches the config in config.json · facebook/bart-base at main 1 nor config.json · facebook/bart-large at main. But it didn’t.
from transformers import BartForConditionalGeneration, BartConfig
myconfig = BartConfig(d_model=1024, max_position_embeddings=256,
    encoder_attention_heads=8, decoder_attention_heads=8,
    encoder_layers=10, decoder_layers=10)

model = BartForConditionalGeneration(myconfig)
model = model.from_pretrained('facebook/bart-base')
model = BartForConditionalGeneration(myconfig)
model = model.from_pretrained('facebook/bart-large')

Just wonder why…

Context: calling from_pretrained initializes a new model using the new config and returns it, which is why there is no error


The location of script for “Training an Abstractive Summarization Model”

Question:
Hi, I have a silly question.
I want to try the scripts on Training an Abstractive Summarization Model 5.
But somehow I could not find out the github url for that.
I mean I want to run the scripts below on the page.
python main.py \
--mode abstractive \
--model_name_or_path bert-base-uncased \
--decoder_model_name_or_path bert-base-uncased \
--cache_file_path data \
--max_epochs 4 \
--do_train --do_test \
--batch_size 4 \
--weights_save_path model_weights \
--no_wandb_logger_log_model \
--accumulate_grad_batches 5 \
--use_scheduler linear \
--warmup_steps 8000 \
--gradient_clip_val 1.0 \
--custom_checkpoint_every_n 300

Could someone give me the github url?
Thanks in advance.

Context:
Hi @kouohhashi, as far as I know the summarisation scripts have been migrated to the seq2seq examples here: transformers/examples/seq2seq at master · huggingface/transformers · GitHub 9
There you can find BART, T5, and Pegasus, although I suggest starting with Pegasus since it produces decent summaries and is relatively light at ~500M params.
Note that the URL you link to is a different library to transformers: if you have questions about that library I suggest you open an issue in their repo / forum


How to train gpt-2 from scratch? (no fine-tuning)

Question:
Hi, I would like to train GPT-2 from scratch. I don’t want to fine-tuning an existing model, but actually train it from scratch with my own tokenizer. How could I do it?
Thanks.

Context:
Hi @iamnotapenguin, the place I would start is by adapting the following script for causal language modelling to your dataset: transformers/run_clm.py at master · huggingface/transformers · GitHub 129
This script allows you to specify both the tokenizer and the model architecture, plus you can do multi-gpu training which is advisable if you’re training from scratch.
Hope that helps!


Multi class text classification tutorial: how does he get away with one out_feature on linear layer?

Question:
I’ve read this tutorial: https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_multiclass_classification.ipynb 25
As I see it, the dataset uses 4 labels. I thought that that would imply having 4 out_features on your last linear layer. But when I check his model, he uses just 1.
Please help me to fix my misunderstanding

Context:
Hi @dickdanieljr, are you referring to this class?
class DistillBERTClass(torch.nn.Module):
    def __init__(self):
        super(DistillBERTClass, self).__init__()
        self.l1 = DistilBertModel.from_pretrained("distilbert-base-uncased")
        self.pre_classifier = torch.nn.Linear(768, 768)
        self.dropout = torch.nn.Dropout(0.3)
        self.classifier = torch.nn.Linear(768, 4)

    def forward(self, input_ids, attention_mask):
        output_1 = self.l1(input_ids=input_ids, attention_mask=attention_mask)
        hidden_state = output_1[0]
        pooler = hidden_state[:, 0]
        pooler = self.pre_classifier(pooler)
        pooler = torch.nn.ReLU()(pooler)
        pooler = self.dropout(pooler)
        output = self.classifier(pooler)
        return output

Here you can see that the last layer is torch.nn.Linear of shape [hidden_dim, num_labels] - or am I missing something?


Trainer epoch does not go through all training data?

Question:
Hello
I’m training a model with transformers Trainer but when I set the number of epoch to eg: 1000 then it seems the training just does 1000 steps however an epoch is normally the number of times the model goes through the entire dataset. Thus, how can we use the trainer such that each epoch goes through the full training dataset (and that we see the progression of these)
Thanks!

Context:
Hi there!
Please post the command/code you are executing as we can’t really help without that.


Not sure why padding isn’t working for me

Question:
There doesn’t seem to be any padding occurring here:
     train_dataset = dataset.shard(10, 1)

    train_dataset.set_format(columns=['text'])

    train_dataset.cleanup_cache_files()

    encoded_dataset = train_dataset.map(lambda examples: tokenizer(examples['text'], padding=True))

    encoded_dataset[:1]

    {'attention_mask': [[1, 1, 1, 1, 1, 1, 1]],
     'input_ids': [[101, 1714, 22233, 21365, 4515, 8618, 102]],
     'text': ['free instagram followers '],
     'token_type_ids': [[0, 0, 0, 0, 0, 0, 0]]}

What am I missing?

Context:
You are using the strategy to pad to the length of the longest sample while also passing your samples one by one to the tokenizer, so no padding happens.
If you want to pass several samples at once, using batched=True in your call to map. If you want to pass to a specific max_length, pass max_length=xxx and padding="max_length" to your call to the tokenizer.


Predicting answers using DistilBertForQuestionAnswering

Question:
I have fine-tuned DistilBertForQuestionAnswering model using custom data. And I saved that model in colab. I can load the model as well.
I need to know how I can use that loaded model to input a question and get a answer(prediction).

Context: You can find this info in the docs 24. Always nice to check docs before asking here


[Beginner] fine-tune Bart with custom dataset in other language?

Question:
@valhalla @sshleifer
Hi, I’m new to the seq2seq model. And I want to fine-tune Bart/T5 for the summarization task. There are some documents related to the fine-tuning procedure.
Such as

      github.com

huggingface/transformers 24

master/examples/seq2seq

  🤗Transformers: State-of-the-art Natural Language Processing for Pytorch and TensorFlow 2.0.

      ohmeow.github.io

modeling.seq2seq.summarization | blurr 16

This module contains custom models, custom splitters, etc... summarization tasks.

And also thanks for the distilbart version.
But my custom dataset is in Japanese. Directly fine-tuning might be impossible. Is it necessary to train a new bpe tokenizer with Japanese data? But I don’t know how to do it.
The second way is to use an existing Japanese tokenizer like bert-japanese, but could I just use it for Bart? How to modify it?
The third way is to use a multilingual model like MBart or MT5. I haven’t tested it. Could I just fine-tune them with the Japanese dataset ?
Please forgive me if this is a stupid question. Thanks in advance.
Thanks in advance.

Context:
Hi @HeroadZ
Bart is trained in English so I don’t think fine-tuning it directly will help.  If you want to train a model from scratch in a new language then yes, you should train a new tokenizer. To train a new tokenizer checkout the tokenizers 14 library.
And both MBart and MT5 support Japanese so that would be a good starting point.
Another option is to leverage language-specific encoder only bert model (in your case  bert-japanese) to create a seq2seq model using the EncoderDecoder framework. See this notebook to know more about EncoderDecoder models
Leverage BERT for Encoder-Decoder Summarization on CNN/Dailymail 59


Question:
An open question for anyone that has used Transformer models in a Streamlit app.  I am using:::
pipeline(“summarization”, model=“sshleifer/distilbart-cnn-6-6”, tokenizer=“sshleifer/distilbart-cnn-6-6”,framework=“pt”)
::: to do summarization in the app. However, it takes about 55 seconds to create the summary, and it appears that 35 seconds or more of that time is spent downloading the model.  Is there another way to access the model quicker?  Perhaps by pre-loading the model to Streamlit Sharing (via the github repo the app sits in)?
Also, the summary generation part of the app appears to work once or twice, but if done any more times the app crashes.  Has anyone else had this experience?

Context: No experience with Streamlit itself, but you can always download the model 6 locally. Usage is a bit different then: you need to provide a directory to the model argument instead of just the model name. So download all those files to a directory, and then use that directory as your arguments.


Creating Trainer object is deleting my ‘labels’ feature

Question:
I have a dataset called tokenized_datasets:
>>> tokenized_datasets
Dataset({
    features: ['attention_mask', 'input_ids', 'labels', 'token_type_ids'],
    num_rows: 755988
})
>>> tokenized_datasets[0].keys()
dict_keys(['attention_mask', 'input_ids', 'labels', 'token_type_ids'])

But when I create a Trainer object, the labels key just disappears!
>>> training_args = TrainingArguments(
    output_dir='./results',          # output directory
    num_train_epochs=3,              # total # of training epochs
    per_device_train_batch_size=16,  # batch size per device during training
    per_device_eval_batch_size=64,   # batch size for evaluation
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./logs',            # directory for storing logs
)
>>> trainer = Trainer(
       model=model,                         # the instantiated 🤗 Transformers model to be trained
       args=training_args,                  # training arguments, defined above
       train_dataset=tokenized_datasets         # training dataset
)
>>> tokenized_datasets[0].keys()                             # WHAT???
dict_keys(['attention_mask', 'input_ids', 'token_type_ids'])

This causes training to fail like so:
>>> trainer.train()
KeyError                                  Traceback (most recent call last)
<ipython-input-108-21d21c7948cc> in <module>()
----> 1 trainer.train()

[SNIPPED]

/home/sbendl/.local/lib/python3.6/site-packages/transformers/file_utils.py in __getitem__(self, k)
   1414         if isinstance(k, str):
   1415             inner_dict = {k: v for (k, v) in self.items()}
-> 1416             return inner_dict[k]
   1417         else:
   1418             return self.to_tuple()[k]

KeyError: 'loss'

I’m at a loss here, and quite frustrated, why on earth is this happening? It doesn’t when I follow the (very similar) code  here. All the code snippets are sequentially run in my notebook, there’s not “hidden” code. I have a dataset, I pass it to the trainer, and as a result my dataset is broken.

Context: How did you create your model? If the key is dropped by the Trainer, it means the model signature does not accept it. You cna also deactivate that behavior by passing remove_unused_columns=False in your TraininingArguments.


Question:
Hello, in RoBERTa article, authors refer to the model’s perplexity. However, I have yet to find a clear definition of what perplexity means in the context of a model training on the Masked Language Modeling Objective as opposed to the Causal Language Modeling task.
Could someone give me a clear definition? Thanks!

Context: It’s the exponential of the cross-entropy loss, like for CLM.


Need of beginning and end of speech tokens for causal language modeling

Question: Hello everyone, I’m trying to learn more about language modeling using huggingface, and how some problems can be modeled as a language to train a model to predict the next token in an arbitrary sequence. In this huggingface tutorial, they mention the use of a [BOS token]. Why is this needed? Does a causal language model need tokens to denote when a sequence begins and ends? What might happen if tokens like this are not included in the training dataset? Will this significantly affect the ability of the model to generate sequences that begin and end properly?

Context: Transformer models create embeddings based on context. If the model doesn’t know where a sentence or document begins and ends, it’s harder for it to determine context, and the resulting embeddings could be affected. If information from an irrelevant context bleeds into the embedding for a word, it can only be a bad thing. As for whether it’s significant, it’s a huge “it depends”. But, unless by some luck things turn out just right, the model will always be a little bit worse.


“table-question-answering” is not an available task under pipeline

Question:
I am trying to load the “table-question-answering” task using the pipeline but I keep getting the message that -
"Unknown task table-question-answering, available tasks are ['feature-extraction',
'sentiment-analysis',............

Below are the lines I run.
from transformers import pipeline
import pandas as pd

tqa = pipeline("table-question-answering")

Context: You should check your version of transformers, it looks like it’s not up-to-date.


Create a transformer model from a pytorch model (model.bin))

Question:
How to go from a model created from Bert and trained in a specific domain stored as model.bin to generate a transformer, tokenizer, model and optimizer??
Sorry if it is very easy but I’m getting crazy!!!
ck=torch.load(’/content/drive/MyDrive/saved_models/pytorch_model.bin’)
print(ck.keys())
odict_keys([‘bert.embeddings.word_embeddings.weight’, ‘bert.embeddings.position_embeddings.weight’, ‘bert.embeddings.token_type_embeddings.weight’, ‘bert.embeddings.LayerNorm.weight’, ‘bert.embeddings.LayerNorm.bias’, ‘bert.encoder.layer.0.attention.self.query.weight’, ‘bert.encoder.layer.0.attention.self.query.bias’, ‘bert.encoder.layer.0.attention.self.key.weight’, ‘bert.encoder.layer.0.attention.self.key.bias’, ‘bert.encoder.layer.0.attention.self.value.weight’, ‘bert.encoder.layer.0.attention.self.value.bias’, ‘bert.encoder.layer.0.attention.output.dense.weight’, ‘bert.encoder.layer.0.attention.output.dense.bias’, ‘bert.encoder.layer.0.attention.output.LayerNorm.weight’, ‘bert.encoder.layer.0.attention.output.LayerNorm.bias’, ‘bert.encoder.layer.0.intermediate.dense.weight’, ‘bert.encoder.layer.0.intermediate.dense.bias’, ‘bert.encoder.layer.0.output.dense.weight’, ‘bert.encoder.layer.0.output.dense.bias’, ‘bert.encoder.layer.0.output.LayerNorm.weight’, ‘bert.encoder.layer.0.output.LayerNorm.bias’, ‘bert.encoder.layer.1.attention.self.query.weight’, ‘bert.encoder.layer.1.attention.self.query.bias’, ‘bert.encoder.layer.1.attention.self.key.weight’, ‘bert.encoder.layer.1.attention.self.key.bias’, ‘bert.encoder.layer.1.attention.self.value.weight’, ‘bert.encoder.layer.1.attention.self.value.bias’, ‘bert.encoder.layer.1.attention.output.dense.weight’, ‘bert.encoder.layer.1.attention.output.dense.bias’, ‘bert.encoder.layer.1.attention.output.LayerNorm.weight’, ‘bert.encoder.layer.1.attention.output.LayerNorm.bias’, ‘bert.encoder.layer.1.intermediate.dense.weight’, ‘bert.encoder.layer.1.intermediate.dense.bias’, ‘bert.encoder.layer.1.output.dense.weight’, ‘bert.encoder.layer.1.output.dense.bias’, ‘bert.encoder.layer.1.output.LayerNorm.weight’, ‘bert.encoder.layer.1.output.LayerNorm.bias’, ‘bert.encoder.layer.2.attention.self.query.weight’, ‘bert.encoder.layer.2.attention.self.query.bias’, ‘bert.encoder.layer.2.attention.self.key.weight’, ‘bert.encoder.layer.2.attention.self.key.bias’, ‘bert.encoder.layer.2.attention.self.value.weight’, ‘bert.encoder.layer.2.attention.self.value.bias’, ‘bert.encoder.layer.2.attention.output.dense.weight’, ‘bert.encoder.layer.2.attention.output.dense.bias’, ‘bert.encoder.layer.2.attention.output.LayerNorm.weight’, ‘bert.encoder.layer.2.attention.output.LayerNorm.bias’, ‘bert.encoder.layer.2.intermediate.dense.weight’, ‘bert.encoder.layer.2.intermediate.dense.bias’, ‘bert.encoder.layer.2.output.dense.weight’, ‘bert.encoder.layer.2.output.dense.bias’, ‘bert.encoder.layer.2.output.LayerNorm.weight’, ‘bert.encoder.layer.2.output.LayerNorm.bias’, ‘bert.encoder.layer.3.attention.self.query.weight’, ‘bert.encoder.layer.3.attention.self.query.bias’, ‘bert.encoder.layer.3.attention.self.key.weight’, ‘bert.encoder.layer.3.attention.self.key.bias’, ‘bert.encoder.layer.3.attention.self.value.weight’, ‘bert.encoder.layer.3.attention.self.value.bias’, ‘bert.encoder.layer.3.attention.output.dense.weight’, ‘bert.encoder.layer.3.attention.output.dense.bias’, ‘bert.encoder.layer.3.attention.output.LayerNorm.weight’, ‘bert.encoder.layer.3.attention.output.LayerNorm.bias’, ‘bert.encoder.layer.3.intermediate.dense.weight’, ‘bert.encoder.layer.3.intermediate.dense.bias’, ‘bert.encoder.layer.3.output.dense.weight’, ‘bert.encoder.layer.3.output.dense.bias’, ‘bert.encoder.layer.3.output.LayerNorm.weight’, ‘bert.encoder.layer.3.output.LayerNorm.bias’, ‘bert.encoder.layer.4.attention.self.query.weight’, ‘bert.encoder.layer.4.attention.self.query.bias’, ‘bert.encoder.layer.4.attention.self.key.weight’, ‘bert.encoder.layer.4.attention.self.key.bias’, ‘bert.encoder.layer.4.attention.self.value.weight’, ‘bert.encoder.layer.4.attention.self.value.bias’, ‘bert.encoder.layer.4.attention.output.dense.weight’, ‘bert.encoder.layer.4.attention.output.dense.bias’, ‘bert.encoder.layer.4.attention.output.LayerNorm.weight’, ‘bert.encoder.layer.4.attention.output.LayerNorm.bias’, ‘bert.encoder.layer.4.intermediate.dense.weight’, ‘bert.encoder.layer.4.intermediate.dense.bias’, ‘bert.encoder.layer.4.output.dense.weight’, ‘bert.encoder.layer.4.output.dense.bias’, ‘bert.encoder.layer.4.output.LayerNorm.weight’, ‘bert.encoder.layer.4.output.LayerNorm.bias’, ‘bert.encoder.layer.5.attention.self.query.weight’, ‘bert.encoder.layer.5.attention.self.query.bias’, ‘bert.encoder.layer.5.attention.self.key.weight’, ‘bert.encoder.layer.5.attention.self.key.bias’, ‘bert.encoder.layer.5.attention.self.value.weight’, ‘bert.encoder.layer.5.attention.self.value.bias’, ‘bert.encoder.layer.5.attention.output.dense.weight’, ‘bert.encoder.layer.5.attention.output.dense.bias’, ‘bert.encoder.layer.5.attention.output.LayerNorm.weight’, ‘bert.encoder.layer.5.attention.output.LayerNorm.bias’, ‘bert.encoder.layer.5.intermediate.dense.weight’, ‘bert.encoder.layer.5.intermediate.dense.bias’, ‘bert.encoder.layer.5.output.dense.weight’, ‘bert.encoder.layer.5.output.dense.bias’, ‘bert.encoder.layer.5.output.LayerNorm.weight’, ‘bert.encoder.layer.5.output.LayerNorm.bias’, ‘bert.encoder.layer.6.attention.self.query.weight’, ‘bert.encoder.layer.6.attention.self.query.bias’, ‘bert.encoder.layer.6.attention.self.key.weight’, ‘bert.encoder.layer.6.attention.self.key.bias’, ‘bert.encoder.layer.6.attention.self.value.weight’, ‘bert.encoder.layer.6.attention.self.value.bias’, ‘bert.encoder.layer.6.attention.output.dense.weight’, ‘bert.encoder.layer.6.attention.output.dense.bias’, ‘bert.encoder.layer.6.attention.output.LayerNorm.weight’, ‘bert.encoder.layer.6.attention.output.LayerNorm.bias’, ‘bert.encoder.layer.6.intermediate.dense.weight’, ‘bert.encoder.layer.6.intermediate.dense.bias’, ‘bert.encoder.layer.6.output.dense.weight’, ‘bert.encoder.layer.6.output.dense.bias’, ‘bert.encoder.layer.6.output.LayerNorm.weight’, ‘bert.encoder.layer.6.output.LayerNorm.bias’, ‘bert.encoder.layer.7.attention.self.query.weight’, ‘bert.encoder.layer.7.attention.self.query.bias’, ‘bert.encoder.layer.7.attention.self.key.weight’, ‘bert.encoder.layer.7.attention.self.key.bias’, ‘bert.encoder.layer.7.attention.self.value.weight’, ‘bert.encoder.layer.7.attention.self.value.bias’, ‘bert.encoder.layer.7.attention.output.dense.weight’, ‘bert.encoder.layer.7.attention.output.dense.bias’, ‘bert.encoder.layer.7.attention.output.LayerNorm.weight’, ‘bert.encoder.layer.7.attention.output.LayerNorm.bias’, ‘bert.encoder.layer.7.intermediate.dense.weight’, ‘bert.encoder.layer.7.intermediate.dense.bias’, ‘bert.encoder.layer.7.output.dense.weight’, ‘bert.encoder.layer.7.output.dense.bias’, ‘bert.encoder.layer.7.output.LayerNorm.weight’, ‘bert.encoder.layer.7.output.LayerNorm.bias’, ‘bert.encoder.layer.8.attention.self.query.weight’, ‘bert.encoder.layer.8.attention.self.query.bias’, ‘bert.encoder.layer.8.attention.self.key.weight’, ‘bert.encoder.layer.8.attention.self.key.bias’, ‘bert.encoder.layer.8.attention.self.value.weight’, ‘bert.encoder.layer.8.attention.self.value.bias’, ‘bert.encoder.layer.8.attention.output.dense.weight’, ‘bert.encoder.layer.8.attention.output.dense.bias’, ‘bert.encoder.layer.8.attention.output.LayerNorm.weight’, ‘bert.encoder.layer.8.attention.output.LayerNorm.bias’, ‘bert.encoder.layer.8.intermediate.dense.weight’, ‘bert.encoder.layer.8.intermediate.dense.bias’, ‘bert.encoder.layer.8.output.dense.weight’, ‘bert.encoder.layer.8.output.dense.bias’, ‘bert.encoder.layer.8.output.LayerNorm.weight’, ‘bert.encoder.layer.8.output.LayerNorm.bias’, ‘bert.encoder.layer.9.attention.self.query.weight’, ‘bert.encoder.layer.9.attention.self.query.bias’, ‘bert.encoder.layer.9.attention.self.key.weight’, ‘bert.encoder.layer.9.attention.self.key.bias’, ‘bert.encoder.layer.9.attention.self.value.weight’, ‘bert.encoder.layer.9.attention.self.value.bias’, ‘bert.encoder.layer.9.attention.output.dense.weight’, ‘bert.encoder.layer.9.attention.output.dense.bias’, ‘bert.encoder.layer.9.attention.output.LayerNorm.weight’, ‘bert.encoder.layer.9.attention.output.LayerNorm.bias’, ‘bert.encoder.layer.9.intermediate.dense.weight’, ‘bert.encoder.layer.9.intermediate.dense.bias’, ‘bert.encoder.layer.9.output.dense.weight’, ‘bert.encoder.layer.9.output.dense.bias’, ‘bert.encoder.layer.9.output.LayerNorm.weight’, ‘bert.encoder.layer.9.output.LayerNorm.bias’, ‘bert.encoder.layer.10.attention.self.query.weight’, ‘bert.encoder.layer.10.attention.self.query.bias’, ‘bert.encoder.layer.10.attention.self.key.weight’, ‘bert.encoder.layer.10.attention.self.key.bias’, ‘bert.encoder.layer.10.attention.self.value.weight’, ‘bert.encoder.layer.10.attention.self.value.bias’, ‘bert.encoder.layer.10.attention.output.dense.weight’, ‘bert.encoder.layer.10.attention.output.dense.bias’, ‘bert.encoder.layer.10.attention.output.LayerNorm.weight’, ‘bert.encoder.layer.10.attention.output.LayerNorm.bias’, ‘bert.encoder.layer.10.intermediate.dense.weight’, ‘bert.encoder.layer.10.intermediate.dense.bias’, ‘bert.encoder.layer.10.output.dense.weight’, ‘bert.encoder.layer.10.output.dense.bias’, ‘bert.encoder.layer.10.output.LayerNorm.weight’, ‘bert.encoder.layer.10.output.LayerNorm.bias’, ‘bert.encoder.layer.11.attention.self.query.weight’, ‘bert.encoder.layer.11.attention.self.query.bias’, ‘bert.encoder.layer.11.attention.self.key.weight’, ‘bert.encoder.layer.11.attention.self.key.bias’, ‘bert.encoder.layer.11.attention.self.value.weight’, ‘bert.encoder.layer.11.attention.self.value.bias’, ‘bert.encoder.layer.11.attention.output.dense.weight’, ‘bert.encoder.layer.11.attention.output.dense.bias’, ‘bert.encoder.layer.11.attention.output.LayerNorm.weight’, ‘bert.encoder.layer.11.attention.output.LayerNorm.bias’, ‘bert.encoder.layer.11.intermediate.dense.weight’, ‘bert.encoder.layer.11.intermediate.dense.bias’, ‘bert.encoder.layer.11.output.dense.weight’, ‘bert.encoder.layer.11.output.dense.bias’, ‘bert.encoder.layer.11.output.LayerNorm.weight’, ‘bert.encoder.layer.11.output.LayerNorm.bias’, ‘bert.pooler.dense.weight’, ‘bert.pooler.dense.bias’, ‘cls.predictions.bias’, ‘cls.predictions.transform.dense.weight’, ‘cls.predictions.transform.dense.bias’, ‘cls.predictions.transform.LayerNorm.weight’, ‘cls.predictions.transform.LayerNorm.bias’, ‘cls.predictions.decoder.weight’, ‘cls.seq_relationship.weight’, ‘cls.seq_relationship.bias’])
Thanks a lot

Context:
Not sure what you mean could please rephrase the question?
What do you mean by

generate a transformer, tokenizer, model and optimizer


Add data augmentation process during training every epoch

Question:
Hello,
I’d like to process my training dataset every epoch.
I want to add random processing as data augmentation, and I want to do it during training, not preprocessing.
I think I can do it with Trainer, DataCollator, or __getitem__ of datasets.arrow_dataset.Dataset, but where should I do it?
For the evaluation set and test set, I plan to do a preprocess using datasets.arrow_dataset.Dataset.map.
Thank you in advance.

Context:
The DataCollator can help if you have something randomized in the call that returns the batch. A getitem in your Dataset can also help, it all depends on what you are trying to do exactly.
The Trainer in itself has nothing implemented for data augmentation, so it won’t help you.


Calculate F1 score in a NER task with BERT

Question:
Hi everyone,
I fine tuned a BERT model to perform a NER task using a BILUO scheme and I have to calculate F1 score.
However, in named-entity recognition, f1 score is calculated per entity, not token.
Moreover, there is the Word-Piece “problem” and the BILUO format, so I should:

aggregate the subwords in words
remove the prefixes “B-”, “I-”, “L-” from each entity
calculate the F1 score on the entity

Before I spend hours (if not days) to try to implement such code, I would like to know if an implemented solution already exists.
Thanks in advance

Context: You should use the datasets metric seqeval that will do all of this for you. Check the new run_ner script 116 for an example.


[Tensorflow Export] How to export a fine tuned GPT2 model to a tensorflow model file?

Question:
Sorry if this is a silly question, but I cant seem to find any proper solution to this.
I am using transformers==2.8.0 and have fine-tuned a gpt2 model with my own dataset.  I know that during training it creates checkpoints in pytorch and that can be used for text generation, but I want to save/load model in tensorflow.
I know that TFGPT2LMHeadModel exists and that it can be used, but I havent found an example online doing this.
Can someone help me please? How can I export a fine-tuned model into tensorflow, so that I can then generate text using that model?
Thanks

Context: You can check this part of the docs 9 that shows how to reload a model trained with PyTorch into TensorFlow (and vice versa).


Question:
Hello,
Could you please point me out the direction to look for a possibility to do multi-class text classification?
I need to assign multiple tags (genres) for texts describing movies/books.

Context:
Should it be done using class BertForMultipleChoice(BertPreTrainedModel)?
Found also transformers-tutorials/transformers_multi_label_classification.ipynb at master · abhimishra91/transformers-tutorials · GitHub 18 , but authors didn’t use BertForMultipleChoice there.


Loading a model from local with best checkpoint

Question:
Hi all,
I have trained a model and saved it, tokenizer as well. During the training I set the load_best_checkpoint_at_end to True and can see the test results, which are good
Now I have another file where I load the model and observe results on test data set. I want to be able to do this without training over and over again. But the test results in the second file where I load the model are worse than the ones right after training.
Is there a way to load the model with best validation checkpoint ?
This is how I save:
tokenizer.save_pretrained(model_directory)
trainer.save_model()
and this is how i load:
tokenizer = T5Tokenizer.from_pretrained(model_directory)
model = T5ForConditionalGeneration.from_pretrained(model_directory, return_dict=False)

Context:
I don’t understand the question. With load_best_model_at_end the model loaded at the end of training is the one that had the best performance on your validation set. So when you save that model, you have the best model on this validation set.
If it’s crap on another set, it means your validation set was not representative of the performance you wanted and there is nothing we can do on Trainer to fix that.


Data shape needed for training TransformerXL from scratch

Question:
Hello everyone,
I’m having trouble getting my data into the proper shape to train a TransformerXL model from scratch. I have a custom Pytorch Dataset that returns a dict from __getitem__ with the keys input_ids, and labels both assigned the same 1-D Tensor of ids for a particular sequence. When I pass this Dataset to the Trainer object with the default_collator I get a grad can be implicitly created only for scalar outputs error. What am I missing here? Am I missing a needed field for the TransfoXLLMHeadModel's forward method? I’ve tried just about everything and cannot figure it out.

Context:
Hi @jodiak
Would be hard to answer this without looking at the code, could you post a small code snippet ?
Also TransformerXL is a language model so the required inputs are input_ids and labels, which are of shape [batch_size, seq_len].


What happens in the MT5 documentation example?

Question:
Hi,
I’m trying to understand the provided example to the MT5 model 11 but have some difficulties.
Here is the example:
from transformers import MT5Model, T5Tokenizer
model = MT5Model.from_pretrained(“google/mt5-small”)
tokenizer = T5Tokenizer.from_pretrained(“google/mt5-small”)
article = “UN Offizier sagt, dass weiter verhandelt werden muss in Syrien.”
summary = “Weiter Verhandlung in Syrien.”
batch = tokenizer.prepare_seq2seq_batch(src_texts=[article], tgt_texts=[summary], return_tensors=“pt”)
outputs = model(input_ids=batch.input_ids, decoder_input_ids=batch.labels)
hidden_states = outputs.last_hidden_state
So I understand that tokenizer.prepare_seq2seq_batch is to encode the input to provide to the model. It is a BatchEncoding containting the input_ids, labels and attention_mask.
However, I don’t understand what follows, what happens in : model(input_ids=batch.input_ids, decoder_input_ids=batch.labels) ? This does not train or fine tune the model but what does it do ?
Why do we provide it a source and target then ? What if we wanted the model to generate the target (summary) ?
Thanks !

Context:
The example is just a general example of how to do a forward pass through the model, just like you can do in any model.  In practice, you’d see something like this:

      github.com

    huggingface/transformers/blob/4f7022d68d4bae4b5e6a748b7a7323515c6fdcd3/examples/seq2seq/seq2seq_trainer.py#L162-L176 4

def _compute_loss(self, model, inputs, labels):    if self.args.label_smoothing == 0:        if self.data_args is not None and self.data_args.ignore_pad_token_for_loss:            # force training to ignore pad token            logits = model(**inputs, use_cache=False)[0]            loss = self.loss_fn(logits.view(-1, logits.shape[-1]), labels.view(-1))        else:            # compute usual loss via models            loss, logits = model(**inputs, labels=labels, use_cache=False)[:2]    else:        # compute label smoothed loss        logits = model(**inputs, use_cache=False)[0]        lprobs = torch.nn.functional.log_softmax(logits, dim=-1)        loss, _ = self.loss_fn(lprobs, labels, self.args.label_smoothing, ignore_index=self.config.pad_token_id)    return loss, logits


Question:
Hi,
I am very new to transformers and found out about it when looking for a LayoutLM implementation.
Now from my understanding, LayoutLM can be used to extract information from a document based on the layout it guessed.
When browsing the documentation, I could only see examples using plain text and I don’t know where to begin to put an image instead.
If it would be possible to help a newbie like me, showing how to pass it an image and how to interpret the results, you would really make me an happy man!!
I really hope someone can help me.
Have a great day

Context:
Hi eveningkid,
transformer models are designed for text.
It might be possible to force the model to accept a numeric representation of an image (after all, it’s all ones and noughts), but it would be unlikely to do anything useful.


Question:
Hi all,
I am new to Hugging Face and am looking to solve a problem of extracting entity relationships as a diagram/graph from company documents. For instance, Bob reports to Sally.
My first guess would be to use NER to label relationships, but is that the right track? All thoughts welcome.
Many thanks in advance,
Ari

Context:
I’m guessing you would need to combine NER with aspect analysis to understand reports to points to Sally and topic modelling/classification to understand how they go together. But I am not sure how you would go about combining the three. I believe a lot of the BERT question-answering models are able to understand and answer about these relationships, so it is happening somewhere inside those models.I guess you would need to look deeper into how those work.
Thanks & Regards,
Daryl


Question: hi, i’m using HuggingFace for multi-label classification. i am curious if there is a way to change/customize the classifier head on top? and second question is, the default type of classifier is linear. i heard of classifiers like SVM, decision tree etc. can anyone explain their connection? between linear and those ones?

Context:
You can extend a pretrained model with your own layers as much as you want. Something like this can work:
class BertCustomClassification(BertPreTrainedModel):
    def __init__(self, config):
        super().__init__(config)
        self.num_labels = config.num_labels

        self.bert = BertModel(config)
        # Add classifier layers below
        self.dropout = nn.Dropout(config.hidden_dropout_prob)
        self.preclassifier = nn.Linear(config.hidden_size, config.hidden_size)
        self.act = nn.GELU()
        self.classifier = nn.Linear(config.hidden_size, config.num_labels)

        # don't forget to init the weights for the new layers
        self.init_weights()

You should then also change the forward pass, of course.
Your question about other ML architectures like SVMs and decision trees is too broad for this forum and fall outside of the scope of HuggingFace Transformers. You can ask such question on a website like Stack Overflow (but search first because this question has been asked a billion times).


How does trainer handle lists with None items?

Question:
I am working through SquAD 2.0 QA with Bert and I am trying to figure out how to deal with data that contains None in it with PyTorch.
For QA tasks, for validation/test data only, we have data contained under the key: 'offset_mapping' and some of its data looks like:
# [[None, None, None, None, None, None, None, None, None, (0, 3), (4, 10), (10, 11), (12, 13), (13, 19), (19, 20), (21, 23), (23, 25), (25, 28), (28, 30), (30, 3]]
In this guide: https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/question_answering.ipynb#scrollTo=s3cuvZsNleG2&line=4&uniqifier=1, data that looks like above is successfully fed into trainer.predict() as well as into torch data loaders. However, torch data loaders will not accept tensors with None in them. How is huggingface handling and getting around these None values?
I am having a hard time figuring that out from the source code here: transformers.trainer — transformers 4.1.1 documentation

Context: The Trainer automatically ignores columns that do not match model arguments. In this case, “offset_mapping” is not an argument of the QA model, so it’s ignored inside the Trainer. That’s why it’s not a problem if it has None items.


way to make inference Zero Shot pipeline faster?

Question:
Hi
Can you guys give me tips how to make Zero Shot pipeline inference faster?
My current approach right now is reducing the model size/parameter
(trying to train “base model” instead of "large model)
Is there another approach?

Context:
There’s some discussion in this topic 56 that you could check out.
Here are a few things you can do:

Try out one of the community-uploaded distilled models 56 on the hub (thx @valhalla) . I’ve found them to get pretty similar performance on zero shot classification and some of them are much smaller and faster. I’d start with valhalla/distilbart-mnli-12-3 (models can be specified by passing e.g. pipeline("zero-shot-classification", model="valhalla/distilbart-mnli-12-3") when you construct a model.
If you’re on GPU, make sure you’re passing device=0 to the pipeline factory in to utilize cuda.
If you’re on CPU, try running the pipeline with ONNX Runtime. You should get a boost. Here’s a project 81 (thx again @valhalla)  that lets you use HF pipelines with ORT automatically.
If you have a lot of candidate labels, try to get clever about passing just the most likely ones to the pipeline. Passing a large # of labels for each sentence is really going to slow you down since each sentence/label pair has to be passed to the model together. If you have 100 possible labels but you can use some kind of heuristic or simpler model to narrow it down, that will help a lot.
Use mixed precision. This is pretty easy 70 if using PyTorch 1.6.


Question: hi, wanted to play with the tapas online API on [https://huggingface.co/google/tapas-base-finetuned-wtq?] 5 but seems it does not work if any element of the table or the question is changed (got the message “Model google/tapas-base-finetuned-wtq is currently loading” for ever)…Any help on this please ? Best. L

Context: cc @lysandre


Remove dataset downloaded by dataset library from local computer

Question:
Hi ,
Currently I have downloaded dataset “wmt19” with python library dataset in my WSL2 Ubuntu distro but now I want to delete this dataset but I cannot find where it located.
Could somebody tell me where is the dataset locate, or did it have function for remove dataset from local.
Also I accidently delete cache data in .cache/huggingface. I’m not sure it will be any consequence here ?

Context: I just found that It was a problem with WSL unable to reclaim space after delete file.


Resources for model design (number of layers, attention heads, etc)

Question:
I’ve been using the transformers libraries for the last several months in creative generative text projects. I’m just a hobbyist – I mostly understand how everything works abstractly, but definitely don’t have a firm grasp on the underlying math. I started out by fine-tuning GPT-2, but lately I’ve been playing around with training models from scratch (e.g. arpabet, where “phonetic english” is represented as “fA0nE1tI0k I1GglI0S”), and I’m looking for resources/tips/pointers for how the various parameters will affect the models and their output.
I started my from-scratch model training by loading a GPT2LMHeadModel with the pre-trained gpt2 config, and that worked great. But I figured I could get better results if I went bigger. I tried training again using the gpt2-xl config, but the memory requirements were too high. So now I’m dialing in the parameters based on gpt2-large.
I’ve noticed that with all other values the same, I can fit the model into memory with the default 36 layers, 20 heads, but also something like 18 layers and 64 heads. I’d experiment with these (and other) values, but since training for 3 epochs through my data set would take weeks at a time, I was wondering if anyone could point me in the right direction as to what the tradeoffs are between number of layers, hidden, attention heads, etc.
Thanks!

Context:
This may not be exactly what you’re looking for, but this paper explores the performance impacts of different configurations of decoder, attention heads, hidden layers and intermediate layers.

      arxiv.org

2010.10499.pdf 5

552.32 KB


Strange output using BioBERT for imputing MASK tokens

Question:
I’m trying to use BioBERT (downloaded from the HuggingFace models repository at dmis-lab/biobert-v1.1) to fill in MASK tokens in text, and I’m getting some unexpected behavior with the suggested tokens.
I pasted a screenshot below comparing bert-base-uncased (which behaves as expected and has sensible most-likely tokens) with BioBERT:
Screen Shot 2020-12-30 at 10.31.42 PM1844×1818 543 KB
Here’s the code to reproduce this:
from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch

text = 'heart disease is [MASK] leading cause of death in the united states.'

tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
model = AutoModelForMaskedLM.from_pretrained('bert-base-uncased')
tokenized = tokenizer(text, return_tensors='pt')
idx = tokenizer.convert_ids_to_tokens(tokenized.input_ids[0]).index(tokenizer.mask_token)
output = model(**tokenized, return_dict=True)
print(tokenizer.convert_ids_to_tokens(torch.topk(output.logits[0, idx, :], 10).indices))

tokenizer = AutoTokenizer.from_pretrained('dmis-lab/biobert-v1.1')
model = AutoModelForMaskedLM.from_pretrained('dmis-lab/biobert-v1.1')
tokenized = tokenizer(text, return_tensors='pt')
idx = tokenizer.convert_ids_to_tokens(tokenized.input_ids[0]).index(tokenizer.mask_token)
output = model(**tokenized, return_dict=True)
print(tokenizer.convert_ids_to_tokens(torch.topk(output.logits[0, idx, :], 10).indices))

And here’s my output from running transformers-cli env:
- `transformers` version: 4.1.1
- Platform: macOS-10.11.6-x86_64-i386-64bit
- Python version: 3.8.5
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

I also asked about similar issues with PubMedBERT as a Github issue 2 a while back, but haven’t gotten a response.
Do the pretrained weights for these models not contain the components necessary for doing masked language modeling/imputing MASK tokens? Is there any way to fix this issue?

Context:
Hi,
I am not an expert, but that is what it looks like to me.
Masked Language Modelling is usually used during pre-training, and is often not needed during fine-tuning, so I guess the DIMS team didn’t think the MLM parameters would be required.
I notice that the DIMS team have provided 5 models.  Do any of the other models have MLM parameters?
It should certainly be possible to copy the DIMS weights into your own model, where your own model does include an MLM head.  I expect you would then need to train your model before it would give sensible answers, unless you could find a suitable MLM head to copy (probably not…).


Transformers, am i only using a Encoder for Binary Classification?

Question:
Hi Guys, i’ve got a basic, begginers Questions. If im doing something like a Binary Classification (Sentiment Analysis) of Text and im using Transformers (like Bert for example). Am i using both Encoder, and the Decoder Part of the Transformer Network? If i understood the basics right im just using an Encoder for Sequences to Binary - and Sequences to Sequence would use both Decoder and Encoder?
Ty in advanced.

Context:
Hi @unknownTransformer,
BERT uses only the Encoder.
See this page of the docs: https://huggingface.co/transformers/model_summary.html 25
See also the Devlin paper: https://arxiv.org/abs/1810.04805 1
Also try Jay Alammar’s blogs, for example this one: alammar.github.io/illustrated-bert/ 4
When you do sentiment analysis, you are using the basic BERT to code your text as numbers, and then you tune a final layer to your task.   The Huggingface models such as BertForSequenceClassification already include a “final” layer for you, which is randomly initialized.  See this page: https://huggingface.co/transformers/model_doc/bert.html 15
You will probably want to Freeze most of the BERT layers while you fine-tune the last layer, at least initially. See this post and the reply by sgugger: How to freeze some layers of BertModel 9
Good luck with it all.


Evaluating your model on more than one dataset

Question:
Hi,
Transformer’s Trainer and Trainingarguments classes allow for only one dataset to use for evaluation. Is there a simple way of adding another one? So, after after an epoch of training my model I could evaluate it on both training and developmental datasets and get metrics for both of them as one output? I know I could alter the training_args.py or trainer.py but I am pretty sure I would only mess things up…

Context: I think the easiest way to do this is to use the new system of TrainerCallback 16 and write a callback that performs a new evaluation on your other datasets during the event on_validate.


Question:
I finally got the  transformer library installed with CUDA support under WSL 2 Ubuntu. Yay.
- `transformers` version: 4.2.0dev0
- Platform: Linux-5.4.72-microsoft-standard-WSL2-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

However, something in the stack is giving me an unreliable network connection that  reacts badly to.
This is what I’m trying to do (standard install test - I added the resume_download optional argument after reading a bug fix report. It doesn’t help).   The question is, is there either a way to MANUALLY download models from HF into its cache, or is there a setting that does re-tries from the current file position in such cases?
python -c "from transformers import pipeline; print(pipeline('sentiment-analysis',resume_download=True)('we love you'))"

A few megabytes into the download (it varies randomly, but never more than 60), I get the following exception:
(“read error: Error([(‘SSL routines’, ‘ssl3_get_record’, ‘decryption failed or bad record mac’)])”,)
`("read error: Error([('SSL routines', 'ssl3_get_record', 'decryption failed or bad record mac')])",)                            | 6.65M/268M [00:02<01:12, 3.60MB/s]`
    Traceback (most recent call last):
      File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 313, in recv_into
        return self.connection.recv_into(*args, **kwargs)
      File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1822, in recv_into
        self._raise_ssl_error(self._ssl, result)
      File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1647, in _raise_ssl_error
        _raise_current_error()
      File "/usr/lib/python3/dist-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
        raise exception_type(errors)
    OpenSSL.SSL.Error: [('SSL routines', 'ssl3_get_record', 'decryption failed or bad record mac')]

The exception handler for that apparently is subject to the same exception:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/mjw/transformers/src/transformers/modeling_utils.py", line 1003, in from_pretrained
    resolved_archive_file = cached_path(
  File "/home/mjw/transformers/src/transformers/file_utils.py", line 1077, in cached_path
    output_path = get_from_cache(
  File "/home/mjw/transformers/src/transformers/file_utils.py", line 1303, in get_from_cache
    http_get(url_to_download, temp_file, proxies=proxies, resume_size=resume_size, headers=headers)
  File "/home/mjw/transformers/src/transformers/file_utils.py", line 1166, in http_get
    for chunk in r.iter_content(chunk_size=1024):
  File "/usr/lib/python3/dist-packages/requests/models.py", line 750, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 564, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 507, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "/usr/lib/python3.8/http/client.py", line 458, in read
    n = self.readinto(b)
  File "/usr/lib/python3.8/http/client.py", line 502, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 332, in recv_into
    raise ssl.SSLError("read error: %r" % e)
ssl.SSLError: ("read error: Error([('SSL routines', 'ssl3_get_record', 'decryption failed or bad record mac')])",)

Context: That extra resume-download argument turns out to be invalid, once the download succeeds randomly once. So ignore that


Question:
I am working on a school project which is to classify news headlines. It’s a binary classification. I scraped the news headlines, used sklearn train_test_split to split them. Then used ktrain 6 - distilBert to classify them. There is a learn rate finder function, I run that and get an abnormal learn rate curve as shown in below image:
Screenshot 2021-01-01 at 13.54.54792×564 28.3 KB
while the normal learn rate should be somehow in a U-shape, falls gradually from a higher loss then up again.
What does that abnormal learn rate curve imply? Is it to do with overfitting or anything? I am really new to the transformer thing and there are not many resources on the internet so I try to ask here. Thanks.

Context:
Hi, not sure if I understand correctly

Which “learn rate finder function” you use ? ( just curious as I am familiar of the idea coming from fast.ai team )
spike in 10^-1 looks plausible to me since it’s too much big learning rate
Did you initialize your model from some checkpoints ? If yes, maybe the model is already good and have small loss in the beginning, so small loss with LR=10^-7 is also plausible


Question:
I’ve been prototyping my model training code on my local machine (2x RTX 3090 GPUs), and I’m now trying to migrate it over for a full training run on the university HPC cluster. What’s confusing me is that training on the cluster node (which has 4x RTX 8000s) is reporting completion times that are a lot longer than what I was seeing locally (same dataset and batch size).
On my local machine, one epoch is projected to take ~84 hours:
49/586086 [00:28<84:02:02,  1.94it/s]
On the HPC, it’s predicting 455 hours(!):
76/293043 [07:13<455:24:38,  5.60s/it]
(note the different units: it/s vs s/it)
I’ve checked with nvidia-smi and all four GPUs are at 100%. The dataset is being stored on a local disk in both cases. So I’m running out of ideas for what could be happening…

Context: I’ve looked into this more and I think it’s a performance bug related to excessive GPU-GPU communication: https://github.com/huggingface/transformers/issues/9371 12


Is beam search always better than greedy search?

Question:
How to generate text 7 states:

Beam search will always find an output sequence with higher probability than greedy search

It’s not clear to me why that is the case. Consider this example, comparing greedy search with beam search with beam width 2:
image551×665 24.1 KB
By the 3rd step, beam search with beam width 2 has found two sequences BFI and BGK, each with probability .49*.5*.51 = 0.12495, but greedy search found ADH, with probability .51*.4*.99 = 0.20196
Exploring more of the tree may cause beam search to get stuck on paths that seem more promising at first, but end up with lower probability than the greedy path. Is this right, or have I misunderstood how beam search works?
Thanks!
Robby

Context:
I think this is an interesting adversarial case for beam-search
So I agree with you that in this case Greedy found the more probable path.


Question:
Hi all.  I’m trying to get this script to run, but I’m getting errors.  Any ideas? Here’s my output:
chmod +x run_squad.py
❯ ./run_squad.py
./run_squad.py: 16:  Finetuning the library models for question-answering on SQuAD (DistilBERT, Bert, XLM, XLNet).: not found
import-im6.q16: attempt to perform an operation not allowed by the security policy PS' @ error/constitute.c/IsCoderAuthorized/408. import-im6.q16: attempt to perform an operation not allowed by the security policy PS’ @ error/constitute.c/IsCoderAuthorized/408.
import-im6.q16: attempt to perform an operation not allowed by the security policy PS' @ error/constitute.c/IsCoderAuthorized/408. import-im6.q16: attempt to perform an operation not allowed by the security policy PS’ @ error/constitute.c/IsCoderAuthorized/408.
import-im6.q16: attempt to perform an operation not allowed by the security policy PS' @ error/constitute.c/IsCoderAuthorized/408. import-im6.q16: attempt to perform an operation not allowed by the security policy PS’ @ error/constitute.c/IsCoderAuthorized/408.
import-im6.q16: unable to grab mouse ': No such file or directory @ error/xwindow.c/XSelectWindow/9187. import-im6.q16: unable to grab mouse ‘: No such file or directory @ error/xwindow.c/XSelectWindow/9187.
from: can’t read /var/mail/torch.utils.data
from: can’t read /var/mail/torch.utils.data.distributed
from: can’t read /var/mail/tqdm
import-im6.q16: unable to grab mouse `’: No such file or directory @ error/xwindow.c/XSelectWindow/9187.
./run_squad.py: 33: Syntax error: “(” unexpected
THanks, and Happy Holidays!

Context: Never mind.  I found the README


Question:
Hi, I am trying to know how to use Rag/DPR, but first I want to get familiar with faiss usage.
I checked the official example in

      huggingface.co

Main classes — datasets 1.1.3 documentation 6

But it seems the snippet code is not self-executable.
So I did some modification, aiming to retrieve similar examples in the sst2 dataset with query ‘I am happy’.
import datasets
from transformers import pipeline
embed = pipeline('sentiment-analysis', model="nlptown/bert-base-multilingual-uncased-sentiment")
ds = datasets.load_dataset('glue', 'sst2', split='test')
ds_with_embeddings = ds.map(lambda example: {'embeddings': embed(example['sentence'])})
ds_with_embeddings.add_faiss_index(column='embeddings')
# query
scores, retrieved_examples = ds_with_embeddings.get_nearest_examples('embeddings', embed('I am happy.'), k=10)
# save index
ds_with_embeddings.save_faiss_index('embeddings', 'my_index.faiss')

ds = datasets.load_dataset('glue', 'sst2', split='test')
# load index
ds.load_faiss_index('embeddings', 'my_index.faiss')
# query
scores, retrieved_examples = ds.get_nearest_examples('embeddings', embed('I am happy.'), k=10)

My problem is at ds_with_embeddings.add_faiss_index(column='embeddings')
I got error with there with " TypeError: float() argument must be a string or a number, not ‘dict’ "
If I changed it to
ds_with_embeddings_score = ds_with_embeddings.map(lambda example: {'embeddings_score': example['embeddings'][0]['score']})
ds_with_embeddings_score.add_faiss_index(column='embeddings_score')

Then I got error " TypeError: len() of unsized object "
Any adivce?Thanks.

Context:
I have little experience with pipelines, but I think the issue is that embed(example['sentence']) should return a vector representation for example['sentence']. However, calling a text classification pipeline returns a dict with labels and scores. Instead, you need to run a feature extraction pipeline which should return vectors. (You may need to unpack it though, as the return type is a nested list, presumably for batched processing.)
So (untested) you can try something like:
embed = pipeline('feature-extraction', model="nlptown/bert-base-multilingual-uncased-sentiment")
...
ds_with_embeddings = ds.map(lambda example: {'embeddings': embed(example['sentence'])[0]})

You may need to test a bit with the [0]. Not sure whether it is necessary.


Question:
Hi,
I am just wondering what is the purpose of the “Authorization” http header in the inference API. If I remove this header, the request is still working. Example:
curl 'https://api-inference.huggingface.co/models/bert-base-uncased' \
  -H 'Connection: keep-alive' \
  -H 'Content-Type: text/plain;charset=UTF-8' \
  -H 'Accept: */*' \
  -H 'Accept-Language: en,en-US;q=0.9,id;q=0.8,de;q=0.7,ms;q=0.6' \
  --data-binary '{"inputs":"If I am hungry, I will make [MASK]."}' \
  --compressed

Context:
@Narsil and @jeffboudier can chime in, but we offer a certain number of non-authed requests, with IP-based rate limiting.
For production workloads you’ll need a token.


Pegasus finetuning, should we always start with pegasus-large?

Question:
I’m fine-tuning pegasus on my own data, which is about 15,000 examples.
I am finding, when fine-tuning Pegasus, using pegasus-large , that the RAM requirements for even just a batch size of 1 are so extreme, that a Nvidia card with 16GB of memory is required… just to run the batch size of 1 ! So at this point I am thinking that maybe my training will run better on the CPU, using a machine with a huge amount of ram… like 512GB of ram… as this seems to allow a much bigger batch size, like up to 64 or 128 .
My guess is that the RAM requirements are so extreme because I am using pegasus-large. I’m doing this based on my understanding of this page:
: https://huggingface.co/transformers/model_doc/pegasus.html#checkpoints 2

All the checkpoints 9 are fine-tuned for summarization, besides pegasus-large, whence the other checkpoints are fine-tuned

My understanding from this is that, if we, as the newbie user, have some data we want to use with Pegasus, we should do this:

Start with pegasus-large: https://huggingface.co/google/pegasus-large 4

Fine tune it on our own data
Use the pytorch_model.bin output from this fine tuning process to run inference on our own data.

Am I getting something wrong here?  Given that I have 15,000 examples, have I made the correct determination that I should fine-tune pegasus-large, and that this will lead to the best results, even though the memory requirements are huge?
I looked for  distilled model, here: https://huggingface.co/models?search=pegasus 9
… But my understanding (possibly wrong?) is that these distilled models are ALREADY fine-tuned, so they would not be appropriate to use, given that I have a lot of my OWN data to fine-tune with.
Thanks!

Context:
To answer your second questions,
the student models are a smaller version of the finetuned-pegasus models, created by choosing alternating layers from the decoder. This method is described in the Pre-trained Summarization Distillation 18. To use these models you should fine-tune them


Using MarianModel’s in pytorch is too slow to do back translation (not parallelised correctly)

Question:
Hi.
I’m trying to use MarianModels for back translation as data augmentation. However, it’s too slow even using multiple GPUs. and I also can not use a batch size larger than 16 setting the max length to 300 though.  Indeed it takes one day to complete half an epoch.
following is the code I’m using
target_langs = ['fr,wa,frp,oc,ca,rm,lld,fur,lij,lmo,es,pt,gl,lad,an,mwl,it,co,nap,scn,vec,sc,ro,la']

def translate(texts, model, tokenizer, language="fr"):

    with torch.no_grad():
        template = lambda text: f"{text}" if language == "en" else f">>{language}<< {text}"
        src_texts = [template(text) for text in texts]
        encoded = tokenizer.prepare_seq2seq_batch(src_texts,
                                                      truncation=True,
                                                      max_length=300, return_tensors="pt").to(device)
        translated = model.module.generate(**encoded).to(device)
        translated_texts = tokenizer.batch_decode(translated, skip_special_tokens=True)
        return translated_texts

def back_translate(texts, source_lang="en", target_lang="fr"):
    # Translate from source to target language
    fr_texts = translate(texts, target_model, target_tokenizer,
                         language=target_lang)

    # Translate from target language back to source language
    back_translated_texts = translate(fr_texts, en_model, en_tokenizer,
                                      language=source_lang)

    return back_translated_texts

target_model_name = 'Helsinki-NLP/opus-mt-en-de'
target_tokenizer = MarianTokenizer.from_pretrained(target_model_name)
target_model = MarianMTModel.from_pretrained(target_model_name)

en_model_name = 'Helsinki-NLP/opus-mt-de-en'
en_tokenizer = MarianTokenizer.from_pretrained(en_model_name)
en_model = MarianMTModel.from_pretrained(en_model_name)

target_model = nn.DataParallel(target_model)
target_model = target_model.to(device) # same performance  if I add .half()
target_model.eval()

en_model = nn.DataParallel(en_model)
en_model = en_model.to(device)# same performance if I add .half()
en_model.eval()

## x1 and x2 are batches of strings.
bk_x1 = back_translate(x1, source_lang="en", target_lang=np.random.choice(target_langs))
bk_x2 = back_translate(x2, source_lang="en", target_lang=np.random.choice(target_langs))

here are GPU’s performances:   low utilization due to small batch size 16 but if I increase the batch size I got Cuda out of memory error. also, I can see only one gpu is used for processing so might be that the Marian model can not be parallelized correctly. if so what would be the solution?
 |-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:1B:00.0 Off |                  N/A |
| 42%   78C    P2   199W / 250W |   9777MiB / 11178MiB |     91%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 108...  Off  | 00000000:1C:00.0 Off |                  N/A |
| 29%   36C    P8    10W / 250W |      2MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 108...  Off  | 00000000:1D:00.0 Off |                  N/A |
| 31%   36C    P8     9W / 250W |      2MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX 108...  Off  | 00000000:1E:00.0 Off |                  N/A |
| 35%   41C    P8     9W / 250W |      2MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   4  GeForce GTX 108...  Off  | 00000000:3D:00.0 Off |                  N/A |
| 29%   34C    P8     9W / 250W |      2MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  GeForce GTX 108...  Off  | 00000000:3F:00.0 Off |                  N/A |
| 30%   31C    P8     8W / 250W |      2MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  GeForce GTX 108...  Off  | 00000000:40:00.0 Off |                  N/A |
| 31%   38C    P8     9W / 250W |      2MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   7  GeForce GTX 108...  Off  | 00000000:41:00.0 Off |                  N/A |
| 30%   37C    P8     9W / 250W |      2MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     58780      C   python                          10407MiB |
|    1   N/A  N/A     58780      C   python                              0MiB |
|    2   N/A  N/A     58780      C   python                              0MiB |
|    3   N/A  N/A     58780      C   python                              0MiB |
|    4   N/A  N/A     58780      C   python                              0MiB |
|    5   N/A  N/A     58780      C   python                              0MiB |
|    6   N/A  N/A     58780      C   python                              0MiB |
|    7   N/A  N/A     58780      C   python                              0MiB |
+-----------------------------------------------------------------------------+

FYI: I’m using
pytorch 1. 1.7.0
transformers 4.0.1
cudda 10.1

Context: The problem might be related to the tokenizer rather than the model. The MarianTokenizer does not have a rust (fast) implementation which lay cause a bottleneck, no matter how many GPUs you use. It might be a good idea to preprocess (tokenizer) your dataset once and use datasets for on-the fly superfast access to that cached dataset.


Trainer.hyperparameter_search doesn’t work for me

Question:
HI, as a beginner I try to follow few examples. Looking at this repo " Fine-tuning a model on a text classification task " (here - https://bit.ly/3mEBqTM 3) I tried to implement the trainer.hyperparameter_search() method with either “optuna” and “ray [tune]” .
!pip install optuna --> seems to work fine
When running the example shown I keep getting:
RuntimeError: At least one of optuna or ray should be installed. To install optuna run pip install optuna.To install ray run pip install ray[tune].
I tried installing both but none works… any guidance would be highly appreciated…
thanks, -Ofer

Context: There is some environment problem it seems, since transformers didn’t detect you have optuna installed. I’d try restarting your kernel since you seem to be in a notebook.


How to deploy a fine tuned t5 model in production

Question:
Hi All,
I am trying to deploy a fine-tuned t5 model in production. This is something new to me, to deploy a PyTorch model in production. I went through the presentation from Hugging Face on youtube, about how they deploy the model. And some of the other blog posts.
It is mentioned by HF that they deploy the model on Cython environment as it gives a ~100 times boost to the inference. So, is it always advisable to run a model in production on Cython?
Converting a model in Pytorch to TF does it help and is advisable or not?
What is the preferred container approach to adopt to run multiple models on a set of GPUs?
I know some of these questions would be basic, I apologize for it, but I want to make sure that I follow the correct guidelines to deploy a model in production.
Thank you
Amit

Context:
Hi @as-stevens,
I don’t know what blog post you’re referring to for using Cython to get 100x but I guess it really depends where the bottleneck is.
For t5 models, they are Seq2Seq models, and I would recommend to stick to PyTorch and  finding a way to optimize the hot path (decoder path). TF could work, but transformers currently can’t use various graph optimizations in TF (we’re working on it).
Or you can try to run it on our hosted inference API to alleviate the hassle of managing all the different layers: https://huggingface.co/pricing 33 (Some optimizations are only enabled for customers)
Hope that helps.
Cheers,
Nicolas


What is best way to serve huggingface model with API?

Question:
Use TF or PyTorch?
For PyTorch TorchServe or Pipelines with something like flask?

Context:
You have a few different options, here are some in increasing level of difficulty

You can use the Hugging Face Inference API via Model Hub if you are just looking for a demo.
You can use a hosted model deployment platform: GCP AI predictions, SageMaker, https://modelzoo.dev/ 211. Full disclaimer, I am the developer behind Model Zoo, happy to give you some credits for experimentation.
You can roll your own model server with something like https://fastapi.tiangolo.com/ 252 and deploy it on a generic serving platform like AWS Elastic Beanstalk or Heroku. This is the most flexible option.


Question:
I’m trying to run text classification inference on a XLNet model using onnx, but when trying to run the inference I’m getting the following error:
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:‘Add_26’ Status Message: /onnxruntime_src/onnxruntime/core/providers/cpu/math/element_wise_ops.h:475 void onnxruntime::BroadcastIterator::Init(int64_t, int64_t) axis == 1 || axis == largest was false. Attempting to broadcast an axis by a dimension other than 1. 6 by 48
Any clue about it?
Thanks in advance for your help!

Context:
Hi @uyjco0
Something seems to be running wrong when you’re trying to export to ONNX.
How did you extract your ONNX graph ?
Are you sure about the tensors you’re feeding your session ?
Without a little more information it’s hard to give you more feedback than what google will.
Cheers,


Question:
I’m having all sorts of issues training transformers on my 3090 - this card requires the cuda 11.1 which in turn requires torch 1.7.1
Is this supported? I’m using python 3.7 (I tried 3.9 but there’s no wheel for one of the dependencies for datasets and it wouldn’t build so I rolled back).
I installed pip install torch==1.7.1+cu110  -f https://download.pytorch.org/whl/torch_stable.html
Training of RobertaForMaskedLM frequently crashes with CUDA exceptions. If this should work I’ll create specific issues.

Context:
OK So I tracked down the crash. The problem was the position embedding. I had max_seq_length == max_position_embeddings, this results in a position index > max_position_embeddings for any sequence which is truncated.
This is because create_position_ids_from_input_ids in modeling_roberta.py' below adds pdding_idx to the cumsum - if there are no masked input_ids this will be > max_seq_length`
mask = input_ids.ne(padding_idx).int()
incremental_indices = torch.cumsum(mask, dim=1).type_as(mask) * mask
return incremental_indices.long() + padding_idx


Applying Tapas/TableQuestionAnswering pipelines on a csv via Pandas?

Question:
Hi guys!
Great work on Tapas and the v4.1.1 releaser!
Is there any guidance on how to apply this pipeline 6 to dataframes uploaded via pandas.read_csv?
Thanks,
Charly

Context:
Hello! Here’s how I would setup a pipeline with a pd.DataFrame
from transformers import pipeline
import pandas as pd

tqa_pipeline = pipeline("table-question-answering")

data = {
    "Repository": ["Transformers", "Datasets", "Tokenizers"],
    "Stars": ["36542", "4512", "3934"],
    "Contributors": ["651", "77", "34"],
    "Programming language": ["Python", "Python", "Rust, Python and NodeJS"],
}

queries = "What repository has the largest number of stars?"
table = pd.DataFrame.from_dict(data)

output = tqa_pipeline(table, queries)
# {'answer': 'Transformers', 'coordinates': [(0, 0)], 'cells': ['Transformers']}

If you want to use a CSV file, you also can; here’s the previous example converted to CSV and saved in ~/pipeline.csv:
Repository,Stars,Contributors,Programming language
Transformers,36542,651,Python
Datasets,4512,77,Python
Tokenizers,3934,34,"Rust, Python and NodeJS"

Here’s how I would do (note the type conversion):
from transformers import pipeline
import pandas as pd

tqa_pipeline = pipeline("table-question-answering")

queries = "What repository has the largest number of stars?"
# Convert everything to a string, as the tokenizer can only handle strings
table = pd.read_csv("~/pipeline.csv").astype(str)

output = tqa_pipeline(table, queries)
# {'answer': 'Transformers', 'coordinates': [(0, 0)], 'cells': ['Transformers']}

Hope that helps!