Question:
Hello,
I am new in this forum and Hugging face models. Could someone help with this:
I want to use model ‘Helsinki-NLP/opus-mt-en-sla’. I am using code from the site:
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained(“Helsinki-NLP/opus-mt-en-sla”)
model = AutoModelForSeq2SeqLM.from_pretrained(“Helsinki-NLP/opus-mt-en-sla”)
but I get this error: ValueError: Connection error, and we cannot find the requested files in the cached path. Please try again or make sure your Internet connection is on.
I have installed all necessary libraries and my internet connection is good…could someone help with this?
Thanks

Context:
Hello @Katarina, I was able to run your code snippet without any problems on my machine, so I wonder whether there is some firewall / proxy you need to configure on your end?
A simple test that your connection is fine would be to spin up a Google Colab notebook and see if your code works there. Alternatively, you could try upgrading to the latest version of transformers just to be sure it’s not an old bug that got fixed recently.


Question:
Trying to follow the instructions for training an Roberta-base mlm-model, as described here:

      github.com

    huggingface/transformers 1

  master/examples/flax/language-modeling

  🤗Transformers: State-of-the-art Natural Language Processing for Pytorch, TensorFlow, and JAX.

Everything is easy to follow until the start of training.  Immediately ends in a core dump, with this error message:
ttcmalloc: large alloc 500236124160 bytes == (nil) @  0x7f51b5df3680 0x7f51b5e13ff4 0x7f51b590a309 0x7f51b590bfb9 0x7f51b590c056 0x7f4e5cc6a659 0x7f4e526a0954 0x7f51b5fe7b8a 0x7f51b5fe7c91 0x7f51b5d46915 0x7f51b5fec0bf 0x7f51b5d468b8 0x7f51b5feb5fa 0x7f51b5bbb34c 0x7f51b5d468b8 0x7f51b5d46983 0x7f51b5bbbb59 0x7f51b5bbb3da 0x67299f 0x682dcb 0x684321 0x5c3cb0 0x5f257d 0x56fcb6 0x56822a 0x5f6033 0x56ef97 0x5f5e56 0x56a136 0x5f5e56 0x569f5e
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
https://symbolize.stripped_domain/r/?trace=7f51b5c2918b,7f51b5c2920f&map=
*** SIGABRT received by PID 14088 (TID 14088) on cpu 95 from PID 14088; stack trace: ***
PC: @     0x7f51b5c2918b  (unknown)  raise
    @     0x7f4f86e6d800        976  (unknown)
    @     0x7f51b5c29210  (unknown)  (unknown)
https://symbolize.stripped_domain/r/?trace=7f51b5c2918b,7f4f86e6d7ff,7f51b5c2920f&map=2a762cd764e70bc90ae4c7f9747c08d7:7f4f79f2b000-7f4f871ac280
E0628 16:55:18.669807   14088 coredump_hook.cc:292] RAW: Remote crash data gathering hook invoked.
E0628 16:55:18.669833   14088 coredump_hook.cc:384] RAW: Skipping coredump since rlimit was 0 at process start.
E0628 16:55:18.669843   14088 client.cc:222] RAW: Coroner client retries enabled (b/136286901), will retry for up to 30 sec.
E0628 16:55:18.669852   14088 coredump_hook.cc:447] RAW: Sending fingerprint to remote end.
E0628 16:55:18.669864   14088 coredump_socket.cc:124] RAW: Stat failed errno=2 on socket /var/google/services/logmanagerd/remote_coredump.socket
E0628 16:55:18.669874   14088 coredump_hook.cc:451] RAW: Cannot send fingerprint to Coroner: [NOT_FOUND] Missing crash reporting socket. Is the listener running?
E0628 16:55:18.669881   14088 coredump_hook.cc:525] RAW: Discarding core.
E0628 16:55:18.673655   14088 process_state.cc:771] RAW: Raising signal 6 with default behavior
Aborted (core dumped)

Any ideas about what is causing this?

Context: OOM for TPU cores; try with smaller batch sizes (1 for starters) or reduce data/model size


[HELP]Bart summarization output exactly the same as labels

Question:
Hii, I’m trying to finetune BART on summariztion task using Tensorflow TPU. I first tokenized the data, stored them in *.tfrecords using the  datasets export() function, then, created TF Datasets using them, I have given the preprocessing and finetuning code below.
Problem: I am getting exact copy of labels as outputs. Its like BART is not learning anything.
eg:
['The ICSI Meeting Recorder Dialog Act (MRDA) Corpus\nWe describe a new corpus of over 180,000 handannotated dialog act tags and accompanying adjacency pair annotations for roughly 72 hours of speech from 75 naturally-occurring meetings.\nWe provide a brief summary of the annotation system and labeling procedure, inter-annotator reliability statistics, overall distributional statistics, a description of auxiliary files distributed with the corpus, and information on how to obtain the data.', 'Templates-Based Information Extraction without the Templates\nStandard algorithms for template-based information extraction (IE) require predefined template schemas, and often labeled data, to learn to extract slot fillers (e.g., a template).\nThis paper describes an approach to template- based IE that removes this requirement and performs extraction without knowing the template structure in advance.\nOur algorithm instead learns the template structures automatically from raw text, inducing template Schema schemas as sets of linked events associated with semantic roles.\nWe also solve', 'Get Out The Vote: Determining Support Or Opposition From Congressional Floor-Debate Transcripts\nWe investigate whether one can determine from the transcripts of U.S. Congressional floor debates whether the speeches represent support of or opposition to proposed legislation.\nTo address this problem, we exploit the fact that these speeches occur as part of a discussion; this allows us to use sources of information regarding relationships between discourse segments, such as whether a given utterance indicates agreement with the opinion expressed by another.\nWe find that the incorporation of such information yields substantial improvements over classifying speeches']
['The ICSI Meeting Recorder Dialog Act (MRDA) Corpus\nWe describe a new corpus of over 180,000 hand-annotated dialog act tags and accompanying adjacency pair annotations for roughly 72 hours of speech from 75 naturally-occurring meetings.\nWe provide a brief summary of the annotation system and labeling procedure, inter-annotator reliability statistics, overall distributional statistics, a description of auxiliary files distributed with the corpus, and information on how to obtain the data.', 'Template-Based Information Extraction without the Templates\nStandard algorithms for template-based information extraction (IE) require predefined template schemas, and often labeled data, to learn to extract their slot fillers (e.g., an embassy is the Target of a Bombing template).\nThis paper describes an approach to template-based IE that removes this requirement and performs extraction without knowing the template structure in advance.\nOur algorithm instead learns the template structure automatically from raw text, inducing template schemas as sets of linked events (e.g., bombings include detonate, set off, and destroy events) associated with semantic', 'Get Out The Vote: Determining Support Or Opposition From Congressional Floor-Debate Transcripts\nWe investigate whether one can determine from the transcripts of U.S. Congressional floor debates whether the speeches represent support of or opposition to proposed legislation.\nTo address this problem, we exploit the fact that these speeches occur as part of a discussion; this allows us to use sources of information regarding relationships between discourse segments, such as whether a given utterance indicates agreement with the opinion expressed by another.\nWe find that the incorporation of such information yields substantial improvements over classifying speeches in isolation.\nWe present a method based on support']
Validation results:---
 {'rouge1': 78.3223, 'rouge2': 72.4416, 'rougeL': 76.0222, 'rougeLsum': 77.904}

I am using facebook/bart-large-cnn as my checkpoint. At first, I was getting gibberish output. So, I passed in BartConfig.from_pretrained("facebook/bart-large-cnn") to the model, and it started copying the labels. I searched on the forum and found this thread, @valhalla suggested using prepare_seq2seq_batch() at that time. Now, since it is going to be deprecated in transformers 5.x, and the suggested way is to use tokenizer.as_target_tokenizer(), I did that.
The code:
class Config:
    num_epochs=3
    train_batch_size=2
    val_batch_size=4
    test_batch_size=4
    learning_rate=2e-5
    num_warmup_steps=0
    num_beams=4
    max_input_length=1024
    max_target_length=128
    val_max_target_length=None
    ignore_pad_token_for_loss=True
    padding="max_length"
    train_data_len=None
    valid_data_len=None
    test_data_len=None
    num_val_take=6
    num_test_take=6
    num_val_examples=num_val_take * val_batch_size * REPLICAS
    num_test_examples=num_test_take * test_batch_size * REPLICAS

def read_tfrecord(example, max_input_length, max_target_length):
    feature_description = {
        'input_ids': tf.io.FixedLenFeature([max_input_length], tf.int64, default_value=[0]*max_input_length),
        'attention_mask': tf.io.FixedLenFeature([max_input_length], tf.int64, default_value=[0]*max_input_length),
        'decoder_input_ids': tf.io.FixedLenFeature([max_target_length], tf.int64, default_value=[0]*max_target_length),
        'decoder_attention_mask': tf.io.FixedLenFeature([max_target_length], tf.int64, default_value=[0]*max_target_length),
        'labels': tf.io.FixedLenFeature([max_target_length], tf.int64, default_value=[0]*max_target_length),
    }

    example = tf.io.parse_single_example(example, feature_description)
    return example, example["labels"]

def preprocess_function(examples, article_column="article", summary_column="target",
                        max_input_length=1024, max_output_length=128, prefix="summarize: "):

    inputs = [prefix + article for article in examples[article_column]]

    tokenized_inputs = tokenizer(inputs, max_length=max_input_length, padding="max_length", truncation=True)

    with tokenizer.as_target_tokenizer():
        tokenized_outputs = tokenizer(examples[summary_column], max_length=max_output_length, padding="max_length", truncation=True)

    return {"input_ids": tokenized_inputs["input_ids"],
            "attention_mask": tokenized_inputs["attention_mask"],
            "decoder_input_ids": tokenized_outputs["input_ids"],
            "decoder_attention_mask": tokenized_outputs["attention_mask"],
            "labels": tokenized_outputs["input_ids"]}

model_config = AutoConfig.from_pretrained("facebook/bart-large-cnn")
tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)
with strategy.scope():
     model = TFAutoModelForSeq2SeqLM.from_pretrained(model_checkpoint, config=model_config)
     model.compile(optimizer=optimizer,
              loss={"logits": masked_sparse_categorical_crossentropy})

metric = load_metric("rouge")

def postprocess_text(preds, labels):
     preds = [pred.strip() for pred in preds]
     labels = [label.strip() for label in labels]

     # rougeLSum expects newline after each sentence
     preds = ["\n".join(nltk.sent_tokenize(pred)) for pred in preds]
     labels = ["\n".join(nltk.sent_tokenize(label)) for label in labels]

     return preds, labels

def eval_fn(model, tokenizer,
            tokenized_tf_dataset,
            tf_dataset_take_size,
            pre_train_val_check=False):

            if Config.val_max_target_length is None:
                Config.val_max_target_length = Config.max_target_length

            gen_kwargs = {
                "max_length": Config.val_max_target_length,
                "num_beams": Config.num_beams,
            }

            if pre_train_val_check: # Checks Validation Loop before starting fine-tuning
                tokenized_tf_dataset = tokenized_tf_dataset.take(2)
                total = 2
            else:
                total = tf_dataset_take_size

            decoded_labels = None
            decoded_pred = None

            for batch, labels in tqdm(
                    tokenized_tf_dataset,
                    total=total,
                    unit="batchs"
                ):
                    temp_batch = {
                        "input_ids": batch["input_ids"],
                        #"attention_mask": batch["attention_mask"],
                    }
                    temp_batch.update(gen_kwargs)
                    generated_tokens = model.generate(**temp_batch)
                    if isinstance(generated_tokens, tuple):
                        generated_tokens = generated_tokens[0]
                    decoded_preds = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
                    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
                    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)
                    decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

                    metric.add_batch(predictions=decoded_preds, references=decoded_labels)

            result = metric.compute(use_stemmer=True)
            # Extract a few results from ROUGE
            result = {key: value.mid.fmeasure * 100 for key, value in result.items()}

            result = {k: round(v, 4) for k, v in result.items()}

            if pre_train_val_check:
                result_integrity = np.array([False if v == 0 else True for k, v in result.items()])
                if False in result_integrity:
                    print("Result: ", result)
                    print("Result integrity failed")
                    return False
                else:
                    print(result)
                    print("Valiation Epoch working correctly....")
                    return True
            else:
                return result

Thanks

Context: try increasing the epochs maybe that will help I don’t have any experience with this model but every time the model doesn’t predict right increasing the epochs almost always helps


Question:
Hi,
I pass a test dataset to trainer.predict but I have many samples. Therefore, I get a memory error. Does the library support a way of batch based trainer.predict? or do I have to implement it myself?

Context: You can pass eval_accumulation_steps=xxx to pass the predictions to the CPU every xxx steps, this should help.


Num_labels creates an error for some models

Question:
Hello,
I am training a classifier for three classes (bad, medium, good) using distilbert-base-uncased-finetuned-sst-2-english but after saving the model to disk and re-loading it I am getting a strange error
from transformers import AutoTokenizer, TFAutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")
model = TFAutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")

tokenizer.save_pretrained('finetuned')
model.save_pretrained('finetuned')
model.from_pretrained('finetuned', num_labels = 3)

ValueError: cannot reshape array of size 1536 into shape (768,3)

This approach worked with bert-base-uncased. Am I doing something wrong here?
Thanks!

Context:
With the latest version installed, you need to add ignore_mismatched_sizes=True to your from_pretrained call for this to work.
Otherwise, you try to load a model with 2 labels inside a model with 3 labels, and you get mismatched sizes like that.


Transformers suddenly complaining about pytorch?

Question:
Hello the team,
Huggingface is suddenly complaining about pytorch … but I have been only using tensorflow all along so far! Do you know what could cause this issue?
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")

model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")
ImportError:
AutoModelForSequenceClassification requires the PyTorch library but it was not found in your environment. Checkout the instructions on the
installation page: https://pytorch.org/get-started/locally/ and follow the ones that match your environment.

Thanks!

Context: If you want to use tensorflow models, try TFAutoModelForSequenceClassification module.


Duplicate hyperparameter tuning trials with Ray and `hyperparameter_search`

Question:
I’m using hyperparameter_search for hyperparamter tuning with the following configurations:
    def tune_config_ray(trial):
        return {"learning_rate": tune.choice([5e-5, 4e-5, 3e-5, 2e-5]),
                "num_train_epochs": tune.choice([4]),
                "per_device_train_batch_size": tune.choice([16])
               }

    best_trial = trainer.hyperparameter_search(hp_space=tune_config_ray,
                                               backend='ray',
                                               direction='maximize',
                                               n_trials=4,
                                               )

Based on the config, there are 4 unique combinations for the learning_rate, num_train_epochs, and per_device_train_batch_size. However, when I run the tuning (as you can see below), I see some duplicates among the trials. I wonder why this happens and how I can have non-duplicate trials?  Is this possibly because ray is also tuning some other hyperparameters that are not listed in the report and that is why I see some duplicates?
+------------------------+----------+-------+-----------------+--------------------+-------------------------------+
| Trial name             | status   | loc   |   learning_rate |   num_train_epochs |   per_device_train_batch_size |
|------------------------+----------+-------+-----------------+--------------------+-------------------------------|
| _objective_7efa7_00000 | RUNNING  |       |           3e-05 |                  4 |                            16 |
| _objective_7efa7_00001 | PENDING  |       |           2e-05 |                  4 |                            16 |
| _objective_7efa7_00002 | PENDING  |       |           5e-05 |                  4 |                            16 |
| _objective_7efa7_00003 | PENDING  |       |           3e-05 |                  4 |                            16 |
+------------------------+----------+-------+-----------------+--------------------+-------------------------------+

Context: cc @richardliaw


Question:
from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="/gdrive/MyDrive/Thesis/GPT2/checkpoints",
    overwrite_output_dir=False,
    num_train_epochs=5,
    per_device_train_batch_size=6, #previous was 6
    save_steps=100,
    save_total_limit=5,
    fp16 = True,
    dataloader_drop_last=True,
    #evaluate_during_training=True,
    warmup_steps=200
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
    # prediction_loss_only = True
)

trainer.train()

I want to save the checkpoints directly to my google drive. The problem is the code above saves my checkpoints upto to save limit all well. But after the limit it can’t delete or save any new checkpoints. Although it says checkpoints saved/deleted in the console. Any help?

Context: I think just wait a bit and they appear up in drive , that’s what happened with me


How to save the best trial’s model using `trainer.hyperparameter_search`

Question:
I’m using hyperparameter_search for hyperparameter tuning in the following way:
trainer = Trainer(
            model_init=model_init,
            args=training_args,
            train_dataset=train_set,
            eval_dataset=dev_set,
            tokenizer=tokenizer,
            compute_metrics=compute_metrics,
)

best_trial = trainer.hyperparameter_search(
            backend="ray",
            direction='maximize',
            n_trials=10,
       )

Everything’s working well and I can see the information for the best trial in the best_trial. However, my question is how can I save the actual best model from the best trial? I tried saving the model using the trainer’s save_model like trainer.save_model(path/to/a/folder), but I get the following error:
trainer.save_model(path/to/a/folder)
  File "/home/ubuntu/anaconda3/envs/ccr/lib/python3.6/site-packages/transformers/trainer.py", line 1885, in save_model
    self._save(output_dir)
  File "/home/ubuntu/anaconda3/envs/ccr/lib/python3.6/site-packages/transformers/trainer.py", line 1930, in _save
    state_dict = self.model.state_dict()
AttributeError: 'NoneType' object has no attribute 'state_dict'

It looks like the trainer does not have the actual best model found as a result of hyperparameter tuning (?). My goal is simple, I basically want to use the best model from hyperparameter tuning to evaluate it on my final test set. But I can’t find a way to save the best model from hyperparameter tuning. Also, someone may say I can get the info from the best trial and fine-tune the model again, but I don’t want to do that and I just simply want to get the model from the hyperparameter tuning. Is there any way to do that? Thanks.

Context: @sgugger Any thoughts on this?


Need help on wav2vec 2.0 models training

Question:
Hello guys, I’m using transformers library and I want to build speech recognition systems base on wav2vec 2.0
I have different problems

Base on the exemple here Wav2Vec2 — transformers 4.7.0 documentation 2
I have tried to pretrain a model. But I have seen in the github on fairseq wav2vec that the loss object  at the end, I can call a backward method on, but here I can.

how can I perform batch training in wav2vec.

Guys please I need help, even another documentation or examples.

Context:
https://github.com/huggingface/blog/blob/master/fine-tune-wav2vec2-english.md 12
try this


Custom SQuAD2.0 dataset gives an error when using run_qa.py script

Question:
Hello,
I am trying to follow the PyTorch Question Answering example 4. However, when running the run_qa.py script using my own (Dutch machine-translated) SQuAD train and test files (JSON), I get the following error: pyarrow.lib.ArrowInvalid: cannot mix list and non-list, non-null values.
I use the following hyperparameters:
python run_qa.py \
--model_name_or_path GroNLP/bert-base-dutch-cased \
--version_2_with_negative \
--do_train \
--do_eval \
--train_file "C:\Users\myname\data\squad\nl_squad_train_clean.json" \
--test_file "C:\Users\myname\data\squad\nl_squad_dev_clean.json" \
--per_device_train_batch_size 12 \
--num_train_epochs 2 \
--max_seq_length 384 \
--doc_stride 128 \
--save_steps=800 \
--output_dir ../output

When replacing the train and test by --dataset_name squad it works fine. What could be the problem with my own SQuAD files?
Thanks in advance! Cheers!

Context:
hey @julifelipe, my guess is that you have your squad data in the conventional json format, while the run_qa.py expects the examples to be line-delimited json
you can find a simple function to do the conversion here: Question answering bot: fine-tuning with custom dataset - #2 by lewtun 4


Tokenizer from tokenizers library cannot be used in transformers.Trainer

Question:
Hi,
I am trying to train my own model with Trainer with a pre-trained SentencePieceBPETokenizer from tokenizers library. However, it is missing several attributes as well as methods (e.g., pad ), which makes it incompatible with transformers.Trainer . Is there an easy way to convert it to PretrainedTokenizer from transformers ?
Thanks!

Context: If you want SentencePiece Tokenizer, you should use the sentencepiece library, then pass in the trained model as a parameter into the desired tokenizer model like T5, Bart etc. By doing this the vocab will be yours and the desired tokenizer will handle the padding, I am not sure about whether it will handle the special tokens though.


Feed output from one transformer model as input to another

Question:
So I am trying to train an automated essay scoring system, that combines the loss of predicting scores with predicting whether a sentence is grammatically correct. To do this I have split each sentence in the essay with a sep and a cls token so that an essay is fed into Bert like this:
essay 1 : [cls] … sent 1 … [sep][cls] … sent 2 … [sep][cls] … sent 3 … [sep][cls] … etc
essay 2 : [cls] … sent 1 … [sep][cls] … sent 2 … [sep][cls] … sent 3 … [sep][cls] … etc
As well as a list of labels for each sentence whether it contains a grammatical error or not and a score for the essay i.e
essay 1 : labels: [1,0,0,1,etc…],score:38
essay 2 : labels: [1,1,0,1,etc…],score:24
(So that the list of labels can be passed into a dataset they must have the same length as the input_ids, attention_masks etc. so I have padded them with 2’s at each word that is not a cls token, so they actually look something like this…
[1,2,2,0,2,2,2,0,2,2,2,2,1,etc…]
which would correspond to a sent
[cls,w,w,cls,w,w,w,cls,w,w,w,w,cls…]
(w=word)
I then use this to get the index of the cls tokens in each essay.
So the performance of the model on the grammatical error detection is comparable to feeding in a sentence normally to bert:
sent 1 :  [cls] … sent 1 … [sep]
sent2 :   [cls] … sent 2 … [sep]
However, unsurprisingly the additional sep and cls tokens decrease the model performance when compared to feeding an essay into Bert normally:
essay 1 :  [cls] … essay 1 … [sep]
essay 2 :  [cls] … essay 2 … [sep]
So to combat this I am trying to use the vector representation of each cls token in the output of each essay as input to another smaller transformer model as done in this paper here https://arxiv.org/pdf/1903.10318.pdf). However I cannot figure out how to do this, I have tried feeding the output into a embedding layer with input_embeds = True but this does not work.
My is a simplified version of my code so far with just a mini batch I have used as a test:
# encoded_dataset_train is my training dataset type = (datasets.arrow_dataset.Dataset)
mini_batch = encoded_dataset_train[:2]
print(mini_batch)
'''{'attention_mask': tensor([[1, 1, 1,  ..., 0, 0, 0],
         [1, 1, 1,  ..., 0, 0, 0]]),
 'input_ids': tensor([[    0, 23314,  5348,  ...,     1,     1,     1],
         [    0,  1360,    73,  ...,     1,     1,     1]]),
 'label': tensor([[1, 2, 2,  ..., 2, 2, 2],
         [1, 2, 2,  ..., 2, 2, 2]]),
 'score': tensor([31, 23])}'''

from transformers import AutoModel
model = AutoModel.from_pretrained('distilroberta-base')

# get the output of the final attention layer of the model
input_ids = mini_batch['input_ids']
attention_mask = mini_batch['attention_mask']
output = model(input_ids=input_ids,attention_mask=attention_mask)

print(output.shape)
'''
torch.Size([2, 512, 768])
'''

# get a mask where each cls token in each batch is 1 and 0 for all other tokens
bs,tok_len = mini_batch['label'].shape

active_labels = torch.where(mini_batch['label']<2,1,0).reshape(bs,tok_len,1)expand(active_labels.shape[0],active_labels.shape[1],768)

print(active_labels.shape)
'''
torch.Size([2, 512, 768])
'''

# multiply the output by the mask
active_loss = output*active_labels

print(active_loss.shape)
'''
torch.Size([2, 512, 768])
'''

# Create my smaller model (so far only extracted the layers, I want have not combined them yet)

embeds = model.embeddings
layers = model.encoder.layer[:2]
classifier = model.classifier

# Result for trying to pass the output into the embedding layer

embeds.forward(input_ids=active_loss,inputs_embeds=True)

'''
RuntimeError: The size of tensor a (512) must match the size of tensor b (768) at non-singleton dimension 2
'''

Cheers in advance

Context: Is this possible or do I have to perform pooling at each sentence vector to get single value to represent that sentence and use that as an input?


How to monitor both train and validation metrics at the same step?

Question:
Hi,
I am finetuning BertForSequenceClassification using run_glue.py and I would like to output every logging_steps all the performance metrics of my model.
Currently, in the logs I see entries like
{'loss': 0.1867809295654297, 'learning_rate': 1.3071895424836603e-07, 'epoch': 2.980392156862745, 'total_flos': 2121344853980160, 'step': 456}
for the training loss and
{'eval_loss': 0.4489470714636048, 'eval_mcc': 0.6251852674757565, 'epoch': 3.0, 'total_flos': 2133905557962240, 'step': 459}
for the evaluation loss.
They are being printed separately, with validation losses in output only at the end of the run.
Where is the actual logging taking place in trainer.py? I’d like to know that so that I can output a single dictionary containing all the metrics.
I am using transformers 3.3.0 and run_glue.py with the flag --evaluation_strategy steps, setting low values =32 for both --logging_steps and --eval_steps. I am confused because evaluation using the validation set doesn’t seem to occur every eval_steps.
I revised Trainer doesn't show the loss at each step 23 but I am still not sure about how to do this.

Context:
Hi @davidefiocco
logging_steps and eval_steps have different meaning,
logging_steps will only log the train loss , lr, epoch etc info and not the metrics,
eval_steps logs the metrics on valid set.
Here the steps refer to actual optimization  steps , so if you are using 2 grad accumulation steps and your BS is 4 then 1 optimization step is equal to 8 trainer steps, so in this case if your eval_steps is 2 then they will logged at trainer step 16.
In the latest version if  eval_steps are not specified, it’ll be set logging_steps by default
logging is done in the log method here 74
and it’s invoked here 37 and here 24 in the train method.
Hope this helps.


What does “generate_with_predict=True” actually do?

Question:
Hello, I am currently trying to finetuning T5 for summarization task using PyTorch/XLA, and I want to know what is the purpose of generate_with_predict. I saw the documentation and know its supposed to be used with ROUGE/BLEU. But, I am confused what it actually does. If I give generate_with_predict=True, then, will the output be decoded on its own and the metric will be directly calculated if I pass it like given below:
from datasets import load_metric

rouge_metric = load_metric("rouge")
                    .
                    .
                    .
trainer = Seq2SeqTrainer(
        model=model,
        args=training_args,
        compute_metrics=rouge_metric,
        train_dataset=train_dataset,
        eval_dataset=dev_dataset,
        data_collator=data_collator,
    )

Or, Do I have to wrap rouge inside another function like below, and then pass it:
def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    decoded_preds = tokenizer.batch_decode(predictions, skip_special_tokens=True)
    # Replace -100 in the labels as we can't decode them.
    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

    result = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)
    # Extract a few results
    result = {key: value.mid.fmeasure * 100 for key, value in result.items()}

    return {k: round(v, 4) for k, v in result.items()}

PS: While using wrapper method, colab/kaggle crashes due to RAM usage exceeding 100%. Even when I use TPU.

Context: Look at the example notebook 12 or the example script 9 for summarization. You will see you have to pass along the latter.


How to make `pipeline` automatically scale?

Question:
Hi, I have trained a model for text classification. When I load it using the transformers’ pipeline, it just works well. The problem comes when I give as input a very big list of sentences: I get a CUDA out of memory error. When I take each example one by one in a for loop, I don’t get this error.
Is there an option to pass when instantiating the pipeline() object that enables to make predictions on a very large sequence automatically (for example by setting a batch size and iterating through the batches)? Or do I have to code this myself?
@sgugger
Thanks.

Context: No you will have to code one yourself, the pipeline API is not designed to handle a large number of inputs automatically.


Generate logits from hidden state embeddings and decoder weights

Question:
Hi,
I am trying to compute prediction_logits using BertForPreTraining model. For some reason, I don’t want to use outputs.prediction_logits and I want to be able to generate them by multiplying the last hidden state with decoder weights. The problem is that when I do this the results I get are not equal to outputs.prediction_logits. Here is the code:
model = BertForPreTraining.from_pretrained("bert-base-multilingual-cased", output_hidden_states=True).to(device)

w = model.state_dict()['cls.predictions.decoder.weight'].cpu().numpy()
b = model.state_dict()['cls.predictions.decoder.bias'].cpu().numpy()

with torch.no_grad():
    outputs = model(**inputs)

output_logits = outputs.prediction_logits.cpu().numpy()
last_hidden_states = outputs.hidden_states[-1].cpu().numpy()

preds = output_logits[i, token_idx]
h = last_hidden_states[i, token_idx]
h_transformed = np.dot(w, h) + b

Basically, I expect h_transformed to be equal to preds, but it is not.
Thanks for your help

Context: I suspect dropout might be to blame here. You can create a small fully connected layer with dropout, and  initialize it with decoder weights to use instead.


Question:
Hello,
Thank you for sharing such nice model again on this framework.
I am trying to finetune a wav2vec2 model on a custom dataset (so not from the dataset package of huggingface). I have tried to follow these two tutorials :

Fine-Tune Wav2Vec2 for English ASR in Hugging Face with 🤗 Transformers 12

Fine-tuning with custom datasets — transformers 4.7.0 documentation 20
but I did not find how to use multiprocessing when using trainer on a custom dataset ? Should I use dataloader class from torch ? For now I am using the regular Dataset class from torch.

I also encoutered memory issue with the GPU (16 gb) with a base Wav2vec2 model, even with batch size = 1. What is the maximum batch size for a base and large model for 16 gb, and with what length of sample ? (using fp16).
I thank you for the help

Context:
You can convert your custom dataset if its a dataframe into Hugging Face format using this.
from datasets import Dataset, load_metric

train_data = Dataset.from_pandas(train_df)
test_data = Dataset.from_pandas(test_df)

If you are facing memory related problem, set num_proc = 1
It solved my problem.


How to customize dataloader creation in trainer?

Question: Is there a way I could re-create the dataloader after each epoch through a custom function when using trainer?

Context: No, you should probably use a manual training loop for that, using the Accelerate library.


Question: Does anyone have an end-to-end example of how to do multi-gpu, multi-node distributed training using the trainer? I can’t seem to find one anywhere.

Context: All the examples using the Trainer run in multi-gpu multi-node, you just have to use the PyTorch launcher to properly launch a multi-GPU multinode training.


Question:
Creating generative chatbot with BertGenerationEncoder and BertGenerationDecoder like:
encoder = BertGenerationEncoder.from_pretrained("bert-large-uncased", bos_token_id=101, eos_token_id=102)
# add cross attention layers and use BERT's cls token as BOS token and sep token as EOS token
decoder = BertGenerationDecoder.from_pretrained("bert-large-uncased", add_cross_attention=True, is_decoder=True, bos_token_id=101, eos_token_id=102)
bert2bert = EncoderDecoderModel(encoder=encoder, decoder=decoder)
bert2bert.to(device)
# create tokenizer...
tokenizer = BertTokenizer.from_pretrained("bert-large-uncased")

and training the model like:
progress_bar = tqdm(range(num_training_steps))

bert2bert.train()

for epch in range(num_epochs):
  for i in range(len(query)):
    input_ids = tokenizer(query[i], add_special_tokens=False, return_tensors="pt").input_ids
    labels = tokenizer(response[i], add_special_tokens = False, return_tensors="pt").input_ids
    loss = bert2bert(input_ids=input_ids.to(device), decoder_input_ids=labels.to(device), labels=labels.to(device)).loss
    loss.backward()
    optimizer.step()
    lr_scheduler.step()
    optimizer.zero_grad()
    progress_bar.update(1)
    progress_bar.set_postfix_str(f'Loss: {loss.item():.5f}')

Here query = [‘Hi’, ‘How are you’]
response = [‘Hello’, ‘I’m good’]
After training the model, getting strange generations like:
input_ids = tokenizer('Hi', add_special_tokens=False, return_tensors="pt").input_ids

#bert2bert.eval()
outputs = bert2bert.generate(input_ids.to(device))

print(tokenizer.decode(outputs[0]))

[CLS] is is is is is is is is is is is is is is is is is is is
Kindly suggest where is the flaw.
Thanks

Context: I’m not particularly familiar with BERT-based generation models but in general, .generate(input_ids, do_sample=True) is an easy way to diversify the output of the model. Bear in mind that the output text can still lack coherence but it will less likely be so repetitive. You can also experiment with the repetition_penalty argument of the generate method as explained here


How to convert wav2vec2 checkpoint to Huggingface processor and model?

Question:
Hi, I have a finetuned checkpoint.pt, which I trained using fairseq repo’s CLI tools. Now I want to infer this model with Huggingface Transformers library. Like this example from here 7 -
processor = Wav2Vec2Processor.from_pretrained("facebook/wav2vec2-base-960h")
model = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-base-960h")

How can I do that? Any help appreciated. Thanks.

Context: did u figure this out ? i am stuck at the same problem


How to measure accuracy while fine-tuning bert-base model?

Question:
Hello everybody,
While I am fine-tunning ‘dbmdz/bert-base-turkish-uncased model’, I can see the loss value during training the model as below:
outputs = model(b_input_ids, attention_mask=b_input_mask, labels=b_labels)
loss = outputs[0]  #  get loss

but I could not measure accuracy value during training the model.
any suggestion, please?

Context:
I’m not sure if it’s possible to print metrics with that object, but with model.fit it automatically displays the metrics as in TensorFlow.
Check out the course page 14
However, I don’t know if it’s possible to include the attention layers here as you do in your example.
model.fit(
    x=training['train']['input_ids'],
    y=training['labels_train'],
    validation_data=(
        validation['valid']['input_ids'],
        validation['labels_test'],
    )
    ,batch_size = batch_size
    ,epochs = epochs
)


Question:
Was anyone able to fine-tune GPT2-XL (or similar models with similar sizes) on COLAB with Deepspeed enabled?
I tried it with a V100 and 25GB RAM instance, w/ and w/out cpu offloading and fp16 with a batch size of 1 but it is still giving OOM on the GPU side and CPU side.

Context: I’m encountering the same problem with gpt-neo. Have you found any solutions?


Train GPT2 from scratch (Tensorflow) - Loss function

Question:
Hello.
I’m trying to train a GPT2 model (actually GPT2LMHeadModel) using tensorflow2.
I found this post 35 where the author shows how to do it with great detail. However, there’s a special aspect regarding the definition of the loss function when compiling the model.
Initially, when I was trying to implement it on my own, I defined the loss function as usual for the Keras models:
model.compile(optimizer=optimizer,  loss=loss_fn, metrics=[metric])

However, when trying to execute the fit method, it threw an error:
history = model.fit(dataset, epochs=EPOCHS)

ValueError: Shape mismatch: The shape of labels (received (490,)) should equal the shape of logits except for the last dimension (received (11760, 64))

In this case, the dateset corresponds to a tensorflow dataset, which shape is <TakeDataset shapes: ((10, 49), (10, 49)), types: (tf.int32, tf.int32)> (Batch size: 10, sequences length: 49).
After that, I realized the loss function parameter is defined differently in the tutorial mentioned above. The loss parameter corresponds to a list, which actually happens to be a function followed by a collection of None values, depending on the number of layers defined for the model architecture. Under this approach, it works fine.

image822×150 12.7 KB

In both cases, the loss function corresponds to tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True).
The documentation for TFGPT2LMHeadModel specifies: “The GPT2 Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings).”. So, I think it makes sense to define the loss as that list-based parameter (Basically the loss function would work for the top layer).
After seeing this, I have some questions:

Is it correct to define the loss function that way?
Does it have any implications for the inference process?

Additionally, I would like to mention that I’ve also tried to train the model using the Trainer class, unfortunately, it throws a similar error when running the train method.
 training_args = TFTrainingArguments(
output_dir='./results',          # output directory
num_train_epochs=EPOCHS,         # total # of training epochs
per_device_train_batch_size=10,  # batch size per device during training
per_device_eval_batch_size=10,   # batch size for evaluation
warmup_steps=500,                # number of warmup steps for learning rate scheduler
weight_decay=0.01,               # strength of weight decay
logging_dir='./logs',            # directory for storing logs)

trainer = TFTrainer(model=model, args=training_args,  train_dataset=dataset,   eval_dataset=dataset)
trainer.train()

ValueError: Shapes (248040,) and (4410,) are incompatible

I’m using transformers v 3.5.0 and tokenizers v 0.9.3
Sorry for the long post, thanks in advance for your help!

Context: I am having the same question, hope someone could answer us


Question:
I am trying to get the key and query vectors out of the Transformer layers but am confused by the docs regarding the embedding tensors provided for each layer. I have hidden_states=True. And the docs say:

hidden_states ( tuple(torch.FloatTensor) , optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True ) –
Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size) .
Hidden-states of the model at the output of each layer plus the initial embedding outputs.

Where are these embeddings coming from in each layer? At first I thought the embedding was from the token embedding layer as the very first input to the model but the embeddings across layers are not the same and it would not make sense to append this to the tuple for every layer. So where are these embeddings coming from?
In addition are they at the 0 index of the tuple? The docs say " output of the embeddings + one for the output" implying index 0 but then say “model at the output of each layer plus the initial embedding outputs.” implying index 1.
Bonus question: how can I get out the query vectors? If I have the keys and attentions then I think I can work out what the query vector is but is there an easier way?
Thank you,
Trenton

Context:
What it means is not “embeddings for each layer” but “output of embeddings” + “outputs of each layer”. So for a 12 layer model, you’d get the embedding output (1 tensor) + the outputs of the 12 following layers (12) = 13 tensors in total.
You can get the attention values back (output_attentions=True), but AFAIK not the query vectors.


ValueError: `mask_length` has to be smaller than `sequence_length`, while finetuning Wav2vec2.0

Question:
As in the title, I’m getting ValueError saying that mask length is longer than sequence length while finetuning wav2vec2 models. So far, I’ve tried wav2vec2-base and wav2vec2-large-xlsr-53 and the same error occurred for both of them.
I am getting around this error by filtering out examples short than a certain length. It seemed the error is coming from the model predicting some sequences to be shorter than a mask length and failing in the forward pass of the training loop.
Is this the desired behavior for wav2vec? It seems reasonable in some sense as (english) words can’t be too short. However, because of this, I’m dropping >50% of my data.
Did anyone had the same issue? If so, how did you get around this error? Below is the full traceback of the error:
    Traceback (most recent call last):
    File "/home/admin/projects/voice-assessment/wav2vec/kr/run.py", line 165, in <module>
        trainer.train()
    File "/home/admin/.miniconda3/envs/voice/lib/python3.9/site-packages/transformers/trainer.py", line 1269, in train
        tr_loss += self.training_step(model, inputs)
    File "/home/admin/.miniconda3/envs/voice/lib/python3.9/site-packages/transformers/trainer.py", line 1760, in training_step
        loss = self.compute_loss(model, inputs)
    File "/home/admin/.miniconda3/envs/voice/lib/python3.9/site-packages/transformers/trainer.py", line 1794, in compute_loss
        outputs = model(**inputs)
    File "/home/admin/.miniconda3/envs/voice/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
        return forward_call(*input, **kwargs)
    File "/home/admin/.miniconda3/envs/voice/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 168, in forward
        outputs = self.parallel_apply(replicas, inputs, kwargs)
    File "/home/admin/.miniconda3/envs/voice/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 178, in parallel_apply
        return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
    File "/home/admin/.miniconda3/envs/voice/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
        output.reraise()
    File "/home/admin/.miniconda3/envs/voice/lib/python3.9/site-packages/torch/_utils.py", line 425, in reraise
        raise self.exc_type(msg)
    ValueError: Caught ValueError in replica 0 on device 0.
    Original Traceback (most recent call last):
    File "/home/admin/.miniconda3/envs/voice/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
        output = module(*input, **kwargs)
    File "/home/admin/.miniconda3/envs/voice/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
        return forward_call(*input, **kwargs)
    File "/home/admin/.miniconda3/envs/voice/lib/python3.9/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1467, in forward
        outputs = self.wav2vec2(
    File "/home/admin/.miniconda3/envs/voice/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
        return forward_call(*input, **kwargs)
    File "/home/admin/.miniconda3/envs/voice/lib/python3.9/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1067, in forward
        hidden_states = self._mask_hidden_states(hidden_states)
    File "/home/admin/.miniconda3/envs/voice/lib/python3.9/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 982, in _mask_hidden_states
        mask_time_indices = _compute_mask_indices(
    File "/home/admin/.miniconda3/envs/voice/lib/python3.9/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 146, in _compute_mask_indices
        raise ValueError(
    ValueError: `mask_length` has to be smaller than `sequence_length`, but got `mask_length`: 10 and `sequence_length`: 9`
    '''

Context: cc @patrickvonplaten


A potential in-place operation that caused an RuntimeError

Question:
Hi, I’m using transformers to generate sentences containing gradients using model.generate with the modification of removing @torch.no_grad() ahead of def generate(...):, since the current version (4.2.1) of model.generate doesn’t support keeping gradient.
And because I set do_sample=True and num_beams>1 in generate, the return type is BeamSampleEncoderDecoderOutput. According to the documents,  the scores of BeamSampleEncoderDecoderOutput consists of log softmax scores for each vocabulary token and the sum of log softmax of previously generated tokens in this beam.
What I want to do is gathering the non-inf value from the last beam and applying gradient descent to train the network later. The key pseudo-code is:
outputs = self.generate(input_ids, ..., **model_kwargs)
# The type of outputs is BeamSampleEncoderDecoderOutput
scores = outputs.scores
last_step_score = scores[-1]
last_step_score = last_step_score[torch.where(last_step_score != -float('inf'))]
last_step_score = last_step_score[::num_beams]

However, when I run the program, I receive an error:
RuntimeError:
one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [16, 50265]], which is output 0 of LogSoftmaxBackward, is at version 17; expected version 0 instead.
Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

which means that there is an in-place operation in generate, and I guess the in-place operation lies in BeamSearchScorer.finalize, but I can’t figure out what source code to change to make it viable.

Context: cc @patrickvonplaten


Differences between Config.from_pretrained and Model.from_pretrained

Question:
Hey there! I have a question regarding the differences between loading a multilingual BERT model from pretrained weights and from a pretrained Config:
Shouldn’t the two models defined below have the same weights?
from transformers import BertConfig, BertModel

mbert_model_1 = BertModel.from_pretrained("bert-base-multilingual-uncased")

mbert_config = BertConfig.from_pretrained("bert-base-multilingual-uncased")
mbert_model_2 = BertModel(mbert_config)

I have checked and they have the same architecture, but the layer weights (and the results obtained when using them) are different.
Sorry if it’s a well-known question but I had never loaded models from Configs and I’ve found this discrepancy. (I’ve looked for a previous question related to this topic but I haven’t found any).
Thanks for your help!

Context:
You should have a look at the relevant section in the course 3 and the correspond video 2 where all of this is explained.
The first model is initialized with the pretrained weights, the second is the same architecture but is initialized randomly.


Adjusting parameters for the FC layers at the end

Question:
I am trying to fine-tune a pre-trained BigBird on a custom task; the pre-training dataset has about 25k samples for a model of 76M parameters while the target datasets is about 800 samples.
During fine-tuning, I am unable to get the loss to converge which is highly volatile (appears like a noise sine wave) - it seems to be that my model might be underfitting on the dataset due to its high sequence length and/or complexity.
For the BERT model at the core,

image1024×589 127 KB

the size or complexity of the ‘Linear’ block can be adjusted to accommodate tasks however for the configuration accessible by Transformers
- vocab_size (int, optional, defaults to 50358) – Vocabulary size of the BigBird model. Defines the number of different tokens that can be represented by the inputs_ids passed when calling BigBirdModel.

- hidden_size (int, optional, defaults to 768) – Dimension of the encoder layers and the pooler layer.

- num_hidden_layers (int, optional, defaults to 12) – Number of hidden layers in the Transformer encoder.

- num_attention_heads (int, optional, defaults to 12) – Number of attention heads for each attention layer in the Transformer encoder.

- intermediate_size (int, optional, defaults to 3072) – Dimension of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.

- hidden_act (str or function, optional, defaults to "gelu_new") – The non-linear activation function (function or string) in the encoder and pooler. If string, "gelu", "relu", "selu" and "gelu_new" are supported.

- hidden_dropout_prob (float, optional, defaults to 0.1) – The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.

- attention_probs_dropout_prob (float, optional, defaults to 0.1) – The dropout ratio for the attention probabilities.

There is not such way to adjust that.
Does anyone have any idea how I might be able to customize BigBird - or should I instead extract the decoder output for use on my own network?

Context:
Gm!
from the docs, it seems that obtaining the sequence embeddings from the pre-trained model is pretty easy using the last_hidden_state or hidden_states returned by BigBird model.
However, I would still prefer if there might be a way to modify the size of the Linear Layer in-place as that might be easier than using the embeddings and constructing another Pytorch model to interface with that.


Get output embeddings out of a transformer model

Question:
Assuming that I am using a language model like BertForMaskedLM. how can I get the embeddings for each word in the sequence after passing the input_ids to the model. In the docs, I found the function get_out_embeddings but it returns an nn.Module linear(seq_length, vocab_size)
My full model is something similar to
class BYOLLM(AlbertPreTrainedModel):
    def __init__(self, config):
        super().__init__(config)

        self.albert = AlbertModel(config)
        self.predictions = AlbertMLMHead(config)
        self.init_weights()
        self.tie_weights()
        self.config = config

        self.mlp = nn.Sequential(
            nn.Linear(config.hidden_size, 4096),
            nn.BatchNorm1d(4096),
            nn.ReLU(inplace=True),
            nn.Linear(4096, config.hidden_size),
        )

    def tie_weights(self):
        self._tie_or_clone_weights(
            self.predictions.decoder, self.albert.embeddings.word_embeddings
        )

    def get_output_embeddings(self):
        return self.predictions.decoder

    def forward(
        self,
        input_ids=None,
        attention_mask=None,
        token_type_ids=None,
        position_ids=None,
        head_mask=None,
        inputs_embeds=None,
        labels=None,
        output_attentions=None,
        output_hidden_states=None,
        masked_index = None,
        **kwargs
    ):

        outputs = self.albert(
            input_ids=input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
        )
        sequence_outputs = outputs[0]

        prediction_scores = self.predictions(sequence_outputs)

        return prediction_scores

What I want is to access the predictions scores logits and get the corresponding embedding for a specific word. getting the embedding is the part that I am asking about.

Context:
I have found this in the docs 16
hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) – Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).
But when using this option, i thought i should get Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size). but after passing this to the model, the one for the output embedding is in shape (1, hidden_size) instead of (1, seq_lenght, hidden_size)
note: the Batch size is 1


Multiple training will give exactly the same result except for the first time

Question:
Hi, I have a function that will load a pre-trained model and fine-tune it for sentiment analysis then calculates the F1 score and returns the result.
The problem is when I call this function multiple times with the exact same arguments, it will give the exact same metric score which is expected, except for the first time which is different, how is that possible?
This is my function which is written based on this tutorial in hugging face:
import uuid

import numpy as np

from datasets import (
    load_dataset,
    load_metric,
    DatasetDict,
    concatenate_datasets
)

from transformers import (
    AutoTokenizer,
    AutoModelForSequenceClassification,
    DataCollatorWithPadding,
    TrainingArguments,
    Trainer,
)

CHECKPOINT = "distilbert-base-uncased"
SAVING_FOLDER = "sst2"
def custom_train(datasets, checkpoint=CHECKPOINT, saving_folder=SAVING_FOLDER):

    model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)
    tokenizer = AutoTokenizer.from_pretrained(checkpoint)

    def tokenize_function(example):
        return tokenizer(example["sentence"], truncation=True)

    tokenized_datasets = datasets.map(tokenize_function, batched=True)
    data_collator = DataCollatorWithPadding(tokenizer=tokenizer)

    saving_folder = f"{SAVING_FOLDER}_{str(uuid.uuid1())}"
    training_args = TrainingArguments(saving_folder)

    trainer = Trainer(
        model,
        training_args,
        train_dataset=tokenized_datasets["train"],
        eval_dataset=tokenized_datasets["validation"],
        data_collator=data_collator,
        tokenizer=tokenizer,
    )

    trainer.train()

    predictions = trainer.predict(tokenized_datasets["test"])
    print(predictions.predictions.shape, predictions.label_ids.shape)
    preds = np.argmax(predictions.predictions, axis=-1)

    metric_fun = load_metric("f1")
    metric_result = metric_fun.compute(predictions=preds, references=predictions.label_ids)

    return metric_result

And then I will run this function several times with the same datasets, and append the result of the returned F1 score each time:
raw_datasets = load_dataset("glue", "sst2")

small_datasets = DatasetDict({
    "train": raw_datasets["train"].select(range(100)).flatten_indices(),
    "validation": raw_datasets["validation"].select(range(100)).flatten_indices(),
    "test": raw_datasets["validation"].select(range(100, 200)).flatten_indices(),
})

results = []
for i in range(4):
    result = custom_train(small_datasets)
    results.append(result)

And then when I check the results list:
[{'f1': 0.7755102040816325}, {'f1': 0.5797101449275361}, {'f1': 0.5797101449275361}, {'f1': 0.5797101449275361}]

Something that may come to mind is that when I load a pre-trained model, the head will be initialized with random weights and that is why the results are different, if that is the case why only the first one is different and the others are exactly the same?

Context:
You need to set the seed before instantiating your model, otherwise the random head is not initialized the same way, that’s why the first run will always be different.
The subsequent runs are all the same because the seed has been set by the Trainer in the train method.
To set the seed:
from transformers import set_seed

set_seed(42)


Question:
Hello
I have two questions:
1 We can  view the. pooled layer by using  output_hidden_states=true and follow the logic, but is there a  generic way to do so?
2 When we are doing inference, can we output the hidden_states from trainer.predict() as we do by model(input/output_hidden_states=True)?
Thanks

Context:
I have a similar doubt as regards point 2.
I am working on Question Answering with Distilbert.
The predict function in the Trainer does not work if output_hidden_states = True. It works fine if the same argument is set to False.
Is this a bug? If not, then how  is one to use a model for prediction if one has set the argument output_hidden_states = True while initializing the model ?


What is the `tie_word_embeddings` option exactly doing?

Question:
Hi,
for some models there is this tie_word_embeddings parameter. I think it is for the text 2 text models.
Can someone please explain what exactly this parameter is doing?
Many thanks
Philip

Context: No this is for all models that have a language modeling head (so even masked language models like BERT or causal language models like GPT-2). The idea is that the embedding weights (vocab_size by hidden_size) are tied with the decoder (hidden_size by vocab_size) so the model only learns one representation of the words (that is a big matrix!)


Minor Bug: HF (run_text_classification) attempts to use XLA on CUDA device

Question:
A minor inconsistency: on a GPU runtime, when I execute:
!pip install -q cloud-tpu-client==0.10 https://storage.googleapis.com/tpu-pytorch/wheels/torch_xla-1.9-cp37-cp37m-linux_x86_64.whl

to install the TPU client, HuggingFace will try to use XLA even if a CUDA device is present.
RuntimeError: tensorflow/compiler/xla/xla_client/computation_client.cc:273 : Missing XLA configuration

Shouldn’t there be checks to verify if XLA/TPU cores flag is not passed, it should fall back to CUDA->CPU rather than trying to run via XLA?

Context: Good point, which flags were you thinking of? If you feel up to it, don’t hesitate to open a PR with those changes!


Question:
I have read previous posts on the similar topic but could not conclude if there is a workaround to get only the best model saved and not the checkpoint at every step, my disk space goes full even after I add savetotallimit as 5 as the trainer saves every checkpoint to disk at the start.
Please suggest.
Thanks

Context: You can set save_strategy to NO to avoid saving anything and save the final model once training is done with trainer.save_model().


`run_glue.py` with my own dataset of one-sentence input

Question:
Hello,
This post is related to `run_glue.py` fails when using my own dataset of regression task · Issue #9393 · huggingface/transformers · GitHub 17 and [examples/text-classification] `do_predict` for the test set of local datasets · Issue #9442 · huggingface/transformers · GitHub 4.
While I was writing the text to open an issue, I realized that it seemed to be a simple mistake on my part.
If anyone gives the detail about it, I would appreciate your comments.
Information
Model I am using (Bert, XLNet …): Bert
The problem arises when using:

[ ] the official example scripts: (give details below)
[x] my own modified scripts: (give details below)

almost the same as run_glue.py, but add some modifications in evaluation metrics, using test sets
The tasks I am working on is:

[ ] an official GLUE/SQUaD task: (give the name)
[x] my own task or dataset: (give details below)

To reproduce
It seems that an error occurs when I use run_glue.py with my own dataset of a regression task.
CUDA_VISIBLE_DEVICES=0 python <my_modified_run_glue.py> \
  --model_name_or_path bert-base-cased \
  --train_file data/****.csv \
  --validation_file data/****.csv \
  --test_file data/****.csv \   # this arg is added for issue #9442
  --do_train \
  --do_eval \
  --do_predict \   # this arg is related to issue #9442
  --max_seq_length 64 \
  --per_device_train_batch_size 32 \
  --per_device_eval_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 10.0 \
  --load_best_model_at_end \
  --evaluation_strategy epoch \
  --metric_for_best_model eval_pearson \
  --output_dir **** \
  --overwrite_output_dir

An example of the train/valid CSV file is as below:
id,label,sentence1
__id_as_string__,3.0,__string__

Then, the trainer gives me the information below.
[INFO|trainer.py:387] 2021-01-07 12:52:02,202 >> The following columns in the training set don't have a corresponding argument in `BertForSequenceClassification.forward` and have been ignored: id, sentenc
e1.
[INFO|trainer.py:387] 2021-01-07 12:52:02,204 >> The following columns in the evaluation set don't have a corresponding argument in `BertForSequenceClassification.forward` and have been ignored: id, sente
nce1.

Expected behavior
it is natural that id column is ignored, but I didn’t know why sentence1 is ignored.
I checked again the task_to_keys in the original script:
task_to_keys = {
    "cola": ("sentence", None),
    "mnli": ("premise", "hypothesis"),
    "mrpc": ("sentence1", "sentence2"),
    "qnli": ("question", "sentence"),
    "qqp": ("question1", "question2"),
    "rte": ("sentence1", "sentence2"),
    "sst2": ("sentence", None),
    "stsb": ("sentence1", "sentence2"),
    "wnli": ("sentence1", "sentence2"),
}

Should I use “sentence” instead of "sentence" if there is only one sentence in the input (in other words, sentence2 is None`)?
Thank you in advance.

Context:
I’ve changed sentence1 to sentence, but the almost same info appears:
[INFO|trainer.py:387] 2021-01-07 13:22:18,233 >> The following columns in the training set don't have a corresponding argument in `BertForSequenceClassification.forward` and have been ignored: sentence, id.
[INFO|trainer.py:387] 2021-01-07 13:22:18,233 >> The following columns in the evaluation set don't have a corresponding argument in `BertForSequenceClassification.forward` and have been ignored: sentence, id.

Is it related to the code snippet below?
    # Preprocessing the datasets
    if data_args.task_name is not None:
        sentence1_key, sentence2_key = task_to_keys[data_args.task_name]
    else:
        # Again, we try to have some nice defaults but don't hesitate to tweak to your use case.
        non_label_column_names = [name for name in datasets["train"].column_names if name != "label"]
        if "sentence1" in non_label_column_names and "sentence2" in non_label_column_names:
            sentence1_key, sentence2_key = "sentence1", "sentence2"
        else:
            if len(non_label_column_names) >= 2:
                sentence1_key, sentence2_key = non_label_column_names[:2]
            else:
                sentence1_key, sentence2_key = non_label_column_names[0], None

Should I change the order of the columns?


Question:
For those wishing to [MASK] several tokens, here this is.
My question, however, relates to the output. I added “top_k” assuming I’d be able to return multiple sentences, but that was not the case. I am not sure how exactly I can achieve this.
import torch
from transformers import BertTokenizer, BertModel,BertForMaskedLM

tokenizer = BertTokenizer.from_pretrained('bert-base-cased')

input_tx = "[CLS] [MASK] [MASK] [MASK] of the United States mismangement of the Coronavirus is its distrust of science. [SEP]"
tokenized_text = tokenizer.tokenize(input_tx)
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
top_k = 10
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([[0]*25])

model = BertForMaskedLM.from_pretrained('bert-base-cased')

outputs = model(tokens_tensor, token_type_ids=segments_tensors)
predictions = outputs[0]
predicted_index = [torch.argmax(predictions[0, i]).item() for i in range(0,24)]
predicted_token = [tokenizer.convert_ids_to_tokens([predicted_index[x]])[0] for x in range(1,24)]
print(predicted_token)

`Output: 'The', 'main', 'cause', 'of', 'the', 'United', 'States', 'mi', '##sman', '##gement', 'of', 'the', 'Co', '##rona', '##virus', 'is', 'its', 'di', '##st', '##rust', 'of', 'science', '`

Context:
Hi there! First of all, please note that in the latest release, the recommended way to preprocess your input is just to call the tokenizer on your test:
import torch
from transformers import BertTokenizer, BertModel,BertForMaskedLM

tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
input_txt = "[MASK] [MASK] [MASK] of the United States mismanagement of the Coronavirus is its distrust of science."
inputs = tokenizer(input_txt, return_tensors='pt')

This returns a dict string to tensors (since I asked to return pytorch tensors with the last argument) and you can then directly call your model on it:
model = BertForMaskedLM.from_pretrained('bert-base-cased')

outputs = model(**inputs)
predictions = outputs[0]

At this stage, predictions is the output of our language model before the softmax (we won’t care about that since the probabilities after the softmax or the activations before are in the same order). You ask for the most probable token, so it only returns that. If you want, say, the most probable 10 tokens, you could go:
sorted_preds, sorted_idx = predictions[0].sort(dim=-1, descending=True)
for k in range(10):
    predicted_index = [sorted_idx[i, k].item() for i in range(0,24)]
    predicted_token = [tokenizer.convert_ids_to_tokens([predicted_index[x]])[0] for x in range(1,24)]
    print(predicted_token)


How is the “Auto Model For Sequence Classification” architecture?

Question: How is the architecture of AutoModelForSequenceClassification? I suppose it’s some pre-trained transformer with some dense layer for classification, however where could I see the forward details of this model?

Context: The auto classes are just abstractions that work for every architecture. You can see the actual forward passes in each modeling files. For instance, if you are using a BERT checkpoint, you will get a BertForSequenceClassification model, which forward pass is defined in `transformers.models.bert.modeling_bert 33


Question:
the first 3 tutorials are nowhere to be found. 404 error

      huggingface.co

🤗 Transformers Notebooks 2

  You can find here a list of the official notebooks provided by Hugging Face. Also, we would like to list here interesting content created by the community. I...

Context: Yes, those have been replaced by more recent versions, check out the master docs 4 for the new links.


MT5 Fine Tuning - KeyError: ‘source_ids’

Question:
Hi,
I am trying to fine tune MT5 for multitask question answer and question generation similar to @valhalla model. I prepared the dataset by using datasets library as follows:
    train_dataset = Dataset.from_pandas(pd.DataFrame(generate_data(mode="train")))
    valid_dataset = Dataset.from_pandas(pd.DataFrame(generate_data(mode="valid")))

    processor = DataProcessor(
        tokenizer,
        max_source_length=data_args.max_source_length,
        max_target_length=data_args.max_target_length
    )

    train_dataset = processor.process(train_dataset)
    valid_dataset = processor.process(valid_dataset)

    columns = ["source_ids", "target_ids", "attention_mask"]

However, when I try to train my model as below:
!python3 run_multi.py \
    --model_name_or_path google/mt5-small \
    --model_type mt5 \
    --tokenizer_name_or_path mt5_qg_tokenizer \
    --output_dir mt5-small-multi \
    --train_file_path data/train_data_qa_qg_mt5.pt \
    --valid_file_path data/valid_data_qa_qg_mt5.pt \
    --per_device_train_batch_size 16 \
    --per_device_eval_batch_size 16 \
    --gradient_accumulation_steps 2 \
    --learning_rate 1e-4 \
    --num_train_epochs 2 \
    --seed 42 \
    --do_train \
    --do_eval \
    --logging_steps 100 \
    --prediction_loss_only True

it says KeyError: 'source_ids'
I am sure the dataset has “source_ids” field.
train_dataset=torch.load(r"train_data_qa_qg_mt5.pt")
train_dataset
> Dataset({
    features: ['attention_mask', 'source_ids', 'source_text', 'target_ids', 'target_text', 'task'],
    num_rows: 3449
})

What might cause this?
The versions of the libraries are:
transformers == 4.4.2
datasets == 1.5.0
Thank you for the reply in advance.

Context: yes I was using Trainer. But I solved the problem. I was loading dataset via Datasets library, when I replaced it with nlp.load_dataset, it worked seamlessly. But thank you for the response, I did not pass --remove_unused_columns False


Question:
Can we use Huggingface library with Pytext 6? Any guides or references if possible? I seek to deploy models like huggingface-conv-ai or RAG or DialoGPT in scalable way. Is there any other good alternative?
Thanks

Context: Hi I actually created a deployment platform with a few friends of mine as a small project - https://backprop.co 4. Is this something that could potentially help you out?


Question:
Hi,
There is any way to select the beam size of the evaluation step of Seq2SeqTrainer?

Context:
You can set it in trainer.evaluate(num_beams=...) or set the default you like when instantiating your model:
model = AutoModelForXXX(checkpoint, num_beams=...)


Translation - MBART, translation with identical source and target language, for text normalization

Question:
Hi,
This is rather a general question about translation and I am aware that I don’t follow exactly your guidelines, so I am sorry for that.
(We could run the examples mentioned in your readme, great tool!)
We try to conceive normalization for Dutch, as a ‘translation’ task.
So, is it for instance possible to use source + target language, defined as the same language, for instance
--source_lang nl_XX \
--target_lang nl_XX \

{“translation”: {“nl_XX”: “liefst geen energie vandaag . waar is m’n oplaadstation ?”, “nl_XX”: “liefst geen energie vandaag . waar is mijn oplaadstation ?”}}
or
--source_lang source\
--target_lang target \

{“translation”: {“source”: “liefst geen energie vandaag . waar is m’n oplaadstation ?”, “target”: “liefst geen energie vandaag . waar is mijn oplaadstation ?”}}

Environment info

      github.com

    huggingface/transformers 2

  master/examples/pytorch/translation

  🤗 Transformers: State-of-the-art Natural Language Processing for Pytorch, TensorFlow, and JAX.

–model_name_or_path facebook/mbart-large-50-many-to-many-mmt
Thanks for your answer!

Context: You should change the data processing in the script as the source_lang and target_lang are also used to set the langauges on multilingual tokenizers (if you’re using mBart for instance, this approach wouldn’t work).


Question:
Hello everyone. I’m trying to upload my fine-tuned GPT-2 Model to Model Hub.
When I try to use the uploading function push_to_hub I get the following error:
AttributeError: 'GPT2Model' object has no attribute 'push_to_hub'
In the documentation it says that I can push the model with this function.
Can anybody help please?

Context: You need to update your Transformers library to the latest version.


How to modify each decoding step in ProphetNet Transformer

Question: Hi everyone, I am working on a project which needs modification on each decoding step of the ProphetNet model. At each decoding step, I want to concatenate an embedding representation with decoding output representation before it passes to the softmax layer. I am not sure which script/code needs modification for this purpose. I am new to the transformer libaray. Can you please suggest or provide some references? It will be helpful. Thank you!

Context: Hi @valhalla and  @s4sarath, any suggestions for this.  It need not be a prophet-net model, it can t5 or BART model in general. Thank you!


Is there a notebook or document for hyperparameter search?

Question:
Trainer has a function named hyperparameter_search(), I wonder is there a notebook or document to describe how to use this function? This function seems so hard for me to understand o(╥﹏╥)o
Thank you~

Context: hey @yc1999 you can find an example on how the hyperparameter search in the Trainer works in this tutorial notebook here: Google Colaboratory 12


ValueError: too many values to unpack (expected 2) when using BertTokenizer

Question:
Hi everyone,
I get an error when using BertTokenizer .
I do
encoding = tokenizer([[prompt, prompt, prompt], [choice0, choice1, choice2]], return_tensors='tf', padding=True))
and get
ValueError: too many values to unpack (expected 2) .
When I do
encoding = tokenizer([[prompt, prompt], [choice0, choice1]], return_tensors='tf', padding=True)
it works. Any idea why? I want to fine-tune TFBertForMultipleChoice such that each question ( prompt ) has three choices and not two as in the documentation BERT — transformers 4.7.0 documentation 2
Below is the complete code
import os
import numpy as np
import pandas as pd
import tensorflow as tf

from transformers import BertTokenizer, TFBertForMultipleChoice

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = TFBertForMultipleChoice.from_pretrained('bert-base-uncased')

prompt = "Accept and check containers of mail from large volume mailers, couriers, and contractors."
choice0 = "Time Management"
choice1 = "Writing"
choice2 = "Reading Comprehension"

encoding = tokenizer([[prompt, prompt, prompt], [choice0, choice1, choice2]], return_tensors='tf', padding=True)
inputs = {k: tf.expand_dims(v, 0) for k, v in encoding.items()}
outputs = model(inputs)  # batch size is 1

logits = outputs.logits

Thanks!
Ayala

Context:
Oh the example has one pairs of [] that is not necessary, will fix. It should be:
encoding = tokenizer([prompt, prompt, prompt], [choice0, choice1, choice2], return_tensors='tf', padding=True)


Question: How to calculate the accuracy of the testing dataset when we build MLM model using scrach?

Context: have you got the answers?


Key Error ‘loss’ while fine tuning GPT-2 with the Trainer utility

Question:
training_args = TrainingArguments(
  output_dir='./results',          # output directory
  num_train_epochs=3,              # total # of training epochs
  per_device_train_batch_size=16,  # batch size per device during training
  per_device_eval_batch_size=16,   # batch size for evaluation
  logging_dir='./logs',            # directory for storing logs
)

trainer = Trainer(
  model=model,
  args=training_args,
  train_dataset=tokenized_datasets["train"],
  eval_dataset=tokenized_datasets["test"],
  tokenizer=tokenizer
)

Error Log:
/usr/local/lib/python3.6/dist-packages/transformers/trainer.py in train(self, model_path, trial)
745                         tr_loss += self.training_step(model, inputs)
746                 else:
–> 747                     tr_loss += self.training_step(model, inputs)
748                 self._total_flos += self.floating_point_ops(inputs)
749
/usr/local/lib/python3.6/dist-packages/transformers/trainer.py in training_step(self, model, inputs)
1073                 loss = self.compute_loss(model, inputs)
1074         else:
-> 1075             loss = self.compute_loss(model, inputs)
1076
1077         if self.args.n_gpu > 1:
/usr/local/lib/python3.6/dist-packages/transformers/trainer.py in compute_loss(self, model, inputs)
1103             self._past = outputs[self.args.past_index]
1104         # We don’t use .loss here since the model may return tuples instead of ModelOutput.
-> 1105         return outputs[“loss”] if isinstance(outputs, dict) else outputs[0]
1106
1107     def is_local_process_zero(self) -> bool:
/usr/local/lib/python3.6/dist-packages/transformers/file_utils.py in getitem(self, k)
1356         if isinstance(k, str):
1357             inner_dict = {k: v for (k, v) in self.items()}
-> 1358             return inner_dict[k]
1359         else:
1360             return self.to_tuple()[k]
KeyError: ‘loss’

Context: If you have this error, it’s probably because you are not passing any labels to your model. It’s hard to know for sure since you don’t explain how you built your dataset.


Question:
I noticed that when loading a pretrained model (BertForSequenceClassification in my case) while running a virtual environment that an error occurs:
HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /bert-base-uncased/resolve/main/config.json (Caused by ProxyError('Cannot connect to proxy.', OSError(0, 'Error')))
Traceback (most recent call last):
  File "/home/aclifton/venvs/hf_test/lib/python3.7/site-packages/urllib3/connectionpool.py", line 696, in urlopen
    self._prepare_proxy(conn)
  File "/home/aclifton/venvs/hf_test/lib/python3.7/site-packages/urllib3/connectionpool.py", line 964, in _prepare_proxy
    conn.connect()
  File "/home/aclifton/venvs/hf_test/lib/python3.7/site-packages/urllib3/connection.py", line 359, in connect
    conn = self._connect_tls_proxy(hostname, conn)
  File "/home/aclifton/venvs/hf_test/lib/python3.7/site-packages/urllib3/connection.py", line 506, in _connect_tls_proxy
    ssl_context=ssl_context,
  File "/home/aclifton/venvs/hf_test/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 450, in ssl_wrap_socket
    sock, context, tls_in_tls, server_hostname=server_hostname
  File "/home/aclifton/venvs/hf_test/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/home/aclifton/anaconda3/lib/python3.7/ssl.py", line 423, in wrap_socket
    session=session
  File "/home/aclifton/anaconda3/lib/python3.7/ssl.py", line 870, in _create
    self.do_handshake()
  File "/home/aclifton/anaconda3/lib/python3.7/ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
OSError: [Errno 0] Error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aclifton/venvs/hf_test/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/aclifton/venvs/hf_test/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/aclifton/venvs/hf_test/lib/python3.7/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /bert-base-uncased/resolve/main/config.json (Caused by ProxyError('Cannot connect to proxy.', OSError(0, 'Error')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aclifton/venvs/hf_test/lib/python3.7/site-packages/transformers/configuration_utils.py", line 505, in get_config_dict
    user_agent=user_agent,
  File "/home/aclifton/venvs/hf_test/lib/python3.7/site-packages/transformers/file_utils.py", line 1337, in cached_path
    local_files_only=local_files_only,
  File "/home/aclifton/venvs/hf_test/lib/python3.7/site-packages/transformers/file_utils.py", line 1499, in get_from_cache
    r = requests.head(url, headers=headers, allow_redirects=False, proxies=proxies, timeout=etag_timeout)
  File "/home/aclifton/venvs/hf_test/lib/python3.7/site-packages/requests/api.py", line 104, in head
    return request('head', url, **kwargs)
  File "/home/aclifton/venvs/hf_test/lib/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/aclifton/venvs/hf_test/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/aclifton/venvs/hf_test/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/aclifton/venvs/hf_test/lib/python3.7/site-packages/requests/adapters.py", line 510, in send
    raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /bert-base-uncased/resolve/main/config.json (Caused by ProxyError('Cannot connect to proxy.', OSError(0, 'Error')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/aclifton/venvs/hf_test/lib/python3.7/site-packages/transformers/modeling_utils.py", line 1086, in from_pretrained
    **kwargs,
  File "/home/aclifton/venvs/hf_test/lib/python3.7/site-packages/transformers/configuration_utils.py", line 440, in from_pretrained
    config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/aclifton/venvs/hf_test/lib/python3.7/site-packages/transformers/configuration_utils.py", line 517, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load config for 'bert-base-uncased'. Make sure that:

- 'bert-base-uncased' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'bert-base-uncased' is the correct path to a directory containing a config.json file

However, not running the venv, the pretrained model loads fine. Here is an example:
from transformers import BertForSequenceClassification
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

Context: Any thoughts on this?


Question:
Hi all.
I’m creating a pretrianed T5 model with:
T5ForConditionalGeneration.from_pretrained("t5-small")

How is this model pretrained? It seems to me that the model weights I get here were trained at least on the GLUE dataset (and probably others).
I’d like it to only be pretrained on C4. Are those weights around somewhere? How do I get a model pretrained that way?
Thanks!

Context:
This is quite a general question. You should find  everything you need in their paper.
[1910.10683] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (arxiv.org) 5


For BERT LMs … are the random tasks created on just the first sentence or the second as well?

Question:
Just wondering if the 15% masked tokens are selected randomly on the first sentence or over the entire sequence (first and 2nd sentence)?
Thanks - wg

Context:
Looks like all input tokens, except for special tokens are masked.
So, yes the second sentence will have tokens masked as well, but not the [SEP] between it and the first (and the other special tokens).
Why I think so:
Looks like probability_matrix is just a tensor the shape of (all) inputs, filled with the 15% probability of being masked…, except for the special tokens where are set to have a 0% probability of masking before being passed to torch.bernoulli here:

      github.com

    huggingface/transformers/blob/master/src/transformers/data/data_collator.py#L379-L380

          probability_matrix.masked_fill_(special_tokens_mask, value=0.0)
          masked_indices = torch.bernoulli(probability_matrix).bool()


Has anyone here deployed a transformers model on Google Cloud using AI Platform?

Question:
Hi,
I have a fine tuned distilgpt2 model that I want to deploy using GCP ai-platform.
I’ve followed all the documentation for deploying a custom prediction routine on GCP but when creating the model I get the error:

Create Version failed. Bad model detected with error: Model requires more memory than allowed. Please try to decrease the model size and re-deploy.

Here is my setup.py file:
from setuptools import setup

setup(
    name="generator_package",
    version="0.2",
    include_package_data=True,
    scripts=["generator_class.py"],
    install_requires=['transformers==2.8.0']
)

I then create a model version using:
gcloud beta ai-platform versions create v1 --model my_model \
 --origin=gs://my_bucket/model/ \
 --python-version=3.7 \
 --runtime-version=2.3 \
 --package-uris=gs://my_bucket/packages/gpt2-0.1.tar.gz,gs://cloud-ai-pytorch/torch-1.3.1+cpu-cp37-cp37m-linux_x86_64.whl \
 --prediction-class=model_prediction.CustomModelPrediction

I have tried every suggested route and cant get this to work and I’m still getting the above error. I’m using the smallest gpt2 model and am well within memory.
Can anyone who have successfully deployed to GCP please give some insight here.
Thank you

Context:
Hi @farazk86,
Any updates about this? Did you manage to use AI platform to serve your model’s predictions?


Question: I wanted to use T5 for multiple choice task. Since there’s no class as T5ForMultipleChoice defined, I was wondering if there’s any specific reason for why this hasn’t been done.

Context: Can anyone suggest how to use T5ForConditionalGeneration for multiple choice task ? I can define my own MLP in the same way as AutoModelForMultipleChoice but was wondering if there’s any specific formatting of inputs required to deal with Multiple Choice task within the library.


Is there a way to backpropagate through multiple steps while using Trainer API

Question: I was wondering if there’s a way to backpropagate the accumulation of loss through multiple optimizer steps while using Trainer API? Since it’s easy to get cuda out of memory and would like to avoid it.

Context: If you are talking about gradient accumulation, you can set it with gradient_accumulation_steps=xxx in your TrainingArguments.


Calling changed functions in transformers/training_args

Question:
I am maintaining a program that used to do things like
TrainingArguments.to_json_string(my_object) and TrainingArguments.to_sanitized_dict(my_object).
But in the latest version of transformers/training_args (v4.8.2),  these two functions do not take arguments anymore. Instead, they act on ‘self’, which is a TrainingArgument object:  to_json_string(self) and to_sanitized_dict(self) -> Dict[str, Any].
Therefore, I wonder how do I change my implementation accordingly in order to make the same functionality works. Copying functions from transformers/training_args to my object class solves part of the issue, but works bad when the function has dependencies on other things in transformers/training_args, and it doesn’t make a lot of sense to copy everything from the transformers/training_args file to my object class.
What is the best way of doing this? Thank you!

Context: Those functions have not changed since the 14 months ago, so quite a few versions. They have always been methods, which you should call directly on an instance of a TrainingArguments.


TrainingArguments changing the GPU by iteslf

Question:
I have 4 gpus available, out of which i have selected the second one using
if torch.cuda.is_available():
    torch.cuda.set_device(2)

However when i compute the TrainingArgument() command :
training_args = TrainingArguments('mydirectory')
then torch.cuda.current_device(). is giving 0
Any idea why this is happening?

Context:
Yes, the training argument set the GPU corresponding to its local_rank value (for distributed training), so you have to make sure to pass along local_rank=2 when you instantiate them.
Though to execute a script on a given GPU, you would be better off setting the global env variable CUDA_VISIBLE_DEVICES.


Question:
Q: Max model input size varies between checkpoints, what is the max num input tokens that each model can process?
A:
max_model_length = {
    "xsum": 512,
    "cnn_dailymail": 1024,
    "newsroom": 512,
    "wikihow": 512,
    "multi_news": 1024,
    "reddit_tifu": 512,
    "big_patent": 1024,
    "arxiv": 1024,
    "pubmed": 1024,
    "gigaword": 128,
    "aeslc": 512,
    "billsum": 1024,
    "large": 1024,
}

that constant is defined here:

      github.com

    sshleifer/transformers_fork/blob/f69cac3347641beaba5037b9a6fca1a46f423639/src/transformers/configuration_pegasus.py#L67-L67 19

max_model_length = {

Context: So does that mean max_position_embedding is reduced for fine-tuned models (gigaword, wikihow). i.e if the max_position_embedding for the pre-trained model is 1024 then all fine-tuned models should also have same, right ?


What is the correct form of decoder_input_ids for LEDForConditionalGeneration?

Question:
I have been looking into the articles on the web, but unfortunately I cannot find the clear answer. I guess one of them is the correct decoder_input_ids (label should be decoder_input_ids[1:] ?):
1) <s>...</s><pad>...<pad>
2) </s><s>...</s><pad>...<pad>
3) <pad><pad>...</s><s>...</s>

Thanks in advance.
+) I am going to fine-tune this model for free form QA.

Context:
I guess decoder_input_ids should be </s><s>... (without </s>), given label as <s>...</s>, according to the code below used for generating decoder_input_ids:
def shift_tokens_right(input_ids: torch.Tensor, pad_token_id: int, decoder_start_token_id: int):
    """
    Shift input ids one token to the right.
    """
    shifted_input_ids = input_ids.new_zeros(input_ids.shape)
    shifted_input_ids[:, 1:] = input_ids[:, :-1].clone()
    shifted_input_ids[:, 0] = decoder_start_token_id

    assert pad_token_id is not None, "config.pad_token_id has to be defined."
    # replace possible -100 values in labels by `pad_token_id`
    shifted_input_ids.masked_fill_(shifted_input_ids == -100, pad_token_id)

    return shifted_input_ids


Question:
I’m trying to train RoBERTa from scratch on proprietary dataset using the script from HF repo (https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py 1). When I run the training on machine with 32 cores and 8x V100, GPUs are not utilized in 100% all the time and it seems like there is a bottleneck on transfer between CPU and GPU. Even when I set number of workers in DataLoaders to 32, the performance does not change at all.
My batch size is 8 (max I could fit into 16GB V100 on Google Cloud), all examples have 512 tokens.
How improve GPU usage? Are there any additional parameters that need to be configured to utilize the GPU better?
I’ve also captured a few seconds of watch on nvidia-smi output to give you a full picture:
https://1drv.ms/v/s!AkfjsmHCRwTChtVDsCiowniO7rMLSA?e=zjNAkF 14
I’m using the following model parameters:
{
  "architectures": [
    "RobertaForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "eos_token_id": 2,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 514,
  "model_type": "roberta",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 1,
  "type_vocab_size": 1,
  "vocab_size": 32768
}

Context: Bump - anyone?


What arguments need to be changed when using deepeed in trainer?

Question:
I understand that we will need to change a regular call function like python run_trainer.py ... to deepspeed --hostfile <hostfile> run_trainer.py ... --deepspeed <deepspeed_config_file>.
Besides the deepspeed argument, is there anything else I should change, for example, sharded_ddp, ddp_find_unused_parameters, skip_memory_metrics, etc.?

Context: You should find everything that you need over here: DeepSpeed Integration — transformers 4.7.0 documentation (huggingface.co) 3


Trainer with load_best_model_at_end doesn’t work

Question:
I was trying to save the checkpoint after each epoch by setting load_best_model_at_end=True with p3.16xlarge parallel training. However, the following error occurs:
OSError
Can’t load config for ‘bert_model/checkpoint-156’. Make sure that:

‘bert_model/checkpoint-156’ is a correct model identifier listed on ‘Hugging Face – The AI community building the future. 1’
or ‘bert_model/checkpoint-156’ is the correct path to a directory containing config.json file.

The error occurs only if the output_dir folder is empty but not occurs if there were checkpoints from last training. Does anyone face the same issue or have an idea on this?

Context: What is the version of Transformers you are using? Also what were the training arguments for this training?


Trainer API to log both Training and Validation Metrics

Question:
I am fine-tuning for a classification task - I am trying to replicate (and potentially replace) my native PyTorch training and evaluation loops with the Trainer API.
I usually log metrics for both training and validation across each batch/epoch.
Here is what I can achieve with Trainer API.

The accuracy and F1 are of validation sets and I want to also see the same set of metrics at each step for training data as well. Can someone guide on how I can achieve this?
Thanks.

Context: This is not implemented in the Trainer. You can manually evaluate yourself at the end of training on any dataset you want (training set included).