Question:
I am working in a Google Coalab session with a HuggingFace DistilBERT model which I have fine tuned against some data.
I am getting the following error when I try to evaluate a restored copy of my model:-
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking arugment for argument index in method wrapper_index_select)

I run the following piece of code TWICE. Once just after fitting the model, and then once after saving and restoring the model.
metric= load_metric("accuracy")
model.eval()
for batch in test_dataloader:
    batch = {k: v.to(device) for k, v in batch.items()}
    with torch.no_grad():
        outputs = model(**batch)

    logits = outputs.logits
    predictions = torch.argmax(logits, dim=-1)
    metric.add_batch(predictions=predictions, references=batch["labels"])

metric.compute()

If I run the evaluation straight after training there is no problem:-
/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:10: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  # Remove the CWD from sys.path while we load stuff.
{'accuracy': 0.6692307692307692}

If I run the above code after saving and restoring the model then I get the error quoted above, the full traceback for which is:-
/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:10: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  # Remove the CWD from sys.path while we load stuff.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-102-33bf1579632a> in <module>()
      4     batch = {k: v.to(device) for k, v in batch.items()}
      5     with torch.no_grad():
----> 6         outputs = model(**batch)
      7
      8     logits = outputs.logits

8 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   2041         # remove once script supports set_grad_enabled
   2042         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 2043     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   2044
   2045

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking arugment for argument index in method wrapper_index_select)

The steps I take for saving and restoring are as follows:-

Write the model to the colab session’s local disc:-
Write from local disc (of the colab session) to Google Drive
Write back from Google Drive to the colab session’s local disc
Use the copy on the local drive to load the model

The code for step 1 has been adapted from that at run_glue.py 3 and is as follows:-
# Saving best-practices: if you use defaults names for the model, you can reload it using from_pretrained()

output_dir = './a_local_copy/'

# Create output directory if needed

if not os.path.exists(output_dir):

    os.makedirs(output_dir)

#logger.info("Saving model checkpoint to %s", args.output_dir)

print("Saving model checkpoint to %s" % output_dir)

# Save a trained model, configuration and tokenizer using `save_pretrained()`.

# They can then be reloaded using `from_pretrained()`

model_to_save = model.module if hasattr(model, 'module') else model  # Take care of distributed/parallel training

model_to_save.save_pretrained(output_dir)

tokenizer.save_pretrained(output_dir)

# Good practice: save your training arguments together with the trained model

# torch.save(args, os.path.join(output_dir, 'training_args.bin'))

Step 4 is the straightforward:-
model = AutoModelForSequenceClassification.from_pretrained(output_dir)
tokenizer = AutoTokenizer.from_pretrained(output_dir)

I am happy to load further code if you could give me some guidance as to what would be useful.

Context:
I think after you load the model, it is no longer on GPU, try
model = AutoModelForSequenceClassification.from_pretrained(output_dir).to(device)


KeyError: “Invalid key: slice(0, 1000, None). Please first select a split

Question:
Hi,
I am trying to train a tokenizer and execute the following line of code:
new_tokenizer = tokenizer.train_new_from_iterator(batch_iterator(), vocab_size=25000)

Though, when I execute it, I get this error:
KeyError: "Invalid key: slice(0, 1000, None). Please first select a split. For example: `my_dataset_dictionary['train'][slice(0, 1000, None)]`. Available splits: ['train']"

Context:
Hi ! Your batch_iterator must be iterating on a Dataset object, however it looks like you try to iterate over a DatasetDict (it maps split names to Dataset objects).
To fix your code, you just have to replace dataset by dataset["train"] in your definition of batch_iterator.
Let me know if that works or if if you have other questions


What is the best way to fine-tune ViT with a custom dataset?

Question:
I have checked out the course and I have come across tutorials for fine-tuning pre-trained models for NLP tasks.
But I would really like to use the Vision Transformer 1 model for classifying images that I have. I have about 1.8k images belonging to 3 categories, and I would like to use ViT for classification. I want to fine-tune the model to my dataset and thus leverage transfer learning.
This is a task of single-label classification.
How can I do this? What is the best way to fine-tune the pretrained ViT model for a classification task to a smaller dataset?
Can anyone point me towards any recipes or tutorials or other forms of how-tos?
Thanks.

Context:
Hi there! I made some demos on how to fine-tune ViT on a custom dataset here:

      github.com

    Transformers-Tutorials/VisionTransformer at master ·... 10

  master/VisionTransformer

  This repository contains demos I made with the Transformers library by HuggingFace. - Transformers-Tutorials/VisionTransformer at master · NielsRogge/Transformers-Tutorials


Is there a way to correctly load a pre-trained transformers model without the configuration file?

Question:
I would like to fine-tune a pre-trained transformers model on Question Answering. The model was pre-trained on large engineering & science related corpora.
I have been provided a “checkpoint.pt” file containing the weights of the model. They have also provided me with a “bert_config.json” file but I am not sure if this is the correct configuration file.
from transformers import AutoModel, AutoTokenizer, AutoConfig

MODEL_PATH = "./checkpoint.pt"
config = AutoConfig.from_pretrained("./bert_config.json")
model = AutoModel.from_pretrained(MODEL_PATH, config=config)

The reason I believe that bert_config.json doesn’t match “./checkpoint.pt” file is that,  when I load the model with the code above, I get the error that goes as below.

Some weights of the model checkpoint at ./aerobert/phase2_ckpt_4302592.pt were not used when initializing BertModel: [‘files’, ‘optimizer’, ‘model’, ‘master params’]

This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertModel were not initialized from the model checkpoint at ./aerobert/phase2_ckpt_4302592.pt and are newly initialized: [‘encoder.layer.2.attention.output.LayerNorm.weight’, ‘encoder.layer.6.output.LayerNorm.bias’, ‘encoder.layer.7.intermediate.dense.bias’, ‘encoder.layer.2.output.LayerNorm.bias’, ‘encoder.layer.21.attention.self.value.bias’, ‘encoder.layer.11.attention.self.value.bias’, …

If I am correct in assuming that “bert_config.json” is not the correct one, is there a way to load this model correctly without the config.json file?

Context:
This is telling you that the checkpoint that they gave you also includes the state of other things. So they also saved the state of the optimizer and not just the state of the model. It seems that you need to only load the “model” key. Maybe there is a better way than this, but I think you can do:

MODEL_PATH = "./checkpoint.pt"
state_dict = torch.load(MODEL_PATH)["model"]
config = AutoConfig.from_pretrained("./bert_config.json")
model = BertModel(config)

model = BertModel._load_state_dict_into_model(
    model,
    state_dict,
    MODEL_PATH
)[0]

# make sure token embedding weights are still tied if needed
model.tie_weights()

# Set model in evaluation mode to deactivate DropOut modules by default
model.eval()

I did not test this. See this for more:

      github.com

    huggingface/transformers/blob/e46ad22cd6cb28f78f4d9b6314e7581d8fd97dc5/src/transformers/modeling_utils.py#L1381-L1384 1

          @classmethod
          def _load_state_dict_into_model(
              cls, model, state_dict, pretrained_model_name_or_path, ignore_mismatched_sizes=False, _fast_init=True
          ):


How to save my tokenizer using save_pretrained?

Question:
I have just followed this tutorial on how to train my own tokenizer.
Now, from training my tokenizer, I have wrapped it inside a Transformers object, so that I can use it with the transformers library:
from transformers import BertTokenizerFast

new_tokenizer = BertTokenizerFast(tokenizer_object=tokenizer)

Then, I try to save my tokenizer using this code:
tokenizer.save_pretrained('/content/drive/MyDrive/Tokenzier')

However, from executing the code above, I get this error:
AttributeError: 'tokenizers.Tokenizer' object has no attribute 'save_pretrained'

Am I saving the tokenizer wrong?
If so, what is the correct approach to save it to my local files, so I can use it later?

Context: You are saving the wrong tokenizer ;-). new_tokenizer.save_pretrained(xxx) should work.


What is the purpose of save_pretrained()?

Question:
Hello everyone. I hope my question is not too silly, but there is something that confuses me.
Let’s say I load a huggingface model using from_pretrained() method, and then finetune it using the Trainer class. Now, via TrainingArguments, I get the chance to define an argument called output_dir. If I specify a directory here, won’t my model be saved in this directory, thus enabling me to load it again in the future from this folder, using from_pretrained()?
Here lies my question: if this argument lets me save the model, what is the purpose of save_pretrained()?
Looking on their respective documentations, it is clear that they do something different however: the first one saves something called checkpoints, while the second one saves “the model and its configuration”.
Can someone explain to me their difference, in other words, what is the difference between saving the checkpoints or the model? Won’t the last checkpoint be the same as the model and its weights?
Thanks in advance.

Context:
Hi there! The question is a bit weird in the sense you are asking: “Why does the model have this method when the Trainer has that model?”. The base answer is: " because they are two different objects."
Not everyone uses the Trainer to train their model, so there needs to be a method directly on the model to properly save it.
Now inside the Trainer, you could very well never save any checkpoint (save_strategy="no") or have the last checkpoint saved be before the end of training (with a save_strategy="steps") so you won’t necessarily automatically have the last model saved inside a checkpoint.
A checkpoint, by the way, is just a folder with your model, tokenizer (if it was passed to the Trainer) and all necessary files to resume training from there (optimizer state, lr scheduler state, trainer state etc).
To save your model at the end of training, you should use trainer.save_model(optional_output_dir), which will behind the scenes call the save_pretrained of your model (optional_output_dir is optional and will default to the output_dir you set).


Unable to use custom dataset when training a tokenizer

Question:
Hello,
I am following this tutorial here: notebooks/tokenizer_training.ipynb at master · huggingface/notebooks · GitHub 1
So, using this code, I add my custom dataset:
from datasets import load_dataset
dataset = load_dataset('csv', data_files=['/content/drive/MyDrive/mydata.csv'])

Then, I use this code to take a look at the dataset:
dataset

Access an element:
dataset['train'][1]

Access a slice directory:
dataset['train'][:5]

After executing the above code successfully, I try to execute this here:
new_tokenizer = tokenizer.train_new_from_iterator(batch_iterator(), vocab_size=25000)

However, I get this error:
KeyError: "Invalid key: slice(0, 1000, None). Please first select a split. For example: `my_dataset_dictionary['train'][slice(0, 1000, None)]`. Available splits: ['train']"

How do I fix this?
I am trying to train my own tokenizer, and this seems to be an issue.
Any help would be appreciated!

Context:
When asking for help on the forum, please paste all relevant code. In this case, you did not past the definition of batch_iterator.
If you are following the notebook, you did not load one dataset, but several (with a split for train/validation/test) which is why you get this error. You should add the split="train" argument when you load your dataset, or adapt the code of batch_iterator to index in your dataset dictionary.


Question:
I am running the following code:-
from tqdm.auto import tqdm

progress_bar = tqdm(range(num_training_steps))

model.train()
for epoch in range(num_epochs):
    for batch in train_dataloader:
        batch = {k: v.to(device) for k, v in batch.items()}
        outputs = model(**batch)
        loss = outputs.loss
        loss.backward()

        optimizer.step()
        lr_scheduler.step()
        optimizer.zero_grad()
        progress_bar.update(1)

This is a straight copy and paste from the HuggingFace documentation. Unfortunately, the code is returning the following error:-
'TFDistilBertForSequenceClassification' object has no attribute 'parameters'

I used the following boilerplate for the data loaders:-
from torch.utils.data import DataLoader

batch_size = 8

num_epochs = 3

train_dataloader = DataLoader(train_dataset, shuffle=True, batch_size=batch_size)

eval_dataloader = DataLoader(val_dataset, batch_size=batch_size)

I would be happy to share any further code that might be useful though there is rather a lot of it so please just describe what you need.
The full Traceback is:-
0%
1/390 [00:00<03:43, 1.74it/s]
/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:10: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  # Remove the CWD from sys.path while we load stuff.
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2897             try:
-> 2898                 return self._engine.get_loc(casted_key)
   2899             except KeyError as err:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

KeyError: 1002

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
8 frames
<ipython-input-167-b86b10a2f8e7> in <module>()
      5 model.train()
      6 for epoch in range(num_epochs):
----> 7     for batch in train_dataloader:
      8         batch = {k: v.to(device) for k, v in batch.items()}
      9         outputs = model(**batch)

/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py in __next__(self)
    519             if self._sampler_iter is None:
    520                 self._reset()
--> 521             data = self._next_data()
    522             self._num_yielded += 1
    523             if self._dataset_kind == _DatasetKind.Iterable and \

/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py in _next_data(self)
    559     def _next_data(self):
    560         index = self._next_index()  # may raise StopIteration
--> 561         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    562         if self._pin_memory:
    563             data = _utils.pin_memory.pin_memory(data)

/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

<ipython-input-154-c53f3c6a8f83> in __getitem__(self, idx)
      9     def __getitem__(self, idx):
     10         item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
---> 11         item['labels'] = torch.tensor(self.labels[idx])
     12         return item
     13

/usr/local/lib/python3.7/dist-packages/pandas/core/series.py in __getitem__(self, key)
    880
    881         elif key_is_scalar:
--> 882             return self._get_value(key)
    883
    884         if is_hashable(key):

/usr/local/lib/python3.7/dist-packages/pandas/core/series.py in _get_value(self, label, takeable)
    988
    989         # Similar to Index.get_value, but we do not fall back to positional
--> 990         loc = self.index.get_loc(label)
    991         return self.index._get_values_for_loc(self, loc, label)
    992

/usr/local/lib/python3.7/dist-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2898                 return self._engine.get_loc(casted_key)
   2899             except KeyError as err:
-> 2900                 raise KeyError(key) from err
   2901
   2902         if tolerance is not None:

KeyError: 1002

Context: I’m confused as the stack trace you are copying is not with the error you mention. The stack trace has a problem of indexing in the dataframe you apparently used for the labels, and as the final error says, it can’t find the label at index 1002.


Question:
Hello guys,
I am trying out multiclass text classification using DistilBERT. It is a dataset with user feedback on a product and there are 4 categories.
0 - Good - “easy to use”
1 - Bad - “slow and constant crashing”
2 - Questions - “can you add feature x and y?”
3 - Others - “NIL”
I am following the guide here 1, I padded the datasets like that:
train_encodings = tokenizer(train_texts, truncation=True, padding='max_length', max_length=128)
test_encodings = tokenizer(test_texts, truncation=True, padding='max_length', max_length=128)

I’ve set it to 128 for both as I thought having different pads would cause issues but some other posts here seem to suggest otherwise.
From my limited knowledge, in order to get new predictions from the model, I would have to encode the new texts and also tokenize them while padding it to 128, that is what I did but the prediction done was wrong. Meanwhile if I didn’t pad it at all, it would predict correctly.
Code:
new_prediction = tokenizer('good', truncation=True, padding='max_length', max_length=128)
new_text = torch.Tensor(new_prediction['input_ids']).long().reshape(1, len(new_prediction['input_ids'])).to('cuda:0')
print("Padded with all 0s version")
print(sentence)
print(tokenizer.batch_decode(sentence, skip_special_tokens=True))
print(model(sentence)[0].argmax(1))
print(model(sentence))

print("\nNo 0s version")
new_prediction = tokenizer('good', truncation=True, padding=True)
new_text = torch.Tensor(new_prediction['input_ids']).long().reshape(1, len(new_prediction['input_ids'])).to('cuda:0')
print(new_text)
print(tokenizer.batch_decode(new_text, skip_special_tokens=True))
print(model(new_text)[0].argmax(1))
print(model(new_text))

Output:
Padded with all 0s version
tensor([[ 101, 2204,  102,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0]], device='cuda:0')
['good']
tensor([1], device='cuda:0')
SequenceClassifierOutput(loss=None, logits=tensor([[ 0.5761,  1.5903, -1.0050, -1.0793]], device='cuda:0',
       grad_fn=<AddmmBackward>), hidden_states=None, attentions=None)

No 0s version
tensor([[ 101, 2204,  102]], device='cuda:0')
['good']
tensor([0], device='cuda:0')
SequenceClassifierOutput(loss=None, logits=tensor([[ 6.5631, -2.2075, -1.9927, -2.2027]], device='cuda:0',
       grad_fn=<AddmmBackward>), hidden_states=None, attentions=None)

As you can see, the word ‘good’ should return category 0, the padded version does not while the non-padded version returns correctly.
I am a beginner and am having a hard time understanding why, can someone enlighten me, thanks!

Context: Please have a look at the course, in particular this section 9. You need to pass the attention mask returned by the tokenizer to have you model ignore padding.


I can’t concatenate_datasets because features are not sorted. How do I sort it?

Question:
Hi guys,
I’m trying to concatenate two datasets that share some common features.
But these two datasets have features in a different order.
It’s like:
DatasetDict({
    train: Dataset({
        features: ['__index_level_0__', 'answers', 'context', 'document_id', 'id', 'question', 'title'],
        num_rows: 3952
    })
    validation: Dataset({
        features: ['__index_level_0__', 'answers', 'context', 'document_id', 'id', 'question', 'title'],
        num_rows: 240
    })
})

and
DatasetDict({
    train: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers'],
        num_rows: 60407
    })
    validation: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers'],
        num_rows: 5774
    })
})

so I erased uncommon features like: __index_level_0__, document_id, etc by using .remove_columns
Now the two DatasetDicts have the same features. However the order is different.
1st DatasetDict: features: ['answers', 'context', 'question', 'title'],
2nd DatasetDict: features: ['title', 'context', 'question', 'answers'],
So when I try to concatenate them by using datasets.concatenate_datasets([1stDatasetDict, 2ndDatasetDict]),
I get an error that says:

ValueError: Features must match for all datasets

Context:
hey @jeffnlp, i don’t think you can concatenate DatasetDict objects with concatenate_datasets - as described in the docs 3 this function expects a list of Dataset objects.
what happens if you try iterating over both DatasetDict objects and building up a new one that concatenates the Datasets objects as follows:
ds1 = DatasetDict(...)
ds2 = DatasetDict(...)
# Create empty DatasetDict
ds3 = DatasetDict()

for (split1, x), (split2, y) in zip(ds1.items(), ds2.items()):
    ds3[split1] = concatenate_datasets([x, y])

this should work since concatenate_datasets can handle out-of-order columns. if not, you might have a problem with the answers columns if they’re nested and the sub-columns don’t match. in that case you might be better off flattening the columns of each dataset and casting the features into the same types (see e.g. here 5 for casting details)


Question:
Hi
I’m a HuggingFace Newbie and I’m trying to fine tune DistilBERT for a three label sentiment classification task.
To do so I am using as a guide the HuggingFace Course 2. Hence I am using the following code to train my model:-
model = TFAutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=3)

lr_scheduler = PolynomialDecay(

    initial_learning_rate=5e-5,

    end_learning_rate=0.,

    decay_steps=num_train_steps

    )

opt = Adam(learning_rate=lr_scheduler)

model.compile(optimizer=opt, loss=loss, metrics=['accuracy', F1_metric()])

model.fit(

    encoded_train,

    np.array(y_train),

    validation_data=(encoded_val, np.array(y_val)),

    batch_size=8,

    epochs=3

)

The loss function is:-
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
The number of training steps is calculated like so:-
batch_size = 8

num_epochs = 3

num_train_steps = (len(encoded_train['input_ids']) // batch_size) * num_epochs

So far then, very much like the boiler-plate code in the course.
My encoded training data looks like this:-
{'input_ids': <tf.Tensor: shape=(1040, 512), dtype=int32, numpy=
array([[  101,   155,  1942, ...,     0,     0,     0],
       [  101, 27900,  7641, ...,     0,     0,     0],
       [  101,   155,  1942, ...,     0,     0,     0],
       ...,
       [  101,   109,  7414, ...,     0,     0,     0],
       [  101,  2809,  1141, ...,     0,     0,     0],
       [  101,  1448,  1111, ...,     0,     0,     0]],
dtype=int32)>, 'attention_mask': <tf.Tensor: shape=(1040, 512), dtype=int32, numpy=
array([[1, 1, 1, ..., 0, 0, 0],
       [1, 1, 1, ..., 0, 0, 0],
       [1, 1, 1, ..., 0, 0, 0],
       ...,
       [1, 1, 1, ..., 0, 0, 0],
       [1, 1, 1, ..., 0, 0, 0],
       [1, 1, 1, ..., 0, 0, 0]], dtype=int32)>}

Printing with y_train.head() my labels look like this (though my code turns this into a numpy array):-
10     2
147    1
342    1
999    3
811    3
Name: sentiment, dtype: int64

I am receiving the following error message:-
Epoch 1/3
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-56-2902befb3adf> in <module>()
     16     validation_data=(encoded_val, np.array(y_val)),
     17     batch_size=8,
---> 18     epochs=3
     19 )

14 frames
/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/type_spec.py in __make_cmp_key(self, value)
    381     raise ValueError("Unsupported value type %s returned by "
    382                      "%s._serialize" %
--> 383                      (type(value).__name__, type(self).__name__))
    384
    385   @staticmethod

ValueError: Unsupported value type BatchEncoding returned by IteratorSpec._serialize

My code is being run in Google Collaboratory using GPUs.

Context: The problem lies in the training data, but you did not share how you built it, so we can’t help you see what’s wrong.


Is there a way to finetune GPT2 775M on 16GB VRAM and 24GB RAM?

Question:
I was able to finetune GPT2 355M 2048 sequence, without FP16, all fit in VRAM.
But no luck with GPT2 755M. Obviously didn’t fit to VRAM, so I used FP16 and DeepSpeed CPU offload, That way I got 9GB VRAM free, but out of RAM.
Did someone succeed with GPT2 training for 775M with 16GB VRAM?

Context:
Now I’m able to run training with block size 1568.
Free VRAM 1593MiB, free RAM 14GB
Feel like there is a room to run block size 2048…


Question:
I am using Hugginface’s Trainer. How to adjust the learning rate after N number of epochs? For example, I have an initial learning rate set to lr=2e-6 , and I would like to change the learning rate to lr=1e-6 after the first epoch and stay on it the rest of the training.
I tried this so far:
optimizer = AdamW(model.parameters(),
              lr = 2e-5,
              eps = 1e-8
            )

epochs = 5
batch_number = len(small_train_dataset) / 8
total_steps = batch_number * epochs

scheduler = get_linear_schedule_with_warmup(optimizer,
                                            num_warmup_steps = 0,
                                            num_training_steps = total_steps,
                                            last_epoch=-1
                                            )

I know that there is LambdaLR — PyTorch 1.9.0 documentation 2 but here it drops learning rate every epoch but that is not what i want to do. I want it to drop after 1 epoch and then stay on it rest of the training process.

Context: Were you able to resolve this? I have a similar problem where I want to implement adaptive learning rate during training.


Pre-training & fine-tuning BERT on specific domain with custom dataset

Question:
Hallo,
I need some help with training BERT and thought maybe I can ask you here…
I am trying to train a BERT model for a specific domain, similar to BioBERT, but for some other field.
So, for achieving my plans, I run the run_mlm.py script which I found on transformers/examples/pytorch/language-modeling at master · huggingface/transformers · GitHub 31 for bert-base-uncased with some custom dataset, which is a large txt-file containing my corpus.
In the next step I wanted to fine-tune my model on the NER tasks using run_ner.py script I found on the same GitHub in: examples/pytorch/token-classification.
For a small example dataset the fine-tuning works, but if I use my whole dataset I get the following error: Map method to tokenize raises index error - #10 by rainman020 7.
Maybe you can tell me if my approach is correct?
Another point where I am struggling is that when using run_ner.py I get the warning that I should train this model on a down-stream task. But I thought this is what I am doing using this script to fine-tune on NER. Do I have to do some extra steps in the pre-training phase?
I googled a lot but it is still not 100% clear for me. If you could help me I would be very glad.
Thank you!

Context:
Hi there!
I’m using your question to ask one related to the run_ner.py script (maybe you could help on this one, since it is extremely basic)! I’m trying to build an extractive summariser using this latter.
I’m starting with all this so I am at level 0, and am trying to understand how to fit this script to my own data? I have my train dev and test datasets tokenized and labelled, in csv formats. But when i run the command on my data, the following appears. (the run_ner.py script is loaded to the environment)

Screenshot 2021-06-14 at 16.00.122212×562 107 KB

Any guidance on how to make this work…? Thank you!!


How do I fine-tune a zero-shot learning model to my task?

Question:
How would you improve the performance of zero-shot models considering you will be obtaining 1) feedback on the predictions made by the model and 2) labeled examples.
I want to do classification in Spanish and with very specialized legal text. The facebook/bart-large-mnli and joeddav/xlm-roberta-large-xnli models seem to work decently. However, I already have a couple of thousand examples labeled that might help to fine-tune the zero-shot learning models. Furthermore, I will be providing feedback on the predictions made by the model.
Thank you and greetings from Mexico!

Context: Hey, Any luck on zero-shot finetuning?


I want to custom my data set in speech recognition wav2vec

Question:
hello,
i’m really bignner but i have to make my own data set wich is in wave format fine tuned to wav2vec model so i really need help about what to do in details .

Context:
Hello wzr97.
I am not sure if I understand your question correctly, and I am also fairly new to everything myself.
But this blog might contain the answer to your question.

      huggingface.co

Fine-Tune Wav2Vec2 for English ASR in Hugging Face with 🤗 Transformers 15


Creating a Rick Sanchez chat bot with Transformers and Chai

Question:
Hey,
I made a notebook tutorial to try and make it as simple as possible to train and deploy a DialoGPT bot.
It uses dialogue from Rick and Morty to train a Rick bot and then deploys the bot onto Chai 5, a platform for creating and interacting with conversational AI’s.

RickBotChai1530×2892 277 KB

By the end of this tutorial you will have your very own chatbot, like the one pictured above

The notebook can be found here 7.
Thanks!

Context: This is cool! In your notebook it says you upload the model to huggingface - do you have a link to the model page? I wanna check it out!


Question:
Hi there,
I uploaded public and private models a week ago (and the public model has over a hundred downloads so far). However, I just found out that they are gone in my profile page. Can anyone tell me what is going on here? I certainly didn’t delete the models. Is this related to model sharing policy that I might be missing? Thank you!
best,

Context:
Oh my! I thought I was the only one going crazy! My models are also gone from my page, nor can they be searched. I similarly didn’t touch the models at all over the past few days… What’s odd is that I can still view and clone them if I visit their old direct links as per usual, like this 1 for example.
There seems to be something going on. I hope it can be resolved soon!


Different versions of ‘wav2vec2’ model and their differences

Question:
Hey everyone. I want to use wav2vec2 to perform ASR using data in my language (Greek). As such, I took a look at the various wav2vec2 pretrained models that exist in the model hub, and there are two things I don’t understand:
Some versions, like this facebook/wav2vec2-large-lv60 · Hugging Face, say in the description that the model ‘should be fine-tuned on a downstream task, like Automatic Speech Recognition’. On the other hand, other versions like ‘facebook/wav2vec2-large-960h-lv60’ (sorry, can’t post more than 2 links), impose no such requirement and also provide code snippets as an example of how to use the particular model.
Furthermore, the group of models mentioned first, do not have code examples, but contain links to this amazing blog post Fine-Tune XLSR-Wav2Vec2 for low-resource ASR with 🤗 Transformers which I studied.
Forgive me if my question is getting too big, but I’d like to ask something more related to this blog post. I noticed 2 things: (1) the author does not load the tokenizer and feature extractor using the ‘from_pretrained()’ method, but instantiates them by themselves and (2) the second group of models mentioned earlier (those who include code examples in their page), do not use a feature extractor at all. What are the reasons behind these distinctions?
Sorry again for the lengthy question. I’d really appreciate any help. Thanks in advance!

Context: A mistake I noticed in my post: the first link is meant to redirect to this version of wav2vec: facebook/wav2vec2-large-xlsr-53 · Hugging Face 1


Continue pre-training of Greek BERT with domain specific dataset

Question:
Hello,
I want to further pre-train Greek BERT of the library on a domain specific dataset in MLM task to improve results. The downstream task of BERT will be sequence classification. I found that the library also provides scripts 35 for that. In the example RoBERTa is further trained on wikitext-2-raw-v1. As I saw here 10, the dataset is formatted as:
{
“text”: “” The gold dollar or gold one @-@ dollar …
}
although, I downloaded the dataset from the link provided in that site and saw that the texts in the dataset are one after the other separated by titles within =.
My question is, what format should the dataset that I will further pre-train BERT have and how should they provided as train and dev?
If there is any source, it would be very helpful.
p.s. BERT was pre-trained in two tasks, MLM and NSP. Since my downstream task is Sequence Labeling, I thought that I should continue the pre-training with just the MLM task.

Context:
You should fine-tune the model on whatever task you want to perform. If you’d like to use it for sequence classification, then that’s what you should train it on i.e. exchange the head for a Sequence-Classification one.
This should be of help: BERT — transformers 4.3.0 documentation 46
This may help you understand how to format your input: BERT Fine-Tuning Tutorial with PyTorch · Chris McCormick 57 (the tokenizer does most of the heavy lifting for you)


Question:
Hello to all,
I’m following this tutorial: https://huggingface.co/blog/how-to-generate 78
which says: " Auto-regressive language generation is now available for  GPT2 ,  XLNet ,  OpenAi-GPT ,  CTRL ,  TransfoXL ,  XLM ,  Bart ,  T5  in both PyTorch and Tensorflow >= 2.0!"
so I wanted to try to do the same, they just change the model to T5. However even though the model runs, the output is very strange. This is the code:
!pip install transformers
import tensorflow as tf
from transformers import TFT5ForConditionalGeneration, T5Tokenizer

tokenizer = T5Tokenizer.from_pretrained("t5-small")
model = TFT5ForConditionalGeneration.from_pretrained("t5-small", pad_token_id=tokenizer.eos_token_id)
input_ids = tokenizer.encode('Hello, my dog is cute', return_tensors='tf')
greedy_output = model.generate(input_ids, max_length=50)
print("Output:\n" + 100 * '-')

print(tokenizer.decode(greedy_output[0], skip_special_tokens=True))

I get this output:
Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello
Because of this, and taking into account that I have not found many text-generation examples with t5, I would like to ask if this is possible? if so, why my output is so strange?

Context: My mistake, reading the documentation is required that the first token is the task, for example ‘summarize’.


[Question] Why does vocab size determine training parameters

Question:
I have a not so smart question: Why does the vocab size increase training parameters by a lot?
The following configuration:
from transformers import RobertaConfig
config = RobertaConfig(
    vocab_size=48000,
    max_position_embeddings=514,
    num_attention_heads=12,
    num_hidden_layers=6,
    type_vocab_size=1,
)

Passed to the model gives 80 mil parameters
model = RobertaForMaskedLM(config=config)
model.num_parameters() → 80.4 milion

If vocab_size is reduced to 16k,
model = RobertaForMaskedLM(config=config)
model.num_parameters() → 55 milion

The Embedding layers at the start increases training parameters so much?
        self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=config.pad_token_id)
        self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.hidden_size)
        self.token_type_embeddings = nn.Embedding(config.type_vocab_size, config.hidden_size)

The layer embeding takes:

Embedding(vocab_size, embedding_space, input_length=max_length))

Where in my case embedding_space is 768 (where 768*48000 - 768 * 16000 ~ 25 mil.)
Therefore the number of training parameters would be vocab_size * embedding_space?

Context: I’m not sure where you question is, all your math is correct and yes, the embedding matrix is responsible for a looot of the model parameters.


[HELP] Special tokens not appearing as predicted tokens!

Question:
Hello,
I require some urgent help!
I trained a masked language model on a Twitter dataset, with each tweet containing one emoji. Then, I used the following code to add the emojis as special tokens:
num_added_toks = tokenizer.add_tokens(['😃',
'😄',
'😁',
'😆',
'😅',
'😂',
'🤣',
'🧔🏿♂'])
print('We have added', num_added_toks, 'tokens')
model.resize_token_embeddings(len(tokenizer))  # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e. the length of the tokenizer

From adding the special tokens, I added 3311 different emojis successfully, which increased the embedding to (53575, 768) as shown below:
We have added 3311 tokens

Embedding(53575, 768)

Now, here’s the issue I am facing… When I add the <mask> token to a sentence and input the top_k as the total number of embeddings, which is 53575, not a single emoji shows up in the predictions.
I used this line of code:
mask_filler("Are you happy today <mask>", top_k=53575)

As you can see in the code above, the top_k is 53575, the total number of embeddings which should include the 3311 emojis I added, right?
However, when I make the predictions and scroll through the list of 53575, not a single emoji is there!
Why is this is happening? Like, I have added the emojis to the vocabulary, but they are simple not there when making predictions.
SEE FULL CODE HERE: MLM-EMOJIS/mlm_emojis.ipynb at main · saucyhambon/MLM-EMOJIS · GitHub 4

Context: I see one emoji in the last predictions. Mostly, it’s that the model is not used to seeing those, so it probably needs to be trained longer.


Question: Hi, I am new to NLP and trying to reproduce fine-tune results of Bert. However, the STS-B task troubles me, from what I understand, the STS-B task is a regression task, but Bert treats it as a classification task. I do not quite know the transformation between scores and labels in detail, is anybody willing to give me a hint?

Context: This is all dealt with in the loss function: a model that is tasked with classification or regression is the same roughly, it just outputs a different number of labels. Inside the code of BertModelForSequenceClassification 16, you can see there is a test that picks a different loss function depending on the problem_type, and by default 1 label (like in STS-B) corresponds to a regression, so the mean-squared error is selected as a loss, instead of cross-entropy.


How can I convert a model created with fairseq?

Question:
Hi,
I fine tuned facebook’s model mbart.cc25 for machine translation with Fairseq, it saved its model as checkpoint_*.pt. How can I use it now with Transformers, is it possible? Thanks

Context:
Unless the naming conventions that are used in transformers are the same as in fairseq, this is not possible out of the box. However, with a bit of digging you should be able to map them.
@sshleifer will know the answer.


Data format for BertForSequenceClassification with num_labels > 2

Question:
Hi,
I have a multilabel task (num_labels=8) and I want to use BertForSequenceClassification using Trainer to train the model.
But I get the following error:

ValueError: Expected input batch_size (8) to match target batch_size (64).

I assume that the problem is the data format of the labels. Currently, my label is a 8-dim list (e.g., [1,0,0,0,0,1,0,0]).
What is the right format for the label data?
Here my code:

class EmotionDataset(torch.utils.data.Dataset):
def init(self, encodings, labels):
self.encodings = encodings
self.labels = labels
def __getitem__(self, idx):
    item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
    item['labels'] = torch.tensor(self.labels[idx])
    return item

def __len__(self):
    return len(self.labels)

MODEL_NAME = ‘dbmdz/bert-base-german-uncased’
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = BertForSequenceClassification.from_pretrained(MODEL_NAME, num_labels=8)
tokenize data
dataset_train = Dataset.from_pandas(df_train)
train_encodings = tokenizer(dataset_train['text], truncation=True, padding=True)
train_dataset = EmotionDataset(train_encodings, dataset_train['label])
training_args = TrainingArguments(
output_dir=’./results’,          # output directory
num_train_epochs=1,              # total # of training epochs
per_device_train_batch_size=8,  # batch size per device during training
per_device_eval_batch_size=32,   # batch size for evaluation
warmup_steps=500,                # number of warmup steps for learning rate scheduler
weight_decay=0.01,               # strength of weight decay
logging_dir=’./logs’,            # directory for storing logs
)
trainer = Trainer(
model=model,                         # the instantiated  Transformers model to be trained
args=training_args,                  # training arguments, defined above
train_dataset=train_dataset,         # training dataset
eval_dataset=test_dataset            # evaluation dataset
)
_ = trainer.train()
trainer.evaluate()

Thanks,
Max

Context: Hi @maxpower, I think the format of your dataset is fine but I think you have to change the model’s loss function to use a sigmoid instead of a softmax on the logits (i.e. BCEWithLogitsLoss). You can see a skeleton + hacky Colab in this thread: Fine-Tune for MultiClass or MultiLabel-MultiClass - #8 by lewtun 100


How to move huggingface’s pretrained model to another machine?

Question:
Hi everyone, I’m having two computers, let’s call them A and B.
On my computer A, I have my huggingface’s textattack/bert-base-uncased-imdb model already downloaded.
My computer B, however, does not have access to the internet and cannot download the model.
A and B can communicate through ssh. Now I’m trying to move the downloaded model from A to B by moving the entire ~/.cache/huggingface to B, but when I run my code on computer B, it still attempts to download the model from the internet.
Maybe the pretrained model in A is located somewhere else? How do I move my model from A to B?
Thanks.

Context: Computer B is still trying to access the internet to check if there is a new version of the model you are using. You have to set the environment variable TRANSFORMERS_OFFLINE to some truthy value (like yes)


Question: And how do I close my account? Thanks.

Context: cc @pierric


Question:
RobertaTokenizerFast seems to be ignoring my Lowercase() normaliser. I’ve created a custom tokeniser as follows:
tokenizer = Tokenizer(BPE(unk_token="<unk>", end_of_word_suffix="</w>"))
tokenizer.normalizer = Lowercase()
tokenizer.pre_tokenizer = Sequence([Whitespace(), Digits(individual_digits=False), Punctuation()])

trainer = BpeTrainer(
    vocab_size=3000,
    special_tokens=["<s>", "</s>", "<unk>", "<pad>", "<mask>"]
)
tokenizer.train(trainer, files)
tokenizer.post_processor = RobertaProcessing(
    cls=("<s>", tokenizer.token_to_id("<s>")),
    sep=("</s>", tokenizer.token_to_id("</s>")),
)
tokenizer.decoder = BPEDecoder(suffix="</w>")

(I’m not 100% sure if the BPE suffix is required?)
I then save as
tokenizer.model.save("./models/roberta")
tokenizer.save("./models/roberta/config.json")

The reason I need a custom tokeniser is my examples aren’t white-space delimited, i…e
tokenizer.encode("AHU-01-SAT").tokens

[’<s>’, ‘ahu’, ‘-’, ‘01’, ‘-’, ‘sat’, ‘</s>’]

The following doesn’t return the correct tokens:
from transformers import RobertaTokenizerFast
tokenizer = RobertaTokenizerFast.from_pretrained(".models/roberta", max_len=512)
tokenizer("AHU-01-SAT")

{‘input_ids’: [0, 40, 112, 40, 1], ‘attention_mask’: [1, 1, 1, 1, 1]}

It’s missing the first and last tokens (plus it’s not even replacing them with “”?
If I manually apply the normalisation I get the correct tokenisation
tokenizer("ahu-01-sat")

{‘input_ids’: [0, 109, 40, 112, 40, 598, 1], ‘attention_mask’: [1, 1, 1, 1, 1, 1, 1]}
I tried AutoTokenizer and observed the same issue - am I doing something wrong?

Context:
Ahh, figured it out, it should be
tokenizer.save("./models/roberta/tokenizer.json")

not
tokenizer.save("./models/roberta/config.json")


Which encoding does GPT2 vocabulary file use?

Question: I wanted to investigate a little, how tokenizer works. But the model trained in Russian language (mostly), and I see garbage instead of tokens. I tried obvious variants like open in UTF-8, that didn’t help.

Context:
Well, I checked that json vocabulary loads as gibberish:
import json

with open("/content/notebooks/ru-gpts/models/gpt3large/vocab.json", "r", encoding="utf-8") as f:

  vocab = json.load(f)

list(vocab.items())[1000:1010]

[('ÑĥÐ¼Ð°', 1000),
 ('ĠÐ¿Ð¸', 1001),
 ('Ġn', 1002),
 ('ĠÐ½ÐµÑĤ', 1003),
 ('Ð¸ÑĤÐ°', 1004),
 ('ÑĢÑĥÐ¿', 1005),
 ('ec', 1006),
 ('ÐµÐ½Ñĭ', 1007),
 ('ĠÑıÐ²', 1008),
 ('Ðĵ', 1009)]

The same exact gibberish is in tokenizer itself, which I view with list(tok1.get_vocab().items())[1000:1010]
Yet, the model seems to work somehow, and produce meaningful results in Russian.


Are images generated by DALL·E mini free to use?

Question: Hi, I found DALL·E mini via Twitter. My question is in the topic title: May I use images generated from my prompts in DALL·E mini? Thanks.

Context: cc @boris who may know more about the licensing of dall-e’s outputs (my guess is it’s fine!)


How to save, load and use my text classification model?

Question:
Hi,
I have followed this text classification tutorial: notebooks/text_classification.ipynb at master · huggingface/notebooks · GitHub 4
However, the tutorial does not show how to save, load and use my text classification model.
Any help on how to use my saved model for classifying a sentence?
Thanks!

Context: Once you have pushed your model to the hub, you can use it like any other model, in a pipeline or directly with the from_pretrained method.


Question:
Hello,
I’m working thru the Question Answering with SQuAD 2.0 module.
I’m stuck in this function:
def add_token_positions(encodings, answers):
    start_positions = []
    end_positions = []
    for i in range(len(answers)):
        start_positions.append(encodings.char_to_token(i, answers[i]['answer_start']))
        end_positions.append(encodings.char_to_token(i, answers[i]['answer_end'] - 1))

        # if start position is None, the answer passage has been truncated
        if start_positions[-1] is None:
            start_positions[-1] = tokenizer.model_max_length
        if end_positions[-1] is None:
            end_positions[-1] = tokenizer.model_max_length

    encodings.update({'start_positions': start_positions, 'end_positions': end_positions})

add_token_positions(train_encodings, train_answers)
add_token_positions(val_encodings, val_answers)

and wondering what
    encodings.update({'start_positions': start_positions, 'end_positions': end_positions})
is actualing doing.
I don’t find documentation or helpful information for this. Is there any information available?
And how can I see the result of this function?
Thanks in advance!

Context: This is a Python method for any dictionary, it adds the content of the dictionary passed (so here the "start_positions" and "end_positions") to the encodings.


How can i use torch.optim.lr_scheduler.MultiStepLR with Trainer?

Question: Is there any way to change learning rate scheduler by using Pytorch’s MultiStepLR with Trainer?

Context: You can pass your own optimizer and scheduler to the Trainer. See the documentation 11 for more information.


Question:
Hello. I’m trying to use multiprocessing when generating summaries on text within a data frame. The pool.map() command just hangs when I run it on a custom generate function. I tried debugging by removing every line of the generate function (shown below) and it seems to work fine if I remove the model.generate() part.
Is this not the way to run multiprocessing on the generate function? Or is multiprocessing on CPUs not possible with the generate function?
import torch.multiprocessing as mp
def generate(i):

  content = df.loc[i, 'content_web']
  location = df.loc[i, 'location']
  content = content + '. ' + location

  tokenized_text = tokenizer(content, truncation=True, padding=True, return_tensors='pt')
  source_ids = tokenized_text['input_ids'].to(device, dtype = torch.long)
  source_mask = tokenized_text['attention_mask'].to(device, dtype = torch.long)

  generated_ids = model.generate(
      input_ids = source_ids,
      attention_mask = source_mask,
      max_length=512,
      min_length=50,
      num_beams=4,
      repetition_penalty=2.5,
      length_penalty=2.0,
      early_stopping=True,
      no_repeat_ngram_size=8,
  )

pool = mp.Pool(processes=4)
results = pool.map(generate2, (range(0, 2)))

Context: I have the same issue for prediction with AutoModelForSequenceClassification. pool.map() just hangs, while a Python map works fine.


Question:
I want to fine-tune T5 model. but there is issue in running this script.
my code:-
!python /content/transformers/examples/language-modeling/run_clm.py
–model_name_or_path t5-base
–train_file /content/train.txt
–do_train
–learning_rate=1e-4
–per_device_train_batch_size=4
–output_dir /tmp/test-clm
This code is not working. How can I fine-tune the t5-base model?

Context:
Please use Preformatted text block for your codes, you can create this block automatically by pressing Ctrl+E:
!python /content/transformers/examples/language-modeling/run_clm.py
–model_name_or_path t5-base
–train_file /content/train.txt
–do_train
–learning_rate=1e-4
–per_device_train_batch_size=4
–output_dir /tmp/test-clm

Can you share any kind of error or unexpected output with us?


Question:
I have a pytorch model with BertModel as the main part and a custom head.  I want to freeze the embedding layer and the first few encoding layers, so that I can fine-tune the attention weights of the last few encoding layers and the weights of the custom layers.
I tried:
ct = 0
for child in model.children():
ct += 1
if ct < 11:          # ########## change value - this freezes layers 1-10
    for param in child.parameters():
        param.requires_grad = False

but I’m not sure that did what I want.
I then ran this to check, but the layer names aren’t recognized

print(L1bb.embeddings.word_embeddings.weight.requires_grad)
print(L1bb.encoder.layer.0.output.dense.weight.requires_grad)
print(L1bb.encoder.layer.3.output.dense.weight.requires_grad)
print(L1bb.encoder.layer.6.output.dense.weight.requires_grad)
print(L1bb.encoder.layer.9.output.dense.weight.requires_grad)
print(L1bb.pooler.dense.weight.requires_grad)
print(L4Lin.requires_grad)

L1bb is the name of the BertModel section in my model, and L1bb.embeddings.word_embeddings.weight is shown in the output of the code that instantiates the model.
How can I freeze the first n layers?
What counts as a layer?
What are the names of the layers in BertModel?
How can I check which layers are frozen?
PS how can I format this pasted code as code in the forum post? One section has done it automatically, but nothing seems to affect it.

Context:
You should not rely on the order returned by the parameters method as it does not necessarily match the order of the layers in your model. Instead, you should use it on specific part of your models:
modules = [L1bb.embeddings, *L1bb.encoder.layer[:5]] #Replace 5 by what you want
for module in mdoules:
    for param in module.parameters():
        param.requires_grad = False

will freeze the embeddings layer and the first 5 transformer layers.


Ideas for beginner-friendlier TPU-VM clm training

Question:
Hello All,
I’ve recently started trying out TPU-VMs and wanted to train distilgpt2 from scratch on an non English language. I’ve had a rather rough start but did manage to overcome and get the training going.
Following the advice from Suraj Patil 3, I decided to write a list of things which can make this experience a bit smoother for others.
If you too think these items might help, perhaps some could be added to the relevant tutorials, docs or even the example code itself.
GIT credentials for Huggingface

Perhaps explain / demonstrate how to store and cache them? (Extra useful if the model is “Private”)
Because I did not cache properly at first, and because I had junk in the keyboard buffer, at some point when the script wanted to “Push to hub”, it failed. Now it seems like it does not retry, nor did  ‘run_clm_flax.py’ saves  the checkpoint to disk when such an error happen, so basically all the hours of training were lost.
I’d suggest that if the training scripts have failed auth for some reason, they should not exit before it has written the checkpoint to disk.
Is it possible to somehow make sure that we have cached GIT credentials when the training starts?

Eval phase

Same as above, in case of an error, do not exit before you write the checkpoint to avoid loss of training time

Log

Is there a built-in option to have all the “on screen” data be written to a log file? - If so, I couldn’t find it
Why are all of those “tcmalloc” lines being printed (even on “Info” log level)? Is there a way to have them not print? - It really clutters the screen to the point where it’s hard to find the actual log prints.
I “worked around” it by piping everything to grep -v “tcmalloc” but, is there a better solution?

dtype

float16 - Doesn’t really work well with TPU, right ? Perhaps not allow it if training with JAX?
bfloat16 - Can it convert to pytorch? Only after I had my first checkpoint, I discovered that I can not use Flax models it with the online inference-box and that AutoModelForCausalLM fails to load bfloat16 ones
float32 - I decided to re-start with float32 as it was loaded well by AutoModelForCausalLM and was saved well as pytorch. Is this the only/best option if I want to have my FLAX model also as pytorch?

Upgrade CLU

Somehow I missed this upgrade step (pip install --upgrade clu) which resulted in a lot of weird errors and malloc failures.
If it’s not just me, which it totally might be  Perhaps it should be more  prominent?

Anyway,
This is my feedback,
I hope someone finds it useful.
Great work everyone!

Doron Adler
@Norod78

Context:
Follow this tutorial 7 to set-up and connect to your TPU-VM

Add your local bin path to the PATH environment variable, If you do not know your local user name, type:
whoami
#In my case, the user is ‘dadler’, so replace ‘dadler’ in the following block with your own user:
nano ~/.bashrc
#Add the following line at the bottom, replace dadler with your own user name
export PATH="/home/dadler/.local/bin:$PATH"
#Save (CTRL O) #Exit (Ctrl X)
#Realod bashrc
source ~/.bashrc

Install and upgrade libraries

pip install datasets
git clone https://github.com/huggingface/transformers.git
sudo pip install --user -e transformers
pip install --upgrade tokenizers
pip install --upgrade clu
git clone https://github.com/google/flax.git
sudo pip install --user -e flax
pip install git+https://github.com/deepmind/optax.git

Setup git-lfs

sudo apt install git-lfs
git lfs install

Login to your huggingface account
huggingface-cli login

Save your git credentials on the local VM (Not secure, do this only if you are the only person who has access the TPU-VM instance)

git config --global credential.helper 'store --file ~/.git-credentials'
git credential fill

#Type/Paste the following two lines:
protocol=https
host="huggingface.co"

#Now hit Enter until you are prompt to enter your huggingface user and password

N̶o̶t̶ ̶s̶u̶r̶e̶ ̶i̶f̶ ̶t̶h̶i̶s̶ ̶i̶s̶ ̶n̶e̶e̶d̶e̶d̶,̶ ̶b̶u̶t̶ ̶i̶n̶ ̶t̶h̶e̶ ̶t̶e̶r̶m̶i̶n̶a̶l̶ ̶s̶e̶s̶s̶i̶o̶n̶ ̶y̶o̶u̶ ̶a̶r̶e̶ ̶a̶b̶o̶u̶t̶ ̶t̶o̶ ̶r̶u̶n̶ ̶y̶o̶u̶r̶ ̶t̶r̶a̶i̶n̶i̶n̶g̶ ̶s̶c̶r̶i̶p̶t̶ ̶i̶n̶,̶ ̶y̶o̶u̶ ̶m̶i̶g̶h̶t̶ ̶w̶a̶n̶t̶ ̶t̶o̶ ̶t̶y̶p̶e̶:̶
̶̶e̶x̶p̶o̶r̶t̶ ̶X̶R̶T̶_̶T̶P̶U̶_̶C̶O̶N̶F̶I̶G̶=̶"̶l̶o̶c̶a̶l̶s̶e̶r̶v̶i̶c̶e̶;̶0̶;̶l̶o̶c̶a̶l̶h̶o̶s̶t̶:̶5̶1̶0̶1̶1̶"̶̶

Continue by following the instructions in this tutorial 6

Hope it helps


Custom Distilbert does not use CUDA for predition

Question:
Hi,
I am using the model classifier = pipeline(“text-classification”,model=‘bhadresh-savani/distilbert-base-uncased-emotion’, return_all_scores=True) for emotion classification of my datatset of 1.2m client feedbacks. I am not training the model, I am just doing a prediction. I noticed that the model uses CPU and does not use CUDA (I have RTX5000) and the prediction takes ages to compute.
Can you explain why it is the case? Is there a way to use CUDA for this model predictions?
Thank you
Krzysztof

Context: I found a solution, add parameter device=0. But I have to classify in small batches as the GPU RAM is a serious limit, even my GPU has 16 GB


Trying to Build Datasets, Random Items Get Added

Question:
Hi all,
I’m currently trying to load in fastai’s version of the IMDB dataset, to learn how to build a Dataset from a folder of .txt's. I’m preparing my data with the following:
# Downloading the dataset
from fastai.data.external import untar_data, URLs
from fastai.data.transforms import get_files

path = untar_data(URLs.IMDB, dest='./IMDB')

From there I can get all the training .txt's with:
texts = get_files(path/'train', extensions='.txt')
texts = [str(t) for t in texts]

In turn, this is a list of 25,000 text files. However, when I use the load_dataset api to bring this in, suddenly my dataset has 25,682 items! Can anyone help me figure out why? This is an issue as I need to use add_column to add in a label, and there is a mismatch between the number of actual training items vs the ones Datasets picked up. Here is how I’m building the dataset:
dset = load_dataset('text', data_files={'train':texts})

TIA!

Context:
I think it’s because the “text” loader creates a new sample for each “\n” it sees, so the texts you have that contain some of those are then split into several samples. @lhoestq or @albertvillanova if you could conifrm?
PS: it would be easier to just do dsets = load_dataset('imdb') :-p


Evaluation became slower and slower during Trainer.train()

Question:
When I used Trainer.train() to fine-tune BartBase, I found something weird that the speed shown in progress bar became slower and slower (from 6 item/s to 0.29 item/s. Please help me, I’m new to transformers.
Here are my codes.
training_args = TrainingArguments(
    output_dir="Model/BartBase",
    overwrite_output_dir=True,

    per_device_train_batch_size=8,
    per_device_eval_batch_size=16,
    learning_rate=1e-5,
    num_train_epochs=20,
    lr_scheduler_type='linear',
    label_smoothing_factor=0,

#     logging_dir='runs',
    logging_strategy='steps', # log according to log_steps
    logging_steps=1,

    save_strategy='steps', # log according to save_steps
    save_steps=4000,
    save_total_limit=10, # limit the total amount of checkpoints

    evaluation_strategy="steps", # log according to eval_steps
    eval_steps=1, # I set eval_steps=1 to debug
    eval_accumulation_steps=1,

    seed=42,

    load_best_model_at_end=True, # load best model according to metric_for_best_model
    metric_for_best_model='f1' # the string should be
    )

from datasets import load_metric
import numpy as np

def compute_metrics(eval_pred):
    f1_metric = load_metric('f1')
    accuracy_metric = load_metric('accuracy')
    pred, label = eval_pred
    pred = np.argmax(pred, axis=-1)
    f1_score = f1_metric.compute(predictions=pred, references=label, average='micro')
    accuracy = accuracy_metric.compute(predictions=pred, references=label)
    return f1_socre.update(accuracy)

from transformers import Trainer
trainer = Trainer(
    model=model,
    tokenizer=tokenizer,
    args=training_args,
    data_collator=collator, # if tokenizer is provided, no need to provide it explicitly

    train_dataset=train_dataset, # torch.utils.data.dataset.Dataset
    eval_dataset=eval_dataset,
    compute_metrics=compute_metrics
)

trainer.train()

Context:
After debugging step by step, I found that

If I remove the compute_metrics=compute_metrics in Trainer, the evaluation went well.
Even if I use a quite simple compute_metrics, the evaluation became slow and stopped eventually (without finishing progress) .def compute_metrics(eval_pred):
      return {'f1': 1}

Please give me some helps. Thanks a lot!!!


Passing additional tensors into metric evaluation function

Question:
Hi,
I was wondering if it is possible to pass additional labels into metric evaluation function. In more details, I want to calculate metrics using not all predicted tokens but only a small subset, and to do this I need to pass a tensor of indices of that tokens into the metric evaluation function.  But I don’t need this tensor for anything else, e.g. for calculating the loss.
I tried to add this tensor into evaluation dataset, using label_names and remove_unused_columns = False options. But in this case the model (T5 encoder-decoder) sees unrecognized keywords and fails. Default remove_unused_columns = True  just deletes the tensor from the batch.
I am using  Transfomers version 4.7.0
Any help is much appreciated! Thanks.

Context: If you leave it in your dataset, you will have an error since your model can’t consume it. You will need to write a manual evaluation loop for this use case.


Error while training a custom pretrained model

Question:
Hi,
I trained a model as follows:
checkpoint = “bert-base-uncased”
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=5)
After 3 epochs I wanted to finetune this model again with the following code (I tried all three versions):
#1
model = BertForSequenceClassification.from_pretrained(“C:/Users/THINK/Dysk Google/_Priv/_Courses/Huggingface/Fine_tuned_models/rmp_eval_bert_base_uncased/”)
#2
model = BertModel.from_pretrained(‘C:/Users/THINK/Dysk Google/_Priv/_Courses/Huggingface/Fine_tuned_models/rmp_eval_bert_base_uncased/’)
#3
config = BertConfig.from_json_file(‘C:/Users/THINK/Dysk Google/_Priv/_Courses/Huggingface/Fine_tuned_models/rmp_eval_bert_base_uncased/config.json’)
model = BertModel.from_pretrained(‘C:/Users/THINK/Dysk Google/_Priv/_Courses/Huggingface/Fine_tuned_models/rmp_eval_bert_base_uncased/’, config=config)
But in each case I got the error:
TypeError: forward() got an unexpected keyword argument ‘labels’
The input format is the same as before.
The model was saved as:
output_dir = ‘C:/Users/THINK/Dysk Google/_Priv/_Courses/Huggingface/Fine_tuned_models/rmp_diff_bert_base_uncased’
model.save_pretrained(output_dir)
Can you help me, what code I should use to load a custom pretrained bert_base_uncased model?
Thank you.

Context: It’s logical you got that error for the last two lines since BertModel does not accept a labels argument. I don’t think you had that exact error for the first one as it does accept a labels argument.


Question:
I am very new to nlp so sorry for the question.
I am wondering if there a name for this task and if I can do it in deep learning or even in huggingdace
Let say I have some sentences, for example:
s1:  I have a dog
s2: I am new to this,
I convert my sentences to  embedded features and now I have an inputs with shape of 1x2048x4 for s1 and 1x2048x5 for s2.
Now I want to classify each word to a category out of 10 categories, so the output will be like:
o1: 1x4x10
and
o1: 1x5x10
the dependency between the word in each sentence is important in the classification. it is kinda like token classification but the labels are not representing the token, but instead the feeling that that word will imply to the sentence.
so the length of input may change and the length of output will change as well, but classes are the same.
Can I do it with transformers and self-attention models? should I look for any specific model or task?
Sorry for the boring question, nlp newbie here with a lot of passion

Context: I’m not sure that I understood your question entirely. But classifying words like you said, would be close to a Named Entity Recognition task I think. BERT like models can produce word embeddings for input sentences, which means separate embedding vector for each word. Maybe you could use such model for your task.


How to specify different batch sizes for different GPUs when training with rum_mlm.py?

Question:
Hi,
I’m using run_mlm.py to train a custom BERT model. I have two GPUs available. One has 24GB of memory and the other has 11 GB of memory. I want to use the batch size of 64 for the larger GPU and the batch size of 16 for the smaller GPU. How can I do so? The --per_device_train_batch_size parameter only takes one number. Or can I just give the combined batch size (80) and let the script figure out how to split the data between GPUs?
Thanks!

Context: This use case is not supported by the Trainer API, it would require custom scripts (one per GPU) to work, and even then, I’m not sure you will see any speed gain by training on two different GPUs that are not of the same type.


Question: Could someone give some insight to the “model.compute_loss” function which is used when fine-tuning the models without the trainer API (e.g- Keras native training). How does it work (i.e.- for different tasks how it’s adapted.)? It says that it takes the first element of the output, how is the loss calculated then?

Context: as shown in here in Huggingface documentation, Fine-tuning with custom datasets — transformers 4.7.0 documentation 1


BERT: AttributeError: ‘RobertaForMaskedLM’ object has no attribute ‘bert’

Question:
I am trying to freeze some layers of my masked language model using the following code:
for param in model.bert.parameters():
    param.requires_grad = False

However, when I execute the code above, I get this error:
AttributeError: 'RobertaForMaskedLM' object has no attribute 'bert'

In my code, I have the following imports for my masked language model, but I am unsure what is causing the error above:
from transformers import AutoModelForMaskedLM
model = AutoModelForMaskedLM.from_pretrained(model_checkpoint)

So far, I have tried to replace bert with model in my code, but that did not work.
Any help would be good.
Thanks.

Context: The name of the body of the model roberta for Roberta models, not bert. So you should loop on for param in model.roberta.parameters(). In general, the attribute that is model agnostic is base_model, so for param in model.base_model.parameters() should work anywhere.


Accessing uncontextualized BERT word embeddings

Question: Hi there! Once I’ve imported a BERT model from HuggingFace, is there a way to convert a sequence of encoded tokens into BERT’s raw embeddings without contextualizing them using self-attention, or otherwise extract the raw embedding for a given token?

Context:
Try this.
I think the apis change a bit between models so take a look before you copy paste
            model = DistilBertForTokenClassification.from_pretrained(
                "distilbert-base-cased", num_labels=self.num_labels
            )
            word_embeddings = model.distilbert.embeddings.word_embeddings(["my token ids here"])
            word_embeddings_with_positions = model.distilbert.embeddings(["my token ids here"])


Converting TF-Bert to Torch using conversion script works, but

Question:
Hi there - we are using this BERT architecture from Google:
* attention_probs_dropout_prob:0.1
* hidden_act:"gelu"
* hidden_dropout_prob:0.1
* hidden_size:768
* initializer_range:0.02
* intermediate_size:3072
* max_position_embeddings:512
* num_attention_heads:12
* num_hidden_layers:12
* type_vocab_size:2
* vocab_size:32000

we trained it from scratch with 10 millions of documents from our very specific domain and also changed optimizers and sentence tokenizer. Now, our BERT works wonderfully, it is evaluated on mask_token and next sentence prediction and we can also fine tune it for e.g. downstream Classfication tasks - all of that works. As long as we stay in the TensorFlow/Nvidia world.
We would however love to move into all the possibilities of PyTorch as well. So we applied the following script: " convert_bert_original_tf_checkpoint_to_pytorch.py 2"
This script initially fails, because our optimizer are not recognized.  By adding them to the list of skipped attributes
if any(n in ["adam_v", "adam_m", "*AdamWeightDecayOptimizer", "AdamWeightDecayOptimizer_1", "global_step"*] for n in name):

The scripts runs through fine and we get a working torch model. And here things become very strange. Initially we tried to fine tune the torch model of our BERT and nothing happened - inspecting the attention values and outputs for input tokens we realized that for any given input sequence ALL attention values are identical, yet different when the input changes. Also the output values in the hidden stated of the converted BASE model are identical for each token, irrespective of the input.
Nevertheless the conversion seems to work in that way that the model can be loaded and is a fully valid Pytorch model file.
I understand that all of that is quite vague but maybe it sounds familiar to somebody and or we get a hint where to look next.
The manifestation of that effect:
outputs = torch_bert_model(**example_input)
print(outputs[0]) # last hidden state, irrespective of input always this:

tensor([[[ 1.5048e-04, -4.1375e-06,  1.4493e-04,  ...,  2.1820e-04,
          -6.6411e-05,  2.1333e-04],
         [ 1.5048e-04, -4.1375e-06,  1.4493e-04,  ...,  2.1820e-04,
          -6.6411e-05,  2.1333e-04],
         [ 1.5048e-04, -4.1375e-06,  1.4493e-04,  ...,  2.1820e-04,
          -6.6411e-05,  2.1333e-04],
         ...,
         [ 1.5048e-04, -4.1375e-06,  1.4493e-04,  ...,  2.1820e-04,
          -6.6411e-05,  2.1333e-04],
         [ 1.5048e-04, -4.1375e-06,  1.4493e-04,  ...,  2.1820e-04,
          -6.6411e-05,  2.1333e-04],
         [ 1.5048e-04, -4.1375e-06,  1.4493e-04,  ...,  2.1820e-04,
          -6.6411e-05,  2.1333e-04]]], grad_fn=<NativeLayerNormBackward>)

And the attention heads:
4=CLS, 5=SEP
Always different for any given input but always the identical value on all heads.
tensor([[   4,   13,    8, 6060,    5,   13, 2840,  350,    8, 6060,    5]])
(tensor([[[[0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          ...,
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909]],

         [[0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          ...,
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909]],

         [[0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          ...,
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909]],

         ...,

         [[0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          ...,
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909]],

         [[0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          ...,
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909]],

         [[0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          ...,
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909]]]],
       grad_fn=<SoftmaxBackward>), tensor([[[[0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],
          [0.0909, 0.0909, 0.0909,  ..., 0.0909, 0.0909, 0.0909],

Thanks a lot for any help! Much appreciated.

Context: Just in case someone runs into the same issue: It was a problem with the naming of the layers  our naming came from an Nvidia TF package and it is differed from standard naming - we did the mapping ourselves and now the model is working and producing the same output for identical input. the script was still useful to see how it is done in principle.


Question:
Hello,
I want to train a reformer for a sequence classification task. The sequences are of protein so I thought of making a new tokenizer and then loaded as a reformer tokenizer which is defined as below.
spm.SentencePieceTrainer.train(input='./sequences_scope.txt', model_prefix='REFORM', max_sentence_length=2000, vocab_size=25)

tokenizer = ReformerTokenizer("REFORM.model", padding=True)
tokenizer.add_special_tokens({'pad_token': '[PAD]'})

The dataset was created as below -
with open("sequences_class_scope.csv", "w") as fp:
  fp.write("idx,sequence,label\n")
  for n,i in enumerate(sequences):
    fp.write(str(n)+","+i+","+str(labels[n])+"\n")

dataset = load_dataset('csv', data_files='sequences_class_scope.csv', split='train[:60%]')

And the model was then defined as follows -
config = ReformerConfig(
    vocab_size=25,
    max_position_embeddings=2000,
    num_attention_heads=12,
    num_hidden_layers=6,
)
model = ReformerForSequenceClassification(config=config)

data_collator = DataCollatorForTokenClassification(
    tokenizer=tokenizer, max_length = 2000
)
training_args = TrainingArguments(
    output_dir="./pREFORMo",
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_gpu_train_batch_size=64,
    save_steps=10_000,
    save_total_limit=2,
    prediction_loss_only=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
)

trainer.train()

I am currently getting the error ValueError: You should supply an encoding or a list of encodings to this method that includes input_ids, but you provided ['label'] and if I map my dataset to include theinput_ids and attention_masks, I get the error
/usr/local/lib/python3.7/dist-packages/transformers/data/data_collator.py in <listcomp>(.0)
    186         padding_side = self.tokenizer.padding_side
    187         if padding_side == "right":
--> 188             batch["labels"] = [label + [self.label_pad_token_id] * (sequence_length - len(label)) for label in labels]
    189         else:
    190             batch["labels"] = [[self.label_pad_token_id] * (sequence_length - len(label)) + label for label in labels]

TypeError: object of type 'int' has no len()

Also, my end goal is to train the reformer and then use it to generate embeddings rather than classification. Is this the correct approach to do so?

Context:
hey @choke what does an example in your tokenized inputs look like?
from the error, i think you could try renaming the target column to labels which is what the Trainer expects by default


How does BERT know which contextualised embedding to choose for a word?

Question:
Hi!
I am trying to explain BERT. I understand the concept of contextualised embedding, where one word has different embeddings depending on the context. I also understand that when by using bidirectionality, BERT can learn these contextualised embedding during pretraining.
My question is when finetuning BERT for a task and feeding it a sentence, how does BERT know which contextualised embedding to used for the word, given there are several to choose from?

Context: I am new to the topic of Transformers, but my understanding (limited as it is) of BERT is that whilst the pre-trained embedding is created through semi-supervised learning (just large unannoted text corpora), when it comes to fine-tuning BERT for a specific task then this is usually a supervised learning process. So effectively you tell it what the right answers are and as part of the learning process in fine-tuning it will learn which contextualised embedding (and maybe a lot of other related knowledge) is relevant to what you want it to do.


Question:
I’m instantiating a model with this
tokenizer = AutoTokenizer.from_pretrained("nlptown/bert-base-multilingual-uncased-sentiment")
model = AutoModelForSequenceClassification.from_pretrained("nlptown/bert-base-multilingual-uncased-sentiment")

Then running a for loop to get prediction over 10k sentences on a G4 instance (T4 GPU). GPU usage (averaged by minute) is a flat 0.0%. What is wrong? How to use GPU with Transformers?

Context: This like with every PyTorch model, you need to put it on the GPU, as well as your batches of inputs.


Small Doubt :- Is there any mistake while finding results from DeBERTa model for SNLI Dataset

Question:
Hi everyone.
from transformers import DebertaTokenizer, DebertaForSequenceClassification
import torch

max_length = 512

premise = "I do not love you"
hypothesis = "I love you"

hg_model_hub_name = "microsoft/deberta-base-mnli"

tokenizer = DebertaTokenizer.from_pretrained(hg_model_hub_name)
model = DebertaForSequenceClassification.from_pretrained(hg_model_hub_name)

tokenized_input_seq_pair = tokenizer.encode_plus(premise, hypothesis,
                                                  max_length=max_length,
                                                  return_token_type_ids=True, truncation=True)

input_ids = torch.Tensor(tokenized_input_seq_pair['input_ids']).long().unsqueeze(0)
# remember bart doesn't have 'token_type_ids', remove the line below if you are using bart.
token_type_ids = torch.Tensor(tokenized_input_seq_pair['token_type_ids']).long().unsqueeze(0)
attention_mask = torch.Tensor(tokenized_input_seq_pair['attention_mask']).long().unsqueeze(0)

outputs = model(input_ids,
                attention_mask=attention_mask,
                token_type_ids=token_type_ids,
                labels=None)
# Note:
# "id2label": {
#     "0": "entailment",
#     "1": "neutral",
#     "2": "contradiction"
# },

predicted_probability = torch.softmax(outputs[0], dim=1)[0].tolist()  # batch_size only one

print("Premise:", premise)
print("Hypothesis:", hypothesis)
print("Entailment:", predicted_probability[0])
print("Neutral:", predicted_probability[1])
print("Contradiction:", predicted_probability[2])

The output is :-
Premise: I do not love you
Hypothesis: I love you
Entailment: 0.9993366599082947
Neutral: 0.0004206844314467162
Contradiction: 0.00024267268599942327

While calculating accuracy using above code on SNLI Dataset I am getting 29% which should not be…
Resluts from SNLI dataset
              precision    recall  f1-score   support

           0       0.04      0.04      0.04      3329
           1       0.82      0.81      0.82      3235
           2       0.02      0.03      0.02      3278

    accuracy                           0.29      9842
   macro avg       0.30      0.29      0.29      9842
weighted avg       0.29      0.29      0.29      9842

Can anybody guide me where i am doing mistake or how should i approach to find the entailment, neutral, contradiction prediction from DeBERTa model
@DeBERTa  @lewtun  can you please help??

Context:
hey @akshat-suwalka i think the reason why you’re getting a much lower score on the snli dataset is due to a misalignment between the label → label_id mappings in the model and dataset.
to explain what i mean, note that the config.json of the deberta model has the following mappings:
  "id2label": {
    "0": "CONTRADICTION",
    "1": "NEUTRAL",
    "2": "ENTAILMENT"
  },
  "label2id": {
    "CONTRADICTION": 0,
    "ENTAILMENT": 2,
    "NEUTRAL": 1
  }

while the snli dataset has the contradiction and neutral labels swapped:
id2label = {0: 'entailment', 1: 'neutral', 2: 'contradiction'}
label2id = {'contradiction': 2, 'entailment': 0, 'neutral': 1}

To fix this you can use the Dataset.align_labels_with_mapping function (docs):
hg_model_hub_name = "microsoft/deberta-base-mnli"

model = DebertaForSequenceClassification.from_pretrained(hg_model_hub_name)
config = model.config

snli = load_dataset("snli")
snli_aligned = snli.align_labels_with_mapping(label2id=config.label2id, label_column="label")

hth!


Question:
Hi, it is glad to find the behavior of “Trainer” can be customized by overriding its methods. However, I am facing a problem with the originally existed functions. For example:
def training_step(self, model: nn.Module, inputs: Dict[str, Union[torch.Tensor, Any]]) -> torch.Tensor:
    ...
    if is_sagemaker_mp_enabled():
            scaler = self.scaler if self.use_amp else None
            loss_mb = smp_forward_backward(model, inputs, self.args.gradient_accumulation_steps, scaler=scaler)
            return loss_mb.reduce_mean().detach().to(self.args.device)
    ...
    loss = self.compute_loss(model, inputs)
    print(loss) # This is the only place I want to change
    return loss.detach()

This is the part of the code of method “training_step”, which I want to rewrite. Suppose I just want to print the loss in each training step without changing other codes. But apparently, I cannot import the function “is_sagemaker_mp_enabled()”, and thus I have to delete them. I don’t think this is a good solution and is there any elegant way?
Thanks for the help!

Context: There is no reason you shouldn’t be able to import is_sagemaker_mp_enabled from its location (transformers.file_utils).


Compare the likelihood of various sentences in a LM?

Question:
Hi!
I have the beginning of a sentence (2-3 words) and couple candidate full sentences starting or ending with those words. Is there an easy way to use Transformers to know which of the full sentences was more likely to be output by a language model?

Context:
Hi Olivier,
What about comparing the average log-likelihoods 4 of yours candidates?


Question:
The default behavior of Trainer(...) when evaluating model is disabling Dropout. Concretely, y_pred for M runs will be exactly the same
for i in range(M):
    logits, labels, metrics = trainer.predict(tokenized_datasets["eval"])
    y_pred = np.argmax(logits, axis=2)
    ...

Now I am trying to apply Monte Carlo Dropout trick introduced this this answer 5. This requires to turn the Dropout on while making predictions on the validation set.
I am wondering how I achieve this goal. Any input is appreciated

Context:
I don’t think this is possible with the Trainer class as it is, but you can derive this class and then change the relevant methods.
In your case, I think you need to change the evaluation_loop method and delete the model.eval() line.
I think it would be better to keep the model.eval() line and set only the Dropout layer to train mode like it is shown in this post 2.
There might be some more changes that you need to make.
You can find the Trainer source code here.
Hope this helps  .


Question:
Hi,
I was looking at the  implementation of the AdamW optimizer 5 and I didn’t understand why you put the weight decay at the end.
Shouldn’t you swap between this line:
p.data.addcdiv_(exp_avg, denom, value=-step_size)
and the weight decay part?
Thanks.
The AdamW algorithm from the “DECOUPLED WEIGHT DECAY REGULARIZATION” paper & The relevant source code for transformers.AdamW:

image1681×794 201 KB

Context: The two lines substract an independent thing to the model parameters, so executing them in any order will give the same results.


Question:
when passing all along with datasets to the Seq2SeqTrainercall I got :
HTTPError: 400 Client Error: Bad Request for url: https://huggingface.co/api/repos/create - Only regular characters and ‘-’, ‘_’, ‘.’ accepted

Context: You should post relevant code when asking for help, otherwise no one can really understand what’s going on. In this instance, the problem is very likely when your define your Seq2SeqTrainingArguments, so please share how you defined those, or if you used your code as a script, what arguments you passed to it,


Question:
With the help of awesome transformers library I have trained a multilabel classificator, which predicts topics for comments. I’m using TFDistilBertModel as a layer, wrapped in Keras functional API. So the architecture of a model is following: 2 input layers, then a layer, respawned as such:
distilbert_layer = TFDistilBertModel.from_pretrained(’/path/to/distilbert_model’), then some LSTM and Dense layers. The model is trained, everything is perfect at this time. Now comes the prediction time.
We serve our models on AWS containers, managed by kubernetes. Each time a user wants to generate topic of a new comment, we send a prediction task. To make a prediction I initiate model architecture, including respawning distilbert_layer as described above, then I load model weights, which I saved after training with the help of keras save_weights() function. Now the model is ready to predict.
The issue:
Each time a prediction task is sent, it consumes around 600mb of memory, which is not released after prediction. I assume there works a loadbalancer principle, so each time the task can be sent to a different process on the container, which uses different ram, which is why caching the model does not help. RAM quickly gets exhausted, then the worker just freezes and gets rebooted or initiates load of another worker.
Has anyone experienced such issues? Any help is very much appreciated.

Context:
I am currently facing the same issue. I deployed a question and answering model on digital ocean server droplet. After sending it a few texts the server stopped running and went down.
My take is ONNX runtime 2 would help; have you tried it? I am converting the model to onnx and redeploying it to see how it would perform.
Please let me know if you have found a solution.
Thanks.


Deploy multilingual sentence tansformer into cloud

Question:
Hi community,
I am new to transformer models and particularly interested by a multilingual sentence transformer (stsb-xlm-r-multilingual, 1.1 Go).
I have passed multiple times to search on how to deploy this model on cloud with the highest throughput (up to 1000 requests/sec) and lowest latency (<1 sec), while trying to saving costs. It seems to be painful.
Would anyone have advice on deployment requirements (CPU, GPU, RAM,…), clouds (AWS, GCP,…),  and frameworks (torchserve, triton,…) to meet my needs?
Thanks!

Context:
Hi,
Would anyone have advice?
Thanks !


Question:
Hi all,
Thanks for making this forum!
I have a list of tests, one of which apparently happens to be 516 tokens long.  I have been using the feature-extraction pipeline to process the texts, just using the simple function:
nlp = pipeline('feature-extraction')

When it gets up to the long text, I get an error:
Token indices sequence length is longer than the specified maximum sequence length for this model (516 > 512). Running this sequence through the model will result in indexing errors

Alternately, if I do the sentiment-analysis pipeline (created by nlp2 = pipeline('sentiment-analysis'), I did not get the error.
Is there a way for me put an argument in the pipeline function to make it truncate at the max model input length?  I tried reading this 37, but I was not sure how to make everything else in pipeline the same/default, except for this truncation.

Context:
One quick follow-up – I just realized that the message earlier is just a warning, and not an error, which comes from the tokenizer portion.  I then get an error on the model portion:
IndexError: index out of range in self

So I have two questions:

Is there a way to just add an argument somewhere that does the truncation automatically?
Is there a way for me to split out the tokenizer/model, truncate in the tokenizer, and then run that truncated in the model?

Thank you!


Do we need to fine-tune Wav2Vec2FeatureExtractor?

Question:
Hi,
I’m thinking about training wav2vec2 model for Japanese.
And I have a question.
Do we need Wav2Vec2FeatureExtractor as well?
Or can we use Wav2Vec2FeatureExtractor for any languages?
Thanks in advacne.

Context:
[Found answer]It seems there are 2 kinds of feature extractors.
1st one is just normalize raw audio and 2nd one is part of architecture.
Since 1st one is just normalizing raw audio, I don’t think we need to train it.


Question:
Hi I have a question about how to pad audio when training wav2vec2 model.
The tutorial explains how to handle batch size of one.
input_values = processor(ds["speech"][0], return_tensors="pt").input_values  # Batch size 1
logits = model(input_values).logits
predicted_ids = torch.argmax(logits, dim=-1)

But I think I need to pad shorter audio when batch size is bigger than 2.
Can I pad shorter audio with 0 ?
or is there convenient function for that?
Thanks in advance.

Context:
[Found Answer]
It’s going to pad with 0.0. I found it…


Is it possible to create a Résumé parser using a Huggingface model?

Question:
In other words, is it possible to train a supervised transformer model to pull out specific from unstructured or semi-structured text and if so, which pretrained model would be best for this?
In the resume example, I’d want to input the text version of a person’s resume and get a json like the following as output: {‘Education’: [‘BS Harvard University 2010’, ‘MS Stanford University 2012’], ‘Experience’: [‘Microsoft, 2012-2016’, ‘Google, 2016 - Present’]}
Obviously, I’ll need to label hundreds or thousands of resumes with their relevant Education and Experience fields before I’ll have a model that is capable of the above.
Here’s another example 31 of the solution that I’m talking about although this person seems to be using GPT-3 and didn’t have any code provided.  Is this something that any of the huggingface pipelines 7 is capable of and if so, which pipeline would be most appropriate?

Context:
Is there any reason you’re looking to do this with a transformer?
This is a common vision problem, and transformers aren’t usually the first port of call for a problem like this.


Question:
Hey,
I am trying to figure out how to freeze layers of a model and read that I had to use
for param in model.base_model.parameters():
    param.requires_grad = False

if I wanted to freeze the encoder of a pretrained MLM for example. But how do I use this with the Trainer?
I tried the following:
from transformers import BertTokenizer, BertForMaskedLM. LineByLineTextDataset, DataCollatorForLanguageModeling, Trainer, TrainingArguments
model = BertForMaskedLM.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

for param in model.base_model.parameters():
    param.requires_grad = False

dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path=in_path,
    block_size=512,
)

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)

training_args = TrainingArguments(
    output_dir=out_path,
    overwrite_output_dir=True,
    num_train_epochs=25,
    per_device_train_batch_size=48,
    save_steps=500,
    save_total_limit=2,
    seed=1
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset
)

trainer.train()

If the encoder was frozen I would expect it to produce the same outputs as a fresh instance of the pretrained encoder, but it doesn’t:
model_fresh = BertForMaskedLM.from_pretrained('bert-base-uncased')
inputs = tokenizer("This is a boring test sentence", return_tensors="pt")
torch.all(model.bert(**inputs)[0].eq(model_fresh.bert(**inputs)[0]))
--> tensor(false)

So I must be doing somethin wrong here, I guess the Trainer is reseting the requires_grad attribute and I have to overwrite it somehow after I instanciated the trainer?
Thanks in advance!
Johannes

Context:
Looking at the source code of BertForMaskedLM 33, the base model is the “bert” attribute, not the “base_model” attribute. So if you want to freeze the parameters of the base model before training, you should type
for param in model.bert.parameters():
    param.requires_grad = False

instead.


Get Optuna study from hyperparameter-search in Trainer?

Question:
Hi there,
I use hyperparameter-search in Trainer with Optuna and wanted to know if there is an easy option to access the study itself. From what I’ve read in the implementation, only the BestRun is returned by run_hp_search_optuna() and not the study itself.
(I’m asking because I wanted to try out the plot functions of Optuna that work on the study)

Context: Hello, I have this confusion too. Have you solved this problem? After some trying, I find this page is helpful——Saving/Resuming Study with RDB Backend 23.


Predicting only ” ” after training (S2T) Wav2Vec2CTC

Question:
My work so far:

      colab.research.google.com

Google Colaboratory 2

So I have copied the code from Fine-Tune XLSR-Wav2Vec2 for low-resource ASR with 🤗 Transformers 2 and tried to implement things separately.
But, after I run trainer.train and try to get the predictions, my model just predicts “” empty strings.
So, what does trainer.train do to my model? It should update the weights so as to increase performance, but it apparently is deleting the model…
Can someone help me please?

Context:
I have the same issue, the loss is nan and after 1 epoch the model predicts empty strings
please, have you found the root of the issue? thanks.