Training DistilGPT2, running into import error

Question:
Hello!
I am trying to train distilgpt2 with my own data on colab. The problem is, even after installing the packages from requirements.txt, it is running into an import error. I am reproducing the entire error below.
Traceback (most recent call last):

File “train.py”, line 28, in
from distiller import Distiller
File “/content/transformers/examples/distillation/distiller.py”, line 31, in
from grouped_batch_sampler import GroupedBatchSampler, create_lengths_groups
File “/content/transformers/examples/distillation/grouped_batch_sampler.py”, line 24, in
from utils import logger
File “/content/transformers/examples/distillation/utils.py”, line 23, in
import git
File “/usr/local/lib/python3.6/dist-packages/git/init.py”, line 38, in
from git.exc import *                       # @NoMove @IgnorePep8
File “/usr/local/lib/python3.6/dist-packages/git/exc.py”, line 9, in
from git.compat import UnicodeMixin, safe_decode, string_types
File “/usr/local/lib/python3.6/dist-packages/git/compat.py”, line 16, in
from gitdb.utils.compat import (
ModuleNotFoundError: No module named ‘gitdb.utils.compat’
Any help is appreciated. Is this because of an improper version of python?
NB : I have restarted the runtime after installing packages from requirements.txt , but still the error persists.
Thank you for your time!

Context:
Are you sure you ran
pip install -r requirements.txt

in the distillation subfolder of the repo? It looks like you are missing some required packages.


Question:
Hi, I was going through the documentation and got a confusion
trainer = Trainer(
model=model,                         # the instantiated  Transformers model to be trained
args=training_args,                  # training arguments, defined above
train_dataset=train_dataset,         # training dataset
eval_dataset=test_dataset            # evaluation dataset
)
I couldn’t understand what is the type of train_dataset and how the target for loss calculation is selected.
In Fine-tuning in native TensorFlow 2 also there is no target value. Am I missing something?
model.fit(train_dataset, epochs=2, steps_per_epoch=115)
Thank you

Context:
For more context, he/she is talking about this page: https://huggingface.co/transformers/training.html 11
I also got confused by this bit of the documentation, but I think this code expects datasets like the ones provided by Hugging Face’s NLP package 9.
I think they are all based on Pytorch’s Dataset Class 6, but I could be mistaken.
Try to use one of the datasets provided by their NLP package and check if it works correctly.
Hope this helps!


Question:
Hello everyone,
I am working on multiclass text classification, currently using XLM-Roberta as a classifier. I have a doubt concerning padding strategies.
My first intuition was to tokenize my training and validation sets separately (as they were two distinct super-batches) using padding = True; the result of this was having all training examples padded to length l1 (the length of the longest sequence in the training set), and validation examples padded to a different l2.
An alternative approach (and the one that seems to be used in the GlueDataset and related methods) is to use padding = max_length, and thus have all examples padded to the same provided length (possibly, 512, which is the maximum sequence length allowed for this model).
Would you mind sharing your thoughts on what strategy might work best and makes more sense from a “theoretical” point of view?
Thank you very much!

Context:
Hi Isabella,
My understanding is that so long as you have your padding mask correctly implemented, the model will not “pay attention” to the pad tokens and so the predictions should be consistent across models (regardless of padding length).
If you do not use a padding mask then the predictions could differ, because the attention weights for the pad tokens will have some affect on your predicted outcome. By having an effect, I mean that the pad tokens contribute to the attention score, and therefore to the loss as a result.
In terms of time complexity, my understanding is somewhat murkier. I suspect that the first method you suggest is faster but it depends on the implementation. If we calculate all attention scores and set those relating to padding to zero, then the max_length version could be much slower. However, I think it is reasonable to assume that the implementation performs a check first i.e. “should I calculate attention here?” which would be only slightly slower than the l1, l2 padding. This point is open to correction!
I could be incorrect in my approach, but I prefer to pad to max_length for the sheer convenience of it. In the l1, l2 approach for example, methods such as cross validation require the repetition of the length calculation.


[Bart] Question for BartModel Output shape

Question:
Hi I’m trying to fine-tune multilingual-Bart with Korean to generate some texts.
while I tried to pass my datas into model, I can’t under stand why the output shape of the model is different from what I expected.
Settings : I used MBartTokenizer and BartForConditionalGeneration
for batch, I used prepare_translation_batch to make datas as batch, inputs_ids and target_ids.
I also need decoder_input_ids(form like [tgt_lang_code, seqence, eos]), so I made it.
Here’s the problem.
BartForConditionalGeneration’s forward pass needs input_ids necessarily. From the Docs of Bart, If I pass only the ‘input_ids’ to the model(it can include attention_mask), the model’s decoder wouldn’t have it’s own input, so It takes ‘input_ids’ as their input.
And the shape of the prediction_scores in returns should be (batch_size, seq_len, vocab_size) and It worked right
But, When I pass the ‘input_ids’ and ‘decoder_input_ids’ together to the model, the shape of the prediction_scores shows (batch_size, 1 , vocab_size) all the time.
I think when I pass the inputs together, The shape of prediction_scores in returns should be like (batch_size, decoder_input_seq_len, vocab_size)
I don’t know why this happen. Maybe I totally misunderstood the model at all.
The reason I made this topic is I need a clean view of this problem.
Any advice would be appreciated.
Thank you.
q1904×785 15 KB

Context: pass use_cache=False to forward. Confusing, I know.


Continue training XLNet on a specific closed-domain dataset

Question:
I’m wondering what if we want to leverage the already pre-trained XLNet model (and its language knowledge) and fine-tune on a specific closed-domain dataset, say legal domain for example.
I have already corpora I’m just missing how to do this with XLNet like models.
Any thoughts on how to do that?

Context: Hi @krannnN, you can use the run_language_modelling script to fine-tune xlnet. You can fine it here 12. You’ll just need to provide the dataset in the required format.


How to get probability of the first generated token?

Question: When doing conditional generation with (m)BART how can I get the probability of the first generated token? I would like to use it as confidence to filter my results. (I generate really short summaries which are really extracted answers from text). The generate method only generates token ids and I could not find out which number to use from the raw model output (after softmax on dim=-1). Also why does it have dimensions like batch_size x 638 x 250027? 250027 is the vocabulary size I guess but what is 638? I thought it should be max_source_length which is 1024 (assuming max output length equals max input length).

Context:
Hi marton,
As you can see in the doc 23

You should apply argmax rather than softmax on the output
638 is the actual sequence length, usually we pad sentences just to the length of the longest sequence in the batch.

What you have got is called prediction_scores, now you can do
predicts = prediction_scores.argmax(dim=-1) # (batch_size, sequence_length) dtype=torch.Long
scores = prediction_scores[predicts] # (batch_size, sequence_length) dtype=torch.float

scores now is the probability of all predicted tokens, if you just want the first token of every sentence, just modify the code above.


Question:
Hello!
Very basic question - is there an official way to install huggingface using conda or does anybody have any insight into whether this is on the to-do list?
Thanks!

Context: Not yet but should be possible in the mid-term. The main limitation right now is that SentencePiece doesn’t like conda but @anthony is working on getting SentencePiece support in our tokenizers library should we should be able to have a conda install as well when this is finished (might still take a little bit of time though, it’s a big chunk of work)


Are WikinewsSum models for text summarization?

Question:
Can anyone tell me if the following models can be used for text summarization?
WikinewsSum/t5-base-multi-en-wiki-news
WikinewsSum/t5-base-multi-combine-wiki-news
WikinewsSum/t5-base-with-title-multi-en-wiki-news
WikinewsSum/bart-large-cnn-multi-en-wiki-news
WikinewsSum/bart-large-multi-en-wiki-news
Thank you,
Efstathios
PS: @valhalla If you know I will appreciate it.

Context: I’m not sure, you can try it and verify


Which of the sshleifer/* models can be used as-is for text summarization?

Question:
Hi @sshleifer,
I would like to ask which of the following models can be used for text summarization:
• shleifer/distilbart-cnn-12-6
• sshleifer/distilbart-xsum-12-6
• sshleifer/distilbart-cnn-6-6
• sshleifer/distilbart-xsum-12-3
• sshleifer/distilbart-xsum-9-6
• sshleifer/distilbart-xsum-12-1
• sshleifer/distilbart-xsum-6-6
• sshleifer/distilbart-xsum-1-1
• sshleifer/distilbart-cnn-12-3
I would also like to know about:
• sshleifer/mbart-large-cc25
Thank you!
Efstathios

Context:
You can use any of these models as-is, you can use cnn checkpoints if you want longer summaries or xsum if you want short summaries,
you can also use facebook/bart-large-cnn and facebook/bart-large-xsum


Question:
I have gone through this code train from scratch 17 and understood how to pre-train a model from scratch. I have the following doubts in this code

What does block_size in LineByLineTextDataset represent?

If I want to further pretrain Robert-base model (instead of training from scratch) using my own corpus, what are the changes I have to make in the above code besides the following changes

from transformers import RobertaForMaskedLM, RobertaTokenizerFast
tokenizer = RobertaTokenizerFast.from_pretrained("roberta-base")
model = RobertaForMaskedLM('roberta-base')

I am aware that I need not to train tokenizer from scratch.
@thomwolf @julien-c

Context:
Hi @mr-nlp, I think you can use the same run_language_modelling script to further pre-train roberta, just provide your own datasets.
block_size is used for max_length


Question:
Hi all,
I’ve recently started into the NLP world, so this might seem like an obvious question but are there models to generate tags from text? E.g. lets say you have a corpus of text describing houses and you would like to retrieve the house type, size and color. The goal would be to get an output like this for every house description {house_type: ‘apartment’, size: ‘1000sqft’, color: ‘grey’}
It seems that question answering model are able to understand that information (or at least answer to one of the questions at a time), but I’m not sure if there’s a way to adapt these for the above task.
It seems to me that it could be pretty useful to train other models (e.g. vision) in a self-supervised way (e.g. get the house type size and color from a house photo using the output of the model above as target variable)
Thanks!
Elliot

Context:
Hi Eliot,
If this information is explicitly available in the text then you can try to achieve this with a QA model. Asking questions like “What is the house type ?”, “What’s the color of the house ?” etc.
There are other methods for such type of semantic parsing tasks, but one way you can approach this using  is using text2text approach with T5 (it’s seq-to-seq model where you can feed in some text and ask the model to output some text). i.e given your text you can train T5  to output a structured text, something like
house_type: apartment <sep> color: grey <sep> house_size: 1000
This might be a overkill but I tried this as a experiment in my work and so far it’s doing really well.
One other approach would be to frame this as an entity extraction task, you entities will be house_type, color and size. Something like spacy could really help. If you are new to Entity extraction see this demo 9 to get an idea


Question:
Fairly new to ML and very new to transformers. Want to make sure I’m doing the right thing … I’m trying to do text classification with a small data set and though this would be a good option (is it?)
Here’s the basics of my code:
texts = ["random text string...", ...]
labels = [1, 0, ...]

tokenized_sents = []
attention_masks = []

for sentence in sentences:
    tokenized_sents.append(tokenizer.encode(sentence, add_special_tokens=True, ...))
input_ids = pad_sequences(tokenized_sents)

for sentence in input_ids:
  att_mask = [int(token_id > 0) for token_id in sentence]
  attention_masks.append(att_mask)

dataset = tf.data.Dataset.from_tensor_slices(input_ids, attention_masks, labels)

# copied from https://huggingface.co/transformers/training.html
optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
model.compile(optimizer=optimizer, loss=loss)
model.fit(dataset, epochs=2, steps_per_epoch=115)

I was pretty confident this all worked, but then when I did the following test:
sent = ["I like to watch movies"]
sent = tokenizer.encode(sentence, add_special_tokens=True, ...)
att_mask = [int(token_id > 0) for token_id in sent]
ds = tf.data.Dataset.from_tensor_slices(sent, att_mask)
model.predict(ds)

I got a super long array. But the labels can only be 1 or 0 and there’s only one sample, so I was expecting a 1 by 2 array. Any idea why this doesn’t work?
Also, what’s the best way to save this model and use it for predictions later.
Thank you.

Context: I would suggest expanding this example instead: https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/trainer/01_text_classification.ipynb 28


What exact inputs does bleu_metric.compute() require?

Question:
Very basic question. In my project I want switch from Google Translate API to Marian MT models and therefore I want to compare them using BLEU score. I want to use the BLEU metric from the nlp library, but I’m having problems getting it to work correctly.
Here is my code:
from transformers import MarianTokenizer, MarianMTModel
src = 'de'
trg = 'en'
mname = f'Helsinki-NLP/opus-mt-{src}-{trg}'
tokenizer = MarianTokenizer.from_pretrained(mname)
model = MarianMTModel.from_pretrained(mname)
src_texts = ["Ich bin ein kleiner Frosch.", "Tom bat seinen Lehrer um Rat."]
tgt_texts = ["I am a small frog.", "Tom asked his teacher for advice."]
batch = tokenizer.prepare_translation_batch(src_texts=src_texts, tgt_texts=tgt_texts)
import nlp
bleu_metric = nlp.load_metric('bleu')
preds = model(batch.input_ids)
targets = batch.decoder_input_ids
bleu_metric.compute(preds, targets)

And this is the error message I get:
---------------------------------------------------------------------------
ArrowTypeError                            Traceback (most recent call last)
<ipython-input-89-33da8475e3a1> in <module>
----> 1 bleu_metric.compute(preds, targets)

/opt/conda/envs/fastai/lib/python3.7/site-packages/nlp/metric.py in compute(self, predictions, references, timeout, **metrics_kwargs)
    191         """
    192         if predictions is not None:
--> 193             self.add_batch(predictions=predictions, references=references)
    194         self.finalize(timeout=timeout)
    195

/opt/conda/envs/fastai/lib/python3.7/site-packages/nlp/metric.py in add_batch(self, predictions, references, **kwargs)
    207         if self.writer is None:
    208             self._init_writer()
--> 209         self.writer.write_batch(batch)
    210
    211     def add(self, prediction=None, reference=None, **kwargs):

/opt/conda/envs/fastai/lib/python3.7/site-packages/nlp/arrow_writer.py in write_batch(self, batch_examples, writer_batch_size)
    155         if self.pa_writer is None:
    156             self._build_writer(pa_table=pa.Table.from_pydict(batch_examples))
--> 157         pa_table: pa.Table = pa.Table.from_pydict(batch_examples, schema=self._schema)
    158         if writer_batch_size is None:
    159             writer_batch_size = self.writer_batch_size

/opt/conda/envs/fastai/lib/python3.7/site-packages/pyarrow/types.pxi in __iter__()

/opt/conda/envs/fastai/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.asarray()

/opt/conda/envs/fastai/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.array()

/opt/conda/envs/fastai/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

/opt/conda/envs/fastai/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowTypeError: Could not convert tensor([[[ 7.0599, -1.6253,  7.5354,  ..., -1.6600, -1.6035,  0.0000],
         [ 6.7303, -2.1037,  7.4463,  ..., -2.0789, -2.0494,  0.0000],
         [ 6.1458, -1.4315,  7.7637,  ..., -1.3350, -1.3450,  0.0000],
         ...,
         [ 5.6321,  0.7129,  9.5273,  ...,  0.7408,  0.7298,  0.0000],
         [ 5.4492, -0.6234,  9.0366,  ..., -0.6214, -0.6522,  0.0000],
         [ 7.1594, -3.3825,  4.1902,  ..., -3.4131, -3.3912,  0.0000]],

        [[ 5.8872, -3.5864,  5.8050,  ..., -3.5273, -3.4877,  0.0000],
         [ 6.4951, -2.9127,  7.8423,  ..., -2.7731, -2.8157,  0.0000],
         [ 6.4846, -2.8267,  8.0983,  ..., -2.6857, -2.7147,  0.0000],
         ...,
         [ 7.0786, -2.7071,  7.8688,  ..., -2.6743, -2.6513,  0.0000],
         [ 5.6782, -1.9020,  7.4212,  ..., -2.0306, -2.0242,  0.0000],
         [ 7.0517, -2.7702,  5.3939,  ..., -2.7920, -2.7438,  0.0000]]],
       grad_fn=<AddBackward0>) with type Tensor: was not a sequence or recognized null for conversion to list type

I’m following the MarianMT docs and the nlp notebook on Colab. I’m not sure in which exact form the targets have to go into bleu_metric.compute(). On the other hand it looks like the error message is triggered by the format of preds. I tried many different ways but cannot get it to work.
Any help is appreciated

Context: Could you try to convert preds to lists instead of torch.Tensor ?


Question:
Hello,
I am training a smaller RoBERTa model (6layers) with BPE tokenizer on a domain specific corpus (not a large corpus) for MLM and it seems to do a good job in filling in masked words in testing. Also, the loss in Tensorboard looks much better compared to my BERT model (12layers) with wordpiece tokenizer.
When I try to use this small RoBERTa model on fine-tuning for QA (on squad2.0), predictions are mostly always empty. It returns almost always blanks as answer.  I followed this https://huggingface.co/blog/how-to-train 5  but QA seems to be not doing great.  Any recommendations to make the smaller RoBERTa model to fine-tune on squad?   or should I go with bigger models (12-24 layers)?   Is there a pruning while training method available in Transformers?  I would appreciate any ideas… Thank you!

Context:
hi @zb1 the official examples include  Roberta distillation using the DistillBert method. You can find it here

      GitHub

huggingface/transformers 28

🤗Transformers: State-of-the-art Natural Language Processing for Pytorch and TensorFlow 2.0. - huggingface/transformers


Pretrained Models to Heroku Production Environment

Question:
Hi! I’m new to hugging face, so excuse this if it’s a dumb question!
I’m trying to expose the T5 small pre_trained model as a REST endpoint. I have both the model code and the python Flask API working in a virtual environment in python on my computer. When I try to host the API on heroku, the code size exceeds the RAM limit. Are there any best practices for storing these models as static assets that can be called by an API? Is there a better way to do this to avoid hitting the RAM limit?
FYI: I used this tutorial for the model code - https://towardsdatascience.com/simple-abstractive-text-summarization-with-pretrained-t5-text-to-text-transfer-transformer-10f6d602c426 9

Context: Not sure if it can help for your usecase but we are providing an API inference endpoint for T5: https://huggingface.co/t5-small?text=My+name+is+Sarah+and+I+live+in+London 18. Will soon be usable for other tasks than translation.


Question Answering using RoBERTa in Hindi

Question: Can the run_squad.py 3 file be used for fine-tuning models with datasets in languages other than English?

Context: You can use it with a pretrained model in any language, but you would need your pretrained model to be in the same language as the dataset you are using (otherwise, the texts seen won’t make sense to the model!)


How to give equal importance of all labels while dealing with unbalanced samples

Question:
for ex: I have four classes
class 1: 200 samples
class 2: 100 samples
class 3: 20 samples
class 4: 10 samples
so during predictions, most of my samples are predicting class 1 or 2 because samples are high even I have unlabeled samples of class3 and class4.
How can I overcome this in HF?
is it possible to tell me the model that concentrates more on class3 and class4?
Help me very soon.
Thank you in advance.

Context:
Hi Para, I’m afraid there is no magic bullet in HF that can help solve this problem. It’s a common ML challenge and as such the standard approaches apply: (1) Under-sampling: Randomly delete records from the over-represented classes (2) Over-sampling: Duplicate records in the under-represented classes (3) Synthetic sampling/data augmentation: In NLP, there are actually quite some interesting techniques with which you can augment your data in the under-represented classes to increase record count. Check out this library: GitHub - makcedward/nlpaug: Data augmentation for NLP
Hope that helps, let me know if any questions.
Cheers
Heiko


Common practice, using the hidden state associated with [cls] as an input feature for a classification task?

Question:
In the upcoming book “Natural Language Processing with Transformers”, it’s teaching us how to do a classification task on sentences by using Transformers as Feature Extractors. We process the sentences through a transformer to get the hidden state.
To train a classifier, we take the token embedding for just the first token, namely the “[CLS]” and ignore the rest of the sentence. The book says that it’s common practice to do that.
It doesn’t make much sense to me to ignore the rest of the embeddings. Shouldn’t they be averaged or something?
The only reasoning I can think of is that the attention layers of the encoder make the CLS token absorb the meaningful context?
Thank you! The book is awesome by the way, highly recommended!

Context:
Hi @carlosaguayo thanks for your question and I’m glad you’re enjoying the book
In general, we need a way to represent the sequence of embeddings as as single vector and there are several “pooling” techniques that people use in the literature:

[CLS] pooling: just take the embedding of the [CLS] token as the representation for the whole sequence
mean pooling: take the average of token embeddings
max pooling: take the token embedding with the largest values

A related question is whether pooling should be applied to the last hidden states, or some earlier layers (or concatenation thereof).
Now, which pooling method + layer(s) provides the best feature representation tends to depend on the task at hand, the domain of the data, length of the texts and so on. We picked [CLS] pooling in this early chapter because it’s simple and tends to be “good enough” for text classification tasks. You can find a nice ablation study that examined some of these issues here.
As to why this even works, you’re insight that it’s due to self-attention is spot on! Each token embedding in the sequence is contextualised through the attention mechanism, so the [CLS] token does contain information about subsequent tokens in the sequence (we explain this in more detail in Chapter 3).
Hope that helps!


Using datacollator for multi-task training

Question:
Hey everyone,
Actually, I am trying to accommodate multiple subtasks including STS-b and NER into a multi-task model, however am unable to issue the tokens from the conll dataset into the DataCollator. Can anyone help me with this? The code snippet is shown below.
class NLPDataCollator(DefaultDataCollator):
    """
    Extending the existing DataCollator to work with NLP dataset batches
    """
    def collate_batch(self, features: List[Union[InputDataClass, Dict]]) -> Dict[str, torch.Tensor]:
        first = features[0]
        if isinstance(first, dict):
          # NLP data sets current works presents features as lists of dictionary
          # (one per example), so we  will adapt the collate_batch logic for that
          if "labels" in first and first["labels"] is not None:
              if first["labels"].dtype == torch.int64:
                  labels = torch.tensor([f["labels"] for f in features], dtype=torch.long)
              else:
                  labels = torch.tensor([f["labels"] for f in features], dtype=torch.float)
              batch = {"labels": labels}
          for k, v in first.items():
              if k != "labels" and v is not None and not isinstance(v, str):
                  batch[k] = torch.stack([f[k] for f in features])
          return batch
        else:
          # otherwise, revert to using the default collate_batch
          return DefaultDataCollator().collate_batch(features)

Context:
This error is displayed when I am trying to accommodate the NER dataset specifically entities extracted in the fashion show here

image1121×432 19.3 KB


Question:
I am interested in creating embedding to HTML tags in a web. Does Bert have such model?
Other works that can be relevant

Context:
Hi,
MarkupLM 4 is such a model. I’ll add it soon to HuggingFace Transformers.


Split compound words (windfall = wind + fall)

Question:
Is there a way to use any of the model to split words into word parts that some might use downstream ? For example, consider a custom domain where words like windfall, firewall exists - but user may search for “wind fall” or “fire wall” downstream. Most basic way I thought was to split them randomly into multiple parts and “accept” a split whose sub parts make sense. For example, windfall = w + indfull, wi + ndfull, win + dfull and so on…Then apply existing language model to see if the subparts words exist in vocabulary.
Appreciate if anyone has pointers.

Context:
Hi @mangled,
I believe subword tokenizers (i.e. PreTrained Tokenizer from HuggingFace which is based on SentencePiece) already do this sort of word splitting.
But most models rely on their own pretrained tokenizer with their own fixed vocab, so you may not have the same subword units. You may try to pre-train your own tokenizer.
Hope this helps.


Using .generate with TAPAS as encoder in EncoderDecoder

Question:
Hi,
I’m trying to to train the model to perform text generation conditioned on tables.
Since TAPAS can encode the semi-structured meaning in tables, I guessed it was a good choice to use it as an encoder and say GPT2 (or any other CLM) as a decoder.
I however encountered a problem when trying to generate from that EncoderDecoder model, this:

image597×651 195 KB

I guess this is since model.generate() for EncoderDecoder does not expect to have the extra dimension of token_type_ids that TAPAS has.
Can anyone think of a way I can make this work?
Thanks!

Context:
Update: it works for me when overriding the _update_model_kwargs_for_generation method. The token_type_ids shouldn’t be updated, as a table only needs to get encoded once.
Notebook: Google Colab 1


Question:
Hello all,
I run a Hyperparameter search using Optuna and got a model giving me 83% accuracy. When I then try and repeat this by retraining using the same hyperparameter (including seed), I cannot repeat the results. This is my trainer arguments and optuna search;
# Define the trainig arguments
training_args = TrainingArguments(
    output_dir='./results',          # output directory
    seed = 0,
    num_train_epochs=3,              # total number of training epochs
    per_device_train_batch_size=16,  # batch size per device during training
    per_device_eval_batch_size=16,   # batch size for evaluation
    warmup_steps=22,                 # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    learning_rate=5e-5,              # initial learning rate for AdamW optimizer.
    load_best_model_at_end=False,     # load the best model when finished training (default metric is loss)
    do_train=True,                   # Perform training
    do_eval=True,                    # Perform evaluation
    logging_dir='./logs',            # directory for storing logs
    logging_steps=10,
    gradient_accumulation_steps=2,   # total number of steps before back propagation
    fp16=True,                       # Use mixed precision
    fp16_opt_level="02",             # mixed precision mode
    evaluation_strategy="epoch",     # evaluate each `logging_steps`
    save_strategy = 'no',           # The checkpoint save strategy to adopt during training. I dont want to save, probably why it did save and take up disk space in HP search
    #save_total_limit = 1.           # Trying this to stop octuna from saving
)

trainer = Trainer(
    model_init=model_init,
    args=training_args,                  # training arguments, defined above
    train_dataset=train_dataset,         # training dataset
    eval_dataset=val_dataset,             # evaluation dataset
    compute_metrics=compute_metrics,
    #callbacks=[EarlyStoppingCallback(3, 0.0)] # early stopping if results dont improve after 3 epochs
)

best_run = trainer.hyperparameter_search(direction="maximize",
                                         hp_space=my_hp_space,
                                         compute_objective=my_objective, # cant get this working, for now work with loss
                                         n_trials=50,
                                         pruner=optuna.pruners.NopPruner(),
                                         sampler=optuna.samplers.GridSampler(search_space),
                                         study_name=name,
                                         storage="sqlite:////content/drive/MyDrive/{}.db".format(name), #change this to a local directory if you want to save to disk
                                         load_if_exists=True # you can change this to true, for continuing the search
                                         )

best_run

I have now also fixed the seed for numpy and torch
RANDOM_SEED = 42

np.random.seed(RANDOM_SEED)

torch.manual_seed(RANDOM_SEED)

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

Could it be that the classification head that being reinitialised every time I retrain is random, resulting in different results?

Context: Same issue


Pre-training a BERT model from scratch with custom tokenizer

Question:
Hi all,
I’ve spent a couple days trying to get this to work. I’m trying to pretrain BERT from scratch using the standard MLM approach. I’m pretraining since my input is not a natural language per se.
Here is my code:

from tokenizers import Tokenizer
from tokenizers.models import WordLevel
from tokenizers import normalizers
from tokenizers.normalizers import Lowercase, NFD, StripAccents
from tokenizers.pre_tokenizers import Whitespace
from tokenizers.processors import BertProcessing
from tokenizers.trainers import WordLevelTrainer
from tokenizers.pre_tokenizers import Split
from tokenizers.normalizers import Strip
from tokenizers import Regex

exp = Regex("(^((\w)+(?=\s)))|((\[ENTRY\]\ (\w|\||\.)+)\s)|((\[CALL\]\ (\w|\||\.|\s)+)(?=\ \[))+|(\[EXIT\])")

pre_tokenizer = Split(pattern=exp, behavior="removed",invert=True)
#print(pre_tokenizer.pre_tokenize_str("performExpensiveLogSetup [ENTRY] void [CALL] java.io.PrintStream println java.lang.String void [CALL] java.lang.Math pow double|double double [CALL] java.lang.Math sqrt double double [CALL] java.io.PrintStream println java.lang.String void [EXIT]"))

trace_tokenizer = Tokenizer(WordLevel(unk_token="[UNK]"))
trace_tokenizer.add_special_tokens(["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])
trace_tokenizer.normalizer = Strip()
trace_tokenizer.pre_tokenizer = pre_tokenizer
trace_tokenizer.post_processor = BertProcessing(sep=("[SEP]", 0),cls=("[CLS]", 1))
VOCAB_SIZE = 5000
trace_tokenizer.add_special_tokens(['[PAD]'])
trace_tokenizer.add_tokens([' '])
trainer = WordLevelTrainer(
    vocab_size=VOCAB_SIZE, special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"]
)
files = ["10k_smaller_dataset.txt"]
trace_tokenizer.train(files, trainer)

trace_tokenizer.save("data/trace.json")

from transformers import BertConfig, BertForMaskedLM
scale_factor = 0.25
config = BertConfig(
    vocab_size=VOCAB_SIZE,
    max_position_embeddings=int(768*scale_factor),
    intermediate_size=int(2048*scale_factor),
    hidden_size=int(512*scale_factor),
    num_attention_heads=8,
    num_hidden_layers=6,
    type_vocab_size=5,
    hidden_dropout_prob=0.1,
    attention_probs_dropout_prob=0.1,
)

from transformers import PreTrainedTokenizerFast
fast_tokenizer = PreTrainedTokenizerFast(tokenizer_object=trace_tokenizer,
                                         return_special_tokens_mask=True, mask_token='[MASK]', return_token_type_ids=False)
fast_tokenizer.add_special_tokens({'pad_token': '[PAD]', 'mask_token': '[MASK]'})

from datasets import load_dataset
dataset = load_dataset('text', data_files={'train': '10k_smaller_dataset.txt', 'test': 'tiny_eval.txt', 'eval': 'tiny_eval.txt'})
small_train_dataset = dataset["train"]
small_eval_dataset = dataset["test"]

model = BertForMaskedLM(config)
model.tokenizer = fast_tokenizer

def preprocess_function(examples):
    return fast_tokenizer(examples["text"], max_length = 128, truncation=True, padding=True)

encoded_dataset_train = small_train_dataset.map(preprocess_function, batched=True)
encoded_dataset_test = small_eval_dataset.map(preprocess_function, batched=True)

import numpy as np
from datasets import load_metric

metric = load_metric("accuracy")

def compute_metric(eval_pred):
    return metric.compute(predictions=eval_pred.predictions, references=eval_pred.label_ids)

from transformers import TrainingArguments
from transformers import DataCollatorForWholeWordMask
data_collator = DataCollatorForWholeWordMask(tokenizer=fast_tokenizer, mlm=True, mlm_probability=0.15)
training_args = TrainingArguments("test_trainer_bert_pre",
                                  num_train_epochs=1,
                                     # prediction_loss_only=True,
)
from transformers import Trainer

trainer = Trainer(
    model=model,
    tokenizer=fast_tokenizer,
    data_collator=data_collator,
    args=training_args, train_dataset=encoded_dataset_train,
    eval_dataset=encoded_dataset_test,
    compute_metrics=compute_metric,
)

train_result = trainer.train(resume_from_checkpoint=True)
train_result

trainer.evaluate(encoded_dataset_test)

The problem is in the last line, I never see the accuracy metric I define.
{'epoch': 1.0,
 'eval_loss': 0.0025006113573908806,
 'eval_runtime': 1.9859,
 'eval_samples_per_second': 503.54,
 'eval_steps_per_second': 62.942}

I’m sure there’s a super simple mistake I’m making that’s resulting in it being ignored. Any ideas?
Thank you in advance.
Best,
Claudio

Context: If you do a metric.compute(…) will you get the accuracy which you are looking for?


Grouping Tokens after Token Classification

Question:
Is there a way to group tokens after token classification via HF? I see something similar in Rasa 1. However, I am not sure it is the best way to do it as they are giving group numbers to the model to train on. However, If a document contains more groups than the documents in the training data, the RASA implementation fails.
I think I am looking for a solution like (kinda supervised) clustering which is independent of the number of groups in the documents.

Context:
Hi,
The token classification pipeline has the ability to group tokens, as seen here 1.
The front-facing API is an “aggregation strategy”. See the docs for more info.


Finetuning for feature-extraction? I.e. unsupervised fine tuning?

Question: I noticed the facebook/bart-large-mnli · Hugging Face 1 model card doesn’t show the feature-extraction task under Train menu, but it is under the Deploy menu. I haven’t been able to find an example of fine tuning a feature-extraction model, so is fine tuning not an option for feature extraction tasks? If it is, I’d love to see a working example somewhere…the examples I’ve been able to find are all for supervised learning, which makes me wonder if one needs labelled data to do fine tuning?

Context:
Hey @MaximusDecimusMeridi, the term “feature extraction” usually means to extract or “pool” the last hidden states from a pretrained model. So fine-tuning a model for feature extraction is equivalent to fine-tuning the language model, e.g. via masked or autoregressive language modelling. (You can find a BERT-like example of fine-tuning here 4, and indeed one does not need any labelled data for this step).
For BART, the situation is a bit more complex because it is a seq2seq architecture, so you would likely need to frame your fine-tuning task in that manner (e.g. as a translation or summarization task).
Most applications that need feature extraction (e.g. neural search) perform best with encoder-based models like BERT and friends - I recommend checking out sentence-transformers  (link) which provides many state-of-the-art models for these applications


Can BERT for mlm predict never seen words?

Question: Is it possible to my BERT model predict a word seen in my training samples, but not contained in my wordpiece tokenizer vocab?

Context: I am not sure I understand your question. Any word in your training sample would be broken down into word-piece tokens, and then converted into IDs. So yes, with the MLM task BERT can predict a word from the training sample.


BERT Multilabel - Different Training Dataset For Each Label?

Question:
Hi everyone,
I have successfully built a multi label classifier (10 labels, somewhat balanced) on sentence level with my own subclass of transformers library BertForSequenceClassification.
The classification performance is okay-ish. When I was first testing BERT on a binary classification task for a single label in my dataset it was very benefitial towards performance to include adversarial sentences that did not hold the same label. Thus, I tried the same approach in this multi label setup by adding more 0 labeled sentences to the dataset. But this worsened the performance.
My question would be if it is possible to have different training/evaluation datasets for each label in a Multilabel classification setup?
Some more background on the dataset and labels:
The labels of a sentence are indicated by a vector, e.g. (0, 0, 0, 1). It’s also possible that no label is present, i.e. the corresponding vector is (0, 0, 0, 0).
In total there are 10 labels.
Thank you!

Context: Just out of curiosity, what does it mean to have “different dataset(s) for each label”?


Question:
I am looking for some code examples on how to train with backbone transformers to both forward and reverse or detokenize.
The problem being that all my datasets (including validation) are pre-labelled for scoring and I  believe it would help if I made the text more clean and similar to what the my pretrained transformers were trained on (wikipedia text, I believe).   This means removing contractions, among other things.
The biggest challenge, of course, is the reverse trip and what techniques can be used to tokenize in such a way that we can appropriately map to and re-generate the original labels.  The text is very messy and requires quite a bit of cleansing.
Any pointers would be greatly appreciated!  Cheers

Context:
Another approach is using augmentation, eg:  What’s in the Dataset object — datasets 1.11.0 documentation 1
The idea being there is that I train/predict on augmented datasets so that I retain the original.   Is this a typical approach used?
Any feedback appreciated


Question:
Hi everyone,
I am running several models in production using the huggingface library to perform some tasks continuously. Unfortunately, the “continuously” part is very much lacking.
This is due to the fact that very often (every couple of hours), the following error pops-up:

requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://huggingface.co/.../resolve/main/tokenizer.json 2 or config.json

I have even tried to avoid this error by running transformers in offline mode using:

TRANSFORMERS_OFFLINE=1
HF_DATASETS_OFFLINE=1

Unfortunately, the systeem still seems to request the model from the web, though its cached and offline mode. Any ideas how to avoid this problem? (except for saving model offline)

Context: A colleague of mine and I have recently experienced a similar issue. Even if the models are cached locally, a momentary internet disconnection results in an error if it coincides with a model/tokenizer load in a series of training scripts. My thoughts on potential reasons go no further than speculations so, any suggestions on what might be the issue and how to solve it is pretty much welcome.


Does batching in the standard question-answering pipeline provide a speedup?

Question:
Hi there.
I am using the QA pipeline and hoping to get speedup through batching. Basically what I want (this is pseudo code, not exact code; I would appreciate your guidance in doing it right):
       _classifier = pipeline("question-answering", model="deberta")
        result = _classifier(question=["What is the goal?","When did it happen?","Who did it?"], context="Once upon a time many years ago an engineer set up a question answering machine and it was hoped it would run really fast")

So the basic goal is to get this to work at roughly the same speed as a single question, assuming I give it a decent GPU capable of parallelizing the 3 questions.
I did search before asking this question I found a few others who have asked it in different ways but not yet answered:
Batched pipeline · Issue #6327 · huggingface/transformers · GitHub 5
[Benchmark] Pipeline for question answering · Issue #3007 · huggingface/transformers · GitHub 1
Many thanks for any pointers or help!

Context:
In my experience it doesn’t help.
Also, when you look at the code, you can see that once it encode all the given examples, it iterate over them one by one (transformers.pipelines — transformers 4.0.0 documentation):
all_answers = []
for features, example in zip(features_list, examples):
    model_input_names = self.tokenizer.model_input_names + ["input_ids"]
    fw_args = {k: [feature.__dict__[k] for feature in features] for k in model_input_names}

    # Manage tensor allocation on correct device
    with self.device_placement():
        if self.framework == "tf":
            fw_args = {k: tf.constant(v) for (k, v) in fw_args.items()}
            start, end = self.model(fw_args)[:2]
            start, end = start.numpy(), end.numpy()
        else:
            with torch.no_grad():
                # Retrieve the score for the context tokens only (removing question tokens)
                fw_args = {k: torch.tensor(v, device=self.device) for (k, v) in fw_args.items()}
                start, end = self.model(**fw_args)[:2]
                start, end = start.cpu().numpy(), end.cpu().numpy()


Weights and biases not showing train loss correctly

Question:
Screen Shot 2021-12-06 at 5.55.37 PM1367×677 41.2 KB

Hi all, I’m training a binary text classification model. In order to debug, I’m training and evaluating on a small subset of the data (around 16 data points) to see if the model can successfully overfit. However, the train_loss logged to Weights and Biases are not showing correctly – as you can see from the screenshot, it’s just a single point. Any idea on why this happened?
Below are my training code:
model = AutoModelForSequenceClassification.from_pretrained("roberta-large")

training_args = TrainingArguments(
    output_dir='./results',
    learning_rate=1e-3,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    num_train_epochs=5,
    evaluation_strategy="epoch",
    logging_steps=1,
    # weight_decay=0.01,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=encoded_ds,
    eval_dataset=encoded_ds,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
    # data_collator=data_collator,
)

Context:
What happens if you drop your batch size (both train and eval) to 1? And set
max_steps = 16,
evaluation_strategy="steps"

Not sure if it will work fully, but might help diagnose what is going on


Question:
Building a custom label NER model for custom medical data. In my dataset, there are times where an entity may have non-entity words splitting it.
For a simple example, say I was designing data for to train for a person label that looked for first names. The sentence “His name was John Smith” would be O, O, O, B-PER, I-PER. That makes sense, but free-text gets messy. Imagine situations like this.
John, the man and legend, Smith, will be remembered forever.
Would Bert understand…
B-PER, O, O, O, O, I-PER, O O O O.
See how the split occurred? These should cause the same label, but I’m not sure if I should create IOB data as above, or have two separate instances of B-PER. The issue being. I want to model to understand that they are connected.
I’m playing with Bio-clinicalBert and it’s done well for ner. Just trying to get it to the next level.
Thanks in advanced, and I’d be happy to share more data if needed.

Context:
Anyone?
Will keep playing on my own in the meantime


Question: Hello! Is there a way to enable online learning on a model deployed on a Huggingface space (or the Model Hub). Like the flag option in gradio apps to flag some examples, is there an option for the user to correct the output predicted? Thanks.

Context:
Thats a super cool idea. The machinery to build this exists, but you would need to actually hook the building blocks together.
We’ll work on a tutorial to streamline this at some point


OSError: Unable to load weights from pytorch checkpoint file

Question:
Hi, everyone. I need some help. I have been developing the Flask website that has embedded one of Transformer’s fine-tuned models within it. I fine-tuned the model with PyTorch. I’ve tested the web on my local machine and it worked at all.
I used fine-tuned model that I’ve already saved the weight to use locally, as pictured in the figure below:
Annotation 2021-01-27 2031381142×127 37.7 KB
The saved results contain:

config.json
pytorch_model.bin
special_tokens_map.json
tokenizer_config.json
vocab.txt

Then, I tried to deploy it to the cloud instance that I have reserved. Everything worked well until the model loading step and it said:
OSError: Unable to load weights from PyTorch checkpoint file at <my model path/pytorch_model.bin>. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.
I’ve searched around the internet to solve it but still nil.  Can I get some enlightenment?
By the way, I’m using Ubuntu 18.04 instance and the environments that I’m used are:

torch 1.7.0
transformers 3.5.1

Thank you before!

Context: Hi @aswincandra were you able to load the tokenizer in the Flask app without problems? My first guess is that the path you are pointing to in the app is not correct.


Generate without using the generate method

Question:
Posting this here for visibility. What if you want to decode the output of a generative seq2seq model (like T5, BART, etc.) yourself, without using the .generate() method? The code example below illustrates this.
Suppose that the model is given a long text, for which it needs to generate a summary. We illustrate here how to manually decode the generated ids autoregressively. In each iteration, we add the predicted token id by the model to the decoder_input_ids, which are then fed as input to the next time step. At the beginning, we only feed the decoder_start_token_id to the decoder of the model.
from transformers import BartTokenizer, BartForConditionalGeneration
import torch

model_name = "sshleifer/distilbart-cnn-6-6"
tokenizer = BartTokenizer.from_pretrained(model_name)
model = BartForConditionalGeneration.from_pretrained(model_name)

text = """The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct."""

input_ids = tokenizer(text, return_tensors="pt").input_ids

decoder_input_ids = [model.config.decoder_start_token_id]
predicted_ids = []
for i in range(20):
    outputs = model(input_ids=input_ids, decoder_input_ids=torch.tensor([decoder_input_ids]))
    logits = outputs.logits[:,i,:]
    # perform argmax on the last dimension (i.e. greedy decoding)
    predicted_id = logits.argmax(-1)
    predicted_ids.append(predicted_id.item())
    print(tokenizer.decode([predicted_id.squeeze()]))
    # add predicted id to decoder_input_ids
    decoder_input_ids = decoder_input_ids + [predicted_id]

This will print:
The
 E
iff
el
 Tower
 is
 324
 metres
 (
1
,
06
3
 ft
)
 tall
,
 about
 the
 same

The final result can also be printed using print(tokenizer.decode(predicted_ids)):
The Eiffel Tower is 324 metres (1,063 ft) tall, about the same

Note that we’ve only done 20 time steps here. Normally, one continues until the model generates the EOS (end of sequence) token, which for BART is </s>.

Context: Hi Niels, thanks for sharing the code. Would you mind also sharing some examples of situations in which you would prefer not to use the .generate() method?


Starting my first pull request, but transformers tests stall

Question:
Hello,
I am trying to contribute to the library by fixing a bug that I’ve found some time ago. This is about to be my first PR here, so I am following exactly the contributing guide 1. Before making the changes to the current code, I’m running the tests to see how that goes:
make test
And this command takes way too long to be true (on a powerful machine with lots of CPUs and GPUs). In partihular, it hangs completely at some point and does not proceed at all with the recent output messages looking OK. The last message is below, but the tests run in parallel on 48 cores, so this may be useless.
[gw0] SKIPPED tests/test_modeling_flax_mbart.py::FlaxMBartModelIntegrationTest::test_batch_generation_en_ro
Is there a way to see, which test gets stuck? Can I skip it? Could anyone give me a piece of advice on how I should proceed?

Context: You should only test the test files related to your PR and let the CI double-check for you, to go faster.


How to deal with differences between CoNLL 2003 dataset tokenisation and BER tokeniser when fine tuning NER model?

Question:
Hello,
I am about to fine-tune a BERT model on the NER task using a legal dataset with custom entities, and would like to know how the fine tuning on the ConLL 2003 dataset was handled at the time in order to create a pertained BertForTokenClassification model, because I’m facing similar issues. The NER dataset here contains one token (or rather word) per line. However, the HuggingFace BERT tokenizer (e.g. “bert-base-cased” or any other) will not produce a one-to-one match with this dataset. Just to give an example, the word “precautionary” (which on the conll 2003 dataset would appear in one line) is split by the HuggingFace tokenizer into ['pre', '##ca', '##ution', '##ary'], and I assume the opposite might be true as well, although perhaps much rarer (i.e. that tokens which were split into two lines in the conll 2003 dataset would be tokenized by HuggingFace as a single token).
Therefore, I was wondering what transformation was done to convert the CoNLL 2003 dataset (in the format I linked above) to a set of token-level labels corresponding to the BERT tokenizer suitable for creating a pytorch’s DataLoader.

Context:
What is typically done is, you tokenize each word of the annotated dataset you have, check how many tokens it has been tokenized into, and then either only label the first wordpiece token, or label them all.
Small example:
Suppose you have the sentence “hello my name is Niels”, and the CoNLL dataset has this labeled as:
hello O
my O
name O
is O
niels B-PER

Then what we do is the following:

option 1, label all tokens of a word (i.e. propagate the label on all tokens)

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

words = ["hello", "my", "name", "is", "niels"]
word_labels = ["O", "O", "O", "O", "B-PER"]
# convert word labels to integer labels
label2id  = {"O": 0, "B-PER":1}
word_labels = [label2id[label] for label in word_labels]

tokens = []
labels = []
for word, label in zip(words, word_labels):
     # tokenize the word
     word_tokens = tokenizer.tokenize(word)
     # propagate label to all tokens of the given word
     tokens.extend(word_tokens)
     labels.extend([label] * len(word_tokens))

option 2, only label the first token of a word, set labels of all remaining tokens to -100

words = ["hello", "my", "name", "is", "niels"]
word_labels = ["O", "O", "O", "O", "B-PER"]
# convert word labels to integer labels
label2id  = {"O": 0, "B-PER":1}
word_labels = [label2id[label] for label in word_labels]

tokens = []
labels = []
for word, label in zip(words, word_labels):
     # tokenize the word
     word_tokens = tokenizer.tokenize(word)
     tokens.extend(word_tokens)
     # only label the first wordpiece
     labels.extend([label] + [-100] * (len(word_tokens) - 1))

The reason we set the remaining tokens to -100 is because this is the ignore_index of PyTorch’ CrossEntropyLoss 1. This means that those labels will not be taken into account by the loss function, hence no gradients will be computed for those.
Which of the 2 options you choose is a design choice, mainly. In practice, both perform well. I’ve made 2 notebooks (one for each of both options), in my repo here 2.


Question:
I tried look into this similar issue: Custom GPT2 Model won't load after training
but still…
class CustomClass(PreTrainedModel):
    def __init__(self, config, num_labels):
        super().__init__(config, num_labels)
        self.distilbert = DistilBertModel.from_pretrained('distilbert-base-uncased')
        self.pre_classifier = torch.nn.Linear(768, 768)
        self.dropout = torch.nn.Dropout(0.1)
        self.classifier = torch.nn.Linear(768, num_labels)

    def forward(self, input_ids, attention_mask):
        distilbert_output = self.distilbert(input_ids=input_ids, attention_mask=attention_mask)
        hidden_state = distilbert_output[0]
        pooled_output = hidden_state[:, 0]
        pooled_output = self.pre_classifier(pooled_output)
        pooled_output = torch.nn.Tanh()(pooled_output)
        pooled_output = self.dropout(pooled_output)
        output = self.classifier(pooled_output)
        return output

I was able to fine tune with a linear classifier for a classification job.
config = PretrainedConfig(name_or_path='own-model', num_labels=100, output_hidden_states=True)

model = CustomClass(config, 100)

I can also save it with
model.save_pretrained(PATH)

But when I try to load it with
new_model=PreTrainedModel.from_pretrained('./PATH/')

i got 'NoneType' object has no attribute 'from_pretrained' which is really strange
Alternatively, with a config file, this doesnt throw error though the model weights wont load
new_config=PretrainedConfig.from_pretrained('./PATH/') new_model=PreTrainedModel.from_pretrained('./PATH/', config=new_config)

please help. I am running out of idea. Oh and I am at v4.6.0
Thank you!

Context: You should use CustomClass.from_pretrained. PreTrainedModel.from_pretrained won’t work directly.


Is there a standard way to handle leftover batches when using gradient accumulation?

Question:
Let’s say that I have the following training loop that uses accumulated gradients (taken from here 1). Let’s also say that I have a batch size of 4 and want to accumulate gradients for 10 steps, which gives me an “effective” batch size of 40.
model.zero_grad()
for i, (inputs, labels) in enumerate(training_set):
    predictions = model(inputs)
    loss = loss_function(predictions, labels)
    loss = loss / accumulation_steps
    loss.backward()
    if (i+1) % accumulation_steps == 0:
        optimizer.step()
        model.zero_grad()
        if (i+1) % evaluation_steps == 0:
            evaluate_model()

I was wondering what we do with gradient accumulation when we have leftover batches in our training loop.
For the above code, we might see this if we have (say) 57 batches in our training step. This would lead to 5 successful training steps of gradient accumulation (corresponding to the first 50 batches), but the last 7 batches would be ignored.
I’m guessing the convention is just to ignore the leftover batches (particularly if you are shuffling the batches in each epoch), but perhaps it might be better to do a training step instead?
Thoughts appreciated.

Context: You should use a step counter that goes over all the training loop instead of the counter step, so that you will finish your batch of epoch 0 during epoch 1 (unless your dataset is pretty small, the probablity of having the same samples twice is not super high).


Error Training Vision Encoder Decoder for Image Captioning

Question:
I am trying to train Vision Encoder Decoder with VIT encoder and Hindi GPT2(surajp/gpt2-hindi at main) decoder, for Hindi Image captioning, which my team are doing as a part of Huggingface course project.
Currently my code is this
For creating a dataset
import torch
from torch.utils.data import Dataset
from PIL import Image

class Image_Caption_Dataset(Dataset):
    def __init__(self,root_dir,df, feature_extractor,tokenizer,max_target_length=128):
        self.root_dir = root_dir
        self.df = df
        self.feature_extractor = feature_extractor
        self.tokenizer = tokenizer
        self.max_length=max_target_length

    def __len__(self,df):
        return self.df.shape[0]

    def __getitem__(self,idx):
        #return image
        image_path = self.df['images'][idx]
        text = self.df['text'][idx]
        #prepare image
        image = Image.open(self.root_dir+'/'+image_path).convert("RGB")
        pixel_values = self.feature_extractor(image, return_tensors="pt").pixel_values
        #add captions by encoding the input
        captions = self.tokenizer(text,
                                 padding='max_length',
                                 max_length=self.max_length).input_ids
        captions = [caption if caption != self.tokenizer.pad_token_id else -100 for caption in captions]
        encoding = {"pixel_values": pixel_values.squeeze(), "labels": torch.tensor(captions)}
        return encoding

from transformers import ViTFeatureExtractor,AutoTokenizer

encoder_checkpoint = 'google/vit-base-patch16-224'
decoder_checkpoint = 'surajp/gpt2-hindi'

feature_extractor = ViTFeatureExtractor.from_pretrained(encoder_checkpoint)
tokenizer = AutoTokenizer.from_pretrained(decoder_checkpoint)

root_dir = "../input/flickr8k/Images"

train_dataset = Image_Caption_Dataset(root_dir=root_dir,
                           df=train_df,
                           feature_extractor=feature_extractor,
                                     tokenizer=tokenizer)
val_dataset = Image_Caption_Dataset(root_dir=root_dir,
                           df=test_df,
                           feature_extractor=feature_extractor,
                                     tokenizer=tokenizer)

from transformers import VisionEncoderDecoderModel
# initialize a vit-bert from a pretrained ViT and a pretrained BERT model. Note that the cross-attention layers will be randomly initialized
model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained(encoder_checkpoint, decoder_checkpoint)
#model.to(device)

After initializing the model I configured the model arguments

# set special tokens used for creating the decoder_input_ids from the labels
model.config.decoder_start_token_id = tokenizer.cls_token_id
model.config.pad_token_id = tokenizer.pad_token_id
# make sure vocab size is set correctly
model.config.vocab_size = model.config.decoder.vocab_size

# set beam search parameters
model.config.eos_token_id = tokenizer.sep_token_id
model.config.max_length = 128
model.config.early_stopping = True
model.config.no_repeat_ngram_size = 3
model.config.length_penalty = 2.0
model.config.num_beams = 4

Then started Training

from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments

training_args = Seq2SeqTrainingArguments(
    predict_with_generate=True,
    evaluation_strategy="steps",
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    fp16=True,
    output_dir="./",
    logging_steps=2,
    save_steps=1000,
    eval_steps=200,
)

from transformers import default_data_collator

# instantiate trainer
trainer = Seq2SeqTrainer(
    model=model,
    tokenizer=feature_extractor,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    data_collator=default_data_collator,
)
trainer.train()

But then I got the error, ValueError: Make sure to set the decoder_start_token_id attribute of the model’s configuration
If you noticed the above code, I have already set this, but when I checked that the tokenizer.cls_token_id was None so I manually set the tokenizer.cls_token_id=’’
but then I got the Index out of range in self, error
Is there any workaround for this, the code is inspired for` here written by @nielsr
I have also tried with custom training loop, I get the same error

Context:
Hi,
You were already on the good way! The only “mistake” I see here is that GPT2 doesn’t have a CLS token. The CLS token is only defined for encoder-only Transformers such as BERT, RoBERTa. So in this case, the decoder start token can be set to the bos (beginning of sequence) token:
model.config.decoder_start_token_id = tokenizer.bos_token_id


How to save model in Colab during TPU training with Accelerate

Question:
I’m training a model on a TPU on Colab and my code is based on this example (with the training loop wrapped in a large function): accelerate/nlp_example.py at main · huggingface/accelerate · GitHub
How do I save to model trained on the TPU to my google drive?
I understand that I need to run a code like this in the training function:
accelerator.wait_for_everyone()
unwrapped_model = accelerator.unwrap_model(model)
accelerator.save(unwrapped_model.state_dict(), f'./results/{training_directory}')

But then I get the error below. I know how to save a model when it’s in my environment, but I’m not sure how to save it when it’s in the large training function and on the TPU.
Exception in device=TPU:0: [Errno 21] Is a directory: './results/nli-few-shot/TPU/'
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 329, in _mp_start_fn
    _start_fn(index, pf_cfg, fn, args)
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 323, in _start_fn
    fn(gindex, *args)
  File "/usr/local/lib/python3.7/dist-packages/accelerate/utils.py", line 570, in __call__
    self.launcher(*args)
  File "<ipython-input-84-275217ef36aa>", line 97, in training_function
    accelerator.save(unwrapped_model.state_dict(), f'./results/{training_directory}')
  File "/usr/local/lib/python3.7/dist-packages/accelerate/accelerator.py", line 507, in save
    save(obj, f)
  File "/usr/local/lib/python3.7/dist-packages/accelerate/utils.py", line 544, in save
    xm.save(obj, f)
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/core/xla_model.py", line 818, in save
    torch.save(cpu_data, file_or_path)
  File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 376, in save
    with _open_file_like(f, 'wb') as opened_file:
  File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 230, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 211, in __init__
    super(_open_file, self).__init__(open(name, mode))
IsADirectoryError: [Errno 21] Is a directory: './results/nli-few-shot/TPU/'
Exception in device=TPU:6: tensorflow/compiler/xla/xla_client/mesh_service.cc:364 : Failed to meet rendezvous 'torch_xla.core.xla_model.save': Socket closed (14)
Traceback (most recent call last):
Exception in device=TPU:5: tensorflow/compiler/xla/xla_client/mesh_service.cc:364 : Failed to meet rendezvous 'torch_xla.core.xla_model.save': Socket closed (14)
Exception in device=TPU:7: tensorflow/compiler/xla/xla_client/mesh_service.cc:364 : Failed to meet rendezvous 'torch_xla.core.xla_model.save': Socket closed (14)
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 329, in _mp_start_fn
    _start_fn(index, pf_cfg, fn, args)
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 329, in _mp_start_fn
    _start_fn(index, pf_cfg, fn, args)
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 329, in _mp_start_fn
    _start_fn(index, pf_cfg, fn, args)
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 323, in _start_fn
    fn(gindex, *args)
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 323, in _start_fn
    fn(gindex, *args)
  File "/usr/local/lib/python3.7/dist-packages/accelerate/utils.py", line 570, in __call__
    self.launcher(*args)
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 323, in _start_fn
    fn(gindex, *args)
  File "/usr/local/lib/python3.7/dist-packages/accelerate/utils.py", line 570, in __call__
    self.launcher(*args)
  File "/usr/local/lib/python3.7/dist-packages/accelerate/utils.py", line 570, in __call__
    self.launcher(*args)
  File "<ipython-input-84-275217ef36aa>", line 97, in training_function
    accelerator.save(unwrapped_model.state_dict(), f'./results/{training_directory}')
  File "<ipython-input-84-275217ef36aa>", line 97, in training_function
    accelerator.save(unwrapped_model.state_dict(), f'./results/{training_directory}')
  File "/usr/local/lib/python3.7/dist-packages/accelerate/accelerator.py", line 507, in save
    save(obj, f)
  File "/usr/local/lib/python3.7/dist-packages/accelerate/accelerator.py", line 507, in save
    save(obj, f)
  File "/usr/local/lib/python3.7/dist-packages/accelerate/utils.py", line 544, in save
    xm.save(obj, f)
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/core/xla_model.py", line 819, in save
    rendezvous('torch_xla.core.xla_model.save')
  File "/usr/local/lib/python3.7/dist-packages/accelerate/utils.py", line 544, in save
    xm.save(obj, f)
  File "<ipython-input-84-275217ef36aa>", line 97, in training_function
    accelerator.save(unwrapped_model.state_dict(), f'./results/{training_directory}')
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/core/xla_model.py", line 819, in save
    rendezvous('torch_xla.core.xla_model.save')
  File "/usr/local/lib/python3.7/dist-packages/accelerate/accelerator.py", line 507, in save
    save(obj, f)
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/core/xla_model.py", line 863, in rendezvous
    return torch_xla._XLAC._xla_rendezvous(get_ordinal(), tag, payload, replicas)
RuntimeError: tensorflow/compiler/xla/xla_client/mesh_service.cc:364 : Failed to meet rendezvous 'torch_xla.core.xla_model.save': Socket closed (14)
  File "/usr/local/lib/python3.7/dist-packages/accelerate/utils.py", line 544, in save
    xm.save(obj, f)
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/core/xla_model.py", line 863, in rendezvous
    return torch_xla._XLAC._xla_rendezvous(get_ordinal(), tag, payload, replicas)
RuntimeError: tensorflow/compiler/xla/xla_client/mesh_service.cc:364 : Failed to meet rendezvous 'torch_xla.core.xla_model.save': Socket closed (14)
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/core/xla_model.py", line 819, in save
    rendezvous('torch_xla.core.xla_model.save')
  File "/usr/local/lib/python3.7/dist-packages/torch_xla/core/xla_model.py", line 863, in rendezvous
    return torch_xla._XLAC._xla_rendezvous(get_ordinal(), tag, payload, replicas)
RuntimeError: tensorflow/compiler/xla/xla_client/mesh_service.cc:364 : Failed to meet rendezvous 'torch_xla.core.xla_model.save': Socket closed (14)
---------------------------------------------------------------------------
ProcessExitedException                    Traceback (most recent call last)
<ipython-input-85-a91f3c0bb4fd> in <module>()
      1 from accelerate import notebook_launcher
      2
----> 3 notebook_launcher(training_function)

3 frames
/usr/local/lib/python3.7/dist-packages/torch/multiprocessing/spawn.py in join(self, timeout)
    142                     error_index=error_index,
    143                     error_pid=failed_process.pid,
--> 144                     exit_code=exitcode
    145                 )
    146

ProcessExitedException: process 0 terminated with exit code 17

Context: As the error indicates, you are trying to save in a directory, and not a file.


Question:
Hi, I was wondering if it is possible to extract the most positive word or the most negative word from a sentence. Say, we have a movie review like, “This movie is amazing” and the most relevant word for sentiment classification would be the word “amazing”. Is it possible to use a trained BERT model to extract/return such most prominent words from a sentence?
Thanks.

Context:
Hi,
Yes that’s possible. There are some cool libraries out there that can be used to do that:

ELI5: GitHub - TeamHG-Memex/eli5: A library for debugging/inspecting machine learning classifiers and explaining their predictions

LIME: GitHub - marcotcr/lime: Lime: Explaining the predictions of any machine learning classifier

I believe it requires some tweaks to make it work on Transformer-based models, but I did this myself some time last year, so it’s definitely possible.
Edit: this Github thread 3 might be helpful.


Multilabel classification performance metrics using Trainer API

Question:
Hello,
My goal is to output certain model performance metrics for my multilabel classification problem (I am using a DistilBERT architecture by the way). If I look at each of the labels individually you can say most of the labels are really unbalanced. Given this I also want to correct for the label (or class) imbalance.
I am fairly new to this and by looking at some examples, and trying myself I have done the following:
def accuracy_thresh(y_pred, y_true, thresh=0.5, sigmoid=True):
    y_pred = torch.from_numpy(y_pred)
    y_true = torch.from_numpy(y_true)
    if sigmoid:
        y_pred = y_pred.sigmoid()
    return ((y_pred>thresh)==y_true.bool()).float().mean().item()

The above code calculates model accuracy given a threshold of 0.5.
Next I used this code and I also included the above function as an output
def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    y_true = labels
    y_pred = sigmoid(eval_pred.predictions)
    y_pred = (y_pred>0.5).astype(float)

    clf_dict = classification_report(y_true, y_pred, target_names=all_labels,
                                         zero_division=0, output_dict=True)

    return {"accuracy_thresh": accuracy_thresh(predictions, labels), "micro f1": clf_dict['micro avg']['f1-score'], "macro f1": clf_dict['macro avg']['f1-score'],
           "weighted f1": clf_dict['weighted avg']['f1-score']}

It looks a bit hacky, but it works (it runs). I have the Trainer als follows:
class MultilabelTrainer(Trainer):
    def compute_loss(self, model, inputs, return_outputs=False):
        labels = inputs.pop("labels") #keeps the labels
        outputs = model(**inputs)
        logits = outputs.logits
        loss_fct = torch.nn.BCEWithLogitsLoss(pos_weight = class_weights)
        loss = loss_fct(logits.view(-1, self.model.config.num_labels),
                        labels.float().view(-1, self.model.config.num_labels))
        return (loss, outputs) if return_outputs else loss

Note: I am actually using pos_weights. Why? Since I am dealing with imbalanced labels as said above I have a tensor which contains for each label a weight calculated as number of negative cases / positive cases.
The trainer then is
multi_trainer = MultilabelTrainer(
    model,
    args,
    train_dataset=train_dataset,
    eval_dataset=test_dataset,
    compute_metrics=compute_metrics,
    tokenizer=tokenizer)

My main question is: Does it actually make sense for what I have done? That is am I actually getting the right performance metrics taking into account I want to correct for imbalance? Or is there a better alternative (e.g. less verbose) to achieve this?

Context:
Hi,
I’ve created a notebook for you to illustrate this: Transformers-Tutorials/Fine_tuning_BERT_(and_friends)_for_multi_label_text_classification.ipynb at master · NielsRogge/Transformers-Tutorials · GitHub 7
Actually, there’s no need for a MultilabelTrainer anymore, as you can just set the problem_type of the model’s configuration to “multi_label_classification”.


MLflowCallback TypeError: can only concatenate list (not “type”) to list

Question:
I’m trying to capture autolog parameters with MLFlow. I’m using the MLflowCallback class written in the transformers.integrations.MLflowCallback and interfacing with transformers.TrainerCallback class.
Here’s the relevant code that tells MLFlow to create an experiment, send it to a host tracking server, and tells transformers what type of logging to do (as defined by the MLflowCallback class).
import torch
from transformers import RobertaTokenizer, RobertaForSequenceClassification, RobertaConfig, Trainer, TrainingArguments
import mlflow
from transformers import TrainerCallback
from transformers.integrations import MLflowCallback

remote_server_uri = [PRIVATE SERVER URL]
mlflow.set_tracking_uri(remote_server_uri)

# After loading and tokenizing data, here we run the training experiment
experiment_name = "ht_vp_roberta_randomSearch"
mlflow.set_experiment(experiment_name)             # server creates experiment folder at this point
with mlflow.start_run():
    training_args = TrainingArguments(
        output_dir=experiment_name,
        evaluation_strategy='epoch',
        eval_steps=500,
        gradient_accumulation_steps=1000,
        eval_accumulation_steps=1,
    )
    model = RobertaForSequenceClassification.from_pretrained("roberta-base")
    trainer = Trainer(
        args=training_args,
        tokenizer=tokenizer,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        model=model,
        compute_metrics=hf.compute_metrics,
        callbacks=MLflowCallback,                       # This triggers error
    )
    trainer.train()
    trainer.evaluate()

I’m getting the below error:
Traceback (most recent call last):
  File "mlflow_test_simple.py", line 80, in <module>
    trainer = Trainer(
  File "/home/jovyan/conda/dsEnv/lib/python3.8/site-packages/transformers/trainer.py", line 385, in __init__
    callbacks = default_callbacks if callbacks is None else default_callbacks + callbacks
TypeError: can only concatenate list (not "type") to list

Context:
I just ran into the same problem.
I have just encountered the same problem

However, now I’m getting a new error, about nested processes, and mlflow wanting me to set nested=True in the start_run call that is embedded in the MLflowCallback class

And found out the cause of this error.
The cause of this problem was that Trainer’s __init__ automatically added a callback based on the installed package.
As a result, the callback given as an argument to Trainer was duplicated and mlflow.start_run() was executed twice.
The solution is to not specify the mlflow callback as an argument.
trainer = Trainer(
    args=training_args,
    tokenizer=tokenizer,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    model=model,
    compute_metrics=compute_metrics,
)

As a side note, I found the cause in lines 391-395 of trainer.py.
get_reporting_integration_callbacks(self.args.report_to) returns all integration callbacks installed and added it into callback handler.
default_callbacks = DEFAULT_CALLBACKS + get_reporting_integration_callbacks(self.args.report_to)
callbacks = default_callbacks if callbacks is None else default_callbacks + callbacks
self.callback_handler = CallbackHandler(
    callbacks, self.model, self.tokenizer, self.optimizer, self.lr_scheduler
)

If you’ve already solved it, I’ll leave the solution for those who are facing this problem anew.


Saving model per some step when using Trainer

Question:
When using the Trainer and TrainingArguments from transformers, I notice that by default, the Trainer save a model every 500 steps. How can I change this value so that it save the model more/less frequent?
here is a snipet that i use
  training_args = TrainingArguments(
        output_dir=output_directory,          # output directory
        num_train_epochs=10,              # total number of training epochs
        per_device_train_batch_size=16,  # batch size per device during training
        per_device_eval_batch_size=64,   # batch size for evaluation
        warmup_steps=500,                # number of warmup steps for learning rate scheduler
        weight_decay=0.01,               # strength of weight decay
        logging_dir=log_directory,            # directory for storing logs
    )

    trainer = Trainer(
        model=model,                         # the instantiated 🤗 Transformers model to be trained
        args=training_args,                  # training arguments, defined above
        train_dataset=train_dataset,         # training dataset
        eval_dataset=val_dataset             # evaluation dataset
    )

    trainer.train()

Context:
This is explained in the documentation 2.
You can change the argument “save_steps”, which defaults to 500.


Question:
I have a special non-language use case using a fixed vocabulary—i.e., a relatively small set of generated tokens that represent the entire vocabulary of our “language.” I’d like to be able to use this with any of the different models and I’m wondering what would be the best approach? It’s just a vocab.txt file of short strings, which I don’t think will work with any of the BPE tokenizers. Am I correct in that assumption? Also, is there a way to “force” a vocabulary onto any of the tokenizers?
Any help much appreciated.

Context:
What if I just instantiate a tokenizer (e.g., BigBirdTokenizer), then used add_tokens() to add my entire vocabulary? That is, start with nothing, then “force” them in with the add_tokens() function… ??
UPDATE: Hmm… no… I can’t instantiate without a vocab file, so that won’t work…


Resuming training BERT from scratch with run_mlm.py

Question:
Initiated training BERT from scratch with run_mlm.py as follows:
python run_mlm.py     --model_type bert
–train_file ./data/mk.txt     --output_dir ./models/bert-base-uncased
–overwrite_output_dir     --tokenizer_name ./models/bert-base-uncased
–line_by_line True     --do_train
–per_device_train_batch_size 4     --num_train_epochs 100
–save_steps 100000     --save_total_limit 500
–max_seq_length 512     --logging_steps 500
–use_fast_tokenizer     --report_to wandb
–disable_tqdm True `
Training stopped due to power outage, having saved latest checkpoint:
.\models\bert-base-uncased\checkpoint-1700000
Which is the most appropriate command, give initial one, to resume training from the last saved checkpoint, and preserving all of the parameters mentioned above?

Context:
hi @striki-ai
if you remove the --overwrite_output_dir option and run the same command again, then the script will detect the last checkpoint and resume training from there.


How can I make a Img2Text transformer using the existent modules?

Question:
I am trying to build a captioning system that inputs an image and outputs a caption.
This is an attempt to solve the Kaggle competition: Bristol-Myers Squibb – Molecular Translation | Kaggle.
For that I tried to build a Bert encoder-decoder module with ViT Embeddings using the following code:
class ViTBert(nn.Module):

  def __init__(self, vocab_size):
    super().__init__()
    config_encoder = BertGenerationConfig(
            vocab_size = vocab_size,
            hidden_size = 256,
            num_hidden_layers = 4,
            num_attention_heads = 4,
            intermediate_size = 1024,
            bos_toke_id = 0,
            eos_token_id = 2
    )
    config_decoder = BertGenerationConfig(
            vocab_size = vocab_size,
            hidden_size = 256,
            num_hidden_layers = 4,
            num_attention_heads = 4,
            intermediate_size = 1024,
            add_cross_attention=True,
            is_decoder=True,
            bos_toke_id = 0,
            eos_token_id = 2
    )       config_encdec = EncoderDecoderConfig.from_encoder_decoder_configs(config_encoder, config_decoder)
    config_emb = ViTConfig(
        hidden_size = 256,
        num_hidden_layers = 4,
        num_attention_heads = 4,
        intermediate_size = 1024,
        image_size = 224,
        patch_size = 16,
        num_channels = 1
    )
    self.emb = ViTModel(config_emb).embeddings
    self.model = EncoderDecoderModel(config_encdec)

  def forward(self, x, y = None, l = None):

    return self.model(
        inputs_embeds = self.emb(x),
        decoder_input_ids = y,
        labels = l
        )

  def beam_search(self, x, y, beam_scorer, criteria):
    lhs = self.model.encoder(inputs_embeds = self.emb(x)).last_hidden_state

    return self.model.decoder.beam_search(input_ids = y,
                                   encoder_hidden_states = lhs,
                                   beam_scorer=beam_scorer,
                                   stopping_criteria = criteria)

model = ViTBert(len(vocab))

The first thing I did was to check if it would work passing random inputs, here I mimic a batch of 3 images of sizes [1, 224, 224] and text inputs and labels of shape [batch, seq_len] in the integer range of 0 to 41 (my vocab length).
x = torch.rand((3, 1, 224, 224))
y = torch.randint(0, 41, (3, 50))
l = torch.randint(0, 41, (3, 50))
model(x, y, l).keys()

odict_keys([‘loss’, ‘logits’, ‘encoder_last_hidden_state’])

I train it using the ‘loss’ provided by the model for a few epochs but when I try to run predictions I get the same output for different images. I did some testing and although the logits are slightly different for different input images, their argmax are always the same.
x = torch.rand((2, 1, 224, 224))
y = torch.cat([torch.randint(0, 41, (1, 50))]*2)
preds= model(x, y).logits.argmax(dim=2)
all(preds[0] == preds[1])

True

Not surprisingly, when I run the beam_search I always get the same result
beam_scorer = BeamSearchScorer(
    batch_size=2,
    num_beams=6,
    device=model.model.decoder.device
)
criteria = StoppingCriteriaList([MaxLengthCriteria(100)])
res = model.beam_search(
     torch.randn((2, 1, 224, 224)),
     torch.cat([torch.tensor([[0]])]*12),
     beam_scorer, criteria)
all(res[0] == res[1])

True

I am just started learning transformers a few weeks ago and my knowledge is very shallow, so any advice would be helpful (even if not directly related with this problem).
If you interested in checking my colab notebook can be accessed here: Google Colaboratory 6
The notebook is a bit messy but I enabled commentary on that notebook so if you feel like writing down anything feel free to do so =)
Looking forward to any comment.
Sincerely,
Passos.

Context: Any luck with this task?


Stopping `model.generate()` based on custom token

Question: Hello everyone, I’ve managed to train a huggingface model that generates coherent sequences based on my training data and am using generate to create these new sequences. This has worked well enough so far however I need to stop sequence generation based on the count of a particular token that denotes the start of a subsequence in my domain. Is there a way to leverage the generate() method to do this? ie rather than generate based on length generate until n number of a particular token are generated.

Context: I found that the best way to do this is by directly calling the model with the necessary inputs rather than using the generate method, and to build logic around this that checks the number of a particular token in the resulting sequence and stops once its reached.


Question:
My code is written in pytorch, thus I use torch.optim.adam as my optimizer. However, I need to do use Adam wright decay with some layer excluded. To be more specifically, I am trying to reproduce this tensor flow code
optimizer = AdamWeightDecayOptimizer(*some parameter setting here,*    exclude_from_weight_decay=["LayerNorm", "layer_norm", "bias"])
What should I do to exclude [“LayerNorm”, “layer_norm”, “bias”] in weight decay in pytorch? Could I use tensor flow optimizer in pytorch?
Thank you.

Context: You can look at how we do this for the Trainer here 23.


How to train the embedding of special token?

Question:
I would like to add some special tokens and train the tokens. For instance, this is an input example of BERT    "[CLS] this is a special token special_token [SEP] The special token is ‘special_token’ ". I guess I should use some functions like  tokenizer. additional_special_tokens(‘special_token’) to add the special token. How do I train the embedding of the token?
Thank you

Context:
There are 2 things you need to do in order to train additional special tokens:

Add new tokens to the tokenizer.

You can either add “regular” tokens, as follows:
tokenizer.add_tokens(['newWord', 'newWord2'])
Or you can add them as special tokens (similar to [CLS] and [SEP]) by passing the additional argument special_tokens=True. This is equivalent to calling tokenizer.add_special_tokens (the latter accepts a dictionary rather than a list).
special_tokens_dict = {'additional_special_tokens': ['[C1]','[C2]','[C3]','[C4]']}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)

Resize the token embedding matrix of the model, so that it matches with the tokenizer:

model.resize_token_embeddings(len(tokenizer))
Next, you can fine-tune your model on your custom dataset, and train these additional tokens.
Sources:

How to add some new special tokens to a pretrained tokenizer? · Issue #247 · huggingface/tokenizers · GitHub 2
how can i finetune BertTokenizer? · Issue #2691 · huggingface/transformers · GitHub 4

source code 1 of tokenization_utils_base.py


Other aggregation on TAPAS beyond (SUM/COUNT/AVERAGE/NONE)

Question:
In the current way to fine-tune the model, is it possible to train TAPAS to learn other aggregations such difference, percentages etc ?
If it is possible, can you please point to some documentation?

Context:
Hi,
Yes it is possible to train TAPAS on other custom aggregations. You can change the number of aggregation operators in TapasConfig, like so:
from transformers import TapasConfig

config = TapasConfig(num_aggregation_heads=10)

and then initialize a TapasForQuestionAnswering model with a pre-trained base and your custom head on top:
from transformers import TapasForQuestionAnswering

model = TapasForQuestionAnswering.from_pretrained('google/tapas-base', config=config)

For more information, see the fine-tuning guide of TAPAS here 1.


Using Transformers with DistributedDataParallel — any examples?

Question:
Hi!
I’ve been consulting this page:

      huggingface.co

Model Parallelism 2

  Parallelism overview: In the modern machine learning the various approaches to parallelism are used to: fit very large models onto limited hardware - e.g. t5...

that says using DDP with transformers is “almost trivial”. But is there an example available?
Am I supposed to follow
https://pytorch.org/tutorials/intermediate/ddp_tutorial.html
just as if I was working with a regular PyTorch model and its optimizer exposed (as opposed to having it abstracted via transformers.Trainer)?
Also, I have some Dataset-related questions. I’ve written a custom dataset class that extends torch.Dataset. My dataset class yields samples from stored binary chunks with pre-shuffled pre-tokenized data (to maximize reading speed within a chunk). Therefore, I had to disable Trainer’s shuffling behavior by replacing RandomSampler with SequentialSampler within Trainer._get_train_sampler.
Will this hack work with DDP? Would it work if I switched to another distributed backend, like deepspeed? Is there a better way to do this?

Context: Then you just need to properly launch your training script, see here 1.


Question:
[ EDIT ] : there is a bug in the 4.11.0. Back to 4.9.2 solves the issue related here (but can create others? like this one  ByT5 tokenizer gives indices of chars instead of bytes?)

Hi.
I’ve created a Colab notebook to show a problem when using google/byt5-small from the model hub of Hugging Face and model.generate() .
Observations:

More especifically, the problem comes from the method tokenizer.convert_tokens_to_string() in source code for transformers.models.byt5.tokenization_byt5.

The same problem happens with google/byt5-base.

If someone could run my notebook and tell me what I did wrong or what could be a solution, I would appreciate it because this problem, besides preventing using ByT5 in inference, prevents its finetuning since when evaluating the model at the end of an epoch, the method tokenizer.convert_tokens_to_string() is called by the script … which suddenly fails). Thanks.
cc @patrickvonplaten, @valhalla, @sshleifer

Screen shots from the notebook

image1502×562 107 KB

image1720×950 72 KB

image1920×1030 82.4 KB

Context: @patrickvonplaten closed this issue (see explanation) with the return of errors="ignore" in decode("utf-8", errors="ignore") (see commit).


How to get a model on patent data for question answering

Question:
Dear list,
I want to have a question answering model for US patent text. For example, I want to ask it to read a patent’s text and ask questions such as ‘what is the specific problem to solve in this text?’. I tried with some general question answering models such as ‘distilbert-base-cased-distilled-squad’ but the answers were not satisfactory.
Now I am considering if I can get a better model through fine-tuning the model with patent data. So, I wonder if this is the right approach and if it is, then how can I fine-tune a model with patent data so that I can get more satisfactory answers?
Thanks in advance.

Context:
seunghon:

a question answering model for US patent text. For example, I want to ask it to read a patent’s text and ask questions such as ‘what is the specific problem to solve in this text?’. I tried with some general question answering models such as ‘distilbert-base-cased-

You’re going to have to finetune as you said, luckily you can finetune Squad pretty easily. See here:

      github.com

    transformers/examples/pytorch/question-answering at master ·... 1

  master/examples/pytorch/question-answering

  🤗 Transformers: State-of-the-art Natural Language Processing for Pytorch, TensorFlow, and JAX. - transformers/examples/pytorch/question-answering at master · huggingface/transformers

In a Python Notebook, import your data into a Pandas dataframe and export the table so that it matches the schema of the Squad dataset (see Streamlit 1  ). In this case, you need 5 fields in your exported file: id, title, context, question and answers.
Once you’ve formatted your data to the schema and exported the JSON/CSV locally, run the run_qa.py file and pass the train and test/validation files like so;
python run_qa.py \
  --model_name_or_path bert-base-uncased \
  --train_file=train-v1.1.json \
  --validation_file=dev-v1.1.json

And of course pass any other (hyper)parameters that you have for your finetuning task.


ERROR: vars() argument must have __dict__ attribute when trying to use trainer.train()?

Question:
I have the following model that I am trying to fine-tune (CLIP_ViT + classification head). Here’s my model definition:
class CLIPNN(nn.Module):

    def __init__(self, num_labels, pretrained_name="openai/clip-vit-base-patch32", dropout=0.1):
        super().__init__()
        self.num_labels = num_labels
        # load pre-trained transformer & processor
        self.transformer = CLIPVisionModel.from_pretrained(pretrained_name)
        self.processor = CLIPProcessor.from_pretrained(pretrained_name)
        # initialize other layers (head after the transformer body)
        self.classifier = nn.Sequential(
            nn.Linear(512, 128, bias=True),
            nn.ReLU(inplace=True),
            nn.Dropout(p=dropout, inplace=False),
            nn.Linear(128, self.num_labels, bias=True))

        def forward(self, inputs, labels=None, **kwargs):
            logits = self.classifier(inputs)
            loss = None
            if labels is not None:
                loss_fct = nn.CrossEntropyLoss()
                loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))

            return SequenceClassifierOutput(
                loss=loss,
                logits=logits,
            )

I also have the following definition for a dataset:
class CLIPDataset(nn.utils.data.Dataset):
    def __init__(self, embeddings, labels):
        self.embeddings = embeddings
        self.labels = labels

    def __getitem__(self, idx):
        item = {"embeddings": nn.Tensor(self.embeddings[idx])}
        item['labels'] = nn.LongTensor([self.labels[idx]])
        return item

    def __len__(self):
        return len(self.labels)

Note: here I am assuming that the model is fed pre-computed embeddings and does not compute embeddings, I know this is not the right logic if I want to fine-tune the CLIP base model, I am just trying to get my code to work.
Something like this throws an error:
model = CLIPNN(num_labels=2)
train_data = CLIPDataset(train_data, y_train)
test_data = CLIPDataset(test_data, y_test)

trainer = Trainer(
    model=model, args=training_args, train_dataset=train_data, eval_dataset=test_data
)
trainer.train()

TypeError                                 Traceback (most recent call last)
 in
----> 1 trainer.train()
~/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/transformers/trainer.py in train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
1256             self.control = self.callback_handler.on_epoch_begin(args, self.state, self.control)
1257
 → 1258             for step, inputs in enumerate(epoch_iterator):
1259
1260                 # Skip past any already trained steps if resuming training
~/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/utils/data/dataloader.py in next(self)
515             if self._sampler_iter is None:
516                 self._reset()
 → 517             data = self._next_data()
518             self._num_yielded += 1
519             if self._dataset_kind == _DatasetKind.Iterable and \
~/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/utils/data/dataloader.py in _next_data(self)
555     def _next_data(self):
556         index = self._next_index()  # may raise StopIteration
 → 557         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
558         if self._pin_memory:
559             data = _utils.pin_memory.pin_memory(data)
~/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
45         else:
46             data = self.dataset[possibly_batched_index]
—> 47         return self.collate_fn(data)
~/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/transformers/data/data_collator.py in default_data_collator(features, return_tensors)
64
65     if return_tensors == “pt”:
—> 66         return torch_default_data_collator(features)
67     elif return_tensors == “tf”:
68         return tf_default_data_collator(features)
~/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/transformers/data/data_collator.py in torch_default_data_collator(features)
80
81     if not isinstance(features[0], (dict, BatchEncoding)):
—> 82         features = [vars(f) for f in features]
83     first = features[0]
84     batch = {}
~/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/transformers/data/data_collator.py in (.0)
80
81     if not isinstance(features[0], (dict, BatchEncoding)):
—> 82         features = [vars(f) for f in features]
83     first = features[0]
84     batch = {}
TypeError: vars() argument must have dict attribute

Any clue where I’m going wrong?

Context: It looks like your train_data variable is used for two different things. Are you sure you passed the instance of your CLIPDataset to the Trainer? Cause it looks like the elements of the training dataset of the Trainer are not dictionaries from the error message.