“AttributeError: ‘Seq2SeqTrainer’ object has no attribute ‘repo’” after running trainer.push_to_hub()

Question:
Just fine-tuned pegasus-large on Google Colab Pro.
I create a Seq2SeqTrainer like so:
trainer = Seq2SeqTrainer(
    model=model,
    args=args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["test"],
    data_collator=data_collator,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics

And after running trainer.train(), I execute:
!huggingface-cli login
!pip install hf-lfs
!git config --global user.email "jakemsc@example.com"
!git config --global user.name "JakeMSc"
trainer.push_to_hub("test_model")

Which then gives this output:
Saving model checkpoint to ./results
Configuration saved in ./results/config.json
Model weights saved in ./results/pytorch_model.bin
tokenizer config file saved in ./results/tokenizer_config.json
Special tokens file saved in ./results/special_tokens_map.json
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-21-a676d2b17752> in <module>()
----> 1 trainer.push_to_hub('test-model')

/usr/local/lib/python3.7/dist-packages/transformers/trainer.py in push_to_hub(self, commit_message, **kwargs)
   2513             return
   2514
-> 2515         return self.repo.push_to_hub(commit_message=commit_message)
   2516
   2517     #

AttributeError: 'Seq2SeqTrainer' object has no attribute 'repo'

System information:

transformers version: 4.8.2
Platform: Linux-5.4.104±x86_64-with-Ubuntu-18.04-bionic
Python version: 3.7.10
PyTorch version (GPU?): 1.9.0+cu102 (True)
Tensorflow version (GPU?): 2.5.0 (True)
Flax version (CPU?/GPU?/TPU?): not installed (NA)
Jax version: not installed
JaxLib version: not installed
Using GPU in script?: Yes
Using distributed or parallel set-up in script?: No

Can’t see this error being discussed anywhere. Any advice?

Context:
hey @JakeMSc
this error is a bit odd because it suggests the Trainer.init_git_repo function is not being called which can only happen if TrainingArguments.push_to_hub is not set to True.
in particular, i could not reproduce your error by pushing a seq2seq model to the hub with the official translation tutorial here: Google Colaboratory
perhaps you can also try running the push to hub part of the tutorial notebook in your environment to see if it’s a problem in your configuration?


Question:
I was referring to the following blog to convert bert model to onnx.

      Medium – 17 Jun 20

Accelerate your NLP pipelines using Hugging Face Transformers and ONNX Runtime 2

  This post was written by Morgan Funtowicz from Hugging Face and Tianlei Wu from Microsoft

    Reading time: 6 min read

here, to take the inference of bert tokenizer, I’ll have to pass the 2d arrays.
Is there a way, where I’ll be able to pass sentence as input to the onnx tokenizer and get encodings as output, so that I’ll be able to use the model platform-independent

Context:
hi @hasak,

 hasak:

Is there a way, where I’ll be able to pass sentence as input to the onnx tokenizer and get encodings as output, so that I’ll be able to use the model platform-independent

the tokenizer is independent of onnx / onnxruntime, so you could create a simple function that converts your string inputs into the numpy format that the onnxruntime session expects:
tokenizer = ...

def prepare_for_session(input: str):
    tokens = tokenizer(input, return_tensors="pt")
    return {k: v.cpu().detach().numpy() for k, v in tokens.items()}

does this answer your question?


Set the format of the datasets to return pytorch tensors return list of tensors but why?

Question:
Hello,
I am folllowing this tutorial to use Fine-tuning a pretrained model — transformers 4.7.0 documentation 4 in order to use the flauBert to produce embeddings to train my classifier. In one of the lines , I have to set my dataset to pytorch tensors but when applying that line I get a list format which I do not understand. When printing element of the dataset I get tensors but when trying to pass the “input_ids” to the model , it is actually a list so the model cannot treat the data. Could help me figure it out why I get list and not a pytoch tensors when using 'set_format to torch.

def get_flaubert_layer(texte, path_to_lge): # last version

    tokenized_dataset, lge_size = preprocessed_with_flaubert(texte, path_to_lge)
    print("Set data to torch format...")
    tokenized_dataset.set_format(type='torch', columns=['input_ids', 'token_type_ids', 'attention_mask'])
    print("Format of data after -set.format- :", type(tokenized_dataset))
    print("Format of input_ids after -set.format- :", type(tokenized_dataset['input_ids']))
    print("Format of one element in the dataset", type(tokenized_dataset['input_ids'][0]))
    print(type(tokenized_dataset))
    print('Loading model...')
    flaubert = FlaubertModel.from_pretrained(path_to_lge)
    hidden_state = flaubert(input_ids=tokenized_dataset['input_ids'],
                            attention_mask=tokenized_dataset['attention_mask'])
    print(hidden_state[0][:, 0])
    cls_embedding = hidden_state[0][:, 0]
    print(cls_embedding)

#  test with data
path = '/gpfswork/rech/kpf/umg16uw/expe_5/model/sm'
print("Load model...")
flaubert = FlaubertModel.from_pretrained(path)
emb, s = get_flaubert_layer(data1, path)

Stacktrace and results
  0%|          | 0/4 [00:00<?, ?ba/s]
 25%|██▌       | 1/4 [00:00<00:01,  1.63ba/s]
 50%|█████     | 2/4 [00:01<00:01,  1.72ba/s]
 75%|███████▌  | 3/4 [00:01<00:00,  1.81ba/s]
100%|██████████| 4/4 [00:01<00:00,  2.41ba/s]
Traceback (most recent call last):
  File "/gpfs7kw/linkhome/rech/genlig01/umg16uw/test/expe_5/traitements/remove_noise.py", line 130, in <module>
    emb, s = get_flaubert_layer(data1, path)
  File "/gpfs7kw/linkhome/rech/genlig01/umg16uw/test/expe_5/traitements/functions_for_processing.py", line 206, in get_flaubert_layer
    hidden_state = flaubert(input_ids=tokenized_dataset['input_ids'],
  File "/linkhome/rech/genlig01/umg16uw/.conda/envs/bert/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/linkhome/rech/genlig01/umg16uw/.conda/envs/bert/lib/python3.9/site-packages/transformers/models/flaubert/modeling_flaubert.py", line 174, in forward
    bs, slen = input_ids.size()
AttributeError: 'list' object has no attribute 'size'
srun: error: r11i0n6: task 0: Exited with exit code 1
srun: Terminating job step 381611.0

real	1m37.690s
user	0m0.013s

format of the data passed to the flaubert model :
Loading tokenizer...
Transform data to format Dataset...
Set data to torch format...
Format of data after -set.format- : <class 'datasets.arrow_dataset.Dataset'>
Format of input_ids after -set.format- : <class 'list'>
Format of one element in the dataset <class 'torch.Tensor'>

As you see the columns input_ids is in a formal list and not tensors

Context: The reason could be that during tokenization, padding and/or truncation is not enabled which results in encoded inputs with different lengths. This would prevent the type conversion to convert input_ids to a tensor since its elements are of different size and the result would be a list of tensors.


Question:
Hello.
I was surprised that I only need to add a few lines of code to solve various tasks with the help of Bert. For exampe below is downstream task code for ML one:
  (cls): BertOnlyMLMHead(
    (predictions): BertLMPredictionHead(
      (transform): BertPredictionHeadTransform(
        (dense): Linear(in_features=768, out_features=768, bias=True)
        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
      )
      (decoder): Linear(in_features=768, out_features=30522, bias=True)
    )
)

or for next sequence prediction one:
    (pooler): BertPooler(
      (dense): Linear(in_features=768, out_features=768, bias=True)
      (activation): Tanh()
    )
  )
  (cls): BertOnlyNSPHead(
    (seq_relationship): Linear(in_features=768, out_features=2, bias=True)
  )

But in other side for me not clear why it works… Why for example solving ML task I use last_hidden_state from Bert model - but for BertForNextSentencePrediction the pooler output? Or if I want to suggest own downstream task - what should I do at first? I found a lot of materials explaining how BERTworks - but nothing explaining how to build downstream task.

Context:
During the pretraining procedure of BERT, there are two tasks: Masked Language Modeling and Next Sequence Prediction. Masked Language Prediction requires the model to make predictions for every token in the model, including the [MASK] token which is modeled by a layer that generates outputs over the entire vocabulary, hence the size 30522. The Next Sequence Prediction task uses the pooler layer in order to do a binary classification, namely whether the next sequence is a suitable continuation of the first sequence.
Therefore, the BertForPreTraining contains a BertPreTrainingHeads layer including both a language modeling head and a prediction head. However, for downstream language modeling tasks using BertForMaskedLM or BertForTokenClassification, the pooler layer is discarded since there is no need for sequence classification. Conversely, for downstream sequence classification tasks using BertForNextSentencePrediction or BertForSequencePrediction, the pooler layer is retained.


“Dump_all() got an unexpected keyword argument ‘sort_keys’” after running trainer.push_to_hub()

Question:
Just fine-tuned pegasus-large on Google Colab Pro with trainer.train(), then executed the following commands:
!huggingface-cli login
!pip install hf-lfs
!git config --global user.email "jakemsc@example.com"
!git config --global user.name "JakeMSc"
trainer.push_to_hub("test_model")

Which leads to the following error:
TypeError                                 Traceback (most recent call last)
<ipython-input-29-0cd57aa71f88> in <module>()
----> 1 trainer.push_to_hub("test_model")

3 frames
/usr/local/lib/python3.7/dist-packages/yaml/__init__.py in dump(data, stream, Dumper, **kwds)
    198     If stream is None, return the produced string instead.
    199     """
--> 200     return dump_all([data], stream, Dumper=Dumper, **kwds)
    201
    202 def safe_dump_all(documents, stream=None, **kwds):

TypeError: dump_all() got an unexpected keyword argument 'sort_keys'

Here are details of my system installation:

transformers version: 4.8.2
Platform: Linux-5.4.104±x86_64-with-Ubuntu-18.04-bionic
Python version: 3.7.10
PyTorch version (GPU?): 1.9.0+cu102 (True)
Tensorflow version (GPU?): 2.5.0 (True)
Flax version (CPU?/GPU?/TPU?): not installed (NA)
Jax version: not installed
JaxLib version: not installed
Using GPU in script?: Yes
Using distributed or parallel set-up in script?: No

Any ideas on how to resolve this?

Context: What is your version of pyaml? I think you need a more recent version (pip install --upgrade pyyaml). Will adjust the setup.


Question:
Hi,
can somebody points out where I can find the metric computation code in the HF?
e.g.  CIDEr, METEOR, ROUGE, BLEU
Thanks.

Context: hey @zuujhyt you can find all the scripts to compute metrics in the datasets library here: datasets/metrics at master · huggingface/datasets · GitHub 5


Defining a custom dataset for fine-tuning translation

Question:
I’m a first time user of the huggingface library. I am struggling to convert my custom dataset into one that can be used by the hugginface trainer for translation task with MBART-50 2. The languages I am trying to train on are a part of the pre-trained model, I am simply trying to improve the model’s translation capability for that specific pair.
My data is in the form of two plaintext files, with each file containing the sentences in one of the languages (sentences at the same line number form a pair.) I have tried using the csv file data loader, but I am unsure of what the column names need to be so that the tokenizer can identify which is the source and which the target. Also, trainer.train() function needs to correctly pick up on the same detail. There is also the issue that I’m trying to train the model to translate back and forth using the same dataset and I am unsure on how to accommodate for that. In the simpletransformers library, with MT5, I could easily do it by specifying the task using a prefix column.
Can someone point me towards a tutorial to load custom datasets for a translation task, similar to the tutorial given for loading a custom dataset for the sequence classification task 24? I assume the method is the same for any of the translation oriented models, so any assistance would be appreciated. If such a tutorial exists for other tasks such as summarization and sentiment analysis, those would also be very helpful. I apologize if it is an overly trivial question.

Context: This is exactly what I am trying to do too with no luck yet. Please let me know if you have found a way to do this.


Add a classification head to a fine-tuned language model

Question:
We I have fine-tuned a GPT-2 model with a language model head on medical triage text, and would like to use this model as a classifier. However, as far as I can tell, the Automodel Huggingface library allows me to have either a LM or a classifier etc. head, but I don’t see a way to add a classifier on top of a fine-tuned LM.
Am I mistaken in my understanding of the possibilities? We want to preserve the fine-tuning we have done for the LM task, so don’t want to swap this head for a classifier and lose all of those weights. I suppose we could unlock the top layers of the GPT2 model and try and push the training back into the model, but I suspect that would soon become untenable for our somewhat limited resources. I see this question has been asked here 7, and my fellow worker has posted a follow-up question in that thread, but we didn’t see any answer we could relate to, so I am asking again in this new topic.

Context: You need to use the AutoModelForSequenceClassification class to add a classification head on top of your pretrained model.


What should be shifted for decoder input for Bart

Question:
Hi,
In HF’s doc 1, regarding decoder_input_ids it says  ... create this tensor by shifting the `input_ids` to the right.....
Shouldn’t it be shifting the labels? because the input is noisy?
Thank you.

Context: I think this is copy pasted from the causal language modeling doc (for which your labels are your inputs shifted to the right).


Getting the MLM accuracy for the BERT model I am training from scratch

Question:
Hi
I am training a BERTforMaskedLM model from scratch.
This is my tokenizer (previously trained)
tokenizer = BertTokenizer('vocab.txt')

This is my config:
config = BertConfig(
    vocab_size=20000,
    max_position_embeddings=258
)

This is how I load the model from the last checkpoint:
model = BertForMaskedLM.from_pretrained("/BERT/bert-checkpoints/checkpoint-1726500",config=config)

My data collator:
data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)

The compute metrics function:

from datasets import load_metric

metric = load_metric("accuracy")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return {metric.compute(predictions=predictions, references=labels)}

The training arguments:
from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="./bert-checkpoints",
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=32,
    save_steps=500,
    save_total_limit=2,
    prediction_loss_only=True,
    evaluation_strategy = 'steps'
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
    eval_dataset=dataset,
    compute_metrics=compute_metrics
)

And finally, I use this to train the model (which works fine):
trainer.train()

However, when the model is being trained, I only see 3 metrics: Step, Training Loss, and	Validation Loss. I also want to see the “Accuracy of the Masked Language Model” (MLM accuracy). How should I do that? Note that I have already defined the “compute_metrics” function which has the “accuracy”. I do not know what is wrong. But the accuracy is not being shown.
Note: by the way, my dataset is an instance of the from torch.utils.data import Dataset object which has a member called “examples”. For instance, dataset.examples[0] is [2, 507, 157, 3656, 117, 2100, 521, 122, 280, 3]

Context: How many total steps are there in your training? Since you chose the "steps" strategy, I wonder if it’s just because evaluation is never run?


Question: I am new in this area. Please advise some good models to generate a summary on conversation between two persons. Thank you!

Context:
hey @MattJan a good place to start would be by looking at models fine-tuned on the samsum dataset (dialogues between two people + their summary): Hugging Face – The AI community building the future. 12
if you want to fine-tune your own model, a good start would be to use a pegasus model that has already be trained for summarisation, e.g. google/pegasus-cnn_dailymail · Hugging Face 13


Training a Tokenizer on a Streamed Dataset

Question:
Hi, I’m trying to train a tokenizer on a dataset that uses streaming. I followed the instructions provided here 4, with the addition of streaming=True during the dataset loading step. However, it quickly failed as the IterableDataset class does not have a length property (unlike the normal Dataset class).
How can I work around this issue without having to download the dataset file entirely, i.e. purely relying on dataset streaming? Here’s the Colab link to reproduce the error.
Any help would be greatly appreciated. Thanks!
Note: This issue 4 seems to be related to mine.

Context:
Hi !
You can follow the instructions here 4 and use this batch iterator instead:
def batch_iterator(batch_size=1000):
    batch = []
    for example in dataset:
        batch.append(example["text"])
        if len(batch) == batch_size:
            yield batch
            batch = []
    if batch:  # yield last batch
        yield batch


Question:
I am trying to load dataset in streaming model.
The current datasets version I am using is 1.8.
But it is producing the following error.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-518060a18801> in <module>()
----> 1 dataset = load_dataset('oscar', "unshuffled_deduplicated_en", split='train', streaming=True)

3 frames
/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _create_builder_config(self, name, custom_features, **config_kwargs)
    339                 if value is not None:
    340                     if not hasattr(builder_config, key):
--> 341                         raise ValueError(f"BuilderConfig {builder_config} doesn't have a '{key}' key.")
    342                     setattr(builder_config, key, value)
    343

ValueError: BuilderConfig OscarConfig(name='unshuffled_deduplicated_en', version=1.0.0, data_dir=None, data_files=None, description='Unshuffled and deduplicated, English OSCAR dataset') doesn't have a 'streaming' key.

Quick notebook link 2.

Context: @valhalla  Can you take a look?


ValueError: not enough values to unpack (expected 2, got 1)

Question:
i am trying to create xlnet classification
  def __init__(self,n_classes):
    super(SentimentClassifier, self).__init__()
    self.xlnet = XLNetModel.from_pretrained(PRE_TRAINED_MODEL_NAME)
    self.drop = nn.Dropout(p=0.3)
    self.out = nn.Linear(self.xlnet.config.hidden_size,  n_classes)
  def forward(self, input_ids,
              attention_mask):
    _, pooled_output = self.xlnet(
      input_ids=input_ids,
      attention_mask=attention_mask)
    output = self.drop(pooled_output)
    return self.out(output)
class Classification(Dataset):

    def __init__(self, texts, labels, tokenizer, max_len):
        self.texts = texts
        self.labels = labels
        self.tokenizer = tokenizer
        self.max_len = max_len

    def __len__(self):
        return len(self.texts)

    def __getitem__(self, item):
        text = str(self.texts[item])
        label = self.labels[item]

        encoding = self.tokenizer.encode_plus(
        text,
        add_special_tokens=True,
        max_length=self.max_len,
        return_token_type_ids=False,
        pad_to_max_length=False,
        return_attention_mask=True,
        return_tensors='pt',
        )

        return {
        'review_text': text,
        'input_ids': encoding['input_ids'].flatten(),
        'attention_mask': encoding['attention_mask'].flatten(),
         'labelss': torch.tensor(label, dtype=torch.long)
        }
def train_epoch(
  model,
  data_loader,
  loss_fn,
  optimizer,
  device,
  scheduler,
  n_examples
):
  model = xlnet_model.train()
  losses = []
  correct_predictions = 0
  for d in data_loader:
    input_ids = d["input_ids"].reshape(4,512).to(device)
    print(d['input_ids'].shape)
    attention_mask = d["attention_mask"].to(device)
    labels = d["labels"].to(device)
    outputs = xlnet_model(input_ids=input_ids, attention_mask=attention_mask)

    _, preds = torch.max(outputs, dim=1)
    loss = loss_fn(outputs, labels)
    correct_predictions += torch.sum(preds == labels)
    losses.append(loss.item())
    loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
    optimizer.step()
    scheduler.step()
    optimizer.zero_grad()
  return correct_predictions.double() / n_examples, np.mean(loss)```

Context:
Hi @sru,
Can you please include the stack trace so we can help out more?


Download model without the trained weights

Question:
Hey community, i hope you’re doing fine.
I’m new to the huggingface framework, so my question is if there any way to download hugging face models(like bert…) without it’s pretrained weights? the architecture only? i’m using pytorch
Thank you so much.

Context:
you can construct one using your own defined configuration.
from transformers import BertForMaskedLM

model = BertForMaskedLM(config=config)

where in the config variable, you provide the parameters of the model - the no. of heads for attention, FCN size etc.
So you can train from scratch, but you won’t need to download its pre-trained weights and use BERT however you wish.


Question:
Hello,
Is there any code snippet of how to use T5 pretrained model in order to do summarization?

Context:
I used the following code to do my task:
from transformers import T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained('t5-small')

model = T5ForConditionalGeneration.from_pretrained('t5-small', return_dict=True)

input = "This  is a summarization example. This is a large  sentence."
input_ids = tokenizer("summarize: "+input, return_tensors="pt").input_ids  # Batch size 1

outputs = model.generate(input_ids)

decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(decoded)


Separate LM fine tuning and classification head training

Question:
I have a large text corpus, and a small subset of it that is labelled for a mutli-label text classification task. I’ve seen many (excellent!) examples of fine-tuning different models for sequence classification, but I couldn’t find one in which the training is separated to two distinct stages:

Fine-tune a specific language model on a specific corpus (unsupervised).
Train a sequence classification model on top of it on a labelled subset of the original dataset (supervised).

I assume this is a pretty common (and simple) scenario, but I couldn’t find any relevant docs. If anyone can provide any pointers, it would be much appreciated!

Context:
Hi @adamh
You can use the run_language_modeling script here 16 to finetune the pre-trained model for ex BertForMaksedLM.
Then you should be able to load the model using BertForSequenceClassification model which will take the base model and add a classification head on  top, which you can then fine-tune for classification.


UnicodeDecodeError with xprophetnet-large-wiki100-cased-xglue-qg model

Question:
Hi I’m new to the transformer model and when I run this code
from transformers import ProphetNetTokenizer, ProphetNetForConditionalGeneration, ProphetNetConfig

model = ProphetNetForConditionalGeneration.from_pretrained('microsoft/xprophetnet-large-wiki100-cased-xglue-qg')
tokenizer = ProphetNetTokenizer.from_pretrained('microsoft/xprophetnet-large-wiki100-cased-xglue-qg')
```from https://huggingface.co/microsoft/xprophetnet-large-wiki100-cased-xglue-qg. I got an "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xaf in position 51: invalid start byte" error when I run "tokenizer = ProphetNetTokenizer.from_pretrained('microsoft/xprophetnet-large-wiki100-cased-xglue-qg')"  Can any one help? Thank you!

Context:
Slight hack to fix the decoder error - I tried editing ProphetNetTokenizer itself, specifically the load_vocab method and changed the encoding to ‘latin-1’.

image998×420 40 KB

The edit is made here 2, I did it locally so I’m not sure whether this could be done for Colab notebooks.
This stackoverflow page 3 gave some suggestions for a fix, of which the latin-1 encoding seemed to work.
Caveat: I haven’t tried using the tokenizer any further than just getting the ProphetNetTokenizer.from_pretrained() step working.
It gave a warning of "Special tokens have been added in the vocabulary, make sure the associated word embedding are fine-tuned or trained.", so there may be some extra steps needed after.
The overall issue is something that can’t be encoded to utf-8 in the vocabulary used. Hope this helps


Running custom modifications in modeling_bart.py

Question:
Hi,
I am planning to modify the output attention weights from the decoder in modeling_bart.py for conditional abstractive summarization. To achieve this I changed the transformers module to mytransformers and made the changes I wanted in the modeling_bart.py script. Now, when I run the run_summarization.py, I cannot import the mytransformers with a FILENOTFOUND Error and I am prompted to install transformers. If I install/import transformers I will be running the original bart instead of my modified code. Can you help me understand how to run my custom code changes in the transformers library ?

Context: You can’t add files to the library like that, but you can have your updated model in the same folder as your example script with the name mytransformers.py which will then allow you to import from it.


Understanding data of dataset_infos.json

Question:
Hi everyone,
I was exploring dataset_infos.json , and I couldn’t figure out what some of the keys represent in the file. Could someone please point me to a reference, which I could use as column descriptions.
eg of some confusing columns: “download_size”, “dataset_size”, “size_in_bytes”, “post_processing_size” and “num_bytes”(splits).
Another set of keys I couldn’t understand/interpret what they represent, were “post_processed” and “supervised_keys”.
Is the structure documentation available, or is diving into the code from the command dataset-cli test would be the correct approach to figure this out?
Example from Cifar-10 (canonical):

image650×558 48.3 KB

Context:
hey @dk-crazydiv you can find a description of all the DatasetInfo fields in the docs: Main classes — datasets 1.8.0 documentation 2
if something is unclear / could be improved, feel free to open a pr!


Question:
Hi,
I am following this tutorial on masked language modelling using my own dataset: notebooks/language_modeling.ipynb at master · huggingface/notebooks · GitHub 2, and I am coming across this error:
Input:
lm_datasets = tokenized_datasets.map(
    group_texts,
    batched=True,
    batch_size=1000,
    num_proc=4,
)

Output:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/multiprocess/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py", line 186, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/datasets/fingerprint.py", line 397, in wrapper
    out = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py", line 1977, in _map_single
    writer.write_batch(batch)
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py", line 383, in write_batch
    pa_table = pa.Table.from_pydict(typed_sequence_examples)
  File "pyarrow/table.pxi", line 1559, in pyarrow.lib.Table.from_pydict
    arrays.append(asarray(v))
  File "pyarrow/array.pxi", line 331, in pyarrow.lib.asarray
    return array(values, type=type)
  File "pyarrow/array.pxi", line 222, in pyarrow.lib.array
    return _handle_arrow_array_protocol(obj, type, mask, size)
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
    res = obj.__arrow_array__(type=type)
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py", line 100, in __arrow_array__
    if trying_type and out[0].as_py() != self.data[0]:
  File "pyarrow/array.pxi", line 1067, in pyarrow.lib.Array.__getitem__
    return self.getitem(_normalize_index(key, self.length()))
  File "pyarrow/array.pxi", line 549, in pyarrow.lib._normalize_index
    raise IndexError("index out of bounds")
IndexError: index out of bounds
"""

The above exception was the direct cause of the following exception:

IndexError                                Traceback (most recent call last)

<ipython-input-34-e35eeb51570c> in <module>()
      3     batched=True,
      4     batch_size=1000,
----> 5     num_proc=4,
      6 )

16 frames

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib._normalize_index()
    547             raise IndexError("index out of bounds")
    548     elif index >= length:
--> 549         raise IndexError("index out of bounds")
    550     return index
    551

IndexError: index out of bounds

Context: You should deactivate multiprocessing to have a clearer error message (remove num_proc=4). It looks like an indexing error in your dataset.


Does task specific prefix matters for T5 fine-tuning?

Question: If I understand correctly pre-trained T5 models were pre-trained with an unsupervised objective without any task specific prefix like “translate”, “summarize”, etc. Is it important then to create my summarization dataset for fine-tuning in a way that every input starts with "summarize: "?

Context: I think it is important, but am not totally certain why. You could test it pretty easily I bet.


Why do I get ‘Ġ’ when adding emojis to the tokenizer?

Question:
Hello,
I have added custom tokens to my tokenizer, which are emojis. This is the code I have used, which adds the new tokens:
model = AutoModelForMaskedLM.from_pretrained(model_checkpoint)

num_added_toks = tokenizer.add_tokens(['👏'])
print('We have added', num_added_toks, 'tokens')
model.resize_token_embeddings(len(tokenizer))  # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e. the length of the tokenizer

Output:
We have added 1 tokens
Embedding(50270, 768)

Though, when I try to tokenize a phrase using this code:
print(tokenizer.tokenize('Congrats 👏'))

I get this output with that strange 'Ġ' symbol:
['Cong', 'rats', 'Ġ', '👏']

Context:
github.com/pytorch/fairseq

      In the vocab of bart.bpe.bpe.decoder,  what does Ġ mean for those words prefixed with 'Ġ'? 15

        opened

        Feb 17, 2020

          closed

        Feb 21, 2020

          v381654729

        question

    ## ❓ Questions and Help

#### What is your question?

In the vocab of bart….bpe.bpe.decoder,  what does Ġ mean for those words prefixed with 'Ġ'?

#### Code

    keywords = { "checkpoint_file" : "model.pt"}
    bart = BARTModel.from_pretrained(config_translation_zh_en.bart_english_directory,   **keywords )
    # bart.bpe.bpe.decoder is a dict, and it contains many 'strange' words like 'Ġthe' 'Ġand' 'Ġof' and also many normal words like 'playing' 'bound' etc.
![image](https://user-images.githubusercontent.com/43415394/74647258-52024680-51b6-11ea-97dd-4631e0404d17.png)

bart.bpe.bpe.decoder is a dict, and it contains many 'strange' words like 'Ġthe' 'Ġand' 'Ġof' and also many normal words like 'playing' 'bound' etc. At first glance, words prefixed with Ġ seem to be complete words and words not prefixed with Ġ are incomplete words (like BertTokenizer's ##-prefixed words ) like 'ing' 'th' and ''re'. However, a high proportion of words not prefixed with Ġ seem not incomplete words, like frequency, relation, etc.

#### Question

What does Ġ mean for those words prefixed with 'Ġ'?  And how does the tokenizer of BART tokenize with the set of 50265-word vocabulary (bart.task.source_dictionary, 50257 (integer) words of which map to bart.bpe.bpe.decoder)? I haven't found documentations on the Internet or in the codes.


XLM-R classifier predictions produce errors

Question:
Hi,
I am using tf-xlm-r-base model for a sentiment classification (multi-class) task with 4 classes. I used both trainer() api and keras native method. Initially, I got some acceptable result but later it predicts only one class for the same data set. I am following this guide. Below are my outputs and code. My inputs and labels are similar to the example mentioned in the guide(mentioned above) having class labels 0,1,2,3. I am trying this on a dataset of 1000 training data points. I also tried with 7K and 14K training data sets which did not solve the error. Only the single predicted class changed.
X_train, X_test, y_train, y_test = train_test_split(comment_texts, comment_labels, test_size=0.1, random_state=0)

model_checkpoint = "jplu/tf-xlm-roberta-base"

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)

train_encodings = tokenizer(X_train, truncation=True, padding=True)
val_encodings = tokenizer(val_texts, truncation=True, padding=True)
test_encodings = tokenizer(X_test, truncation=True, padding=True)

train_dataset = tf.data.Dataset.from_tensor_slices(( # convert to dataset objects
    dict(train_encodings),
    y_train
))
val_dataset = tf.data.Dataset.from_tensor_slices((
    dict(val_encodings),
    val_labels
))
test_dataset = tf.data.Dataset.from_tensor_slices((
    dict(test_encodings),
    y_test
))

from transformers import TFAutoModelForSequenceClassification, TFTrainingArguments, TFTrainer
num_labels=4#
model = TFAutoModelForSequenceClassification.from_pretrained(model_checkpoint, num_labels=num_labels)

Model: "tfxlm_roberta_for_sequence_classification_3"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
roberta (TFRobertaMainLayer) multiple                  277453056
_________________________________________________________________
classifier (TFRobertaClassif multiple                  593668
=================================================================
Total params: 278,046,724
Trainable params: 278,046,724
Non-trainable params: 0

from sklearn.metrics import accuracy_score,precision_score,recall_score

def compute_metrics(p):
    pred, labels = p
    pred = np.argmax(pred, axis=1)
    accuracy = accuracy_score(y_true=labels, y_pred=pred)
    recall = recall_score(y_true=labels, y_pred=pred, average='weighted')
    precision = precision_score(y_true=labels, y_pred=pred, average='weighted')
    #f1 = f1_score(y_true=labels, y_pred=pred)
    return {"accuracy": accuracy, "precision": precision, "recall": recall}

raining_args = TFTrainingArguments(
    output_dir='/content/drive/MyDrive/test_transformer/results',          # output directory
    num_train_epochs=3,              # total number of training epochs
    evaluation_strategy = "epoch",
    per_device_train_batch_size=8,  # batch size per device during training
    per_device_eval_batch_size=8,   # batch size for evaluation
    warmup_steps=100,                # number of warmup steps for learning rate scheduler
    weight_decay=0.001,               # strength of weight decay
    logging_dir='/content/drive/MyDrive/test_transformer/logs',            # directory for storing logs
    logging_steps=10,
)

with training_args.strategy.scope():
    model = TFAutoModelForSequenceClassification.from_pretrained(model_checkpoint, num_labels=num_labels, output_attentions=True)

trainer = TFTrainer(
    model=model,                         # the instantiated Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    train_dataset=train_dataset,         # training dataset
    eval_dataset=val_dataset,             # evaluation dataset # should not be the test dataset although I am using it
    compute_metrics=compute_metrics
)

trainer.train()

For Keras native training,
optimizer = tf.keras.optimizers.Adam(learning_rate=5e-5)
model.compile(optimizer=optimizer, loss=model.compute_loss, metrics=['accuracy'])# can also use any keras loss fn
his=model.fit(train_dataset.shuffle(1000).batch(16), epochs=10, batch_size=16)

out=model.predict(test_dataset.batch(16))
y_pred=np.array((np.argmax((tf.nn.softmax(out.logits,axis=-1)),axis=-1)))

which gives results for a test dataset like,
              precision    recall  f1-score   support

           0     0.0000    0.0000    0.0000        31
           1     0.4200    1.0000    0.5915        42
           2     0.0000    0.0000    0.0000        26
           3     0.0000    0.0000    0.0000         1

    accuracy                         0.4200       100
   macro avg     0.1050    0.2500    0.1479       100
weighted avg     0.1764    0.4200    0.2485       100

The logits outputs tally with these, giving high value for a one particular class,
PredictionOutput(predictions=array([[ 1.0289115 ,  1.1212691 ,  0.35617405, -2.9331322 ],
       [ 1.0295717 ,  1.1219745 ,  0.35612756, -2.934678  ],
       [ 1.0289289 ,  1.1211748 ,  0.3562293 , -2.9330313 ],
       [ 1.0290331 ,  1.121373  ,  0.35619155, -2.9334006 ],
       [ 1.0291184 ,  1.121406  ,  0.3562282 , -2.933531  ],
       [ 1.0291708 ,  1.121502  ,  0.35619184, -2.933691  ],
       [ 1.0289183 ,  1.121173  ,  0.35626405, -2.933055  ],
       [ 1.0282533 ,  1.1204101 ,  0.35632664, -2.9314802 ],
       [ 1.0289795 ,  1.1212872 ,  0.35624185, -2.933268  ],
       [ 1.0289418 ,  1.1212839 ,  0.35617486, -2.933162  ],
       [ 1.0290436 ,  1.1213527 ,  0.35621163, -2.933384  ],
       [ 1.0290844 ,  1.1214646 ,  0.35614225, -2.9335442 ],
       [ 1.0289063 ,  1.1212507 ,  0.35618058, -2.9330966 ],
       [ 1.0291401 ,  1.1214608 ,  0.3562286 , -2.9336445 ],
       [ 1.0290921 ,  1.1214745 ,  0.3561555 , -2.9335837 ],
       [ 1.0290128 ,  1.1212608 ,  0.35625243, -2.9332502 ],
       [ 1.0291014 ,  1.1213605 ,  0.3562572 , -2.9334743 ],
       [ 1.0291253 ,  1.1214253 ,  0.35621867, -2.9335544 ],
       [ 1.0290672 ,  1.1213386 ,  0.35625044, -2.933417  ],
       [ 1.0289694 ,  1.1212484 ,  0.35623503, -2.9331877 ],
       [ 1.0291487 ,  1.1214706 ,  0.35619804, -2.93363   ],
       [ 1.0289078 ,  1.1212181 ,  0.3562105 , -2.933069  ],
       [ 1.0291108 ,  1.1214286 ,  0.35619968, -2.9335427 ],
       [ 1.0291494 ,  1.121471  ,  0.35619843, -2.9336333 ],
       [ 1.0291737 ,  1.1215005 ,  0.35622108, -2.9337227 ],
       [ 1.0291069 ,  1.1215074 ,  0.3561285 , -2.9336267 ],
       [ 1.0289413 ,  1.1212887 ,  0.3561762 , -2.9331977 ],
       [ 1.0293871 ,  1.1218258 ,  0.35609013, -2.9342983 ],
       [ 1.0290092 ,  1.1213552 ,  0.3561896 , -2.9333515 ],
       [ 1.0283587 ,  1.1205777 ,  0.3562444 , -2.93175   ],
       [ 1.0289493 ,  1.1212447 ,  0.3562152 , -2.9331517 ],
       [ 1.0290315 ,  1.1213847 ,  0.3561749 , -2.93341   ],
       [ 1.0290761 ,  1.1214103 ,  0.35618582, -2.9334972 ],
       [ 1.0289084 ,  1.1212412 ,  0.35619718, -2.9331186 ],
       [ 1.0290228 ,  1.1213146 ,  0.35622573, -2.933303  ],
       [ 1.028934  ,  1.1211748 ,  0.3562406 , -2.933056  ],
       [ 1.0292109 ,  1.1215464 ,  0.356189  , -2.9337885 ],
       [ 1.0292367 ,  1.1215956 ,  0.3561677 , -2.9338856 ],
       [ 1.0290639 ,  1.1214087 ,  0.35618123, -2.9334571 ],
       [ 1.0290803 ,  1.1214298 ,  0.35617745, -2.933514  ],
       [ 1.0291355 ,  1.1214267 ,  0.35623217, -2.9335864 ],
       [ 1.0289494 ,  1.1212884 ,  0.35619822, -2.9332087 ],
       [ 1.0293379 ,  1.1216819 ,  0.35617125, -2.9340885 ],
       [ 1.0293596 ,  1.1217119 ,  0.35619378, -2.934169  ],
       [ 1.0291876 ,  1.1215029 ,  0.35619885, -2.9337032 ],
       [ 1.0288249 ,  1.1211317 ,  0.35620478, -2.9328792 ],
       [ 1.0291348 ,  1.1215305 ,  0.35612702, -2.93368   ],
       [ 1.0292716 ,  1.1215887 ,  0.3561728 , -2.9338777 ],
       [ 1.029365  ,  1.1217095 ,  0.35616824, -2.9341416 ],
       [ 1.0290942 ,  1.1214304 ,  0.35618833, -2.9335175 ],
       [ 1.0289066 ,  1.1211537 ,  0.3562669 , -2.933033  ],
       [ 1.0291075 ,  1.121446  ,  0.35618696, -2.9335594 ],
       [ 1.0294148 ,  1.121831  ,  0.3561264 , -2.9343624 ],
       [ 1.0292593 ,  1.1216727 ,  0.35611537, -2.9339767 ],
       [ 1.0290027 ,  1.1213193 ,  0.35620502, -2.9333057 ],
       [ 1.0292455 ,  1.1215812 ,  0.3561885 , -2.933863  ],
       [ 1.0291928 ,  1.1215296 ,  0.35619065, -2.9337544 ],
       [ 1.0289562 ,  1.12131   ,  0.3561774 , -2.9332416 ],
       [ 1.0292183 ,  1.1215689 ,  0.35616964, -2.9338312 ],
       [ 1.0292469 ,  1.121585  ,  0.35619286, -2.9338806 ],
       [ 1.0292145 ,  1.1215852 ,  0.35615656, -2.9338334 ],
       [ 1.0290105 ,  1.1213479 ,  0.3561993 , -2.933348  ],
       [ 1.0290651 ,  1.1213914 ,  0.35617766, -2.9334235 ],
       [ 1.0290747 ,  1.1214159 ,  0.3561734 , -2.9334788 ],
       [ 1.0290564 ,  1.1214483 ,  0.35613787, -2.933495  ],
       [ 1.0290995 ,  1.1214293 ,  0.3561933 , -2.9335384 ],
       [ 1.0292302 ,  1.1215992 ,  0.356152  , -2.933872  ],
       [ 1.0294645 ,  1.1218221 ,  0.35615602, -2.9344025 ],
       [ 1.0291889 ,  1.1215235 ,  0.35620224, -2.9337523 ],
       [ 1.0292081 ,  1.1215614 ,  0.35616958, -2.9337964 ],
       [ 1.0291911 ,  1.1215253 ,  0.35619062, -2.9337654 ],
       [ 1.028842  ,  1.1211593 ,  0.35621268, -2.9329333 ],
       [ 1.0290655 ,  1.1213945 ,  0.3562099 , -2.933466  ],
       [ 1.027889  ,  1.1199136 ,  0.35638046, -2.9304833 ],
       [ 1.0290161 ,  1.1213044 ,  0.35623237, -2.9333255 ],
       [ 1.0292108 ,  1.1215161 ,  0.3562079 , -2.9337723 ],
       [ 1.0290184 ,  1.1213537 ,  0.35618034, -2.9333456 ],
       [ 1.0289462 ,  1.1212633 ,  0.35620436, -2.9331503 ],
       [ 1.0292357 ,  1.1216115 ,  0.3561552 , -2.933901  ],
       [ 1.0290319 ,  1.1213702 ,  0.35618973, -2.933375  ],
       [ 1.0292478 ,  1.1216022 ,  0.35619214, -2.93391   ],
       [ 1.0288923 ,  1.1211102 ,  0.35628372, -2.932956  ],
       [ 1.0272568 ,  1.1193739 ,  0.3562229 , -2.929129  ],
       [ 1.0290308 ,  1.1214021 ,  0.35614964, -2.9334133 ],
       [ 1.0291088 ,  1.1214335 ,  0.356207  , -2.933562  ],
       [ 1.028747  ,  1.1209638 ,  0.3562615 , -2.932617  ],
       [ 1.0292501 ,  1.1215758 ,  0.35620457, -2.933898  ],
       [ 1.0288965 ,  1.1211853 ,  0.3562436 , -2.933039  ],
       [ 1.0290757 ,  1.121396  ,  0.3562069 , -2.9334598 ],
       [ 1.0290804 ,  1.1213692 ,  0.3562515 , -2.9334688 ],
       [ 1.0292016 ,  1.1215339 ,  0.35618612, -2.9337628 ],
       [ 1.0291742 ,  1.1215011 ,  0.35619456, -2.9337041 ],
       [ 1.0290596 ,  1.1214275 ,  0.35615835, -2.9334834 ],
       [ 1.0293068 ,  1.1216859 ,  0.35614944, -2.9340594 ],
       [ 1.0291089 ,  1.1214725 ,  0.35615313, -2.9335957 ],
       [ 1.0290648 ,  1.1213634 ,  0.35623825, -2.933438  ],
       [ 1.0292054 ,  1.1215473 ,  0.35618582, -2.9337878 ],
       [ 1.0281092 ,  1.1202368 ,  0.356357  , -2.931106  ],
       [ 1.0292268 ,  1.1216258 ,  0.3561134 , -2.9338837 ],
       [ 1.0290968 ,  1.1214894 ,  0.35614622, -2.9336073 ]],
      dtype=float32), label_ids=array([0, 1, 2, 0, 0, 2, 0, 1, 1, 1, 1, 1, 1, 2, 1, 0, 1, 2, 2, 0, 0, 2,
       1, 1, 1, 3, 0, 2, 2, 2, 0, 1, 2, 2, 2, 0, 1, 0, 0, 1, 0, 1, 1, 2,
       2, 1, 1, 1, 0, 1, 0, 0, 2, 1, 0, 0, 1, 0, 0, 0, 0, 1, 2, 2, 1, 0,
       2, 1, 2, 0, 1, 1, 0, 2, 1, 2, 0, 1, 1, 1, 2, 0, 1, 0, 1, 2, 2, 1,
       1, 0, 0, 1, 1, 1, 1, 2, 0, 2, 1, 1]), metrics={'eval_loss': 1.13915282029372, 'eval_accuracy': 0.42, 'eval_precision': 0.1764, 'eval_recall': 0.42})

Can someone please suggest a tip or tell me what could have gone wrong? (I also tried batch sizes 8/16, running on both CPU and GPU with same datasets and parameters, changing learning rates/epochs and down sampling training dataset to balance classes).

Context: also found that for binary classification, (two from above four classes), it behaves similarly. And tried changing labels to float type and different loss function too.


Question:
Hey,
curious question to illuminate my understanding.
Fine Tuning a BERT model for you downstream task can be important. So I like to tune the BERT weights. Thus, I can extract them from the BertForSequenceClassification which I can fine tune.
if you fine tune eg. BertForSequenceClassification you tune the weights of the BERT model and the classifier layer too.
But for making right fine tune, you would first need freeze the BERT weights, and tune the classifier. Afterwards you fine tune the BERT weights too, right?
Now, there are myriads of ways to finetune the BERT weights?
If I just use the main BERT model together with arbitrary neural network architecture afterwards I could fine tune the BERT weights in this way too, right?

Context:
Any suggestions?
Also what came in my mind. That TFBertForSequence is using the pooled_output. So the model is finetuned viad this pooled_output.
But instead I could use the cls embedding or the globalaveragepooling of the hiddensequence for finetuning (pass to the classifier layer), right?


Question:
Hi everyone!
I have two datasets, loaded as CSV files, which have the same features/columns. I would like to know if there is a way to merge both datasets into a larger one (like I would do with pd.concat((df_1, df_2))using pandas.
In case that such method does not exist, would it be interesting to implement such functionality?
Thanks in advance

Context: I would rather combine the csv’s


Question:
Hello
I am running distilbert-base-multilingual-cased’ on Pytorch. My model has 4 classes in the target.
In the code in models/distilbert/modeling_distilbert.py. I am reaching this state
elif self.config.problem_type == “multi_label_classification”:
**                loss_fct = BCEWithLogitsLoss()**
**                loss = loss_fct(logits, labels)**
I have two questions:
1 BCEWithLogitsloss  must receive the labels as one hot rather integers . must I take care of doing one_hot?
2 If I wish to add regulation functions to the loss , what is the best practice for doing so?
Thanks

Context:
Note that multi_label_classification is only for problems where you can have multiple labels for one example, so you should use the default if your samples can only have one label.
If you are in a true multiple label problem, then it’s very likely your labels are already in a one-hot format.
For your second question, you should just output the logits of your model and then compute the loss manually with your penalty. If you’re using the Trainer API, you can subclass and write a compute_loss function with that, see here 3 for an example.


How to test masked language model after training it?

Question:
Hi,
I have followed and trained my masked language model using this tutorial: notebooks/language_modeling.ipynb at master · huggingface/notebooks · GitHub 10
Now, once the model as been saved using this code below:
trainer.save_model("my_model")

But, the notebook does not seem to include any code to allow me to test my model, so I am unsure how to do this.
I have saved my model, but I now want to mask a sentence using my model by doing something like this:
The [MASK] of France is Paris

Thanks!

Context:
You can load it in a pipeline by using the folder where you saved it:
mask_filler = pipeline("fill-mask", model="my_model")


Is it normal of more memory use of DistributedDataParallel than single

Question:
Hello , I am new here.
I try to Fine-turn MBart model ( mbart-large-cc25 ), My device : one pc(ubuntu), gpu(10GB) memory * 2
When i fine-turn on single gpu, first load model to gpu , only cost 3(GB) memory and start train it, increase to 8(GB), so i can fine-turn with small batch.
I use DistributedDataParallel to have more batch in fine-turn
But, i load model to 2 gpu , than cost 7GB both, didn’t start training.
Is it normal ?
Here is my init memory code
class SummarizationModule() :
    def __init__(self,local_rank) -> None:
        self.mBartModel = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-cc25")
        self.mBartTokenizer = MBartTokenizer.from_pretrained("facebook/mbart-large-cc25", src_lang="en_XX", tgt_lang="zh_CN")
        self.mLocal_rank = local_rank
        torch.cuda.set_device(self.mLocal_rank)
        print('Rank : [' , args.local_rank , '] is ready.')
        os.environ['CUDA_VISIBLE_DEVICES'] = '0,1'
        os.environ['MASTER_ADDR'] = 'localhost'
        os.environ['MASTER_PORT'] = '8787'
        torch.distributed.init_process_group(backend="nccl", init_method="env://", world_size=1 ,rank=self.mLocal_rank )
        self.mBartModel = self.mBartModel.cuda()
        self.mBartModel = DistributedDataParallel(self.mBartModel, find_unused_parameters=True)
        os.system('nvidia-smi')

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument('--local_rank', default=-1, type=int, help='node rank for distributed training')
    args = parser.parse_args()
    mModel = SummarizationModule(args.local_rank)

Use :
CUDA_VISIBLE_DEVICES=0,1 python3 -m torch.distributed.launch --nproc_per_node=2 main_mbart_multi_gpu.py

Context: There is a slight overhead when using DistributedDataParallel so it’s normal to see a bit more GPU usage yes.


Question: Hello, is there a way I can renew my API Key? I’ve seen a few forum posts about it but I’d like to reset it.

Context: cc @julien-c or @pierric


Question:
Hi there,
In order to debug something I need to make data non-shuffle. Can you please tell me how to turn off the shuffle?
I am using from transformers import Trainer for training and from datasets import load_dataset for data loading with default arguments.

Context: There is no option to do this natively in the Trainer, you can either make a source install and change the line that creates the training dataloader, or subclass Trainer and override the get_train_dataloader method.


Question:
Hi @joeddav and @bhadresh-savani I am using your text-classification models, but I am encountering a problem, nothing I try allows me to retrieve all the values: always just the first value is returned, would you be able to advise me, thanks so much.

image1040×157 7.12 KB

      huggingface.co

joeddav/distilbert-base-uncased-go-emotions-student · Hugging Face 2

      huggingface.co

bhadresh-savani/distilbert-base-uncased-emotion · Hugging Face 1

Context:
Hi @snowdere,
I am really glad you use this model, you can use it like either of the below ways
from transformers import pipeline
classifier = pipeline("text-classification",model='bhadresh-savani/distilbert-base-uncased-emotion', return_all_scores=True)
prediction = classifier("I love using transformers. The best part is wide range of support and its easy to use", )
print(prediction)

or
from transformers import pipeline
classifier = pipeline("sentiment-analysis",model='bhadresh-savani/distilbert-base-uncased-emotion', return_all_scores=True)
prediction = classifier("I love using transformers. The best part is wide range of support and its easy to use", )
print(prediction)

return_all_scores=True should be in the pipeline() argument.
Output:
[[
{'label': 'sadness', 'score': 0.0006792712374590337},
{'label': 'joy', 'score': 0.9959300756454468},
{'label': 'love', 'score': 0.0009452480007894337},
{'label': 'anger', 'score': 0.0018055217806249857},
{'label': 'fear', 'score': 0.00041110432357527316},
{'label': 'surprise', 'score': 0.0002288572577526793}
]]

Thanks


Showing individual token and corresponding score during beam search

Question:
Hello,
I am using beam search with a pre-trained T5 model for summarization. I would like to visualize the beam search process by showing the tokens with the highest scores, and eventually the chosen beam like this diagram:
image894×669 56.8 KB
(Taken from How to generate text: using different decoding methods for language generation with Transformers 5)
I am unsure how I can show the tokens and their corresponding scores.
I followed the discussion [Announcement] GenerationOutputs: Scores, Attentions and Hidden States now available as outputs to generate 6 and https://github.com/huggingface/transformers/pull/9150.
Following the docs, upon calling generate, I have set return_dict_in_generate=True, output_scores=True
generated_outputs = model_t5summary.generate(
  input_ids=input_ids.to(device),
  attention_mask=features['attention_mask'].to(device),
  max_length=input_ids.shape[-1] + 2,
  return_dict_in_generate=True,
  output_scores=True,
  output_hidden_states=True,
  output_attentions=True,
  no_repeat_ngram_size=2,
  early_stopping=True,
  num_return_sequences=3,
  num_beams=5,
)

Now I have an instance of BeamSearchEncoderDecoderOutput.
If I understand the docs (Utilities for Generation — transformers 4.2.0 documentation 5) correctly, scores will provide me with what I want but I am unsure on how to use the scores.
Any help/pointers from the community would be greatly appreciated, thank you

Context: Tagging @patrickvonplaten reposted my question from Github, thanks for directing me to the forum


Token Classification (ValueError: NumPy boolean array indexing assignment)

Question:
I was reading and working through “Token Classification with W-NUT Emerging Entities” tutorial on Fine-tuning with custom datasets — transformers 4.5.0.dev0 documentation 6 using a different data.
To replicate the data structure of the tutorial, I used the code below to insert a blank space between sentences/tags
# insert blank row in python dataframe when value in column changes
mask = DF['sentence_id'].ne(DF['sentence_id'].shift(-1))
DF1 = pd.DataFrame('',index=mask.index[mask] + .5, columns=DF.columns)

DF2 = pd.concat([DF, DF1]).sort_index().reset_index(drop=True).iloc[:-1]
DF2.head(18)

I then wrote the data to a file
DF2.to_csv(r'/content/drive/MyDrive/Colab Notebooks/model/dataset.txt',

          header=None, index=None, sep='\t', mode='a', encoding="utf-8")

I read the data back using the code provided in the tutorial
def read_wnut(file_path):
    file_path = Path(file_path, encoding='utf8')

    raw_text = file_path.read_text().strip()
    raw_docs = re.split(r'\n\t?\n', raw_text)
    token_docs = []
    tag_docs = []
    for doc in raw_docs:
        tokens = []
        tags = []
        for line in doc.split('\n'):
            token, tag = line.split('\t')
            tokens.append(token)
            tags.append(tag)
        token_docs.append(tokens)
        tag_docs.append(tags)

    return token_docs, tag_docs

texts_df, tags_df = read_wnut(colab_file_path)

Next, I split up the data into training, validation, and test sets.
I then created encoding for the tags and the tokens as seen below
tag2id = {tag: id for id, tag in enumerate(unique_tags)}
id2tag = {id: tag for tag, id in tag2id.items()}

# import the transformers module
from transformers import BertTokenizerFast

# import the small bert tokenizer
model_name = "google/bert_uncased_L-4_H-512_A-8"
tokenizer = BertTokenizerFast.from_pretrained(model_name)

train_encodings = tokenizer(train_texts, is_split_into_words=True, return_offsets_mapping=True, padding=True, truncation=True)
val_encodings = tokenizer(val_texts, is_split_into_words=True, return_offsets_mapping=True, padding=True, truncation=True)
test_encodings = tokenizer(test_texts, is_split_into_words=True, return_offsets_mapping=True, padding=True, truncation=True)

It’s when I run the offset function that I get an error message
import numpy as np

def encode_tags(tags, encodings):
    labels = [[tag2id[tag] for tag in doc] for doc in tags]
    encoded_labels = []
    for doc_labels, doc_offset in zip(labels, encodings.offset_mapping):
        # create an empty array of -100
        doc_enc_labels = np.ones(len(doc_offset),dtype=int) * -100
        arr_offset = np.array(doc_offset)

        # set labels whose first offset position is 0 and the second is not 0
        doc_enc_labels[(arr_offset[:,0] == 0) & (arr_offset[:,1] != 0)] = doc_labels
        encoded_labels.append(doc_enc_labels.tolist())

    return encoded_labels

# return the encoded labels
train_labels = encode_tags(train_tags, train_encodings)
val_labels = encode_tags(val_tags, val_encodings)
test_labels = encode_tags(test_tags, test_encodings)

The error message is below:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-18-7290e6aeda9c> in <module>()
     17
     18 # return the encoded labels
---> 19 train_labels = encode_tags(train_tags, train_encodings)
     20 val_labels = encode_tags(val_tags, val_encodings)
     21 test_labels = encode_tags(test_tags, test_encodings)

<ipython-input-18-7290e6aeda9c> in encode_tags(tags, encodings)
     11
     12         # set labels whose first offset position is 0 and the second is not 0
---> 13         doc_enc_labels[(arr_offset[:,0] == 0) & (arr_offset[:,1] != 0)] = doc_labels
     14         encoded_labels.append(doc_enc_labels.tolist())
     15

ValueError: NumPy boolean array indexing assignment cannot assign 236 input values to the 120 output values where the mask is true

I am not sure where my error is, as I have tried to replicate what is in the tutorial.

Context:
I have the same problem, maybe because of the training data problem, you can try to UTF-8 encoding to UTF-8-SIG
raw_text = file_path.read_text(encoding='UTF-8-sig').strip()

Because UTF-8 may cause an extra \ufeff character in the encoded data


Trouble with the built in inference API example

Question:
Hi I’m just starting out with the inference API examples. While I can get other examples to work using the same formatting , I get this error
{“error”:“d argument needs to be of type (SquadExample, dict)”}
When using the example for deepset/roberta-base-squad2 · Hugging Face
import json
import requests

API_URL = "https://api-inference.huggingface.co/models/deepset/roberta-base-squad2"
headers = {"Authorization": "Bearer api_xxx"}

def query(payload):
	data = json.dumps(payload)
	response = requests.request("POST", API_URL, headers=headers, data=data)
	return json.loads(response.content.decode("utf-8"))

data = query(
    {
        "inputs": {
            "question": "What's my name?",
            "context": "My name is Clara and I live in Berkeley.",
        }
    }
)

I’d love know how else to format the query, or what else I should double check.
Thanks for reading.

Context:
Hi @LowellR
I just tried your exact pasted code and it worked fine.
{'answer': 'Clara', 'end': 16, 'score': 0.9326569437980652, 'start': 11}
Could you confirm if you are running this same code?
Thanks!


Question:
Dear All,
I have some experience with BERT and text analysis, but I’m a beginner at . And so, please bear my question which might be a little silly.
Here is the situation, I would like to solve a text comprehension task with a proper model.
However, some model seems not supported in TensorFlow according to the following screenshot.

image1782×1398 215 KB

But it also mentioned that “any model saved as before can be loaded back either in PyTorch or TensorFlow” on the “Quick tour” page.
I am a little confused by those two descriptions above. For say, I would like to choose Bert Generation for my task and I have no experience with PyTorch. Would the model work fine or just have some limited function while I fine-tune it in TensorFlow?
I deeply appreciate your kind assistance.
Sincerely,

Context: The documentation should be interpreted as “any model saved as before can be loaded back either in PyTorch or TensorFlow as long as there is an implementation for both frameworks”, so no, you wouldn’t be able to use Bert Generation in TensorFlow at all.


Cannot download translation models in Colab

Question:
I am trying to translate English text to German. And so I run this-
translator = pipeline("translation", model="Helsinki-NLP/opus-mt-en-de")

But I get thrown an error-

ValueError: This tokenizer cannot be instantiated. Please make sure you have sentencepiece installed in order to use this tokenizer.

Full error message
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-65-accbe9f8763e> in <module>()
----> 1 translator = pipeline("translation", model="Helsinki-NLP/opus-mt-en-de")

1 frames
/usr/local/lib/python3.7/dist-packages/transformers/pipelines/__init__.py in pipeline(task, model, config, tokenizer, feature_extractor, framework, revision, use_fast, use_auth_token, model_kwargs, **kwargs)
    441
    442             tokenizer = AutoTokenizer.from_pretrained(
--> 443                 tokenizer_identifier, revision=revision, use_fast=use_fast, _from_pipeline=task, **tokenizer_kwargs
    444             )
    445

/usr/local/lib/python3.7/dist-packages/transformers/models/auto/tokenization_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    449                 else:
    450                     raise ValueError(
--> 451                         "This tokenizer cannot be instantiated. Please make sure you have `sentencepiece` installed "
    452                         "in order to use this tokenizer."
    453                     )

ValueError: This tokenizer cannot be instantiated. Please make sure you have `sentencepiece` installed in order to use this tokenizer.

As it is suggested that I should have sentencepiece installed, I installed it via pip, but that does not help. I have tried importing it so that its namespace is available, but it still does not work.
Note: Besides the  Helsinki-NLP/opus-mt-en-de model, I have also tried using the Helsinki-NLP/opus-mt-fr-en model as shown in the course video 1, but it does not work either.
What am I missing?

Context:
okay, I tried to run this locally (not in Colab):
from transformers import pipeline

translator = pipeline("translation", model="Helsinki-NLP/opus-mt-en-de")

translation = translator("hello, my name is Bob")

print(translation)

and it printed out:
[{'translation_text': 'Hallo, mein Name ist Bob.'}]

I don’t know where you try run that code, but seems to work ok for me. Have you installed latest package of transformers?
pip install transformers -U
Where do you run that code?
Can you share the full code to see if something else is going on there?


Model Parallelism, how to parallelize transformer?

Question:
Hi there,
I am pretty new, I hope to do it right:)
I have two gpus nvidia, which work fine. I can train model on each of them, I can use data parallelism. I wonder if I can parallelize the model itself. Surfing the internet I found it is possible but no one tells how. Some frameworks do it as torchgpipe, deepspeed PipelineModule, Fairscale but they wants sequential models but transformers are hard to turn sequential.
Can you point me in the right direction?
Specs:
I want to parallelize BERT model on two gpus titan xp.
Thank you, every hints or helps will be appreciated
valgi0

Context:
hey @valgi0 my suggestion would be to try out the new accelerate library: GitHub - huggingface/accelerate: 🚀 A simple way to train and use PyTorch models with multi-GPU, TPU, mixed-precision 151
in particular, there is an nlp example that shows you how to configure accelerate for the multi-GPU case here: accelerate/examples at main · huggingface/accelerate · GitHub 74


How to do sentiment analysis on my own dataset?

Question:
Hi,
I have seen this 3 example, which does sentiment analysis on product reviews.
Therefore, is there a way to do this with my own dataset? I want to classify the sentiment of text as “happy”, “sad” or “neutral” using my own dataset.
Thanks.

Context: You have to define your own torch.utils.data.Dataset and torch.utils.data.DataLoader to load your own labeled text. Then choose an LM like BERT from the huggingface library to finetune.


Get word embeddings from transformer model

Question:
Hi
I would like to plot semantic space for specific words. Usually, we use word embeddings for this. But model I use (xlm-roberta) deala with language on the level of part of words (BPE tokens). It means that if I give to the model a word ‘hello’ I will get 2-3 vectors for each part of this word, right?
model(**tokenizer('hello', return_tensors="tf"),output_hidden_states=True)
# hidden states are embedding I suppose

So, what is the best approach to get one single vector to plot?
To be clear what do I mean by semantic space:

850×719 102 KB

Context:
I’m not sure what’s the best approach since I’m not an expert in this  , but you can always do mean pooling to the output. Here is a working example

from transformers import AutoTokenizer, AutoModelForMaskedLM

def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    sum_embeddings = torch.sum(token_embeddings * input_mask_expanded, 1)
    sum_mask = torch.clamp(input_mask_expanded.sum(1), min=1e-9)
    return sum_embeddings / sum_mask

tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-base")
model = AutoModelForMaskedLM.from_pretrained("xlm-roberta-base")
encoded_input = tokenizer("hello", return_tensors='pt')
model_output = model(**encoded_input)
mean_pooling(model_output, encoded_input['attention_mask'])

This is inspired in sentence transformers 62


Adding New Tokens - IndexError: index out of range in self

Question:
Hi,
I have added custom tokens using this code:
# Let's see how to increase the vocabulary of Bert model and tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = AutoModelForMaskedLM.from_pretrained('bert-base-uncased')

num_added_toks = tokenizer.add_tokens(['😎', '🤬'])
print('We have added', num_added_toks, 'tokens')
model.resize_token_embeddings(len(tokenizer))  # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e. the length of the tokenizer.

However, when I execute this code:
trainer.train()

I get this error:
---------------------------------------------------------------------------

IndexError                                Traceback (most recent call last)

<ipython-input-42-3435b262f1ae> in <module>()
----> 1 trainer.train()

11 frames

/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1914         # remove once script supports set_grad_enabled
   1915         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1916     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1917
   1918

IndexError: index out of range in self

Context: hey @anon58275033 without seeing the full code it’s a bit hard to debug, but my first question would be whether you tokenized the corpus after adding the new tokens? if yes, did you observe whether the tokenization is working as expected?


Reducing output size when performing hyperparameter search

Question:
Hello everyone!
I’ve been trying to perform a simple hyperparameter research on ‘distilroberta-base’. When using Kaggle notebooks there is a 20 GB limit on outputs. Even when i am doing only 5 trials the output directory fills up. I am used to using GridSearchCV for HP-search which only keeps track of the best parameters and retrains at the end without saving individual models during the search.
I have tried using  ‘overwrite_output_dir=True’ in TrainingArguments but this doesn’t seem to reduce any output.
I apologize if this is documented somewhere in the documentation, but I have not been able to find it.
My code:
model_training_arguments = TrainingArguments(
    "./model_output",
    evaluation_strategy = "epoch",
    fp16=True,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    num_train_epochs=4,
    seed=2,
    load_best_model_at_end=True,
    overwrite_output_dir=True
)

model_trainer = Trainer(
    model_init=model_init,
    args=model_training_arguments,
    train_dataset=train_dataset,
    eval_dataset=valid_dataset,
    tokenizer=tokenizer,
    compute_metrics=metrics,
)

model_trainer.hyperparameter_search(
    direction = "minimize",
    backend = "optuna",
    n_trials=5

)

Thanks in advance!

Context:
Set
 load_best_model_at_end=False,
and add
    save_strategy = 'no',           # The checkpoint save strategy to adopt during training.
On the training arguments


Extracting token embeddings from pretrained language models

Question:
I am interested in extracting feature embedding from famous and recent language models such as GPT-2, XLNeT or Transformer-XL.
Is there any sample code to learn how to do that?
Thanks in advance

Context:
Hello!
You can use the feature-extraction pipeline for this.
from transformers import pipeline
pipeline = pipeline('feature-extraction', model='xlnet-base-cased')
data = pipeline("this is a test")
print(data)

You can also do this through the Inference API.


Saving-Loading Model in Colab and Making Predictions

Question:
I’m fairly new to Python and HuggingFace and have what is probably a simple question about saving and loading a model. I can’t figure out how to save a trained classifier model and then reload so to make target variable predictions on new data. As an example, I trained a model to predict imbd ratings with an example from the HuggingFace resources, shown below. I’ve tried a number of ways (save_model, save_pretrained) and either am struggling to save it at all or when loaded, can’t figure out what to call to get predictions. Any help would be incredibly appreciated on the steps that involve saving/loading/predicting new scores.
#example mainly from here: https://huggingface.co/transformers/training.html
!pip install transformers
!pip install datasets

from datasets import load_dataset
raw_datasets = load_dataset("imdb")

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

def tokenize_function(examples):
    return tokenizer(examples["text"], max_length = 128, padding="max_length", truncation=True)

tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)

#choosing small datasets for example#
small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(1000))
small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(500))

### TRAINING classification ###
from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2)

from transformers import TrainingArguments
from transformers import Trainer

training_args = TrainingArguments("test_trainer", evaluation_strategy="epoch", num_train_epochs=2, weight_decay=.0001, learning_rate=0.00001, per_device_train_batch_size=32)

trainer = Trainer(model=model, args=training_args, train_dataset=small_train_dataset, eval_dataset=small_eval_dataset)
trainer.train()

y_test_predicted_original = model_loaded.predict(small_eval_dataset)

#### Saving ###
from google.colab import drive
drive.mount('/content/gdrive')
%cd /content/gdrive/My\ Drive/FOLDER

trainer.save_pretrained ("Trained model") #assumed this would save but did not
model.save_pretrained ("Trained model") #did save

### Loading Model and Creating Predicted Scores ###

#perhaps this....#
from transformers import BertConfig, BertModel
conf = BertConfig.from_pretrained("Trained model", num_labels=2)
model_loaded = AutoModelForSequenceClassification.from_pretrained("Trained model", config=conf)

#or...#
model_loaded = AutoModelForSequenceClassification.from_pretrained("Trained model", local_files_only=True)
model_loaded

#with ultimate goal of getting predicted scores (not sure what to call here)...
y_test_predicted_loaded = model_loaded.predict(small_eval_dataset)

Context: Any insights on this? I can’t find any examples start to finish, which seems like it should be straightforward


Does it make sense to train DistilBERT from scratch in a new corpus

Question:
Hi! First post in the forums, excited to start getting deep into this great library!
I have a rookie, theoretical question. I have been reading the DistilBERT paper (fantastic!) and was wondering if it makes sense to pretrain a DistilBERT model from scratch.
In the paper 2, the authors specify that “The student is trained with a distillation loss over the soft target probabilities of the teacher.”. My question is, when pretraining DistilBERT on a new corpus (say, another language) what are the ‘probabilities of the teacher’? AFAIK, the teacher does not have any interesting probabilites to show since it has never seen the corpus either.
So my question is, how does the transfomers library distill knowledge into the model when I train DistilBertForMaskedLM 3 from scratch in a brand new corpus? Sorry in advance if there is something really obvious I’m missing, I’m quite new to using transformers.
Just to be extra explicit, I would load my model like this:
config = DistilBertConfig(vocab_size=VOCAB_SIZE)
model = DistilBertForMaskedLM(config)

and train it like this:
trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=tokenized_datasets["train"],
        eval_dataset=tokenized_datasets["valid"],
        tokenizer=tokenizer,
        data_collator=data_collator,
    )
trainer.train()

Context:
Hi @lesscomfortable welcome to the forum!
In the DistilBERT paper they use bert-base-uncased as the teacher for pretraining (i.e. masked language modelling). In particular, the DistilBERT student is pretrained on the same corpus as BERT (Toronto Books + Wikipedia) which is probably quite important for being able to effectively transfer the knowledge from the teacher to the student.
So the answer to your question

My question is, when pretraining DistilBERT on a new corpus (say, another language) what are the ‘probabilities of the teacher’? AFAIK, the teacher does not have any interesting probabilites to show since it has never seen the corpus either.

is that the pretrained BERT teacher generates logits and hidden states that can be used to guide the pretraining of the student (through the KL divergence and “cosine embedding” terms in the loss function).
You can find more technical details here 29 and you should check out the distiller.py module to see how the loss is implemented and train.py for the pretraining logic.
If you want to use a trainer my suggestion would be to subclass Trainer, add the teacher as an attribute and override the compute_loss function, e.g.
class DistillationTrainer(Trainer):
    def __init__(self, *args, teacher_model=None, **kwargs):
        super().__init__(*args, **kwargs)
        self.teacher = teacher_model

    def compute_loss(self, model, inputs):
        # adapt the code from distiller.py here

Then you could initialise the teacher and student along the lines you did
teacher_model = BertForMaskedLM.from_pretrained('bert-base-uncased')

student_config = DistilBertConfig(vocab_size=VOCAB_SIZE)
student_model = DistilBertForMaskedLM(student_config)

trainer = DistillationTrainer(
        model=student_model,
        teacher = teacher_model,
        args=training_args,
        train_dataset=tokenized_datasets["train"],
        eval_dataset=tokenized_datasets["valid"],
        tokenizer=tokenizer,
        data_collator=data_collator,
    )

Caveats:

I do not know how well this will work if your corpus is significantly different from the one BERT was pretrained on. For example, if your corpus is in another language then you’d be better off using a different teacher model in that language
This approach is likely to be error-prone and expensive ($$$), so my suggestion is to use the battle-tested scripts from the link above

Hope that helps!


Question:
hi, I was trying to run bert but was getting the error “IndexError: index out of range in self”. after troubleshooting for a couple of days I figured out it was the word “screwing” that was breaking my code. is this a bug or is there certain words you cant use with bert? or am I just doing something wrong?  thanks.
here’s the code example:
import transformers
import torch
import torch.nn as nn
class SentimentClassifier(nn.Module):
def init(self, n_classes):
super(SentimentClassifier, self).init()
self.bert = transformers.BertModel.from_pretrained(‘bert-base-cased’)
self.drop = nn.Dropout(p=0.3)
self.out = nn.Linear(self.bert.config.hidden_size, n_classes)
def forward(self, input_ids, mask):
    output = self.bert(
      input_ids=input_ids,
      attention_mask=mask
    )
    output = self.drop(output['pooler_output'])
    return self.out(output)

this doesn’t work
batch_sentences = [
‘screwing’,
]

running this works
batch_sentences2 = [
‘this is a test sentence’,
‘another one’
]
bert_model = transformers.BertModel.from_pretrained(‘bert-base-uncased’)
tokenizer = transformers.BertTokenizer.from_pretrained(‘bert-base-uncased’)
encoded_inputs = tokenizer(batch_sentences, padding=True, truncation=True, add_special_tokens=True)
samples = torch.tensor(encoded_inputs[‘input_ids’])
targets = torch.zeros(samples.shape[0]).long()
mask = (samples != 0)
print(samples.shape)
model = SentimentClassifier(3)
EPOCHS = 10
optimizer = transformers.AdamW(model.parameters(), lr=2e-5, correct_bias=False)
total_steps = 1 * EPOCHS
scheduler = transformers.get_linear_schedule_with_warmup(
optimizer,
num_warmup_steps=0,
num_training_steps=total_steps
)
criterion = nn.CrossEntropyLoss()
model = model.train()
for i in range(EPOCHS):
print(i)
preds = model(input_ids=samples, mask=mask)
loss = criterion(preds, targets)
print(loss)
loss.backward()
nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
scheduler.step()
optimizer.zero_grad()

Context: I hope you’ve figured out the solution already but it seems that the tokenizer you use bert-base-uncased and the model initialized in the SentimentClassifier class bert-base-cased do not match. There may be overlaps in the vocabulary of the cased and uncased tokenizers which may seem working fine in some cases but the same token ids can decode to totally different text sequences with different tokenizers. In general, using an uncased tokenizer for a cased model or vice versa should always be erroneous, even if there is no error message showing up.


Question:
Hello all, I am training a model using Huggingface and when evaluating it I get approx 80% accuracy.
I then try to plot the confusion matrix (using pycm) but I see no error.
Any ideas what this might be?

Context: Could you be providing the same labels to the confusion matrix plotting function, like plotting predictions versus predictions or ground truth versus ground truth? That kind of mistake is one that I make all the time.


How to stop Optuna saving checkpoints during Hyperparameter Search

Question:
Hello I am running a Hyperparameter search using Optuna.
As I am using Colab, I have limited diskspace, so I was wondering how to stop saving checkpoints, I only care about the final result and don’t need all the intermediate steps saved.
I tried the following argument sin my TrainingArguments parameter, but its not working
# Define the trainig arguments
training_args = TrainingArguments(
    output_dir='./results',          # output directory
    seed = 0,
    num_train_epochs=5,              # total number of training epochs
    per_device_train_batch_size=16,  # batch size per device during training
    per_device_eval_batch_size=16,   # batch size for evaluation
    warmup_steps=22,                 # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    learning_rate=5e-5,              # initial learning rate for AdamW optimizer.
    load_best_model_at_end=True,     # load the best model when finished training (default metric is loss)
    do_train=True,                   # Perform training
    do_eval=True,                    # Perform evaluation
    logging_dir='./logs',            # directory for storing logs
    logging_steps=10,
    gradient_accumulation_steps=2,   # total number of steps before back propagation
    fp16=True,                       # Use mixed precision
    fp16_opt_level="02",             # mixed precision mode
    evaluation_strategy="epoch",     # evaluate each `logging_steps`
    save_strategy = 'no',           # The checkpoint save strategy to adopt during training. I dont want to save, probably why it did save and take up disk space in HP search
    save_steps = 100000,
    save_total_limit = 1.           # Trying this to stop octuna from saving

Any help would be appreciated, thank you!

Context:
Ok after reading the documentation carefully, it turns out setting
load_best_model_at_end=True, overrides the strategy. Took it off and now it works.


NameError: name ‘BertTokenizer’ is not defined

Question:
Hi,
I am trying to add custom tokens using this code below:
# Let's see how to increase the vocabulary of Bert model and tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = AutoModelForMaskedLM.from_pretrained('bert-base-uncased')

num_added_toks = tokenizer.add_tokens(['token_1'])
print('We have added', num_added_toks, 'tokens')
model.resize_token_embeddings(len(tokenizer))  # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e. the length of the tokenizer.

Though, when executing the above code, I get this error:
---------------------------------------------------------------------------

NameError                                 Traceback (most recent call last)

<ipython-input-36-31798d520617> in <module>()
      1 # Let's see how to increase the vocabulary of Bert model and tokenizer
----> 2 tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
      3 model = AutoModelForMaskedLM.from_pretrained('bert-base-uncased')
      4
      5 num_added_toks = tokenizer.add_tokens(['token_1'])

NameError: name 'BertTokenizer' is not defined

Context: hey @anon58275033 what version of transformers are you using? i was not able to reproduce the error in v4.6.1


How to add new tokens for existing masked language modelling?

Question:
Hi,
I have followed this tutorial from GitHub on masked language modelling: notebooks/language_modeling.ipynb at master · huggingface/notebooks · GitHub 3
But, I am wondering, how do I modfiy this code below for the masked language modelling task, and where in my code do I place it?
In the tutorial, this line of code is used:
from transformers import AutoModelForMaskedLM
model = AutoModelForMaskedLM.from_pretrained(model_checkpoint)

This is the code I need to modify to satisfy MLM:
 Let's see how to increase the vocabulary of Bert model and tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')

num_added_toks = tokenizer.add_tokens(['new_tok1', 'my_new-tok2'])
print('We have added', num_added_toks, 'tokens')
model.resize_token_embeddings(len(tokenizer))  # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e. the length of the tokenizer.

Context: First of all, I guess you want to use BertForMaskedLM instead of BertModel. The other parts should work AFAIK.


API Rest with several models loaded using GPU but not at same time

Question:
I am creating an API Rest (using Flask) that does inference with several models given a list. For example summarization, sequence-to-sequence classification, etc …
The problem is that all the models don’t fit at GPU at the same time.
Is there a way of loading a model into GPU make inference with that model and move it to CPU and load next model to GPU for inference then to CPU…

Context:
UPDATE
The Summarization task works on GPU if I run the script on the Virtual Machine without calling it on flask. However, once I start it on Flask I get:
RuntimeError: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling `cublasCreate(handle)`


How to test my text classification model after training it?

Question:
Hello,
I have followed this tutorial on text classification: notebooks/text_classification.ipynb at master · huggingface/notebooks · GitHub 5
Now, I have trained it using my own data, but I am unsure how to actually deploy it to carry out a classification task.
For example, I want to input the following sentence: “You look good today.” And, from there, I want to see if the classification is positive or negative, but I do not know how to do this.
Any help is much appreciated.

Context:
That’s a good question. cc @sgugger, would be great if the several notebooks also include an inference part. I had to look into several notebooks before finding out you can access the trained model using trainer.model. Here’s how to do inference on a new, unseen sentence:
sentence = “You look good today.”
# encode sentence (i.e. create input_ids, attention_mask)
encoding = tokenizer(sentence)
# make sure the keys of the "encoding" dict are on the same device as the model
encoding = {k: v.to(trainer.args.device) for k, v in encoding.items()}
# forward pass through the model
with torch.no_grad():
    outputs = trainer.model(**encoding)
logits = outputs.logits
print("Predicted class index:", logits.argmax(-1))

The predicted class index will be either a zero or a one (I guess one represents positive).


Which loss function in bertforsequenceclassification regression

Question:
BertForSequenceClassification can be used for regression when number of classes is set to 1.  The documentation says that BertForSequenceClassification calculates cross-entropy loss for classification.  What kind of loss does it return for regression?
(I’ve been assuming it is root mean square error, but I read recently that there are several other possibilities such as Huber or Negative Log Likelihood.)
Which is it?
How should I find out / where is the code?

Context:
This is the GitHub 135 link
At line 1354, you have the condition to check the labels (if it is one or more)
if self.num_labels == 1:
#  We are doing regression
loss_fct = MSELoss()
loss = loss_fct(logits.view(-1), labels.view(-1))
else:
loss_fct = CrossEntropyLoss()
loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))


Question: Any way to modify the length of the output text generated by the GPT-neo inference API?

Context: Does anyone know a solution for this?


Question:
Hi guys,
i am kinda new to hugginface and have a question regarding the PPL.
So what i have is, i fine-tuned a model and at the end of the traning i get the PPL
for the dev dataset by doing:
eval_results = trainer.evaluate()
print(f"Perplexity: {math.exp(eval_results['eval_loss']):.2f}")

However, in my DatasetsDict I have train, dev and test and also want to get the PPL
for the test set. Is there any simple way do get it done?
Best regards
Chris

Context: Hey Chris, i’m a begginer as well but i think you could try using EvalPrediction 4. You pass your test_set and it’s labels and then you should get your loss at the specific set.


How to load ckpt into my model base on tf2.x

Question:
I need to load ckpt file from google-search’s bert ckpt.
I have read all questions related to this and tried some methods but it seem still doesn’t work .
ckpt files are like this:

I have tried method like:
bert_config = transformers.BertConfig.from_json_file('./bert/bert_config.json')
bert = transformers.TFBertModel.from_pretrained('./bert/bert_model.ckpt', config=bert_config)

I find a method said that use TFBertForPreTraining instead TFBertModel
bert_config = transformers.BertConfig.from_json_file('./bert/bert_config.json')
bert = transformers.TFBertModel.from_pretrained('./bert/bert_model.ckpt', config=bert_config)

but it still doesn’t work.
Also ,
 bert.load_weights('./bert/bert_model.ckpt')

doesn’t work too.
I really can’t understand why bert.load_weights('./bert/bert_model.ckpt') can’t work.Maybe rewrite it?
so how can a model base on tf2 can load a ckpt file?

Context: hello？Can someone help me？


What’s the difference between wordpiece and sentencepiece?

Question:
I found that the wordpiece training and the sentencepiece are almost the same.
So what’s the difference ?? Thanks

Context: See for example this post: https://towardsdatascience.com/a-comprehensive-guide-to-subword-tokenisers-4bbd3bad9a7c 7


Evaluating Finetuned BERT Model for Sequence Classification

Question:
Python 3.7.6
Transformers 4.4.2
Pytorch 1.8.0
Hi HF Community!
I would like to finetune BERT for sequence classification on some training data I have and also evaluate the resulting model. I am using the Trainer class to do the training and am a little confused on what the evaluation is doing. Below is my code:
import torch
from torch.utils.data import Dataset
from transformers import BertForSequenceClassification, BertTokenizer, Trainer, TrainingArguments
import pandas as pd

class MyDataset(Dataset):
    def __init__(self, csv_file: str):
        self.df = pd.read_csv(csv_file, encoding='ISO-8859-1')
        self.tokenizer = BertTokenizer.from_pretrained("bert-base-uncased", padding_side='right', local_files_only=True)
        self.label_list = self.df['label'].value_counts().keys().to_list()

    def __len__(self) -> int:
        return len(self.df)

    def __getitem__(self, idx: int) -> str:
        if torch.is_tensor(idx):
            idx = idx.tolist()

        text = self.df.iloc[idx, 1]
        tmp_label = self.df.iloc[idx, 3]
        if tmp_label != 'label_a':
            label = 1
        else:
            label = 0
        return (text, label)

def data_collator(self, dataset_samples_list):
    tokenizer = BertTokenizer.from_pretrained("bert-base-uncased", padding_side='right', local_files_only=True)
    examples = [example[0] for example in dataset_samples_list]
    encoded_results = tokenizer(examples, padding=True, truncation=True, return_tensors='pt',
                                return_attention_mask=True)

    batch = {}
    batch['input_ids'] = torch.stack([result for result in encoded_results['input_ids']])
    batch['attention_mask'] = torch.stack([result for result in encoded_results['attention_mask']])
    batch['labels'] = torch.stack([torch.tensor(example[1]) for example in dataset_samples_list])
    return batch

train_data_obj = MyDataset('/path/to/train/data.csv')
eval_data_obj = MyDataset('/path/to/eval/data.csv')
model = BertForSequenceClassification.from_pretrained("bert-base-uncased")

training_args = TrainingArguments(
    output_dir='/path/to/output/dir',
    do_train=True,
    do_eval=True,
    per_device_train_batch_size=2,
    per_device_eval_batch_size=2,
    evaluation_strategy='epoch',
    num_train_epochs=2,
    save_steps=10,
    gradient_accumulation_steps=4,
    dataloader_drop_last=True
)

trainer = Trainer(
            model=model,
            args=training_args,
            train_dataset=train_data_obj,
            eval_dataset=eval_data_obj,
            data_collator=data_collator
        )

trainer.train()
trainer.save_model("/path/to/model/save/dir")
trainer.evaluate()

As I understand, once trainer.train() is called, after each epoch the model will be evaluated on the dataset from eval_data_obj and those results will be displayed. After the training is done and the model is saved using trainer.save_model("/path/to/model/save/dir"), trainer.evaluate() will evaluate the saved model on the eval_data_obj and return a dict containing the evaluation loss. Are there other metrics like accuracy that are included in this dict by default? Thank you in advance for your help!

Context: If you want other metrics, you have to indicate that to the Trainer by passing a compute_metrics function. See for instance our official GLUE example 20 or the corresponding notebook 30.