Open-sourcing better cross-encoders for STILTS and better IR?

Question:
Hi @nreimers,
I find your research on bi-encoders and models on sbert.net 3 super helpful. Based on your research I understand that cross-encoders generally perform better than bi-encoders, while their main disadvantage is computational speed.
I’m very interested in deepening my research in cross-encoders, but I noticed that you’ve only published comparatively few cross-encoders here: cross-encoder (Sentence Transformers - Cross-Encoders) 2.
My question: Could you consider to publish improved cross-encoders, either trained on your paraphrase data or the ‘all’ data from the FLAX event (‘all-mpnet…’ etc.)?
I feel like this would have great added value for the HF- and research-community, because:
- Improved cross-encoders trained on more diverse data could be great improved STILTS for sequential transfer learning applications. (see here https://arxiv.org/pdf/1811.01088.pdf 1)
- Your bi-encoders are probably already good STILTS, but I imagine that cross-encoders would be even better. Using these intermediate models for task-specific fine-tuning would probably be a super easy way for people to get improved performance on many tasks - just by taking your cross-encoder as the base model instead of BERT-base etc.
- Having high-performance cross-encoders would also be useful for implementing BM25 & cross-encoder reranking for information retrieval applications etc.
Could you consider to published improved cross-encoders?
(Maybe there are technical reasons why your paraphrase or ‘all’ data cannot be used for cross-encoders and that’s the reason why non are published with this data?)
Best,
Moritz

Context:
Hi,
Happy to hear that
Better cross encoders that are trained on larger datasets are on my agenda. However, training is not so straightforward. For bi-encoders, you use the other examples in a batch as negative.
For cross-encoders, you have to create the negative pairs. Here, the creation of the negative pairs plays an extremely important role.
I hope I will soon be able to train these models. But setting up the training etc takes some effort.
Best
Nils


Pipelines for mutliple inputs don’t produce reliable results

Question:
I am using a text classification pipeline (‘sentiment-analysis’) with a fine-tuned ELECTRA model and transformers version 4.5.1
For some reason, calling the pipeline for a list of inputs will result in different outputs for each input than with applying the pipeline to each input! Why is this like that? I went through the patch notes but couldn’t see any fix for this issue, so I’m not sure if this still persists in recent versions.

Context:
I found out that this is not related to the transformers package, but is probably due to PyTorch optimisations with operations potentially happening in different orders depending on the input tensor - as float operations are inaccurate, this may lead to different results for the same inputs to a pipeline if combined with other input sentences. As this does not (only) depend on the shape of the tensor but also on the content, the only safe way to generate the exact same output is by applying the pipeline only for one input sentence one at a time.
For most use cases, class probabilities changing by values around 10^-6 doesn’t matter, but if you require exact results, be aware of this issue!


A new dataset for multi-label text classification

Question:
Soumik and I are pleased to share a new NLP dataset for multi-label text classification. The dataset consists of paper titles, abstracts, and term categories scraped from arXiv. Find the dataset on Kaggle: arXiv Paper Abstracts | Kaggle 2.
We are also releasing our data collection pipeline which is based on Apache Beam that can be run on Cloud Dataflow (GCP) at scale and can be used to accumulate an even bigger dataset at ease. To help the community get started quickly we have authored this blog post that shows how to build a simple baseline model for a smaller version of the dataset.
More details are here: GitHub - soumik12345/multi-label-text-classification 7

Context: Cool! Would be great if you upload this dataset to the hub  here’s a guide: Share — datasets 1.12.1 documentation 2


FlaxGPTNeoForCausalLM generates the same text regardless of seed, temperature, top_k and top_p values

Question:
Hello,
I was trying to generate text using flax (just as an experiment to see if it works well on a TPU-VM machine). However, no matter how I tried, it always generates the exact same text for a given prompt. This happened both on a TPU-VM as well as local CPU inference.
Here is a short code snippet which demonstrates the problem I encountered:
from transformers import FlaxGPTNeoForCausalLM, AutoTokenizer

model_name = 'EleutherAI/gpt-neo-125M'
model = FlaxGPTNeoForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt_text = "Hello there, my name is"
generated_max_length = 50

#Changing the seed value, does not seem to change the outcome
seed = 1001
model.seed = seed
model.config.pad_token_id = model.config.eos_token_id

inputs = tokenizer(prompt_text, return_tensors="jax")

#Changing temperature, top_k and top_p does not seem to change the outcome
outputs = model.generate(
    input_ids = inputs["input_ids"],
    max_length=generated_max_length,
    do_sample=True,
    temperature=0.8,
    early_stopping=True,
    top_k=50,
    top_p=0.90)

output_sequence = outputs['sequences'].squeeze(0)
text = tokenizer.decode(output_sequence, clean_up_tokenization_spaces=True)

print(text)

#Always prints:
#Hello there, my name isergus, and I was presented a competition looking for a library keeper with the 31-K--Goods Wallace Leisure library manager on 10-11-08 447-5721. It involves teaching a Royally

I also tried calling jax.random.PRNGKey(seed) which didn’t help as well as other methods such as:
model.top_p = 0.9
model.top_k = 50

jit_generate = jax.jit(model.generate)
#jit_generate( inputs["input_ids"],  ....

I assume I’m doing something very wrong, but I was not able to find any example code for generating text with FlaxGPTNeoForCausalLM (I did find examples for training it).
I hope I posted this in the right forum.
Regards,
Doron

Context:
Update: Tried again on the latest Master branch (transformers-4.11.0.dev0) and this time I was able to get it to generate a different output by changing the seed. However, changing temperature, top_k and top_p still does not seem to have an influence on the outcome.
import jax
from transformers import FlaxGPTNeoForCausalLM, AutoTokenizer

model_name = 'EleutherAI/gpt-neo-125M'
model = FlaxGPTNeoForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt_text = "Hello there, my name is"
generated_max_length = 50

#Changing the seed and thus the prng_key value below, does seem to change the outcome.
seed = 1000
model.seed = seed

inputs = tokenizer(prompt_text, return_tensors="np")

#Changing temperature, top_k and top_p does not seem to change the outcome
outputs = model.generate(
    input_ids = inputs["input_ids"],
    max_length=generated_max_length,
    pad_token_id = model.config.eos_token_id,
    prng_key=jax.random.PRNGKey(seed),
    temperature=1.0,
    early_stopping=True,
    top_k=50,
    top_p=0.95,
    do_sample=True,
    no_repeat_ngram_size=4)

output_sequence = outputs['sequences'].squeeze(0)
text = tokenizer.decode(output_sequence, clean_up_tokenization_spaces=True)

print(text)


Custom GPT2 Model won’t load after training

Question:
Environment info

transformers version: 4.10.2
Platform: Linux-5.11.0-34-generic-x86_64-with-glibc2.29
Python version: 3.8.10
PyTorch version (GPU?): 1.8.1+cu102 (True)
Tensorflow version (GPU?): 2.4.1 (False)
Flax version (CPU?/GPU?/TPU?): not installed (NA)
Jax version: not installed
JaxLib version: not installed
Using GPU in script?:
Using distributed or parallel set-up in script?:

Information
Model I am using GPT2PretrainedModel.
The problem arises when using:

[ ] the official example scripts: (give details below)
[ x ] my own modified scripts: (give details below)

The tasks I am working on is:

[ ] an official GLUE/SQUaD task: (give the name)
[ x ] my own task or dataset: (give details below)

The Problem
I was able to train my customly build model but I am not able to load it with the from_pretrained() function. BTW I don’t save the model manually if that is important. The saving is done by the Huggingface-Trainer.
The Error message:
    model = CustomGPTModel.from_pretrained("results/checkpoint-19065", config=config)
File "/home/flo/PycharmProjects/EET2/venv/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1325, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
TypeError: __init__() missing 1 required positional argument: 'config'

I load the model like this:
config = AutoConfig.from_pretrained("results/checkpoint-19065")

model = CustomGPTModel.from_pretrained("dbmdz/german-gpt2", config=config)
# custom = CustomGPTModel(model=model, config=config)
training_args = TrainingArguments(
    output_dir='./results',  # output directory
    per_device_train_batch_size=1,  # batch size per device during training
    per_device_eval_batch_size=1,  # batch size for evaluation
    logging_dir='./logs/event/',  # directory for storing logs
)
trainer = Trainer(
    model=model,  # the instantiated 🤗 Transformers model to be trained
    # model=custom,  # the instantiated 🤗 Transformers model to be trained

    args=training_args,  # training arguments, defined above
    compute_metrics=compute_everything,
    )
trainer.predict(test_dataset=test_dataset)

As you can tell from the commented code, I tried a lot of different approaches to no avail.
Other approaches I tried:
config = AutoConfig.from_pretrained("results/checkpoint-19065")
model = CustomGPTModel.from_pretrained("results/checkpoint-19065", config=config)
# or
config = AutoConfig.from_pretrained("results/checkpoint-19065")
model = CustomGPTModel.from_pretrained("results/checkpoint-19065")

Anyway the question is How do I load my custom model?
I think it is because of the way I initialize the CustomGPTModel (see below).

The Task / More Information on what I am Doing
I am training the “dbmdz/german-gpt2” on a multilabel-classification task. For this I had to create my own model by subclassing the GPT2PretrainedModel. This is what the model looks like:
class CustomGPTModel(GPT2PreTrainedModel):
    def __init__(self, model, config):
        super(CustomGPTModel, self).__init__(config)
        self.num_labels = config.num_labels
        self.init_weights()

        ### Architecture:
        self.transformer = model
        self.linear1 = nn.Linear(config.n_embd, 256)
        self.score = nn.Linear(256, self.num_labels, bias=False)
        self.dropout = nn.Dropout(p=0.2)
        self.sig = nn.Sigmoid()
        self.relu = nn.ReLU()

        # Model parallel
        self.model_parallel = False
        self.device_map = None
    def forward(self, input_ids=None, past_key_values=None, attention_mask=None,
                token_type_ids=None, position_ids=None, head_mask=None, inputs_embeds=None,
                labels=None, use_cache=None, output_attentions=None, output_hidden_states=None,
                return_dict=None, ):

        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        transformer_outputs = self.transformer(
              input_ids,
              past_key_values=past_key_values,
              attention_mask=attention_mask,
              token_type_ids=token_type_ids,
              position_ids=position_ids,
              head_mask=head_mask,
              inputs_embeds=inputs_embeds,
              use_cache=use_cache,
              output_attentions=output_attentions,
              output_hidden_states=output_hidden_states,
              return_dict=return_dict,
          )
        hidden_states = transformer_outputs[0]  # call model
        hdn_2 = self.linear1(hidden_states)  # first linear
        logits = self.score(self.dropout(self.relu(hdn_2)))  # apply activation/dropout and final layer

        if input_ids is not None:
            batch_size, sequence_length = input_ids.shape[:2]
        else:
            batch_size, sequence_length = inputs_embeds.shape[:2]

        assert (
                self.config.pad_token_id is not None or batch_size == 1
        ), "Cannot handle batch sizes > 1 if no padding token is defined."
        if self.config.pad_token_id is None:
            sequence_lengths = -1
        else:
            if input_ids is not None:
                sequence_lengths = torch.ne(input_ids, self.config.pad_token_id).sum(-1) - 1
        pooled_logits = logits[range(batch_size), sequence_lengths]
        loss = None
        if labels is not None:
            loss_fct = BCEWithLogitsLoss()
            loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1, self.num_labels))
            return (loss, pooled_logits)
        else:
            return logits

Here I initialize the model for training:
training_args = TrainingArguments(
        output_dir='./results',  # output directory
        num_train_epochs=10,  # total number of training epochs
        per_device_train_batch_size=1,  # batch size per device during training
        per_device_eval_batch_size=1,  # batch size for evaluation

        warmup_steps=500,  # number of warmup steps for learning rate scheduler
        weight_decay=0.01,  # strength of weight decay
        logging_dir='./logs/event/',  # directory for storing logs
        logging_steps=1000,
        load_best_model_at_end=True,
        evaluation_strategy="epoch",  # Evaluation is done (and logged) every eval_steps
        save_strategy="epoch",

        # logging_first_step = True,
        do_eval=True,
    )
trainer = Trainer(
        model=custom_gpt2,                         # the instantiated 🤗 Transformers model to be trained
        args=training_args,                  # training arguments, defined above
        train_dataset=train_dataset,         # training dataset
        eval_dataset=val_dataset,             # evaluation dataset
        compute_metrics=compute_everything,
        callbacks=[EarlyStoppingCallback(early_stopping_patience=3)],
    )
trainer.train()

Expected behavior
The model should get loaded as expected.
I tried to fix it for two days now so I thought creating an issue is the last straw. Hopefully someone can explain what I am doing wrong  If someone needs more information please tell me!

Context:
You should either:

use the regular torch.load to load the weights of your models
or make sure your custom model class subclasses PreTrainedModel and is initalized with a single config, like all Transformers models if you want to use from_pretrained.


Optimal methods to monitor attention matrices when doing training/inference using BERT-type models

Question:
Our team is using BERT/Roberta from the huggingface transformers library for sequence-classification (amongst other tasks). We are looking for an efficient way to monitor the attention matrices so as to understand what the model is doing during inference (i.e. the model made this prediction because it is focusing on these words, etc). Are there any useful code snippets used for analysis.
Often the models make funny predictions, and it’s hard to understand why… How are other teams managing this process? We want to avoid large bloated (graphical) tools, and would prefer simplicity.
thanks!

Context: Just checking you have seen BertViz 4 as a source of ideas if nothing else.


Question:
Hello everyone,
I know the question isn’t new but I wanted to see if there have been new features/developments to this: Is there a way to easily add class weights while fine tuning a bert model? I am doing the fine tuning with the Trainer() instance.
Cheers

Context: You can override the loss computation anyway you like by subclassing the Trainer, as is documented here 4.


Question:
Hi all!
Is it possible to use a pretrained BERT model to initialize the encoder part of an encoder-decoder model like BART, leaving the decoder uninitialized (or random), and then do fintuning on some seq2seq task?
How should I proceed if its possible? Does someone know of previous instances where something like that has been tried?
Thanks in advance!
Best,
Gabriel.

Context:
I am not entirely sure about Bart but you can check out this: transformers/modeling_encoder_decoder.py at master · huggingface/transformers · GitHub 8
Also you can read the publication linked in comments, I think this is similar to what you want to achieve


Save custom transformer as PreTrainedModel

Question: I have a custom BERT-like model (with modified attention) that I pretrained with PyTorch. Data preparation was done with a Huggingface tokenizer. Now I want to integrate this PyTorch-model in the Huggingface environment so it can be used in pipelines and for finetuning as a PreTrainedModel. How do I generate the necessary config files?

Context:
There might be a better way, but I would:

subclass PretrainedModel with your own class
load your trained model weights into this new class
run yourmodel.save_pretrained() to save the model weights for this class
now you can do YourCustomModel.from_pretrained() as it now can use those HF methods


Question:
Hello everyone, Is there a way to create non bert tokenizer for dpr model? And how much effort it will be to create one?
Maybe someone have experience converting models from original library to transformers (I know there is a script for this, but it doesn’t show what to do with tokenisers).

Context:
just want to add more context.
I’ve been trying to create dpr for my language, but now I can’t use it with transformers because DPRContextEncoderTokenizer inherits specifically from BertTokenizer, not PreTrainedTokenizer, I think it will be helpful to change it to something more general, can someone help me figure out the hierarchy of classes or maybe design some specific behaviour (for example hold actual tokenizer inside DPRTokenizer and then just override from_pretrained?)


Question: In the paper for T5, I noticed that the inputs to the model always a prefix (ex. “summarize: …” or “translate English to German: …”. When I finetune a T5 model, can I use any phrase/word that I want as a prefix, or can T5 only understand a specific predefined list of prefixes?

Context:
T5 only has been trained on a specific set of prefixes. You can find a list here:
https://arxiv.org/pdf/1910.10683.pdf 4 (starting at page 47)
That said, you can just finetune without a prefix (or with a custom prefix) and it should still work out.


Linear learning rate despite lr_scheduler_type=“polynomial”

Question:
Hello,
While fine-tuning my network, I would like to set up a polynomial learning rate scheduler by setting lr_scheduler_type="polynomial" and learning_rate=0.00005.
However, when I visualize the learning rate on the wandb dashboard, I’m observing a linear decrease of the learning rate instead of polynomial. (Screenshot below)
What might be causing the issue? I’ve tested without setting the learning_rate and the behavior was exactly the same.

image872×622 22.8 KB

Context: Polynomial scheduler is not really supported via this argument, as it requires an additional power keyword argument that defaults to 1 and which can’t be set via this API. You should thus set the scheduler directly in the Trainer.


Finetuning from multiclass to mutlilabel

Question:
Hello,
I got a specific classification task where I finetuned a pretrained BERT Model on a specific task concerning customer reviews (classify a text as “customer service text”, “user experience” etc.):
num_labels_cla = 8
model_name_cla = "bert-base-german-dbmdz-uncased"
batch_size_cla = 32

model = AutoModelForSequenceClassification.from_pretrained(model_name_cla, num_labels=num_labels_cla)
tokenizer = AutoTokenizer.from_pretrained(model_name_cla)

As you can see I got 8 distinct classes. My finetuned classification model scores pretty well with unseen data with a f1 score of 80.1. However, it is possible that one text belongs to 2 different classes. My question now is how I have to change my code achieve that? I already transformed my target variable with MultiLabelBinarizer such that my target variable looks like this:
       [0, 0, 0, 1, 0, 0, 0, 0],
       [1, 0, 0, 0, 0, 0, 0, 0],
       [1, 0, 0, 1, 0, 0, 0, 0],

I am using the HuggingFace Trainer instance for finetuning.
Cheers

Context:
You can set the problem_type of an xxxForSequenceClassification model to multi_label_classification when instantiating it, like so:
from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("bert-base-german-dbmz-uncased", problem_type="multi_label_classification", num_labels=num_labels_cla)

This ensures that the BCEWithLogitsLoss is used instead of the CrossEntropyLoss, which is necessary for multi-label classification. You can then fine-tune just like you would do with multi-class classification.


Question:
Hi,
I am pre-training a Bert model from scratch using Tensorflow.
I’ve seen the methid to push pyTorch models, but I don’t know how to do with my TF model.
Here is how I imagine I have to do:
1- Convert my checkpoint from Tf to torch
2- Push to HF
Is this correct?
But an other question is :
I don’t know how to push the tokenizer, all I am having is:

my vocab.txt
tokenizer.vocab
-tokenizer.model

how can I do this please.
Thanks

Context:
If you pre-trained BERT from scratch in TF using the run_mlm.py script, you can easily convert the model from TF to PyTorch, like so:
from transformers import BertForMaskedLM

model = BertForMaskedLM.from_pretrained("name of directory where the run_mlm.py script saved all files", from_tf=True)
model.save_pretrained("name of directory where you'd like to save all model files")

Next, you can easily push it to the hub as follows (I’m assuming you’re in a Colab notebook):
First, install git-LFS:
!sudo apt-get install git-lfs
!git config --global user.email "<your email>"
!git config --global user.name "<your name>"

Next, create a repo on the hub, then git clone it:
git clone <URL of your repository on the hub>

Next, add your files and upload them:
git add .
git commit -m "First commit"
git push


Question:
I want to do Multiclass-Multilabel ( MLMC) classification problem using Conv-BERT model.
Steps that I have taken is:
I downloaded the Conv-Bert model from this link: YituTech/conv-bert-base · Hugging Face 2
<< YituTech/conv-bert-base>>
from pytorch_pretrained_bert import BertTokenizer, BertForSequenceClassification, BertAdam
tokenizer = BertTokenizer.from_pretrained("path_to_Conv-Bert_model", do_lower_case = True)
model = BertForSequenceClassification.from_pretrained("path_to_Conv-Bert_model", num_labels = 240)
model.cuda()

I want to understand can we call any classification module from Hugging face and pass any pre-trained models to it like Roberta, Conv-BERT.. so on. ? << As in above example>>

Context:
I think the question is a bit too vague but if you mean the BERT-like family of models can be loaded using the code above and similarly for others but ConvBERT for example has its own class (see here: ConvBERT — transformers 4.7.0 documentation 2).
The head that will be initialized will be of random weights so the model will need fine-tuning.


BERT finetuning “index out of range in self”

Question:
Hello everyone,
I am trying to build a Multiclass Classifier with a pretrained BERT model. I am completely new to the topic. I have 8 classes and use Huggingface’s Dataset infrastructure to finetune a pretrained model for the german language:
from transformers import AutoModelForSequenceClassification
from transformers import Trainer, TrainingArguments
from sklearn.metrics import accuracy_score, f1_score

num_labels_cla = 8
model_name_cla = "bert-base-german-dbmdz-uncased"
batch_size_cla = 8

model = AutoModelForSequenceClassification.from_pretrained(model_name_cla, num_labels=num_labels_cla)

def tokenize(batch):
    return tokenizer(batch['text'], padding=True, truncation=True,max_length=260)

def compute_metrics(pred):
    labels = pred.label_ids
    preds = pred.predictions.argmax(-1)
    f1 = f1_score(labels, preds, average="weighted")
    acc = accuracy_score(labels,preds)
    return {"accuracy":acc, "f1":f1}

My model shouldn’t be a sentiment classifier but a multilabel classifier which classifies customer reviews based on different label (e.g customer support etc.).
When I train/finetune my model with the Huggingface Trainer() instance:
#Encoding the data
data_encoded = data_dict.map(tokenize, batched=True, batch_size=None)
data_encoded.set_format("torch", columns=["input_ids", "attention_mask", "label"])

#Specify training arguments
logging_steps=len(data_encoded["train"])
training_args = TrainingArguments(output_dir='./results',
                                  num_train_epochs=3,
                                 learning_rate=2e-5,
                                 per_device_train_batch_size=batch_size_cla,
                                 per_device_eval_batch_size=batch_size_cla,
                                 load_best_model_at_end=True,
                                 metric_for_best_model="f1",
                                 weight_decay=0.01,
                                 evaluation_strategy="steps",
                                  eval_steps = 2,
                                 disable_tqdm=False,
                                 logging_steps=logging_steps)

#Specify trainer
trainer = Trainer(
    model=model, args=training_args, compute_metrics=compute_metrics, train_dataset=data_encoded['train'], eval_dataset=data_encoded['test']
)

#Train
trainer.train()

After 6 steps I get the following error:
~/miniconda3/envs/textmallet/lib/python3.9/site-packages/torch/nn/modules/sparse.py in forward(self, input)
    156
    157     def forward(self, input: Tensor) -> Tensor:
--> 158         return F.embedding(
    159             input, self.weight, self.padding_idx, self.max_norm,
    160             self.norm_type, self.scale_grad_by_freq, self.sparse)

~/miniconda3/envs/textmallet/lib/python3.9/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   2041         # remove once script supports set_grad_enabled
   2042         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 2043     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   2044
   2045

IndexError: index out of range in self

Does anyone have any idea what I could change in my code?
Cheers

Context:
Hi @marlon89
I can’t see the tokenizer you are loading. Are you sure it is the correct one (e.g. AutoTokenizer.from_pretrained("bert-base-german-dbmdz-uncased")?


Encoding/decoding NLP model in tensorflow lite (fine-tuned GPT2)

Question:
We are in the process of building a small virtual assistant and would like it to be able to run a fine-tuned version of GPT-2 on a raspberry-pi with a coral accelerator.
So far, we managed to convert our model to a tflite and to get first results. We know how to convert from words to indices with the previous tokenizer but then we need a bigger tensor as input to the interpreter. We miss the conversion from indices to tensors. Is there a way to do this simply?
You can find our pseudo-code here, we are stuck at step 2 and 6 :

import tensorflow as tf

#Prelude
TF_MODEL_PATH_LITE = "/path/model.tflite"

interpreter = tf.lite.Interpreter(model_path=TF_MODEL_PATH_LITE)
interpreter.allocate_tensors()
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
input_shape = input_details[0]['shape']

#1-Encode input, giving you indices
context_idx = tokenizer.encode("Hello world.", return_tensors = "tf")

#2-How to convert the context_idx to appropriate np.array ?
input_data = np.array(np.random.random_sample(input_shape), dtype=np.int32) #dummy input for now

#3- feed input
interpreter.set_tensor(input_details[0]['index'], input_data)

#4- Run model
interpreter.invoke()

#5- Get output as tensor
output_data = interpreter.get_tensor(output_details[0]['index'])

#6- How decode this np array to idx
output_idx=np.random.randint(100) #dummy for now ...

#7- Decode Output from idx to word
string_tf = tokenizer.decode(output_idx, skip_special_tokens=True)

Context: Were you able to accelerate your model with the Coral TPU in the end?


Does fine-tuning a language model modify its hidden weights?

Question:
If i load a pre-trained language model (say BERT) and use a standard pytorch implementation (as shown in the code-block), are the weights of the bert-model updated ? and if not is it recommended to try doing so for a down-stream task ?
Note: the task involves using the bert embeddings for clustering text using a clustering algorithm (like KMeans, DBSCAN, etc…)
class Model(nn.Module):
  def __init__(self, name):
    super(Model, self).__init__()
    self.bert = transformers.BertModel.from_pretrained(config['MODEL_ID'], return_dict=False)
    self.bert_drop = nn.Dropout(0.0)
    self.out = nn.Linear(config['HIDDEN_SIZE'], config['NUM_LABELS'])
    self.model_name = name

  def forward(self, ids, mask, token_type_ids):
    _, o2 = self.bert(ids, attention_mask = mask, token_type_ids = token_type_ids)
    bo = self.bert_drop(o2)
    output = self.out(bo)
    return output

Context:
When you do optimizer = torch.optimizer.SGD(Model("bert").parameters(), lr=1e-3) the parameters of the pretrained transformer are ready to be updated in addition to other layers you introduce. In my experience, updating only the newly introduced layers during fine-tuning resulted in very slow convergence and I would recommend updating the transformer weights as well.
If you would use a clustering algorithm that assumes embeddings close to each other in the vector space are semantically related, I recommend that you use a loss function to enforce such a beahavior in fine-tuning (maybe Triplet Loss with Siamese Modeling like SBERT) because CLS embedding space, so to speak, is not constructed with such concerns like context-independent word embedding  spaces.


Training a language model from scratch with tensorflow (not pytorch)?

Question:
Hello there,
I am interested in training a language model from scratch. Not fine tuning the usual distilbert ---- running the whole thing on my GPU instead !
I found this interesting notebook How to train a new language model from scratch using Transformers and Tokenizers 3 and I would be interested to know if there is one that uses tensorflow instead. I cannot have pytorch on my machine unfortunately.
Is there a huggingface example notebook that would help me do that?
Thanks!

Context:
Hi there!
This might be what you’re after: transformers/examples/tensorflow/language-modeling at master · huggingface/transformers · GitHub 6
You’d use the run_clm script for a GPT-2 like model, and the run_mlm script for a BERT-like model.
EDIT: If you’re able to use docker on your machine, you could also use a huggingface image 2 to run that notebook you linked to.


`serving` signature in TensorFlow Serving blogpost

Question:
Hi everyone!
I am currently working through @jplu 's blogpost 3 on serving a HuggingFace model with TF-Serving, in which he overwrites the model’s serving method to change the signature of the traced graph input to accept embeddings.
That, in turn, led me to discover that this serving signature is part of all TF models (Models — transformers 4.7.0 documentation 3)
Can someone explain to me how exactly this serving method is used by the model server? I can’t find it referenced in the rest of the tutorial and I wasn’t succesful in finding my way around the codebase.
Is that redefined signature used at all in the tutorial? I might be mistaken, but it seems to me that the requests (both REST and gRPC) to the TF-server use the output of the tokenizer, not those of an embedding layer.

Context: cc @Rocketknight1


Question: I fine-tuned a  pre-trained model(wav2vec) in hugging face using the transformers library and converted it from PyTorch to Tensorflow. I want to load this fine-tuned model in Tensorlfow but I can’t seem to find any tutorials showcasing how to. Any help would be appreciated.

Context:
from transformers import TFWav2Vec2ForXxx

TFWav2Vec2ForXxx.from_pretrained(model_name_or_path)

where model_name_or_path is the folder where you stored that model, or the name of the repo you pushed it to on the Hugging Face Hub.


Understanding zero-shot classification in one-shot ;-)

Question:
Hello there!
I LOVE the zero-shot classification API. When you think about it, it could also be used as search engine (find the documents that are related to a particular topic).
My question is: conceptually, how does it work? I saw the original post New pipeline for zero-shot text classification 11 but I cannot find anything that explains the model under the hood.
My understanding is that

the API gets the embedding for the input sequence
the API computes the embedding for the possible labels
a similarity measure is computed between the input sequence and each of the labels and - via a softmax - a probability over the topics is then given

Is that correct?
Thanks and keep up the good work!

Context:
hey @olaffson, conceptually i believe the zero-shot classification pipeline works by adapting a task like natural language inference, where the language model is provided with a “template” like
"<some text you want to classify>. This text is about <MASK>"

and the model fills in the most probable label given the context. joe discusses this in this section of his blog post 12 and you can find one of the underlying zero-shot models here 13


Question:
This might bit a tricky question (because summarization is difficult), but consider the example shown in the documentation
summarizer("An apple a day, keeps the doctor away", min_length=5, max_length=20)
Your max_length is set to 20, but you input_length is only 13. You might consider decreasing max_length manually, e.g. summarizer('...', max_length=50)
Out[49]: [{'summary_text': 'apple a day keeps the doctor away from the doctor.'}]

I find the summarized text a bit… interesting  Granted, this was a very short example. But let’s consider a more real-life example taken from the NYT:

mytext = ‘’’
The highly contagious Delta variant is now responsible for almost all new Covid-19 cases in the United States, and cases are rising rapidly. For the first time since February, there were more than 100,000 confirmed cases on Tuesday, the same day the Centers for Disease Control and Prevention recommended that vaccinated people should resume wearing masks in public indoor spaces in communities where the virus is surging.
That updated guidance was based in part on a new internal report that cited evidence that vaccinated people experiencing breakthrough infections of the Delta variant, which remain infrequent, may be as capable of spreading the virus as infected unvaccinated people.
Several studies, including ones referenced in the C.D.C.’s presentation, have shown that vaccines remain effective against the Delta variant, particularly against hospitalization and death. That has held true in the real world: About 97 percent of those recently hospitalized by the virus were unvaccinated, the C.D.C. said. But in counties where vaccination rates are low, cases are rising fast, and deaths are also on the rise.

summarizer(mytext, min_length=5, max_length=20)
Out[51]: [{'summary_text': 'the highly contagious variant is now responsible for almost all new cases in the united states '}]

But… this is just the first sentence of the whole paragraph almost verbatim! What do you think we can do to improve the output of the summarization pipeline? Can summarization be trained?
Thanks!

Context:
A funny result from NLP work on summarization is that the first sentence of a news article usually turns out to be a pretty challenging baseline to beat  This isn’t necessarily a bad thing—it’s great for us as human readers that news writers do this!
Summarization is an active area of research, with both supervised and unsupervised training approaches. If you check out NAACL 2021, the most recent NLP conference (the very latest, ACL 2021, starts today), there are twenty-seven papers about summarization 7!


Question:
Hello there,
I came across this very interesting post (Sentence Transformers in the Hugging Face Hub 14)  that essentially shows a way to extract the embeddings for a given word or sentence
from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]

model = SentenceTransformer('sentence-transformers/paraphrase-MiniLM-L6-v2')
embeddings = model.encode(sentences)
print(embeddings)

Just a few questions if someone has a few moments:

how are these embeddings different from contextual embeddings that I would get with distilbert and other transformer models?
More importantly, once I have the embeddings I can simply compute a cosine similarity metrics with other sentences to cluster by similarity. If so, what is the need of an API as described here Sentence Transformers in the Hugging Face Hub 14. Am I missing something more subtle here?

Thanks!!

Context:
hey @olaffson, as described in the sentencebert paper 3 uses a siamese network structure to learn the sentence embeddings.

Screen Shot 2021-07-28 at 20.20.542200×1112 179 KB

in general, this approach gives higher-quality embeddings than those you’d get from distilbert etc and you can find a nice performance chart here 5
regarding your second question, i’m not sure which api you’re referring to exactly in the blog post (which is mostly about the integration of sentence-transformers with the hugging face hub). but indeed, once you have the embeddings you can compute metrics / cluster using whatever tools you wish


How can state-of-the-art classifiers be so wrong?

Question:
Hello there! Sorry for the provocative question but consider this simple example
from transformers import pipeline

classifier = pipeline(task = 'sentiment-analysis')

classifier('this is so good!')
Out[4]: [{'label': 'POSITIVE', 'score': 0.9998476505279541}]

classifier('this is so gooood!')
Out[5]: [{'label': 'NEGATIVE', 'score': 0.9922532439231873}]

How can gooood be treated as negative with a very high confidence score? How can I fix this behavior?
Thanks!

Context:
The problem you are encountering has little to do with whether the language model is state-of-the-art, but rather the language dialect(s) used to train the model versus those dialects used at inference time.
Most likely you are using a model trained on standard English, such as the text that appears in Wikipedia, whereas your second query uses the slang term ‘gooood’ which is likely outside the vocabulary used during training.    I suspect that you would encounter similar issues with ‘goooooood’, ‘soooo gooood’, vulgarities, acronyms (e.g.: lol, WTF) or words whose slang definition may differ from standard English (e.g.: bad, fly).
Dialect can be an important issue when dealing with casual speech, tweets, medical literature, clinical records, patents or other domains that use specialized language variants.
Possible approaches:

Select a pre-trained model trained on the language dialect of interest.  For example, cardiffnlp/twitter-roberta-base-sentiment is a variant of RoBERTa trained on tweets.  The huggingface model library also includes models trained on Reddit text.
Fine-tune an existing pre-trained model through supplementary training using examples of the language dialect of interest.  Huggingface includes datasets related to Reddit and tweets that may be of use for this task.
Using an existing base language model trained on the dialect of interest (e.g.: Reddit), and a classification head and train using a standard sentiment analysis training dataset


How to ignore attributes of TrainingArguments?

Question:
I am trying to subclass Huggingface’s Trainer and overwrite it with custom optimizer and lr_scheduler
from transformers import TrainingArguments

training_args = TrainingArguments(
    "rugpt3-headers",
    num_train_epochs=3,
    per_device_train_batch_size=3,
    evaluation_strategy="epoch",
    logging_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
)

from transformers import Trainer, AdamW
from torch.optim.lr_scheduler import MultiStepLR

class MyTrainer(Trainer):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def create_optimizer(self):
      self.optimizer = AdamW(model.parameters(),lr = 1e-7)

    def create_scheduler(self, num_training_steps):
      self.lr_scheduler = MultiStepLR(self.optimizer, milestones=[1,3], gamma=0.5)

trainer = MyTrainer(
    model=model,
    args=training_args,
    train_dataset=small_train_dataset,
    eval_dataset=small_eval_dataset,
    compute_metrics=compute_metrics,
    callbacks=[EarlyStoppingCallback(early_stopping_patience=3)],
)
trainer.train()

Even though I did not specify learning_rate in TrainingArguments , it has a default value of 5e-7 . My attempt to overwrite the optimizer and scheduler is not successful because of that. After my training was completed, I used tensorboard to check which learning rate was used and it is still 5e-07 even though I thought I overwrote it.
How to overcome this issue? I want it to set up my learning rate based on
def create_optimizer(self):
     self.optimizer = AdamW(model.parameters(),lr = 1e-7)

def create_scheduler(self, num_training_steps):
     self.lr_scheduler = MultiStepLR(self.optimizer, milestones=[1,3], gamma=0.5)

I wrote above and ignore the default learning rate of TrainingArguments

Context:
Are you sure you are looking at the right logs? The learning_rate from the TrainingArguments is only used once in the create_optimizer method and you overrode it, so there is no reason for it to be used.
Also, make sure you are on the latest version of Transformers, just in case.


Mismatched target and input size for BCE using “multi_label_classification”

Question:
I am trying to build a multi-label, multi-class classification model. Any input text can have zero or more labels, up to 11 possible classes.
I have been trying to use the problem_type="multi_label_classification" and everything looks OK, but I get ValueError: Target size (torch.Size([16, 11])) must be the same as input size (torch.Size([16, 2])) when it tries to calculate the binary_cross_entropy_with_logits
I presume my data is in the wrong shape somehow, but I can’t see where exactly. Any suggestions?
transformers==4.8.2
Here is a minimal example:
import torch
from torch.utils.data.dataset import Dataset

from transformers import AutoTokenizer, AutoModelForSequenceClassification

# Example data.
# In reality, the strings are usually longer and there are 11 possible classes
texts = [
    "This is the first sentence.",
    "This is the second sentence.",
    "This is another sentence.",
    "Finally, the last sentence.",
]

labels = [
    [0, 0, 0, 0, 1],
    [1, 0, 0, 0, 0],
    [0, 1, 1, 0, 0],
    [0, 0, 0, 0, 0],
]

train_texts = texts[:2]
train_labels = labels[:2]

eval_texts = texts[2:]
eval_labels = labels[2:]

tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)

train_encodings = tokenizer(train_texts, padding="max_length", truncation=True, max_length=512)
eval_encodings = tokenizer(eval_texts, padding="max_length", truncation=True, max_length=512)

class TextClassifierDataset(Dataset):
    def __init__(self, encodings, labels):
        self.encodings = encodings
        self.labels = labels

    def __len__(self):
        return len(self.labels)

    def __getitem__(self, idx):
        item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
        item["labels"] = torch.tensor(self.labels[idx])
        return item

train_dataset = TextClassifierDataset(train_encodings, train_labels)
eval_dataset = TextClassifierDataset(eval_encodings, eval_labels)

model = AutoModelForSequenceClassification.from_pretrained(
    "bert-base-uncased",
    problem_type="multi_label_classification",
)

training_arguments = TrainingArguments(
    output_dir=".",
    evaluation_strategy="epoch",
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=1,
)

trainer = Trainer(
    model=model,
    args=training_arguments,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
)

trainer.train()

# long traceback, but here is the important bit...
~/python3.8/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    711
    712     def forward(self, input: Tensor, target: Tensor) -> Tensor:
--> 713         return F.binary_cross_entropy_with_logits(input, target,
    714                                                   self.weight,
    715                                                   pos_weight=self.pos_weight,

~/python3.8/site-packages/torch/nn/functional.py in binary_cross_entropy_with_logits(input, target, weight, size_average, reduce, reduction, pos_weight)
   2956
   2957     if not (target.size() == input.size()):
-> 2958         raise ValueError("Target size ({}) must be the same as input size ({})".format(target.size(), input.size()))
   2959
   2960     return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)

ValueError: Target size (torch.Size([16, 11])) must be the same as input size (torch.Size([16, 2]))

Context: When defining your model, you did not specify the number of labels (with num_labels=xxx).


Text classification on small dataset (8K)

Question:
Hello,
I’m trying to find a good architecture for a model which has to do text classification. The domain is a chat-bot doing a help-desk. Its goal is to book appointments for customers who need some machines to be repaired. The current model only has to classify single utterances in one of the 20 categories.
I have around 8K examples in my data set. I’m wondering if there is some recommended type of architecture/model based on tranformers for this type of model.
I tried a model with a frozen DistilBERT layer followed by a fully connected layer, before a classification layer.
So basically the same architecture than DistilBertForSequenceClassification presentend here:
https://huggingface.co/transformers/_modules/transformers/models/distilbert/modeling_distilbert.html#DistilBertForSequenceClassification
but with the DistilBERT layer frozen.
But the results were so so. So I’m thinking about two things:

Is there something more appropriate than DistilBERT for my set-up?
Should I maybe keep only some layers of DistilBERT frozen and not all of them?

If anyone has a suggestion, I would be glad to hear about it.
Thank you in advance!

Context:
In my experience (I worked with BERT and RoBERTa), not updating the transformer model parameters during fine-tuning resulted in lower accuracy and slower decrease in the loss value. This might mean that the fully connected layer alone is not enough to model the task at hand. I suggest updating the parameters of the DistilBert model as well, which is what fine-tuning is for.
I should also note that freezing the first 6 layers of BERT-base did not decrease the accuracy of the model significantly, in my case.


Adding Preprocessing to Hosted Inference API

Question: My question answering model requires an extra step of preprocessing the input and the output. How can I add those preprocessing scripts to work with Hosted Inference API (Widget on the website, too)?

Context:
hey @yigitbekir, as far as i know the inference api does not support custom pre- / post-processing logic, but you could easily include these steps within a dedicated web / streamlit application
if you need a custom widget, you can propose one on the huggingface_hub library here: GitHub - huggingface/huggingface_hub: Client library to download and publish models and other files on the huggingface.co hub 4


Segmentation fault (Core dumped) with datasets

Question:
While trying to download a large dataset(~100GB), without streaming mode like this:
from datasets import load_dataset
mc4_dataset = load_dataset("mc4", "hi")

I first got an error:
multiprocessing.pool.RemoteTraceback:
ConnectionError: Couldn't reach https://huggingface.co/datasets/allenai/c4/resolve/1ddc917116b730e1859edef32896ec5c16be51d0/multilingual/c4-hi.tfrecord-00709-of-01024.json.gz

On running the same 2 line script again, the downloads resumed but then crashed with a single line message Segmentation fault (core dumped).
Rerun of same script again gives the following message:
Downloading and preparing dataset mc4/hi (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/user/.cache/huggingface/datasets/mc4/hi/0.0.0/a2bc8f2c4d913b8b16fac4d1a63d673fa6cb22859520dcac7f193feec1f00cae...
Segmentation fault (core dumped)

Any suggestions on how to debug this error?

There’s a lock file in ~/.cache/huggingface/datasets/

~/.cache/huggingface/datasets/mc4/hi/ contains hash.“incomplete” directory, which is empty

~/.cache/huggingface/datasets/downloads/ contains a lot of hash-id files and locks.

In this state, is there anything we could do to repair the state and continue without having to re-download entire dataset from scratch? Also in worst case, is there anydatasets alternative of rm -r purge, as not only mc4 but I think the lock and the downloads dir contents will need to go away.

Context: @lhoestq have you seen this before by any chance?


Question:
Hi! I noticed that in order to get reproducible results across runs when carrying out sequence classification, the type conversion for the labels from e.g. string  to int has to be done consistently, i.e. every class has to be assigned to the same int across runs.
Say I have 5 classes: ‘a’, ‘b’, ‘c’, ‘d’, ‘e’, and they get mapped to 0, 1, 2, 3, 4, respectively. If I run the exact same experiment but just changing this mapping to, say, 4, 3, 2, 1, 0, results won’t be the same. I’m assuming the distribution is not known (i.e. dataset can be imbalanced).
In fact, this is also mentioned in a comment in the run_glue.py example (here).
My question is simply: why? Why does this matter when computing cross entropy loss?
Thanks

Context: Even if you use the same seed, you will not get the same results because the last layer of the model, the classification head, will be initialized randomly the same way if you label are mapped to 0, 1, 2, 3, 4, or another permutation. That means that the initial weight for ‘a’ will be different if ‘a’ is mapped to 0 or 4. Then going from there, you will get different losses, so different gradients and different updates, and will end up with a completely different model.


Calculating perplexity from hidden_states

Question:
Hi all, I am trying to run ray tune for my masked language model, I want to find the best hyperparameters that will minimize perplexity of the model. I am not able to figure out how to calculate perplexity using the model’s hidden_states, which is returned as EvalPrediction.predictions. Any help will be greatly appreciated. Thank you!
following code snippet show the training.
model_checkpoint = "distilroberta-base"
model = AutoModelForMaskedLM.from_pretrained(model_checkpoint).to('cuda')

def compute_custom_metric(eval_pred):
    # following will print (3387, 32, 50265) (beach_size * max_output_len * vocal_size)
    print(eval_pred.predictions.shape)
    # following will print (3387, 32) (batch_size * max_output_len)
    print(eval_pred.label_ids.shape)
    return {'custom_metric': 0}

trainer = Trainer(
    model = model,
    args = training_args,
    train_dataset = train,
    eval_dataset = validation,
    tokenizer = tokenizer,
    data_collator = data_collator,
    compute_metrics = compute_custom_metric
)

trainer.evaluate()

Context:
After extensive searching finally found the solution. Below function calculates perplexity after every epoch. FYI I have also added my training arguments.
def compute_custom_metric(pred):
    logits = torch.from_numpy(pred.predictions)
    labels = torch.from_numpy(pred.label_ids)
    loss = F.cross_entropy(logits.view(-1, tokenizer.vocab_size), labels.view(-1))
    return {'perplexity': math.exp(loss), 'calculated_loss': loss}

training_args = TrainingArguments(
    output_dir='./some_results',
    evaluation_strategy = "epoch",
    num_train_epochs=3,
    learning_rate=1e-5,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    warmup_steps=500,
    weight_decay=0.01,
    logging_dir='./some_logs',
    logging_steps=logging_steps,
    seed=seed,
    fp16=True,
    eval_accumulation_steps=50,
)


Passing Trainer state as an artifact in kfp.v2 pipeline

Question:
I’m trying to create a kfp pipeline fine-tuning BERT on GCP’s VertexAI. I can save a model as an artifact, but have troubles saving the Trainer (Trainer’s state), as I would like to split eval and testing into two separate pipeline components and to achieve that - I need to reconstruct the Trainer in the latter.
I have, created trainer_artifact and would like to save_state() into tainer_artifact.path. However, save_state() does not accept arguments and saves to model’s directory by default. How can I save trainer as an artifact in this situation? Maybe there is a workaround for it, eg. recreating it in the next step?
I attach my code.
@component(
    packages_to_install = [
        "pandas",
        "datasets",
        "transformers"
    ],
)
def fine_tune_modell(
    small_train_dataset: Input[Dataset],
    small_eval_dataset: Input[Dataset],
#     full_train_dataset: Input[Dataset],
#     full_eval_dataset: Input[Dataset],
    model_artifact: Output[Model],
    trainer_artifact: Output[Artifact]
):

    import pandas as pd
    import numpy as np
    import datasets
    from transformers import AutoModelForSequenceClassification
    from transformers import TrainingArguments
    from transformers import Trainer
    from datasets import load_metric

    # create model
    model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2)

    # load data
    train_data = datasets.load_from_disk(small_train_dataset.path)
    eval_data = datasets.load_from_disk(small_eval_dataset.path)

    metric = load_metric("accuracy")

    def compute_metrics(eval_pred):
        logits, labels = eval_pred
        predictions = np.argmax(logits, axis=-1)
        return metric.compute(predictions=predictions, references=labels)

    training_args = TrainingArguments(
        output_dir="test_trainer",
        evaluation_strategy="epoch",
        per_device_train_batch_size=8,
        per_device_eval_batch_size=8,
        num_train_epochs=3,
        seed=0,)

    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_data,
        eval_dataset=eval_data,
        compute_metrics=compute_metrics
    )

    train_output = trainer.train()

    model_artifact.metadata["train_output"] = train_output
    model_artifact.metadata["framework"] = "Pytorch"

    # How to save trainer?

    model.save_model(model_artifact.path)```

Context: Maybe you can move the state file created?


How can I use keyBERT using huggingface inference API?

Question: Hi guys, basically I want to use KeyBERT using the huggingface inference API. can someone please tell me how can I do that? How can I upload KeyBERT on HF inference API?

Context:
Hi @akarshghale!
We have a guide 22 on how to integrate a new library into the HF Inference API which should be added to the main site the coming week. Feel free to open an issue if you have any questions!


Question:
Hey everone. I’m trying to install transformers and datasets package using conda.
I installed pytorch using conda, and I’m using miniconda with python version 3.7. My environment is also using python 3.7.
Installation of transformers using the command

conda install -c huggingface transformers

works, but when testing the installation I get

from transformers import pipeline
Traceback (most recent call last):
File “”, line 1, in
File “/home/nfs/tjviering/envs/torch3/lib/python3.7/site-packages/transformers-4.4.2-py3.8.egg/transformers/init.py”, line 43, in
File “/home/nfs/tjviering/envs/torch3/lib/python3.7/site-packages/transformers-4.4.2-py3.8.egg/transformers/dependency_versions_check.py”, line 36, in
File “/home/nfs/tjviering/envs/torch3/lib/python3.7/site-packages/transformers-4.4.2-py3.8.egg/transformers/file_utils.py”, line 56, in
ModuleNotFoundError: No module named ‘importlib_metadata’

conda install importlib-metadata
File “”, line 1
conda install importlib-metadata
^
SyntaxError: invalid syntax

I thought possibly the following would solve it

conda install importlib-metadata

But when testing the installation I still get trouble:

from transformers import pipeline
Traceback (most recent call last):
File “”, line 1, in
File “/home/nfs/tjviering/envs/torch3/lib/python3.7/site-packages/transformers-4.4.2-py3.8.egg/transformers/init.py”, line 2310, in getattr
File “/home/nfs/tjviering/envs/torch3/lib/python3.7/site-packages/transformers-4.4.2-py3.8.egg/transformers/file_utils.py”, line 1660, in getattr
File “/home/nfs/tjviering/envs/torch3/lib/python3.7/site-packages/transformers-4.4.2-py3.8.egg/transformers/init.py”, line 2304, in _get_module
File “/home/nfs/tjviering/envs/torch3/lib/python3.7/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “/home/nfs/tjviering/envs/torch3/lib/python3.7/site-packages/transformers-4.4.2-py3.8.egg/transformers/pipelines/init.py”, line 24, in
File “/home/nfs/tjviering/envs/torch3/lib/python3.7/site-packages/transformers-4.4.2-py3.8.egg/transformers/modelcard.py”, line 31, in
File “/home/nfs/tjviering/envs/torch3/lib/python3.7/site-packages/transformers-4.4.2-py3.8.egg/transformers/models/init.py”, line 19, in
File “/home/nfs/tjviering/envs/torch3/lib/python3.7/site-packages/transformers-4.4.2-py3.8.egg/transformers/models/mt5/init.py”, line 36, in
File “/home/nfs/tjviering/envs/torch3/lib/python3.7/site-packages/transformers-4.4.2-py3.8.egg/transformers/models/t5/tokenization_t5_fast.py”, line 23, in
File “/home/nfs/tjviering/envs/torch3/lib/python3.7/site-packages/transformers-4.4.2-py3.8.egg/transformers/tokenization_utils_fast.py”, line 25, in
File “/home/nfs/tjviering/envs/torch3/lib/python3.7/site-packages/tokenizers/init.py”, line 79, in
from .tokenizers import (
ImportError: /lib64/libm.so.6: version `GLIBC_2.29’ not found (required by /home/nfs/tjviering/envs/torch3/lib/python3.7/site-packages/tokenizers/tokenizers.cpython-37m-x86_64-linux-gnu.so)

Any help would be greatly appreciated!

Context:
For anyone wondering how to fix this issue, I found that when using the channel conda-forge there is no issue at all. So the fix:
conda install -c conda-forge transformers
conda install importlib-metadata


How to concatenate the word embedding for special tokens and words

Question:
I tried to add an extra dimension to the Huggingface pre-trained BERT tokenizer. The extra column represents the extra label. For example, if the original embedding of the word “dog” was [1,1,1,1,1,1,1], then I might add a special column with index 2 to represent ‘noun’. Thus, the new embedding becomes [1,1,1,1,1,1,1,2]. Then, I will feed the new input [1,1,1,1,1,1,1,2] into the Bert model. How can I do this in Huggingface?
There is something called tokenizer.add_special_tokens which extends the original vocabulary with new tokens. However, I want to concatenate the embedding of the original vocabulary with the embedding of the tokenizer. For example, I want the Bert model to understand that Dog is a noun by connecting the embedding of dog to the embedding of noun. Should I even change the input word embedding of a pre-trained model? Or should I somehow enhance the attention on “dog” and “noun” in the middle layer?
Here is the example of using tokenizer.add_special_tokens

tokenizer = GPT2Tokenizer.from_pretrained(‘gpt2’)
model = GPT2Model.from_pretrained(‘gpt2’)
special_tokens_dict = {‘cls_token’: ‘’}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
print(‘We have added’, num_added_toks, ‘tokens’)
model.resize_token_embeddings(len(tokenizer))
assert tokenizer.cls_token == ‘’

Context: I found solution here : How to use additional input features for NER? 101


Question Answering for generating long answers

Question:
Hello,
Most of the Question Answering model answers are small like 3-4 words. What if I want to build a model which could elaborate my answer rather than a factoid based answer.
Here I have context and a question and I want it to answer a long answer sequence based on the context given.
Is there some method I can control the process to generate longer sequence answer.
Any model, or any idea how to approach this problem that may be helpful.
Thanks.

Context: hey @theainerd it sounds like you’re looking for models that are trained for long form question answering. here’s a great article by Yacine Jernite that shows how you can train such models (on the ELI5 dataset in this case): Long_Form_Question_Answering_with_ELI5_and_Wikipedia 46


Question:
Hi all!
I’m trying to perform continued pre-training on T5. Basically, I’m doing domain adaptation to new data before fine-tuning, and I want to make sure that I’m preprocessing data as similarly as possible to how T5 does it during pre-training (i.e., randomly corrupt 15% of tokens, pack sequences together for training examples of length 512, mask contiguous corrupted spans, reconstruct corrupted spans in target sequence).
I know that there’s a TensorFlow implementation of this from Google (text-to-text-transfer-transformer/preprocessors.py at d72bd861de901d3269f45ec33c6ca6acd18b10b8 · google-research/text-to-text-transfer-transformer · GitHub 15), but has anyone implemented this in PyTorch for use with huggingface models? I’m hoping to make something like the linked span_corruption function work with torch tensors composed of tokenized text, rather than a tf.Dataset object.
Thanks!

Context: hey @amueller i haven’t tried this myself but it seems that pretraining T5 can be done by using sentinel tokens in the tokenizer as described here: T5 — transformers 4.5.0.dev0 documentation 51


T5 cross-attention - inconsistent results

Question:
Environment info
Python version: 3.7.10
PyTorch version (GPU?): '1.7.1+cu110' (True)
Transformer version: '4.5.0.dev0'

Details
I am trying to use the cross-attention from the T5 model for paraphrasing. The idea is to map the input sentence and output generated sequence based on the attention. But the first results I got are very strange.
I generated an example with the following code:
from transformers import T5ForConditionalGeneration, T5Tokenizer
import torch

pretrained_model = "ramsrigouthamg/t5_paraphraser"

model = T5ForConditionalGeneration.from_pretrained(pretrained_model,
                                                   output_attentions=True,
                                                   output_scores=True)

translated_sentence = "I like drinking Fanta and Cola."

text = "paraphrase: " + translated_sentence + " </s>"

encoding = tokenizer.encode_plus(text,
                                 pad_to_max_length=True,
                                 return_tensors="pt")
input_ids, attention_masks = encoding["input_ids"], encoding["attention_mask"].

Then, I gave a look to the cross attention for each generated token by selecting the last layer of the encoder and the first head.

beam_outputs = model.generate(
    input_ids=input_ids,
    attention_mask=attention_masks,
    do_sample=True,
    max_length=256,
    top_k=100,
    top_p=0.95,
    num_return_sequences=1,
    output_attentions = True,
    output_scores=True,
    return_dict_in_generate=True
)

sentence_id = 0
print("Input phrase: ", tokenizer.decode(encoding.input_ids[0],
                                      skip_special_tokens=False,
                                      clean_up_tokenization_spaces=False))
print("Predicted phrase: ", tokenizer.decode(beam_outputs.sequences[sentence_id],
                                      skip_special_tokens=True,
                                      clean_up_tokenization_spaces=True))
for out in range(len(beam_outputs.sequences[sentence_id])-1):
    print(
        "\nPredicted word: ",
        tokenizer.decode(beam_outputs.sequences[sentence_id][out],
                         skip_special_tokens=True,
                         clean_up_tokenization_spaces=True))
    att = torch.stack(beam_outputs.cross_attentions[out])
    # Last layer of the encoder
    att = att[-1]

   # First batch and first head
    att = att[0, 0, :, :]
    att = torch.squeeze(att)

    idx = torch.argsort(att)
    idx = idx.cpu().numpy()

    print("Input words ordered by attention: ")
    for i in range(min(5, len(idx))):
        token_smallest_attention =tokenizer.decode(encoding.input_ids[0][idx[i]],
                                      skip_special_tokens=True,
                                      clean_up_tokenization_spaces=True)
        token_largest_attention =tokenizer.decode(encoding.input_ids[0][idx[-(1+i)]],
                                      skip_special_tokens=True,
                                      clean_up_tokenization_spaces=True)
        print(f"{i+1}: Largest attention: {token_largest_attention} | smallest attention:{token_smallest_attention}")

The attention scores are sorted and each generated token is associated with the input with the highest attention (5 values) and with the lowest attentions (also 5 values).
Input phrase:  paraphrase: I like drinking Fanta and Cola.</s>
Predicted phrase:  I like to drink Fanta and Cola.

Predicted word:  <pad>
Input words ordered by attention:
1: Largest attention: I | smallest attention:Col
2: Largest attention: like | smallest attention:a
3: Largest attention: : | smallest attention:t
4: Largest attention: para | smallest attention:a
5: Largest attention: . | smallest attention:Fan

Predicted word:  I
Input words ordered by attention:
1: Largest attention: phrase | smallest attention:t
2: Largest attention: </s> | smallest attention:a
3: Largest attention: para | smallest attention:a
4: Largest attention: : | smallest attention:Col
5: Largest attention: like | smallest attention:and

Predicted word:  like
Input words ordered by attention:
1: Largest attention: Fan | smallest attention:I
2: Largest attention: Col | smallest attention:.
3: Largest attention: phrase | smallest attention:like
4: Largest attention: a | smallest attention:para
5: Largest attention: </s> | smallest attention:a

Expecting results
I was expecting an almost one-to-one mapping as the paraphrase is very close to the input but it is not the case. The model gives good paraphrases.  Do you think that I made some errors in the interpretation of the cross-attention object?

Thank you for your help!
Hopefully, it is something simple that I am missing.

Context:
We found the problem: the shape of the attention matrix was not correct.
Here are the modifications
# From T5 documentation
# Initial shape: Tuple (one element for each generated token) of tuples (one element for
# each layer of the decoder) of torch.FloatTensor of shape
# (batch_size, num_heads, generated_length, sequence_length).

# combine all cross attention into one tensor
x = [torch.stack(beam_outputs['cross_attentions'][i]) for i in range(len(beam_outputs['cross_attentions']))]
x = torch.stack(x)
# Shape: (nb_generated, nb_layer, (batch_size, num_heads, generated_length, sequence_length))
print(x.shape)

x = x.transpose(1,0)
# (nb_layer, nb_generated, batch_size, num_heads, generated_length, sequence_length)
print(x.shape)

x = x.transpose(1,3)
# (nb_layer, num_heads, batch_size, nb_generated, generated_length, sequence_length)
print(x.shape)

x = torch.squeeze(x, 4)
# (nb_layer, num_heads, batch_size, nb_generated, sequence_length)
print(x.shape)

x = x.transpose(2, 1)
# (nb_layer, batch_size, num_heads, nb_generated, sequence_length)
print(x.shape)
cross_attentions = x

We can compute the encoder/decoder tokens:
encoder_text = tokenizer.convert_ids_to_tokens(input_ids[0])
decoder_text = tokenizer.convert_ids_to_tokens(beam_outputs.sequences[0])

encoder_tokens=np.array(encoder_text)
decoder_tokens=np.array(decoder_text[:-1])

The initial sentence is: In Belgium, summers can be very dry and the heat is burning. and the paraphrase: 'In Belgium the summers can be very dry and the heat is burning.
The associated tokens are:
['▁para', 'phrase', ':', '▁In', '▁Belgium', ',', '▁summer', 's', '▁can', '▁be', '▁very', '▁dry', '▁and', '▁the', '▁heat', '▁is', '▁burning', '.', '</s>']
['<pad>', '▁In', '▁Belgium', '▁the', '▁summer', 's', '▁can', '▁be', '▁very', '▁dry', '▁and', '▁the', '▁heat', '▁is', '▁burning', '.', '</s>']

Here are the attention score sorted by importance:
layer = 0
head = 0 # choose head to analyze
att = cross_attentions[layer, 0, head]
att.shape

for i in range(att.shape[0]):
    idx = np.argsort(att[i].cpu().numpy())[::-1][:6]
    print(f"Predicted token: {decoder_tokens[i]}, input related tokens: {encoder_tokens[idx]}")

Here are some results:
Predicted token: ▁In, input related tokens: ['</s>' '▁In' 'phrase' '.' ':' '▁Belgium']
Predicted token: ▁Belgium, input related tokens: ['▁Belgium' '</s>' 'phrase' ':' '▁para' '▁summer']
Predicted token: ▁the, input related tokens: ['</s>' 'phrase' ',' '.' '▁the' ':']
Predicted token: ▁summer, input related tokens: ['▁Belgium' '</s>' 'phrase' '▁In' 's' '▁summer']

We solved this problem with the help of the decoder encoder version of bertviz 5


Question:
As I understand effective batch size = batch grad accumulation * batch size.
Firstly
Is this the only adjustment that needs to be done, nothing for the learning rate?
Secondly how much can you accumulate. Say I want a batch size of 256, and can only fit a batch size of 4, can I really accumulate gradients over 64 batches or is that way too much and never done in practice?

Context:
From the pure math side, there is no limit to how many times you can accumulate gradients.

\frac{\partial \mathcal L}{\partial \theta} = \sum_{accumulation} \sum_{batch\_size} \frac{\partial \mathcal L}{\partial θ}(x_i, y_i) = \sum_{larger\_batch\_size} \frac{\partial \mathcal L}{\partial θ}(x_i, y_i)

I guess, there would be more numerical accuracy / stability issues. Maybe the results would be a bit different from the “true” batch_size=256, but not too much.
I don’t think you need to adjust the learning rate, use the same value you would use for batch_size = 256.
Also remember that it is deep learning — even if you are sure everything is going to be alright, it may be quite different in practice. Just try it out and see.


XLMR-large not converging on Paws-X paraphrase dataset but mbert does

Question:
Hey everyone,
I tried training Mbert and Xlmr-large on the paws-x English paraphrase detection dataset and looks like Xlmr-large is not converging, while Mbert does. I’ve tried tweaking the hyperparameters for the Xlmr-large but that doesn’t seem to help as well.
Attaching train stats for both models below(Note: I’m evaluating every 100 steps)
image738×755 28.7 KB
image741×689 27.1 KB
I’ve changed the hf run_glue colab example to reproduce this behavior here 1. For hyperparameters, I’m following the hyperparameters used in xtreme 2 paper where they report better results for xlmr-large compared to mbert on paws-x
Would appreciate it if someone took a quick look and have any suggestions.
Thanks

Context:
I revisited this with the latest hf and tried it with fp16 and it seems to work now.
Also had a similar issue with roberta-large models on xnli and paws. Tried with fp16 and fp32 and every time, one of them worked.


Train and inference wav2vec2 using a language model

Question: Hi everyone. Is it possible to use a language model built with KenLM for train and inference steps of Wav2Vec2 model? If so, please share any resource on this. Thanks in advance

Context:
Hey,
We don’t have a lot of code that support LM decoding for Wav2Vec2 sadly.
This notebook: huggingface_notebook/xlsr_gpt.ipynb at main · voidful/huggingface_notebook · GitHub 46 might help


Question:
Chai 3 is an open-source platform which allows you to develop and deploy chat AIs.

Using chai_py you can speak with your AI in just a few lines of code.

The key to using a HuggingFace model for your chat AI on chai is to call their endpoint. You can read a little more about this in the chai docs 2.

Screenshot 2021-04-28 at 07.49.122336×1692 271 KB

Context: Contact us on hello@chai.ml or join our Discord channel if you have any questions/feedback! We’d love to hear from you.


Scaling up BERT-like model Inference on modern CPU - Part 1

Question:
Hi community,
I have come through the nice article by @mfuntowicz : Hugging Face – The AI community building the future. 13
It sounds really interesting how easily you can benchmark your BERT transformer model with CLI and  Facebook AI & Research’s Hydra configuration library 8.
Is it possible however to easily test it on cloud services as AWS and how to deploy it?
Thanks!

Context:
Hey @Matthieu,
Thanks for reading and posting here .
Indeed, everything in the blog was run in AWS (c5.metal) instance(s).
The way I’m currently using it:

git clone https://github.com/huggingface/tune
cd tune
pip install -r requirements.txt
export PYTHONPATH=src
python src/main.py --multirun backend=pytorch batch=1 sequence_length=128,256,512

The overall framework is quite new and I’ll be improving the UX in the coming days, sorry for this user experience
Morgan


Run_mlm.py using –sharded_ddp “zero_dp_3 offload” gives AssertionError

Question:
I’m trying to run the following on a single, multi-gpu machine that has 8 GPUs:
python -m torch.distributed.launch --nproc_per_node=8 \
run_mlm.py \
--model_name_or_path roberta-base \
--use_fast_tokenizer \
--dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \
--do_train --do_eval \
--num_train_epochs 5 \
--output_dir ./experiments/wikitext \
--fp16 \
--sharded_ddp "zero_dp_3 offload"

This fails with the following AssertionError:

Traceback (most recent call last):   File "run_mlm.py", line 492, in <module>     main()   File "run_mlm.py", line 458, in main     train_result = trainer.train(resume_from_checkpoint=checkpoint)   File "/home/me/ve/lib/python3.6/site-packages/transformers/trainer.py", line 1120, in train     tr_loss += self.training_step(model, inputs)   File "/home/me/ve/lib/python3.6/site-packages/transformers/trainer.py", line 1522, in training_step     loss = self.compute_loss(model, inputs)   File "/home/me/ve/lib/python3.6/site-packages/transformers/trainer.py", line 1556, in compute_loss     outputs = model(**inputs)   File "/home/me/ve/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl     result = self.forward(*input, **kwargs)   File "/home/me/ve/lib/python3.6/site-packages/fairscale/nn/data_parallel/fully_sharded_data_parallel.py", line 902, in forward     self._lazy_init()   File "/home/me/ve/lib/python3.6/site-packages/fairscale/nn/data_parallel/fully_sharded_data_parallel.py", line 739, in _lazy_init     self._init_param_attributes(p)   File "/home/me/ve/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context     return func(*args, **kwargs) File "/home/me/ve/lib/python3.6/site-packages/fairscale/nn/data_parallel/fully_sharded_data_parallel.py", line 796, in _init_param_attributes     assert p._fp32_shard.device == torch.device("cpu") AssertionError

If I omit the “offload” option to --sharded_ddp, it runs with no problems
CUDA 11.0
PyTorch 1.7.1+cu110
Huggingface 4.5.1
Has anyone successfully gotten this to work?  Any help much appreciated!

Context:
Last time I checked, it was blocked by a bug on fairscale side 11, but that yielded a different error message than this one. Will take a look this morning.
In any case solving this first bug will only get you in the second one, so you should use deepspeed for ZeRO DP3 with offload.


Question:
Hi all,
What is the best method advised to do domain adaptation to a transformer model ?
Kind regards,
lematmat

Context: One generic method that can be applied to any encoder is, [1505.07818] Domain-Adversarial Training of Neural Networks 52


Multi-Task dataset with Custom Sampler and Sharding

Question:
The current Huggingface Trainer Supports, a single train_dataset (torch.utils.data.dataset.Dataset). While it makes sense for most of the training setups, there are still some cases where it is convenient to have a list of train_dataset. The trainer can randomly select or follow a specific sampling strategy to select the samples from each of the train_dataset.  An example is attached below for the custom sampling streategy code.
The sampling strategy for each of the  train_dataset (torch.utils.data.dataset.Dataset)  (from multiple train dataset) can be varied by a penalty variable (\alpha). The sample code for a custom multinomial distribution based sampling strategy is below (mentioned in XLM 1 paper),
def multinomial_prob(dataset_len, alpha=.5):
    tot_number_of_sent_in_all_lang = 0
    prob = OrderedDict()
    for k, v in dataset_len.items():
        tot_number_of_sent_in_all_lang += v
    for k, v in dataset_len.items():
        neu = v
        den = tot_number_of_sent_in_all_lang
        p = neu/den
        prob[k] = p

    q = OrderedDict()
    q_den = 0.0
    for k, v in prob.items():
        q_den += (v**alpha)
    sum_ = 0.0
    for k, v in prob.items():
        q[k] =  (v**alpha)/q_den
        sum_ += q[k]
    assert math.fabs(1-sum_) < 1e-5
    return q

def iterator_selection_prob(alpha, train_datasets, logger=None):
    dataset_len = OrderedDict()
    for k, v in train_datasets.items():
        dataset_len[k] = len(v)
    for k, v in dataset_len.items():
        logger.info("Total Number of samples in {} : {}".format(k, v))
    prob = multinomial_prob(dataset_len, alpha=alpha)
    logger.info("Language iterator selection probability.")
    ret_prob_index, ret_prob_list  = [], []
    for k,v in prob.items():
        ret_prob_index.append(k)
        ret_prob_list.append(v)
    for k, v in zip(ret_prob_index, ret_prob_list):
        logger.info("{} : {}".format(k, v))
    return dataset_len, ret_prob_index, ret_prob_list

So I have three questions in general,

How to integrate multiple datasets (or sub-dataset) in the same dataset class?
How to apply custom control on the sampling strategy (let’s just say I want to inject the above sampling strategy in my sub-datasets) into different sub-dataset?
Also in the case of the large tenderized dataset that cannot fit into memory how to handle the sharding using huggingface trainer.

Note:

I am not looking for sample codes. A discussion or pointer to the Hf source library is also highly appreciated. However, sample codes are always best.
I would also like to know if you have seen some other repository implements these feature with/without Hf-library.
Discussion on any topic is highly appreciated.

Context: @sgugger  Do you have any idea on these topics?


Pruning a model embedding matrix for memory efficiency

Question:
Hi, I’m trying to finetune the facebook/mbart-large-50-many-to-many-mmt model for machine translation. Unfortunately, I keep maxing out my GPU memory and even with a batch size of 1 sample with gradient accumulation I cannot get it to work.
I was looking through potential solutions and came across this 9 thread where pruning the embeddings has been suggested as a solution. @sshleifer created an issue for the same here 2 and here 2, but I don’t think it saw any progress.
I’m trying to do this by myself right now, and was wondering if my approach was correct -

Run tokenizer on dataset and get a vocabulary of all unique tokens
Copy all the embeddings associated with the vocabulary and create a new embedding matrix
Replace the embedding matrix in the model with the new one
Map the old vocabulary to their corresponding indices on the new embedding matrix
Run tokenizer again but remap tokens to new embedding matrix before passing them to the model

Does anyone here have any idea if this could work?

Context:
Yes this seems like the right approach.
When you get to step 4/5 you can just make a new Tokenizer.
If you get it working please post the solution here!


Question:
Hello everyone,
I am contemplating converting my trained GPT2 model to JavaScript to go wild. The model has been trained in its PyTorch variant.
I have an idea of how to map it to JavaScript and would like to ask your opinion.
Maybe this is a little to much overkill:

Save PyTorch GPT2 as pretrained.
Load the TensorFlow GPT2 from the file I have just saved. (Believing the binaries are framework agnostic).
Convert the TensorFlow model to TensorflowJS.

Would that work?

Context:
hey @TristanBehrens, your approach sounds sensible (although i don’t know anything about the TF → TFJS step). to load the pytorch model in tf, you’ll just need to add a from_pt=True argument, e.g.
tf_model = TFDistilBertForSequenceClassification.from_pretrained("path/to/pytorch-model", from_pt=True)

an alternative would be to use onnx as the intermediate representation (see here 11 for the export), but your approach seems simpler


Question:
Hey guys
I currently got an insufficient GPU memory error with the config below. Training on 8 x V100 GPUs.
It doesn’t appear imidiately though, but rather non-deterministicly far into the training, which rather points to a memory leak somewhere. Would you have some tips or ideas how to approach this?
Any ideas?
training_args = TrainingArguments(
output_dir="./wav2vec2-xlsr-sg-g",
logging_dir=’./logs’,
group_by_length=True,
per_device_train_batch_size=16,
gradient_accumulation_steps=2,
evaluation_strategy=“steps”,
num_train_epochs=30,
fp16=False,
save_steps=400,
eval_steps=400,
logging_steps=400,
learning_rate=3e-4,
warmup_steps=500,
save_total_limit=2,
)

Context:
The non-determinism might arise if your batches aren’t sized uniformly?  Without more detail on your training data, it’s just a wild guess.
You might try enabling fp16.  This will give you a lot more breathing room, even if it doesn’t explain the root cause…


Question:
Hey, I’ve been trying to train Sentence2Vec embeddings and I’ve been wondering what do you think about my approach.
I would be glad to learn about different possible pitfalls in my approach and how to solve them.

What do I have?

I have a small unique corpus of about 4 million sentences in my test language
I have a smaller subset (150k) of labeled sentences of whether the sentence is toxic or not.

For this question’s sake assume there isn’t any pretrained model or another existing corpus I can use.

What am I trying to achieve?
The target is to be able to cluster sentences that have similar meanings together.

Way of Action

Train a RoBERTa language model
Fine tune it for classification of toxic or not
Use a hidden layer of that classifier as an embedding model
Cluster using embeddings and HDBSCAN.

That’s about it, I tried to keep it as clear as I can.
Thanks for anyone that read up until here!!

Context:
What is your main goal? Is it to have a classifier for  toxic vs non-toxic sentences? Or to cluster sentence embeddings by semantic meaning?
For creating sentence embeddings I would recommend sentence transformers 9. It’s an extensions of regular huggingface transformers but optimized for creating text embeddings.


Question:
I’m developing an application similar to a chatbot and I’m curious how people prevent toxic outputs, E.g. references to extreme political groups, Trump. This seems like a pretty simple question, but unfortunately I couldn’t find much on it.
For the fine-tuning step, the dataset is manually labelled and toxic training examples can be filtered out. But for the much larger pre-training corpus, this isn’t feasible. I’m weighing up a few options, such as using a blacklist or a toxicity classifier.
Curious to hear what other approaches I can use?

Context: If you’re using GPT-2, you can use bad_words_ids to filter out unwanted words.


Out of index error when using pre-trained Pegasus model

Question:
Hey everyone,
I’ve been trying to use a pre-trained pegasus model for generating paraphrases of an input sentence using the most popular paraphrasing model on the huggingface model hub. However I’m running into an out of index error, and what’s strange about the error is that it only occasionally happens: most sentences get correctly paraphrased by the model but maybe one in every 100 sentences will run into an error. If I run:
import torch
from transformers import PegasusForConditionalGeneration, PegasusTokenizer

para_model_name = 'tuner007/pegasus_paraphrase'
para_tokenizer = PegasusTokenizer.from_pretrained(para_model_name)
para_model = PegasusForConditionalGeneration.from_pretrained(para_model_name)

text = [' (Chng et al']
batch = para_tokenizer(text, truncation=True, padding='longest', max_length=200,
return_tensors="pt")
translated = para_model.generate(**batch, max_length=200, num_beams=10,
num_return_sequences=1, temperature=1.5)

I get the following error (when run on cpu):
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in embedding(input, weight,
padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
    1911         # remove once script supports set_grad_enabled
    1912         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1913     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
    1914
    1915

IndexError: index out of range in self

I assumed at first it was that the tokenization scheme, and it assigned indices beyond the shape of the embedding matrix, however strangely if I change the input text to:
text = [' (Chng et alt']

then the tokens are very similar, [143, 19152,  4652,  3256, 2700, 1] for the previous error-inducing input and [143, 19152,  4652,  3256,  20913, 1] now, but the model now works. This seems a bit backwards though as the one with a higher maximum tokenized value works and leads to no errors. So I’m a bit stuck, I don’t know if internally the model generates out of vocabulary words but that seems implausible given how popular the model is (it’s been downloaded 80000 times this month), so any help would be greatly appreciated.
Thank you very much!

Context:
It seems that the problem is related to the shape of the “embed_positions” layer of the decoder.
This code snippet:

works fine with a max_length of 60 (or less) in ‘generate’ (throws no error)
and
also works fine with max_length=200 and a different Pegasus model, e.g.: “sshleifer/distill-pegasus-xsum-16-4”
(examples below)

The two models differ in the shape of this layer:
tuner007/pegasus_paraphrase:
(decoder): PegasusDecoder(
(embed_tokens): Embedding(96103, 1024, padding_idx=0)
(embed_positions): PegasusSinusoidalPositionalEmbedding(60, 1024)
sshleifer/distill-pegasus-xsum-16-4:
(decoder): PegasusDecoder(
(embed_tokens): Embedding(96103, 1024, padding_idx=0)
(embed_positions): PegasusSinusoidalPositionalEmbedding(1024, 1024)
tuner007/pegasus_paraphrase, max_length=60, config.max_position_embeddings=60 :
import torch
from transformers import PegasusForConditionalGeneration, PegasusTokenizer

para_model_name = 'tuner007/pegasus_paraphrase'
para_tokenizer = PegasusTokenizer.from_pretrained(para_model_name)
para_model = PegasusForConditionalGeneration.from_pretrained(para_model_name)

text = [' (Chng et al']
batch = para_tokenizer(text, truncation=True, padding='longest', max_length=200,
return_tensors="pt")
translated = para_model.generate(**batch,
                                 #max_length=200,
                                 max_length=60,
                                 num_beams=10,
                                 num_return_sequences=1,
                                 temperature=1.5)

print(para_model.config.max_position_embeddings)

sshleifer/distill-pegasus-xsum-16-4, max_length=200, config.max_position_embeddings=1024 :
import torch
from transformers import PegasusForConditionalGeneration, PegasusTokenizer

para_model_name = "sshleifer/distill-pegasus-xsum-16-4"
para_tokenizer_s = PegasusTokenizer.from_pretrained(para_model_name)
para_model_s = PegasusForConditionalGeneration.from_pretrained(para_model_name)

text = [' (Chng et al']
batch = para_tokenizer_s(text, truncation=True, padding='longest', max_length=200,
return_tensors="pt")
translated = para_model_s.generate(**batch, max_length=200, num_beams=10,
num_return_sequences=1, temperature=1.5)

print(para_model_s.config.max_position_embeddings)


Transformer’s output as input to other model

Question:
Hello,
I want to create a model which generates text and the generated text is input to other model. So basically two models are trained together. How can i achieve this using hugging face?
Thanks

Context:
Welcome to the forum, @omerarshad!
Nice questions, I had the same problem, too. In my opinion this is possible only if you have ground truth for the intermediate step and not only the final reference. What you might do is to train two models separately: the first one with the intermediate reference, and the last one with the final reference.
Schematically:
Input -> MODEL_1 -> Output_1
                        | compare (cross-entropy)
             Intermediate Reference

Intermediate Reference -> MODEL_2 -> Output_2
                                        | compare (cross-entropy)
                                  Final Reference

What do you think?


Inference with Finetuned BERT Model converted to ONNX does not output probabilities

Question:
Environment info

transformers version: 3.5.1
Platform: Linux-4.14.203-116.332.amzn1.x86_64-x86_64-with-glibc2.10
Python version: 3.7.6
PyTorch version (GPU?): 1.7.0 (True)
Tensorflow version (GPU?): 2.3.1 (True)
Using GPU in script?: No
Using distributed or parallel set-up in script?: No

Information
Model I am using (Bert, XLNet …): Bert
The problem arises when using:

my own modified scripts: (give details below)

The tasks I am working on is:

my own task or dataset: (give details below)

To reproduce
Steps to reproduce the behavior:

Trained HuggingFace Transformers model BertForSequenceClassification on custom dataset with PyTorch backend.
Used provided convert_graph_to_onnx.py script to convert model (from saved checkpoint) to ONNX format.
Loaded the model with ONNXRuntime
Instantiated BertTokenizer.from_pretrained(‘bert-based-uncased’) and fed in various input text to encode_plus method.
Fed outputs of this to the ONNXRuntime session.

Expected behavior
The expected behavior is that the output of sess.run on the aforementioned inputs should output an array of dimension (1, 100) (corresponding to 100 classes) with each value between 0 and 1, with all entries summing to 1. We get the correct dimension, however, we get values between about -3.04 and 7.14 (unsure what these values refer to).

Context:
Hi @nsingh, without seeing your code it’s hard to know exactly what’s going wrong but based on this comment

We get the correct dimension, however, we get values between about -3.04 and 7.14 (unsure what these values refer to).

my guess is that you are getting the logits from the model instead of the predicted classes. I ran into this problem recently and the solution was to specify pipeline_name=sentiment-analysis to load the model for a TextClassificationPipeline:
from transformers.convert_graph_to_onnx import convert

model_ckpt = ...
tokenizer = ...
onnx_model_path = ...
convert(framework="pt", model=model_ckpt, tokenizer=tokenizer,
 output=onnx_model_path, opset=12, pipeline_name="sentiment-analysis")

By default, convert_graph_to_onnx uses the feature-extraction pipeline which might explain why you’re seeing negative numbers (i.e. the logits)


404 when instantiating private model/tokenizer

Question:
I think I might be missing something obvious, but when I attempt to load my private model checkpoint with the Auto* classes and use_auth=True I’m getting a 404 response. I couldn’t find anything in the docs about the token/auth setup for the library so I’m not sure what’s wrong.
from transformers import AutoTokenizer, AutoModelWithLMHead

tokenizer = AutoTokenizer.from_pretrained("jodiak/mymodel", use_auth=True)

model = AutoModelWithLMHead.from_pretrained("jodiak/mymodel", use_auth=True)
# 404 Client Error: Not Found for url: https://huggingface.co/jodiak/model/resolve/main/config.json

I’ve verified that I can load this directory locally and also that the path in model hub is correct. Any help here is appreciated.

Context:
Hi @jodiak, did you do the login phase ?
https://huggingface.co/transformers/model_sharing.html?highlight=login#basic-steps 664
That should create a ~/.huggingface/token file (that is being used to see your private model).


Generate ‘continuation’ for seq2seq models

Question:
I am not sure if I missed an obvious way to do this, but I didn’t find any.
Basically the idea is that if we have a seq2seq model, let’s say Bart. Right now, one can input the tokens to the encoder in order to start decoding and generating text using model.generate(), but there doesn’t seem to be a way to add decoder inputs, that is text which we want the generate function to continue.
Using the example at the documentation:
from transformers import BartTokenizer, BartForConditionalGeneration, BartConfig
model = BartForConditionalGeneration.from_pretrained('facebook/bart-base')
tokenizer = BartTokenizer.from_pretrained('facebook/bart-base')
ARTICLE_TO_SUMMARIZE = "My friends are cool but they eat too many carbs."
inputs = tokenizer([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors='pt')
# Generate Summary
summary_ids = model.generate(inputs['input_ids'], num_beams=4, max_length=5, early_stopping=True)
print([tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summary_ids])

And this outputs ['My friends']. Let’s say we want to precondition the generation to start with ‘Friends’ instead of ‘My’, it would be cool to have something like:
decoder_inputs = tokenizer(['Friends'], max_length=1024, return_tensors='pt')
# Generate Summary
summary_ids = model.generate(inputs['input_ids'], decoder_inputs = decoder_inputs['inputs_ids']  num_beams=4, max_length=5, early_stopping=True)
print([tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summary_ids])

Which would output a summary starting with ‘Friends’. I am aware that it is possible to do a forward pass with explicit decoder_inputs but I was wondering if there is a way to do this at generation, to take advantage of beam search and such.
Perhaps with the newly added prefix_allowed_tokens_fn there is a workaround by having it return the desired starting tokens at the beginning of generation but I was wondering if there is a more straight forward way I missed or is this something that would be interesting to add to the generate functionality.
Cheers!

Context:
So after looking at the code behind generate() indeed it already incorporates this functionality if decoder_input_ids is given as input, in a similar fashion as the forward function:
summary_ids = model.generate(inputs['input_ids'], decoder_input_ids = decoder_inputs['inputs_ids']  num_beams=4, max_length=5, early_stopping=True)

So once again I am impressed by the amount of capabilities built into the library. I still think it would be beneficial to clarify this functionality somewhere in the documentation.
I won’t delete the post in case someone comes looking for the same.


BERT: What is the shape of each Transformer Encoder block in the final hidden state?

Question:
Hi everyone,
I am studying BERT paper after I have studied the Transformer.
The thing I can’t understand yet is the output of each Transformer Encoder in the last hidden state (Trm before T1, T2, etc… in the image).
In particular, I should know that thanks (somehow) to the Positional Encoding, the most left Trm represents the embedding of the first token, the second left represents the embedding of the second token and so on.
Hence, the shape of each one of them should be simply the hidden_dim (for example, 768) if what I have said before is true.
However, I am not convinced of this logic; so is the answer correct?
Many thanks in advance

Context:
Your answer is true but your logic seems shaky. The input embedding has nothing to do with the final shape. In other words, you could still have an output of each token by having an 1*n_tokens to 768*n_tokens linear layer and that’s it. In the case of the transformer, these have the embedding has the same size as the hidden states of the encoder, but it is conceivable to have transformations in-between that deal with different shapes, e.g. a smaller embedding shape to reduce dimensionality for computational efficiency.
So in your image, assuming the encoder block has a hidden size of 768, the output will N*768, simply because 768 is the output shape of the last layer in the encoder block.