Continual pre-training vs. Fine-tuning a language model with MLM

Question:
I have some custom data I want to use to further pre-train the BERT model. I’ve tried two following approaches so far:

Starting with a pre-trained BERT checkpoint and continuing the pre-training with Masked Language Modeling (MLM) + Next Sentence Prediction (NSP) heads (e.g. using BertForPreTraining model)
Starting with a pre-trained BERT model with the MLM objective (e.g. using the BertForMaskedLM model assuming we don’t need NSP for the pretraining part.)

But I’m still confused that if using either BertForPreTraining or BertForMaskedLM actually does the continual pre-training on BERT or these are just two models for fine-tuning that use MLM+NSP and MLM for fine-tuning BERT, respectively. Is there even any difference between fine-tuning BERT with MLM+NSP or continually pre-train it using these two heads or this is something we need to test?

Context: I have similar question here. I was following this tutorial 3, but still got quite confused when we call BertForMaskedLM how much weights we retained from the original BERT model? Please let me know if you figure out! many thanks in advance.


How to preserve Html when processing(paraphrasing)

Question: Is there a way to preserve the html when using these the accelerated interference. I am trying to put strings split on html elements through PEGASUS paraphraser. Problem is that the html is removed in the output. How would you go about preserving the html.

Context: Parse the html, while keeping all the indexes of each part and then make prediction. After getting model output place them back at those index. This won’t work without valid tags though


Using Seq2SeqTrainer to eval during training?

Question:
following the instruction of  run_summarization.py in the  example/summarization/ folder.
To further eval the trained model during training, i set the eval_strategy = "steps"  and the bash file is:
CUDA_VISIBLE_DEVICES=4,5,6,7 python -m torch.distributed.launch \
    --nproc_per_node 4 \
    --use_env \
    $(pwd)/run_summarization.py \
    --model_name_or_path /path/to/bart-base \
    --dataset_name cnn_dailymail \
    --dataset_config_name 3.0.0 \
    --per_device_train_batch_size 16 \
    --per_device_eval_batch_size 16 \
    --num_train_epochs 5 \
    --do_train \
    --do_eval \
    --predict_with_generate \
    --learning_rate 3e-5 \
    --label_smoothing_factor 0.1 \
    --weight_decay 0.01 \
    --max_grad_norm 1.0 \
    --logging_strategy 'steps' \
    --logging_steps 1000 \
    --save_strategy 'steps' \
    --save_steps 5000 \
    --save_total_limit 3 \
    --evaluation_strategy 'steps' \
    --eval_steps 5000 \
    --fp16 \
    --output_dir /path/to/output_dir \

But when eval during training, the eval results seem not will.  The max_length  & num_beams are set to default values in the Config files of BART(max_length=20, num_beams=4).
Functions relative to the problem might be:
train -- trainer.py
 _maybe_log_save_evaluate -- trainer.py
evaluate --trainer_seq2seq.py
evaluate -- trainer.py
evaluation_loop -- train.py
prediction_step --trainer_seq2seq.py

maybe in the trainer_seq2seq.py, TheSeq2SeqTrainer should ovrride the _maybe_log_save_evaluate function to explictly provided the max_length & num_beams hyper-parameters ?

Another problems is in the run_summarization_no_trainer.py.
in the Link 293, It didn’t provide the  fp16, so in the Link 444, the accelerator.use_fp16 must be False. I guess the fp16 should be added in the parser as a hyper-parameter ?

Context: The same question for trainer also


How to get the predicted labels per epoch or step for the huggingface.transformers Trainer?

Question:
Hello, I’m new to Hugging Face and I have a question regarding the huggingface.transformers Trainer class. How do we get the predicted labels per epoch or step for the huggingface.transformers Trainer?
Thank you.

Context:
I think you need to define a compute metric function which get your prediction and return the accuracy.
def compute_metrics(pred):
labels = pred.label_ids
preds = pred.predictions.argmax(-1)
precision, recall, f1, _ = precision_recall_fscore_support(labels, preds)
precision_ma, recall_ma, f1_ma, _ = precision_recall_fscore_support(labels, preds, average=‘macro’)
precision_mi, recall_mi, f1_mi, _ = precision_recall_fscore_support(labels, preds, average=‘micro’)
acc = accuracy_score(labels, preds)
return {
‘accuracy’: acc,
‘f1’: f1}  and pass it to the trainer


Question:
Soon we will merge a big refactor of the generate() method: https://github.com/huggingface/transformers/pull/6949 20
All transformer models that have a language model head rely on the generate() method, e.g.

Bart, T5, Marian, ProphetNet for summarization, translation, …
GPT2, XLNet for open-end text generation

The generate() method has become hard to read and more and quite difficult to maintain. The goal of the refactor is thus as follows:

Remove the _no_beam_search_generation and _beam_search_generation methods and replace them with the four different generation methods corresponding to the four different types of generation: greedy_search  (corresponds to num_beams = 1, do_sample=False), sample  (corresponds to num_beams = 1, do_sample=True), beam_search  (corresponds to num_beams > 1, do_sample = False), and beam_sample  (corresponds to num_beams > 1, do_sample = True).
The following philosophy has been adapted here: the original generate() method is kept at 100% backwards compatibility (ping me on github under @patrickvonplaten, if your specific use case has nevertheless been broken by this PR). In addition each of the four specific generation methods can be used directly. The specific methods are as “bare-bone” as possible meaning that they don’t contain any magic pre-processing of input tensors. E.g. generate() automatically runs the input_ids through the encoder if model is encoder-decoder model vs. for each specific generate method they have to be added as encoder_outputs to the model_kwargs; input_ids are automatically created in case they are empty or automatically expanded for num_beams  in generate()  vs. this has to be done manually for each specific generate method. The reason behind this design is that it will give the user much more flexibility for specific use cases of generate() and improves maintainability and readability. The user should not be limited in any way when directly using the specific generate methods. It should therefore pave the way to allow backprop through generate, make beam generation much easier, makes it easier to write “higher level” generate functions as is done for RAG… For more information on this design please read the docs, look into the examples of greedy_search, sample, beam_search and beam_sample.

All of the generate parameters that can be used to tweak the logits distribution for better generation results, e.g. no_repeat_ngram_size, min_length, … are now defined as separate classes that are added to a LogitsProcessorList. This has the following advantages: a) better readability b) much easier to test these functions c) easier to add new logits distribution warpers. A huge thanks goes to https://github.com/turtlesoupy 1 who has had the original idea of this design in this PR: https://github.com/huggingface/transformers/pull/5420 7

Move all beam_search relevant code into its own generation_beam_search.py file and speed up beam search. Beam search has gained more and more in importance thanks to many new and improved seq2seq models. This PR moves the very difficult to understand beam search code into its own file and makes sure that the beam_search generate function is easier to understand this way. Additionally, all Python List operations are now replaced by torch.tensor operations which led to a 5-10% speed up for beam_search generation. This change improves speed, readability and also maintainability. New beam search algorithms, such as https://arxiv.org/abs/2007.03909 2 , should now be easier to add to the library.

Tests have been refactored and new more aggressive tests have been added. The tests can also be very helpful to understand how each of the methods work exactly. Check out test_generation_utils.py, test_generation_beam_search and test_generation_logits_process.

More docstring especially on beam search and logits processors to make generate more accessible and understandable to the user.

TODO:

Do the same refactor for TF
Check possibility of carrying gradients through generate
Add GenerationOutputs similar to ModelOutputs that allows to return attention outputs and hidden states

Context:
Hi Patrick @patrickvonplaten,
Thanks for your great work!! I have a question.
I now almost finish translating TFDPR 1 , and currently also working on TFRag (I already made TFRagModel works like Pytorch, and I am investigating TFRagTokenForGeneration right now) .
Since the current code of RagTokenForGeneration involves with both _no_beam_search_generation  and  _beam_search_generation , regarding the Refactor, I think I should just go ahead with the current version, and fix them later for refactor.
Or would you suggest that I wait for Pytorch Refactor, or wait for both Pytorch & TF Refactor ?


Beam search (FlaxT5) generates PAD tokens mid generation

Question:
Hi,
When I try to use beam search (num_beams > 1) with FlaxT5ForConditionalGeneration, the model generates PAD tokens randomly in the middle of the generation and the output is not usually complete.
Is this a known issue?
I have seen the FlaxMarianMTModel overrides _adapt_logits_for_beam_search method to fix a similar issue. Is something similar required for FlaxT5ForConditionalGeneration as well? If someone can guide me, I can look at implementing it and getting a PR up.
Thank you!

Context:
I have also encountered this issue recently.  I used the official t5_summarization_flax.py, but ran it on an internal dataset.  I get good results using greedy search, but much worse results when using num_beams > 1.  On some examples it generates the whole sequence, but on others it starts generating pad tokens mid-sequence, without outputting an EOS token.
I tried monkeypatching the MarianMT _adapt_logits_for_beam_search onto the FlaxT5Model, as the method description seemed like it was implemented to solve this type of issue, but that didn’t seem to change the outputs for T5 at all.
Any thoughts on why this might be happening, or how we could resolve it for Flax T5 beam search @patrickvonplaten / @patil-suraj?


How to generate a sequence using inputs_embeds instead of input_ids?

Question:
Hello, I am struggling with generating a sequence of tokens using model.generate() with inputs_embeds.
For my research, I have to use inputs_embeds (word embedding vectors) instead of input_ids (token indices) as an input to the GPT2 model.
I want to employ model.generate() which is a convenient tool for generating a sequence of tokens, but there is no argument for inputs_embeds. I tried to edit " transformers.generation_utils", but it was not easy to figure out which lines I should change.
Is there any idea that I can easily generate tokens with default settings for hyper-paremeters as in model.generate()? If there is any idea, help me please.

Context: Were you able to figure something out for this ? My needs are similar. I’m using BART model


Question:
@stas ,
I saw your tweet and had two questions.

Does using deepspeed inference require consolidating the files in the checkpoint dir or it performs that for you and then reshards them across available GPUs?

Does this now support model > GPU RAM or is the comment in the doc 1 still accurate:

Additionally DeepSpeed is currently developing a related product called Deepspeed-Inference which has no relationship to the ZeRO technology, but instead uses tensor parallelism to scale models that can’t fit onto a single GPU. This is a work in progress and we will provide the integration once that product is complete.

In general I am trying to figure out if there exists a sharded checkpoint  format that allows either (a) inference or (b) resuming training on a different world size.

Context:
Deepspeed ZeRO Inference is the same as ZeRO Training except it doesn’t allocate optimizer and lr scheduler and that it requires ZeRO-3.
Therefore it always supports model > single gpu RAM.
During Training it indeed saves a sharded state checkpoint.
During Inference it doesn’t need to do that. While it can load from a sharded trained-with-ZeRO checkpoint - It is meant to be used with a normal single file checkpoint. It uses zero.Init to split the weights to different GPUs while loading the model.
When we do normal training, we need to do validation, so inference happens there already.
So yes, DeepSpeed uses a sharded checkpoint format.
I don’t think you can currently use a sharded checkpoint for N gpus and be able to load it on M gpus automatically.
I wrote a script to consolidate fp32 weights: Zero Redundancy Optimizer (ZeRO) - DeepSpeed 3
which if you use it will allow you to move from N to M gpus, since it generates a single unsharded checkpoint.
You can, of course, write code that will rewrite the checkpoints to change from N to M gpus.
The main reason I wrote it as a separate tool is that it requires huge amount of CPU RAM, so doing this dynamically at run time could be too demanding and may require an instance with a lot of CPU memory

In the case of TP+PP+DP which we use at Megatron-Deepspeed, Tunji has been working on converting any deepspeed PP-style checkpoint from any TP/PP degree to any other TP/PP degree. But the PP checkpoint is slightly different from ZeRO checkpoint. It has each layer in a separate checkpoint, and saves each TP split in a separate checkpoint. So TP is hardwired and can’t be changed once the training started, whereas PP can be changed on the fly. But of course one can reformat the checkpoint to change TP degree as well. The tools are here:

      github.com

    Megatron-DeepSpeed/tools/convert_checkpoint at main ·... 2

  main/tools/convert_checkpoint

  Ongoing research training transformer language models at scale, including: BERT & GPT-2 - Megatron-DeepSpeed/tools/convert_checkpoint at main · microsoft/Megatron-DeepSpeed

they also include conversion from Meg-DS → Megatron-LM → HF Transformers.
More tools are planned to be added there.

Deepspeed Inference (not ZeRO) uses Tensor Parallelism and at the moment this is still a WIP - I think right now it works with a single checkpoint. But it’d probably make sense to pre-shard it for a faster loading.

Please let me know if I have addresses all the questions.


Error when Fine-tuning pretrained Masked Language Model

Question:
My whole question is here:- python - TypeError: zeros_like(): argument 'input' when fine-tuning on MLM - Stack Overflow 8
Basically, I am having this error when fine-tuning my pretrained model:
ValueError: expected sequence of length 2033 at dim 1 (got 2036)

Anyone have any idea how I can solve this?

Context: Anyone  ? I have put padding=True, so such issues should not exist


Improve the performance of model prediction of transformers model

Question:
Hi I am new to transformers. I am using the some models of it for many tasks. One is the summarization using google pegasus-xum model, the performance is good on GPU but when I try to do it on CPU it takes around 16-18 seconds. I also started using the parrot-paraphrase library which uses the T5 model in the backend, it also performs the same in GPU but on CPU it is taking time around 5-8 seconds to process the result. Due to some limitation of GPU on my server I have to optimize it for CPU, to take down the response time to 2-4 seconds max.
Here are the link of models I am using:
Pegasus-XSUM: google/pegasus-xsum · Hugging Face
Parrot=Paraphrase: prithivida/parrot_paraphraser_on_T5 · Hugging Face 1
Code for pegasus model:
from transformers import PegasusTokenizer, PegasusForConditionalGeneration
from trained_model import ModelFactory
import os

project_root = os.path.dirname(os.path.dirname(__file__))

path = os.path.join(project_root, 'models/')

class Summarization:
    def __init__(self):
        self.mod = ModelFactory("summary")
        self.tokenizer = PegasusTokenizer.from_pretrained(path)

    def generate_abstractive_summary(self, text):
        """ To generate summary of text using pegasus model"""
        model = self.mod.get_preferred_model("summarization")
        inputs = self.tokenizer([text], max_length=1024, return_tensors='pt', truncation=True)
        summary_ids = model.generate(inputs['input_ids'])
        summary = [self.tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summary_ids]
        return summary[0]

Is there any way to improve the performance, any small improvement also appreciated…

Context:
I have not used those two specific models before, however without a small reproducible example of your code is impossible to provide personalised feedback. You didn’t mention what you’re doing or using the model for, but I’ll assume you’re doing inference (not training or fine tuning) based on the run times you reported.
Ultimately doing inference on a CPU will be much slower than on a GPU, there is absolutely no way around it (although the time difference if you’re doing training in a   CPU vs GPU is even bigger). Whether there is still some margin for CPU optimisation in your specific case, is something you would need to share the code for the community to be able to comment on.
The only thing that comes to my mind right now (without knowing anything at all about your code), is to only initialise the model once, and then do inference on the same instance, so avoid calling something like
trained_model = AutoModel.from_pretrained(...)

at every iteration inside a for loop, as that would be very detrimental, but without seeing the code I can’t say much else.


Question:
I resumed training from checkpoint. I set the learning rate in TrainingArguments to 5e-5. Now the learning rate in the first logging step is 2.38e-05. Its value decreases in subsequent steps. How can I set the learning rate to the desired value? I do not understand where this 2.38e-05 comes from.
These are my training arguments.
training_args = Seq2SeqTrainingArguments(
        output_dir=output_dir,
        num_train_epochs=8,
        max_steps=-1,
        evaluation_strategy='epoch',
        eval_steps=0,
        per_device_train_batch_size=8,
        per_device_eval_batch_size=8,
        learning_rate=5e-5,
        warmup_ratio=0.1,
        warmup_steps=0,
        logging_dir=None,
        logging_strategy='steps',
        logging_steps=50,
        disable_tqdm=disable_tqdm,
        save_strategy='epoch',
        save_steps=0,
        load_best_model_at_end=True,
        metric_for_best_model='eval_loss',
        seed=random_state,
        predict_with_generate=True,
        dataloader_num_workers=4,
        save_total_limit=10,
    )

Context: The scheduler used by default is a linear decay, so that’s why you see this learning rate, since you’re logging after 50 steps.


NLP Pretrained model model doesn’t use GPU when making inference

Question:
I am using Marian MT Pretrained model for Inference for machine Translation task integrated with a flask Service  . I am running the Model on Cuda enabled device .While inferencing the model not using the GPU ,it is using the CPU only .I don’t want to use the cpu for inference as it is taking very long time for processing the request. Even if i am passing 1 sentence it is taking very long . Please help on this . Below is the code snippet and model i am using
model_name = ‘Helsinki-NLP/opus-mt-ROMANCE-en’
tokenizer = MarianTokenizer.from_pretrained(model_name)
print(tokenizer.supported_language_codes)
model = MarianMTModel.from_pretrained(model_name)
translated = model.generate(**tokenizer.prepare_translation_batch(src_text))
tgt_text = [tokenizer.decode(t, skip_special_tokens=True) for t in translated]
I have downloaded the pytorch model.bin and other tokenizer  files from s3 environment and saved on my local …Please help on this how i can put the things on GPU for faster inference

Context: Have you tried model.to(‘cuda’), to make the model use the GPU?


Why fine-tuning BERT mlm on specific domain doesn’t work? What am I doing wrong?

Question:
I’m new. I’m trying to fine-tuned a BERT MLM (bert-base-uncased) on a target domain. Unfortunately, results are not good.
Before fine-tuning, the pre-trained model fills the mask of a sentence with words in line of human expectations.
E.g. Wikipedia is a free online [MASK], created and edited by volunteers around the world.
The most probable prediction are encyclopedia (score: 0.650) and resource (score:0.087).
After fine-tuning, the prediction are completely wrong. Often stopwords are predicted as result.
E.g. Wikipedia is a free online [MASK], created and edited by volunteers around the world.
The most probable prediction are the (score: 0.052) and be (score:0.033).
I experimented with different epochs (from 1 to 10) and different datasets (from a few MB to a few GB) but I got the same issue. What am I doing wrong? I’m using the following code, I hope you can help me.
from transformers import AutoConfig, AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
config = AutoConfig.from_pretrained('bert-base-uncased', output_hidden_states=True)
model = AutoModelForMaskedLM.from_config(config)  # BertForMaskedLM.from_pretrained(path)

from transformers import LineByLineTextDataset

dataset = LineByLineTextDataset(tokenizer=tokenizer,
                                file_path="data/english/corpora.txt",
                                block_size = 512)

from transformers import DataCollatorForLanguageModeling

data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=True, mlm_probability=0.15)

from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(output_dir="output/models/english",
                                 overwrite_output_dir=True,
                                 num_train_epochs=5,
                                 per_gpu_train_batch_size=8,
                                 save_steps = 22222222,
                                 save_total_limit=2)

trainer = Trainer(model=model, args=training_args, data_collator=data_collator, train_dataset=dataset)

trainer.train()
trainer.save_model("output/models/english")

from transformers import pipeline

# Initialize MLM pipeline
mlm = pipeline('fill-mask', model="output/models/english", tokenizer="output/models/english")

# Get mask token
mask = mlm.tokenizer.mask_token

# Get result for particular masked phrase
phrase = f'Wikipedia is a free online {mask}, created and edited by volunteers around the world'

result = mlm(phrase)

# Print result
print(result)

Context:
What I believe happening here is, when you call
config = AutoConfig.from_pretrained('bert-base-uncased', output_hidden_states=True) model = AutoModelForMaskedLM.from_config(config)
the model is a randomly initialized BERT model having the same configuration with “bert-base-uncased” model. So, after fine-tuning, model predictions are not satisfactory because the initial model is not a pretrained one. I have checked it on a colab notebook like this:

image669×736 35.6 KB

If your intention is just to enable output_hidden_states functionality, you can try:
AutoModelForMaskedLM.from_pretrained("bert-base-uncased", output_hidden_states=True)
When I ran the line I recommended on colab, the download time was noticably greater which is a potential indicator that it is indeed the model parameters that are being loaded, not just the configuration file.


Finetuned I-Bert for question answering task

Question: How can we fine-tune an Integer Bert for question answering task?

Context:
Hi,
As explained in IBERT’s model card: fine-tuning the model consists of 3 stages:

full precision fine-tuning
quantization
quantization-aware training.

So for the first step, you can fine-tune IBERT just as any BERT model. You can take a look at the example scripts 1 as well as the official QA notebook 2.
Step 2 is simply setting the quantize attribute of the model’s configuration to True.
Step 3 is the same as step 1, but now with your quantized model.


Language generation with torchscript model?

Question:
I have fine-tuned a summarization model following the Hugging Face seq2seq guide (starting from sshleifer/distilbart-xsum-12-6).
Our team is interested in using AWS elastic inference for deployment for cost reduction. (e.g. similar to this https://aws.amazon.com/blogs/machine-learning/fine-tuning-a-pytorch-bert-model-and-deploying-it-with-amazon-elastic-inference-on-amazon-sagemaker/ 6)
I was wondering whether there’s any examples or any suggested way to use the beam searching logic in BartForConditionalGeneration with model inference from a torchscript model. Most of the examples for torchscript I’ve found are with classification tasks where this isn’t necessary.

Context:
I’ve had success with deploying a BartForConditionalGeneration model using SageMaker with EI.
Try:
model = BartForConditionalGeneration.from_pretrained(model_dir, torchscript=True)


ModuleNotFoundError: No module named ‘transformers’

Question:
Hi! I’ve been having trouble getting transformers to work in Spaces.
When tested in my environment using python -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('we love you'))", the results show it’s been properly installed. When imported in Colab it works fine too, but whenever deployed to Spaces it always returns the same ModuleNotFound error. Full traceback message:
Traceback:
File "/home/user/.local/lib/python3.8/site-packages/streamlit/script_runner.py", line 354, in _run_script
    exec(code, module.__dict__)File "/home/user/app/app.py", line 1, in <module>
    from transformers import pipeline

It’s a simple test app using transformers and streamlit, - both of which were reinstalled with pip after creating a new venv and reinstalling tensorflow and pytorch. I also tried cleaning, uninstalling, and reinstalling conda based on advice from another forum. No dice.
Currently using:
Python 3.9.4
Tensorflow 2.7.0
PyTorch 1.10.0
Transformers 4.12.3
Streamlit 1.2.0
Any help greatly appreciated! Thanks

Context: it might be due to not having a requirements file. Here is an example of what your spaces app should have -  flax-community/image-captioning at main 1 try adding the requirements as they till the environment what packages to load. Hope this helps.


Question:
So I tried doing from transformers import FlaxVisionEncoderDecoderModel but it throws me the following exception ImportError: cannot import name 'FlaxVisionEncoderDecoderModel' from 'transformers' (/.local/lib/python3.8/site-packages/transformers/__init__.py)
I have Flax installed, (with jax as well) I upgraded Transformers to the latest version.
As a reference: I can run GPT-J with transformers on this TPUVM just fine, so I don’t know why it doesnt like this flax model-thing?
I’m totally lost, help needed.

Context: user Cahya on discord helped me on this, turns out FlaxVisionEncoderDecoderModel is not on 4.12.5, so simply install it from the repo in order to have it!


[Ray] How to get the best model per trial

Question:
Hi everyone,
I’m trying to use the feature “hyperparameter_search” but cannot get the best model per the trial. Each trial always get the last value.
I see the code line here calling the Ray tune function to get the best model without the “scope” parameter passed. Default value of that function is “last”.

      github.com

    huggingface/transformers/blob/123cce6ffcafe6ecece7eb9914395d2ebcc98a48/src/transformers/integrations.py#L294

              # special attr set by tune.with_parameters
              if hasattr(trainable, "__mixins__"):
                  dynamic_modules_import_trainable.__mixins__ = trainable.__mixins__

    analysis = ray.tune.run(
                  dynamic_modules_import_trainable,
                  config=trainer.hp_space(None),
                  num_samples=n_trials,
                  **kwargs,
              )
              best_trial = analysis.get_best_trial(metric="objective", mode=direction[:3])
              best_run = BestRun(best_trial.trial_id, best_trial.last_result["objective"], best_trial.config)
              if _tb_writer is not None:
                  trainer.add_callback(_tb_writer)
              return best_run

def run_hp_search_sigopt(trainer, n_trials: int, direction: str, **kwargs) -> BestRun:

    from sigopt import Connection

Is there any way to workaround this?
Thanks!

Context:
Thanks for raising this.
I am having the same problem with the Optuna backend.


Got `ONNXRuntimeError` when try to run BART in ONNX format #12851

Question:
Environment info

transformers version: 4.9.0
Platform: Linux-5.4.104±x86_64-with-Ubuntu-18.04-bionic
Python version: 3.7.11
PyTorch version (GPU?): 1.9.0+cu102 (True)
Using GPU in script?: Yes

I was using Google Colab and trying to export model facebook/bart-large-cnn to the onnx format. I ran the command python -m transformers.onnx -m=facebook/bart-large-cnn onnx/bart-large-cnn , and the outputs seem okay.
2021-07-22 23:14:33.821472: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
Using framework PyTorch: 1.9.0+cu102
Overriding 1 configuration item(s)
	- use_cache -> False
/usr/local/lib/python3.7/dist-packages/transformers/models/bart/modeling_bart.py:212: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
/usr/local/lib/python3.7/dist-packages/transformers/models/bart/modeling_bart.py:218: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attention_mask.size() != (bsz, 1, tgt_len, src_len):
/usr/local/lib/python3.7/dist-packages/transformers/models/bart/modeling_bart.py:249: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
/usr/local/lib/python3.7/dist-packages/transformers/models/bart/modeling_bart.py:863: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if input_shape[-1] > 1:
tcmalloc: large alloc 1625399296 bytes == 0x5595ce83a000 @  0x7f1780d9f887 0x7f177f695c29 0x7f177f696afb 0x7f177f696bb4 0x7f177f696f9c 0x7f17670dcbb7 0x7f17670dd064 0x7f175b75ba1c 0x7f176bf8eaff 0x7f176b949b88 0x55949fda8bf8 0x55949fe1c6f2 0x55949fe16c35 0x55949fda973a 0x55949fe1893b 0x55949fe16c35 0x55949fda973a 0x55949fe1bf40 0x55949fe16c35 0x55949fda973a 0x55949fe1893b 0x55949fda965a 0x55949fe17b0e 0x55949fda965a 0x55949fe17b0e 0x55949fe16c35 0x55949fe16933 0x55949fe14da0 0x55949fda7ea9 0x55949fda7da0 0x55949fe1bbb3
tcmalloc: large alloc 1625399296 bytes == 0x55962f654000 @  0x7f1780d9f887 0x7f177f695c29 0x7f177f696afb 0x7f177f696bb4 0x7f177f696f9c 0x7f17670dcbb7 0x7f17670dd064 0x7f175b75ba1c 0x7f176bf8ecab 0x7f176b949b88 0x55949fda8bf8 0x55949fe1c6f2 0x55949fe16c35 0x55949fda973a 0x55949fe1893b 0x55949fe16c35 0x55949fda973a 0x55949fe1bf40 0x55949fe16c35 0x55949fda973a 0x55949fe1893b 0x55949fda965a 0x55949fe17b0e 0x55949fda965a 0x55949fe17b0e 0x55949fe16c35 0x55949fe16933 0x55949fe14da0 0x55949fda7ea9 0x55949fda7da0 0x55949fe1bbb3
tcmalloc: large alloc 1625399296 bytes == 0x5595ce83a000 @  0x7f1780d9d1e7 0x55949fdd9a18 0x55949fda4987 0x7f176bf8ece2 0x7f176b949b88 0x55949fda8bf8 0x55949fe1c6f2 0x55949fe16c35 0x55949fda973a 0x55949fe1893b 0x55949fe16c35 0x55949fda973a 0x55949fe1bf40 0x55949fe16c35 0x55949fda973a 0x55949fe1893b 0x55949fda965a 0x55949fe17b0e 0x55949fda965a 0x55949fe17b0e 0x55949fe16c35 0x55949fe16933 0x55949fe14da0 0x55949fda7ea9 0x55949fda7da0 0x55949fe1bbb3 0x55949fe16c35 0x55949fda973a 0x55949fe17b0e 0x55949fe16c35 0x55949fce8eb1
tcmalloc: large alloc 1625399296 bytes == 0x55962f654000 @  0x7f1780d9f887 0x7f177f695c29 0x7f177f695d47 0x7f177f6977a5 0x7f176bd60368 0x7f176bfbc844 0x7f176b949b88 0x55949fda8010 0x55949fda7da0 0x55949fe1bbb3 0x55949fe16c35 0x55949fda973a 0x55949fe1893b 0x55949fe16c35 0x55949fda973a 0x55949fe1bf40 0x55949fe16c35 0x55949fda973a 0x55949fe1893b 0x55949fda965a 0x55949fe17b0e 0x55949fda965a 0x55949fe17b0e 0x55949fe16c35 0x55949fe16933 0x55949fe14da0 0x55949fda7ea9 0x55949fda7da0 0x55949fe1bbb3 0x55949fe16c35 0x55949fda973a
Validating ONNX model...
	-[✓] ONNX model outputs' name match reference model ({'last_hidden_state', 'encoder_last_hidden_state'}
	- Validating ONNX Model output "last_hidden_state":
		-[✓] (2, 8, 1024) matchs (2, 8, 1024)
		-[✓] all values close (atol: 0.0001)
	- Validating ONNX Model output "encoder_last_hidden_state":
		-[✓] (2, 8, 1024) matchs (2, 8, 1024)
		-[✓] all values close (atol: 0.0001)
All good, model saved at: onnx/bart-large-cnn/model.onnx

Then I tried to execute the model in onnxruntime ,
import onnxruntime as ort

ort_session = ort.InferenceSession('onnx/bart-large-cnn/model.onnx')

# Got input_ids and attention_mask using tokenizer

outputs = ort_session.run(None, {'input_ids': input_ids.detach().cpu().numpy(), 'attention_mask': attention_mask.detach().cpu().numpy()})

And I got the error,
---------------------------------------------------------------------------
RuntimeException                          Traceback (most recent call last)
<ipython-input-30-380e6a0e1085> in <module>()
----> 1 outputs = ort_session.run(None, {'input_ids': input_ids.detach().cpu().numpy(), 'attention_mask': attention_mask.detach().cpu().numpy()})

/usr/local/lib/python3.7/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py in run(self, output_names, input_feed, run_options)
    186             output_names = [output.name for output in self._outputs_meta]
    187         try:
--> 188             return self._sess.run(output_names, input_feed, run_options)
    189         except C.EPFail as err:
    190             if self._enable_fallback:

RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Reshape node. Name:'Reshape_109' Status Message: /onnxruntime_src/onnxruntime/core/providers/cpu/tensor/reshape_helper.h:42 onnxruntime::ReshapeHelper::ReshapeHelper(const onnxruntime::TensorShape&, std::vector<long int>&, bool) gsl::narrow_cast<int64_t>(input_shape.Size()) == size was false. The input tensor cannot be reshaped to the requested shape. Input shape:{2}, requested shape:{1,1}

I see that BART is recently supported for ONNX in the latest release, but there isn’t any code to exactly explain how to run the inference in onnxruntime . Maybe I’m doing something wrong here, so any help will be appreciated!

Context:
Hi @AlfredWGA,
Thanks for bringing this up to us.
As a sanity check, can you please provide the shape of input_ids and attention_mask? Also if you can share the ONNX Runtime version you’re using, that would be very helpful.
Thanks!


Question:
I added few tokens to the tokenizer, and would like now to train roberta model. Will it automatically also tune the embedding layer (the layer that embeds the tokens), or is there any flag or anything else I should change so that embedding layer will be tuned? Schematically, my code looks like that:

model = RobertaForSequenceClassification.from_pretrained(‘roberta-base’, num_labels=2)
trainer = Trainer(
model=model,
args=training_args,  # training arguments, defined above
…
)
trainer.train()

and the input data is the tokenized data.

Context:
[I am not an expert, but I believe this is right]
If you need to add different tokens, then you will need to train the RoBerta model from scratch.
(You probably don’t want to do that.)
It doesn’t work to change the tokens after the model has been pre-trained.
Do you definitely need to add different tokens?
If you just include your different tokens in your data, the tokenizer will probably deal with them OK, by representing them as combinations or tokens it already knows.
I recommend Chris McCormick’s blog posts about this, BERT Word Embeddings Tutorial · Chris McCormick 43
By default, if you fine-tune a pre-trained RoBerta model, then the embedding layer will be very slightly tuned.  Most of the tuning change will happen in the last few layers, especially the Classification head layer.
If you want to tune ONLY the last layer(s), you can Freeze the earlier layers.   (It isn’t possible to freeze the later layers and tune the earlier ones).


ModuleNotFoundError: No module named ‘transformers.modeling_roberta’

Question:
im trying to use longformer and in its code it has from transformers.modeling_roberta import RobertaConfig, RobertaModel, RobertaForMaskedLM
but although I install the transformers and I can do import transformers I still get an error when I do:
from transformers.modeling_roberta import RobertaConfig, RobertaModel, RobertaForMaskedLM
any suggestion on how to install it? or what to do?

Context: You should import those objects from transformers directly.


Wav2Vec2 Speech Pre-Training After a few epochs the contrastive loss was decreased to zero and the model stopped changing

Question:
I tried to re-run the demo script with the same parameters on Colab. After a few epochs, the contrastive loss was decreased to zero and the model stopped changing. Original Script can be found here:

      github.com

    transformers/examples/pytorch/speech-pretraining at master ·... 2

  master/examples/pytorch/speech-pretraining

  🤗 Transformers: State-of-the-art Natural Language Processing for Pytorch, TensorFlow, and JAX. - transformers/examples/pytorch/speech-pretraining at master · huggingface/transformers

Here is sample code Colab 4:
sample output:
| loss: 9.969e-02| constrast_loss: 0.000e+00| div_loss: 9.969e-01| %_mask_idx: 5.137e-01| ppl:
2.000e+00| lr: 1.572e-03| temp: 1.902e+00| grad_norm: 8.068e-19

| loss: 9.969e-02| constrast_loss: 0.000e+00| div_loss: 9.969e-01| %_mask_idx: 4.952e-01| ppl:
2.000e+00| lr: 1.572e-03| temp: 1.902e+00| grad_norm: 4.017e-19

| loss: 9.969e-02| constrast_loss: 0.000e+00| div_loss: 9.969e-01| %_mask_idx: 4.831e-01| ppl:
2.000e+00| lr: 1.572e-03| temp: 1.902e+00| grad_norm: 5.166e-19

I used the parameters given in the README file so this behavior is unexpected and may indicate a different problem.
Is this a bug in the official feature or am I doing some mistake if so please help me how can I fix this problem?

Context: I am struggling with this issue since a month ago, Has anyone found an explanation such behaviour? Thanks.


Question: How to do batch generation with the GPT2 model?

Context:
Batch generation is now possible for GPT2 in master by leveraging the functionality shown in this PR: https://github.com/huggingface/transformers/pull/7552?notification_referrer_id=MDE4Ok5vdGlmaWNhdGlvblRocmVhZDEyMTMzNzA0MDA6MjM0MjM2MTk%3D#event-3876130796 156 .
For more info on how to prepare a GPT2 for batch generation, you can checkout this test:

      github.com

    huggingface/transformers/blob/890e790e16084e58a1ecb9329c98ec3e76c45994/tests/test_modeling_gpt2.py#L430 94

def test_gpt2_sequence_classification_model(self):
    config_and_inputs = self.model_tester.prepare_config_and_inputs()
    self.model_tester.create_and_check_gpt2_for_sequence_classification(*config_and_inputs)

def test_gpt2_gradient_checkpointing(self):
    config_and_inputs = self.model_tester.prepare_config_and_inputs(gradient_checkpointing=True)
    self.model_tester.create_and_check_forward_and_backwards(*config_and_inputs)

@slow
def test_batch_generation(self):
    model = GPT2LMHeadModel.from_pretrained("gpt2")
    model.to(torch_device)
    tokenizer = GPT2Tokenizer.from_pretrained("gpt2")

    tokenizer.padding_side = "left"

    # Define PAD Token = EOS Token = 50256
    tokenizer.pad_token = tokenizer.eos_token
    model.config.pad_token_id = model.config.eos_token_id


Whitelist specific tokens in beam search

Question:
I’m using model.generate() for text generation.
I’m wondering is there a way to whitelist specific tokens so that they are returned during the beam search phase.
For example - I want to the “force” the response to contain a question mark or a speccific phrase

Context: Every token is “whitelisted” in the sense that it is considered during beam search. However, you can modify/boost certain tokens you would like to see generated, using the LogitsProcessor 10. Just implement your own class that boosts your favourite tokens or sequence of tokens. If you are looking for a question mark - maybe you can “blacklist” other end of sentence punctuations in your LogitsProcessor, too?


Question:
Hello everybody,
I recently fine-tuned a GPT model (GPT2LMHeadModel) and tried to push it to hub.  I followed the tutorial but got this strange error: ‘GPT2LMHeadModel’ object has no attribute ‘push_to_hub’

Screenshot 2021-11-10 at 21.31.481182×727 218 KB

I’ve updated the transformers library to the latest version, as shown in the screenshot. Can anybody help me with that? I’ve been working on this problem for 3 hours but still in vain…
(By the way I was coding on Kaggle)
Any help is appreciated!

Context:
Hi,
The correct way to push a model to the hub is by typing:
model.save_pretrained(".",
        push_to_hub=True,
        commit_message="First commit",
        repo_path_or_name: Optional[str] = None,
        repo_url: Optional[str] = None,
        use_temp_dir: bool = False,
        commit_message: Optional[str] = None,
        organization: Optional[str] = None,
        private: Optional[bool] = None,
        use_auth_token: Optional[Union[bool, str]] = None,)

As can be seen here.


Output of ‘bert-base-NER-uncased’ is different when using website and different when used via python

Question:
Hi,
I’m trying to use the above mentioned model for token classification.
Below is my sample text:

00:00:02 Speaker 1: hi john, it’s nice to see you again. how was your weekend? do anything special? 00:00:06 Speaker 2: yep, all good thanks. i was with my sister in derby. We saw, you know, that james bond film. what’s it called? then got a couple of drinks at the pitcher and piano, back in nottingham. 00:00:18 Speaker 1: that’s close to your flat, right? 00:00:25 Speaker 2: yeah, about five minutes away. i live on parliament street, remember? 00:00:39 Speaker 1: of course, i remember. you moved last year after you left your parents’ place. 00:00:39 Speaker 2: yeah, it was my sister’s birthday on sunday, susie, the older one. i told you last time about that new job she got. sainsbury’s, the one by victoria centre.

When using the hosted interface API 1, the output is excellent:

And here is the json output from the hosted API:
[
  {
    "entity_group": "PER",
    "score": 0.9778427481651306,
    "word": "john",
    "start": 23,
    "end": 27
  },
  {
    "entity_group": "LOC",
    "score": 0.9929279685020447,
    "word": "derby",
    "start": 166,
    "end": 171
  },
  {
    "entity_group": "MISC",
    "score": 0.7170370817184448,
    "word": "james bond",
    "start": 196,
    "end": 206
  },
  {
    "entity_group": "LOC",
    "score": 0.993842363357544,
    "word": "nottingham",
    "start": 293,
    "end": 303
  },
  {
    "entity_group": "LOC",
    "score": 0.9108084440231323,
    "word": "parliament street",
    "start": 420,
    "end": 437
  },
  {
    "entity_group": "PER",
    "score": 0.9840036034584045,
    "word": "susie",
    "start": 613,
    "end": 618
  },
  {
    "entity_group": "ORG",
    "score": 0.9001737236976624,
    "word": "sai",
    "start": 684,
    "end": 687
  },
  {
    "entity_group": "LOC",
    "score": 0.9343950748443604,
    "word": "##nsbury's",
    "start": 687,
    "end": 695
  },
  {
    "entity_group": "LOC",
    "score": 0.7310423851013184,
    "word": "victoria centre",
    "start": 708,
    "end": 723
  }
]

But when used via the python API using the following code:
from transformers import AutoTokenizer, AutoModelForTokenClassification

from transformers import pipeline

tokenizer = AutoTokenizer.from_pretrained("dslim/bert-base-NER-uncased")

model = AutoModelForTokenClassification.from_pretrained("dslim/bert-base-NER-uncased")

nlp = pipeline("token-classification", model=model, tokenizer=tokenizer)

example = """00:00:02 Speaker 1: hi john, it's nice to see you again. how was your weekend? do anything special? 00:00:06 Speaker 2: yep, all good thanks. i was with my sister in derby. We saw, you know, that james bond film. what's it called? then got a couple of drinks at the pitcher and piano, back in nottingham. 00:00:18 Speaker 1: that's close to your flat, right? 00:00:25 Speaker 2: yeah, about five minutes away. i live on parliament street, remember? 00:00:39 Speaker 1: of course, i remember. you moved last year after you left your parents' place. 00:00:39 Speaker 2: yeah, it was my sister's birthday on sunday, susie, the older one. i told you last time about that new job she got. sainsbury's, the one by victoria centre."""

ner_results = nlp(example)

print(ner_results)

print(len(ner_results))

I get very different results, here is the output of the code:
[{'entity': 'B-PER', 'score': 0.97784275, 'index': 10, 'word': 'john', 'start': 23, 'end': 27}, {'entity': 'B-LOC', 'score': 0.99292797, 'index': 50, 'word': 'derby', 'start': 166, 'end': 171}, {'entity': 'B-MISC', 'score': 0.8592305, 'index': 59, 'word': 'james', 'start': 196, 'end': 201}, {'entity': 'I-MISC', 'score': 0.5748464, 'index': 60, 'word': 'bond', 'start': 202, 'end': 206}, {'entity': 'B-LOC', 'score': 0.9938424, 'index': 83, 'word': 'nottingham', 'start': 293, 'end': 303}, {'entity': 'B-LOC', 'score': 0.8480199, 'index': 121, 'word': 'parliament', 'start': 420, 'end': 430}, {'entity': 'I-LOC', 'score': 0.973597, 'index': 122, 'word': 'street', 'start': 431, 'end': 437}, {'entity': 'B-PER', 'score': 0.9840036, 'index': 172, 'word': 'susie', 'start': 613, 'end': 618}, {'entity': 'B-ORG', 'score': 0.90017325, 'index': 190, 'word': 'sai', 'start': 684, 'end': 687}, {'entity': 'I-LOC', 'score': 0.93890965, 'index': 191, 'word': '##ns', 'start': 687, 'end': 689}, {'entity': 'I-LOC', 'score': 0.8916274, 'index': 192, 'word': '##bury', 'start': 689, 'end': 693}, {'entity': 'I-LOC', 'score': 0.9475074, 'index': 193, 'word': "'", 'start': 693, 'end': 694}, {'entity': 'I-LOC', 'score': 0.9595369, 'index': 194, 'word': 's', 'start': 694, 'end': 695}, {'entity': 'B-LOC', 'score': 0.55478203, 'index': 199, 'word': 'victoria', 'start': 708, 'end': 716}, {'entity': 'I-LOC', 'score': 0.90730333, 'index': 200, 'word': 'centre', 'start': 717, 'end': 723}]
As can be seen its detecting 15 entities which is way more than the hosted API. And it is even detecting 's as an I-LOC which is very wrong and makes the result unusable.
Why the difference in results? Am I doing something wrong in the code?
Thanks

Context: You need to add agregation_strategy="simple" to your pipeline creation, to tell the pipeline to group together the tokens in the same entities.


MMBT Model (Resnet and BERT) for multimodal embeddings

Question:
Hi! I’m trying to use the librarys implementation of Multimodal Bitransformers (Kiela et all.) on how to classify images and text simultaneously.
I’ve found it hard, because there is very little documentation, and no examples.
In particular i’ve been having a hard time figuring out how to pass the encoded image with the tokenized text to the already intialized model.
If anyone has already worked with this implementation i could really use some help
Thanks

Context: Hey, check out this example 179.


Getting IndexError: list index out of range when fine-tuning

Question:
Hi everyone! I want to fine-tune my pre-trained Longformer model and am getting this error:-
---------------------------------------------------------------------------

IndexError                                Traceback (most recent call last)

<ipython-input-54-2f2d9c2c00fc> in <module>()
     45     )
     46
---> 47 train_results = trainer.train()

6 frames

/usr/local/lib/python3.7/dist-packages/transformers/trainer.py in train(self, resume_from_checkpoint, trial, **kwargs)
   1032             self.control = self.callback_handler.on_epoch_begin(self.args, self.state, self.control)
   1033
-> 1034             for step, inputs in enumerate(epoch_iterator):
   1035
   1036                 # Skip past any already trained steps if resuming training

/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py in __next__(self)
    515             if self._sampler_iter is None:
    516                 self._reset()
--> 517             data = self._next_data()
    518             self._num_yielded += 1
    519             if self._dataset_kind == _DatasetKind.Iterable and \

/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py in _next_data(self)
    555     def _next_data(self):
    556         index = self._next_index()  # may raise StopIteration
--> 557         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    558         if self._pin_memory:
    559             data = _utils.pin_memory.pin_memory(data)

/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

<ipython-input-53-5e4959dcf50c> in __getitem__(self, idx)
      7
      8     def __getitem__(self, idx):
----> 9         item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
     10         item['labels'] = torch.tensor(self.labels[idx])
     11         return item

<ipython-input-53-5e4959dcf50c> in <dictcomp>(.0)
      7
      8     def __getitem__(self, idx):
----> 9         item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
     10         item['labels'] = torch.tensor(self.labels[idx])
     11         return item

IndexError: list index out of range

Evidently, it’s a problem with my tokenization. but I can’t it - for training the LM, I ensured length argument is set for tokenizer:
tokenizer = LongformerTokenizerFast.from_pretrained("./ny_model", max_len=3500)
with a hefty 52000 vocab size. next, when fine-tuning:
train_encodings = tokenizer(list(train_text), truncation=True, padding=True, max_length=3500)
val_encodings = tokenizer(list(val_text), truncation=True, padding=True, max_length=3500)

you can see I truncate the sequences. I tried with some dummy data (ensuring they are of equal length), same problem.
So what could the problem be? Any ideas?

Note that I am fine-tuning the model after uploading the LM on Huggingface.
Also, I have attached the code required to train the LM:------
https://colab.research.google.com/drive/153754DbFXRhKdHvjdSUUp9VSB5JqtZwX?usp=sharing 11

Context: have you figured out the reason? getting the same error here


Freeze Lower Layers with Auto Classification Model

Question:
I’ve been unsuccessful in freezing lower pretrained BERT layers when training a classifier using Huggingface. I’m using AutoModelForSequenceClassification particularly, via code below, and I want to freeze the lower X layers (ex: lower 9 layers). Is this possible in HuggingFace, and if so what code would I add to this for functionality?

tokenizer = AutoTokenizer.from_pretrained(“bert-base-cased”)
def tokenize_function(examples):
return tokenizer(examples[“text”], max_length = 512, padding=“max_length”, truncation=True)
tokenized_train = train.map (tokenize_function, batched=True)
tokenized_test= test.map (tokenize_function, batched=True)
model = AutoModelForSequenceClassification.from_pretrained(“bert-base-cased”, num_labels=1)
for w in model.bert.parameters():
w._trainable= False
training_args = TrainingArguments(“test_trainer”, evaluation_strategy=“epoch”, per_device_train_batch_size=8)
trainer = Trainer(model=model, args=training_args, train_dataset=tokenized_train, eval_dataset=tokenized_test)
trainer.train()

Context:
Yes, in PyTorch freezing layers is quite easy. It can be done as follows:
from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained(“bert-base-cased”, num_labels=1)

for name, param in model.named_parameters():
     if name.startswith("..."): # choose whatever you like here
        param.requires_grad = False


How to use Adaptive Learning rate during training?

Question:
Hi,
I am trying to train a Siamese network and want to incorporate adaptive learning rate during training.
So I have a SiameseTrainer which is a subclass of Trainer class. My question is am I doing something wrong that I get the error
TypeError: Object of type Tensor is not JSON serializable

in this line of my code:
train_result = trainer.train(resume_from_checkpoint=checkpoint)

My code runs for few epochs and then crashes. So what I am doing is below:
def compute_metrics(p: EvalPrediction):
  correct_predictions = p.predictions[1] if isinstance(p.predictions, tuple) else p.predictions
  labels = p.label_ids
  return {
    "MSE": np.mean((correct_predictions-labels)**2)
  }

optimizer = Adafactor(
            model.parameters(),
            scale_parameter=True, # If True, learning rate is scaled by root mean square
        )
lr_scheduler = AdafactorSchedule(optimizer)

trainer = SiameseTrainer(
            model=model,
            args=training_args,
            train_dataset=train_dataset,
            eval_dataset=eval_dataset,
            data_collator=data_collator,
            compute_metrics=compute_metrics,
            tokenizer=tokenizer,
            callbacks=[EarlyStoppingCallback(early_stopping_patience=2)],
            optimizers=(optimizer, lr_scheduler)
)

When I comment out the optimizers=(optimizer, lr_scheduler) parameter and run the code, it does not crash in the middle of epochs.
Appreciate any help!

Context:
Hi, anzaman
I am stuck on the same problem as you do. Have You fixed this one yet?
This is mine error.

image1480×565 44 KB

##########UPDATE
I found a way to trick it by adding logging_steps=9999999 in the trainer argument to make it not log between training , but is there any better solutions for this one.


Question:
The example “use this in Transformers” on facebook/wav2vec2-large-robust · Hugging Face fails with an OSError, seemingly due to a problem with the uploaded model on the Hub.

image1784×532 97.4 KB

Notebook to replicate error: Google Colab
It looks like there is a problem specifically with this model on the hub, because other models work with the same code snippet e.g. if you swap out “wav2vec2-large-robust” for “facebook/wav2vec2-large-960h-lv60-self”, that one works fine.
See also Can't load tokenizer for 'facebook/wav2vec2-large-robust' 1

Context:
Here’s the state of the cache after I try instantiating both tokenizers, the robust one and the 960h one.

image1595×343 99.3 KB

And here’s what happens when I do a search in that directory using ag
~/.cache/huggingface/transformers# ll
total 48
drwxr-xr-x 2 root root 4096 Oct 19 18:32 ./
drwxr-xr-x 3 root root 4096 Oct 19 18:21 ../
-rw-r--r-- 1 root root 1606 Oct 19 18:32 5681e9346f90f9fc4d72503284e96e6bbdc8bf5a38cafeb6ebf3791120b7570d.e0d02e2ed52b244ae1896cccc2beab5caccc2478b8b3d1131c14666c6e14cfdc
-rw-r--r-- 1 root root  153 Oct 19 18:32 5681e9346f90f9fc4d72503284e96e6bbdc8bf5a38cafeb6ebf3791120b7570d.e0d02e2ed52b244ae1896cccc2beab5caccc2478b8b3d1131c14666c6e14cfdc.json
-rwxr-xr-x 1 root root    0 Oct 19 18:32 5681e9346f90f9fc4d72503284e96e6bbdc8bf5a38cafeb6ebf3791120b7570d.e0d02e2ed52b244ae1896cccc2beab5caccc2478b8b3d1131c14666c6e14cfdc.lock*
-rw-r--r-- 1 root root  162 Oct 19 18:32 814e23f251e4a5cd4763cf9b9b6ecb43e43f6a219ec036d9db3419f8dc9d93c3.6685801c836773b383173a1d86dd10317cc4f4eeadcf01f689918a50fdda946b
-rw-r--r-- 1 root root  163 Oct 19 18:32 814e23f251e4a5cd4763cf9b9b6ecb43e43f6a219ec036d9db3419f8dc9d93c3.6685801c836773b383173a1d86dd10317cc4f4eeadcf01f689918a50fdda946b.json
-rwxr-xr-x 1 root root    0 Oct 19 18:31 814e23f251e4a5cd4763cf9b9b6ecb43e43f6a219ec036d9db3419f8dc9d93c3.6685801c836773b383173a1d86dd10317cc4f4eeadcf01f689918a50fdda946b.lock*
-rw-r--r-- 1 root root   85 Oct 19 18:32 de1143309c04207e22168c4563b24770c49eb4e933dbad506eadae8e43a7b422.9d6cd81ef646692fb1c169a880161ea1cb95f49694f220aced9b704b457e51dd
-rw-r--r-- 1 root root  165 Oct 19 18:32 de1143309c04207e22168c4563b24770c49eb4e933dbad506eadae8e43a7b422.9d6cd81ef646692fb1c169a880161ea1cb95f49694f220aced9b704b457e51dd.json
-rwxr-xr-x 1 root root    0 Oct 19 18:32 de1143309c04207e22168c4563b24770c49eb4e933dbad506eadae8e43a7b422.9d6cd81ef646692fb1c169a880161ea1cb95f49694f220aced9b704b457e51dd.lock*
-rw-r--r-- 1 root root  291 Oct 19 18:32 e1f77599caea3f1f7004987f2f7a354d0fd31966b1b6bca5db52b63a8a8cb995.7c838a0a103758bad6ef4922531682da23a8b1c45d25f8d8e7a6d857c0b26544
-rw-r--r-- 1 root root  152 Oct 19 18:32 e1f77599caea3f1f7004987f2f7a354d0fd31966b1b6bca5db52b63a8a8cb995.7c838a0a103758bad6ef4922531682da23a8b1c45d25f8d8e7a6d857c0b26544.json
-rwxr-xr-x 1 root root    0 Oct 19 18:32 e1f77599caea3f1f7004987f2f7a354d0fd31966b1b6bca5db52b63a8a8cb995.7c838a0a103758bad6ef4922531682da23a8b1c45d25f8d8e7a6d857c0b26544.lock*
-rw-r--r-- 1 root root 1583 Oct 19 18:21 f4ed1cd2d2b55e3401644b177d5a166863754c345f98ed09260d0dce9a385d9a.2523c04309986c65617e9a8f2f66c3d656ba969fe07a994af31a3a0cf7b19b78
-rw-r--r-- 1 root root  145 Oct 19 18:21 f4ed1cd2d2b55e3401644b177d5a166863754c345f98ed09260d0dce9a385d9a.2523c04309986c65617e9a8f2f66c3d656ba969fe07a994af31a3a0cf7b19b78.json
-rwxr-xr-x 1 root root    0 Oct 19 18:21 f4ed1cd2d2b55e3401644b177d5a166863754c345f98ed09260d0dce9a385d9a.2523c04309986c65617e9a8f2f66c3d656ba969fe07a994af31a3a0cf7b19b78.lock*
~/.cache/huggingface/transformers# ag 960h
e1f77599caea3f1f7004987f2f7a354d0fd31966b1b6bca5db52b63a8a8cb995.7c838a0a103758bad6ef4922531682da23a8b1c45d25f8d8e7a6d857c0b26544.json
1:{"url": "https://huggingface.co/facebook/wav2vec2-large-960h-lv60-self/resolve/main/vocab.json", "etag": "\"88181b954aa14df68be9b444b3c36585f3078c0a\""}

5681e9346f90f9fc4d72503284e96e6bbdc8bf5a38cafeb6ebf3791120b7570d.e0d02e2ed52b244ae1896cccc2beab5caccc2478b8b3d1131c14666c6e14cfdc.json
1:{"url": "https://huggingface.co/facebook/wav2vec2-large-960h-lv60-self/resolve/main/config.json", "etag": "\"674493ec11ad5d90eaf72d07f69a4bb60203f46b\""}

de1143309c04207e22168c4563b24770c49eb4e933dbad506eadae8e43a7b422.9d6cd81ef646692fb1c169a880161ea1cb95f49694f220aced9b704b457e51dd.json
1:{"url": "https://huggingface.co/facebook/wav2vec2-large-960h-lv60-self/resolve/main/special_tokens_map.json", "etag": "\"25bc39604f72700b3b8e10bd69bb2f227157edd1\""}

814e23f251e4a5cd4763cf9b9b6ecb43e43f6a219ec036d9db3419f8dc9d93c3.6685801c836773b383173a1d86dd10317cc4f4eeadcf01f689918a50fdda946b.json
1:{"url": "https://huggingface.co/facebook/wav2vec2-large-960h-lv60-self/resolve/main/tokenizer_config.json", "etag": "\"97d4216be71590fae568725f363d52f00eb7c944\""}

5681e9346f90f9fc4d72503284e96e6bbdc8bf5a38cafeb6ebf3791120b7570d.e0d02e2ed52b244ae1896cccc2beab5caccc2478b8b3d1131c14666c6e14cfdc
2:  "_name_or_path": "facebook/wav2vec2-large-960h-lv60-self",
~/.cache/huggingface/transformers# ag robust
f4ed1cd2d2b55e3401644b177d5a166863754c345f98ed09260d0dce9a385d9a.2523c04309986c65617e9a8f2f66c3d656ba969fe07a994af31a3a0cf7b19b78.json
1:{"url": "https://huggingface.co/facebook/wav2vec2-large-robust/resolve/main/config.json", "etag": "\"a52cf9097910107f4e0d1bccf82fd4e08d4e4b66\""}


Question:
Hello,
I want to use HateBERT from the paper’s repository 1, which is a BERT model that its pre-training was extended in abusive language.
In order to do that, I created a BERT model (bert-base-uncased in PyTorch) and tried to load HateBERT’s weights with load_state_dict() (after having made minor changes in parameters names, to match BERT’s).
load_state_dict() throws the error:
RuntimeError: Error(s) in loading state_dict for BertModel:
	Missing key(s) in state_dict: "embeddings.position_ids".

which means that the BERT model requires embeddings.position_ids. I checked this tensor and it is not a PyTorch parameters tensor, just a tensor. From the error message, it is also evident that all other parameters match, otherwise their names would also be mentioned. Can someone explain?

Context:
HateBERT is available on the hub. GroNLP/hateBERT · Hugging Face 1
from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("GroNLP/hateBERT")

model = AutoModelForMaskedLM.from_pretrained("GroNLP/hateBERT")


Question:
Hi, I am looking for a tensorflow model that is capable of converting an audio file to text. Can we do this with tensorflow and/or huggingface? The only models I find on the hub are for pytorch…
Thanks!

Context: If you are looking for inference with TF based speech to text model, Here is TFwav2vec2 or are you looking for fine-tuning a TF based model? You can look Here 3 for that.


Prepare data to fine-tune T5 model on unsupervised objective

Question: Hi, I couldn’t find a way to fine-tune the T5 model on a dataset in a specific domain (let’s say medical domain) using the unsupervised objective. Does the current version of Huggingface support this? Basically, all I need is to prepare the dataset to train the T5 model on the unsupervised objective, which could itself be very tricky. Any pointer on this is highly appreciated. P.S: I am looking for something in PyTorch and not Tensorflow. @valhalla @clem

Context: Hi! Have you found a solution? I also can’t figure out how to fine-tune the pretrained model (mT5) on unlabeled domain specific data using Transformer library. Thank you.


Continue pre-training Greek BERT with domain specific dataset

Question:
Hello,
I want to further pre-train Greek BERT in a domain specific dataset and the library provides scripts 35 for this. There is also a BERT model, BertForPreTraining, which has a head for masked language modeling  and a head for next sentence prediction.
Can this model be used for continuing pre-training as well?
If it can should I use the script or the model?

Context:
Hi,
Yes the script is only for masked language modeling (MLM), so you would have to modify this script if you want to also perform next sentence prediction.
But what you could do is the following:

First use the run_mlm.py script to continue pre-training Greek BERT on your domain specific dataset for masked language modeling.
Define a BertForPreTraining model (which includes both the masked language modeling head as well as a sequence classification head), load in the weights of the model that you trained in step 1, and then train on the next sentence prediction task.


Question:
My friend who wishes to remain anonymous asked a good question about T5 that I couldn’t answer:
Say we have a model that predicts sentiment – answers are “positive/negative/neutral” – for something like RoBERTa we’d add a layer, slap on a softmax – and we get both argmax predictions, and some notion of probability amongst the three classes (as well as entropy).
For T5, we just get a text reply.
But of course if we looked at the outputs to the softmax in T5, we’d see p(“positive”) etc – assuming the response is 1 token.
Has anyone tried to do this already or seen examples/notebooks like this?
Ideally, we want  to ask our model several questions. Without worrying too much about the conditional logic, we’d like to be able to measure the probability of text outputs, including some rare categories (that nonetheless are present in our training set). As well as to look for low and high entropy predictions.
If nobody has done this, any code pointers where to look would be helpful.

Context: I’m not sure I understand what is the difference between what you are describing and how the GLUE dataset is handled in the T5 paper


Using onnx for text-generation with GPT-2

Question:
Hi @valhalla @patrickvonplaten , I was working with onnx_transformers 1 and using onnx for GPT-2 model and text-generation task. I used transformer pipeline for text-generation and the runtime for generating text was a bit high(20~30s) and I’ve tried using different approaches like using cronjobs to handle it but it didn’t help. and I found your repo and think of using onnx to accelerate the text generation. As I read the README on the repo there is no text-generation for onnx_transformers. I also used some mehtods in this notebook: Inference_GPT2_with_OnnxRuntime_on_CPU 2 but the qulity of generated text was not even near transformer pipline, would you please give me some insight about this runtime issue and how can I accelerate text-generation besides increasing resources.
Thanks

Context:
Hi,
We’ve recently added an example of exporting BART with ONNX, including beam search generation: transformers/examples/onnx/pytorch/translation at master · huggingface/transformers · GitHub 4
However, it doesn’t include a README right now, which could be very useful to explain how exactly the model can be used. I’ve asked the author to add it.


Question: Has anyone run benchmark studies to evaluate the generation/summarization performance of GPT2 on datasets such as “xsum” ? If so could you share the performance numbers (in-terms of ROUGE scores) you got?  I search for these results online, but couldn’t find any.

Context:
Hi,
I can suggest starting looking here:

      paperswithcode.com

Papers with Code - XSum Dataset 12

  The Extreme Summarization (XSum) dataset is a dataset for evaluation of abstractive single-document summarization systems. The goal is to create a short, one-sentence new summary answering the question “What is the article about?”. The dataset...

I haven’t found neither to be honest. But!
As I believe it seems smarter to use encoder-decoder style models (like PEGASUS or BART) for summarization. Decoder-only language models like GPT were not only trained to continue texts (unlike ~BART), but also they don’t extract the idea of the given texts. Though encoder can be interpreted as an “idea extractor” and the decoder as the generator for natural language text.
P.S. I know one paper that tries to prove the point on summarizing using GPT3 is better than using BART. Even though it heavily relies on Russian language in experiments - you can use the references in the paper to look deeper and find what you are looking for. Paper arxiv link:

      arXiv.org

Fine-tuning GPT-3 for Russian Text Summarization 12

  Automatic summarization techniques aim to shorten and generalize information
given in the text while preserving its core message and the most relevant
ideas. This task can be approached and treated with a variety of methods,
however, not many...

Good luck and let me know if you find anything,
Kirill


Question:
I’m trying to use the new T0 model (bigscience/T0pp · Hugging Face) but when I try following the instructions, I get the following error:
from transformers import AutoTokenizer
from transformers import AutoModelForCausalLM, AutoModelForSeq2SeqLM, GPT2Model, GPT2Config, pipeline
t0_tokenizer = AutoTokenizer.from_pretrained("bigscience/T0pp")
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/home/rschaef/CoCoSci-Language-Distillation/CLIP_prefix_caption/clipgpt_venv/lib/python3.7/site-packages/transformers/models/auto/tokenization_auto.py", line 469, in from_pretrained
    return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/rschaef/CoCoSci-Language-Distillation/CLIP_prefix_caption/clipgpt_venv/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 1742, in from_pretrained
    resolved_vocab_files, pretrained_model_name_or_path, init_configuration, *init_inputs, **kwargs
  File "/home/rschaef/CoCoSci-Language-Distillation/CLIP_prefix_caption/clipgpt_venv/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 1858, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/rschaef/CoCoSci-Language-Distillation/CLIP_prefix_caption/clipgpt_venv/lib/python3.7/site-packages/transformers/models/t5/tokenization_t5_fast.py", line 136, in __init__
    **kwargs,
  File "/home/rschaef/CoCoSci-Language-Distillation/CLIP_prefix_caption/clipgpt_venv/lib/python3.7/site-packages/transformers/tokenization_utils_fast.py", line 117, in __init__
    "Couldn't instantiate the backend tokenizer from one of: \n"
ValueError: Couldn't instantiate the backend tokenizer from one of:
(1) a `tokenizers` library serialization file,
(2) a slow tokenizer instance to convert or
(3) an equivalent slow tokenizer class to instantiate and convert.
You need to have sentencepiece installed to convert a slow tokenizer to a fast one.

Am I missing something from the instructions on that page?

Context: According to python - Transformers v4.x: Convert slow tokenizer to fast tokenizer - Stack Overflow 2, I need to separately install a library called sentencepiece - is that correct?


Question:
Hi! We would like to start a discussion about adding Intel OpenVINO backend in Transformers library.
If you have not heard about OpenVINO before, it’s a library which accelerates deep learning inference (not training, but inference of pretrained models) on Intel Architecture (CPU, GPU, VPU and others). The library is distributed in PyPI and developed in open source.
Currently, there is an issue here: Intel OpenVINO inference backend · Issue #13987 · huggingface/transformers · GitHub 1 (for GitHub discussions) and the latest proposal here: Intel OpenVINO backend by dkurt · Pull Request #1 · dkurt/transformers · GitHub 1
Example (QA):
from transformers import AutoTokenizer, OVAutoModelForQuestionAnswering

tok = AutoTokenizer.from_pretrained("dkurt/bert-large-uncased-whole-word-masking-squad-int8-0001")
model = OVAutoModelForQuestionAnswering.from_pretrained("dkurt/bert-large-uncased-whole-word-masking-squad-int8-0001")

context = """
Soon her eye fell on a little glass box that
was lying under the table: she opened it, and
found in it a very small cake, on which the
words “EAT ME” were beautifully marked in
currants. “Well, I’ll eat it,” said Alice, “ and if
it makes me grow larger, I can reach the key ;
and if it makes me grow smaller, I can creep
under the door; so either way I’ll get into the
garden, and I don’t care which happens !”
"""

question = "Where Alice should go?"

input_ids = tok.encode(question + " " + tok.sep_token + " " + context, return_tensors="pt")

outputs = model(input_ids)

start_pos = outputs.start_logits.argmax()
end_pos = outputs.end_logits.argmax() + 1

answer_ids = input_ids[0, start_pos:end_pos]
answer = tok.convert_tokens_to_string(tok.convert_ids_to_tokens(answer_ids))

print("Question:", question)
print("Answer:", answer)

Context: Opened a pull request at Intel OpenVINO backend (inference only) by dkurt · Pull Request #14203 · huggingface/transformers · GitHub 2. Any feedback welcome!


Question:
A user of Panel asked for help creating a data app based on the GPT2 transformer and bokeh plots Not able to update Bokeh bar plot based on button click which updates source.data - Panel - HoloViz Discourse 1.
I took up the challenge and in an hour I had created this. I can only agree with the prediction

image1920×1080 229 KB

Check out the code Hugging Face GPT2 Transformer Example · GitHub 1.

Context: Does any one know if I can host the app on spaces?


‘BertEncoder’ object has no attribute ‘gradient_checkpointing’

Question:
I’m getting a strange error that previously worked OK. I’m only trying to use a previously trained NLP model to predict a label.
Source Code:
BATCH_SIZE = 128
MAX_LEN = 64
def generate_labels(model, tokenizer, df, segment_label):
  segments = list(df[segment_label].values)
  pred_labels = []
  num_batches = int(len(segments) / BATCH_SIZE) + 1
  print('Total num batches: ', num_batches)
  model.eval()
  with torch.no_grad():
    for i in range(num_batches):
      if (i + 1) % 100 == 0:
        print('Processing batch #', (i + 1))

      batch = segments[i * BATCH_SIZE : (i + 1) * BATCH_SIZE]
      tokenized_text = tokenizer(batch, return_tensors="pt", padding = 'max_length', \
                                             truncation = True, max_length = MAX_LEN)
      input_ids = tokenized_text['input_ids'].to('cuda')
      attention_mask = tokenized_text['attention_mask'].to('cuda')
      outputs = model(input_ids, attention_mask = attention_mask).logits
      pred_labels_batch = torch.argmax(outputs, dim = 1).cpu().numpy()
      pred_labels.extend(pred_labels_batch)
  return pred_labels`Preformatted text`

pred_labels = generate_labels(trained_model, tokenizer, sent_df, 'text')

ERROR:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<command-98410652084852> in <module>
----> 1 pred_labels = generate_labels(trained_model, tokenizer, sent_df, 'text')
      2 sent_df['PREDICTED_DISCOURSE_TAG'] = [pubmed_rct_idx_to_label[x] for x in pred_labels]

<command-4224228297735130> in generate_labels(model, tokenizer, df, segment_label)
     17       input_ids = tokenized_text['input_ids'].to('cuda')
     18       attention_mask = tokenized_text['attention_mask'].to('cuda')
---> 19       outputs = model(input_ids, attention_mask = attention_mask).logits
     20       pred_labels_batch = torch.argmax(outputs, dim = 1).cpu().numpy()
     21       pred_labels.extend(pred_labels_batch)

/databricks/python/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

/databricks/python/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict)
   1528         return_dict = return_dict if return_dict is not None else self.config.use_return_dict
   1529
-> 1530         outputs = self.bert(
   1531             input_ids,
   1532             attention_mask=attention_mask,

/databricks/python/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

/databricks/python/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)
    994             past_key_values_length=past_key_values_length,
    995         )
--> 996         encoder_outputs = self.encoder(
    997             embedding_output,
    998             attention_mask=extended_attention_mask,

/databricks/python/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

/databricks/python/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py in forward(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)
    558             past_key_value = past_key_values[i] if past_key_values is not None else None
    559
--> 560             if self.gradient_checkpointing and self.training:
    561
    562                 if use_cache:

/databricks/python/lib/python3.8/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
   1128             if name in modules:
   1129                 return modules[name]
-> 1130         raise AttributeError("'{}' object has no attribute '{}'".format(
   1131             type(self).__name__, name))
   1132

AttributeError: 'BertEncoder' object has no attribute 'gradient_checkpointing'

No idea where this is coming from. Any ideas or help?
Note that this error goes away completely if I use transformers==4.10.0

Context:
You did not show how you create your model in your post, so no one can help you debug the problem.
Form the error message, it looks like you used torch.save to save your whole model (and not the weights), which is not recommended at all because when the model changes (like it did between 4.10 and 4.11) you then can’t reload it directly with torch.load.
Our advice is to always use save_pretrained/from_pretrained to save/load your models or if it’s not possible, to save the weights (model.state_dict) with torch.save and then reload them with model.load_state_dict, as this will works across different versions of the models.


Document Similarity of long documents e.g. legal contracts

Question:
Is there any way of getting similarities between very long text documents. I know about the ways to get similarity between sentences using sentence transformers but is there a model that can give me a one shot output similar or not. Something like a siamese network that can tell if 2 random images are similar or not. I might be wrong about the analogy but it seems very similar.
If such models don’t exist then is there a method where I can make use of transformers to get similarities between long documents.

Context:
Hi @hemangr8, a very simple thing you can try is:

Split the document into passages or sentences
Embed each passage / sentence as a vector
Take the average of the vectors to get a single vector representation of the document
Compare documents using your favourite similarity metric (e.g. cosine etc)

Depending on the length of your documents, you could also try using the Longformer Encoder-Decoder which has a context size of 16K tokens: allenai/led-large-16384 · Hugging Face 81
If your document fit within the 16K limit you could embed them in one go. There’s some related ideas also in this thread: Summarization on long documents 93


Error when fine-tuning imdb with the script

Question:
Hi,
I’m using the run_glue.py script I found here to fine tune imdb.
I follow the example, training looks fine, eval looks fine but there is no result at the end.
python run_glue.py \
  --model_name_or_path bert-base-cased \
  --dataset_name imdb  \
  --do_train \
  --do_predict \
  --max_seq_length 128 \
  --per_device_train_batch_size 4 \
  --learning_rate 2e-5 \
  --num_train_epochs 1 \
  --output_dir tmp/imdb/

I tried to add --remove_unused_columns False :
python run_glue.py \
  --model_name_or_path bert-base-cased \
  --dataset_name imdb  \
  --do_train \
  --do_predict \
  --max_seq_length 128 \
  --per_device_train_batch_size 4 \
  --learning_rate 2e-5 \
  --num_train_epochs 1 \
  --remove_unused_columns False \
  --output_dir tmp/imdb/

I’m getting this error at the end for each attempt :
10/29/2021 08:07:12 - INFO - __main__ - ***** Predict results None *****
[INFO|modelcard.py:449] 2021-10-29 08:07:12,801 >> Dropping the following result as it does not have all the necessary fields:
{'task': {'name': 'Text Classification', 'type': 'text-classification'}, 'dataset': {'name': 'imdb', 'type': 'imdb', 'args': 'plain_text'}}

I don’t really know what is happening because there is no crash.
Transformers, datasets and tokenizers are up to date.
Thank you.

Context: You need to adapt the script a bit for it to work on the IMDB dataset. It has no “validation” set (the set is named “test”) so you have to change the name of that column.


Does anyone else observer RoBERTa fine-tuning instability?

Question:
I have come across this for two different tasks now, where my setup basically looks as follows:
A smaller dataset for fine-tuning (500k samples), as well as a larger version of this data set (2 million samples).
Hyperparameters are 1000 iterations warmup, 3 epochs training duration, and otherwise default.
Previous runs with the small dataset gave decent results for BERT, and slightly better results with RoBERTa. However, once I go and train with the larger dataset, RoBERTa models no longer show any signs of convergence and instead just predict nonsense. Note that the BERT model still (consistently) performs fine.
This is a problem across several (6) random seeds!
My question now is whether someone else has observed a similar behavior, or whether there are some caveats to the parameters that only let selective models reach a stable training state. Generally the RoBERTa results were better on the smaller data, so obviously I’d like to go with a stable run on the larger data as well.

Context:
hey @dennlinger when you say:

 dennlinger:

However, once I go and train with the larger dataset, RoBERTa models no longer show any signs of convergence and instead just predict nonsense.

do you mean that both the training and validation loss don’t decrease or something else?
the two main parameters i’ve needed to tune XLM-R (not RoBERTa exactly, but close enough) effectively for text classification have been the learning rate and number of warmup steps, with the former generally needing to be in the 10^-6-10-5 range.
in the RoBERTa paper they used 6% of the total training steps for the warmup, so perhaps you could see whether increasing this to say 100,000 steps helps in your 2M samples training set.
what task are you working on?


Using EXTREMELY small dataset to finetune BERT

Question:
Hi, I have a domain-specific language classification problem that I am attempting to use a bert model for.
My approach has been to take the standard pretrained bert model and run further unsupervised learning using domain-specific language corpora (using TSDAE training from the Sentence-Transformer framework 6).
I am now trying to take this domain-trained model and finetune it for a classification task. The problem is I only have an extremely small labelled dataset (~1000 samples), I have been running a few training experiments and surprisingly have received very good results that I am very sceptical of.
The task is to take natural language text and classify it to 1 of 5 classes. Here is my training setup:
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(model.parameters(), lr=0.0001, momentum=0.9)

class BertClassification(nn.Module):
    def __init__(self):
        super(BertClassification, self).__init__()
        self.bert = BertModel.from_pretrained("TSDAE_model/0_Transformer")
        self.to_class = nn.Linear(768, 5)

    def forward(self, x):
        x = self.bert(x)[0][:,0,:]
        x = self.to_class(x)
        return x

And here are the training results:

I am very unsure of how trust worthy these results are as the dataset is so small. I have also tried freezing the bert weights and just training the self.to_class linear layer (~4000 params) but the model peaks at only about 50% accuracy.
I was hoping someone may be able to help me decide if this is an appropriate training strategy for this dataset or if maybe I should look at alternatives. Thanks!

Context:
hey @JoshuaP, are your 5 classes equally balanced? if not, you might be better off charting a metric like the f1-score which tends to be less biased by cases where you have a lot of examples in just a few classes.
another idea would be to implement a baseline (e.g. the classic naive bayes ) and see how that compares against your transformer model.
finally you could try cross-validation (with a stratified split if your classes aren’t balanced) to mitigate some of the problems that come from doing a train/test split with small datasets


Error when trying Deberta for tensorflow version, Can’t import the model

Question:
i found that Deberta is available on tensorflow. i want to try the example for TFDebertaForSequenceClassification but i got error which said like this

Screenshot_1541457×441 44.5 KB

transformers version= 4.5.1

Context: The TensorFlow Deberta model was added more recently, so you will need to update your version of Transformers.


Question:
Hi,
i have tried to combine the ViT (BeiT weights 16patch-384) as Encoder with a Bert Model as Decoder.
(Like Microsofts new arxiv TrOCR paper )

u1301×590 110 KB

If i use the EncoderDecoderModel it does not support pixel_values for the encoder.
feat_extractor = ViTFeatureExtractor.from_pretrained("microsoft/beit-base-patch16-384")
tokenizer = XLMRobertaTokenizer.from_pretrained("microsoft/Multilingual-MiniLM-L12-H384")

encoder = ViTModel.from_pretrained("microsoft/beit-base-patch16-384", output_attentions=True, output_hidden_states=True, return_dict=True, is_decoder=False)

decoder = BertModel.from_pretrained("microsoft/Multilingual-MiniLM-L12-H384", is_decoder=True, add_cross_attention=True, return_dict=True)

encoder_inputs = feat_extractor(torch.randn(3, 512, 512), return_tensors='pt')
decoder_input = tokenizer.encode_plus("Hello World", return_tensors='pt')
print(decoder_input)

#encoder_outputs = encoder(**encoder_inputs)
#decoder_outputs = decoder(input_ids=decoder_input['input_ids'],encoder_hidden_states=encoder_outputs.last_hidden_state)

model = EncoderDecoderModel(encoder=encoder, decoder=decoder)
outputs = model(input_embeds=encoder_inputs['pixel_values'], decoder_input_ids=decoder_input)

  File "/home/felix/anaconda3/envs/work/lib/python3.8/site-packages/transformers/models/encoder_decoder/modeling_encoder_decoder.py", line 425, in forward
    encoder_outputs = self.encoder(
  File "/home/felix/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'input_ids'

And if i try it to combine with different specified:
feat_extractor = ViTFeatureExtractor.from_pretrained("microsoft/beit-base-patch16-384")

tokenizer = XLMRobertaTokenizer.from_pretrained("microsoft/Multilingual-MiniLM-L12-H384")

encoder = ViTModel.from_pretrained("microsoft/beit-base-patch16-384", output_attentions=True, output_hidden_states=True, return_dict=True, is_decoder=False)

decoder = BertModel.from_pretrained("microsoft/Multilingual-MiniLM-L12-H384", is_decoder=True, add_cross_attention=True, return_dict=True)

encoder_inputs = feat_extractor(torch.randn(3, 512, 512), return_tensors='pt')

decoder_input = tokenizer.encode_plus("Hello World", return_tensors='pt')

print(decoder_input)

encoder_outputs = encoder(**encoder_inputs)

decoder_outputs = decoder(input_ids=decoder_input['input_ids'], encoder_hidden_states=encoder_outputs.last_hidden_state)

#model = EncoderDecoderModel(encoder=encoder, decoder=decoder)

#outputs = model(input_embeds=encoder_inputs['pixel_values'], decoder_input_ids=decoder_input)

  File "/home/felix/anaconda3/envs/work/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 990, in forward
    encoder_outputs = self.encoder(
  File "/home/felix/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/felix/anaconda3/envs/work/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 582, in forward
    layer_outputs = layer_module(
  File "/home/felix/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/felix/anaconda3/envs/work/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 494, in forward
    cross_attention_outputs = self.crossattention(
  File "/home/felix/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/felix/anaconda3/envs/work/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 401, in forward
    self_outputs = self.self(
  File "/home/felix/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/felix/anaconda3/envs/work/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py", line 280, in forward
    key_layer = self.transpose_for_scores(self.key(encoder_hidden_states))
  File "/home/felix/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/felix/.local/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 96, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/felix/.local/lib/python3.8/site-packages/torch/nn/functional.py", line 1847, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (577x768 and 384x384)

Is there currently a way to do this with the transformers lib ?
Else i think its also a way to use the ViT model pretrained from transformers and build the Decoder via nn.TransformerDecoder or from scratch and init with a pretrained
Thanks

Context:
Hi,
EncoderDecoderModel is meant to combine any bidirectional text encoder (e.g. BERT) with any autoregressive text decoder (e.g. GPT2). We’re planning to add a VisionEncoderDecoderModel (recently we’ve added SpeechEncoderDecoderModel 6, which allows you to combine any speech autoencoding model such as Wav2Vec2 with any autoregressive text decoder).
Feel free to contribute this if you are interested!


Dealing with T5 for Multiple Choice Task

Question: I wanted to learn how others deal typically with a training text-to-text model for performing multiple choice task such as SWAG. Any pointers will be appreciated.

Context: If anyone is interested: There’s a SWAG notebook in community section notebook that can be used for reference. The input can be context: sentence. 1: option_1, 2: option_2 where labels can be option_numbers.


Using Huggingface for computer vision (Tensorflow)?

Question:
Hi,
I would like to to use huggingface to train some computer vision models. The issue is that I use tensorflow and I can only see pytorch models on the hub.
Is tensorflow supported for computer vision? Is there a TF notebook I could use to train my own models?
Thanks!

Context:
We’re currently adding a TF implementation of the Vision Transformer: Add TFViTModel by ydshieh · Pull Request #13778 · huggingface/transformers · GitHub 5
This will also make it easier to add the other vision models (DeiT, BEiT), which are very similar to ViT.


Are albert-base-v1( and v2) pretrained enough?

Question:
Hi all,
I have questions on albert-base-v1 1 and v2 models uploaded in the huggingface model hub. I’ve checked the MLM loss of albert base models using book corpus dataset and squad context data(which is basically similar to Wikipedia data) based on this example script from transformer repo, in order to validate initial model performance (without additional training). It appears that the average MLM losses are around 2.5(squad context) and 3.2(book corpus) in albert-base-v1, which is much worse than I expected given that those two datasets must have been used for the pertaining albert base model. The value was much worse in v2.
I would like to ask whether albert-base models are trained until the losses converge, or the model has been trained on one or two epochs. Also, it would be great if there is an actual training script and/or MLM loss history on albert pertaining models.
Thanks!
Hojin

Context: The models on the hub are not trained by HuggingFace (unless explicitly mentioned), so the weights are the original implementation weights, ported/converted to the implementation by HF. Whether or not the model is trained “well enough” is a question for the original authors of the model.


BertForMaskedLM’s loss and scores, how the loss is computed?

Question:
I have a simple MaskedLM model with one masked token at position 7. The model returns 20.2516 and 18.0698 as loss and score respectively. However, not sure how the loss is computed from the score. I assumed the loss should be
loss = - log(softmax(score[prediction])
but computing this loss returns 0.0002. I’m confused about how the loss is computed in the model.
import copy
from transformers import BertForMaskedLM, BertTokenizerFast
import torch
model = BertForMaskedLM.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')

text = "Who was Jim Paterson ? Jim Paterson is a doctor".lower()
inputs  =  tokenizer.encode_plus(text,  return_tensors="pt", add_special_tokens = True, truncation=True, pad_to_max_length = True,
                                         return_attention_mask = True,  max_length=64)
input_ids = inputs['input_ids']
masked  = copy.deepcopy(inputs['input_ids'])
masked[0][7] = 103
for t in range(len(masked[0])):
  if masked[0][t] != 103:
    masked[0][t] = -100
loss, scores = model(input_ids = input_ids, attention_mask = inputs['attention_mask'] , token_type_ids=inputs['token_type_ids'] , labels=masked)
print('loss',loss)
print(scores.shape)
pred = torch.argmax( scores[0][7]).item()
print("predicted token:", pred, tokenizer.convert_ids_to_tokens([pred])  )
print("score:", scores[0][7][pred])

logSoftmax = torch.nn.LogSoftmax(dim=1)
NLLLos = torch.nn.NLLLoss()
output = NLLLos( logSoftmax(torch.unsqueeze(logit[0][7], 0)), torch.tensor([pred]))
print(output)

Context:
Hi @sanaz,
I can see few mistakes here

You need to mask tokens in the input_ids not labels. And to prepare lables for masked LM set every position  to -100 (ignore index) except the masked positions.
masked loss is then calculated simply using the CrossEntropy loss between the logits and labels.

So correct usage would be
text = "Who was Jim Paterson ? Jim Paterson is a doctor".lower()
inputs  =  tokenizer([text],  return_tensors="pt")

input_ids = inputs["input_ids"]
# mask the token
input_ids[0][7] = tokenizer.mask_token_id

labels = inputs["input_ids"].clone()
labels[labels != tokenizer.mask_token_id] = -100 # only calculate loss on masked tokens

loss, logits = model(
    input_ids=input_ids,
    labels=labels,
    attention_mask=inputs["attention_mask"],
    token_type_ids=inputs["token_type_ids"]
)
# loss => 18.2054

# calculate loss manually
import torch.nn.functional as F
loss2 = F.cross_entropy(logits.view(-1, tokenizer.vocab_size), labels.view(-1))
# loss2 => 18.2054

Hope this helps.


Hidden States of OpenAI GPT2 inconsistent

Question:
Hi, I am trying to use the OpenAI GPT2 and I just realized that the hidden states change every time I run the model and I cannot figure out why. When I use BertModel this does not happen.
Does anyone have an explanation for that?
Thank you so much in advance!

Context:
Do you run the model in evaluation mode?
i.e. model.eval()
=> this will turn off any dropout modules


Jointly train two-stage models using Trainer

Question:
Hi all,
I want to train a two-stage model containing model1 and model2, in which model2 takes model1’s result as input. For now, I could train model2 properly through Huggingface Seq2SeqTrainer, but I have no clue of how to jointly train model1 and model2 through the Huggingface Trainer. Could someone give me some advice? Thank you very much

Context: I don’t think this is possible. For suc a use case, you should definitely check out Accelerate 2 and use your custom training loop.


I could not able to use save_pretrained on my T5 Model

Question:
I have fine tuned T5 model on a task and i am trying to save the model using save_pretrained  method available for huggingface Model  but i could not able to do so .

image789×323 18.2 KB

can some say what is the error ?

Context:
The class seems pointless because it doesn’t add functionality on top of T5 as currently written. You could have just used T5 as-is. save_pretrained is part of HF subclassed models, not of PyTorch’s nn.Module’s.
Either don’t use this class and just finetune the given T5 model. Or instead of subclassing nn.Module, subclass transformers.PreTrainedModel.


Question:
Hi, everyone~ I have defined my model via huggingface, but I don’t know how to save and load the model, hopefully someone can help me out, thanks!
class MyModel(nn.Module):
  def __init__(self, num_classes):
    super(MyModel, self).__init__()
    self.bert = BertModel.from_pretrained('hfl/chinese-roberta-wwm-ext', return_dict=True).to(device)
    self.fc = nn.Linear(768, num_classes, bias=False)

  def forward(self, x_input_ids, x_type_ids, attn_mask):
    outputs = self.bert(x_input_ids, token_type_ids=x_type_ids, attention_mask=attn_mask)
    pred = self.fc(outputs.pooler_output)
    return pred

model = MyModel(num_classes).to(device)
# save

# load

Context: If you make your model a subclass of PreTrainedModel, then you can use our methods save_pretrained and from_pretrained. Otherwise it’s regular PyTorch code to save and load (using torch.save and torch.load).