There is always something going wrong with hyper parameter tuning

Question:
Hi everyone,
I’ve been trying to do grid search for hyper parameter tuning with the new trainer API and ray tune.
I have to say there is always something messing up during that time.
Has anyone successfully done grid search using Trainer API and BERT?

tune_config = {

        "per_device_train_batch_size": 32,
        "per_device_eval_batch_size": 32,
        "num_train_epochs": tune.choice([2, 3, 4, 5]),
        "weight_decay": tune.uniform(0.0, 0.3),
        "num_epochs": tune.choice([2, 3, 4, 5]),
        #"max_steps": 1 if smoke_test else -1,  # Used for smoke test.
    }

training_args = TrainingArguments("test", eval_steps=500, disable_tqdm=True)

trainer = Trainer(
    args=training_args,
    tokenizer=tokenizer,
    train_dataset=tokenized_datasets_train,
    eval_dataset=tokenized_datasets_val,
    model_init=model_init,
    compute_metrics=compute_metrics,
)

trainer.hyperparameter_search(
    direction="maximize",
    backend="ray",
    hp_space=lambda _: tune_config)

I’ve used ray tune but it tends to give errors like these:

(pid=991) 2021-08-30 12:30:53,900	ERROR function_runner.py:266 -- Runner Thread raised error.
(pid=991) Traceback (most recent call last):
(pid=991)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 260, in run
(pid=991)     self._entrypoint()
(pid=991)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 329, in entrypoint
(pid=991)     self._status_reporter.get_checkpoint())
(pid=991)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 594, in _trainable_func
(pid=991)     output = fn()
(pid=991)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/utils/trainable.py", line 344, in inner
(pid=991)     trainable(config, **fn_kwargs)
(pid=991)   File "/usr/local/lib/python3.7/dist-packages/transformers/integrations.py", line 162, in _objective
(pid=991)     local_trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
(pid=991)   File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1031, in train
(pid=991)     self._hp_search_setup(trial)
(pid=991)   File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 860, in _hp_search_setup
(pid=991)     f"Trying to set {key} in the hyperparameter search but there is no corresponding field in `TrainingArguments`."
(pid=991) AttributeError: Trying to set num_epochs in the hyperparameter search but there is no corresponding field in `TrainingArguments`.
(pid=991) Exception in thread Thread-2:
(pid=991) Traceback (most recent call last):
(pid=991)   File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
(pid=991)     self.run()
(pid=991)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 279, in run
(pid=991)     raise e
(pid=991)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 260, in run
(pid=991)     self._entrypoint()
(pid=991)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 329, in entrypoint
(pid=991)     self._status_reporter.get_checkpoint())
(pid=991)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 594, in _trainable_func
(pid=991)     output = fn()
(pid=991)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/utils/trainable.py", line 344, in inner
(pid=991)     trainable(config, **fn_kwargs)
(pid=991)   File "/usr/local/lib/python3.7/dist-packages/transformers/integrations.py", line 162, in _objective
(pid=991)     local_trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
(pid=991)   File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1031, in train
(pid=991)     self._hp_search_setup(trial)
(pid=991)   File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 860, in _hp_search_setup
(pid=991)     f"Trying to set {key} in the hyperparameter search but there is no corresponding field in `TrainingArguments`."
(pid=991) AttributeError: Trying to set num_epochs in the hyperparameter search but there is no corresponding field in `TrainingArguments`.
(pid=991)
2021-08-30 12:30:54,010	ERROR trial_runner.py:773 -- Trial _objective_19f77_00000: Error processing event.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trial_runner.py", line 739, in _process_trial
    results = self.trial_executor.fetch_result(trial)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/ray_trial_executor.py", line 746, in fetch_result
    result = ray.get(trial_future[0], timeout=DEFAULT_GET_TIMEOUT)
  File "/usr/local/lib/python3.7/dist-packages/ray/_private/client_mode_hook.py", line 82, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/ray/worker.py", line 1621, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(TuneError): ray::ImplicitFunc.train_buffered() (pid=991, ip=172.28.0.2, repr=<ray.tune.function_runner.ImplicitFunc object at 0x7f208feca750>)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trainable.py", line 178, in train_buffered
    result = self.train()
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trainable.py", line 237, in train
    result = self.step()
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 379, in step
    self._report_thread_runner_error(block=True)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 527, in _report_thread_runner_error
    ("Trial raised an exception. Traceback:\n{}".format(err_tb_str)
ray.tune.error.TuneError: Trial raised an exception. Traceback:
ray::ImplicitFunc.train_buffered() (pid=991, ip=172.28.0.2, repr=<ray.tune.function_runner.ImplicitFunc object at 0x7f208feca750>)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 260, in run
    self._entrypoint()
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 329, in entrypoint
    self._status_reporter.get_checkpoint())
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 594, in _trainable_func
    output = fn()
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/utils/trainable.py", line 344, in inner
    trainable(config, **fn_kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/integrations.py", line 162, in _objective
    local_trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1031, in train
    self._hp_search_setup(trial)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 860, in _hp_search_setup
    f"Trying to set {key} in the hyperparameter search but there is no corresponding field in `TrainingArguments`."
AttributeError: Trying to set num_epochs in the hyperparameter search but there is no corresponding field in `TrainingArguments`.
Result for _objective_19f77_00000:
  {}

== Status ==
Memory usage on this node: 5.0/25.5 GiB
Using FIFO scheduling algorithm.
Resources requested: 1.0/4 CPUs, 1.0/1 GPUs, 0.0/15.0 GiB heap, 0.0/7.5 GiB objects (0.0/1.0 accelerator_type:T4)
Result logdir: /root/ray_results/_objective_2021-08-30_12-30-46
Number of trials: 18/20 (1 ERROR, 16 PENDING, 1 RUNNING)
+------------------------+----------+-------+--------------+--------------------+----------------+
| Trial name             | status   | loc   |   num_epochs |   num_train_epochs |   weight_decay |
|------------------------+----------+-------+--------------+--------------------+----------------|
| _objective_19f77_00001 | RUNNING  |       |            2 |                  4 |     0.233907   |
| _objective_19f77_00002 | PENDING  |       |            4 |                  4 |     0.13375    |
| _objective_19f77_00003 | PENDING  |       |            2 |                  4 |     0.137775   |
| _objective_19f77_00004 | PENDING  |       |            4 |                  5 |     0.04286    |
| _objective_19f77_00005 | PENDING  |       |            5 |                  3 |     0.0169235  |
| _objective_19f77_00006 | PENDING  |       |            3 |                  5 |     0.281566   |
| _objective_19f77_00007 | PENDING  |       |            2 |                  5 |     0.297663   |
| _objective_19f77_00008 | PENDING  |       |            2 |                  5 |     0.183496   |
| _objective_19f77_00009 | PENDING  |       |            4 |                  5 |     0.00691873 |
| _objective_19f77_00010 | PENDING  |       |            5 |                  4 |     0.119958   |
| _objective_19f77_00011 | PENDING  |       |            4 |                  5 |     0.292127   |
| _objective_19f77_00012 | PENDING  |       |            3 |                  3 |     0.0271819  |
| _objective_19f77_00013 | PENDING  |       |            5 |                  4 |     0.114739   |
| _objective_19f77_00014 | PENDING  |       |            2 |                  5 |     0.140029   |
| _objective_19f77_00015 | PENDING  |       |            2 |                  4 |     0.204092   |
| _objective_19f77_00016 | PENDING  |       |            2 |                  4 |     0.00397949 |
| _objective_19f77_00017 | PENDING  |       |            3 |                  5 |     0.168986   |
| _objective_19f77_00000 | ERROR    |       |            4 |                  4 |     0.238963   |
+------------------------+----------+-------+--------------+--------------------+----------------+
Number of errored trials: 1
+------------------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Trial name             |   # failures | error file                                                                                |
|------------------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------|
| _objective_19f77_00000 |            1 | /root/ray_results/_objective_2021-08-30_12-30-46/_objective_19f77_00000_0_num_epochs=4,num_train_epochs=4,weight_decay=0.23896_2021-08-30_12-30-46/error.txt |
+------------------------+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+

(pid=989) 2021-08-30 12:31:01,040	ERROR function_runner.py:266 -- Runner Thread raised error.
(pid=989) Traceback (most recent call last):
(pid=989)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 260, in run
(pid=989)     self._entrypoint()
(pid=989)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 329, in entrypoint
(pid=989)     self._status_reporter.get_checkpoint())
(pid=989)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 594, in _trainable_func
(pid=989)     output = fn()
(pid=989)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/utils/trainable.py", line 344, in inner
(pid=989)     trainable(config, **fn_kwargs)
(pid=989)   File "/usr/local/lib/python3.7/dist-packages/transformers/integrations.py", line 162, in _objective
(pid=989)     local_trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
(pid=989)   File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1031, in train
(pid=989)     self._hp_search_setup(trial)
(pid=989)   File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 860, in _hp_search_setup
(pid=989)     f"Trying to set {key} in the hyperparameter search but there is no corresponding field in `TrainingArguments`."
(pid=989) AttributeError: Trying to set num_epochs in the hyperparameter search but there is no corresponding field in `TrainingArguments`.
(pid=989) Exception in thread Thread-2:
(pid=989) Traceback (most recent call last):
(pid=989)   File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
(pid=989)     self.run()
(pid=989)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 279, in run
(pid=989)     raise e
(pid=989)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 260, in run
(pid=989)     self._entrypoint()
(pid=989)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 329, in entrypoint
(pid=989)     self._status_reporter.get_checkpoint())
(pid=989)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 594, in _trainable_func
(pid=989)     output = fn()
(pid=989)   File "/usr/local/lib/python3.7/dist-packages/ray/tune/utils/trainable.py", line 344, in inner
(pid=989)     trainable(config, **fn_kwargs)
(pid=989)   File "/usr/local/lib/python3.7/dist-packages/transformers/integrations.py", line 162, in _objective
(pid=989)     local_trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
(pid=989)   File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1031, in train
(pid=989)     self._hp_search_setup(trial)
(pid=989)   File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 860, in _hp_search_setup
(pid=989)     f"Trying to set {key} in the hyperparameter search but there is no corresponding field in `TrainingArguments`."
(pid=989) AttributeError: Trying to set num_epochs in the hyperparameter search but there is no corresponding field in `TrainingArguments`.
(pid=989)
2021-08-30 12:31:01,211	ERROR trial_runner.py:773 -- Trial _objective_19f77_00001: Error processing event.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trial_runner.py", line 739, in _process_trial
    results = self.trial_executor.fetch_result(trial)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/ray_trial_executor.py", line 746, in fetch_result
    result = ray.get(trial_future[0], timeout=DEFAULT_GET_TIMEOUT)
  File "/usr/local/lib/python3.7/dist-packages/ray/_private/client_mode_hook.py", line 82, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/ray/worker.py", line 1621, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(TuneError): ray::ImplicitFunc.train_buffered() (pid=989, ip=172.28.0.2, repr=<ray.tune.function_runner.ImplicitFunc object at 0x7f4ad6a4cc50>)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trainable.py", line 178, in train_buffered
    result = self.train()
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trainable.py", line 237, in train
    result = self.step()
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 379, in step
    self._report_thread_runner_error(block=True)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 527, in _report_thread_runner_error
    ("Trial raised an exception. Traceback:\n{}".format(err_tb_str)
ray.tune.error.TuneError: Trial raised an exception. Traceback:
ray::ImplicitFunc.train_buffered() (pid=989, ip=172.28.0.2, repr=<ray.tune.function_runner.ImplicitFunc object at 0x7f4ad6a4cc50>)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 260, in run
    self._entrypoint()
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 329, in entrypoint
    self._status_reporter.get_checkpoint())
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/function_runner.py", line 594, in _trainable_func
    output = fn()
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/utils/trainable.py", line 344, in inner
    trainable(config, **fn_kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/integrations.py", line 162, in _objective
    local_trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1031, in train
    self._hp_search_setup(trial)
  File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 860, in _hp_search_setup
    f"Trying to set {key} in the hyperparameter search but there is no corresponding field in `TrainingArguments`."
AttributeError: Trying to set num_epochs in the hyperparameter search but there is no corresponding field in `TrainingArguments`.
Result for _objective_19f77_00001:
  {}

Let me know, please!

Context:
The traceback is telling you

Trying to set num_epochs in the hyperparameter search but there is no corresponding field in TrainingArguments

So you problably should set some default value there, which will then be overwritten by ray during search.


Predicted Start_index < Predicted End_index in BertForQuestionAnswering

Question:
We want to fine-tuned a QA model, which is based on BertForQuestionAnswering.
After training, we can get a span-start/end scores by input_ids/token_type_ids/attention_mask and choose the indices with maximum span-start/end scores as predicted start_index and predicted end_index .
But, sometimes predicted start_index would less than predicted end_index .
If any reasonable method to solve this situation, thanks~
Ex:
span-start scores = [-0.1, -2.1, 0.7, 1.3, 4.1]
span-end scores = [-0.7, 3, 5, -0.7, 3.3]
=>
predicted start_index = 4
predicted end_index = 2
It is not reasonable.

Context: Those predictions are not considered usually.


Cross Entropy Loss and loss of HuggingFace T5ForConditionalGeneration does not matches

Question:
Hello, I am using T5ForConditionalGeneration for Question & Answering Model and Finetuning it, but In the train step, hugginface loss and my loss is not being matched, I want it for some experiment purpose.
class UQAFineTuneModel(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.model = T5ForConditionalGeneration.from_pretrained(
            "allenai/unifiedqa-t5-small", return_dict=True
        )
        self.model.train()
    def forward(
        self,
        source_text_input_ids,
        source_text_attention_mask,
        target_text_input_ids=None,
    ):
        output = self.model(
            input_ids=source_text_input_ids,
            attention_mask=source_text_attention_mask,
            labels=target_text_input_ids,
        )
        return output.loss, output.logits

    def training_step(self, batch, batch_idx):
        source_text_input_ids = batch["source_text_input_ids"]
        source_text_attention_mask = batch["source_text_attention_mask"]
        target_text_input_ids = batch["target_text_input_ids"]
        # labels_attention_mask = batch["target_text_attention_mask"]
        loss, outputs = self(
            source_text_input_ids, source_text_attention_mask, target_text_input_ids
        )
        loss_mine = None

        output = self.model(
            input_ids=source_text_input_ids,
            attention_mask=source_text_attention_mask,
            labels=target_text_input_ids,
        )
        labels = batch["target_text_input_ids"].clone()
        labels[labels == 0] = -100
        if target_text_input_ids is not None:
            loss_fct = CrossEntropyLoss(ignore_index=-100)
            loss_mine = loss_fct(output.logits.view(-1, outputs.size(-1)), labels.view(-1))
            print(f"loss_huggingface: {loss.item()}, loss_mine : {loss_mine.item()}")
        self.log("train_loss", loss, prog_bar=True, logger=True)
        return {"loss": loss, "predictions": outputs}

But my loss is different then huggingface loss, however they both are using CrossEntropy,

image1027×274 14.8 KB

Context: @valhalla , @BramVanroy  Plz have a look at this


Question:
How to apply TranslationPipeline from English to Brazilian Portuguese?
I’ve tried the fowling approach with no success:
from transformers import pipeline

translator = pipeline(
    model="t5-small",
    task="translation_en_to_br"
    )

translator("How old are you?", src_lang="en", tgt_lang="br")
# [{'translation_text': '         '}]

Could you give me some directions?

Context: As far as I can tell, T5 has only been trained/finetuned on English, German, French, Romanian. You can read Section 3.1.3 in their paper. I am not aware of Brazilian Portuguese models. Also, I don’t think it has an official language code so “br” is not likely to work anyway.


Problems Subclassing Trainer Class for Custom Evaluation Loop

Question:
Hello Everybody,
While training my model with deepspeed on 4GPUs, I was trying to Inject some custom behaviour in the evaluation loop.
According to the Trainer docs under evaluate function it says.

You can also subclass and override this method to inject custom behavior

However when I tried doing this, I get the following Error:
Traceback (most recent call last):
  File "GPT2nr.py", line 109, in <module>
Traceback (most recent call last):
  File "GPT2nr.py", line 109, in <module>
    values = trainer.evaluate()
  File "/home/dagbert-skunkworks/src/skunkworks/models/GPT2Trainer.py", line 42, in evaluate
    values = trainer.evaluate()
  File "/home/dagbert-skunkworks/src/skunkworks/models/GPT2Trainer.py", line 42, in evaluate
    eval_loop = self.prediction_loop if self.args.use_legacy_prediction_loop else self.evaluation_loop
AttributeError: 'TrainingArguments' object has no attribute 'use_legacy_prediction_loop'
    eval_loop = self.prediction_loop if self.args.use_legacy_prediction_loop else self.evaluation_loop
AttributeError: 'TrainingArguments' object has no attribute 'use_legacy_prediction_loop'
Traceback (most recent call last):
  File "GPT2nr.py", line 109, in <module>
    values = trainer.evaluate()
  File "/home/dagbert-skunkworks/src/skunkworks/models/GPT2Trainer.py", line 42, in evaluate
    eval_loop = self.prediction_loop if self.args.use_legacy_prediction_loop else self.evaluation_loop
AttributeError: 'TrainingArguments' object has no attribute 'use_legacy_prediction_loop'
Traceback (most recent call last):
  File "GPT2nr.py", line 109, in <module>
    values = trainer.evaluate()
  File "/home/dagbert-skunkworks/src/skunkworks/models/GPT2Trainer.py", line 42, in evaluate
    eval_loop = self.prediction_loop if self.args.use_legacy_prediction_loop else self.evaluation_loop
AttributeError: 'TrainingArguments' object has no attribute 'use_legacy_prediction_loop'
Killing subprocess 136
Killing subprocess 137
Killing subprocess 138
Killing subprocess 139
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/dist-packages/deepspeed/launcher/launch.py", line 171, in <module>
    main()
  File "/usr/local/lib/python3.6/dist-packages/deepspeed/launcher/launch.py", line 161, in main
    sigkill_handler(signal.SIGTERM, None)  # not coming back
  File "/usr/local/lib/python3.6/dist-packages/deepspeed/launcher/launch.py", line 139, in sigkill_handler
    raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python', '-u', 'GPT2nr.py', '--local_rank=3']' returned non-zero exit status 1.

GPT2Trainer.py
import math
import time
from typing import Dict, List, NamedTuple, Optional, Tuple, Union

import numpy as np
from datasets import Dataset
from torch.utils.data.dataloader import DataLoader
from transformers.trainer import Trainer

class EvalPrediction(NamedTuple):
    predictions: Union[np.ndarray, Tuple[np.ndarray]]
    label_ids: np.ndarray
    classes: List

class EvalLoopOutput(NamedTuple):
    predictions: Union[np.ndarray, Tuple[np.ndarray]]
    label_ids: Optional[np.ndarray]
    metrics: Optional[Dict[str, float]]
    num_samples: Optional[int]
    classes: Optional[List]

class GPT2Trainer(Trainer):
    def __init__(self, model,args = None,data_collator = None,train_dataset = None,eval_dataset = None,tokenizer = None,
            model_init = None,compute_metrics = None,callbacks = None,optimizers = (None,None)):

            super(GPT2Trainer,self).__init__(model, args, data_collator, train_dataset, eval_dataset, tokenizer, model_init,
              compute_metrics, callbacks, optimizers)

    def evaluate(self,
                eval_dataset: Optional[Dataset] = None,
                ignore_keys: Optional[List[str]] = None,
                metric_key_prefix: str = "eval",) -> Dict[str, float]:

        self._memory_tracker.start()

        eval_dataloader = self.get_eval_dataloader(eval_dataset)
        start_time = time.time()

        eval_loop = self.prediction_loop if self.args.use_legacy_prediction_loop else self.evaluation_loop
        output = eval_loop(
            eval_dataloader,
            description="Evaluation",
            # No point gathering the predictions if there are no metrics, otherwise we defer to
            # self.args.prediction_loss_only
            prediction_loss_only=True if self.compute_metrics is None else None,
            ignore_keys=ignore_keys,
            metric_key_prefix=metric_key_prefix,
        )

        total_batch_size = self.args.eval_batch_size * self.args.world_size

        output.metrics.update(
            speed_metrics(
                metric_key_prefix,
                start_time,
                num_samples=output.num_samples,
                num_steps=math.ceil(output.num_samples / total_batch_size),
            )
        )

        self.log(output.metrics)

        if DebugOption.TPU_METRICS_DEBUG in self.args.debug:
            # tpu-comment: Logging debug metrics for PyTorch/XLA (compile, execute times, ops, etc.)
            xm.master_print(met.metrics_report())

        self.control = self.callback_handler.on_evaluate(self.args, self.state, self.control, output.metrics)

        self._memory_tracker.stop_and_update_metrics(output.metrics)

        return output.metrics

    def evaluation_loop(
        self,
        dataloader: DataLoader,
        description: str,
        prediction_loss_only: Optional[bool] = None,
        ignore_keys: Optional[List[str]] = None,
        metric_key_prefix: str = "eval",
    ) -> EvalLoopOutput:
        """
        Prediction/evaluation loop, shared by :obj:`Trainer.evaluate()` and :obj:`Trainer.predict()`.

        Works both with or without labels.
        """
        prediction_loss_only = (
            prediction_loss_only if prediction_loss_only is not None else self.args.prediction_loss_only
        )

        # if eval is called w/o train init deepspeed here
        if self.args.deepspeed and not self.deepspeed:

            # XXX: eval doesn't have `resume_from_checkpoint` arg but we should be able to do eval
            # from the checkpoint eventually
            deepspeed_engine, _, _ = deepspeed_init(self, num_training_steps=0, resume_from_checkpoint=None)
            self.model = deepspeed_engine.module
            self.model_wrapped = deepspeed_engine
            self.deepspeed = deepspeed_engine
            # XXX: we don't need optim/sched for inference, but this needs to be sorted out, since
            # for example the Z3-optimizer is a must for zero3 to work even for inference - what we
            # don't need is the deepspeed basic optimizer which is self.optimizer.optimizer
            deepspeed_engine.optimizer.optimizer = None
            deepspeed_engine.lr_scheduler = None

        model = self._wrap_model(self.model, training=False)

        # if full fp16 is wanted on eval and this ``evaluation`` or ``predict`` isn't called while
        # ``train`` is running, halve it first and then put on device
        if not self.is_in_train and self.args.fp16_full_eval:
            model = model.half().to(self.args.device)

        batch_size = dataloader.batch_size

        logger.info(f"***** Running {description} *****")
        if isinstance(dataloader.dataset, collections.abc.Sized):
            logger.info(f"  Num examples = {self.num_examples(dataloader)}")
        else:
            logger.info("  Num examples: Unknown")
        logger.info(f"  Batch size = {batch_size}")

        model.eval()

        self.callback_handler.eval_dataloader = dataloader
        # Do this before wrapping.
        eval_dataset = dataloader.dataset

        if is_torch_tpu_available():
            dataloader = pl.ParallelLoader(dataloader, [self.args.device]).per_device_loader(self.args.device)

        if self.args.past_index >= 0:
            self._past = None

        # Initialize containers
        # losses/preds/labels on GPU/TPU (accumulated for eval_accumulation_steps)
        losses_host = None
        preds_host = None
        labels_host = None
        # losses/preds/labels on CPU (final containers)
        all_losses = None
        all_preds = None
        all_labels = None
        # Will be useful when we have an iterable dataset so don't know its length.

        observed_num_examples = 0
        # Main evaluation loop
        for step, inputs in enumerate(dataloader):
            # Update the observed num examples
            inputs, class_labels = inputs.get('input_ids'),inputs.get('labels')
            observed_batch_size = find_batch_size(inputs)
            if observed_batch_size is not None:
                observed_num_examples += observed_batch_size

            # Prediction step
            loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)

            # Update containers on host
            if loss is not None:
                losses = self._nested_gather(loss.repeat(batch_size))
                losses_host = losses if losses_host is None else torch.cat((losses_host, losses), dim=0)
            if logits is not None:
                logits = self._pad_across_processes(logits)
                logits = self._nested_gather(logits)
                preds_host = logits if preds_host is None else nested_concat(preds_host, logits, padding_index=-100)
            if labels is not None:
                labels = self._pad_across_processes(labels)
                labels = self._nested_gather(labels)
                labels_host = labels if labels_host is None else nested_concat(labels_host, labels, padding_index=-100)
            self.control = self.callback_handler.on_prediction_step(self.args, self.state, self.control)

            # Gather all tensors and put them back on the CPU if we have done enough accumulation steps.
            if self.args.eval_accumulation_steps is not None and (step + 1) % self.args.eval_accumulation_steps == 0:
                if losses_host is not None:
                    losses = nested_numpify(losses_host)
                    all_losses = losses if all_losses is None else np.concatenate((all_losses, losses), axis=0)
                if preds_host is not None:
                    logits = nested_numpify(preds_host)
                    all_preds = logits if all_preds is None else nested_concat(all_preds, logits, padding_index=-100)
                if labels_host is not None:
                    labels = nested_numpify(labels_host)
                    all_labels = (
                        labels if all_labels is None else nested_concat(all_labels, labels, padding_index=-100)
                    )

                # Set back to None to begin a new accumulation
                losses_host, preds_host, labels_host = None, None, None

        if self.args.past_index and hasattr(self, "_past"):
            # Clean the state at the end of the evaluation loop
            delattr(self, "_past")

        # Gather all remaining tensors and put them back on the CPU
        if losses_host is not None:
            losses = nested_numpify(losses_host)
            all_losses = losses if all_losses is None else np.concatenate((all_losses, losses), axis=0)
        if preds_host is not None:
            logits = nested_numpify(preds_host)
            all_preds = logits if all_preds is None else nested_concat(all_preds, logits, padding_index=-100)
        if labels_host is not None:
            labels = nested_numpify(labels_host)
            all_labels = labels if all_labels is None else nested_concat(all_labels, labels, padding_index=-100)

        # Number of samples
        if not isinstance(eval_dataset, IterableDataset):
            num_samples = len(eval_dataset)
        # The instance check is weird and does not actually check for the type, but whether the dataset has the right
        # methods. Therefore we need to make sure it also has the attribute.
        elif isinstance(eval_dataset, IterableDatasetShard) and hasattr(eval_dataset, "num_examples"):
            num_samples = eval_dataset.num_examples
        else:
            num_samples = observed_num_examples

        # Number of losses has been rounded to a multiple of batch_size and in a distributed training, the number of
        # samplers has been rounded to a multiple of batch_size, so we truncate.
        if all_losses is not None:
            all_losses = all_losses[:num_samples]
        if all_preds is not None:
            all_preds = nested_truncate(all_preds, num_samples)
        if all_labels is not None:
            all_labels = nested_truncate(all_labels, num_samples)

        # Metrics!
        if self.compute_metrics is not None and all_preds is not None and all_labels is not None:
            metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels,classes=class_labels))
        else:
            metrics = {}

        # To be JSON-serializable, we need to remove numpy types or zero-d tensors
        metrics = denumpify_detensorize(metrics)

        if all_losses is not None:
            metrics[f"{metric_key_prefix}_loss"] = all_losses.mean().item()

        # Prefix all keys with metric_key_prefix + '_'
        for key in list(metrics.keys()):
            if not key.startswith(f"{metric_key_prefix}_"):
                metrics[f"{metric_key_prefix}_{key}"] = metrics.pop(key)

        return EvalLoopOutput(predictions=all_preds, label_ids=all_labels, metrics=metrics, num_samples=num_samples)

GPT2nr.py
import json
from typing import Dict, List, Optional

import deepspeed
import numpy as np
import pandas as pd
import torch
from datasets import Dataset, load_dataset, load_metric
from skunkworks.models.GPT2Trainer import (EvalLoopOutput, EvalPrediction,
                                                GPT2Trainer)
from transformers import (GPT2LMHeadModel, GPT2TokenizerFast, Trainer,
                          TrainingArguments)
from transformers.trainer_utils import get_last_checkpoint, is_main_process

def compute_metrics(eval_pred: EvalPrediction) -> Dict:
    logits, label_ids,clas_labels = eval_pred
    logits = torch.Tensor(logits)
    logits = torch.argmax(logits,axis=-1)
    predictions = tokenizer.batch_decode(logits)
    label_ids = tokenizer.batch_decode(label_ids)
    metric_values = metric.compute(predictions=predictions, references=label_ids)
    avg_divergence_df = pd.DataFrame({"labels":class_labels,"scores":metric_values['raw_scores']})
    return metric_values

deepspeed_dict = json.load(open('ds_config_zero2manual.json','r'))
metric = load_metric("../metrics/hf_metric.py")
block_size = 128

def tokenize_function(examples,field):
    return tokenizer(examples[field], padding="max_length", truncation=True)

def group_texts(examples):
    # Concatenate all texts.
    concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
    total_length = len(concatenated_examples[list(examples.keys())[0]])
    # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
        # customize this part to your needs.
    total_length = (total_length // block_size) * block_size
    # Split by chunks of max_len.
    result = {
        k: [t[i : i + block_size] for i in range(0, total_length, block_size)]
        for k, t in concatenated_examples.items()
    }
    result["labels"] = result["input_ids"].copy()
    return result

model = GPT2LMHeadModel.from_pretrained('gpt2-medium')
tokenizer = GPT2TokenizerFast.from_pretrained('gpt2-medium')
tokenizer.pad_token = tokenizer.eos_token

train_dataset = load_dataset('csv', data_files={'train': 'train.csv'})

test_dataset = load_dataset('csv', data_files={'test': 'test.csv'})

tokenized_train_dataset = train_dataset.map(lambda x: tokenize_function(x,'text'), batched=True, num_proc=4, remove_columns=["text"])
tokenized_train_dataset = tokenized_train_dataset.map(group_texts,batched=True,batch_size=1,num_proc=4)

tokenized_test_dataset = test_dataset.map(lambda x: tokenize_function(x,'data'), batched=True, num_proc=4, remove_columns=["data"])

training_args = TrainingArguments("test-trainer",per_device_train_batch_size=8,
                                    per_device_eval_batch_size=8,
                                    num_train_epochs=1,
                                    learning_rate=2e-5,
                                    weight_decay=0.01,
                                    eval_accumulation_steps=2,
                                    fp16=True,
                                    deepspeed=deepspeed_dict)

trainer = GPT2Trainer(
    model=model,args=training_args,
    train_dataset=tokenized_train_dataset["train"],
     eval_dataset=tokenized_test_dataset["test"],
     tokenizer=tokenizer,
     compute_metrics=compute_metrics
     )

trainer.train()
values = trainer.evaluate()

Transformers Version: 4.5.2
Training GPUs: 4
Training GPU Model: Tesla T4

Context: It looks like you are not using the latest version of Transformers.


Question:
I am using the BertTokenizerFast from transformers library to encode my text data. I am using pre-trained model for that “bert-base-uncased”. I have a data set of 7 Million rows which is around 2GB. I am using 64GB RAM.
When I am trying to convert into vectors using the model BertModel with same pre-trained model"bert-base-uncased", It cannot convert even 10,000 encoded vectors. It is allocating by saying very high memory needed (100s of GB).
I am using the reference code from here reference blog 3

Context: You are probably trying to do “all data at once” which of course will not work. Try chunking your data into batches and process them one by one.


Cardinality issue when training bert from scratch (tensorflow)

Question:
Hello there!
I am trying to adapt the official  Google Colab for language generation to tensorflow and everything seems to work wonderfully by simply appending TF to most of the huggingface function calls (TFAutoModel, etc)
Unfortunately, this strategy fails at the training step:
from transformers import TFTrainer, TFTrainingArguments
import tensorflow as tf

training_args = TFTrainingArguments(
    "test-clm",
    evaluation_strategy = "epoch",
    learning_rate=2e-5)

trainer = TFTrainer(
    model=model,
    args = training_args,
    train_dataset=lm_datasets[0:1000],
    eval_dataset=lm_datasets[1000:])

trainer.train()

 self.num_train_examples = self.train_dataset.cardinality().numpy()
AttributeError: 'dict' object has no attribute 'cardinality'

I have absolutely no idea what this cardinality is. Do you know what the issue can be?
Thanks!

Context:
I saw a related issue here TensorFlow Question-Answering example fails to run (cardinality error) · Issue #10246 · huggingface/transformers · GitHub 13 … huggingface masters, do you have an idea?
Thanks!


How to structure labels for token classification?

Question:
The documentation for the label parameter for BertForTokenClassification says that

Indices should be in  [0, ..., config.num_labels - 1]

But BertConfig doesn’t have a num_labels parameter as far as I can tell, so what is this config.num_labels argument?
Also, in this 3 tutorial it says that we can set the labels we want the model to ignore, to -100. If that is correct, why doesn’t the documentation for BertForTokenClassification mention it? Maybe it’s not correct, because when I make my labels like this, I get the error

/opt/conda/conda-bld/pytorch_1591914880026/work/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [27,0,0] Assertion t >= 0 && t < n_classes failed

which indicates to me that I can not have labels with values outside the interval [0, n_classes].
What I have are labels for which class each wordpiece belongs to, after tokenization. Then I add the special tokens and padding, and I’m setting labels for the special tokens to -100. So for example if I want a sequence length of 10, and I want to classify wordpieces with an ‘o’ in them as class 1, and wordpieces with a ‘p’ in them as class 2, I would have, for the sentence “Oh, that school is pretty cool”:
Tokens: [‘oh’, ‘,’, ‘that’, ‘school’, ‘is’, ‘pretty’, ‘cool’]
With special tokens: [’[CLS]’, ‘oh’, ‘,’, ‘that’, ‘school’, ‘is’, ‘pretty’, ‘cool’, ‘[SEP]’, ‘[PAD]’]
Labels: [-100, 1, 0, 0, 1, 0, 2, 0, -100, -100]

Context:
You should be able to do something like this:
config = AutoConfig.from_pretrained("bert-base-cased", num_labels=3)
model = AutoModel.from_pretrained("bert-base-cased", config=config)

Note that in your example you have three possible labels: with o, with p, and with neither. If you set num_labels to 2, you will have gotten the error that you described.
-100 is the default ignore index for NLLLoss 14. When a target item has this index, it will be ignored from loss computation.


Question:
Hi does someone know a fine tuning example with the HF trainer API?
Have never fintined a gpt before so any tipis is much appreciated

Context: I’ve never used GPT models before. However, if they are available on huggingface, I guess fine-tuning it won’t be different than fine-tuning any other model. I use PyTorch Lightning for training and replacing any huggingface models with each other wouldn’t change the process I guess…


MLM train loss is very different after version update

Question:
Hi,
I use the run_mlm.py script to pretrain from scratch BERT model. Not sure if this is the most updated version of the script since I’ve been using it for a couple of months.
When I used transofrmers 4.5.1 I used to get train loss of ~1.9 but after updating to transformers 4.9.2 the train loss is ~4.5.
I’m training from scratch on my data file + trained tokenizer and I ran the exact same command on both times.
This are the training argument on 4.5.1:
output_dir=test-mlm-wiki, overwrite_output_dir=True, do_train=True, do_eval=None, do_predict=False, evaluation_strategy=IntervalStrategy.NO, prediction_loss_only=False, per_device_train_batch_size=32, per_device_eval_batch_size=8, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=5e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=1.0, max_steps=-1, lr_scheduler_type=SchedulerType.LINEAR, warmup_ratio=0.1, warmup_steps=0, logging_dir=runs/Aug08_11-51-18, logging_strategy=IntervalStrategy.STEPS, logging_first_step=False, logging_steps=500, save_strategy=IntervalStrategy.STEPS, save_steps=5000, save_total_limit=1, no_cuda=False, seed=42, fp16=False, fp16_opt_level=O1, fp16_backend=auto, fp16_full_eval=False, local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=500, dataloader_num_workers=0, past_index=-1, run_name=test-mlm-wiki, disable_tqdm=True, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=None, ignore_data_skip=False, sharded_ddp=[], deepspeed=None, label_smoothing_factor=0.0, adafactor=False, group_by_length=False, length_column_name=length, report_to=[‘tensorboard’], ddp_find_unused_parameters=None, dataloader_pin_memory=True, skip_memory_metrics=False, _n_gpu=1, mp_parameters=
These are the training arguments on 4.9.2:
TrainingArguments(_n_gpu=1, adafactor=False,  adam_beta1=0.9,
adam_beta2=0.999, adam_epsilon=1e-08, dataloader_drop_last=False, dataloader_num_workers=0, dataloader_pin_memory=True, ddp_find_unused_parameters=None, debug=[], deepspeed=None, disable_tqdm=True, do_eval=False, do_predict=False, do_train=True, eval_accumulation_steps=None, eval_steps=None, evaluation_strategy=IntervalStrategy.NO, fp16=False, fp16_backend=auto, fp16_full_eval=False, fp16_opt_level=O1, gradient_accumulation_steps=1, greater_is_better=None, group_by_length=False, ignore_data_skip=False, label_names=None, label_smoothing_factor=0.0, learning_rate=5e-05, length_column_name=length, load_best_model_at_end=False, local_rank=-1, log_level=-1, log_level_replica=-1, log_on_each_node=True, logging_dir=test-mlm-wiki/runs/Aug27_17-12-33, logging_first_step=False, logging_steps=500, logging_strategy=IntervalStrategy.STEPS, lr_scheduler_type=SchedulerType.LINEAR, max_grad_norm=1.0, max_steps=-1, metric_for_best_model=None, mp_parameters=, no_cuda=False, num_train_epochs=1.0, output_dir=test-mlm-wiki, overwrite_output_dir=True, past_index=-1, per_device_eval_batch_size=8, per_device_train_batch_size=32, prediction_loss_only=False, push_to_hub=False, push_to_hub_model_id=test-mlm-wiki-sst5-100, push_to_hub_organization=None, push_to_hub_token=None, remove_unused_columns=True, report_to=[‘tensorboard’], resume_from_checkpoint=None, run_name=test-mlm-wik, save_on_each_node=False, save_steps=5000, save_strategy=IntervalStrategy.STEPS,
save_total_limit=1, seed=42, sharded_ddp=[], skip_memory_metrics=True, tpu_metrics_debug=False, tpu_num_cores=None, use_legacy_prediction_loop=False, warmup_ratio=0.1, warmup_steps=0, weight_decay=0.0)
These are the difference I’ve found:

4.5.1
4.9.2

debug=False
debug=[]

eval_steps=500
eval_steps=None

logging_dir=runs/Aug08_11-51-18_dogfish-01
logging_dir=est-mlm-wiki/runs/Aug27_17-12-33_dogfish-01

-
log_level=-1

-
log_level_replica=-1

-
log_on_each_node=True

-
push_to_hub=False

-
push_to_hub_model_id=test-mlm-wiki

-
push_to_hub_organization=None

-
push_to_hub_token=None

-
save_on_each_node=False

-
use_legacy_prediction_loop=False

-
resume_from_checkpoint=None

Any ideas why this happens?

Context:
I wonder if something is going on with the tokenizer…
When running on the 4.5.1 version, I changed in run_mlm script AutoTokenizer to BertTokenizer (it had a bug perhaps, wouldn’t recognize my tokenizer).
On 4.9.2 I get this error when running the script:
“The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization.
The tokenizer class you load from this checkpoint is ‘T5TokenizerFast’.
The class this function is called from is ‘BertTokenizer’.”
That’s the code I’m using to train the tokenizer:
tokenizer = BertWordPieceTokenizer()
tokenizer.train(files=[‘wikipedia_1e6.txt], vocab_size=30000)
tokenizer.save_model(’/tokenizers/vocab_folder’)
tokenizer = BertTokenizer.from_pretrained(’/tokenizers/vocab_folder’)
tokenizer.save_pretrained(’/tokenizers/wikipedia_1e6/’)
However, when training tokenizer with transformers 4.9.2 and using it in run_mlm I still get a bigger train loss (than 4.5.1), but no warning.


Bart outputing </s> in start of every decoded sentence

Question:
Any sentence that I input into the pretrained model of  BartforConditionalGeneration, it would outputs a sentence with the prefix . Am I doing something wrong?
For example, if I input “Obama is the best president”, it gives back </s.><s.>Obama is the greatest president ever</s.>…Same is the case with every input sentence.
TIA

Context: This is expected behavior, if you see BartforConditionalGeneration docs, the decoder_start_token_id is the EOS token by default (which is ). This is in contrast to other models, a BERT decoder for instance has a CLS token as the starting token given to the decoder.


How do i get Training and Validation Loss during fine tuning

Question:
Hi,
The trained API doesn’t seem to throw training loss.
Only shows the validation loss.
Could some one show me why this is the case?
Here is my code snippet.
def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    predictions = predictions[:, 0]
    return metric.compute(predictions=predictions, references=labels)

#Training Arguments
args = TrainingArguments(
    output_dir = "/Content/mod",
    evaluation_strategy = "epoch", #Can be epoch or steps
    learning_rate=2e-5, #According to original bert paper
    per_device_train_batch_size=32, #According to original bert paper
    per_device_eval_batch_size=32,
    num_train_epochs=3, #should be inbetween 2-4 according to the paper
    weight_decay=0.01,
    prediction_loss_only = True
)

#Initializing Data Collator: this is for dynamic padding of tokens
#Helps the training cycle to be quicker according to Hugging face Dcumentations
data_collator_ = DataCollatorWithPadding(tokenizer=tokenizer)

#Trainer itself.
trainer = Trainer(
    model,
    args,
    train_dataset=tokenized_datasets_train,
    eval_dataset=tokenized_datasets_val,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
    data_collator = data_collator_
)

torch.cuda.empty_cache() #This line is unnecessary but i still kept it cuz why not.
trainer.train()

Context: For example if you use evaluation_strategy="steps" and eval_steps=2000 in the TrainingArguments, you will get training and validation loss for every 2000 steps. If you wanna do it on an epoch level I think you need to set evaluation_strategy="epoch" and logging_strategy="epoch" in the TrainingArguments class.


Saving only the best performing checkpoint

Question:
Hi,
Is there a parameter in config that allows us to save only the best performing checkpoint ?
Currently, multiple checkpoints are saved based on save_steps (, batch_size and dataset size). If we want to train the model for lets say 10 epochs and 7th epoch gives the best performance on validation set, then how can we just save the checkpoint from 7th epoch and ignore the rest.
Thanks.

Context: There is no parameter for that yet, keeping this in mind for the future development of Trainer.


How to make single-input inference faster? Create my own pipeline?

Question:
Hello there!
I was able to fine-tune my own model for text classification (from a distilbert-base-uncased-finetuned-sst-2-english · Hugging Face 1 model). Everything works correctly on my PC.
Now comes the app development time but inference - even on a single sentence - is quite slow. I am processing one sentence at a time and using the simple function predict_single_sentence(['this is my input sentence'])

tokenizer = AutoTokenizer.from_pretrained('C:\\Users\\mytrainedmodel')
mymodel = TFAutoModelForSequenceClassification.from_pretrained('C:\\Users\\mytrainedmodel')

def predict_single_sentence(text_list, model, tokenizer):
    #tokenize the text
    encodings = tokenizer(text_list,
                          max_length=280,
                          truncation=True,
                          padding=True)
    #transform to tf.Dataset
    dataset = tf.data.Dataset.from_tensor_slices((dict(encodings)))
    #predict
    preds = model.predict(dataset.batch(1)).logits

    #transform to array with probabilities
    res = tf.nn.softmax(preds, axis=1).numpy()

    return res

Despite my big RTX, inference is quite slow (about 5 sec for a single sentence). Am I missing something here? Should I create my own pipeline to speed things up?
Thanks!

Context: @sgugger sorry to pull you in, but I would really appreciate your input here. Am I doing things wrong? Is there anything huggingface provides that allows me to speed up inference? THank you so much!!


Question:
Hello there,
I am struggling with two simple questions about transformers models. I hope somebody in the huggingface communitity can shed some light on these points.

transformers models create contextual embeddings. That is, the embedding of a word depend on the words around it in the sentence. Does this means a word has a different embedding for every possible sentence?
At inference time, I am feeding a sentence to a transformer model for classification. How can the attention mechanism work (this word is related to this word, etc) if the model has never seen the sentence? I get the intuition at training time (we minimize the loss) but what happens at inference time?

Any insights would be greatly appreciated!
Thanks!

Context:
The final output of a word will differ depending on its position in the sentence as well as the words surrounding it. So, yes - for every different sentence, the word will have a different output.
This is the whole point of training: the model learns which words it should pay attention to and which ones it should not for a given input vector. In practice it is more mathematical than that. This illustration 19 may help you get your head around it.


How to change max_length of a fine tuned model

Question:
Hello Everyone,
I trained and shared a custom model based on gpt2 and now in config.json file of my model in the Model Hub I have the max_length as 50. I don’t remember passing that number as a training argument or such. However I want to use the whole capability of gpt-2 model and generate texts of length 1024 tokens. How can I update my model on Model Hub so that it’s possible to generate longer outputs using a pipeline?
Here’s the config file of my model : config.json · YusufSahin99/IFIS_ZORK_AI_FANTASY at main 6
Any help would be appreciated

Context: This number is just a default for when you use the model in a text-generation pipeline, which can be overridden by passing a new max_length in a call.


Is the huggingface run_mlm Script dynamically masked?

Question:
Hi,
In the roberta paper, the model is trained by dynamic masking of sentences.
If a roberta model is further pre-trained using the run_mlm.py , is the sentences going to be dynamically masked during pre-training using this file or is it statically masked like vanilla BERT?
The script is found here

      github.com

    transformers/examples/pytorch/language-modeling at master ·... 9

  master/examples/pytorch/language-modeling

  🤗 Transformers: State-of-the-art Natural Language Processing for Pytorch, TensorFlow, and JAX. - transformers/examples/pytorch/language-modeling at master · huggingface/transformers

Context: Apparently, dynamic masking is always used, no matter the model. See this: dynamic masking for RoBERTa model · Issue #5979 · huggingface/transformers · GitHub 20


[HELP] NER task single sentence/sample prediction

Question:
Hello everybody.
I am trying to predict with the NER model, as in the tutorial from huggingface (it contains only the training+evaluation part).
I am following this exact tutorial here : notebooks/token_classification.ipynb at master · huggingface/notebooks · GitHub.
It works flawlessly, but the problems that I have begin when I try to predict on a simple sample.
loaded_model = AutoModel.from_pretrained('./my_model_own_custom_training.pth', from_tf=False)

input_sentence = "John Nash is a great mathematician, he lives in France"
tokenized_input_sentence = tokenizer([input_sentence],
                                     truncation=True,
                                     is_split_into_words=False,
                                     return_tensors='pt')
predictions = loaded_model(tokenized_input_sentence["input_ids"])[0]

predictions is of shape (1,13,768)
How can I arrive at the final result of the form [JOHN <-> ‘B-PER’, … France <-> “B-LOC”], where B-PER and B-LOC are two ground truth labels, representing the tag for a person and location respectively?
The result of the prediction is:
torch.Size([1, 13, 768])
If I write:
print(predictions.shape)
print(predictions.argmax(axis=2))
tensor([613, 705, 244, 620, 206, 206, 206, 620, 620, 620, 477, 693, 308])

I get the tensor above.
However I would have expected to get the tensor representing the ground truth [0…8] labels from the ground truth annotations.
Summary when loading the model :

loading configuration file ./my_model_own_custom_training.pth/config.json
Model config DistilBertConfig {
“name_or_path": “distilbert-base-uncased”,
“activation”: “gelu”,
“architectures”: [
“DistilBertForTokenClassification”
],
“attention_dropout”: 0.1,
“dim”: 768,
“dropout”: 0.1,
“hidden_dim”: 3072,
“id2label”: {
“0”: “LABEL_0”,
“1”: “LABEL_1”,
“2”: “LABEL_2”,
“3”: “LABEL_3”,
“4”: “LABEL_4”,
“5”: “LABEL_5”,
“6”: “LABEL_6”,
“7”: “LABEL_7”,
“8”: “LABEL_8”
},
“initializer_range”: 0.02,
“label2id”: {
“LABEL_0”: 0,
“LABEL_1”: 1,
“LABEL_2”: 2,
“LABEL_3”: 3,
“LABEL_4”: 4,
“LABEL_5”: 5,
“LABEL_6”: 6,
“LABEL_7”: 7,
“LABEL_8”: 8
},
“max_position_embeddings”: 512,
“model_type”: “distilbert”,
“n_heads”: 12,
“n_layers”: 6,
“pad_token_id”: 0,
“qa_dropout”: 0.1,
“seq_classif_dropout”: 0.2,
“sinusoidal_pos_embds”: false,
"tie_weights”: true,
“transformers_version”: “4.8.1”,
“vocab_size”: 30522
}

Context:
Calin:

notebooks/token_classification.ipynb at master · huggingface/notebooks · GitHub

cc’ing @sgugger, these demo notebooks really need an inference part.
To do inference with NER, you need to load an AutoModelForTokenClassification rather than an AutoModel, like so:
from transformers import AutoTokenizer, AutoModelForTokenClassification

tokenizer = AutoTokenizer.from_pretrained("path_to_directory")
model = AutoModelForTokenClassification.from_pretrained("path_to_directory")

input_sentence = "John Nash is a great mathematician, he lives in France"
encoding = tokenizer([input_sentence], return_tensors='pt')

# forward pass
outputs = model(**encoding)
logits = outputs.logits

predictions = logits.argmax(-1)

The logits will be of shape (batch_size, seq_len, num_labels).


Why is the code for DataCollatorForSeq2Seq overwriting the labels?

Question:
Hey Guys,
I can’t figure out why in the source code for DataCollatorForSeq2Seq, the feature[‘label’] is being overwritten? Does it not lead to code break when the training sample are shuffled?

Context:
I guess you are talking about this line (in the future please link to the code that you are talking about so that we can easily look it up).

      github.com

    huggingface/transformers/blob/143738214cb83e471f3a43652617c8881370342c/src/transformers/data/data_collator.py#L281-L287 1

          if labels is not None:
              max_label_length = max(len(l) for l in labels)
              padding_side = self.tokenizer.padding_side
              for feature in features:
                  remainder = [self.label_pad_token_id] * (max_label_length - len(feature["labels"]))
                  feature["labels"] = (
                      feature["labels"] + remainder if padding_side == "right" else remainder + feature["labels"]

It is not changing the given labels but it is padding them to ensure that all items in the batch have the same length. Pad tokens are ignored when the loss is calculated.
It is a collator, so it happens after the shuffling process of the dataloader. It receives a number of items from the dataloader (possibly shuffled) and then collates them (prepares them for the model).


Question:
Hello everyone,
I successfully fine-tuned a model for text classification. Now I would like to run my trained model to get labels for a large test dataset (around 20,000 texts).
So I had the idea to instantiate a Trainer with my model and use the trainer.predict() method on my data. This works fine, but I was wondering if it makes sense (and it’s efficient, advisable, & so on) to use a Trainer (which, of course, was meant to be used for training models) just for inference.
If not, what would be a better way to perform inference on a large dataset? I cannot just pass all data to model() as I get out of memory errors. I would need to explicitly batch my data, I guess (while Trainer takes care of that part implicitly)…
Thank you in advance for your thoughts on this!

Context:
Normally, the Trainer saves your trained model in a directory. You can specify this with the output_dir argument when instantiating the TrainingArguments.
You can then instantiate your trained model using the .from_pretrained() method. Suppose that you have fine-tuned a BertForSequenceClassification model, then you can instantiate it as follows:
from transformers import BertForSequenceClassification

model = BertForSequenceClassification.from_pretrained("path_to_the_directory")

You can then make batched predictions as follows:
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("path_to_the_directory")

text = ["this is one sentence", "this is another sentence"]
encoding = tokenizer(text, return_tensors="pt")

# forward pass
outputs = model(**encoding)
predictions = outputs.logits.argmax(-1)


Model trains with Seq2SeqTrainer but gets stuck using Trainer

Question:
Hi,
I’ve been trying to finetune the BART large pre-trained on MNLI with the Financial Phrasebank dataset to build a model for news sentiment analysis. I’m just a beginner and so, I mostly use the code from GEM Getting Started.
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, Seq2SeqTrainer, Seq2SeqTrainingArguments
tokenizer = AutoTokenizer.from_pretrained(‘facebook/bart-large-mnli’)
model = AutoModelForSeq2SeqLM.from_pretrained(‘facebook/bart-large-mnli’)
The model only trains when I use the AutoModelForSeq2SeqLM, Seq2SeqTrainer andSeq2SeqTrainingArguments. When I use model = AutoModelForSequenceClassification.from_pretrained(‘facebook/bart-large-mnli’) with the Trainer and TrainingArguments, the model does not train.
Is it appropriate to use seq2seq for sentiment classification tasks?
Any suggestions would be immensely helpful.
Thanks in advance.

Context:
Are you framing your classification problem as a sequence generation task? what types of labels do you have for your training data? Are the labels text/sequence or a finite number of categories? If your task is classification I believe you’re using the wrong model class. You could probably use BertForSequenceClassification for a sentiment analysis task as has been done in the link below:

      huggingface.co

nlptown/bert-base-multilingual-uncased-sentiment at main 3

And instead of using Seq2SeqTrainer, just use Trainer and TrainingArguments.


Onnx tf bert sentiment-analysis input and outputs

Question:
I trained a model based on bert-large-cased using the run_text_classification.py 3 tensorflow example. I converted the model to onnx using the convert_graph_to_onnx.py tool.
That looks like this:
convert_graph_to_onnx.convert(
        framework='tf',
        model="output",
        output=Path("model/model.onnx"),
        opset=12,
        tokenizer="output",
        use_external_format=True,
        pipeline_name="sentiment-analysis")

I’m having a slight difficulty understanding the inputs and outputs. I am trying to use Rust and the onnxruntime 1. The onnx model input shape is (?, 5). I am using input_ids, attention mask, token ids constructed from the tokenizers Rust library. For example
the inputs I’m using
[[101, 1, 0, 0, 0],
 [1104, 1, 0, 0, 0],
 [23120, 1, 0, 0, 0],
 [188, 1, 0, 0, 0],
 [19091, 1, 0, 0, 0],
 [8124, 1, 0, 0, 0],
 [1111, 1, 0, 0, 0],
 [3062, 1, 0, 0, 0],
 [1105, 1, 0, 0, 0],
 [15470, 1, 0, 0, 0],
 [119, 1, 0, 0, 0],
 [102, 1, 0, 0, 0],
 [0, 0, 0, 0, 0],
 [0, 0, 0, 0, 0],
...
...

I truncate + pad the tokens out to 128 length, since I used a max length of 128 when training the model.
the outputs are logits (I think) in the shape (?, 4). My sentiment analysis task has 4 classes, so I think this makes sense.
[[0.87741166, 0.4000733, -2.557633, 1.5771139],
 [-0.7227318, -0.14528184, 2.4809465, -1.7312673],
 [0.88585603, 0.392128, -2.54713, 1.5852065],
 [-0.89909637, 0.5074229, 2.3639672, -1.8381689],
 [0.8940967, 0.40258378, -2.5756738, 1.5999701],
...
...

Since this is a sentiment-analysis task why are there logits for each token?
Any advice on things to check? Any things stand out as obviously wrong?

Context:
Usually, the last layer of a classification model (in your case TFBertForSequenceClassification), produces raw prediction values as real numbers ranging from [-infinity, +infinity]  These raw, unconstrained prediction values are commonly known as logits
Now after this, we usually want to normalize these outputs to a probability distribution over predicted output classes. To do this, we usually use a normalization layer such as sigmoid and softmax for binary and multi-class classification respectively.
The output of your softmax function is now a probability and to convert that into your classes, we just take the max of these.
To do that with your onnx outputs, you can do something like this:
import numpy as np
from scipy.special import softmax

np.argmax(softmax(outputs[0][0], axis=0))


Question:
Hi,
If I am not mistaken, there are two types of trainers in the library. The standard trainer and the seq2seq trainer.
It seems that the Trainer works for every model since I am using it for a Seq2Seq model (T5).
MY question is: What advantages does seq2seq trainer have over the standard one?
And why does not the library handle the switch in the background or does it?
I mean that the user can use Trainer all the time and in the background, it will be a seq2seqtrainer if the corresponding model needs it.
Thank you!

Context:
Hi @berkayberabi
You are right, in general, Trainer can be used to train almost any library model including seq2seq.
Seq2SeqTrainer is a subclass of Trainer and provides the following additional features.

lets you use SortishSampler

lets you compute generative metrics such as BLEU, ROUGE, etc by doing generation inside the evaluation loop.

The reason to add this as a separate class is that for calculating generative metrics we need to do generation using the .generate method in the predict step which is different from how other models to prediction, to support this you need to override the prediction related methods such as (prediction_step,  predict) to customize the behaviour,  hence the Seq2SeqTrainer.
Hope this answers your question.


How to use transformers for batch inference

Question:
I use transformers to train text classification models，for a single text, it can be inferred normally. The code is as follows
from transformers import BertTokenizer, TFAlbertForSequenceClassification
text = 'This is a sentence'
model_path ='../albert_chinese_tiny'
tokenizer = BertTokenizer.from_pretrained(model_path)
model = TFAlbertForSequenceClassification.from_pretrained('../model_tf/20210818')
encoding = tokenizer(text, truncation=True, padding=True, max_length=30, return_tensors="tf")
result = model(encoding)

When I predict more than one text at a time, an error will be reported. The code is as follows
texts = ['This is a sentence', 'This is another sentence']
encodings = []
model_path ='../albert_chinese_tiny'
tokenizer = BertTokenizer.from_pretrained(model_path)
model = TFAlbertForSequenceClassification.from_pretrained('../model_tf/20210818')
for text in texts:
    encoding = tokenizer(text, truncation=True, padding=True, max_length=30, return_tensors="tf")
    encodings.append(encoding)
result = model(np.array(encodings))

The error information is as follows:
tensorflow.python.framework.errors_impl.InvalidArgumentError: Value for attr ‘Tindices’ of string is not in the list of allowed values: int32, int64
; NodeDef: {{node ResourceGather}}; Op<name=ResourceGather; signature=resource:resource, indices:Tindices → output:dtype; attr=batch_dims:int,default=0; attr=validate_indices:bool,default=true; attr=dtype:type; attr=Tindices:type,allowed=[DT_INT32, DT_INT64]; is_stateful=true> [Op:ResourceGather]

Context: This question has been answered here 375.


Multiple choice with variable number of choices

Question:
Hi all,
Similar question to

    Multiple choice with variable length options 🤗Transformers

    Hello! I have a beginner question. I am trying to create a model that makes predictions on the QAngaroo dataset with DistilBert. In this dataset, we get a list of supports and some candidate answers (between 2~100), and we need to choose the right answer for the model. Right now, I am trying to use TFDistilBertForMultipleChoice, but I am running into a problem since num_choices is a value that is fixed with the entire batch size. I was wondering how I could go about making that value dynamic.
I…

The reply there is to go full-blown text-to-text—which is a great idea!—but I’m interested in getting a discriminative BERT-esque baseline if possible (due to the dataset’s particular size, structure, and text content).
Since multiple choice models (like RobertaForMultipleChoice) detect the number of questions dynamically per batch, it seems like the main challenge is getting each batch to have a consistent number of choices.
Going off of the run_swag.py example, there are two main user-provided data processing functions:

a preprocess_function() — for adding new features to the Dataset
a collate function — for turning a raw batch into tensors

Unfortunately, while it’s no problem to add the number of choices for an example in 1., by the time 2. comes around, we’ve already been given a batch, so it’s too late to ensure they all have the same number of choices.
In other words, it seems like the sampler is the place where we’d make sure the number of choices is consistent per batch.
To my delight, I found the --group_by_length and --length_column_name options, which enable the (Distributed)LengthGroupedSampler. This opens up a potential way for doing this:

Add a feature with the number of choices
Pass this feature as the --length_column_name  and use --group_by_length

Unfortunately for me, this constraint is a soft one rather than a hard one. (This makes sense for the original purpose, of course, which is just helping with padding.) This means that some batches do end up with multiple “lengths” (choices).
I wrote a quick test that injects a number from 1-4 as the feature and checked how many batches had multiple “lengths.” I was hoping it would be just three batches (at the borders between 1-2, 2-3, 3-4). Over 500 batches, there were 14 that ended up with mixed numbers. I could truncate these batches, which would only lose ~3% of the data. Not a huge loss, but it makes me wonder whether I can do better!
So, to complete the very long-winded question, I wonder whether anyone more familiar with Huggingface Transformers can recommend an approach to implement multiple choice with a variable number of choices. Right now my main options are:

Write my own sampler to do this. Given this is just for a baseline, and my trepidation at debugging a custom distributed sampler, I worry this might not be worth the investment.

Create multiple Dataset objects, each with a consistent number of choices. Do an outer training loop. (This would be less ideal because each number of choices corresponds to a question format, so this would increase coarse patterns into training / reduce how shuffled it is.)

Just throw away the ~3% of mixed-data batches (less if we take the majority, so maybe ~1.5%).

??? (better approach I can’t see?)

Huge thanks for your time!

Context:
In case anyone in the future is reading this, the above does work pretty well, but only for training. For evaluation, the (Distributed)LengthGroupedSampler is not used. Furthermore, even if it was, by nature of throwing some data away in the collator, we skip some of the evaluation set (which is a no-no for comparing results between methods).
I provided an example implementation of a batch sampler that groups based on a provided feature in the following comment:
Option for `(Distributed)LengthGroupedSampler` to treat groups as a hard constraint · Issue #12995 · huggingface/transformers · GitHub 5
It also requires a change to Transformers itself to support a batch sampler, which is in a PR linked to that issue.


Question:
Hello there,
I am facing a rather annoying issue. I fine-tuned a Bert model for classification and saved the resulting model to disk. However, I am unable to subsequently load the model.
This is essentially what I do:
tokenizer = AutoTokenizer.from_pretrained('C:\\Users\\bert-sentiment')
#tokenize text...
model = TFAutoModelForSequenceClassification.from_pretrained('C:\\Users\\bert-sentiment',
                                                                  num_labels =3,
                                                                  ignore_mismatched_sizes = True)
#fine tune model...
model.fit()
model.save_pretrained('C:\\Users\\mydir')

mydir contains two files tf_model.h5 and config.json. However, when I tried to load the model by simply running
mymodel = TFAutoModelForSequenceClassification.from_pretrained("C:\\Users\\mydir")

I get the error
OSError: Unable to load weights from h5 file. If you tried to load a TF 2.0 model from a PyTorch checkpoint, please set from_pt=True.
 do you know what the issue could be?
Thanks!

Context: Strange bug. I re-generated the saved models and then loading worked fine. Perhaps some corruption at saving time?


Question:
I want to start a new pre training language. When I use ’bertformaskedLM‘, I find that the model has a decoder layer, and the output of the model should be 768 dimensions, not 20000 (20000 is the number of tokens). What is the reason for this problem ？and how can I pretrain a language correctly？  thank you.
this code:
config = BertConfig(
    vocab_size=20000,
    hidden_size=768,
    num_hidden_layers=6,
    num_attention_heads=12,
    max_position_embeddings=512
)

model = BertForMaskedLM(config)

training_args = TrainingArguments(
    output_dir='./bangla_data/working/',
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_device_train_batch_size=32,
    save_steps=1000, #10000
    save_total_limit=2,
    prediction_loss_only=True,
    learning_rate: float = 5e-05,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
)

model = BertForMaskedLM(config)
print('No of parameters: ', model.num_parameters())
this model:
BertForMaskedLM(
(bert): BertModel(
(embeddings): BertEmbeddings(
(word_embeddings): Embedding(20000, 768, padding_idx=0)
(position_embeddings): Embedding(512, 768)
(token_type_embeddings): Embedding(2, 768)
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(encoder): BertEncoder(
(layer): ModuleList(
(0): BertLayer(
(attention): BertAttention(
(self): BertSelfAttention(
(query): Linear(in_features=768, out_features=768, bias=True)
(key): Linear(in_features=768, out_features=768, bias=True)
(value): Linear(in_features=768, out_features=768, bias=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(output): BertSelfOutput(
(dense): Linear(in_features=768, out_features=768, bias=True)
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
)
(intermediate): BertIntermediate(
(dense): Linear(in_features=768, out_features=3072, bias=True)
)
(output): BertOutput(
(dense): Linear(in_features=3072, out_features=768, bias=True)
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
)
(1): BertLayer(
(attention): BertAttention(
(self): BertSelfAttention(
(query): Linear(in_features=768, out_features=768, bias=True)
(key): Linear(in_features=768, out_features=768, bias=True)
(value): Linear(in_features=768, out_features=768, bias=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(output): BertSelfOutput(
(dense): Linear(in_features=768, out_features=768, bias=True)
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
)
(intermediate): BertIntermediate(
(dense): Linear(in_features=768, out_features=3072, bias=True)
)
(output): BertOutput(
(dense): Linear(in_features=3072, out_features=768, bias=True)
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
)
(2): BertLayer(
(attention): BertAttention(
(self): BertSelfAttention(
(query): Linear(in_features=768, out_features=768, bias=True)
(key): Linear(in_features=768, out_features=768, bias=True)
(value): Linear(in_features=768, out_features=768, bias=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(output): BertSelfOutput(
(dense): Linear(in_features=768, out_features=768, bias=True)
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
)
(intermediate): BertIntermediate(
(dense): Linear(in_features=768, out_features=3072, bias=True)
)
(output): BertOutput(
(dense): Linear(in_features=3072, out_features=768, bias=True)
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
)
(3): BertLayer(
(attention): BertAttention(
(self): BertSelfAttention(
(query): Linear(in_features=768, out_features=768, bias=True)
(key): Linear(in_features=768, out_features=768, bias=True)
(value): Linear(in_features=768, out_features=768, bias=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(output): BertSelfOutput(
(dense): Linear(in_features=768, out_features=768, bias=True)
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
)
(intermediate): BertIntermediate(
(dense): Linear(in_features=768, out_features=3072, bias=True)
)
(output): BertOutput(
(dense): Linear(in_features=3072, out_features=768, bias=True)
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
)
(4): BertLayer(
(attention): BertAttention(
(self): BertSelfAttention(
(query): Linear(in_features=768, out_features=768, bias=True)
(key): Linear(in_features=768, out_features=768, bias=True)
(value): Linear(in_features=768, out_features=768, bias=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(output): BertSelfOutput(
(dense): Linear(in_features=768, out_features=768, bias=True)
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
)
(intermediate): BertIntermediate(
(dense): Linear(in_features=768, out_features=3072, bias=True)
)
(output): BertOutput(
(dense): Linear(in_features=3072, out_features=768, bias=True)
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
)
(5): BertLayer(
(attention): BertAttention(
(self): BertSelfAttention(
(query): Linear(in_features=768, out_features=768, bias=True)
(key): Linear(in_features=768, out_features=768, bias=True)
(value): Linear(in_features=768, out_features=768, bias=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(output): BertSelfOutput(
(dense): Linear(in_features=768, out_features=768, bias=True)
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
)
(intermediate): BertIntermediate(
(dense): Linear(in_features=768, out_features=3072, bias=True)
)
(output): BertOutput(
(dense): Linear(in_features=3072, out_features=768, bias=True)
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
)
)
)
)
(cls): BertOnlyMLMHead(
(predictions): BertLMPredictionHead(
(transform): BertPredictionHeadTransform(
(dense): Linear(in_features=768, out_features=768, bias=True)
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
)
(decoder): Linear(in_features=768, out_features=20000, bias=True)
)
)
)

Context:
Hi @ccfeidao
For your first question about the decoder and the hidden size and output size: Internally, the model projects the input tokens which have dimension of the vocabulary size (20’000 in your case) to the hidden size (768 in your case). Inside the layers of BERT the embeddings of 768 are processed. Finally, after the last BERT layer we need to get back from the hidden size to the vocabulary size which corresponds to proper tokens. That’s what the decoder layer does: it takes embeddings of dim=768 and projects them to dim=20000.
As for your second question about pretraining: there is a tutorial on Google Colab 7.


Unable to torch.jit.trace quantized BigBird (0INTERNAL ASSERT FAILED runtime error) but works for BERT and RoBERTa

Question:
Hello,
I am trying to torch.jit.trace Transformers’ implementation of BigBird. But I’m encountering a runtime error that I’m not very familiar with, specifically:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-4-1dfdd2340788> in <module>
      4 )
      5
----> 6 traced_model = torch.jit.trace(model, (input_ids, attention_mask))
      7 torch.jit.save(traced_model, "traced_bigbird.pt")

/opt/conda/lib/python3.7/site-packages/torch/jit/_trace.py in trace(func, example_inputs, optimize, check_trace, check_inputs, check_tolerance, strict, _force_outplace, _module_class, _compilation_unit)
    742             strict,
    743             _force_outplace,
--> 744             _module_class,
    745         )
    746

/opt/conda/lib/python3.7/site-packages/torch/jit/_trace.py in trace_module(mod, inputs, optimize, check_trace, check_inputs, check_tolerance, strict, _force_outplace, _module_class, _compilation_unit)
    957                 strict,
    958                 _force_outplace,
--> 959                 argument_names,
    960             )
    961             check_trace_method = module._c._get_method(method_name)

RuntimeError: 0INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/jit/ir/alias_analysis.cpp":532, please report a bug to PyTorch. We don't have an op for aten::constant_pad_nd but it isn't a special case.  Argument types: Tensor, int[], bool,

I’m trying to locate constant_pad_nd in the code base for BigBird to figure out how it relates to aten, but i’m also that familiar with aten. That said, I also ran the same code for BERT and RoBERTa but did not encounter the same issue and was able to trace the quantized models for both respectively.
To reproduce this error,

Git clone this repo

Run example.ipynb

Anyone familiar with this matter or knows enough to help debug this issue?

Context:
Created a Github issue here 17.
Also tagging @patrickvonplaten @sgugger @lewtun for more visibility


Huge difference in speed when finetuning summarization with different scripts

Question:
From transformers 3.x, I have been using examples/seq2seq/finetune.py (huggingface-transformers/finetune.py at v3.5.1 · microsoft/huggingface-transformers · GitHub 4) to finetune pegasus model and it’s been working fine.
After upgrading to transformers 4.x, that script has been moved to legacy and/so I’m thinking of using the examples/seq2seq/run_summarization.py (transformers/run_summarization.py at v4.4.2 · huggingface/transformers · GitHub 1) for the same training.
I moved pieces around to make the old finetune.py work under transformers 4.x as well to have fair comparison. Other than the dataset format difference between the two, I mainly noticed the huge difference in training speed between the two.
For the same training dataset (~6 million data points):

finetune.py: with 4 v100 GPU, taking ~6.5h/epoch.

run_summarization.py with 4 v100 GPU, taking ~15h/epoch.

Upon reading the code, finetune.py from transformers 3.x uses pytorch-lightning but run_summarization.py I believe just uses pytorch. Is this mainly causing the difference in speed? Or are there any signification implementation difference between the two?
I can provide more details if needed.
Thanks!

Context:
The two scripts don’t have the same defaults at all, so there could be plenty of reasons for the differences in speed. Could you tell us what command lines you use to run them in both cases?
Thanks!


Question:
Hi,
When adding a new token in the vocabulary, there is a difference between Tokenizer and FastTokenizer.
from transformers import BartTokenizer, BartTokenizerFast

tokenizer = BartTokenizer.from_pretrained('facebook/bart-large')
tokenizer_fast = BartTokenizerFast.from_pretrained('facebook/bart-large')

tokenizer.add_tokens("<NEW_TOKEN>")
tokenizer_fast.add_tokens("<NEW_TOKEN>")

sentence = "I added a <NEW_TOKEN> in the vocabulary."

print(tokenizer.encode(sentence))
# [0, 100, 355, 10, 50265, 179, 5, 32644, 4, 2]

print(tokenizer_fast.encode(sentence))
# [0, 100, 355, 10, 1437, 50265, 11, 5, 32644, 4, 2]

The fast tokenizer adds a space token before the <NEW_TOKEN> (1437) while the standard tokenizer removes the automatic space from the next token (179 vs. 11).
I tried with RoBERTa and got the same problem.
Thanks!

Context: Technically speaking overall implementation of tokenizers wrt to Sentencepiece is kind of hacky in HuggingFace.


How can I get the score from Question-Answer Pipeline? Is there a bug when Question-answer pipeline is used?

Question:
When I run the following code
from transformers import AutoTokenizer, AutoModelForQuestionAnswering

import torch

tokenizer = AutoTokenizer.from_pretrained("bert-large-uncased-whole-word-masking-finetuned-squad") model = AutoModelForQuestionAnswering.from_pretrained("bert-large-uncased-whole-word-masking-finetuned-squad")

text = r""" As checked Dis is not yet on boarded to ARB portal, hence we cannot upload the invoices in portal """

questions = [ "Dis asked if it is possible to post the two invoice in ARB.I have not access so I wanted to check if you would be able to do it.", ]

for question in questions: inputs = tokenizer.encode_plus(question, text, add_special_tokens=True, return_tensors="pt") input_ids = inputs["input_ids"].tolist()[0]

text_tokens = tokenizer.convert_ids_to_tokens(input_ids)
answer_start_scores, answer_end_scores = model(**inputs)

answer_start = torch.argmax(
    answer_start_scores
)  # Get the most likely beginning of answer with the argmax of the score
answer_end = torch.argmax(answer_end_scores) + 1  # Get the most likely end of answer with the argmax of the score

answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[answer_start:answer_end]))

print(f"Question: {question}")
print(f"Answer: {answer}\n")

The answer that I get here is:
Question: Dis asked if it is possible to post the two invoice in ARB.I have not access so I wanted to check if you would be able to do it. Answer: dis is not yet on boarded to ARB portal
How do I get a score for this answer? Score here is very similar to what is I get when I run Question-Answer pipeline .
I have to take this approach since Question-Answer pipeline when used is giving me Key Error for the below code
from transformers import pipeline
nlp = pipeline("question-answering")

context = r""" As checked Dis is not yet on boarded to ARB portal, hence we cannot upload the invoices in portal. """
print(nlp(question="Dis asked if it is possible to post the two invoice in ARB?", context=context))

Context:
See how decode is called here 1 and defined here.
You can modify decode like this to get the scores:
def decode(start: np.ndarray, end: np.ndarray, topk: int, max_answer_len: int):

    # Compute the score of each tuple(start, end) to be the real answer
    outer = np.matmul(np.expand_dims(start, -1), np.expand_dims(end, 1))

    # Remove candidate with end < start and end - start > max_answer_len
    candidates = np.tril(np.triu(outer), max_answer_len - 1)

    #  Inspired by Chen & al. (https://github.com/facebookresearch/DrQA)
    scores_flat = candidates.flatten()

    if topk == 1:
        idx_sort = [np.argmax(scores_flat)]
    elif len(scores_flat) < topk:
        idx_sort = np.argsort(-scores_flat)
    else:
        idx = np.argpartition(-scores_flat, topk)[0:topk]
        idx_sort = idx[np.argsort(-scores_flat[idx])]

    idx = np.argpartition(-scores_flat, topk)[0:topk]
    idx_sort = idx[np.argsort(-scores_flat[idx])]

    starts, ends = np.unravel_index(idx_sort, candidates.shape)[1:]
    scores = candidates[0, starts, ends]
    return starts, ends, scores


Fine-tuning Wav2Vec2 for English ASR with  on local machine Transformers

Question:
I’m running https://huggingface.co/blog/fine-tune-wav2vec2-english#training–evaluation 7 example on my local machine and getting Training Loss=nan

Step
Training Loss
Validation Loss
Wer
Runtime
Samples Per Second

200
nan
13.842948
2.703102
204.199500
8.227000

400
nan
13.842948
2.703102
204.301000
8.223000

600
nan
13.842948
2.703102
204.371700
8.220000

Local modifications to the example
training_args = TrainingArguments(
output_dir="./wav2vec2-base-timit-demo",
group_by_length=True,
per_device_train_batch_size=4, ** changes from 32 **
…
save_steps=200,
eval_steps=200,
logging_steps=100,
…
)

Context:
Update - by moving the job to the CPU the Training loss is getting values.
steps so far

make cuda unavailable
import torch

torch.cuda.is_available = lambda : False

disabling Mixed precision
training_args = TrainingArguments(
…
# fp16=True,
…)

*Mixed precision training with AMP or APEX (--fp16) and FP16 evaluation can only be used on CUDA devices.  *


Question:
Hello there! I have a question regarding the fine-tuning of mbart. I did the training like the example here transformers/examples/pytorch/translation at v4.6.1 · huggingface/transformers · GitHub and obtained a model pytorch_model.bin
However when trying to use the model to translate I get an UnicodeDecodeError UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 64: invalid start byte
The complete error is the following. As far as I can see it produces when loading the model I obtained from fine-tuning.
Traceback (most recent call last):
  File "mbart/predict.py", line 41, in <module>
    main()
  File "mbart/predict.py", line 21, in main
    model = MBartForConditionalGeneration.from_pretrained(opt.model)
  File "/home/claudia/anaconda3/envs/speechenv/lib/python3.7/site-packages/transformers/modeling_utils.py", line 1080, in from_pretrained
    **kwargs,
  File "/home/claudia/anaconda3/envs/speechenv/lib/python3.7/site-packages/transformers/configuration_utils.py", line 427, in from_pretrained
    config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/claudia/anaconda3/envs/speechenv/lib/python3.7/site-packages/transformers/configuration_utils.py", line 495, in get_config_dict
    config_dict = cls._dict_from_json_file(resolved_config_file)
  File "/home/claudia/anaconda3/envs/speechenv/lib/python3.7/site-packages/transformers/configuration_utils.py", line 578, in _dict_from_json_>
    text = reader.read()
  File "/home/claudia/anaconda3/envs/speechenv/lib/python3.7/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 64: invalid start byte

Any ideas on how to solve this?
Thanks in advance

Context:
maybe your inputpath is xxxxxxxxxxxxx/pytorch_model.bin
change to xxxxxxxxxxxxx/
have a try


Can’t load pre-trained tokenizer with additional new tokens

Question:
I first pretrained masked language model by adding additional list of words to the tokenizer. Then I saved the pretrained model and tokenizer.
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
model = AutoModelForMaskedLM.from_pretrained(
        'bert-base-uncased'
)
tokenizer.add_tokens(list_of_words)
model.resize_token_embeddings(len(tokenizer))

trainer.train()
model_to_save = model.module if hasattr(model, 'module') else model
model_to_save.save_pretrained(data_args.output_file)
tokenizer.save_pretrained(data_args.output_file)

After that, I want to load the pre-trained tokenizer and model by
tokenizer = BertTokenizer.from_pretrained(model_args.model_name_or_path)
encoder = BertModel.from_pretrained(model_args.model_name_or_path, num_labels=num_classes)
tokenizer.add_tokens(list_of_words)
encoder.resize_token_embeddings(len(tokenizer))

However, an error occurred as shown below and it seems that the pretrained tokenizer couldn’t be loaded correctly.
AssertionError: Non-consecutive added token 'IncelTears' found. Should have index 30525 but has index 30526 in saved vocabulary.

Does anyone have an idea on this? Thanks a lot.

Context: Which line threw the error? If it’s tokenizer.add_tokens(list_of_words), it’s because your tokenizer already has those words added from the first sample, so you can’t re-add them.


Why transformer overfit quickly? how to solve it?

Question:
Hi
I have a general question and appreciate your feedback on it.
I am new to transformers. My main problem is that it overfits so quickly, I am using regularization methods such as augmentation and dropout, but after 2 epochs my validation accuracy starts to drop while the training accuracy reaches to highest (basically my model overfit).
do you have any suggestions?
Interestingly I never see this behavior when I use convolutions…

Context:
My personal thought is if your data is less, it will overfit quickly. If you want to avoid it, reduce epochs. But, best way is to gather more data.
Having said that neither transformer nor neural networks suffer too much from overfitting. Some papers are there I guess. It’s good in generalizing most of the times. Remember transformer like models have quite good number of parameters, that’s also one reason of overfitting. But in downstream tasks, even if it overfits, it’s useful right.
Pre training is like a person who graduate with Masters. Fine tuning is like doing PhD ( except here it is quick  , use your graduation skills to be an expert in specific field. So overfitting is okay. Personal opinion only.


Question:
Hi everyone,
in my code I instantiate a trainer as follows:
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    compute_metrics=compute_metrics,
)

I don’t specify anything in the “optimizers” field as I’ve always used the default one (AdamW).
I tried to create an optimizer instance similar to the default one so I could try to change the learning rate (lr).
The code I used to simulate the default optimizer is the following:
no_decay = ["bias", "LayerNorm.weight"]
optimizer_grouped_parameters = [
    {
        "params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],
        "weight_decay": 0.0,
    },
    {
        "params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],
        "weight_decay": 0.0,
    },
]
optimizer = AdamW(optimizer_grouped_parameters, lr=5e-05, eps=1e-08)
scheduler = get_linear_schedule_with_warmup(
    optimizer, num_warmup_steps=0, num_training_steps=750
)

and then pass the parameters optimizer and scheduler to the optimizer field of the trainer.
The problem is that with this definition of optimizer, I have different results than the default one (even if I thought they were identical). What should I change to create an optimizer identical to the default one, but where I can change the lr directly from my code?
Thanks!

Context: At first glance, it might be linked to the number of training_steps? Are you sure your other training does 750 steps? Also I don’t know what your training_args are but if any of them don’t use the default value, that could also be the reason for the change in results.


Question:
I’m pre training a distillBert model from scratch and saving the model every 300 steps , When trying to load a checkpoint to continue training from the Trainer show that it’s skipping the trained steps but it just starts from 0 and doesn’t start logging or saving until the trainer passes the number of skipped steps.
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = DistilBertForMaskedLM.from_pretrained("/content/drive/My Drive/AIMBert/output_gpu/checkpoint-48000",config=config).to(device)

training_args = TrainingArguments(
output_dir="/content/drive/My Drive/AIMBert/output_gpu",
logging_dir='/content/drive/My Drive/AIMBert/logs_gpu',
overwrite_output_dir=True,
num_train_epochs=10,
per_device_train_batch_size=32,
per_device_eval_batch_size = 32,
logging_steps = 100,
save_steps=300,
save_total_limit=5,
evaluation_strategy = "steps",
eval_steps=50000,
seed = 42,
prediction_loss_only=True,
fp16 = True
)

trainer = Trainer(
model=model,
args=training_args,
data_collator=data_collator,
train_dataset= processed_train_dataset,
eval_dataset = processed_valid_dataset
)

trainer.train('/content/drive/My Drive/AIMBert/output_gpu/checkpoint-48000')

Context: What do you mean by “It starts at 0?” Also, which version of transformers are you using?


Extracting embeddings with distilbert? (in tensorflow)

Question:
Hello,
I am trying to understand the transformers architecture better and in particular to extract the contextual embeddings for a given sentence.
I know I can use the pipeline feature-extraction but I would like to extract them manually, but consider the small example below. Unfortunately, the last hidden states cannot be the contextual embeddings : I get a 2-dimensional vector whereas the embeddings have hundreds of dimensions.
import tensorflow as tf
from transformers import AutoTokenizer, TFAutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained('distilbert-base-uncased-finetuned-sst-2-english')
model = TFAutoModelForSequenceClassification.from_pretrained('distilbert-base-uncased-finetuned-sst-2-english')

input_ids = tf.constant(tokenizer.encode("Hello I am a dog."))[None, :]
outputs = model(input_ids)
last_hidden_states = outputs[0]

last_hidden_states.numpy()
Out[22]: array([[-1.651872 ,  1.6822953]], dtype=float32)

What is the issue here? Thanks!

Context: You should use the model without head for this, given by the class TFAutoModel. Here you use a model for sequence classification.


Question:
Hello,
I am trying to upgrade to the latest distribution of transformers (4.9.1 according to Release v4.9.1: Patch release · huggingface/transformers · GitHub).
I currently have the version 4.3.3 installed and it works fine. So, I downloaded the 4.9.1 zip file, extracted to a directory, cd to that directory and run python setup.py install.
The install went fine (except for the huggingface-hub requirement at the end — which is blocked by my firewall). However, when I start spyder again the version is still 4.3.3.
Am I doing something wrong to update the package offline (on windows)?
Thanks!

Context: I was able to upgrade by using pip instead. Weird that the good old python setup.py install would not work in this case.


Question:
image1511×592 50.1 KB

can anyone help me out?

Context: I don’t think Wav2Vec2 has beem updated to work with saved models, cc @Rocketknight1


[Spaces] Streamlit does not support external components

Question:
Hello,
I have created streamlit tags but I was playing around with streamlit in Spaces Beta and I realised that it does not support any of the custom components like streamlit-tags
Any support available for spaces?
Regards,
Gagan Bhatia

Context: HF Spaces now supports Streamlit components (see Changelog 2)