noamaanMulla-03/transformers-issues huggingface dataset:
Loading a set of tokenized files for training
Author: nlpravi
Pull request: no
Labels: name: wontfix

I have a directory of files that are already tokenized using a pretrained tokenizer. Each file is a pickle file containing a list of objects where each object corresponds to a text sequence containing input_ids and attention_masks. The directory has thousands of files. I'm looking for an efficient way to load the data for training using Trainer. Do I have to write my own Dataloader or do I create a custom dataset using Datasets? 

Thank you. 

Comments:
In this case, you could write a custom `dataset` that will read your pickle files and return the examples from `__getitem__` method.

If you are looking for an efficient way of pre-tokenizing the dataset, saving/caching it for future use, and loading it for training then I would recommend you to take a look at [datasets](https://github.com/huggingface/datasets) library. It takes care of caching your pre-processed data and loading it efficiently (lazy loading, so memory won't blow up)

Thanks Suraj 

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
Training loss not getting logged
Author: kunalpagarey
Pull request: no
Labels: name: wontfix

While training GPT2 using run_clm.py I wanted to track the training loss as well but could not find a way to do that with evaluation strategy = epoch. So I tried to look deeper into the code and found that may be adding `control.should_log = True` after line referred below will start logging training loss after every epoch. 
https://github.com/huggingface/transformers/blob/ae333d04b29a25be1a70eaccd6260c294c243c5b/src/transformers/trainer_callback.py#L422

Please correct me if I am wrong and suggest how should I track training loss per epoch?

Thanks in advance.

Comments:
@sgugger is the best suited to answer you

This option is not implemented in Trainer.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
Prediction problem of glue task
Author: anbo724
Pull request: no
Labels: name: wontfix


Comments:
I have trained the glue task for mrpc, and I want to load the pretrained model and predict for new sentence pairs.

```py
eval_dataset = load_dataset( "json", data_files={"test": "/home/aa/paraphrase/data/qqp/tt.json"})
eval_dataset = eval_dataset.map(preprocess_function, batched=False, load_from_cache_file=True)
print(eval_dataset['test']['idx'])

eval_dataset.remove_columns_("label")

trainer = Trainer(model=model, tokenizer=tokenizer)
predictions = trainer.predict(test_dataset=eval_dataset).predictions
print(predictions)
predictions = np.array([softmax(element) for element in predictions])[:, 1]
```

And I got this :

```
load model finish
Using custom data configuration default
Reusing dataset json (/home/aa/.cache/huggingface/datasets/json/default-8988cd19f10ded6e/0.0.0/70d89ed4db1394f028c651589fcab6d6b28dddcabbe39d3b21b4d41f9a708514)
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:00<00:00, 1413.70ex/s]
[0, 1, 2, 3, 4, 5, 6, 7, 8]
Traceback (most recent call last):
  File "predict.py", line 89, in <module>
    predictions = trainer.predict(test_dataset=eval_dataset).predictions
  File "/home/aa/anaconda3/envs/transformer/lib/python3.7/site-packages/transformers/trainer.py", line 1381, in predict
    test_dataloader, description="Prediction", ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix
  File "/home/aa/anaconda3/envs/transformer/lib/python3.7/site-packages/transformers/trainer.py", line 1441, in prediction_loop
    for step, inputs in enumerate(dataloader):
  File "/home/aa/anaconda3/envs/transformer/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/home/aa/anaconda3/envs/transformer/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/aa/anaconda3/envs/transformer/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/aaanbo/anaconda3/envs/transformer/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
KeyError: 0
```

Why I got keyerror?
Anyone can help or show me how to use the pretrained models for sentence pair prediction?


This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
[s2s] test_finetune_trainer_slow fails when run in group
Author: stas00
Pull request: no
Labels: name: wontfix

On dual gpu when running `test_finetune_trainer.py::TestFinetuneTrainer::test_finetune_trainer_slow` alone - all is good.

when running it with all the other tests in that file it fails:

```
RUN_SLOW=1 pytest -sv test_finetune_trainer.py
[...]
self = <seq2seq.test_finetune_trainer.TestFinetuneTrainer testMethod=test_finetune_trainer_slow>

    @slow
    def test_finetune_trainer_slow(self):
        # There is a missing call to __init__process_group somewhere
        output_dir = self.run_trainer(
            eval_steps=2, max_len="128", model_name=MARIAN_MODEL, num_train_epochs=10, distributed=False
        )

        # Check metrics
        logs = TrainerState.load_from_json(os.path.join(output_dir, "trainer_state.json")).log_history
        eval_metrics = [log for log in logs if "eval_loss" in log.keys()]
        first_step_stats = eval_metrics[0]
        last_step_stats = eval_metrics[-1]

>       assert first_step_stats["eval_bleu"] < last_step_stats["eval_bleu"]  # model learned nothing
E       AssertionError: assert 0.0 < 0.0

test_finetune_trainer.py:130: AssertionError
----------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------
WARNING  seq2seq.finetune_trainer:finetune_trainer.py:160 Process rank: -1, device: cuda:0, n_gpu: 2, distributed training: False, 16-bits training: False

==================================================================== short test summary info ====================================================================
FAILED test_finetune_trainer.py::TestFinetuneTrainer::test_finetune_trainer_slow - AssertionError: assert 0.0 < 0.0
================================================ 1 failed, 7 passed, 1 skipped, 17 warnings in 102.82s (0:01:42) ================================================
```

For some reason it fails to learn anything when some other tests run before it. 

tested with pytorch-nightly + py38.


Comments:
I also have the same failure on one GPU on my side FYI (but no failure when run on its own).

Thank you, @sgugger  - your input helped a lot to reduce the sequence quickly! So this sequence fails:
```
CUDA_VISIBLE_DEVICES=0 RUN_SLOW=1 pytest \
test_finetune_trainer.py::TestFinetuneTrainer::test_finetune_trainer_apex \ 
test_finetune_trainer.py::TestFinetuneTrainer::test_finetune_trainer_slow
```
Something about apex.

This sequence with another similar test before it but no apex doesn't fail:
```
CUDA_VISIBLE_DEVICES=0 RUN_SLOW=1 pytest \
test_finetune_trainer.py::TestFinetuneTrainer::test_finetune_trainer_no_dist \
test_finetune_trainer.py::TestFinetuneTrainer::test_finetune_trainer_slow
```

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
Load from a TF 1.0 checkpoint in modeling_tf_utils.py
Author: vsuarezpaniagua
Pull request: no
Labels: name: wontfix

In the file modeling_utils.py, we can load a TF 1.0 checkpoint as is indicated in this [line](https://github.com/huggingface/transformers/blob/fb650df8590f796663226132482d09da5b0fb613/src/transformers/modeling_utils.py#L930). However, in the file modeling_tf_utils.py, which is the same version for TF, we can not load models from TF 1.0, and it says expecifically that you can as:
`            >>> model = BertModel.from_pretrained('./tf_model/my_tf_checkpoint.ckpt.index', from_tf=True, config=config)`
But there is no _if_ for
`os.path.isfile(os.path.join(pretrained_model_name_or_path, TF_WEIGHTS_NAME + ".index"))`



Comments:
Hey @vsuarezpaniagua,

Great point! I noticed the same thing actually a couple of days ago as well with @jplu. I think we should add this functionality to `modeling_tf_utils.py`. It should be very similar to how it's done in the corresponding code in `modeling_utils.py`, and would require a new `load_tf1_weights` for TF2 models. 

Pinging @jplu, @LysandreJik, @sgugger here as well for some brainstorming on the importance of this feature request and how to best design it if neeed.

Thank you for taking it into consideration. Also, I saw that the _**EvaluationStrategy**_ for _epoch_ is not working using it in _training_args_tf.py_ for building a [TFTrainer](https://github.com/huggingface/transformers/blob/ec07da65e25562040581febaf9b400a462962961/src/transformers/trainer_tf.py#L49) in _trainer_tf.py_. And I think this is because there are not _self.control.should_evaluate_ or _self.control.should_save_ as there are in the Torch implementations _trainer.py_ and _training_args.py_. Having similar code for both implementations could solve all these problems and easier to follow.

> Hey @vsuarezpaniagua,
> 
> Great point! I noticed the same thing actually a couple of days ago as well with @jplu. I think we should add this functionality to `modeling_tf_utils.py`. It should be very similar to how it's done in the corresponding code in `modeling_utils.py`, and would require a new `load_tf1_weights` for TF2 models.
> 
> Pinging @jplu, @LysandreJik, @sgugger here as well for some brainstorming on the importance of this feature request and how to best design it if neeed.



The TF Trainer is off of maintenance since a while in order to be rethought when we can dedicate a bit of time to it. Not the current TF priority unfortunately. But at some point it is our plan to make the TF Trainer catching up his late on the PT one.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
Save underlying BertModel only
Author: pugantsov
Pull request: no
Labels: name: wontfix

I currently have a custom model on top of a pretrained BERT Model, which is effectively just dropout and a linear layer on top for classification. I want to be able to train this classifier, updating the underlying weights, but then to save the `BERTModel` underneath (w/o linear classification layer) so that I can load from file and use this as my input to the same custom model. Is there a way I can access and save the underlying transformer to then be used again like this?

Code for clarification:
```
class BERTBinaryClassifier(torch.nn.Module):

    def __init__(self, model_name_or_path: str):
        super(BERTBinaryClassifier, self).__init__()
        self.bert = ModelSelection.get_model(
            model_name=model_name_or_path)
        self.drop = torch.nn.Dropout(p=0.3)
        self.out = torch.nn.Linear(self.bert.config.hidden_size, 2).cuda()

    def forward(self, inputs):
        logits, embs, *_ = self.bert(**inputs)
        output = self.drop(logits)
        return self.out(output), embs
```

Right now I'm just outputting the logits and hidden_state for future usage but I'd like to be able to use this same function to effectively load a `BertModel` like I do at the start here with `self.bert` (which just loads `BertModel.from_pretrained` and save my Pytorch classifier as a whole, separately.

Would it be as simple as say accessing `self.bert.bert` and then saving it that way?

Comments:
If your bert model is an instance of `BertModel` then you should be able to save it using `self.your_model.bert.save_pretrained(path)` and load using `.from_pretrained`

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
bert-mlm-converting from tensorflow to pytorch
Author: ahmedkotb98
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
hello, i convert bert model from tensorflow to pytorch to fine tune it and in train function specially in loss , pred_masks = model(inputs, labels) i got this error not enough values to unpack (expected 2, got 1) why model return one only it should return two one for loss and the second for pred_masks

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
LXMERT cross_modality_matching logits order in code vs. documentation
Author: LetiP
Pull request: no
Labels: name: wontfix

Issue #7266 evolved into an entirely different issue described below.

# Possible documentation error
Independently of considering #8333 or not, it is clear that the results of the cross-modality matching in LXMERT (for example in [this demo](https://colab.research.google.com/drive/18TyuMfZYlgQ_nXo-tr8LCnzUaoX0KS-h?usp=sharing)) are better (below random chance vs. above random chance) when the first logit of the `output_lxmert['cross_relationship_score']` *– (torch.FloatTensor of shape (batch_size, 2))* represents *a mismatch* rather than *a match*.

This **contradicts the [documentation](https://huggingface.co/transformers/model_doc/lxmert.html)** which in my understanding assigns the first logit to `is_match` (True):
> cross_relationship_score – (torch.FloatTensor of shape (batch_size, 2)): Prediction scores of the textual matching objective (classification) head (scores of True/False continuation before SoftMax).

@LysandreJik, @eltoto1219: Is the documentation right or wrong? How was the model trained and which logit was intended to predict the match and which one the mismatch?

I would be very happy to have a clear answer about the order of the logits in training and code and its correspondence to the documentation. I know how easy it is to flip the logits around and choose the one delivering better results on the data at hand. But in my understanding of scientific conduct, I can not just choose this by wishful thinking and ignoring the documentation: there might be other things going on exhibiting this behavior.

Thank you in advance!

Comments:
This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
MLFlow logger breaks training
Author: david-waterworth
Pull request: no
Labels: name: wontfix

I'm running into two issues when I try and train with mflow installed.

The main one is transformers logs every single parameter and mlflow has restrictions on parameter size. So when I tried to train a `AutoModelForSequenceClassification` using the code below

```
num_labels=25
config = AutoConfig.from_pretrained(
    './models/roberta',
    num_labels=num_labels)
model = AutoModelForSequenceClassification.from_pretrained(
    './models/roberta', 
    config=config)
```

An exception is thrown by the mlflow/utils/validation.py - specifically that the length of the value of parameter with key "id2label" is > 250 (this is a character limit so it's trying to send the dict as a string)

The second issue is if I resume training after this error I get a further mlflow exception saying that the run is still in progress - I have to manually call

`mlflow.end_run()`

I'm not sure how to work around this (for now I've uninstalled mlflow). Can I tell the trainer not to log certain parameters?


Comments:
This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
Saving Pretrained Tokenizer
Author: david-waterworth
Pull request: no
Labels: name: wontfix

I created a custom `tokenizers.Tokenizer` and saved it as follows

```
tokenizer.model.save("./tokenizer")
tokenizer.save("./tokenizer.json") 
```

This produces 3 files, merges.txt, vocab.json and tokenizer.json

Then I created a `transformers.RobertaTokenizerFast` and saved it to the same folder

```
tokenizer = RobertaTokenizerFast.from_pretrained("./tokenizer")  
tokenizer.save_pretrained("./tokenizer") 
```

This adds `special_tokens_map.json` and `tokenizer_config.json`

I then saved it to another folder to simulate what happens when I train my model

```
tokenizer.save_pretrained("./model") 
tokenizer = RobertaTokenizerFast.from_pretrained("./model")  
```

What I noticed was `tokenizer_config.json` contains a key `name_or_path` which still points to `./tokenizer`, so what seems to be happening is `RobertaTokenizerFast.from_pretrained("./model") ` is loading files from two places (./model and ./tokenizer)

Not sure if this is expected, it seems that the tokenizer_config.json should be updated in save_pretrained, and tokenizer.json should be saved with it?

Or perhaps this is just an issue because I'm training my tokenizer in a subdirectory of the model folder?


Comments:
I realised that

`tokenizer.model.save("./tokenizer")`

Is unnecessary. I've started saving only the `tokenizer.json` since this contains not only the merges and vocab but also the pipeline.

And I noticed that `tokenizer.save_pretrained()` has a parameter  `legacy_format` which defaults to True. When I set it to false it properly round trips (i.e. saves out the unified tokenizer.json rather than the model files (merges.txt and vocab.json)

The only issue now is if I create a trainer

```
trainer = Trainer(
    model=model,
    tokenizer=tokenizer
    args=training_args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["validation"] ,
    data_collator=data_collator,
)

logger.info("*** Train ***")
trainer.train(model_path=model_path)
trainer.save_model()
```

The last line doesn't accept kwargs so it saves the tokenizer in the legacy format.

A workaround is to do it manually ie.

```
trainer = Trainer(
    model=model,
    #tokenizer=tokenizer
    args=training_args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["validation"] ,
    data_collator=data_collator,
)

logger.info("*** Train ***")
trainer.train(model_path=model_path)
trainer.save_model()
tokenizer.save_pretrained(output_dir, legacy_format=False)
```

Only issue is I don't see a workaround to ensure the checkpoints are created using `legacy_format=False`





This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
when to use sortish sampler 
Author: rabeehkarimimahabadi
Pull request: no
Labels: name: wontfix

Hi,
I appreciate adding to the documentation on when to use sortish sampler and which platform this works/how much it impacts the speed, this is related to seq2seq folder of huggingface, for Seq2seqDataset.
thanks


Comments:
Is there any update on this? 

Seems like it allows dynamic batching. 

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
Log metrics along with hparams in TensorBoardCallback
Author: howardlau1999
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

Log metrics along with hparams in TensorBoardCallback.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

It seems useless to log hparams with empty metric dict because the training arguments are already logged in the text section. Now users can see only a blank screen when they click on the HPARAMS of the TensorBoard. So I think it will be better to call `add_hparams` with evaluation metrics when available. Otherwise just don't call this function.



Comments:
This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.