noamaanMulla-03/transformers-issues huggingface dataset:
MNLI: BERT No Training Progress
Author: ghost
Pull request: no
Labels: name: wontfix

## 🐛 Bug

I am using BERT and have successfully set up pipelines for 7/8 GLUE tasks, and I find comparably good accuracy on all of them. However, for MNLI task, training loss does not converge at all. I am correctly using 3 classes with `num_classes`. In fact, I have even tried reduced the scope of MNLI to a 2-class problem (entailment vs neutral/contradiction) for testing purposes to see if this is the issue, and the model fails to converge on this training task as well. However, the exact same code (with only the dataset switched) works for MRPC, for example.

Model I am using (Bert, XLNet....): **BERT**

Language I am using the model on (English, Chinese....): **English**

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: **see co-lab**

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: **MNLI**
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

See linked co-lab for minimum viable example
https://colab.research.google.com/drive/1Thdrpvp0uX2TCaCpqsLoUVn7DuRLlILQ

## Expected behavior

I expect to see good training progress; instead, training loss does not converge.

## Environment

* OS: **Linux**
* Python version: **3.6**
* PyTorch version: **1.3.0**
* PyTorch Transformers version (or branch): **2.1.1**
* Using GPU ? **Yes**
* Distributed of parallel setup ? **No, single GPU**
* Any other relevant information: **None**


Comments:
I'm assuming you did multiple runs with ≠ seeds?

Yes -- I have tried with multiple different seeds.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> Yes -- I have tried with multiple different seeds.

@evanweissburg Have you found a solution? I'm also having issues with MNLI data. 

@anthonyfuller7 Nope. I think one resolution was to not use BERT large.

I too am facing the same issue with BERT and Albert as well. The model does not converge on fine-tuning and the loss does not decrease even over 5 epochs. Did someone manage to solve this issue?
comment correction in test_retrieval_rag.py?
Author: zuujhyt
Pull request: no

HI, in
https://github.com/huggingface/transformers/blob/master/tests/test_retrieval_rag.py#L223

comments of L223 and L224 are the same, maybe one of it should be "min inner product is reached with ...:
But I am not sure which one.
Pardon me if it is already correct.

Comments:
Both comments are right.

```python
self.assertEqual(doc_dicts[0]["id"][0], "1")  # max inner product is reached with second doc
```
makes sure the first retrieved document of the first query is the second document of the corpus (it's the one that maximizes the inner product)

while
```python
self.assertEqual(doc_dicts[1]["id"][0], "0")  # max inner product is reached with first doc
```
makes sure that the first retrieved document of the second query is the first document of the corpus (it's the one that maximizes the inner product)

To be clearer the indices in those statements could be written as 
```python
query_idx = 0
retrieved_document_idx = 0
expected_id = "0"
self.assertEqual(doc_dicts[query_idx]["id"][retrieved_document_idx], expected_id)
```

Thanks for your reply!
[apex.normalizations.FusedLayerNorm] torch.cuda.is_available() is redundant as apex handles that internally
Author: stas00
Pull request: yes

This PR is a follow up to https://github.com/huggingface/transformers/issues/9338

According to https://github.com/huggingface/transformers/issues/9338#issuecomment-752242098 we can just remove the `torch.cuda.is_available()` check before importing `apex.normalizations.FusedLayerNorm` and the multiprocess problem will go away.

Fixes #9338

@patrickvonplaten 

Comments:
Thanks a lot for digging into this @stas00
Fixes crash when `compute_metrics` is not passed to `Trainer` in run_mlm example
Author: galtay
Pull request: yes

# What does this PR do?

If I'm understanding the example run_mlm code correctly, the `metrics` attribute will not be present on the `train_result` object if `compute_metrics` is not passed to `Trainer`. This edit prevents the script from attempting to write the metrics to file if they don't exist. 

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@sgugger 
@stas00 
 


Comments:
In master `train` always returns `metrics`.

Is it possible that you are running the script from master but loading `transformers` that is pre-installed and it is not master? This metrics was added just recently.

Do you still get the error if you do:
```
git clone https://github.com/huggingface/transformers/
cd transformers
PYTHONPATH=src examples/language-modeling/run_mlm.py ...
```
This ensures that you're using the master version in the script.

Or alternatively if you tend to use the master a lot, install it with `pip install -e .[dev]` which allows you to `git pull` and not needing to reinstall anything.

To verify that there is no problem in master I have just run:

```
python run_mlm.py --model_name_or_path roberta-base --dataset_name wikitext \
--dataset_config_name wikitext-2-raw-v1 --do_train --output_dir /tmp/test-mlm
```
and got:
```
INFO|trainer.py:1248] 2020-12-29 22:49:16,276 >> Saving model checkpoint to /tmp/test-mlm
[INFO|configuration_utils.py:289] 2020-12-29 22:49:16,277 >> Configuration saved in /tmp/test-mlm/config.json
[INFO|modeling_utils.py:814] 2020-12-29 22:49:16,818 >> Model weights saved in /tmp/test-mlm/pytorch_model.bin
12/29/2020 22:49:16 - INFO - __main__ -   ***** Train results *****
12/29/2020 22:49:16 - INFO - __main__ -     epoch = 3.0
12/29/2020 22:49:16 - INFO - __main__ -     train_runtime = 383.9452
12/29/2020 22:49:16 - INFO - __main__ -     train_samples_per_second = 4.688
```

So all seems to be in norm.


Thanks for taking a look @stas00 and for the examples ! You are correct, I had transformers 4.1.1. Looks fine when run on master. I'll close this PR
TextDataset support for tensorflow?
Author: mymusise
Pull request: no
Labels: name: wontfix

Hey, guys. I find [`TextDataset`](https://github.com/huggingface/transformers/blob/9a21b50614991889f11dbe0743af25923765f9e9/src/transformers/data/datasets/language_modeling.py#L20) and `LineByLineTextDataset` is a great design, it help people build input data more faster. But it's a pity that it only support **pytorch** now. Is there any possible to support **tensorflow**?

Comments:
There is no plan for that as both those APIs will be deprecated soon. Users should directly use the [Datasets](https://github.com/huggingface/datasets) library which works for both PyTorch and TF. There are examples of how to replicate `TextDataset` and `LineByLineTextDataset` using that library in the new [`run_clm`](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_clm.py) and [`run_mlm`](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py) scripts. To convert the datasets to the TF format, just use their `set_format` method (see the [doc here](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.set_format)). 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Perplexity in T5/BART
Author: Palipoor
Pull request: no

Hi,
Is there a way to compute perplexity for T5 or BART outputs?

Comments:
T5 uses CrossEntropy loss so I think you can use `torch.exp(loss)` to calculate perplexity.

Thanks!

@patil-suraj Sorry to bother you again, I thought my problem has been solved, but I don't know how to deal with the loss. It's a (1,1,vocab_size) tensor. I don't even have an intuition about what that means. Can you help me?

It seems that you are not passing `lm_labels`. When you pass `lm_labels` to T5/BART's forward methods then the first element of the output tuple is the loss, otherwise its `prediction_scores` tensor which you are receiving.

hi @patil-suraj bart doesnt have lm_labels has argument

`lm_labels` is renamed  to `labels`

Thanks for the prompt response, @patil-suraj , previously before trying Bart, I trained it with t5,the valid loss gets to almost zero, but still found it difficult generating correct translation. I would love to know reasons for that. Thanks

I'm not sure which problem you are talking about, hard to answer without knowing the context.

Hi patil,

I am working on a translation task/competition, so as a text2text
framework, I used t5 even though it is not pretrained on my source language
just for the knowledge sake,
The notebook link here

https://colab.research.google.com/drive/1w0brh9KULV4zDmgMme4csXeCpfAR960N?usp=sharing

the test and valid loss was declining to zero even though the later
generated predictions wasn't good so I changed to Bart even though no
improvement, I have no idea why? I would love to know the reason.

I found MarianMT just yesterday which was pretrained for my task yo-en, I
have started working with it. I think it would also be the case for MarianMt
Thank you.

On Fri, Jan 1, 2021, 8:38 AM Suraj Patil <notifications@github.com> wrote:

> I'm not sure which problem you are talking about, hard to answer without
> knowing the context.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/4210#issuecomment-753280527>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AOP3ZJFLJLLFGUSFCU3WKB3SXV3X7ANCNFSM4M3PUPXA>
> .
>
seq2seq/finetune.py: remove useless check
Author: sshleifer
Pull request: no
Labels: name: wontfix

Removed on built-in already
```python
        self.dataset_class = (
            Seq2SeqDataset if hasattr(self.tokenizer, "prepare_seq2seq_batch") else LegacySeq2SeqDataset
        )

```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
BertEncoder has no attribute 'bias' when convert tf checkpoint
Author: heraclex12
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help
BertEncoder object has no attribute 'bias' when convert tf checkpoint
<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
I tried to convert my pretrained BERT tf checkpoint but I got this error:
```
/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py in load_tf_weights_in_bert(model, config, tf_checkpoint_path)
    133                 pointer = getattr(pointer, "weight")
    134             elif scope_names[0] == "output_bias" or scope_names[0] == "beta":
--> 135                 pointer = getattr(pointer, "bias")
    136             elif scope_names[0] == "output_weights":
    137                 pointer = getattr(pointer, "weight")
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __getattr__(self, name)
    770                 return modules[name]
    771         raise ModuleAttributeError("'{}' object has no attribute '{}'".format(
--> 772             type(self).__name__, name))
    773 
    774     def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -> None:
ModuleAttributeError: 'BertEncoder' object has no attribute 'bias'
```


I used both load_tf_weights_in_bert function and BertForPreTraining but it was not working.

My code:
```
config = BertConfig.from_json_file('bertvn_base/bertvn_base_config.json')
model = BertForPreTraining.from_pretrained('bertvn_base/model.ckpt', from_tf=True, config=config)
```

My config:
```
{"attention_probs_dropout_prob": 0,
"hidden_act": "gelu",
  "hidden_dropout_prob": 0,
  "embedding_size": 768,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "max_position_embeddings": 192,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "num_hidden_groups": 12,
  "net_structure_type": 0,
  "gap_size": 0,
  "num_memory_blocks": 0,
  "inner_group_num": 1,
  "down_scale_factor": 1,
  "type_vocab_size": 2,
  "vocab_size": 120000
}
```

Thanks for your help!
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
ModelUtilsTest.test_model_from_pretrained failiing on CUDA
Author: sshleifer
Pull request: no
Labels: name: wontfix

Seems as though an `architecture` key is being added.

Not sure who to assign this to @LysandreJik.
```python
__________________ ModelUtilsTest.test_model_from_pretrained ___________________
[gw0] linux -- Python 3.7.6 /home/hf/actions-runner_transformers/_work/transformers/transformers/.env/bin/python

self = <tests.test_modeling_common.ModelUtilsTest testMethod=test_model_from_pretrained>

    @slow
    def test_model_from_pretrained(self):
        for model_name in BERT_PRETRAINED_MODEL_ARCHIVE_LIST[:1]:
            config = BertConfig.from_pretrained(model_name)
            self.assertIsNotNone(config)
            self.assertIsInstance(config, PretrainedConfig)
    
            model = BertModel.from_pretrained(model_name)
            model, loading_info = BertModel.from_pretrained(model_name, output_loading_info=True)
            self.assertIsNotNone(model)
            self.assertIsInstance(model, PreTrainedModel)
            for value in loading_info.values():
                self.assertEqual(len(value), 0)
    
            config = BertConfig.from_pretrained(model_name, output_attentions=True, output_hidden_states=True)
            model = BertModel.from_pretrained(model_name, output_attentions=True, output_hidden_states=True)
            self.assertEqual(model.config.output_hidden_states, True)
>           self.assertEqual(model.config, config)
E           AssertionError: BertConfig {
E             "_name_or_path": "bert-base-uncased",
E             "a[518 chars]22
E           }
E            != BertConfig {
E             "architectures": [
E               "BertForMaskedLM"
E            [478 chars]22
E           }

tests/test_modeling_common.py:1092: AssertionError

```

https://github.com/huggingface/transformers/runs/1303479917?check_suite_focus=true

Comments:
Assign me!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
tutorial document
Author: jc-hou
Pull request: no
Labels: name: wontfix

In the Translation section in https://huggingface.co/transformers/task_summary.html
"
Here is an example of doing translation using a model and a tokenizer. The process is the following:
1.Instantiate a tokenizer and a model from the checkpoint name. Summarization is usually done using an encoder-decoder model, such as Bart or T5.
2.Define the article that should be summarizaed.
3.Add the T5 specific prefix “translate English to German: “
4.Use the PreTrainedModel.generate() method to perform the translation.
"
1 and 2 seemed copies from Summarization section and not be modified accordingly.



Comments:
You're correct! @patrickvonplaten, git blame shows you as the author, want to fix?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to load tokenizer for models without vocab.txt?
Author: havetry
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
I want to use xlm-roberta-large model, but "https://huggingface.co/"just give the file named "xlm-roberta-large-tokenizer.json", and have no "vocab.txt", so how to use the package “XLMRobertaTokenizer” to load the the file  "xlm-roberta-large-tokenizer.json"?
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
Hello! In order to use the xlm-roberta-large model, why don't you use the `from_pretrained` method?

```py
from transformers import XLMRobertaModel

model = XLMRobertaModel.from_pretrained("xlm-roberta-large")
```

> Hello! In order to use the xlm-roberta-large model, why don't you use the `from_pretrained` method?
> 
> ```python
> from transformers import XLMRobertaModel
> 
> model = XLMRobertaModel.from_pretrained("xlm-roberta-large")
> ```
Thanks for your answer!
I tried do it like what you said, but I couldn't linked the URL to download the model, so I try to download the model、cofig、tokenizer to local and load it. 
so, The question what I said was I have not find the vocab.txt  to generate the tokenizer.

You don't need the URL to download the model, you can just use the identifier as its shown in my message. Or is there a reason why you want to have the URLs? 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Bart Caching: do we need encoder outputs after step 1?
Author: sshleifer
Pull request: no
Labels: name: wontfix

since projected cross attention k,v are cached, this doesn't seem like it's actually needed.
Maybe we could save some mem if we deleted them after we were done.

Comments:
Very good point! But I'm actually not sure if the memory consumption of `num_layers * cached k, v states` already makes memory consumption of `encoder_outputs` negligible. Would be cool to try out.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
TextDataset bug with big files
Author: paulomann
Pull request: no
Labels: name: wontfix

- `transformers` version: 3.0.2
- Platform: Linux-4.4.0-137-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help

@mfuntowicz

## Information

The model I am using (Bert, XLNet ...): XLMRobertaTokenizer

The problem arises when using:
* [ x ] my own modified script: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run the following code 
 ```Python
from transformers import XLMRobertaTokenizer, TextDataset
max_pos = 4096
tokenizer = XLMRobertaTokenizer.from_pretrained("xlm-roberta-base", model_max_length=max_pos)
tokenizer.model_max_length = max_pos
tokenizer.init_kwargs["model_max_length"] = max_pos
train_datapath = "path/to/train.raw"
train_dataset = TextDataset(
    tokenizer=tokenizer,
    file_path=train_datapath,
    block_size=tokenizer.max_len
)
```

## Error Messages

```
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted
```

## Expected behavior

My file `train.raw` contains 349912098 words with 2,3G. When I try with a small dataset (262703214 words and 251M), it works fine. I have modified this line in the TextDataset class https://github.com/huggingface/transformers/blob/a16e568f22a4d07813ba76343309ec20096115a5/src/transformers/data/datasets/language_modeling.py#L68 to understand where is the problem. It happens in the `tokenizer.tokenize(text)` part. I have changed it not to tokenize the entire text directly but to process chunks of the text each time, concatenating the result in a final list. Although memory hungry, this method works fine (and memory is not my problem).

**Note:** I have executed this script in a machine with 504G of RAM, and the script used approx. 36G when it died.


Comments:
Hi! Could you put the whole stack trace here? I fear it might be an internal sentencepiece error, for which we'll be unable to help and you would have more luck opening an issue on the sentencepiece repo directly.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Loading a pytorch quantized model
Author: amanpreet692
Pull request: no
Labels: name: wontfix

Hi, I quantized a pre-trained model via **torch.quantization.quantize_dynamic** and saved it using **save_pretrained**.
based on https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/dynamic_quantization_bert_tutorial.ipynb#scrollTo=foe-dVxHIgOC . 
On reloading it using **from_pretrained** again the model blows up to its original size and the resultant predictions are garbage as well.
Is there a way to properly save the quantized weights and reload them?
Opening an issue because I couldn't find a resolution for the same.
@mfuntowicz @VictorSanh @patrickvonplaten 

Comments:
Hello @amanpreet692 
Are you making sure you are reloading the saved (quantized) weights into a quantized model?
A dense checkpoint and its quantized counterpart have two different state dicts and I suspect you are re-initializing random matrices because when loading into from_pretrained it doesn't find the right keys in the state dict.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.