noamaanMulla-03/transformers-issues huggingface dataset:
support local_files_only option for tf models
Author: ogarin
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5116?src=pr&el=h1) Report
> Merging [#5116](https://codecov.io/gh/huggingface/transformers/pull/5116?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/355954ffca798bb81d9db8886e30ce10f11e8a40&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5116/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5116?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #5116   +/-   ##
=======================================
  Coverage   77.28%   77.29%           
=======================================
  Files         133      133           
  Lines       22134    22135    +1     
=======================================
+ Hits        17107    17109    +2     
+ Misses       5027     5026    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5116?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5116/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.44% <100.00%> (+0.01%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5116/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `75.00% <0.00%> (-0.41%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5116/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.06% <0.00%> (+0.12%)` | :arrow_up: |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/5116/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `39.62% <0.00%> (+0.23%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5116?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5116?src=pr&el=footer). Last update [355954f...fa3e6dc](https://codecov.io/gh/huggingface/transformers/pull/5116?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
[cleanup] generate_beam_search comments
Author: sshleifer
Pull request: yes
Labels: name: cleanup

This PR does two small things:

1) try to make `generate_beam_search`'s inline comments work with the code, and fix some typos.
2) make the `cur_len` argument to `BeamHypothesis.is_done` mandatory, since it is always specified. This simplifies the logic a tiny bit.



Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5115?src=pr&el=h1) Report
> Merging [#5115](https://codecov.io/gh/huggingface/transformers/pull/5115?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/355954ffca798bb81d9db8886e30ce10f11e8a40&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5115/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5115?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5115      +/-   ##
==========================================
+ Coverage   77.28%   77.30%   +0.01%     
==========================================
  Files         133      133              
  Lines       22134    22130       -4     
==========================================
+ Hits        17107    17108       +1     
+ Misses       5027     5022       -5     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5115?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5115/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.85% <100.00%> (+0.42%)` | :arrow_up: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5115/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.04% <100.00%> (+0.09%)` | :arrow_up: |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/5115/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `39.62% <0.00%> (+0.23%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5115?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5115?src=pr&el=footer). Last update [355954f...5d77afc](https://codecov.io/gh/huggingface/transformers/pull/5115?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great!
SLOW GPU test Failures
Author: sshleifer
Pull request: no
Labels: name: marian

```bash
=========================== short test summary info ============================
FAILED tests/test_modeling_marian.py::TestMarian_RU_FR::test_batch_generation_ru_fr
FAILED tests/test_modeling_marian.py::TestMarian_MT_EN::test_batch_generation_mt_en
==== 2 failed, 1110 passed, 392 skipped, 361 warnings in 1347.28s (0:22:27) ====
```

https://github.com/huggingface/transformers/runs/782480196?check_suite_focus=true

Comments:
Status Badge:
![Self-hosted runner (scheduled)](https://github.com/huggingface/transformers/workflows/Self-hosted%20runner%20(scheduled)/badge.svg)
Fine-Tuning GPT2
Author: apteryxlabs
Pull request: no

This post references a page that no longer exists (the run_lm_finetuning.py). Where is the updated page?

"You can look into GPT-2's training on the CLM task, which is done on WikiText-2 in this example."

_Originally posted by @LysandreJik in https://github.com/huggingface/transformers/issues/1145#issuecomment-526322616_

Comments:
Hi! This script has been replaced by `language-modeling/run_language_modeling.py`, because it now handles pre-training as well, not only fine-tuning. [Here's a useful link.](https://github.com/huggingface/transformers/tree/master/examples/language-modeling)
data_collator.py does not allow NoneType labels for test set predictions on Glue
Author: rsanjaykamath
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Distilbert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below) 
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)


## To reproduce

Steps to reproduce the behavior:

1. Open the  example Colab for text-classification from here https://huggingface.co/transformers/examples.html
2. Try to run the prediction function found in run_glue.py to predict on the official Glue test set. 
3. The error is as shown below. 

Earlier, this has worked with the exact same program, ever since an update recently, this error shows up. 

`TypeError                                 Traceback (most recent call last)
<ipython-input-16-9eecdd4d48b1> in <module>()
      2 output_mode = "classification"
      3 
----> 4 predictions = trainer.predict(test_dataset=test_dataset).predictions
      5 if output_mode == "classification":
      6     predictions = np.argmax(predictions, axis=1)

7 frames
/usr/local/lib/python3.6/dist-packages/transformers/data/data_collator.py in default_data_collator(features)
     45     if "label" in first:
     46         dtype = torch.long if type(first["label"]) is int else torch.float
---> 47         batch["labels"] = torch.tensor([f["label"] for f in features], dtype=dtype)
     48     elif "label_ids" in first:
     49         if isinstance(first["label_ids"], torch.Tensor):

TypeError: must be real number, not NoneType`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
The error can be seen in the Colab notebook here https://colab.research.google.com/drive/1H_92qdsOOql2hS210qNrfMEEMRcAoHD_?usp=sharing

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform: Colab
- Python version: NA
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?:


Comments:
Recommended batch size and epochs for finetuning on large data
Author: okgrammer
Pull request: no
Labels: name: wontfix

In the original paper, BERT model is fine-tuned on downstream NLP tasks, where the number of instances for each task is in the order of thousands to hundreds of thousands. In my case, I have about 5 million samples. I'm curious whether there are recommended batch size and epochs for such training size? I'm fine-tuning bert-base-multilingual on 4 GPUs and there is a lot of unused GPU memory with the default batch size of 32. Even after increasing it to 128 there is still free available memory.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@okgrammer Larger batch size often means lower accuracy but faster epochs.  You can try it by doing several runs of varying batch size while keeping other params constant. 

See, especially,  https://arxiv.org/pdf/1804.07612.pdf

> In the original paper, BERT model is fine-tuned on downstream NLP tasks, where the number of instances for each task is in the order of thousands to hundreds of thousands. In my case, I have about 5 million samples. I'm curious whether there are recommended batch size and epochs for such training size? I'm fine-tuning bert-base-multilingual on 4 GPUs and there is a lot of unused GPU memory with the default batch size of 32. Even after increasing it to 128 there is still free available memory.

I have exactly the same issue. Can anyone help? 
The pretraining is really slow with more than 90% GPU memory available. No matter how I increase the batch size, the GPU memory usage is minimal.
[cleanup] remove redundant code in SummarizationDataset
Author: sshleifer
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5119?src=pr&el=h1) Report
> Merging [#5119](https://codecov.io/gh/huggingface/transformers/pull/5119?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/355954ffca798bb81d9db8886e30ce10f11e8a40&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5119/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5119?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5119      +/-   ##
==========================================
- Coverage   77.28%   77.28%   -0.01%     
==========================================
  Files         133      133              
  Lines       22134    22134              
==========================================
- Hits        17107    17106       -1     
- Misses       5027     5028       +1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5119?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5119/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `75.00% <0.00%> (-0.41%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5119/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.26% <0.00%> (-0.16%)` | :arrow_down: |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/5119/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `39.62% <0.00%> (+0.23%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5119?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5119?src=pr&el=footer). Last update [355954f...09ff62a](https://codecov.io/gh/huggingface/transformers/pull/5119?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Unable to evaluate on fine-tuned bart for summarization
Author: cshivade
Pull request: no

Hello,

I used the [finetune_bart.sh ](https://github.com/huggingface/transformers/blob/master/examples/summarization/finetune_bart.sh) and was able to finetune the model on my task. I checked the output directory and checkpoints were stored and there were no errors.

Following the training, i ran evaluate_cnn.py using this command. 
```
python evaluate_cnn.py <path_to_test.source> test_generations.txt <model-name>  --score_path rouge_scores.txt
```
specified [here](https://github.com/huggingface/transformers/tree/master/examples/summarization). I made sure the above command points to test.source for the finetuning task.

However, the evaluation is unable to load the checkpoint and throws the following error
```
OSError: Can't load config for '/..PATH_TO../wiki_bart'. Make sure that:

- '/..PATH_TO../wiki_bart' is a correct model identifier listed on 'https://huggingface.co/models'

- or '/..PATH_TO../wiki_bart' is the correct path to a directory containing a config.json file
```
Seems like it is looking for the config.json which is not present along with the checkpoint files. 

I saw the finetune.py has a do_predict argument like do_train. Is that to be used instead of evaluate_cnn.py?

Can you please help?

Comments:
Got around this by running the following code to generate the config.json and then running evaluate_cnn.py as above.

```
from lightning_base import BaseTransformer
from finetune import SummarizationTrainer
import torch
from argparse import Namespace

args = Namespace(adam_epsilon=1e-08, cache_dir='', config_name='', data_dir='PATH_TO_DATA', do_predict=False, do_train=False, eval_batch_size=4, learning_rate=3e-05, max_source_length=400, max_target_length=400, model_name_or_path='facebook/bart-large', n_gpu=8, num_train_epochs=3, output_dir='PATH_TO_OUTPUT', tokenizer_name='', train_batch_size=4, warmup_steps=0, weight_decay=0.0)

model = SummarizationTrainer(args)
model = model.load_from_checkpoint('PATH_TO_CHECKPOINT')
torch.save(model.state_dict(), args.output_dir + '/pytorch_model.bin')
model.config.to_json_file(args.output_dir + '/config.json')
```
Input format for a BertTokenClassification task
Author: Kc2fresh
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help
https://stackoverflow.com/questions/61168882/processing-and-handling-input-ids-when-using-bert-for-token-classification

So i want to use BERT for semantic entity extraction. This is not quite the same as NER or POS tagging.

for example eg- Given a sentence:
```
A=The leak could have been stopped the same hour it was discovered if the well had a working shut-off valve
```
it returns two separate phrases
```
B= if the well had a working shut-off valve, and C= The leak could have been stopped the same hour it was discovered.
```
thus i pd read a three column csv file of  A, B, C, similar data. and BERTtokenised them and all, so my question, whats the appropriate way to load the data for training, does it have to be converted into a CONLL format? 

```
from torch.utils.data import TensorDataset, random_split
dataset = TensorDataset(input_ids, attention_masks, labels)
```
How do i put the data into input_ids


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[WIP] Add early stopping to the trainer
Author: primaprashant
Pull request: yes

closes #4894 

Comments:
Well that was quick. Awesome! Let me know if you need some help (for the pytorch part) or code review.

Looks like a duplicate of https://github.com/huggingface/transformers/pull/4186

> Looks like a duplicate of #4186

You are absolutely right. Closing this in favour of https://github.com/huggingface/transformers/pull/4186
`facebook/bart-large-mnli` - random accuracy on MNLI
Author: VictorSanh
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): BART

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: MNLI
* [ ] my own task or dataset: (give details below)

## To reproduce

There might be something wrong with the `facebook/bart-large-mnli` checkpoint (https://huggingface.co/facebook/bart-large-mnli). I can only get a random accuracy (0.35) on MNLI:
`python examples/text-classification/run_glue.py --data_dir <GLUE DATA DIR/MNLI> --task MNLI --model_name_or_path facebook/bart-large-mnli --output_dir ./dbg/ --max_seq_length 128 --overwrite_cache --do_eval`

I checked and the checkpoint has a classification head (`'classification_head.dense.bias', 'classification_head.out_proj.weight', 'classification_head.out_proj.bias'`) i.e. the classification head is not initialized randomly.

cc @sshleifer as discussed

Comments:
[bart-mnli] Fix class flipping bug
Author: sshleifer
Pull request: yes

Previously, `eval_mnli/acc` was 0.35, now .9. Same issue as other models ported from fairseq.
New results of Victor's command.
```bash
wandb:   eval_mnli-mm/acc 0.9001220504475184
wandb:           _runtime 274.7172155380249
wandb:          eval_loss 0.3496225342093929
wandb:      eval_mnli/acc 0.9011716760061131
wandb:         _timestamp 1592585765.2334569
```

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5141?src=pr&el=h1) Report
> Merging [#5141](https://codecov.io/gh/huggingface/transformers/pull/5141?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/e33929ef1e98a71ba7fc411f39cbb5451396ef02&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5141/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5141?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #5141   +/-   ##
=======================================
  Coverage   77.19%   77.20%           
=======================================
  Files         133      133           
  Lines       22232    22233    +1     
=======================================
+ Hits        17162    17164    +2     
+ Misses       5070     5069    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5141?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/data/datasets/glue.py](https://codecov.io/gh/huggingface/transformers/pull/5141/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL2RhdGFzZXRzL2dsdWUucHk=) | `86.56% <100.00%> (+0.20%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5141/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `75.00% <0.00%> (-0.41%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5141/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.28% <0.00%> (+0.12%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5141/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `85.96% <0.00%> (+0.14%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5141?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5141?src=pr&el=footer). Last update [e33929e...70c5736](https://codecov.io/gh/huggingface/transformers/pull/5141?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Why the weights are not intialized ?
Author: lemo2012
Pull request: no

03/03/2019 14:13:01 - INFO - pytorch_pretrained_bert.modeling -   Weights of BertForMultiLabelSequenceClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']
03/03/2019 14:13:01 - INFO - pytorch_pretrained_bert.modeling -   Weights from pretrained model not used in BertForMultiLabelSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias']

Comments:
I was wondering this myself.  It looks like there's some configuration mismatch -- some parameters found which aren't used, and a few expected that aren't found.

I'm not sure if this is expected, since the top-level task-specific classifier is correctly NOT pre-trained... or if it's something more.

(question about lower performance moved into new issue)


I dug up some related issues which confirms my guess above -- this kind of message is expected since the models are not yet find-tuned to the task.

https://github.com/huggingface/pytorch-pretrained-BERT/issues/161
https://github.com/huggingface/pytorch-pretrained-BERT/issues/180



Yes this is the expected behavior.
I don't want to make the warning messages says this is "all good" because in some case, depending on the model you are loading in, this could be an unwanted behavior (not loading all the weights).

Hello @thomwolf : I continued pre-training with bert-base-uncased without fine tuning on round about 22K sequences and the precision @ K for MaskedLM task did not change at all. Is the result legitimate or do I rather have a problem loading the weights? I received the same warning message/ INFO. The data set is from the automotive domain. At what point can I expect the weights to change?  Thank you very much for experience values. 

@viva2202, I did the same here using directly the "run_language_modeling.py" script, but with 11k sequences (I continued pretraining using training data only), and then fine-tuned it using BertForSequenceClassification. Got 1.75% increase in accuracy compared to not continuing pretraining.