noamaanMulla-03/transformers-issues huggingface dataset:
Cache directory
Author: abhigenie92
Pull request: no
Labels: name: wontfix

Where is the cache directory?
```
BertTokenizer.from_pretrained('bert-base-uncased')
```

I want to download manually and place it there.
```
OSError: Couldn't reach server at '{}' to download vocabulary files.```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Exception in device=TPU:1: 'ascii' codec can't decode byte 0xc2 in position 37: ordinal not in range(128)
Author: marton-avrios
Pull request: no

I get the error message when trying to run the seq2seq example on 8 TPU cores on XSUM and bart-large.

Run this from `examples/seq2seq`:
```
export PYTHONPATH="../":"${PYTHONPATH}"

python finetune.py \
    --learning_rate=3e-5 \
    --gpus 0 \
    --n_tpu_cores 8 \
    --do_train \
    --do_predict \
    --n_val 1000 \
    --val_check_interval 1.0 \
    --sortish_sampler \
    --data_dir ${PWD}/xsum \
    --train_batch_size=1 \
    --eval_batch_size=1 \
    --output_dir=xsum_results \
    --num_train_epochs 1 \
    --model_name_or_path facebook/bart-large
```

BTW there is also a bug in `lightning_base.py`. The `gpus` argument is given twice if `n_tpu_cores` is specified (second time through `**train_params`) so I replaced Trainer creation lines with these:
```
    n_gpus = train_params.pop("gpus", args.gpus)

    trainer = pl.Trainer(
        logger=logger,
        accumulate_grad_batches=args.gradient_accumulation_steps,
        gpus=n_gpus,
        max_epochs=args.num_train_epochs,
        early_stop_callback=early_stopping_callback,
        gradient_clip_val=args.max_grad_norm,
        checkpoint_callback=checkpoint_callback,
        callbacks=[logging_callback] + extra_callbacks,
        fast_dev_run=args.fast_dev_run,
        val_check_interval=args.val_check_interval,
        weights_summary=None,
        resume_from_checkpoint=args.resume_from_checkpoint,
        **train_params,
    )
```

Comments:
...it also happens on CPU

I solved it. If you happen to use a Google Cloud VM with the `Debian GNU/Linux 9 Stretch + PyTorch/XLA` image then locale is not set correctly. Add the following to `/etc/default/locale`:
```
LC_ALL=en_US.UTF-8
```

...thank you, it works!
DataCollatorForLanguageModeling - Shift labels for left-to-right LM?
Author: shtoshni
Pull request: no

Hi,

Shouldn't the labels be shifted 1 step right in the data collation step at -https://github.com/huggingface/transformers/blob/09a2f40684f77e62d0fd8485fe9d2d610390453f/src/transformers/data/data_collator.py#L88


Comments:
Actually never mind. At least GPT2 does the shifting internally.
TracerWarning on modeling_gpt2.py:147 when using Torchscript
Author: Damiox
Pull request: no
Labels: name: wontfix

The following script:
```
    model = GPT2LMHeadModel.from_pretrained('gpt2-medium')
    model.eval()
    example = torch.randn(100, 50).abs().mul(1000).type(torch.int64)
    traced_model = torch.jit.trace(model, example)
```
I

It's returning the following Warning:
```
python3.7/site-packages/transformers/modeling_gpt2.py:147: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  mask = self.bias[:, :, ns - nd : ns, :ns]
```

Is this something to worry about? I see there was a similar issue with sqrt in  https://github.com/huggingface/transformers/issues/3954 

Additionally, if I wanted to use Torchscript with past, then I should still trace the model in such way so that the past is being recorded in the model. Is that correct?

- `transformers` version: 2.9.0
- Platform: macos
- Python version: 3.7
- PyTorch version: 1.5.0 cpu
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[Request] NER Scripts on CoNLL 2003 dataset 
Author: gyuwankim
Pull request: no
Labels: name: wontfix

name: Ex: Named Entity Recognition

name: Documentation

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation
Current NER scripts (data preparation, train, and evaluation) is for the GermEval 2014 dataset.
It is very nice. However, according to the original BERT paper, experiments are done on the CoNLL-2003 dataset. It would be much better if the NER script could reproduce the results on this dataset.

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution
Looking forward to the feedback. Thanks in advance!

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Comments:
Hi @gyuwankim the problem with English CoNLL dataset is, that it is not freely available (because it requires access to Reuters corpora in order to create it from scratch with the provided scripts).

You can find the dataset somehow on GitHub, but legally this is very dubious (and you have no guarantee that it is the original dataset).

That's the reason why I did use the GermEval 2014 dataset :)

But what you think about adding a detailed description of training a NER model on the WNUT17 dataset?

Data can be accessed here: https://github.com/juand-r/entity-recognition-datasets/tree/master/data/WNUT17

I could add a short example with some resuls (for other Transformer-based models as well). So we have both German and an English example of how to use the NER script :)

/cc @julien-c  

Additionally, the BERT evaluation strategy cannot be simply reproduced, see this [thread](https://github.com/allenai/allennlp/pull/2067#issuecomment-443961816) on the problem.



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Ability to specify directory of tensorboard logs in BART finetuning example
Author: isabelcachola
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

Using [this script](https://github.com/huggingface/transformers/blob/839bfaedb21e42edee093b9e21e2c2f1ea7514f0/examples/summarization/bart/finetune.py) to finetune BART, it'd be useful if we could specify the directory to store the tensorboard logs. Right now, it looks like the logs are saved to `lightning_logs/version_X`. 

Ideally, this is the command I would run:
```{bash}
python finetune.py \
--data_dir=./cnn-dailymail/cnn_dm \
--model_name_or_path=bart-large \
--learning_rate=3e-5 \
--train_batch_size=4 \
--eval_batch_size=4 \
--output_dir=$OUTPUT_DIR \
--do_train  \
--logging_dir logs $@
```

## Motivation

Being able to specify the directory of your tensorboard logs makes it easier to compare multiple experiments.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
RobertaForSequenceClassification for BERT
Author: pashok3d
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

Could you please explain, what are the consequences of using RobertaForSequenceClassification head with BERT pre-trained model? Will it be the same effect as using BertForSequenceClassification?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Slow training time on BERT pretraining on multiple gpu compare to single gpu
Author: ntubertchen
Pull request: no

# ❓ Questions & Help

Hello, I'm pretraining BERT on two RTX 2080Ti with 15 batch size on each gpu. They update every 40 steps.

The training time to 28000 steps of 2 2080Ti is around 9 days and it only takes 1 2080Ti 10 days to achieve the same training time. I was wondering whether is training time is accurate or I can do something to improve it. 

First problem I can think of is although I expand to batch size to the most one GPU memory can hold. The gpu usage is only around 50% all the time. I'm not sure how to improve the overall gpu usage, is it possible that 50% usage is normal? Please give me some advice.

Sincerely

Comments:
I solve this problem with distributed training.

The DataParallel is really slow with a lot of defects, just don't use it.



Hi @ntubertchen, how did you make distributed training to work with the `run_language_modeling.py` script? Or did you do everything from scratch?

run_language_modeling.py has written the distributed part, just use the command.
fine-tuning BERT, next sentence prediction loss is not decreasing
Author: sszzsupersupersupersuper
Pull request: no
Labels: name: wontfix

I run simple_lm_finetuning and monitor the loss change of next_sentence_loss and masked_lm_loss. The loss of masked token can converge but the next_sentence_loss is not decreasing.

For now I tried: tuning learning rate, change to optimizers from pytorch.optim, I checked the input_ids and the input looks good... I also ignore the loss from masked tokens and specifically train next sentence prediction but it didn't work.

 Did anyone face the same problem? Or where should I check to make it work? Thanks.





Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


same not working for me. NSP loss is not converging even though MLM loss is converging.
torch num_samples=0 error on XLMnet @ run_language_modeling.py
Author: ysig
Pull request: no
Labels: name: wontfix

# 🐛 Bug
While finetuning a language model on an XLNET:

```
python transformers/examples/run_language_modeling.py \
    --output_dir=lms/xlnet_custom \
    --model_type=xlnet \
    --model_name_or_path=xlnet-base-cased \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --per_gpu_train_batch_size 1 \
```

at the step after the features are produced, I get the error:

```
Traceback (most recent call last):
  File "transformers/examples/run_language_modeling.py", line 284, in <module>
    main()
  File "transformers/examples/run_language_modeling.py", line 254, in main
    trainer.train(model_path=model_path)
  File "/home/ysig/miniconda3/envs/hug/lib/python3.7/site-packages/transformers/trainer.py", line 218, in train
    train_dataloader = self.get_train_dataloader()
  File "/home/ysig/miniconda3/envs/hug/lib/python3.7/site-packages/transformers/trainer.py", line 160, in get_train_dataloader
    RandomSampler(self.train_dataset) if self.args.local_rank == -1 else DistributedSampler(self.train_dataset)
  File "/home/ysig/miniconda3/envs/hug/lib/python3.7/site-packages/torch/utils/data/sampler.py", line 94, in __init__
    "value, but got num_samples={}".format(self.num_samples))
ValueError: num_samples should be a positive integer value, but got num_samples=0

```

My text is a raw text of the size of MBs.

I am running this on a CUDA 10.2 in a python 3.7.6 conda environment with a torch version of 1.5.0.

Is there something I am possibly doing wrong?


Comments:
Hi! Are you using a multi-GPU setup ?

No I am running it on a single GPU.

Several other threads discuss this issue: 
 - Either your datasets are too small:  https://github.com/kaushaltrivedi/fast-bert/issues/181#issuecomment-596462172 
 - Or you haven't set `--block_size`:  https://github.com/huggingface/transformers/issues/2380#issuecomment-572762207 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
fix typo in training_args_tf.py
Author: adelevie
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5894?src=pr&el=h1) Report
> Merging [#5894](https://codecov.io/gh/huggingface/transformers/pull/5894?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/09a2f40684f77e62d0fd8485fe9d2d610390453f&el=desc) will **decrease** coverage by `1.19%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5894/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5894?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5894      +/-   ##
==========================================
- Coverage   78.49%   77.30%   -1.20%     
==========================================
  Files         146      146              
  Lines       26210    26210              
==========================================
- Hits        20573    20261     -312     
- Misses       5637     5949     +312     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5894?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/training\_args\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/5894/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmluZ19hcmdzX3RmLnB5) | `47.45% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_mobilebert.py](https://codecov.io/gh/huggingface/transformers/pull/5894/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9tb2JpbGViZXJ0LnB5) | `23.38% <0.00%> (-73.39%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5894/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.71% <0.00%> (+0.50%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/5894/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9yb2JlcnRhLnB5) | `93.36% <0.00%> (+49.37%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5894?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5894?src=pr&el=footer). Last update [09a2f40...b9754e4](https://codecov.io/gh/huggingface/transformers/pull/5894?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
fix typo in training_args.py
Author: adelevie
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5893?src=pr&el=h1) Report
> Merging [#5893](https://codecov.io/gh/huggingface/transformers/pull/5893?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/09a2f40684f77e62d0fd8485fe9d2d610390453f&el=desc) will **increase** coverage by `0.17%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5893/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5893?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5893      +/-   ##
==========================================
+ Coverage   78.49%   78.66%   +0.17%     
==========================================
  Files         146      146              
  Lines       26210    26210              
==========================================
+ Hits        20573    20619      +46     
+ Misses       5637     5591      -46     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5893?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/training\_args.py](https://codecov.io/gh/huggingface/transformers/pull/5893/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmluZ19hcmdzLnB5) | `77.55% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/5893/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `64.90% <0.00%> (-33.90%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_electra.py](https://codecov.io/gh/huggingface/transformers/pull/5893/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9lbGVjdHJhLnB5) | `95.53% <0.00%> (+69.51%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5893?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5893?src=pr&el=footer). Last update [09a2f40...762ff74](https://codecov.io/gh/huggingface/transformers/pull/5893?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Create README.md
Author: mrm8488
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5813?src=pr&el=h1) Report
> Merging [#5813](https://codecov.io/gh/huggingface/transformers/pull/5813?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/89a78be51f1c2afd263da66ec76d3297432f0c2a&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5813/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5813?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #5813   +/-   ##
=======================================
  Coverage   78.19%   78.19%           
=======================================
  Files         146      146           
  Lines       26047    26047           
=======================================
  Hits        20367    20367           
  Misses       5680     5680           
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5813?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5813/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `85.96% <0.00%> (-0.26%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5813/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `81.49% <0.00%> (+0.29%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5813?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5813?src=pr&el=footer). Last update [89a78be...1965bb0](https://codecov.io/gh/huggingface/transformers/pull/5813?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Delay decay schedule until the end of warmup
Author: amodaresi
Pull request: yes

The decay schedule should start at the end of the warmup steps. Without this simple edit, learning rate will drop at the end of the warmup steps, like the figure shown below:
<img width="378" alt="image" src="https://user-images.githubusercontent.com/15783079/84426617-c5820500-ac38-11ea-9579-2c0d5f374e33.png">


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4940?src=pr&el=h1) Report
> Merging [#4940](https://codecov.io/gh/huggingface/transformers/pull/4940?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/6293eb04dfef704e87a6e0b358848ffc41587b4f&el=desc) will **increase** coverage by `0.40%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4940/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4940?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4940      +/-   ##
==========================================
+ Coverage   77.20%   77.60%   +0.40%     
==========================================
  Files         128      128              
  Lines       21746    21746              
==========================================
+ Hits        16789    16877      +88     
+ Misses       4957     4869      -88     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4940?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/optimization\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/4940/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9vcHRpbWl6YXRpb25fdGYucHk=) | `57.27% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4940/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.42% <0.00%> (+0.15%)` | :arrow_up: |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/4940/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `78.65% <0.00%> (+2.29%)` | :arrow_up: |
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/4940/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `56.68% <0.00%> (+23.24%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4940?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4940?src=pr&el=footer). Last update [6293eb0...96ae67a](https://codecov.io/gh/huggingface/transformers/pull/4940?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@jplu what do you think of this?

Fix #5098