noamaanMulla-03/transformers-issues huggingface dataset:
Clean-up code
Author: sgugger
Pull request: yes

Looks like #4747 introduced some bad formatting, fixing so CI is happy again.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4790?src=pr&el=h1) Report
> Merging [#4790](https://codecov.io/gh/huggingface/transformers/pull/4790?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/fa661ce749b0d14ae1999d1b097866248624a842&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4790/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4790?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4790      +/-   ##
==========================================
+ Coverage   76.28%   76.29%   +0.01%     
==========================================
  Files         128      128              
  Lines       21500    21500              
==========================================
+ Hits        16401    16404       +3     
+ Misses       5099     5096       -3     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4790?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/configuration\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/4790/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3hsbmV0LnB5) | `94.00% <100.00%> (ø)` | |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4790/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.51% <0.00%> (+0.31%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4790/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `74.27% <0.00%> (+0.41%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4790?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4790?src=pr&el=footer). Last update [fa661ce...a3f8a97](https://codecov.io/gh/huggingface/transformers/pull/4790?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


👍
[cleanup/marian] pipelines test and new kwarg
Author: sshleifer
Pull request: yes

avoids DeprecationWarning (because `max_len` kwarg is being deprecated)

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4812?src=pr&el=h1) Report
> Merging [#4812](https://codecov.io/gh/huggingface/transformers/pull/4812?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/875288b344d2181b789746e27e7b5bc62df8cae1&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4812/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4812?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4812      +/-   ##
==========================================
- Coverage   76.18%   76.15%   -0.03%     
==========================================
  Files         128      128              
  Lines       21497    21497              
==========================================
- Hits        16377    16371       -6     
- Misses       5120     5126       +6     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4812?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_marian.py](https://codecov.io/gh/huggingface/transformers/pull/4812/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fbWFyaWFuLnB5) | `92.79% <ø> (ø)` | |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/4812/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `80.00% <0.00%> (-1.36%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4812/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.85% <0.00%> (-0.42%)` | :arrow_down: |
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4812/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `89.68% <0.00%> (-0.12%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4812?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4812?src=pr&el=footer). Last update [875288b...b7b9470](https://codecov.io/gh/huggingface/transformers/pull/4812?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Add model and doc badges
Author: sgugger
Pull request: yes

Add badges at each model for:
- the page with all community models
- the documentation of the model

Remove the manual doc links as a result. 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4811?src=pr&el=h1) Report
> Merging [#4811](https://codecov.io/gh/huggingface/transformers/pull/4811?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/875288b344d2181b789746e27e7b5bc62df8cae1&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4811/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4811?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4811      +/-   ##
==========================================
- Coverage   76.18%   76.15%   -0.03%     
==========================================
  Files         128      128              
  Lines       21497    21497              
==========================================
- Hits        16377    16372       -5     
- Misses       5120     5125       +5     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4811?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4811/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `86.41% <0.00%> (-0.79%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4811/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `74.27% <0.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4811?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4811?src=pr&el=footer). Last update [875288b...8ccd73a](https://codecov.io/gh/huggingface/transformers/pull/4811?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great
Can I train question-answering on TPU using Huggingface
Author: thak123
Pull request: no

# ❓ Questions & Help
I am trying to run run_squad from question answering on TPU but I get the following error.

## Details
export SQUAD_DIR=/path/to/SQUAD
python examples/xla_spawn.py --num_cores 8 \
 run_squad.py \
  --model_type bert \
  --model_name_or_path bert-base-uncased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/

**run_squad.py: error: unrecognized arguments: --tpu_num_cores 1**




Comments:
as you can see in https://github.com/huggingface/transformers/tree/master/examples#the-big-table-of-tasks `question-answering` is not implemented using Trainer, yet – so doesn't have TPU support. 

This is on our todo-list but might take a few weeks. Feel free to open a PR though – or use TFTrainer is TF is an option

I think I can live with TFTrainer ... Thanks for the reply

how do i use TFTrainer with TPU

Hi! The [running on TPU](https://github.com/huggingface/transformers/tree/master/examples#running-on-tpus) section of the examples covers that.

Basically, if on a correctly setup TPU, the TF trainer will automatically launch on TPU.
Is albert lm finetuning with SOP in Pytorch supported?
Author: faddyai
Pull request: no

# ❓ Questions & Help


Hello, 

I am trying to use transfer learning on the albert language model. Before i train it on Squad. 

Does run_language_modeling.py support albert models and SOP ?

Thank you 

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments:
Hi, `run_language_modeling.py` does support the Albert model. It only does MLM though, no SOP.

I see, am i correct in assuming that pretraining/finetuning the albert model with run_language_modeling.py which only supports MLM task,   would result in lower performance, vs training with a script from another library (such as the original Albert repo from google) which supports SOP? 

Thank you 

It might result in lower performance, indeed. Adding the SOP task shouldn't be too hard, as the layer used for SOP are implemented. You can check this issue for more information https://github.com/huggingface/transformers/issues/2671.
Correcting path to pplm examples
Author: Mr-Ruben
Pull request: no

# 🐛 Bug

On https://github.com/huggingface/transformers/tree/master/examples/text-generation/pplm#setup

It says

```
git clone https://github.com/huggingface/transformers && cd transformers
pip install .
pip install nltk torchtext # additional requirements.
cd examples/pplm


```

and as you can guess from the url, the correct path is

```
git clone https://github.com/huggingface/transformers && cd transformers
pip install .
pip install nltk torchtext # additional requirements.
cd examples/text-generation/pplm
```


cd examples/**text-generation**/pplm


Comments:
Thanks for the report – care to open a PR?

Done
https://github.com/huggingface/transformers/pull/4778

I hope it is done well.
Better None gradients handling in TF Trainer
Author: jplu
Pull request: yes

Update the TF Trainer to better handle None gradients in order to have something generic and not anymore task dependent.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4469?src=pr&el=h1) Report
> Merging [#4469](https://codecov.io/gh/huggingface/transformers/pull/4469?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/5856999a9f2926923f037ecd8d27b8058bcf9dae&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `0.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4469/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4469?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4469      +/-   ##
==========================================
+ Coverage   77.98%   78.00%   +0.01%     
==========================================
  Files         123      123              
  Lines       20436    20431       -5     
==========================================
  Hits        15938    15938              
+ Misses       4498     4493       -5     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4469?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/trainer\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/4469/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyX3RmLnB5) | `17.92% <0.00%> (+0.41%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4469/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.44% <0.00%> (-0.42%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4469/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.66% <0.00%> (ø)` | |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4469/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.53% <0.00%> (+0.11%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4469?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4469?src=pr&el=footer). Last update [5856999...095c8d2](https://codecov.io/gh/huggingface/transformers/pull/4469?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Tests for more examples
Author: srush
Pull request: no
Labels: name: wontfix


# Add tests for more of the examples/  

We need to have more testing code for examples. This is particularly true with NER which recently had a tokenizer issue. 

(Self-assigning)

Comments:
I agree. Tried for an hour last night to add coverage for `run_bart_sum.py` and got stuck on two things.

1) A `circleci` job that installs `pytorch_lightning` (and potentially other dependencies)
2) the ability to import `examples/transformer_base.py`

### Suggested Approach:
- add the aforementioned circleci job
- a flag like `@require_lightning` to decorate some tests
- some code changes to get the imports working sanely. 
- Checklist/Instructions for how new examples/ contributors can add test coverage.

Happy to help with this!
CC: @LysandreJik , @julien-c, @thomwolf @patrickvonplaten 

Cool. Most of these seem easy. Sharing code between examples is a bit harder. I am not sure if we should have and examples package, or symlink that shared file somehow. Currently we add it to the path before running the code. 

With the Pytorch-Lightning examples, I've been running into an [issue](https://github.com/huggingface/transformers/pull/3437) loading trained models with `--do_predict` (when not also using `--do_train`), so it would be helpful to add some model-loading test as well :) 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
pretrained EsperBERTo
Author: vr25
Pull request: no
Labels: name: wontfix

name: Ex: LM (Pretraining)

name: Usage

Hi,

I am trying to replicate the pretraining process mentioned in this blog post: https://huggingface.co/blog/how-to-train

I have time-restricted access to the GPU I'm currently working on and so I wanted to know how to save the checkpoint and resume the pretraining process from the latest checkpoint.

Thanks!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
model name '../data/bert_models/chinese_finetuned_lm/pytorch_model.bin' was not found in model name list . Creating an empty model card.
Author: zbbwss
Pull request: no
Labels: name: wontfix

nlp = pipeline('fill-mask',
                   # model=args.bert_model_path,
                   # config=args.bert_config_path,
                   # tokenizer=args.bert_model_dir
                    model = '../data/bert_models/chinese_finetuned_lm/pytorch_model.bin',
                    config = '../data/bert_models/chinese_finetuned_lm/config.json',
                    tokenizer = "../data/bert_models/chinese_finetuned_lm/"
                   )
i use fine-tune model (chinese bert) ,when i end fine-tune  i can not load model  !

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
GPT2TokenizerFast does not preserve special tokens' ids after a save and load.
Author: s-jse
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

name: Should Fix

name: Fast Tokenizers

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT2 Fast Tokenizer

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

This problem only happens with `GPT2TokenizerFast` and not `GPT2Tokenizer`

## To reproduce

Steps to reproduce the behavior:

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```python
from transformers import GPT2TokenizerFast

tokenizer = GPT2TokenizerFast.from_pretrained('gpt2')
print('special tokens: ', tokenizer.additional_special_tokens, tokenizer.additional_special_tokens_ids)
tokenizer.add_special_tokens({'additional_special_tokens': ['<special_token>'], 'pad_token': '<pad>'})
print('special tokens: ', tokenizer.additional_special_tokens, tokenizer.additional_special_tokens_ids)
print(tokenizer.pad_token, tokenizer.convert_tokens_to_ids(tokenizer.pad_token))
tokenizer.save_pretrained('./save_dir/')

tokenizer = GPT2TokenizerFast.from_pretrained('./save_dir/')
print('special tokens: ', tokenizer.additional_special_tokens, tokenizer.additional_special_tokens_ids)
print(tokenizer.pad_token, tokenizer.convert_tokens_to_ids(tokenizer.pad_token))
```

It outputs

```
special tokens:  [] []
special tokens:  ['<special_token>'] [50257]
<pad> 50258
special tokens:  ['<special_token>'] [50258]
<pad> 50257
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

```
special tokens:  [] []
special tokens:  ['<special_token>'] [50257]
<pad> 50258
special tokens:  ['<special_token>'] [50257]
<pad> 50258
```

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.5.1
- Platform: Linux-5.4.19-100.fc30.x86_64-x86_64-with-fedora-30-Thirty
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.1.0 (True)
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
BART.generate: possible to reduce time/memory?
Author: astariul
Pull request: no
Labels: name: wontfix

# 🐛 Performance issues

I did a quick benchmark between HuggingFace's implementation of **BART** and FairSeq's implementation.

You can find the benchmark code [here](https://gist.github.com/Colanim/cc418d19e5e107f462bac306f53ba994).

---

Here is my results, on a single GPU GTX 1080 (12 GiB of memory) :

| FP16 - Batch size 16 | s/batch | s/sample |
|----------------------|---------|----------|
| FairSeq              | 8.8676  | 0.5664   |
| HuggingFace          | 12.3358 | 0.7879   |

| FP16 - Batch size 32 | s/batch | s/sample |
|----------------------|---------|----------|
| FairSeq              | 17.1247 | 0.5469   |
| HuggingFace          | OOM     | OOM      |

| FP16 - Batch size 1 | s/sample |
|---------------------|----------|
| FairSeq             | 1.6743   |
| HuggingFace         | 1.8856   |

| FP32 - Batch size 1 | s/sample |
|---------------------|----------|
| FairSeq             | 1.7865   |
| HuggingFace         | 2.0670   |

---

**FairSeq is consistently faster than HuggingFace on all my experiments.**

---

This sparks a few questions :

* Do you have similar results on your side ? Did I mess my benchmark ?
* Why HuggingFace's implementation is significantly slower ?
* Why HuggingFace's implementation takes more space in memory (illustrated by `OOM` with batch size of 32) ?
* Is the release of the `Summarization Pipeline` going to improve this ?

@sshleifer 

 

Comments:
1) Identical to my benchmark for speed. Hadn't tested memory but I'm not surprised that their implementation is less.

For both memory and speed, they have a lot of clever tricks that we haven't implemented yet.

4) Summarization Pipeline will not help, but I will take a longer look at this tomorrow and see if we can improve.


On master, the gap has closed considerably!
<16GB GPU RAM for fp16, bs=32, and timings much closer:
![image](https://user-images.githubusercontent.com/6045025/77931576-55d7bd00-727a-11ea-8904-e57070635087.png)

My numbers are a bit lower than yours because I am on an NVIDIA RTX GPU.


I tested again and I have similar results ! Thanks for the fix.

I now have exact same GPU memory utilization.

---

About the (now) small difference of inference time between implementations, do you know from where it comes from ?

Haven't investigated. So far, I just investigated memory and the speed improvements were a happy side effect.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.