noamaanMulla-03/transformers-issues huggingface dataset:
[s2s] Script to save wmt data to disk
Author: sshleifer
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6403?src=pr&el=h1) Report
> Merging [#6403](https://codecov.io/gh/huggingface/transformers/pull/6403?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/00bb0b25ed66a4878f2e0ffdd1ca65b7684db57e&el=desc) will **decrease** coverage by `0.63%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6403/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6403?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6403      +/-   ##
==========================================
- Coverage   80.24%   79.60%   -0.64%     
==========================================
  Files         149      149              
  Lines       27680    27680              
==========================================
- Hits        22211    22035     -176     
- Misses       5469     5645     +176     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6403?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/6403/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25feGxtLnB5) | `16.26% <0.00%> (-66.67%)` | :arrow_down: |
| [src/transformers/tokenization\_marian.py](https://codecov.io/gh/huggingface/transformers/pull/6403/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fbWFyaWFuLnB5) | `66.95% <0.00%> (-25.22%)` | :arrow_down: |
| [src/transformers/tokenization\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/6403/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYXV0by5weQ==) | `95.55% <0.00%> (-2.23%)` | :arrow_down: |
| [src/transformers/configuration\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6403/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3V0aWxzLnB5) | `95.89% <0.00%> (-0.69%)` | :arrow_down: |
| [src/transformers/tokenization\_dpr.py](https://codecov.io/gh/huggingface/transformers/pull/6403/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fZHByLnB5) | `57.65% <0.00%> (+4.50%)` | :arrow_up: |
| [src/transformers/tokenization\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/6403/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fY3RybC5weQ==) | `96.11% <0.00%> (+17.47%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6403?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6403?src=pr&el=footer). Last update [00bb0b2...19f2c61](https://codecov.io/gh/huggingface/transformers/pull/6403?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Fix tokenizer saving and loading error
Author: yobekiko
Pull request: yes

This PR is to address #6025

Comments:
> This is great! Do you mind adding a test under `tests/test_tokenization_common.py` so that we may ensure this doesn't fail in the future?

Hi I added into an existing test. Please check if it's appropriate.

Hello! I added a more robust test and fixed the style issue. Thanks a lot for your contribution, merging as soon as all the tests show green!
Failed to save tokenizer with AddedToken in additional_special_tokens
Author: yobekiko
Pull request: no

# 🐛 Bug

I tried to add new special tokens to some tokenizer. I wanted them to be `AddedToken` in order to handle specific whitespace striping. But I got the following error when saving the tokenizer.

## To reproduce

Steps to reproduce the behavior:

```python
>>> from transformers import BertTokenizer
>>> from tokenizers import AddedToken
>>> tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
>>> new_token = AddedToken('new_token', lstrip=True)
>>> tokenizer.add_special_tokens({'additional_special_tokens': [new_token]})
1
>>> tokenizer.save_pretrained('.')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/junyuan/anaconda3/envs/deep/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 1371, in save_pretrained
    f.write(json.dumps(write_dict, ensure_ascii=False))
  File "/home/junyuan/anaconda3/envs/deep/lib/python3.7/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/home/junyuan/anaconda3/envs/deep/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/home/junyuan/anaconda3/envs/deep/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/home/junyuan/anaconda3/envs/deep/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type AddedToken is not JSON serializable
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->


Comments:
Fixed by #6026
Debug flag to `run_language_modeling` triggers error
Author: dmlap
Pull request: no

## Environment info

- `transformers` version: 3.0.2
- Platform: Linux-5.4.0-42-generic-x86_64-with-glibc2.29
- Python version: 3.8.2
- PyTorch version (GPU?): 1.6.0+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes, run_language_modeling.py
- Using distributed or parallel set-up in script?: no

### Who can help
 
I'd guess @sgugger or @julien-c

## Information
I'm using [run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py) and turned on debug output to double check things were working as I expected. Unfortunately, [trainer.py](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L628) keys off that debug option to invoke `xm.master_print(...)` and `xm`/`torch_xla.core.xla_model` isn't loaded because I'm not working on a TPU-based system.

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

All steps should be run on a system with a GPU but no TPU. Steps to reproduce the behavior:

1. Run `run_language_modeling.py` with the debug flag:
```sh
python run_language_modeling.py \
  --output_dir ./output \
  --model_type gpt2 \
  --model_name_or_path gpt2 \
  --do_train \
  --train_data_file ./train.txt \
  --learning_rate 1e-4 \
  --num_train_epochs 1 \
  --save_total_limit 2 \
  --save_steps 200 \
  --do_eval \
  --eval_data_file ./eval.txt \
  --debug
```
2. Allow the script to run.

The command will error towards the end with this traceback:
```sh
Epoch:   0%|                                                                                                                                                   | 0/1 [54:06<?, ?it/s]
Traceback (most recent call last):
  File "run_language_modeling.py", line 281, in <module>
    main()
  File "run_language_modeling.py", line 245, in main
    trainer.train(model_path=model_path)
  File "/home/user/project/env/lib/python3.8/site-packages/transformers/trainer.py", line 570, in train
    xm.master_print(met.metrics_report())
NameError: name 'xm' is not defined
```
## Expected behavior

The script exits without error.


Comments:
The documentation of `debug` clearly points out this parameter is only used for TPU-training:
```
debug (:obj:`bool`, `optional`, defaults to :obj:`False`):
            When training on TPU, whether to print debug metrics or not.
```
It's called simply debug (and not `tpu_debug`) for harmonization with `TFTrainer`.

Ah, my bad for not fully reading the documentation. Would you be open to a PR with a guard or more-specific error message for this scenario?

Sure!

closed by #6390
How to add Dense layer on top of TFBertForSequenceClassification model?
Author: sainimohit23
Pull request: no

I am having a really hard time adding the dense layers on the top of this model. I have tried to add the layers of `TFBertForSequenceClassification` in a sequential model with some dense layers like this:

```
bert_model = TFBertForSequenceClassification.from_pretrained("bert-base-cased", config=config)

model = keras.models.Sequential()
model.add(bert_model.layers[0])
model.add(keras.layers.Dense(10, activation='relu'))
model.add(keras.layers.Dense(1, activation='sigmoid'))
```
But when I fit the model using: 
``` 
model.fit(
    [padded, attention_mask],
    [np.array(df[1][:2000])],
    epochs=100,
)
```

I am getting this error:
```AttributeError: 'list' object has no attribute 'shape'```


I have also tried to use the layers of the `TFBertForSequenceClassification` in `keras.models.Model() class`. But, again there is no way to get the input layer. For example using `bert_model.layers[0].input_shape` gives the following error:
```
   1571     """
   1572     if not self._inbound_nodes:
-> 1573       raise AttributeError('The layer has never been called '
   1574                            'and thus has no defined input shape.')
   1575     all_input_shapes = set(

AttributeError: The layer has never been called and thus has no defined input shape.
```

What is the right way to add layers on the top of this model. 

Comments:
Found the solution on [1936](https://github.com/huggingface/transformers/issues/1936). Closing.

Can you write down the solution here? @sainimohit23
Warn if debug requested without TPU fixes (#6308)
Author: dmlap
Pull request: yes

Check whether a PyTorch compatible TPU is available before attempting to print TPU metrics after training has completed. This way, users who apply `--debug` without reading the documentation aren't suprised by a stacktrace.

Comments:
The CircleCI failures look like a pre-existing line length violation in `trainer.py` and a checksum mismatch downloading transformers itself for `run_tests_torch`. I don't believe either are related to my change – I was able to run the examples test suite locally and everything passed. I'd be happy to fix the line length issue, if that helps. I think it would take me awhile to figure out what's going on with the checksum mismatch.

Hi! There was an issue with the style in your PR, I pushed the fix. Will merge once all the tests are green!

Thanks for your contribution :)

No problem! Thanks for the style-fixup, @LysandreJik.
Create README.md
Author: abedkhooli
Pull request: yes
Labels: name: model card

For GPT-2 Arabic Poetry - https://huggingface.co/akhooli/gpt2-small-arabic-poetry

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6397?src=pr&el=h1) Report
> Merging [#6397](https://codecov.io/gh/huggingface/transformers/pull/6397?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/7ea9b2db3732904014b9121fb8a5c896ae00d4cf&el=desc) will **increase** coverage by `0.96%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6397/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6397?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6397      +/-   ##
==========================================
+ Coverage   77.31%   78.27%   +0.96%     
==========================================
  Files         146      146              
  Lines       26597    26597              
==========================================
+ Hits        20563    20820     +257     
+ Misses       6034     5777     -257     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6397?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/6397/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `64.40% <0.00%> (-34.39%)` | :arrow_down: |
| [src/transformers/tokenization\_marian.py](https://codecov.io/gh/huggingface/transformers/pull/6397/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fbWFyaWFuLnB5) | `68.14% <0.00%> (-25.67%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_mobilebert.py](https://codecov.io/gh/huggingface/transformers/pull/6397/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9tb2JpbGViZXJ0LnB5) | `96.75% <0.00%> (+73.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6397?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6397?src=pr&el=footer). Last update [7ea9b2d...141f941](https://codecov.io/gh/huggingface/transformers/pull/6397?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


[Thanks for sharing](https://huggingface.co/akhooli/gpt2-small-arabic-poetry)

If you'd like, you could submit some simple inputs for Arabic to https://github.com/huggingface/widgets-server/blob/master/DefaultWidget.ts – let me know if you need any help
switch Hindi-BERT to S3 README
Author: mapmeld
Pull request: yes
Labels: name: model card

The Markdown parser currently cuts off the CoLab URL (the last char is an underscore) on https://huggingface.co/monsoon-nlp/hindi-bert

There are some other necessary updates, and I'd rather update this model card by pushing to S3 in the future

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6396?src=pr&el=h1) Report
> Merging [#6396](https://codecov.io/gh/huggingface/transformers/pull/6396?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/3e0fe3cf5c1059c04535de8f04f4efed7251adbe&el=desc) will **increase** coverage by `0.11%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6396/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6396?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6396      +/-   ##
==========================================
+ Coverage   79.40%   79.51%   +0.11%     
==========================================
  Files         148      148              
  Lines       27200    27200              
==========================================
+ Hits        21598    21628      +30     
+ Misses       5602     5572      -30     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6396?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6396/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.71% <0.00%> (+7.51%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6396?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6396?src=pr&el=footer). Last update [06bc347...106d0f3](https://codecov.io/gh/huggingface/transformers/pull/6396?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@mapmeld we have a better version control/preview system coming in the future. In the meantime, merging this
Confuse by  "All learning rates are 0"
Author: BeHappyForMe
Pull request: no

transformers/examples/seq2seq/finetune.py  219row :
        if max(scheduler.get_last_lr()) > 0:
            warnings.warn("All learning rates are 0")

why, is there something i left?


Comments:
I've got the same problem here. Have you solved it?

Same here. Any solution to solve? Or is this just a warning message that can be ignored?

Pinging @sshleifer 

Ignore it. I will try to get it out of the code. I think the lr starts at 0 and climbs up for `warmup_steps` iterations.
how to fine tune t5 model for summarization task using tensorflow2?
Author: banunitte
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
please see: https://discuss.huggingface.co/t/how-to-train-t5-with-tensorflow/641.
My finetuned gpt2 model is taking wayy too long to generate samples, like 5-8 minutes
Author: krishnerkar
Pull request: no

I fine tuned the gpt2 model using transformers, i trained it on a lyrics dataset, and after successful training, when i do model.generate(args), it takes like a hell lot of time to genrate results
What Should i do?


Comments:
Hi @Krish-Nerkar , [this](https://discuss.huggingface.co/t/speeding-up-gpt2-generation/470) might help

@patil-suraj Thanks Alot! will check those methods out
Where do the Masked Language Model perform mask on the input data
Author: SusanSun8
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->

I am trying to pre-train a bert from scratch with my own word set using only the Masked Language Model. 
I have a trouble finding where exactly the code masks 15% of the words and replaces it with 80% mask, 10% random, and 10% original. 

I noticed that the input "labels" kind of refers to the places where words are masked. Does it mean that when I preprocess the data, I need to masked it myself and then indicates the position of masks in the "labels" input? If so, is "labels" the only input that would be affect? Is there any other input variables, such as the "masked_lm_positions" and "masked_lm_ids" in google-bert that I need to take care of?


<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
Hey @SusanSun8, we are trying to move "non-bugs" questions to the forum: https://discuss.huggingface.co/ . Could you maybe post your question there again? 

Here is the code that is responsible for Masked Language Modeling: https://github.com/huggingface/transformers/blob/f6cb0f806efecb64df40c946dacaad0adad33d53/src/transformers/data/data_collator.py#L107.
Faster bert basic tokenizer
Author: GuillemGSubies
Pull request: yes
Labels: name: wontfix

In this PR I tried 2 things:

* First I change some comparisons in the form of `a == sth or a == otherthing` for `a in {sth, otherthing}`. It is faster and more readable.

* I noticed that some methods could actually be functions because they did not use anything from the class.

I am a bit new to this, so I will accept any feedback.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4721?src=pr&el=h1) Report
> Merging [#4721](https://codecov.io/gh/huggingface/transformers/pull/4721?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/47a551d17b6ed2eaf03301f049006d559fca5cf3&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4721/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4721?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4721   +/-   ##
=======================================
  Coverage   77.14%   77.14%           
=======================================
  Files         128      128           
  Lines       21073    21072    -1     
=======================================
  Hits        16256    16256           
+ Misses       4817     4816    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4721?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/4721/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmVydC5weQ==) | `94.97% <100.00%> (-0.03%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4721/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.83% <0.00%> (+0.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4721?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4721?src=pr&el=footer). Last update [47a551d...6f7ff85](https://codecov.io/gh/huggingface/transformers/pull/4721?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Rather than making them regular functions I'd make them into static methods since they're still very much related to the class. You just don't need `self`.

> Rather than making them regular functions I'd make them into static methods since they're still very much related to the class. You just don't need `self`.

Done

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.