noamaanMulla-03/transformers-issues huggingface dataset:
[Bart: example] drop columns that are exclusively pad_token_id from input_ids
Author: sshleifer
Pull request: yes

reasoning: These columns slow down computation, but do not change output.
impact: this reduces the runtime to compute EVAL on the CNN examples from 2h to 1:37 before any other changes.

I'm open to putting this as a method on `PretrainedTokenizer` if others find it useful.

@joeddav you might find this useful.


### Code for easy copy paste
```python
def trim_batch(
    input_ids, pad_token_id, attention_mask=None,
):
    """Remove columns that are populated exclusively by pad_token_id"""
    keep_column_mask = input_ids.ne(pad_token_id).any(dim=0)
    if attention_mask is None:
        return input_ids[:, keep_column_mask]
    else:
        return (input_ids[:, keep_column_mask], attention_mask[:, keep_column_mask])
```

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3400?src=pr&el=h1) Report
> Merging [#3400](https://codecov.io/gh/huggingface/transformers/pull/3400?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/f7dcf8fcea4d486544f221032625a97ad7dc5405&el=desc) will **not change** coverage by `%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3400/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3400?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3400   +/-   ##
=======================================
  Coverage   77.55%   77.55%           
=======================================
  Files         100      100           
  Lines       16970    16970           
=======================================
  Hits        13161    13161           
  Misses       3809     3809           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3400?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3400?src=pr&el=footer). Last update [f7dcf8f...9cfd3da](https://codecov.io/gh/huggingface/transformers/pull/3400?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Model Cards: Fix grammar error
Author: mrm8488
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3467?src=pr&el=h1) Report
> Merging [#3467](https://codecov.io/gh/huggingface/transformers/pull/3467?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/63f4d8cad010f1972254007ad56b22fe5ed203fe&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3467/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3467?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3467   +/-   ##
=======================================
  Coverage   77.84%   77.84%           
=======================================
  Files         100      100           
  Lines       17060    17060           
=======================================
+ Hits        13280    13281    +1     
+ Misses       3780     3779    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3467?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3467/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.32% <0.00%> (+0.17%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3467?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3467?src=pr&el=footer). Last update [63f4d8c...1681ac8](https://codecov.io/gh/huggingface/transformers/pull/3467?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
[Bart/Memory] Two separate, smaller decoder attention masks
Author: sshleifer
Pull request: yes

### Background

The bart decoder requires two masks: one to ignore padding tokens, the other (`causal_mask`), to avoid attending to future tokens during training.
Previously, `_prepare_bart_decoder_inputs` combined these two masks into one  float_mask of shape `(bsz, 1, tgt_len, tgt_len)` filled with -inf for tokens that should be ignored. This mask was subsequently added to the attention activations.

Now, we return the two masks separately:
`decoder_padding_mask`: shape `(bs, tgt_len)`, `bool`
`causal_mask`: shape `(tgt_len, tgt_len)`, `float`

### Impact
saves 800 MB for bs=6, tgt_len=1024, with negligible speed impact.

### Notes
- The distinct data types (bool and float) are used to minimize code change. 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3371?src=pr&el=h1) Report
> Merging [#3371](https://codecov.io/gh/huggingface/transformers/pull/3371?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/cf72479bf11bf7fbc499a518896dfd3cafdd0b21&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `93.75%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3371/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3371?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3371      +/-   ##
==========================================
- Coverage   77.55%   77.52%   -0.03%     
==========================================
  Files         100      100              
  Lines       16970    16957      -13     
==========================================
- Hits        13161    13146      -15     
- Misses       3809     3811       +2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3371?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_bart.py](https://codecov.io/gh/huggingface/transformers/pull/3371/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iYXJ0LnB5) | `97.59% <93.75%> (-0.50%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3371?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3371?src=pr&el=footer). Last update [cf72479...8db65c1](https://codecov.io/gh/huggingface/transformers/pull/3371?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
ModuleNotFoundError: No module named 'model_bertabs' AND RuntimeError: CUDA error: device-side assert triggered
Author: TLCFYBJJHYYSND
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):Bert

Language I am using the model on (English, Chinese....):Chinese

The problem arise when using:
* [ ] the official example scripts: 

[1]ModuleNotFoundError: No module named 'model_bertabs' when run convert_bertabs_original_pytorch_checkpoint.py
[2] RuntimeError: CUDA error: device-side assert triggered when run  run_lm_finetuning.py




* [ ] my own modified scripts: (give details)
python convert_bertabs_original_pytorch_checkpoint.py \
    --bertabs_checkpoint_path /home/jhzhou/code/transformers-master/examples/summarization/data \
    --pytorch_dump_folder_path /home/jhzhou/code/transformers-master/examples/summarization/outputs/


export TRAIN_FILE=/home/jhzhou/transformers/examples/path/to/dataset/wiki.train.raw
export TEST_FILE=/home/jhzhou/transformers/examples/path/to/dataset/wiki.test.raw
export DataFile=/home/jhzhou/transformers/examples/path/to/dataset/out


CUDA_VISIBLE_DEVICES=0 python run_lm_finetuning.py \
    --output_dir /home/jhzhou/transformers/examples/path/to/dataset/out \
    --model_type=bert \
    --model_name_or_path bert-base-chinese \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE \
    --mlm \
    --overwrite_output_dir 

    
    


The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name):1.convert_bertabs_original_pytorch_checkpoint.py  
                                                                                2.run_lm_finetuning.py
* [ ] my own task or dataset: (give details):wiki.train.raw

## To Reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:
* Python version:3.6
* PyTorch version:1.3.1
* PyTorch Transformers version (or branch):  2.1.1
* Using GPU ? YES
* Distributed or parallel setup ? One GPU

## Additional context

First my task was fine-tuning with MLM,then i with the problem happened i found the way seems to be the solution  to convert the pytorch_model.bin to stuff that can be used in the program.https://github.com/huggingface/transformers/issues/1615 but it doesnt work to me. 


Comments:
I have the same CUDA error with a finetuning of offical bert downloaded from s3. I'll wait for clarifications too.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
PyTorch 1.2 has released API 'torch.nn.Transformer'，so it's better to modify the source code with the official  python API
Author: daydayfun
Pull request: no
Labels: name: wontfix

## 🚀 Feature

<!-- A clear and concise description of the feature proposal. Please provide a link to the paper and code in case they exist. -->

## Motivation

It's better to Modify modeling_bert.py with official API 'torch.nn.Transformer' of PyTorch 1.2

## Additional context

https://pytorch.org/docs/stable/nn.html?highlight=transformer#torch.nn.Transformer


Comments:
This has been suggested a while back when this was first introduced (we're at 1.4 now). This is possibly impractical to do since it is likely that many people are still on 1.0<=x<1.2. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Add option to choose T5 model size.
Author: LysandreJik
Pull request: yes

I believe the error mentioned in #3469 is due to the example tests loading T5-large in memory. One of the workers load that model which fills up the machine's memory, and other workers crash with an oom error.

This PR gives the option to choose the T5 model size, and changes the tests to only use the small model.

Comments:
Add T5 to docs
Author: patrickvonplaten
Pull request: yes

- [x] Copy past from bart docs
- [x] polish main docs page
- [x] improve docstring in `modeling_t5.py` and `modeling_tf_t5.py`

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3461?src=pr&el=h1) Report
> Merging [#3461](https://codecov.io/gh/huggingface/transformers/pull/3461?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/3ee431dd4c720e67e35a449b453d3dc2b15ccfff&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3461/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3461?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3461   +/-   ##
=======================================
  Coverage   77.79%   77.80%           
=======================================
  Files         100      100           
  Lines       17049    17051    +2     
=======================================
+ Hits        13264    13267    +3     
+ Misses       3785     3784    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3461?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_bart.py](https://codecov.io/gh/huggingface/transformers/pull/3461/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iYXJ0LnB5) | `97.58% <ø> (ø)` | |
| [src/transformers/tokenization\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/3461/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdDUucHk=) | `95.83% <ø> (ø)` | |
| [src/transformers/modeling\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/3461/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190NS5weQ==) | `81.29% <100.00%> (+0.08%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/3461/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl90NS5weQ==) | `94.98% <100.00%> (ø)` | |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3461/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.32% <0.00%> (+0.17%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3461?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3461?src=pr&el=footer). Last update [3ee431d...d57c03b](https://codecov.io/gh/huggingface/transformers/pull/3461?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Docstring cannot be build anymore
Author: patrickvonplaten
Pull request: no

# 🐛 Bug

## Information

Docstring

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. `pip install -e ".[docs]"` at `transformers` root folder
2. `cd docs` 
3. `make html`

## Expected behavior

It should work, but an error message is displayed:

> /home/patrick/hugging_face/transformers/src/transformers/modeling_utils.py:docstring of transformers.PreTrainedModel.from_pretrained:23: WARNING: Unexpected indentation.
> /home/patrick/hugging_face/transformers/src/transformers/modeling_tf_utils.py:docstring of transformers.TFPreTrainedModel.from_pretrained:20: WARNING: Definition list ends without a blank line; unexpected unindent.
> 
> Exception occurred:
>   File "/home/patrick/hugging_face/transformers_venv/lib/python3.6/site-packages/sphinx/util/docfields.py", line 260, in transform
>     assert len(field) == 2
> AssertionError
> The full traceback has been saved in /tmp/sphinx-err-mblqzztk.log, if you want to report the issue to the developers.
> Please also report this if it was a user error, so that a better error message can be provided next time.
> A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
> Makefile:19: recipe for target 'html' failed


## Environment info

- `transformers` version: 2.6.0
- Platform: Linux-5.3.0-42-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0+cpu (False)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?:No
- Using distributed or parallel set-up in script?: No


Comments:
Was fixed with https://github.com/huggingface/transformers/commit/e2c05f06ef58ea77103d2c64492dd8d9a0b21c3f
setup.py succeeds, then can't import transformers
Author: sshleifer
Pull request: no

```bash
pip install -e .
```
works succesfully and makes `transformers.egg-info/`

Then,
```python
import transformers
```
fails with

```
ModuleNotFoundError: No module named 'transformers'
```


env
```
- `transformers` version: 2.6.0
- Platform: Darwin-19.0.0-x86_64-i386-64bit
- Python version: 3.7.5
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

Comments:
Do you use the `zsh` shell? cause in this case I think you have to do `pip install -e "."`

I think this has something to do with the shells. Can you compare `zsh` to pure `bash` shell?

Fails in bash identically.
One clue, this is also happening for `pip install tokenizers` followed by import `tokenizers`, but not for `import numpy`.

And with `pip install -e "."` it works ?

I fixed it by **restarting** my zsh shell. Now I can't reproduce the bug in a new zsh shell. Closing.
Fix circle ci flaky fail of wmt example 
Author: patrickvonplaten
Pull request: yes

Weird bug which might be fixed by forcing bleu scorer. 

Test pass locally, but seem to fail on circle ci.

Comments:
This is from this right? https://circleci.com/gh/huggingface/transformers/26455?utm_campaign=workflow-failed&utm_medium=email&utm_source=notification

Yeah exactly - it should be fixed now. 
I will push a slightly cleaner version in a second :-) 

In general, every example test that creates folder or files should deleted them afterwards to avoid same file naming collisions. Will open a PR about this. @julien-c @sshleifer @LysandreJik
Attibute Error：‘NoneType’ object has no attribute 'seek' and OSError
Author: JacksonChan-X
Pull request: no
Labels: name: wontfix

My pytorch version is 1.4.0+cpu and tensorflow version is 2.0.0-dev20191002.
/torch/serialization.py,line 289,in_check_seekable
‘NoneType’ object has no attribute 'seek'
You can only torch.load from a file that is seekable.Please pre_load the data into a buffer like io.BytesIO and try to load from it instead.
But how should I do to sovle the quetions?
Another question.
OSError:Unable to load weights from pytorch checkpoint file.If you tried to load a PyTorch model from a TF 2.0 checkpoint,please set from_tf=true.

Comments:
python version is 3.7.3

Hi, you would need to provide more information than that for us to help you. What code made you run into this error? 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
PPLM with Tensorflow
Author: cakiki
Pull request: no

## ❓ Questions & Help

Hello,

I am still quite new to the library, so I do apologize if the answer is straightforward. 

The latest release (https://github.com/huggingface/transformers/releases/tag/v2.3.0) mentions the inclusion of PPLM as a new architecture, both as Pytorch and TF. 

I can't however seem to figure out how to import **PPLM** as a **Tensorflow** model. Any help would be greatly appreciated. 

Thank you

Comments:
I think this is a mistake on our part. cc @LysandreJik @w4nderlust 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Yes, the code I / we contributed is only for PyTorch. I think porting it to TensorFlow is feasible, but it's not in there at the moment.

The release notes were fixed then. Thanks!
No such file or directory: '..\\VERSION'
Author: balkon16
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

While trying to install `pytorch-transformers` I get the following error:
```

    ERROR: Command errored out with exit status 1:
     command: 'c:\users\pawel.lonca\appdata\local\programs\python\python35\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\PAWEL~1.LON\\AppData\\Local\\Temp\\pip-install-b5eog20_\\sentencepiece\\setup.py'"'"'; __file__='"'"'C:\\Users\\PAWEL~1.LON\\AppData\\Local\\Temp\\pip-install-b5eog20_\\sentencepiece\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
         cwd: C:\Users\PAWEL~1.LON\AppData\Local\Temp\pip-install-b5eog20_\sentencepiece\
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\PAWEL~1.LON\AppData\Local\Temp\pip-install-b5eog20_\sentencepiece\setup.py", line 29, in <module>
        with codecs.open(os.path.join('..', 'VERSION'), 'r', 'utf-8') as f:
      File "c:\users\pawel.lonca\appdata\local\programs\python\python35\lib\codecs.py", line 895, in open
        file = builtins.open(filename, mode, buffering)
    FileNotFoundError: [Errno 2] No such file or directory: '..\\VERSION'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
```

Google search suggests that the recommended solution is upgrading the `setuptools` but it didn't work in my case.


## Environment

* OS: Windows 10
* Python version: 3.5.2
* PyTorch version: 1.2.0+cpu

Comments:
The same bug occurs when installing.

This looks like an issue with sentencepiece and python 3.5. Do you want to have a look there maybe? https://github.com/google/sentencepiece

> This looks like an issue with sentencepiece and python 3.5. Do you want to have a look there maybe? https://github.com/google/sentencepiece

Python version maybe the issue.
I switched to python 3.6 and successfully installed it.

I'm getting the same issue. Changing to 3.6 or 3.7 did not fix it.

Download the wheel file from https://github.com/google/sentencepiece/releases for your python version and install it with
 pip install sentencepiece-xxx-cpxx-xx.whl

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> Download the wheel file from https://github.com/google/sentencepiece/releases for your python version and install it with
> pip install sentencepiece-xxx-cpxx-xx.whl

this trick works fantastically, many thanks!
Write With Transformer returning a 502 on gpt2/xl model
Author: sergalbutt
Pull request: no

When setting the model size to **gpt2/xl**, WwT gets stuck on loading the autocomplete.
Checking Chrome's console tells me
"Failed to load resource: the server responded with a status of 502 (Bad Gateway)"

Having a quick look through the older tickets, I saw that this has happened before. #2121 

Comments:
We had to turn that model off because it was really expensive/hard to operationalize.

We'll add a warning to that particular webpage. (cc @LysandreJik)