noamaanMulla-03/transformers-issues huggingface dataset:
Memory leak
Author: cahya-wirawan
Pull request: no

## Environment info

- `transformers` version: 3.1.0
- Platform: Linux-3.10.0-1127.el7.x86_64-x86_64-with-debian-buster-sid
- Python version: 3.7.0
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: distributed


### Who can help
@LysandreJik, @sgugger, @patrickvonplaten

## Information

Model I am using (Bert, GPT2):

The problem arises when using:
* [ X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X ] my own task or dataset: (give details below)

## To reproduce
When I pretrain or fine tune a model (in my case BERT and GPT2) using torch.distributed.launch, the CPU memory usage will grow up to the memory limit (>500GB) until the first process is killed due to this issue.  If I train bert-base, it takes around 30 epochs until the first process is killed, but when I train gpt-large, it just need 3 epochs until it is killed. Following is the command line I run to train/fine tune the bert-base (similar with gpt2). The script run_language_modeling.py is a copy of transformers/examples/language-modeling/run_language_modeling.py (vers. 3.1.0)

python -m torch.distributed.launch --nproc_per_node=8 \
  ../run_language_modeling.py \
  --output_dir $model_target \
  --model_name_or_path $model_source \
  --config_name $model_source \
  --tokenizer_name $model_source \
  --train_data_file $target_train \
  --eval_data_file $target_test \
  --save_total_limit 5 \
  --block_size 128 \
  --overwrite_output_dir \
  --fp16 \
  --num_train_epochs 50 \
  --do_train --do_eval \
  --per_device_train_batch_size 32 \
  --per_device_eval_batch_size 4 \
  --mlm

## Expected behavior
I would expect that the distributed training run until it is done without any memory issue. 
Thanks for checking it.


Comments:
This looks to be a duplicate of #7169

But I think my problem is run out of the cpu memory, not the GPU memory

Ah my bad, I misread one letter ;-)
To fully understand your error, what's the dataset (particularly its size) you are training on?

The size of dataset (indonesian Wikipedia) is around 522MB.

just additional info, running the script in single process doesn't have this issue. In my case, the memory usage is stable, and stay at 16GB after few epochs. 
But I want to run it in multiple GPU, it is just too slow with only one :-)

#6999 

I tried the fix from #6999 manually (which is just a one liner `return loss` to `return loss.detach()`, and it seems to solve my memory leak issue. The fix is actually available since version 3.2.0, but when I used the version 3.2.0 with multi gpu, the process just stuck after the 500 steps, maybe there is deadlock among processes? Maybe I will write another ticket regarding this issue.
Example Format of Data for token classification
Author: Michael95-m
Pull request: no

Hi!! I'd like to train the token classification model but I don't know what is the right format of data for token classification training. Thank you.

Comments:
Hi @Michael95-m ,

the dataset format for e.g. the normal NER task is relatively simple: one token - label pair per line and an empty line specified a new sentence.

So here's a good example from Spanish CoNLL dataset for NER:

```bash
Melbourne B-LOC
( O
Australia B-LOC
) O
, O
25 O
may O
( O
EFE B-ORG
) O
. O

- O

El O
Abogado B-PER
General I-PER
del I-PER
Estado I-PER
, O
Daryl B-PER
Williams I-PER
, O
subrayó O
hoy O
la O
necesidad O
de O
tomar O
medidas O
para O
proteger O
al O
sistema O
judicial O
australiano O
frente O
a O
una O
página O
de O
internet O
que O
imposibilita O
el O
cumplimiento O
de O
los O
principios O
básicos O
de O
la O
Ley B-MISC
. O

```

Each line consists of token/word and its corresponding label, delimited by a space. An empty line denotes a new sentence.

Technically, the parsing of your input file is done here:

https://github.com/huggingface/transformers/blob/f5518e56318a79056ba3c80cbece29d9fe98558c/examples/token-classification/tasks.py#L18-L44

I hope this helps :)



Thanks for your kind answer !!!
[s2s] add create student script
Author: patil-suraj
Pull request: yes

This PR adds `create_student.py` script to create the student models for distillbart/pegasus/marian and t5.

```bash
python make_student.py \
  facebook/bart-large-cnn \
  --e 12 \
  --d 6 \
  --save_path student-bart-cnn-12-6 \
```

Comments:
What do you think @patil-suraj ? 
+ Added t5 support, renamed create_bart_student -> make_student
+ made the code return which layers it copied so that distillation.py can use it.
+ Deleted a bunch of broken stuff!
+ added `save_randomly_initialized_model.py` script.


LGTM, much cleaner now. 
Just left couple of nits, hope you don't mind me doing this :)

Love the nits!
Bug fix: NER pipeline shouldn't group separate entities of same type
Author: cceyda
Pull request: yes
Labels: name: wontfix

## Effects
nlp=pipeline('ner', ... , grouped_entities=True)

## Fixes 
Separate entities of same type shouldn't be grouped together even if they are same type 
( B-type1 B-type1 ) != ( B-type1 I-type1 )

## Example 
"something something Istanbul Los Angeles something something"

Current output: [ (O O) (B-type1 B-type1 I-type1) (O O) ]
Fixed output: [ (O O) (B-type1) (B-type1 I-type1) (O O) ]




Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Adding extra inputs when fine-tuning BERT
Author: nadavborenstein
Pull request: no
Labels: name: wontfix

I am trying to fine-tune BERT for a sequence classification task where in addition to the sequences, I have extra features such as the writer age, tags, etc. I want to use those extra features, and I was thinking about concatenating them to the input of the final linear layer.
Is there a way of doing such a thing? If not, what is the best way for integrating extra features in the fine-tuning process?

 

Comments:
You could try stacking a linear layer over-top of BERT that takes as input the BERT sequence representation + your features. You would have to fine-tune through all of BERT.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> I am trying to fine-tune BERT for a sequence classification task where in addition to the sequences, I have extra features such as the writer age, tags, etc. I want to use those extra features, and I was thinking about concatenating them to the input of the final linear layer.
> Is there a way of doing such a thing? If not, what is the best way for integrating extra features in the fine-tuning process?

Did you find a way to add extra features then fine-tuning BERT?
Flos fix
Author: marrrcin
Pull request: yes

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #7146 

This basically unwraps the model that is used during training and can be either plain `Module` or `DataParallel`/`DistributedDataParallel`.

Comments:
Please merge at will, as this fix is blocking us (https://github.com/huggingface/transformers/issues/7146#issuecomment-698852274).
How to get parameters from a Query.
Author: thiagomoeng
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

Hi maybe someone can help me with a problem, I need to get parameters from a query.
I already use question answering pipeline to identify a report and now I want to get some parameters to use as filter, example:
 
Query: How many electric cars sold in 2019?

Output: (Item: eletric cars / Year:2019)


Comments:
It seems you're trying to identify entities in your query? You could use an NER model to do so, but I'm not sure the entities the NER models on the hub are trained on would work in your use-case. You can still check them out, and check the [NER script](https://github.com/huggingface/transformers/tree/master/examples/token-classification) if you want to train a model with your data.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Reorganize documentation navbar
Author: sgugger
Pull request: yes

With the library containing so many models now, the documentation navigation bar was started to get unreadable, especially if someone was looking for a specific model. To make this cleaner I:
- removed the PACKAGE REFENCE caption (not happy about this but there is no way to have a caption with empty content :-( )
- added three captions to separate the documentation between main classes, models and internals
- sorted alphabetically the sections

Also, make the background color of the section headers a bit darker to make the distinction with the rest of the toc clearer.

The result can be checked [here](https://92916-155220641-gh.circle-artifacts.com/0/docs/_build/html/index.html).

Comments:
Cool, thanks @sgugger
should mBART-large-en-ro have decoder_start_token_id by default?
Author: sshleifer
Pull request: no
Labels: name: Help wanted

name: translation

Hypothesis: since the argument `prepend_bos` is set to "False" in fairseq/examples/README.md, mbart-large-en-ro does not need `decoder_start_token_id`.

TODO:
- create branch that deletes `decoder_start_token_id`. Setting it to None in the config might not be enough.
- verify that decoder_start_token_id is in fact not being used by setting a breakpoint in `generate`.
- run_eval.py on wmt-en-ro/test and see if BLEU is >= 26.46, the score with decoder_start_token_id=250020.



Comments:
Hi @sshleifer, I'd like to contribute and help out here if still needed. My thinking is to remove ```decoder_start_token_id```  from run_eval.py and generation_utils.py and change the following code:

https://github.com/huggingface/transformers/blob/6028ed92bd9e5471e6f8a1d23cfd95a3a63018fb/src/transformers/generation_utils.py#L403-L409

to:

            input_ids = torch.full(
                (effective_batch_size * num_beams, 1),
                250020,
                dtype=torch.long,
                device=next(self.parameters()).device,
            )

I dont think that change will do anything since decoder_start_token_id = 250020.

What I would do is change the 250020 to a bos_token_id (0, I think) or a pad_token_id (1) and see what the BLEU score is. 


Ah yes that makes sense. I tried those two and the eos_token_id and got the following results:

ID | BLEU Score
-- | --
eos_token_id (2) | 28.22
decoder_start_token_id (250020) | 28.06
pad_token_id (1) | 26.79
bos_token_id (0) | 26.01



Super interesting, thanks for running that. It seems like I should change decoder_start_token_id in the mbart-large-en-ro config to 2. Do you have opinions on mbart-large-cc25?

No problem! Yes I think configuring decoder_start_token_id to 2 is a good idea. Unfortunately, I'm getting the same issues you're getting with mbart-large-cc25 (output's in English not Romanian and missing the first word when I use bos_token_id or 250020 and gibberish with eos/pad_token_id) and don't understand why that's the case. I'll investigate and post any useful findings.




I think I fixed this another way in #6526 
on master
```
python run_eval.py facebook/mbart-large-en-ro $ENRO_DIR/test.source eos_baseline_enro_test_generations.txt \
--reference_path $ENRO_DIR/test.target \
--score_path baseline_test_bleu_eos.json --bs 32 --task translation --fp16
```
=> {'bleu': 26.81}


```
python run_eval.py facebook/mbart-large-en-ro $ENRO_DIR/test.source \
eos_baseline_enro_test_generations.txt --reference_path $ENRO_DIR/test.target \
--score_path baseline_test_bleu_eos.json --bs 32 --task translation --fp16  \
--decoder_start_token_id 2
```
{'bleu': 11.57} (and takes 40 mins!)

in the original fairseq I get 26.83.

Gunna close this since the score is now basically the same as fairseq. Thanks for your help!
Uploading/Sharing large models to HuggingFace
Author: MXueguang
Pull request: no

Hi,

I am trying to upload a `t5-3b` based model to HuggingFace. The folder to upload has 11G.
When I am uploading, it will gives `'Connection aborted.', BrokenPipeError(32, 'Broken pipe')`.
Does it because the model is too large and there is a limitation? How could I deal with that?
Thank you for your help!

Comments:
There is no limit to the file sizes on the model hub, however, for uploads that large and if your connection is even slightly unstable, it can indeed fail.

If you have another host (S3 bucket or whatever) you can upload the file to, I can handle `cp`ing it to your namespace on huggingface.co

Actually It will abort at the very beginning of uploading process for the large file every time. All my other smaller models could be uploaded smoothly. so I feel it might not be my network issue.
My `pytorch_model.bin` is about 11G. I tried to use `truncate` to truncate the file size and noticed that I will keep aborting until I `truncate` the file to 5G

Btw, we have a public google cloud storage host. Does it work for you if i am still not able to upload the model?

I can indeed reproduce. For now, can you upload to a GCS or S3 bucket, post the url here, and I'll cp the file? 

Will take a note to investigate/fix this in the future.

```
gs://ron-random/castorini/monot5-3b-med-msmarco/
gs://ron-random/castorini/monot5-3b-msmarco/
```
Could you help us cp these two models to our organization `castorini`

Thank you very much for your help!

Here you go: https://huggingface.co/castorini

Will close this for now but we are tracking the "large file upload" issue internally
CUDA out of memory (ALBERT) - run_squad.py ignores --per_gpu_train_batch_size
Author: chbensch
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.3.0
- Platform: Colab Pro (P100) / Anaconda (Windows / 2080 Ti)
- Python version: 3.6
- PyTorch version (GPU?): 1.6.0 / CUDA 10.2
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


## Information

Model I am using (Bert, XLNet ...): ALBERT

The problem arises when using:
* [x] the official example scripts: run_squad.py
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: SQUaD 2.0
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Fine-tune ALBERT-xlarge or xxlarge and  set --per_gpu_train_batch_size 8 or 10
2. try to finetune

```
!python transformers\examples\question-answering\run_squad.py \
  --model_type albert \
  --model_name_or_path albert-large-v2 \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file train-v2.0.json \
  --predict_file dev-v2.0.json \
  --per_gpu_train_batch_size 8 \
  --learning_rate 3e-5 \
  --num_train_epochs 1.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /content/model_output \
  --save_steps 1000 \
  --threads 4 \
  --version_2_with_negative \
  --overwrite_output_dir
```
## Expected behavior

The model should be training, but despite the 8gb limit there is an out of memory error:
```
RuntimeError: CUDA out of memory. Tried to allocate 36.00 MiB (GPU 0; 15.90 GiB total capacity; 15.01 GiB already allocated; 7.88 MiB free; 15.03 GiB reserved in total by PyTorch)
```

Comments:
Do you get the same error when using a batch size of 1?

Thanks, with a batch size of 1 it works! 

I never thought that even the "A light BERT" models are so big. :)

The `large` "light BERT" model is quite large indeed ;) The `base` model is smaller if you want to use bigger batch sizes.
deleted
Author: av-maslov
Pull request: yes

Deleted

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/7450?src=pr&el=h1) Report
> Merging [#7450](https://codecov.io/gh/huggingface/transformers/pull/7450?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/1fc4de69ed024e18b88cb6f040021630599de2f7?el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `0.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/7450/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/7450?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #7450      +/-   ##
==========================================
- Coverage   79.35%   79.35%   -0.01%     
==========================================
  Files         181      181              
  Lines       35800    35801       +1     
==========================================
  Hits        28410    28410              
- Misses       7390     7391       +1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/7450?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_layoutlm.py](https://codecov.io/gh/huggingface/transformers/pull/7450/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19sYXlvdXRsbS5weQ==) | `25.00% <0.00%> (-0.07%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/7450/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.21% <0.00%> (-0.26%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/7450/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `83.37% <0.00%> (+0.24%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/7450?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/7450?src=pr&el=footer). Last update [1fc4de6...c5759b1](https://codecov.io/gh/huggingface/transformers/pull/7450?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Add a code of conduct
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds a code of conduct to the project, inspired by the [Contributor Covenant](https://www.contributor-covenant.org/). To make it clearly visible it also adds:
- a badge that displays under Transformers on the main README.
- a link in the contributing guide.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/7433?src=pr&el=h1) Report
> Merging [#7433](https://codecov.io/gh/huggingface/transformers/pull/7433?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a1a8ffa5126ced93c12dfb677cbe3a069f48dcf3?el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/7433/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/7433?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #7433      +/-   ##
==========================================
- Coverage   76.85%   76.84%   -0.02%     
==========================================
  Files         181      181              
  Lines       35800    35800              
==========================================
- Hits        27513    27509       -4     
- Misses       8287     8291       +4     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/7433?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/7433/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdDUucHk=) | `66.34% <0.00%> (-28.85%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/7433/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `83.37% <0.00%> (+0.24%)` | :arrow_up: |
| [src/transformers/tokenization\_mbart.py](https://codecov.io/gh/huggingface/transformers/pull/7433/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fbWJhcnQucHk=) | `96.82% <0.00%> (+39.68%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/7433?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/7433?src=pr&el=footer). Last update [a1a8ffa...f8d87a2](https://codecov.io/gh/huggingface/transformers/pull/7433?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


(before merging, let's check out the thread we had internally on this)
remove codecov PR comments
Author: sshleifer
Pull request: yes



#### Problem
+ @stas00 has tried very hard to get codecov working to no avail in https://github.com/huggingface/transformers/issues/6317
+ Files that are not affected by a PR show changes in coverage.
+ Codecoverage information is rarely/never useful
+ lots of distracting spam emails
+ lots of vertical space that could otherwise be used for reading discussion history.

#### Proposed solution:
The idea of codecov -- to warn people before they introduce untested code -- is good, but the current implementation is worse than nothing, and after significant effort (mostly from @stas00) I think it is time to give up. If we see another tool we like, or manage to reconfigure this one to work well, that's great, but I think that should happen without broken codecov on master.



Comments:
The page is nice but the data seems wrong -- 
https://codecov.io/gh/huggingface/transformers/src/master/src/transformers/modeling_layoutlm.py
says that `__init__` is not covered, but I checked and it is.. in `test_modeling_layoutlm.py`.

I can also just disable PR comments if that works better for you.



> Is there another tool we could leverage to give us more reliable information on our code coverage?

The tool is not the problem, the problem lies in our test suite being not idempotent. codecov only compares the old coverage to new coverage. It can't give correct coverage if the data it works with is invalid. Garbage in garbage out.

If you want an approximate coverage, you can just add  `cov: pytest --cov` in Makefile. It has a bunch of formats if you want the report in a particular format. It should be within 98% of correctness based on the current state of the test suite. 

I understand the issue. Could we simply disable the PR comments for now @sshleifer, as that's the only pain point?

Done. Will merge once checks pass!