noamaanMulla-03/transformers-issues huggingface dataset:
Download assets directly to the specified cache_dir
Author: n-a-sz
Pull request: no

## 🚀 Feature

```
import torch
from transformers import *
TRANSFORMERS_CACHE='/path/to/my/transformers-cache'
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', cache_dir=TRANSFORMERS_CACHE) 
```

Actual behavior: It downloads the asset into a temp folder and then copies it to the specified cache_dir.

Proposed behavior: Download the asset directly to the specified cache_dir.

Impacted code part: https://github.com/huggingface/transformers/blob/master/transformers/file_utils.py#L295-L322

## Motivation

We have an environment setup where the tmp folders have limited space, because it is not a mounted docker volume. If the `"asset size" > 10GB - "docker image size"` then it won't be able to download the asset. (The 10GB limitation is a docker limitation)



Comments:
I have read the comment part of the reference code:
```
# Download to temporary file, then copy to cache dir once finished.
# Otherwise you get corrupt cache entries if the download gets interrupted.
```
So I would change my proposal:
* Either let it be configurable to skip the tmp folder and download directly to the cache folder -> the user will know what he is doing and will know that the asset could get corrupted
* Or check the file in the cache before usage - e.g. using checksums
* Or write "download has started" and "download has finished" information to the meta data file that can be checked before asset usage.


I would propose to download to the cache_dir with a specific temporary name (like a `.part` suffix) and copy + rename at the end.

Probably best to activate that with an option `use_cache_dir_as_tmp`. To not clutter the cache dir with temporary files in the default settings.

Do you want to submit a PR for that? Would be happy to review it

Yes that is also a good approach. For now, we seem to be okay with this limitation, but I'll do a pr if we face this as an issue or have some free time.

Same problem here. On cluster, /tmp folder is small. Keep getting no space on device.

I fixed this in b67fa1a8d2302d808ecb9d95355181eaf21ee3b6.

Until there's a release with this fix, you can set $TMPDIR to an appropriate location if /tmp is too small.

Cool, thank you @aaugustin !
Do Hugging Face GPT-2 Transformer Models Automatically Does the Absolute Position Embedding for Users?
Author: h56cho
Pull request: no

Hello,

According to Hugging Face ```GPT2DoubleHeadsModel``` documentation (https://huggingface.co/transformers/model_doc/gpt2.html#gpt2doubleheadsmodel)

```
"Indices of input sequence tokens in the vocabulary. 
GPT-2 is a model with absolute position embeddings"
```

So does this mean that, when we implement any Hugging Face GPT-2 Models (```GPT2DoubleHeadsModel```,```GPT2LMHeadsModel```, etc.) via the ```model( )``` statement, the 'absolute position embedding' is _automatically_ done for the user, so that the user actually does not need to specify anything in the ```model( )``` statement to ensure the absolute position embedding?

If the answer is 'yes', then why do we have an option of specifying ```position_ids``` in the ```model( )``` statement?

Thank you,



Comments:
Indeed, as you can see from the source code [here](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_gpt2.py#L414-L417), when no position ids are passed, they are created as absolute position embeddings.

You could have trained a model with a GPT-2 architecture that was using an other type of position embeddings, in which case passing your specific embeddings would be necessary. I'm sure several other use-cases would make sure of specific position embeddings.

Ooohh, ok,

so to clarify, absolute position embedding _**is automatically done**_ by the ```model( )``` statement, but if we want to use our custom position embedding (i.e. other than the absolute position embedding), we can use the ```position_ids``` option inside the ```model( )``` statement......is what I said above correct?

Thank you,

Yes, that is correct!

Thank you :) !
Improve logging message in feature conversion functions
Author: simonepri
Pull request: yes

This PR adds the total number of examples to process to the log message produced during the feature conversion step.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2361?src=pr&el=h1) Report
> Merging [#2361](https://codecov.io/gh/huggingface/transformers/pull/2361?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/f75bf05ce6a05ef316363de129c29f2e00cacd7b?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2361/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2361?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2361   +/-   ##
=======================================
  Coverage   73.23%   73.23%           
=======================================
  Files          87       87           
  Lines       14985    14985           
=======================================
  Hits        10975    10975           
  Misses       4010     4010
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2361?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/data/processors/utils.py](https://codecov.io/gh/huggingface/transformers/pull/2361/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvdXRpbHMucHk=) | `19.6% <0%> (ø)` | :arrow_up: |
| [src/transformers/data/processors/glue.py](https://codecov.io/gh/huggingface/transformers/pull/2361/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvZ2x1ZS5weQ==) | `27.86% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2361?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2361?src=pr&el=footer). Last update [f75bf05...dc69c5c](https://codecov.io/gh/huggingface/transformers/pull/2361?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great, thanks @simonepri
Is there a uncased gpt2?
Author: cloudygoose
Pull request: no

## ❓ Questions & Help

Hi, thanks for everything. Quick question: Is there a pre-trained uncased gpt2, like bert-uncased?

Comments:
Hi, all the available models are listed in the [pretrained models section of the documentation](https://huggingface.co/transformers/pretrained_models.html). For GPT-2, there are four different models (`gpt2`, `gpt2-medium`, `gpt2-large`, `gpt2-xl`), which are all cased.
Parallel data preprocessing for distillation
Author: jianwolf
Pull request: no
Labels: name: wontfix

## 🚀 Feature

Use the `multiprocessing.Pool` function to parallelize the text tokenization and uint16 conversion in `transformers/examples/distillation/scripts/binarized_data.py`.

## Motivation

I tried to preprocess a 2.6 GB txt file using the python script, but the expected time is 2.4 hours. I tried to parallelize it myself and the total time decreased to 10 minutes on my server.

## Additional context

My code is something like this:

```
def process_data(text):
    return tokenizer.encode(f'{bos} {text.strip()} {sep}')

pool = Pool()
rslt = pool.map(process_data, data)
rslt_ = pool.map(np.uint16, rslt)
```

Comments:
What is your suggestion, then? Adding a mp_encode function? Perhaps this is something that should stay at the user's side. 

Hello @jianwolf,
Yes indeed, I've never taken the time to do it (mainly because most of the I do pre-processing are one-shot: I launch it before leaving the office 😴).
If you feel like opening a pull request with your suggestion, I would happy to add it.

@BramVanroy do you see any drawbacks of having parallelized pre-processing by default?

I tried to integrate your few lines and had this error:
```
  File "/usr/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'main.<locals>.process_data'
```

It seems like `process_data` should be outside of the `main`, that shouldn't be too complicated.

(Also, how many parallel processes/cpus do you have on your server for this order of magnitude in reduction?)

Victor


@VictorSanh At first reading I thought the suggestion was to implement a default multiprocessing encoding for tokenizers. That would seem like a large change that needs a lot of testing across multiple platforms (note the different between fork and spawn) as well as a possible reproducibility issue when retrieving results from different threads, and thus different batch orders. Of course these problems could be mitigated but it seemed like a lot of work to suddenly overhaul all tokenizers in this way.

Now that it's clear that it's only for the distillation script, I'm sure there's no big issue here even though I would like to see this implemented in a deterministic way, i.e. order of return values should always be identical. 

Hi! Yeah I will create a pull request for this code! On my machine there are 80 CPU threads available!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Maximum length of out put generated in run_generation.py is of length (1021 ) despite changing position id length and length parameter
Author: anubhakabra
Pull request: no
Labels: name: wontfix

I would like to generate a text of about 3000 words.
However the run_generation.py file limits it to 1024, and produces only 1021 words. I have tried changing the internal parameters for the same but in vain. 

Comments:
What model are you using?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Add support for Albert and XLMRoberta for the Glue example
Author: simonepri
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2403?src=pr&el=h1) Report
> Merging [#2403](https://codecov.io/gh/huggingface/transformers/pull/2403?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/80faf22b4ac194061a08fde09ad8b202118c151e?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2403/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2403?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2403   +/-   ##
=======================================
  Coverage   73.24%   73.24%           
=======================================
  Files          87       87           
  Lines       14989    14989           
=======================================
  Hits        10979    10979           
  Misses       4010     4010
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2403?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2403?src=pr&el=footer). Last update [80faf22...ff6dacf](https://codecov.io/gh/huggingface/transformers/pull/2403?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
update the config.is_decoder=True before initialize the decoder
Author: zlinao
Pull request: yes

Currently the `PreTrainedEncoderDecoder`  class fails to initialize the "cross-attention layer" since it updates  `decoder.config.is_decoder = True` after decoder initialization.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2389?src=pr&el=h1) Report
> Merging [#2389](https://codecov.io/gh/huggingface/transformers/pull/2389?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/9261c7f771fccfa2a2cb78ae544adef2f6eb402b?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2389/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2389?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2389   +/-   ##
=======================================
  Coverage   73.24%   73.24%           
=======================================
  Files          87       87           
  Lines       15001    15001           
=======================================
  Hits        10988    10988           
  Misses       4013     4013
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2389?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2389?src=pr&el=footer). Last update [9261c7f...9261c7f](https://codecov.io/gh/huggingface/transformers/pull/2389?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


But these are not decoders, they're encoders with an additional language modeling head?

> But these are not decoders, they're encoders with an additional language modeling head?

Oh, thanks to point out my mistake, I should actually modify the `modeling_encoder_decoder.py` file. I accidentally closed this pull request and made a new one #2435 .
make test problem
Author: OsbertTay
Pull request: no

## ❓ Questions & Help
Hello all,

I recently installed this library/module and wanted to run a test with: `make test`      
However, things do not go right and I got the following message:

> python -m pytest -n auto --dist=loadfile -s -v ./tests/
> /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: No module named pytest
> make: *** [test] Error 1

I'm using macOS X Catalina, with Python3.7 (3.7.5), pytest is installed. 
(I have no clue why it returns an error on Py2.7 )

Thanks in advance


Comments:
Fix typograpical errors
Author: gentaiscool
Pull request: yes

Fixed few typos.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2438?src=pr&el=h1) Report
> Merging [#2438](https://codecov.io/gh/huggingface/transformers/pull/2438?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/fb2ab869c6894ea05df97a1372ac9e016ec9c662?src=pr&el=desc) will **decrease** coverage by `0.17%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2438/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2438?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2438      +/-   ##
==========================================
- Coverage   73.24%   73.06%   -0.18%     
==========================================
  Files          87       87              
  Lines       15001    15001              
==========================================
- Hits        10988    10961      -27     
- Misses       4013     4040      +27
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2438?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/2438/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25feGxtLnB5) | `83.26% <100%> (ø)` | :arrow_up: |
| [src/transformers/modeling\_tf\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/2438/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9hdXRvLnB5) | `25% <0%> (-7.15%)` | :arrow_down: |
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/2438/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `66.37% <0%> (-2.3%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2438/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `91.53% <0%> (-1.59%)` | :arrow_down: |
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2438/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `87.19% <0%> (-0.65%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2438?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2438?src=pr&el=footer). Last update [fb2ab86...58ca488](https://codecov.io/gh/huggingface/transformers/pull/2438?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great, thanks @gentaiscool !
spelling correction
Author: orena1
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2434?src=pr&el=h1) Report
> Merging [#2434](https://codecov.io/gh/huggingface/transformers/pull/2434?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/176d3b30798fce556613da31c698d31cfdfd02aa?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2434/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2434?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2434   +/-   ##
=======================================
  Coverage   73.24%   73.24%           
=======================================
  Files          87       87           
  Lines       15001    15001           
=======================================
  Hits        10988    10988           
  Misses       4013     4013
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2434?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2434?src=pr&el=footer). Last update [176d3b3...7bce837](https://codecov.io/gh/huggingface/transformers/pull/2434?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great, thanks @orena1 !
Make doc regarding masked indices more clear
Author: r0mainK
Pull request: yes

See this [issue](https://github.com/huggingface/transformers/issues/2418) for details, basically there used to be different ways of specifying masked indices (either -1 or -100), which was fixed by this [commit](https://github.com/huggingface/transformers/commit/418589244d263087f1d48655f621a65f2a5fcba6).

However the doc remains unclear, this PR fixes this.

I had an issue originally because I was using a version which did not incorporate the uniformisation yet.

Comments:
Fantastic, thanks @r0mainK!
Update Squ
Author: Santosh-Gupta
Pull request: yes

Update https://github.com/huggingface/transformers/blob/master/examples/run_squad.py

Implemented solution from 

https://github.com/huggingface/transformers/issues/1837#issuecomment-554206349

Comments:
Could you give at least some explanation in the title and the body of your post explaining what fixed or changed? Now it's very vague. 

> Could you give at least some explanation in the title and the body of your post explaining what fixed or changed? Now it's very vague.

Sorry about the vagueness, I'll do this for any new pull requests. As for this one, it's obsolete now since this script has been refactored.
How is the interactive GPT-2 implemented?
Author: ChanningPing
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
I came across this online demo from HuggingFace for GPT-2 writing: https://transformer.huggingface.co/doc/gpt2-large. The demo is really amazing, both accurate and fast. My major observation is that the service actually uses user's earlier writing examples in later prediction, almost instantly. I'm very curious how it is implemented? It seems to me that it is not fine-tuned in real-time, then is there some other mechanism behind it? Any ideas are appreciated.
Context examples I typed in:

> Set the a c to low level = the room is very cold
> turn down the volume = the music is too loud

Then when i try:
> turn on the lights = 

It gives me
> the room is too bright

Also, I tried fine-tuning the entire model with much more than 2 examples (around 30), however the result for "turn on the lights = "  after fine-tuning is a lot worse than the demo:

> about 0.016 (lit a second and a 100 pixels)

Is it that the demo only fine-tune, e.g. the very last layer of the model?


Comments:
Hi, the models are not fine-tuned on the fly. Language models like GPT-2 are very context-aware and are strong at generating words related to the inputs they were given. 

We are not training the models in that demo, we are only using them for inference.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Typo in XLM moses pipeline.
Author: alvations
Pull request: no

The replacement on for the unicode punct replacement has a mistake at https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_xlm.py#L477

Comments:
Indeed, thanks @alvations !

BTW, https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_xlm.py#L621 could also be simplified to the normalizer object from https://github.com/alvations/sacremoses/blob/master/sacremoses/normalize.py#L129


```python
    def moses_punct_norm(self, text, lang):
        if lang not in self.cache_moses_punct_normalizer:
            punct_normalizer = sm.MosesPunctNormalizer(lang=lang, 
                pre_replace_unicode_punct=True, post_remove_control_chars=True)
            self.cache_moses_punct_normalizer[lang] = punct_normalizer
        else:
            punct_normalizer = self.cache_moses_punct_normalizer[lang]
        return punct_normalizer.normalize(text)
```

Then the pipeline at https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_xlm.py#L635 would just be 


```python
    def moses_pipeline(self, text, lang):
        text = self.moses_punct_norm(text, lang)
        return text
```