noamaanMulla-03/transformers-issues huggingface dataset:
Fix GPT2 docstring from #1906
Author: bilal2vec
Pull request: yes

Fixes #1906

Changes the GPT2 Tokenizer's docstrings to correctly explain the reason for `add_prefix_space` parameter

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1911?src=pr&el=h1) Report
> Merging [#1911](https://codecov.io/gh/huggingface/transformers/pull/1911?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/26db31e0c09a8b5e1ca7a61c454b159eab9d86be?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1911/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1911?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1911   +/-   ##
=======================================
  Coverage   84.04%   84.04%           
=======================================
  Files          97       97           
  Lines       14333    14333           
=======================================
  Hits        12046    12046           
  Misses       2287     2287
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1911?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/tokenization\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/1911/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl9ncHQyLnB5) | `96.72% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1911?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1911?src=pr&el=footer). Last update [26db31e...65c5080](https://codecov.io/gh/huggingface/transformers/pull/1911?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great, thanks!
Fix issue: #1962, input shape problem
Author: billpku
Pull request: yes

Hi,

To Fix #1962 
The input's shape seem to cause error in 2.2.0 version tf_albert_model

Hope that it can help.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1966?src=pr&el=h1) Report
> Merging [#1966](https://codecov.io/gh/huggingface/transformers/pull/1966?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/cc7968227e08858df4a5c618c739e1a3ca050196?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1966/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1966?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1966      +/-   ##
==========================================
- Coverage   84.26%   84.24%   -0.02%     
==========================================
  Files         104      104              
  Lines       15431    15431              
==========================================
- Hits        13003    13000       -3     
- Misses       2428     2431       +3
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1966?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_tf\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/1966/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RmX2FsYmVydC5weQ==) | `85.49% <100%> (ø)` | :arrow_up: |
| [transformers/tests/modeling\_tf\_common\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1966/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX3RmX2NvbW1vbl90ZXN0LnB5) | `95.63% <0%> (-1.46%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1966?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1966?src=pr&el=footer). Last update [cc79682...a1aec9c](https://codecov.io/gh/huggingface/transformers/pull/1966?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great, thank you for this @billpku !
suggest to track repo w/ https rather than ssh
Author: rlouf
Pull request: yes

cf #1943 we tell users to track the repository via https rather than ssh (as it requires us to enable ssh).

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1952?src=pr&el=h1) Report
> Merging [#1952](https://codecov.io/gh/huggingface/transformers/pull/1952?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/8e5d84fcc1a645d3c13b8a2f64fa995637440dad?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1952/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1952?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master   #1952   +/-   ##
======================================
  Coverage      84%     84%           
======================================
  Files          97      97           
  Lines       14340   14340           
======================================
  Hits        12047   12047           
  Misses       2293    2293
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1952?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1952?src=pr&el=footer). Last update [8e5d84f...c6edc47](https://codecov.io/gh/huggingface/transformers/pull/1952?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Fix encode_plus
Author: thomwolf
Pull request: yes

Add options to control more precisely the output of `encode_plus`.

All the outputs that can't be ingested by a model as deactivated by default.
`token_type_ids` can be ingested by most models and if thus activated by default but can be turned off.

Fix #1532 among others

Comments:
Looks good to me, this philosophy (only model-inputs and optional non-model-inputs) seems way better to me than the previous one.
PyTorch library dependency
Author: makcedward
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): XLNet

Language I am using the model on (English, Chinese....): English

```
pytorch_transformers\modeling_xlnet.py in forward(self, input_ids, token_type_ids, input_mask, attention_mask, mems, perm_mask, target_mapping, head_mask)
    925             # `1` indicates not in the same segment [qlen x klen x bsz]
    926             seg_mat = (token_type_ids[:, None] != cat_ids[None, :]).long()
--> 927             seg_mat = F.one_hot(seg_mat, num_classes=2).to(dtype_float)
    928         else:
    929             seg_mat = None

AttributeError: module 'torch.nn.functional' has no attribute 'one_hot'
```

## To Reproduce

Steps to reproduce the behavior:

1. Install pytprch 1.0.1 version
2. Use XLNet model to do the prediction.


'torch.nn.functional''s 'one_hot' function introduced from [1.1.0](https://pytorch.org/docs/1.1.0/nn.html#one-hot) while [requirements.txt](https://github.com/huggingface/pytorch-transformers/blob/master/requirements.txt#L2 ) requests 1.0.0+

Comments:
I have the same issue with `pytorch 1.0.1`. When pytorch version is upgraded (to `1.2.0` for instance), this error is removed, however I get an import error:
```
ImportError: /opt/conda/lib/python3.7/site-packages/fused_layer_norm_cuda.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN2at19UndefinedTensorImpl10_singletonE
```


> I have the same issue with `pytorch 1.0.1`. When pytorch version is upgraded (to `1.2.0` for instance), this error is removed, however I get an import error:
> 
> ```
> ImportError: /opt/conda/lib/python3.7/site-packages/fused_layer_norm_cuda.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN2at19UndefinedTensorImpl10_singletonE
> ```

My issue is fixed after upgrading pytorch to 1.2.0. Which model/ function do you call ?

For some reason, I had to reinstall apex after upgrading pytorch. 

> For some reason, I had to reinstall apex after upgrading pytorch.

So have you fixed your ImportError？ I met this error when initializing BertAdam. I will try to migrate my code from pytorch-pretrained-bert to pytorch-transformers.

Yes, upgraded pytorch, then reinstalled apex.

> Yes, upgraded pytorch, then reinstalled apex.

Thank you. I will try.

Just to confirm, I had the same issue (ImportError). As @tayciryahmed said, re-installing apex would do the trick as it did for me :) 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
git@github.com: Permission denied (publickey) when fetching
Author: iedmrc
Pull request: no

When you wanted to sync forked repository with base, as described [here](https://github.com/huggingface/transformers/blame/aa92a184d2b92faadec975139ad55e2ae749362c/CONTRIBUTING.md#L140)

You get:
```
➜  transformers (master) ✗ git fetch upstream
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
```
[This](https://stackoverflow.com/a/34081606) solution introduced on stackoverflow fixes the problem. [This](https://stackoverflow.com/questions/13509293/git-fatal-could-not-read-from-remote-repository#comment85002398_34081606) one also says: 

> If the repo owner has not set up ssh keys then you will likely have this issue. The fix as indicated is to use https instead, or have the repo owner set up ssh

Could you please fix this (by setting up ssh?), in order to make contributing easy?
Thanks!

Comments:
Yeah you can clone using https, it's usually easier (github actually recommends it for simple workflows)

cc @rlouf 

Nice! Then, we might update  [this line](https://github.com/huggingface/transformers/blame/5d3b8daad2cc6287d30f03f8a96d0a1f7bc8d0dc/CONTRIBUTING.md#L109) since Github encourages https instead of ssh.

Thanks for pointing this out, I just made the change.
Added Mish Activation Function
Author: digantamisra98
Pull request: yes

Mish is a new activation function proposed here - https://arxiv.org/abs/1908.08681
It has seen some recent success and has been adopted in SpaCy, Thic, TensorFlow Addons and FastAI-dev. 
All benchmarks recorded till now (including against ReLU, Swish and GELU) is present in the repository - https://github.com/digantamisra98/Mish
Might be a good addition to experiment with especially in the Bert Model.

Comments:
Ok, why not.
Load output file from fine-tuned bert language model
Author: avinashsai
Pull request: no

Hi,
I have fine-tuned bert cased language model using **run_lm_finetuning.py**. 'output' is the output file directory and '**bert-base-cased.txt**' is another file created by the model.

1. Does the .txt file mentioned has the output of fine-tuned model??
2. If so, how should I open the file?? I am getting 'UTF-8' encoding issue with the file.

Thank you.


Comments:
Hmm this script should not output any `.txt` files except `eval_results.txt`. What is inside this output directory except for this file?

I didn't provide any evaluation file.
However logging info is as follows:

- Creating features from dataset file at 
- Saving features into cached file **bert-base-cased_cached_lm_32.txt**
Fix CTRL past
Author: LysandreJik
Pull request: yes

Fixes the issue with the re-usable past in CTRL.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1740?src=pr&el=h1) Report
> Merging [#1740](https://codecov.io/gh/huggingface/transformers/pull/1740?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/5afca00b4732f57329824e1538897e791e02e894?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1740/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1740?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1740      +/-   ##
==========================================
+ Coverage   84.06%   84.06%   +<.01%     
==========================================
  Files         105      105              
  Lines       15536    15537       +1     
==========================================
+ Hits        13060    13061       +1     
  Misses       2476     2476
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1740?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/1740/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2N0cmwucHk=) | `96.47% <100%> (+0.01%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1740?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1740?src=pr&el=footer). Last update [5afca00...8da47b0](https://codecov.io/gh/huggingface/transformers/pull/1740?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Ok great, nice catch and fix!
[multiple-choice] Simplify and use tokenizer.encode_plus
Author: julien-c
Pull request: yes

Our base tokenizer `PreTrainedTokenizer` now has the ability to encode a sentence pair up to a `max_length`, adding special tokens for each model and returning a mask of `token_type_ids`.

In this PR we upgrade `run_multiple_choice` by adopting this factorized tokenizer API.

To ensure the results are strictly the same as before, we implement a new `TruncatingStrategy` (ideally this could be an enum).

@erenup as you spent a lot of time on this script, would you be able to review this PR?

Result of eval with parameters from [examples/readme](https://github.com/huggingface/transformers/blob/julien_multiple-choice/examples/README.md#multiple-choice):
```
eval_acc = 0.8352494251724483
eval_loss = 0.42866929549320487
```

Comments:
Great addition. I feel like using enums would be especially helpful for the truncating strategy, indeed.

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1385?src=pr&el=h1) Report
> Merging [#1385](https://codecov.io/gh/huggingface/transformers/pull/1385?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/5c3b32d44d0164aaa9b91405f48e53cf53a82b35?src=pr&el=desc) will **decrease** coverage by `0.07%`.
> The diff coverage is `41.66%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1385/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1385?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1385      +/-   ##
==========================================
- Coverage   84.69%   84.61%   -0.08%     
==========================================
  Files          84       84              
  Lines       12596    12610      +14     
==========================================
+ Hits        10668    10670       +2     
- Misses       1928     1940      +12
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1385?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1385/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl91dGlscy5weQ==) | `87.73% <41.66%> (-2.46%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1385?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1385?src=pr&el=footer). Last update [5c3b32d...9e136ff](https://codecov.io/gh/huggingface/transformers/pull/1385?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


I have reviewed this PR and It looks good to me. Thank you! @julien-c . I added two lines of comments above. Hope they are useful. Thank you. 

Merged in, and superseded by, #1384
a BertForMaskedLM.from_pretrained error
Author: zhujun5164
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (BertForMaskedLM):

Language I am using the model on (Chinese):

When i want to do a limit vocabulary fine-tune in BerForMaskedLM, the new config/ vocab and model.bin in word_embedding are all been change. 

for example:

`config = BertConfig.from_pretrained('bert-base-chinese')`
`config.vocab_size = 2`
`model_dict = torch.load(old_model_path)`
`model_dict['bert.embeddings.word_embeddings.weight'] = model_dict['bert.embeddings.word_embeddings.weight'][:2]`
`torch.save(model_dict, new_model_path)`

`model = BertForMaskedLM.from_pretrained(new_model_path, config=config)`

But when i use the from_pretrained code for the model, an errors has appeared:

` Error(s) in loading state_dict for BertForMaskedLM:
	size mismatch for cls.predictions.bias: copying a param with shape torch.Size([21128]) from checkpoint, the shape in current model is torch.Size([2]).
	size mismatch for cls.predictions.decoder.weight: copying a param with shape torch.Size([21128, 768]) from checkpoint, the shape in current model is torch.Size([2, 768]).` 

And i have try to use the BertModel.from_pretrained, the error does not appear, and use the code to check the data shape.

`torch.load(new_model_path)['bert.embeddings.word_embeddings.weight'].shape `
it appear   torch.Size([2, 768])

Comments:
Masked language modeling is an example of autoencoding language modeling , typically mask one or more of words in a sentence and have the model predict those masked words given the other words in sentence. When you changed the vocab size, the number of word to predic from the model also changed.
This unofficial methods leaved the output embedding and prediction bias unchanged while changed the vocab size, and caused these errors . 
I think the output embeddings should be copied from input embeddings, but the bias needs you to tune-up manually.

------ 
For now, these two lines may make you avoid the error
```
model_dict['cls.predictions.bias'] = model_dict['cls.predictions.bias'][:2]
model_dict['cls.predictions.decoder.weight'] = model_dict['cls.predictions.decoder.weight'][:2]


```

thank!
'convert_tf_checkpoint_to_pytorch.py' file is missing
Author: imayachita
Pull request: no

Hi all,
I pre-trained BERT base model on my domain-specific corpus using ```https://github.com/google-research/bert```  ```create_pretraining_data.py``` and ```run_pretraining.py```.
Now, I want to use it with this ```pytorch-transformers```. I saw from this page https://devhub.io/repos/huggingface-pytorch-pretrained-BERT that there is conversion script from tf checkpoints to ```pytorch_model.bin``` called ```convert_tf_checkpoint_to_pytorch.py``` but the file no longer exists.
Does anyone have solution? Thanks!


Comments:
_PyTorch-pretrained-BERT_ is a older name of this library; now its name is **Transformers**.
You can check the latest docs of the library and install it from PyPi with `pip install transformers` (you have to install manually TensorFlow 2.0 and PyTorch as well through `pip install tensorflow==2.0.0` and `pip install torch`).

Said this, you can read [this](https://github.com/huggingface/transformers/blob/17ea43cf985829634bd86b36b44e5410c6f83e36/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py) Python script whose goal is to convert BERT original TensorFlow checkpoint to PyTorch. The input of this script are the following three:
- the path to the TensorFlow checkpoint, through `--tf_checkpoint_path` parameter
- a JSON file which specifies the model architecture through `--bert_config_file` parameter
- the path to the output converted PyTorch model through `--pytorch_dump_path` parameter

> Hi all,
> I pre-trained BERT base model on my domain-specific corpus using `https://github.com/google-research/bert` `create_pretraining_data.py` and `run_pretraining.py`.
> Now, I want to use it with this `pytorch-transformers`. I saw from this page https://devhub.io/repos/huggingface-pytorch-pretrained-BERT that there is conversion script from tf checkpoints to `pytorch_model.bin` called `convert_tf_checkpoint_to_pytorch.py` but the file no longer exists.
> Does anyone have solution? Thanks!

Thanks @TheEdoardo93!
Planned support for new Grover 1.5B models?
Author: GenTxt
Pull request: no
Labels: name: wontfix

Thanks for the great repo.

Just wondering if there's any planned support for the new Grover 1.5B models?

https://github.com/rowanz/grover (original 1.5B now available via download_model.py)

https://github.com/vanyacohen/opengpt2-1.5B-gpu-inference (slightly different variation)

Cheers



Comments:
No short-term plan to implement this ourselves, but we'd welcome a PR (especially one involving the original authors for validation).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
parameter never_split not added in BasicTokenizer's tokenize
Author: jjyunlp
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):
Bert

Language I am using the model on (English, Chinese....):
English

The problem arise when using:
* [ ] my own modified scripts:  I need to add some special tokens that will not been split during tokenizing. And my special tokens contain punc, like [E1]. This will be split in _run_split_on_punc() if omit parameter never_split in this function.

## To Reproduce

Steps to reproduce the behavior:

1.
 omit parameter never_split when invoke self._run_split_on_punc(token)

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->


```python
    # tokenization_bert.py
    def tokenize(self, text, never_split=None):
       ....
        orig_tokens = whitespace_tokenize(text)
        split_tokens = []
        for token in orig_tokens:
            if self.do_lower_case and token not in never_split:
                token = token.lower()
                token = self._run_strip_accents(token)

            split_tokens.extend(self._run_split_on_punc(token))
        output_tokens = whitespace_tokenize(" ".join(split_tokens))

        return output_tokens
```
Modify 
```python
            split_tokens.extend(self._run_split_on_punc(token))
```
to 
```python
            split_tokens.extend(self._run_split_on_punc(token, never_split))
```
wolud solve this problem.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.