noamaanMulla-03/transformers-issues huggingface dataset:
Performance dramatically drops down after replacing pytorch-pretrained-bert with pytorch-transformers
Author: YuxiXie
Pull request: no

I am trying to run a baseline model, whose encoder is the pretrained BERT ('bert-base-uncased'). I have tried both versions of this package and found that the performance of the pytorch-transformers-BERT is much worse than pytorch-pretrained-bert-BERT, i.e. the BLeU-4 has dropped from 8. to 2.
Below is my codes, I wanna see if there is some important difference between the two versions that will lead to the drop, or it's the wrong way to call the functions in my codes that causes the bad performance.

(1) pytorch-pretrained-bert:
```python
from pytorch_pretrained_bert import BertModel

pretrained = BertModel.from_pretrained('bert-base-uncased')

enc_outputs, *_ = pretrained(src_seq, token_type_ids=src_sep, output_all_encoded_layers=True)
enc_output = enc_outputs[-1]
```

(2) pytorch-transformers
```python
from pytorch_transformers import BertModel, BertConfig

config = BertConfig.from_pretrained('bert-base-uncased')
config.output_hidden_states = True
pretrained = BertModel(config)

enc_outputs = pretrained(src_seq, token_type_ids=src_sep)
enc_output = enc_outputs[0]
enc_outputs = enc_outputs[2][1:]
```

Comments:
Same here. I am finetuning language models on new dataset. Once I change from `pytorch-pretrained-bert` to `pytorch-transformers`, generation quality dramatically drops.  

I  have the same problem. I refer to a example of Named Entity Recognition which used pytorch-pretrained-bert. I changed it to pytorch-transformers, but I got a bad F1 score. It's suppsed to be  0.78, I got 0.41.  

I'm also seeing similar problems after the refactoring related to BertForMultipleChoice models (issue here: https://github.com/huggingface/pytorch-transformers/issues/931) 

Similar issue here. Working on custom adaptation of BERT for STS benchmark dataset. Spearman correlation drops by about 2 points (.78 -> .76) after refactoring 0.6.1 to 1.0.0, even though all parameters are the same. If I reload my old models, I still get the old (higher) scores.

I suspect that this might be due to a different linear warmup function used in 0.6.1 (compared to 0.6.2 and 1.0.0), that returns smaller learning rates.

I think these differences originate from different modifications so it's not really possible to have all of them in one issue like here with no specific description of the setup and condition of each of you.

I've set up templates for the issues to incite people to give more information.

Please re-open separate issues with more details on each setup.

In particular, there is a template called "MIGRATION" which is specifically concerned with giving information on migration issues from pytorch-pretrained-bert.

In the meantime, I will close this issue.

@YuxiXie @dykang @teng1996 Any updates on this?
Tokenize whole sentence vs. tokenize words in sentence then concat
Author: bigkunzi
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
When using En-Fr XLMModel in transformers library,
I found that result from tokenizing a whole sentence is different from tokenizing words in the sentence and then concatenate.

My configuration is as below
**(XLMModel, XLMTokenizer, XLMConfig, 'xlm-mlm-enfr-1024')**

The result is as below
![image](https://user-images.githubusercontent.com/12389429/71879066-84a93000-3170-11ea-91fe-25227ebe478e.png)

The ultimate goal is to 'detokenize' tokenized sentence which is
['I', 'love', 'swim', '##ing'] -> ['I', 'love', 'swimming']
In order to do this, I have to know the raw token's index for each tokenized tokens.

It would be great if anyone can help with this problem.

Comments:
Seems like this is a duplicate of https://github.com/huggingface/transformers/issues/2140

Thank you for mentioning the same issue!
Can't download models or model config
Author: cnzjhdx
Pull request: no

## ❓ Questions & Help
when I run fine-tune examples as  run_squad.py,it turns out error like this:
 E:\tensorflow_natural_question\transformers\examples>python run_squad.py --model_type bert --model_name_or_path bert-base-cased --do_train --do_eval --do_lower_case --train_file $SQUAD_DIR/train-v2.0.json --predict_file $SQUAD_DIR/dev-v2.0.json --per_gpu_train_batch_size 2 --learning_rate 3e-5 --num_train_epochs 1.0 --max_seq_length 384 --doc_stride 128 --output_dir /tmp/debug_squad/
2020-01-05 23:58:07.219057: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
01/05/2020 23:58:08 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 1, distributed training: False, 16-bits training: False
01/05/2020 23:58:13 - INFO - filelock -   Lock 2170508660632 acquired on C:\Users\Administrator\.cache\torch\transformers\b945b69218e98b3e2c95acf911789741307dec43c698d35fad11c1ae28bda352.d7a3af18ce3a2ab7c0f48f04dc8daff45ed9a3ed333b9e9a79d012a0dedf87a6.lock
01/05/2020 23:58:13 - INFO - transformers.file_utils -   https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-config.json not found in cache or force_download set to True, downloading to C:\Users\Administrator\.cache\torch\transformers\tmptcqtrh98

win10+pytorch-gpu1.2.0+python3.7.3



Comments:
Hi, I'm not sure I see what exactly is your problem ? Was there something following this message, like an error or a warning ?

OSError: Couldn't reach server at 'https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-pytorch_model.bin' to download pretrained weights.  
it is disconnect service

ok,thanks.
bias weights not used in T5Model
Author: BramVanroy
Pull request: no

## 🐛 Bug
Running the T5Model on v2.3.0 can show the info message that all bias weights are not used:

> Weights from pretrained model not used in T5Model: ['encoder.block.0.layer.0.layer_norm.bias', 'encoder.block.0.layer.1.layer_norm.bias', 'encoder.block.1.layer.0.layer_norm.bias', 'encoder.block.1.layer.1.layer_norm.bias', 'encoder.block.2.layer.0.layer_norm.bias', 'encoder.block.2.layer.1.layer_norm.bias', 'encoder.block.3.layer.0.layer_norm.bias', 'encoder.block.3.layer.1.layer_norm.bias', 'encoder.block.4.layer.0.layer_norm.bias', 'encoder.block.4.layer.1.layer_norm.bias', 'encoder.block.5.layer.0.layer_norm.bias', 'encoder.block.5.layer.1.layer_norm.bias', 'encoder.final_layer_norm.bias', 'decoder.block.0.layer.0.layer_norm.bias', 'decoder.block.0.layer.1.layer_norm.bias', 'decoder.block.0.layer.2.layer_norm.bias', 'decoder.block.1.layer.0.layer_norm.bias', 'decoder.block.1.layer.1.layer_norm.bias', 'decoder.block.1.layer.2.layer_norm.bias', 'decoder.block.2.layer.0.layer_norm.bias', 'decoder.block.2.layer.1.layer_norm.bias', 'decoder.block.2.layer.2.layer_norm.bias', 'decoder.block.3.layer.0.layer_norm.bias', 'decoder.block.3.layer.1.layer_norm.bias', 'decoder.block.3.layer.2.layer_norm.bias', 'decoder.block.4.layer.0.layer_norm.bias', 'decoder.block.4.layer.1.layer_norm.bias', 'decoder.block.4.layer.2.layer_norm.bias', 'decoder.block.5.layer.0.layer_norm.bias', 'decoder.block.5.layer.1.layer_norm.bias', 'decoder.block.5.layer.2.layer_norm.bias', 'decoder.final_layer_norm.bias']

I think these are to be expected (and that this is actually not a bug), but I'm not sure. https://github.com/huggingface/transformers/issues/180#issuecomment-453937845 mentions that in some cases like this an additional message could be shown indicating whether this is expected behaviour or not but that has not been implemented here.


```python
from transformers import T5Model
import logging

logging.basicConfig(format='%(asctime)s - [%(levelname)s]: %(message)s',
                    datefmt='%d-%b %H:%M:%S',
                    level=logging.INFO)

model = T5Model.from_pretrained('t5-small')

for name, _ in model.named_parameters():
    print(name)
```

Comments:
Yes, all the layer norms in T5 have no bias (so we keep the default value of 0)
How to use GPU to do inference ?
Author: rxy1212
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
When I use pretrained model to infer, how can I use GPU?
```
    tokenizer = XLNetTokenizer.from_pretrained('your-folder-name')
    model = XLNetModel.from_pretrained('your-folder-name')
    inputs = torch.tensor([tokenizer.encode("你好GitHub!")])
    states = model(inputs)[0][0]
```
Like the code above, If I wanna use GPU when `states = model(inputs)[0][0]`, What should I do ?

Comments:
Ok, I find the way. Just do it like the naive Pytorch code.
Fix Tokenizer.from_pretrained `raise OSError`
Author: tamuhey
Pull request: yes

`raise` before OSError seems to be forgotten.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2465?src=pr&el=h1) Report
> Merging [#2465](https://codecov.io/gh/huggingface/transformers/pull/2465?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/f599623a99b808e3d5926d89cd13237457b9eeba?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2465/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2465?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2465   +/-   ##
=======================================
  Coverage   73.23%   73.23%           
=======================================
  Files          87       87           
  Lines       15003    15003           
=======================================
  Hits        10988    10988           
  Misses       4015     4015
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2465?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2465/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `87.56% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2465?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2465?src=pr&el=footer). Last update [f599623...c217821](https://codecov.io/gh/huggingface/transformers/pull/2465?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Numpy compatibility for sentence piece
Author: LysandreJik
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2191?src=pr&el=h1) Report
> Merging [#2191](https://codecov.io/gh/huggingface/transformers/pull/2191?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ceae85ad60da38cacb14eca49f752669a4fe31dc?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2191/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2191?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2191      +/-   ##
==========================================
- Coverage   79.92%   79.92%   -0.01%     
==========================================
  Files         131      131              
  Lines       19469    19470       +1     
==========================================
  Hits        15561    15561              
- Misses       3908     3909       +1
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2191?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2191/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl91dGlscy5weQ==) | `92.1% <100%> (+0.01%)` | :arrow_up: |
| [transformers/tests/utils.py](https://codecov.io/gh/huggingface/transformers/pull/2191/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL3V0aWxzLnB5) | `90.62% <0%> (-3.13%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2191?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2191?src=pr&el=footer). Last update [ceae85a...cb6d54b](https://codecov.io/gh/huggingface/transformers/pull/2191?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Awesome!
Quality of life enhancements in encoding + patch MLM masking
Author: LysandreJik
Pull request: yes

This PR aims to add quality of life features to the encoding mechanism and patches an issue with the masked language modeling masking function.

1 - ~It introduces an `always_truncate` argument to the `encode` method.~ The `always_truncate` argument is now used as default, with no option to set it to `False` when a `max_length` is specified. Currently, if a `max_length` is specified to the `encode` method with a sequence pair, with both sequences being longer than the max length, then the sequence pair won't be truncated. This may then result in a sequence longer than the specified max length, which may crash the preprocessing mechanism (see current `run_glue.py` with the QNLI task). This argument may be further improved by truncating according to the pair of sequences length ratio.

2 - It adds a new return to the `encode_plus` return dictionary: `sequence_ids`. This is a list of numbers corresponding to the position of special/sequence ids. As an example:

```py
sequence = "This is a sequence"
input_ids_no_special = tok.encode(sequence) # [1188, 1110, 170, 4954]
input_ids = tok.encode(sequence, add_special_tokens=True) # [101, 1188, 1110, 170, 4954, 102]
# Special tokens ─────────────────────────────────────────────┴───────────────────────────┘
```

The new method offers several choices: single sequence (with or without special tokens), sequence pairs, and already existing special tokens:

```py
tok.get_sequence_ids(input_ids_no_special) # [0, 1, 1, 1, 1, 0]
tok.get_sequence_ids(input_ids, special_tokens_present=True) # [0, 1, 1, 1, 1, 0]
```

This offers several quality of life changes:
1 - The users are now aware of the location of the encoded sequences in their input ids: they can have custom truncating methods while leveraging model agnostic encoding
2 - Being aware of the location of special tokens is essential in the case of masked language modeling: we do not want to mask special tokens. An example of this is shown in the modified `run_lm_finetuning.py` script.

Considering sequence ids, the naming may not be optimal, therefore I'm especially open to propositions @thomwolf. Furthermore, I'm not sure it is necessary to consider the cases where no special tokens are currently in the sequence.

Comments:
I think we should drop the `always_truncate` param, and just set it to `True` iff `max_length is not None`

Other than that I like it.

As seen with @julien-c , `always_truncate` really should be enabled by default when a `max_length` is specified.
[Proposal] GLUE processors included in library
Author: LysandreJik
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1325?src=pr&el=h1) Report
> Merging [#1325](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1325?src=pr&el=desc) into [glue-example](https://codecov.io/gh/huggingface/pytorch-transformers/commit/a6981076eca5494b9d230f13217c14b93443888a?src=pr&el=desc) will **decrease** coverage by `1.63%`.
> The diff coverage is `34.48%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1325/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1325?src=pr&el=tree)

```diff
@@               Coverage Diff                @@
##           glue-example    #1325      +/-   ##
================================================
- Coverage         81.07%   79.44%   -1.64%     
================================================
  Files                57       62       +5     
  Lines              8207     8489     +282     
================================================
+ Hits               6654     6744      +90     
- Misses             1553     1745     +192
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1325?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/data/processors/\_\_init\_\_.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1325/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvZGF0YS9wcm9jZXNzb3JzL19faW5pdF9fLnB5) | `100% <100%> (ø)` | |
| [pytorch\_transformers/data/\_\_init\_\_.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1325/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvZGF0YS9fX2luaXRfXy5weQ==) | `100% <100%> (ø)` | |
| [pytorch\_transformers/data/processors/glue.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1325/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvZGF0YS9wcm9jZXNzb3JzL2dsdWUucHk=) | `27.45% <15.78%> (ø)` | |
| [pytorch\_transformers/data/metrics/\_\_init\_\_.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1325/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvZGF0YS9tZXRyaWNzL19faW5pdF9fLnB5) | `34.88% <34.88%> (ø)` | |
| [pytorch\_transformers/data/processors/utils.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1325/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvZGF0YS9wcm9jZXNzb3JzL3V0aWxzLnB5) | `42.85% <42.85%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1325?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1325?src=pr&el=footer). Last update [a698107...789ea72](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1325?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Adding gpt-2 large (774M parameters) model
Author: thomwolf
Pull request: yes

Per request #1061

Also, fix a small restriction in a few conversion scripts (easier loading from original JSON configuration files).

Comments:
Oops @LysandreJik forgot to add it in the [list of pretrained models of the doc](https://huggingface.co/pytorch-transformers/pretrained_models.html)

@thomwolf Added it with 2f93971


You're the best!
Updating schedules for state_dict saving/loading
Author: thomwolf
Pull request: yes

This PR updates the schedules so that they can be saved/reloaded using the standard `state_dict()` and `load_state_dict()` methods of PyTorch [`LambdaLR` model](https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.LambdaLR.load_state_dict).

Useful for continuing stopped training as mentioned in #839

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/872?src=pr&el=h1) Report
> Merging [#872](https://codecov.io/gh/huggingface/pytorch-transformers/pull/872?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/268c6cc160ba046d6a91747c5f281f82bd88a4d8?src=pr&el=desc) will **increase** coverage by `0.12%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/872/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/872?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #872      +/-   ##
==========================================
+ Coverage    78.9%   79.03%   +0.12%     
==========================================
  Files          34       34              
  Lines        6192     6228      +36     
==========================================
+ Hits         4886     4922      +36     
  Misses       1306     1306
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/872?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/tests/optimization\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/872/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvb3B0aW1pemF0aW9uX3Rlc3QucHk=) | `98.97% <100%> (+0.4%)` | :arrow_up: |
| [pytorch\_transformers/optimization.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/872/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvb3B0aW1pemF0aW9uLnB5) | `96.62% <100%> (+0.33%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/872?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/872?src=pr&el=footer). Last update [268c6cc...0740e63](https://codecov.io/gh/huggingface/pytorch-transformers/pull/872?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Templates and explanation for adding a new model and example script
Author: thomwolf
Pull request: yes

This PR adds:
- templates and explantations for all the steps needed to add a new model
- a simple template for adding a new example script (basically the current `run_squad` example).
- links to them in the `README` and `CONTRIBUTING` docs.

@LysandreJik and @rlouf, feel free to update if you want to add stuff or fix the wording.

Comments:
Thanks @stefan-it, feel free to give your opinion on the explanation/templates as well, always happy to have your feedback
bugs with run_summarization script
Author: jimkim3
Pull request: no
Labels: name: wontfix

Hi
The evaluation part of this script is missing, I tried to complete it myself but this code does not generation proper string during decoding part for me and does not converege, encoder and decoder weights are not tied as the paper, could you please add the generated text as the output of evaluation code, also please complete the evaluation code, maybe you could use some codes in run_generation.py for this, thanks

Comments:
The summarization/generation scripts are still work in progress. They should be included in the next release.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.