noamaanMulla-03/transformers-issues huggingface dataset:
How to get the softmax probabilities from the TransfoXLLMModel
Author: Shashi456
Pull request: no
Labels: name: wontfix

```
A tuple of (last_hidden_state, new_mems)
        `softmax_output`: output of the (adaptive) softmax:
            if target is None:
                Negative log likelihood of shape [batch_size, sequence_length] 
            else:
                log probabilities of tokens, shape [batch_size, sequence_length, n_tokens]
        `new_mems`: list (num layers) of updated mem states at the entry of each layer
            each mem state is a torch.FloatTensor of size [self.config.mem_len, batch_size, self.config.d_model]
            Note that the first two dimensions are transposed in `mems` with regards to `input_ids` and `target`
```


how do i get the negative log likelihood of a given sentence, like say modifying the example : 
```
# Tokenized input
text_1 = "Who was Jim Henson ?"
text_2 = "Jim Henson was a puppeteer"
tokenized_text_1 = tokenizer.tokenize(text_1)
tokenized_text_2 = tokenizer.tokenize(text_2)

# Convert token to vocabulary indices
indexed_tokens_1 = tokenizer.convert_tokens_to_ids(tokenized_text_1)
indexed_tokens_2 = tokenizer.convert_tokens_to_ids(tokenized_text_2)

# Convert inputs to PyTorch tensors
tokens_tensor_1 = torch.tensor([indexed_tokens_1])
tokens_tensor_2 = torch.tensor([indexed_tokens_2])
# Load pre-trained model (weights)
model = TransfoXLLMHeadModel.from_pretrained('transfo-xl-wt103')
model.eval()

# If you have a GPU, put everything on cuda
tokens_tensor_1 = tokens_tensor_1.to('cuda')
tokens_tensor_2 = tokens_tensor_2.to('cuda')
model.to('cuda')

with torch.no_grad():
    # Predict all tokens
    predictions_1, mems_1 = model(tokens_tensor_1)
    # We can re-use the memory cells in a subsequent call to attend a longer context
    predictions_2, mems_2 = model(tokens_tensor_2, mems=mems_1)

```


now when i print the shape of predictions_1 i get `torch.Size([1, 5, 267735])`

this is in relation with #477, and also one final thing, is there any way i could fasten the model loading and inference time in eval mode, i just need to use this as a reward in an RL setup, but it takes a tremendous amount of time to evaluate sentence, is there any way i could fasten the process?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
read().splitlines() -> readlines()
Author: Yiqing-Zhou
Pull request: yes

splitlines() does not work as what we expect here for bert-base-chinese because there is a '\u2028' (unicode line seperator) token in vocab file. Value of '\u2028'.splitlines() is ['', ''].
Perhaps we should use readlines() instead.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/860?src=pr&el=h1) Report
> Merging [#860](https://codecov.io/gh/huggingface/pytorch-transformers/pull/860?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/2f869dc6651f9cf9253f4c5a43279027a0eccfc5?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/860/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/860?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master    #860   +/-   ##
======================================
  Coverage    78.9%   78.9%           
======================================
  Files          34      34           
  Lines        6192    6192           
======================================
  Hits         4886    4886           
  Misses       1306    1306
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/860?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/860/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX2JlcnQucHk=) | `93.2% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/860?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/860?src=pr&el=footer). Last update [2f869dc...bef0c62](https://codecov.io/gh/huggingface/pytorch-transformers/pull/860?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great, thanks!
typos
Author: lpq29743
Pull request: yes

"ouputs" -> "outputs"

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/847?src=pr&el=h1) Report
> Merging [#847](https://codecov.io/gh/huggingface/pytorch-transformers/pull/847?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/a615499076a67dceb8907ecdf8eadaff04bb8d6a?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/847/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/847?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master    #847   +/-   ##
======================================
  Coverage    78.9%   78.9%           
======================================
  Files          34      34           
  Lines        6192    6192           
======================================
  Hits         4886    4886           
  Misses       1306    1306
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/847?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/847?src=pr&el=footer). Last update [a615499...76be189](https://codecov.io/gh/huggingface/pytorch-transformers/pull/847?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Indeed!
fixed version issues in run_openai_gpt
Author: rabeehk
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/845?src=pr&el=h1) Report
> Merging [#845](https://codecov.io/gh/huggingface/pytorch-transformers/pull/845?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/a615499076a67dceb8907ecdf8eadaff04bb8d6a?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/845/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/845?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master    #845   +/-   ##
======================================
  Coverage    78.9%   78.9%           
======================================
  Files          34      34           
  Lines        6192    6192           
======================================
  Hits         4886    4886           
  Misses       1306    1306
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/845?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/845?src=pr&el=footer). Last update [a615499...f63ff53](https://codecov.io/gh/huggingface/pytorch-transformers/pull/845?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


LGTM, thanks @rabeehk
Fixed PreTrainedModel.from_pretrained(...) not passing cache_dir to PretrainedConfig.from_pretrained(...)
Author: mbugert
Pull request: yes

See #863 

It's not a beautiful solution, but neither is the practice of modifying incoming parameters via pop. 🤷‍♂ 

Comments:
Indeed thanks. We'll subsume this PR with #866 which add a few other stuff.

I agree with you on the `pop` pattern. We'll move away from this when the first one of these two events happens: (i) google stop open-sourcing interesting new models or (ii) google stop using python 2 internally ;)

Okay! :+1:
CLS segment_id for BERT
Author: guillaume-be
Pull request: no

Hello,

In your example for GLUE you set the CLS segment id token to 1 for BERT: https://github.com/huggingface/pytorch-transformers/blob/2f869dc6651f9cf9253f4c5a43279027a0eccfc5/examples/run_glue.py#L259

Reading the original reference implementation it seems that CLS should have a segment_id=0. This is also aligned with several comments & docstrings you have around the code. Is this a design choice? What is the impact on general performance?

Comments:
Yes, this one was also mentioned in https://github.com/huggingface/pytorch-transformers/issues/810#issuecomment-512991164.

It is fixed now.
fp16 is broken
Author: zsk423200
Pull request: no

run run_glue.py with the parameter of --fp16, and return error:
```
RuntimeError: Incoming model is an instance of torch.nn.parallel.DistributedDataParallel. Parallel wrappers should only be applied to the model(s) AFTER
the model(s) have been returned from amp.initialize.
```

i find the reason is the wrong order of `amp.initialize` and `model = torch.nn.DataParallel`, [iIf DDP wrapping occurs before amp.initialize, amp.initialize will raise an error](https://github.com/NVIDIA/apex/blob/master/examples/imagenet/README.md), and it worked after i change the order

Comments:
Indeed, thanks! Fixed in master
fp16 is not work
Author: zsk423200
Pull request: no

GPU:v100

run run_glue.py with the command in the README:
```
python ./examples/run_glue.py \
    --model_type bert \
    --model_name_or_path bert-base-uncased \
    --task_name $TASK_NAME \
    --do_train \
    --do_eval \
    --do_lower_case \
    --data_dir $GLUE_DIR/$TASK_NAME \
    --max_seq_length 128 \
    --per_gpu_eval_batch_size=8   \
    --per_gpu_train_batch_size=8   \
    --learning_rate 2e-5 \
    --num_train_epochs 3.0 \
    --output_dir /tmp/$TASK_NAME/
```
and i also run with --fp16,  but spend the same time and gpu memory, why fp16 does not work

Comments:
Duplicate of #868
PreTrainedModel.from_pretrained(...) doesn't pass cache_dir to PretrainedConfig.from_pretrained(...)
Author: mbugert
Pull request: no

The cache_dir key-value parameter does not work as intended in `PreTrainedModel.from_pretrained(...)`. It is popped from the kwargs, then `PretrainedConfig.from_pretrained(...)` is called which expects this parameter in the kwargs, but it's obviously not there anymore. A default location is used as a fallback, but this leads to strange behaviour if this default location doesn't exist or isn't writable (as it was in my case).

Comments:
Fix with #866
Fix openai-gpt ROCStories example's issues with AdamW optimizer
Author: prrao87
Pull request: yes

Fixes the `AdamW` optimizer instance in the `openai-gpt` ROCStories example as per the new API. The default arguments for it are now set as per the [documentation](https://huggingface.co/pytorch-transformers/model_doc/bert.html?highlight=adamw#pytorch_transformers.AdamW).

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/811?src=pr&el=h1) Report
> Merging [#811](https://codecov.io/gh/huggingface/pytorch-transformers/pull/811?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/71d597dad0a28ccc397308146844486e0031d701?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/811/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/811?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master    #811   +/-   ##
======================================
  Coverage    78.9%   78.9%           
======================================
  Files          34      34           
  Lines        6192    6192           
======================================
  Hits         4886    4886           
  Misses       1306    1306
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/811?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/811?src=pr&el=footer). Last update [71d597d...51d66f1](https://codecov.io/gh/huggingface/pytorch-transformers/pull/811?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


We should also probably add a linearly decreasing schedule (not in the optimizer anymore).
Did you try this version with flat learning rate? Does it have good performances?

I didn't run the training loop to completion but I looked at it now and it seems there's an issue with the evaluation routine (it crashes with an error). Will look into this as soon as I have time. 

@thomwolf I used the code from #845 and the flat learning rate does the trick - eval accuracy of 87.2% after 3 training epochs. The defaults args in `run_openai_gpt.py` are good, just confirming some of them used for this result below. 

```
python run_openai_gpt.py  \
  --model_name openai-gpt  \
  --do_train  \
  --do_eval  \
  --train_dataset "./ROCStories/cloze_test_val__spring2016 - cloze_test_ALL_val.csv"  \
  --eval_dataset "./ROCStories/cloze_test_test__spring2016 - cloze_test_ALL_test.csv"  \
  --train_batch_size 8  \
  --eval_batch_size 16  \
  --num_train_epochs  3
```
It makes sense to go ahead and close this.  Thanks!
Bug of BertTokenizer
Author: LeeJuly30
Pull request: no

when load a tokenizer from pretrain
```python
tokenizer = BertTokenizer.from_pretrained(vocab_path)
```
the vocab length is:
```python
len(tokenizer.vocab)
21128
```
but the last token of vocab is:
```python
next(reversed(tokenizer.vocab.items()))
('##😎', 21129)
```


Comments:
If you are loading the chinese model, this is probably related to #860  and #825.
Should be fixed now.

thanks!
Error loading converted pytorch checkpoint
Author: hguan6
Pull request: no

I am using BioBert. After converting the tensorflow checkpoint to pytorch checkpoint, I want to load it to Bert model. I found that the old pytorch-pretrained-bert works perfect but the new pytorch-transformer fails.
Here is the successful run using pytorch-pretrained-bert:

> from pytorch_pretrained_bert import BertForSequenceClassification
> model = BertForSequenceClassification.from_pretrained("biobert_v1.1_pubmed/", num_labels=1)
> exit()


Here's the failure in pytorch_transformers:

> from pytorch_transformers import BertForSequenceClassification
> model = BertForSequenceClassification.from_pretrained("biobert_v1.1_pubmed/", num_labels=1)
> 
> Model name 'biobert_v1.1_pubmed/' was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc). We assumed 'biobert_v1.1_pubmed/config.json' was a path or url but couldn't find any file associated to this path or url.
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/home/michael/.local/lib/python3.7/site-packages/pytorch_transformers/modeling_utils.py", line 403, in from_pretrained
>     model = cls(config)
>   File "/home/michael/.local/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 958, in __init__
>     super(BertForSequenceClassification, self).__init__(config)
>   File "/home/michael/.local/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 548, in __init__
>     super(BertPreTrainedModel, self).__init__(*inputs, **kwargs)
>   File "/home/michael/.local/lib/python3.7/site-packages/pytorch_transformers/modeling_utils.py", line 206, in __init__
>     self.__class__.__name__, self.__class__.__name__
> ValueError: Parameter config in `BertForSequenceClassification(config)` should be an instance of class `PretrainedConfig`. To create a model from a pretrained model use `model = BertForSequenceClassification.from_pretrained(PRETRAINED_MODEL_NAME)`



Comments:
Do you have a file named `biobert_v1.1_pubmed/config.json` as mentioned in the error?

Oh thanks. There is a config file name "bert_config.json" in the directory "biobert_v1.1_pubmed/". I changed the file name to "config.json" and it works! 
I am wondering why the pytorch-pretrained-bert can load the checkpoint. Maybe it reads "bert_config.json" and pytorch-transformer reads "config.json"?
can not convert_tf_checkpoint_to_pytorch
Author: Zhangxuri
Pull request: no

```
.
├── convert_tf_checkpoint_to_pytorch.py
├── uncased_L-12_H-768_A-12
│   ├── bert_config.json
│   ├── bert_model.ckpt.data-00000-of-00001
│   ├── bert_model.ckpt.index
│   ├── bert_model.ckpt.meta
│   └── vocab.txt
├── uncased_L-12_H-768_A-12.zip
└── Untitled.ipynb
```

```
(base) ➜  ckpt_to_bin git:(master) ✗ python convert.py --tf_checkpoint_path=./uncased_L-12_H-768_A-12 --bert_config_file=./uncased_L-12_H-768_A-12/bert_config.json --pytorch_dump_path=./uncased_L-12_H-768_A-12
Building PyTorch model from configuration: {
  "attention_probs_dropout_prob": 0.1,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "num_labels": 2,
  "output_attentions": false,
  "output_hidden_states": false,
  "torchscript": false,
  "type_vocab_size": 2,
  "vocab_size": 30522
}

INFO:pytorch_transformers.modeling_bert:Converting TensorFlow checkpoint from /home/zxr/summary/bertsum/src/ckpt_to_bin/uncased_L-12_H-768_A-12
Traceback (most recent call last):
  File "convert.py", line 65, in <module>
    args.pytorch_dump_path)
  File "convert.py", line 36, in convert_tf_checkpoint_to_pytorch
    load_tf_weights_in_bert(model, config, tf_checkpoint_path)
  File "/home/zxr/anaconda3/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 83, in load_tf_weights_in_bert
    init_vars = tf.train.list_variables(tf_path)
  File "/home/zxr/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/checkpoint_utils.py", line 95, in list_variables
    reader = load_checkpoint(ckpt_dir_or_file)
  File "/home/zxr/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/checkpoint_utils.py", line 63, in load_checkpoint
    "given directory %s" % ckpt_dir_or_file)
ValueError: Couldn't find 'checkpoint' file or checkpoints in given directory /home/zxr/summary/bertsum/src/ckpt_to_bin/uncased_L-12_H-768_A-12

```

Comments:
```
python convert.py --tf_checkpoint_path=./uncased_L-12_H-768_A-12/bert_model.ckpt --bert_config_file=./uncased_L-12_H-768_A-12/bert_config.json --pytorch_dump_path=./uncased_L-12_H-768_A-12/bert_model.bin
```
Custom data, gradient explosion, accuracy is 0
Author: pbamotra
Pull request: no
Labels: name: wontfix

Hi,

I have 16000+ labels to predict using sequence classifier. I tried running the code with BertAdam (no gradient clipping) and low LR of 1e-5. But my loss doesn't not improve and the accuracy stays at zero. Gradient clipping doesn't help either. I've check my inputs and they are correct. Any help is welcome.

model: base uncased

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Can lm_finetuning be used with non-english data ? 
Author: ereday
Pull request: no

Hi, 
My target domain is in German. Can I still use the scripts &codes under `lm_finetuning` folder to finetune pre-trained Bert models  or are those only for English target domains? 

Comments:
@ereday Take the `simple_lm_finetuning.py` script for example. It has a `--bert_model` argument. When your target domain is German, then you should use the recently introduced [BERT model for german](https://github.com/huggingface/pytorch-transformers/pull/688) via passing `bert-base-german-cased`. 

This should fine-tune the German BERT model :)

@stefan-it  thanks alot. I wasn't aware of german specific bert model. Awesome!
make save_pretrained do the right thing with added tokens
Author: joelgrus
Pull request: yes

right now it's dumping the *decoder* when it should be dumping the *encoder*. and then (for obvious reasons) you get an error when you try to load "from_pretrained" using that dump.

this PR fixes that.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/893?src=pr&el=h1) Report
> Merging [#893](https://codecov.io/gh/huggingface/pytorch-transformers/pull/893?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/067923d3267325f525f4e46f357360c191ba562e?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/893/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/893?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #893   +/-   ##
=======================================
  Coverage   79.03%   79.03%           
=======================================
  Files          34       34           
  Lines        6234     6234           
=======================================
  Hits         4927     4927           
  Misses       1307     1307
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/893?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/893/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX3V0aWxzLnB5) | `86.53% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/893?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/893?src=pr&el=footer). Last update [067923d...ae152ce](https://codecov.io/gh/huggingface/pytorch-transformers/pull/893?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Indeed, thanks Joel!