noamaanMulla-03/transformers-issues huggingface dataset:
typos
Author: lpq29743
Pull request: yes

README.md: "formely known as" -> "formerly known as"

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/816?src=pr&el=h1) Report
> Merging [#816](https://codecov.io/gh/huggingface/pytorch-transformers/pull/816?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/71d597dad0a28ccc397308146844486e0031d701?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/816/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/816?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master    #816   +/-   ##
======================================
  Coverage    78.9%   78.9%           
======================================
  Files          34      34           
  Lines        6192    6192           
======================================
  Hits         4886    4886           
  Misses       1306    1306
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/816?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/816?src=pr&el=footer). Last update [71d597d...e5a18b3](https://codecov.io/gh/huggingface/pytorch-transformers/pull/816?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks!
Update Readme link for Fine Tune/Usage section
Author: praateekmahajan
Pull request: yes

Incorrect link for `Quick tour: Fine-tuning/usage scripts`

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/815?src=pr&el=h1) Report
> Merging [#815](https://codecov.io/gh/huggingface/pytorch-transformers/pull/815?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/71d597dad0a28ccc397308146844486e0031d701?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/815/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/815?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master    #815   +/-   ##
======================================
  Coverage    78.9%   78.9%           
======================================
  Files          34      34           
  Lines        6192    6192           
======================================
  Hits         4886    4886           
  Misses       1306    1306
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/815?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/815?src=pr&el=footer). Last update [71d597d...0d46b17](https://codecov.io/gh/huggingface/pytorch-transformers/pull/815?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
fix some errors for distributed lm_finetuning
Author: yzy5630
Pull request: yes

1. makedirs 
2. save models

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/797?src=pr&el=h1) Report
> Merging [#797](https://codecov.io/gh/huggingface/pytorch-transformers/pull/797?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/5fe0b378d899f81eb0a7f2db0c4eb0234748e915?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/797/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/797?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #797   +/-   ##
=======================================
  Coverage   78.91%   78.91%           
=======================================
  Files          34       34           
  Lines        6193     6193           
=======================================
  Hits         4887     4887           
  Misses       1306     1306
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/797?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/797?src=pr&el=footer). Last update [5fe0b37...a7ba27b](https://codecov.io/gh/huggingface/pytorch-transformers/pull/797?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hi, do you want to update these examples to the new `pytorch-transformers` API at the same time?
Models now return `tuple` so we should take the first element of the model output as the loss and we should also update `BertAdam` to `AdamW`.

ok， I  have a try, I‘m stilll use the old API. 

Hi,  I have updated the opt to the new API , please check. 

Thanks @yzy5630!
SEG_ID constants for XLNet misleading/off
Author: sleepinyourhat
Pull request: no

https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/tokenization_xlnet.py#L47 shows:
```
# Segments (not really needed)
SEG_ID_A   = 0
SEG_ID_B   = 1
SEG_ID_CLS = 2
SEG_ID_SEP = 3
SEG_ID_PAD = 4
```

These don't seem to be used anywhere in the repo, but I tried using them as a shortcut myself, and I'm not sure they're right. In contrast, for xlnet-base-cased, I get:

```
        self._sep_id = tokenizer.convert_tokens_to_ids("<sep>")
        self._cls_id = tokenizer.convert_tokens_to_ids("<cls>")
        self._pad_id = tokenizer.convert_tokens_to_ids("<pad>")
        print(self._cls_id, self._sep_id, self._pad_id)
```
```
3 4 5
```

Comments:
Yes I will remove them. They are used in the `run_glue.py` example (like in the original TF repo) but they don't have any reason to be in the library it-self.

In XLNet segment ids (what we call `token_type_ids in the repo) don't correspond to embeddings, they are just numbers and the only important thing is that they have to be different for tokens which belong to different segments, hence the flexibility in the exact values (XLNet is using relative segment difference with just two segment embeddings: 0 if the segment id of two tokens are the same, 1 if not). See [here](https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_xlnet.py#L926-L928).

It's in the XLNet paper but I should probably add a word or two in the docstring as well.

Ah, got it. Thanks! I had assumed that these were part of the token vocabulary. I didn't realize that there were more than two segment types.

By the way, the default [here](https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_glue.py#L259) seems off—the [CLS] token for BERT is marked as part of segment 1/B, while the paper shows it as part of 0/A with the rest of the first input.

But that's another issue, and I'm not fully certain. Closing.
GPT sentence log loss: average or summed loss?
Author: jhlau
Pull request: no

>>> config = GPT2Config.from_pretrained('gpt2')
>>> tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
 >>> model = GPT2LMHeadModel(config)
 >>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
 >>> outputs = model(input_ids, labels=input_ids)
 >>> loss, logits = outputs[:2]

For the loss value computed for the sentence, is it an average log loss or summed log loss? I had a look at CrossEntropyLoss in torch.nn and it seems to be an average loss, but thought I'd double check.

If there are multiple sentences in the input instead (so batch size > 1), what does it return? The average logloss over all tokens in the two sentences?

Comments:
Yes, it's the average

Thanks for the prompt reply. Much appreciated.
why use  self.apply(self.init_bert_weights) in inhiritance class？
Author: g-jing
Pull request: no
Labels: name: wontfix

self.apply(self.init_bert_weights) is already used in BertModel class, why do we still need to use self.apply(self.init_bert_weights) in all inhiritance model such as BertTokenClassificaiton model?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How can we import cased bert model?
Author: g-jing
Pull request: no
Labels: name: wontfix

I notice that the author only shows us how to use uncased model, could anyone show me how to import cased model in both BertModel and BertTokenClassifier model? Thanks

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[bug]BertAdam change to AdamW in example
Author: shibing624
Pull request: no

https://github.com/huggingface/pytorch-transformers/blob/master/examples/lm_finetuning/simple_lm_finetuning.py#L35  BertAdam change to AdamW

Comments:
Changing that line still causes an error in line 568, BertAdam has to be changed to AdamW as well and the warmup kwarg has to be removed.

#797 (specifically d6522e28732fd14a926440ef5f315e6a8e13792c) 

have fix the error! I tested it on toy dataset.

@shibing624   this bug can be closed
Providing older documentation
Author: Ricocotam
Pull request: no

Hey, would it be possible to release the previous documentation ? I'm working on previous version and can't find proper doc right now. 

Thanks if you can help

Comments:
Hi, you may go to https://github.com/huggingface/pytorch-transformers/releases, select the release you are working with and in its "Assets" download the repo and navigate the code, together with documentation

Hi, here is the older documentation: https://github.com/huggingface/pytorch-transformers/tree/v0.6.2
CUDA error: invalid configuration argument when not using DataParallel
Author: Phirefly9
Pull request: no

Good Evening,

We have a DGX2 system running the latest Nvidia pytorch docker container - 19.06.  When attempting to use the gpt2 or gpt2-medium models to extract out embeddings we are getting the following error, but only when not using dataparallel: (note we are using apex here but an optimization level of 0, this issue occurs without apex)

```
[...]                                                                                                                                           
  File "[removed]", line 167, in forward
    x1_emb, past = self.embedding(x1)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 494, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_transformers/modeling_gpt2.py", line 515, in forward
    outputs = block(hidden_states, layer_past, head_mask[i])
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 494, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_transformers/modeling_gpt2.py", line 332, in forward
    output_attn = self.attn(self.ln_1(x), layer_past=layer_past, head_mask=head_mask)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 494, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_transformers/modeling_gpt2.py", line 285, in forward
    x = self.c_attn(x)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 494, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_transformers/modeling_utils.py", line 490, in forward
    x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
RuntimeError: CUDA error: invalid configuration argument
```

I found reference to this issue from Pytorch:
https://github.com/pytorch/pytorch/issues/2080
but it is reported as fixed so I'm not sure if this issue belongs to this repo or pytorch.

Comments:
Further testing showed this was caused by the batch size being too high and the card running out of memory, and providing a misleading error.
How do you train custom corpus with bert?
Author: shadylpstan
Pull request: no
Labels: name: Need more information

name: wontfix

I am using a domain specific dataset for text classification.
But major of my data points are treated with [UNK] token in Bert.

Can I please get help on how to keep my custom corpus tokens?

Comments:
Maybe it's a problem of not setting `do_lower_case=False` and using a `cased` model like in #436.
Which pretrained model are you using?

@thomwolf 
sorry but this is completely not what I am asking for.
Suppose I have medical data and the names of the medicines and diseases are not in the bert pre-trained corpus. So bert gives me [UNK] token for important medical terms. How can I train it on my custom data?

Using lower cased multilingual pre-trained model.

**PS: Medical data was just an example.** 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi, I'm also facing similar issue plz let me know if you have found any solution @shadylpstan 


> Hi, I'm also facing similar issue plz let me know if you have found any solution @shadylpstan

I was successful in modifying a couple of bert files and train it using my corpus.
Though it was really a mess doing so because by default bert has a classification model of a certain number of classes.
The accuracy wasn't good enough. So no idea it is a worth giving it shot from your end.
Import Error
Author: goyalsaransh97
Pull request: no
Labels: name: wontfix

I'm getting error " ImportError: cannot import name 'WEIGHTS_NAME' from 'pytorch_pretrained_bert.file_utils' " on running run_squad.py. I've already tried building from source but the problem persists.

Comments:
This should be fixed with the new release (0.6.2).

Unfortunately, I still get this error with the new release. Could that be because I had installed the package before some time ago (and removed it afterwards)? 

Never mind, got it running by cleaning up the environments/paths.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Fixed typo
Author: rish-16
Pull request: yes

Fixed typo in README.md

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/844?src=pr&el=h1) Report
> Merging [#844](https://codecov.io/gh/huggingface/pytorch-transformers/pull/844?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/a615499076a67dceb8907ecdf8eadaff04bb8d6a?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/844/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/844?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master    #844   +/-   ##
======================================
  Coverage    78.9%   78.9%           
======================================
  Files          34      34           
  Lines        6192    6192           
======================================
  Hits         4886    4886           
  Misses       1306    1306
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/844?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/844?src=pr&el=footer). Last update [a615499...6b3d9ad](https://codecov.io/gh/huggingface/pytorch-transformers/pull/844?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Learning from scratch not working
Author: kilyyaoyao
Pull request: no
Labels: name: wontfix

I'm using simple_lm_learning as it was, except I didn't use the model from pretrained, but a new BertForPreTraining model with the same config as bert-base, how come it's not learning anything and predicting the same token 1996 ("the") for every output?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
global grad norm clipping (#581)
Author: lukovnikov
Pull request: yes
Labels: name: wontfix

(see #581 )
- norm-based gradient clipping was being done per param group
- when more than one param group, this is different from global norm grad clipping

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Reproduce the results on CoLA 
Author: cooelf
Pull request: no
Labels: name: Discussion

name: wontfix

I try to reproduce the CoLA results reported in the BERT paper but the numbers are far from the reported one. My best mcc (BERT large) for dev is 64.79% and the test result is 56.9% while the reported test result is 60.5%. The learning rate is 2e-5 and the total number of epochs is 5. For BERT base，the result is also lower by 3-5%.

As the paper said, 
`for BERTLARGE we found that fine-tuning was sometimes unstable on  small data sets (i.e., some runs would produce degenerate results), so we ran several random restarts and selected the model that performed best on the Dev set. `

I also tried several restarts with different learning rates and random seeds but it seems no improvement. I'm quite confused for the reproduction. Any suggestions would be greatly appreciated.

Comments:
Cola is probably one of the most unstable tasks for BERT. For us it mostly boiled down to running many seeds. If all you care about is a good pre-trained model checkpoint, we have a 65 / 61 run at https://github.com/zphang/bert_on_stilts 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@cooelf hi, can you reproduce cola score of BERT now? I work on this recently but still can't reach the reported score on the test set, even if restart with different seed multiple times, and I further noticed that improvement on dev set is inconsistent with that on the test set at all.

Thanks for all the feedbacks! 
@scissorsy The results of dev and test are indeed incosistent. I have changed the seeds, lr, warmup rates and max_epochs, and finally the test result reached about 60% for some runs. Using multi-tasking or transfer from bigger dataset such as MNLI seems to be more stable.



> Thanks for all the feedbacks!
> @scissorsy The results of dev and test are indeed incosistent. I have changed the seeds, lr, warmup rates and max_epochs, and finally the test result reached about 60% for some runs. Using multi-tasking or transfer from bigger dataset such as MNLI seems to be more stable.

Thanks!

Hi @cooelf, what parameter number did you change in order to fit a better result, thanks!
Speedup using NVIDIA Apex 
Author: llidev
Pull request: no

Hi,

According to PR https://github.com/huggingface/pytorch-pretrained-BERT/pull/116, we should be able to achieve a 3-4 x speed up for both bert-base and bert-large. However, I can only achieve 2x speed up with  bert-base. My docker image uses CUDA9.0 while the discussion in the PR https://github.com/huggingface/pytorch-pretrained-BERT/pull/116 is based on CUDA10.0... I am wondering if that makes the difference....

Thanks

Comments:
Maybe. You can try with pytorch docker image `dockerhub 1.0-cuda10.0-cudnn7` to debug, as we did in the discussion in PR #116.

Just verified that CUDA10.0 makes 4x speedup. It should be better to include this in the main document.

What GPU do you run ... and how do you increase such a speedup? Is this possible with gtx 1080?

Thanks
xlnet input_mask and attention_mask type error
Author: Saner3
Pull request: no

when I use:

```input_mask = (input_ids == 0)```
```perm_mask = perm_mask = torch.zeros((1, input_ids.shape[1], input_ids.shape[1]), dtype=torch.float, device=device)```
```perm_mask[:, :, -1] = 1.0  # Previous tokens don't see last token```

File "pytorch-transformers/pytorch_transformers/modeling_xlnet.py", line 881, in forward
    data_mask = input_mask[None] + perm_mask
RuntimeError: expected backend CUDA and dtype Float but got backend CUDA and dtype Byte

input_mask is a Tensor but perm_mask is FloatTensor

Comments:
Humm you are right, the docstrings are off, it would be more clear if they were all indicated as `torch.FloatTensor` (even though officially torch.Tensor is an alias for the default tensor type (torch.FloatTensor)).
finetune_on_pregenerate Loss.backwards() throw an error 
Author: sophiapeng0426
Pull request: no

In finetune_on_pregenerated.py, loss are tuples and thus loss.backward() is not going to work.

Original:
loss = model(input_ids, segment_ids, input_mask, lm_label_ids, is_next)

Update:
loss, _ , _ = model(input_ids, segment_ids, input_mask, lm_label_ids, is_next)

Is this fix correct?

Comments:
Yes this example should have been updated now with #797.