noamaanMulla-03/transformers-issues huggingface dataset:
GPT-2 FineTuning on Cloze/ ROC
Author: rohuns
Pull request: no
Labels: name: Discussion

Hi, wrote some code to finetune GPT2 on rocstories using the DoubleHeads model mirroring the GPT1 code. However, I'm only getting performance of 68% on the eval. Was wondering if anyone else had tried it and seen this drop in performance. Thanks

Comments:
Hi rohuns, I was wondering what padding value have you used for the lm_labels, since the -1 specified in the docs doesn't work for me on GPT2LMHead model. See #577. 

> Hi rohuns, I was wondering what padding value have you used for the lm_labels, since the -1 specified in the docs doesn't work for me on GPT2LMHead model. See #577.

I had just used -1, can take a look at your stack trace and respond on that chat

Also to close this issue it appears others also achieved similar performance on the MC task, more details on the thread issue #468 

> > Hi rohuns, I was wondering what padding value have you used for the lm_labels, since the -1 specified in the docs doesn't work for me on GPT2LMHead model. See #577.
> 
> I had just used -1, can take a look at your stack trace and respond on that chat

Yes, please do have a look. Here is a toy example with a hand-coded dataset to prove that the -1 throws an error. It looks like it's a library issue.
[gpt2_simplified.py.zip](https://github.com/huggingface/pytorch-pretrained-BERT/files/3144185/gpt2_simplified.py.zip)

Regards,
Adrian
move pytroch_pretrained_bert cache folder under same path as torch
Author: ailzhang
Pull request: yes

This PR does two things: 

* Envs available: 
PYTORCH_PRETRAINED_BERT_CACHE > TORCH_HOME > XDG_CACHE_HOME > `~/.cache`
* If no env is set, the default path is 
`~/.cache/torch/pytorch_pretrained_bert` where `pytorch_pretrained_bert` is visible instead of hidden `.pytorch_pretrained_bert`. (since this is the cache folder, I feel it makes sense to make it visible, please correct me if I'm wrong :) 
* minor: fix typo in `hubconf.py` example.


Comments:
Ok, looks good, thanks @ailzhang!
[Question] Cross-lingual sentence representations
Author: shoegazerstella
Pull request: no

Hi, 
Would it be possible to integrate also a BERT model for cross-lingual sentence representations? 
Something like, for example, the `XNLI-15` model in [https://github.com/facebookresearch/XLM](https://github.com/facebookresearch/XLM).
Thanks!

Comments:
Hi @shoegazerstella well XLM is already pretty much as powerful as BERT and focused on cross-lingual sentence representations so I would go directly for it instead of BERT.

Thanks @thomwolf, 
Are you considering integrating something for cross-lingual representations in the `pytorch-pretrained-BERT` library in the near future?

Not in the short-term
Fix gradient overflow issue during attention mask
Author: SudoSharma
Pull request: yes

This fix is in reference to issue #382. GPT2 can now be trained in mixed precision, which I've confirmed with testing. I also tested unconditional generation on multiple seeds before and after changing 1e10 to 1e4 and there was no difference. Please let me know if there is anything else I can do to make this pull request better. Thanks for all your work!

Comments:
Ok, great, thanks @SudoSharma!

While the outputs are the same between 1e10 and 1e4, I shouldn't expect the outputs between fp32 and fp16 to be the same, should I? I get different outputs between the two when doing unconditional/conditional generation with top_k=40 but even with top_k=1. Usually they're the same for a while and then deviate. This is with Apex installed, so using FusedLayerNorm.

If I turn on Apex's AMP with `from apex import amp; amp.init()` then they still deviate but after a longer time (I think it makes the attention nn.Softmax use fp32). Have to remove the `model.half()` call when using AMP.

Perhaps it's not realistic to have the outputs be the same when fp16 errors in the "past" tensors are compounding as the sequence gets longer? But it is surprising to see them differ for top_k=1 (deterministic) since only the largest logit affects the output there.

P.S. For my site it's been enormously helpful to have this PyTorch implementation. @thomwolf Thank you!

Hi @AdamDanielKing,
Congratulation on your demo!
Are you using the updated API for apex Amp? (https://nvidia.github.io/apex/amp.html)
Also, we should discuss this in a new issue? At first, I thought this was related to this PR but I understand it's not, right?

@thomwolf You're probably right that a new issue is best. I've created one at #602.

Thanks for pointing out I was using the old Apex API. Switching to the new one unfortunately didn't fix the issue though.
Anyone tried this model to write a next sentence?
Author: qnkhuat
Pull request: no
Labels: name: Discussion

name: wontfix


Comments:
Closing for now.

Why ? @thomwolf 

I'm trying to clean up the issues to get a better view of what needs to be fixed.
But you are right opening/closing issue is too binary. Let's add labels instead.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Transformer-XL: wrong encoding in the vocab
Author: akhti
Pull request: no
Labels: name: Help wanted

name: wontfix

Seems that something odd happened during vocab serialization as many symbols with non-latin symbols are broken.
E.g.:
```
In [1]: import pytorch_pretrained_bert

In [2]: tokenizer = pytorch_pretrained_bert.TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')

In [4]: print(tokenizer.idx2sym[224178])
'EnquÃªtes
```

The correct token should be "'Enquêtes". And there are around 10k tokens like this.

Could it be 'encoding="latin1"' here?
https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/convert_transfo_xl_checkpoint_to_pytorch.py#L54

Comments:
Yeah, the re-encoding seems to fix the bug:
```
In [6]: "'EnquÃªtes".encode('latin1').decode('utf8')
Out[6]: "'Enquêtes"
```

which version of python are you using?

It's python3.6. Does the snippet above gives different result on other version?

JFYI: I'm using the script below to create a fixed vocab and save in the current folder.
```(python)
import collections
import urllib.request

from pytorch_pretrained_bert import tokenization_transfo_xl
import torch


def fix(x):
    return x.encode('latin1').decode('utf8')


def main():
    basedir = '.'
    good_vocab_path = basedir + '/' + tokenization_transfo_xl.VOCAB_NAME
    vocab_url = tokenization_transfo_xl.PRETRAINED_VOCAB_ARCHIVE_MAP['transfo-xl-wt103']
    urllib.request.urlretrieve(vocab_url, basedir + '/vocab.buggy.bin')
    vocab = torch.load(basedir + '/vocab.buggy.bin')

    vocab['counter'] = collections.Counter({fix(k): v for k, v in vocab['counter'].items()})
    vocab['sym2idx'] = {fix(k): v for k, v in vocab['sym2idx'].items()}
    vocab['idx2sym'] = [fix(k) for k in vocab['idx2sym']]
    torch.save(vocab, good_vocab_path)


if __name__ == "__main__":
    main()
```

Maybe indeed. Do you want to submit a PR to fix this?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
padded positions are ignored when embedding position ids
Author: guxd
Pull request: no
Labels: name: Discussion

name: wontfix

 When embedding position ids, all positions are considered. 
```
seq_length = input_ids.size(1)
position_ids = torch.arange(seq_length, dtype=torch.long, device=input_ids.device)
```
This is different from most transformer implementations. 
Should it be
```
position_ids = np.array([
        [pos_i+1 if w_i != PAD else 0
         for pos_i, w_i in enumerate(seq)] for seq in batch_seq])
```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
run_lm_finetuning
Author: Shi-Linqing-Jason
Pull request: no
Labels: name: Need more information

name: wontfix

When I run_lm_finetuning with the exemplary training corpus (small_wiki_sentence_corpus.txt), I printed the tr_loss every 20 steps. I found that the tr_loss increases very fast. I wonder what the reason is.
![image](https://user-images.githubusercontent.com/40857896/53220384-35b0f400-369f-11e9-8902-27f97a1adacc.png)
![image](https://user-images.githubusercontent.com/40857896/53220402-4c574b00-369f-11e9-899b-c7962f28cde0.png)



Comments:
What tr_loss are you exactly printing here? Is it possible that you just print this one here? https://github.com/huggingface/pytorch-pretrained-BERT/blob/2152bfeae82439600dc5b5deab057a3c4331c62d/examples/run_lm_finetuning.py#L600 If yes, you should divide it by the number of training steps (nb_tr_steps) first to get your average train loss. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Tokenizer defaults lowercase even when bert_model is cased
Author: chmccreery
Pull request: no
Labels: name: wontfix

https://github.com/huggingface/pytorch-pretrained-BERT/blob/2152bfeae82439600dc5b5deab057a3c4331c62d/pytorch_pretrained_bert/tokenization.py#L77

A more clear behavior would be to use whether or not 'uncased' is in bert_model, and set the default behavior of do_lower_case accordingly.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Unable to train (fine-tuning) BERT with small training set
Author: shuvadibp
Pull request: no
Labels: name: Need more information

name: wontfix

I am trying to train BERT with 1 context and 1 answer in the train.json, I am getting the below error.
_lr_this_step = args.learning_rate * warmup_linear(global_step/t_total, args.warmup_proportion)
ZeroDivisionError: division by zero_
After training with 1 context and 5 answers, the error is avoided, but I do not see any change in the answer obtained from BERT. Please help regarding this and let me know if anyone has tried this fine tuning training.


Comments:
Probably an issue with `t_total` and the number of training optimization steps similarly to #329.
Could you check the number of total training step sent to the optimizer? Which example script are you using?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
run_lm_finetuning - ZeroDivisionError
Author: naga-dsalgo
Pull request: no
Labels: name: Need more information

name: wontfix

Trying to get run_lm_finetunning example on working on below GPU machine but finding getting zeroDivisonError.Any idea what could be causing this error?

![image](https://user-images.githubusercontent.com/47925301/53542425-6c966800-3aec-11e9-9681-87432bcd6aee.png)


python /home/ec2-user/SageMaker/bert_pytorch/pytorch-pretrained-BERT/examples/run_lm_finetuning.py   --bert_model bert-base-uncased   --do_train   --train_file /home/ec2-user/SageMaker/bert_pytorch/pytorch-pretrained-BERT/samples/sample_text.txt   --output_dir /home/ec2-user/SageMaker/bert_pytorch/pytorch-pretrained-BERT/models   --num_train_epochs 5.0   --learning_rate 3e-5   --train_batch_size 32   --max_seq_length 32



![image](https://user-images.githubusercontent.com/47925301/53542336-01e52c80-3aec-11e9-9966-beecb8755d01.png)



Comments:
Seems like an error with `t_total`. `t_total` is the number of training optimization steps of the optimizer defined [here](num_train_optimization_steps) in the `run_lm_finetuning` example.
Can you make sure it's not zero?

Your `batch_size` of 32 is too big for such a small `train_file`, i.e. sample_text.txt. Try setting `batch_size` to 16 or send in a larger train file with more text. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
pretrain for chinese dataset
Author: Jason-kid
Pull request: no

Hi , I want to pretrain my model for chinese dataset . can i use my own vocab.txt ? 
and what is the format for vocab.txt ? thanks a lot

Comments:
Hi, you should probably turn to the TensorFlow version for pre-training.
This package is mostly intended to be used for fine-tuning pre-trained models.
Another option for BERT-like pre-training is to use Facebook's [XLM](https://github.com/facebookresearch/XLM)
ADD ERNIE
Author: nghuyong
Pull request: no
Labels: name: Help wanted

Can we add a new model ERNIE?
ERNIE is based on the Bert model and has better performance on Chinese NLP tasks.
Github address: https://github.com/PaddlePaddle/LARK/tree/develop/ERNIE
paper:  https://arxiv.org/abs/1904.09223
Thanks

Comments:
Hi @nghuyong, I won't convert ERNIE but I'm open to welcome a PR if somebody want to give it a try.

Also, note that unlike examples, a PR with a new model should have a configuration class, tests, a conversion script and be documented like the other models in the library.


I do implement that converting ERNIE to huggingface's format
The address is https://github.com/nghuyong/ERNIE-Pytorch
Welcome to use and open issue if have problems
[Question] Best choice for Sentence Compression model? 
Author: Hellisotherpeople
Pull request: no
Labels: name: Discussion

name: wontfix

I'm trying to develop a model that will do "word level extractive summarization" e.g. that it will delete unimportant words or tokens and summarize a document. This is also known as "Sentence Compression" in the NLP community. 

I'm thinking using the BertforTokenClassification module. Will it work with a large dataset, or must it all fit in my VRAM at once? 

In my case, I also have access to a human made abstractive summary of each document - I was wondering if I could do contextual Sentence Compression by having a model do sequence to sequence conversion between the Abstractive Summary and the original document, and extract all words with the highest attention scores. Anyone know if this is a good idea or not? 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Fix for computing t_total in examples
Author: lukovnikov
Pull request: yes

Examples had wrongly computed t_total, resulting in warning messages (Issue #556 )

Added fixes in several examples but:
- only tested MRPC in `run_classifier.py` so far
- `finetune_on_pregenerated.py` still needs fixing (not sure why lines 221-227 are as they are)

Comments:
Closed in favour of #604
Fix ineffective no_decay bug when using BERTAdam
Author: xiaoda99
Pull request: yes

With the original code, all parameters are decayed because the condition "parameter_name in no_decay" will never be satisfied.

Comments:
thanks!

Question - wouldn't `.named_parameters()` for the model return a tuple `(name, param_tensor)`, where name looks similar to these
```
['bert.embeddings.word_embeddings.weight',
 'bert.embeddings.position_embeddings.weight',
 'bert.embeddings.token_type_embeddings.weight',
 'bert.embeddings.LayerNorm.weight',
 'bert.embeddings.LayerNorm.bias',
 'bert.encoder.layer.0.attention.self.query.weight',
 'bert.encoder.layer.0.attention.self.query.bias',
 'bert.encoder.layer.0.attention.self.key.weight',
 'bert.encoder.layer.0.attention.self.key.bias',
 'bert.encoder.layer.0.attention.self.value.weight',
 'bert.encoder.layer.0.attention.self.value.bias',
 'bert.encoder.layer.0.attention.output.dense.weight',
 'bert.encoder.layer.0.attention.output.dense.bias',
 'bert.encoder.layer.0.attention.output.LayerNorm.weight',
 'bert.encoder.layer.0.attention.output.LayerNorm.bias',
...
...
'classifier.linear.weight',
'classifier.linear.bias']
``` 
therefore requiring slightly smarter conditions than just `in`? Something along the lines?
```
[p for n, p in param_optimizer if any(True for x in no_decay if n.endswith(x))]
```

Don't mind my comment, tested it further this morning and everything seems to work as expected!
t_total
Author: zhangatao
Pull request: no

Traceback (most recent call last):
  File "finetune_on_pregenerated.py", line 333, in <module>
    main()
  File "finetune_on_pregenerated.py", line 321, in main
    optimizer.step()
  File "/home/py36/lib/python3.6/site-packages/pytorch_pretrained_bert/optimization.py", line 290, in step
    lr_scheduled *= group['schedule'].get_lr(state['step'])
  File "/home/py36/lib/python3.6/site-packages/pytorch_pretrained_bert/optimization.py", line 61, in get_lr
    progress = float(step) / self.t_total
ZeroDivisionError: float division by zero

Excuse me, what is the cause of this situation, why is t_total equal to 0? ? ?

Comments:

I found the reason. When the data is relatively small, this happens. After I added the data, it is normal now.
Separator token for custom QA input (multi paragraph, longer than 512)
Author: bugtig
Pull request: no
Labels: name: Discussion

name: wontfix

Hello!
I'm trying to extract features for a QA task where the document is composed of multiple disparate paragraphs. So my input is:
question ||| document
where document is {para1 SEP para2 SEP para3 SEP}, so overall, it's something like:
question ||| para1 SEP para2 SEP para3 SEP

My question is: Is it okay to use the default BERT [SEP] token for the paragraph separation token as above? Or should I use something like the NULL token instead, or simply remove the paragraph separation token completely?

Secondly, my input is longer than 512, so I'm thinking of doing sliding windows like:
question ||| doc[:512]
question ||| doc[256:768]
and so on, finally merging the overlaps by averaging. Would this be correct?

Thanks!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Little training has no impact
Author: ghost
Pull request: no
Labels: name: Help wanted

name: Need more information

name: wontfix

When tried to enter few training data in trainxx.json (few questions and few answers) and ran the training, then new pytorch_model.bin file got generated ( = uncased + squad training + few my questions).  
 However, when same question was put in devxx.json the answer is not same which was put in training.   

Why is there no positive impact of training?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Ranking predictions with BertForQuestionAnswering
Author: gqoew
Pull request: no
Labels: name: Discussion

name: wontfix

I am using `BertForQuestionAnswering`

I am trying to make a prediction from the same question asked on different paragraphs. It outputs an `OrderedDict` of tuples with format `(paragraphID, answer)`. How can I rank those predictions to get the most probable answer across all paragraphs?

Thanks for great repo!

Comments:
I am also interested in this, it looks like we would have to append the prediction probability to the `all_predictions` JSON output.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Loss function of run_classifier.py takes in 2 inputs of different dimensions. 
Author: datduong
Pull request: no

I am having an error here https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_classifier.py#L836

In this line, `loss = loss_fct(logits.view(-1, num_labels), label_ids.view(-1))`, suppose we have 2 labels (entailment vs. not_entailment like QNLI task), then, 

`logits` is already in dimension batch_size x num_labels. Using `logits.view(-1, num_labels)` will convert logits into an array of length (2 times batch_size) x 1. 

So, this logits does not match `label_ids.view(-1)` which is batch_size x 1. 

Does anyone else see this error when running the code ? 

Thanks. 





Comments:
Closing issue, because I pass in the num_labels as 1 instead of 2 for the QNLI task. I was thinking that giving 1 label is enough because the 2nd label can be inferred from the 1st one.
BERT accuracy reduced after providing custom training..The answer is also not correct
Author: shuvadibp
Pull request: no
Labels: name: Help wanted

name: Need more information

name: wontfix

I have trained Google BERT with a custom training.
I have included the exact question and answer along with the context from the input document in the training file and trained BERT.

With new generated checkpoints (ckpt) I am still getting the same wrong answer as obtained before training. However it is observed the probability returned is reduced this time, in nbest_predictions.json.

Comments:
Can you give a simple self-contained script to reproduce your issue?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Vocabularly file not available for Squad predictions
Author: beyondbeneath
Pull request: no
Labels: name: Help wanted

name: wontfix

There appears to be a bug in the way the vocabulary file is handled.

For example, if we execute [`run_squad.py`](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_squad.py) with `--do_train`, and set the `--output_dir` to `/tmp/debug_squad/`, we successfully build a model and the resulting model files (`bert_config.json` and `pytorch_model.bin`) are saved in the appropriate directory.

Then we execute [`run_squad.py`](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_squad.py) with `--do_predict`, and this time set `--bert_model` to `/tmp/debug_squad`, which according to [`modelling.py`](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling.py) is all that is required (see the [`from_pretrained`](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling.py#L532) method).

However, this raises a bug as the tokenizer cannot load a vocabulary file. If we inspect [`tokenizer.py`](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/tokenization.py#L138) we can see what happens: if you are using a pre-trained BERT model, it will look for the vocab file at a specific URL. If you input a directory as your `bert_model`, it assumes you have a file `vocab.txt` (`VOCAB_NAME`) in that same directory. It also appears to check the cache which may or may not be present.

We were able to fix this by simply downloading the appropriate vocab file for our base BERT model (e.g., `'bert-base-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt"`), renaming it to `vocab.txt`, and placing it in `/tmp/debug_squad`, however it feels as though this should be better handled by the train/predict pipeline.

Comments:
Indeed, this example could be improved. I would happy to welcome a PR on that.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.