noamaanMulla-03/transformers-issues huggingface dataset:
Error running openai-gpt on ROCstories
Author: roholazandie
Pull request: no

## 🐛 Bug

<!-- Important information -->

The model I am using: OpenAIGPT

The language I am using the model on: English

The problem arises when using:
* [ ] the official example scripts: When I try to run examples/single_model_script/run_openai_gpt.py I get this error:

```
Traceback (most recent call last):
  File "/home/rohola/Codes/Python/pytorch-transformers/examples/single_model_scripts/run_openai_gpt.py", line 288, in <module>
    main()
  File "/home/rohola/Codes/Python/pytorch-transformers/examples/single_model_scripts/run_openai_gpt.py", line 158, in main
    model = OpenAIGPTDoubleHeadsModel.from_pretrained(args.model_name, num_special_tokens=len(special_tokens))
  File "/home/rohola/Codes/Python/pytorch-transformers/pytorch_transformers/modeling_utils.py", line 330, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
TypeError: __init__() got an unexpected keyword argument 'num_special_tokens'
```


The tasks I am working on is:
*  ROCstories

## To Reproduce

Steps to reproduce the behavior:

1. Just run the "run_openai_gpt.py "


## Environment

* OS: Ubuntu 16.04
* Python version: 3.6
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): The last commit
* Using GPU: True
* Distributed of parallel setup: No
* Any other relevant information:

## Additional context

Even when I remove that argument I get another error:
```
Traceback (most recent call last):
  File "/home/rohola/Codes/Python/pytorch-transformers/examples/single_model_scripts/run_openai_gpt.py", line 288, in <module>
    main()
  File "/home/rohola/Codes/Python/pytorch-transformers/examples/single_model_scripts/run_openai_gpt.py", line 224, in main
    losses = model(input_ids, mc_token_ids, lm_labels, mc_labels)
  File "/home/rohola/Codes/Python/pytorch-transformers/env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/rohola/Codes/Python/pytorch-transformers/pytorch_transformers/modeling_openai.py", line 601, in forward
    head_mask=head_mask)
  File "/home/rohola/Codes/Python/pytorch-transformers/env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/rohola/Codes/Python/pytorch-transformers/pytorch_transformers/modeling_openai.py", line 425, in forward
    hidden_states = inputs_embeds + position_embeds + token_type_embeds
RuntimeError: The size of tensor a (78) must match the size of tensor b (16) at non-singleton dimension 1
```

Comments:
Ok should be fixed now on master with e768f23
start_position=0 in utils_squad.py when span is impossible
Author: YeDeming
Pull request: no

Hi, 

https://github.com/huggingface/pytorch-transformers/blob/e768f2322abd2a2f60a3a6d64a6a94c2d957fe89/examples/utils_squad.py#L340-L351

when answer is out of span, start_position should be cls_index rather than 0 as L350


And in

https://github.com/huggingface/pytorch-transformers/blob/e768f2322abd2a2f60a3a6d64a6a94c2d957fe89/examples/run_squad.py#L253-L259

when using multi gpu and set evaluate_during_training=True, we may add 
`model_tmp = model.module if hasattr(model, 'module') else model` in order to get `model_tmp.config. start_n_top` rather than `model.config.start_n_top`

Comments:
How can I load a fine-tuned model? 
Author: daz261
Pull request: no
Labels: name: wontfix

I finetuned a new model by running pregenerate_training_data.py and finetune_on_pregenerated.py and the output is saved as pytorch_model.bin.

How do I load the model to run the regular run_classfier,py predictions? To which files do I have to add code?

Comments:
you can use the path to the folder containing your fine-tuned model as `--bert_model`.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
BERT returns different embedding for same sentence
Author: rshah1990
Pull request: no

I am using pre-trained BERT for creating features, for same sentence it produces different result in two different runs. Do we have to set some random state to produce consistent result? I am using pytorch-transformers for reading pre-trained model.


Comments:
Are you initializing from a pretrained model? If no, than this is normal behaviour: your weights are randomly initialized. If yes, make sure your model is in evaluation mode (```model.eval()```), this disables dropout and other random modules.

Thank you for quick response @srslynow . how to initialize weights and biases fora pre-trained model ? I thought weights and biases freezes after training.

you were right , I missed the model.eval() that's the reason I was getting slightly different embedding on each run becz of dropout layer.
fix annotation
Author: Maxpa1n
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1298?src=pr&el=h1) Report
> Merging [#1298](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1298?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/9f995b99d4c4067662c3bd4f1274315c0839deeb?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1298/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1298?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1298   +/-   ##
=======================================
  Coverage   80.77%   80.77%           
=======================================
  Files          57       57           
  Lines        8092     8092           
=======================================
  Hits         6536     6536           
  Misses       1556     1556
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1298?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/tokenization\_xlnet.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1298/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX3hsbmV0LnB5) | `89.18% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1298?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1298?src=pr&el=footer). Last update [9f995b9...51decd5](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1298?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks!
bert-large config file
Author: desperadoola
Pull request: no
Labels: name: wontfix

Here is the config file I download from path in modelling for bert large, 
{
  "attention_probs_dropout_prob": 0.1, 
  "directionality": "bidi", 
  "hidden_act": "gelu", 
  "hidden_dropout_prob": 0.1, 
  "hidden_size": 1024, 
  "initializer_range": 0.02, 
  "intermediate_size": 4096, 
  "max_position_embeddings": 512, 
  "num_attention_heads": 16, 
  "num_hidden_layers": 24, 
  "pooler_fc_size": 768, 
  "pooler_num_attention_heads": 12, 
  "pooler_num_fc_layers": 3, 
  "pooler_size_per_head": 128, 
  "pooler_type": "first_token_transform", 
  "type_vocab_size": 2, 
  "vocab_size": 28996
}

I am wondering what are the following params for? I can't find them in the modelling file and the checkpoint I download.

"pooler_fc_size": 768, 
"pooler_num_attention_heads": 12, 
"pooler_num_fc_layers": 3, 
"pooler_size_per_head": 128, 
"pooler_type": "first_token_transform", 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
how to correctly do classifying?
Author: junchen14
Pull request: no
Labels: name: Help wanted

name: Need more information

name: Discussion

name: wontfix

when I do the classifying with task cola, the result is like this?
eval_loss = 0.0
global_step = 49173
loss = 0.0
mcc = 0.0

what is my prediction result?
and how should I use the output model????


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I'm having a similar issue where I run a BERT tutorial  by Chris McCormick and receive 65-78% accuracy on training data and 0.0% for test data.  Was anyone able to diagnose a result of 0.0%?  One theory I have is a division by zero error for very small numbers that are rounded to zero...
No language embedding weights in pre-trained xlm models. 
Author: BalazsHoranyi
Pull request: no

I'm trying to train a one-shot classification model using the given XLM pre-trained weights.  
However, I noticed that for both `xlm-mlm-17-1280` and `xlm-mlm-100-1280`, That I kept receiving the warning `weights of XLMForSequenceClassification not initialized from pre-trained model: ['lang_embeddings.weight']`. I then looked into the state_dict of those two checkpoints and saw that indeed there were no weights that matched that key. Should that weight exist in the state_dict somewhere?

Comments:
Indeed, I've checked and the 100 and 17 language models don't use language indices.
Just supply `langs=None`.
You can see that in the official notebook from Facebook: https://github.com/facebookresearch/XLM/blob/master/generate-embeddings.ipynb

I see. I missed this issue here which explains it pretty well. https://github.com/huggingface/pytorch-transformers/issues/1034
Thanks for the help!
Bertology example is probably broken
Author: davidefiocco
Pull request: no
Labels: name: wontfix

Hello!

I tried to run `run_bertology.py` in the example dir calling it with 

```
export TASK_NAME=CoLA
python ./run_bertology.py     --data_dir $GLUE_DIR/$TASK_NAME     
                              --model_name bert-base-uncased     
                              --task_name $TASK_NAME     
                              --max_seq_length 128     
                              --output_dir ./tmp/$TASK_NAME/     
                              --try_masking 
                              --metric_name mcc
```

But it fails with 

> Traceback (most recent call last):
>   File "./run_bertology.py", line 346, in <module>
>     main()
>   File "./run_bertology.py", line 327, in main
>     eval_data = load_and_cache_examples(args, args.task_name, tokenizer, evaluate=True)
>   File "/data/home/dfiocco/BERT/run_glue.py", line 245, in load_and_cache_examples
>     list(filter(None, args.model_name_or_path.split('/'))).pop(),
> AttributeError: 'Namespace' object has no attribute 'model_name_or_path'

One fix to that problem should be (?) replacing all occurrences of `model_name` with `model_name_or_path` in `run_bertology.py`. Still, even with that "patch" running the code gives

> Traceback (most recent call last):
>   File "./run_bertology.py", line 346, in <module>
>     main()
>   File "./run_bertology.py", line 341, in main
>     head_mask = mask_heads(args, model, eval_dataloader)
>   File "./run_bertology.py", line 175, in mask_heads
>     print_2d_tensor(head_mask)
> UnboundLocalError: local variable 'head_mask' referenced before assignment

Trying another task (MRPC) I get instead

> 
> Traceback (most recent call last):
>   File "./run_bertology.py", line 346, in <module>
>     main()
>   File "./run_bertology.py", line 341, in main
>     head_mask = mask_heads(args, model, eval_dataloader)
>   File "./run_bertology.py", line 169, in mask_heads
>     _, head_importance, preds, labels = compute_heads_importance(args, model, eval_dataloader, compute_entropy=False, head_mask=new_head_mask)
>   File "./run_bertology.py", line 97, in compute_heads_importance
>     head_importance += head_mask.grad.abs().detach()
> AttributeError: 'NoneType' object has no attribute 'abs'

Did anybody manage to run the Bertology example without hiccups?

 


Comments:
Yes, this example is still work in progress. Hopefully, I can finish it before ACL (but not sure).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Order of tokens in vocabulary of German model
Author: schoennenbeck
Pull request: no
Labels: name: wontfix

The vocabulary for the German model ('bert-base-german-cased') has the token '[unused3001]' at position 0 (and the '[PAD]' token at position 1). However, the BertEmbedding has padding_idx=0 as usual.

Is this behaviour intended and if so would it be possible to  get some insight into the rationale behind it?


Comments:
@tholor and @timoeller may have some insights on these

Hey Sebastian, thanks for using the German Bert and digging into its details. The mysterious [unused3001] token was actually a special comma symbol to get rid of [UNK] tokens in some of our training texts. But we covered it up later on in the process + didn't anticipate it would be coming back to us : ) 

So agreed, it is unwanted behaviour. 

Though TL;DR, we don't believe it is impacting either pretraining or downstream task training. 



Apparently the token at index 0 (= [unused3001]) is used as padding token in TF Bert and pytorch Bert and the implementations do not really care if it is called [unused3001] [PAD] or [something].

To be a bit more intuitive we now swapped [unused3001] and [PAD] in the vocab files (pytorch and TF) only. 
Might be that future code somehow substitutes "[PAD]" input strings, which could cause problems.

The only thing that seems worrisome to us is that the embedding values for this padding token are non-zero (and change over the course of training) for our German Bert but also for Googles open sourced models. 

I tried to check how the padding embedding is handled in TF but am not familiar with debugging there...

Maybe you want to dig more into it and raise an issue in the original TF Bert repro? Maybe this closed issue could be related to a rather unwanted padding embedding handling: https://github.com/google-research/bert/issues/113

Hope that helps, good luck!

Thanks a lot for the input @Timoeller (not quite sure who Christian is, though ;) ). I also got the feeling that it doesn't really impact downstream applications (NER in this case). At least not heavily.

I'll do some more experiments and raise an issue with the original repo if it feel it is warranted. 

Thanks again and all the best,
Sebastian



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Answers to Bullet/List Items by bert
Author: ghost
Pull request: no
Labels: name: wontfix

Hi,
There are lists in document (Bullet items), and I am running BERT (non trained as well as squad trained). But seems BERT does not understands Bullets/lines starting with number or star.  
Will any text preprocessing help?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to use Bert QA model for predictions?
Author: Swathygsb
Pull request: no
Labels: name: wontfix

Hi,
Can you give sample codes for how to use Bert QA model for predicting an answer given a text corpus and a question?

Comments:
You can write your own code like the prediction phase [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/78462aad6113d50063d8251e27dbaadb7f44fbf0/examples/run_squad.py#L345) 

@Swathygsb have you figured it out? I have the same use case as you and I'm struggling to understand the source code. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Delete n_special reference in docstring
Author: sshleifer
Pull request: yes

I don't think the `n_special` param is used, even in `**kwargs`.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1294?src=pr&el=h1) Report
> Merging [#1294](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1294?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/0d1dad6d5323cf627cb8d7ddd428856ab8475f6b?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1294/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1294?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1294   +/-   ##
=======================================
  Coverage   80.77%   80.77%           
=======================================
  Files          57       57           
  Lines        8092     8092           
=======================================
  Hits         6536     6536           
  Misses       1556     1556
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1294?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/configuration\_openai.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1294/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvY29uZmlndXJhdGlvbl9vcGVuYWkucHk=) | `89.13% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1294?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1294?src=pr&el=footer). Last update [0d1dad6...119610b](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1294?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Indeed, thanks Sam
Sudden catastrophic classification output during NER training
Author: fabiocapsouza
Pull request: no

Hi,
I am fine-tuning BERT model (based on `BertForTokenClassification`) to a NER task with 9 labels ("O" + BILU tags for 2 classes) and sometimes during training I run into this odd behavior: a network with 99% accuracy that is showing a converging trend suddenly shifts all of its predictions to a single class. This happens during the interval of a single epoch.

Below are the confusion matrices and some other metrics one epoch before the event and after the event:
```
Epoch 7/10: 150.57s/it, val_acc=99.718% (53391/53542), val_acc_bilu=87.568% (162/185), val_rec=98.780%, val_prec=55.862%, val_f1=71.366%
Confusion matrix:
 [[53229     2    66    25     2    25     8]
 [    0     7     0     7     0     0     0]
 [    0     0    14     0     0     0     0]
 [    0     0     0    67     0     0     1]
 [    1     0     0     3    11     0     1]
 [    1     0     1     0     0    14     0]
 [    0     0     0     7     1     0    49]]

Epoch 8/10: 150.64s/it, val_acc=0.030% (16/53542), val_acc_bilu=8.649% (16/185), val_rec=100.000%, val_prec=0.030%, val_f1=0.060%
Confusion matrix:
 [[    0     0     0     0 53357     0     0]
 [    0     0     0     0    14     0     0]
 [    0     0     0     0    14     0     0]
 [    0     0     0     0    68     0     0]
 [    0     0     0     0    16     0     0]
 [    0     0     0     0    16     0     0]
 [    0     0     0     0    57     0     0]]
```
I am using the default configs for `bert-base-multilingual-cased` and standard `CrossEntropyLoss`. The optimizer is `BertAdam` untouched with learning rate 1e-5. The dataset is highly unbalanced (very few named entities, so >99% of the tokens are "O" tags), so I use a weight of 0.01 to the "O" tag in CE.

Has anyone faced a similar issue?

Thanks in advance

Comments:
I manage to solve this problem. There is an issue in the calculation of the total optimization steps in `run_squad.py` example that results in a negative learning rate because of the `warmup_linear` schedule. This happens because `t_total` is calculated based on `len(train_examples)` instead of `len(train_features)`. That may not be a problem for datasets with short sentences, but, for long sentences, one example may generate many entries in `train_features` due to the strategy of dividing an example in `DocSpan's`.

@fabiocapsouza I am trying to handle text classification but my dataset is also highly unbalanced. I am trying to find where I can adjust the class weights when training transformers. Which parameter you changed in your case?

@MendesSP , since the provided BERT model classes have the loss function hardcoded in the `forward`method, I had to write a subclass to override the `CrossEntropyLoss` definition passing a `weight` tensor.
from pytorch-pretrained-bert to pytorch-transformers，some problem
Author: SStarLib
Pull request: no
Labels: name: wontfix

TypeError: forward() got an unexpected keyword argument 'output_all_encoded_layers'


Comments:
now you should use:
```
model = BertModel.from_pretrained('bert-base-cased', output_hidden_states=True)
outputs = model(input_ids)
all_hidden_states = outputs[-1]
```
Note that the first element in `all_hidden_states` (`all_hidden_states[0]`) is the output of the embedding layers (hence the fact that there is `num_layers + 1` elements in `all_hidden_states`).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Is there any plan of developing softmax-weight function for using 12 hidden BERT layer? 
Author: izuna385
Pull request: no
Labels: name: wontfix

Thanks for developing very nice/useful library.

My question is about using 12/(or in large model, more) hidden layer.
First, Does how to use hidden layers depend on the downstream task?
(Say, concat, average, only final layer, only mean of top 4 layer, etc...)

For using all layer, I think it's good to use softmax weight. During training , hidden layer's feature is fix but weight is learned for the task. So second question is, Is there any plan of developing softmax-weight function for using 12 hidden BERT layer? 

Thanks

Comments:
Yes we might add a module for scalar mixture of layers like the one of AllenNLP, for instance (https://github.com/allenai/allennlp/blob/master/allennlp/modules/scalar_mix.py).

I'm looking forward to see that in also pytorch-transformer.
Again, thanks! I'll keep track on this repository.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
16 GB dataset for finetuning fail on reduce_memory 
Author: g-jing
Pull request: no
Labels: name: wontfix

Hi, I am using 16GB dataset to finetune bert Model. When I do not use reduce_memory, which is loading dataset into memory first, it will use all of my 120GB memory and then crush because of out of memory. Now I am using reduce_memory model, with the increase of loading lines, the memory use is still increasing. But it is much slower in reduce_memory setting. So, I am thinking whether it would crush at the end. Could anyone have an answer for that ?

Sorry for the bothering, but it is too slow to run with reduce_memory setting, I have no idea weather it would crush or go well. I am afraid that it would keep loading for days and crushed finally. Thanks in advance for any suggestions

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.