noamaanMulla-03/transformers-issues huggingface dataset:
Brackets are not aligned in the DocString of Bert.
Author: jiaxin96
Pull request: no

The Brackets in the file https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_bert.py#L606 are not aligned, which will cause some highlight mistakes in some editers (i.e. VSCODE).
it should be fixed as : [0, config.max_position_embeddings - 1]

Comments:
You're right, will fix! cc @LysandreJik
Issue: Possibly wrong documentation about labels in BERT classifier
Author: domaala
Pull request: no

Possibly also elsewhere, but when discussing the proper format of labels for BERT classification, the documentation states the following:

https://github.com/huggingface/pytorch-transformers/blob/44dd941efb602433b7edc29612cbdd0a03bf14dc/pytorch_transformers/modeling_bert.py#L935

However, shouldn't it be `[0, ..., config.num_labels - 1]`? After all, [`CrossEntropyLoss`](https://pytorch.org/docs/stable/nn.html#crossentropyloss) is being used here. 

Comments:
Indeed. @LysandreJik I think it should be `Indices should be in ``[0, ..., config.num_labels-1]`` for classification or torch.floats for regression`, what do you think?
missing 1 required positional argument: 'num_classes' in 'from_pretrained'
Author: desireevl
Pull request: no

I am running a [multiclass BERT classification](https://github.com/desireevl/Bert-Multi-Label-Text-Classification/blob/master/train_bert_multi_label.py) model and am receiving the following error:

`
Traceback (most recent call last):
  File "train_bert_multi_label.py", line 144, in <module>
    main()
  File "train_bert_multi_label.py", line 78, in main
    num_classes = len(id2label))
  File "/opt/miniconda3/envs/tempenv/lib/python3.6/site-packages/pytorch_transformers/modeling_utils.py", line 403, in from_pretrained
    model = cls(config)
TypeError: __init__() missing 1 required positional argument: 'num_classes'
`

The script worked fine when using `pytorch-pretrained-bert` and am guessing there is an issue from the new release. 

Thanks for the great tools :)

Comments:
Yes, @xanlsh is working on an update to reduce the effect of this breaking change in #866.

You should be able to keep your script unchanged.

@desireevl Since #866 has been merged, your code should work now

Thanks!
Fine-tuning Bert
Author: goyalsaransh97
Pull request: no
Labels: name: wontfix

I want to fine-tune Bert's LM for a specific corpora. I converted the test into the format specified in the documentation and ran the fine-tuning codes given. I'm getting the following error:
File "simple_lm_finetuning.py", line 156, in random_sent
    assert len(t2) > 0
AssertionError
I'm getting similar error in pregenerate_.... script. What could be the reason for the same? Is it due to some possible OOV words? My corpus does contain some emoticons.
Thanks in advance

Comments:
I was facing the same issue when finetuning using `finetune_on_pregenerated.py`. The problem was in the fact that I have some empty sentences in my dataset. Also there are some special characters, like `\t` (tabulation) which can make a mess and should be cleared. 
I preprocess the text like this:
```
for_train = for_train.dropna(subset=['text'])

import re
with open('for_pretraining_full.txt', "w", encoding='utf-8') as writer:
    for doc in for_train['text'].tolist():
        doc = doc.replace(u'\xa0', u' ').replace(u'\u200b', u' ').replace(u'\u206f', u' ').replace(u'\u206e', u' ').replace(u'\u206b', u' ').replace(u'\u206c', u' ').replace(u'\u2063', u' ').replace(u'\u200d', u' ').strip() # replace some special unicode chars
        doc = re.sub('\t+', '', doc) # replace tabs
        doc = doc.replace('. ', '\n')
        doc = re.sub('\n+( )*(\n+)*', '\n', doc) # replace several consecutive new lines by a single one
        doc = doc.strip()
        if (doc != ''):
            writer.write(doc)
            writer.write('\n\n')
```

Also you can try to find problem sentances, debuging your `simple_lm_finetuning.py`.

I ran into an issue on this where some sneaky \n were still sneaking through even with the above code. I remedied this by just doing 

`if "\n" in doc:

    doc = re.sub('\n', ' ', doc)`

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Is XLA feature existed in current repo?
Author: geekan
Pull request: no

## ❓ Questions & Help

I find https://news.developer.nvidia.com/nvidia-achieves-4x-speedup-on-bert-neural-network/ says tensorflow XLA has higher speed on bert, however, the pull request in this repo it mentioned https://github.com/huggingface/pytorch-pretrained-BERT/pull/116 didn't implement something like XLA. Is the XLA feature already exist?

Comments:
Hi, there is no XLA/TPU support with the current library. Maybe in a future release!
Training with wrong GPU count
Author: hairzooc
Pull request: no

Hi,
Thank you for your repo :)

I'm fine-tuning with 4 GPU (run_squad, bert model)
And I found that gpu count is wrong when to do distributed training.
I've got 1 GPU count and that's caused by source code below
Is there any reason to set n_gpu = 1 when to do distributed training?

 if args.local_rank == -1 or args.no_cuda:
        device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu")
        n_gpu = torch.cuda.device_count()
    else:
        torch.cuda.set_device(args.local_rank)
        device = torch.device("cuda", args.local_rank)
        n_gpu = 1 <= this!!!
        # Initializes the distributed backend which will take care of sychronizing nodes/GPUs
        torch.distributed.init_process_group(backend='nccl')


Comments:
Yes, this is expected behavior. Each script in distributed training has ownership over one GPU only.

You can read this blog post for details on parallel and distributed training: https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255
How to use the pretrain script with only token classification task ?
Author: g-jing
Pull request: no

Hi, I need to train on my own twitter corpus, but most of the twitter contains only one sentence. Therefore I can not use the sentence prediction task to train the model. Will the script automatically use only token classification task when there is no next sentence ? Thanks in advance.

Comments:
No, you probably need to adapt the example script to your exact task.
BertForNextSentencePrediction labels
Author: ArthurCamara
Pull request: no

Hi everyone!
I was reading trough the documentation, and, according to https://huggingface.co/pytorch-transformers/model_doc/bert.html#bertforsequenceclassification, it expects that `next_sentence_label` is `1` if B is **not** a next sequence for A, and `0` if B **is** a sequence for B. 
That's somewhat counterintuitive, since most of the datasets (I believe) will assume this problem to be a binary classification. 

Is my assumption correct? Should I flip my dataset before fine-tuning the model?

Comments:
You should supply your own labels when using the `BertForSequenceClassification` class (`labels` input to the forward method). You can choose the labels you like.

The `BertForSequenceClassification` class is **not** related to the Next Sentence Classification task used during Bert pretraining. You can use the [BertForNextSentencePrediction](https://huggingface.co/pytorch-transformers/model_doc/bert.html#bertfornextsentenceprediction) class if you want to do next sentence prediction (classification).

Yes, I'm sorry I meant the  [BertForNextSentencePrediction](https://huggingface.co/pytorch-transformers/model_doc/bert.html#bertfornextsentenceprediction) class.

Yes, in this case, you should follow the docstring: `0 indicates sequence B is a continuation of sequence A, 1 indicates sequence B is a random sequence.`
Pretrained GPT2 mdoels does not load unk special symbol
Author: pywirrarika
Pull request: no

## 🐛 Bug

<!-- Important information -->

Im using GPT2 (on pytorch-transformers 1.0.0) using the introductory tutorial but it seems that the tokenizer does not load the unk special symbol from the pretrained dictionary.

`
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2Model.from_pretrained('gpt2')
print(tokenizer.unk_token)

`
The output from the above code is as follows:

`Using unk_token, but it is not set yet.
None`

Is this the expected behavior? 

Thank you!


## Environment

* OS: Linux
* Python version: 3.7.3
* PyTorch version:1.1.0
* PyTorch Transformers version (or branch):1.0.0
* Using GPU ? yes
* Distributed of parallel setup ? no
* Any other relevant information: - 



Comments:
Hi! GPT-2 does not have an unknown token because of its byte-level BPE. This is a warning so it should not affect your code, but maybe we should do something about this warning for models that do not have unknown tokens. cc @thomwolf.

However, it seems that having a defined _unk_ symbol is necessary to run other methods, like 
`def add_tokens(self, new_tokens)`
If the unk_token is set to None, add_special_tokens() breaks calling add_tokens() because the None type that is returned from convert_tokens_to_ids.

Yes, it's already on master if you compile from source and will be in the next (1.1) release (which will likely be released next week).

Thank you very much!
SQuAD 1.1 very low evaluation score when using `--fp16`
Author: knuser
Pull request: no
Labels: name: wontfix

I'm replicating SQuAD 1.1 https://github.com/huggingface/pytorch-pretrained-BERT/tree/v0.6.2#squad on latest release `v0.6.2`.

My setup: 
* GeForce RTX 2080 Ti
   * Driver Version: 418.43
   * CUDA Version: 10.1
* Linux Ubuntu 18.10
* pytorch 1.1.0 (installed via conda: py3.7_cuda10.0.130_cudnn7.5.1_0)
* latest `apex` package

I'm on latest release:
```
$ git status 
HEAD detached at v0.6.2
```
-------------

I'm replicating fine tuning bert on SQuAD 1.1. When executing without `--fp16` I'm getting expected result:
```
$ python evaluate-v1.1.py dev-v1.1.json /tmp/debug_squad/predictions.json
{"exact_match": 81.58940397350993, "f1": 88.6984251786611}
```

Full command:
```
export SQUAD_DIR=squad_11_data

python run_squad.py \
  --bert_model bert-base-uncased \
  --do_train \
  --do_predict \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --train_batch_size 8 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/
```

Same experiment with `--fp16` I'm getting very poor results:
```
$ python evaluate-v1.1.py dev-v1.1.json /tmp/debug_squad_fp16_apex/predictions.json 
{"exact_match": 0.47303689687795647, "f1": 8.678859681492447}
```

Full command:
```
export SQUAD_DIR=squad_11_data

python run_squad.py \
  --bert_model bert-base-uncased \
  --do_train \
  --do_predict \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --train_batch_size 8 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad_fp16/ \
  --fp16
```
I remember that I saw information about gradient overflow several times (sorry, don't have more details I have lost output logs).

How to get decent results when using `--fp16`?

Comments:
I took example code from https://github.com/huggingface/pytorch-pretrained-BERT#fine-tuning-bert-large-on-gpus (which has additional option `--loss_scale 128`). Still getting very low test scores:
```
$ python evaluate-v1.1.py dev-v1.1.json ../output/debug_squad_fp16/predictions.json 
{"exact_match": 0.5771050141911069, "f1": 8.853750220358535}
```

Is there any known bug in current (on `v0.6.2`) PyTorch BERT implementation or is this my setup?

---

**UPDATE**
Probably there is a bug in current stable `v0.6.2` version. When running same command on latest `master` branch I'm getting good results:
```
$ python evaluate-v1.1.py dev-v1.1.json ../output/debug_squad_fp16/predictions.json
{"exact_match": 81.45695364238411, "f1": 88.71433452234619}
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Future attention masking in GPT/GPT-2?
Author: sksq96
Pull request: no

Based on my understanding, [this](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling_gpt2.py#L288) is the place where future attention masking for the causal model happens.

If this is the case

- Why is it called `bias`? 
- Why is this in the core of `Attention` module and not passed as a `attention_mask` parameter similar to BERT?



Comments:
Hi Shubham,
This is a legacy from the original Tensorflow code (https://github.com/openai/finetune-transformer-lm/blob/master/train.py#L64-L69).

Thanks for the link to the original reference.
How can I use only one layer transformer via this repository?
Author: LLLLLLI
Pull request: no

## ❓ Questions & Help
I want to use only one layer transformer on the head of some backbone model. Can I use this repository in a simple way?

Comments:
This repository is especially useful if you're looking to use a pre-trained transformer of the same architecture than that of BERT, GPT, GPT-2, XLM, XLNet or TransfoXL.

If you're looking at using a simple transformer of your own making, how about using the newly released [torch.nn.Transformer](https://pytorch.org/docs/stable/nn.html?highlight=transformer#torch.nn.Transformer)?

Thanks! But since I use pytorch v1.0.1 with cuda8.0, it's not convenient to upgrade to v1.2.0 and use torch.nn.Transformer.

If you're looking to use an existing architecture and modifying a few things (like the number of layers, or embedding size), you can always do so by specifying these values in a config file.

As you were saying you would like to use a one-layer transformer on top of some backbone model, you could create a config and specify `num_hidden_layers = 1` and `num_attention_heads = 1` to have a very simple one-layer single-headed transformer.

The documentation for the `BertConfig` file can be found [here](https://huggingface.co/pytorch-transformers/model_doc/bert.html#bertconfig). Each model has its own configuration file.

If you're looking to build a Transformer from scratch, something you could do is re-use some of our model's logic to create your own transformer. For example if you want to use our Attention for GPT-2, you could always import it like this:

```python
from pytorch_transformers.modeling_gpt2 import Attention
```

You can then re-use it as a part of your code, building your own Transformer architecture.

Hope that helps.

Thanks for your patience! I will try what you told me. It really helps me a lot!
Thanks again.
Can't get attribute 'Corpus' on <module '__main__' from 'convert_transfo_xl_checkpoint_to_pytorch.py'>
Author: Pydataman
Pull request: no

I trained my data with the original transformer_xl  repo, but I use convert_transfo_xl_checkpoint_to_pytorch.py to transfer tf to pytorch,
error occurs：
AttributeError: Can't get attribute 'Corpus' on <module '__main__' from 'convert_transfo_xl_checkpoint_to_pytorch.py'>

to use my data, What code do I want to change?

Comments:
Fine-tuning approach for Bert and GPT2 classifiers
Author: amity137
Pull request: no

## ❓ Questions & Help

Hey folks, when we are fine-tuning BERT or GPT2 model for a classification task via classes like GPT2DoubleHeadsModel or BertForSequenceClassification, what is the recommended fine-tuning strategy? I assume all transformer layers of the base model are unfrozen for fine-tuning. Does this result in catastrophic forgetting in practice? Do people use gradual unfreezing (as in ULMFiT)?

Comments:
AFAIK, BERT doesn't make use of gradual unfreezing. Instead, during fine-tuning all model parameters are trainable. It can result in catastrophic forgetting, if you train it for long enough/ large enough learning rate, which is why we usually fine tune for 1-2 epochs at a low learning rate.

When it comes to doing it yourself, you'll should be able to just tweak the number of epochs/train steps and then find which number gives you the best results. IMO anymore than a couple epochs will result in overfitting/forgetting.

Hope that helps.

https://arxiv.org/pdf/1905.05583.pdf

This issue can be closed.

Thanks @andrewpatterson2018.
GPT2 Tokenizer got an expected argument `skip_special_tokens`
Author: sashank06
Pull request: no

## ❓ Questions & Help

I keep running into this error when trying to use the GPT2 model and GPT2 tokenizer while decoding. 
Keep getting the following error when I run the piece of code below:
``tokenizer.decode(response_ids, skip_special_tokens=True)``
Error:
``TypeError: decode() got an unexpected keyword argument 'skip_special_tokens'``


Comments:
Could you please submit a bug report with the version of the library you're using?

will close this issue. Opened a bug report.
How to use the fine tuned model for classification (CoLa) task?
Author: GeetDsa
Pull request: no
Labels: name: wontfix

How to use the fine-tuned model for classification (CoLa) task?


I do not see the argument `--do_predict`, in `/examples/run_classifier.py`. 

However, `--do_predict` exists in the original implementation of the Bert.

The fine-tuned model is getting saving in the BERT_OUTPUT_DIR as `pytorch_model.bin`, but is there a simple way to reuse it through the command line?

Comments:
Can you please confirm that the "/examples/run_classifier.py" file is indeed an example for simple sentence classification?
It looks like the code here uses the "BertForSequenceClassification" model where the tf model uses the "BertModel" (line 577 here https://github.com/google-research/bert/blob/master/run_classifier.py) - why is it different?

> How to use the fine-tuned model for classification (CoLa) task?
> 
> I do not see the argument `--do_predict`, in `/examples/run_classifier.py`.
> 
> However, `--do_predict` exists in the original implementation of the Bert.
> 
> The fine-tuned model is getting saving in the BERT_OUTPUT_DIR as `pytorch_model.bin`, but is there a simple way to reuse it through the command line?

I got an solution of QNLI task in GLUE.  You can add an arg-parser (--do_predict) and [these lines](https://github.com/weidafeng/NLU2019/blob/master/model/run_classifier.py#L743:#L792),  and  run this command: 
```bash
python ./model/run_classifier.py \
	--task_name QNLI \
	--do_predict \
	--do_lower_case \
	--data_dir ./glue_data/QNLI \
	--bert_model bert-base-uncased \
	--output_dir ./glue_data/QNLI/results     **# path to your trained model**
```
You will get the QNLI.tsv file as you want. Hope this works for you.


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
GPT2 - support data type torch.DoubleTensor for Position embedding
Author: adigoryl
Pull request: no
Labels: name: wontfix

Hi,

Could the team add support for floating data types for the position embedding? Currently, it only allows torch.LongTensor between 0 - config.n_positions - 1 and must be of the same shape as the input. I see this a restriction as one may use floating values between e.g. 0-2 to represent the sequence.

What would be even better, if the team could add build-in position encoding techniques like the one used in Attention Is All You Need paper, where they use cos and sin: 
PE(pos,2i) = sin(pos/10000**(2i/dmodel))
PE(pos,2i+1) = cos(pos/10000**(2i/dmodel))

@thomwolf Could you please share your thoughts with me on that?

Regards,
Adrian.


Comments:
Hi, we won't be able to add these feature as they would render the PyTorch model not compatible with the pretrained model open-sourced by OpenAI.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to use GPT2 to predict and fit a word into an existing sentence?
Author: padmalcom
Pull request: no
Labels: name: wontfix

Hi, I'd like to know if I can use GPT2 to decorate a simple sentence such as "Peter was sad because his sister had eaten all his candy." to get sth like "Tuesday morning the ten years old Peter was sitting in his room and was sad because his mean sister Clara had eaten all his tasty candy with her friends."

Using BERT I can use BertForMaskedLM to insert [MASK] tokens into my simple sentence but results are not very good (a lot of repetitions and words do not really fit in).

Since I heard GPT2 was better for text generation I'd now like to experiment with your fantastic library but I can not really find a starting point how to insert text in an existing text instead of (what all tutorials to) add it to the end of the sentence.

Comments:
You would need an insertion-based transformer model like Google's recent KERMIT (http://arxiv.org/abs/1906.01604). But unfortunately, we currently don't have this model in the library.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Adding new tokens to GPT tokenizer
Author: sashank06
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): GPT2

Language I am using the model on (English, Chinese....): English

I am having saving GPT2Tokenizer when custom new tokens are added to it. Tried out two specific methods.
1. Using  special_mapping dictionary and save_vocabulary and save_pretrained method
2. Using special_mapping list and save_vocabulary and save_pretrained method

## To Reproduce
```SPECIAL_TOKENS = ["<bos>", "<eos>", "PersonA", "PersonB", "<pad>"]
tokenizer_class = GPT2Tokenizer if "gpt2" in args.model_checkpoint else OpenAIGPTTokenizer
tokenizer = tokenizer_class.from_pretrained(args.model_checkpoint,unk_token="unk_token")
tokenizer.add_tokens(SPECIAL_TOKENS)
model.resize_token_embeddings(len(tokenizer))
tokenize.save_vocabulary(filedir)
```
the above method only save the current vocab json without any of the new tokens being added. When save_vocabulary is replaced with save_pretrained(filedir) a new file called special_mappings.json is created with only 3 special tokens `{"bos_token": "
", "eos_token": "
", "unk_token": "unk_token"}`
## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Ubunti
* Python version: 3.6.8
* PyTorch version:1.1
* PyTorch Transformers version (or branch):1
* Using GPU ?Yes

## Additional context
If there is anything wrong with the code, please do let me know.

Comments: