noamaanMulla-03/transformers-issues huggingface dataset:
tf.keras.mixed_precision.experimental.Policy
Author: ydshieh
Pull request: no

## ❓ Questions & Help

I want to use `mixed_precision`, and I found [tf.keras.mixed_precision.experimental.Policy](https://www.tensorflow.org/api_docs/python/tf/keras/mixed_precision/experimental/Policy).

So I put `tf.keras.mixed_precision.experimental.set_policy("mixed_float16")` before `TFBertModel.from_pretrained(pretrained_weights)`. When I run the code, I got the following error:

> InvalidArgumentError: cannot compute AddV2 as input #1(zero-based) was expected to be a half tensor but is a float tensor [Op:AddV2] name: tf_bert_model_1/bert/embeddings/add/

which happened at `ret = model(model.dummy_inputs, training=False)  # build the network with dummy inputs`.

I am not sure if I used it correctly. I think `tf.keras.mixed_precision.experimental.set_policy` is supposed to be used before constructing / build the model, as the tf page says `Policies can be passed to the 'dtype' argument of layer constructors, or a global policy can be set with 'tf.keras.mixed_precision.experimental.set_policy'`.

I wonder if I can use AMP with tf based transformer models and how. Thanks.

[error.txt](https://github.com/huggingface/transformers/files/3907032/error.txt)


Comments:
Sorry, I created this duplicated issue as the previous one. Please delete this one, thank you.
how to get distilbert-base-uncased-distilled-squad?
Author: RyanHuangNLP
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I think it is initialize a six layer bert and distill with a 12 layer bert, then save the checkpoint file, is it right?

Comments:
Hello @RyanHuangNLP,
I am not sure to get your question.
Do you mean that you want to play with the model (do inferences)? If so, did you try `qa_model = DistilBertForQuestionAnswering.from_pretrained('distilbert-base-uncased-distilled-squad')`?

@VictorSanh I wonder how to get the 'distilbert-base-uncased-distilled-squad' pretrain model, just use the first six layer of the base one or initialize a six layer bert?

If you do:
```
qa_model = DistilBertForQuestionAnswering.from_pretrained('distilbert-base-uncased-distilled-squad')
```
You will get the 'distilbert-base-uncased-distilled-squad' pretrain model.
Nothing more to do.

I am sorry not clearly expressed, my question is how to get the 'distilbert-base-uncased-distilled-squad' pretrain model,  I know that use that code can get the six layer layer, but how to train the pretrain model is what I concern, there is no six layer bert release

Ok, I understand your question now @RyanHuangNLP.
It is finetuned from `distilbert-base-uncased`. More precisely, the model we release is finetuned AND distilled at the same time: the loss is computed from the classic qa loss (see `run_squad.py` and `DistilBertForQuestionAnswering`) plus the distillation supervision from a BERT SQuAD model (second loss).
We haven't released the script for doing that (and we plan to do it in the near future) but it is a simple adaptation of `run_squad.py` (mostly adding a second loss i.e. distillation).

do we have any release date of that run_squad_adapted.py @VictorSanh ? :D

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
modeling_xlnet.py lines 798  torch.eisum('i,d->id', pos_seq, inv_freq)
Author: lcy081099
Pull request: no
Labels: name: wontfix

hi  I run position embedding in modeling_xlnet.py , but it not work  , why not torch.eisum('i,d->id', [pos_seq, inv_freq])  ?i use pytorch 0.4.1

Comments:
I don't understand your question.

Can you give more details and point to the exact code lines you are referring to?

You cannot provide specific position indices to XLNet if that's what you are trying to do. You have to use the built-in relative embeddings.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


For new PyTorch 1.0, the syntax should be `torch.eisum('i,d->id', pos_seq, inv_freq) `

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Dataset format and Best Practices For Language Model Fine-tuning
Author: HanGuo97
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hi, thanks for making this code base available!

I have two questions, one on the input format of for fine-tuning the language model on custom dataset, and one on (unreasonably-)long data preprocessing time. Thanks in advance for any help!

- I'm trying to fine-tune the BERT Model on an extra dataset, and am using the `run_lm_finetuning.py` script in the `examples/` directory. However, I'm having trouble locating instructions on the proper format of the input data. There used to be some instructions in the `examples/lm_finetuning/` directory, but they seem deprecated now.

- As a start, I followed the `run_lm_finetuning.py` example and changed nothing but `--train_data_file` argument with a bigger text file (arbitrary format). The training, however, hangs on the data preprocessing part for about 10 hours, and the last standard output is shown below.

```
pytorch_transformers.tokenization_utils -   Token indices sequence length is longer than the specified maximum sequence length for this model (164229992 > 512). Running this sequence through the model will result in indexing errors
```

Comments:
I am facing the same issue as there is no proper format available for defining the train and test dataset. 
As usual, I use .csv file in a format of columns with (UID, Text, and Labels). But according to the Wiki.txt its more of arbitrary format.

Any help would be appreciated.

I'm having the same issue. I think it's counting the total length of the tokenized corpus not only the tokenized document length. I tried to run the wiki raw files as mentioned in the read me and still get this warning of total tokenized corpus length.

I tried to the following formats with no success:
1. sentence per line with a blank line in between docs
2. document per line with a blank line in between docs

Update:
After looking at the code again it looks like even though this warning is showing the sequence length being longer than 512 it is still chunking the corpus into 512 tokens and training it that way. This raises the question of whether it is problematic to just separate the corpus based on token length alone especially that BERT for example is training on predicting next sentence. What happens to the probably recurring case of the data being chunked mid-way the sentence?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
RoBERTa : add_special_tokens=True
Author: HongyanJiao
Pull request: no
Labels: name: wontfix

I set add_special_tokens=True
but I still get:
A sequence with no special tokens has been passed to the RoBERTa model. This model requires special tokens in order to work. Please specify add_special_tokens=True in your encoding.

Comments:
I'm getting the same warning, also here: https://github.com/huggingface/pytorch-transformers/issues/1318

I think you should add < s >  without spaces before as well as after sentences.

Please share a self contained script exhibiting the behavior and allthe information on the python/pytorch/pytorch-transformers versions you are using.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to use GPT-2 text generator in spanish
Author: erdos-ml
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I would like to know if there is a way to use the gpt-xl model for text generation in spanish.

The command I use to run the english text generation model is the following:

$ python ./examples/run_generation.py     --model_type=gpt2     --length=50     --model_name_or_path=gpt2-xl

What other parameter must I use to allow spanish text generation?


Comments:
At the moment, there is **no pre-trained model in Spanish language**. If you want, you can use a **multi-lingual** pre-trained model, such as BERT or XLM. In particular, Transformers offer the following settings of multi-lingual models:
- **bert-base-multilingual-cased** (Masked language modeling + Next sentence prediction, 104 languages)
- **bert-base-multilingual-uncased** (Masked language modeling + Next sentence prediction, 102 languages)
- **xlm-mlm-17-1280** (Masked language modeling, 17 languages)
- **xlm-mlm-100-1280** (Masked language modeling, 100 languages)

You can find more information in the  [official documentation](https://huggingface.co/transformers/multilingual.html). 

Thank you. I will close the issue.
How to add weighted CrossEntropy loss in sequence classification task?
Author: TinaChen95
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I am doing a sequence classification task base on run_ner.py. My dataset is an imbalenced dataset and apperantly 'O' label appears far more than other labels. I'm wondering if it is helpful to use weighted CrossEntropy loss.
If the answer is 'yes', how to add weight to the loss function? Currently I hard code the weight in BertForSequenceClassification class (in modeling_bert.py). I feel it is not a smart way to do it. Do you have some advice on how to add the weight infomation?
Many thanks.

Comments:
Hi, no specific advice other than computing the loss your-self outside of the model's forward method.
Is it okay to define and use new model by using only a part of full GPT model blocks? Or anyone tried to do so?
Author: duzani
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi,

As far as I know, the smallest GPT2 model that comes with pretrained weight has 768 hidden size with 12 transformer blocks.

However, I'm considering to define 'smaller' GPT model with only 1 or 2 transformer blocks and load those weight from corresponding blocks in the pretrained GPT weight.

This is basically because the original model size is too big to fit in my gpu environment.

so I'm wondering if it will work or not. I think it would be at least better than start training from randomly initialized weight, but not sure about it.

So I'd like to ask how you think. 
Thanks!

Comments:
### Hi, 
As far as I remember from my experiments with GPT2, the GPT2 blocks will work on its own. What I mean with that is that **the model will produce reasonable outputs, even if you leave out some layers from the original model.** So your idea should work. 
What I'm not quite sure about is the performance of an ONLY 1-2 layers model. I would guess the model will produce valid sentences, however the quality might be not optimal. 

### Another Tip: 
If you want to build this model, load the original GPT2 model from the lib and delete the layers you don't want manually. This is a relatively ugly solution but its very time efficient ). Also you dont have to define you own forward function etc.

Did you take a look at `distilgpt2`? It has 6-layer, 768-hidden, 12-heads, 82M parameters.

cc @VictorSanh @LysandreJik 

Hi! Indeed, if you're looking for a smaller GPT-2 model then DistilGPT-2 has half of the small GPT-2 layers. However, if you want to create your own GPT-2 model with an arbitrary number of layers, you can do so very easily with the configuration file:

```py
from transformers import GPT2Model, GPT2Config

config = GPT2Config(n_layer=2)
model = GPT2Model(config)
```

This will define a model with only two layers. Please be aware that the weights will be randomly initialized if you go down this route, whereas the weights will already be trained if you use DistilGPT-2.

Thank you guys! Really helped me a lot. I changed the GPT2Config as @LysandreJik told me and loaded pretrained weight to those layers :)
What is the real parameters to weight the triple loss (L_{ce}, L_{mlm}, L_{cos}) in DistilBert?
Author: voidism
Pull request: no

Hello! Thanks for your great work DistilBert. I want to ask what is the real parameters "alpha" you used in DistilBert to weight the triple loss (L_{ce}, L_{mlm}, L_{cos})?

You did not mention this detail in your NIPS workshop paper (http://arxiv.org/abs/1910.01108). In the [README](https://github.com/huggingface/transformers/blob/master/examples/distillation/README.md) file, you listed two different setups: `--alpha_ce 5.0 --alpha_mlm 2.0 --alpha_cos 1.0 --alpha_clm 0.0` for single GPU training and `--alpha_ce 0.33 --alpha_mlm 0.33 --alpha_cos 0.33 --alpha_clm 0.0` for distributed training. Can you tell me what is the best setting?

Actually, I have tried to reproduce your results of DistilBert. I trained the DistilBert with the corpus used by BERT, but the performance of GLUE seemed slightly fall behind your pre-trained `distilbert-base-uncased` by 2 points. I would be appreciated if you can tell me the parameters for reproducibility. Thanks!


Comments:
Hello @voidism,
Thank you for your interest!
The parameters we used for training DistilBERT are the first one you listed: `--alpha_ce 5.0 --alpha_mlm 2.0 --alpha_cos 1.0 --alpha_clm 0.0`.
Victor

@VictorSanh Thank you very much!
Why is the vocabulary of token_type_ids and input_ids shared? 
Author: ZeweiChu
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

https://github.com/huggingface/transformers/blob/17ea43cf985829634bd86b36b44e5410c6f83e36/transformers/modeling_gpt2.py#L421

In GPT2Model, forward method, it seems the vocabulary of token_type_ids and input_ids is shared. I checked the vocabulary table, 0 and 1 corresponds to the exclamation sign and the quote sign. What is the reason of sharing the vocabulary? Is it on purpose? 


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
add special tokens
Author: ghost
Pull request: no

Hello
I tried to add special tokens to bert tokenizer via add_special_tokens: 
```
tokenizer.add_special_tokens({'additional_special_tokens':['SS']})
```
But I got CUDA error
```
CUDA error: device-side assert triggered
```
The code runs without adding additional_special_tokens!
Any idea? 

Comments:
Do you load everything (model, data) on GPU?

> Hello
> I tried to add special tokens to bert tokenizer via add_special_tokens:
> 
> ```
> tokenizer.add_special_tokens({'additional_special_tokens':['SS']})
> ```
> 
> But I got CUDA error
> 
> ```
> CUDA error: device-side assert triggered
> ```
> 
> The code runs without adding additional_special_tokens!
> Any idea?

Yes, I did. The code runs without this line : 
```
tokenizer.add_special_tokens({'additional_special_tokens':['SS']})
```
Do you think it is a resourcse issue? 
Thanks 

I got this lately: 
```
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1370         ret = torch.addmm(bias, input, weight.t())
   1371     else:
-> 1372         output = input.matmul(weight.t())
   1373         if bias is not None:
   1374             output += bias

RuntimeError: cublas runtime error : resource allocation failed at /pytorch/aten/src/THC/THCGeneral.cpp:216
```

Are you changing the #vocab count in the config as well?


No, I did not. 

you can check them with``` tokenizer.convert_ids_to_tokens(id)``` (I don’t remember exactly, but I think, from 100 to 1000 are free, maybe, from 5 to 1000 even free, crosscheck please, also it depends on the "case" of the model)
Generally there's a pack in the beginning and then somewhere in the between we have these free unused tokens..!

Try this,

```
### Let's load a model and tokenizer
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

### Do some stuff to our model and tokenizer
# Ex: add new tokens to the vocabulary and embeddings of our model
tokenizer.add_tokens(['[SPECIAL_TOKEN_1]', '[SPECIAL_TOKEN_2]'])
model.resize_token_embeddings(len(tokenizer))
# Train our model
train(model)

### Now let's save our model and tokenizer to a directory
model.save_pretrained('./my_saved_model_directory/')
tokenizer.save_pretrained('./my_saved_model_directory/')

### Reload the model and the tokenizer
model = BertForSequenceClassification.from_pretrained('./my_saved_model_directory/')
tokenizer = BertTokenizer.from_pretrained('./my_saved_model_directory/')
```

Thank you, 
I missed this line. Silly mistake 
```
model.resize_token_embeddings(len(tokenizer))
```
it worked!

You can close the issue :)
[CamemBERT] Potential error in the docs
Author: manueltonneau
Pull request: no

Thanks for the great work on this repo! As I was going through the details about the available pre-trained models (https://huggingface.co/transformers/v2.2.0/pretrained_models.html), I spotted what I think is an error in the description of camembert-base (12-layer, 768-hidden, 12-heads, 110M parameters; CamemBERT using the BERT-base architecture). Isn't it RoBERTa-based?


Comments:
RoBERTa and BERT (and CamemBERT) share mostly the same model architecture. Most of the differences lie in:
- the tokenizers
- the pre-training method

cc @LysandreJik 

Cool, thanks for the reply! :)
Ram utilisation of DistilBERT
Author: 008karan
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I was checking the memory consumption of RoBERTa and DistilBERT. I found there is no significant change in memory usage. Although Inference time is around 1sec for DistilBERT and for RoBERTa is 2sec.
Memory usage on CPU:
Port 9000: DistilBERT
Port 9002: RoBERTa
![compute](https://user-images.githubusercontent.com/18630864/65751708-e8ab6980-e128-11e9-93a7-937cc0211009.png)

Have you guys seen any significant change in memory usage or am I missing something here?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
run_lm_finetuning.py script CLM inputs and labels preparing
Author: AliOsm
Pull request: no

I'm trying to finetune the GPT-2 on my own dataset, while I'm reading the code in `run_lm_finetuning.py` script, I found a weird thing in line `227`. When the script preparing CLM batch inputs and labels, it gives the model the same `batch` variable as inputs and labels:

```
inputs, labels = mask_tokens(batch, tokenizer, args) if args.mlm else (batch, batch)
```

In this way, the model will learn to take the token as an input and predict it directly, right?

Can anyone explain what happens?

Comments:
Ok, in [modeling_gpt2.py](https://github.com/huggingface/transformers/blob/master/transformers/modeling_gpt2.py) file I found this comment in line `495`:

```
Note that the labels **are shifted** inside the model, i.e. you can set ``lm_labels = input_ids``
```

So, the model takes care of the shifting process.
Use PyTorch's GELU activation
Author: BramVanroy
Pull request: no
Labels: name: wontfix

## 🚀 Feature

PyTorch 1.2 provides a built-in, GPU-accelerated GELU function at `torch.nn.functional.gelu`. Reading through the merged pull request (https://github.com/pytorch/pytorch/pull/20665) it seems that this is optimised for CUDA, too. Therefore I would propose trying to import the built-in gelu function first, and use the back-off gelu definition if it's not found for torch < 1.2.

## Additional context

I started _very_ basic changes over at https://github.com/BramVanroy/transformers/tree/pytorch_gelu by changing the gelu definition in e.g. BERT to something like

```python
def gelu(x):
    """ Original Implementation of the gelu activation function in Google Bert repo when initialy created.
        For information: OpenAI GPT's gelu is slightly different (and gives slightly different results):
        0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
        Also see https://arxiv.org/abs/1606.08415
    """
    return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))

ACT2FN = {"relu": torch.nn.functional.relu, "swish": swish, "gelu_new": gelu_new}

try:
    ACT2FN["gelu"] = torch.nn.functional.gelu
except AttributeError:
    ACT2FN["gelu"] = gelu
```

However, I wonder whether it wouldn't be cleaner to have all activation functions in an importable constant `ACT2FN` somewhere. Maybe under `modeling_utils`? This should make it easier to keep a good overview of all activation functions that can be used. If requested, I can put some time in refactoring this.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Where I could find the vocab.json for XLNet
Author: kugwzk
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I notice in tokenizer_xlnet.py there is not the vocab,json only spiece model. So I want to know where I could find the vocab.json? And what I should rename the file ?

Comments:
If you want to have `xlnet_config.json`, which is a JSON file which specifies the hyper-parameters of the XLNet model, you can download the .zip file from [here](https://github.com/zihangdai/xlnet/blob/5cd50bc451436e188a8e7fea15358d5a8c916b72/README.md) which contains the pre-trained weights of XLNet model.

> ##  Questions & Help
> I notice in tokenizer_xlnet.py there is not the vocab,json only spiece model. So I want to know where I could find the vocab.json? And what I should rename the file ?

I downloaded the pre-trained model, the config file and the sentence piece model, but when I run the code I found the vocab_size = -1. Did I miss something?

An example of the content of `xlnet_config.json` is the following:
```
{
    "d_head": 64, 
    "d_inner": 3072, 
    "d_model": 768, 
    "ff_activation": "gelu", 
    "n_head": 12, 
    "n_layer": 12, 
    "n_token": 32000, 
    "untie_r": true
}
```

> When i run the code I found the vocab_size=-1

Which code you are talking about?


You can either download from the S3 repo or the script is supposed to automatically download the vocab file. Make sure you have a working internet connection.
wwm-bert lm_finetune 
Author: yangDDDD
Pull request: no
Labels: name: wontfix

## 🚀 Feature
in run_lm_finetuning.py
present how to finetune language model with dataset

## Motivation

But there isn't option to finetune whole word masking bert models
I suggest to add it

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.