noamaanMulla-03/transformers-issues huggingface dataset:
Format problem when training DistilBert
Author: alexandrabenamar
Pull request: no

## Format problem when training DistilBert

Hello,

I'm trying to train DistilBert from scratch on French language with the official "trainin with distillation task" script.

## To Reproduce

Steps to reproduce the behavior:
The problem arise when I invoke the script : 
https://github.com/huggingface/transformers/blob/master/examples/distillation/distiller.py
With the command line : 
```bash
python train.py --student_type distilbert --student_config training_configs/distilbert-base-uncased.json \
    --teacher_type bert --teacher_name bert-base-uncased --mlm --dump_path train_model/my_first_training --data_file data/binarized_text.bert-base-multilingual-cased.pickle \
    --token_counts data/token_counts.bert-base-uncased.pickle --force --n_gpu 1
```

I did not modify the script in any way, and I get the error : 

```bash
Traceback (most recent call last):
  File "train.py", line 286, in <module>
    main()
  File "train.py", line 281, in main
    distiller.train()
  File "/dds/work/distil/transformers/examples/distillation/distiller.py", line 335, in train
    token_ids, attn_mask, lm_labels = self.prepare_batch_mlm(batch=batch)
  File "/dds/work/distil/transformers/examples/distillation/distiller.py", line 227, in prepare_batch_mlm
    token_ids = token_ids.masked_scatter(pred_mask, _token_ids)
RuntimeError: Expected object of scalar type Byte but got scalar type Bool for argument #2 'mask'
```

## Environment

* OS: Windows
* Python version: 3.6
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): 2.1.1
* Using 1 GPU

Could you help me resolve this ?


Comments:
Hi, I believe that `torch.bool` was introduced in PyTorch 1.2.0. Do you think you could try to upgrade it to 1.2.0 to try out the distillation scripts?

Problem fixed, the problem was the PyTorch version as you said, thank you so much! :)
bert-base-multilingual-uncased vocabulary not consecutive
Author: ntubertchen
Pull request: no
Labels: name: wontfix

## 🐛 Bug

When I was checking out bert-base-multilingual-uncased vocabulary. I receive the warning "Saving vocabulary to ./vocab.txt: vocabulary indices are not consecutive. Please check that the vocabulary is not corrupted"

I ran the similar command on two different machine and got the same warning.

from pytorch_transformers import *
tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-uncased',do_lower_case=True)
tokenizer.save_vocabulary('./')

I ran it on 

* OS:
* Python version: python3.5 
* PyTorch version: pytorch1.0.1.post2
* PyTorch Transformers version (or branch): 1.0
* Using GPU ? Yes
* Distributed of parallel setup ?no

Comments:
H! Could you please specify on which OS you have this error? I cannot reproduce this on MacOS 10.15, nor on Ubuntu 18.04 with both Python 3.5 and 3.6.

Hi, @LysandreJik 

I tried these on Ubuntu 16.04.6 LTS.

@ntubertchen
Just in case helpful for you -- I had exactly the same issue with release 1.0 of pytorch-transformer, when I worked with multilingual BERT base models.  (Ubuntu 19.04, training MRPC model with run_glue.py script.) Always at the end of training (e.g. MRPC), it gave me the above warning. And the eval results were quite strange. (e.g. often much lower than expected). Due to this, and also Chinese model broken issue; I installed master branch as of now (commits after Roberta models added), and the error went away. no more such warnings, and stable result. 

Maybe you should try to install from current master (which will install locally built 1.1); it helped me. I think some code has been changed on handling multilingual vocab file. 

Thanks @gilnoh. @ntubertchen can you let me know if you still have the same problem with the current (1.1.0) release ?

Yes, we had an issue in Bert tokenizer that made it lose a token in the Chinese vocabulary.
This is fixed now with the merge of #860.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Initialising XLMTokenizer 
Author: hiyingnn
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
To initialise the XLMTokenizer, both the vocab file and the merges.txt file are needed. If I am pre-training XLM, how do I obtain the merges.txt file? 


Comments:
To pretrain XLM, you should use the original (PyTorch) codebase and training scripts which are [here](https://github.com/facebookresearch/XLM)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
wrong generation of training sentence pairs. method: get_corpus_line, in finetune_on_pregenerated.py
Author: Evgeneus
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using:
BERT

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [ ] the official example scripts: (give details)

The tasks I am working on is:
I am running on my own text corpus the official example to fine-tune BERT

Steps to reproduce the behavior:

1. create my_corpus.txt:
AAA
BBB

CCC
DDD

EEE
FFF
2. run python3 simple_lm_finetuning.py 
--train_corpus my_corpus.txt 
--bert_model bert-base-uncased 
--do_lower_case 
--output_dir finetuned_lm/
--do_train

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior
I expected to see the following first 3 outputs of get_corpus_line method, ie, t1 t2:
1)
t1=AAA
t2=BBB
2)
t1=CCC
t2=DDD
3)
t1=EEE
t2=FFF

But received:
1)
t1=AAA
t2=BBB
2)
t1=CCC
t2=DDD
3) **!!!!!!! (HERE)
t1=DDD
t2=AAA


## Additional context 

It seems we need to make self.line_buffer equal to None whenever we close the file. 
Possible silution:
line 118:
if cur_id != 0 and (cur_id % len(self) == 0):
    self.file.close()
    self.file = open(self.corpus_path, "r", encoding=self.encoding)
    ***self.line_buffer = None***
<!-- Add any other context about the problem here. -->

Comments:
Hi @Evgeneus, does your proposed solution solve the problem on your side?

> Hi @Evgeneus, does your proposed solution solve the problem on your side?

Hi @thomwolf, seems yes.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
`run_squad.py` not using the dev cache
Author: VictorSanh
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert (but it's independent of the model)

Language I am using the model on (English, Chinese....): English (but it's independent of the language)

The problem arise when using:
* [ X] the official example scripts: `examples/run_squad.py`

The tasks I am working on is:
* [ X] an official GLUE/SQUaD task: (give the name)

It's not really a bug, more an unnecessary repetition of some operations.

It seems like the dev set is binarized (tokenized + tokens_to_id) for every single evaluation of a checkpoint even if the binarized data are already cached (from the previous evaluation for instance). It is particularly striking when adding the flag `--eval_all_checkpoints`.

It arises when calling `dataset, examples, features = load_and_cache_examples(args, tokenizer, evaluate=True, output_examples=True)` in the `evaluate` function. The cache is never used because of the argument `output_examples=True`:

```python
if os.path.exists(cached_features_file) and not args.overwrite_cache and not output_examples:
    logger.info("Loading features from cached file %s", cached_features_file)
    features = torch.load(cached_features_file)
```
From my understanding, except if the tokenizer changes between two checkpoints (which is not the case), the computed features are always the same.

The command I use:

```bash
python -m torch.distributed.launch --nproc_per_node=8 ./examples/run_squad.py \
    --model_type bert \
    --model_name_or_path bert-base-uncased \
    --do_train \
    --do_eval \
    --do_lower_case \
    --train_file $SQUAD_DIR/train-v1.1.json \
    --predict_file $SQUAD_DIR/dev-v1.1.json \
    --learning_rate 3e-5 \
    --num_train_epochs 2 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --output_dir ../models/bert-base-uncased_finetuned_squad/ \
    --per_gpu_eval_batch_size=3   \
    --per_gpu_train_batch_size=3   \
    --eval_all_checkpoints 
```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
GPT2Tokenizer for Hindi Data
Author: DEBADRIBASAK
Pull request: no
Labels: name: wontfix

I was trying to fine-tune GPT2LMHeadModel with Hindi data corpus. It is performing well. But when I looked at the tokens that are generated from the GPT2Tokenizer, I saw that they are containing tokens of almost character level. I am not understanding how is this kind of encoding handling Hindi data, or any form of non Roman script correctly. Can anyone explain the working of GPT2Tokenizer from this aspect?

Comments:
I might be wrong, but I think GPT2Tokenizer uses byte pair encoding, a form of subword-level encoding. On an intuitive level, this is a between character-level and word level, and akin to breaking the word apart by syllable (in reality it's breaking the word apart by the highest frequency patterns). I know some people use Sentence-Piece tokenization for working with Chinese in BERT, so it might be worthwhile to see if there's a similar effort for a Sentence-Piece GPT-2

@DEBADRIBASAK 
Can you share the steps for fine-tuning on hindi dataset? Thanks.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Why DistilBertTokenizer and BertTokenizer are creating different number of features?? 
Author: vamshi975
Pull request: no

Hi, 

I tried working with DistilBertTokenizer and BertTokenizer  from transformers. And according to documentation DistilBertTokenizer was identical to the BertTokenizer . But while creating features for a particular Dataset it creates different number of examples. Why? I also tried using distilbert model with BertTokenizer but still it doesnot work.

could you please explain me this ?? or How can i get same number of features ??



Comments:
How to split consecutive numbers?
Author: ZacBi
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

&emsp;&emsp; In some NER datasets of BIO(ES) format, each number of a consecutive number string labeled with a corresponding tag, e.g., "All Jhon need is only 10 yuan" will be labeled as "O, PER, O, O, O, O, O, O". In this case, "10" is labeled as "O, O". But in **BertTokenizer** and **PreTrainedTokenizer**, I don't find(or know?)  effective params to deal with this situation.
<!-- A clear and concise description of the question. -->

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
ALBERT
Author: duyvuleo
Pull request: no

# 🌟New model addition

## Model description

ALBERT is "A Lite" version of BERT, a popular unsupervised language representation learning algorithm. ALBERT uses parameter-reduction techniques that allow for large-scale configurations, overcome previous memory limitations, and achieve better behavior with respect to model degradation.

For a technical description of the algorithm, see our paper:

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut

## Open Source status

* [ ] the model implementation is available: https://github.com/google-research/google-research/tree/master/albert.

I just want to ask whether you guys have plan to add ALBERT in near future. 


Comments:
Merging with #1370
distilroberta-base unavailable in pip install transformers
Author: duyvuleo
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hi,

Would you please update the `pip install transformers` with the addition of `distilroberta-base`? 

As of 28 Nov 2019, I tried `pip install transformers` or `pip install --upgrade transformers` but the `distilroberta-base` model is not available. But I can see it from the master branch and if I install it from source, it will work btw. 

Thanks!


Comments:
Yes, we should push a new pip release this coming week. In the meantime please use master.
Fix for mlm evaluation in run_lm_finetuning.py
Author: altsoph
Pull request: yes

No masking is done in the original evaluation code, so the resulting perplexity is always something like 1.0. In this PR a simple fix is proposed, using just the same masked scheme as in a training code.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1633?src=pr&el=h1) Report
> Merging [#1633](https://codecov.io/gh/huggingface/transformers/pull/1633?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ae1d03fc51bb22ed59517ee6f92c560417fdb049?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1633/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1633?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master   #1633   +/-   ##
======================================
  Coverage    85.9%   85.9%           
======================================
  Files          91      91           
  Lines       13653   13653           
======================================
  Hits        11728   11728           
  Misses       1925    1925
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1633?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1633?src=pr&el=footer). Last update [ae1d03f...a9b7ec4](https://codecov.io/gh/huggingface/transformers/pull/1633?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This is great, thanks!
Getting tokenization ERROR while running run_generation.py 
Author: dxganta
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (GPT-2....):

Language I am using the model on (English):

The problem arise when using:
* [ ] the official example scripts: (give details)
pytorch-transformers/examples/run_generation.py \


The tasks I am working on is:
* [ ] my own task or dataset: (give details)
-> just simple next sentence prediction. My actual text 'Saw her in the park yesterday'

## To Reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
Error message I got is
ERROR - pytorch_transformers.tokenization_utils -   Using sep_token, but it is not set yet.

And then the next sentence that it predicts has nothing to do with my given sentence.


## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
-> Not give the error. and work like it is supposed to,

## Environment

* OS: Google Colab
* Python version:
* PyTorch version:
* PyTorch Transformers version (or branch):
* Using GPU ? Yes
* Distributed of parallel setup ?
* Any other relevant information:

## Additional context

I am pretty sure that the problem is not so hard to solve. But I am a noob here. So please forgive me .

Comments:
Hi! Yes currently there's a small issue with the tokenizer that outputs this warning during the decoding of the sentence. It will be fixed very shortly. 

It won't affect your training however, as it is only a warning :)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Xlnet for multi-label classification
Author: ghaith-khlifi
Pull request: no
Labels: name: wontfix

Can you provide me with the xlnet code to deal with the multi-label classification task, please

Comments:
you can try fast-bert.  https://github.com/kaushaltrivedi/fast-bert.

its built on top of pytorch-transformers and supports multi-label classification for both BERT and XLNet.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
❓ Why in `run_squad.py` using XLNet, CLS token is not set at the end ?
Author: astariul
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

[This line](https://github.com/huggingface/pytorch-transformers/blob/e00b4ff1de0591d5093407b16e665e5c86028f04/examples/run_squad.py#L292)  of the file `run_squad.py` create the features for the dataset.

No matter which model is used (BERT or XLNet), the function will create the format :

> CLS A SEP B SEP

But for XLNet case, we want :

> A SEP B SEP CLS

---

**Isn' it wrong ? Did I miss something ?**

Comments:
Humm I think you are right.

The SquAD example looks a bit broken in pytorch-transformers, we will have to review it @LysandreJik.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Output of BertModel does not match the last hidden layer from fixed feature vectors
Author: sasaadi
Pull request: no
Labels: name: wontfix

Based on BERT documentation (https://github.com/google-research/bert#using-bert-to-extract-fixed-feature-vectors-like-elmo) we can extract the contextualized token embeddings of each hidden layer separately. However, when I extract the last hidden layer (layer -1), it does not match the `outputs[0]` from `pytorch_transformers.BertModel()` as described here: https://huggingface.co/pytorch-transformers/model_doc/bert.html#bertmodel

Just to remind that I am using the same pre-trained model (e.g. `bert-base-uncased`) and the same input (e.g. 'here is an example .') for both.

Comments:
What is your exact command to _extract the last hidden layer (layer -1)_?
And what is your exact command to get _the outputs[0] from pytorch_transformers.BertModel()_ ?

To extract the last hidden layer (layer -1) from BERT, I run the `extract_features.py` as follows:
`python extract_features.py   --input_file=tmp/input.txt   --output_file=tmp/output.json   --vocab_file=cased_L-12_H-768_A-12/vocab.txt   --bert_config_file=cased_L-12_H-768_A-12/bert_config.json   --init_checkpoint=cased_L-12_H-768_A-12/bert_model.ckpt   --layers=-1   --max_seq_length=128   --batch_size=1`

where the `input_file` contains only one line e.g. 'here is an example .'
The output gives me the -1 hidden layer of each token separately.

To get the embeddings from the `outputs[0]`:

```
config = BertConfig.from_pretrained('bert-base-cased')
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
model = BertModel(config)
input_ids = torch.tensor(tokenizer.encode("here is an example .")).unsqueeze(0)  # Batch size 1
outputs = model(input_ids)
last_hidden_states = outputs[0]
```
where `last_hidden_states` gives me a list of embeddings. I presume one for each token in the sentence in the same order they appear in the sentence.

Thanks




Help me, i having same problem, how to extract feature from tuned .bin file, in bert's original doc, only init ckpt checkpoint used  

@sasaadi, you should load the pretrained model with `model = BertModel.from_pretrained('bert-base-cased')`. In your example only the config (a dict of hyper-parameters) is loaded from the pretrained model, not the weights. 

@thomwolf pytorch_transformers.BertModel.from_pretrained('bert-base-multilingual-cased', state_dict=model_state_dict)
Is this solution when you load from tuned model ?

@hungph-dev-ict to load from a fine-tuned checkpoint you reference it directly: `BertModel.from_pretrained('/path/to/finetuned/model')`.

The doc for the method referenced by @LysandreJik is [here](https://huggingface.co/pytorch-transformers/main_classes/model.html#pytorch_transformers.PreTrainedModel.from_pretrained)

@LysandreJik @thomwolf thank you very much.
Now this library has just added RoBERTa, I want tune it with my corpus, do you have any solution ?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Allowing PR#1455 to be merged in the master 
Author: ghost
Pull request: no

Hi Thomas
Remi was saying in PR:#1455 it has the bert seq2seq ready, could you move in a gradual way please and allow this PR to be merged at this stage that is working for BERT? Then people can use the BERT one, this is already great, then after a while when this is ready for also other encoders, you can add them later, I really appreciate adding the BERT ones thanks

Comments:
Please don't post issues like this. I'm sure the maintainers work as hard as they can. Asking them to _work faster_ doesn't help. In fact, adding these kind of non-issues only distract the maintainers from actually working on the actual issues at hand. Please close this question.
Moving model from GPU -> CPU doesn't work
Author: ranamihir
Pull request: no

## 🐛 Bug

Hi,

I tried creating a model (doesn't matter which one from my experiments), moving it first to multiple GPUs and then back to CPU. But I think it doesn't work as intended.

The following is the code to reproduce the error:

```python
import torch
import torch.nn as nn
from transformers import BertTokenizer, BertModel

>>> model = BertModel.from_pretrained('bert-base-uncased')
>>> model.to('cuda:0')
>>> model = nn.DataParallel(model, device_ids=range(torch.cuda.device_count()))
>>> print(model.device_ids)
[0, 1]
>>> model.to('cpu')
>>> print(model.device_ids) # Still on GPUs
[0, 1]
```

Comments:
`.to()` is not an in-place operation. You should use `model = model.to('cpu')`. If that doesn't work, it might be that you need to access the module as part of the DataParallel object, like this:

```python
model = model.module.to('cpu')
```

Ahh gotcha. Thanks for the quick reply!