noamaanMulla-03/transformers-issues huggingface dataset:
--seed does not change the fintuning results of the xlnet model
Author: zhaoguangxiang
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using  (XLNet):

Language I am using the model on (English):

The problem arise when using:
* [ ] the official example scripts
gpu=3
seed=1
task=MRPC
bsz=32
learning_rate=5e-5
max_steps=800
warmup_steps=200
save_steps=400
export CUDA_VISIBLE_DEVICES=${gpu}
export GLUE_DIR=/home/zhaoguangxiang/bert/glue_data
python3 ./examples/run_glue.py \
    --model_type xlnet \
    --model_name_or_path xlnet-large-cased \
    --do_train  \
    --do_eval   \
    --task_name=${task}     \
    --data_dir=${GLUE_DIR}/${task}  \
    --output_dir=checkpoint/xl_${task}_seed${seed}/   \
    --max_seq_length=128   \
    --per_gpu_eval_batch_size=${bsz}   \
    --per_gpu_train_batch_size=${bsz}   \
    --gradient_accumulation_steps=1 \
    --max_steps=${max_steps}  \
    --model_name=xlnet-large-cased   \
    --overwrite_output_dir   \
    --overwrite_cache \
    --save_steps ${save_steps} \
    --learning_rate ${learning_rate} \
    --warmup_steps=${warmup_steps}
The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)

## To Reproduce

Steps to reproduce the behavior:

1. change the seed
2. results does not change, and keep acc=0.8774509803921569 for every seed


## Environment

* OS:linux
* Python version:3.6
* PyTorch version: 1.2
* PyTorch Transformers version (or branch): latest
* Using GPU ?  1 * TITAN RTX



Comments:
In the command line you are showing, you should add a `--seed ${seed}` argument to set the seed, otherwise, it will stay the same.

> In the command line you are showing, you should add a --seed ${seed} argument to set the seed, otherwise, it will stay the same.

Sorry, i forgot it.
How can i use bert for finding word embeddings
Author: gkv91
Pull request: no
Labels: name: Discussion

name: wontfix

Hi all,
Can I use pre-trained BERT for finding fixed sized word embeddings, like 300D Glove or Word2vec word embeddings?. 
  

Comments:
I extract features like examples in extarct_features.py. But went I used these features(the last encoded_layers) as word embeddings, I got a worse result than using 300D Glove. I also used these features to compute the cos similarity for each word in  sentences, I found that all values were around 0.6

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi. I really need someone's help regarding using the bert model to extract the word embeddings: This script: https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/extract_features.py doesn't exist anymore.

Any suggestions instead that replaced this script?
where is "pytorch_model.bin"?
Author: jufengada
Pull request: no
Labels: name: wontfix


Comments:
@jufengada 

Assuming that you've installed pytorch_pretrained_bert package properly. If you load any of the `BERT` models ex: `BertForSequenceClassification` with `.from_pretrained` method with arguments for type of Bert architectures say `bert-base-uncased`; pytorch_model.bin will be downloaded from an s3 bucket to a temporary folder in your environment.

Another way is downloading entire set of pre-trained weights in to a folder from the github repository and 
 pointing the `path to pre trained weights` in the call for `.from_pretrained` method would also do the trick.



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
DistilBERT baseline
Author: jeremyasapp
Pull request: no

Nice work on DistillBert! I was wondering if you had baseline numbers for the student model trained without the distillation objective? Seems like a very important baseline to understand if distillation was useful.

Comments:
Hello @jeremyasapp,
That's a good question!

Here are the results on the pre-training solely using MLM training signal (the small model is initialized from the teacher though).
![image](https://user-images.githubusercontent.com/16107619/64210993-c0ef1b80-ce72-11e9-806b-171313e8ae9e.png)

This is the same table presented in the blog post to which I added the last row. The drop in performance is consistent across the tasks (except for QNLI or QQP). I believe though that you could slightly improve these figures by continuing the pre-training for a few more epochs.

I should also refer you to my answer to [this question](https://medium.com/@weadhsu_77395/can-you-provide-a-comparison-with-the-small-model-that-is-trained-from-scratch-d75001057e15#--responses) in the blogpost. It is also related to the influence of the knowledge distillation loss.

(Btw, it's `DistilBERT` with a single `L` 🤣no worries though, I found your issue ;))
Victor

Hey @VictorSanh, thanks for the response.

Definitely non trivial improvement! I actually attempted something similar to you guys this summer, but with less success. It's really great to see that this works :)
added tokens may split a normal token into halves
Author: askerlee
Pull request: yes

In the tokenizer base class, split_on_token() attempts to split input text by each of the added tokens. Because it uses text.split(tok), it may accidentally split a token at the middle. 

For example a new token "ht" is added to the vocabulary. Then "light" will be split into "lig" and "". But as "light" is in the original vocabulary, it should be left intact to be processed by self._tokenize(). 

Hence I'd suggest to replace it with re.split, which will split only at word boundaries ([0-9a-zA-Z_]). 

But in languages whose word boundaries are different from English, this behavior may be undesirable and the user can revert to the old text.split(). It is controlled by a newly added flag split_added_on_word_boundary. 


Comments:
This looks great, thanks a lot @askerlee.

Can you add a docstring for the new argument?

Ideally also a test in [tokenization_tests_commons.py](https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/tests/tokenization_tests_commons.py)

Thanks @thomwolf . I updated my patch to make it neater. Will close this pull request and submit a new request soon.
Implementing XLNet in pytorch
Author: roholazandie
Pull request: no

Due to the new work of [XLNet](arxiv.org/abs/1906.08237 ) and the implementation of it in tensorflow maybe we need to add this one to the current repository.

https://github.com/zihangdai/xlnet

Comments:
FYI @roholazandie we are currently working on XLNet with pytorch over here
https://github.com/pingpong-ai/XLNet-pytorch/tree/dev/poc

I'll add it here also. I was working on a coming release this week anyway.
It's a mix of BERT/Transformer-XL and something I was also playing with (Two-Stream Self-Attention) so hopefully, it won't delay too much the release. 

Awesome, downstream libraries like [flair](https://github.com/zalandoresearch/flair) are really looking forward to use XLNet 🤗 (So expect results on CoNLL and PoS tagging whenever XLNet is implemented here)

and [jiant](https://jiant.info)!

I had finished Simple XLNet implementation with Pytorch Wrapper here : https://github.com/graykode/xlnet-Pytorch

How is this project going?

I'm debating whether to try to build XLNet into jiant directly, but I'm not eager to replicate your hard work. Tx!

Yes we're on track to finish the release this week I think (or next Monday in the worse case).

We reproduced the results of XLNet on STS-B (Pearson R > 0.918), the GLUE task showcased on the TF repo, with the same hyper-parameters (didn't try the others tasks but the model is the same for all).

It's taking a bit more time than planned because we took the opportunity to refactor the library's back-end. I'm really excited about the new release.

We will now have 6 different architectures (BERT, GPT, GPT-2, Transformer-XL, XLNet and XLM) and over 25 associated pretrained weights, all with the same API (for instance the GLUE training script is now the same for all the models!) and direct access to all the models' internals.

And a lot of other things (a lot of tests to avoid future regressions, compatibility with TorchScript, easy serialization and loading of fine-tuned models...)

> I'll add it here also. I was working on a coming release this week anyway.
> It's a mix of BERT/Transformer-XL and something I was also playing with (Two-Stream Self-Attention) so hopefully, it won't delay too much the release.

@thomwolf , very much looking forward to your implementation of BERT/Transformer-XL. Wondering if you were planning to release that too, and if so where you are planning to make it available. Thanks so much 

Is there any direct wrapper for QA task using XLNet?

I appriciate your hard work. I just saw it today, and you almost implemented.

Is it really better than Bert, which i find to be amazing so far?

Under the conditions in their paper, yes. Question is whether it always is: in what tasks, using the same data set; with same compute time (or energy consumption) for training (+on what hardware as same speed on hardware X doesn't imply same speed on hardware Y); with same number of parameters (model size); with architecture size for same prediction speed; consistently / on average (+confidence interval) for repeated runs with random initialisation...
How to get perplexity score of a sentence using anyone of the given Language Models?
Author: vikas-baghel
Pull request: no
Labels: name: wontfix

I want to find the perlexity score of sentence. I know that we can find the perplexity if we have the loss as perplexity = 2^(entropy loss). Can you tell me how to do it with the models you have listed?
It will be of great help.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
gpt-2 model doesn't output hidden states of all layers.
Author: fabrahman
Pull request: no
Labels: name: wontfix

Using GPT2Model , it seems like it outputs the hidden states of only 1 layer. However, according to code and documentation it is expected to output hidden states features for each layer.

Am I making a mistake?

Thanks for the advise,

Comments:
Currently not indeed. This option will be in the coming release.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
some words not in xlnet vocabulary ,especially name 
Author: lagka
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I face the problem when xlnet tokenizer encodes name 

    from pytorch_transformers import *
    import torch
    tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased') 
    model = XLNetModel.from_pretrained('xlnet-base-cased')
    sents = ["here is a dog","michael love mary <pad>"]
    input_ids =[tokenizer.encode(sent) for sent in sents]

    model(torch.tensor(input_ids))[0]
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: expected sequence of length 4 at dim 1 (got 8)

I think the reason is that xlnet can't encode the name "michael " and "mary" , instead it encodes "michael" to "_mi" "cha" "el" and mary  to "m" "ary"  respectively . And this result leads input_ids to has two different length id list . As the result , when I fed the torch.tensor(input_ids) to XLNetModel would cause "ValueError" problem , how could I fix it?

    input_ids
    [[193, 27, 24, 2288], [12430, 2409, 530, 564, 17, 98, 1449, 5]]
    [tokenizer.tokenize(sent) for sent in sents]
    [['▁here', '▁is', '▁a', '▁dog'], ['▁mi', 'cha', 'el', '▁love', '▁', 'm', 'ary', '<pad>']]


Comments:
Hi!

XLNet uses a SentencePiece tokenizer which splits the words into subword units. In your case, it splits the two examples in different sized sequences, which can't be parsed as a Tensor which requires an input matrix (and not a list of lists).

You should pad your sequences after they have been tokenized so that they all are of equal size. Then they can be converted to a tensor and fed to the model.

> Hi!
> 
> XLNet uses a SentencePiece tokenizer which splits the words into subword units. In your case, it splits the two examples in different sized sequences, which can't be parsed as a Tensor which requires an input matrix (and not a list of lists).
> 
> You should pad your sequences after they have been tokenized so that they all are of equal size. Then they can be converted to a tensor and fed to the model.

Thanks for your reply.
My purpose is to fed each word's contextual embedding to transformer layer to obtain sentence embedding
However ,  by this way I can't get each word's contextual embedding , is there any solution to fix this problem or just use xlnet function to get sentence embedding?

If you're looking to create sentence embeddings based on transformers, I'd like to redirect you to [this issue](https://github.com/huggingface/pytorch-transformers/issues/876) that discusses exactly this.

It discusses the use of the [UKPLab sentence-transformers library](https://github.com/UKPLab/sentence-transformers) which is built on top of our library and which can provide sentence embeddings based on XLNet.
Finetuning BertModel to extract textual features for VQA shows bad results
Author: ggaemo
Pull request: no

## ❓ Questions & Help
I am trying to use Bert as a textual feature extractor for VQA.

This is the code for tokenizing question text in VQA.
```
    self.tokenizer = pytorch_transformers.BertTokenizer.from_pretrained(
            'bert-base-uncased')
    q = self.tokenizer.encode(q_text, add_special_tokens=True)
```

This code is for extracting features from Bert.
```

  self.bert = pytorch_transformers.BertModel.from_pretrained('bert-base-uncased',
                                                                   output_attentions=True)
    question_mask_cls = torch.arange(lengths[0]).to(self.device)
    lengths_cls = lengths
    q_mask_cls = question_mask_cls[None, :] < lengths_cls[:, None]
    q_mask_cls = q_mask_cls.to(torch.float32)
    question_embed_t, _, src_attn_list = self.bert(question_padded,
                                                 attention_mask=q_mask_cls)

    output, tgt_attn_list, tgt_src_attn_list = self.q_decoder(
            tgt=obj_feature_bbox_cls.permute(1, 0, 2),
                       memory=question_embed_t.permute(1, 0, 2),
                       memory_key_padding_mask=memory_key_padding_mask,
                       tgt_key_padding_mask=tgt_key_padding_mask)
```

If I free the parameters of Bert, it gives better results. But when I fine tune the whole model, the model does not seem to learn. I've tried the Adam optimizer of pytorch and AdamW provided by this repository. Both of them does not work.

```
optimizer = pytorch_transformers.optimization.AdamW(model.parameters(), lr=3e-5)
```

![image](https://user-images.githubusercontent.com/8081512/64243428-4ab9ea00-cf42-11e9-8e36-70121e0453ae.png)


The orange curve shows the model with Bert parameters freezed, while the pink and skyblue curve shows the model that trains Bert parameters.


Are there any potential issues I am missing?
<!-- A clear and concise description of the question. -->

Comments:
It turned out the learning rate was too high (1e-4) using 3e-5 showed good results.
ImportError: cannot import name 'DistilBertModel'
Author: zbloss
Pull request: no

## 🐛 Bug

Can you update the pypi package? I cannot import DistilBERT on pytorch_transformers==1.1.0

Comments:
Yes, you should install it from source right now if you want to use DistilBERT. We're planning a new release 1.2.0 that includes DistilBERT + GPT-2 Large + XLM 100/17 sometimes this week :).
[README] link to Write With Transformer
Author: julien-c
Pull request: yes

Tomorrow we'll release a new version of Write With Transformer that's gonna let you:
- experiment with different models (gpt2, xlnet) and/or model checkpoints (example for gpt2: small, large, arxiv)
- Share links to your documents.

With those two changes transfomer.huggingface.co is graduating to being an official demo for `pytorch-transformers`'s text generation capabilities.

Comments:
Fix typo
Author: tm4roon
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1205?src=pr&el=h1) Report
> Merging [#1205](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1205?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/0b52642d379bed155e8aa4f4088588bfd8ceaa88?src=pr&el=desc) will **increase** coverage by `0.44%`.
> The diff coverage is `95.65%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1205/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1205?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1205      +/-   ##
==========================================
+ Coverage   80.83%   81.27%   +0.44%     
==========================================
  Files          46       46              
  Lines        7878     7877       -1     
==========================================
+ Hits         6368     6402      +34     
+ Misses       1510     1475      -35
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1205?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...h\_transformers/tests/tokenization\_tests\_commons.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1205/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvdG9rZW5pemF0aW9uX3Rlc3RzX2NvbW1vbnMucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [pytorch\_transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1205/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX3V0aWxzLnB5) | `89.37% <91.66%> (+8.89%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1205?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1205?src=pr&el=footer). Last update [0b52642...5c6cac1](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1205?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks a lot for that!
Took the occasion to add regression tests and clean up a bit the base class.
how does the run_squad.py deal with non-answerable questions 
Author: elephantomkk
Pull request: no
Labels: name: Help wanted

name: Discussion

Hi, 

I want to do a similar reading comprehension task with non-answerable questions but i didn't figure out how you deal with it from the codes. Did you add additional token on this?  Or only outputs the no-answer when the start logit = end logit = -1? Thanks! 

Comments:
@Liangtaiwan and @abeljim were the contributors of the `run_squad.py` example.
Maybe they can help.

Hi @elephantomkk,

The non-answerable solving is using [CLS] token as the ground truth.
As a result, the start login = and end logit = -1.

You can find it out in this repo code or official Bert code.
Also, the method is mentioned in Jacob Devlin's slide.
https://nlp.stanford.edu/seminar/details/jdevlin.pdf?fbclid=IwAR2TBFCJOeZ9cGhxB-z5cJJ17vHN4W25oWsjI8NqJoTEmlYIYEKG7oh4tlY

Thanks for the reply! I tested that and found the performance on the non-answerable is not so good compared with the answerables:(
BertTokenizer never_split issue
Author: goPikachu88
Pull request: no
Labels: name: wontfix

Hi,

I'm using the BertTokenizer to tokenize a piece of text where I use some entity markers to mark the beginning and end of entities, e.g.:

> This was among a batch of paperback [E1] Oxford World [/E1] ' s Classics

I've manually added such entity markers to _vocab file_ and the _never_split_ tuple in _BertTokenizer_. My purpose is to retain such markers as a token, not to be split into wordpieces.

However, when I test the code from command line in linux terminal, the _never_split_ does not work. Entity markers are split into wordpieces. Here is the printout: 
```
06/28/2019 11:41:20 - INFO - __main__ -   Writing example 0 of 1000
['In', '1983', ',', 'a', 'year', 'after', 'the', 'rally', ',', '[', 'E', '##1', ']', 'For', '##sberg', '[', '/', 'E', '##1', ']', 'received', 'the', 'so', '-', 'called', '`', '`', 'genius', 'award', "'", "'", 'from', 'the', '[', 'E', '##2', ']', 'John', 'D', '.', '[', '/', 'E', '##2', ']', 'and', 'Catherine', 'T', '.', 'MacArthur', 'Foundation', '.']
```

The strange thing is that, the _never_split_ works pretty fine when I test it in PyCharm, and I get my desired output:

```
06/28/2019 11:53:01 - INFO - __main__ -   Writing example 0 of 1000
06/28/2019 11:53:01 - INFO - __main__ -   *** Example ***
06/28/2019 11:53:01 - INFO - __main__ -   guid: train-61b3a65fb9b7111c4ca4
06/28/2019 11:53:01 - INFO - __main__ -   tokens: [CLS] In 1983 , a year after the rally , [E1] For ##sberg [/E1] received the so - called ` ` genius award ' ' from the [E2] John D . [/E2] and Catherine T . MacArthur Foundation . [SEP]
06/28/2019 11:53:01 - INFO - __main__ -   input_ids: 101 1130 2278 117 170 1214 1170 1103 11158 117 20 1370 19945 21 1460 1103 1177 118 1270 169 169 13533 2574 112 112 1121 1103 22 1287 141 119 23 1105 6017 157 119 21045 2974 119 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
``` 
The tests are done on the same machine (my local PC) under same virtual environment. The code, pre-trained bert model and parameters used in Pycharm and terminal are the **same**. 

Since I need to migrate the code to a server for model training, I really need to resolve this issue. I spent some time debugging but have no idea what could be the cause. Could anyone please provide some hints?

Thanks in advance.


 

Comments:
which version of python do you use in these environments?

> which version of python do you use in these environments?

Hi Thomwolf,

I'm using Python 3.6.8 for all these environments.



In case it's helpful, I create a gist to include some details of this issue: https://gist.github.com/ardellelee/4d80ee7a07166bb6d1a203fdd4d7cc07

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Recommended multilingual bert cased model returns similar embeddings 
Author: oleg-yaroshevskiy
Pull request: no
Labels: name: wontfix

I'm trying to get embeddings for multilingual input: 
```
tokenizer = BertTokenizer.from_pretrained("bert-base-multilingual-cased", do_lower_case=False)

class NeuralNet(BertPreTrainedModel):
    def __init__(self, config):
        super(NeuralNet, self).__init__(config)
        self.bert = BertModel(config)
        self.apply(self.init_bert_weights)
    
    def forward(self, input_ids, token_type_ids=None, attention_mask=None):
        _, pooled_output = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)

        return pooled_output

model = NeuralNet.from_pretrained("bert-base-multilingual-cased")
```

and for some reason all `pooled_output` vectors are very similar with 1e-3 cosine distance for semantically different inputs. Changing model to `bert-base-multilingual-UNcased` works just okay. Any ideas?

Comments:
I second this issue #735 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.