noamaanMulla-03/transformers-issues huggingface dataset:
model = GPT2LMHeadModel.from_pretrained(args.model_path) try loads in json format
Author: lucasjinreal
Pull request: no

## 🐛 Bug

Initial a model in this code:

```
logging.info('loading model from: {}'.format(args.model_path))
    model = GPT2LMHeadModel.from_pretrained(args.model_path)
```



It seems inside transforms reading our model in json format which cause error:

```
INFO 11-10 16:18:49 generate.py:167 - loading model from: ./weights/ProseInChinese/pytorch_model.bin
I1110 16:18:49.226168 139982371047232 configuration_utils.py:148] loading configuration file ./weights/ProseInChinese/pytorch_model.bin
Traceback (most recent call last):
  File "generate.py", line 225, in <module>
    main()
  File "generate.py", line 168, in main
    model = GPT2LMHeadModel.from_pretrained(args.model_path)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py", line 287, in from_pretrained
    **kwargs
  File "/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py", line 154, in from_pretrained
    config = cls.from_json_file(resolved_config_file)
  File "/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py", line 186, in from_json_file
    text = reader.read()
  File "/usr/lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
```

Comments:
What string was passed under `args.model_path`? Is it a directory or a file name? (It should be a directory)

model_path is a model trained with xxx.pth format.

Is that a directory? contains what？

The `model_path` should link to a directory holding the pytorch model and the configuration file associated with it, as if it was saved with `model.save_pretrained(directory)`.

What is your `xxx.pth` format, how did you obtain such a file?

Oh, no, I just got others trained xx.pth model, no config file.

Why a pytorch model need additional config.json? And people really don't know what's the format inside that file at all..... no convinient

The Pytorch models require configuration files because the architectures in the library can have very different sizes. For example GPT-2 small, medium, large and XL all share the same model but have different sizes. Configuration files are essential for this purpose.

The format is defined by the `save_pretrained` method on the models. Models acquired from other sources than our library may not be loadable, or may need to be converted using one of our conversion scripts.

Please read the [quickstart portion of the documentation](https://huggingface.co/transformers/quickstart.html) to get a deeper understanding of the way the library works.
Hi there, is bert-large-uncased-whole-word-masking-finetuned-squad trained for Squad 1.0 or 2.0?
Author: JianLiu91
Pull request: no
Labels: name: wontfix

I mean, whether the training data contains examples with no answer?

Comments:
Hi! I believe this checkpoint originates from the training specified [there](https://huggingface.co/pytorch-transformers/examples.html#squad).

The SQuAD version would then be 1.1!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Update conversion script to convert XLM-R
Author: Tiiiger
Pull request: yes

It requires a few changes to the existing RoBERTa conversion script because `fairseq` has updated the RoBERTa model definition. 

Specifically, the definition of multiheaded attention is changed at https://github.com/pytorch/fairseq/blob/2a9b4ec2374574cd0315f95e126788e4fe795f0d/fairseq/modules/multihead_attention.py#L42

With this new script, you should be able to convert the XLM-R model released at https://github.com/pytorch/fairseq/tree/master/examples/xlmr and requested by #1769. 

I have successfully converted the two released model weights. 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1812?src=pr&el=h1) Report
> Merging [#1812](https://codecov.io/gh/huggingface/transformers/pull/1812?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/155c782a2ccd103cf63ad48a2becd7c76a7d2115?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1812/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1812?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1812   +/-   ##
=======================================
  Coverage   84.16%   84.16%           
=======================================
  Files          94       94           
  Lines       14185    14185           
=======================================
  Hits        11939    11939           
  Misses       2246     2246
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1812?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1812?src=pr&el=footer). Last update [155c782...6d2a5bf](https://codecov.io/gh/huggingface/transformers/pull/1812?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This is great indeed but we are losing backward compatibility to convert the original RoBERTa weights aren't we?

Maybe we can add a flag to control which version of fairseq RoBERTa the original weight come from? For instance a flag as a string selected between `roberta` and `XLM-R`.

I think the situation is a bit more complicated than I thought. We also need to change the vocabulary size, etc. I think it would be better to have a separate treatment for XLM-R. So I am closing this for now.
Whether it belongs to the bug of class trainedtokenizer decode?
Author: yuanxiaosc
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

![1](https://user-images.githubusercontent.com/16183570/68684557-a4bed780-05a3-11ea-8d06-1c5638be6ac6.png)
![2](https://user-images.githubusercontent.com/16183570/68684572-abe5e580-05a3-11ea-9620-847264507fae.png)




Comments:
You're right, this is an error. The PR #1811 aims to fix that issue!

It should be fixed now, thanks! Feel free to re-open if the error persists.
Remove duplicate hidden_states of the last layer in BertEncoder in modeling_bert.py
Author: usertomlin
Pull request: no
Labels: name: wontfix

## 🚀 Feature

<!-- A clear and concise description of the feature proposal. Please provide a link to the paper and code in case they exist. -->
For  "class BertEncoder" in modeling_bert.py, remove duplicate hidden_states of the last layer 

## Motivation

 
![image](https://user-images.githubusercontent.com/2401439/64491964-fb0c5300-d2a0-11e9-9873-d86ee9041db3.png)


Bert-Base models have 12 layers instead of 13 layers. But when config.output_hidden_states is true, "len(all_hidden_states)" is printed 13 instead of 12. It seems that the two lines under "# Add last layer" is improper, since the last layer's hidden_states are already added.



Comments:
Hi! Indeed, the BERT-base only has 12 layers. The `all_hidden_states` is 13-dimensional however because it keeps track of the inputs as well.

In the code you have shown, the `hidden_states` variable is computed between the two underlined variables you mentioned. None of it is redundant :)! 

> Hi! Indeed, the BERT-base only has 12 layers. The `all_hidden_states` is 13-dimensional however because it keeps track of the inputs as well.
> 
> In the code you have shown, the `hidden_states` variable is computed between the two underlined variables you mentioned. None of it is redundant :)!

Thanks, I see :)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
how to use 'spiece.model' to create the xlnet_tokenizer
Author: yangzh9106
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
For some reason, my computer can not connect to Internet, which means I can not use the code "tokenizer = tokenizer_class.from_pretrained('xlnet-base-cased', do_lower_case = True)" to create the tokenizer.
The Piece model (spiece.model)  is used for (de)tokenization, how can I use it to create the tokenizer?
I have tried :
`
from pytorch_transformers import XLNetTokenizer
config = {
'vocab_path' :  path.sep.join([BASIC_DIR,'pretrained/pytorch_xlnet_pretrained/spiece.model'])      
        }
tokenizer = XLNetTokenizer.from_pretrained(config['vocab_path'], do_lower_case = True)
`
it doesn't work. 

Comments:
First, note that you can give proxies to the `from_pretrained` method if your connection problem comes from a proxy (see the doc/docstring for an example).

You can also download the sentence piece model from our S3 bucket (see the top of the `tokenization_xlnet.py` file for the url) and save it in a folder with the name `spiece.model`, then just give this folder path to the `from_pretrained` method.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
class DistilBertForMultiLabelSequenceClassification()
Author: emtropyml
Pull request: no
Labels: name: wontfix

## 🚀 Feature

Distil BERT For Multi-Label Sequence Classification

## Motivation

To do multi-label text classification using DistilBERT

## Additional context

None

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Token indices sequence length is longer than the specified maximum sequence length for this model
Author: LysandreJik
Pull request: yes

As of now the tokenizers output a specific warning when an encoded sequence is longer than the maximum specified sequence length, which is model-specific:

```
Token indices sequence length is longer than the specified maximum sequence length for this model (X > 1024). Running this sequence through the model will result in indexing errors
```

It is currently in the `convert_tokens_to_ids` and this leads to two issues:

- using `encode` or `encode_plus` methods with a `max_length` specified will still output that warning as the `convert_tokens_to_ids` method is used before the truncation is done. (cf #1791)
- since `prepare_for_model` was introduced, I personally feel that all modifications related to the model should happen in that method and not in `tokenize` or `convert_tokens_to_ids`. 

This PR aims to slightly change the behavior so that both aforementioned issues may be solved by putting the warning in the `prepare_for_model` method if no `max_length` is specified.

Comments:
replace LambdaLR scheduler wrappers by function
Author: rlouf
Pull request: yes

Custom schedulers are currently initiated by wrapping Pytorch's LambdaLR
class and passing a method of the wrapping class to the __init__
function of LambdaLR. This approach is not appropriate for several
reasons:

1. one does not need to define a class when it only defines a
__init__() method;
2. instantiating the parent class by passing a method of the child class
creates a cyclical reference which leads to memory leaks. See issues #1742 and #1134.

In this commit we replace the wrapper classes with functions that
instantiate `LambdaLR` with a custom learning rate function. We use a
closure to specify the parameter of the latter. We also do a bit of
renaming within the function to explicit the behaviour and removed
docstrings that were subsequently not necessary.

Comments:
Yes, great job tracking and fixing this.
Can you update all the examples as well?

Great job finding the issue!

I updated the docs and examples. I am confused because the tests fail on this function:

```
=================================== FAILURES ===================================
______________ TFDistilBertModelTest.test_pt_tf_model_equivalence ______________

self = <transformers.tests.modeling_tf_distilbert_test.TFDistilBertModelTest testMethod=test_pt_tf_model_equivalence>

    def test_pt_tf_model_equivalence(self):
        if not is_torch_available():
            return
    
        import torch
        import transformers
    
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()
    
        for model_class in self.all_model_classes:
            pt_model_class_name = model_class.__name__[2:]  # Skip the "TF" at the beggining
            pt_model_class = getattr(transformers, pt_model_class_name)
    
            config.output_hidden_states = True
            tf_model = model_class(config)
            pt_model = pt_model_class(config)
    
            # Check we can load pt model in tf and vice-versa with model => model functions
            tf_model = transformers.load_pytorch_model_in_tf2_model(tf_model, pt_model, tf_inputs=inputs_dict)
            pt_model = transformers.load_tf2_model_in_pytorch_model(pt_model, tf_model)
    
            # Check predictions on first output (logits/hidden-states) are close enought given low-level computational differences
            pt_model.eval()
            pt_inputs_dict = dict((name, torch.from_numpy(key.numpy()).to(torch.long))
                                  for name, key in inputs_dict.items())
            with torch.no_grad():
                pto = pt_model(**pt_inputs_dict)
            tfo = tf_model(inputs_dict)
            max_diff = np.amax(np.abs(tfo[0].numpy() - pto[0].numpy()))
>           self.assertLessEqual(max_diff, 2e-2)
E           AssertionError: nan not less than or equal to 0.02
```

which has no apparent link with my changes.

This test has been failing on and off for a week or so now; I'll look into it soon.
Quick questions about details
Author: syahrulhamdani
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Can someone explain the difference between `mem_len`,  `mlen`, and `ext_len` in `TransfoXLModel`? While the documentation has stated below

![Screen Shot 2019-09-10 at 11 18 40](https://user-images.githubusercontent.com/23093968/64583914-110f3600-d3bd-11e9-9571-5e9c607a1744.png)

Unfortunately, I'm still confused especially `mem_len` and `mlen`. Thank you so much.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to set the weight decay in other layers after BERT output?
Author: g-jing
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I notice that we should set weight decay of bias and LayerNorm.weight to zero and set weight decay of other parameter in BERT to 0.01. But how to set the weight decay of other layer such as the classifier after BERT? Thanks 

Comments:
@RoderickGu. maybe try:
```python
    bert_param_optimizer = list(model.bert.named_parameters())
    lstm_param_optimizer = list(model.bilstm.named_parameters())
    crf_param_optimizer = list(model.crf.named_parameters())
    linear_param_optimizer = list(model.classifier.named_parameters())
    no_decay = ['bias', 'LayerNorm.weight']
    optimizer_grouped_parameters = [
        {'params': [p for n, p in bert_param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01,
         'lr': args.learning_rate},
        {'params': [p for n, p in bert_param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0,
         'lr': args.learning_rate},
        {'params': [p for n, p in lstm_param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01,
         'lr': 0.001},
        {'params': [p for n, p in lstm_param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0,
         'lr': 0.001},
        {'params': [p for n, p in crf_param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01,
         'lr': 0.001},
        {'params': [p for n, p in crf_param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0,
         'lr': 0.001},
        {'params': [p for n, p in linear_param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01,
         'lr': 0.001},
        {'params': [p for n, p in linear_param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0,
         'lr': 0.001}
    ]
```

@lonePatient  Thanks for your answer! Do you mean that if there is a linear layer after the BERT, the weights of the linear layer will get a weight decay, but the bias of the linear layer will not? Besides, I wonder since your code covers each part in the model, if your answer is equivalent to:

`param_optimizer = list(model.named_parameters())
optimizer_grouped_parameters = [
        {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01, 'lr': args.learning_rate},
        {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0, 'lr': args.learning_rate}
]`

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Can pytorch-transformers be used to get XLM sentence embeddings for multiple languages?
Author: pbutenee
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I tried to create a class to get the XLM sentence embeddings in multiple languages.
```
class XLMSentenceEmbeddings(pt.XLMPreTrainedModel):
    def __init__(self, config):
        super(XLMSentenceEmbeddings, self).__init__(config)
        self.transformer = pt.XLMModel(config)
    def forward(self, input_ids, lengths=None, position_ids=None, langs=None, token_type_ids=None,
                attention_mask=None, cache=None, labels=None, head_mask=None):
        transformer_outputs = self.transformer(input_ids, lengths=lengths, position_ids=position_ids, token_type_ids=token_type_ids, langs=langs, attention_mask=attention_mask, cache=cache, head_mask=head_mask)
    return transformer_outputs[0][:, 0, :]
```

But if I try this code I get the same result if I set the language ids to Dutch or if I don't.
```
tokenizer = pt.XLMTokenizer.from_pretrained('xlm-mlm-100-1280')
xlm_model = XLMSentenceEmbeddings.from_pretrained('xlm-mlm-100-1280')

sentence = 'een nederlandstalige zin'
lang = 'nl'
input_ids = torch.tensor(tokenizer.encode(sentence, lang=lang)).unsqueeze(0)
with torch.no_grad():
    xlm_model.eval()
    output_without_lang = xlm_model(input_ids).numpy()
    output_with_lang = xlm_model(input_ids, langs=xlm_model.transformer.config.lang2id[lang]*torch.ones(input_ids.size(), dtype=int)).numpy()

np.sum(output_without_lang - output_with_lang)
```
The output of this sum at the end always returns zero.

If I change the config to use the language embeddings like so: `config.use_lang_emb = True` my results are random each time which seems to suggest the embeddings are not included in the model. Should I change a different config? Or is the only way to get the sentence embeddings to train the model again?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Output of BertModel does not match fixed feature vectors extracted from the last hidden layer  
Author: sysuzyx
Pull request: no
Labels: name: wontfix

I want to finetune bert in my task. However, the bert ouput does not match the fixed feature vectors.

I extracted the fixed feature vectors like this:
I use `extract_features.py` to  extract the fixed feature vectors of last hidden layer (layer -1). The command line is below:
`python extract_features.py --input_file=input.txt --output_file=output.json --vocab_file=model_path/vocab.txt --bert_config_file=model_path/bert_config.json --init_checkpoint=model_path/bert_model.ckpt --layers=-1 --max_seq_length=128 --batch_size=8`


The bert model output extracted like below:
The load model code is like below:
 `model_dict = model.state_dict()`
`#load pretrained released bert model`
  `pretrained_dict = torch.load( 'pytorch_model.bin')`
  `pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in model_dict}`
`#update params`
  `model_dict.update(pretrained_dict)`
  `model.load_state_dict(model_dict)`

The init model code is like below:
`class A(nn.Module):`
` def __init__(self):`
` self.config = BertConfig.from_json_file('config.json')`
` self.bert = BertModel(self.config)`
`def inference(self, input_ids):`
` all_encoder_layers, _ = self.bert(input_ids, token_type_ids=None, attention_mask=input_mask)`
`return all_encoder_layers[-1]`
(I've omitted some irrelevant code.)

Then I ouput tensor all_encoder_layers[-1]. 
all_encoder_layers[-1] doesn't match the feature vectors extracted by `extract_features.py` .

I checked the params in my model. Bert params have been loaded and it is consistent with pretrained params. Also. the input sequence is consistent.
 Can anybody help me? Is there any settings I forget?






Comments:
Hi, have you solved this problem?
And does anyone know the order of all_encoder_layers?
Thanks.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
model_type for gpt
Author: tuhinjubcse
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

To run openai-gpt model in run_glue.py what should we provide as model_type? The model classes contains 
MODEL_CLASSES = {
    'bert'
    'xlnet'
    'xlm'
    'roberta'
}

Comments:
Hi Tuhin, adding GPT/GPT2 is a tiny bit more complicated since these models need to have special tokens added to their vocabulary prior to fine-tuning (just a few more lines of code though).

Do you want to try to make a PR with these other models?

Okay I will add bert as the model type i was just not sure if that d be the right thing to do , hence asked :)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
XLnet output attentions doesn't work
Author: aviclu
Pull request: no

## 🐛 Bug

Model I am using - XLNet.

Language I am using the model on English.

The problem arise when using the flag `output_attentions=True`

## To Reproduce

## Expected behavior
When executing
`        outputs= self.model(input_ids) 
`
I would expect to have a tuple with outputs and attentions but it fails.
The problem probably roots from the lines:
`            if self.output_attentions:
                attentions.append(outputs[2])
`
I receive Nones instead of the attentions, or sometimes the error:
`IndexError: tuple index out of range`
For the following line in the forward function:
`attentions.append(outputs[2])`
## Environment

* OS: Windows
* Python version: 3.7.3
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): Master branch version from 31.08.19
* Using GPU: Yes

Comments:
why did you close it?

I'm checking, it might be my own but.