noamaanMulla-03/transformers-issues huggingface dataset:
Extending run_language_modeling.py for XLNet
Author: shngt
Pull request: no

# 🚀 Feature request

The run_language_modeling.py script in examples/language-modeling/ currently works for BERT, RoBERTa, GPT and related models. It would be helpful if it also allowed XLNet. I believe this would involve writing functions to generate `perm_mask`, `target_mapping` and `labels` for the inputs sequences as per the paper and `https://github.com/zihangdai/xlnet/blob/master/data_utils.py`, but I'm not a 100% sure about this.  

## Motivation

I have to adapt XLNet onto a specialized domain (finance) and am yet to find a decent guide or implementation which discusses how to do this. I've decided to do this myself (with the help of this library), and would like to share my code in case others find it useful.

## Your contribution

Since I will be trying to do this anyway, I would like to submit a relevant PR when it is done. I was also hoping to receive guidance or feedback as appropriate by other contributors to ensure correctness and utility.

Comments:
See also https://github.com/huggingface/transformers/issues/2008

@shngt can you please share your code for adapting xlnet to the domain specific data ? ( the cli command or the code you used after the merged PR ) 
I was following the latest changes to transformers but I still don't understand how to continue training xlnet on the domain specific data.
thanks 

@shngt @krannnn Are there any updates on this? I'm interested as well (: Thanks! 

@matthiaslmz @krannnn I'm afraid I no longer have the cli command I used to run the example script, but I think it was quite similar to what's given in the docs. Are you facing some specific issue?
Fix ignore files behavior in doctests
Author: bryant1410
Pull request: yes

In the doc tests (which btw I'm aware they're disabled), the `ignore_files` uses a mutable default value, so it's modified (e.g., when `__init__.py` is appended), it modifies the value for the next function calls that also use the default value (that don't set the arg).

I also fixed typing issues in the file and other minor issues.

Comments:
Longformer inference time
Author: cmdllx
Pull request: no
Labels: name: wontfix

The longformer-base-4096 model should be faster than bert-base-cased ,However I find that the former one takes more time to inference

Comments:
hi @cmdllx , I don't think longformer will  be necessarily faster than bert-base-cased. The goal of longformer is to make memory complexity of attention layer liner w.r.t to seq length instead of quadratic, so it saves memory, not compute. More the seq-length , more will be inference time.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
is config argument necessary for XXModel.from_pretrained method? And when is needed?
Author: xixiaoyao
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

I find the model can be loaded with `XXModel.from_pretrained` function even if no `config` argument is given. But if the `config` is given, the argument `gradient_checkpointing` cannot be enabled. I wonder why the config argument can lead to the unable of `gradient_checkpointing`??? 

I test the conclusion with longformer model. Thanks!


Comments:
Hi @xixiaoyao 
The `config` argument is not required for `from_pretrained`, when `config` is not passed it loads the saved config from the model dir. However, it's necessary to pass `config`  when you wan't to override some config value, like setting `gradient_checkpointing` to `True`. Here's how you can override the `config`  and pass it to `from_pretrained`.

```python
config = XXConfig.from_pretrained(path, gradient_checkpointing=True)
model = XXModel.from_pretrained(path, config=config)
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
demo website: i info icon should link to resource about parameters
Author: rugk
Pull request: no
Labels: name: wontfix

name: Write With Transformer

The website/demo at https://transformer.huggingface.co does have a "Model & decoder settings" box at the bottom left.
Next to it, at the right, there is an "(i)" sign.

STR: Click on it.
**What happens:** It links to this repo.
**What should happen:** It should link to somewhere, where the parameters are actually explained.

Reason:
The generic link to this repo is way to – well – generic. I've searched the Readme for the parameter names and could not find anything useful/an explanation of what they actually do.
The reason I click on that icon is to know what I can adjust there/what happens if I change an option, I don't just want to see the general project.

There should be a page that just explains, what each option means (in simple terms):
* Model size: what model to use (from small to big), the bigger the more useful/accurate(?) it may be
* Top p:
* Temperature: 
* Max time: how long the model should – at most – compute to search for results

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi @rugk, open to any suggestion for a better documentation link – e.g. should we link to https://huggingface.co/blog/how-to-generate ?

Maybe https://medium.com/huggingface/how-to-write-with-transformer-5ee58d6f51fa ?

Or to https://medium.com/huggingface/how-to-build-a-state-of-the-art-conversational-ai-with-transfer-learning-2d818ac26313#79c5 like on https://convai.huggingface.co/

cc @patrickvonplaten @sgugger 

I dunno. My complaint was just an UX complaint from a user with less scientific background, so linking to very in-deep document is likely bad.

How you solve this hmm?
Maybe just write a new doc/small wiki entry here on GitHub or so?

Though if I check your links the "Advanced settings" section in https://medium.com/huggingface/how-to-write-with-transformer-5ee58d6f51fa It does what I want: explains the parameter.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


shh #badbot :no_bell::robot::no_bell: This is still a TODO.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Feature request: State the goals and none goals of the library in the README
Author: talolard
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

I'd like the library's README to say what it's goals and none goals are.
For example, as a (new) user I really like [Typescripts non-goals](https://github.com/microsoft/TypeScript/wiki/TypeScript-Design-Goals) because it made it clear what to expect from Typescript and thus how to best use it. I'd like the same for transformers. 


## Motivation

As a new user  I look at the README and think this library can and wants to do everything that I do.


But I think it has a tighter more well defined (implicit) scope and that peripheral use cases are better handled elsewhere, such as aligning [offset annotations](https://github.com/huggingface/transformers/issues/7019#issuecomment-691965153) or having trainers [support other data structures](https://github.com/huggingface/transformers/issues/6860).

I wish that was stated (as clearly as possible) so that I could make the best use of transformers instead of forcing it to be something it doesn't want to be. 

## Your contribution

I think it's up to the Huggingface team to decide what transformers is and what it will be. After digging around for a few days I would suggest something like
> Transformers makes pre-trained language models available to everyone through clear, consistent and cross platform APIs. Transformer's goal(s) are to make pretrained models accessible and plug into end users existing infrastructure, as well as support common academic tasks. 

> Transformers does not aim to
* Implement end to end applications of NLP
* Implement domain specific models (Custom entities in Swahili, joint classification and entity annotation) 
* .... 


Comments:
That's a good idea! The closest document we have to this right now is our [philosophy](https://huggingface.co/transformers/philosophy.html), have you checked it out?

No, I just saw it now. 
It in fact has exactly what I would have wanted to find. Particularly the sentence
> As a consequence, this library is NOT a modular toolbox of building blocks for neural nets. If you want to extend/build-upon the library, just use regular Python/PyTorch/TensorFlow/Keras modules and inherit from the base classes of the library to reuse functionalities like model loading/saving.



We could clean up the readme indeed. It's probably time to remove the migration sections about `pytorh-pretrained-bert` and `pytorch-transformers` and the quick-tour about the examples could be shortened.

Cleaning the README has been on my TODO for a long time, but I have been actively procrastinating. Will do that this week.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
MobileBERT inconsistent output (padded / not padded text)
Author: sweco
Pull request: no
Labels: name: wontfix

## Environment info
- `transformers` version: 3.0.2
- Platform: Linux-5.4.0-47-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.3
- PyTorch version (GPU?): 1.5.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@LysandreJik, @mfuntowicz

## Information

When using MobileBERT model `google/mobilebert-uncased`, I get different model outputs for the same text when it contains padding and when it does not contain any padding (is the longest text in the batch). The same code produces correct results for other models, such as `bert-base-uncased`, so the issue is probably only with MobileBERT. The problem happens for both `CLS` output and all features.

Model I am using (Bert, XLNet ...): MobileBERT

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce
```python
from transformers import AutoModel, AutoTokenizer

model = 'google/mobilebert-uncased'
tokenizer = AutoTokenizer.from_pretrained(model)
model = AutoModel.from_pretrained(model)
text = 'Hey, how are you?'

i1 = tokenizer.batch_encode_plus([text], padding=True, return_tensors='pt')
emb1 = model(**i1)[1][0]  # Only one in batch (not padded): [-2.5088e+07,  7.3279e+04,  1.7544e+05, ...]

i2 = tokenizer.batch_encode_plus([text, text + ' hey'], padding=True, return_tensors='pt')
emb2 = model(**i2)[1][0]  # Not longest (padded): [-2.4871e+07,  8.1873e+04,  1.6693e+05, ...]

i3 = tokenizer.batch_encode_plus([text, text[:-10]], padding=True, return_tensors='pt')
emb3 = model(**i3)[1][0]  # Longest in batch (not padded): [-2.5088e+07,  7.3281e+04,  1.7544e+05, ...]

i4 = tokenizer.encode(text, return_tensors='pt')
emb4 = model(i4)[1][0]  # Without attention masks (not padded): [-2.5088e+07,  7.3279e+04,  1.7544e+05, ...]

i5 = tokenizer.encode_plus(text, return_tensors='pt')
emb5 = model(**i5)[1][0]  # With attention masks (not padded): [-2.5088e+07,  7.3279e+04,  1.7544e+05, ...]
```

## Expected behavior

I would expect the model to always return the same embeddings, but I don't really know which of the two variants is correct.

Comments:
Just confirming that the issue happens also for the latest `transformers`, PyTorch and Python 3.8.

- `transformers` version: 3.1.0
- Platform: Linux-5.4.0-47-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Can't load weights for
Author: saburbutt
Pull request: no
Labels: name: New model

# 🌟 New model addition

**Getting the following error after training a question answering problem using ALBERT.**

404 Client Error: Not Found for url: https://huggingface.co/saburbutt/albert_xxlarge_tweetqa_v2/resolve/main/tf_model.h5
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    950             try:
--> 951                 state_dict = torch.load(resolved_archive_file, map_location="cpu")
    952             except Exception:

13 frames
RuntimeError: [enforce fail at inline_container.cc:145] . PytorchStreamReader failed reading zip archive: failed finding central directory

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
OSError: Unable to load weights from pytorch checkpoint file for 'saburbutt/albert_xxlarge_tweetqa_v2' at '/root/.cache/torch/transformers/280e3f03092e3b52d227bc27519ff98aff017abcc160fc5138df7ce1bddcff1e.b5346cd8c01b1d2591b342ede0146ce26b68ad0a84ff87e5dc8f9d5a03a79910'If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True. 

During handling of the above exception, another exception occurred:

HTTPError                                 Traceback (most recent call last)
HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/saburbutt/albert_xxlarge_tweetqa_v2/resolve/main/tf_model.h5

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    683                     f"- or '{pretrained_model_name_or_path}' is the correct path to a directory containing a file named one of {TF2_WEIGHTS_NAME}, {WEIGHTS_NAME}.\n\n"
    684                 )
--> 685                 raise EnvironmentError(msg)
    686             if resolved_archive_file == archive_file:
    687                 logger.info("loading weights file {}".format(archive_file))

OSError: Can't load weights for 'saburbutt/albert_xxlarge_tweetqa_v2'. Make sure that:

- 'saburbutt/albert_xxlarge_tweetqa_v2' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'saburbutt/albert_xxlarge_tweetqa_v2' is the correct path to a directory containing a file named one of tf_model.h5, pytorch_model.bin.

It was working for all the previous models I have tried.



Comments:
DPRReaderTokenizers returns, for multiple passages given, only the tokens & masks of one passage 
Author: omarsou
Pull request: no

## Environment info
     
- `transformers` version: 3.5.1
- Platform: Colab Notebook
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
 tokenizers: @mfuntowicz 
 -->

## Information

Model I am using : DPRReaderTokenizer

The problem arises when using:
DPRReaderTokenizer, instead of returning as many tensor as passages, he returns only one (On the documentation it must return (n_passages, sequence_length), but it returns (1, sequence_length) on basic examples.


The tasks I am working on is:
* Tokenization with DPRReaderTokenizer on multiple passages (texts)

## To reproduce

from transformers import AlbertTokenizer, AlbertForQuestionAnswering, DPRReader, DPRReaderTokenizer, AutoTokenizer
import torch
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

tokenizer_DPR = DPRReaderTokenizer.from_pretrained('facebook/dpr-reader-single-nq-base')
model_DPR = DPRReader.from_pretrained('facebook/dpr-reader-single-nq-base', return_dict=True).cuda()

encoded_inputs = tokenizer_DPR(
         questions=["What is Transformers?"],
         titles=['Attention is all you need', 'One famous library'],
         texts=['Attention is a new mechanism designed to improve the performance of the seq2seq models', 'One of the most
famous NLP library is called Transformers' ],
         padding=True,
         return_tensors='pt'
)

encoded_inputs
{'input_ids': tensor([[  101,  2054,  2003, 19081,  1029,   102,  3086,  2003,  2035,  2017,
          2342,   102,  3086,  2003,  1037,  2047,  7337,  2881,  2000,  5335,
          1996,  2836,  1997,  1996,  7367,  4160,  2475,  3366,  4160,  4275]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1]])}

## Remarks
The expected outputs is two tensors ... and I got only one .. 👎 

Comments:
Hello, 
A little update, I think I fixed this issue. 
The tokenizer returns a tensor of shape (n_passages, n_sequence_length) but only because I have duplicated the question like [questions] * n_passages. It was not clear on the documentation since I thought it was automatically done.
BartTokenizer prepare_seq2seq_batch() does not return decoder_input_ids,  decoder_attention_mask as per document after passing tgt_texts
Author: MojammelHossain
Pull request: no

I am trying to train a seq2seq model using BartModel. As per BartTokenizer documentation if I pass tgt_texts then it should return decoder_attention_mask and decoder_input_ids please check the attachment for clarity. 
![image](https://user-images.githubusercontent.com/47693507/96223213-5d10af00-0faf-11eb-9253-5985990f15f9.png)
But I am only getting input_ids, attention_mask and labels.
![image](https://user-images.githubusercontent.com/47693507/96223257-71ed4280-0faf-11eb-9c8f-b26b7af8dd9a.png)


Comments:
I am facing the same issue and I noticed that the method indeed returns  the `["input_ids"]` of `tgt_texts` as labels. I think I could easily fix this to return both `input_ids` and `attention_mask` of `tgt_texts` (as `decoder_...`) but I noticed the same pattern in other seq2seq models, like T5. I am not sure what's the proper solution but if it is similar to what I suggest, than I'd be happy to make a pull request.

@LysandreJik I'd be happy to hear an opinion and start working on this.

I think https://github.com/huggingface/transformers/pull/6654/ and https://github.com/huggingface/transformers/issues/6624 are related - the PR changed `decoder_input_ids` to `labels`. Probably the documentation should be changed but I have to get more familiar with the respective issue and PR to be sure.

Thanks for the feedback @freespirit. Hopefully, they will update the documentation as it is a little bit confusing. But what I found that the modeling_bart.py file already handles the problem. _prepare_bart_decoder_inputs() and shift_tokens_right() solving that if I am not wrong. But I think I have to go deeper for understanding which I am trying to.
![image](https://user-images.githubusercontent.com/47693507/97541324-2c893600-19ef-11eb-9a3e-3a1057c70d71.png)
![image](https://user-images.githubusercontent.com/47693507/97541362-3c087f00-19ef-11eb-948b-842378cf0039.png)


Pinging @sshleifer for advice

@MojammelHossain is correct, the docs are wrong. 
The correct usage is to allow `_prepare_bart_decoder_inputs` to make `decoder_input_ids` and `decoder_attention_mask` for you. For training, you only need to pass the 3 keys returned by `prepare_seq2seq_batch`.
Unable to Tie Encoder Decoder Parameters When Using EncoderDecoderModel Constructor
Author: ncoop57
Pull request: no

## Environment info
     
- `transformers` version: 3.5.0
- Platform: Linux-5.4.0-53-generic-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
@patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): Encoder (RoBERTa) Decoder (GPT2) model

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

```
from transformers import (
    AutoConfig,
    AutoModel,
    AutoModelForCausalLM,
    EncoderDecoderModel,
    EncoderDecoderConfig,
    GPT2Config,
)

encoder_config = AutoConfig.from_pretrained('microsoft/codebert-base')
encoder = AutoModel.from_pretrained('microsoft/codebert-base')
decoder_config = GPT2Config(
    n_layer = 6,
    n_head = encoder_config.num_attention_heads,
    add_cross_attention= True,
)
decoder = AutoModelForCausalLM.from_config(decoder_config)
encoder_decoder_config = EncoderDecoderConfig.from_encoder_decoder_configs(encoder_config, decoder_config)
encoder_decoder_config.tie_encoder_decoder = True
shared_codebert2gpt = EncoderDecoderModel(encoder = encoder, decoder = decoder, config = encoder_decoder_config)
```

The tasks I am working on is: N/A

## To reproduce

Steps to reproduce the behavior:

Running the above code produces the following message:

```
The following encoder weights were not tied to the decoder ['transformer/pooler', 'transformer/embeddings', 'transformer/encoder']
```

When checking the number of parameters of the model produces a model with `shared_codebert2gpt: 220,741,632` parameters, which is the same number of parameters if I were to not attempt to tie the encoder and decoder parameters :(.

## Expected behavior

The above snippet should produce a model with roughly `172,503,552` parameters.

My big question is, am I doing this correctly? I can correctly tie the model parameters if I use the `EncoderDecoderModel.from_encoder_decoder_pretrained` constructor and pass `tie_encoder_decoder=True`. However, for my task, I don't want to use a pretrained decoder and so am unable to use this constructor.

Any help with this would be greatly appreciated!


Comments:
Oops realized it was because this weight tying doesn't work across different architectures.