noamaanMulla-03/transformers-issues huggingface dataset:
Bug in squad example with XLNet
Author: zhiqihuang
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2
- Platform: Linux-3.10.0-957.21.2.el7.x86_64-x86_64-with-centos-7.6.1810-Core
- Python version: 3.7.4
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): 2.0.0 (False)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 examples/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...):
XLNet

The problem arises when using:
the official example scripts: (give details below)

The tasks I am working on is:
an official GLUE/SQUaD task: (give the name)


## To reproduce

Steps to reproduce the behavior:

1. run_squad.py with xlnet as the mode type
2. I think because the AutoModelForQuestionAnswering is mapping xlnet to XLNetForQuestionAnsweringSimple, it will have input error. Since XLNetForQuestionAnsweringSimple does not require cls_index, it will throw an error
3. https://github.com/huggingface/transformers/blob/d9149f00d1a4650bafa7e1cd73e10398193c852c/examples/question-answering/run_squad.py#L194

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior


<!-- A clear and concise description of what you would expect to happen. -->


Comments:
see also #3535

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Pretrained GPT2DoubleHeadsModel
Author: h56cho
Pull request: no
Labels: name: wontfix

#6667 
Hello,
This is a follow-up post for #6667.

So just so that I am understanding this correctly, the main body (excluding the 2 output heads) of the pre-trained `GPT2DoubleHeadsModel`  are indeed the pre-trained weights and biases, but the 2 output heads are not pre-trained?

Thank you,

Comments:
One of the two output heads is the language modeling head, which is tied to the embeddings. This is already trained, as the embeddings were trained during pre-training.

The second output head is a multiple choice head, which was not pre-trained. You would need to fine-tune it on a multiple choice dataset so that it works in your case.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Integer division of tensors using div or / is no longer supported torch
Author: dhruvrnaik
Pull request: no
Labels: name: wontfix

https://github.com/huggingface/transformers/blob/97bb2497abbbf978a0f78f1d414a7b45539e795b/examples/seq2seq/bertabs/modeling_bertabs.py#L885

Line throws an error "integer division of tensors using div or / is no longer supported torch" when executing `bertabs/run_summarization.py`

Is replacing `.div()` with `.floor_divide()` the correct solution here?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Sequence packing
Author: saareliad
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

Add sequence packing support

## Motivation

Faster training, higher utilization, replicate experiments. 
See https://github.com/google-research/text-to-text-transfer-transformer/issues/365

## Your contribution

I think it makes sense doing something similar to other frameworks which already have this implemented (e.g https://github.com/tensorflow/mesh/blob/6a812c8bb847e081e976533ed497c7c5016bb1ec/mesh_tensorflow/transformer/dataset.py#L474-L504 )

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to use 16 token types in pretrained Albert/BERT?
Author: Traeyee
Pull request: no
Labels: name: wontfix

I have a dialogue task and I use token type to distinguish the diffenrent state of the different speeches, but all the pretrained models I can find are of type_vocab_size=2. To accomplish my goal, I have to rewrite many codes in a dirty way. So I want to ask is there an elegant way to restore the pretrained weights and ignore the token type embeddings at the same time? Roughly modifying the type_vocab_size in the given config.json will certainly raise an error

Comments:
Maybe I don't get it, but can't you simply do:

```
from transformers.modeling_bert import BertConfig, BertModel

bconfig = BertConfig.from_pretrained('bert-base-uncased')
bconfig.type_vocab_size = 16
model = BertModel(bconfig)
model.parameters
# ...
#    (token_type_embeddings): Embedding(16, 768)
# ...
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> Maybe I don't get it, but can't you simply do:
> 
> ```
> from transformers.modeling_bert import BertConfig, BertModel
> 
> bconfig = BertConfig.from_pretrained('bert-base-uncased')
> bconfig.type_vocab_size = 16
> model = BertModel(bconfig)
> model.parameters
> # ...
> #    (token_type_embeddings): Embedding(16, 768)
> # ...
> ```

Yes. What I can do is to reassign the token type embeddings after init, and thing is if there is any risk to do this. But I don't continue on this because my dialogue task is too difficult for almost language model even with BERT hahhhhh

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Minor style improvements for the Flax BERT and RoBERTa examples
Author: avital
Pull request: yes

1. Use `@nn.compact` rather than `@compact` (as to not make it seem
   like compact is a standard Python decorator.
2. Move attribute docstrings from two `__call__` methods to comments
   on the attributes themselves. (This was probably a remnant from
   the pre-Linen version where the attributes were arguments to
   `call`.)

# What does this PR do?

    Minor style improvements:
    
    1. Use `@nn.compact` rather than `@compact` (as to not make it seem
       like `compact` is a standard Python decorator.
    2. Move attribute docstrings from two `__call__` methods to comments
       on the attributes themselves. (This was probably a remnant from
       the pre-Linen version where the attributes were arguments to
       `call`.)


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? No. It's just adjusting the Flax example to the current best practices (I work on Flax)
- [x] Did you make sure to update the documentation with your changes? No doc changes.Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).

It's not clear what the right pattern is for docstrings of dataclass attributes. I went with something pragmatic here. I couldn't find any online references for the "correct Pythonic pattern" here -- LMK if there's another form you prefer.

- [x] Did you write any new necessary tests? No new tests. Exists tests pass.

## Who can review?

Comments:
cc @LysandreJik @mfuntowicz 

Offline approval from @mfuntowicz!
Does to T5 Transformer training scale to multiple GPUs?
Author: abhisheknovoic
Pull request: no
Labels: name: wontfix

Hello team,

  I have a large set of sequence to sequence dataset. Basically, a huge bunch of input text sequences to output text sequences. I want to train a T5 network on this. I have the following specific questions.

a. Can I use the sample code here (along with my own code) to train T5 on my data? - https://huggingface.co/transformers/model_doc/t5.html

b. Will that automatically scale to multiple GPUs? What if I want to further scale to tens of GPUs across different machines. Does HuggingFace support that?

Thanks


Comments:
Hi @abhisheknovoic , you can use T5 on multiple GPU's. Have a look at this community notebook https://github.com/patil-suraj/exploring-T5/blob/master/t5_fine_tuning.ipynb.

It uses pytorch-lightning, so its very easy to setup multi-gpu training.See this guide https://pytorch-lightning.readthedocs.io/en/latest/multi_gpu.html

@patil-suraj , thanks for the reference. I will take a look at it. Just to confirm, if I use the code as is in the notebook, will it run on multiple GPUs, or do I need to learn a bit about pytorch-lightning and then make some more changes for multi GPU support? Thanks Suraj !

You won't need to make any changes to the code, you'll just to specify number of gpus when initialising  lightning  trainer. You can check their multi-gpu docs for more info 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> Hi @abhisheknovoic , you can use T5 on multiple GPU's. Have a look at this community notebook https://github.com/patil-suraj/exploring-T5/blob/master/t5_fine_tuning.ipynb.
> 
> It uses pytorch-lightning, so its very easy to setup multi-gpu training.See this guide https://pytorch-lightning.readthedocs.io/en/latest/multi_gpu.html

Hello, Suraj. Is there any way to to achieve model parallelism using your solution? Or maybe this tool https://towardsdatascience.com/model-parallelism-in-one-line-of-code-352b7de5645a can be applied to your code?

Hi @patil-suraj  I am trying to use your notebook with pytorch lightening for multiple tasks, in this case one have multiple eval dataloaders, and metrics are also multiple, do you have an idea how I can extend your notebook for multiple tasks to handle this? thanks a lot
[Bug fix] Fixed value for BlenderBot pad token
Author: guillaume-be
Pull request: yes

# What does this PR do?

The current `BlenderbotSmallTokenizer` has an incorrect (probably a typo) value for the `pad_token`. This causes the BlenderBot model to crash on padded sequences (currently pads with a value that exceeds the embedding matrix size).

This PR fixes the behaviour and the tokenizer now pads correctly with `0`.

## Who can review?

 Blenderbot, Bart, Marian, Pegasus: @sshleifer


Comments:
How to get translation of one batch of sentences after batch_encode_plus? 
Author: OOF-dura
Pull request: no

```
model = AutoModelWithLMHead.from_pretrained("Helsinki-NLP/opus-mt-es-en")
tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-es-en")
batch_input_str = (("Mary spends $20 on pizza"), ("She likes eating it"), ("The pizza was great"))
encoded = (tokenizer.batch_encode_plus(batch_input_str, pad_to_max_length=True))
```

The ```encoded```is like:
```
{'input_ids': [[4963, 10154, 5021, 9, 25, 1326, 2255, 35, 17462, 0], [552, 3996, 2274, 9, 129, 75, 2223, 25, 1370, 0], [42, 17462, 12378, 9, 25, 5807, 1949, 0, 65000, 65000]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 0, 0]]}
```

Then, should I just pass the ```encoded``` to 
```
output = model.generate(a)
```
And then use
```
res = tokenizer.decode(output)
```


Thanks!

Comments:
Hello, have you read the docs concerning the translation task? It is [available here](https://huggingface.co/transformers/task_summary.html#translation).

Since you're specifically asking about a Helsinki model, you can find the documentation, with examples, [here](https://huggingface.co/transformers/model_doc/marian.html#multilingual-models).
Roberta's Positional Embedding Offset
Author: h324yang
Pull request: no
Labels: name: wontfix

https://github.com/huggingface/transformers/blob/d4c2cb402d6674211726fd5f4803d1090664e438/src/transformers/modeling_bart.py#L754

https://github.com/huggingface/transformers/blob/d4c2cb402d6674211726fd5f4803d1090664e438/src/transformers/modeling_bart.py#L763

So this offset is added because the function `create_position_ids_from_input_ids` shifts the position ids by padding_idx + 1. However, I wonder if other models should also include this? 

https://github.com/huggingface/transformers/blob/d4c2cb402d6674211726fd5f4803d1090664e438/src/transformers/modeling_roberta.py#L54
For instance, when I am using `Longformer`, it looks like the offset is not added to `Roberta`, so I need to add such a offset to config.max_position_embeddings



Comments:
That's certainly possible. As you can see from my comment, and PR #5188 , I don't fully understand the motivation for the offset. It is very tricky.

I figured out why. See here https://github.com/pytorch/fairseq/issues/1177
So basically the purpose is to make positional embedding = 0 on padding positions (positions where token is padding token), using the `padding_idx` parameter in torch.nn.Embedding. 

I think we can simply use masked_fill() to make positional embedding = 0 on padding positions, so the code is easier to understand (no need for the offset).

Exactly! 
Would love to do that, but the migration of the existing bart state dicts is non trivial, since they already store the extra position embedding. Even if we tracked down all bart models with `config.static_position_embeddings=False` and resized their positional embeddings, we would break code that is not up to date w master (lots of code).

So I think we must settle for documenting what is going on better in `LearnedPositionalEmbedding` and accept the unfortunate reality that we are stuck with the offset forever (or until we have some futuristic model hub tooling to version state dicts).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
FillMaskPipeline return special tokens i.e. <mask> as prediction
Author: hungluumfc
Pull request: no
Labels: name: wontfix

# ❓ FillMaskPipeline return special tokens i.e. \<mask\> as prediction


## Details
Im training a new language model from scratch using ByteLevelBPETokenizer for tokenizer and RobertaForMaskedLM for self-supervised language model. The config is as follow:

Tokenizer:

```
tokenizer = ByteLevelBPETokenizer()

tokenizer.train(files=paths, vocab_size=100000, min_frequency=2, special_tokens=[
    "
",
    "<pad>",
    "
",
    "<unk>",
    "<mask>",
])
```

After the tokenizer was trained, it was loaded using

`tokenizer = RobertaTokenizerFast.from_pretrained("./BERTmese/Tokenizer", max_length=512)`

And the RobertaForMaskedLM  is configed as follow:

```
config = RobertaConfig(
    vocab_size=100000,
    max_position_embeddings=514,
    num_attention_heads=12,
    num_hidden_layers=6,
)
model = RobertaForMaskedLM(config=config)
```

Then I train the RobertaForMaskedLM  model using custom dataset and Adam optimizer as follow: 
```
class CustomTextDataset(Dataset):
    def __init__(self, tokenizer, dataset, max_length):
        self.tokenizer = tokenizer
        self.dataset = dataset
        self.max_length = max_length

    def __len__(self):
        return len(self.dataset)

    def __getitem__(self, i) -> torch.Tensor:
        encoding = self.tokenizer(self.dataset[i]['text'],
                                add_special_tokens=True, 
                                max_length=self.max_length, 
                                padding='max_length',
                                truncation=True)
        return torch.tensor(encoding["input_ids"], dtype=torch.long)
```
```
for step in data_loader:
    input_ids = next(iter(train_loader))
    input_ids = input_ids.to(device)

    outputs = model(input_ids, labels=input_ids)
    loss, _ = outputs[:2]
    loss.backward()
    ...
```
For every 100 steps, I test the trained model using this code snippet:
```
model.eval()
# test on a sample text

fill_mask = FillMaskPipeline(
    model=model,
    tokenizer=tokenizer,
    topk=1
)

mask = fill_mask("ယခုလတွင်ပျားရည်နှင့်ပျားဖယောင်းများကိုစုဆောင်း<mask>သည်ဟုခန့်မှန်းနိုင်သည်။")[0] 

print(mask['token'], decoder.decode([mask['token_str']]), mask['score'])

# set model back to train mode

model.train()
```
In the first hundred steps, the predict token is fined (it predict some Burmese word). However, after that, the model start to predict the '\<mask\>' token as the result. Im not sure if predicting \<mask\> token for masked language is normal?


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
src/transformers/trainer.py relies on path to infer global training steps, skips training for glue example
Author: alexeifigueroa
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): bert-base-uncased that has been retrained stored somewhere else, with a timestamp in the path

Language I am using the model on (English, Chinese ...):EN

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name) MRPC
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. store a pretrained model with a different name, say /funny/path/2020-06-19_120000/retrained , 
with the tokenizer and stuff inside.
2. Run the glue example run_glue on that path with --do_train, --do_eval
3. The training will be skipped because there's [this](https://github.com/huggingface/transformers/blob/84be482f6698fac822a5113735f2242c6d3abc76/src/transformers/trainer.py#L431) funny line in the trainer that will bump the global steps because of the parsing of the date.. 

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
I would not expect the trainer to rely on an arbitrary internal convention for naming checkpoints or any kind of path to yield a spurious global step count.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.11.0
- Platform: Linux-5.4.0-29-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.5
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?):
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
I have the same problem. The script shouldn't run differently depending on what naming scheme you decide to use for a model.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
fine tuning with Chinese data LCQMC val_acc not increase
Author: ares5221
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
hello everyone
when i use huggingface bert-base-chinese model fine tuning LCQMC dataset ,
the train_acc will raise normal like 0.6->0.8->0.9,but val_acc not increase, it alway same number like 0.56
do you konw what happent?

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.