noamaanMulla-03/transformers-issues huggingface dataset:
run_pl_glue.py token_type_id error on fresh install
Author: ethanjperez
Pull request: no
Labels: name: wontfix

If you try to run the run_glue.py example with e.g. roberta from a fresh install of the library, it errors out with the following error:

```
Traceback (most recent call last):
  File "examples/text-classification/run_pl_glue.py", line 228, in <module>
    main()
  File "examples/text-classification/run_pl_glue.py", line 218, in main
    trainer = generic_train(model, args)
  File "/home/ejp416/complexity/examples/lightning_base.py", line 400, in generic_train
    trainer.fit(model)
  File "/home/ejp416/miniconda3/envs/complexity2/lib/python3.7/site-packages/pytorch_lightning/trainer/states.py", line 48, in wrapped_fn
    result = fn(self, *args, **kwargs)
  File "/home/ejp416/miniconda3/envs/complexity2/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1072, in fit
    model = self.accelerator_backend.setup(model)
  File "/home/ejp416/miniconda3/envs/complexity2/lib/python3.7/site-packages/pytorch_lightning/accelerators/gpu_backend.py", line 34, in setup
    self.trainer.call_setup_hook(model)
  File "/home/ejp416/miniconda3/envs/complexity2/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1444, in call_setup_hook
    model.setup(stage_name)
  File "/home/ejp416/complexity/examples/lightning_base.py", line 175, in setup
    self.train_loader = self.get_dataloader("train", self.hparams.train_batch_size, shuffle=True)
  File "examples/text-classification/run_pl_glue.py", line 98, in get_dataloader
    all_token_type_ids = torch.tensor([f.token_type_ids for f in features], dtype=torch.long)
TypeError: an integer is required (got type NoneType)
```

To reproduce, run e.g.

```
python examples/text-classification/run_pl_glue.py --model_name_or_path roberta-base --output_dir ./blah --task mnli --do_train --data_dir ./glue_data/MNLI --max_seq_length 512 --max_grad_norm inf --adam_epsilon 1e-6 --weight_decay 0.1 --num_train_epochs 2 --train_batch_size 2 --eval_batch_size 4 --learning_rate 1e-5 --seed 12 --gradient_accumulation_steps 8 --gpus 1
```

The reason is that roberta does not have segment ids so token_type_ids is set to null in the data loader, causing torch.tensor to freak out. There's probably a more elegant long-term solution for this, but it's easy to fix by just setting it to 0 instead of null for those models. This issue has come up before in other scripts:
- https://github.com/huggingface/transformers/pull/3801
- https://github.com/huggingface/transformers/issues/3810

Comments:
Ah, indeed. Out of curiosity, have you tried using `run_glue.py` instead of `run_pl_glue.py`? Does the error still happen?

Closed by mistake.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
Allow the user to input positional embeddings
Author: anicolson
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

Hi, 

I think that allowing the user to input positional embeddings in the same way that `inputs_embeds` can be fed directly to the Transformer would be greatly appreciated. (If this is already possible, please let me know :) ).

## Motivation

This is important for multimodal approaches, such as **VisualBERT** (https://arxiv.org/pdf/1908.03557.pdf), where each modality requires a separate positional embedding. The user could then use a `torch.nn.Embedding` per modality and concatenate the positional embeddings and feed this to the Transformer along with the concatenated input embeddings of the modalities.

This would also resolve other related issues:

https://github.com/huggingface/transformers/issues/5095
https://github.com/huggingface/transformers/issues/3395

## Your contribution

I can help in any form. 


Comments:
Hey @anicolson,

I'm really not sure whether such a functionality is general enough to be added to the lib. It would also be very different from our current design of the library in that we would allow `torch.nn.Embedding` types as input, so I'd rather not add it. @LysandreJik what do you think? 

I think this is specifically where tweaking the library so that it supports your use-case is the way to go. It should be easy enough to modify the files directly in order to do this, but it would add unnecessary complexity to several model files.

Let's keep this issue open, and if other users are interested in this, we'll have a deeper look.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
Bertabs example: index_select(): Expected dtype int64 for index
Author: TheTimKiely
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.3.1
- Platform: Ubuntu 18.04
- Python version: 3.8.0
- PyTorch version (GPU?): torch==1.7.0
- Tensorflow version (GPU?):
- Using GPU in script?:yes
- Using distributed or parallel set-up in script?: no

### Who can help
@patil-suraj

## Information

Following the example in the seq2seq/bertabs readme.md.
I am getting this error:
```
File "/code/tools/transformers/examples/seq2seq/bertabs/modeling_bertabs.py", line 919, in _fast_translate_batch
    alive_seq = torch.cat([alive_seq.index_select(0, select_indices), topk_ids.view(-1, 1)], -1)
RuntimeError: index_select(): Expected dtype int64 for index
```
In a debugger, I see that the 'select_indices' parameter is a tensor of floats.

I don't understand the beam mechanism, so I don't know where to start troubleshooting this.

Any help would be great!

-Tim

Comments:
Hello! The `bert_abs` example is not maintained anymore, and should be moved to `examples/contrib/legacy`.

The recommended way of training sequence-to-sequence models is described in the `examples/seq2seq/README.md` file. What are you trying to do with `bertabs`, so that we may help you find what you need?

Hi!
Thanks for your response.
I'm just starting to experiment with abstractive text summarization.
Is this something I should look for in the Hugging Face tools and samples?
Thanks again,
Tim

I believe abstractive text summarization is implemented in the `seq2seq` examples, as the XSUM models were trained to do abstractive text summarization.

Have you taken a look at the summarization examples in https://github.com/huggingface/transformers/tree/master/examples/seq2seq?

@patil-suraj may also be of help.

Thanks again!
I’ll take a look at the seq2seq examples.  
-Tim

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
Pytorch Vs Onnx: Pytorch is faster and provides different output
Author: agemagician
Pull request: no
Labels: name: wontfix

## Environment info

- `transformers` version: 3.5.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help

albert, bert, GPT2, XLM: @LysandreJik 
Onnx: @mfuntowicz


## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

https://colab.research.google.com/drive/1UwgWgUF4k_GPJ5TcziHo4eH_rRFQeNVL?usp=sharing


## Expected behavior

I have followed the  the onnx export tutorial:
https://github.com/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb
However, I have found 2 issues:
1. Pytorch is faster than Onnx.
2. Onnx produce different embedding output than Pytorch.

Could anyone help me to figure out the issue ?

Comments:
I also got different results 🤦‍

I have figured out the problem, but I don't have the solution.
When you use a single sample per batch it works correctly, but when you use more than one sample per batch, the results are totally different.

@mfuntowicz and @LysandreJik , it will be great if you could show us an example for how to correctly use batch processing for onnx inference.

Hey @agemagician! I have exactly the same problem, great inference time with onnx when batch size = 1 but when batch size is increased, raw pytorch wins over onnx. Did you find any solution for the issue?

unfortunately, not.

That's unfortunate..

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
[s2s] Option to aggregate rouge deterministically
Author: sshleifer
Pull request: yes
Labels: name: wontfix

Optionally take randomness/sampling out of calculate_rouge_score.
Not breaking, the default is unchanged.

Comments:
This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
BertTokenizer loses unicode character
Author: arvieFrydenlund
Pull request: no
Labels: name: wontfix

## Environment info

- `transformers` version: 3.0.2
- Platform: Linux-4.15.0-109-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Who can help
 @LysandreJik 


## Information

The tokenizer seems to lose a specific unicode character on tokenization.  From the sentence

`General Saprang Kalayanamitr ( Thai : <unk> ่ ง กัลยาณมิตร ;`

from line 2557 of the Wiki02 training dataset, there is a little dot after and above `<unk>`
however the tokenizer produces 

` ['general', 'sap', '##rang', 'kala', '##yana', '##mit', '##r', '(', 'thai', ':', '<', 'un', '##k', '>', 'ง', '[UNK]', ';']`


## To reproduce

`t = BertTokenizer.from_pretrained('bert-base-uncased')`
`o = t.tokenize('General Saprang Kalayanamitr ( Thai : <unk> ่ ง กัลยาณมิตร ;')`
 `o`
`['general', 'sap', '##rang', 'kala', '##yana', '##mit', '##r', '(', 'thai', ':', '<', 'un', '##k', '>', 'ง', '[UNK]', ';']`

## Expected behavior

A subword should be given for the ' ่ ' token.  Otherwise there should be an option to be given a warning for removed characters.


Comments:
I'm having a similar problem since upgrading to 4.0 where certain unicode characters are being "eaten" even though I set 'use_fast' = False.

## Environment info

- `transformers` version: 4.0.0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.6
- PyTorch version (GPU?): 1.7.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
 
### Who can help
@mfuntowicz

## Information

BertTokenizer

The problem comes from certain unicode character being not turned correctly into UNK characters by the tokenizers when they did in earlier versions.

## To reproduce

```
import transformers
import torch
from transformers import BertTokenizer

print(torch.__version__)
print(transformers.__version__)

# THERE IS A UNICODE character between the , and '' (specifically \U+200D\U+200D\U+200D\U+200D)
sentence = ": ءُپَاعَر سِيْا , ‍‍‍‍ '' Upal"

tokenizer = BertTokenizer.from_pretrained(
            "bert-base-cased",
            do_lower_case=False,
            use_fast=False
        )

print(tokenizer.tokenize(sentence))

#  output 4.0.0

[':', '[UNK]', '[UNK]', ',', "'", "'", 'Up', '##al']

# output from 3.0.1

[':', '[UNK]', '[UNK]', ',', '[UNK]', "'", "'", 'Up', '##al'
```

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
MMI-antiLM decoding
Author: AADeLucia
Pull request: yes
Labels: name: wontfix

# What does this PR do?

Implements **Maximum Mutual Information anti-Language Model objective (MMI anti-LM)** decoding from [A Diversity-Promoting Objective Function for Neural Conversation Models](https://www.aclweb.org/anthology/N16-1014.pdf). 


Comments:
This technique looks really cool! Unfortunately running `generate` with two models will break lots of our assumptions, so maybe you could write a standalone:

```python
def generate_anti_lm(model, lm_model, **kwargs):
	... logic ...
    return generated_ids
```

,put it in `examples/anti_mlm_generation/`, and add a test that it runs with `sshleifer/tiny-gpt2`, for example?

Does that make sense to you @patrickvonplaten ?

Hey @AADeLucia - thanks for your PR! The `generate()` function is a very central part of the library and thus we have to be super careful when implementing new features. I agree with @sshleifer that MMI-antiLM decoding probably fits better in an example (has its own `generate()` function in an example file) to begin with - would that be ok for you?

> Hey @AADeLucia - thanks for your PR! The `generate()` function is a very central part of the library and thus we have to be super careful when implementing new features. I agree with @sshleifer that MMI-antiLM decoding probably fits better in an example (has its own `generate()` function in an example file) to begin with - would that be ok for you?

Thank you so much for your quick responses! Yes, putting it as its own example is okay with me.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
Fixing issue #7810
Author: lucadiliello
Pull request: yes
Labels: name: wontfix

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # 7810


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), 
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? 


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.
@LysandreJik you suggested to open this PR here: https://github.com/huggingface/transformers/issues/7810

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 

Comments:
This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
simple fix for spurious PyTorch->TF BERT weight conversion warning
Author: dslim23
Pull request: yes
Labels: name: wontfix


# What does this PR do?
Fixes https://github.com/huggingface/transformers/issues/7797

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [ X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), 
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? 


## Who can review?
 @LysandreJik 



Comments:
@LysandreJik yeah I think you're right, thanks! made the change

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
Correctly tokenize sentence pairs
Author: datistiquo
Pull request: no
Labels: name: wontfix

Hey,

I saw different ways to tokenize sentence pairs and the intuitive one is not shown here:

https://huggingface.co/transformers/preprocessing.html#preprocessing-pairs-of-sentences

So, I am asking here if I do right.

I encode pairs of sentence using list of lists. Instead handing over two seperate lists for each sentences, I handover a list of list, where each element is a list with  a single sentence pair. So:

pairs=[[sen1, sen2],[sen1,sen2],....]

Is this hopefully right too?

Comments:
This would be still interesting.

For tokenizing a list of pairs I get 

```
input_ids = tokenizer(pairs, max_length=50, padding="max_length",truncation=True, return_tensors="tf")

 'token_type_ids': <tf.Tensor: shape=(1, 50), dtype=int32, numpy=
array([[0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
        1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0],
....
```
So I wonder I am doing right as it seems the padded values are connetced to the 1. sentences (0 id)
SO token type IDS are 0 for the padded places, is that right?

Hey @datistiquo ,

As one can see in the following script:

```python
#!/usr/bin/env python3
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

input_ids = tokenizer([["hey hello there", "what is going on"], ["peter is in the", "what is going on"]], max_length=20, padding="max_length", truncation=True, return_tensors="tf")

print("List of pairs", input_ids)

input_ids = tokenizer("hey hello there", "what is going on", max_length=20, padding="max_length", truncation=True, return_tensors="tf")

print("Pair", input_ids)
```

tokenizing a list of pairs should be done exactly as proposed by you. Regarding the token_type_ids it is also correct that padded places should have a value of 0. In general if a model does not make use of `token_type_ids`, we return a 0 for such a model, see: https://github.com/huggingface/transformers/blob/6b034309ca4ca2ec6e5c3cacda92a448fa10b921/src/transformers/models/roberta/tokenization_roberta.py#L233 . So for padded tokens that should be discarded in the model, 0 seems like the most sensible choice to me.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
[broken] tf generate: use model_kwargs
Author: sshleifer
Pull request: yes
Labels: name: wontfix

@patrickvonplaten , I started trying to get tf generation/cache to be consistent with pytorch, but got stuck trying to get T5 working. I figured I would share in case you see an easy fix/want to take over. Otherwise, feel free to ignore :)

Comments:
This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
Checklist for Model Hub
Author: prajjwal1
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

I want to propose a checklist for model hub. Model hub is increasingly being used by `transformers` users. It is imperative that users exactly know what model they are downloading and importing for transparency and research-related reasons. It should be mandatory for users to upload a `REAME.md` while uploading model on model hub. Additionally, it will reduce the need to send a seperate PR for model cards.

Checklist should contain the following elements:
- Details of training and validation set used. Specify the split if not using default.
- A table with results on the validation set
- Model name (can be inferred from `config.json`)
- Objective used for training
- Hyperparameter configurations if default ones are not being used
- Link to implemented code (if available, and is different from default examples)
- For how many epochs the model was trained for ?
- Associated paper (if published)

These are just the general ones. Happy to hear about other suggestions. This is just a suggestion which would help many users. Thanks for this feature.

Comments:
Yes! This is something we'll build, step by step. I'll post more about our roadmap for this in the coming weeks.

> Yes! This is something we'll build, step by step. I'll post more about our roadmap for this in the coming weeks.

Maybe you can create a web-based interface with input fields/drop downs that can generate the model card automatically. Just a thought. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


posting so that the bot doesn't close this issue.

@prajjwal1 Have you checked out our new model storage/model versioning features on huggingface.co? You can also now edit your model card directly from the website (or from `git`) and we'll make this workflow more prominent (vs. adding model cards to `transformers`) in the coming days/weeks.

Feedback welcome, here or on the Forum.

No, I haven't. I am aware of it. I will check out soon. Loving the direction where HF is headed.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.