noamaanMulla-03/transformers-issues huggingface dataset:
Set num_beams=4 for all Helsinki-NLP models
Author: sshleifer
Pull request: no
Labels: name: marian

Currently it is 6. Empirically, I tested 77 random models and num_beams=4 was about 50% faster and on average slightly higher BLEU (22.5 vs 22.4). 

We also have @jorgtied 's approval for the change.
On slack, he wrote

>  Again, no systematic evaluation - more like a feeling. I had the impression that 1 or 2 is worse and I didn’t want to set 10 or 12 that I have seen otherwise, because it may slow down things quite substantially. If you make some more tests then let me know what you will find … Thanks!


There are about 1300 affected model's, so this feels like the type of thing @patrickvonplaten 's script could do well.

Comments:
I can do the change! Will wait until the new git model hub is merged and then apply it :-) 

Thanks!


Should be decently easy to do it @patrickvonplaten let me know when you get to it :)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@julien-c, @jorgtied - sorry forgot about this issue...changing `num_beams=4` for all opus models now.
Remove duplicated extras["retrieval"]
Author: n1t0
Pull request: yes

The `extras["retrieval"]` is defined a few lines above as:
https://github.com/huggingface/transformers/blob/28b26013abea3a49afeb46d36993a568ec98f39e/setup.py#L217-L222
and then it seems to be overridden just below, probably linking to `faiss-cpu` being included even on windows.

This PR removes the second assignment. 

cc @LysandreJik @sgugger @stas00 

Comments:
Error in GPT2 while using gradient checkpointing.
Author: devrimcavusoglu
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.2.0
- Platform: Linux | 5.4.0-60-generic | 18.04.1-Ubuntu SMP | x86_64
- Python version: 3.7.7
- PyTorch version (GPU?): 1.7.1
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
 @LysandreJik

## Information

Model I am using: GPT2

The problem arises when using:
* GPT2LMHeadModel with config `gradient_checkpointing: True` 

When using GPT2 pretrained model, with the latest releases (4.x), `gpt2_modeling.py` fails due to the behavior arising from pytorch. It was due to the fact that `torch.utils.checkpoint.checkpoint` import is malfunctioning, see [this](https://discuss.pytorch.org/t/attributeerror-module-torch-utils-has-no-attribute-checkpoint/101543) discussion, but I tried with python3.8 as well, and the problem still occured. When I observe the other scripts for modeling on different models (like BERT, etc.) the import statement for `checkpoint` is handled successfully, but GPT2 script fails. It is discussed that the problem arises due to the import behaviour of python.

```
    File
    "/home/username/.miniconda3/envs/prj/lib/python3.7/site-packages/torch/nn/modules/module.py", line
    727, in _call_impl
    result = self.forward(*input, **kwargs)
    File
    "/home/username/.miniconda3/envs/prj/lib/python3.7/site-packages/transformers/models/gpt2/modeling_gpt2.py", line
    901, in forward
    return_dict = return_dict,
    File
    "/home/username/.miniconda3/envs/prj/lib/python3.7/site-packages/torch/nn/modules/module.py", line
    727, in _call_impl
    result = self.forward(*input, **kwargs)
    File
    "/home/username/.miniconda3/envs/prj/lib/python3.7/site-packages/transformers/models/gpt2/modeling_gpt2.py", line
    728, in forward
    outputs = torch.utils.checkpoint.checkpoint(
            AttributeError: module 'torch.utils' has no attribute 'checkpoint'
```

## Suggestion
in `modeling_gpt2.py` add this import `import torch.utils.checkpoint`.

Comments:
Hitting this issue as well.
Tokenizstion
Author: Syavaprd
Pull request: no

Hi all, help me please. I'm trying to solve multilingual WiC problem using XLM-R. 
I have a word and a sentence, that has this word, but may be in different form. So I want to find the position of its token ids in an encoded sentence. 
There problems with Chinese, cause they don't have space between many symbols and tokenizer can encode a pair of symbols differently.

Comments:
So in short:
- in public open-source projects like this you can say "Hi all" or "Hi folks", this way you'll address contributors of all genders
- also, do you mind closing this issue and opening a thread on the forum at https://discuss.huggingface.co? We keep the issues for bug reports and feature requests which this issue is not.
- last note for your future issues on open-source projects: when there is an issue template like here you should fill it that's typically the first thing the maintainers will ask you if you didn't do it.

This looks like a nice project good luck with it and I hope you'll succeed!
Odd predictions of T5 models in recent versions 
Author: danyaljj
Pull request: no

We are seeing odd predictions by T5 models ([UnifiedQA models](https://github.com/allenai/unifiedqa/)) when using the recent HF version (4.2.1).  Here is the discussion: https://github.com/allenai/unifiedqa/issues/11 

### Who can help
@TevenLeScao @patrickvonplaten

## To reproduce
Try running the following script: 
```python 
from transformers import AutoTokenizer, T5ForConditionalGeneration

model_name = "allenai/unifiedqa-t5-small" # you can specify the model size here
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = T5ForConditionalGeneration.from_pretrained(model_name)

def run_model(input_string, **generator_args):
    input_ids = tokenizer.encode(input_string, return_tensors="pt")
    res = model.generate(input_ids, **generator_args)
    return [tokenizer.decode(x) for x in res]

run_model("which is best conductor? \\n (a) iron (b) feather")
```

 - For `transformers==4.2.1`,  I am getting `['<pad> iron
']`, which is not good. 
 - However, `transformers==3.5.1`and `transformers==3.1.0` give me `['iron']`which is the expected response. 



Comments:
Hello @danyaljj,

This is expected actually. Could you change your code as follows to get the previous results:

```python
from transformers import AutoTokenizer, T5ForConditionalGeneration

model_name = "allenai/unifiedqa-t5-small" # you can specify the model size here
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = T5ForConditionalGeneration.from_pretrained(model_name)

def run_model(input_string, **generator_args):
    input_ids = tokenizer.encode(input_string, return_tensors="pt")
    res = model.generate(input_ids, **generator_args)
    return tokenizer.batch_decode(res, skip_special_tokens=True)

run_model("which is best conductor? \\n (a) iron (b) feather")
```

Thanks for the quick reply! The new code works! Thanks!
A layman wants to train DistilBERT
Author: WalterZhong
Pull request: no
Labels: name: wontfix

    I want to train DistilBERT, but I don't  know how to get the training data.
In the article, it describes the training data as a concatenation of Toronto Book Corpus and English Wikipedia (same training data as the English version of BERT). So how can I get it?
![image](https://user-images.githubusercontent.com/38905736/98760883-378c8f00-240f-11eb-881a-2048e2967617.png)


[https://github.com/huggingface/transformers/blob/1ab8dc44b3d84ed1894f5b6a6fab58fb39298fc7/examples/distillation/README.md](url)

Comments:
[https://github.com/google-research/bert](url)

In this url, I find some files.
![image](https://user-images.githubusercontent.com/38905736/98761374-3c057780-2410-11eb-8028-4f896e86aa24.png)

But I don't whether they are the data which used to be the training data of distilbert.
And the distilbert model need "dump.txt ", not ".json", whether the data we need is included in it?

I find a text in the goole project bert.
[https://github.com/google-research/bert/blob/master/sample_text.txt](url)
Is it the training data of this project?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to print out the probability for each bean search result in gpt2 text generator?
Author: beibeic
Pull request: no
Labels: name: wontfix

# 🚀 I want to see the probability for the text result generated by gpt2 model 
<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->


See a sample code as below, when we used a pre-trained model,  can we also print out the probability outcome with each beam_output? 

beam_outputs = model.generate(
    input_ids, 
    max_length=50, 
    num_beams=5, 
    no_repeat_ngram_size=2, 
    num_return_sequences=5, 
    early_stopping=True
)

for i, beam_output in enumerate(beam_outputs):
  **print("{}: {}".format(i, tokenizer.decode(beam_output, skip_special_tokens=True)))**


<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->



<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Comments:
Take a look at https://github.com/huggingface/transformers/issues/5164

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Gradient accumulation averages over gradients
Author: MarktHart
Pull request: no
Labels: name: wontfix

https://github.com/huggingface/transformers/blob/121c24efa4453e4e726b5f0b2cf7095b14b7e74e/src/transformers/trainer.py#L1118

So I have been looking at this for the past day and a half. Please explain to me. Gradient accumulation should accumulate the gradient, not average it, right? That makes this scaling plain wrong? Am I missing something? 

Comments:
Hi @MarktHart do you mind posting this on the forum rather? It's here: https://discuss.huggingface.co

We are trying to focus the issues on bug reports and features/model requests.

Thanks a lot.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Error w/Transformers 4.2.0 and TF Nightly
Author: ANarayan
Pull request: no

@jplu I am running into issues when running transformers w/tf-nightly. 

I get the error when I am trying to load the TFDistilBERT model:

`model = TFDistilBertModel.from_pretrained('distilbert-base-uncased')`

this is the error message:
```
ImportError: 
TFDistilBertModel requires the TensorFlow library but it was not found in your environment. Checkout the instructions on the
installation page: https://www.tensorflow.org/install and follow the ones that match your environment.
```

I came across this bug when running a CI test for Ludwig. I think many projects use tf-nightly in their CIs tests to make sure that the integrations are future proof!

Comments:
Hello!

Please update to v4.2.1

This worked, thanks!
Missing `return_dict` in Doc example
Author: arnaudsm
Pull request: yes

# What does this PR do?
Fixes a crash in [Summary of the tasks](https://huggingface.co/transformers/task_summary.html) documentation, by adding `return_dict=True` to the model() function, as we need `start_logits` and `end_logits` afterwards.

## Fixes
[Issue 9043](https://github.com/huggingface/transformers/issues/9043) (reproduced in 4.2.1)

## Who can review?
@LysandreJik @sgugger


Comments:
Hello! Are you sure you checked in v4.2.1? I just checked on both `master` and v4.2.1 and the code executes (as it should!).

The `return_dict` was set to be `True` by default in v4.0.0.

My bad, I was in the wrong pyenv. Closing the PR.
Tapas v4 (tres)
Author: NielsRogge
Pull request: yes
Labels: name: model card

name: PR for Model Addition

Here we are again, opening a new PR based on the former (#8988) which had some Github issues. 
cc @LysandreJik 

Comments:
Don't worry about the TF tests, these are because of TF2.4 which are fixed on `master`.

The conversion script currently includes a line in which I'm importing a local `vocab.txt`, I know this should be removed in the future.
jplu/tf-xlm-roberta-large showing random performance
Author: mobassir94
Pull request: no
Labels: name: wontfix

# 🐛 jplu/tf-xlm-roberta-large showing random performance on tpu model training

## i have observed several times that jplu/tf-xlm-roberta-large showing random performance on several experiments,for example if i train a model and get validation accuracy 61% then if i run the model again (without changing anythin or doing 0% change in code) then second i get over 96% accuracy and sometimes i observed that validation accuracy stay same for all epoch and again this same code will get me totally different train statistics next time,why is this happening?

Model I am using jplu/tf-xlm-roberta-large

Language I am using the model on (English, Chinese ,turkish,spanish etc):

The problem arises when using:
jplu/tf-xlm-roberta-large for finetuning 

The tasks I am working on is:
kaggle ongoing competition : jigsaw multilingual toxic comment classification where train set contails all english comments and 0/1(toxic/non toxic) label and validation+test data contains non English comments

## To reproduce

Steps to reproduce the behavior: 

1. just simply fork and run this kernel : https://www.kaggle.com/mobassir/understanding-cross-lingual-models and you will never get same result and probably will get bit more worst result



## Expected behavior

same model training several times should always produce close result if not exact same, the randomness is high,,sometimes model with 91% validation accuracy can get us 61% validation accuracy on next run and sometimes validation accuracy of that same model get frozen and seems like the model is not learning anything (i repeat the model name : it's tf xlm roberta large)

## Environment info
kaggle kernels/notebooks using tpu
     



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


same here, mine is torch version, though.
@mobassir94 Did you fix it?

@HenryPaik1 no
i also faced this issue in both pytorch xla and in tf tpu
bert_tokenizer.decode(bert_tokenizer.encode(sentence))!=sentence
Author: youngornever
Pull request: no

from transformers import AutoTokenizer       # transformers==4.2.1
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
paragraph = "no_passages_used __knowledge__ no_passages_used"
print(tokenizer.encode(paragraph))
print(tokenizer.decode(tokenizer.encode(paragraph)))

"""
>[101, 2053, 1035, 13768, 1035, 2109, 1035, 1035, 3716, 1035, 1035, 2053, 1035, 13768, 1035, 2109, 102]
>[CLS] no _ passages _ used _ _ knowledge _ _ no _ passages _ used [SEP]
"""

Comments:
Hi! This is a normal behavior of the BERT tokenizer. You can add the tokens you do not wish to see split to the vocabulary:
```py
>>> tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
... tokenizer.add_tokens(["no_passages_used"]) # <--------------------------------- Here
... paragraph = "no_passages_used knowledge no_passages_used"
... print(tokenizer.encode(paragraph))
... print(tokenizer.decode(tokenizer.encode(paragraph)))
[101, 30522, 3716, 30522, 102]
[CLS] no_passages_used knowledge no_passages_used [SEP]
```

Don't forget to resize the embedding matrix of your model if you add new tokens to the vocabulary: [docs for add_tokens method](https://huggingface.co/transformers/internal/tokenization_utils.html?highlight=resize_token_embeddings#transformers.tokenization_utils_base.SpecialTokensMixin.add_tokens)
Fix imports in conversion scripts
Author: sgugger
Pull request: yes

# What does this PR do?

During the rework of the new init for fast imports, all absolute imports were switched to relative ones indiscriminately (because they usually don't work anymore for the core of the lib). However, the conversion scripts are supposed to be executed as scripts and relative imports can't work there (that's how Python works). This PR fixes those, and it seems that it doesn't hurt the transformers-cli convert command (which import things from those modules).

Comments:
Fix TF Flaubert and XLM
Author: jplu
Pull request: yes

# What does this PR do?

By doing some experiments on Flaubert and XLM I realized that building a model with a `None` argument forces this value to be `None`  when served. Then to fix this issue, the build takes a proper input without a `None` value.


Comments:
I'm checking what is going wrong as the tests of equivalence should not fail.

Ok, now it works 👍
TypeError: an integer is required (got type NoneType)
Author: parthplc
Pull request: no

While using trainer class to finetune gpt-2 on Hindi dataset. It's outputting the following error
```
TypeError                                 Traceback (most recent call last)
<ipython-input-44-3435b262f1ae> in <module>()
----> 1 trainer.train()

5 frames
/usr/local/lib/python3.6/dist-packages/transformers/data/datasets/language_modeling.py in __getitem__(self, i)
     99 
    100     def __getitem__(self, i) -> torch.Tensor:
--> 101         return torch.tensor(self.examples[i], dtype=torch.long)
    102 
    103 

TypeError: an integer is required (got type NoneType)
```
Here is the link: https://colab.research.google.com/drive/1um5UeY9hasmjPNcR1WkBe2uDDFhLUBrX?usp=sharing


Comments:
Hey @parthplc,

Could you try to post a very short code snippet that can reproduce the error. It's too time-consuming to go through such a big notebook sadly

@parthplc Please post a solution if you close the issue. There is none in your colab as far as I can see (`model.resize_token_embeddings(len(tokenizer))` is the only relevant code I can see and it doesn't fix the problem).
AttributeError: 'Seq2SeqTrainer' object has no attribute '_actual_model'
Author: caralen
Pull request: no

_actual_model method is not defined in the Seq2SeqTrainer class, nor in the Trainer class from which is derived 

https://github.com/huggingface/transformers/blob/12c1b5b8f448d652f5e1fa0f069b9569f4540948/examples/seq2seq/seq2seq_trainer.py#L63

Comments:
Hi @caralen 

the `Seq2SeqTrainer` is now integrated with the main lib, now it's under `src/trainer_seq2seq.py`, and the seq2seq_trainer in examples is about to be deprecated, this bug is fixed in the new version, so I would recommend you to use the new`Seq2SeqTrainer` from the lib rather than examples folder,

you could directly import it from transformers using
```python
from transformers import Seq2SeqTrainer
```

Hi @patil-suraj, thanks for the quick reply. I will close this issue now.
Missing argument: decoder_head_mask for T5
Author: stancld
Pull request: no

# 🚀 Feature request

Despite the encoder-decoder architecture of T5, the models use a single `head_mask` argument instead of having separate `head_mask` and `decoder_head_mask` as it will be for BART-based models after merging the PR #9569.

## Your contribution

I'm going to send a PR soon. (I'll try to prepare this feature both for PyTorch and TensorFlow in two separate PRs.)

## Reviewer
@patrickvonplaten 

Comments:
Solved in #9634.
Restrain tokenizer.model_max_length default
Author: sgugger
Pull request: yes

# What does this PR do?

Apply the same fix to `run_mlm` (when line_by_line is not selected) as we did previously in `run_clm`. Since the tokenizer model_max_length can be excessively large, we should restrain it when no `max_seq_length` is passed.

Fixes #9665

Comments:
Generating sentence embeddings from pretrained transformers model
Author: Prithvi103
Pull request: no

Hi, I have a pretrained BERT based model hosted on huggingface. 
https://huggingface.co/microsoft/SportsBERT

How do I generate sentence vectors using this model? I have explored sentence bert but it doesn't allow you to use custom trained models. I have also seen Bert as a client. It works but for my current scenario, I was wondering if there's something which could be done without running a server for converting to vectors.

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!
SQuAD 2.0 metric not supported
Author: yonatanbitton
Pull request: no

Hello.
I'm trying to run the official `run_qa.py` code for SQuAD 2.0.

You have an open TODO here that is causing a bug: https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_qa.py#L436

I would like to know what is the status of this TODO, and if it is going to be updated, or is there a way around it.

This is the current code:

```python
    current_dir = os.path.sep.join(os.path.join(__file__).split(os.path.sep)[:-1])
    metric = load_metric(os.path.join(current_dir, "squad_v2_local") if data_args.version_2_with_negative else "squad")
```

I receive: 
```
FileNotFoundError: Couldn't find file locally at .../squad_v2_local/squad_v2_local.py,
```

I've tried to change it to: 
```python
metric = load_metric("squad_v2" if data_args.version_2_with_negative else "squad")
```

But this is the stacktrace I receive: 
```
Traceback (most recent call last):
  File "/data/users/yonatab/transformers_pip/QA/run_qa_val_more_valueable.py", line 557, in <module>
    main()
  File "/data/users/yonatab/transformers_pip/QA/run_qa_val_more_valueable.py", line 538, in main
    results = trainer.evaluate()
  File "/data/users/yonatab/transformers_pip/QA/trainer_qa.py", line 63, in evaluate
    metrics = self.compute_metrics(eval_preds)
  File "/data/users/yonatab/transformers_pip/QA/run_qa_val_more_valueable.py", line 499, in compute_metrics
    return metric.compute(predictions=p.predictions, references=p.label_ids)
  File "/data/users/yonatab/transformers_pip/trans_pip/lib/python3.6/site-packages/datasets/metric.py", line 398, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/ec2-user/.cache/huggingface/modules/datasets_modules/metrics/squad_v2/7529efd518b03f775290694e7b797412cb2253e90b4f843af83cf7434cccb3a8/squad_v2.py", line 108, in _compute
    exact_raw, f1_raw = get_raw_scores(dataset, predictions)
  File "/home/ec2-user/.cache/huggingface/modules/datasets_modules/metrics/squad_v2/7529efd518b03f775290694e7b797412cb2253e90b4f843af83cf7434cccb3a8/evaluate.py", line 111, in get_raw_scores
    gold_answers = [a["text"] for a in qa["answers"] if normalize_answer(a["text"])]
  File "/home/ec2-user/.cache/huggingface/modules/datasets_modules/metrics/squad_v2/7529efd518b03f775290694e7b797412cb2253e90b4f843af83cf7434cccb3a8/evaluate.py", line 111, in <listcomp>
    gold_answers = [a["text"] for a in qa["answers"] if normalize_answer(a["text"])]
TypeError: string indices must be integers
100%|███████████████████████████████████████████| 13/13 [00:05<00:00,  2.51it/s]
```

How can I solve it? 

Thanks

Comments:
@sgugger would know about this TODO; I think the fix has landed in `datasets`, right?

Yes, this should be fixed directly from `datasets` now, will update the script this afternoon.