noamaanMulla-03/transformers-issues huggingface dataset:
MRPC Reproducibility with transformers-4.1.0
Author: Linwenye
Pull request: no

I always get lower precision following the MRPC example, what's the reason?
```
python run_glue.py \
  --model_name_or_path bert-base-cased \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3.0 \
  --output_dir /tmp/$TASK_NAME/
```
and get 
```
12/18/2020 17:16:38 - INFO - __main__ -   ***** Eval results mrpc *****
12/18/2020 17:16:38 - INFO - __main__ -     eval_loss = 0.5318707227706909
12/18/2020 17:16:38 - INFO - __main__ -     eval_accuracy = 0.7622549019607843
12/18/2020 17:16:38 - INFO - __main__ -     eval_f1 = 0.8417618270799347
12/18/2020 17:16:38 - INFO - __main__ -     eval_combined_score = 0.8020083645203595
12/18/2020 17:16:38 - INFO - __main__ -     epoch = 3.0

12/18/2020 16:45:29 - INFO - __main__ -   ***** Eval results mrpc *****
12/18/2020 16:45:29 - INFO - __main__ -     eval_loss = 0.47723284363746643
12/18/2020 16:45:29 - INFO - __main__ -     eval_accuracy = 0.8063725490196079
12/18/2020 16:45:29 - INFO - __main__ -     eval_f1 = 0.868988391376451
12/18/2020 16:45:29 - INFO - __main__ -     eval_combined_score = 0.8376804701980294
12/18/2020 16:45:29 - INFO - __main__ -     epoch = 3.0

12/18/2020 16:34:37 - INFO - __main__ -   ***** Eval results mrpc *****
12/18/2020 16:34:37 - INFO - __main__ -     eval_loss = 0.571368932723999
12/18/2020 16:34:37 - INFO - __main__ -     eval_accuracy = 0.6838235294117647
12/18/2020 16:34:37 - INFO - __main__ -     eval_f1 = 0.8122270742358079
12/18/2020 16:34:37 - INFO - __main__ -     eval_combined_score = 0.7480253018237863
12/18/2020 16:34:37 - INFO - __main__ -     epoch = 3.0
```

GPU: GTX 1080
transformers: 4.1.0
Torch: 1.6.0
python: 3.8
Server: Ubuntu 18.04


Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!
[core] transformers version number normalization
Author: stas00
Pull request: no

It looks that we need to either normalize the `transformers` version number to one of the accepted formats:

```
x.y.z
x.y.z.dev0
x.y.z.rc1
```
or silence the warning.

Currently, `setuptools` doesn't like `-rc-1` and `-dev`, as you can see from:

```
python setup.py --name
.../python3.8/site-packages/setuptools/dist.py:452: UserWarning: Normalizing '4.0.0-rc-1' to '4.0.0rc1'
```
with `4.0.0-dev`
```
python3.8/site-packages/setuptools/dist.py:452: UserWarning: Normalizing '4.0.0-dev' to '4.0.0.dev0'
```

Otherwise, this warning will be showing up all the time during `make style` and friends once https://github.com/huggingface/transformers/pull/8645 is merged.


@sgugger, @LysandreJik 


Comments:
We're making a release today, so will change the version number to the proper format just after :-)

Following what @sgugger said!
[setup] correct transformers version format
Author: stas00
Pull request: yes

setuptools has a pretty fixed expectation of version numbers.

```
x.y.z
x.y.z.dev0
x.y.z.rc1
```

This PR fixes the dev version number and adds a comment with correct formats for the future editors

This fix removes this warning on `make fixup|style|etc` or any other time `setup.py` is being run.
```
setuptools/dist.py:452: UserWarning: Normalizing '4.2.0dev0' to '4.2.0.dev0'
  warnings.warn(tmpl.format(**locals()))
```
and the alternative:
```
/setuptools/dist.py:452: UserWarning: Normalizing '4.0.0-rc-1' to '4.0.0rc1'
```

Fixes: #8749

@LysandreJik, @sgugger


Comments:
Fix link to old SQUAD fine-tuning script
Author: mrm8488
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Comments:
Add timing inside Trainer
Author: sgugger
Pull request: yes

# What does this PR do?

Add timing reports for training/evaluation and test inside the Trainer. Also, change the default repr of `TrainingArguments` to avoid printing the deprecated arguments.

There is a breaking change in this PR: the output of `Trainer.train` gains a new field `metrics`, so the length of the namedtuple changes. I don't think it's too bad since all scripts and examples I've seen never store the result of this `train` method. After discussion with @LysandreJik we proposed to merge this breaking change and revert it before the next release if users complain.

Comments:
may I ask for one more bit, while you're at it - sorting the metrics before printing them out?

I added this already into the final json file writing, but it'd make it easier to read the info logs.

Now we have:

```
2020-12-18 11:51:55 | INFO | __main__ |   val_loss = 368.4116
2020-12-18 11:51:55 | INFO | __main__ |   val_bleu = 26.3465
2020-12-18 11:51:55 | INFO | __main__ |   val_gen_len = 31.2
2020-12-18 11:51:55 | INFO | __main__ |   val_runtime = 22.1214
2020-12-18 11:51:55 | INFO | __main__ |   val_samples_per_second = 9.041
2020-12-18 11:51:55 | INFO | __main__ |   epoch = 1.0
2020-12-18 11:51:55 | INFO | __main__ |   val_n_objs = 200
```

as compared to sorted:
```
2020-12-18 11:51:55 | INFO | __main__ |   epoch = 1.0
2020-12-18 11:51:55 | INFO | __main__ |   val_bleu = 26.3465
2020-12-18 11:51:55 | INFO | __main__ |   val_gen_len = 31.2
2020-12-18 11:51:55 | INFO | __main__ |   val_loss = 368.4116
2020-12-18 11:51:55 | INFO | __main__ |   val_n_objs = 200
2020-12-18 11:51:55 | INFO | __main__ |   val_runtime = 22.1214
2020-12-18 11:51:55 | INFO | __main__ |   val_samples_per_second = 9.041
```

or I could make a PR later if it's too unrelated... I probably should do that and not waste your time.

Thanks.
example code for fine-tuning CLM does not work for GPT
Author: TalitaGroetzinger
Pull request: no

## Environment info
- `transformers` version: 3.5.0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.6
- PyTorch version (GPU?): 1.7.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: NO 

### Who can help
@patrickvonplaten @TevenLeScao


## Information

Model I am using: open-ai GPT. 

The problem arises when using:
* [x] the official example scripts: (give details below)

```
python run_clm.py --model_name_or_path openai-gpt --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 --do_train --do_eval --output_dir /tmp/test-clm
```

It's the example script on: [(https://github.com/huggingface/transformers/tree/master/examples/language-modeling], which is used to fine-tune a casual language model. 

## To reproduce

Steps to reproduce the behavior:

1. go to transformers/examples/language-modeling 
2. run the following command 

`python run_clm.py --model_name_or_path openai-gpt --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 --do_train --do_eval --output_dir /tmp/test-clm`

3 The following error occurs: 
`
RuntimeError: The size of tensor a (1024) must match the size of tensor b (512) at non-singleton dimension 1
`

In my case, the error does not occur when using `--model_name_or_path gpt2`. 

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

The progress bar should be filled and the language model should be finetuned. 

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Could you try putting `--block_size=512` in your command to see if it changes something?

@LysandreJik  Thank you it solved the issue!
GPT-model attention heads pruning example
Author: altsoph
Pull request: yes

# What does this PR do?

This script is adapted from the [BERT attention heads pruning code](https://github.com/huggingface/transformers/blob/master/examples/research_projects/bertology/run_bertology.py) AKA Bertology to make it possible to prune GPT-model heads as well.

It basically works the same way as run_bertology.py, but can deal with GPT-models.


Comments:
One last detail: could you run `make style` on your branch? There seems to be some bad formatting.

> One last detail: could you run `make style` on your branch? There seems to be some bad formatting.

I did, but let me recheck it
T5 with allowing model changes 
Author: rabeehk
Pull request: no

Hi
I would like to be able to change the model of T5, in addition to training on multiple tasks, I think currently the script can work for summarization only. Do you know possible other scripts allowing training on multiple tasks? 
One more question that in T5 in tensorflow repo, they have a small example for using your repo, does it work for large-scale? Is training on TPU working? could it allow model changes so one can change the model architecture? 
thanks a lot

Comments:
Hey @rabeehk - for more specific cases we recommend that you fork master and tweak the model however you would like to so that it fits your purpose. 
I didn't understand 100% what kind of script you are looking for, but here you can browse some of the T5 scripts, we have collected: https://github.com/huggingface/transformers/tree/master/notebooks#-transformers-notebooks

Hi
thanks for getting back to me. I am looking for a script showing to train
T5 with multiple tasks. this is when they create a T5 registery mixture
dataset in their original code and train one model on a mixture of several
dataset.
Do you know if huggingface version works fine for handling multiple
datasets? and do you know how performance is different from JAX
implementation? is this more or less the same? can one train the base T5
with a mixture of datasets with huggingface code?

thank you very much.
Best
Rabeeh

On Fri, Oct 23, 2020, 8:20 AM Patrick von Platen <notifications@github.com>
wrote:

> Hey @rabeehk <https://github.com/rabeehk> - for more specific cases we
> recommend that you fork master and tweak the model however you would like
> to so that it fits your purpose.
> I didn't understand 100% what kind of script you are looking for, but here
> you can browse some of the T5 scripts, we have collected:
> https://github.com/huggingface/transformers/tree/master/notebooks#-transformers-notebooks
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/7966#issuecomment-714945934>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABP4ZCEVU5IEVMEYHK4VLSTSMEOELANCNFSM4S2XT3OA>
> .
>
Getting Import error ImportError: cannot import name 'quantize' from 'transformers.convert_graph_to_onnx' (/opt/conda/lib/python3.7/site-packages/transformers/convert_graph_to_onnx.py)
Author: pinexai
Pull request: no
Labels: name: wontfix

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-19-60bf04c6de64> in <module>
----> 1 from simpletransformers.classification import MultiLabelClassificationModel
      2 
      3 
      4 model = MultiLabelClassificationModel('roberta', 'roberta-base', num_labels=6, args={'train_batch_size':2, 'gradient_accumulation_steps':16, 'learning_rate': 3e-5, 'num_train_epochs': 3, 'max_seq_length': 512})

/opt/conda/lib/python3.7/site-packages/simpletransformers/classification/__init__.py in <module>
----> 1 from simpletransformers.classification.classification_model import ClassificationModel
      2 from simpletransformers.classification.multi_label_classification_model import MultiLabelClassificationModel
      3 from simpletransformers.classification.multi_modal_classification_model import MultiModalClassificationModel
      4 from simpletransformers.config.model_args import (
      5     ClassificationArgs,

/opt/conda/lib/python3.7/site-packages/simpletransformers/classification/classification_model.py in <module>
     62     get_linear_schedule_with_warmup,
     63 )
---> 64 from transformers.convert_graph_to_onnx import convert, quantize
     65 
     66 from simpletransformers.classification.classification_utils import (

ImportError: cannot import name 'quantize' from 'transformers.convert_graph_to_onnx' (/opt/conda/lib/python3.7/site-packages/transformers/convert_graph_to_onnx.py)

Comments:
Any Updates on the above issue ?

@mfuntowicz any updates on this ?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Downloading DPR model ('facebook/dpr-ctx_encoder-single-nq-base')
Author: abesalom10
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!--Hi, I want to use model build on the natural questions. I have to question:
1.. I see an example of the above-mentioned model. here it is 
```
from transformers import DPRReader, DPRReaderTokenizer
tokenizer = DPRReaderTokenizer.from_pretrained('facebook/dpr-reader-single-nq-base')
model = DPRReader.from_pretrained('facebook/dpr-reader-single-nq-base', return_dict=True)
encoded_inputs = tokenizer(
        questions=["What is love ?"],
        titles=["Haddaway"],
        texts=["'What Is Love' is a song recorded by the artist Haddaway"],
        return_tensors='pt'
    )
outputs = model(**encoded_inputs)
start_logits = outputs.stat_logits
end_logits = outputs.end_logits
relevance_logits = outputs.relevance_logits
```  So I want to run the command which gives me the answers here. It gives me just the start and end position of answer.
2.. Can I use my context (my document) where I want to search model the answer? is it possible here?
Thanks in advance-->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
Hi, could you mention the issue you're having?

I want to use a natural question dataset and model trained on that. I have seen this code:
```
from transformers import DPRReader, DPRReaderTokenizer
tokenizer = DPRReaderTokenizer.from_pretrained('facebook/dpr-reader-single-nq-base')
model = DPRReader.from_pretrained('facebook/dpr-reader-single-nq-base', return_dict=True)
encoded_inputs = tokenizer(
        questions=["What is love ?"],
        titles=["Haddaway"],
        texts=["'What Is Love' is a song recorded by the artist Haddaway"],
        return_tensors='pt'
    )
outputs = model(**encoded_inputs)
start_logits = outputs.stat_logits
end_logits = outputs.end_logits
relevance_logits = outputs.relevance_logits
```
But from here I can not know how to return the answer from the model. It gives us just starting and ending position of the answer as far as I understand.
And also CAn I use my document as a context, and model search answer in this document. If yes please tell me how it is possible.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Feature Request: Support training/evaluation on Squad-format (json) files in SquadDataset for quick Squad fine-tuning
Author: raperry
Pull request: no
Labels: name: wontfix

I am currently working on a project using [run_squad_trainer.py](https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad_trainer.py)
to quickly fine-tune on new Squad-format (v2.0 json) files. However, the current SquadDataset class only allows training/evaluating on the original Squad jsons (train-v2.0.json, dev-v2.0.json). I have used a quick work around by using softlinks, to the actual training/evaluation files, but feel this is a little contrived.

I believe if the arguments `train_file` and `predict_file` are added in [SquadArguments](https://github.com/huggingface/transformers/blob/aa6c3c14b4ff8fbd5d40b100f4aae71fb359d6ae/src/transformers/data/datasets/squad.py#L36) and also line 152 in [SquadDataset](https://github.com/huggingface/transformers/blob/aa6c3c14b4ff8fbd5d40b100f4aae71fb359d6ae/src/transformers/data/datasets/squad.py#L152) is changed to 
                   
 `self.examples = self.processor.get_dev_examples(args.data_dir, filename=args.predict_file)`

and line 154 in [SquadDataset](https://github.com/huggingface/transformers/blob/aa6c3c14b4ff8fbd5d40b100f4aae71fb359d6ae/src/transformers/data/datasets/squad.py#L154)



`self.examples = self.processor.get_train_examples(args.data_dir, filename=arg.train_file)`

that may do the trick. At least this approach works in [run_squad.py](https://github.com/huggingface/transformers/blob/aa6c3c14b4ff8fbd5d40b100f4aae71fb359d6ae/examples/question-answering/run_squad.py#L444).

Thanks for your great work!


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Unique names for dataset cache for each tokenizer 
Author: Santosh-Gupta
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

Currently in the examples, dataset caches are named for the family of tokenizer used. For example, 'cached_train_BertTokenizer_128'. This may lead to unexpected behavior when running multiple models/tokenizers within the same model type, but different variations on the model/tokenizer. 

In this colab notebook, NER training is used run using scibert and then bert-base-cased. Even though the old data files are removed, the code still uses the old cache, resulting in an indexing error due to the mismatched token indices. 

https://colab.research.google.com/drive/1q4uBFm81yBWVNzG3Si2ByBh1nw8fk-Q5?usp=sharing

In this colab notebook, NER training is run on scibert-cased and then scibert-uncased. In this notebook, no explicit error occurs since there isn't an indexing error, but it seems that the wrong dataset is being used. In the output of scibert-uncased, there are many warnings of unpredicted tokens, and a lower than expected score. Both of these do not occur if scibert-cased is not run before scibert-cased

https://colab.research.google.com/drive/1pnpWfRqX4nknc0RRe9A2CArbVok3NbhC?usp=sharing


## Motivation

Prevent unexpected behavior when testing on multiple variations on the same transformer architecture. 




Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
quick question about `BertForMaskedLM`
Author: h56cho
Pull request: no
Labels: name: wontfix

Hello,
I have a question about the example code that can be found in the documentation for `BertForMaskedLM` model. The example from the documentation is shown below:
```python
from transformers import BertTokenizer, BertForMaskedLM
import torch
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForMaskedLM.from_pretrained('bert-base-uncased', return_dict=True)
input_ids = tokenizer("Hello, my dog is cute", return_tensors="pt")["input_ids"]
outputs = model(input_ids, labels=input_ids)
loss = outputs.loss
prediction_logits = outputs.logits
```
In this example, from the input string "Hello, my dog is cute", I don't see any `mask_token` in it. Also, the example code simply passes the label as `label=input_ids`. 

So in this particular example, how exactly does the `BertForMaskedLM` model calculate the masked-LM loss (since the `mask_token` is not specified in the input string)? when I simply pass `labels = input_ids`, does `BertForMaskedLM` model automatically place the `mask_token` over the the first token of the input string (or something similar to this)?

I don't think that the code provided in the documentation is wrong, because when I run the code on my machine, it runs smoothly without generating any error.

Thank you,

Comments:
This has been fixed already but is only visible in the master documentation: see [here](https://huggingface.co/transformers/master/model_doc/bert.html#bertformaskedlm). The documentation that is shown by default corresponds to the last release and the fix in the docstrings has been done since then :-)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.