noamaanMulla-03/transformers-issues huggingface dataset:
placing the run dir only in the output_dir
Author: rabeehk
Pull request: no
Labels: name: wontfix

Hi
In training_args.py, there is a code which creates a run dir, then anyplace user runs the code there it would create a run dir, could you create it only in the output_dir? thanks 


```
def default_logdir() -> str:
    """
    Same default as PyTorch
    """
    import socket
    from datetime import datetime

    current_time = datetime.now().strftime("%b%d_%H-%M-%S")
    return os.path.join("runs", current_time + "_" + socket.gethostname())

```



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[BUG] Wrong Scores for many SQUAD models
Author: elronbandel
Pull request: no
Labels: name: wontfix

@julien-c
@VictorSanh

## Information

All models trained with run_squad.py have abstaining threshold of 0.0 and possibly wrong evaluation scores.
Those model have their wrong score in their model cards many people rely on. 

For example: ahotrod/electra_large_discriminator_squad2_512

The results with current evaluation script:

```
"exact": 87.09677419354838,
  "f1": 89.98343832723452,
  "total": 11873,
  "HasAns_exact": 84.66599190283401,
  "HasAns_f1": 90.44759839056285,
  "HasAns_total": 5928,
  "NoAns_exact": 89.52060555088309,
  "NoAns_f1": 89.52060555088309,
  "NoAns_total": 5945,
  "best_exact": 87.09677419354838,
  "best_exact_thresh": 0.0,
  "best_f1": 89.98343832723432,
  "best_f1_thresh": 0.0
```

The problem and its fix can be found in: [[Bug Fix] Fix run_squad.py evaluation code doesn't use probabilities](https://github.com/huggingface/transformers/pull/7319) #7319
The problem arises when using:
* [ ] run_squad.py

The tasks I am working on is:
* [ ] SQuAD

## To reproduce

Steps to reproduce the behaviour:

1. run `run_squad.py`

## Expected behaviour:
Resulting 'best_f1_thresh' wont be 0.0 .


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
providing the user with possibility to set the cache path 
Author: rabeehk
Pull request: no
Labels: name: wontfix

Dear HuggingFace team,

In the codes of huggingface, most of the time there is some cache_path which is on the home directory,

/idiap/home/rkarimi/.cache/huggingface/datasets/downloads

could you provide me with a command to set a different cache_path? to me this is hard-coded in the codes or I am missing something.

Thank you.

Best regards
Rabeeh



Comments:
This path is the default `cache_path` of the datasets library, not transformers. You can change it by setting an environment variable named `HF_HOME` to the path you want, the datasets will then be cached in this path suffixed with "/datasets/"

Hi
thank you, the huggingface codes gebnerally also create an empty folder
titled '  ' when I run it, which is specifies the caching folder address,
could it be possible not to create this folder?
thanks
Best
Rabeeh

On Sat, Nov 21, 2020 at 6:44 PM Sylvain Gugger <notifications@github.com>
wrote:

> This path is the default cache_path of the datasets library, not
> transformers. You can change it by setting an environment variable named
> HF_HOME to the path you want, the datasets will then be cached in this
> path suffixed with "/datasets/"
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/8703#issuecomment-731611519>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABP4ZCDCFJDCXZAPZ73ORHDSQ74AZANCNFSM4T5ZWXDA>
> .
>


We would need to see the code you are running that creates this empty folder named `" "` to be able to help.

Hi there
I am training seq2seq_trainer codes. I have adapted it for my use case but
in the original version of codes should also happen.
thanks
Best
Rabeeh

On Sun, Nov 22, 2020, 4:46 AM Sylvain Gugger <notifications@github.com>
wrote:

> We would need to see the code you are running that creates this empty
> folder named " " to be able to help.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/8703#issuecomment-731693963>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABP4ZCCKYJ5TFIKGT2N7LTLSRCCP5ANCNFSM4T5ZWXDA>
> .
>


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
training text_classification with tpu using xla_spawn gives wrong result
Author: kiyoungkim1
Pull request: no
Labels: name: wontfix

I tested text_classification code with tpu and gpu versions shown below.
tpu(colab with 8cores) and gpu(colab) versions take 2min and 17min, respectively, which is nice.
Training gpu version gives good behavior, where loss decreases continuously.

However, for tpu version, the dataset is almost divided into 8 segments(so 1/8 times consuming), but each node may not be connected. Results from tpu of each node are just result with gpu with 1/8 of the original dataset size.

1.
Should I change something?
Or if I`d like to use TPU, I have to use TF version?

2.
My final goal is to train Roberta on TPU in Korean.
There are three options.
1. Huggingface Trainer with xla  --> I am here.
2. Huggingface TFTrainer --> TFTrainer supports TPU, but need to make mlm datasets.
3. Fairseq with xla
If there are any sources or examples, please let me know.

Thanks,

# tpu version (same one shown in the document 'https://github.com/huggingface/transformers/tree/master/examples/text-classification'):
python examples/xla_spawn.py \
  --num_cores=8 \
  transformers/examples/text-classification/run_glue.py \
  --do_train \
  --do_eval \
  --task_name=mrpc \
  --num_train_epochs=3 \
  --max_seq_length=128 \
  --learning_rate=5e-5 \
  --output_dir=/tmp/mrpc \
  --overwrite_output_dir \
  --logging_steps=5 \
  --save_steps=5 \
  --tpu_metrics_debug \
  --model_name_or_path=bert-base-cased \
  --per_device_train_batch_size=64 \
  --per_device_eval_batch_size=64

# single gpu version (remove   --num_cores=8,   --tpu_metrics_debug)
python examples/xla_spawn.py \
  transformers/examples/text-classification/run_glue.py \
  --do_train \
  --do_eval \
  --task_name=mrpc \
  --num_train_epochs=3 \
  --max_seq_length=128 \
  --learning_rate=5e-5 \
  --output_dir=/tmp/mrpc \
  --overwrite_output_dir \
  --logging_steps=5 \
  --save_steps=5 \
  --model_name_or_path=bert-base-cased \
  --per_device_train_batch_size=64 \
  --per_device_eval_batch_size=64

Comments:
Hello! I'm not sure I completely understand the issue. Are you saying you do not obtain the results you expected when using the 8 cores of the TPU, vs using a single TPU core?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Cannot load tokenizer in community T5 pretrained model
Author: minhtam2048
Pull request: no
Labels: name: wontfix

## Environment info
- `transformers` version: 3.5
- Platform: Window 10
- Python version: 3.7.9
- PyTorch version (GPU?): 1.7.0+cpu (False)
- Tensorflow version (GPU?): 2.3.1 (False)
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False

### Who can help

 T5: @patrickvonplaten

## Information

I'm trying to use the **sshleifer/t5-base-cnn** for summarization task. But It seem likes there is something wrong with the tokenizer. 

    tokenizer = T5Tokenizer.from_pretrained('sshleifer/t5-base-cnn')
    model = T5ForConditionalGeneration.from_pretrained('sshleifer/t5-base-cnn')
    
This code return an error:

    OSError: Can't load tokenizer for 'sshleifer/t5-base-cnn'. Make sure that:

    - 'sshleifer/t5-base-cnn' is a correct model identifier listed on 'https://huggingface.co/models'

    - or 'sshleifer/t5-base-cnn' is the correct path to a directory containing relevant tokenizer files

Can someone point out what I am missing or Is there any problem with my code ?
Many thanks.

Comments:
Yes, that model from @sshleifer does not bundle its own tokenizer, as you can see in the list of files: https://huggingface.co/sshleifer/t5-base-cnn/tree/main

We'll add this info to the model card, but you can just use the one from t5: `T5Tokenizer.from_pretrained("t5-base")`

@julien-c Thank you for your help

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Bert variants pretrained on Wikipedia are easily downloaded. Are the optimizers from the pretraining also available?
Author: pkadambi
Pull request: no
Labels: name: wontfix

Is there a pretrained optimizer checkpoint available that can be loaded in the same way as a pretrained model?

I noticed that though the pretrained models are available trained on Wikipedia (ex can load a pretrained distillbert using: <br/>`model = DistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased'`)

But, I cannot find the optimizer from the end of the training run on wikipedia. There is no `checkpoint['optimizer']`

For my task, looking at optimizer internals (momentum, second moment, etc) from the end of training on wikipedia may be more useful to me than looking at optimizer internals from training on a downstream task (eg. GLUE). Does such a checkpoint exist (either for TF or Torch?)

 Environment info (not really relevant)
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.1.0
- Platform: Linux-4.15.0-123-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.7
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 Trainer: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)





Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Result changes if we don't pass attension mask in TFDistilbert model on SQUADv1 dataset
Author: bhadreshpsavani
Pull request: no
Labels: name: wontfix

## Environment info
- `transformers` version: latest
- Platform: Colab
- Python version: 
- Tensorflow version (GPU?): 2.3.0
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
I used the below code for getting Model
```
from transformers import  AutoTokenizer, TFAutoModelForQuestionAnswering
tokenizer = AutoTokenizer.from_pretrained('distilbert-base-uncased-distilled-squad')
model =  TFAutoModelForQuestionAnswering.from_pretrained('distilbert-base-uncased-distilled-squad', return_dict=True)
```
 tokenizers: @mfuntowicz
 examples/seq2seq: @patil-suraj
 tensorflow: @jplu 

## Information

The model I am using **TFDistilbert** pretrained.

The problem arises when using:
* my own modified scripts:
This is the Notebook
[Colab](https://github.com/bhadreshpsavani/UnderstandingNLP/blob/master/TFLiteExperimentsQALatest.ipynb)

The tasks I am working on is:
*  an official SQUaD task

## To reproduce

Steps to reproduce the behavior: [Colab](https://github.com/bhadreshpsavani/UnderstandingNLP/blob/master/TFLiteExperimentsQALatest.ipynb)

## Expected behavior

The performance should be the same because the Attention mask is the optional argument, if we don't pass it will create it internally. 

With Attention Mask:
```
OrderedDict([('exact', 77.71050141912077),
           ('f1', 85.5370981182013),
           ('total', 10570)])
```
Without Attention Mask:
```
OrderedDict([('exact', 72.82876064333927),
             ('f1', 80.71521545953475),
             ('total', 10570)])
```
     


Comments:
`attention_mask` is an optional argument, but that doesn't mean that it should not be passed to the function. If `attention_mask` is `None` then it is initialized to attend all tokens (all 1's in the `attention_mask` tensor), which is incorrect if the input is a batch that includes padding tokens. => It's better to simply pass the `attention_mask` to the forward function.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[WIP] Rewrite ProphetNet to adapt converting ONNX friendly
Author: jiafatom
Pull request: yes

# What does this PR do?
We want to convert ProphetNet (pytorch model) to ONNX, but it needs some source code change to adapt it. The current code cannot convert to ONNX because 
(1) The current pytorch model generates very large TorchScript IR graph (38k for decoder). We rewrite the way it generates bias: ~~Let's use numpy and then formulate the torch tensor finally.~~
Numpy way can help convert ONNX via tracing, but we prefer using scripting here.
Add script decorator so that the model can be converted via scripting. This reduces IR graph to 5k for decoder.
(2) `torch.new` generates constant dimension for Tensor in IR graph, which is not suitable if we want to do dynamic input axes for the converter. So we use `torch.full` instead.
This PR does not (should not) change any model behavior.

Fixes # (issue)
After this PR, the model can be converted to ONNX via scripting.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@qiweizhen @patrickvonplaten @Zhylkaaa 

Comments:
@mfuntowicz - could you take a look maybe? :-) 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Colab pro -fine RoBERTa error tcmalloc: large alloc 6325288960
Author: Shafi2016
Pull request: no
Labels: name: wontfix

I want to fine-tune RoBERTa on my newspaper data  (around 8GB) using Colab pro. It works fine on small data. I have given my codes below. Are my codes correct? **Is there any way to handle this memory problem**? The error crushes the colab.

tcmalloc: large alloc 6325288960 bytes == 0x447fa000 @  0x7f3438dcc1e7 0x59221c 0x4ca6f4 0x566daa 0x5a4df1 0x5a5eea 0x4ce082 0x566c02 0x5a4df1 0x5a60ae 0x5bd138 0x50a47f 0x50c1f4 0x507f24 0x509202 0x594b01 0x54a17f 0x5517c1 0x5a9eec 0x50a783 0x50cfd6 0x507f24 0x509c50 0x50a64d 0x50cfd6 0x509918 

 ```

!python "/content/transformers/examples/language-modeling/run_language_modeling.py" \
    --output_dir "/content/drive/My Drive/EPU-NLP/FinetuneModel/output" \
    --model_type roberta \
    --model_name_or_path roberta-base \
    --do_train \
    --per_gpu_train_batch_size 16 \
    --seed 22 \
    --train_data_file "/content/drive/My Drive/EPU-NLP/FinetuneModel/data_all.txt" \
    --block_size 256 \
    --line_by_line \
    --weight_decay 0.01 \
    --adam_epsilon 1e-6 \
    --save_total_limit 500 \
    --learning_rate 6e-4 \
    --num_train_epochs 3 \
    --save_total_limit 500 \
    --save_steps 500 \
    --mlm
```


Comments:
Got the same with StyleGAN using Colab Pro

What is your data size? First, try it with 2GB of data.  Also check again by decreasing the GPU batch size to 8 or 4.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Add token_type_ids to prepare_inputs_for_generation for gpt/gpt2
Author: bhedayat
Pull request: yes

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}


Comments:
Maybe related to https://github.com/huggingface/transformers/pull/6601 and https://github.com/huggingface/transformers/pull/7552

Yes seems to be related to both. https://github.com/huggingface/transformers/pull/7355 doesn't seem to have token_type_ids passed in though, but if those PRs get merged in I'll close mine

We have the same problem here as explained in https://github.com/huggingface/transformers/pull/6601#issuecomment-708029212. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Updating checkpoint for GPT2ForSequenceClassification #11334
Author: abiolaTresor
Pull request: yes


# What does this PR do?

This PR fixes the checkpoint for GPT2ForSequenceClassification. It sets it from `microsoft/dialogrpt` to `microsoft/DialogRPT-updown`


Fixes # (issue)
The identifier  `microsoft/dialogrpt` is incorrect. When used, the weights of the linear layer at top are differently initialized at each execution, which gives different prediction results  for  same inputs. The checkpoint `microsoft/DialogRPT-updown` fixes that issue since it offers a pretrained classification head.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section? Yes
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. Yes
[https://github.com/huggingface/transformers/issues/11334](url)
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? No


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR. Hello @LysandreJik 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments: