noamaanMulla-03/transformers-issues huggingface dataset:
CUDA out of memory error on Trainer hyperparameter_search
Author: gchhablani
Pull request: no

Hi, 

I am using Colab to GridSearch on a dataset. The dataset has 7000 samples. I use 1/5 shard and batch_size is 1. No matter what I do, I get this error.

Env Related Info
-----------
tranformers (4.2.1)
datasets (1.2.1)

Model Used
----------------
[SpanBERT Large](https://huggingface.co/SpanBERT/spanbert-large-cased)

Snippets
------------
My dataset has 7934  train examples and 690  eval examples, with maximum number of tokens around 300 per example.

Tokenization Details:
```
  max_length: 384
  doc_stride: 128
```
```
def model_init():
    return AutoModelForQuestionAnswering.from_pretrained(model_checkpoint_name, num_labels=2)

args = TrainingArguments(
    output_dir = os.path.join(checkpoint_dir,'/grid_search/'),
    logging_dir= os.path.join(runs_dir,'/grid_search/'),
    evaluation_strategy='epoch',
    per_device_train_batch_size=2,
    per_device_eval_batch_size=2,
    gradient_accumulation_steps=1,
    learning_rate=2e-5,
    weight_decay=0.01,
    num_train_epochs=3.0,
    lr_scheduler_type="linear",
    warmup_steps=0,
    logging_steps=200,
    save_steps=200,
    seed=42
)

grid_search_trainer = Trainer(
    model_init=model_init,
    args=args,
    train_dataset=tokenized_train.shard(index=1,num_shards=5),
    eval_dataset=tokenized_val.shard(index=1,num_shards=5),
    data_collator=data_collator,
    tokenizer=tokenizer
)

best_run = grid_search_trainer.hyperparameter_search(n_trials=10, direction="minimize")
```

Sometimes, this happens after the trainer is done with one epoch. Is the trainer initializing the model without deleting/clearing the previous one? Would that affect the GPU? How can I prevent this issue?

![image](https://user-images.githubusercontent.com/29076344/105278992-a551f580-5bcc-11eb-9056-91560a6a99d4.png)

@patil-suraj 

Comments:
There's not enough information here to help you. Please provide env info, the model/tasks you are using, and a short text code snippet to reproduce the error.  

@patil-suraj I have edited the comment. Please help me out. Thanks.

@patil-suraj Same issue here


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
NAN return from F.softmax function in pytorch implementation of BART self-attention
Author: KaiQiangSong
Pull request: no

Pytorch 1.7.1 with GPU
transformers 3.0.2

Filling all masked positions with "-inf" may cause a NAN issue for softmax function returns.

Comments:
It may cause similar issues in other models and other versions of the same model as well.

HI @KaiQiangSong 

We haven't yet observed `NaN`s with BART specifically, could you post a code snippet where the model returns `NaN` so we could take a look ?

> HI @KaiQiangSong
> 
> We haven't yet observed `NaN`s with BART specifically, could you post a code snippet where the model returns `NaN` so we could take a look ?

Sorry that, I couldn't publish my code now due to it is unpublished research.
I've fixed the issue myself with changing the mask_fill of float("-inf") to -1e5 (for supporting AMP as well).
Just post this issue here to let you know there might be a potential issue.

I have the same exact problem.

Ill try with the -1e5 trick and see if it helps me too.

Thanks a lot!

> I have the same exact problem.
> 
> Ill try with the -1e5 trick and see if it helps me too.
> 
> Thanks a lot!

glad that my solution helps.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
RAG Fine Tuning
Author: Shree-Lakshmi-G-Prakash
Pull request: no

How do we train RAG mode with custom data set.
Can we have detailed document on this.

Thanks

Comments:
It is already [here](https://github.com/huggingface/transformers/tree/master/examples/research_projects/rag) .

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Renamed `nlp` variables #9455
Author: terrenceedmonds
Pull request: yes

* Give better names to pipeline variables named nlp

* This was desired because nlp was not a descriptive variable name 

Fixes # 9455

@Narsil , @sgugger 


Comments:
I can change `unmask` to `unmasker`. And I'll go back and take care of the merge conflicts and `make style` . 

I think your rebase went wrong as the diff as suddenly become unreadable. Could you close this PR and open a new one from your branch? Don't hesitate to tag me on it.

Hi @terrenceedmonds I don't think you ever opened a new clean PR from your branch (might need a new rebase first since it's been a while). You had done all the work for this issue so it would be great to merge it!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Fixing bug in generation
Author: nicola-decao
Pull request: yes

When passing `inputs_embeds` and not `input_ids=None` the generation function fails because `input_ids` is created but the function but it should not.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@patrickvonplaten


Comments:
Circle CI is unrelated - merging! Thanks a lot @nicola-decao
Fix cross-attention head mask for Torch encoder-decoder models
Author: stancld
Pull request: yes

1. This PR fixes head masking for the cross-attention module in the following models:

- BART,
- Blenderbot,
- Blenderbot_small,
- FSMT,
- LED,
- M2M_100,
- Marian,
- MBart,
- Pegasus.
- T5

2. This PR also contains slight changes in docstrings so that it will be clear that `head_mask` is related to the config of an encoder and the shape of `decoder_head_mask` and `cross_head_mask` depends on the config of a decoder.
 
3. This PR enables `test_headmasking` for M2M_100 model.

<hr>

**Reviewers:** @patrickvonplaten @patil-suraj 

<hr>

Fixes: #10540 

Comments:
Hi @patrickvonplaten & @patil-suraj, my PR does not pass one test, however, I am not able to reproduce this error on my local (I can't even find the file at `src/transformers/models/new_enc_dec/modeling_new_enc_dec.py` in the repo, which is the one where should be a problem with a copy inconsistency)

Hi @stancld thank you for your work! The issue is because you have updated all model files (thank you!!), but you haven't updated the template. The template is used when adding a new model, it's available [here](https://github.com/huggingface/transformers/tree/master/templates/adding_a_new_model).

For example, these lines should probably be updated to include the `cross_head_mask`:

https://github.com/huggingface/transformers/blob/44f64132a5f50726f9de4467ed745421c3b11ab3/templates/adding_a_new_model/cookiecutter-template-%7B%7Bcookiecutter.modelname%7D%7D/modeling_%7B%7Bcookiecutter.lowercase_modelname%7D%7D.py#L2070-L2085

@LysandreJik - Thank you very much for the clarification :)

Wonderful, the tests pass! Thanks for handling the templates.

@patil-suraj Thank you for your review and the suggestions. I agree with the change in variable naming. The new one is more accurate. I will change it everywhere accordingly :) 

Hi @patrickvonplaten, I've just rebased this branch to the current master to keep this PR up to date. Could you please review this one? Thanks a lot! :) 

Hey @stancld, Patrick is off for a couple of weeks but will take a look at this as soon as he's back :)

The PR looks good to me :-) 

Think we just need to fix the docstring. @stancld - let me know if you need help regarding the docstring

@patrickvonplaten The docstring should be fixed now. I forgot a line from a conflict there.. :)
🐛 Bug in attention head mask for cross-attention module in encoder-decoder models
Author: stancld
Pull request: no

Currently, encoder-decoder models use either `head_mask` or `decoder_head_mask` for masking attention heads in cross-attention modules. Both cases are not perfectly correct. Furthermore, MHA in cross-attention modules shares the parameters with the decoder, i.e. `shape = (decoder.num_layers, decoder.num_attention_heads)`, therefore, the usage of encoder `head_mask` in the cross-attention module may lead to errors due to the shape mismatch.

<hr>

**My contribution:** I will take care of this issue this weekend.

<hr>

**Reviewers:** @patil-suraj @patrickvonplaten 



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Unstale

Hey @stancld,

Sorry for being so unresponsive here - I'm happy to change the behavior and provide 3 masks
wrong parentclass in documentation
Author: cronoik
Pull request: yes

# What does this PR do?

The documentation linked to the parent class PreTrainedTokenizerFast but it should be the slow tokenizer class


## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


@sgugger


Comments:
unsubscribe<https://github.com/notifications/unsubscribe-auth/ATPBJ2FNGKBXXXSN6ZH6POLTKIDLZANCNFSM43PQRSGA>.
‏/
0Merged #11410<https://github.com/huggingface/transformers/pull/11410> into master.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<https://github.com/huggingface/transformers/pull/11410#event-4639275602>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ATPBJ2CW5M6UJ4RCYT2YIOLTKIFCTANCNFSM43PQRSGA>.
BigBird
Author: slvcsl
Pull request: no

## Environment info
- `transformers` version: 4.4.0.dev0
- Platform: Linux-5.3.0-64-generic-x86_64-with-glibc2.30
- Python version: 3.9.2
- PyTorch version (GPU?): 1.8.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Y
- Using distributed or parallel set-up in script?: N

### Who can help
@patrickvonplaten, @patil-suraj (guessing)
## Information

Model I am using **BigBird**:

The problem arises when using: the official example scripts: **seq2seq**

The tasks I am working on is: the official **BigPatent** dataset

## To reproduce
I'm trying to use BigBird for a summarization task (on the BigPatent dataset). I'm using the official seq2seq script, which I run as (all lengths/batches are small for testing)

```
python run_seq2seq.py \
    --model_name_or_path google/bigbird-roberta-base \
    --dataset_name big_patent \
    --max_source_length 3 \
    --max_target_length 3 \
    --val_max_target_length 3 \
    --do_eval --do_predict \
    --per_gpu_train_batch_size 1 \
    --per_gpu_eval_batch_size 1 \
    --num_train_epochs 1 \
    --output_dir tmp 
```

However, I get the following error message:
```
Traceback (most recent call last):
  File "/home/scasola/factuality/factuality/transformers/examples/seq2seq/run_seq2seq.py", line 657, in <module>     
    main()
  File "/home/scasola/factuality/factuality/transformers/examples/seq2seq/run_seq2seq.py", line 344, in main
    config = AutoConfig.from_pretrained(
  File "/home/scasola/anaconda3/envs/factuality/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 382, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
KeyError: 'big_bird'
```

Comments:
hi @slvcsl 

`BigBird` is till WIP and not yet added into the lib,. You can follow the progress here #10183



Thank you for your response. I'll check it out!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
load wav2vec model from local path
Author: roboticsai
Pull request: no

I'm trying to run the wav2vec based ASR on my machine.  This is my code:
```
import soundfile as sf
import torch
from transformers import Wav2Vec2ForMaskedLM, Wav2Vec2Tokenizer

# load pretrained model
cp = "./my_model_directory/wav2vec_small.pt"
tokenizer = Wav2Vec2Tokenizer.from_pretrained(cp)
model = Wav2Vec2ForMaskedLM.from_pretrained(cp)

# load audio
audio_input, _ = sf.read("/home/robot/Music/dinesh.flac")

# transcribe
input_values = tokenizer(audio_input, return_tensors="pt").input_values
logits = model(input_values).logits
predicted_ids = torch.argmax(logits, dim=-1)
transcription = tokenizer.batch_decode(predicted_ids)[0]
print(transcription)

```
Here cp is the path to the wav2ved local model file. But when i try to run this i'm getting error;
`- or './my_model_directory' is the correct path to a directory containing relevant tokenizer files`
Here when i use the model present in the cloud eg. cp = "facebook/wav2vec2-base-960h". This works perfectly. 
Isn't this possible to run the transformers wav2vec without cloud? 

Comments:
hi @roboticsai 

`from_pretrained` expects path of the directory where it can find `config.json` and `pytorch_model.bin` files. It seems that you haven't saved the model using `save_pretrained`. 

To use `from_pretrained`, the model should be saved using `save_pretrained` method.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Position ids in RoBERTa
Author: qhd1996
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: Ubuntu 20.04
- Python version: Python 3.7.10
- PyTorch version (GPU?): 1.6.0_py3.7_cuda10.1.243_cudnn7.6.3_0
- Tensorflow version (GPU?): No
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@LysandreJik 

## Information
Position ids in RoBERTa is not implemented properly.
The problem arises when using:
create_position_ids_from_input_ids in transformers.models.roberta.modeling_roberta.py
Based on this function, position id 0 is never used. This may cause problem when the sequence is long, for example, 512. Token whose id >= 511 will not get its corresponding token ids.



Comments:
There's a reason why position ids don't start at 0 for RoBERTa, see #5285

`RoBERTa` never uses 0 and 1 positional ids, in `ROBERTa`, all pad tokens have position id of 1, and the rest of the tokens have position ids in the range `(2, seq_length - num_pad_tokens)`.  It's implemented like this to match the original implementation in fairseq.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
run_clm.py does not work with any other block_size other than 1024
Author: sytelus
Pull request: no

**Note:** This issue can be fixed with one character change as described in last section.

## Environment info
- `transformers` version: 4.4.0.dev0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Who can help
- maintained examples (not research project or legacy): @sgugger, @patil-suraj

## Information

Model I am using (Bert, XLNet ...): GPT2

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: Causal language modelling
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Use GPT based model which has block_size different than 1024
2. Try to train it or fine tune with run_clm.py setting setting block_size in data_args.

In GPU mode, you will get following error:
```
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`
```

In CPU mode, you will get following error:

```
index out of range in self
```

## Expected behavior
Above error should not occur.

## Cause and Proposed Fix

The issue is because block_size on [line 337](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_clm.py#L337) always get set to 1024 because of wrong indentation:

```
    if data_args.block_size is None:
        block_size = tokenizer.model_max_length
        if block_size > 1024:
            logger.warn(
                f"The tokenizer picked seems to have a very large `model_max_length` ({tokenizer.model_max_length}). "
                "Picking 1024 instead. You can change that default value by passing --block_size xxx."
            )
        block_size = 1024 # <<< THIS LINE NEEDS TO BE INTENDED!!!
```
 So just tabbing that line should fix the issue.

Comments:
Indeed! Would you mind making a PR with that change since you found the correct fix?

Sure, I'll get that prepared!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Albert quantized
Author: Zjq9409
Pull request: no

I use onnxruntime to optimize and quantize transformers model 'albert-base-v2' ,but the quantized result is different from original result,so,Does it support transformers albert quantized right now?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Needed a feature to convert facebook mbart many - many model to ONNX runtime inorder to reduce the inference time
Author: Vimal0307
Pull request: no

We need a feature which makes the facebook mbart many - many model to convert to ONNX runtime which reduces the inference time, current mbart many-many model takes 9 secs to translate and we need to quantize it further to reduce the inference time  

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
How to Reduce the inference time of Facebook/many to many model?
Author: sankarsiva123
Pull request: no

Facebook/many to many model takes 9s on cpu to translate , how to reduce the inference time on cpu ?
It will be helpful if an method is suggested .

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Guidance for continued pre-training of BART with de-noising.
Author: griff4692
Pull request: no

# 🚀 Feature request

An example of continued pre-training of BART with de-noising.

## Motivation

I'm using the run causal LM [script](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_clm.py), but it seems on line 340, it's simply copying input to output (learning the identify function).  

1. Would line 340 be the best place to add the de-noising or should I introduce it as part of the collator?
2. Is there any code which implements de-noising in HuggingFace?  BART defines 4-5 main operations which should be easy to reproduce - I just don't want to introduce new code if well-tested code already exists.


Thanks!!!


Comments:
also it is my questions, thanks 

denoising function is a part of T5 pretraining as well, is there a denoising function implementation in Huggingface repo ? Any advice is appreciated. thanks 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.