noamaanMulla-03/transformers-issues huggingface dataset:
RuntimeError: CUDA error: device-side assert triggered
Author: abb4s
Pull request: no

```
if torch.cuda.is_available():  
    dev = "cuda:0" 
else:  
    dev = "cpu"
device = torch.device(dev)

bert = BertForSequenceClassification.from_pretrained(args.model_name_or_path)
bert = bert.to(device)
```
raise RuntimeError: CUDA error: device-side assert triggered
nvidia-smi: 
![image](https://user-images.githubusercontent.com/7654832/115616848-ed84cf00-a305-11eb-951d-751619b9f8a8.png)


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Training a TimeSFormer for video classification 
Author: slimaneaymen
Pull request: no

My input data are the feature maps, instead of raw images. and have the form : (4,50,1,1,256)
mini_batch=4 / frames=50 / channels=1 / H=1 / W= 256
The parameters of the TimeSformer are : 
TimeSformer(
    dim = 128,
    image_size = 256,
    patch_size = 16,
    num_frames = 50,
    num_classes = 2,
    depth = 12,
    heads = 8,
    dim_head =  32,
    attn_dropout = 0.,
    ff_dropout = 0.
) 
In order to check if my network is working, I have tried to make it overfit by using only 6 training data and 2 validation data of the same shape as before (4,50,1,1,256).
But the accuracy I'm getting is in oscillation and never reaches a value >80% and my training loss is not decreasing it's always around 0.6900 - 06950
My training function and parameters are:
![Capture1](https://user-images.githubusercontent.com/24354915/115587902-7c75f500-a2ce-11eb-9a6c-88266f3c233a.PNG)
![Capture2](https://user-images.githubusercontent.com/24354915/115588104-ae875700-a2ce-11eb-81f0-d6e31e177d5d.PNG)
![Capture3](https://user-images.githubusercontent.com/24354915/115588124-b47d3800-a2ce-11eb-82bc-c80e87670bd3.PNG)
I would appreciate any suggestion.
thank you


Comments:
Hello! Is this a `transformers` issue? Where does `transformers` come into play?

> 
> 
> Hello! Is this a `transformers` issue? Where does `transformers` come into play?
Hi !! Actually, here I am showing only the issues I'm getting when training the TimeSFormer model,
you can find more information about the model in this link: https://github.com/lucidrains/TimeSformer-pytorch 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Bart tokenizer and bart model for conditional generation have different vocab size 
Author: clementbernardd
Pull request: no

## Environment info

- `transformers` version: 4.6.1
- Platform: Google colab
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1
- Using GPU in script?: No


### Who can help
@patrickvonplaten, @patil-suraj

Models:

- bart


Library:

- text generation: @patrickvonplaten
- tokenizers: @LysandreJik



## Information

Model I am using (BART):

The problem arises when using the pretrained model of bart : 'bart-large-xsum'. 
I tried to load the tokenizer and the model from `bart-large-xsum`.
I then tried to add mask to the inputs (as mentionned in the original paper section 5.1 (https://arxiv.org/pdf/1910.13461.pdf)).
But the tokenizer and the bart model don't have the same vocab size.
Does it mean the `bart-large-xsum` model doesn't take masks as inputs ? Do I need to add it to the vocabulary myself ? 



## To reproduce

Steps to reproduce the behavior:

```python
tokenizer =  BartTokenizer.from_pretrained('facebook/bart-large-xsum')
bart = BartForConditionalGeneration.from_pretrained('facebook/bart-large-xsum')
print(tokenizer.vocab_size)
print(bart.config.to_dict()['vocab_size'])
```
![Screenshot 2021-05-25 at 11 59 44](https://user-images.githubusercontent.com/44928643/119479244-bebca700-bd50-11eb-9a9f-e438abd66592.png)





Comments:
Hi there,

This is because the [orginal](https://github.com/pytorch/fairseq/tree/master/examples/bart#pre-trained-models) `bart-large-xsum` model uses 50264 for token_embedding, so you would probably need to extend the token embedding layer.
Use `self.assertEqual` instead of `assert` in tests.
Author: PhilipMay
Pull request: no

IMO the test should use `self.assertEqual` instead of `assert` here:

https://github.com/huggingface/transformers/blob/8d171628fe84bdf92ee40b5375d7265278180f14/tests/test_tokenization_deberta_v2.py#L105-L106

I can provide a PR...

Comments:
> I can provide a PR...

see #11935
[WIP] Expand `past_key_values` also during beam search in EncoderDecoder models
Author: seongminp
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #11781


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. **Not yet - issue was just submitted**
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation). **Internal change**
- [x] Did you write any new necessary tests? **No coverage branch added**


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj


Comments:
Changed to `WIP` because right now the PR does not account for cross attentions in `past_key_values` (indices 2 and 3). 

Could not be certain if cross-attention matrix for each layer in `past_key_values` is always a 4-tuple for all encoder-decoder models (maybe some model does not use cross-attention even though it is an encoder-decoder model..?). 

The doc does say key/value indices 2 and 3 in `past_key_values` are optional.
Add MT5ForConditionalGeneration as supported arch. to summarization README
Author: PhilipMay
Pull request: yes

see #11960

Comments:
Thanks a lot for fixing this!

Could you also update [translation readme](https://github.com/huggingface/transformers/blob/master/examples/pytorch/translation/README.md) ?

> Could you also update [translation readme](https://github.com/huggingface/transformers/blob/master/examples/pytorch/translation/README.md) ?

Done.
Summarization also supports MT5ForConditionalGeneration
Author: PhilipMay
Pull request: no

The [README.md](https://github.com/huggingface/transformers/blob/master/examples/pytorch/summarization/README.md) of the summarization examples sais it supports `T5ForConditionalGeneration` IMO `MT5ForConditionalGeneration` should be added - right?

Tagging @sgugger and @sshleifer ...

Comments:
And @patil-suraj

Yes, you are right!

We could also add it  in the[ translation readme](https://github.com/huggingface/transformers/blob/master/examples/pytorch/translation/README.md) as well.

Feel free to open a PR :)

> Feel free to open a PR :)

see #11961



Fixed by #11961
Remove redundant `nn.log_softmax` in `run_flax_glue.py`
Author: n2cholas
Pull request: yes

# What does this PR do?

`optax.softmax_cross_entropy` expects unscaled logits, so it already calls `nn.log_softmax` ([here](https://github.com/deepmind/optax/blob/master/optax/_src/loss.py#L166)). `nn.log_softmax` is idempotent so mathematically it shouldn't have made a difference.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@marcvanzee @patrickvonplaten 


Comments:
Great catch @n2cholas! Could you also remove the line:

```python
import flax.linen as nn
```

to make our code quality checks happy? Happy to merge right after :-)

Done @patrickvonplaten!
Mask token mismatch with the model on hosted inference API of Model Hub
Author: Ethan-yt
Pull request: no

### Who can help

@LysandreJik 
@julien-c 
@mfuntowicz 

## Information

In my model card: https://huggingface.co/ethanyt/guwenbert-base, I used to be able to run the hosted inference successfully, but recently it prompted an error: `"<mask>" must be present in your input.`

My model uses RoBERTa MLM and BERT Tokenizer. So the mask token is actually "[MASK]". I have already set it in `tokenizer_confg.json` but the inference API still mismatches with that.

In the past it is OK but recently it turns to prompt an error. Seems like the front-end start to double-check the mask token. How can I set the mask token in an appropriate way? Is it documented to set mask token in inference API?

Thanks!

## To reproduce

Steps to reproduce the behavior:

1. Go to https://huggingface.co/ethanyt/guwenbert-base
2. Run an example with "[MASK]"

## Expected behavior
In the past it was OK. See snapshot in  https://github.com/ethan-yt/guwenbert/blob/main/README_EN.md


Comments:
In the past, it was no error.
<img width="510" alt="lm-demo" src="https://user-images.githubusercontent.com/9592150/120172538-bd4e1b80-c235-11eb-8576-6446a2dd0ed8.png">
I don't know when it starts to emit an error.
<img width="544" alt="image" src="https://user-images.githubusercontent.com/9592150/120172600-ce972800-c235-11eb-91b0-a231cfaf4f5f.png">


I've fixed it by explicitly specifying the `mask_token` in your model card metadata: https://huggingface.co/ethanyt/guwenbert-base/commit/30aaff24928389096312600511a9ca2fad1b3974

thanks for reporting!

> thanks for reporting!

Thanks!
Add FlaxCLIP
Author: patil-suraj
Pull request: yes
Labels: name: Flax

# What does this PR do?

This PR adds the CLIP model in JAX/Flax.

Comments:
- Added jitted tests for `get_image_features` and `get_text_features`
- `__init__(...)` now takes `input_shape` as an arg, when `None` it's set using default values in `config`.

Merging!
Add new token to pretrained GPT2 tokenizer
Author: sooftware
Pull request: no

Hi! Thank you for awesome project!    
I want to add several tokens to pre-trained GPT2 tokenizer.   
Can I?

Comments:
I will use "EleutherAI/gpt-neo-1.3B" tokenizer. Is there a 

This should help you out: https://huggingface.co/transformers/internal/tokenization_utils.html?highlight=add_tokens#transformers.tokenization_utils_base.SpecialTokensMixin.add_tokens

Thank you! @LysandreJik
modify qa-trainer
Author: zhangfanTJU
Pull request: yes

I fixed the evaluation failure for the TrainerQA-based script [`examples/pytorch/question-answering/run_qa.py`](https://github.com/huggingface/transformers/blob/master/examples/pytorch/question-answering/run_qa.py) when using distributed training, which has been partially fixed in  #11746.

```
Traceback (most recent call last):
  File "run_qa.py", line 543, in <module>  
    main()
  File "run_qa.py", line 509, in main
    metrics = trainer.evaluate()
  File "trainer_qa.py", line 44, in evaluate
    ignore_keys=ignore_keys,
  File "/home/x/anaconda3/envs/torch18/lib/python3.6/site-packages/transformers/trainer.py", line 2162, in evaluation_loop
    logits = self._nested_gather(logits)
  File "/home/x/anaconda3/envs/torch18/lib/python3.6/site-packages/transformers/trainer.py", line 2252, in _nested_gather
    tensors = distributed_concat(tensors)
  File "/home/x/anaconda3/envs/torch18/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 154, in distributed_concat
    return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)
  File "/home/x/anaconda3/envs/torch18/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 154, in <genexpr>
    return type(tensor)(distributed_concat(t, num_total_examples) for t in tensor)
  File "/home/x/anaconda3/envs/torch18/lib/python3.6/site-packages/transformers/trainer_pt_utils.py", line 156, in distributed_concat
    dist.all_gather(output_tensors, tensor)
  File "/home/x/anaconda3/envs/torch18/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 1863, in all_gather
    work = default_pg.allgather([tensor_list], [tensor])
RuntimeError: Tensors must be non-overlapping and dense
```

This failure is similar to the the previous commit (https://github.com/huggingface/transformers/pull/404/commits/fda2f623953bfe2290cd65429eb008f02ebdb152), but it also happens in pytorch 1.8 now.

Meanwhile, I added **Evaluation** and **Prediction** log for the script [`run_qa_no_trainer.py`](https://github.com/huggingface/transformers/blob/master/examples/pytorch/question-answering/run_qa_no_trainer.py) according to the TrainerQA.

Thanks! @sgugger

Comments:
I encountered an unexpected error in ci, could you help me to finish this pr? @LysandreJik 

Seems like a CircleCI failure, I just relaunched all tests.

the CircleCI encountered a HTTPError, what should I do? @LysandreJik
get_ordinal(local=True) replaced with get_local_ordinal() in training_args.py
Author: BassaniRiccardo
Pull request: yes

## Fixed
Wrong method call fixed. Modified according to:
https://pytorch.org/xla/release/1.8.1/_modules/torch_xla/core/xla_model.html

TPU training as called by the following or similar scripts now works:
```ruby
python xla_spawn.py \
        --num_cores=8 \
        language-modeling/run_mlm.py \
            --train_file $TRAIN_FILE \
            --model_name_or_path bert-base-uncased \
            --output_dir $OUTPUT_DIR \
            --overwrite_output_dir True \
            --do_train True \
```

## Discussed/approved
https://github.com/huggingface/transformers/issues/11910

## Who can review?
 @sgugger


Comments:
Thanks a lot!
save only the best performing checkpoint
Author: yuvalkirstain
Pull request: no

# 🚀 Feature request

In the Trainer - Enable an option to save only the best performing checkpoints (rather than the newsest) 

## Motivation

Usually when we train a model we would like to keep only the best performing checkpoints (on the dev set according to the specified metric) rather than the newest checkpoints.



Comments:
The checkpoints are saved to resume training in case you are interrupted, so saving only the best checkpoints wouldn't work with this.

The `load_best_model_at_end` functionality already keeps track of the best checkpoint during training and reloads it at the end, I think it should cover what you need.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

So，when I  finish training，how can I load the best performing checkpoint ？ @sgugger 

 When  I check the `trainer_state.json`  file I found these mssage:
```
  "best_metric": null,
  "best_model_checkpoint": null,
  "epoch": 100.0,
  "global_step": 559300,
  "is_hyper_param_search": false,
  "is_local_process_zero": true,
  "is_world_process_zero": true,
```

as shown above，"best_model_checkpoint"   is null.



If you did not specify `--load_best_model_at_end` for your script, you won't get it automatically.
reinitialize wandb config for each hyperparameter search run
Author: Mindful
Pull request: yes

# What does this PR do?

Fixes #11944 

This is the quick/easy fix I'm using to work around the issue locally by just rerunning the WandbCallback integration `setup()` method for each run. This works fine for me, but if for some reason it's not safe/desirable to rerun the `WandbCallback.setup()` please feel free to just close this PR. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Unsure; probably whoever did the wandb integration is best. Otherwise maybe @sgugger because it's Trainer related? 


Comments:
I usually do the search directly with wandb sweeps so didn't notice this issue.
Looks good on my side, thanks!
Neptune.ai integration
Author: vbyno
Pull request: yes

# What does this PR do?
The PR integrates the trainer with [neptune.ai](https://neptune.ai/) 

To start with neptune.ai logging:
1) Set env variables:
	NEPTUNE_PROJECT
	NEPTUNE_API_TOKEN
2) Add an option that turns on Neptune logging
```
--report_to 'neptune'
```
# Who can review?
@sgugger 

Comments:
Config File
Author: ghost
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

Hi everybody, is there a way to automatically obtain the configuration file for pre-trained TensorFlow T5 network? Specifically the config.json file which I want to use to convert a tf model into pytorch using the script provided by HuggingFace.
Just to be more clear, I want something like that: https://s3.amazonaws.com/models.huggingface.co/bert/t5-small-config.json, with the specifications for my model.

Thanks in advance :)



Comments:
You get the default config file as written on AWS with 
```python 
config = T5Config.from_pretrained("t5-small")
```

If you want to change anything on the config after you can update it with a dict of your specifications:
```python 
config.update(your_dict)
```

Hello @patrickvonplaten !
I have tried to update the config file using new dict 
```
new_dict={  "d_ff": 2048,
        "d_kv": 64,
        "d_model": 512,
        "dropout_rate": 0.1,
        "eos_token_id": 1,
        "feed_forward_proj": "relu",
        "initializer_factor": 1.0,
        "is_encoder_decoder": True,
        "layer_norm_epsilon": 1e-06,
        "model_type": "t5",
        "num_decoder_layers": 6,
        "num_heads": 8,
        "num_layers": 6,
        "pad_token_id": 0,
        "relative_attention_num_buckets": 32,
        "transformers_version": "4.5.1",
        "use_cache": False,
        "vocab_size": 32128,
        "num_input_sent": 4,   # TODO: ADDED
        "max_source_len ": 25, # TODO: ADDED
        "max_target_len": 25,  # TODO: ADDED
        
     }
```
print(onfig.d_ff)  gives 2048

but:  
print(max_source_len) gives me an error  "AttributeError: 'T5Config' object has no attribute 'max_source_len'"

how to use the new added configurations after they are added?

Hey @Arij-Aladel, 

What would be the purpose of adding parameters to the config, such as `max_target_len` that are not used in the corresponding model `T5Model`?

@patrickvonplaten  I am trying to build my model depending on t5 hugging face API and I need to build my own config. Whatever the purpose is , the question still what is th benefit of updating the parameters if I can not reuse them?

Ok thanks I have solved it it was my bad, so sorry! every thing works well
What do these model parameters mean?
Author: roshan-k-patel
Pull request: no


        "params_classifier.dense.weight"
        "params_classifier.dense.bias"
        "params_classifier.out_proj.weight"
        "params_classifier.out_proj.bias"

Could someone please briefly explain these parameters to me? Using the DeBERTa model

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
How to use GPU when running run_summarization.py
Author: xuyeliu
Pull request: no

When I run run_summarization.py on my computer and my computer has 2 GPUs. But the code didn't run with gpu cuda and is very very slow. Can anyone tell me how to use transformers with GPU? Thank you very much!

Comments:
Hi @liubest 

Please make sure that your torch installation can detect the GPU. All scripts will run on GPU if it's available.
If you want to run on multiple GPUs, follow the docs [here](https://github.com/huggingface/transformers/tree/master/examples/pytorch#distributed-training-and-mixed-precision).

And if you just want to use a single GPU, you could select the device by setting `CUDA_VISIBLE_DEVICES=0" which will select the first GPU.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.