noamaanMulla-03/transformers-issues huggingface dataset:
enable mixed precision for Tensorflow training benchmarks
Author: harryzorus
Pull request: no

# 🚀 Feature request

Currently [Tensorflow Benchmarks](https://github.com/huggingface/transformers/blob/master/src/transformers/benchmark/benchmark_tf.py) implemented in transformers package only supports training in FP32 mode and FP16 support is [unimplemented](https://github.com/huggingface/transformers/blob/master/src/transformers/benchmark/benchmark_tf.py#L173). It could be helpful for the community to be able benchmark the training for the models in FP16 mode as using mixed precision greatly improves the performance of training.

## Motivation

Enabling mixed precision in training is [shown](https://medium.com/tensorflow/automatic-mixed-precision-in-tensorflow-for-faster-ai-training-on-nvidia-gpus-6033234b2540) to significantly improve the throughput of the training process. We wanted  to implement the missing support for fp16 for training in [Tensorflow Benchmarks](https://github.com/huggingface/transformers/blob/master/src/transformers/benchmark/benchmark_tf.py) to gauge the performance uplift we noticed for 1.5x improvement in performance for `bert-base-uncased` using batch size of `8` and sequence length of `128`.

## Your contribution

The code for this is implemented in [amp_tf_training_benchmarks](https://github.com/huggingface/transformers/compare/master...harishneit:amp_tf_training_benchmarks) branch in a fork. I can submit a pull request with tests if the community is interested in this.

Comments:
Sorry for the slow reply - this is definitely something we'd be interested in seeing! Can I ask why you used `tf.compat.v1.mixed_precision.enable_mixed_precision_graph_rewrite` in your fork rather than `set_global_policy` or similar? It's not necessarily wrong, but I'm curious what the tradeoffs are there!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Correct & simplify check_dummies regex
Author: cosine0
Pull request: yes

# What does this PR do?
Add necessary `\` escapes omitted for `()` and remove unnecessary `\` in the check_dummies' code matching regex.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Feature Request: Add support for --do_train/eval/predict arguments in the TF examples script for token classification
Author: MalteHB
Pull request: no

It would be truly awesome if the TensorFlow example of token classification could mimic the capabilities of the PyTorch implementation, by providing additional argument-functionality, including `--do_train`, `--do_eval` and `--do_predict`. 

Furthermore, giving the user the opportunity to provide a custom dataset through the `--predict_file` argument.  💯 

I see that you, @Rocketknight1, are already doing some awesome work, so perhaps you know whether this will be implemented anytime soon?

https://github.com/huggingface/transformers/blob/9ff672fc4d84db3b077e03ea22e2dafbd5d99fa4/examples/tensorflow/token-classification/run_ner.py#L494-L527



Comments:
Hi, you can use the `--train_file` and `--validation_file` arguments to pass custom data to the model! Are you specifically interested in doing predictions too?

Yes! It is the predictions I think would be awesome to have the option to do! 🥇 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hi, I didn't mean to let this go stale! We're planning a rewrite of our examples with the new data pipeline soon - I'll try to make sure we include the option for a `--predict_file` when that happens.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Very strange Training Data Loss Pattern when fitting MT5 for Summarization
Author: PhilipMay
Pull request: no

Hi,

I am training a MT5-small summarization model.
On tensorboard the loss function looks like this (see below).

The orange one is with batch size 3 and gradient_accumulation_steps 2 and 10 epochs. It looks normal.

The red one is with the same parameters but gradient_accumulation_steps 3 instead of 2.

The graph looks very strange. After each epoch it makes a huge jump down. Then it seems like it goes up again
between the "jumps".

For me this somehow looks like maybe a bug with gradient_accumulation.

Can someone please check this?

PS: I am using only one GPU.

PPS: The orange line is with default learing rate of 5e-5 the red graph is with learning_rate=5e-4

![grafik](https://user-images.githubusercontent.com/229382/126282933-293d8d1d-e6cc-401a-9560-7353c1d63600.png)


Comments:
Pinging @patil-suraj and @patrickvonplaten for MT5

PS: I am using only one GPU.

Hi, @PhilipMay Could please mention which script are you using for this? (Official examples/custom script, Pytrich/TF/FLax ?) 

> Hi, @PhilipMay Could please mention which script are you using for this? (Official examples/custom script, Pytrich/TF/FLax ?)

Oops. Sure @patil-suraj 

I am using this: https://github.com/huggingface/transformers/blob/master/examples/pytorch/summarization/run_summarization.py

The params are:
```bash
python run_summarization.py --model_name_or_path "google/mt5-small" --do_train --do_eval   \
--source_prefix "summarize: " --output_dir ./checkpoints --per_device_train_batch_size=3 \
--per_device_eval_batch_size=3 --overwrite_output_dir --predict_with_generate --max_source_length 800 \
--max_target_length 96 --save_strategy="epoch" --warmup_ratio=0.3  --dataset_name de_en_mit \
--num_train_epochs 10 --report_to tensorboard --gradient_accumulation_steps=3  --learning_rate=5e-4
```

I modified the script so that `--dataset_name de_en_mit` loads my custom csv dataset.

PPS: The orange line is with default learing rate of 5e-5 the red graph is with learning_rate=5e-4

Sorry that I had forgotten this info. I have added it again above.

Small update until the end of the training

![grafik](https://user-images.githubusercontent.com/229382/126457759-e4a5e72e-ba5f-4524-9a6f-5509ae6bb156.png)


@patil-suraj so what do you think about this?

@patil-suraj and news about this?

Gently pinging @sgugger on this one as well since it's based on the trainer. I'm not really sure it's a bug to be honest

I'm not sure what bug in the Trainer I should be looking at. What I see is a bumpier training with a higher learning rate, which seems completely normal.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[Examples] Add an official audio classification example
Author: anton-l
Pull request: yes

This PR adds an example for fine-tuning Wav2Vec2-like models on any labeled audio dataset.

The single-GPU script runs on one V100 and trains on [SUPERB KS](https://huggingface.co/datasets/superb#ks)
The multi-GPU one runs on 4 V100s and trains on [CommonLanguage](https://huggingface.co/datasets/anton-l/common_language) (`datasets` PR: https://github.com/huggingface/datasets/pull/2989)

Comments:
Added a multi-gpu example and trained the demo models:
* https://huggingface.co/anton-l/wav2vec2-base-keyword-spotting
* https://huggingface.co/anton-l/wav2vec2-base-langid

Ready for a full review now :)

Very nice addition!
Some weights of GPT2LMHeadModel were not initialized from the model checkpoint at 'yyy' and are newly initialized
Author: BalajiAJ
Pull request: no

Hi Everyone,

I am trying to use a fine tuned GPT Neo Model for my inferencing, am getting the below warning - 

### **_Some weights of GPT2LMHeadModel were not initialized from the model checkpoint at 'yyy' and are newly initialized..._**

I have used deep speed to fine tune the model. Because of this warning, **_the generated text during inference is giving some random meaningless text_**. Please suggest how to load those weights.


### Inference code is below (highlighted in bold is the line of code)

from transformers import GPT2Tokenizer, GPT2LMHeadModel
import torch

device = 'cuda' if torch.cuda.is_available() else 'cpu'

tokenizer = GPT2Tokenizer.from_pretrained('finetuned')
tokenizer.padding_side = "left"
tokenizer.pad_token = tokenizer.eos_token
### model = GPTNeoForCausalLM.from_pretrained("finetuned").half().to("cuda")

Thanks,
Balaji

Comments:
Hi, I'm not very familiar with GPT Neo, but should [this](https://huggingface.co/transformers/model_doc/gpt_neo.html#transformers.GPTNeoForCausalLM) be the correct model for fine-tuned GPT Neo?

Apologies for pasting the wrong code, below is the one. Thanks @qqaatw for pointing it out. 
This is the line of code where am seeing the warning.

### model = GPTNeoForCausalLM.from_pretrained("finetuned").half().to("cuda")

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Docs: TrainingArguments call incorrect
Author: UrosOgrizovic
Pull request: no

On [this](https://huggingface.co/transformers/training.html) page, there's the following code:

```
from transformers import TrainingArguments
training_args = TrainingArguments("test_trainer", evaluation_strategy="epoch")
```

However, `evaluation_strategy` needs to be an `IntervalStrategy` instead of a `string`. The way `TrainingArguments` actually needs to be called is:

```
training_args = TrainingArguments("test_trainer", evaluation_strategy=IntervalStrategy.EPOCH)
```

Comments:
No, this example is correct, as both syntax work.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Label Smoothing for Question Answering task
Author: ubamba98
Pull request: no

# 🚀 Feature request

Currently label smoothing is only applied when "label" is present in the input in compute_loss function and not in case of question answering default in Trainer class.

## Your contribution

I would like to work on this issue and submit a PR by modifying compute_loss to use label_names to apply label smoothing for question answering. 

Comments:
cc @sgugger 

I'm not entirely sure how straightforward this could be, plus it seems like a very narrow use case. I think this should be implemented independently by the user with a custom `compute_loss`  function.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Why do we need to use `Loss.repeat(eval_batch_size)` in accelerator gather loop? 
Author: thakursc1
Pull request: no

https://github.com/huggingface/transformers/blob/b1198a8440cc05f569b0bc22038993a1e5e707ab/examples/pytorch/language-modeling/run_mlm_no_trainer.py#L510

If I do not use this, and simple do acclerator.gather(loss) my code is stuck at this point. 
But if I repeat the loss it seems to work. Can you explain why is this the case ?

Why do we also later use `losses = losses[: len(eval_dataset)]` ?


Comments:
@sgugger Can you please help ?

Hi @thakursc1, Sylvain is currently off until next week - he'll answer your query when he's back from his break. Thanks for your understanding.

I'll look at why it does not work for a 0d tensor when I have a bit of bandwith (lots to do when coming back from vacation!)

The main reason we do it this way is to be able to compute the true average of the loss across all the processes at the end (otherwise we won't get the exact value of the loss). That being said, there is no reason why `accelerator.gather(loss)` in your code should be stuck. 

https://github.com/huggingface/accelerate/pull/152 should fix the problem on the Accelerate side.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Remove a duplicated bullet point in the GPT-J doc
Author: yaserabdelaziz
Pull request: yes

# What does this PR do?

Remove a duplicated bullet point in the GPT-J doc.

## Before submitting
- [x] This PR fixes a typo or improves the docs.

## Who can review?
@sgugger, @patrickvonplaten, @LysandreJik 

Comments:
Thanks!
Upcasting of attention computation for reliable pretraining of GPT-2 models
Author: lvwerra
Pull request: no

# 🚀 Feature request

In a recent [talk](https://youtu.be/AYPOzc50PHw?t=3662) about pretraining language models as part of the [Mistral](https://github.com/stanford-crfm/mistral/) project @siddk mentioned that in order to achieve stable pretraining a slight modification in the GPT-2 code is necessary. The issue is a numerical instability when training with mixed precision in the attention mechanism which can be solved by upcasting the attention computation (see [here](https://github.com/stanford-crfm/mistral/blob/53ebb290e55fe367dcaebb54ab63de4a137802db/src/models/mistral_gpt2.py#L324)).

## Motivation

Enable reliable pretraining of GPT-2 models.

## Your contribution

I can create a PR if adding this is an option.

cc @thomwolf 

Comments:
Also related are https://github.com/huggingface/huggingface_hub/issues/300 and https://github.com/stanford-crfm/mistral/issues/86 

Hey folks, sorry I'm late to the party. Replying here to just to centralize things.

The upcasting + scaled-dot product attn reordering + scaling implemented in Mistral is a pretty straightforward tweak on top of the existing GPT-2 model definition in `transformers`. The only other change we made was the weight initialization procedure for GPT-2 models, which shouldn't affect anyone downstream.

If you give me a day or two, I can do the following:
- Submit a PR to `transformers` with a flag for turning on "mistral" (upcasting of scaled-dot product attention)
- Edit the GPT2Config and Arguments to reflect this flag... ensure `.from_pretrained()` works as expected.
- Fix the GPT2 weight initialization.

This would 1) be simple, 2) be easy for anyone looking to use the Mistral models in the future, and 3) would stop us from defining a new "MistralGPT" class (which we might do anyway for v2 when we add other types of parallelism and the like.

What do y'all think?

@osanseviero @lvwerra @thomwolf @LysandreJik 

Hi @siddk, that sounds good to me. I would like to start training a larger model in the coming days so that would be very welcome on my side :)
Can I train an Rnd2GPT model through HuggingFace "Encoder-Decoder Model"?
Author: soda-lsq
Pull request: no

Hi, 

I would like to train an Rnd2GPT model, whose encoder is a randomly initialized transformer encoder and the decoder utilizes the pre-trained GPT2 model. I found that HuggingFace's "Encoder-Decoder Model" could implement the architecture of "Bert2Bert" and "Bert2GPT" models. However, my source input is not a sentence that could be represented directly through the Bert model, it may be better to be encoded through an initialized transformer encoder. 

So, I would like to know how can I achieve the Rnd2GPT model through the Hugging Face "Encoder-Decoder Model"?

Very grateful for your help! Thanks!

Comments:
You can initialize an `EncoderDecoderModel` with any autoencoding text encoder and any autoregressive text decoder. These can be randomly initialized, or you can start from pre-trained checkpoints.

So yes, it's totally possible to instantiate an `EncoderDecoderModel` with a randomly initialized BERT and a pre-trained GPT-2 model, like so:

```
from transformers import EncoderDecoderModel, BertConfig, BertModel, GPT2LMHeadModel

# randomly initialized BERT
encoder_config = BertConfig()
encoder = BertModel(encoder_config)

# pre-trained GPT-2
decoder = GPT2LMHeadModel.from_pretrained("gpt2")

model = EncoderDecoderModel(encoder=encoder, decoder=decoder) 
```

> You can initialize an `EncoderDecoderModel` with any autoencoding text encoder and any autoregressive text decoder. These can be randomly initialized, or you can start from pre-trained checkpoints.
> 
> So yes, it's totally possible to instantiate an `EncoderDecoderModel` with a randomly initialized BERT and a pre-trained GPT-2 model, like so:
> 
> ```
> from transformers import EncoderDecoderModel, BertConfig, BertModel, GPT2LMHeadModel
> 
> # randomly initialized BERT
> encoder_config = BertConfig()
> encoder = BertModel(encoder_config)
> 
> # pre-trained GPT-2
> decoder = GPT2LMHeadModel.from_pretrained("gpt2")
> 
> model = EncoderDecoderModel(encoder=encoder, decoder=decoder) 
> ```

I see! Thank you so much for your kind reply! It means a lot to me!
DistilBertTokenizer for distilbert-base-multilingual-cased is unable to encode / decode Japanese characters properly adding unnecessary characters in between
Author: shreyajain4
Pull request: no

I have used a google collab notebook.
`transformers` version used is 4.9.2

Model used :
distilbert-base-multilingual-cased 
@LysandreJik
I am facing an issue with the tokenizer: @LysandreJik 

The problem arises when using the tokenizer in the case of Japanese text. I have attached an example script of what I am going to suggest
 
![image](https://user-images.githubusercontent.com/23450481/131085602-be7ffceb-c408-4243-92b4-8abd99b4ec5f.png)

I wanted to obtain the token ids for string "祝い めでた 動画". When I used the list of token ids to obtain the corresponding string, I obtained another string "祝い めでた 動 画" seems like there  is a bug in either of the functions.

Steps to reproduce the behavior:
```python
from transformers import DistilBertTokenizer
import torch
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-multilingual-cased')
print(tokenizer.decode([  101, 5914, 15221, 1965, 12236, 20058, 2621, 115384, 102]))
print(tokenizer.decode([  101,  5914,  1906,  1965, 12236, 20058,  2621,  5618,   102]))
inputs = tokenizer("祝い めでた 動画", return_tensors="pt")
print(inputs)

```

Expected behavior:
I should have obtained the same string when I was decoding the token ids obtained from the input string.




Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Pretraining T5-v1_1 on Flax
Author: peregilk
Pull request: no

@patrickvonplaten 

In the[Flax tutorial](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling) it is recommended loading the config from t5-v1_1-base when pretraining, using:

`config = T5Config.from_pretrained("google/t5-v1_1-base", vocab_size=tokenizer.get_vocab_size())`

This basically copies this [config](https://huggingface.co/google/t5-v1_1-base/blob/main/config.json)

It seems like this is tuned for finetuning, since it has the line:
'"dropout_rate": 0.1'

Google [states](https://huggingface.co/google/t5-v1_1-base) that _"Dropout was turned off in pre-training (quality win). Dropout should be re-enabled during fine-tuning."._ Should this be modified for pretraining?

Google also state that "no parameter sharing between embedding and classifier layer". How is this achieved?


Comments:
Think both using `dropout_rate: 0.1` and not using it is fine! It also depends on the dataset you are using. The Flax T5 demo trains on Norwegian which is much smaller than English so it makes more sense here to use dropout for regularization

OK. Thanks. Im training a much larger model here, just using the Flax T5 Demo as a starting point. But if I understand you correctly, just simply changing this manually to  'dropout_rate: 0', would then be more in line with what Google describes in v1.1 - and then changing it back before finetuning. 

What about the change that is called _"no parameter sharing between embedding and classifier layer"_ that T5 v1.1 is using? I was unable to see how this is implemented in the example code. Is this a setting in config.json, or does it require changing the architecture.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.