noamaanMulla-03/transformers-issues huggingface dataset:
New training arg: warmup_ratio
Author: GanjinZero
Pull request: no
Labels: name: Feature request

# 🚀 Feature request

When training or fine-tuning a transformer model, people usually warmup for 10% training steps. For now, transformers only provide the parameter of warmup_steps. A warmup_ratio parameter can be helpful, it means warmup for some percentage of total training steps.

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

To use the parameter of warmup_steps, people need to know the total training steps. When people use the training epoch parameter instead of max_steps, it is hard to know the total training steps. A warmup_ratio parameter get rid of people knowing total training steps. Another reason for using warmup_ratio parameter is it can help people write less hard code. People have different total training steps for different dataset, but people usually set warmup_ratio as 10% as default.

Original Usage may like this:
`
python run_ner.py --data_dir some_data_dir \
--model_name_or_path some_model \
--output_dir some_output_dir \
--max_seq_length 512 \
--num_train_epochs 10 \
--warmup_steps 35 \
--per_device_train_batch_size 8 \
--do_train \
`

New usage may like this:
`
python run_ner.py --data_dir some_data_dir \
--model_name_or_path some_model \
--output_dir some_output_dir \
--max_seq_length 512 \
--num_train_epochs 10 \
--warmup_ratio 0.1 \
--per_device_train_batch_size 8 \
--do_train \
`

Also, we can merge warmup_step and warmup_ratio into one parameter. If user input a number 0 <= x < 1, it will be considered as warmup_ratio. If user input an interger, it will be considered as warmup_step.

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

I can submit a PR to complete this feature. If similar feature is alreadly in this repo, please just close this issue.

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Well, I think it is a useful feature since PLMs like RoBERTa require setting task-specific warmup-steps, which is annoying. But the design of merge warmup-steps and warmup-ratio together may not be a good idea.  

Hi @TobiasLee,
Agreed with not merging both `warmup_steps` and `warmup_ratio` into a single parameter. It seems cleaner to give higher precedence to one over the other in case both are given by user.  

I have raised a [PR](https://github.com/huggingface/transformers/pull/10229) with the same implemented.  
Your review is appreciated!
Request to add Switch Transformer
Author: coderpotter
Pull request: no
Labels: name: New model

Google has come up with yet another transformer: https://arxiv.org/pdf/2101.03961.pdf 

Comments:
 Google released the source code for transformer-based mixture-of-experts (the switch architecture): https://github.com/tensorflow/mesh/blob/master/mesh_tensorflow/transformer/moe.py

According to https://www.infoq.com/news/2021/02/google-trillion-parameter-ai/ the model weights are not available yet.
[trainer] implement support for full fp16 in evaluation/predict
Author: stas00
Pull request: yes

This PR allows users to use `model.half()` in evaluation/predict, which may or may not deliver results identical to fp32 eval. The outcome depends on how the model was trained and the application. e.g. if I use `--label_smoothing` with t5-small I get `eval loss = nan`, but bleu scores are exactly the same as with fp32.

### Need

Besides users asking for it in the past, the real need that prompted me to implement this is based on this Issue:  https://github.com/huggingface/transformers/issues/10161. To explain - DeepSpeed trains in fp16, while keeping master copy of fp32 weights on cpu, which allows fitting a model like t5-11b (45GB in params) onto a 40GB gpu (only 22.5GB in fp16). But then the user wants to eval and deepspeed is of no use here at the moment. So we need to give a way to users to run full fp16 in eval, which is what this PR proposes.

This PR:
* [x] adds `is_in_train` public Trainer attribute which helps to tell whether `evaluation` is running on its own, or called from `train`
* [x] adds `--fp16_full_eval` to enable the full fp16 mode under eval/predict (while `full-fp16` would read better, I picked the name starting with `--fp16_` to align/group well with other3 `--fp16_*` args.
* [x] adds the first test that measures gpu mem deltas - let's hope it proves to work across different gpus 

The logic is a bit tricky since we must not `model.to(device)` before `model.half()` or otherwise the model loading will OOM, but I hope I was able to keep it simple and not error-prone. Perhaps instead of replaying `place_on_device` logic at the end of `train` in the deepspeed clean up section - it'd be better to re-play the full logic in the `predict_loop`? So that each stage can decide at its beginning how and when to put the model on device. 

A few small related fixes:
* [x] fixes `_wrap_model` to do nothing under deepspeed
* [x] fixes `--fp16` help to remove apex-only comment, as it's outdated.

Questions:
* [ ] Should I add a log, saying that half is used at `model.half()` activation
* [ ] I put it inside `prediction_step` which seems to be the right place, it won't run if it's a re-entrant eval-inside-train
* [ ] as the `inputs` are `ints` I don't think we need to switch them to `half()` as well.

@sgugger 

Comments:
BERT with regression head cannot fit one datapoint
Author: lucky-bai
Pull request: no

Hi, I am trying to use BERT for a token-level regression task (predict a continuous value for each token), and I'm having trouble getting my model to train. As a debugging strategy, I'm trying to get it to overfit one datapoint, which should be easy, but it's failing that also.

Here is a minimal reproducing source code. The model is BERT which feeds into a ``nn.Linear(768, 1)``. To keep things simple, I am feeding it a sequence of length 1, and training it to output 0.5.

```
import torch
import transformers

class RegressionModel(torch.nn.Module):
  def __init__(self):
    super(RegressionModel, self).__init__()

    self.bert = transformers.BertModel.from_pretrained('bert-base-uncased')
    self.linear = torch.nn.Linear(768, 1)

  def forward(self, X_ids):
    return self.linear(self.bert(X_ids).last_hidden_state)


model = RegressionModel().cuda()
model.train()

opt = torch.optim.Adam(model.parameters())
X_ids = torch.LongTensor([[12345]]).cuda()
Y_true = torch.Tensor([[0.5]]).cuda()

steps = 0
while True:
  opt.zero_grad()
  Y_pred = model(X_ids)
  loss = (Y_true - Y_pred)**2
  loss.backward()
  print(steps, Y_pred, float(loss))
  steps += 1
  opt.step()
```

After a few thousand iterations, it predicts around 0.5 but not exactly:

```
2315 tensor([[[0.4669]]], device='cuda:0', grad_fn=<AddBackward0>) 0.0010972624877467752
2316 tensor([[[0.5115]]], device='cuda:0', grad_fn=<AddBackward0>) 0.00013136999041307718
2317 tensor([[[0.4788]]], device='cuda:0', grad_fn=<AddBackward0>) 0.00045129822683520615
2318 tensor([[[0.4658]]], device='cuda:0', grad_fn=<AddBackward0>) 0.0011675604619085789
```

Note that if I set `model.eval()` instead of `model.train()`, then the model is able to fit as expected (predicts 0.5000 after about 200 iterations). The problem exists in the RoBERTa model as well.

## Version information

- `transformers` version: 4.3.2
- Platform: Linux-4.15.0-112-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.5
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!
Labels are now added to model config under id2label and label2id
Author: marma
Pull request: yes

Fixes huggingface/transformers#2487

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2945?src=pr&el=h1) Report
> Merging [#2945](https://codecov.io/gh/huggingface/transformers/pull/2945?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/53ce3854a16ad2a715bc6ac8af3e30c18b5a1d11?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2945/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2945?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master   #2945   +/-   ##
======================================
  Coverage    76.1%   76.1%           
======================================
  Files          98      98           
  Lines       15946   15946           
======================================
  Hits        12136   12136           
  Misses       3810    3810
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2945?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2945?src=pr&el=footer). Last update [53ce385...0a5ab3f](https://codecov.io/gh/huggingface/transformers/pull/2945?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


I had noticed this indeed, thanks for fixing!

@marma So the labels are supposed to be the actual labels instead of placeholders like "Label_0", "Label_1" ... right? I tried it out in 3.3.1 but it still generates the placeholder labels in config. I guess I am missing something? I would be grateful if you could help! :) 



I would also be interested in learning how I can store the actual label2id and id2label dictionaries along with a pre-trained model. Is this possible?

@konstantinmiller & @naveenjafer You can pass label2id and id2label to config then pass that config to model like in below snippet:
```py
config = AutoConfig.from_pretrained(
        model_args.config_name if model_args.config_name else model_args.model_name_or_path,
        num_labels=num_labels,
        id2label={i: label for i, label in enumerate(label_list)},
        label2id={label: i for i, label in enumerate(label_list)},
        finetuning_task=data_args.task_name)
model = AutoModelForSequenceClassification.from_pretrained(
        model_args.model_name_or_path,
        from_tf=bool(".ckpt" in model_args.model_name_or_path),
        config=config )
```
Making TF TransfoXL model compliant with AMP
Author: jplu
Pull request: yes

# What does this PR do?

This PR makes the TF TransfoXL model compliant with AMP. All the slow tests are passing as well for these models.

These two models cannot be XLA compliant for now, as it seems that tf.where cannot be used in XLA if the x and y parameters are None. See the _get_global_attn_indices method which has this case. I have opened [an issue](https://github.com/tensorflow/tensorflow/issues/47211) on the TF repo in order to ask if it is an expected behavior or a bug.


Comments:
logging_epochs argument for TrainingArguments
Author: hasansalimkanmaz
Pull request: no
Labels: name: Feature request

# 🚀 Feature request

There is no `logging_epochs` argument in `TrainingArguments`. When someone wants to train with `EvaluationStrategy.EPOCH`, he/she wants to see the logs after each epoch. Currently it is not possible. 

## Motivation

Better logging for training

## Your contribution

If I have time, I would like to add it. However, I am not available in a coming couple of weeks.


Comments:
Just like `evaluation_strategy` chooses between `'steps'` and `'epoch'`, to maintain consistency I think it is better to introduce either of:
1. a new enumeration `LoggingStrategy` with values
    1. `'epoch'` for per-epoch functionality
    2. `'steps'` functionality by falling back on `logging_steps`
2. a new bool argument `log_per_epoch` to decide between `epoch` or `steps` functionality and proceed similarly as above

@hasansalimkanmaz If you're still occupied, is it okay if I take a stab at this?

Feel free to go ahead @tanmay17061 I am still busy with some other staff. Thanks for your interest.
Return correct Bart hidden state tensors
Author: joeddav
Pull request: yes

# What does this PR do?

Fixes #8601. When `output_hidden_states=True`, hidden states transposing is done _before_ being fed through the next layer. This ensures that the returned hidden state tensors lie upstream in the graph from the model outputs (allowing their gradients to be computed).

Comments:
## UPDATE:

This PR is ready for review.

@sgugger, @LysandreJik - this very-well documented issue: https://github.com/huggingface/transformers/issues/8601 shows that for some models the gradient of the outputted `hidden_states` and the `attentions` cannot be computed because the tensors are excluded from the computation graph via some `transpose`, `permute`, or `slice` operations. 
@joeddav found a great fix for Bart and I applied the same fix now for all other models and added a test.

The only models that are not capable of keeping the gradient in `attentions` and `hidden_states` are 
- Longformer -> chunked attention slice operations don't allow keeping grad
- Reformer -> customized  backward doesn't help here
- ProphetNet -> Decoder part can't keep grad because of slice operations
- TransfoXL, XLNet -> two stream attention doesn't allow to keep the grad either

All other models can keep the grad which is ensured by the test.
Random Word Replacement Probability
Author: BradSegal
Pull request: no

Hi,

It appears that the token masking function replaces tokens with random words 50% of the time instead of the commented 10%. 

https://github.com/huggingface/transformers/blob/709c86b5a925f1efe650e24ee8b1f52bdc5a3acb/src/transformers/data/data_collator.py#L381

Comments:
Nevermind, didn't see the not for replaced indices so it's 50% of the remaining 20% after masking
set tgt_lang of MBart Tokenizer for summarization
Author: HeroadZ
Pull request: yes

# What does this PR do?

To set tgt_lang of MBart Tokenizer for summarization. 
Otherwise, the error `AttributeError: 'MBartTokenizerFast' object has no attribute 'tgt_lang'` occurred.
I have read your discussion and know that you will modify the part of MBart later. So this PR will be meaningless at that time. 
But at least it will be useful now :)

Sorry that I didn't take any tests, but it works well on my machine for summarization using MBart.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@stas00 @patil-suraj  @patrickvonplaten @ @sgugger



Comments:
Thanks a lot for fixing this!

Other models/tasks have this issue as well https://github.com/huggingface/transformers/issues/10292

These features require tests. Without tests this is an endless work-work-work - otherwise we keep on breaking what was working before.
fixes #10303
Author: cronoik
Pull request: yes

# What does this PR do?

Fixes #10303 


## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

Documentation: @sgugger


Comments:
Thanks for fixing!
convert_tokens_to_string documentation bug
Author: cronoik
Pull request: no

The [documentation](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.convert_tokens_to_string) states that convert_tokens_to_string would convert _a sequence of token ids in a single string._ 

That is actually not correct as it converts a sequence of tokens. The method that converts a sequence of token ids is the decode method.

Documentation: @sgugger





Comments:
Minor documentation issue
Author: gwc4github
Pull request: no

## Minor issue in the Fine tuning docs
In the "Named Entity Recognition" section of the "Summary of tasks" documentation page there are some bad links.
Here is a link to the section: https://huggingface.co/transformers/task_summary.html#named-entity-recognition

The text in question is:
##Named Entity Recognition
Named Entity Recognition (NER) is the task of classifying tokens according to a class, for example, identifying a token as a person, an organisation or a location. An example of a named entity recognition dataset is the CoNLL-2003 dataset, which is entirely based on that task. If you would like to fine-tune a model on an NER task, you may leverage the run_ner.py (PyTorch), run_pl_ner.py (leveraging pytorch-lightning) or the run_tf_ner.py (TensorFlow) scripts.

## ISSUE:
The links for the following give a 404 error:
       run_pl_ner.py, run_tf_ner.py



Comments:
Thanks for flagging! Those have not been updated in a while so I made a pass over that file.

I still see the bad links.  Is the change getting pushed/merged later?

It will only be seen in the [master documentation](https://huggingface.co/transformers/master/) for now. At the next release, it will become visible in the stable documentation.

Will the new run_ner.py work with PyTorch and TF?  PyTorch-Lightening too?
Documentation of the decode method is missing
Author: cronoik
Pull request: no

The tokenizer documentation [page](https://huggingface.co/transformers/main_classes/tokenizer.html) is generated from the following files:

- tokenization_utils_base.py  
- tokenization_utils_fast.py  
- tokenization_utils.py  

At least the documentation of the decode method is missing even if it is properly documented in the [source file](https://github.com/huggingface/transformers/blob/9a7e63729f3ff6ddf065fd0d443421e46b1a2ffb/src/transformers/tokenization_utils_base.py#L3099).

@sgugger 
Could you please have a look?






Comments:
This has been fixed a few days ago, I believe. Look at the [master doc tokenizer page](https://huggingface.co/transformers/master/main_classes/tokenizer.html) (the stable documentation is only updated at each release).

Yes, you are right.
fix typo in conversion script
Author: tagucci
Pull request: yes

# What does this PR do?

Fix typo in `convert_fsmt_original_pytorch_checkpoint_to_pytorch.py`

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.
@stas00

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Wonderful! Thank you for this fix, @tagucci! 

(I tweaked your PR to run `make style` to appease to auto-formatters to have CI pass)
BERT for speech
Author: arunraja-hub
Pull request: no

How can I use HF's BERT models for speech-to-text training?

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Also, [here's the doc](https://huggingface.co/transformers/model_doc/wav2vec2.html#transformers.Wav2Vec2ForCTC) for `Wav2Vec2ForCTC` which seems to be the model you're interested in.

Thanks!
[Question] Add a new token to tokenizer and bart model
Author: mwang98
Pull request: no

Hi,  
I have extended the word embedding of a tokenizer and a bart model through `tokenizer.add_token()` and `model.resize_token_embeddings(len(tokenizer))`. Because the ground truth consist of a newly added token, the dimension of the decoder output should be extended as well.

But I can't figure out how to extend the model. Can anyone give me some help?

Thanks in advance ! 💯 

Comments:
Hello! You should use the [resize_token_embeddings](https://huggingface.co/transformers/main_classes/model.html?highlight=resize_token_embeddings#transformers.PreTrainedModel.resize_token_embeddings) method for that. Will add that to the documentation.
Making TF Longformer-like models compliant with AMP
Author: jplu
Pull request: yes

# What does this PR do?

This PR makes the TF Longformer-like models compliant with AMP. All the slow tests are passing as well for these models.

These two models cannot be XLA compliant for now, as it seems that `tf.where` cannot be used in XLA if the `x` and `y` parameters are `None`. See the `_get_global_attn_indices` method which has this case. I have opened [an issue](https://github.com/tensorflow/tensorflow/issues/47211) on the TF repo in order to ask if it is an expected behavior or a bug.


Comments: