noamaanMulla-03/transformers-issues huggingface dataset:
Adding type hints of TF:OpenAIGPT
Author: Mathews-Tom
Pull request: yes

Issue related: #16059

As the title suggests, this PR adds type hints to the `Tensorflow` `OpenAIGPT` model class.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hey @Mathews-Tom, thanks for working on providing type hints for both OpenAIGPT and CTRL! To get a review quicker, don't hesitate to ping @Rocketknight1 directly :)
Generalize decay_mask_fn to apply mask to all LayerNorm params
Author: duongna21
Pull request: yes

# What does this PR do?

Fixes the problem of `decay_mask_fn` not applying mask to all LayerNorm params. 

For example, when running `run_mlm_flax.py` with `roberta-base`'s config, current code fails to apply mask to the LayerNorm's `scale` param of the `lm_head`, 
![Imgur](https://i.imgur.com/87Oy6QJ.png)
This is because `("layer_norm", "scale")` is omitted from `decay_mask_fn`:
```python
def decay_mask_fn(params):
    flat_params = traverse_util.flatten_dict(params)
    flat_mask = {path: (path[-1] != "bias" and path[-2:] != ("LayerNorm", "scale")) for path in flat_params}
    print('flat_mask: ', flat_mask)
    return traverse_util.unflatten_dict(flat_mask)
```

Another example, `run_t5_mlm_flax.py` with `t5-base`'s config omitted all the LayerNorm params, 
![Imgur](https://i.imgur.com/isNQDSf.png) 
This is because `decay_mask_fn` only takes into account `scale` while the T5LayerNorm's param is `weight`.
```python
def decay_mask_fn(params):
    flat_params = traverse_util.flatten_dict(params)
    flat_mask = {
        path: (path[-1] != "bias" and path[-2:] not in [("layer_norm", "scale"), ("final_layer_norm", "scale")])
        for path in flat_params
    }
    print('flat_mask: ', flat_mask)
    return traverse_util.unflatten_dict(flat_mask)
```

## Fix
Generalize `decay_mask_fn` to apply mask to all params whose lowered name containing `layernorm`, `layer_norm` or `ln`.
## Who can review?

potential reviewers: @patrickvonplaten, @sgugger, @patil-suraj


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sanchit-gandhi Thank you for pointing them out! It's been done.

Amazing, thanks for the PR @duongna21!
[bugfix] Loading sharded model with torch_dtype='auto' causes TypeError
Author: koreyou
Pull request: yes

# What does this PR do?

Fixes #18314

## Before submitting
- [n] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [y] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [n] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [na] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [n] Did you write any new necessary tests?

## Who can review?

Anyone can review as it is a very small bugfix, but I will tag @sgugger based on git blame.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi there, thanks for the fix! This is a duplicate of #18061 so we will merge this one as it came first :-)

Hi. Sorry for dup and thanks for the fix! I am closing this PR (and the issue) as it is no longer necessary.
Owlvit test fixes
Author: alaradirik
Pull request: yes

# What does this PR do?

- Fixes assertion error in slow OWL-ViT integration tests
- Fixes incompatible cpu/gpu test errors caused by `OwlViTForObjectDetection.normalize_grid_corner_coordinates()`
- Sets `attention_mask` input argument  as optional

Comments:
_The documentation is not available anymore as the PR was closed or merged._
RHO loss
Author: flozi00
Pull request: no

### Feature request

https://github.com/oatml/rho-loss

Abstract

> Training on web-scale data can take months. But most computation and time is wasted on redundant and noisy points that are already learnt or not learnable. To accelerate training, we introduce Reducible Holdout Loss Selection (RHO-LOSS), a simple but principled technique which selects approximately those points for training that most reduce the model's generalization loss. As a result, RHO-LOSS mitigates the weaknesses of existing data selection methods: techniques from the optimization literature typically select 'hard' (e.g. high loss) points, but such points are often noisy (not learnable) or less task-relevant. Conversely, curriculum learning prioritizes 'easy' points, but such points need not be trained on once learned. In contrast, RHO-LOSS selects points that are learnable, worth learning, and not yet learnt. RHO-LOSS trains in far fewer steps than prior art, improves accuracy, and speeds up training on a wide range of datasets, hyperparameters, and architectures (MLPs, CNNs, and BERT). On the large web-scraped image dataset Clothing-1M, RHO-LOSS trains in 18x fewer steps and reaches 2% higher final accuracy than uniform data shuffling.

### Motivation

It's always nice to speed up training :)

### Your contribution

I am not sure if I am able to add this to transformers on myself but would be happy to give it a try with advices how to design the API 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[Flax] Fix incomplete batches in example scripts
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Currently in our Flax examples scripts, we drop the last incomplete batch during training and inference:
https://github.com/huggingface/transformers/blob/09178705101b9803e7b9ea7f79a46b4c242dd4bf/examples/flax/summarization/run_summarization_flax.py#L350
We do this for two reasons:

1. Because XLA is not shape polymorphic, forming the last batch with shape different from the preceding batches triggers a recompilation of the `pmap`'d function .
2. If the batch size is not divisible by the number devices, then the last step must be executed on a single device (or a subset of devices), potentially leading to OOMs.

During training, dropping the last batch isn't an issue: since we shuffle the data and train for multiple epochs, all of the training data is eventually used and the effects of dropping the last batch amortised.

However, during evaluation and prediction, dropping the last batch leads to incorrect results: since we don't account for the examples in the last batch, we do not evaluate over the whole dataset, and thus have partial results.

This PR corrects for the incomplete batches in the relevant Flax training examples.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Just waiting to double check that the slow tests pass from [`test_flax_examples.py`](https://github.com/huggingface/transformers/blob/main/examples/flax/test_flax_examples.py) before merging. Working with @patil-suraj to verify this ✅

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@patil-suraj @sanchit-gandhi can we merge this one?

Just verifying the slow tests from [`test_flax_examples.py`](https://github.com/huggingface/transformers/blob/main/examples/flax/test_flax_examples.py) pass on a 
v3-8
 v100 GPU!
Checkpoint sharding
Author: sgugger
Pull request: yes

# What does this PR do?

This PR introduces the ability to create and load sharded checkpoints. It introduces a new argument in `save_pretrained` that controls the maximum size of a checkpoint before being auto-sharded into smaller parts (which defaults to 10GB after internal discussion, which should be good with the Hub and environment with low RAM like Colab).

When the model total size is less than this maximum size, it's saved exactly like before. When the model size is bigger, while traversing the state dict, each time a new weight tips the size above that threshold, a new shard is created. Therefore each shard is usually of size less than the max size, but if an individual weight has a size bigger than this threshold, it will spawn a shard containing only itself that will be of a bigger size.

On the `from_pretrained` side, a bit of refactoring was necessary to make the API deal with several state dict files. The main part is isolating the code that loads a state dict into a model in a separate function, so I can call it for each shard. I'm leaving comments on the PR to facilitate the review and I will follow up with another PR that refactors `from_pretrained` even more for cleaning but with no change of actual code.

cc @julien-c @thomwolf @stas00 @Narsil who interacted in the RFC.

Linked issue: #13548 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger is it possibile to apply sharding to a current pretrained model that is bigger than 10gb (let's say t0pp like), using 

load_pretrained-> save_pretrained

To save a sharded version, and the again

load_pretrained

That should handle therefore load the sharded checkpoint?

Thanks

Yes, that's exactly what you should do!
We'll also create a new branch of the t0pp checkpoint with a sharded checkpoint (we can't do it on the main branch or it would break compatibility with older versions of Transformers).

Hahaha amazing! Where is it? 

@sgugger we could possibly upload the sharded checkpoint **in addition to** the current checkpoint in the same repo branch no?

I thought that's what we wanted to do to preserve backward compat while still upgrading those big models


If you put them in the same branch, `from_pretrained` will only download the full checkpoint as it's the first in order of priority.

I see. should we consider changing this behavior? we've seen that git-branches are not super practical for those large models (cc @osanseviero )

(yep, i know this is going to require to call something like /api/models/xxx at the start of `from_pretrained`... :) )

or maybe we can say that we'll just push sharded models from now on (for example the bigscience models will be sharded-only) I think that's actually fine

or we can default to the shared model for the new version of `transformers`?

yep that was what i was suggesting but implementation wise is a bit more complex (and affects all model repos not just the sharded ones)
Swin Transformer V2
Author: RyanHuangNLP
Pull request: no
Labels: name: New model

name: Good First Issue

### Model description

[Swin Transformer V2: Scaling Up Capacity and Resolution](https://arxiv.org/pdf/2111.09883.pdf)

repo origin: [Swin Transformer V2](https://github.com/microsoft/Swin-Transformer#updates)

repo timm: [Swin Transformer V2](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/swin_transformer_v2.py)

all the model the pretrain is ready

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

repo origin: [Swin Transformer](https://github.com/microsoft/Swin-Transformer#updates)

Comments:
Marking this as a good first issue as Swin v2 only adds a couple of small design improvements compared to Swin v1.

One could use the [add new-model-like](https://github.com/huggingface/transformers/tree/main/templates/adding_a_new_model#add-new-model-like-command) feature to copy all Swin files, and then implement Swin v2 by tweaking these files. 

Hey! Can I give it a shot?

@NielsRogge I would like to add this model.  

Hi :) sure, maybe you can give me your email addresses such that we can set up a Slack channel for coordination.

> Hi :) sure, maybe you can give me your email addresses such that we can set up a Slack channel for coordination.

Here is mine : ritiknandwal021@gmail.com

Hey @NielsRogge, I'd like to help out as well. My email is srinivasansabarish@gmail.com

my email is joaquinrivero94@gmail.com

Thanks, I'll create one. You should receive an invite later today

Hi all is this work complete, I'd love to help if possible.
Run ViT examples on PyTorch XLA out of the box
Author: miladm
Pull request: yes

Check if Vision Transformers (ViT) model and its example script are supported on PyTorch/XLA out of the box. If not, produce the code to bring the needed support.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15599). All of your documentation changes will be reflected on that endpoint.

It turns out the model is mostly supported. 
I had to make a couple of change to successfully run the model on the TPU using PyTorch/XLA. Below is the command to repro:

```
python examples/pytorch/xla_spawn.py --num_cores 1 examples/pytorch/image-classification/run_image_classification.py   \
--dataset_name cats_vs_dogs     --output_dir ./cats_vs_dogs_outputs/  \
--remove_unused_columns False     --do_train     --do_eval  --learning_rate 2e-4 \
--num_train_epochs 5     --per_device_train_batch_size 32     --per_device_eval_batch_size 32 \
--logging_strategy steps     --logging_steps 10     --evaluation_strategy epoch     --save_strategy epoch  \
--load_best_model_at_end True     --save_total_limit 3     --seed 1337
```

Hi @sgugger. 

Basic questions: I just locally ran the code quality test. Not sure if I understand what it requires to correct the issue given the small size of my code change. How do you usually handle this issue?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I dropped the ball here.
@steventk-g can please follow up on this PR and help land it in HF stack?
Faster mBART finetuning
Author: sshleifer
Pull request: no
Labels: name: Help wanted

name: seq2seq

name: Examples

Goal: Get BLEU 20 in 1 epoch on wmt-en-ro.
Can't run a bs=1 without `--freeze_embeds`.
1 epoch takes 6H on 16GB GPU with fp16, with `--freeze_embeds` and `--freeze_encoder`. Max bs=4


Ideas:

- [ ] Dataset that fits as many sentences as possible into an example, to increase gpu utilization.
- [ ] Only store embeddings once
- [ ] prune embeddings: https://github.com/pytorch/fairseq/issues/2120
- [ ] `label_smoothing=0.1`
- [ ] TPU?


Fairseq finetune command:
https://github.com/pytorch/fairseq/issues/2179


Comments:
Hi @sshleifer , can embed pruning affect accuracy(bleu) ? TPU option seems good as it won't result in less accuracy 

https://github.com/pytorch/fairseq/issues/2120#issuecomment-647915216 suggests it costs 2 BLEU points.

One of our goals with examples is to make train loops that people can run on their hardware+data. If we can get 36 on a 16GB gpu that is super useful, just like `--freeze_encoder` and `--freeze_embeds` are useful, even if they probably hurt final performance a bit.


Broke into smaller issues, but leaving this open in case people have other ideas!

note: fairseq wmt_en_de batch wps=14721.8 on V100 with https://github.com/pytorch/fairseq/issues/2506#issuecomment-678630596

check tpb/seconds for seq2seq/finetune.py



I am trying to fine-tune on wmt-en-ro (facebook/mbart-large-cc25) using Colab and Kaggle 16GB gpu. On Colab, I get cuda oom and on Kaggle I get out of disk apace (20 GB limit). Is there a way I can skip checkpoints to fit the process in 20GB disk space? 
I even tried 1 bs, 8 max len, fp16 and freeze_encoder.

High level, I would use an opus-mt model instead of mbart for most tasks. They are smaller and tend to be nearly as good at translation, if not better. I have run ~50~ experiments finetuning mbart on wmt-en-ro and it was not a particularly pleasant experience.

Disk space: You can see what happens if you remove `checkpoint_callback` from 
https://github.com/huggingface/transformers/blob/9336086ab5d232cccd9512333518cf4299528882/examples/seq2seq/finetune.py#L362

and just call `model.model.save_pretrained`  `model.model.half().save_pretrained` after.






Thanks for the advice @sshleifer - just wanted to try it out and test other language pairs. 
On Kaggle, as I just found out, the trick is to create the output_dir outside the working directory (where total disk space is just 5GB). Kaggle won't save it with kernel commit though.


Cool! LMK if you have good results!

I got it working using a small subset of data, then went on training English -> Arabic translator (just a proof of concept). Model uploaded to [HF](https://huggingface.co/akhooli/mbart-large-cc25-en-ar).

Hi, I had written the code to do vocab pruning and because of the number of people wanting help with it I converted the code into a standalone library. I hope its okay if I link it here for other people to find. 

[Link to repo.](https://github.com/IamAdiSri/hf-trim)

Referencing issues #5896 #6132
Unexpected behavior when using PubMedBERT with AutoModelForMaskedLM
Author: rahuln
Pull request: no
Labels: name: wontfix

## Information

I'm getting some strange behavior when using `AutoModelForMaskedLM` with PubMedBERT to impute masked tokens. The screenshot below shows a simple example where I would expect PubMedBERT to give reasonable values, but the suggested tokens are really strange. As shown just below this, `bert-base-uncased` seems to behave reasonably.

<img width="1013" alt="Screen Shot 2020-11-12 at 10 28 13 AM" src="https://user-images.githubusercontent.com/3958904/98984623-4e96b400-24d7-11eb-9a5f-faac70b1b399.png">

Same code as above, in text:

```python
import torch
from transformers import AutoTokenizer, AutoModelForMaskedLM

model_name = 'microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name).to('cuda')

text = f'Heart disease is {tokenizer.mask_token} leading cause of death in the United States.'
tokenized = tokenizer(text, return_tensors='pt').to('cuda')
print(tokenizer.convert_ids_to_tokens(tokenized.input_ids.squeeze()))
output = model(**tokenized, return_dict=True)
output.logits.size()
print(tokenizer.convert_ids_to_tokens(torch.topk(output.logits[0, 4, :], 10).indices))

model_name = 'bert-base-uncased'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name).to('cuda')

tokenized = tokenizer(text, return_tensors='pt').to('cuda')
print(tokenizer.convert_ids_to_tokens(tokenized.input_ids.squeeze()))
output = model(**tokenized, return_dict=True)
print(tokenizer.convert_ids_to_tokens(torch.topk(output.logits[0, 4, :], 10).indices))
```

## Environment info

- `transformers` version: 3.3.1
- Platform: Linux-3.10.0-957.5.1.el7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I know it's an old issue but I just came across this page.

The original PubMedBERT checkpoint didn't have the mask prediction heads, but we updated the checkpoint ~10 months ago 
https://huggingface.co/microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract

Also, we have a new biomed+clinical domain-specific model if you're interested: https://huggingface.co/microsoft/BiomedVLP-CXR-BERT-general

@rahuln
updated translation
Author: banda-larga
Pull request: yes

Left the term fine-tuning since there is no correct translation into Italian and the English term is generally used. The same was done with some terms like "learning rate"

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @banda-larga @sgugger we discuss a lot with @mfumanelli for the better italian translation. For me this version is ok, you can merge the PR