noamaanMulla-03/transformers-issues huggingface dataset:
[Pipeline] skip feature extraction test if in `IMAGE_PROCESSOR_MAPPING`
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes the [following failing test](https://app.circleci.com/pipelines/github/huggingface/transformers/53884/workflows/8df76bfb-b6d2-493e-afdf-257b59672b02/jobs/648580)

## Context:

Currently `FeatureExtractionPipelineTests` are skipped for multi-modal models by checking if the model config is in `FEATURE_EXTRACTOR_MAPPING`. The check is done [on this line](https://github.com/huggingface/transformers/blob/1543cee7c8c95ef47f832b1f37625ba2923c4994/tests/pipelines/test_pipelines_feature_extraction.py#L181) 
Recent vision and multimodal models will deprecate the usage of `xxxFeatureExtractor` in favor of `xxxImageProcessors`. For [Blip](https://github.com/huggingface/transformers/pull/20716), the test fails because `BlipFeatureExtractor` is not implemented at all in favor of `BlipImageProcessor`.

## Why this fix is relevant?

Blip seems to be the first multimodal model that relies on `xxxImageProcessor` only. 

cc @Narsil @amyeroberts @NielsRogge 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks
I am not sure about how do we want to approach that exactly, but I think that's the plan at some point cc @amyeroberts :D 

> Thanks I am not sure about how do we want to approach that exactly, but I think that's the plan at some point cc @amyeroberts :D

I think it's good to disambiguate Audio from Vision (both are currently named `FeatureExtractor` I think).

In that regard I'd like to stress to include no-code (or almost none) into `Processor` the general class that encapsulates `Tokenizer`, `FeatureExtractor` and `ImageProcessor` . It's great for demos and quick hacks, but it's much more cumbersome to reason about within a lib, as it's impossible to know what it should be able to do since it's by definition not standard. (It doesn't have any invariant).

For instance `Tokenizer.encode(text)` is always going to be a valid call and will return ids (that's an invariant).
Add Universal Segmentation class + mapping
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds the `AutoModelForUniversalSegmentation` class and corresponding mapping. Models that can be added to this mapping include DETR, MaskFormer, Mask2Former and OneFormer.

To do:

- [x] update pipeline

@sgugger for some reason make fixup complains:

```
Traceback (most recent call last):
  File "/Users/nielsrogge/Documents/python_projecten/transformers/utils/check_repo.py", line 827, in <module>
    check_repo_quality()
  File "/Users/nielsrogge/Documents/python_projecten/transformers/utils/check_repo.py", line 816, in check_repo_quality
    check_models_are_in_init()
  File "/Users/nielsrogge/Documents/python_projecten/transformers/utils/check_repo.py", line 369, in check_models_are_in_init
    raise Exception(f"The following models should be in the main init: {','.join(models_not_in_init)}.")
Exception: The following models should be in the main init: MaskFormerForUniversalSegmentation.
```

Comments:
I think the error message is pretty clear on what to do. The model would also need to be added to the doc page of maskformer if we go through with this.

I'm not convinced we should however. While adding a new auto-model API could make sense (I'd wait to have more than one model though), renaming the model class a year after the model has been released is not something we should do (the same way we keep `GPTLMHeadModel` for instance). 

_The documentation is not available anymore as the PR was closed or merged._

I'm unsure why you keep changing more of the `MaskFormerForInstanceSegmentation`. I think I may have been unclear in my previous comments. I am against changing that name in a model that has been around for 10 months now, for the same reason we did not change the name of `GPTLMHeadModel` or `BertLMHeadModel` even if those are not ideal.

@sgugger I'll keep the old names, but add them to the same (new) mapping.
Fix gradient clipping on XLA device
Author: ymwangg
Pull request: yes

# What does this PR do?
This PR fixed the gradient clipping logic on XLA device. We found all_reduce is wrongfully disabled in fp16 mode when `max_grad_norm=0` on XLA gpu device. To be consistent with native pytorch behavior, all_reduce should be placed immediately after calling `self.training_step`.

Tested with the run_mlm.py example script on 8 Nvidia V100 GPU:
```sh
GPU_NUM_DEVICES=8 python -m torch_xla.distributed.xla_spawn --num_gpus 8 run_mlm.py \
    --model_name_or_path bert-base-uncased \
    --dataset_name wikitext \
    --dataset_config_name wikitext-2-raw-v1 \
    --overwrite_output_dir true \
    --output_dir /tmp/test-mlm \
    --per_gpu_train_batch_size 16 \
    --do_eval \
    --fp16 true \
    --max_grad_norm 0 \
    --do_train \
    --num_train_epochs 3
```
Results of final training losses:
| Backend | fp16 | fp32       | fp16+max_grad_norm=0| fp32+max_grad_norm=0|
| --- | ----------- | --------| ----------- | --------| 
| pytorch cuda| 1.8649 | 1.8575 | 1.8753 | 1.8694 |
| torch_xla cuda| 1.86 | 1.8576 | 1.8694 | 1.867|

cc @sgugger

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20116). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Roberta model seemingly unable to take embeddings as input
Author: shahbuland
Pull request: no

### System Info

Python 3.9 on linux, transformers 4.24.0



### Who can help?

@lysandrejik

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Trying to implement cross encoder with RoBERTa. I embed token ids separate from the model forward call:
```python
model = AutoModel.from_pretrained("roberta-base")
text_features : TensorType["batch", "sequence_length", "d_model"] = model.embeddings(text.input_ids)
embs = .... # same shape as text features
```
AFAIK this incorporates word embeddings, positional embeddings and special token embeddings, since those are all in the models embeddings. Sequence length here is max_posititon_embeddings from the models config. I then add on ViT embeddings to start of sequence and truncate accordingly (shape of tensor stays the same)
```python
out = model(
            inputs_embeds=embs,
            attention_mask=attn_mask,
            output_hidden_states=True,
            return_dict=True
        )
```
This gives error: 
```python
IndexError: index out of range in self
```
The traceback (image linked [here](https://media.discordapp.net/attachments/738882678711123980/1039276109423984711/unknown.png?width=719&height=146)) suggests it is trying to perform word and position embeddings again, and an error is occurring when position embeddings are called. If I'm understanding correctly, since I already did embeddings, and am providing them rather than tokens, it should not be calling on position embeddings again, yes? Running the same code with "bert-base-uncased" functions as expected with no errors.

### Expected behavior

Expect a successful forward call through model using the embeddings provided

Comments:
Even if you provide input embeddings, the RoBERTa model will still add the token type embeddings and position embeddings.

As for the exact reason you get an error, we would need a full reproducer to be able to investigate.

> Even if you provide input embeddings, the RoBERTa model will still add the token type embeddings and position embeddings.

Is there any way to circumvent this? In my use-case position embeddings would probably be harmful to training.






You'll need to modify the model code for that.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Document BLOOM lm_logits original training behavior
Author: shijie-wu
Pull request: yes

# What does this PR do?

~Compute softmax of Bloom in fp32 during half percision~

Document BLOOM lm_logits original training behavior

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@thomasw21 @stas00 @TevenLeScao

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20027). All of your documentation changes will be reflected on that endpoint.

cc @thomasw21 and @younesbelkada but I don't think this is necessary as bfloat16 is more numerically stable than float16?

we noticed considerable performance difference between softmax in bf16 and softmax in fp32 in an internal bloom-based model. since during training the softmax is conducted in fp32, this change better reflects the model behavior in megatron-deepspeed.

@shijie-wu can you please share a bit more about how you run the model? 

We redid the experiment regarding fp32 in SA. It seems like the gap we observed earlier is caused by some other issues. I have removed that part of the PR. After discussion with @thomasw21, instead of enforcing a type conversion, this PR will instead document the original behavior so that advanced users could recover it.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add Swin2SR
Author: NielsRogge
Pull request: no
Labels: name: New model

name: Good First Issue

### Model description

Swin2SR is a Swinv2-based model for image super resolution and compression.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

https://github.com/mv-lab/swin2sr

Comments:
If there is consensus for this, can I work on it?

Sure!

cool, I will start with this.

Check out some tips on contributing a model here:

* https://github.com/huggingface/transformers/tree/main/templates/adding_a_new_model#add-new-model-like-command
* https://huggingface.co/docs/transformers/contributing
* https://huggingface.co/docs/transformers/add_new_model


Am I supposed to add the model here:
https://github.com/huggingface/transformers/blob/main/src/transformers/models/swinv2/modeling_swinv2.py

Also, is there any Super resolution models already present in Transformers?

Hi,

No each model in the library has its own folder and implementation files. We duplicate a lot of code in favor of easily readable code.

There's no super resolution model already available in Transformers, it would be the first one.

Thanks.
Can I reuse the code from https://github.com/mv-lab/swin2sr repo in a new folder or  build on top of the model from here `modeling_swinv2.py` ?

You can start from modeling_swinv2.py, copy it over and tweak it for the new model.
Add Swin2SR
Author: NielsRogge
Pull request: yes

# What does this PR do?

Fixes #19568 and replaces #19667 

This PR adds Swin2SR, a Swinv2-based model for image super resolution, compression and restoration.

To do:

- [x] finish `Swin2SRImageProcessor` - should incorporate padding
- [x] fix integration test
- [x] transfer checkpoints to the appropriate organization

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Gently pinging @sgugger here
Install `sentencepiece` in `DeepSpeed` CI image
Author: ydshieh
Pull request: yes

# What does this PR do?

Install `sentencepiece` in `DeepSpeed` CI image. 
  - The new base image has no `sentencepiece` pre-installed, but it's required for DeepSpeed CI tests
  - With this PR, the tests all pass (on single GPU runner)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

It's done :-) as suggested
Is AMD supported for transformers and text generation?
Author: oobabooga
Pull request: no
Labels: name: bug

### System Info

Hello,

I would like to ask if it is possible to run models like GPT-J and OPT-6.7b using an AMD GPU like RX 6800 16GB.

Specifically using `AutoModelForCausalLM.from_pretrained`. I have similar models (like OPT-1.3B) working on an NVIDIA GPU with half precision, but I don't know if it would seamlessly work on a different GPU brand.

The operating system is Ubuntu.

Thank you.

### Who can help?

@patil-suraj @patrickvonplaten @Narsil @gante_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Example script

```
model = AutoModelForCausalLM.from_pretrained("/home/me/models/opt-1.3b/", torch_dtype=torch.float16).cuda()
tokenizer = AutoTokenizer.from_pretrained("/home/me/models/opt-1.3b/")

input_text = f"Hello"
input_ids = tokenizer.encode(str(input_text), return_tensors='pt').cuda()

output = model.generate(
    input_ids,
    do_sample=True,
    max_length=200,
    temperature=0.9,
).cuda()

reply = tokenizer.decode(output[0], skip_special_tokens=True)
```

### Expected behavior

Working on an AMD GPU (RX 6800 16GB) seamlessly

Comments:
Hey @oobabooga, are you getting an error when running this code currently on AMD? Could you maybe just try it out and add a stack trace here if it doesn't work? :-) 

Thanks!

Hello @patrickvonplaten, I am not getting an error, I would just like to know if transformers also works on AMD or if it is exclusive to NVIDIA. I have not found this information anywhere.

Hey @oobabooga, we're not exclusive to either NVIDIA or AMD, but we do rely on several backends to run transformer models: either PyTorch, TensorFlow, or JAX.

If you setup either of those 3 to run with AMD GPUs, it should run with transformer models without issue.

Thank you for the clarification, @LysandreJik. I am closing the issue as that answers the question.

> If you setup either of those 3 to run with AMD GPUs, it should run with transformer models without issue.

Can you please link some resources on how to do that?
[wip: testing doc-builder]
Author: mishig25
Pull request: yes


Comments:
_The documentation is not available anymore as the PR was closed or merged._

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Problems with layoutlm language model
Author: mv96
Pull request: no

### Feature request

Hi,

I was recently reading Layoutlm and it's variants and I figured out that they do exist in hugging face but there are several problems with It or I am not very sure to understand it just by reading the documentation.

Q1) [Layoutlm](https://arxiv.org/abs/1912.13318) is a multimodal machine learning transformer then why is it listed in the text transformers category on HuggingFace ?

Q2) Even if it is multimodal, layoutlm does not tend to use the image anywhere in the examples of layoutlm [Layoutlm](https://huggingface.co/docs/transformers/main/en/model_doc/layoutlm) ? 

Q3) For [Layoutlmv2](https://huggingface.co/docs/transformers/main/en/model_doc/layoutlmv2) there is no tensorflow class like TFlayoutlm as it is available for version1 ?

Q4) there is no MLM head class as it available for it's version 1 ? so I am not sure if I want to pretrain this model from scratch how do I do that ?

Q5) Same as Q4 there is not MLM head class, so in this case if I have my own tokenizer and I want to pretrain from scratch layoutlm and then simply want to change the transformer with one line of code change, that's not possible because they have different heads ?

I am a bit new to HF interface so forgive me if I asked something super basic. 
I don't know if I have something wrong with understanding layoutlm in the first place or these are valid questions, 
But I would be very happy if anyone can shed some light on this !!
Once again thanks for taking the time to read 😊😊 , have a good day !!



### Motivation

Just find it very difficult to understand the implementation of the specific model from the transformers library

### Your contribution

I can try to look into it, but first I need to know if the problem is really a problem or it is just my wrong understanding of the library.

Comments:
Please use the [forums](https://discuss.huggingface.co/) to ask such questions as we keep issues for bugs and feature requests only :-)
cc @NielsRogge 

Please link your question on the forum, I'll answer there!

@NielsRogge can you send me the link to access forums, I tried to look at the discord channel of hugging face but I am not exactly sure where to ask questions.

Thanks

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Support fairseq encoder-normalize-before in RoBERTa
Author: AndreasMadsen
Pull request: no

### Feature request

Support Fairseq's `--encoder-normalize-before` variation of transformer-based models, (particularly RoBERTa, not sure how many it applies to) that "apply layernorm before each encoder block". See: https://fairseq.readthedocs.io/en/v0.7.0/models.html

### Motivation

There currently exist unofficial hacks of Huggingface's transformer model at https://github.com/princeton-nlp/dinkytrain which uses `--encoder-normalize-before` with fairseq and then applies a series of hacks to make it work with Huggingface's transformer library. See in particular [their custom version of RoBERTa](https://github.com/princeton-nlp/DinkyTrain/blob/main/huggingface/modeling_roberta_prelayernorm.py) which is a hack of Huggingface's RoBERTa implementation and depends on several internal components.

The weights for these hacked models are currently distributed on https://huggingface.co/princeton-nlp but are not actually compatible with the official transformer library. (https://arxiv.org/abs/2202.08005 is the related paper).

It would be great to implement feature parity for `--encoder-normalize-before` in Huggingface's transformer such that these hacks can be prevented.

Note that I have not affiliation with fairseq, the https://github.com/princeton-nlp/DinkyTrain authors, nor the https://arxiv.org/abs/2202.08005 authors.

### Your contribution

I'm happy to contribute a PR for RoBERTa that adds feature parity for the `--encoder-normalize-before` fairseq flag.

Note that I have no affiliation with fairseq nor the https://github.com/princeton-nlp/DinkyTrain authors.

Comments:
Transformers is very opinionated in that regard and is not a building block library like fairseq. The only way to add support for those models in Transformers would be to add a new modeling file which adapts the code of RoBERTa to only include the code path corresponding to `--encoder-normalize-before`, as putting both in the same modeling file hurts readability.

You can learn more about our philosophy in this regard in [this blog post](https://huggingface.co/blog/transformers-design-philosophy) and if you're interested in making a PR to add this new model, we're looking forward to it!


Thanks, I will look at adding the new model. Do you have a policy regarding its name? In this case, the authors did not give it a dedicated name as they see it as just RoBERTa.

Please see the implementation provided in https://github.com/huggingface/transformers/pull/20305

That's pretty cool! We had this kind of "problem" when adding support for XLM-R XL models:

https://github.com/huggingface/transformers/pull/12082#issue-665786049

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Vilt - use image_transforms pad
Author: amyeroberts
Pull request: yes

# What does this PR do?

When Vilt was first implemented, image transforms library didn't have `pad` implemented. This PR removes the old pad implementation in `image_processing_vilt.py` and uses the standard library. 

It also adds some missing `# Copied from ` statements that apply to other module level functionality in `image_processing_detr.py` spotted when comparing `pad` between the models. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._