noamaanMulla-03/transformers-issues huggingface dataset:
[run_clm example] add torch_dtype option for model load.
Author: sywangyi
Pull request: yes

for BLOOM 175B model. peak memory will reduce about 350G for inference. the weight of BLOOM in model hub is bfloat16

Signed-off-by: Wang, Yi A <yi.a.wang@intel.com>

# What does this PR do?

reduce the peak memory for BLOOM inference.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

- trainer: @sgugger


Comments:
@yao-matrix @jiqing-feng @sgugger please help to review

_The documentation is not available anymore as the PR was closed or merged._

@sgugger done,  add other torch dtypes per review comment
Set Transformer
Author: leondz
Pull request: no
Labels: name: New model

### Model description

This issue proposes addition of the Set Transformer, a set2seq transformer for learning to order sets of items. 

## Short description of the model and link to the paper

The transformer is one of a family of models that implements permutation invariance in order to learn ordering relations. This particular implementation uses stacked attention blocks to achieve the invariance. Set transformers are good for a multitude of problems - the toy problem is the TSP, where vertices are ordered optimally, though the framing can also be applied to any sequence generation tasks where the sequence items are known ahead of time. See [this review](https://jair.org/index.php/jair/article/view/12839) for a description of the family of problems.

This particular transformer is the Set Transformer, presented in [Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks](http://proceedings.mlr.press/v97/lee19d.html).

This isn't immediately designed for text or images or speech, but is a distinct transformer architecture that has been applied to text and image data.

## Link to the implementation if it is open-source

There's an official PyTorch implementation at [https://github.com/juho-lee/set_transformer](https://github.com/juho-lee/set_transformer)

We've already got this up & running as a baseline in an upcoming IJCAI paper

## Link to the model weights if they are available.

not immediately available, but we could work something out

### Open source status

- [x] The model implementation is available
- [ ] The model weights are available

### Provide useful links for the implementation

Me & a colleague can get this up onto HF, we have a running implementation and the reference implementation is both on github and licensed MIT. Reference implementation by @juho-lee (author) and @yoonholee.

Comments:
@leondz can I work on this issue?
Unable to load weights from pytorch checkpoint file
Author: manandey
Pull request: no

### System Info

I trained a model and uploaded the checkpoint in the Hub [here](https://huggingface.co/taskydata/deberta-v3-base_10xp3nirstbbflanseuni_10xc4). When I try to load the model, I get the following error message:
`OSError: Unable to load weights from pytorch checkpoint file for '/root/.cache/huggingface/hub/models--taskydata--deberta-v3-base_10xp3nirstbbflanseuni_10xc4/snapshots/f5d6b49731ea0b36601f151dd67623380462a3cb/pytorch_model.bin' at '/root/.cache/huggingface/hub/models--taskydata--deberta-v3-base_10xp3nirstbbflanseuni_10xc4/snapshots/f5d6b49731ea0b36601f151dd67623380462a3cb/pytorch_model.bin'. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.`

Also, when I try to run it using the inference API, I get:
`Could not load model taskydata/deberta-v3-base_10xp3nirstbbflanseuni_10xc4 with any of the following classes: (<class 'transformers.models.deberta_v2.modeling_deberta_v2.DebertaV2ForSequenceClassification'>, <class 'transformers.models.deberta_v2.modeling_tf_deberta_v2.TFDebertaV2ForSequenceClassification'>).`

Transformer version: `4.25.1`

Any help on how to resolve this would be greatly appreciated. Thanks!

cc. @LysandreJik @younesbelkada @ArthurZucker 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Mentioned above.

### Expected behavior

The model should be loading since all the files are uploaded.

Comments:
Hi @manandey,
Here is the error I get: 
```
RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory
```
Your saved model is probably corrupted, could you tell us how did you saved the model, or alternatively give it another try?

Hi @younesbelkada, when I try to load the model by connecting my colab to the GCP instance where the checkpoints are saved, the model seems to be loading perfectly fine as can be seen in the attached snapshot. But when I try to download the checkpoint and upload it to hub, the downloaded files seem to become corrupted. 

To download the checkpoint, I am using something like this: `!tar -zcvf checkpoints.tar.gz checkpoint/checkpoint-300000`

![image](https://user-images.githubusercontent.com/6687858/212846041-0a7223f6-063f-449e-900c-ce60a5a6f96d.png)


If I understood correctly you are manually uploading the weights on the Hub?
Can you maybe try: 
```
model.push_to_hub("taskydata/deberta-v3-base_10xp3nirstbbflanseuni_10xc4")
```
after the lines that you have attached above. 
Make sure to login from your notebook with:
```
from huggingface_hub import notebook_login
notebook_login()
```

Thanks a lot, @younesbelkada! It worked! 

Very happy that it worked! Thanks for guiding us precisely through your issue
MT5ForConditionalGeneration: forward() got an unexpected keyword argument 'penalty_alpha'
Author: Mahyar-Ali
Pull request: no

When I use the MT5ForConditionalGeneration class to load an mt5 model and then specify the `penalty_alpha` parameter in the generate function, the library raises the error `forward() got an unexpected keyword argument 'penalty_alpha'`. But when I load the same model with `AutoModelForSeq2SeqLM` class, it doesn't raise that issue.

This shouldn't be happening because the `AutoModel` class automatically selects the relevant `MT5ForConditionalGeneration` for all T5 models. So, why does this raises an issue when I use the `MT5ForConditionalGeneration` directly?

Also, this is particularly interesting because when you fine-tune a t5 model using the `MT5ForConditionalGeneration` class but load that model(after training) using the `AutoModelForSeq2SeqLM` and then use the `penalty_alpha`, it still raises the same issue.

Code:

```
model = MT5ForConditionalGeneration.from_pretrained(f"{model_dir}")
tokenizer = MT5Tokenizer.from_pretrained(f"{model_dir}")



input_ids = tokenizer.encode(
            source_text, return_tensors="pt", add_special_tokens=True
        )
input_ids = input_ids.to(device)
generated_ids = model.generate(
            input_ids=input_ids,
            penalty_alpha=0.6, top_k=4
        )
preds = [
    tokenizer.decode(
        g,
        skip_special_tokens=skip_special_tokens,
        clean_up_tokenization_spaces=clean_up_tokenization_spaces,
    )
    for g in generated_ids
]
```

Transformers version: 4.16.2
Python version: 3.9.15

Comments:
cc @gante 

Hi, @Mahyar-Ali 
I have transformers - 4.25.1 and used the code you gave (above) and it ran without any error, then I switched back to transformers - 4.16.2(same as your version) and it gave the error regarding "penalty alpha", I believe it is fixed in later versions of transformers, since I got the output in version 4.25.1 .

Maybe you can try to upgrade to latest version and check if it works for you or not.

It's working now. Didn't pay attention to the version (as it was working well for `AutoModel`). Thanks!

@susnato thank you for pitching in ;)
Loading wav2vec2 pre-trained models
Author: jvel07
Pull request: no

Hi, my aim is to get embeddings from a pre-trained wav2vec2 model using my own data (over 9k samples for each: train, dev, and test; the wavs are from 1 to 4 secs of duration) 
I got two main inquiries:
1. I am trying to load a pre-trained model in the following way and I get "CUDA out of memory" error:
```device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_name = "facebook/wav2vec2-large-xlsr-53-german"
feature_extractor = Wav2Vec2Processor.from_pretrained(model_name)
model = Wav2Vec2Model.from_pretrained(model_name)
model.to(device)
```
Am I missing something here? The only way I could load the model is when I don't do `.to(device)` so it loads in the RAM, but in this way, inferencing is way too slow.

2. After loading the model without using GPU, I could get the embeddings in this way:
```
input_values = feature_extractor(train_dataset["speech"], return_tensors="pt", padding=True, 
                                 feature_size=1, sampling_rate=16000 )
outs = model(**input_values)
last_hiddens = outs.last_hidden_state   # embeddings here correspond to the sequence of last_hidden_states
last_cnn_embs = outs.extract_features  # sequence of features of the last conv layer of the model
```
Saving the embeddings to a binary file resulted in over 20 GB for the `last_hiddens` and over 10 GB for the `last_cnn_embs`.
Is that a normal thing?

Comments:
Hey @jvel07 - you should be able to load the model to GPU. The model is "only" 1GB in size so it should be easily useable on most GPUs. Are you sure there aren't any other processes occupying the GPU when you try loading the model? 

Regarding the binary files - I can't really say anything here as I've never saved the embedding outputs to binaries.

Can you please check the forum: https://discuss.huggingface.co/ for more specific Wav2Vec2 questions? :-) Also these blog posts might be useful:

- https://huggingface.co/blog/fine-tune-wav2vec2-english
- https://huggingface.co/blog/fine-tune-xlsr-wav2vec2
Fixes to TF collators
Author: Rocketknight1
Pull request: yes

This PR makes a couple of fixes to TF data collators:

1) Fixes an incorrect call to `is_integer` when the default data collator receives `tf.Tensor` input and is outputting `tf.Tensor` as well
2) Prefer "np" tensors rather than "tf" tensors when calling our collators via `to_tf_dataset`. This is because data preprocessing is generally done with `np.ndarray` rather than `tf.Tensor` anyway, and Keras/`tf.data` can do the final conversion to `tf.Tensor` for us.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Rename test_feature_extraction files
Author: amyeroberts
Pull request: yes

# What does this PR do?

Renames the tests for image processors: `test_feature_extraction_xxx.py` -> `test_image_processing_xxx.py`

A follow up PR will change the feature extractor references to equivalent image processor ones in the files. 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you @amyeroberts !

I think there are a few lines in `tests/utils/test_add_new_model_like.py` to be changed, like

https://github.com/amyeroberts/transformers/blob/6b10c045e259a47f3786ceb11089fe418828346e/tests/utils/test_add_new_model_like.py#L62


@ydshieh Done!
feat: add standalone guide on XLA support.
Author: sayakpaul
Pull request: yes

# What does this PR do?

We have had XLA support for our TF generation models (GPT2, Whisper, for example) for a while. This PR adds a standalone guide in the doc to discuss it. 

Cc: @Rocketknight1 @amyeroberts @ydshieh 

Comments:
@sayakpaul 

It seems there is an issue with your CircleCI permissions.
Could you try refreshing your permissions as shown [here](https://support.circleci.com/hc/en-us/articles/360048210711-How-to-Refresh-User-Permissions-)?

@ydshieh I don't have CircleCI signed in nor have I installed it here: https://github.com/settings/applications. 

Could you login CircleCI with your GitHub account, and follow `huggingface/transformers`?

See https://circleci.com/docs/projects/

(but it's kinda strange - you have opened a lot of PRs before, so not sure why we have this issue now)

Should be all good now I guess:

<img width="896" alt="image" src="https://user-images.githubusercontent.com/22957388/212717447-19e1b042-d39f-44f0-bf8c-faef7b97c2ea.png">


Yeah, then we can try to push an empty commit to see if the CI will run :-)
```
git commit --allow-empty -m "Empty commit to trigger CI"
git push
```


_The documentation is not available anymore as the PR was closed or merged._

@sgugger addressed your comments including [this one](https://github.com/huggingface/transformers/pull/21141/files#r1072172489). Let me know if I am good to merge given all tests are green. 

Yes, all good!
Small simplification to TopKLogitsWarper
Author: njhill
Pull request: yes

# What does this PR do?

The max of `top_k` and `min_tokens_to_keep` performed on every call can just be done once up-front.

Apologies if there's some reason for it being this way that I overlooked!

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] (N/A) Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] (N/A) Did you write any new necessary tests?


## Who can review?

@sgugger @patrickvonplaten 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @gante
Add batch of resources
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds a batch of resources, primilarly for all image classifiers.

Comments:
Thanks for your review! It's unclear to me why the "build PR documentation" check is failing, thought it had to do with the pipeline tags, but it's still failing. Any insight would be greatly appreciated

You will have to isolate which file triggers the issue and then which line inside that file by trial and error I'm afraid. That's one of the reason smaller PRs are easier to deal with :-)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

_The documentation is not available anymore as the PR was closed or merged._
Change variable name to prevent shadowing
Author: sayakpaul
Pull request: yes

This PR replaces the `input` variable with `input_string` to prevent shadowing caused by the `input()` function. 

Thanks to @LysandreJik for catching it. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Since the changes are small, I guess it's good to merge but still confirming. 

Yes, one core maintainer approval is enough to merge.
CLI: update hub PR URL
Author: gante
Pull request: yes

# What does this PR do?

Keeps up with hub API changes, and updates the instruction to get the PR URL (which was returning an object)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

It is already recent enough: the change was added in `v0.10.0` (see `1.` in `v0.10.0` breaking changes), which is our current minimum version
Add tensorflow example for the image classification task guide
Author: MKhalusova
Pull request: no

For the same dataset and steps as in https://huggingface.co/docs/transformers/tasks/image_classification, 
add sample code for the TensorFlow part. 

This example can supplement the existing guide and can be helpful to those who choose TensorFlow over PyTorch and would like to use Transformers for image classification. 
Related PR: https://github.com/huggingface/transformers/pull/21038

Comments:
VideoMAE with `num_channels!=3` needs a small fix
Author: layjain
Pull request: no

## Issue

The VideoMAE model doesn't work for non-RGB videos (when `num_channels!=3`). I believe this is caused by the harcoded image-net means and stds in the following lines:

https://github.com/huggingface/transformers/blob/d51e7c7e8265d69db506828dce77eb4ef9b72157/src/transformers/models/videomae/modeling_videomae.py#L824L826


## Code to Reproduce

```
import torch
from transformers import VideoMAEConfig, VideoMAEForPreTraining

NUM_CHANNELS = 1

config = VideoMAEConfig(num_channels=NUM_CHANNELS)
model = VideoMAEForPreTraining(config)

pixel_values = torch.rand(1, 16, NUM_CHANNELS, 224, 224)

num_patches_per_frame = (model.config.image_size // model.config.patch_size) ** 2
seq_length = (model.config.num_frames // model.config.tubelet_size) * num_patches_per_frame
bool_masked_pos = torch.randint(0, 2, (1, seq_length)).bool()

outputs = model(pixel_values, bool_masked_pos=bool_masked_pos)
loss = outputs.loss
```
This produces the following error: ```File ".../lib/python3.10/site-packages/torch/functional.py", line 74, in broadcast_tensors
    return _VF.broadcast_tensors(tensors)  # type: ignore[attr-defined]
RuntimeError: The size of tensor a (512) must match the size of tensor b (1536) at non-singleton dimension 2```

Setting NUM_CHANNELS = 3 works fine.

## Potential Fix

Since we don't have a `_DEFAULT_MEAN/STD` for non-RGB images, we can just replace `frames=pixel_values` and disallow `norm_pix_loss=False` if `num_channels!=3`. With this modification, the code works on my machine. I am willing to fix and submit a PR.

Comments:
Hi,

This is related to #19913. Would be great to open a PR! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fixed num_channels!=3 normalization training
Author: layjain
Pull request: yes

Fixes #20580 and #19913 

## Who can review?

@NielsRogge 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Not exactly, the issue with your CircleCI permissions, the tests won't run.

Could you try refreshing your permissions as shown [here](https://support.circleci.com/hc/en-us/articles/360048210711-How-to-Refresh-User-Permissions-)?

I have been now stuck on this for a while. I refreshed the permissions and re-ran the CircleCI checks, and I get the error:
"Resource class docker for xlarge is not available for your project, or is not a valid resource class. This message will often appear if the pricing plan for this project does not support docker use."




You might need to push an empty commit to re-trigger the tests after refreshing your permissions.

Hi @layjain let me know whether you could pick this up :)

FYI I pushed an empty commit to trigger CI

Hi @NielsRogge , I have fixed the CircleCI permissions, can this be merged.

@layjain The CI is currently running under your profile and not the Hugging Face profile, and as such our tests are mostly not run (there should be 22 checks here). If you rebase your PR on main you will see a new check failing (we added a fix to detect this recently).
Adding doc page for the object detection task 
Author: MKhalusova
Pull request: yes

This is a PR for the https://github.com/huggingface/transformers/issues/20805  issue. 
The guide has content and working code examples for: 
- [x] Introduction  
- [x] Loading CPPE-5 dataset from Hub
- [x] Preprocessing both images and annotations. Images are augmented, annotations are reformatted to be in the format DETR expects
- [x] Training with `Trainer`
- [x] Evaluation
- [x] Inference 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

If I missed someone who has to be invited as a reviewer, please feel free to add them. 

@MKhalusova thanks for doing this! I will take a look tomorrow my time. 

I think you can follow the instructions noted [here](https://github.com/huggingface/transformers/pull/16255#discussion_r830432539) to resolve the quality bug in the CI. Let me know if anything's unclear.  

Did a rebase in an attempt to fix the CI issue. Accidentally added a whole bunch of unrelated commits to the PR. Figuring out how to remove them. 

> Did a rebase in an attempt to fix the CI issue. Accidentally added a whole bunch of unrelated commits to the PR. Figuring out how to remove them.

You might want to revert to the previous commit. [This thread](https://stackoverflow.com/questions/4114095/how-do-i-revert-a-git-repository-to-a-previous-commit) might be helpful in that regard. And then from there:

* Create a separate Python virtual environment. 
* Make sure you're in the virtual environment you just created and then from the `transformers` directory root run `pip install -e .[quality]`.
* Now, once the dependencies have been installed to the new virtual environment, run `make style`. 

This should likely fix it. Since the code quality errors were previously coming from a doc page (c.f. https://app.circleci.com/pipelines/github/huggingface/transformers/54549/workflows/18b53122-1cc3-4c87-ad12-486853427500/jobs/657389), I suspect this to be stemming from the task page we're adding in this PR. 

Let me know if anything is unclear. 

Closing this due to messed up rebase. The new PR is now here https://github.com/huggingface/transformers/pull/20925