noamaanMulla-03/transformers-issues huggingface dataset:
Regression in CLIPProcessor from 4.24.0 -> 4.25.0.dev0
Author: patrickvonplaten
Pull request: no

### System Info

- `transformers` version: 4.24.0 / 4.25.0.dev0
- Platform: Linux-5.18.10-76051810-generic-x86_64-with-glibc2.34
- Python version: 3.9.7
- Huggingface_hub version: 0.11.0.dev0
- PyTorch version (GPU?): 1.11.0+cpu (False)
- Tensorflow version (GPU?): 2.9.1 (False)
- Flax version (CPU?/GPU?/TPU?): 0.6.0 (cpu)
- Jax version: 0.3.16
- JaxLib version: 0.3.15
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@amyeroberts @sg

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

There seems to be a regression of `CLIPProcessor` between current `main` and `4.24` 

You can easily reproduce it by running the following script with current main `4.25.0.dev0` and `4.24` to see a difference:

```python
#!/usr/bin/env python3
from transformers import CLIPProcessor
import transformers
from PIL import Image
import PIL.Image
import numpy as np
import torchvision.transforms as tvtrans
import requests
from io import BytesIO

print(transformers.__version__)

url = "https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg"
response = requests.get(url)
image = Image.open(BytesIO(response.content)).convert("RGB")

BICUBIC = PIL.Image.Resampling.BICUBIC
image = image.resize([512, 512], resample=BICUBIC)
image = tvtrans.ToTensor()(image)

np_image = np.asarray(image)
processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")

pixel_values = processor(images=2 * [np_image], return_tensors="pt").pixel_values

print(pixel_values.abs().sum())
print(pixel_values.abs().mean())
```

The outputs for the different versions are as follows:
```
4.24.0
tensor(287002.5000)
tensor(0.9533)
```
```
4.25.0.dev0
tensor(503418.8125)
tensor(1.6722)
```

The code snippet above comes from reproducing a problem that happens when updating `transformers` to main for https://github.com/SHI-Labs/Versatile-Diffusion .
https://github.com/SHI-Labs/Versatile-Diffusion only works with `transformers==4.24.0` - the pipeline gives random results when using `transformers==4.25.0.dev0` 

### Expected behavior

It seems like a bug was introduced for after the 4.24. release. The code snippet above might seem a bit edge-casy but I believe people have started to build any kind of image processing pipelines with CLIP already.

Comments:
[OWL VIT] make daily CI happy
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes a slow test for OWLViT while trying to integrate `accelerate` support for this model! 

cc @ydshieh @alaradirik 

Slow test that was failing: `tests/models/owlvit/test_modeling_owlvit.py::OwlViTModelIntegrationTest::test_inference_one_shot_object_detection`

all slow tests pass!


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks @ydshieh , just updated the description! 
Yes without this change, the test would fail on GPU (`accelerate` tests excluded)
[BNB] fix nasty `bnb` bug
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR fixes a very nasty bug that can be reproduced with the following script: 
```
from transformers import AutoModel

model_16bit = AutoModel.from_pretrained("bigscience/bloom-560m", device_map="auto", load_in_8bit=False)
model_8bit = AutoModel.from_pretrained("bigscience/bloom-560m", device_map="auto", load_in_8bit=True)

print(model_16bit.is_loaded_in_8bit)
>>> True
```
In fact we should assign the attribute `is_loaded_in_8bit` to the variable `model` instead of `cls`. Otherwise `is_loaded_in_8bit` will be overriden by the next model that is loaded in 8bit

@sgugger @ydshieh 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Cached model files can't be referred in docker container on AWS Lambda
Author: chenye-814
Pull request: no

### System Info


{
  "errorMessage": "Can't load tokenizer for 'model_name'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'model_name' is the correct path to a directory containing all relevant files for a BlenderbotTokenizer tokenizer.",
  "errorType": "OSError",
  "requestId": "",
  "stackTrace": [
    "  File \"/var/lang/lib/python3.9/importlib/__init__.py\", line 127, in import_module\n    return _bootstrap._gcd_import(name[level:], package, level)\n",
    "  File \"<frozen importlib._bootstrap>\", line 1030, in _gcd_import\n",
    "  File \"<frozen importlib._bootstrap>\", line 1007, in _find_and_load\n",
    "  File \"<frozen importlib._bootstrap>\", line 986, in _find_and_load_unlocked\n",
    "  File \"<frozen importlib._bootstrap>\", line 680, in _load_unlocked\n",
    "  File \"<frozen importlib._bootstrap_external>\", line 850, in exec_module\n",
    "  File \"<frozen importlib._bootstrap>\", line 228, in _call_with_frames_removed\n",
    "  File \"/var/task/app.py\", line 22, in <module>\n    tokenizer = BlenderbotTokenizer.from_pretrained(MODEL_NM,local_files_only=True)\n",
    "  File \"/var/task/transformers/tokenization_utils_base.py\", line 1761, in from_pretrained\n    raise EnvironmentError(\n"
  ]
}



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am pretty sure cached model files are included in my image and TRANSFORMERS_CACHE is also set properly when building the image.
I can see that my model files exist in container through log output:
```
models--xxxxxxx : ['blobs', 'refs', 'snapshots']
TRANSFORMERS_CACHE = ./parent_dir_for_model/
```

Code is simply like this, and it works well on my local python 3.10 environment (can automatically go for the path defined in `TRANSFORMERS_CACHE` without trying to download) .

```
tokenizer = BlenderbotTokenizer.from_pretrained(path,local_files_only=True)
model = BlenderbotForConditionalGeneration.from_pretrained(path,local_files_only=True)
```

My version of transformers is 4.24.
I suspect something wrong about the compatibility with Lambda python3.9 runtime?

### Expected behavior

Cached model files should be loaded properly based on the path defined by `TRANSFORMERS_CACHE` .

Comments:
[BNB] Throw `ValueError` when trying to cast or assign
Author: younesbelkada
Pull request: yes

# What does this PR do?

I have seen several piece of code where users try to assign `8-bit` loaded models into a new device and/or `dtype` this PR aims to throw an error to the users that perform this.

Also added a set of slow tests

cc @sgugger @ydshieh 

Do not merge before #20408


Comments:
_The documentation is not available anymore as the PR was closed or merged._

What happens if we don't have the changes in this PR and users try to do assign 8-bit loaded models into a new device and/or dtype?

Users can face unexpected behaviors such as: https://github.com/huggingface/transformers/issues/20361#issuecomment-1324113579
some tokenizer(s) don't save the updated attributes
Author: ydshieh
Pull request: no

### System Info

transformers version: 4.25.0.dev0
Torch version: 1.13.0+cpu
Cuda available: False
Cuda version: None
CuDNN version: None
Number of GPUs available: 0

### Description

For `GPT2Tokenizer(Fast)`, Set `tokenizer.model_max_length` to `128` (originally `1024`), save it then reload, will give `tokenizer.model_max_length` being `1024`.

### Reproduction

```python
from transformers import GPT2Tokenizer, GPT2TokenizerFast

tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")
print(tokenizer.model_max_length)

tokenizer.model_max_length = 128
print(tokenizer.model_max_length)

tokenizer.save_pretrained("my-gpt2")
tokenizer_loaded = GPT2TokenizerFast.from_pretrained("my-gpt2")
print(tokenizer_loaded.model_max_length)
```

The output is

```bash
1024
128
1024

```


### Expected behavior

`tokenizer_loaded.model_max_length` should be `128` in the above example. In general, the updated attribute(s) should be saved.

Comments:
It turns out that we use
https://github.com/huggingface/transformers/blob/0ee71188ff184ee5f8b70081665858301fe4afb1/src/transformers/tokenization_utils_fast.py#L735
which is defined
https://github.com/huggingface/transformers/blob/0ee71188ff184ee5f8b70081665858301fe4afb1/src/transformers/tokenization_utils_base.py#L1475

Is this the expected behavior, i.e. we don't want to save the modified attributes like `model_max_length`?

Hi @ydshieh, 

This is a very good remark! I've also often wondered... what I'm afraid of is that for some attributes the tokenizer's behavior is different between:
1. A tokenizer initialized with some parameters and then with a parameter that is modified on the fly
2. A tokenizer that would be initialized with the final parameters of the previous tokenizer

What is complicated with tokenizers is that all tokenizers share the lines of code you mention but many of them have specificities implemented on top of them and it's quite hard to be sure that we won't break more things than we fix (and this if we consider that we don't add a breaking change...). I think there must be a reason why historically it was chosen to only save the `init_kwargs` for the `tokenizer_config`:

https://github.com/huggingface/transformers/blob/0ee71188ff184ee5f8b70081665858301fe4afb1/src/transformers/tokenization_utils_base.py#L2084
Use updated attributes when saving tokenizers
Author: ydshieh
Pull request: yes

# What does this PR do?

Use updated attributes when saving tokenizers.

Fix #20395

Comments:
_The documentation is not available anymore as the PR was closed or merged._

For the record: running slow tests with the generic fix:

2 are (By-)T5 issues, other 3 from Bert/RocBert/NLLB

```bash
=FAILED tests/models/bert/test_tokenization_bert_tf.py::BertTokenizationTest::test_saved_model - ValueError: The two structures don't have the same nested structure.
FAILED tests/models/byt5/test_tokenization_byt5.py::ByT5TokenizationTest::test_added_token_serializable - ValueError: Both extra_ids (125) and additional_special_tokens (['new_token']) are provided to ByT5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokens
FAILED tests/models/nllb/test_tokenization_nllb.py::NllbTokenizationTest::test_save_pretrained - ValueError: Non-consecutive added token 'ar_AR' found. Should have index 1229 but has index 1204 in saved vocabulary.
FAILED tests/models/roc_bert/test_tokenization_roc_bert.py::BertTokenizationTest::test_sequence_builders - assert [1, 5, 6, 2] == [101, 5, 6, 102]
FAILED tests/models/t5/test_tokenization_t5.py::T5TokenizationTest::test_added_token_serializable - ValueError: Both extra_ids (100) and additional_special_tokens (['new_token']) are provided to T5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokens

```

Will try to make tokenization code better, but need to merge in order to unblock tiny model creation (so for pipeline testing and ONNX testing)
Fix ModelOutput instantiation when there is only one tuple
Author: sgugger
Pull request: yes

# What does this PR do?

This PR fixes a bug discovered by @NielsRogge [here](https://github.com/huggingface/transformers/pull/20407#discussion_r1030465759).

To behave like a dict, `ModelOutput` need to accept a single iterator of key/value pairs as its first argument (otherwise many properties of dictionaries instantiation are left) which caused an issue here since Niels is creating one with a single tuple (but not of key/value pairs). To fix this, added some stronger checks, and new tests.

Comments:
Add Spanish translation of pr_checks.mdx
Author: donelianc
Pull request: yes

# What does this PR do?

Add the Spanish translation for `pr_checks.mdx` as part of the #15947 issue.
Changes include the Spanish version of the original document and the updated `_toctree.yml` file.


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests) Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? _Task assignment [here](https://github.com/huggingface/transformers/issues/15947#issuecomment-1321245149)_.
- [x] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @osanseviero. Can you help review this PR, please? 

Only minor typos were spotted during the first review. I already addressed them with my latest commit. 
@sgugger, if you agree, we can skip a second review from [osanseviero](https://github.com/osanseviero) and merge this PR. 

Thanks!
Fix device in longformer onnx path
Author: fxmarty
Pull request: yes

# What does this PR do?

Longformer has a custom path in its `_chunk()` method, in order to be tracable (to some extent) + exportable to ONNX. https://github.com/huggingface/transformers/pull/20292 fixed a bug where this special path was always registering a non-general case:

https://github.com/huggingface/transformers/blob/9ef46659da45f6b605873ca59124d03976990b33/src/transformers/models/longformer/modeling_longformer.py#L785-L787

It seems that the `else` path that should be taken in the export was actually never tested, and notably never tested on GPU. This PR fixes a device assignment.

The test `RUN_SLOW=1 python -m pytest -v tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_102_longformer_token_classification` is now running fine.

## Who can review?

@lewtun @ydshieh 

Comments:
gently pinging @sgugger for final approval :)

_The documentation is not available anymore as the PR was closed or merged._
Raised Exceptions under conditions that are contrary to specified conditions that assert statements
Author: JuheonChu
Pull request: yes

Co-author: @Batese2001
Test file: src/transformers/models/distilbert/modeling_distilbert.py
Local testing: ![testDistilbert](https://user-images.githubusercontent.com/35699839/203280712-a8a9b823-0f4f-4d69-8913-d444d1cd9988.png)


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for your PR. As you can see, it shows a diff of 191 files. Could you open a fresh PR with just your proposed changes?

Yes, I will do that. Thanks for the feedback! 

I will make a new PR after improving the validity checks on ci/circleci: check_code_quality.
Integrate Timm models as vision encoders in Vision encoder decoder models 
Author: gagan3012
Pull request: no

### Feature request

Hello, 
I would like to use timm models as vision encoders in vision encoder-decoder model who can I do so?

### Motivation

Timm models are very powerful and allow a lot of image processing which would allow us to make better document AI models

### Your contribution

I would be down to help out with PR

Comments:
@NielsRogge 

Hi,

I don't think that makes sense as for the Vision encoder-decoder framework, one needs a Transformer-based encoder or at least an encoder that outputs a sequence of hidden states, which can be used for cross-attention with the Transformer-based language decoder. Given that most backbones in timm are convolution-based, which output 3D feature maps, one would first need to flatten/project them into a sequence of hidden states.

The VisionEncoderDecoderModel [framework](https://huggingface.co/docs/transformers/model_doc/vision-encoder-decoder) currently supports ViT, DeiT, BEiT and Swin Transformer, which is already a fair amount of models.

I was more looking to use Swin with non-square image sizes and the Timm implementation allows us to do that and I would like to implement SwinV2 with bert

>  I was more looking to use Swin with non-square image sizes

Our implementation also allows that, did you try it?

I havent tried it can you share an example?


```
from transformers import AutoModelForImageClassification
import torch

model = AutoModelForImageClassification.from_pretrained("microsoft/swinv2-tiny-patch4-window8-256")

pixel_values = torch.randn(1, 3, 244, 522)

outputs = model(pixel_values)
```
Last hidden states different for same input even when evaluating ViT model
Author: robbohua
Pull request: no

### System Info

Google Colab

### Who can help?

@NielsRogge @sg

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I've created a public Colab notebook: https://colab.research.google.com/drive/1CUpyNInQg2kw7gL-mYwX8ov8fc9HuAV-#scrollTo=DAXcuQXVtTkz

where a VitMAE model is created, and its weights saved using Trainer.

Then it is loaded again, and placed in eval mode. However the last hidden state changes when it is passed the same input - why might that be happening?

### Expected behavior

A model in eval mode should have a deterministic output for the same input

Comments:
Hi,

Yes ViTMAE generates a random boolean mask to mask patches internally. This results in non-deterministic hidden states when you perform a forward pass twice on the same image.

To get deterministic behaviour, you can pass a noise tensor yourself as shown [here](https://github.com/huggingface/transformers/blob/afce73bd9d891b55dcb8d4d875d17718ffa01ff0/tests/models/vit_mae/test_modeling_vit_mae.py#L321). 

Thanks!

Btw another way (in case you don't want to mask patches) is to set `config.mask_ratio=0.0`
add FAN model (vision)
Author: johnnv1
Pull request: no
Labels: name: New model

### Model description

Fully attentional networks (FAN) is a family of general-purpose Vision Transformer backbones that are highly robust to unseen natural corruptions in various visual recognition tasks.

### Open source status

- [X] The model implementation is available
- [ ] The model weights are available

### Provide useful links for the implementation

* https://github.com/NVlabs/FAN
* https://arxiv.org/abs/2204.12451

Comments:
Hi there, I would like to implement this model @NielsRogge 

Hi @NielsRogge.
To my knowledge there would still be two pending tasks
- [X] Update README.md to include FAN model
- [ ] Migrate Files, weights to NVIDA organization space
Please let me know what additional tasks you think might be pending
[WIP] Add Efficientformer
Author: novice03
Pull request: yes

# What does this PR do?

This PR adds Efficientformer, a model that has similar latency as MobileNets, but achieves better accuracy on ImageNet.

Paper: https://arxiv.org/abs/2206.01191
Code and weights: https://github.com/snap-research/EfficientFormer

To-do:

- [ ] Improve documentation
- [ ] Verify tests pass

Fixes #18041

## Who can review?

@NielsRogge 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @novice03 @NielsRogge @alaradirik :) - what is the status of adding this model? @novice03 if you do not what to continue working on it I can pick it up from here.

Hi @Bearnardd, I'm afraid I don't have time in the next few weeks to continue working on this model. But, please pick it up from here if you can. Thanks!

Hi @alaradirik - I have just started working on this PR locally and I have applied most of the style changes that you have requested and in the following days I will apply the rest of the changes that you have asked for. Is there any possibility to catch you on Slack or on the other medium since I am pretty confident I will have some questions regarding this task. Thanks!

Hi @Bearnardd, of course, could you give me your email address? I can create a Slack channel and invite you.

If possible, it'd be great if @novice03 could add you as a collaborator to his transformers repo and you work on the same repo/branch so that he gets credit for the PR too.

Hi @alaradirik I have sent you my email address on LinkedIn also great idea about the collaboration. @novice03 if it suits you well you can send me an invitation :)

Hi @novice03 ! - I have done some work on top of your changes. Not everything is ready but I think that I am in good spot to push the changes and get some review before going further. Would you mind adding me as a collaborator to your repo so I can push there directly or do you want me to open a new PR on top of yours?

Hello @Bearnardd, I added you as a collaborator on my repo. Thanks!
[wip test doc build]
Author: mishig25
Pull request: yes


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19867). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19867). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.