noamaanMulla-03/transformers-issues huggingface dataset:
[`pipeline`] Add pool option to image feature extraction pipeline
Author: amyeroberts
Pull request: yes

# What does this PR do?

Adds the flag `pool` which will return the pooled output, rather than the raw hidden states. 

Doesn't work for data2vecvision as the model doesn't [add the pooling layer by default](https://github.com/huggingface/transformers/blob/78ba9f4617370a41c436126bbbb6f8d75924837c/src/transformers/models/data2vec/modeling_data2vec_vision.py#L637). At  the moment, [model kwargs aren't passed to the model constructor](https://github.com/huggingface/transformers/blob/78ba9f4617370a41c436126bbbb6f8d75924837c/src/transformers/pipelines/base.py#L817), so it's not simple to add passing `add_pooling_layer` here. 

I chose to raise an error when getting the outputs of the model. Although this means we fail quite late, it avoids any complex inspection needed on the model.

From the list of models in #28944: 

Models which work with the `pool` option:
- beit
- bit
- convnext
- convnextv2
- deit
- dinov2
- dpt
- efficientnet
- focalnet
- levit
- mobilenet_v1
- mobilenet_v2
- mobilevit
- mobilevitv2
- nat
- regnet
- resnet
- swin
- swinv2
- van
- vit
- vit_hybrid
- vivit
- yolos

Models that don't have a pooling layer: 
- conditional_detr
- deformable_detr
- deta
- detr
- dinat
- efficientformer
- glpn
- imagegpt
- poolformer
- pvt
- segformer
- siglip_vision_model
- swiftformer
- swin2sr
- table
- timesformer
- timm_backbone
- videomae
- vit_msn
- vitdet
- vit_mae

Not working
- data2vec-vision. 
 



## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28985). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker Updated the error message and added tests for exact outputs in [b94d08c](https://github.com/huggingface/transformers/pull/28985/commits/b94d08cb0bebd6f7854422b989b4bd93a9b56e46)
support SDPA Attention in stablelm
Author: eaidova
Pull request: yes

# What does this PR do?
enable SDPA attention in stablelm architecture

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
cc @fxmarty 

> Looks alright but we need to add an integration test iMO :)

added

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29106). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[`Maskformer`] safely get backbone config
Author: amyeroberts
Pull request: yes

# What does this PR do?

Don't fail if `backbone_config` is `None`

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29166). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Plans to add DoRA?
Author: RonanKMcGovern
Pull request: no
Labels: name: Feature request

### Feature request

Improves on LoRA by allowing magnitude fine-tuning.

### Motivation

Improved perplexity.

### Your contribution

Sebastien Bubeck has published demo code. https://github.com/rasbt/dora-from-scratch

Comments:
cc @younesbelkada @pacman100 

Hi @RonanKMcGovern ! 
Thanks for the feature request! There is already an ongoing work from @BenjaminBossan to add DoRA in PEFT: https://github.com/huggingface/peft/pull/1474

Closing as there is a PR underway.

OK thank you @RonanKMcGovern !
[ `gemma`] Adds support for Gemma 💎
Author: ArthurZucker
Pull request: yes
Labels: name: New model

# What does this PR do?
Adds support for Gemma 💎

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29167). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
🚨 Llama: update rope scaling to match static cache changes
Author: gante
Pull request: yes

# What does this PR do?

(see title :))

What's breaking? The shape of the returned sin/cos caches are changed (sin/cos for all positions -> sin/cos for the positions in `position_ids`). Note that this breaking change was also present in the static cache PR, for the main RoPE class (#27931).

Review suggestion:
1. Review changes in Llama
2. Review the rest

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29143). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
FIX [`Gemma`] Fix bad rebase with transformers main
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes the current failing CI

cc @ArthurZucker @amyeroberts 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29170). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Generate: low memory tests are flaky
Author: gante
Pull request: yes

# What does this PR do?

As identified by @molbap -- generate tests with the `low_memory` flag are flaky. The full reason is the same as explained in [this comment](https://github.com/huggingface/transformers/issues/25420#issuecomment-1775317535).

The error likelihood has low (~3%), but still quite disruptive for transformers devs.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29136). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts #29109 seems to have fixed most of the issue (this test does compare batched vs unbatched generation under the hood, which the PR linked above fixes)

The root issue about this and other tests being non-deterministic tests persists, though :) I'm going to close the PR and move the discussion to slack at a future time :)
Cache: standardize cache interface
Author: gante
Pull request: yes

# What does this PR do?

In #27931, where the static cache was introduced, we noticed it had the following hard requirements:
1. The model instance holds the cache, as opposed to being a tensor passed around;
2. Each layer has its own cache, as opposed to a single cache for all layers.

This contrasts with previous implementations (e.g. `DynamicCache`). Given the hard requirements of the static cache, and the resulting benefits, this PR migrates the interface for all cache classes so as to match the static cache. As a result, the modeling code becomes slightly simpler 🤗  

Comments:
Model correctness depends on #28937, rebasing with its contents

Closed in place of #29180 (merge conflicts 🤷 )
Typo in `modeling_clip.ClipVisionTransformer`
Author: AdityaKane2001
Pull request: yes

Fixed a typo in `modeling_clip.py`.

Comments:
@amyeroberts 

Yes, I realized that when I tried to incorporate the change in my fork. Maybe the maintainers might have to do this, but what would be a solution in this case? Apart from the brute-force one, i.e. changing names in _all_ hosted clip weights.



Maybe overloading a try-catch mechanism somewhere such that loading the weights to the misspelled one also reflects in the correct one?

@AdityaKane2001 It's not something that we're going to try to fix. It's a bit annoying but effectively not consequential. The solution is to put up with the original spelling :) 

Gotcha. Closing the PR.
How do I load a pretrained file offline?
Author: zysNLP
Pull request: no

## ❓ Questions & Help

Hi, when I use "RobertaModel.from_pretrained(roberta.large)" to load model. A progress bar appears to download the pre-training model. I've already downloaded files like "roberta-large-pytorch_model.bin ". How can I stop automatically downloading files to the ".cache" folder and instead specify these pre-training files I downloaded?


Comments:
You can do it, instead of loading `from_pretrained(roberta.large)` like this download the respective `config.json` and `<mode_name>.bin` and save it on your folder then just write 
`.from_pretrained('Users/<location>/<your folder name>')` and thats about it.

OK, Thank you very much!

@shashankMadan-designEsthetics' solution may require git-lfs to download the files of some models. If you are not a sudoer, this can be a problem. The most reliable and easy solution I've found is this:
```
from transformers import AutoModel, AutoTokenizer

# Do this on a machine with internet access
model = AutoModel.from_pretrained("model-name")
tokenizer = AutoTokenizer.from_pretrained("model-name")

_ = model.save_pretrained("./model-dir")
_ = tokenizer.save_pretrained("./model-dir")
```
Then you can do whatever you want with your model -- send it to a computing cluster, put it on a flash drive etc. Then you just do:
```
model = AutoModel.from_pretrained("path/model-dir")
tokenizer = AutoTokenizer.from_pretrained("path/model-dir")
```

> You can do it, instead of loading `from_pretrained(roberta.large)` like this download the respective `config.json` and `<mode_name>.bin` and save it on your folder then just write `.from_pretrained('Users/<location>/<your folder name>')` and thats about it.

This approach worked for facebook/m2m100_418M only after I downloaded every file at https://huggingface.co/facebook/m2m100_418M/tree/main except .gitattributes and README.md. (I can't swear that every single one is required, but I leave trial and error to determine the minimal set as an exercise for the reader.)
Wrong issue
Author: elon-3b
Pull request: no

### System Info

- `transformers` version: 4.38.0.dev0
- Platform: Linux-5.4.0-169-generic-x86_64-with-glibc2.31
- Python version: 3.10.13
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.26.1
- Accelerate config: 	
	- compute_environment: LOCAL_MACHINE
	- distributed_type: FSDP
	- mixed_precision: bf16
	- use_cpu: False
	- debug: False
	- num_processes: 2
	- machine_rank: 0
	- num_machines: 1
	- rdzv_backend: static
	- same_network: True
	- main_training_function: main
	- fsdp_config: {'fsdp_auto_wrap_policy': 'TRANSFORMER_BASED_WRAP', 'fsdp_backward_prefetch': 'BACKWARD_PRE', 'fsdp_cpu_ram_efficient_loading': True, 'fsdp_forward_prefetch': False, 'fsdp_offload_params': False, 'fsdp_sharding_strategy': 'FULL_SHARD', 'fsdp_state_dict_type': 'SHARDED_STATE_DICT', 'fsdp_sync_module_states': True, 'fsdp_transformer_layer_cls_to_wrap': 'MistralDecoderLayer', 'fsdp_use_orig_params': True}
	- downcast_bf16: no
	- tpu_use_cluster: False
	- tpu_use_sudo: False
	- tpu_env: []
- PyTorch version (GPU?): 2.2.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction


### Expected behavior



Comments:
AutoModelForCausalLM.from_pretrained Stuck
Author: Marvinmw
Pull request: no

### System Info

torch 2.2.0+cu121
transformer 4.38.1

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
model_name_or_path = "google/gemma-7b" 
transformers.logging.set_verbosity_debug()
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map="auto")
```
The code gets stuck AutoModelForCausalLM.from_pretrained

### Expected behavior

It should not be stuck and continue running.

Comments:
If I interact with the Python interpreter, it can load the model. No stuck.

If I use another new file with the same code, it works....

The problem seems very trick..  python script.py 2>&1 | tee log.txt . It does not work.. but if worked before.
[Mistral, Mixtral] Improve docs
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR improves the docs of Mistral and Mixtral, by including:

- explaining the difference between base models vs instruction tuned ones, including the newer [v2 checkpoint](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) of Mistral-7B
- use of chat templates
- quantization tips and tricks
- memory requirements
- resources such as the [Alignment Handbook](https://github.com/huggingface/alignment-handbook) by HF

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29084). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Skipping test_save_load_low_cpu_mem_usage() for all failing models
Author: hackyon
Pull request: yes

# What does this PR do?

Skips all failing unit tests for test_save_load_low_cpu_mem_usage(). This should be temporary for some of them until the correct tie_weights() have been added to the models.

I created this temporary PR just in case it takes longer to make progress with #29024, and we don't want to block other PRs while we wait for feedback. 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts 


Comments:
Hello @amyeroberts,

I came up this PR to simply ignores all the failing tests for test_save_load_low_cpu_mem_usage(). This change should be safe as it only touches tests.

This should help unblock any PRs from being merged, while we work on getting tie_weights() into some of these models with #29024.

I leave it up to you to decide this is worth merging.

Thanks! 🙏

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29118). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@hackyon Thanks for opening this and all of your work on adding this feature to our models! 

We're going to revert #28948 and then add it back after the next release, which should happen tomorrow. 

Unfortunately, our test suite doesn't seem to be correctly picking up all of the necessary tests. I know you've run a check against all models but we want to be confident that everything is in a good and stable state before the release. 

We can then combine the work in #29024 with all models having similar changes which address the current review comments. 

Sounds good, thanks for taking care of this.

Once this rollback goes in, I can come up with a single PR that covers everything all in one.
ZeroDivisonError in inverse_sqrt scheduler
Author: Sangh0
Pull request: no

Hello! I've been findiing your repository very useful.

However, I've encountered a bug. It occurs when trying to train using the inverse_sqrt scheduler, leading to a ZeroDivisionError.
Please refer to the following code link for details.
Therefore, I suggest modifying the code from `decay = 1.0 / math.sqrt((current_step + shift) / timescale)` to `decay = 1.0 / math.sqrt((current_step + shift) / (timescale + 1e-9))`.

[Bug code](https://github.com/huggingface/transformers/blob/ec29d25d9f7109f3fdaadfa51515eb6745a136ba/src/transformers/optimization.py#L292)

![zerodivisionerror](https://github.com/huggingface/transformers/assets/47784418/5ce8c101-5da9-49c6-b8c4-2a79c9ab5f80)

@muellerzr @pacman100
Could you please check this issue?

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
args = TrainingArguments(
    output_dir="./checkpoints",
    lr_scheduler_type="inverse_sqrt",  # ZeroDivisonError
)
```

### Expected behavior
```python
decay = 1.0 / math.sqrt((current_step + shift) / (timescale + 1e-9))
```

Comments:
cc @muellerzr 

Thank you for your interest in this issue. I will close this issue now.

Yoissa
Inference quality drastic change with torch.compile() and Qwen1.5-7B
Author: Qubitium
Pull request: no

### System Info

``` 
model: finetuned Qwen1.5-7B
transformers: 3.38.1
pytorch: 2.2
accelerate: 0.27.2
flash-attn: 2.5.3
os: ubuntu 22.04
gpu: 4090
```



### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Not sure how we can further debug this and provide more data but we are seeing drastic output quality difference between no-torch.compile and torch.compile.  We are new to torch.compile().

Does torch.compile inherently cause accuracy issues? Not sure if just natural part of the beast. 


model load code:
```
model = AutoModelForCausalLM.from_pretrained(
    adapter_path,
    device_map="cuda",
    config=config,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",

model = torch.compile(model, mode="reduce-overhead", fullgraph=True)
```

gen code and gen-params are same for generation. 

The difference is as if we are running a vastly different, a more inaccurate model when torch.compile() is enabled.

### Expected behavior

Minimal difference if any.

Comments:
Closed and moved to more appropriate pytorch issue: https://github.com/pytorch/pytorch/issues/120462
ImportError: cannot import name 'is_g2p_en_available' from 'transformers.utils' (/usr/local/lib/python3.10/dist-packages/transformers/utils/__init__.py)
Author: kli017
Pull request: no

### System Info

env: colab
python=3.10
trasformers=4.37.0.dev0

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi, I was run the peft_bnb_whisper_large_v2_training.ipynb from peft project. Everything is good till I met the error when I run the code `import evaluate`. I also try transformers=4.27.4, 4.33,1 and 4.36.2 and get the same error.

### Expected behavior

Anyone can help?

Comments:
I don't know which section this is, but thefunction is defined here: 
https://github.com/huggingface/transformers/blob/07f5cdcac14b672ea6934b16da432518717f5b74/src/transformers/utils/import_utils.py#L448

it was merged to transformers last week, so can't be there for earlier versions. Can you share a small reproducer? 


I have the same problem following this example: https://github.com/huggingface/notebooks/blob/main/examples/image_classification.ipynb

at AutoImageProcessor.from_pretrained(model_checkpoint)

with transformers 4.37.1

Looks like there is also this issue: https://github.com/huggingface/peft/issues/1351

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.