noamaanMulla-03/transformers-issues huggingface dataset:
Avoid unnecessary warnings when loading `CLIPConfig`
Author: ydshieh
Pull request: yes

# What does this PR do?

Avoid unnecessary warnings when loading `CLIPConfig`: when a user doesn't change something inside `text_config`.

Fix #28042

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28108). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

>Hard-coding values like this is brittle

This is fine IMO for this situation

> The intention of the user is ambiguous. If I deliberately set "bos_token_id": 0 in the config - then I still would have unexpected behaviour but the warning wouldn't show.

Very much agreed for this!

Since we don't have a perfect solution to this issue caused by the original design, and 2 core maintainers accept change it to `info` --> Let's do this. (At least, we have this handling block, so it's easy to figure thing out when someone get trapped by [this issue](https://github.com/huggingface/transformers/pull/19954#issuecomment-1295182328))

> Do we know if users ever modify the log verbosity for debugging purposes? i.e. would it be enough to set everything to info?

The ratio of this is quite low I believe.

Changed to `info`. There are are config classes in the clip family like `altclip`, `chineseclip` etc. Will change them too before merge unless there is other opinions.
[docs] Trainer docs
Author: stevhliu
Pull request: yes

Part 2 of #27986 to finish cleaning up the `Trainer` API docs. This includes:

- moving the CUDA extension installation problems to the performance and scalability debugging [doc](https://huggingface.co/docs/transformers/main/en/debugging) where it is more appropriate
- GPU selection has its own section in the multiple GPU training [doc](https://huggingface.co/docs/transformers/main/en/perf_train_gpu_many)
- spin out the FSDP sections into their own docs
- add a link from the Trainer guide to the FSDP guide

Comments:
Test is passing! 

I rebased to incorporate the changes from #28135, but that didn't work for some reason, so I manually edited `tests/utils/test_doc_samples.py` to reflect the latest changes. Is this still ok since the change already exists on `main`?

@stevhliu Yep - it should be fine!
Generate: fix speculative decoding
Author: gante
Pull request: yes

# What does this PR do?

This PR:
- Fixes speculative decoding quality:
  - Incorrect indexing operation
  - The assistant model should sample when the larger model also samples (more generally, it should take the original model's `generation_config`)
  - Custom logits processors should also be passed to the assistant model
- Changes docs to put an emphasis on "speculative decoding" as opposed to "assisted generation", as the former is more popular

________

`RUN_SLOW=1 py.test tests/ -k speculative` was run locally to confirm that slow assisted generation whisper tests were passing.

Comments:
@amyeroberts all tests were passing, the quality of the outputs was meh in some cases. 

Going to add a slow test with a parameterization corresponding to a bad output in the previous commit, so we don't regress :)
`YolosImageProcessor` violates `longest_edge` constraint for certain images
Author: xenova
Pull request: no

### System Info

- `transformers` version: 4.35.0
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.0+cu118 (False)
- Tensorflow version (GPU?): 2.14.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.7.4 (cpu)
- Jax version: 0.4.16
- JaxLib version: 0.4.16
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@NielsRogge @amyeroberts 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction


```py
from transformers import AutoProcessor
from PIL import Image
import requests

processor = AutoProcessor.from_pretrained("Xenova/yolos-small-300") # or hustvl/yolos-small-300
url = 'https://i.imgur.com/qOp3m0N.png' # very thin image

image = Image.open(requests.get(url, stream=True).raw).convert('RGB')
output = processor(image)
print(output['pixel_values'][0].shape)  # (3, 89, 1335)
```

A shape of (3, 89, 1335) is printed out, but this shouldn't be possible due to the `longest_edge` constraint in the [config.json](https://huggingface.co/Xenova/yolos-small-300/blob/main/preprocessor_config.json#L22):
```json
"size": {
  "longest_edge": 1333,
  "shortest_edge": 800
}
```

Here is the image used:
![image](https://github.com/huggingface/transformers/assets/26504141/74c75ab1-4678-4ff0-860b-b6b35a462eb8)


### Expected behavior

The image should have the maximum edge length be at most 1333 (1335 should not be possible)

Comments:
Hi @xenova, thanks for reporting! 

Looking into it 🕵️‍♀️ 

@xenova Apologies for the delay - I've opened a PR to fix! 

@amyeroberts No worries! ;) Thanks so much!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

bump :)
Fix yolos resizing
Author: amyeroberts
Pull request: yes

# What does this PR do?

On main - running the following produces an image outside of the limits set by "longest_edge" in an images config: 

```py
from transformers import AutoProcessor
from PIL import Image
import requests

processor = AutoProcessor.from_pretrained("Xenova/yolos-small-300") # or hustvl/yolos-small-300
url = 'https://i.imgur.com/qOp3m0N.png' # very thin image

image = Image.open(requests.get(url, stream=True).raw).convert('RGB')
output = processor(image)
# Result
# main: (3, 89, 1335)
# branch: (3, 80, 1328)
print(output['pixel_values'][0].shape) 
``` 

This logic was copied from the DETR [image processing logic](https://github.com/huggingface/transformers/blob/8aca43bdb3cb9a5020f6d57589d85679dc873b1c/src/transformers/models/detr/image_processing_detr.py#L93) which comes from the [original DETR repo ](https://github.com/facebookresearch/detr/blob/3af9fa878e73b6894ce3596450a8d9b89d918ca9/datasets/transforms.py#L76). Note: this means the DETR models' image processors won't respect `longest_edge`.  

Yolos' image processor has been updated to reflect [the original model](https://github.com/hustvl/YOLOS/blob/5717fc29d727dab84ad585c56457b4de1225eddc/datasets/transforms.py#L76).

Note - this also differers from the output image size that would be obtained from using `torchvision`s `resize`. In the above example, it would be `(3, 88, 1333)`


Fixes #27381


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
disable test_retain_grad_hidden_states_attentions on SeamlessM4TModelWithTextInputTest
Author: dwyatte
Pull request: yes

# What does this PR do?

Disables `tests/models/seamless_m4t/test_modeling_seamless_m4t.py::SeamlessM4TModelWithTextInputTest::test_retain_grad_hidden_states_attentions` as discussed in https://github.com/huggingface/transformers/pull/28144#issuecomment-1864990888

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts 


Comments:
Related to #28035 and #28060 which should have already skipped it
Fix Vip-llava docs
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes some nits on the Vip-Llava docs, in fact users should be aware that the correct prompt format is different from the llava one, as stated on the model card: https://huggingface.co/llava-hf/vip-llava-7b-hf#how-to-use-the-model

Also updated the docs in the modeling

cc @amyeroberts 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28085). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Question about the output of the decision transformer
Author: Pulsar110
Pull request: no

From the code in here: https://github.com/huggingface/transformers/blob/v4.35.2/src/transformers/models/decision_transformer/modeling_decision_transformer.py#L920-L927
```      
        # reshape x so that the second dimension corresponds to the original
        # returns (0), states (1), or actions (2); i.e. x[:,1,t] is the token for s_t
        x = x.reshape(batch_size, seq_length, 3, self.hidden_size).permute(0, 2, 1, 3)

        # get predictions
        return_preds = self.predict_return(x[:, 2])  # predict next return given state and action
        state_preds = self.predict_state(x[:, 2])  # predict next state given state and action
        action_preds = self.predict_action(x[:, 1])  # predict next action given state
````
I'm not sure I understand why ` self.predict_return(x[:, 2])` or `self.predict_state(x[:, 2])` is predicting the return/next state given the state and action. From the comment on the top, `x[:, 2]` is only the action? Am I missing something?

And if this code is correct, what is the use of `x[:, 0]`?

Comments:
Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!

Thanks!

Thank you. I have created a post here: https://discuss.huggingface.co/t/question-about-the-output-of-the-decision-transformer/65384
So far no one has commented there. I'm not sure if there is a bug in the code, or maybe I do not understand it correctly, that's also why I wanted to post here. 

I don't know this model at all so pinging @edbeeching the author of the PR! 

Hi @Pulsar110 , thanks for your question. It would probably be best to reach out to the authors with this question as our implementation aims to match the author's codebase: https://github.com/kzl/decision-transformer/blob/e2d82e68f330c00f763507b3b01d774740bee53f/gym/decision_transformer/models/decision_transformer.py#L97

If I were to hazard a guess I would think that there is a mistake in their implementation and we should be indexing entry 0 at some point. 

Let us know what they say and perhaps we can update our implementation with any changes they suggest. I will close the issue for now but feel free to reopen it with more questions or if you hear back from them.
[Whisper] Use torch for stft if available
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

PoC PR to highlight what using the `torch` backend for STFT would look like. In a toy benchmark over 2500 samples, the torch backend is approx 4x faster than the native `numpy` one: https://github.com/sanchit-gandhi/codesnippets/blob/main/benchmark_preprocess.ipynb


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26119). All of your documentation changes will be reflected on that endpoint.

cc @Vaibhavs10 you'll probably get a speed-up again using the torch backend for STFT if you install from `main`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Given the increasing usage of the Whisper model as the 'de-facto' speech recognition model in the library, I propose merging this PR to improve the pre-processing time for Whisper, and then generalising the changes for all other ASR models in the library in a follow-up PR (since this will be more involved and take some more time). WDYT @ArthurZucker @ylacombe?

I also ran some long-form transcription algorithms for Flax Whisper to demonstrate the WER equivalence with using the torch implementation, and the huge speed-up we get to the overall transcription time: https://wandb.ai/sanchit-gandhi/distil-whisper-long-form-test/reports/Whisper-FE-numpy-vs-torch--Vmlldzo2MzAwNzYy

Yep for sure! Already approved, make sure to rebase on main! 

I think it's a great idea given the visibility of the model and the speedup you found!
Error while importing the transformers 
Author: iamshreeram
Pull request: no

### System Info

**Transformers Version** : 4.36.0.dev0 
**Platform** : Mac OS 
**Python** : 3.9.18


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce : 
1. Run the following program to translate to the target language:
```
from transformers import pipeline

pipeline_generator = pipeline(
    "automatic-speech-recognition",
    "facebook/seamless-m4t-v2-large",
)

transcript = pipeline_generator("https://www2.cs.uic.edu/~i101/SoundFiles/preamble10.wav", generate_kwargs={"tgt_lang": "spa",   },)
```

2. This throws the following exception:

```
Traceback (most recent call last):
  File "/Users/home/ram/project/python/text-translation-speech/ttrans.py", line 12, in <module>
    transcript = pipeline_generator("https://www2.cs.uic.edu/~i101/SoundFiles/preamble10.wav", generate_kwargs={"tgt_lang": "ta",   },)
  File "/Applications/anaconda3/envs/dub/lib/python3.9/site-packages/transformers/pipelines/automatic_speech_recognition.py", line 357, in __call__
    return super().__call__(inputs, **kwargs)
  File "/Applications/anaconda3/envs/dub/lib/python3.9/site-packages/transformers/pipelines/base.py", line 1134, in __call__
    self.get_iterator(
  File "/Applications/anaconda3/envs/dub/lib/python3.9/site-packages/transformers/pipelines/base.py", line 1182, in get_iterator
    feature_extractor = self.feature_extractor if self.feature_extractor is not None else self.image_processor
AttributeError: 'AutomaticSpeechRecognitionPipeline' object has no attribute 'image_processor'
```

3. Despite not importing `image_processor`, the exception is thrown.



### Expected behavior

Produce output in the target language as seen in this [thread](https://github.com/facebookresearch/seamless_communication/issues/237#issuecomment-1864534911), running with the expected results.


Comments:
@sanchit-gandhi @ylacombe same issue as reported in #28162 



@amyeroberts , Thank you for investigating. Everything is functioning properly after the installation of the stable version of transformers (`4.36.2`).
In PreTrainedTokenizerBase add missing word in error message
Author: petergtz
Pull request: yes

# What does this PR do?

This is a minor cosmetic change in the error message when invoking the tokenizer:

"text input must of type" -> "text input must be of type"


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker
@sgugger 
<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27949). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Even more TF test fixes
Author: Rocketknight1
Pull request: yes

This PR hopefully fixes the last remaining issues from the `build()` PR and gets the CI back to normal!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28146). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This should be ready to go now, and finally fixes the remaining CI issues after the `build()` PR!
Fix slow backbone tests - out_indices must match stage name ordering
Author: amyeroberts
Pull request: yes

# What does this PR do?

Fixes slow autobackbone tests failing on nightly after #27606 

#27606 enforces the out_indices and out_features to be in the same order as the stage names. This ensures backbone selects the correct features in its forward pass. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28186). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
AWQ quantization example in colab failures
Author: poedator
Pull request: no

### System Info

colab

### Who can help?

@younesbelkada @SunMarc 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

page https://huggingface.co/docs/transformers/main_classes/quantization refers to colab example for AWQ quantization.
this example fails:

1) [optional] After the 4.35.0 release it fails with error `ImportError: libcudart.so.12: cannot open shared object file: No such file or directory` during `from awq import AutoAWQForCausalLM`. I thought that it should work now without git+https installs with 4.35.

2)quantization_config = AwqConfig(... fails, because it expects `AWQLinearVersion.GEMM` and gets string "GEMM" instead. see my other issue today. 

3) the resulting model has no config at all. Which looks odd to me. It has `model.quant_config` though.


### Expected behavior

more normal

Comments:
Thanks @poedator for reporting, everything should be back to normal now. 
We tested it with autoawq==0.1.4 which did not had the strong requirement for CUDA>=12.0, that has been introduced in the most recent release: https://github.com/casper-hansen/AutoAWQ/releases/tag/v0.1.6 . To install autoawq with CUDA < 12.0 I had to call `pip install https://github.com/casper-hansen/AutoAWQ/releases/download/v0.1.6/autoawq-0.1.6+cu118-cp310-cp310-linux_x86_64.whl` (cc @casper-hansen in case I am mistaken somewhere).  
To retrive the config from the resulting model you need to check the attribute `model.model.config`, as done per the line:
```python
# the pretrained transformers model is stored in the model attribute + we need to pass a dict
model.model.config.quantization_config = quantization_config
```
The point 2 is temporarly solved if you call `lower()` to the version string, until https://github.com/huggingface/transformers/pull/27320 gets merged where it should perform str to enum conversion.
Again thanks for reporting, on my end everything looks resolved, I'll leave this issue open but if you think that everything has been addressed feel free to close the issue! 

@younesbelkada You are correct. PyPi only allows one release file, so I chose to update to the latest configuration of torch 2.1.0 and CUDA 12.1.1 that PyTorch offers. Extra wheels are made available with torch 2.0.1 and CUDA 11.8.0 for compatibility on the GitHub release itself.

I updated the installation section to reflect the above:
https://github.com/casper-hansen/AutoAWQ#install

Ok makes sense, therefore I believe there is no fix to upstream on autoawq side, the only fix is on the notebook, which is done!

@casper-hansen , @younesbelkada - thank you for the quick response!
Add CogVLM
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds CogVLM natively into the Transformers library (it's already usable with `trust_remote_code=True`, but with this PR one can run it without the xformers, einops and triton dependencies).

To do:

- [x] remove triton dependency for rotary embeddings (or make `FastRotaryEmbedding` optional)
- [x] wait for #27690 to be merged
- [ ] decide on attributes to be saved for multimodal processors: see https://github.com/huggingface/transformers/pull/27761

Comments:
A cleaner implementation I'm working on is here: https://github.com/NielsRogge/transformers/tree/add_cogvlm_cleaner. It implements the model like llava, by adding the image tokens inside the model, rather than creating them in the processor class.

Closing this one in favor of the PR above.