noamaanMulla-03/transformers-issues huggingface dataset:
[Whisper] Make tokenizer normalization public
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Using the Whisper English normalizer is common practice when evaluating Whisper models on English ASR. Here, we have to normalize the predictions, e.g. using the argument `normalize=True` to the tokenizer `.decode` method:
https://github.com/huggingface/transformers/blob/5aec50ecaf9c1c039cde85881f0586110f845859/src/transformers/models/whisper/tokenization_whisper.py#L633

However, we also have to normalize the reference, which is most easily done by calling the **private** method `_normalize`: https://github.com/huggingface/transformers/blob/5aec50ecaf9c1c039cde85881f0586110f845859/src/transformers/models/whisper/tokenization_whisper.py#L509

This PR updates the tokenizer to use a **public** method for the second normalization step, the recommended design for exposed methods. Note that I have chosen here to deprecate the existing private method `_normalize`, rather than removing it blindly, since I anticipate that it has been accessed by some users already and want to prevent a hard breaking change. Happy to remove it in one go if we feel it's ok removing a private method.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28136). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add French translation: french README.md
Author: ThibaultLengagne
Pull request: yes

# What does this PR do?

Add the French version of README.md

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@stevhliu and @MKhalusova


Comments:
Hello,

Thanks @stevhliu and @Sarapuce for the review. I have fixed every remark. 

Seems good to go :+1: 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28696). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Adding CrossMAE
Author: johko
Pull request: no
Labels: name: New model

### Model description

Hey,
the recently released [CrossMAE](https://crossmae.github.io/) seems like it would be a nice addition to transformers.

Basically the model improves MAE by using Cross-Attention instead of Self-Attention on the tokens and thereby decreasing the needed FLOPS quite significantly. At the same time it seems to be able to keep the performance of MAE or even improve it a bit.

Maybe there are already plans of integrating it @NielsRogge ?

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Project Page: https://crossmae.github.io/
GitHub Repo: https://github.com/TonyLianLong/CrossMAE
Paper: https://arxiv.org/pdf/2401.14391.pdf

Comments:
Opened a PR which leverages [PyTorchModelHubMixin](https://huggingface.co/docs/huggingface_hub/package_reference/mixins#huggingface_hub.PyTorchModelHubMixin) on the original repo: https://github.com/TonyLianLong/CrossMAE/pull/2
[DO NOT MERGE] Hf quantizer refactor
Author: younesbelkada
Pull request: yes

# What does this PR do?

Built on top of https://github.com/huggingface/transformers/pull/26610 - this PR is just to see if I don't any surprising diff similar as in https://github.com/poedator/transformers/pull/4

Comments:
Do you see possibility of people mixing and matching different quantization params or even different quantizers? Use cases:
- less compression in MLP vs attention
- wild mix of methods to get that extra 1/1000s of perplexity

If supported, what should be the way? 
1) from_pretrained() calling the right quntizer as it loads each module?
1) custom HFQuantizer subclass that applies methods as they fit the modules

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28703). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks @poedator for your comments and ideas - I think the way forward would be to extend the `xxxQuantizer` by adding new arguments in the corresponding quantization config object - e.g.  `quantize_mlp_only` . I feel mixing different quantization approaches (your second point) might be a bit too much of an edge case but contributors can always create a new quantizer for it `MixedQuantizer` with a `MixedQuantizationConfig`. 

I will merge the commits of this PR directly in https://github.com/huggingface/transformers/pull/26610 to properly credit @poedator from his great work ! closing this - thanks @ArthurZucker for the review and offline discussions!
Don't allow passing `load_in_8bit` and `load_in_4bit` at the same time
Author: osanseviero
Pull request: yes

# What does this PR do?

This PR disallows having both `load_in_8bit` and `load_in_4bit` set simultaneously. This would help avoid unexpected behaviors or broken configurations.

## Example

**Without the PR**

```
from transformers import BitsAndBytesConfig

BitsAndBytesConfig(load_in_8bit=True, load_in_4bit=True)
ValueError: load_in_4bit and load_in_8bit are both True, but only one can be used at the same time
```

works fine

**With the PR**

```
from transformers import BitsAndBytesConfig

BitsAndBytesConfig(load_in_8bit=True, load_in_4bit=True)
ValueError: load_in_4bit and load_in_8bit are both True, but only one can be used at the same time
```

errors as I would expect

Comments:
Hey folks, should we merge this PR? :D 

Definitely, @Titus-von-Koeller if you don't see any issue what do you think of merging this PR?
cc @amyeroberts @ArthurZucker for a core-maintainer review before merging
Move CLIP _no_split_modules to CLIPPreTrainedModel
Author: lz1oceani
Pull request: yes

# What does this PR do?
Move _no_split_modules to CLIPPreTrainedModel so that device_map=auto can support all CLIP models.

## Before submitting
- [] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [] Did you write any new necessary tests?

## Who can review?
@amyeroberts

Comments:
> Hey! Thanks, I think #27851 already partly fixed this for the encoder, let's merge the decoder part if you want!

Hi, I have looked at the current version of master branch and that fix. I guess CLIPModel is not fixed, and the code is shown as following:
```
@add_start_docstrings(CLIP_START_DOCSTRING)
class CLIPModel(CLIPPreTrainedModel):
    config_class = CLIPConfig

    def __init__(self, config: CLIPConfig):
        super().__init__(config)
```
By following llama's code, I suggest to move all _no_split_modules to the PreTrainedModel. I can update my PR if you accept this solution. Also, I need to reformat some files to pass the workflow in the past. Let me check it latter.
```
class LlamaPreTrainedModel(PreTrainedModel):
    config_class = LlamaConfig
    base_model_prefix = "model"
    supports_gradient_checkpointing = True
    _no_split_modules = ["LlamaDecoderLayer"]
    _skip_keys_device_placement = "past_key_values"
    _supports_flash_attn_2 = True
```

Llama is not multimodal, this it only has one actual model while clip has two. The no split modules should be different for the clip encoder and the clip decoder

It is OK.. I just add _no_split for CLIP Model now.. Could you merge the PR to the main branch?

The class CLIPModel contains both text encoder and image encoder. Thus, I include ["CLIPTextEmbeddings", "CLIPEncoderLayer"] to the _no_split.
Fix typo of `Block`.
Author: xkszltl
Pull request: yes

Models:
- text models: @ArthurZucker and @younesbelkada

Was introduced in:
- https://github.com/huggingface/transformers/pull/27942

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28727). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Nice catch on checking on GH !

Sounds good to me, added.

BTW why 4.40?
Any convention and consideration regarding the deprecation cycle?

> BTW why 4.40? Any convention and consideration regarding the deprecation cycle?

We typically deprecate in 2 release cycles and this commit will be part of 4.38. 

@xkszltl For the failing tests on CI, running `make fixup` and pushing the changes should resolve the issue and trigger a new CI run

> @xkszltl For the failing tests on CI, running `make fixup` and pushing the changes should resolve the issue and trigger a new CI run

Would be great if we can actually lint in CI (not just check) and print out the git diff.
This way we don't have to run anything locally and can make whatever change needed directly from the log.

And that's also how I usually setup linter CI, based on observation that not everyone likes the extra step.

> This way we don't have to run anything locally and can make whatever change needed directly from the log.

Isn't having to look at a diff in the CI run and manually apply the changes more work than just having the linter run locally? 

> Isn't having to look at a diff in the CI run and manually apply the changes more work than just having the linter run locally?

Good question.
My opinion is: not always.
- For people continuously developing on the project, yes, a simple linter command is the easiest.
- For people occasionally joining, or currently focus on other projects, spending extra effort to setup another environment locally with possibly conflict dependencies is a lot more than copy-pasting diff from CI log.

Fixed.
Also change to log only once or it may be too annoying.
[`HfQuantizer`] Move it to "Developper guides"
Author: younesbelkada
Pull request: yes

# What does this PR do?

Move the "How to add a new quantization method" tutorial into "Developper Guide" which seems to be a better appropriate place for that tutorial rather than in "Performance and scalability"

cc @stevhliu   @ArthurZucker @amyeroberts 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28768). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Swinv2config isnt working with depth estimator
Author: hackkhai
Pull request: no

### System Info


ValueError: Unrecognized configuration class <class 'transformers.models.swinv2.configuration_swinv2.Swinv2Config'> for this kind of AutoModel: AutoBackbone.
Model type should be one of BeitConfig, BitConfig, ConvNextConfig, ConvNextV2Config, DinatConfig, Dinov2Config, FocalNetConfig, MaskFormerSwinConfig, NatConfig, ResNetConfig, SwinConfig, TimmBackboneConfig, VitDetConfig.


### Who can help?

@amyeroberts @Narsil 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import pipeline

pipe = pipeline(task="depth-estimation", model="Intel/dpt-swinv2-large-384")
result = pipe("http://images.cocodataset.org/val2017/000000039769.jpg")
result["depth"]
```

### Expected behavior


ValueError: Unrecognized configuration class <class 'transformers.models.swinv2.configuration_swinv2.Swinv2Config'> for this kind of AutoModel: AutoBackbone.
Model type should be one of BeitConfig, BitConfig, ConvNextConfig, ConvNextV2Config, DinatConfig, Dinov2Config, FocalNetConfig, MaskFormerSwinConfig, NatConfig, ResNetConfig, SwinConfig, TimmBackboneConfig, VitDetConfig.


Comments:
@amyeroberts would you be able to merge #27742? Will resolve this PR ;) 

Also make sure to use Transformers from source as the PR won't be included until the next release: 
`pip install --upgrade git+https://github.com/huggingface/transformers.git`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[Whisper] Refactor whisper
Author: ArthurZucker
Pull request: yes

# What does this PR do?
The goal of this PR is to allow the users to do the following : 
```python 
...
whisper_model.generate(audio, return_timestamps = True)
whisper_model.generate(audio, return_timestamps = True, task = Transcribe)
```
The language is automatically detected. This also simplifies the pipeline calls, and add a good example of `generation_config` 's intended usage. 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

"The language is automatically detected". From my experience the language detection by Whisper is very unreliable. Will it still be possible to specify language?

Sure, let's make sure we still allow the language to be past! Thanks for pointing this out

Once #21257 is merged, the tests here should also pass ! 

Pipeline tests need #21269 to be merge 😉 

The two failing tests are from the latest modification of the multilingual tokenizer's config
Fix transformers.utils.fx compatibility with torch<2.0
Author: fxmarty
Pull request: yes

Fixes https://github.com/huggingface/transformers/issues/28690

Tested on 1.13 that `pytest tests/models/opt/ -k "test_torch_fx" -s -vvvvv` passes, while it is currently failing with torch<2.0 following https://github.com/huggingface/transformers/pull/28447 (`AttributeError: module 'torch.nn.functional' has no attribute 'scaled_dot_product_attention'`).

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28774). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Further pin pytest version (in a temporary way)
Author: ydshieh
Pull request: yes

# What does this PR do?

#28758 tried to pin `pytest<8.0.0` version, however, in `doc_test_job` job, there is a command 

> pip install --upgrade --upgrade-strategy eager pytest pytest-sugar

which bring it back to `8.0.0` again after the desired version being installed with `pip install -e .[dev]`.

This PR changes it to 

> pip install --upgrade --upgrade-strategy eager 'pytest<8.0.0' pytest-sugar

this is not a very ideal solution, but let's fix it quickly and I will see if I can have a better solution for the long term.

Comments:
A [run](https://app.circleci.com/pipelines/github/huggingface/transformers/83431/workflows/145ee093-f7cb-4912-a64c-6b7f6ca74e98/jobs/1075839) shows we do get the desired version (pytest==7.4.4) now. (see `Show installed libraries and their versions`)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28780). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[`Backbone`] Use `load_backbone` instead of `AutoBackbone.from_config`
Author: amyeroberts
Pull request: yes

# What does this PR do?

Uses `load_backbone` in place of `AutoBackbone.from_config` in the modeling files. This is the first part of a series of changes to enable loading timm or transformers models with the same call i.e. removing the if/else structure [we see in models like DETR](https://github.com/huggingface/transformers/blob/8278b1538ecc89dad8ebca510a31a86bc8645edb/src/transformers/models/detr/modeling_detr.py#L345). 

This forms part of the work to be able to load pretrained backbones from timm or transformers interchangeably into a new model. 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28661). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker This is just internal. We still want to be able to load a backbone from the config. In fact, this still happens inside `load_backbone`. `load_backbone` just enables us to pass the model's config directly, and then figures out how to load the model based on what's there e.g. a backbone checkpoint or a config.
Task-specific pipeline init args
Author: amyeroberts
Pull request: yes

# What does this PR do?

Related to: https://github.com/huggingface/transformers/pull/28439#issue-2074806467

Adds a function to build the pipelines init arguments based on the processing objects it accepts: tokenizer, image processor, feature extractor. 

This replaces `PIPELINE_INIT_ARGS` for the task-specific arguments so as to avoid e.g. `tokenizer` being listed as an input for some models when it's not correct. 

Removes `task` as an input argument for the task-specific pipelines (the task is already specified).


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
Thanks for this work ! I don't see the PR's documentation though, would be nice to take a look at the rendered stuff.

Also would this be interesting @amyeroberts : https://github.com/huggingface/doc-builder/issues/465 ? (I've seen a discrepancy between docstring and typing in safetensors, and thought this might be interesting in transformers if done in core specific places potentially).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28439). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@pcuenca Thank you for your review! I believe I've addressed all of your comments. 

@Narsil We finally have the docs 🥳 Re typing and docstring checks, I'm definitely pro when the types are given. We currently something which does this in the check_docstrings - [specifically here](https://github.com/huggingface/transformers/blob/708b19eb093bcfb4efcfa229925b249397c2bd5e/utils/check_docstrings.py#L976). However, all of the pipeline classes are ignored because they accept arguments which aren't documented e.g. most accept `tokenizer` even if it isn't used. Once possibility is explicitly setting the documented kwargs in the `__init__` and passing the rest through `**kwargs` in this or a follow up PR. WDYT?
Generate: documented function to compute the transition scores
Author: gante
Pull request: yes

# What does this PR do?

Fixes #18616; Addresses comments in #5164, #20008 (and in a few other issues that I've lost track of).

## The issue

A few users would like to have a simple function to obtain the transition scores (i.e. the logits for each selected token at generate time). This is very useful for exploring the generated contents and simplifies the construction of powerful color-coded interfaces (e.g. [this one](https://joel.tools/codegen/)). It is also commonly requested to compare our models against OpenAI's.

We had a function for that in PT, `compute_beam_transition_scores`, but it was unknown to most users. This is because it was limited to Beam-based approaches, was not in our documents, and had no examples.

## The solution

This PR upgrades the function above to a first-class citizen 🥇 :
1. Makes it compatible with all generation strategies (e.g. Sample)
2. Adds a flag to renormalize the logits before fetching the right ones, which is a frequent downstream use
3. Adds it to the documentation
4. Populates the docstring with examples for which I've got questions a few times (how to print the token probabilities and how to recompute the score of the sequences in beam search)

In the process, I've decided to update the name of the function (`compute_beam_transition_scores` -> `compute_transition_scores`), to match better what it does. Although this technically breaks the API, the function was not part of our documented functions and, given the number of related issues, I'd say it was mostly unknown.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger merging as the failing test is a known flaky test (`tests/models/auto/test_modeling_auto.py::AutoModelTest::test_from_pretrained_dynamic_model_distant`)

Thanks! This function is super helpful for my use case.

OMG, I'm just hacking on making something like this. Thrilled to discover this function. Super helpful!
Pin Torch to <2.2.0
Author: Rocketknight1
Pull request: yes

PyTorch 2.2.0 was pushed to `pip` about 30 minutes ago and is causing our CI to fail. It isn't showing up on Pytorch.org yet, so this may be an accidental push from the maintainers (the same thing happened with TF 2.16 last week)

For now, we pin `torch<2.2.0` to fix the CI.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28785). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

thanks . merged now to prevent people crying on their PRs
[`bnb`] Fix bnb slow tests
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes the current failing BNB slow tests on main: https://github.com/huggingface/transformers/actions/runs/7705429360/job/21003940543 

https://github.com/huggingface/transformers/pull/28266 broke the tests which has been merged right before the quantizer refactoring PR. 

Since the attributes `load_in_4bit` and `load_in_8bit` have been removed in favor of a property method, the fix is simply to explicitly pass them in the `to_dict` method.

cc @ArthurZucker 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28788). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.