noamaanMulla-03/transformers-issues huggingface dataset:
filter out callable attributes from tokenizer_config in save_pretrained
Author: shuttie
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/28472

As discussed in the upstream bug report, `add_special_token` can be both kwargs parameter passed to the `from_pretrained` and a method in `SpecialTokensMixin.add_special_tokens`. Not sure that it's the best way for doing this, but this PR:
* ensures that no methods are passed into the tokenizer config
* so it can be safely serialized to json with `json.dumps`


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@ArthurZucker 




Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[i18n-de] Translate CONTRIBUTING.md to German
Author: khipp
Pull request: yes

# What does this PR do?

This PR adds a German translation for the CONTRIBUTING.md.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Documentation: @stevhliu
Contributors: @flozi00


Comments:
Thanks for the great suggestions!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28954). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> # What does this PR do?
> 
> This PR adds a German translation for the CONTRIBUTING.md.
> 
> ## Before submitting
> - [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
> 
> ## Who can review?
> 
> Documentation: @stevhliu
> Contributors: @flozi00
> 

b44567538b48e63354ecd0a87ba0492888bcfbeb
DIT Text Detection model
Author: arvisioncode
Pull request: no

### Feature request

Add the text detection models of the dit that are in https://github.com/microsoft/unilm/tree/master/dit/text_detection to the hugging face, to be able to execute them easily with the transformers library like other models as https://huggingface.co/microsoft/dit-base

### Motivation

These models have very good performances and are very interesting to be able to include in hf both to make simple inferences and to transform them to other formats such as ONNX


Comments:
I know and I desperately want that model to be available... :D however for that, Mask R-CNN first needs to be integrated. I have a PR on that #22973, I need to finish that one up, then we can add it

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@NielsRogge Hi, Do you have any plan for this? I want to export the DiT detection (Layout Analysis) model to ONNX.
Please let me know If there's a new comment about this.

Hello @arvisioncode ,
Have you managed to tackle this? I'm willing to fine-tune the base model for this task on CORD or FUNSD but not sure if masked image modeling is the right paradigm for this. Eventually I need to train the model on an Indic language. I've done the full flow for DONUT but DiT seems to be lighter. Will appreciate any resource you might point me towards.

Thanks
Abstract image processor arg checks.
Author: molbap
Pull request: yes

# What does this PR do?

This refactors existing image processor argument checks that sprawl out on all existing models that have an `ImageProcessor`. 
Lines such as

```python
        if do_resize is not None and size is None:
            raise ValueError("Size and max_size must be specified if do_resize is True.")

        if do_rescale is not None and rescale_factor is None:
            raise ValueError("Rescale factor must be specified if do_rescale is True.")

        if do_normalize is not None and (image_mean is None or image_std is None):
            raise ValueError("Image mean and std must be specified if do_normalize is True.")
```

can be abstracted away in a `validate...` function residing in `image_utils`. 
Also, fixing (when it doesn't break BC) some cases where existence of arguments is checked, but the actual `preprocess` method doesn't seem to call them. `bridgetower` doesn't pass  `center_crop` from init, `chinese_clip` does not `convert_to_rgb`, and so on. 

## Who can review?

 @amyeroberts




Comments:
Also linked to #28711 as I discovered logic flow issues here, seems fitting to abstract them separately and deal with the actual processing in the main PR. Here I'll try to stick to verifications and fix what's necessary to satisfy existing tests.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28843). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts I think it's ok to take another look at this one now! Improved a few things, didn't add much, will rebase the other refactor off of that one
FIX [`bnb` / `tests`] Propagate the changes from #29092 to 4-bit tests
Author: younesbelkada
Pull request: yes

# What does this PR do?

As per title, I overlooked the fix and forgot to push the changes of https://github.com/huggingface/transformers/pull/29092 in 4-bit tests 😢 

cc @amyeroberts @Titus-von-Koeller 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29122). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
FEAT [`Trainer` / `bnb`]: Add RMSProp from `bitsandbytes` to HF `Trainer`
Author: younesbelkada
Pull request: yes

# What does this PR do?

As requested by the community, this PR adds the support for bnb RMSProp optimizers to HF Trainer ! 
`RMSProp` exists in bitsandbytes since its first commit: https://github.com/TimDettmers/bitsandbytes/commit/7439924891496025edf60c9da6a782f362a50c70#diff-8384af03566f84c3055f3fee7b1516696a1546d130d63935714af17781d6202b so this PR is compatible with all versions of bnb. 

I also added nice tests, which all pass on my machine

Fixes: https://github.com/huggingface/trl/issues/1336

cc @amyeroberts and @Titus-von-Koeller FYI !


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29082). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Generate: unset GenerationConfig parameters do not raise warning
Author: gante
Pull request: yes

# What does this PR do?:

Thank you @fxmarty for raising [this issue](https://github.com/huggingface/transformers/pull/25381#issuecomment-1952527813).

This PR allows users to unset (= set to `None`) unused parameters to ensure `generation_config.validate()` doesn't throw a warning. Previously, this was not possible when a parameter had a non-`None` default.

For instance, the following snippet would throw a warning before this PR:
```py
from transformers import GenerationConfig

generation_config = GenerationConfig()
generation_config.update(temperature=None)
generation_config.validate()
# "... UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `None` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`."
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29119). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Move kernel compilation to init rather than at import stage
Author: NielsRogge
Pull request: no
Labels: name: Should Fix

name: bug

### Feature request

Some models like Deformable DETR rely on custom CUDA kernels to be compiled as seen [here](https://github.com/huggingface/transformers/blob/f7ef7cec6c6c162087421f36a17eabdbb223579d/src/transformers/models/deformable_detr/modeling_deformable_detr.py#L54).

Currently these are compiled when importing the Transformers library, but this needs to happen later, when initializing the models.

All custom CUDA kernels are defined here: https://github.com/huggingface/transformers/tree/main/src/transformers/kernels

### Motivation

This is pretty important to fix actually as currently running `make fixup` on machines that have `ninja` installed will compile all these kernels before running the quality checks, making it super slow.. thanks @younesbelkada for the info

### Your contribution

Not sure I can help with this, the current workaround is simply removing `ninja` from the environment

Comments:
Revert low cpu mem tie weights
Author: amyeroberts
Pull request: yes

# What does this PR do?

Reverts #28948 and #29043

See relevant comment: https://github.com/huggingface/transformers/pull/29110#issuecomment-1953847826

cc @hackyon  @ydshieh 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29135). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sounds good, thanks for taking care of this!
Save (circleci) cache at the end of a job
Author: ydshieh
Pull request: yes

# What does this PR do?

This way, `pytest` will run before `cache saving` and we have access to the results earlier in the case of partial or no cache loaded.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29141). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Caching image prototype embeddings for image-guided object detection using OWL-ViT
Author: jakubhejhal
Pull request: no

### Feature request

The [OWL-ViT](https://arxiv.org/abs/2205.06230) model currently supports image-guided one-shot object detection by using reference image embeddings as the input to the classification head instead of the text embedding. This is implemented by the [image_guided_detection](https://huggingface.co/docs/transformers/v4.37.2/en/model_doc/owlvit#transformers.OwlViTForObjectDetection.image_guided_detection) method.

There are 2 problems
- it doesn't support passing multiple reference images as input
- the reference image is passed through the image encoder every time

In practice I'd like to use the model's `image_guided_detection` for inference on larger dataset and computing the reference image query embedding for each image I'm doing an inference on is clearly wasteful, as the query embeddings are not dependent on the target image. 

1) Is there a way to cache the query image embeddings?
2) And is there a way to use multiple query images for one target image?

### Motivation

In practice One-shot learning is an extreme case of Few-Shot learning and it's usually very hard / impossible to represent the whole class with only one reference image. 

Therefore a natural extension is to use multiple prototypical images capturing the detected object in various situations, lightning conditions etc.
But as of now, the running time of the OWL-ViT scales linearly with the number of query images, which makes it impractical for real-world usage. 

### Your contribution

I'm unfortunately not very experienced with hugging face and I don't know even where to start if I tried to implement this myself.

Comments:
Hi, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.
AI2 Olmo 7B does not support Flash-Attention 2.0. ValueError: OLMoForCausalLM does not support Flash Attention 2.0 yet.
Author: KaifAhmad1
Pull request: no
Labels: name: New model

### Model description

Model Name: allenai/OLMo-7B


### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
FIX [`PEFT` / `Trainer` ] Handle better peft + quantized compiled models
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes: https://github.com/huggingface/transformers/issues/29033

Even though quantized models + compile + peft is not really stable (might not work OTB for all users), the current way we deal with peft compiled models leads to errors that are hard to interpret to users such as the one described in https://github.com/huggingface/transformers/issues/29033 

I will run some tests on my see to check if torch.compile + qlora is supported

cc @amyeroberts 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29055). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

LGTM! Let me know when it's out of draft and you want a final review 

It would be good to add a test for compiled models alongside this change 

Thanks @amyeroberts ! This is ready for review ! 
Note currently QLoRA + torch.compile is not really supported as we need to make some changes to make 4bit layers from bnb compatible with torch.compile . Therefore I added a test that simply checks if the trainer is correctly initialized so that we don't get the unappropriate error from #29033 . Once compile + QLoRA will be supported, the integration will be seemless as users will simply have to update `bitsandbytes` to make it work

Thanks @amyeroberts for the review! I just adapted the PR according to your suggestion, one concern I have with this new approach is that if bnb supports `torch.compile` in the next months we'll need to do some version check on trainer's init and make the code slightly bloated - though I am happy with this approach as well as I don't think this is that bad. What do you think? 🙏 

@younesbelkada How soon and with what confidence do you think bnb will support this? 

 I agree we want to avoid bloat as much as possible, but I think it's better overall to be explict and have well defined behaviour for our users. The easiest way to avoid many version checks is to only support the latest version of bnb. 

OK sounds good! I agree with that approach!
Currently we don't have any ETA :/ it might take a bit long, and it will also depend if the community will ask for it 

@younesbelkada I would certainly be interested if bnb could support torch.compile!
Raise unused kwargs image processor
Author: molbap
Pull request: yes

# What does this PR do?

This PR captures all kwargs passed to an `ImageProcessor` `preprocess` method and compares them to what's expected, raising an exception or logging an informative message when a difference is found.
This will 
1) Make preprocess methods more reliable
2) Inform users when an expected kwargs has no actual impact in the preprocessing

- [x] Did you write any new necessary tests?


## Who can review?

- vision models: @amyeroberts



Comments:
Finishing up draft, removing this validation functionality from #28711 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29063). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Main concern I had was using `inspect`, so I moved it to the test suite instead of the actual validation function. That way we don't inspect at runtime, we just compare static lists. For new model additions, `self._valid_processor_keys` remain optional (if it's not there, nothing is run) but it'd be a recommendation.
As to what the validation raises, for now it's a `logger.info`, could be a warning instead but should not raise an exception as a good chunk of existing models /processors from the hub would break down
Generate: missing generation config eos token setting in encoder-decoder tests
Author: gante
Pull request: yes

# What does this PR do?

These tests were failing with low likelihood, all for the same reason as fixed in [this recent PR](https://github.com/huggingface/transformers/pull/28923): there should be no EOS token to enable endless generation, but the generation config still had the default value.

I couldn't find more occurrences of this pattern.

Example of a failed run fixed by this PR: https://app.circleci.com/jobs/github/huggingface/transformers/1099434

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29146). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Added image_captioning version in es and included in toctree file
Author: gisturiz
Pull request: yes

# What does this PR do?

Translated image_captioning from en to es from issue https://github.com/huggingface/transformers/issues/28936 began by @stevhliu. I will continue to go through the documentation and make the correct translations.

(closed previous PR and reopened to rebasing issue)

Fixes # https://github.com/huggingface/transformers/issues/28936


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@stevhliu @aaronjimv

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29104). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Drop path is ignored in DINOv2
Author: fepegar
Pull request: no

### System Info

- `transformers` version: 4.38.0.dev0
- Platform: Linux-5.15.0-91-generic-x86_64-with-glibc2.31
- Python version: 3.11.7
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.2.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: not necessarily
- Using distributed or parallel set-up in script?: no

### Who can help?

@amyeroberts @NielsRogge

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I'm not getting any errors, and I think sharing a script doesn't make sense in this case.

The issue is simply that two `Dinov2DropPath` layers are being instantiated in the `Dinov2Layer`:

https://github.com/huggingface/transformers/blob/7d312ad2e9473cd3a0ea3e9b206b8ed3c147e9be/src/transformers/models/dinov2/modeling_dinov2.py#L374-L392

But they're not being used anywhere else:

https://github.com/huggingface/transformers/blob/7d312ad2e9473cd3a0ea3e9b206b8ed3c147e9be/src/transformers/models/dinov2/modeling_dinov2.py#L394-L423

### Expected behavior

These layers should probably not be ignored. Moreover, I think there's no reason to instantiate two different ones.

I've implemented a fix in https://github.com/fepegar/transformers/pull/1/files. Please let me know if you'd like me to open a PR here.

I've tried running `pytest`, but I'm getting some `pytest`-related errors (not failing tests). Happy to report that somewhere else as well.

Comments:
Hey, thanks for the issue! I've checked out your branch, from what I'm seeing tests are passing on your fix, would you mind opening a PR? 
Also, since this will affect training, do you have a script that compares both in a training scenario? AFAIK current integration tests for Dinov2 are not in a training setting.

Hi @molbap. Thanks for your response! I've created
- #29147.

The high-level effects of these changes would take quite a lot of work to measure, but here's a little snippet:

```python
>>> import torch
>>> from transformers import Dinov2Model
>>> 
>>> torch.set_grad_enabled(False)
>>> torch.manual_seed(0)
>>> 
>>> model = Dinov2Model.from_pretrained("facebook/dinov2-base", drop_path_rate=0.3)
>>> model.train()
>>> 
>>> x = torch.rand(1, 3, 224, 224)
>>> out_1 = model(x)
>>> out_2 = model(x)
>>> torch.all(out_1.last_hidden_state == out_2.last_hidden_state)
```

The output is `tensor(True)` in `main`, indicating that the depth is deterministic because the drop path rate is not being used, where it's `tensor(False)` in my branch due to stochastic depth being properly enabled.