noamaanMulla-03/transformers-issues huggingface dataset:
Please support GGUF format for UMT5EncoderModel
Author: nitinmukesh
Pull request: no
Labels: name: Feature request

### Feature request

```python
import torch
from transformers import UMT5EncoderModel

from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="city96/umt5-xxl-encoder-gguf", filename="umt5-xxl-encoder-Q8_0.gguf"
)
text_encoder = UMT5EncoderModel.from_pretrained(
    "Wan-AI/Wan2.1-I2V-14B-480P-Diffusers", # or any Hub path with the correct config
    subfolder="text_encoder",
    gguf_file=path,
    torch_dtype=torch.bfloat16,
)
```

```
umt5-xxl-encoder-Q8_0.gguf: 100%|████████████████████████████| 6.04G/6.04G [07:46<00:00, 12.9MB/s]
config.json: 100%|███████████████████████████████████████████████████████| 854/854 [00:00<?, ?B/s]
Traceback (most recent call last):
  File "C:\aiOWN\diffuser_webui\UMT.py", line 9, in <module>
    text_encoder = UMT5EncoderModel.from_pretrained(
  File "C:\aiOWN\diffuser_webui\venv\lib\site-packages\transformers\modeling_utils.py", line 271, in _wrapper
    return func(*args, **kwargs)
  File "C:\aiOWN\diffuser_webui\venv\lib\site-packages\transformers\modeling_utils.py", line 4158, in from_pretrained
    state_dict = load_gguf_checkpoint(gguf_path, return_tensors=True, model_to_load=dummy_model)["tensors"]
  File "C:\aiOWN\diffuser_webui\venv\lib\site-packages\transformers\modeling_gguf_pytorch_utils.py", line 399, in load_gguf_checkpoint
    raise ValueError(f"GGUF model with architecture {architecture} is not supported yet.")
ValueError: GGUF model with architecture t5encoder is not supported yet.

```

### Motivation

It will help low VRAM users to run WAN

### Your contribution

I am not a developer so can't contribute

Comments:
Hi, can I take this up if no one else is doing it?

@minhaj3 

Please if you can, thank you.

Hi , Is there any progress on this Issue ?
@minhaj3 

Hi , Is there any progress on this Issue ?
@minhaj3 

Example:
text_encoder = UMT5EncoderModel.from_pretrained(
        "city96/umt5-xxl-encoder-gguf",
        gguf_file="umt5-xxl-encoder-Q8_0.gguf",
        torch_dtype=torch.float16,
    )
All works fine
[trainer] handle case where EOS token is None in `generation_config`
Author: gante
Pull request: yes

# What does this PR do?

Follow-up to #38441: the added special token handling caused crashes when the EOS token is undefined/None in an existing `generation_config`

This PR also:
- improves the thrown warning
- updates the eli5 dataset in the CLM task (whose code I've used to debug this change -- the old dataset is not compatible with `datasets` 4.0) 

Comments:
(@SunMarc I tagged you since I believe you're still the person to tag for trainer-related issues -- let me know otherwise 🤗 )

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40127). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix hidden torchvision>=0.15 dependency issue
Author: yonigozlan
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/39907

Comments:
> Do we really want to have to go through a PIL mapping if it was supposed to be torch only? Let's just define it as a constant explicitly based on version for clarity no? 🤗

It was a way to check the version only in one place but agreed that it doesn't make much sense in the long run

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39928). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: conditional_detr, deformable_detr, detr, eomt, grounding_dino, mask2former, maskformer, mobilenet_v2, mobilevit, oneformer, rt_detr, sam, segformer, yolos
Hidden torchvision>=0.19.0 dependency results in quiet import failures of e.g. PreTrainedModel
Author: tomaarsen
Pull request: no

Hello!

The changes from #37055 introduce a dependency of `torchvision>=0.19.0` as they rely on `InterpolationMode.NEAREST_EXACT` from `torchvision.transforms.InterpolationMode`, whereas previously I was able to use older versions. This interpolation mode was added in https://github.com/pytorch/vision/pull/6754, which was first released in 0.19.0. 

Even worse: when using older versions of `torchvision`, importing parts of `transformers` will quietly (!!!) fail, e.g.:
```python
from transformers import PreTrainedModel
```
```
ImportError: cannot import name 'PreTrainedModel' from 'transformers'
```

The underlying source only becomes apparent when importing with the full path:
```python
from transformers.modeling_utils import PreTrainedModel
```
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "[sic]\lib\site-packages\transformers\modeling_utils.py", line 74, in <module>
    from .loss.loss_utils import LOSS_MAPPING
  File "[sic]\lib\site-packages\transformers\loss\loss_utils.py", line 21, in <module>
    from .loss_d_fine import DFineForObjectDetectionLoss
  File "[sic]\lib\site-packages\transformers\loss\loss_d_fine.py", line 21, in <module>
    from .loss_for_object_detection import (
  File "[sic]\lib\site-packages\transformers\loss\loss_for_object_detection.py", line 32, in <module>
    from transformers.image_transforms import center_to_corners_format
  File "[sic]\lib\site-packages\transformers\image_transforms.py", line 22, in <module>
    from .image_utils import (
  File "[sic]\lib\site-packages\transformers\image_utils.py", line 62, in <module>
    PILImageResampling.NEAREST: InterpolationMode.NEAREST_EXACT,
  File "C:\Users\tom\.conda\envs\setfit\lib\enum.py", line 429, in __getattr__
    raise AttributeError(name) from None
AttributeError: NEAREST_EXACT
```

I'm fine with upgrading my `torchvision`, but perhaps that should be required (via `setup.py`), and we really should be wary with hidden import errors.

cc @yonigozlan @zshn25 @Yann-CV 
cc @ArthurZucker 

- Tom Aarsen

_Originally posted by @tomaarsen in https://github.com/huggingface/transformers/issues/37055#issuecomment-3153954171_
            

Comments:
Thanks @tomaarsen for raising this! Since we don't have torchvision as a dependency (for now at least), it won't be possible to require users to have torchvision>=0.19.0, so I guess the best solution here would be to fall back to NEAREST. I'll open a PR!

Looks like InterpolationMode.NEAREST_EXACT was added in 0.15:
https://github.com/pytorch/vision/blob/98f8b3757c0648724064ca95434b18281c43c5f6/torchvision/transforms/functional.py#L57
Opened a PR to fix this: https://github.com/huggingface/transformers/pull/39928
🌐 [i18n-KO] Translated `jamba.md` to Korean
Author: skwh54
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.md` to Korean" 으로 부탁드립니다 -->
# What does this PR do?

Translated the `jamba.md` file of the documentation to Korean.
Thank you in advance for your review.

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 KREW 팀원들에게 리뷰를 요청하는 아래 주석을 노출해주세요!-->
May you please review this PR?
@jungnerd, @luckyvickyricky, @chelsseeey, @amo33, @maximizemaxwell, @D15M4S
<!-- @harheem, @nsbg, @Youngdong2, @xhaktm00, @ssunbear, @ChoHyoungSeo, @judy-choi -->
<!-- @4N3MONE, @Kim-Ju-won, @ahnjj, @FacerAin, @ssum21, @TaskerJang, @HyunZ118 -->
<!-- @yijun-lee, @songi104, @chhaewxn, @AhnJoonSung, @jihyun-0611, @seopp, @pyapyapya -->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. KREW 팀원들의 리뷰가 끝난 후에 아래 주석을 노출해주세요! -->
@stevhliu May you please review this PR?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39890). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

build-doc

[Building docs for all languages...](https://github.com/huggingface/transformers/actions/runs/16913570860)
🌐 [i18n-KO] Translated `main_classes/optimizer_schedules.md` to Korean
Author: nevertmr
Pull request: yes

# What does this PR do?

Translated the `main_classes/optimizer_schedules.md` file of the documentation to Korean.
Thank you in advance for your review.

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)
May you please review this PR?
@jungnerd, @chelsseeey, @skwh54, @amo33, @maximizemaxwell, @D15M4S
@harheem
@4N3MONE
@yijun-lee

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)
@stevhliu May you please review this PR?

Comments:
LGTM :D
정확한 번역이라 별다른 보완이 필요 없어 보여요!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39713). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I'll merge this for you once we've ironed out an issue with the doc-builder, no action is necessary on your part! 🤗 

Thank you for the information🙂 @stevhliu 


build-doc

[Building docs for all languages...](https://github.com/huggingface/transformers/actions/runs/16913578021)
[ `Core`]  Refactor modeling code
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Refactor LlamaAttention following #34282

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34987). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
🚨🚨  [generate] ignore `cache_implementation="hybrid"` hub defaults
Author: gante
Pull request: yes

# What does this PR do?

Follow-up to #40039 -- on models where the hub checkpoint specifies `cache_implementation="hybrid"` (static sliding window hybrid cache), UNSETS this value. This will make the model use the dynamic sliding window layers by default.

Any user-defined `cache_implementation="hybrid"` will apply the static sliding window hybrid cache.

✅ with this PR, we no longer suffer from super slow 1st `generate` calls on models with hybrid caches. See snippet below.

🚨🚨  
- BC breaking: the cache returned by `generate` calls now will have `DynamicSlidingWindowLayer` layers by default (as opposed to `SlidingWindowLayer`, its static-shaped counterpart).
- Tests: we might have failing slow tests after merging this PR. Tests using checkpoints with `cache_implementation="hybrid"` may have small differences in their `generate` calls.

___________________________________

Time of 1st `generate` call test snippet:
```py
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
from time import time


model_id = "google/gemma-3-1b-it"
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", torch_dtype=torch.bfloat16)

tokenizer = AutoTokenizer.from_pretrained(model_id)
prompt = "What is the capital of France?"
model_inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

torch.cuda.synchronize()
start = time()
generate_outputs = model.generate(
    **model_inputs, max_new_tokens=100, do_sample=False, return_dict_in_generate=True
)
torch.cuda.synchronize()
end = time()
print(f"Time taken: {end - start} seconds (new)")
# time on RTX 4090: 0.79s

torch.cuda.synchronize()
start = time()
generate_outputs = model.generate(
    **model_inputs, max_new_tokens=100, do_sample=False, return_dict_in_generate=True, cache_implementation="hybrid"
)
torch.cuda.synchronize()
end = time()
print(f"Time taken: {end - start} seconds (old)")
# time on RTX 4090: 21.62s

```

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: gemma3

run-slow: gemma3

(On my machine, on `main`, there are 3 slow test failures. This PR sees the same failures)

This comment contains run-slow, running the specified jobs: 

models: ['models/gemma3']
quantizations: [] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40135). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

(Let's hope we don't have too many failures on tomorrow's daily CI due to shapes... 🤞🍀)
🌐 [i18n-KO] Translated `gpt2.md` to Korean
Author: taemincode
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.md` to Korean" 으로 부탁드립니다 -->
# What does this PR do?

Translated the `gpt2.md` file of the documentation to Korean.
Thank you in advance for your review.

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 KREW 팀원들에게 리뷰를 요청하는 아래 주석을 노출해주세요!-->
May you please review this PR?
<!-- @jungnerd, @luckyvickyricky, @chelsseeey, @skwh54, @amo33, @maximizemaxwell, @D15M4S -->
<!-- @harheem, @nsbg, @Youngdong2, @xhaktm00, @ssunbear, @ChoHyoungSeo, @judy-choi -->
<!-- @4N3MONE, @Kim-Ju-won, @ahnjj, @FacerAin, @ssum21, @TaskerJang, @HyunZ118 -->
<!-- @yijun-lee, @songi104, @chhaewxn, @AhnJoonSung, @jihyun-0611, @seopp, @pyapyapya -->
@yijun-lee @harheem @4N3MONE @jungnerd

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. KREW 팀원들의 리뷰가 끝난 후에 아래 주석을 노출해주세요! -->
@stevhliu May you please review this PR?

Comments:
LGTM! 
번역을 정말 잘해주셨네요. 좋은 번역 감사합니다! 👍

@stevhliu Oops 😬 Thanks for the feedback! I've updated the `toctree` with the translated doc as suggested. I'll wait for the doc-builder issue to be resolved before merging 🤗

build-doc

[Building docs for all languages...](https://github.com/huggingface/transformers/actions/runs/16913562887)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39808). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

build-doc

[Building docs for all languages...](https://github.com/huggingface/transformers/actions/runs/16942114342)
🌐 [i18n-KO] Translated `optimizers.md` to Korean
Author: chelsseeey
Pull request: yes

# What does this PR do?

Translated the `optimizers.md` file of the documentation to Korean.
Thank you in advance for your review.

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [X] Check for missing / redundant translations (번역 누락/중복 검사)
- [X] Grammar Check (맞춤법 검사)
- [X] Review or Add new terms to glossary (용어 확인 및 추가)
- [X] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [X] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

May you please review this PR?
@jungnerd, @luckyvickyricky, @chelsseeey, @skwh54, @amo33, @maximizemaxwell, @D15M4S 
<!-- @harheem, @nsbg, @Youngdong2, @xhaktm00, @ssunbear, @ChoHyoungSeo, @judy-choi -->
<!-- @4N3MONE, @Kim-Ju-won, @ahnjj, @FacerAin, @ssum21, @TaskerJang, @HyunZ118 -->
<!-- @yijun-lee, @songi104, @chhaewxn, @AhnJoonSung, @jihyun-0611, @seopp, @pyapyapya -->

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. KREW 팀원들의 리뷰가 끝난 후에 아래 주석을 노출해주세요! -->
@stevhliu May you please review this PR? 

Comments:
build-doc

[Building docs for all languages...](https://github.com/huggingface/transformers/actions/runs/16914017001)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40011). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

build-doc

[Building docs for all languages...](https://github.com/huggingface/transformers/actions/runs/16942121713)
🌐 [i18n-KO] Translated grounding-dino.md to Korean
Author: TaskerJang
Pull request: yes

# What does this PR do?

Translated the `grounding-dino.md` file of the documentation to Korean.
Thank you in advance for your review.

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 KREW 팀원들에게 리뷰를 요청하는 아래 주석을 노출해주세요!-->
May you please review this PR?
<!-- @jungnerd, @luckyvickyricky, @chelsseeey, @skwh54, @amo33, @maximizemaxwell, @D15M4S -->
<!-- @harheem, @nsbg, @Youngdong2, @xhaktm00, @ssunbear, @ChoHyoungSeo, @judy-choi -->
@4N3MONE, @Kim-Ju-won, @ahnjj, @FacerAin, @ssum21, @TaskerJang, @HyunZ118
<!-- @yijun-lee, @songi104, @chhaewxn, @AhnJoonSung, @jihyun-0611, @seopp, @pyapyapya -->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. KREW 팀원들의 리뷰가 끝난 후에 아래 주석을 노출해주세요! -->
 @stevhliu May you please review this PR? 

Comments:
주원님 리뷰 반영 이외에는 수정할 부분 없어 보입니다. 번역 고생하였습니다 :) 

(AP, Average Precision이 이 문서를 읽을 분들이 일반적으로 쉽게 이해할 수 있을지 혼동이 오긴 하는데, 이 부분만 한번 고려해주시면 감사하겠습니다)

Looks like you’ve already wrapped up all the changes 🙂 Great work, and thank you for your effort!
Thanks to your translation, I was able to enjoy such a wonderful read!  Really appreciate the time and care you put into it.
LGTM 👍 

LGTM 👍🏻
좋은 번역 감사합니다! 제가 따로 수정 요청 드릴 부분이 없네요!

build-doc

[Building docs for all languages...](https://github.com/huggingface/transformers/actions/runs/16914051056)

build-doc

[Building docs for all languages...](https://github.com/huggingface/transformers/actions/runs/16914647531)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39861). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

build-doc

[Building docs for all languages...](https://github.com/huggingface/transformers/actions/runs/16941733974)
🚨 Use lru_cache for sine pos embeddings MaskFormer
Author: yonigozlan
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Since sine pos embeddings only depends on a fixed feature shape (for these models at least), this seems like a free speedup. Changed Maskformer and related only for now, as its sine pos embedding module is used in sam2, but happy to extend this to other models in the library!

The speed gains are not negligible, for sam2_video on my machine, it goes from 24.4 to 25.8 fps, a 6% speed improvement


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_40007). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: mask2former, maskformer, oneformer

In this case I don't think it will have any noticeable impact, as the tensors involved are relatively small, and only one will be stored in the cache. But yes definitely something I'll keep in mind. I can also experiment in another PR to register these embeddings with `register_buffer` like we do with rope embedding frequencies, but there might be some differences because of dtype casting

We can also decrease lru_cache max_size to `1`, so that only one tensor is stored at a time.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: mask2former, maskformer, oneformer