noamaanMulla-03/transformers-issues huggingface dataset:
[ViT] Add note about interpolation of position encodings
Author: NielsRogge
Pull request: no

### Feature request

We should add a note to the docs on the fact that, in order to fine-tune ViT on higher resolution (e.g. 512x512), one can set `interpolate_pos_encoding=True` in the forward of the model.

### Motivation

This thread on the forum: https://discuss.huggingface.co/t/fine-tuning-image-transformer-on-higher-resolution/22623/4

### Your contribution

I'll take this!

Comments:
Hi,

I'm findind hard to achieve to set the interporlate_pos_encoding = True by redefining the model's forward method.
Could you make a brief step by step case on how to do so in order to train the model, and not just make a forward pass?

I thought it was just about modifying the model.config which feeds the module parameters, as one can set the output_attentions = True per example, but I see is not the same case for the inteporlate_pos_encoding.

Thank you!
P.D: I have post this same question at the hugginface forum

Could you clarify? The only thing you need to change is pass `interpolate_pos_encoding=True` to the forward when training the model (no need to redefine the forward method).

This issue was fixed in #19103, therefore I'm closing this issue.

> 
I fine tune the model by using the Trainer build class from transformers, not directly by calling the forward method, so I'm not finding the way to set the interpolate_pos_encoding to true in that case.


Hi @NielsRogge 
Is there any method to set the `interpolate_pos_encoding` to `True` while using the Trainer API?
Not able to find a method to pass it as a parameter to the `forward` method without redefining the `forward` method.


Also - if a section to explain this step is added in the example notebooks - would be really helpful for the community.

Pinging @sgugger here - the question is whether one can set a boolean argument in the forward of a model to `True` when using the Trainer API.

No, that is not possible.

Sad to hear. Many code examples use the Trainer API. 

Would be great to bring this feature to the trainer API.

> Could you clarify? The only thing you need to change is pass `interpolate_pos_encoding=True` to the forward when training the model (no need to redefine the forward method).
> 
> This issue was fixed in #19103, therefore I'm closing this issue.

Sorry to bother. Currently there are many models based on ViT. They do not support this argument (e.g. CLIP, MAE, SAM).
And the `interpolate_pos_encoding` is at a very beginning step which makes it hard to hijack. And what I am doing is modifying the source code (paste the interpolate function and run it on the pos_embeddings). Is there any suggestion to do this more elegant since they are all ViTs principlely?
I also tried loading the models to `ViTModel`, some are ok (MAE). but for SAM and CLIP, it cannot match the weights.
LayoutLMv3FeatureExtractor(apply_ocr=True) not returning words 
Author: rahulss14
Pull request: no

I have trained a LILT using my custom dataset , everything works fine but I am not getting ocr words of the detected regions . How can I get the words .

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks! 

Hello 
thank you for your prompt response , I am new to this can you answer this here . I will ask my further questions on forum .


Awaiting your response 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix FA2 import + deprecation cycle 
Author: SunMarc
Pull request: yes

# What does this PR do ?

This PR puts back `is_flash_attn_available` and make it go through a deprecation cycle. It was removed in this [PR](#26785). As this method was never private, some remote modeling files used it. Fixes #27319



Comments:
_The documentation is not available anymore as the PR was closed or merged._

@SunMarc Merging so this can be part of a patch release.
[`Peft`] `modules_to_save` support for peft integration
Author: younesbelkada
Pull request: yes

# What does this PR do?

With https://github.com/huggingface/peft/pull/1112 being merged, we can now extend the PEFT integration to support `modules_to_save`.

What is `modules_to_save`? It is a feature in PEFT that enables users to fine-tune extra parameters in addition to PEFT adapters; in practice, users can for example further fine-tune the lm_head in addition to lora adapters

Added also some tests and few lines in the documentation

cc @amyeroberts @pacman100 @BenjaminBossan 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@amyeroberts I'll merge this PR after the patch release from PEFT !

Link to patch release: https://github.com/huggingface/peft/releases/tag/v0.6.2
Fix M4T weights tying
Author: ylacombe
Pull request: yes

# What does this PR do?

Following some failing tests [here](https://github.com/huggingface/transformers/actions/runs/6806630933/job/18508287020), I've pinpointed the bug to a tying weight issue !  This fixes the issue

cc @ydshieh, @LysandreJik  and @amyeroberts 

Also cc @Narsil who had fix a related seamlessM4t issue recently 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This shouldn't be needed, the weights should already be tied at init time.

Edit: I looked a bit more closely:

- `no_init_weights` means all tensors are initialized as meta tensors (all shared with basically a null pointer for tensor).
- Tensors are loaded where they can, NOT updating the tied weights (since information is lost view device="meta".
- `tie_weights` is called here (currently doing nothing).
- Then things crash in `dispatch_model` (Something tries to attach some data to those still meta tensors).

With the fix, `tie_weights` reaffects the missing tensors.
Something could be a bit off though, since if this shared was actually offloaded (so really kept onto disk for instance) my understanding is that it would still be `meta` at step 2, and other kind of crashes would still happen, no ?

> no_init_weights means all tensors are initialized as meta tensors (all shared with basically a null pointer for tensor).
Tensors are loaded where they can, NOT updating the tied weights (since information is lost view device="meta".
tie_weights is called here (currently doing nothing).
Then things crash in dispatch_model (Something tries to attach some data to those still meta tensors).
With the fix, tie_weights reaffects the missing tensors.

This is exactly what happens here!

> Something could be a bit off though, since if this shared was actually offloaded (so really kept onto disk for instance) my understanding is that it would still be meta at step 2, and other kind of crashes would still happen, no ?

Not sure to have a response here, what kind of crashes could happen ?

It's just information flow, here we're good only because the shared tensor is actually loaded, but with a fully offloaded to disk model it might not, right ?

Oh right, it could happen indeed. But it could happen to every model right ? because `model.tie_weights` happens [here](https://github.com/ylacombe/transformers/blob/1432fae1ee2ae621d1e88bcf70741a74ed4c17ae/src/transformers/modeling_utils.py#L3502) anyways ?

Gentle ping here to ask for a review @LysandreJik ! :hugs:
[Whisper] Fix pipeline test
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Fixes typo in Whisper pipeline test that was causing a failure on the daily CI (cc @ydshieh): https://github.com/huggingface/transformers/actions/runs/6792980649/job/18467224285


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Revert "[time series] Add PatchTST (#25927)"
Author: amyeroberts
Pull request: yes

The model was merged before final review and approval.

This reverts commit 2ac5b9325ed3b54950c6c61fd5838ac6e55a9fe1.

cc @kashif @LysandreJik 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[`CI-test_torch`] skip `test_tf_from_pt_safetensors` for 4 models
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Skip `test_tf_from_pt_safetensors` tests for MobileBert,  TransfoXL, SPeech2text and XGLM

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27481). All of your documentation changes will be reflected on that endpoint.

Hello, @ArthurZucker 
Thanks for fixing the broken tests. I have made a PR and the tests failed because of the broken tests, how could I rerun the tests ? 

https://github.com/huggingface/transformers/pull/26304

I'll re-run them for you! 

@ArthurZucker Hi, I have rebased my PR  #27351 with your latest commits, but the same CI failure still persists with `speech_to_text` (only `speech_to_text_2` is skipped in the current PR).  We need to skip `test_tf_from_pt_safetensors` tests for `speech_to_text` as well. Thanks!

Thanks! @Rocketknight1 is working on a broader fix, I don't really mind skipping this test for all the other failing PRs

The proper fix is here #27490
Update and reorder docs for chat templates
Author: Rocketknight1
Pull request: yes

Based on feedback, some bits of chat templates were unclear even after reading the docs (particularly around using chat templates for training). I've added a section on that, and also reorganized the entire doc to do the following:

- Stop using gated models (so the doctests actually work)
- Push advanced topics (writing chat templates and pushing them to the Hub) to the end, and put the most widely useful information (using chat templates in inference and training) at the start.
- Cut down on the far-too-verbose opening, and get straight to more concrete example code.

Comments:
cc @NielsRogge and @philschmid who gave me the idea for this!

_The documentation is not available anymore as the PR was closed or merged._

Pinging @NielsRogge and @philschmid if you have any comments - I'll merge tomorrow if not!
Generate: `GenerationConfig.from_pretrained` can return unused kwargs
Author: gante
Pull request: yes

# What does this PR do?

I wrote the doctest for the feature described above, but forgot to add the feature itself in the original `GenerationConfig` commit 🙃 

Fixes the failing `pytest --doctest-modules src/transformers/generation/configuration_utils.py::transformers.generation.configuration_utils.GenerationConfig.from_pretrained`

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Generate: fix `ExponentialDecayLengthPenalty` doctest
Author: gante
Pull request: yes

# What does this PR do?

The doctest was failing... but the root cause for the failure in `transformers.generation.logits_process.ExponentialDecayLengthPenalty` is the transition from `torch==2.0` to `torch==2.1` (i.e. installing `torch==2.0` fixes it).

I couldn’t find any related reference in the release notes, but I know they touched `torch.multinomial` after the 2.0 release -- it was sampling things with probability=0.0. This change may impact this doctest, as it is sample-based and it may induce probabilities=0.0.

As such, the fix consists of updating the test's outputs. I took the opportunity to improve the example as well :)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The failing test

```
src/transformers/generation/logits_process.py::transformers.generation.logits_process.WhisperTimeStampLogitsProcessor
```

is known on doctest CI, so good for me to merge. I already ping @sanchit-gandhi on slack.

@amyeroberts would you be able to merge this PR, as the failure is unrelated? (different doctest on the same file) :)

I can do it, but maybe wait @amyeroberts this time (so she knows I can 😄 without surprise)

I can merge! 

cc @sanchit-gandhi as the failing doctest is a whisper one.
dev-ai
Author: devgupta6
Pull request: no
Labels: name: New model

### Model description

It a generative artifical intelligence model. I have the architecture ready but I am facing the problem to interagte it with transformer architure. It would be great if you provide me assistance with this.

### Open source status

- [X] The model implementation is available
- [ ] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
Hi @devgupta6, thanks for opening an issue! 

The easiest and recommended way to make a model available in `transformers` is to add the modeling code directly on the hub: https://huggingface.co/docs/transformers/custom_models. Here is a more general guide on adding models: https://huggingface.co/docs/transformers/add_new_model

Hi @amyeroberts 
I had registered my LLM model using automodel like this - AutoModelForCausalLM.register(CustomAIConfig, CustomAI) 
but it is showing the error - ---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-17-298cb7a13739> in <cell line: 2>()
      1 # Load model directly
----> 2 from transformers import CustomAI
      3 model = CustomAI.from_pretrained("RANITBAG/gan")

ImportError: cannot import name 'CustomAI' from 'transformers' (/usr/local/lib/python3.10/dist-packages/transformers/__init__.py)

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
--------------------------------------------------------------------------- .

I have tried all sorts of things but it would be of great help if you could assist me with this. 




Could you link to your model on the hub? Without seeing the code I'm not able to see what the issue might be. 

Yes, sure. https://github.com/devgupta6/dev-ai this is the model link


@devgupta6 Please read the documentation pages I sent over as these contain all the information you should need. The model at the moment is just a torch implementation and doesn't have any of the necessary adaptations for the transformers library. For example, your model needs to inherit from `PretrainedModel`. We can help with bugs and issues but we can't write your code for you.
Track the number of tokens seen to metrics
Author: muellerzr
Pull request: yes

# What does this PR do?

This PR adds `num_tokens_seen` to the `TrainerState` allowing users to know how many tokens were passed in an individual batch. Uses `gather` to ensure that in DDP this can be known as well.

Fixes https://github.com/huggingface/transformers/issues/27027


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@pacman100 @amyeroberts 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@amyeroberts question: these failures are all because they cannot find the `main_input_name` in the batch. Does this mean some of the models are wrong?

For instance, one of the failures in the examples script is from `SpeechEncoderDecoderModel`, which states `main_input_name` is `"inputs"`, however when running the script it's using `"input_values":

```python
self = {'input_values': tensor([[-0.0177, -0.0188, -0.0202,  ..., -0.0032, -0.0068, -0.0039],
        [-0.0196, -0.0556, -0.0...-100, -100, -100, -100, -100, -100, -100, -100, -100,
         -100, -100, -100, -100, -100, -100, -100, -100, -100]])}
```

@muellerzr I think we don't have to worry about that if it's behind a flag in the training args. This way - if someone want to measure it then they have to make sure that `main_input_name` is properly set, but it won't break trainer for models which are currently compatible but don't have this correctly set. 

@amyeroberts failing tests seem to be unrelated

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27274). All of your documentation changes will be reflected on that endpoint.
Add `tgs` speed metrics
Author: CokeDong
Pull request: yes

add tgs metrics for `trainer`. the motivation is that: current `speed_metrics` only consider `train_samples_per_second`. but the length of each example is  not the same(especailly `cutting_off ` increase). this pr introduce `tgs` metrics, which take `tokens` into considerations.

Comments:
cc @muellerzr @pacman100 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25858). All of your documentation changes will be reflected on that endpoint.

@muellerzr @pacman100 PTAL, thx

https://app.circleci.com/jobs/github/huggingface/transformers/913233 `test_hub` failed， seems not related with current PR :(

@CokeDong please rebase with the main branch of transformers, this should ensure it passes :) 

BTW @CokeDong, you don't have to do force-pushes if you're worried about the commit-bloat post-merge, in `transformers` we squash when merging. 

> BTW @CokeDong, you don't have to do force-pushes if you're worried about the commit-bloat post-merge, in `transformers` we squash when merging. 

Got that

thanks for this great feature! 
quickly tried it with accelerate on 1-3 GPUs and the results confuse me 🤔 
t/s should be higher with 3 GPUs vs 1 GPU, right?

![W B Chart 13_11_2023, 06_25_18](https://github.com/huggingface/transformers/assets/141400217/6fb0e0db-0846-4c31-95ad-056e4922db9c)
![W B Chart 13_11_2023, 06_26_15](https://github.com/huggingface/transformers/assets/141400217/de54babe-29e7-465d-ab87-2ebc5196c313)


> thanks for this great feature! quickly tried it with accelerate on 1-3 GPUs and the results confuse me 🤔 t/s should be higher with 3 GPUs vs 1 GPU, right?
> 
> ![W B Chart 13_11_2023, 06_25_18](https://user-images.githubusercontent.com/141400217/282492227-6fb0e0db-0846-4c31-95ad-056e4922db9c.jpg) ![W B Chart 13_11_2023, 06_26_15](https://user-images.githubusercontent.com/141400217/282492332-de54babe-29e7-465d-ab87-2ebc5196c313.jpg)

hi,  `tokens/sec/gpu`(tgs)  meatures tokens throughput capability per device.

> per device

now it makes sense, thank you!
[`CI-test_torch`] skip test_tf_from_pt_safetensors and `test_assisted_decoding_sample` 
Author: ArthurZucker
Pull request: yes

# What does this PR do?
skip test_tf_from_pt_safetensors and test_assisted_decoding_sample

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27508). All of your documentation changes will be reflected on that endpoint.
The results of tensorrt and model.generate
Author: lin-lcx
Pull request: no

Torch's model inference is using
```
model = VisionEncoderDecoderModel.from_pretrained(
nougat-base)
outputs = model.generate(
    pixel_values.to(device),
    decoder_input_ids=decoder_input_ids.to(device),
    # max_length=model.decoder.config.max_length,
    early_stopping=True,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.eos_token_id,
    use_cache=True,
    num_beams=1,
    bad_words_ids=[[tokenizer.unk_token_id]],
    return_dict_in_generate=True,
)
```
Now I use tensorrt to accelerate it, and the result is the direct output of the network (for example, I have 50,000 tokens), the output size of tensorrt is (50000,). How should I input it into "model.generate"? Or is there another way to get the same result as "model.generate".
To put it simply, it is how to turn the direct output of the model into the result of model.generate.

Comments:
Allow custom logit warpers
Author: lshamis
Pull request: no

### Feature request

Currently, when generation sampling, logit warpers are automatically generated from the generation_config. 
https://github.com/huggingface/transformers/blob/0a55d9f7376f72ad3ff296d4249840021b03bcc4/src/transformers/generation/utils.py#L1641
which does not support any custom logit warpers
https://github.com/huggingface/transformers/blob/0a55d9f7376f72ad3ff296d4249840021b03bcc4/src/transformers/generation/utils.py#L811

Custom logits processor are allowed and follow a nearly identical code path.

### Motivation

I am unable to experiment with changes to sampling without also changing the logits via a LogitsProcessor.

### Your contribution

I can submit a PR, but am not necessarily the best person to do so.

Comments:
I see the same call in beam sampling and assisted generation.  Maybe custom logit warpers can be handled in `generate()` the same way `logits_processor` is used i.e., the caller can pass in a list of custom warpers as an arg `logits_warper` but has to be sure this doesn't collide with any from the config or raise?  I can take a shot at a PR if that seems like the right direction. 

Hey @lshamis, you can probably overload `_get_logits_warper` in the mean time? The goal of transformers is to be easy to develop on top of it, but not necessarly fit your particular use case! If more people from the community require it, we might consider adding this. 
FYI @gante 

Hey @lshamis @hedeershowk 👋 

As @ArthurZucker wrote, we'd like to avoid adding more custom code if possible :) 

Let me attempt to present a solution that doesn't require new code: `logits_processor` and `logits_warper` behave the same way, the only difference being that the latter are only fetched on sampling-based decoding methods. However, nothing prevents you from adding a custom `logits_processor` for sampling purposes, e.g. the `top_k` operation. The only thing you won't have control over is the order of operations (if that's relevant for your use case), in which case I'd suggest overwriting our functions as @ArthurZucker suggested!

I hope this helps 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.