noamaanMulla-03/transformers-issues huggingface dataset:
Add MovieChat Model
Author: kumar-devesh
Pull request: no
Labels: name: New model

### Model description

MovieChat proposes a Vision Foundation model + LLM + Long short-term memory-based solution to long-range video understanding addressing computation, memory, and long-range temporal understanding challenges using transformer model tokens as a fixed amount of memory.
Project Page: https://rese1f.github.io/MovieChat/

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Paper: https://arxiv.org/abs/2307.16449
Github: https://github.com/rese1f/MovieChat

Comments:
I would like to work towards adding this model to huggingface transformers @NielsRogge @amyeroberts 

@kumar-devesh Great! Feel free to open a PR and tag us or @rafaelpadilla when it's ready for review. Let us know if you have any questions about adding the model to the library.

which models can I use as a template for converting this model to huggingface format? @amyeroberts 

Hi @kumar-devesh, 🙂

[Here](https://huggingface.co/docs/transformers/main/en/add_new_model) you can find a useful documentation explaining the expected structure for a newly added model.

MovieChat seems close to MultiModal (Visual Question Answering) models. There's a list of MultiModal models on the left menu [on this page](https://huggingface.co/docs/transformers/main/en/index) that you could use as reference.


Hi @amyeroberts @rafaelpadilla @kumar-devesh Let me know if I can pick this up?

Hi @Dev-Khant i am working on this issue currently :)


@amyeroberts @rafaelpadilla implementation for the model uses opencv and decord for reading videos from paths. When trying to pass a partial hf port model a video tensor, the frames decoded are not the same and lead to slightly different text output for the models. The conversion scripts i went through perform inference on the author's code and the hf port to verify similarity in outputs. Is there any workaround for this?

Model outputs:
hf model: `In the video, a man is standing in a kitchen with his arms outstretched in front of him. He is standing in front of a white sink and appears to be preparing food`
original model: `In the video, a man is standing in a kitchen with his arms outstretched in front of him. He is standing next to a stainless steel kitchen sink.`


> Hi @Dev-Khant i am working on this issue currently :)
> 

OK cool @kumar-devesh :) 

Hi @kumar-devesh ,

Thank you for raising this question. :) 

If I understood your question correctly, the new model should work in the frame/image level, so your model itself shouldn't worry about extracting the frames. 

Your preprocessing should receive an `ImageInput` object, containing the already extracted frames, which could be done with opencv, decord, ffmpeg, etc. So, for that, you can use the same library adopted by the official code.

You can use similar models in the `transformers` as reference, like `timesformer`,`videomae`, `vivit`, etc.

I hope that clarifies your question.

@kumar-devesh Hey, are you still working on it? If not I would like take this on
Use `torch 2.3` for CI
Author: ydshieh
Pull request: yes

# What does this PR do?

It has been sometime torch 2.3 is released. I should have done this earlier.

In particular, the torch compile related tests require it.

(We might have issue on T4 even with torch 2.3 however)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30837). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @amyeroberts For CI on GitHub Actions + such changes, we don't really run CI before merge. We are going to use torch 2.3, it's for sure. We will *try* to fix the failing tests once the reports are sent via CI after merge.

The changes on this CI won't affect pull requests' CircleCI.
Ignore errors when deleting old checkpoints in trainer
Author: akrogager
Pull request: yes

# What does this PR do?

Fixes #17265


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@sgugger


Comments:
_The documentation is not available anymore as the PR was closed or merged._

https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L2371

same logic might needs this fix also ?
Add support for custom checkpoints in MusicGen
Author: jla524
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #29873 (issue)

Add a new argument for custom checkpoint and pass it into`get_pretrained`

## Who can review?

@ElizavetaSedova and @ArthurZucker 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Hey! Can you make sure the CI is green and ping me again for a reveiw!

Hey @ArthurZucker! The CI is green now.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30011). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add missing dependencies in image classification example
Author: jla524
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #30059 (issue)

Tested on an M1 Mac and a 7900 XTX system

## Who can review?

@ArthurZucker @amyeroberts 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
[LLaVa-NeXT] Small fixes
Author: NielsRogge
Pull request: yes

# What does this PR do?

Some small fixes after #29850 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30841). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Adding UDopForTokenClassification to be able to use UDOP for token classification tasks on Documents
Author: fathyshalaby
Pull request: no

### Feature request

As there is a LayoutLMV3ForTokenClassification, why not a UDOPForTokenClassification also I just followed this tutorial by @NielsRogge and the inference seems not to work as this class doesnt exist yet or relased yet for Auto: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/UDOP/Fine_tune_UDOPEncoderModel_on_FUNSD_(HuggingFace_Trainer).ipynb


### Motivation

https://github.com/NielsRogge/Transformers-Tutorials/blob/master/UDOP/Fine_tune_UDOPEncoderModel_on_FUNSD_(HuggingFace_Trainer).ipynb

### Your contribution

Would think that @NielsRogge  is more knowledgable about this, but generally he already added it in the notebook, so would just be a copy paste job. Would be happy to do it as well if needed.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
ImportError: cannot import name 'PaliGemmaForConditionalGeneration' from 'transformers' (/usr/local/lib/python3.10/dist-packages/transformers/__init__.py)
Author: KaifAhmad1
Pull request: no

### System Info

transformers Version: 4.40.2
cuda 12.1 
os Windows 11 x64 
pip 24.0 
python 3.10.10 

### Who can help?

Hi, @amyeroberts, I'm encountering an error. Could you help me with it?


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

``` Python 
from tqdm import tqdm
import bitsandbytes as bnb
import torch
import torch.nn as nn
from peft import LoraConfig, prepare_model_for_kbit_training, get_peft_model
from trl import SFTTrainer
from transformers import (
    PaliGemmaForConditionalGeneration,
    AutoTokenizer,
    TrainingArguments,
    BitsAndBytesConfig,
    pipeline,
    logging
)
from sklearn.model_selection import train_test_split
from datasets import Dataset
```

```
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-15-11114d78412d> in <cell line: 7>()
      5 from peft import LoraConfig, prepare_model_for_kbit_training, get_peft_model
      6 from trl import SFTTrainer
----> 7 from transformers import (
      8     PaliGemmaForConditionalGeneration,
      9     AutoTokenizer,

ImportError: cannot import name 'PaliGemmaForConditionalGeneration' from 'transformers' (/usr/local/lib/python3.10/dist-packages/transformers/__init__.py)

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.

```

### Expected behavior

It will load the model without raising any issues!

Comments:
Hi @KaifAhmad1, PaliGemma has been released on the development branch, but isn't part of the v4.40.2 release. 

To work from the development branch and get all the most recent (but not guaranteed stable) features, you can install from source: `pip install git+https://github.com/huggingface/transformers`. 

We're planning on having a release this week, so you should be able to install from pypi as part of a stable release soon 

Thanks @amyeroberts
FEAT / Bitsandbytes: Add `dequantize` API for bitsandbytes quantized models
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/30177

This PR adds a new feature `dequantize` in order to de-quantize models for interesting usecases such as the one described in #30177 

The API is very simple:

```python
from transformers import AutoModelForCausalLM, BitsAndBytesConfig, AutoTokenizer

model_id = "facebook/opt-125m"

model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=BitsAndBytesConfig(load_in_4bit=True))
tokenizer = AutoTokenizer.from_pretrained(model_id)

model.dequantize()

text = tokenizer("Hello my name is", return_tensors="pt").to(0)

out = model.generate(**text)
print(tokenizer.decode(out[0]))
```

Users just need to make sure they have enough GPU RAM in order to store the unquantized model, otherwise they might face unexpected behaviour

Added the support for 4-bit / 8-bit models and nice tests + docs to educate users on how to use this new API.

cc @amyeroberts @SunMarc 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30806). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Yeah this is great, thanks

Great thanks @RonanKMcGovern ! let us know how it goes
[Idefics2] Improve docs, add resources
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR fixes the code snippet of Idefics2, adds links to resources and explains how to optimize the model using Flash Attention/quantization.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30717). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Feel free to approve and merge
Cache: add new flag to distinguish models that `Cache` but not static cache
Author: gante
Pull request: yes

# What does this PR do?

See title :)

Models like `Jamba` (will only ever support `HybridMambaAttentionDynamicCache`) or `Mistral` (currently only supports dynamic caches, like `SinkCache`) need a distinction in terms of support flags, to run the appropriate checks and tests.

Without this flag, we have the following bad practices:
1. manually skip tests
2. don't do cache support checks in `generate`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30800). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker we can remove this attribute when we update all Llama-copied models to support the static cache :D
Disable the FA backend for SDPA on AMD GPUs
Author: mht-sharma
Pull request: yes

# What does this PR do?

Garbage values may occur during model generation with models like LLama, Mistral, and Mixtral, particularly when utilizing multi-gpu setup and `device_map=auto` alongside `SDPA` and `FA`.

The PR disables the FA on SDPA for ROCM devices

Comments:
cc @SunMarc @ydshieh 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30850). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Ready for a merge @mht-sharma ?
run finetune_rag.py -- errror OSError: Not enough disk space. Needed: 139.13 GiB (download: 66.09 GiB, generated: 73.03 GiB, post-processed: Unknown size)
Author: wangyu185
Pull request: no
Labels: name: bug

### System Info

faiss-cpu                     1.7.2
datasets                      2.4.0
psutil                        5.7.0
torch                         1.12.1+cu113
ray                           2.0.0
pytorch-lightning             1.7.2
transformers                  4.21.1
GitPython                     3.1.27

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

!python3 /content/drive/MyDrive/colab/rag/finetune_rag.py \
    --data_dir /content/drive/MyDrive/colab/rag/data \
    --output_dir /content/drive/MyDrive/colab/rag/output_data \
    --model_name_or_path facebook/rag-token-base \
    --model_type rag_sequence \
    --fp16 \
    --gpus 8 \
    --profile \
    --do_train \
    --do_predict \
    --n_val -1 \
    --train_batch_size 8 \
    --eval_batch_size 1 \
    --max_source_length 128 \
    --max_target_length 25 \
    --val_max_target_length 25 \
    --test_max_target_length 25 \
    --label_smoothing 0.1 \
    --dropout 0.1 \
    --attention_dropout 0.1 \
    --weight_decay 0.001 \
    --adam_epsilon 1e-08 \
    --max_grad_norm 0.1 \
    --lr_scheduler polynomial \
    --learning_rate 3e-05 \
    --num_train_epochs 100 \
    --warmup_steps 500 \
    --gradient_accumulation_steps 1

OSError: Not enough disk space. Needed: 139.13 GiB (download: 66.09 GiB, generated: 73.03 GiB, post-processed: Unknown size)


### Expected behavior

Why rag need 139Gib?

Comments:
Hi @wangyu185 👋 RAG is a retrieval-augmented model, meaning that it queries some database. It needs to have the database in memory. Please check the [model card](https://huggingface.co/facebook/rag-token-nq) and the [documentation](https://huggingface.co/docs/transformers/model_doc/rag) for more info.

As per our [issues guidelines](https://github.com/huggingface/transformers/blob/main/ISSUES.md), we reserve GitHub issues for bugs in the repository and/or feature requests. For any other matters, we'd like to invite you to use our [forum](https://discuss.huggingface.co/) 🤗

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @wangyu185 have you resolve this error?
Video-LLaVa: Fix docs
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes model id in the docs to the correct one

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30855). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Docs: update example with assisted generation + sample
Author: gante
Pull request: yes

# What does this PR do?

Fixes example failing doctest, caused by #30778.

The seed is set at the beginning of the example. Before, the assistant model was doing sampling, so it “consumed” the random state set by the seed. Now, the assistant is not doing sampling, so it doesn’t consume the random state. When the main model produces its samples, it sees a different random state!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30853). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Whisper for audio classification: input_values not present in features
Author: manjualoshious
Pull request: no
Labels: name: Audio

inputs = feature_extractor(sample["array"], sampling_rate=sample["sampling_rate"])

print(f"inputs keys: {list(inputs.keys())}")

print(
    f"Mean: {np.mean(inputs['input_values']):.3}, Variance: {np.var(inputs['input_values']):.3}"
)

     82         If the key is a string, returns the value of the dict associated to `key` ('input_values', 'attention_mask',


Comments:
Hi @manjualoshious, what is the issue here being reported? 

So that we can best help you, please make sure to follow the [issue template](https://github.com/huggingface/transformers/blob/main/.github/ISSUE_TEMPLATE/bug-report.yml) and include: 

* The running environment: run `transformers-cli env` in the terminal and copy-paste the output
* A minimal code reproducer. At the moment, we cannot copy-paste the example code provided as we don't have access to `sample`
* The full error traceback

Could you also update the title to something more meaningful? 


cc @sanchit-gandhi 

> This issue is solved now.

@manjualoshious Could you please share how this was solved for anyone coming to this issue in the future? 


inputs keys: ['input_features']
TST / Quantization: Reverting to torch==2.2.1
Author: younesbelkada
Pull request: yes

# What does this PR do?

Autoawq and other libraries are not compatible with torch 2.3.0: https://github.com/casper-hansen/AutoAWQ/issues/466

To avoid failures in our CI, let's temporary build torch==2.2.1 in our docker images 

cc @ydshieh @SunMarc 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30866). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.