noamaanMulla-03/transformers-issues huggingface dataset:
An example for finetuning FLAVA or any VLP multimodel using trainer (for example for classification)
Author: Ngheissari
Pull request: no
Labels: name: Examples

name: Feature request

### Feature request

There is no example of finetuning any VLP model using trainer. I would appreciate an example

### Motivation

The way to use trainers with any Vision and Language pretrained model is not clear. 

### Your contribution

None. 

Comments:
Hi,

Notebooks for FLAVA will soon be available in https://github.com/NielsRogge/Transformers-Tutorials.

You can find already some tutorials here: https://github.com/apsdehal/flava-tutorials.

cc @apsdehal 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

The issue was auto marked as closed, but there aren't yet any resources on how to fine-tune FLAVA.
Neither of the links posted above by @NielsRogge have instructions on fine-tuning.

I'm posting to also express my interest on this.

are there any finetuning tutorials yet in 2024?
fix _update_model_kwargs_for_generation
Author: Jintao-Huang
Pull request: yes

# What does this PR do?

The default value of `model_inputs` is None, and there is a line of code later on `model_kwargs["cache_position"] = model_inputs.get("cache_position", None)`, which is unsafe. So, if the value of `model_inputs` is `None`, it should be set to `{}`.


When running the following code with the `main` branch code, an error occurs. However, it runs successfully using `transformers==4.38.*`.

```python
# pip install ms-swift -U
import os

from swift.llm import (
    get_model_tokenizer, get_template, inference, ModelType,
    get_default_template_type, inference_stream
)
from swift.utils import seed_everything
import torch

model_type = ModelType.qwen1half_0_5b_chat
template_type = get_default_template_type(model_type)
print(f'template_type: {template_type}')  # template_type: qwen

kwargs = {}
# kwargs['use_flash_attn'] = True  # use flash_attn

model, tokenizer = get_model_tokenizer(model_type, torch.float16,
                                       model_kwargs={'device_map': 'auto'}, **kwargs)
template = get_template(template_type, tokenizer)
seed_everything(42)

model.generation_config.max_new_tokens = 128

query = 'hello'
gen = inference_stream(model, template, query)
print_idx = 0
print(f'query: {query}\nresponse: ', end='')
for response, history in gen:
    delta = response[print_idx:]
    print(delta, end='', flush=True)
    print_idx = len(response)
print()
print(f'history: {history}')

```

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@gante 

Comments:
Hi @Jintao-Huang 👋 

This need is not needed after https://github.com/huggingface/transformers/pull/29467 gets merged in :) As such, I'm not approving/merging this PR so as to not generate conflicts
[PyTorch/XLA] Fix extra TPU compilations introduced by recent changes
Author: alanwaketan
Pull request: yes

# What does this PR do?

This PR tries to fix some extra TPU compilations caused by recent HF changes.
1. PyTorch/XLA doesn't support sdpa yet. So we need to set the default attention implementation to eager.
2. tensor.item() will trigger tpu graph synchronization. We should avoid using it in the training loop.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker @younesbelkada 

Comments:
Some how the pull request is not synced with my branch. Not sure why?

> Some how the pull request is not synced with my branch. Not sure why?

It's resolved. Never mind. Maybe -f is not recognized.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29158). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the reviews!
Refactor TFP call to just sigmoid()
Author: Rocketknight1
Pull request: yes

We're seeing [issues](https://huggingface.co/google/gemma-7b-it/discussions/71) caused by Tensorflow Probability + Keras 3 incompatibilities. Annoyingly, these are being thrown even when people are trying to use unrelated Torch models, as long as they have TF / TFP installed!

The underlying cause is that we import TFP in one of our pipeline files if it's present, and this file is run when `pipeline` is imported, even if the user is using an entirely different pipeline/framework. I removed the import entirely - it was only used to get the probabilities of a Bernoulli distribution given input logits, but this is just the sigmoid function, so I replaced the whole thing with a single call to `sigmoid()` instead.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29641). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix batching tests for new models (Mamba and SegGPT)
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

PR #29297 was merged without rebasing `main`, so some new models were failing the test. This PR fixes Mamba and SegGPT models.

- Mamba was storing its cache in an object, now we skip it
- SegGPT has weird hack of merging half batches, so we have a custom check for the tester now


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@gante @amyeroberts @ydshieh 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29633). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix `multi_gpu_data_parallel_forward` for `MusicgenTest`
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix 

```
tests/models/musicgen/test_modeling_musicgen.py::MusicgenDecoderTest::test_multi_gpu_data_parallel_forward

tests/models/musicgen/test_modeling_musicgen.py::MusicgenTest::test_multi_gpu_data_parallel_forward
```
introduced in #29297

we need batch size to be even number

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29632). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Support PEFT models in pipelines
Author: Rocketknight1
Pull request: yes

Our pipelines check for supported classes but aren't PEFT-aware, so they get confused and throw errors for PEFT models. This PR changes the supported class check to use the actual underlying model of a PEFT model.

Fixes #29395

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29517). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @B-Step62, this branch should fix the issue now! Try `pip install --upgrade git+https://github.com/huggingface/transformers.git@fix_peft_model_in_pipelines` and let me know if you have any issues. The example you sent me seems to be working now.

If you don't find any further problems, I'll ping for core maintainer review and merge this soon.

Thanks for the quick fix, @Rocketknight1! I have tested my examples and confirmed the error message no longer is printed:)

cc @amyeroberts for core maintainer review in that case

Also note creating a brand new PEFT model with `get_peft_model` will lead to having a model with random adapter weights attached on it, therefore you'll mostl likely get gibberish output, I think that's also a bad intent as well :/ 
The typical workflow should be
1- train your adapters with PEFT
2- push them / save them somewhere after training
3- load them back with `AutoModelxxx` interface
4- do whatever 
What do you think?


My bad! Your solution sounds good - I guess we can close this PR and just post something in the original issue at #29395

ok ok sounds good! sorry for being responsive late 😭 ok yes the proposed plan sounds good! 

Thank you for working on this issue @younesbelkada @Rocketknight1 !

> 2- push them / save them somewhere after training
> 3- load them back with AutoModelxxx interface

Unfortunately, this solution doesn't work as it is in our case. We want to save/load the adapter (+ the base model and tokenizer) locally as pipeline, without needing to interaction with the hub, to avoid the flakiness of downloading heavy files over network. Let me ask a few follow-up questions with this regards:

* Is there a way to casting the PEFT model into AutoModelxxx without pushing to the hub?
* Also vice-versa, once loading the model as AutoModelxxx, can we distinguish it is a PEFT model? It seems `is_peft_model` uses `isinstance` so my guess is it doesn't work.



Thanks @B-Step62 for the clarification ! 
1- Yes you can also save it locally 
2- When you load a peft model as `AutoModelxxx` you can distinguish if it's a peft model by looking into the attribute `_hf_peft_config_loaded`: `getattr(model, "_hf_peft_config_loaded", False)`
Experimental loading of MLX files
Author: pcuenca
Pull request: yes

# What does this PR do?

Meant as a working draft to support the discussion in #29333.
Fixes #29333 

This code works for a simple roundtrip conversion in `float16`:
```
python -m mlx_lm.convert --hf-path TinyLlama/TinyLlama-1.1B-Chat-v1.0 --dtype float16 --mlx-path tiny-llama-chat
```

However:
- It requires `mlx` to be installed because it's detected as a distinct framework. We should possibly override with `pt` in this case, as it doesn't make sense to demand `mlx` as a dependency. This may or may not clash with recent work done in #29406.
- Many model weights in MLX adopt the same naming conventions as `transformers`, this was a decision made in the `mlx-examples` project to increase interoperability. However, this is not guaranteed. We should probably error if we find incompatible weights.
- I don't know how (or if) this works with quantized weights. Many MLX repos are about creating 4-bit quantized versions that run efficiently on Apple silicon.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@alexweberk, @ArthurZucker

cc @Vaibhavs10 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29511). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This is now ready for review. It essentially fixes `from_pretrained` to allow loading models saved with `mlx`, and adds a simple test. Quantized models can't be loaded, we can explore it further in another PR.

Gentle ping @amyeroberts or @LysandreJik for merge or additional feedback.
Why is the default `use_reentrant=True` when no kwargs were set?
Author: simon-lund
Pull request: no

https://github.com/huggingface/transformers/blob/350c5d15668aa2f0467699baa795ef751b1167d3/src/transformers/modeling_utils.py#L2110-L2111

Here `use_reentrant=True`  is set when no kwargs are provided. From the docs of [pytorch](https://github.com/pytorch/pytorch/blob/92a2b214f8e9217c5d94c7583817b22f822fa925/torch/utils/checkpoint.py#L343) this seems to be the legacy variant. 

Does this have any performance or other advantages that I am not aware of?

Comments:
Hi @simon-lund, thanks for opening an issue! 

You can check the git blame to find the PRs which add certain lines - which should normally give you the reasons for the code logic. 

In this case, this line was added in #28538 - according to the PR it's for preparation of an upcoming torch release, where it's necessary for us to explicitly pass the `use_reentrant` kwarg. 

The value is set to true based on the current default PT behaviour c.f. [this comment](https://github.com/huggingface/transformers/pull/28538#issuecomment-1902624777)

Ah. I'm sorry, I only searched the issues and found several discussion about this, but I didn't search through the PRs
Thank you very much for the link as well as the hint with the PRs (I was not aware of that) :+1:
Add AMD nightly CI
Author: fxmarty
Pull request: yes

As per title.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26932). All of your documentation changes will be reflected on that endpoint.

Thank you @fxmarty .

But I think the priority is to have a daily (complete) CI run for `latest torch version`. It's like `.github/workflows/self-scheduled.yml` (maybe ignore the deepspeed part etc.).

We have push event for AMD CI, but it is really difficult to follow (as there are always some failures, even for nvidia!). Even for Nvidia CI, others and me almost only look the result of the daily (complete) run.

Of course, if you really want to have this nightly CI (i.e. with nightly torch version) before the `daily CI with stable torch`, I am happy to review and merge once it is all good.

Sounds good @ydshieh, indeed it makes sense to have first a scheduled CI for tests on PyTorch stable, and merge this one later on. It should be pretty similar, I will ping you there.

Thanks @ydshieh, will give the priority as you mentioned:
- daily
- then, nightly

DeepSpeed should works OOB, so maybe let's try to include it right away @fxmarty?

scheduled CI involves more stuff: 4 different docker images, example test job, (torch/tf)pipeline test jobs etc.

we will also have a new slack channel for the report (I can take care of this) + a way to display the diff between the latest run with the previous run (all these could be treated toward the end)

@mfuntowicz DeepSpeed on RoCm currently does not work with PyTorch 2.1. I'll set aside the DeepSpeed tests for now until https://github.com/microsoft/DeepSpeed/pull/4538 is included in a deepspeed release (or patch)
Add support for FSDP+QLoRA and DeepSpeed ZeRO3+QLoRA
Author: pacman100
Pull request: yes

# What does this PR do?
1. `bitsandbytes` has now added `quant_storage` option for 4-bit parameters which is required for FSDP+QLoRA support. This PR adds that option and fixes the corresponding parameter calculation.
2. Enables loading the model layer by layer, quantising it on GPU and then transferring it to CPU so that FSDP or DeepSpeed can later shard it and put it on the GPUs. This reduces the GPU memory required. For example, a 70B model when loaded in 4-bit on each GPU without sharding would require 35Gb per GPU. However, if we load the model on CPU and shard it on 2 GPUs, then the same quantized model would require 35/2=17.5 GB per GPU which now can fit on 24GB GPUs.
3. Dispatch needs to be disabled else it will try to put the quantized weights which are on CPU to GPu before sharding.
4. Disable `zero.init` when using DeepSpeed with QLoRA. 
5. When using FSDP with PEFT LoRA, the auto wrap policy needs to be updated to additionally separately wrap LoRA trainable layers. When using FSDP with QLoRA, the mixed precision policy needs to be updated to use the quantization storage data type.

This PR should be merged after Accelerate PR https://github.com/huggingface/accelerate/pull/2544.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29587). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@younesbelkada, addressed the  comments in the latest commit

Really amazing work @pacman100 Thanks so much for this! ❤️
Referencing #29470 Unable to Create Bert Encodings
Author: melissafeeney
Pull request: no

### System Info

Could you please reopen #29470 ? I left a new comment but seem unable to reopen the issue :( 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

#29470 


### Expected behavior

#29470 

Comments:
@melissafeeney I've reopened the issue.

Next time, can you please comment on the issue directly requesting it to be reopened, rather than opening a new issue? Having many different issues opened for the same thing make it hard to track and isn't sustainable if everyone does it.
Fix TPU checkpointing inside Trainer
Author: shub-kris
Pull request: yes

# What does this PR do?

This PR fixes the following issue:

Whenever `save_steps` is a multiple of `logging_steps` this [step](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L2406) ` if self.control.should_log and self.state.global_step > self._globalstep_last_logged` is invoked which calls the `xm.mark_step` and the checkpointing defined inside `save_tpu` [here](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L3009) works fine. 
 
```python
 def _maybe_log_save_evaluate(self, tr_loss, grad_norm, model, trial, epoch, ignore_keys_for_eval):
        if self.control.should_log and self.state.global_step > self._globalstep_last_logged:
            if is_torch_xla_available():
                xm.mark_step()
```

but the execution halts when `save_steps` is not a multiple of `logging_steps` as the `xm.mark_step` is never called. 
   

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@muellerzr @pacman100 

Comments:
The issue was also discussed [here](https://huggingface.slack.com/archives/C038TF8B7UZ/p1710339607520609)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29657). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add `dataset_revision` argument to `RagConfig`
Author: ydshieh
Pull request: yes

# What does this PR do?

The recent changes (yesterday) on the dataset [wiki_dpr](https://huggingface.co/datasets/wiki_dpr/commits/main) cause failing tests.

This PR just uses a revision to avoid the failures. 

Checked on runner and everything is back to the previous status.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29610). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix PVT v2 tests
Author: ydshieh
Pull request: yes

# What does this PR do?

Same #29362 but for the newly added model pvt v2

failing test is

```
tests/models/pvt_v2/test_modeling_pvt_v2.py::PvtV2ModelTest::test_training

(line 1448)  RuntimeError: Failed to import transformers.models.dinat.modeling_dinat because of the following error (look up to see its traceback):
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29660). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.