noamaanMulla-03/transformers-issues huggingface dataset:
launch deepspeed in mixed precision fp8 using HF Trainer is not working
Author: eljandoubi
Pull request: no
Labels: name: trainer

name: DeepSpeed

name: bug

### System Info

![Capture d'écran 2024-10-08 114022](https://github.com/user-attachments/assets/d953186a-0c9c-4224-8cce-def576aec8f1)
acc_cfg.yml:

compute_environment: LOCAL_MACHINE                                                                                                                                           
debug: false
deepspeed_config:
  deepspeed_multinode_launcher: standard
  gradient_accumulation_steps: auto
  gradient_clipping: 1.0
  offload_optimizer_device: cpu
  offload_param_device: cpu
  zero3_init_flag: true
  zero3_save_16bit_model: true
  zero_stage: 3
distributed_type: DEEPSPEED
downcast_bf16: 'no'
machine_rank: 0
main_process_ip: 0.0.0.0
main_process_port: 0
main_training_function: main
mixed_precision: fp8
fp8_config:
  amax_compute_algorithm: max
  amax_history_length: 1024
  backend: TE
  fp8_format: HYBRID
  interval: 1
  margin: 0
  override_linear_precision: false
  use_autocast_during_eval: true
num_machines: 3
num_processes: 24
rdzv_backend: etcd-v2
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

accelerate launch --config_file acc_cfg.yml train.py $TRAINING_ARGS
the train.py is any training script that train using transformers.Trainer
$TRAINING_ARGS are the TrainingArguments plus some path to data
![image](https://github.com/user-attachments/assets/05c8e0d7-ebfe-402e-ac63-e198f0af6e54)


### Expected behavior

Deepseed do not capture that the mixed precision is fp8 and it switches to bf16.

Comments:
Maybe cc @SunMarc @muellerzr!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
ValueError: decoder_start_token_id or bos_token_id has to be defined for encoder-decoder generation
Author: rabeehk
Pull request: no

Hi
I followed the tutorial for generation with T5, and I made my input as  "Question: Question sequence ", and target as "Target sequence", then I use tokenizer.batch_encode_plus to encode the sentence and then I call generate as:

model.generate(batch[input_ids], attention_mask=..., max_length=..., early_stopping=True)

I got the following error:
 
  File "/opt/conda/envs/pl/lib/python3.7/site-packages/transformers/generation_utils.py", line 398, in generate
    "decoder_start_token_id or bos_token_id has to be defined for encoder-decoder generation"
ValueError: decoder_start_token_id or bos_token_id has to be defined for encoder-decoder generation


but the input format seems to be matching the tutorial. thanks for your help.

Comments:
Hi,
could you assist in adding an example in tutorial for this, I followed the tutorial sample for summarization, Is the way I made my inputs  ("Question: Question sequence ") correct? thanks 
here is the tutorial sample: 
```
>>> input_ids = tokenizer("summarize: studies have shown that owning a dog is good for you ", return_tensors="pt").input_ids  # Batch size 1
>>> outputs = model.generate(input_ids)
```

solved when I load the config from a pretrained model, maybe this helps to add the extra info needed as the default setting :)
Fix new FA2 if `is_causal` is passed explicitly
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

In GPT2 we have to pass `is_causal` explicitly for SDPA, but it causes double occurence for FA2 as highlighted in https://github.com/huggingface/transformers/issues/35380. This fixes it.
Also reverts the `reshape` to simple `view` for simplicity (even if they are technically equivalent, as reshape calls view if possible)




Comments:
cc @ArthurZucker, this should be included in the coming release!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35390). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@Cyrilvallez not sure if `view` is gonna cause issues as non-contiguous tensors will result in an error (which really is the only reason to use reshape instead), i.e. calling contiguous before is safer imo.

We were using 'view' without 'contiguous' before the refactors, so this should be fine!

Sounds good! Was just overly cautious then :)

No worries, thanks for double-checking!
Incorrect file structure in convert_mask2former_original_pytorch_checkpoint_to_pytorch.py?
Author: yujunwei04
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.34.0
- Platform: Linux-6.8.0-31-generic-x86_64-with-glibc2.17
- Python version: 3.8.20
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.5
- Accelerate version: 0.23.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@amyeroberts @qubvel 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I have problem when running convert_mask2former_original_pytorch_checkpoint_to_pytorch.py
1. I have the path: `~/my_project/checkpoints/coco/instance-segmentation/my_model.pkl`
2. I have the path: `~/my_project/configs/coco/instance-segmentation/my_configs.yaml`
3. I run the following in terminal: `python convert_mask2former_original_pytorch_checkpoint_to_pytorch.py     --checkpoints_dir ~/my_project/checkpoints     --configs_dir ~/my_project/configs     --mask2former_dir ~/work/Mask2Former`
4. The error happens: `coco must be wrong since acceptable values are: instance-segmentation, panoptic-segmentation, semantic-segmentation.`

### Expected behavior

In line 907, I print the input path: `~/my_project/checkpoints/coco/instance-segmentation/my_model.pkl`
In line 909, my `model_name_raw` is `instance-segmentation`, while the comment in line 908 says `# model_name_raw is something like maskformer2_swin_small_bs16_50ep.` The model_name_raw should be `my_model` from my understanding

Comments:
Hi @yjwnb6 - I don't understand how that error is happening with those paths. Can you print the value of `segmentation_task_name` on line 912?

Hi @Rocketknight1, thanks for your reply! The value of segmentation_task_name on line 912 is `coco`.
Pass `hidden_states` to `MllamaVisionEncoderLayer` w.o. keyword
Author: oraluben
Pull request: yes

# What does this PR do?

ZeRO3 add hooks without kwargs support, so this PR fixes an issue that cannot set leaf to `MllamaVisionEncoderLayer` when using ZeRO3.


Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@muellerzr

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Translation model M2M100 uses 2 models in cache (from version 4.46.0)
Author: harelfar2
Pull request: no
Labels: name: bug

name: Safetensors

### System Info

I'm using `facebook/m2m100_418M` translation model.
From version 4.46.0 it downloads another model which wieghts ~2 GB.
I'm using python 3.11, in `ubuntu`

### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
import torch
from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer


model = M2M100ForConditionalGeneration.from_pretrained(
    "facebook/m2m100_418M",
    torch_dtype=torch.float16,
).to("cpu").eval()

token = M2M100Tokenizer.from_pretrained("facebook/m2m100_418M")

encoded_text = token("my name is earl", return_tensors="pt")
encoded_text = encoded_text.to("cpu")

target_lang_id = token.get_lang_id("he")
generated_tokens = model.generate(**encoded_text, forced_bos_token_id=target_lang_id)
print(generated_tokens)
```

### Expected behavior

The models are being put in `/home/ubuntu/.cache/huggingface/hub/models--facebook--m2m100_418M/`
until version 4.46.0 there was this hierarchy: 
`snapshots/55c2e61bbf05dfb8d7abccdc3fae6fc8512fd636` which contained 7 files (one of them is the model itself pytorch_model.bin - ~2 GB).
From version 4.46.0, there is a new dir: `snapshots/791dc1c6d300846c9a747d4bd11fcc7f369b750e`, there is one file in there: `model.safetensors`, which is a soft link to another heavy ~2GB file in blobs dir.

Can you please resolve it and make it download and use only one model file? this usage is very wasteful. 

Thanks!

Comments:
Hey @harelfar2, we're slowly moving towards deprecating support of `pytorch_model.bin` files as several security issues have been found with these files.

As such, when there is a `model.safetensors` file available as an equivalent file, we now default to downloading and using this file. As a result, you should be able to safely remove the `pytorch_model.bin` file in your cache and the model should still work as expected.

Please let me know if you run into any issues

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
uniformize kwargs for SAM
Author: tibor-reiss
Pull request: yes
Labels: name: Vision

name: Multimodal

name: Processing

Adds uniformized processors for SAM following https://github.com/huggingface/transformers/issues/31911.

@qubvel @molbap

Comments:
cc @yonigozlan can you take a look! 

@yonigozlan Thanks for the review, I have implemented your suggestions. Could you check again please?

Friendly reminder @yonigozlan 

Pinging @ArthurZucker for a core maintainer review

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34578). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Deprecate _is_quantized_training_enabled
Author: MekkCyber
Pull request: yes

# What does this PR do?
Deprecate `_is_quantized_training_enabled` in `modeling_utils.py` as it's no longer used, and it's replaced by `hf_quantizer.is_trainable`

## Who can review ?
@SunMarc 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34991). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix typing in docstring for `PaliGemmaProcessor`
Author: alvarobartt
Pull request: yes

# What does this PR do?

This PR updates the typing for `tokenizer` in the `PaliGemmaProcessor` to be `GemmaTokenizerFast` instead of `LlamaTokenizerFast`.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Not sure who to tag, maybe @molbap as he contributed to the model AFAIK, feel free to ping anyone else if needed (sorry if that's the case)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35278). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix : VPTQ test
Author: MekkCyber
Pull request: yes

# What does this PR do?
Fix VPTQ test 

## Who can review ? 
@SunMarc 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35394). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix case of nested tensors in BatchMixFeature
Author: Rocketknight1
Pull request: yes
Labels: name: run-slow

Huge kudos to @zucchini-nlp for figuring this one out - we were getting intermittent failures and I couldn't figure out why. The reason was that `BatchMixFeature.to()` was breaking the structure of some inputs, because it used a nested list comprehension that assumed a specific list depth that wasn't always respected by `PixtralProcessor`.

This PR replaces the slightly hacky `.to()` code with a recursion that should preserve nesting structure regardless of list depth.

TODO:
- [ ] Compare against the VLLM implementation of Pixtral to see how they handle this (if their code supports batched?)

Comments:
Hey @Rocketknight1! There is a PR that might be relevant in terms of how args/kwargs are propagated

 - https://github.com/huggingface/transformers/pull/34883
 
 P.S. didn't notice it's a model-related change only, so might be not relevant

@qubvel thanks for letting me know regardless, since we might make the same change in `BatchFeature`! The two PRs shouldn't conflict much, though - if you merge yours first, I can just rebase and incorporate those changes afterwards.

Note - this PR should not be merged yet, as I believe the issue in `.to()` was actually masking the fact that Processor outputs don't match the expectations of our Pixtral modelling code!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35063). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @arthurzucker @lysandrejik for core maintainer review, this one should be ready!

Closing because this change is included in #34801 instead
Fix the structure of images in PixtralProcessor
Author: Rocketknight1
Pull request: yes
Labels: name: run-slow

The `PixtralProcessor` has some issues regarding correct nesting depth of inputs. If we are fully explicit about input nesting, then `text` should be `List[str]` and `images` should be `List[List[Image]]`. This is because each sample only has one text, but can have multiple images.

The start of the Pixtral processor code handles cases when users don't supply fully nested inputs. However, it uses the heuristic that if the user passes `List[str]` for `text` and `List[Image]` for `images`, then this indicates **one image per sample**. This is [very confusing](https://github.com/huggingface/transformers/pull/34332#issuecomment-2519369424) for users, especially because it means output changes when `batch_size==1`, depending on whether that single input is passed as `str` or `[str]`.

With this PR, we avoid that assumption. If the user supplies a single image or flat list of images, then we assign them all to the first sample in the event that `batch_size==1`. If `batch_size>1` then we throw an error, asking them to supply an explicit list of lists instead. This seems much safer!

Comments:
cc @yurkoff-mv, can you try out this PR and confirm it resolves the issues you were having? You can use it with `pip install git+https://github.com/huggingface/transformers.git@pixtral_processor_structure_fix`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35107). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

also cc @zucchini-nlp and @arthurzucker / @LysandreJik for core maintainer review

@Rocketknight1, thank you for responding so quickly. Yes, these fixes work.

Hi @zucchini-nlp, how do the inputs differ here compared to most other processors? I thought `text = List[str]` and `images = List[List[Image]]` was pretty normal!

@Rocketknight1 usually we support both formats, as `List[List[image]]` and simply `List[image]`, except for some cases like in Idefics where the cross attention module needs to know how many images each prompt should attend to. Maybe we can also ask @yonigozlan, he's working on standardizing processor input/output format for VLMs

Closing because this has been included in #34801 instead
bugfix Idefics3 processor - handle gracefully cases with text and no images
Author: mfarre
Pull request: yes

# What does this PR do?

Fixing Idefics3 processor to work with batches that do not include images


## Who can review?
@andimarafioti 



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35363). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Let's wait for @yonigozlan to be sure!

thanks @andimarafioti 
thanks @yonigozlan if you give me your LGTM I will merge the changes: I followed your suggestions adding some tests and adding your code proposal.

> Yes that looks much better! Just suggested a check to avoid hallucinations like this:
> 
> ```python
> messages = [
>     {
>         "role": "user",
>         "content": [
>             {"type": "image"},
>             {"type": "text", "text": "What do we see in this image?"},
>         ]
>     }
> ]
> prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
> inputs = processor(text=prompt, return_tensors="pt")
> inputs = {k: v.to(DEVICE) for k, v in inputs.items()}
> 
> 
> # Generate
> generated_ids = model.generate(**inputs, max_new_tokens=50)
> generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)
> 
> print(generated_texts)
> 
> ['User:What do we see in this image?\nAssistant: The image depicts a scene from a historical or fictional setting, likely from the medieval period, given the attire and architecture. The central focus is on a large, ornate gate, which appears to be the entrance to a castle or a fortified structure.']
> ```
> 
> also in `Idefics3ProcessorTest` could we add a test for text only inference and one to check that an error is raised if we have an image in the conversation, but no images are passed to the processor?



Looks great thanks @mfarre for fixing this! LGTM for me, but let's maybe wait for @ArthurZucker 's review before merging :)
Adding logger.info about update_torch_dtype in some quantizers
Author: MekkCyber
Pull request: yes

# What does this PR do?
Adding missing `logger.info` statements in some quantizers to inform users which dtype is set during quantization.

## Who can review ?
@SunMarc 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35046). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
bitsandbytes: simplify 8bit dequantization
Author: matthewdouglas
Pull request: yes
Labels: name: Quantization

# What does this PR do?

Simplifies the dequantization for bitsandbytes int8 weights. There is a similar PR open in [huggingface/peft#2245](https://github.com/huggingface/peft/pull/2245/files).

In the upcoming bitsandbytes release we will have an added API for for int8 dequantization. For backwards compatibility, a simplified (but functionally equivalent) dequantization operation is performed.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@SunMarc @BenjaminBossan 




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35068). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

LGTM, thanks. The failing test `tests/models/xlm_roberta_xl/test_modeling_xlm_roberta_xl.py::XLMRobertaXLModelTest::test_assisted_decoding_matches_greedy_search_1_same` appears to be unrelated.