noamaanMulla-03/transformers-issues huggingface dataset:
don't zero out the attention_mask when using sliding window with flash attention
Author: winglian
Pull request: yes

# What does this PR do?

Flash attention has it's own sliding window argument,  but currently the attention_mask is clobbered with sliding window leading to incorrect/large loss values (and also breaks sample packing in axolotl):

Here's the existing loss with flash attention and no sample packing (LoRA):

<img width="1163" alt="Screenshot 2024-06-27 at 12 17 12 PM" src="https://github.com/huggingface/transformers/assets/381258/8c3cc076-d47a-4001-b68c-c09bbaf0fc61">

Here's the loss with the fix (no sample packing):

<img width="1170" alt="Screenshot 2024-06-27 at 12 20 09 PM" src="https://github.com/huggingface/transformers/assets/381258/6f35f6b1-012b-4b55-8540-7fd47b32944c">


Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@muellerzr @SunMarc  @ArthurZucker


Comments:
Thanks 🤗 

Checkout https://pypi.org/project/transformers/4.42.2/ which includes this commit!
Libraries import missing, unable to load image for inference and not able to load pipeline with the trained model
Author: dejokz
Pull request: no

### System Info

Running in google colab 
- `transformers` version: 4.40.2
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.23.0
- Safetensors version: 0.4.3
- Accelerate version: 0.30.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.2.1+cu121 (True)
- Tensorflow version (GPU?): 2.15.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.8.3 (gpu)
- Jax version: 0.4.26
- JaxLib version: 0.4.26
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: not sure

### Who can help?

@stevhliu 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

copying the latest github notebook and running in google colab

### Expected behavior

To run without any import errors, and to train without any errors, and load my trained model without any errors

Comments:
https://colab.research.google.com/github/huggingface/notebooks/blob/main/transformers_doc/en/pytorch/semantic_segmentation.ipynb

Hi @dejokz, thanks for raising an issue! 

Could you give some more details about the errors you're experiencing? Specifically the error messages and their full traceback? 

Hi @dejokz, I checked the provided colab notebook and I got the same error and I think that even after running the suggested install of `pip install transformers[torch]` and `pip install accelerate -U` in the error traceback you aren't able to resolve the issue, right?

So, for fixing that just restart the colab session and after that without rerunning the `!pip install` code cells just  run all the rest of the other code cells back 'cause Colab has a script that appends metadata of pip-installed packages to a list-like object during runtime and imports only the search function from the top of this list so, we need to restart the session after `!pip install`

Cheers!

Hi, I recommend copying/pasting the code from the docs on the `main` version for now. I think these issues should've been resolved in #30132, but the doc-builder is not converting the latest changes to the notebooks for some reason.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[HybridCache] Fix `get_seq_length` method
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Fixes #31664

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31661). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Failures seem unrelated; I'll merge it and do a patch release

Patch is up on pypi! Thanks both for the quick PR and review :)

Thanks @LysandreJik for the speedy patch!
return_dict in encodec is always set to True: 
Author: kamilakesbi
Pull request: no
Labels: name: Audio

### System Info


- `transformers` version: 4.42.0.dev0
- Platform: Linux-5.4.0-166-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.23.3
- Safetensors version: 0.4.2
- Accelerate version: 0.29.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.2+cu121 (True)
- Tensorflow version (GPU?): 2.13.1 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.0 (cpu)
- Jax version: 0.4.13
- JaxLib version: 0.4.13
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA DGX Display


### Reproduction

```
from transformers import EncodecConfig, EncodecModel, EncodecFeatureExtractor
import numpy as np 
import torch

signal = [np.random.randn(1065473), np.random.randn(1065473) ]
signal = EncodecFeatureExtractor()(signal)

model = EncodecModel(EncodecConfig())

result = model(torch.tensor(signal.input_values), return_dict = True)
print(type(result))
result2 = model(torch.tensor(signal.input_values), return_dict = False)
print(type(result2))
```

will print `<class 'transformers.models.encodec.modeling_encodec.EncodecOutput'>` even when `return_dict` is set to False. 


### Expected behavior

It should return a tuple when `return_dict` is set to False. 

cc @sanchit-gandhi 

Comments:
https://github.com/huggingface/transformers/blob/dfaadfdcda8d2c2f564c94121d4618309c1ecdd5/src/transformers/models/encodec/modeling_encodec.py#L789

@kamilakesbi 
by default self.config.return_dict  is true so the or condition is always maintained and the function returns a dict.

Hi @AbhinavJangra29, 

Thanks for spotting the issue. @jla524 has opened a PR to fix it!
Remove deprecated config attribute in VLMs
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

This PR removes the deprecated 'vocab_size' in llava and vipllava. One moment to note is that I didn't notice Paligemma added 'vocab_size' to its config, not `text_config`. Since it did not have a deprecation cycle, I am not sure if we can remove it now, so leaving as-is. cc @molbap 

The resize_embedding tests are passing for both models, because we've fixed that in prev PRs to work with "text_config"

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31655). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@zucchini-nlp 

After this update, I am facing an error while making an instance of LlavaProcessor. This is the code I wrote: 
`processor = AutoProcessor.from_pretrained("llava-hf/llava-1.5-7b-hf")`

It gives the following error: 
`TypeError: LlavaProcessor.__init__() got an unexpected keyword argument 'chat_template'`

It was working fine before the update. I have the following versions of transformers and bitsandbytes installed:
`!pip install -q -U transformers==4.37.2`
`!pip install -q bitsandbytes==0.41.3 accelerate==0.25.0`

Is there any issue with the current update or is there any other solution for this problem? 

@prakash-shubham-2003 Which version you're using. Chat templates were introduced in a different PR and should be working. Can you verify that you have at least v4.42?

> @prakash-shubham-2003 Which version you're using. Chat templates were introduced in a different PR and should be working. Can you verify that you have at least v4.42?

Are you asking about the version of the transformer library that I am installing?

You can check this notebook to check further: [https://www.kaggle.com/code/scyther07/notebook49a1fde1f1/edit](url)

@prakash-shubham-2003 Try loading again, now it shouldn't throw an error. I reverted the changed and will apply back after finding a way to make it less breaking.

Yes, the chat feature from transformers version v4.42.

> @prakash-shubham-2003 Try loading again, now it shouldn't throw an error. I reverted the changed and will apply back after finding a way to make it less breaking.
> 
> Yes, the chat feature from transformers version v4.42.

It's working fine again. Thanks.



`!pip install -q -U transformers==4.37.2`

I am installing v4.37.2. (I was following an official demo. this one : [https://colab.research.google.com/drive/1qsl6cd2c8gGtEW1xV5io7S8NHh-Cp1TV?usp=sharing#scrollTo=7RA7j86HhkP5](url))
Build docs for `v4.42.0`
Author: mishig25
Pull request: yes

Build docs for `v4.42.0`

> !WARNING
> do not merge, only for doc-building purposes

Comments:
@mishig25 Do we know why this didn't trigger for the recent release? Is there anything we need to fix in our release scripts / preparation? 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31693). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<img width="490" alt="image" src="https://github.com/huggingface/transformers/assets/11827707/4d5239b4-73c7-40d5-9080-0bf06de9ebba">

done. 

> @mishig25 Do we know why this didn't trigger for the recent release? Is there anything we need to fix in our release scripts preparation?

@amyeroberts yes because the action failed https://github.com/huggingface/transformers/actions/runs/9699541121

Thanks for fixing!
Added HelpingAIForCausalLM
Author: KingNish24
Pull request: yes

I added that HelpingAI Model into Transformers.

For more information about Model #31103,

Thanks!

Comments:
cc @ArthurZucker 

@Divy-Don Please refrain from spamming PRs with comments like this: repetitive and clearly generated from an LLM (non specific, written as a list). This isn't sustainable and makes code reviews harder for everyone. I'm hiding the comments to reduce noise.  

> Hey! Nice work, would be nice if you can isolate the changes and use #Copied from statements, or use the `diff_gemma` as a template to automatically generate the modeling code using #30868!

can u explain?

Sur, `#Copied from` is our core philosophy: re-using code that already exists for an easier maintenant. The PR for the diff explains how to use them otherwise!
Add French version of run scripts tutorial
Author: jadechoghari
Pull request: yes

## What does this PR do?

This pull request adds a French translation of the run_scripts from English to French.

## Changes made

- Translated `run_scripts.md` from English to French.
- Updated the `_toctree.yml` to include the French version of the run_scripts tutorial.

## Why is this change important?

Providing documentation in multiple languages helps make the information accessible to a broader audience and supports users who are more comfortable reading in French. 🇫🇷
More docs to come. 🤗

## Review

Like last time I will tag:
- @amyeroberts (for the merge ✅)
- @younesbelkada (for the review 🇫🇷🎊)


Comments:
@NielsRogge, could you kindly confirm the French part? 
We just need one review from a maintainer with write access to close this small PR, been 5 days. 
🤗 Thank you! 🇫🇷



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31483). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
adalomo is not a valid OptimizerNames
Author: luoruijie
Pull request: no
Labels: name: optimization

### System Info

torch==2.3.0+cu121
transformers==4.41.2
trl==0.9.4

### Who can help?

@ArthurZucker
@muellerzr 
@SunMarc

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

import torch
import datasets
from transformers import TrainingArguments, AutoTokenizer, AutoModelForCausalLM
import trl

train_dataset = datasets.load_dataset('imdb', split='train')

args = TrainingArguments(
    output_dir="./test-lomo",
    max_steps=1000,
    per_device_train_batch_size=2,
    optim="adalomo",
    gradient_checkpointing=False,
    logging_strategy="steps",
    logging_steps=1,
    learning_rate=5e-4,
    save_strategy="no",
    run_name="lomo-imdb",
)

model_id = "mistralai/Mistral-7B-v0.1"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, low_cpu_mem_usage=True).to(0)

trainer = trl.SFTTrainer(
    model=model, 
    args=args,
    train_dataset=train_dataset,
    dataset_text_field='text',
    max_seq_length=512,
)

trainer.train()


## and after run this code ,i meet this error .the error belows :


Traceback (most recent call last):
  File "/root/full_parameter_ft_use_lomo.py", line 9, in <module>
    args = TrainingArguments(
           ^^^^^^^^^^^^^^^^^^
  File "<string>", line 128, in __init__
  File "/root/miniconda3/lib/python3.12/site-packages/transformers/training_args.py", line 1623, in __post_init__
    self.optim = OptimizerNames(self.optim)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/enum.py", line 757, in __call__
    return cls.__new__(cls, value)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/enum.py", line 1179, in __new__
    raise exc
  File "/root/miniconda3/lib/python3.12/enum.py", line 1156, in __new__
    result = cls._missing_(value)
             ^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/transformers/utils/generic.py", line 495, in _missing_
    raise ValueError(
ValueError: adalomo is not a valid OptimizerNames, please select one of ['adamw_hf', 'adamw_torch', 'adamw_torch_fused', 'adamw_torch_xla', 'adamw_torch_npu_fused', 'adamw_apex_fused', 'adafactor', 'adamw_anyprecision', 'sgd', 'adagrad', 'adamw_bnb_8bit', 'adamw_8bit', 'lion_8bit', 'lion_32bit', 'paged_adamw_32bit', 'paged_adamw_8bit', 'paged_lion_32bit', 'paged_lion_8bit', 'rmsprop', 'rmsprop_bnb', 'rmsprop_bnb_8bit', 'rmsprop_bnb_32bit', 'galore_adamw', 'galore_adamw_8bit', 'galore_adafactor', 'galore_adamw_layerwise', 'galore_adamw_8bit_layerwise', 'galore_adafactor_layerwise']

### Expected behavior

i think this code would be run normaly ,thank you .

Comments:
Hi @luoruijie, this is normal since `adalomo` is not on the latest version of transformers. You need use the main branch of transformers or wait until the new release ! 

thank you very much .
Add `bot_token` attribute to `PreTrainedTokenizer` and `PreTrainedTokenizerFast`
Author: aw632
Pull request: no
Labels: name: Feature request

### Feature request

I'm requesting for the attribute `bot_token` (beginning-of-tools token) to be added to the `PreTrainedTokenizer` classes, similar to `eos_token`. This token would be associated with `self.bot_token` and `self.bot_token_id` and would expose the token to downstream consumers like vLLM.

### Motivation

This request builds off [this PR comment](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/discussions/35) as well as the ongoing work to support function calling [in transformers](https://github.com/huggingface/transformers/pull/30621). 

A number of downstream consumers depend on what's available in the `PreTrainedTokenizer` classes, like vLLM's `Sequence` class and `LLMEngine` class [example](https://github.com/vllm-project/vllm/blob/main/vllm/engine/llm_engine.py#L462). For example, the current problem I'm facing is that vLLM doesn't correctly label the finish reason for "tool call" outputs, as, well, tool calls, since the `CompletionOutput.finish_reason` ultimately relies on the attributes available in `PreTrainedTokenizer`. 

As open-source tool calling proliferates, having these attributes exposed would greatly enhance the utility of the library. This token can be set to `None` by default and should be backwards compatible with the right implementation. 

### Your contribution

I can help contribute to the PR and write code. Might need help navigating the library + writing good test cases. 

Comments:
cc @Rocketknight1
A parameter in TrainingArguments: sample_output=True
Author: Tylersuard
Pull request: no
Labels: name: Feature request

### Feature request

It would be nice to have the model output some sample generations while it is training, so that I can see if my training is going well, and at what point I should stop it.

### Motivation

For example let's say I set my training for 5 epochs.  I see the samples from the first epoch are bad, the second look a little better, and the third are exactly what I want.  I could stop the training right there so I don't have to overfit.

### Your contribution

I'm not sure if I can contribute.

Comments:
https://huggingface.co/docs/evaluate/main/en/transformers_integrations#trainer
using evaluation_strategy and compute_metrics may would help

+1, I would be pro of making a similar thing work for decoder-only models, which I believe isn't possible currently by simply passing `compute_metrics` and `predict_with_generate` arguments

Related issue was discussed [here](https://github.com/huggingface/trl/issues/862)

+1. For this feature I think it will be better to show both metrics and sample outputs, which is more clear

Hey @Tylersuard, sorry for the very slow reply here, we're catching up on the backlog.

Rather than adding a dedicated `sample_output` flag to `TrainingArguments`, this is best done with a small custom `TrainerCallback`: every callback event is handed the live `model` and `processing_class`/tokenizer, see https://github.com/huggingface/transformers/blob/150eb7c9ed4091294c829fa0e9466b090cb0f87f/src/transformers/trainer_callback.py#L543-L556
 so you can run `model.generate(...)` on a few fixed prompts inside `on_evaluate` and print/log the decoded text. No core change needed, and it works for decoder-only models too.
We're keeping the base `Trainer` model-agnostic and lean, so a generation-only argument isn't something we want to bake into the core.

If you'd like it ready-made, TRL ships a `LogCompletionsCallback` that does exactly this (logs sample generations to the console / W&B during training).
Mismatch of implement and comment of attention_mask  in clip encoder?
Author: hust-nj
Pull request: no

I found the comment in CLIPEncoder

https://github.com/huggingface/transformers/blob/e65502951593a76844e872fee9c56b805598538a/src/transformers/models/clip/modeling_clip.py#L592-L596

mismatch with the code implement here

https://github.com/huggingface/transformers/blob/e65502951593a76844e872fee9c56b805598538a/src/transformers/models/clip/modeling_clip.py#L296

Comments:
Hi,

The `attention_mask` gets internally turned into a tensor containing either zeros or very low values https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_attn_mask_utils.py

Hi @hust-nj, thanks for opening an issue! 

Is the "mismatch" you're highilghting regarding the values or the shapes? As @NielsRogge mentions, for the attention calculation, the values are converted to floats and values will become large negative numbers or zeros. The masks are also expanded [within the forward pass](https://github.com/huggingface/transformers/blob/e65502951593a76844e872fee9c56b805598538a/src/transformers/models/clip/modeling_clip.py#L702) to create a 4d mask

> Hi @hust-nj, thanks for opening an issue!
> 
> Is the "mismatch" you're highilghting regarding the values or the shapes? As @NielsRogge mentions, for the attention calculation, the values are converted to floats and values will become large negative numbers or zeros. The masks are also expanded [within the forward pass](https://github.com/huggingface/transformers/blob/e65502951593a76844e872fee9c56b805598538a/src/transformers/models/clip/modeling_clip.py#L702) to create a 4d mask

I mean the value of mask in clip `vision` model(not the text part) ranging from 0 to 1 or  0 to -float(inf)

@hust-nj The clip vision model [doesn't take an attention mask](https://github.com/huggingface/transformers/blob/e65502951593a76844e872fee9c56b805598538a/src/transformers/models/clip/modeling_clip.py#L829). 

Got it, thank you 

> @hust-nj The clip vision model [doesn't take an attention mask](https://github.com/huggingface/transformers/blob/e65502951593a76844e872fee9c56b805598538a/src/transformers/models/clip/modeling_clip.py#L829).
Phi3SmallForCausalLM missing?
Author: 0wwafa
Pull request: no

I can't find the Phi3SmallForCausalLM definitions.

Comments:
The model is defined on the hub: https://huggingface.co/microsoft/Phi-3-small-128k-instruct/blob/f80aaa30bfc64c2b8ab214b541d9050e97163bc4/modeling_phi3_small.py#L898

> The model is defined on the hub: https://huggingface.co/microsoft/Phi-3-small-128k-instruct/blob/f80aaa30bfc64c2b8ab214b541d9050e97163bc4/modeling_phi3_small.py#L898

Thanks.