noamaanMulla-03/transformers-issues huggingface dataset:
Generate: v4.42 deprecations 🧹🧹
Author: gante
Pull request: yes

# What does this PR do?

Removes `generate`-related things scheduled to be removed in v4.42 :)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31956). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Generate: handle `logits_warper` update in models with custom generate fn
Author: gante
Pull request: yes

# What does this PR do?

Oops, #31956 written by me broke some models with custom `generate` loop (`logits_warper` is no longer optional).

This PR corrects failing models, and takes the opportunity to standardize (i.e. copy-paste patterns) `generate` on musicgen models.

cc @ylacombe 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31957). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add Mamba2
Author: zzhhjjj
Pull request: yes

# What does this PR do?

<!-- Remove if not applicable -->

Fixes #31204  (issue)


Comments:
cc @ArthurZucker 

This doesn't look like `Mamba2` at all, just a copy-paste of `Jamba`. Is it meant to be a placeholder for now?

One can easily see it as mamba2 doesn't use attention (except for some ablation studies) and the core (trition) functions `mamba_chunk_scan_combined` and `mamba_split_conv1d_scan_combined` are not even imported. Not to be rude but what's the purpose behind this PR? :sweat_smile: 

@vasqu I agree. Do you wanna work on a PR together?


@Adibvafa look into the issue #31204 , you can help me there or I can help here / a different PR if this is not being continued.

Feel free to open a PR and maybe leverage the new diff conversion! #30868 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Using a single 'RecurrentGemmaRglru' layer -  "Trying to backward through the graph a second time" Error
Author: talrub
Pull request: no

### System Info

- `transformers` version: 4.42.0.dev0
- Platform: Linux-4.15.0-65-generic-x86_64-with-glibc2.27
- Python version: 3.11.5
- Huggingface_hub version: 0.23.3
- Safetensors version: 0.4.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.2.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): 0.6.1 (cpu)
- Jax version: 0.3.25
- JaxLib version: 0.3.25
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: Tesla V100-SXM2-32GB


### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Code sample can be found in the following Colab link:

https://colab.research.google.com/drive/16L8RbHhg5EvgHcZq6qoZEmQS7vhV8QTr?usp=drive_link


### Expected behavior

I tried to work with a single 'RecurrentGemmaRglru'  layer followed by a linear layer (see RGLRU class) for an MNIST classification task.
After one successful epoch, i am getting the following error:

RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.

I would appreciate any advice regarding this issue since I didn't find any problem with my code.
I would like to avoid using retain_graph=True unless there is a known issue with using 'RecurrentGemmaRglru' that requires it.



Comments:
Hey! Sorry for the delayed answer here. Are you certain that you set `use_cache` to false to enter the code path where the cache is not used? 🤗 


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
sdpa for bert causes nan output when mix-precision enabled.
Author: Leoyzen
Pull request: no

### System Info

* transformers: transformers==4.41.1
* pytorch: 2.3.0+cu121
* python: 3.10


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

training bert model with f16-mixed /bf16-mixed enabled.

```python
# modeling_bert.py
...
        # Expand the attention mask
        if use_sdpa_attention_masks:
            # Expand the attention mask for SDPA.
            # [bsz, seq_len] -> [bsz, 1, seq_len, seq_len]
            if self.config.is_decoder:
                extended_attention_mask = _prepare_4d_causal_attention_mask_for_sdpa(
                    attention_mask,
                    input_shape,
                    embedding_output,
                    past_key_values_length,
                )
            else:
                extended_attention_mask = _prepare_4d_attention_mask_for_sdpa(
                    attention_mask, embedding_output.dtype, tgt_len=seq_length
                )
            #  print(f"sdpa attn mask:{embedding_output.dtype=} {extended_attention_mask.sum(axis=(2,3))=}")
        else:
            # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length]
            # ourselves in which case we just need to make it broadcastable to all heads.
            extended_attention_mask = self.get_extended_attention_mask(attention_mask, input_shape)
...
```
In the code above, the `embedding_output.dtype` used to produce the attention mask.

But within mixed precision training, the dtype of embedding output should always be float32, which casues the attention mask to be float32, then causes `F.scaled_dot_product_attention` failed and outputs nan.

We should specify the dtype by another way instead of using dtype of embedding outputs.

see also https://github.com/pytorch/pytorch/issues/98814#issuecomment-1796144712

### Expected behavior

training with f16-mixed/bf16-mixed mixed precision should ok and without nan output.

Comments:
cc @ArthurZucker @younesbelkada 

do you want to open a PR for the fix? 🤗 this sounds indeed like a problem that we can avoid


@ArthurZucker I'm not sure which dtype should been used for attention_mask generation. Do you have a recommendation?

It should be the same as the model's dtype no ? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`Gemma2`] Support FA2 softcapping
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Adds support for the new FA2 softcapping following https://github.com/Dao-AILab/flash-attention/pull/1025

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31887). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Good to see this. Can we use it for model fine-tuning, or is it just for inference? Google recommends fine-tuning in 'eager' mode.

Now you can use it for finetuning as well if you have the correct version of FA2. Not sure if finetuning "requires" it

Great! Any plans for sdpa support as well?

Sdpa is a bit more complicated, we need to use flex attention, did not have time to implement. Do you want to open a PR? 

Hi @ArthurZucker, should we also add the sliding window and soft-capping to `flash_attn_func`
https://github.com/huggingface/transformers/blob/fc35907f95459d7a6c5281dfadd680b6f7b620e3/src/transformers/models/gemma2/modeling_gemma2.py#L466-L469
just like
https://github.com/huggingface/transformers/blob/fc35907f95459d7a6c5281dfadd680b6f7b620e3/src/transformers/models/mistral/modeling_mistral.py#L519-L528

It should be here on main: https://github.com/huggingface/transformers/blob/main/src/transformers/models/gemma2/modeling_gemma2.py#L361 we updated the whole FA2 integration .

On the release branch it was there AFAIK

Get it, thanks for replying!
Add Bella 2-8B model
Author: Cognitivess
Pull request: yes

# What does this PR do?

This pull request adds support for the Bella 2-8B model. The following changes were made:
- Added `src/transformers/models/Cognitivess` directory with the following files:
  - `configuration_Cognitivess.py`
  - `modeling_Cognitivess.py`
  - `tokenization_Cognitivess.py`
  - `tokenization_Cognitivess_fast.py`
  - `__init__.py`
- Integrated the model into `auto/modeling_auto.py`.
- Registered the configuration and tokenizer in `__init__.py`.

This model achieves state-of-the-art performance on XYZ benchmarks and offers several unique features including ABC. Please review and merge.
project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Add Bella 2-8B model
Author: Cognitivess
Pull request: yes

This pull request adds support for the Bella 2-8B model. The following changes were made:
- Added `src/transformers/models/Cognitivess` directory with the following files:
  - `configuration_Cognitivess.py`
  - `modeling_Cognitivess.py`
  - `tokenization_Cognitivess.py`
  - `tokenization_Cognitivess_fast.py`
  - `__init__.py`
- Integrated the model into `auto/modeling_auto.py`.
- Registered the configuration and tokenizer in `__init__.py`.

This model achieves state-of-the-art performance on XYZ benchmarks and offers several unique features including ABC. Please review and merge.


Comments:
Fix tests after `huggingface_hub` 0.24
Author: Wauplin
Pull request: yes

See discussion [on slack](https://huggingface.slack.com/archives/C02V5EA0A95/p1721298572212239?thread_ts=1721227457.945549&cid=C02V5EA0A95) (private).

In latest `huggingface_hub` update ([0.24.0!](https://huggingface.co/spaces/Wauplin/huggingface_hub/discussions/7)), some logic has been added to avoid empty commits. This mechanism should prevent bloating git history when no changes have been detected. This change broke 2 tests in `transformers`'s CI which should be fixed by this PR (previously tested on https://github.com/huggingface/transformers/compare/main...ci-test-huggingface-hub-v0.24.0.rc0). **Note:** when an empty commit is prevented, a warning message is logged to the user. This is what we are testing in this PR. 

cc @ydshieh with whom we discussed this


**EDIT:** tests are failing since CI is still running on `huggingface_hub==0.23.5`. I guess rerunning them tomorrow should be good?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32054). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Let me check the version

It is `huggingface-hub==0.23.5` used

> It is huggingface-hub==0.23.5 used

Yes I think we need to wait for tomorrow (building docker image once a day right?)

yes! although we can build manually but it's a bit more involved

I like bonus points :heart_eyes: (though it was thanks to @ydshieh's request :smile:)

I am a lazy programmer so I ask a lot of clear comments 😅
Fix progress callback deepcopy
Author: fozziethebeat
Pull request: yes

# What does this PR do?

When ProgressCallback runs after the WandbCallback, in some scenarios it currently tries to deepcopy an object that can't be pickled. This makes a shallow copy of the log values to avoid any pickling and safely log everything.

Fixes #32064

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@amyeroberts
@muellerzr and @SunMarc


Comments:
Add Bella-2-8b language model
Author: Cognitivess
Pull request: yes

# Introduction
This PR adds support for the Bella-2-8b language model, developed by Cognitivess AI, to the Transformers library.

## Model details
- **Model name**: Bella-2-8b
- **Developer**: Cognitivess AI
- **Languages**: Multilingual (English, Romanian, Spanish, French, German, and others)
- **Tasks**: Text Generation, Question Answering, Summarization, Reasoning, Dialogue, Sentiment Analysis
- **Parameters**: 8.03 billion
- **Hugging Face model**: https://huggingface.co/CognitivessAI/bella-2-8b
- **Demo**: bella.cognitivess.com

## Key features
- Multilingual support for diverse language applications
- Tailored for empathic AI interactions
- Employs floating-point 16 (BF16) tensor type, facilitating speech-to-speech applications
- Transformer-based architecture with a sequence length of 8192 tokens
- Trained between May 2024 and June 2024, with ongoing training to incorporate new data

## Implementation details
- This implementation includes the model architecture, configuration, and tokenizer for Bella-2-8b.
- The model uses a causal language modeling objective and is based on the Transformer architecture.

## Checklist
- [x] Model files added
- [x] Configuration files added
- [x] Tokenizer files added
- [ ] Model card added (if applicable)
- [ ] Tests added (if applicable)

## Additional notes
Bella-2-8b is designed as a general-purpose language model with a focus on empathic AI interactions and multilingual support. It aims to provide high-quality text generation for a wide range of tasks while maintaining efficiency.

## License
Licensed under Apache 2.0

## References
- Model page: https://huggingface.co/CognitivessAI/bella-2-8b
- GitHub repository: https://github.com/Cognitivess/bella-2-8b

I welcome any feedback or suggestions to improve this implementation. Thank you for considering this addition to the Transformers library!

Comments:
unexpected keyword argument 'torch_empty_cache_steps' in TrainingArguments
Author: X-Chen97
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.42.4
- Platform: Linux-4.18.0-477.27.1.el8_8.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.14
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.2
- Accelerate version: 0.32.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA A40

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Received error: "TypeError: transformers-cli env.__init__() got an unexpected keyword argument 'torch_empty_cache_steps'"

![CleanShot 2024-07-18 at 18 01 28@2x](https://github.com/user-attachments/assets/22daa981-348f-46df-8cf1-e21da5a4eb27)


### Expected behavior

torch_empty_cache_steps should be one of the valid args according to the document of TrainingArguments

Comments:
Hi @X-Chen97, thanks for raising an issue! 

Could you link to the docs page you're referencing? When I look the [doc page for TrainingArguments for v4.42](https://huggingface.co/docs/transformers/v4.42.0/en/main_classes/trainer#transformers.TrainingArguments) of transformers `torch_empty_cache_steps` doesn't appear

I believe @X-Chen97 is referring to the documentation present in the code in Github. This change went in recently into the github repo https://github.com/huggingface/transformers/commit/43ffb785c0b2f24948d2011883d40dccb609d341

It might probably come out in a future stable version.

here it is: https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments.torch_empty_cache_steps

@X-Chen97 I see that the version selected in the dropdown on the left is MAIN in the link that you have referenced. I believe MAIN refers the the top of main branch here in Github which is ahead of the stable release. If you want to use the version of transformers that is on top of the main branch then you can do this,
```
pip install git+https://github.com/huggingface/transformers.git
```

ok thanks. I think this issue is solved.
Transformer compatibility Python 3.9, ComfyUI
Author: davedaverave
Pull request: no

### System Info

Paperspace A6000, A10080GB, Python 3.9, ComfyUI with Lastben Repository

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I'm running ComfyUI, i have isntalled Wassquatch as an addon, it doesn't work because cannot import name 'BlipProcessor' from 'transformers',  you support but 3.8+ but is erroring me...

### Expected behavior

Transformer would import correctly bliprocessor in my enviroment of py 3.9

Comments:
Hey @davedaverave, normally this should work seamlessly on 3.9, we test up to 3.8. What's the transformers version under the hood?

Hi the transformer version is 4.42.3
Agents planning
Author: aymeric-roucher
Pull request: yes
Labels: name: Agents

# What does this PR do?

This adds an optional planning steps to agents, to let them insert steps of higher-level planning to their workflow.

It also:
- improves several aspects in the Python code interpreter, particularly to handle pandas series.
- removes an `AgentType` behavior where int and floats or other outputs would be converted to `AgentType`, which made them less accessible.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31702). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
rewoo plan, critique
Author: plaggy
Pull request: yes
Labels: name: Agents

It's a draft, don't merge yet!

I experiment with the facts output format - not 100% sure it's better, subject to test
Added an optional ReWOO planning (https://arxiv.org/abs/2305.18323) - worked very well for me
Added a critic component which refines both a plan and an action - worked well for me

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31567). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @aymeric-roucher
TypeError: BARTDecoder.prepare_inputs_for_inference() got an unexpected keyword argument 'cache_position'
Author: Rlahuerta
Pull request: no

https://github.com/huggingface/transformers/blob/03732dea60fba1da78c79eb59c443ebf975c2be6/src/transformers/generation/utils.py#L2429C9-L2429C21

I got this error in Transformers 4.39.1 when I was trying to use Nogaut from facebook (https://github.com/facebookresearch/nougat) 

I think this method "prepare_inputs_for_generation" is not supporting "cache_position" kwargs

Comments:
@Rlahuerta hey! 

It seems to be failing because nougat has implemented its own "prepare_inputs_for_generation", which differs from `transformers` implementation. I would say that older version of `transformers` should be used in nougat, before the "cache_position" changes were introduced. For ex:

`pip install transformers==4.38.2`

@zucchini-nlp You are alright; I tested with this transformers==4.38.2, and it worked! I have notified them about this issue...

I confirm with `transformers==4.38.2` the error `BARTDecoder.prepare_inputs_for_inference() got an unexpected keyword argument 'cache_position'`

Should reported on nougat ocr github page.
Add support for Apple's DCLM-Baseline-7B model
Author: minipasila
Pull request: no
Labels: name: New model

### Model description

This is a new model released by Apple using a new framework called "openlm" so it doesn't work with Huggingface Transformers currently.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

https://huggingface.co/apple/DCLM-Baseline-7B Link to the model weights.
https://github.com/mlfoundations/open_lm Link to the model framework.
https://github.com/mlfoundations/dclm
https://arxiv.org/abs/2406.11794 Link to the paper.

Comments:
cc @ArthurZucker 

Hey @amyeroberts, I can have a look at this if it's one of the projected additions to the library.

This seems to be a decent workaround: https://github.com/mlfoundations/open_lm/pull/291
Generate: store special token tensors under a unique variable name
Author: gante
Pull request: yes

# What does this PR do?

See this comment in the end-to-end generation PR: https://github.com/huggingface/transformers/pull/30788#issuecomment-2228710326

## Problem TL;DR
Compiled code can't create tensors as `x=torch.tensor(x)` (tensor output var name = non-tensor input var name), the graph overwrites the argument (non-tensor) with the output (tensor), causing the graph to be missing the input to this node the 2nd time it is called (because it was overwritten). It is a [known limitation](https://pytorch.org/docs/stable/torch.compiler_cudagraph_trees.html#limitations) of `torch.compile`.

Our code that converts the special tokens into tensors falls into this pattern.

## Solution

Write the special tokens converted into tensors under a new variable name. A bit annoying, we will now have e.g. `pad_token_id` (integer) and `pad_token_tensor` (tensor) in the `generation_config` object throughout `generate`.

Note: We can't change the variable name where these tokens are read from, which would be much cleaner. This is because: 
1. in end-to-end compilation we can't deepcopy  
2. the attribute is set outside `generate` (at `GenerationConfig` creation time)

as such, even if we create an auxiliary attribute with a different name to read from, the original attribute will be overwritten, leading to the original issue 😢 



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31980). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.