noamaanMulla-03/transformers-issues huggingface dataset:
[Attention Mask] Refactor all encoder-decoder attention mask
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

This PR refactors the attention mask of all PT Seq2Seq models. While this is a nice improvement of life, it is also necessary to effectively add FA2 and SDPA to PT Seq2Seq models (without having to change 54+ files).

In a follow-up PR it'll be much easier to add FA2 to just Bart and most important Bart-like models.
The PR slightly goes against the single-file policy, but attention masks are really the same across models and there is also only so much they can be (causal, non-causal, windowed). I think it doesn't really hurt readability as the function are very clearly defined (create 4d attention mask from 2d).

For some very big exceptions (I found only one which is LED, see comment [here](https://github.com/huggingface/transformers/pull/27086#discussion_r1374403602)), we could just write part of the mask creation separately as is done. 

I could also give the mask creation functions a `_` to make it clearer that they are private mehtods in Transformers. Both keeping as is or changing is good for me.

@amyeroberts @LysandreJik @ArthurZucker this is ready for a review!

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for the fast reviews everyone! Ran slow tests
```
CUDA_VISIBLE_DEVICES="0" RUN_SLOW=1 pytest ...
```

for:
- Mistral
- Falcon
- LLama
- Whisper
- Bart
- LED (exceptional model)
- Bloom (uses boolean mask)

Think this should be good enough! 

@patrickvonplaten I know this PR has been merged, but I have a question regarding the Pytorch version of BART. The implementation assumes the decoder is always used in an autoregressive manner [Pytorch version](https://github.com/huggingface/transformers/blob/8211c59b9a8fe84d2861446b26542f89a0260e64/src/transformers/models/bart/modeling_bart.py#L998) unlike the flax [version](https://github.com/huggingface/transformers/blob/8211c59b9a8fe84d2861446b26542f89a0260e64/src/transformers/models/bart/modeling_flax_bart.py#L262) . There could be cases of the decoder being used as an "encoder" and a "cross attention". In this case, the autoregressive nature is not required. While I think the default should be the autoregressive manner, but if `is_decoder` is set to false, the non-causal masking operation should be performed instead.

> non-causal masking operation should be performed instead.

@DavidAkinpelu I think you linked the FlaxAttention class, not the FlaxDecoder class above. In PT the Attention class can also be used in non-causal model, just like in Flax. If you want to use Bart in non-auto-regressive mode why don't you use BartEncoder? 

@patrickvonplaten This paper got me thinking in that direction [Mores+](https://arxiv.org/pdf/2205.04275.pdf).
Add OWLv2, bis
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds OWLv2 in a way that is more compliant to the Transformers-philosophy, meaning one paper = one model = one file. Rather than modifying the existing OWL-ViT (v1), this PR adds a new standalone Owlv2ForObjectDetection model. It copies 99% of the v1 model, only modifying the object detection head.

Follow-up of #26379 

To do:

- [x] fix image processor
- [x] add doc tests
- [x] verify objectness logits shape

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi, any idea when this PR will land? Eagerly waiting on playing with the model in HF!

@NielsRogge, I think there might be a bug in `post_process_object_detection()` when using cuda device.
You can see that `scale_fct` is created as cpu tensor here:
https://github.com/huggingface/transformers/blob/main/src/transformers/models/owlv2/image_processing_owlv2.py#L507
But here `scale_fct` is moved to device:
https://github.com/huggingface/transformers/blob/main/src/transformers/models/owlv2/image_processing_owlv2.py#L568

Thanks for spotting @assafbot would you be able to open a PR regarding this?
Add an option to decide whether to store the checkpoint and rng_state.
Author: timturing
Pull request: no
Labels: name: Feature request

**Motivation:**
Currently, when using the Transformers library in combination with DeepSpeed for training large language models like LLMs, checkpoints (e.g. `bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt`) are automatically saved along with the `rng_state`, which can lead to significant disk space usage. In scenarios where multiple GPUs are employed for training, this can quickly become a storage bottleneck, especially when shared by a team. Sometimes we just want to keep the bin file (e.g. `pytorch_model-00001-of-00002.bin`) as it's enough for load again.

**Feature Request:**
I propose adding a configurable option to decide whether to store the checkpoint and `rng_state` during training. This will give users the flexibility to choose when to save checkpoints and reduce the disk space required.

**Proposed Solution:**

1. Add a new parameter, such as `save_checkpoint_enabled`, to the DeepSpeed configuration file. Users can set this parameter to `True` or `False` to control whether checkpoints and `rng_state` should be saved during training.

2. Modify the `trainer.py` script in the Transformers library to include a condition for `self.save_checkpoint_enabled` in the `_save_checkpoint` function. Here's a code snippet illustrating the change:

   ```python
   if self.is_deepspeed_enabled and self.save_checkpoint_enabled:
       # Save the checkpoint
   ```

This change will allow users to save disk space by not storing checkpoints when not needed, and it can help alleviate the storage challenges associated with large-scale language model training.

I have already submitted this issue to the DeepSpeed library #https://github.com/microsoft/DeepSpeed/issues/4403#issue-1913025248 , as this feature may require collaboration between both libraries.

Comments:
The issue I reported is still impacting my work as our group is building a pretty big project based on this, and I believe it's an important one to address. I would be grateful if you could help me.


cc @pacman100  and @muellerzr if you think this is something we ought to have! 

@pacman100 @muellerzr I also met the save situtation. Could you provide an option to save disk memory?

Hello @timturing, checkpoints during training are meant for resuming it and therefore save the model, optimizer and scheduler and rng states. What you want is to just save the model without considering the ability to resume training. Is that understanding correct?

@pacman100 Yes, exactly.

Just like the `save_strategy` in the Trainer (https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.save_strategy).
Since SFT is very mature, we do not need to save the intermediate results for resuming.

@pacman100 @muellerzr Hi, could you improve this? This is very useful for me.
[Bug] TypeError: forward() got an unexpected keyword argument 'padding_mask'
Author: SingL3
Pull request: no

### System Info

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.34.0
- Platform: Linux-4.18.0-425.3.1.el8.x86_64-x86_64-with-glibc2.31
- Python version: 3.9.18
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.23.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.0+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Follow https://huggingface.co/blog/ram-efficient-pytorch-fsdp and submit mission with [srun](https://gist.github.com/pacman100/1cb1f17b2f1b3139a63b764263e70b25) to train a llama2 model which is save using transformers==4.33.0.


### Expected behavior

Training successfully.
I think downgrade to 4.33.0 is a solution but I wonder it is a better solution?

Comments:
The issue is that the codebase you are using has [monkey patched in a different `forward` function in the `LlamaAttention` module](https://github.com/pacman100/DHS-LLM-Workshop/blob/main/chat_assistant/training/llama_flash_attn_monkey_patch.py) <-- see the link. The monkey patched `forward` was valid in 4.33 but no longer matches the function signature of `LlamaAttention.forward` in 4.34.0. I had the same issue. This is not a bug in transformers.

I see! Thank you! Closing now.

 I have to use transformers==4.34.0. Any solution for this error?

> Any solution for this error?

1) Remove the monkey patching
2) Load your model with: `model = transformers.AutoModelForCausalLM.from_pretrained(name, use_flash_attention_2=True)`.


> > Any solution for this error?
> 
> 1. Remove the monkey patching
> 2. Load your model with: `model = transformers.AutoModelForCausalLM.from_pretrained(name, use_flash_attention_2=True)`.

 Why this should help? It doesn't change the model behavior?
Thanks

> > > Any solution for this error?
> > 
> > 
> > 
> > 1. Remove the monkey patching
> > 2. Load your model with: `model = transformers.AutoModelForCausalLM.from_pretrained(name, use_flash_attention_2=True)`.
> 
> Why this should help? It doesn't change the model behavior? Thanks

Hugging Face has already integrated flash attention for llama. So there is no need to use monkey patch. Just load model with `use_flash_attention_2=True`
VisualBERT: Low accuracy on VQA v2
Author: nikich28
Pull request: no

I use the exact code from visual bert demo, but only got about 46% accuracy on VQA v2 validation data. Has anyone had the same issue?

Comments:
Has anyone found a problem solution?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi I have the same issues? One possible reason is the id2label and label2id in config.json. The other possbile reason is the frcnn image features are not exact. Can anyone solve this issue? Thank you very much.

@nikich28 Hi nick, have you solve this issue? many thanks
Fix line ending in `utils/not_doctested.txt`
Author: ydshieh
Pull request: yes

# What does this PR do?

#27262 (by me) changed the line ending of this file to `\r\n` which is not good. This PR changes it back, and this would unblock https://github.com/huggingface/transformers/pull/26928#discussion_r1390413692

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fixed typo in error message
Author: cmcmaster1
Pull request: yes

# What does this PR do?

"past key much have a shape" -> "past key must have a shape"

I was getting this error and the typo was bothering me. Still don't know why I'm getting the error, but it least it might bother me less.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27461). All of your documentation changes will be reflected on that endpoint.
deprecation warning from transformers itself
Author: pseudotensor
Pull request: no

### System Info

https://github.com/huggingface/transformers/blob/6cbc1369a330860c128a1ba365f246751382c9e5/src/transformers/generation/configuration_utils.py#L717

This is using use_auth_token even though it's deprecated, so always showing warning despite no user mishandling.

### Who can help?

@gante 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

should be obvious from code line shared

### Expected behavior

No deprecation warning

Comments:
Yep, a CTRL + F in transformers indicates that this is also the case for some peft calls here and there. I'll open a pr for a fix, thanks for reporting!
Any way we can get dropout added to modeling_llama.py?
Author: enn-nafnlaus
Pull request: no
Labels: name: Feature request

### Feature request

Re: https://github.com/OpenAccess-AI-Collective/axolotl/issues/672

We currently lack any way to prevent overweighting of specific neurons, which becomes a problem when trying to finetune on limited datasets. Something like dropout should go a long way towards being able to develop better models at smaller parameter counts.

### Motivation

See above.

### Your contribution

I don't have familiarity with this codebase.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

cc @younesbelkada @ArthurZucker
Fix `from_pt` flag when loading with `safetensors`
Author: LysandreJik
Pull request: yes

`from_pt` flag should continue working when specifying `safe_serialization=True`. The safetensors file will be in the PyTorch format, so it's not a stretch to expect the flag to be here to load it (it works without the flag `from_pt=True` as well).

Not supporting `from_pt=True` for this is a breaking change as existing workflows with `from_pt=True` break when the serialized file is a safetensors.

Comments:
`ci/circleci: tests_torch` is treated in another PR, so not relevant here.

_The documentation is not available anymore as the PR was closed or merged._

> Not sure it's really necessary, but maybe a comment to say those methods not only deal with pytorch checkpoint but also safetensors.
> 
> (The name itself like load_pytorch_checkpoint_in_xxx doesn't match the fact)

It remains a PyTorch checkpoint, but under the safetensors format :)
OWLv2: bug fix in post_process_object_detection() when using cuda device
Author: assafbot
Pull request: yes

Fix an issue in OWLv2 when calling post_process_object_detection() with data on cuda device.
@NielsRogge, I don't know how to add you as a reviewer so I tagged you here

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27468). All of your documentation changes will be reflected on that endpoint.
Fix docstring for `gradient_checkpointing_kwargs`
Author: tomaszcichy98
Pull request: yes

# What does this PR do?

Docstring entry for `gradient_checkpointing_kwargs` was `gradient_checkpointing_args`.

Fixes #27469

@stevhliu @MKhalusova


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27470). All of your documentation changes will be reflected on that endpoint.
Wrong argument name in the documentation for `transformers.TrainingArguments`
Author: tomaszcichy98
Pull request: no

### System Info

- `transformers` version: 4.35.0
- Platform: Linux-5.15.0-1046-gcp-x86_64-with-glibc2.31
- Python version: 3.10.13
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.3.3
- Accelerate version: 0.23.0
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: DEEPSPEED
        - mixed_precision: bf16
        - use_cpu: False
        - debug: False
        - num_processes: 4
        - machine_rank: 0
        - num_machines: 1
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - deepspeed_config: {'gradient_accumulation_steps': 4, 'offload_optimizer_device': 'none', 'offload_param_device': 'none', 'zero3_init_flag': False, 'zero3_save_16bit_model': False, 'zero_stage': 3}
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- PyTorch version (GPU?): 2.1.0+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: YES


### Who can help?

@stevhliu @MKhalusova 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Create training arguments with `gradient_checkpointing_args` as in the [documentation](https://huggingface.co/docs/transformers/v4.35.0/en/main_classes/trainer#transformers.TrainingArguments) `training_args = TrainingArguments(..., gradient_checkpointing=True, gradient_checkpointing_args={"use_reentrant": False})`
2. Get error `TypeError: TrainingArguments.__init__() got an unexpected keyword argument 'gradient_checkpointing_args'`

### Expected behavior

In the documentation `gradient_checkpointing_args` should be `gradient_checkpointing_kwargs`. 

See: https://github.com/huggingface/transformers/blob/f1185a4a73a03d238afce1b40456588d22520dd2/src/transformers/training_args.py#L1137

Comments:
Install `python-Levenshtein` for `nougat` in CI image
Author: ydshieh
Pull request: yes

# What does this PR do?

We have already this in https://github.com/huggingface/transformers/blob/8f577dca4f2e9153d152afffe209fee643a90124/.circleci/create_circleci_config.py#L475. 

This PR just add the same thing to the CI image docker file to fix issues like in doctest

```
nougat.md
UNEXPECTED EXCEPTION: ImportError('\nNougatTokenizerFast requires the python-Levenshtein library but it was not found in your environment. You can install it with pip: pip\ninstall python-Levenshtein. Please note that you may need to restart your runtime after installation.\n')
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Adding Flash Attention Support for StableLMEpochForCausalLM
Author: reletreby
Pull request: no

### Feature request

Please add flash attention 2 support for StableLMEpochForCausalLM. Will help with finetuning Stability LM 3B model.

### Motivation

This will enable fine-tuning stability 3B LLM with flash attention 2

### Your contribution

Happy to help if needed!

Comments:
Hi @reletreby, thanks for opening this feature request! 

The stability LM model has its code on the hub. I'd suggest opening up a discussion to request and/or PR to add the feature there. Let us know if you have any questions about the implementation! 

Sure! Will start a discussion. Thanks
[DataCollator] Warn on identical `eos_token_id` and `pad_token_id`
Author: MustSave
Pull request: yes

# What does this PR do?
This PR displays a warning message when the values of `pad_token_id` and `eos_token_id` are identical. This is to prevent unexpected behavior during multi-turn training.


After the multi-turn data training with [DataCollatorForCompletionOnlyLM](https://github.com/huggingface/trl/blob/9e9f024399b76842ece3552884bbc4f304fd4153/trl/trainer/utils.py#L56), I encountered an issue where the model continued generating outputs even after the assistant's turn had been completed. This issue was due to the equivalence of the tokenizer's eos token and pad token by default, resulting in the eos token not being properly trained.

For instance, in the torch_call() function within [data_collator.py](https://github.com/huggingface/transformers/blob/7ee995fd9c692761c4601ddbffa2ac2ec9f27b0b/src/transformers/data/data_collator.py#L740C10-L740C10), the pad_token_id is converted to ignore_id(-100).

```python
if self.mlm:
    batch["input_ids"], batch["labels"] = self.torch_mask_tokens(
        batch["input_ids"], special_tokens_mask=special_tokens_mask
    )
else:
    labels = batch["input_ids"].clone()
    if self.tokenizer.pad_token_id is not None:
        labels[labels == self.tokenizer.pad_token_id] = -100
    batch["labels"] = labels
```

If the multi-turn data is formatted as shown below, and `eos_token_id` and `pad_token_id` are identical, the eos_token would not be properly trained. This could lead to a scenario where the model continuously generates both user and assistant turns without recognizing the end of sequence (eos) token.
```

### User: user message 1
### Assistant: assistant message 1

### User: user message 2
### Assistant: assistant message 2

```

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
Fix Falcon tokenizer loading in pipeline
Author: Rocketknight1
Pull request: yes

Previously, pipelines checked `config.json` for info on the model tokenizer. However, often the information they want is not here, and is instead stored in `tokenizer_config.json`. This PR changes the pipeline code to more aggressively try `AutoTokenizer.from_pretrained()` even if `model.config` doesn't contain tokenizer information.

This PR is quite experimental for now, don't review/merge yet!

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This turned out to be a more involved fix than I thought, so I'm going to put in a quick hack at @ArthurZucker's suggestion to fix it for Falcon, and come back to it later when the Keras 3 Situation is less urgent!

Some unrelated tests are red even after rebasing, will merge after that's resolved!
[`AWQ` ] Addresses TODO for awq tests
Author: younesbelkada
Pull request: yes

# What does this PR do?

As per title, now that models from TheBloke have been converted we will include the model in the current testing suite to make sure we are always compatible with TheBloke weights on the Hub

cc @amyeroberts 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fixed typo in pipelines.md documentation
Author: adismort14
Pull request: yes

# What does this PR do?

This PR fixes a minor (very minor, to be frank) typo in the documentation that I found while skimming through it. My OCD won't let me ignore it.

Fixes # (issue)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@stevhliu @MKhalusova 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27455). All of your documentation changes will be reflected on that endpoint.