noamaanMulla-03/transformers-issues huggingface dataset:
Enhance Text Grouping with Dynamic Padding for Variable Sequence Lengths
Author: itsmenick212
Pull request: yes

This commit introduces an enhancement to the `group_texts` function, allowing it to dynamically pad sequences within each batch to match the length of the longest sequence. This change ensures that batches fed into the model have uniform sequence lengths, improving model training stability and performance.

# What does this PR do?

## Summary

Implement dynamic padding in the `group_texts` function to handle batches with variable sequence lengths, improving model performance and training efficiency.

## Motivation

Previously, the `group_texts` function would group texts into blocks of a specified size without ensuring that each sequence within a batch was of the same length. This could lead to inefficiencies and complications during model training as machine learning models typically expect inputs of uniform size.

## Changes

The `group_texts` function now includes a padding step that dynamically adjusts the length of sequences within a batch. The sequences are padded with the tokenizer's padding token to match the length of the longest sequence in the batch.

The specific changes include:
- Computing the max sequence length in each batch post-chunking.
- Padding shorter sequences with the `tokenizer.pad_token_id`.
- Adjusting the function's return to include padded `input_ids` and `labels`.

## Impact

This change will:
- Ensure that all batches have sequences of uniform length, which is crucial for many machine learning models, particularly when using hardware accelerators like GPUs.
- Potentially increase the data throughput and efficiency during training, as uniform sequence lengths can be more easily parallelized.
- Improve the ease of use for the `group_texts` function, as it now internally manages variable sequence lengths, abstracting this complexity from the user.

## Tests

- Updated existing tests to account for the new padding behavior.
- Added new tests to specifically check for correct padding behavior with sequences of varying lengths within the same batch.

## Usage Example

```python
block_size = 128  # The desired sequence length after grouping
padded_datasets = tokenized_datasets.map(
    lambda examples: group_texts(examples, block_size, tokenizer),
    batched=True,
    num_proc=data_args.preprocessing_num_workers,
    load_from_cache_file=not data_args.overwrite_cache,
    desc=f"Grouping and padding texts in chunks of {block_size}",
)


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Large Language Models as Optimizers
Author: rajveer43
Pull request: no

### Feature request

Optimization is ubiquitous. While derivative-based algorithms have been powerful tools for various problems, the absence of gradient imposes challenges on many real-world applications. In this work, we propose Optimization by PROmpting (OPRO), a simple and effective approach to leverage large language models (LLMs) as optimizers, where the optimization task is described in natural language. In each optimization step, the LLM generates new solutions from the prompt that contains previously generated solutions with their values, then the new solutions are evaluated and added to the prompt for the next optimization step. We first showcase OPRO on linear regression and traveling salesman problems, then move on to prompt optimization where the goal is to find instructions that maximize the task accuracy. With a variety of LLMs, we demonstrate that the best prompts optimized by OPRO outperform human-designed prompts by up to 8% on GSM8K, and by up to 50% on Big-Bench Hard tasks.

### Motivation

-

### Your contribution

https://github.com/Moocember/Optimization-by-PROmpting
https://arxiv.org/abs/2309.03409

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

what do you think of this one @ArthurZucker 

As it is a feature request, unless someone form  the community tackles it I would recommend trying to put the code on the hub first and share this here! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Okay will do that

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`FA-2`] Fix fa-2 issue when passing `config` to `from_pretrained`
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes: https://github.com/huggingface/transformers/issues/28038

Some users pass the `config` attribute to `from_pretrained` in order to modify model's hyperparameters to modify the undelrying architecture. 

Note in previous versions before the attention refactor, it was possible to perform

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaForCausalLM, AutoConfig

model_id = "tiiuae/falcon-7b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
config = AutoConfig.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    config=config,
    torch_dtype=torch.bfloat16, 
    use_flash_attention_2="flash_attention_2",
    low_cpu_mem_usage=True,
)
```

Now users get an issue while trying to perform the operation above because the logic of handling model's config for fa2 changed a bit.

I propose a simple fix to mitigate this issue which is overwriting the attribute `_attn_implementation` of `config` only in case it has been passed by the user. I can confirm with this fix the snippet:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaForCausalLM, AutoConfig

model_id = "tiiuae/falcon-7b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
config = AutoConfig.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    config=config,
    torch_dtype=torch.bfloat16, 
    attn_implementation="flash_attention_2",
    low_cpu_mem_usage=True,
)
```

Works as expected as in the earlier versions of transformers


cc @amyeroberts @fxmarty 

Comments:
I am not 100% sure this approach is correct cc @fxmarty does this looks good to you (as you took care of the attention refactor) ?

I'm a bit concerned about this - this is effectively a patch inside `from_pretrained` to add backwards compatibility that should have already been handled. The main question this raises for me is whether there other FA parameters/behaviours we need to check? 

Is it still possible to pass in both `use_flash_attention_2` and `config` to `from_pretrained`? If not, it's not clear to me from the diff how this is addressed: `use_flash_attention_2` isn't handled from the model kwargs. 

Didn't do a final review on the recent refactor, so might be missing something. It's also not clear to me from just this PR why passing in a config would change whether or not I can pass in `attn_implementation`.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28043). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Good catch

@amyeroberts Even witout a fix, `use_flash_attention_2=True` along with a provided config IMO works thanks to https://github.com/huggingface/transformers/blob/050e0b44f6a63131b56d493543ab39fb7b4f20ca/src/transformers/modeling_utils.py#L1295-L1299

cc @amyeroberts @fxmarty requesting another round of review!
[`Modeling` / `Mixtral`] Fix GC + PEFT issues with Mixtral
Author: younesbelkada
Pull request: yes

# What does this PR do?

Applies the same fix presented in #28031 for Mixtral, specifically addresses: https://github.com/huggingface/transformers/issues/28023#issuecomment-1856556941

cc @amyeroberts 

Fixes: https://github.com/huggingface/trl/issues/1088

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28061). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks ! As far as I know all models that uses the new cache refactor have been fixed in #28031 + this PR
[Flax BERT] Update deprecated 'split' method
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Fixes #27644. JAX Array split method was deprecated in JAX 0.4.5: https://jax.readthedocs.io/en/latest/changelog.html#jax-0-4-5-mar-2-2023

This PR updates the four uses in the codebase to use the (recommended) `jnp.split` replacement. 


Comments:
Remove SpeechT5 deprecated argument
Author: ylacombe
Pull request: yes

# What does this PR do?

`stop_labels` is an unused argument that was supposed to be removed in `4.30.0`, here I remove it!

cc @amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28062). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Merging !
doc: Correct spelling mistake
Author: caiyili
Pull request: yes

# What does this PR do?

Correct the word "toekn" to "token" in a document.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28064). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[`Mixtral`] update conversion script to reflect new changes
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes: https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1/discussions/41

Sliding window has been recently removed from mixtral config, thus we need to reflect these changes in the conversion script

I also wonder if we should ignore `sliding_window` in MixtralAttention & MixtralFlashAttention as it will be never used

cc @amyeroberts 



Comments:
> Re removal from Mixtral modeling files, I'd say yes if it's never used. Depends on whether you think it'll ever be added again in the future.

Let me check that with Mistral team and if it will not be added again, I also think it should be removed

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28068). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
No module named 'clip'
Author: jzssz
Pull request: no

### System Info


- `transformers` version: 4.35.2
- Platform: Linux-5.4.0-166-generic-x86_64-with-glibc2.17
- Python version: 3.8.18
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 1.13.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in> yes
- Using distributed or parallel set-up in script?: <fill in> yes


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

"src/transformers/models/clip/convert_clip_original_pytorch_to_hf.py"
"from clip import load"
bug：**No module named 'clip'.** 

so I use “pip install clip” to install clip, but:
another bug: **cannot import name 'load' from 'clip'**

so I'm wondering if the way I installed the clip package is wrong? I'm hoping to find the correct way to install this package so I can run this code "src/transformers/models/clip/convert_clip_original_pytorch_to_hf.py"

### Expected behavior

I'm hoping to find the correct way to install “clip” this package so I can run this code "src/transformers/models/clip/convert_clip_original_pytorch_to_hf.py"

Comments:
The instructions for installing the package `clip` can be found on the original implementation's github README: https://github.com/openai/CLIP
Skip M4T `test_retain_grad_hidden_states_attentions` 
Author: ylacombe
Pull request: yes

# What does this PR do?

After investigating the reasons for the `test_retain_grad_hidden_states_attentions` flaky failure, I realized the speech encoder attentions can be `None` with a non-zero probability when `training=True`.  Skipping the test is the fastest fix.

Fixes #28036

cc @gante @amyeroberts @ydshieh 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28060). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for fixing! 

If training is allowed to happen on the model but it can fail e.g. with attentions being None, could you open an issue to track this? Training should either be prevented with an exception or made possible (probably 1 then the other)

Hey @amyeroberts, in theory, training is supported for the tasks that translate inputs (text or audio) into texts, since it's a classic LLM with classic objective.
To improve training, the model randomly skip layers in the speech encoder block (thus having `None` as attention weights), but it doesn't break training when it happens.
[Flax LLaMA] Fix attn dropout
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Attention dropout was not activated in Flax LLaMA, despite it being so in PyTorch LLaMA: https://github.com/huggingface/transformers/blob/1a585c1222a56bcaecc070966d558d4a9d862e83/src/transformers/models/llama/modeling_llama.py#L430

=> this PR unifies the implementations across frameworks

Comments:
Make GPT2 traceable in meta state
Author: kwen2501
Pull request: yes

# What does this PR do?

Before this PR, if we create GPT2 on "meta" device and trace it with dynamo or torch.export, the following line would create an error:
```
mask_value = torch.full([], mask_value, dtype=attn_weights.dtype).to(attn_weights.device)
```

```
torch._dynamo.exc.TorchRuntimeError: Failed running call_method to(*(FakeTensor(..., size=()), device(type='meta')), **{}):
Creating a new Tensor subclass FakeTensor but the raw Tensor object is already associated to a python object of type FakeTensor
```

That is, tracing a `.to("meta")` method on a meta tensor is not yet supported by PT2, even though we were just tracing the code.

A quick workaround is to move the device in `.to` method to the tensor constructor, which is what this PR does.

Longer term, it would be best for dynamo/export to not error out when tracing through the `.to` method in this situation.

(I will file an issue again PyTorch.)


## Who can review?

@younesbelkada
@muellerzr @SunMarc 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28054). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
support megatron-lm plugin in training
Author: xiaojunjie
Pull request: yes

# What does this PR do?
Q: How to train models in trainer.py with megatron_lm_plugin？

I think there are 3 main steps that need to be done：
- support megatron-lm plugin in trainer of transformers.
- give compatible api for transformers in accelerate.
- deveplot a tool to convert checkpoint in Megatron

this pr is for the first step.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] discussed via the forum [50804](https://discuss.huggingface.co/t/how-to-run-trainer-py-with-megatron-lm-plugin/50804)
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@pacman100 


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hello, yes, this is not planned.
Expose `gradient_as_bucket_view` as training argument for `DDP`
Author: chiragjn
Pull request: no
Labels: name: Feature request

### Feature request

As the title says, add `gradient_as_bucket_view` as the training argument (default False)

### Motivation

I have been experimenting with qlora fine-tuning LLMs on multiple A10 GPUs and I am leveraging DDP. I was going through the torch docs and https://pytorch.org/docs/2.1/generated/torch.nn.parallel.DistributedDataParallel.html and it seems the `gradient_as_bucket_view` argument can save a little bit of memory. It would be great to have it added as accelerate's DDP plugin already supports it.

I am already experimenting with it to test it out
```python
class HFTrainer(Trainer):
    def _wrap_model(self, model, training=True, dataloader=None):
        outputs = super()._wrap_model(model, training, dataloader)
        if self.args.parallel_mode == ParallelMode.DISTRIBUTED and self.accelerator.ddp_handler:
            self.accelerator.ddp_handler.gradient_as_bucket_view = True

        return outputs
```



### Your contribution

Let me know, I can also work on a PR for this as the change is relatively small

Comments:
cc @muellerzr @pacman100 

In the start of Q1 I'll be looking at closely integrating an Accelerate configuration as part of the `TrainingArguments` that will let you configure and pass anything you want in, rather than blowing up the `TrainingArguments` with many different arguments. If you can wait a few weeks until then (first few weeks into January) I should have something workable.
Update fixtures-image-utils
Author: lhoestq
Pull request: yes

The [hf-internal-testing/fixtures_image_utils](https://huggingface.co/datasets/hf-internal-testing/fixtures_image_utils) dataset fixture will break with the next release of datasets .
This dataset has a script that writes cache image files that are used in tests.
But in the next release the dataset is loaded from the Parquet files (so there is no local cache image file anymore).

FYI the issue appears because of new security features: `datasets` now loads the datasets Parquet exports by default to not let users run dataset scripts if possible.

To fix this I opened a PR on to remove the datasets script here: https://huggingface.co/datasets/hf-internal-testing/fixtures_image_utils/discussions/1

And in this PR I pass `revision="refs/pr/1"` in the tests to use the fixed dataset fixture and update the tests that rely on it.

IMO later we can merge the PR on HF and remove the `revision` argument (if we do this right now it will break tests in the other PRs on github)

cc @NielsRogge I think it's impacting tests you implemented

Comments:
Fix `low_cpu_mem_usage` Flag Conflict with DeepSpeed Zero 3 in `from_pretrained` for Models with `keep_in_fp32_modules`"
Author: kotarotanahashi
Pull request: yes

## Summary

This pull request addresses a compatibility issue in the `from_pretrained` method. Specifically, when using models with `keep_in_fp32_modules` not set to None (e.g., BLIP2, T5) in conjunction with DeepSpeed's Zero 3, an unexpected error occurs due to the improper handling of the `low_cpu_mem_usage`.

##  Problem Description

Currently, in the `from_pretrained` method, the `low_cpu_mem_usage` flag is set to `True` if `use_keep_in_fp32_modules` is `True` and accelerate is available. This logic does not account for the incompatibility with DeepSpeed Zero 3. When Zero 3 is enabled, setting `low_cpu_mem_usage` to True can lead to unexpected errors.

## Proposed Change

I propose to modify the condition to check whether DeepSpeed Zero3 is enabled. The `low_cpu_mem_usage` should be set to `True` only if Accelerate is available and DeepSpeed Zero3 is not enabled. The revised code snippet is:

```python
if is_accelerate_available() and not is_deepspeed_zero3_enabled():
    low_cpu_mem_usage = True
```

This change prevents the `low_cpu_mem_usage` flag from being incorrectly set in scenarios where DeepSpeed Zero 3 is in use, thereby avoiding the aforementioned issue.




Comments:
[docs] Trainer
Author: stevhliu
Pull request: yes

This PR attempts to clean up some of the current navigational complexity of the [`Trainer`](https://huggingface.co/docs/transformers/main/en/main_classes/trainer) API doc to make it easier to use as a purely reference lookup page. A lot of the content (checkpoints, logging, customization, etc.) is moved and organized into a separate guide.

The API page still has some content that doesn't entirely belong there (specific GPU selection, training on M1, etc.), but that'll be addressed in a separate PR.

Comments:
[docs] MPS
Author: stevhliu
Pull request: yes

As a part of a larger effort to clean up the `Trainer` API docs in #27986, this PR moves the [Trainer for accelerated PyTorch training on Mac](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#using-trainer-for-accelerated-pytorch-training-on-mac) section to the currently empty [Training on Specialized Hardware](https://huggingface.co/docs/transformers/main/en/perf_train_special) page.

Other updates include rewriting it a bit so it doesn't sound like it's copied directly from the blog post and removing the link to the paywalled article for setup 🙂 

Comments: