noamaanMulla-03/transformers-issues huggingface dataset:
DO NOT MERGE: generate compatible with torch.compile(fullgraph=True)
Author: gante
Pull request: yes
Labels: name: WIP

name: Compilation

name: Cache

# What does this PR do?

This PR is a 🔪 mangled🔪  version of `generate` where `torch.compile(model.generate, fullgraph=True)` works and returns the same values. It should NOT be merged, but rather be used as a reference -- other PRs will be created that push the needed changes, once at a time, to ensure we don't break other features.

________________________________________________
### Script to test correctness
```py
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig
import torch
import copy

torch_device = "cuda"

EXPECTED_GENERATION = [
    "The best color is the one that complements the skin tone of the",
    "We should not undermind the issues at hand.\nWe should not undermind the issues",
]

tokenizer = AutoTokenizer.from_pretrained(
    "NousResearch/Llama-2-7b-chat-hf", padding_side="left", pad_token="
"
)
model = AutoModelForCausalLM.from_pretrained(
    "NousResearch/Llama-2-7b-chat-hf",
    torch_dtype=torch.bfloat16,
    attn_implementation="sdpa",
).to(torch_device)
inputs = tokenizer(
    ["The best color is", "We should not undermind the issues at hand"], padding=True, return_tensors="pt"
).to(model.device)

generation_kwargs = {
    "do_sample": False,
    "max_new_tokens": 10,
}

print("Dynamic cache")
gen_out = model.generate(**inputs, **generation_kwargs)
decoded = tokenizer.batch_decode(gen_out, skip_special_tokens=True)
print(decoded)
assert decoded == EXPECTED_GENERATION

print("Static cache")
model.generation_config.cache_implementation = "static"
gen_out = model.generate(**inputs, **generation_kwargs)
decoded = tokenizer.batch_decode(gen_out, skip_special_tokens=True)
assert decoded == EXPECTED_GENERATION
print(decoded)

print("Compiled static cache")
generation_config = copy.deepcopy(model.generation_config)
generation_config.update(**generation_kwargs)
compiled_generate = torch.compile(model.generate, fullgraph=True, mode="reduce-overhead")
gen_out = compiled_generate(**inputs, generation_config=generation_config)
decoded = tokenizer.batch_decode(gen_out, skip_special_tokens=True)
print(decoded)
assert decoded == EXPECTED_GENERATION
```
fixes #27837


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29374). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@gante If this PR is going to be long lived - you can add the `WIP` label and it will stop the bot closing if stale 

Closed in favor of #30788
Cleanup docker build
Author: ydshieh
Pull request: yes

# What does this PR do?

I am staring to go back to other CI workflow (nightly, past, amd) and the first thing is to use `runs-on: [intel-cpu, 8-cpu, ci]` instead of `runs-on: ubuntu-22.04` so we won't have disk space issue.

For `.github/workflows/build-ci-docker-images.yml`, we still keep `runs-on: ubuntu-22.04` as we want to make sure the images there are really small (for CircleCI workflow/jobs)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31119). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> With regards to the issues we were having last week with the images, are we 100% sure these work?

IIRC, that was for CircleCI images. For that part, my comment

> For .github/workflows/build-ci-docker-images.yml, we still keep runs-on: ubuntu-22.04 as we want to make sure the images there are really small (for CircleCI workflow/jobs)



And in this PR, it's only about the images for GitHub Actions.
need better token_type_id processing on transformer GPT2Model 
Author: JohnHerry
Pull request: no
Labels: name: Feature request

### Feature request

in the GPT2Model code, I see that the token_type_id value is embeded with the same **wte** of input tokens. so that the two token type id values: 0, 1,  are in collision with input sentence tokens. We need a more nature design here so that we do not have to reproduce vocab_size and sentence token values when trained with the `token_type_ids` inputs.

### Motivation

make it simply the same  vocab size and sentence token inputs for training GPT2Model with  or without `token_type_ids` input.

### Your contribution

maybe another embedding layer can be used for the token_type_id embeddings.

Comments:
Hey @JohnHerry, I'm not sure I understand what's the issue. Could you provide a code sample of what seems to be limiting you?

> Hey @JohnHerry, I'm not sure I understand what's the issue. Could you provide a code sample of what seems to be limiting you?

sorry for my unclear description.  
the code [GPT2Model](https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt2/modeling_gpt2.py) line 1064-1071:  
```        
        if inputs_embeds is None:
            inputs_embeds = self.wte(input_ids)
        position_embeds = self.wpe(position_ids)
        hidden_states = inputs_embeds + position_embeds

        if token_type_ids is not None:
            token_type_embeds = self.wte(token_type_ids)
            hidden_states = hidden_states + token_type_embeds
```  
the language tokens and `token_type_id` are sharing the same Embedding layer while the set of `token_type_id` values  【0 and 1 as said in the line 795-801】 is actually a subset of the set of language token values.  so I think there need some design to distinguish the two value sets.   maybe  different `token_type_id` values, or another Embedding layer for the two values.
Use the `Index_copy` method to update static cache inplace
Author: huzama
Pull request: yes

Use the `Index_copy` method to update the static cache inplace and avoid recompilation during each iteration in XLA

# What does this PR do?
The PR focuses on avoiding repeated recompilation during each iteration in XLA by performing in-place updates on the static cache. Specifically, it replaces direct tensor indexing assignments with `index_copy_` method calls. This technique ensures that cache updates are executed without triggering full tensor recompilations, improving runtime performance.

### Code Changes

**Original Code:**
```python
k_out[:, :, cache_position] = key_states
v_out[:, :, cache_position] = value_states
```

**Updated Code:**
```python
k_out.index_copy_(2, cache_position, key_states)
v_out.index_copy_(2, cache_position, value_states)
```




## Before submitting
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?

 @zucchini-nlp @gante 



Edit: 

- #### Discussion: [31126](https://github.com/huggingface/transformers/issues/31126)
- The code was further updated to create a new list instead of performing in-place updates to avoid recompilation in XLA.




Comments:
[Idefics2, Mistral] Improve docs
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR improves the documentation regarding Idefics2 and Mistral.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30539). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
FIX / Quantization: Add extra validation for bnb config
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes: https://github.com/huggingface/transformers/issues/31124 

Adds extra validation for bnb config, the guard `getattr(torch, bnb_4bit_quant_storage)` is not sufficient since torch can contain some attributes such as `add`, which makes `getattr(torch "add")` pass without any error

cc @SunMarc @amyeroberts 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31135). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
BitsAndBytesConfig missing a validation
Author: hotab
Pull request: no

### System Info

- `transformers` version: 4.41.1
- Platform: Windows-11-10.0.22631-SP0
- Python version: 3.12.1
- Huggingface_hub version: 0.23.2
- Safetensors version: 0.4.3
- Accelerate version: 0.30.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


### Who can help?

@SunMarc and @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import BitsAndBytesConfig
cfg = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_storage='add'
)
```

Will not fail with a validation error and will just continue as-is

### Expected behavior

Failing with a ValueError, same as `bnb_4bit_compute_dtype`

Comments:
Request for Static Cache Support for XLA Compiler in Transformers
Author: huzama
Pull request: no
Labels: name: Feature request

### Feature request

I am requesting static cache support for the XLA compiler for transformer models. This will prevent recompilation at each step caused by slicing

### Motivation

To enhance the efficiency of autoregressive generation in transformer models, static cache support for the XLA can be implemented to mitigate the need for recompilation at each generation step. 

### Your contribution

[Use the Index_copy method to update static cache inplace](https://github.com/huggingface/transformers/pull/31129)

- Additionally, While out-of-place operations could offer further improvements, they introduce new tensors with different addresses, complicating calculations. Addressing this issue is crucial for fully leveraging the potential of XLA.

Comments:
[Question] padding_side of LlamaTokenizerFast
Author: x54-729
Pull request: no

https://github.com/huggingface/transformers/blob/e42587f596181396e1c4b63660abf0c736b10dae/src/transformers/models/llama/tokenization_llama_fast.py#L100

Why `padding_side` is left here? I found out that this may cause different behaviours between `LlamaTokenizer.from_pretrained` and `AutoTokenizer.from_pretrained`.

Comments:
Hey. We can default to `padding="right"` indeed, not really sure however why/how it can create a different behaviour. Do you have a small reproducer?

> Hey. We can default to `padding="right"` indeed, not really sure however why/how it can create a different behaviour. Do you have a small reproducer?

Not a big problem actually. In some situations I want to test the logits of my model, so `padding_side` may take effect when I want to pad my input. It's not that convenient since I have to set `padding_side` every time.


You can save your tokenizer with `padding_side` and load it from the save folder no? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Would be nice to have this inconsistency resolved. It causes issues when working with logits if you are not aware of the special padding. Would be great if all tokenizers were initialized with the same padding_side value.
get_wsd_schedule gets passed num_training_steps because not handled
Author: MostHumble
Pull request: no

getting:
```
TypeError: get_wsd_schedule() got an unexpected keyword argument 'num_training_steps'
```
because there's not a handling of ```WARMUP_STABLE_DECAY```, get_wsd_schedule gets passed default params.

https://github.com/huggingface/transformers/blob/a3c7b59e31710f76492146db1d87ba77e3ae265d/src/transformers/optimization.py#L547

Comments:
cc @muellerzr
Docs / Quantization: Replace all occurences of `load_in_8bit` with bnb config
Author: younesbelkada
Pull request: yes

# What does this PR do?

As flagged by @osanseviero offline

cc @osanseviero @SunMarc @stevhliu 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31136). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix get_scheduler when name is warmup_stable_decay
Author: zspo
Pull request: yes

# What does this PR do?

fix #31085

I also add some cases for test.
![iShot_2024-05-30_10 52 20](https://github.com/huggingface/transformers/assets/26846598/1a9fcd98-07ee-4ecb-9477-af29ed6f718f)

![iShot_2024-05-30_10 52 52](https://github.com/huggingface/transformers/assets/26846598/ea7ef4a3-d68d-4cd1-b317-761f2eef76ab)


@ArthurZucker and @amyeroberts 


Comments:
@SunMarc  this means that you'd have to pass, num_warmup_steps=num_warmup_steps, AND scheduler_specific_kwargs = {num_stable_steps,  num_decay_steps, num_cycles, min_lr_ratio} as they're required params, this can be confusing as all of them are specific to that scheduler, I'm not sure why scheduler_specific_kwargs  exists

> @SunMarc this means that you'd have to pass, num_warmup_steps=num_warmup_steps, AND scheduler_specific_kwargs = {num_stable_steps, num_decay_steps, num_cycles, min_lr_ratio} as they're required params, this can be confusing as all of them are specific to that scheduler, I'm not sure why scheduler_specific_kwargs exists

I considered whether to put `num_warmup_steps` in `scheduler_specific_kwargs`. But this parameter appears in the function `get_scheduler` explicitly. 

`scheduler_specific_kwargs` exists so that we don't have a `get_scheduler` with too many args + that are specific to only a few schedulers. 

I see so it's juste two different styles clashing, thanks.
Workflow: Remove `IS_GITHUB_CI`
Author: younesbelkada
Pull request: yes

# What does this PR do?

`IS_GITHUB_CI` is a global env variable that is never used anywhere in our repo, it's a copypasta from TRL / PEFT: https://github.com/huggingface/peft/blob/cb0bf077744d11524ec6f68d920f4cfe4ef3e8f3/Makefile#L48

cc @ydshieh 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31147). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
pytest -rsfE
Author: ydshieh
Pull request: yes

# What does this PR do?

In #30318, @ArthurZucker and me decided to use `-rs` to show the skipped tests. However, this won't show the failed tests in the summary, neither the error out tests.

This PR uses `-rsfE` to show all tests except the passing one.

Without this PR
```
SKIPPED [1] tests/test_tokenization_common.py:1583: start to fail after # 29473. See https://github.com/huggingface/transformers/pull/29473#pullrequestreview-1945687810
SKIPPED [1] tests/test_tokenization_common.py:1971: No padding token.
SKIPPED [1] tests/test_tokenization_common.py:2005: No padding token.

```

With this PR
```
SKIPPED [1] tests/test_tokenization_common.py:1583: start to fail after # 29473. See https://github.com/huggingface/transformers/pull/29473#pullrequestreview-1945687810
SKIPPED [1] tests/test_tokenization_common.py:1971: No padding token.
SKIPPED [1] tests/test_tokenization_common.py:2005: No padding token.
FAILED tests/models/codegen/test_tokenization_codegen.py::CodeGenTokenizationTest::test_split_special_tokens - AssertionError: False is not true
FAILED tests/models/codegen/test_tokenization_codegen.py::CodeGenTokenizationTest::test_truncation - AssertionError: '\nif len_a > len_b:\n      result = a\nelse:\n      result = b' != '\nif len_a > len_b:      result = a\nelse:      result = b'

```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31140). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix quantized cache output
Author: SunMarc
Pull request: yes

# What does this PR do ?

This PR fixes the expected output of the quantized cache PR. This was not updated in this prior [PR](https://github.com/huggingface/transformers/pull/31052/files). Since the kernels changed with the latest version of quanto,  it is normal that we get slightly different output. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31143). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update sam.md
Author: asifajrof
Pull request: yes

# What does this PR do?

fixes an error in variable name.

`mask` variable is not defined. probably a writing mistake. it should be `segmentation_map`. `segmentation_map` should be a `1` channel image rather than `RGB`. [on a different note, the `mask_url` is the same as `raw_image`. could provide a better example.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
cc @younesbelkada
Quantization: Enhance bnb error message
Author: younesbelkada
Pull request: yes

# What does this PR do?

It's a simpler command to run `pip install -U bitsandbytes` rather than `pip install -i https://pypi.org/simple/ bitsandbytes`

Context: https://github.com/huggingface/transformers/issues/30887#issuecomment-2141601921

cc @amyeroberts 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31160). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[trainer] add sanity evaluation option 
Author: SunMarc
Pull request: yes

# What does this PR do?
This PR adds the possibility to perform a sanity check for evals before training.
Fixes https://github.com/huggingface/transformers/issues/31047#issuecomment-2137495927 

TODO: add a test 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31146). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add sanity validation steps
Author: dhruvbpai
Pull request: no
Labels: name: Feature request

### Feature request

@RylanSchaeffer

Add support for sanity validation steps before training begins, to ensure evaluation works correctly. For inspiration, Pytorch Lightning has a validation sanity check feature which is similar to what I am requesting.

### Motivation

I would like to have the option to run evaluation before training starts, either for metrics or debugging purposes.

### Your contribution

I can add support for this when given the green light.

Comments:
cc @muellerzr 

Yeah axolotl does this, would be good for us to have this as well. 

I'd be happy to get started but I am relatively new to open-source contributions. Would you be able to provide a few pointers for how you would recommend getting started, so I can take a hack at it?
@muellerzr 

cc @SunMarc not sure how far you got into this :)

Hi @dhruvbpai, I tried to implement what you asked ! LMK if this is what you wanted. I didn't test it yet but feel free to do it !
Add streaming, various fixes
Author: aymeric-roucher
Pull request: yes
Labels: name: Agents

# What does this PR do?

- [Fixes tool pushing/loading from Hub](https://github.com/huggingface/transformers/pull/30838/commits/fed6c74c2c93a2b97e2da0fd8e4fa8a9ee3eeb84#diff-0e9df5d3d8bd1fcd6927c628729420a0ae593e2bf5c4162af579dc138aea67d1).
- [Implements streaming behaviour for `run` method](https://github.com/huggingface/transformers/pull/30838/commits/ca47a4a36ddcd824bde1b720837a569ea995e2f7)
- [Improve default prompts](https://github.com/huggingface/transformers/pull/30838/commits/050ad9473161af3d54a3d54ef1a2ffc4913070e0)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30838). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@LysandreJik this is ready for review. It adds streaming to `run` method, as needed by @freddyaboulton.

The streaming api works great - thanks @aymeric-roucher !

I'd explicitly document the additional imports that can be added to the interpreter, with the `requests` example you had in mind (and a further statement mentioning that the Agent will then be able to run anything with that import, so to be careful what is allowed here).