noamaanMulla-03/transformers-issues huggingface dataset:
FEAT / Trainer: Experimental feature - add `GGUF` conversion when pushing the model to Hub
Author: younesbelkada
Pull request: yes

# What does this PR do?

Introduces a new `quantization_config` that is intended to be used only for `trainer.push_to_hub()`, it calls ` a GGUF conversion Space under the hood - (for now: https://huggingface.co/spaces/ybelkada/gguf-my-repo that we will move to a more "official" org)

The API looks like the following:
```python
from datasets import load_dataset
from trl import SFTTrainer, SFTConfig
from transformers import TrainingArguments, GgufConfig

dataset = load_dataset("imdb", split="train")

args = SFTConfig(output_dir="test-gguf-trainer")

trainer = SFTTrainer(
    "ybelkada/tiny-random-llama",
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=512,
    args=args
)

gguf_config = GgufConfig(
    quantization_method="Q8_0",
    private=True,
    space_name="ybelkada/gguf-my-repo"
)

trainer.push_to_hub(gguf_config=gguf_config)
```
cc @amyeroberts @Vaibhavs10 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30928). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Would love to get a first round of review before adding tests and documentation  ! cc @amyeroberts @muellerzr 

Example generated GGUF repo: https://huggingface.co/ybelkada/test-gguf-trainer-Q8_0-GGUF

This is possible but would require quite a huge work, the conversion script maintained by the community: https://github.com/ggerganov/llama.cpp/blob/master/convert.py evolves quite quickly, which would make maintaining that feature quite hard. On the other hand, the official Space uses docker and is build on that conversion script: https://huggingface.co/spaces/ggml-org/gguf-my-repo/blob/main/app.py#L102 
one direction could be to build a separate python package / or submodule within transformers to convert locally transformers model to GGUF, with the caveat being the maintenance burden between the official llama.cpp conversion script and our submodule. What do you think? cc @Vaibhavs10 as well

cc @LysandreJik wdyt ? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
ImportError: Using `bitsandbytes` 8-bit quantization requires Accelerate: `pip install accelerate` and the latest version of bitsandbytes: `pip install -i https://pypi.org/simple/ bitsandbytes`
Author: AnandUgale
Pull request: no

### System Info

Packages installed with CUDA 11.8:

    torch - 2.3.0+cu118
    llama-index - 0.10.37
    llama-index-llms-huggingface - 0.2.0
    transformers - 4.39.0
    accelerate - 0.27.0
    bitsandbytes - 0.43.1

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

import torch
from llama_index.llms.huggingface import HuggingFaceLLM

# Optional quantization to 4bit
from transformers import BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
)

llm = HuggingFaceLLM(
    model_name="meta-llama/Meta-Llama-3-8B-Instruct",
    model_kwargs={
        "token": hf_token,
        "torch_dtype": torch.bfloat16,  # comment this line and uncomment below to use 4bit
        # "quantization_config": quantization_config
    },
    generate_kwargs={
        "do_sample": True,
        "temperature": 0.6,
        "top_p": 0.9,
    },
    tokenizer_name="meta-llama/Meta-Llama-3-8B-Instruct",
    tokenizer_kwargs={"token": hf_token},
    stopping_ids=stopping_ids,
)


### Expected behavior

able to run LLM model

Comments:
@AnandUgale Have you tried installing accelerate as per the error message?

I have the same issue.  Accelerate is installed. This is while trying to run inference after successfully doing training with bitsandbytes.

The issue seems to be that `is_bitsandbytes_available()` in `import_utils.py` returns false when a cuda device is not available. So one should just not use the 4/8bit stuff at all when the device is CPU, which to be fair makes sense.

@RuABraun Yes, cuda is required for using bitsandbytes

cc @younesbelkada  - maybe we can update the warning to make things clearer? 

Hi ! 
https://github.com/huggingface/transformers/commit/1872bde7fc6a5d6796bd742bc2dc38eaf8069c5d should be included in the latest transformers so whenever you don't have access to a GPU it should error out with a clearer error message (I see you are using transformers==4.39.0)
Though I will enhance the error message to point out users to install bitsandbytes through the simpler command `pip install -U bitsandbytes`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Train on logits instead of one hot vectors
Author: Theodotus1243
Pull request: no
Labels: name: Feature request

### Feature request

Add the ability to train on Bigger LLM logits instead of one hot vectors created from tokenized input example

### Motivation

As stated on [gemma2 paper](https://storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf) train smaller models on logits of bigger models (distilation) produces better result

### Your contribution

Details from gemma2 [here](https://storage.googleapis.com/deepmind-media/gemma/gemma-2-report.pdf)

Comments:
Adds final answer tool for all agents
Author: aymeric-roucher
Pull request: yes
Labels: name: Agents

# What does this PR do?

Fixes a few errors reported by @MoritzLaurer .
- Fixes typo in doc
- Explains `agent.logs` in doc

**Important change:**
The CodeAgent would not always return the right output, because it would always only return the last element, regardless of wether the important element to return was the last one or not.

For instance, when asked to generate an image, the generated code could look like:
```
image = image_generator("Draw me a lamb")
print("Image correctly generated!")
```
Which returned string `"Image correctly generated!"` instead of the expected image.

To fix this, I just make all agents use the `final_answer` tool: thus the agent can return one specific outcome of its run by passing it to `final_answer`.
This also simplifies the code!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31703). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
handle (processor_class, None) returned by ModelPatterns
Author: molbap
Pull request: yes

`transformers-cli add-new-model-like` seems currently broken for models like `llava` where `ModelPatterns` captures a tuple instead of a processor string descriptor. Adding a special case to handle that.

cc @LysandreJik I suppose?


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31753). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add ignore_errors=True to trainer.py rmtree in _inner_training_loop
Author: njbrake
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #17265

ignore_errors was already put into on rmtree call via https://github.com/huggingface/transformers/pull/20984, but appears to have missed the call that happens in the _inner_training_loop


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
Adding 
@amyeroberts @sgugger  since they had been corresponding on the linked issue


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31668). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix RT-DETR cache for generate_anchors
Author: qubvel
Pull request: yes
Labels: name: run-slow

# What does this PR do?

For RT-DETR model:

1. Fix lru_cache for the `generate_anchors` method, so even if we go with dynamic anchors generation we only generate it once per image of the same size.
2. Fix anchors dtype: make anchors generated always in float32, and then convert to the desired type. Otherwise, a difference might be observed between dynamic and static anchors inference in float16/bfloat16 (test is added).

## Who can review?
cc @SangbumChoi 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31671). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Did you also finetuned RT-DETR with bfloat16? Is it okay?

No, I didn't. I tried bfloat16 only for inference 🙂
Fix RT-DETR weights initialization
Author: qubvel
Pull request: yes

# What does this PR do?

Fix RT-DETR bbox and class head weight initialization.

1. In`_init_weight` method bbox and class heads are not reachable for initialization. This sometimes leads to unstable training and lower results (see experiments below).
2. Added config parameter for bias initialization, in the [original code](https://github.com/lyuwenyu/RT-DETR/blob/5b628eaa0a2fc25bdafec7e6148d5296b144af85/rtdetr_paddle/ppdet/modeling/transformers/rtdetr_transformer.py#L325) biases of the head are initialized with `prior_prob=0.01` which is OK for training with 80 classes, however, while fine-tuning this value should be adjusted.

Results of the fine-tuning on `main` vs `fix` branches on CPPE-5 dataset (averaged for 6 runs each):

 - better convergence: +0.1-0.15 mAP50 on average on eval and test sets
 - lower dispersion of results

<img width="680" alt="Screenshot 2024-07-01 at 09 42 12" src="https://github.com/huggingface/transformers/assets/31920396/e2410381-e836-4d7c-b722-ae3575731f69">

<img width="662" alt="Screenshot 2024-07-01 at 09 43 00" src="https://github.com/huggingface/transformers/assets/31920396/a630bf42-4bfb-4afe-9e4c-1ea3b98983fd">


## Who can review?
@amyeroberts 

cc @SangbumChoi @NielsRogge 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31724). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I didn't have a chance to run the fine-tuning with the original code, maybe @SangbumChoi has a fine-tuning script to compare. However, I would say that from my previous experiments with other detection models in `transformers`  RT-DTER with this fix is the best in terms of compute, convergence speed, and results achieved on `cppe-5` dataset.

@qubvel is there anything else that needs to be done? 

@merve there is one more PR for generating anchors cache fix, but it's not that critical
https://github.com/huggingface/transformers/pull/31671

I also implemented `sdpa` attention, but didn't observe any speed-up in inference speed

@qubvel Isn't SDPA is default operation in [MDHA](https://pytorch.org/tutorials/intermediate/scaled_dot_product_attention_tutorial.html)? 
> This function has already been incorporated into torch.nn.MultiheadAttention and torch.nn.TransformerEncoderLayer.

Since there are many FLOPS in encoder (which is not related to Attention module) I guess speed-up with applying attention friendly library such as SDPA, xformers might be marginal.

@qubvel @NielsRogge Thanks for this PR. (Good to here that this is the best result by far) Unfortunately I don't have any results of finetuning raw RTDETR repo. (I have some test result in Transformers RTDETR).

@SangbumChoi I'm talking about `RTDetrMultiheadAttention` [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/rt_detr/modeling_rt_detr.py#L917), I added support for `sdpa` for this class, but didn't observe any speed-up, I will open a separate PR to discuss it :)
`tokenizer.apply_chat_template` not working as expected for Mistral-7B model; it adds `
` despite no system message.
Author: ivsanro1
Pull request: no

### System Info

- Colab
- `transformers==4.34.0`

### Who can help?
Maybe @ArthurZucker or @Rocketknight1

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Colab link to reproduce it [here](https://colab.research.google.com/drive/16xTa_3PawEocUleRzGei9ipXZVlgNjpS?usp=sharing)

### Expected behavior

The resulting instantiated prompt should be like the one [shown in the model card](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1#instruction-format), and it should not contain `
` or `
` delimiters, or system message at all:

```
text = "

 What is your favourite condiment? 
"
"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!
 "
"
 Do you have mayonnaise recipes? 
"
```


Additional note: I was running the same code yesterday on the same version of `transformers` and it was working correctly.

Comments:
Hi @ivsanro1, this is because an update to the Mistral model repo removed the chat template by accident, which caused the tokenizer to fall back to the default LLaMA chat ttemplate. I'm communicating with the team now, hopefully it can be restored very soon!

Hello @Rocketknight1, thanks a lot for the explanation, that makes a lot of sense and explains why it happened without changing transformers version.

As this is not a transformers issue per se, please feel free to close the issue when you consider it appropriate.

Hi @ivsanro1, the fix has been merged [here](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1/discussions/45), so I'll close the issue now. Thank you very much for the quick bug report though - this issue was what alerted us to the problem and let us fix it so quickly!

@Rocketknight1 the model [Qwen/Qwen2-7B](https://huggingface.co/Qwen/Qwen2-7B) contains a chat_template and it adds the `
` tags even though it wasn't trained for them..

Hi @jpiabrantes, thanks for pointing that out! You should open an issue or PR on the relevant model and tag the model authors.
Fix Bug: Gemma2 the past_key_value.update() function has added a new parameter "sliding_window" to support the _sliding_update function.
Author: kkk935208447
Pull request: yes

# What does this PR do?
System Info
transformers 4.42.3

Now gemma2 model generates long text that exceeds the window size, it will report a CUDA error, which seems to be a problem with the failure of the _sliding_update function in HybridCache. The error is as follows:

```python
# pip install accelerate
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-9b-it")
model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2-9b-it",
    device_map="auto",
    torch_dtype=torch.bfloat16
)

input_text = "Write me a poem about Machine Learning." * 800
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
outputs = model.generate(**input_ids,max_new_tokens = 1150)
print(tokenizer.decode(outputs[0]))
```

```python
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
```

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [] Did you write any new necessary tests?


## Who can review?
@ArthurZucker
@zucchini-nlp
@gante (all others)

 -->

Comments:
Question about model training with checkpointing:
Author: mostafamdy
Pull request: no

### System Info

I currently download the dataset and split it into training and evaluation sets using `dataset.train_test_split(test_size=0.005, seed=42)` every time I run the code.

My question is: When I resume training from a checkpoint with `trainer.train(resume_from_checkpoint="trained_weights/checkpoint-1000")`, will the model train on the entire dataset again, or will it skip processing the data until it reaches the saved step (step 1000 in this case)?


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

.

### Expected behavior

.

Comments:
@mostafamdy Have a look at the code from this [line](https://github.com/huggingface/transformers/blob/bdb9106f247fca48a71eb384be25dbbd29b065a8/src/transformers/trainer.py#L2125). So, the `Trainer` class have an arg named `ignore_data_skip`. It governs **"while resuming training, whether or not to skip the first epochs and batches to get to the same training data"**. And its default value is `False`. So, skip processing the data is upto you.

Cheers!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Gemma 2: Update slow tests
Author: gante
Pull request: yes

# What does this PR do?

This PR:
- Updates the change in #31747 to not rely on `locals()` (more readable)
- Adds a test for #31747 
- Updates slow tests on Gemma 2*

### Slow Gemma 2 tests
I'm assuming the tests weren't passing on any related PR: for instance, the expected output included special tokens like `<bos>`, but we decoded the generated text with `skip_special_tokens=True`. As we checkout to commits on `main` [touching Gemma 2](https://github.com/huggingface/transformers/commits/main/src/transformers/models/gemma2), none of the test text outputs is close to the original text -- @ArthurZucker is this expected?

The current diff changes the tests to match the output we currently obtain on `main` 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31759). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
`pytest_num_workers=4` for some CircleCI jobs
Author: ydshieh
Pull request: yes

# What does this PR do?

`pytest_num_workers=4` for circleci jobs that already have `parallelism=6` to avoid/reduce `requests.exceptions.ReadTimeout`.

torch job: 3m37 --> 4m20
tf job: 3m50 --> 6m09

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31764). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix Gemma2 types
Author: hiyouga
Pull request: yes


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker 


Comments: