noamaanMulla-03/transformers-issues huggingface dataset:
Adding FlaxNoRepeatNGramLogitsProcessor
Author: giganttheo
Pull request: yes

# What does this PR do?

Adding the no repeat n-gram logits processor to Flax, compatible with jitting.
I also added the test `test_no_repeat_ngram_dist_processor`, adapted from the torch one, and added the `FlaxNoRepeatNGramLogitsProcessor` to the `test_processor_list` and `test_processor_list_jitted`.

All the tests are passing, as `RUN_SLOW=1 pytest -sv tests/generation/test_flax_logits_process.py` prints:

```python
======================================= test session starts ========================================
platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.4.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /content/transformers
configfile: pyproject.toml
plugins: anyio-3.7.1
collected 9 items                                                                                  

tests/generation/test_flax_logits_process.py::LogitsProcessorTest::test_forced_bos_token_logits_processor PASSED
tests/generation/test_flax_logits_process.py::LogitsProcessorTest::test_forced_eos_token_logits_processor PASSED
tests/generation/test_flax_logits_process.py::LogitsProcessorTest::test_min_length_dist_processor PASSED
tests/generation/test_flax_logits_process.py::LogitsProcessorTest::test_no_repeat_ngram_dist_processor PASSED
tests/generation/test_flax_logits_process.py::LogitsProcessorTest::test_processor_list PASSED
tests/generation/test_flax_logits_process.py::LogitsProcessorTest::test_processor_list_jitted PASSED
tests/generation/test_flax_logits_process.py::LogitsProcessorTest::test_temperature_dist_warper PASSED
tests/generation/test_flax_logits_process.py::LogitsProcessorTest::test_top_k_dist_warper PASSED
tests/generation/test_flax_logits_process.py::LogitsProcessorTest::test_top_p_dist_warper PASSED

========================================= warnings summary =========================================
../../usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py:1373
  /usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py:1373: PytestConfigWarning: Unknown config option: doctest_glob
  
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================== 9 passed, 1 warning in 14.73s ===================================
```

Note: in order to work properly within beam search, this processor needs the fix proposed in PR #29636 for the bug discussed in #29635 

Let me know if you have any comments or questions regarding this feature.

## Who can review?

@gante
@sanchit-gandhi


Comments:
The red CI can be fixed by running `make fixup` on your terminal (in the `transformers` folder), then committing the changes

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29677). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Merging as there is a green light and green CI! 🥳
Update documentation for agents
Author: aymeric-roucher
Pull request: yes
Labels: name: Agents

# What does this PR do?

Update documentation for agents.

cc @LysandreJik @Jofthomas  

Comments:
Is this a duplicate of #29815? 

@ArthurZucker yes, we decided with @LysandreJik to merge a few PRs on the `agents` branch of transformers first rather than directly on `main`.

@ArthurZucker I closed the other Pr and I keep you remark: "make sure the `test_pr_documentation` passes". Thank you for your follow-up!

@LysandreJik this is ready: if you could give this a quick review that'd be awesome!
Making the whole documentation work involved a few tweaks in the other files, thus the other changes on top of the `.md` file.

> Awesome work, thanks again for updating this doc! Most of my suggestions are around how to structure the doc so it's not as granularly defined with so many headers which can be distracting, and creating a more prominent Tool section.
> 
> The rough outline of the doc looks better like this imo with some of the more minor subsections folded into the larger subsection they belong in.
> 
> ```
> # Agents and tools
> ## Agents
> ### Build an agent
> ### Code execution
> ### Customize system prompt
> ## Tools
> ### Create a new tool
> ### Manage agent toolbox
> ### Gradio-tools
> ### LangChain tools
> ```

@stevhliu thank you for the suggestion! I've re-ordered the doc to follow your suggestion, and everything is much more streamlined indeed!

The only thing is that Agents need Tools, so defining tools only after the agent creates some sort of mental ImportError for the reader: but if the short definition "a function that can b e called by the agent" is not enough, they can just click on the table of contents to dive deeper into the second part before going back to the first.

What do you think? 😃

Yeah a short definition of what a "tool" is is totally fine! Both concepts of agents and tools can be described in the opening paragraph in line 20. Something along the lines of the below would suffice :)

```
The agent is empowered by *tools*, functions for performing a task that can be called by the agent.
```

So as to unblock the cascading PRs, let's merge this PR into `agents` now that all comments have been addressed. @stevhliu we're still happy to incorporate changes in the doc if you have proposals!
GenerationConfig: warn if pad token is negative
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Related to the [issue](https://github.com/huggingface/transformers/issues/25426). I believe we should add warnings here so the users know how to fix this type of errors.

The -1 pad token is for now noted only in InstructBlip, which seem to be loaded from a hub repo that does not exist anymore (`decapoda-research/llama-7b-hf`). Anyway we can raise warning for future cases if there are other models based on it.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30187). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Flash Attention Support for Blip2ForConditionalGeneration 
Author: soumya1729
Pull request: no
Labels: name: Feature request

name: Flash Attention

Is there anyone working on a FlashAttention support for Blip2ForConditionalGeneration?

Comments:
Hi @soumya1729, there's an issue tracking model requests and open PRs here: #26350 

If there's a model missing, feel free to comment on this issue asking for it to be added, or to open a PR to add FA2.
Update `make-fixup` to make sure image processor tests present
Author: ydshieh
Pull request: no
Labels: name: Core: Modeling

name: Feature request

### Feature request

Update `make-fixup` to make sure image processor tests present

See [this comment](https://github.com/huggingface/transformers/pull/30136#issuecomment-2068884321)



### Motivation

Make `make-fixup` more robust

### Your contribution

I can work on it

Comments:
Schedule Free Optimizers (pytorch) and Sophia optimizer
Author: CoffeeVampir3
Pull request: no
Labels: name: Feature request

name: optimization

### Feature request

See <https://github.com/facebookresearch/schedule_free> and <https://github.com/Liuhong99/Sophia> -- These optimizers have very different properties and are often useful over the existing choices.

### Motivation

Schedule free optimizer (especially schedule-free SGD) offers increased convergence over the existing implementation with no additional memory requirements. There's some variability on the end gradients but this is an equitable engineering tradeoff in many cases. In the same vein, sophia has been shown to converge twice as fast as adam on some language modeling tasks.

### Your contribution

These would add in additional dependencies if included straight on, I'm not sure what the typical approach for this is. I'd be willing to implement both features but I'm unsure how the addition of dependencies should go.

Comments:
#30079 and #24338 are relevant, not sure if the policy around optimizers has changed but you could try out the branches if it's not merged. 

cc @younesbelkada
`Llama` family, fix `use_cache=False` generation
Author: ArthurZucker
Pull request: yes

# What does this PR do?
fixes #30347

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30380). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Yeah, but it feels like this should be part of the generation mixin tests (and AFAIR it is?)
Update docstrings for text generation pipeline
Author: Rocketknight1
Pull request: yes

Updates the docstrings and doctests for the text generation pipeline, now that it's fully taken over the conversational pipeline role!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30343). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add HelpingAI-3B-v2.2: Emotionally Intelligent Conversational AI
Author: OEvortex
Pull request: no
Labels: name: New model

### Model description

## Introduction

HelpingAI-3B-v2.2 is a state-of-the-art large language model specializing in emotionally intelligent conversation. With advanced emotional understanding capabilities, it can engage in empathetic dialogue tailored to the user's emotional state and context.

## Emotional Intelligence Capabilities

HelpingAI-3B-v2.2 exhibits several key traits that enable emotionally resonant responses:

- Emotion recognition and validation
- Empathetic perspective-taking 
- Generating emotionally supportive language
- Contextual emotional attunement
- Using appropriate tone, word choice and emotional expression

Whether comforting someone grieving, celebrating positive news, or addressing complex feelings, HelpingAI-3B-v2.2 can adapt its communication style with emotional nuance.

## Examples of Emotionally Intelligent Responses


"I'm really sorry to hear about your friend's loss. 😔 Losing a parent can be incredibly difficult and traumatic, and it's completely understandable that they are struggling with grief. It's important to show them empathy and support during this tough time. You can start by simply being there for them, listening to them, and offering a shoulder to lean on. 💕 It's also important to acknowledge their feelings and let them know that it's okay to grieve and take as much time as they need to process their emotions. Offering practical help, such as running errands or cooking for them, can also make a big difference. Remember, small gestures of kindness and compassion can go a long way in comforting someone during difficult times."

"Hey there! 🌟 It's a beautiful day today, so you should cook something delicious and enjoy the sunny weather! How about trying out a new recipe and let your creativity flow? Whether it's a classic dish or something experimental, the kitchen is the perfect place to let your imagination run wild. Write me a list of ingredients you have on hand, and I'll help you come up with a wonderful culinary creation! 🍲🌞"

The model tailors its language, tone and emotional content to be contextually appropriate, combining emotional intelligence with factual knowledge and practical suggestions.



### Open source status

- [ ] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation
https://huggingface.co/collections/OEvortex/emotional-intelligence-6625cc3247851d717e9fa382

Comments:
Jamba: fix left-padding test
Author: gante
Pull request: yes

# What does this PR do?

[This PR](https://github.com/huggingface/transformers/pull/30016) was built on a branch that did not have jamba :) This PR integrates the change from that PR on jamba tests

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30389). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update `transfromers` to use `tokenizers>=0.19`
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Update to latest breaking release of `tokenizers`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29991). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix DETA save_pretrained
Author: qubvel
Pull request: yes

# What does this PR do?

`save_pretrained` is not working for `DetaForObjectDetection`. This PR is supposed to fix this error.

A simple script to reproduce:
```python
from transformers import DetaConfig, DetaForObjectDetection

config = DetaConfig()
model = DetaForObjectDetection(config)
model.save_pretrained("output_dir/")
```

or

```python
from transformers import AutoModelForObjectDetection

model = AutoModelForObjectDetection.from_pretrained("jozhang97/deta-resnet-50")
model.save_pretrained("output_dir/")
```

The following error occurs:

```
RuntimeError: The weights trying to be saved contained shared tensors [{'model.decoder.class_embed.0.weight', 'class_embed.0.weight'}, {'model.decoder.class_embed.0.bias', 'class_embed.0.bias'}, {'model.decoder.class_embed.1.weight', 'class_embed.1.weight'}, {'model.decoder.class_embed.1.bias', 'class_embed.1.bias'}, {'model.decoder.class_embed.2.weight', 'class_embed.2.weight'}, {'model.decoder.class_embed.2.bias', 'class_embed.2.bias'}, {'model.decoder.class_embed.3.weight', 'class_embed.3.weight'}, {'class_embed.3.bias', 'model.decoder.class_embed.3.bias'}, {'model.decoder.class_embed.4.weight', 'class_embed.4.weight'}, {'model.decoder.class_embed.4.bias', 'class_embed.4.bias'}, {'model.decoder.class_embed.5.weight', 'class_embed.5.weight'}, {'class_embed.5.bias', 'model.decoder.class_embed.5.bias'}, {'class_embed.6.weight', 'model.decoder.class_embed.6.weight'}, {'class_embed.6.bias', 'model.decoder.class_embed.6.bias'}] that are mismatching the transformers base configuration. Try saving using `safe_serialization=False` or remove this tensor sharing.
```

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30326). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @NielsRogge
FIX / PEFT: Pass device correctly to peft
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes: https://github.com/huggingface/transformers/issues/30339 - before this PR the underlying loading device was automatically inferred by PEFT that loads the adapter on other devices than one would target it 

Also fixes an issue when passing `{"device": device}` to `adapter_kwargs` as `device` does not get popped from the kwargs

All PEFT tests pass with this fix

cc @amyeroberts 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30397). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add `device` parameter for PEFT `load_adapter`
Author: uwu-420
Pull request: no
Labels: name: Feature request

### Feature request

Hello,

the function `load_adapter` of the `PeftAdapterMixin` class currently doesn't have a `device` parameter but a `device_map` parameter which isn't being used by the `load_peft_weights` function. Instead, the `load_peft_weights` function will use `infer_device` to figure out some device.

See
https://github.com/huggingface/transformers/blob/30b453206d224ee5f747afa33ff216671558e6a0/src/transformers/integrations/peft.py#L193C13-L193C97
and
https://github.com/huggingface/peft/blob/5a4b9cade64bac8afdff5006ee9dd815c90b5469/src/peft/utils/save_and_load.py#L392C9-L392C32

Can it be changed to use the device of the model we want to add the adapter to by default?

### Motivation

I stumbled over this when I had a model on CPU and the loaded adapter was loaded onto the MPS device which caused other issues for me. I think it would be more intuitive if the adapter is directly loaded to the same device as the model.

### Your contribution

Sorry, nothing here. But happy to provide more context.

Comments:
cc @younesbelkada 

Hi @uwu-420 
Thanks for the feature request ! 
Currently you can pass the device through `adapter_kwargs={"device": device}` but I agree we should use the same device as the model if possible. I will work on that and provide a fix
RuntimeError: "triu_tril_cuda_template" not implemented for 'BFloat16'
Author: DogeWatch
Pull request: no
Labels: name: bug

### System Info

transformers==4.19.2

### Who can help?

use bf16 with accelerate config
```
compute_environment: LOCAL_MACHINE
deepspeed_config: {}
distributed_type: MULTI_GPU
fsdp_config: {}
machine_rank: 0
main_process_ip: null
main_process_port: null
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 8
use_cpu: false
```
model is LongformerModel, and get error

File "~/miniconda3/envs/bai/lib/python3.8/site-packages/transformers/models/longformer/modeling_longformer.py", line 788, in _mask_invalid_locations
    beginning_mask_2d = input_tensor.new_ones(affected_seq_len, affected_seq_len + 1).tril().flip(dims=[0])
RuntimeError: "triu_tril_cuda_template" not implemented for 'BFloat16'


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

none

### Expected behavior

none

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I have same error. And it is only in windows.How can I do can resolve it.Although WSL does not have this error, running CUDA programs on Windows always causes VSCode to disconnect or the computer to freeze (the screen is static, all input functions are disabled, including keyboard and mouse), or a blue screen occurs.
when I update torch to 2.2.2 this issue has been resolved
Add Llama 3 support to `convert_llama_weights_to_hf()`
Author: calmitchell617
Pull request: no

### Feature request

Add Llama 3 support to [`convert_llama_weights_to_hf()`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/convert_llama_weights_to_hf.py).

### Motivation

I am using [torchtune](https://github.com/pytorch/torchtune) to fine-tune Llama 3. The easiest way to output checkpoints of that model with that lib is the Meta checkpointer. Checkpoints saved with that checkpointer are not loadable with `from_pretrained()`, which makes it hard to use Llama 3 models fine-tuned with torchtune in the Huggingface ecosystem (such as with Gradio).

### Your contribution

I will contribute a PR if someone from HF will take a look at it once submitted.

Comments:
cc @ArthurZucker 

I successfully performed the following steps to test whether adding Llama 3 support to this script would facilitate the use case I outlined above:

I downloaded Llama 2 in meta format (not HF format) with the torchtune cli. It is important not to test with a model in the HF format, because Llama 3 is not yet officially uploaded in HF format. It wouldn't be an apples to apples comparison.

CLI command used:

```
tune download meta-llama/Llama-2-7b-chat --output-dir <checkpoint_path> --hf-token $HF_TOKEN
```

I then fine tuned the downloaded meta-format Llama 2 with torchtune:

```
tune run \
    --nproc_per_node=4 \
    lora_finetune_distributed \
    --config llama2/7B_lora \
    batch_size=1 \
    seed=29 \
    tokenizer.path=<checkpoint_path> \
    checkpointer.checkpoint_dir=<checkpoint_path> \
    checkpointer.output_dir=<checkpoint_path> \
    dataset=torchtune.datasets.my_custom_dataset \
    checkpointer.checkpoint_files=['consolidated.00.pth'] \
    checkpointer=torchtune.utils.FullModelMetaCheckpointer \
    gradient_accumulation_steps=1 \
    lr_scheduler.num_warmup_steps=5 \
    enable_activation_checkpointing=False \
    dataset.max_rows=100 \
    epochs=1
```

Then I converted the fine-tuned model to format that can be loaded with `from_pretrained()` with the `convert_llama_weights_to_hf()` function. I named the script `convert_checkpoint.py`, but it just contains the `convert_llama_weights_to_hf()` function copy/pasted from Transformers:

```
python convert_checkpoint.py --input_dir <checkpoint_dir> --output_dir <output_dir>  --model_size 7B
```

I then ran a Gradio chatbot on the fine-tuned/converted model, and it worked as expected. So, it does seem that adding Llama 3 support to this script will unlock my particular use case (and probably many others).

see #30334 

Thank you! I will keep an eye on that on that PR.

Just needs the doc 🤗 

Good to know. 

Let me know if that's something I could contribute, otherwise, thanks for the good work.
[docs] LLM inference
Author: stevhliu
Pull request: yes

This PR creates a lightweight LLM inference optimization guide focused on just providing a brief explanation and copy/paste-able code examples so users can instantly get going without necessarily needing to get too in-depth in the underlying conceptual details about how it works. It includes:

- static kv-cache + torch.compile
- speculative and prompt lookup decoding
- attention optimizations (FlashAttention-2 and SDPA)
- quantization

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29791). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

For each of the topics covered, I was wondering if:

- @ArthurZucker, would you mind having a look at the [static kv + torch.compile](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29791/en/llm_optims#static-kv-cache-and-torchcompile) section?
- @gante, would you mind having a look at the [speculative decoding](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29791/en/llm_optims#speculative-decoding) section?
- @younesbelkada or @SunMarc, would you mind having a look at the [attention optimization](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29791/en/llm_optims#attention-optimizations) and [quantization](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29791/en/llm_optims#quantization) sections?