noamaanMulla-03/transformers-issues huggingface dataset:
Make LogitsProcessor compatible with torch.compile
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Small part of the issue #28981 . This PR makes sure that Logits Processor and Stopping Criteria are compatible with `torch.compile` when `fullgraph=True`. The changes were tested with dummy inputs and logits and also with Llama. For now only the Processors used in `generate` were checked, those that are used in bark/whisper models can be checked later if needed.

The below processors are not compatible, exceptions will be added later:
- EncoderNoRepeatNGramLogitsProcessor and NoRepeatNGramLogitsProcessor -> tries to get a value from dict, which is input dependent
- PrefixConstrainedLogitsProcessor -> relies on user provided functions, which mostly probably are also input dependent
- SequenceBiasLogitsProcessor will not work at the same time with NoBadWordsProcessor, only one needs to be defined -> both call the same `_prepare_bias_variables`, which leads to recompiling it the second time we call with new arguments. Can be fixed if we either merge them into one processor or separate as two distinct.
- UnbatchedClassifierFreeGuidanceLogitsProcessor -> calls the model forward, current Llama with sdpa failed due to providing not None attention_mask.
- MaxTimeCriteria -> uses built-in time.time()



FYI @gante 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29018). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@gante I went through the comments and fixed where possible. I am wondering if it is a good idea to add warnings as I did? Maybe there is a better way to do it, so that the users do not see lots of unrelated warning. I guess not everyone will use `compile` to generate 

@gante Ready to review. I fixed tests and the generation utils to work with "cur_len", everything runs successfully in my machine. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
MBartForConditionalGeneration to do mask filling task with mbart-large-50-many-to-many-mmt
Author: Aureole-1210
Pull request: no

### System Info

I also have problem with this.
I want to use 【facebook/mbart-large-50-many-to-many-mmt】 to do mask filling task. But the output is always strange.
I modify the input format as the Model Card from https://huggingface.co/facebook/mbart-large-50-many-to-many-mmt suggested.
My code is as follows:

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import (
AutoTokenizer,
BertForMaskedLM,
MBart50TokenizerFast,
MBartForConditionalGeneration,
DataCollatorForLanguageModeling
)
model_name_or_path = 'my_path/mbart-large-50-many-to-many-mmt'
model = MBartForConditionalGeneration.from_pretrained(model_name_or_path)
tokenizer = MBart50TokenizerFast.from_pretrained(model_name_or_path)

tokenizer.src_lang = 'en_XX'

src = "So that such a thing won’t happen <mask>."
encoded_src = tokenizer([src], return_tensors="pt")
input_ids = encoded_src["input_ids"]
src_tokens = tokenizer.convert_ids_to_tokens(input_ids[0])

model_outputs = model(**encoded_src)
logits = model_outputs.logits

masked_index = torch.nonzero((input_ids[0] == tokenizer.mask_token_id)).item()
probs = logits[0, masked_index].softmax(dim=0)
values, predictions = probs.topk(5)

print(tokenizer.convert_ids_to_tokens(predictions))
```

The output is:
['.', '☎', '↔', '∏', '∴']



### Expected behavior

When I change my input, it always output strange symbols. I think this is wrong.
I think the output should be English words at least.

I am confused whether this model is not suitable for this task. How should I modify to get proper outputs? Thank you so much!

Comments:
cc @ArthurZucker 

This is a duplicate of #25425, you are using `'my_path/mbart-large-50-many-to-many-mmt'` which I cannot access. Could you try with an official checkpoint and read the linked issue? 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[Whisper] Use Attention Cache
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Refactors the Whisper model to use the attention cache abstraction proposed in #26681. This is required to have consistency with the `StaticCache` attention class proposed in #27931.

The complexity with the current `Cache` abstraction comes from the fact that Whisper is an encoder-decoder model, meaning each decoder attention layer consists of:
1. A self-attention layer (k/v cache over the previous decoder input ids)
2. A cross-attention layer (k/v cache from the encoder hidden-states)

=> the problematic layer for static generation is the dynamic k/v cache in the self-attention layer. In anticipation of using a static cache for this module, the proposed design uses a separate cache for each layer. We can't build the k/v cache into a single `Cache` abstraction, as the shapes for the self and cross-attention key-values are different (which would break compile).

The design is therefore:

```python
past_key_values: Tuple[Cache] = (past_self_attn_key_values, past_cross_attn_key_values)
```

Where `past_self_attn_key_values` and `past_cross_attn_key_values` are each `Cache` abstractions. This is not the most elegant design, but is compatible with the current `Cache` abstraction. Another option would be to do a refactor of the `Cache` / `DynamicCache` / `StaticCache` for better compatibility with encoder-decoder models.

cc @ArthurZucker @tomaarsen @gante 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28931). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I have a fully functional working solution for static shaped Whisper, which we have extensively tested on librispeech dataset and get same accuracy as original model. 

@sanchit-gandhi FYI I'm going to change the `Cache` structure a bit, while it's not widespread in the codebase. In a nutshell, given the hard constraints of the static cache (and its obvious benefits), all caches will have an interface similar to the new static cache (which differs from the original `Cache` implementation).

PR in progress here: #29005  

After this PR is done, then we can expand its usage using the same interface, e.g. for encoder-decoder models 🤗 

Thanks for the context @gante! Is there anything I can do to help with the static cache refactor? Pretty keen to implement a compile-compatible cache for Whisper!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
add YaRN RoPE scaling code for LLaMA
Author: jquesnelle
Pull request: yes

# What does this PR do?

This adds the [YaRN RoPE scaling method](https://arxiv.org/abs/2309.00071) to the LLaMA-class of models. It can be activated for finetuned models by setting `rope_scaling.type = 'yarn'` or for non-finetuned models by setting `rope_scaling.type = 'dynamic-yarn'`.

This PR enables the LLaMA family of models (LLaMA, Mistral, SOLAR, etc.) to use YaRN without `trust_remote_code=True`.
While we've [released several models](https://github.com/jquesnelle/yarn) that use `trust_remote_code`, it's nicer to not have to execute arbitrary code 🙂 

cc: @gante 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey @jquesnelle 👋 

Are you folks interested in iterating on this PR? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
the shape of trainer.predict().predictions is inconsistent with the input dataset
Author: SharonJin422
Pull request: no
Labels: name: Migration

when I run under the debug mode, the length of dataset is 500, but the length of prediction is 498, I can't find the causes....

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hello!

I met a similar problem. Did you figured it out? @SharonJin422 
Any leads on how to resolve it would be appreciated.

Thanks!
[NarrowBERT] Add NarrowBERT to HF
Author: lihaoxin2020
Pull request: yes

# What does this PR do?

Add implementation of NarrowBERT. 

- original repo: https://github.com/lihaoxin2020/narrowbert
- paper: https://arxiv.org/abs/2301.04761v2

## Who can review?

- text models: @ArthurZucker and @younesbelkada


Comments:
Hey! Thanks for wanting to contribute I would recommend you to put the model on the hub following [this tutorial](https://huggingface.co/docs/transformers/custom_models) to make it easily available 🤗

Hi. I've uploaded pretrained models to the hub. With the last commit i think I've passed all tests i can see. please let me know anything i missed. Thanks! @ArthurZucker and @younesbelkada

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi. I've uploaded the [model](https://huggingface.co/lihaoxin2020/narrowbert-sparse_attn-uncased) (i'll update model readme soon) and clear all tests i can see. It's been a while since I poked last time. Could you check in and make sure everything looks right? 
Let me know anything I should improve. Thanks! @ArthurZucker @younesbelkada 

Hey! If the model is on the hub then we should close this PR no? 🤗 

Sorry I'm confused. I did add the model weights to be hub, but I'm also adding this model implementation to the transformers library. I suppose you need to review the RP and approve so that it can be merged? 

I've marked this PR as ready for review so you should be able to review and comment/approve. 
Let me know if there's anything I misunderstood. I'm new on open source project. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
loss return from SpeechToSpeechOutput cannot fit to Seq2SeqTrainer
Author: indiejoseph
Pull request: no

Seq2SeqTrainer cannot recognize the loss returned from SpeechToSpeechOutput, and return this error:

```
The model did not return a loss from the inputs, only the following keys: spectrum, encoder_last_hidden_state. For reference, the inputs it received are input_values, attention_mask, labels
```

I cannot find any code or tutorial about training SpeechT5SpeechToSpeech model, I can only found the training source code in fairseq's repo, but it is obsolete and not well documented how to prepare the training dataset.

Comments:
cc @ylacombe and @sanchit-gandhi 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Fixed by #29357
MPNet doesn't have an implemented LMHead subclass
Author: ysahil97
Pull request: no
Labels: name: Feature request

### Feature request

The code of mpnet needs a proper implementation of LMHead class to adapt to Language Generation Tasks

### Motivation

We want to perform some experiments on the MPNet style inference on language generation tasks. On going through the codebase, there is a standby class inherited from `torch.nn.module`, which means there is no proper implementation of LMHead on this model

### Your contribution

I'm eager to contribute to this issue by submitting a PR. This work would be super relevant for us to extend our experiments to MPNet as well for our research projects.

Comments:
MPNet has `MPNetForMaskedLM` no?  with a lm head no? 

It does have the class `MPNetForMaskedLM` but no implementation of `MPNetWithLMHead`

Pinging this issue to revive the discussion.
I would love to contribute to the possible implementation of `MPNetWithLMHead` towards the HuggingFace library

Sorry, but I don't think this is relevant:
1. There are only 3 models that have `WithLMHead` class, and it's probably a mistake
2. It is completely equivalent to `XXXForMaskedLM` AFAIK
How the new version of transformers uses the author's LLaVA weights?
Author: koking0
Pull request: no

I am very excited that the LLaVA model has been added to transformers-4.36. I noticed that the LLaVA model of transformers seems to be different from the LLaVA author's model.

LLaVA model of transformers: [https://huggingface.co/llava-hf/llava-1.5-7b-hf](https://huggingface.co/llava-hf/llava-1.5-7b-hf)
LLaVA author's model: [https://huggingface.co/liuhaotian/llava-v1.5-7b](https://huggingface.co/liuhaotian/llava-v1.5-7b)

We have been using the LLaVA author's model. I would like to ask if transformers-4.36 can load the LLaVA author's model.

Something like this:

```python
from PIL import Image

import torch
from transformers import AutoProcessor, LlavaForConditionalGeneration

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# model_id = "llava-hf/llava-1.5-7b-hf"
model_id = "liuhaotian/llava-v1.5-7b"

prompt = "USER: <image>\nWhat are these?\nASSISTANT:"
image_file = "images/000000039769.jpg"

processor = AutoProcessor.from_pretrained(model_id)
model = LlavaForConditionalGeneration.from_pretrained(
     model_id,
     torch_dtype=torch.float16
).to(device)


raw_image = Image.open(image_file)
inputs = processor(prompt, raw_image, return_tensors='pt').to(0, torch.float16)

output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(processor.decode(output[0][2:], skip_special_tokens=True))
```

Comments:
Hi,

Both should be equal (cc @younesbelkada).

Which difference are you seeing?

I used the above code, and the initial error was that the preprocessor_config.json file was missing. 

I found that there was indeed no such file in https://huggingface.co/liuhaotian/llava-v1.5-7b/tree/main, so I started from https://huggingface.co/llava-hf/llava-1.5-7b-hf/tree/main Download this file, then put it in the model folder of liuhaotian/llava-v1.5-7b, and then run it, I encountered the following error:

```
valueError: The input provided to the model are wrong. The number of image tokens is 0 while the number of image given to the model is 1. This prevents correct indexing and breaks batch generation.
```

![image](https://github.com/huggingface/transformers/assets/45281765/b6d2865c-9673-492a-9636-f7c00291cfe1)


Hi @koking0 
unfortunately the checkpoints are not inter-compatible, if you want to use `liuhaotian/llava-v1.5-7b` you need to use the original author's library and not transformers

ok, fine.

Note however `liuhaotian/llava-v1.5-7b` should be the same as `llava-hf/llava-1.5-7b-hf`, same for the 13b model and so on 

> Hi @koking0 unfortunately the checkpoints are not inter-compatible, if you want to use `liuhaotian/llava-v1.5-7b` you need to use the original author's library and not transformers

Hi @younesbelkada. What is this mean ' not transformers'. Is that mean He should not use AutoProcessor,LlavaForConditionalGeneration from transformers packages?

The `transformers` supports is for the `-hf` checkpoints.
Generate: move misplaced test
Author: gante
Pull request: yes

# What does this PR do?

Moves a integration test in the mixin to its right place :) 

[because it was a mixin, it was running more times than it should 👀 ]

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29902). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Misleading warning message about padding_side when passing tokenizer instance to pipeline
Author: tilmanbeck
Pull request: no

### System Info

System Info
Python=3.11.5
Transformers= '4.37.2'

In my setup I am initializing a tokenizer and want to pass it to the pipeline. My expectation is that if I set the padding_side directly on tokenizer instance, the pipeline should not print the warning `A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set 'padding_side="left"' when initializing the tokenizer.` 

Alternatively, I would like to add it as a parameter to the pipeline instantiation, but I think passing tokenizer parameters in the pipeline generator is currently not envisioned, as discussed in https://github.com/huggingface/transformers/issues/12039 https://github.com/huggingface/transformers/issues/24707 https://github.com/huggingface/transformers/pull/22995

Possibly similar issue origin as reported in #29378 

@Narsil 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer, pipeline, AutoModelForCausalLM, Conversation

msg = 'The capital of France '
modelname = 'microsoft/DialoGPT-small'

# init model and tokenizer
model = AutoModelForCausalLM.from_pretrained(modelname)
tokenizer = AutoTokenizer.from_pretrained(modelname, padding_side='left')
tokenizer.pad_token_id = tokenizer.eos_token_id
#  hand over model & tokenizer instances to pipeline
chatbot = pipeline(task='conversational', model=model, tokenizer=tokenizer, framework='pt')
messages = ([{"role": "system", "content": 'You are a helpful assistant'}, {"role": "user", "content": msg}])
response = chatbot(Conversation(messages=messages), pad_token_id=chatbot.tokenizer.eos_token_id)
```
This prints the warning message: 
A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.

### Expected behavior

The warning message should not be printed as `padding_side` was set during tokenizer initialization.
Alternatively, I would expect to be able to provide `padding_side` parameter directly to the pipeline generation. 

Comments:
cc @Rocketknight1 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

#29614 fixed it so closing
4.38 showing Grad_norm in log (when used with wandb) as tensors thus giving error while saving with json
Author: deshwalmahesh
Pull request: no

### System Info

`transformers==4.38`

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Use `wandb` logging with 4.38. Same script runs smoothly on `4.37` but breaks with the newer one as grad norms are logged too

### Expected behavior

Error while serializing the grad norm tensors in training logs

Comments:
modify  dist-packages/transformers/trainer.py   

self._maybe_log_save_evaluate(tr_loss, grad_norm, model, trial, epoch, ignore_keys_for_eval)
==》
self._maybe_log_save_evaluate(tr_loss, grad_norm.item(), model, trial, epoch, ignore_keys_for_eval)

In my case, I changed the code as follow:

https://github.com/huggingface/transformers/blob/831bc25d8fdb85768402f772cf65cc3d7872b211/src/transformers/trainer.py#L2403

before:
```
logs["grad_norm"] = grad_norm
```

after:
```
logs["grad_norm"] = grad_norm if isinstance(grad_norm, (int, float)) else grad_norm.item()
```

cc @muellerzr 

This should be fixed now I believe, can you try installing from main? (`pip install git+https://github.com/huggingface/transformers`)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[docs] Big model loading
Author: stevhliu
Pull request: yes

This PR consolidates and updates all the relevant info for instantiating and loading big models. Previously, it was split between [Large model loading](https://huggingface.co/docs/transformers/main_classes/model#large-model-loading) section in the Model API and the [Instantiating a big model](https://huggingface.co/docs/transformers/big_models) doc, which caused some important stuff (`low_cpu_mem_usage`) to be missed/overlooked because users weren't expecting it to be in the API docs.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29920). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[`generate`] fix breaking change for patch
Author: ArthurZucker
Pull request: yes

# What does this PR do?

A bug was introduced by #29467 pretty much unrelated to cache positions.
This fixes #29968

cc @gante and @zucchini-nlp. The testing suite is missing this particular test for all generation strategies


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29976). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Failing test is unrelated