noamaanMulla-03/transformers-issues huggingface dataset:
little suggestion on pad_token check in LlamaForSequenceClassification class
Author: Alessa-mo
Pull request: no
Labels: name: bug

### System Info

When I use LlamaForSequenceClassification for training the official llama3.1 8b model on multi label classification task, as we know, llama model doesn't have a pad_token, we should set the pad_token.
Usually, we write 
`tokenizer.pad_token = tokenizer.eos_token` or `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`
However, the code for check pad_token in the LlamaForSequenceClassification class is:
```
if self.config.pad_token_id is None and batch_size != 1:
            raise ValueError("Cannot handle batch sizes > 1 if no padding token is defined.")
```
It reads the model config file and then checks it. So I have to modify the config file instead of simply adding the pad_token set code in my code. Otherwise, it will raise the error. It's quite inconvenient. I think maybe it can be improved.

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

`tokenizer = PreTrainedTokenizerFast.from_pretrained("meta-llama/Meta-Llama-3.1-8B")`
`model = LlamaForSequenceClassification.from_pretrained("meta-llama/Meta-Llama-3.1-8B", num_labels=num_labels)`
`if tokenizer.pad_token is None:
    tokenizer.pad_token =  tokenizer.eos_token`
call any function which invovles the forward process, like training or infering.

### Expected behavior

don't need to change the config file to avoid these exception

Comments:
I misunderstand the LlamaForSequnceClassification code

No worries!

There is an issue with this check. I loaded llama tokenizer and I added the padding token:


```
tokenizer = AutoTokenizer.from_pretrained(model_params["MODEL"])
    if tokenizer.pad_token is None: 
        tokenizer.pad_token = tokenizer.eos_token
```


but then I get the error:

> File ".../decoder_to_encoder/venv/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 1391, in forward                                                     │·····················
>     raise ValueError("Cannot handle batch sizes > 1 if no padding token is defined.")
>
Default setting of GenerationConfig
Author: jiangwangyi
Pull request: no
Labels: name: bug

name: Generation

### System Info

```
- `transformers` version: 4.44.2
- Platform: Linux-5.15.0-91-generic-x86_64-with-glibc2.35
- Python version: 3.11.9
```

### Who can help?

@gante @SunMarc

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When creating a GenerationConfig instance with the following code:
```python
>>> from transformers import GenerationConfig
>>> GenerationConfig(do_sample=False, temperature=0.0)
```
I encounter the following warning:
```
UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.0` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.
```
I don't believe these two parameters conflict with each other. Additionally, I am curious why the default temperature value is set to 1.0 when do_sample is False by default, indicating that greedy sampling is the default setting.

### Expected behavior

1. No warning should be issued when do_sample=False and temperature=0.0.
2. Clarification on why the default temperature is 1.0 when do_sample=False.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Try to convert LlamaForCausalLM to ONNX with input_embeds as input 
Author: OswaldoBornemann
Pull request: no
Labels: name: bug

name: ONNX

name: Deployment

So in my case, the input is `input_embeds` instead of `token ids`. When I try to export LlamaForCausalLM to onnx like this way

```
batch_size = 1  # Adjust the batch size if necessary
sequence_length = 50  # Adjust the sequence length if necessary

input_ids = None
attention_mask = torch.ones((1, sequence_length)).to(device)  # Attention mask (ones as default)
position_ids = torch.arange(0, sequence_length).unsqueeze(0).to(device)  # Position IDs
past_key_values = None  # Assuming no past key values for initial export
inputs_embeds = torch.randn((1, sequence_length, 1792), dtype=torch.float32).to(device)
cache_position = torch.arange(0, sequence_length).to(device)  # Cache position (optional)
```

```
torch.onnx.export(
    model,                             # The model
    (input_ids,
    attention_mask, 
     position_ids, 
     past_key_values, 
     inputs_embeds, 
     cache_position),  # Inputs to the forward function
    "./onnx_dir/t2sllama.onnx",                       # Output ONNX file path
    export_params=True,                # Store the trained parameter weights
    opset_version=17,       # ONNX version to export to
    do_constant_folding=True,          # Whether to execute constant folding for optimization
    input_names=['attention_mask', 
                 'position_ids', 
                 'past_key_values', 
                 'inputs_embeds', 
                 'cache_position'
                 ],  # Input names
    output_names=['output'],          # Output names
    dynamic_axes={
        # 'inputs_embeds': {1: 'length'},
        'attention_mask': {1: 'attention_length'},
        'position_ids': {1: 'position_length'},
        'output': {1: 'length'}
    }  # Enable dynamic batch size and sequence length
)
```

The onnx can be successfully exported. However, I found that the onnx model change the inner dtype of `input_embeds`

```
for input_meta in sess.get_inputs():
    print(f"Input Name: {input_meta.name}, Type: {input_meta.type}, Shape: {input_meta.shape}")

Input Name: attention_mask, Type: tensor(float), Shape: [1, 'attention_length']
Input Name: position_ids, Type: tensor(int64), Shape: [1, 'position_length']
Input Name: past_key_values, Type: tensor(float), Shape: [1, 50, 1792]
Input Name: inputs_embeds, Type: tensor(int64), Shape: [50]
```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Do Transformers onnx export support the input of the Llama is the input_embeds?
Author: OswaldoBornemann
Pull request: no
Labels: name: bug

name: ONNX

Do Transformers onnx export support the input of the Llama is the input_embeds?

Comments:
cc @xenova regarding the onnx export :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
batch inference scales linearly with batch size when input is long
Author: platypus1989
Pull request: no
Labels: name: bug

### System Info

transformers.version=4.42.4

### Who can help?

@Gante

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Hi, I am noticing when running batch inference over Mixtral-8x7B-Instruct-v0.1, model seems to be scale nicely (sublinearly) with batch size if input size is small, but when input size gets large (more than 400 tokens), inference time start to become linearly against batch size.

Some sample code to reproduce what I am seeing

```
import torch
from time import time
import pandas as pd
model_id = 'mistralai/Mixtral-8x7B-Instruct-v0.1'

model = AutoModelForCausalLM.from_pretrained(
        model_id,
        torch_dtype=torch.bfloat16,
        device_map='auto',
    )
tokenizer = AutoTokenizer.from_pretrained(model_id)

def inference_time(input_size, batch_size):
    prompt = "how are you? "*input_size
    prompts = [prompt]*batch_size
    input = tokenizer(prompts)
    input_ids, attention_mask = torch.tensor(input['input_ids']).to(model.device), torch.tensor(input['attention_mask']).to(model.device)
    tic = time()
    with torch.no_grad():
        output = model(input_ids=input_ids, attention_mask=attention_mask)
    return time() - tic


input_sizes = []
batch_sizes = []
wall_time = []
for i in [1, 5, 10, 20, 50, 100]:
    for j in [1, 5, 10, 20, 50, 100]:
        input_sizes.append(i)
        batch_sizes.append(j)
        wall_time.append(inference_time(i, j))

pd.DataFrame({
    'input_size': input_sizes,
    'batch_size': batch_sizes,
    'inference_time': wall_time,
})
```
You can see from the output, the inference time scales sublinearly against batch size when input size is less than 10. Once the input size increased to more than 20, the inference time starts to scale with batch size somewhat linearly. 
<img width="389" alt="Screenshot 2024-09-10 at 12 29 11 AM" src="https://github.com/user-attachments/assets/3ffae2be-8f04-4e89-88b0-019c5296c947">



### Expected behavior

I was expecting batch inference's running time to be scaling sublinearly regardless of the input size to some extent.

Comments:
Btw I am running the experiment on an instance with 4 A100.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
convert_data2vec_audio_original_pytorch_checkpoint_to_pytorch.py works for data2vec 1.0 checkpoint but not data2vec 2.0
Author: Bogsamurai
Pull request: no
Labels: name: Audio

### System Info

transformers version: 4.41
Platform: Red Hat Enterprise Linux 8.4 (Ootpa)
Python version: 3.10.14 
PyTorch version  2.3+cu118 

### Who can help?

@sanchit-gandhi @SunMarc

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run [convert_data2vec_audio_original_pytorch_checkpoint_to_pytorch.py](https://github.com/huggingface/transformers/blob/main/src/transformers/models/data2vec/convert_data2vec_audio_original_pytorch_checkpoint_to_pytorch.py)
AttributeError: 'ModuleList' object has no attribute 'mlp'

### Expected behavior

Expected behavior is that model weights must be loaded via this function: recursively_load_weights(model, hf_wav2vec, not is_finetuned)
but error is:
![image](https://github.com/huggingface/transformers/assets/143995447/f5dd9286-5ecd-4872-b19e-a3be1fae04fb)


Comments:
cc @kamilakesbi 

cc @ylacombe

Hey @Bogsamurai, correct me if I'm wrong, but I'm pretty sure we don't support data2vec 2.0. There's seems to be some architectural differences between 1.0 and 2.0, according to [this](https://github.com/facebookresearch/fairseq/tree/main/examples/data2vec#data2vec-20).
Would you like to work on adding data2vec 2.0 support ?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
FP8 inference and FP8 KV cache
Author: SinanAkkoyun
Pull request: no
Labels: name: Cache

### Feature request

Hi!
Could anyone please help me with using HuggingFace models (LLaMa [or if LLaMa is difficult, MPT-7b]) with the TransformerEngine TE FP8 inference? We really need the speedup

https://github.com/NVIDIA/TransformerEngine/issues/199
This is a somewhat related issue to this topic.

### Motivation

Faster inference and more specialized tensor operations means less cost and less latency.

### Your contribution

I would really love to test suggestions out as I have temporary access to a H100 cloud GPU.
I am not sufficient in porting the models myself which is why I created this issue.

I really appreciate any help, thank you very much.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@SinanAkkoyun  have you find the solution how to use transformerengine with Llama?

Any updates?

Gentle ping @fxmarty 

Another ping @fxmarty. Could you nominate someone to take this over for you? 

cc @IlyasMoutawwakil 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[Distilbert] Torch jit trace failed with `load_in_8bit=True`.
Author: sihanwang41
Pull request: no
Labels: name: Quantization

name: bug

name: Deployment

### System Info

The trace failed with 
```
torch.jit._trace.TracingCheckError: Tracing failed sanity checks!
ERROR: Graphs differed across invocations!
```
version:
transformers==4.43.4
torch==2.1.0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Script to reproduce
```
import torch
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained('distilbert/distilbert-base-uncased')
model = AutoModel.from_pretrained("distilbert/distilbert-base-uncased", load_in_8bit=True,  torchscript=True,)

text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
with torch.inference_mode():
    model.eval()
    traced_model = torch.jit.trace(model, [encoded_input["input_ids"], encoded_input["attention_mask"]], check_trace=True)
```



### Expected behavior

success :)

Comments:
I'm not certain the two are compatible unfortunately, cc @SunMarc 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Generation tests: don't rely on main input name
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Precedes https://github.com/huggingface/transformers/pull/34174/files. 

Before merging the linked PR, we need to stop relying on `main_input_name` as in some cases (BLIP) the main input is not text but rather audio/video/image. So we fail to check generated text length

We can make some assumptions here:
- Decoder-only models always require text as main input, because we generate from given input id(s). So we can always get `input_ids` and use its shape in tests
- Encoder-decoder can take whatever modality input and still generate text. But we don't need the input length in that case to verify generated length
- We still need seq length in checking attentions/hidden states. But we don't have to try and infer it from inputs, because we already register all length in model tester. This PR introduces similar way of getting seq length as in `test_modeling_common`  + add a `text_seq_length`. The latter is for two models that are special, Reformer which changes seq length in train vs inference and GIT which looks like an old-style VLM where images are concated in modeling code
- 

Comments:
Add VITS training script example
Author: ylacombe
Pull request: yes

# What does this PR do?

Vits is a TTS model that have been supported in HF for a few months now. MMS-tts, Meta's models on a thousand languages, also uses VITS architecture.

This PR aims to add Vits training support, following [community interest on the hub](https://huggingface.co/facebook/mms-tts/discussions/1#653c1b47c0d69fe83ded4161).   

Vits is a peculiar model, a GAN, a VAE a flow-based, all at the same time. To train it, one needs a discriminator, which this PR adds to Transformers.

I'll comment on the architecture and on some modeling choices on the comments below.

I still need to write the description of the training code, but I believe the implementation choices I've made should have our core maintainers approval first!
 

cc @amyeroberts and @sanchit-gandhi !

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27244). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Do you fancy finishing this integration @ylacombe? It looks like we're almost there in terms of code clean-up, and it would be nice to round-off the VITS integration.

Alternatively, we can move the modelling code to your [`finetune-hf-vits`](https://github.com/ylacombe/finetune-hf-vits) repo, and maintain it there.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi, could you please merge this. Thanks.
Change Paligemma import logging to work with modular 
Author: yonigozlan
Pull request: yes

# What does this PR do?
Currently, the modeling file of Paligemma uses `from ...utils import logging`, and the processing file uses `import logging`.
That means if we want to use a modular file to add a model inheriting from Paligemma, depending on what import `logging` we add to the modular file, we either have: 
- An error as one logging is used to create a logger with `logging.get_logger` and the other with `logging.getLogger`
- A style issue as logging is imported from two different places in the same file

I'm not sure if we want to modify modular transformers to handle this, or if it's better to enforce the use of the same logging module across the files of a model. Wdyt @ArthurZucker 


Comments:
Request for a clear documentation for .generate()
Author: sedol1339
Pull request: no
Labels: name: Feature request

### Feature request

The `.generate()` function has a lot of parameters, for example `length_penalty` and `diversity_penalty`. However, the [documentation](https://huggingface.co/docs/transformers/v4.45.1/en/main_classes/text_generation#transformers.GenerationMixin.generate) of this function does not document a full list of parameters, hiding them in **kwargs. It says that "For an overview of generation strategies and code examples, check out the [following guide](https://huggingface.co/docs/transformers/v4.45.1/en/generation_strategies)". However, this guide also do not document a full list of parameters. It only says that "For the complete list of the available parameters, refer to the [API documentation](https://huggingface.co/docs/transformers/v4.45.1/en/main_classes/text_generation.md)." However, the link is broken. So, could you please document all the available parameters?

UPD I found the [full documentation](https://huggingface.co/docs/transformers/v4.18.0/en/main_classes/text_generation#transformers.generation_utils.GenerationMixin.generate) through Google in the earlier version of Transformers. Why it was removed?

### Motivation

The full documentation for `.generate()` is either missing or hidden somewhere

### Your contribution

No, sorry, I can't write a doumentation, on the contrary, I need it to understand how the library works.

Comments:
cc @gante

A core functionality of the library is essentially undocumented currently. This should be a lot higher priority.

cc @stevhliu as well
Access to model outputs inside LogitProcessor
Author: AdityaMayukhSom
Pull request: no
Labels: name: Feature request

name: Generation

### Feature request

![image](https://github.com/user-attachments/assets/cb57d5dd-6502-4b31-a948-bb46e535fea5)

The LogitProcessor __call__ method currently has access to the input_ids and the logits generated. It would be really helpful if it has access to the model output for that iteration.

### Motivation

I am trying to implement something similar to [Get To The Point: Summarization with Pointer-Generator Networks](https://arxiv.org/abs/1704.04368) along with transformers, especially Llama models. I am required to add the attention output to the model's generated logit for pointing mechanism, but I do not have access to the attention value inside logit processor. As seen in the picture, if the standard model output object can be passes to the logit processor, then the user can extract required details from the output object and use it to further process the generated logits.

![image](https://github.com/user-attachments/assets/4103b2f8-a09c-4a2e-8916-9ebac8962dd1)


### Your contribution

I can implement the required changed and create a PR if allowed.

Comments:
cc @gante
Sync token-classification pipeline with Hub spec
Author: Rocketknight1
Pull request: yes

One unusual change this time: The TokenClassificationPipeline put the docstrings in an odd place, with a decorator. As far as I can tell, this isn't necessary at all - I just put them in the normal place (which also un-breaks my compatibility checker).

cc @wauplin - there's an output discrepancy for the `zero-shot` pipeline, but I can't figure out what output format the Hub spec actually wants when `multi-label` is `True`. Where did that Hub spec come from?

Comments:
@Rocketknight1 these specs comes from https://github.com/huggingface/huggingface.js/blob/main/packages/tasks/src/tasks/zero-shot-classification/spec/output.json (PR https://github.com/huggingface/huggingface.js/pull/449). When we wrote them we tried to be as consistent as possible with the existing but if there are some discrepancies, we can also update the specs correctly. Some tasks are more popular and have been reviewed/reworked several times but the zero-shot classification is not one of them :confused: 

(This PR is ready, just on hold until the next release of `huggingface_hub` - will retest and merge then!)