noamaanMulla-03/transformers-issues huggingface dataset:
Constraint.advance() docstring is slighly incorrect
Author: alex-calderwood
Pull request: no
Labels: name: bug

### System Info

according to the implementations of the Constraint ABC, the [advance() ](https://github.com/huggingface/transformers/blob/c1357834e86e71792d1cad06a998f5ebf6e07126/src/transformers/generation/beam_constraints.py#L49)function can return either a single token id (as the docstring indicates) or a list of token ids that would advance the beam search. It should be updated to reflect this.

Here is an example fix:

```
@abstractmethod
def advance(self):
    """
    When called, returns the token(s) that would take this constraint one step closer to being fulfilled.

    Return:
        token_ids (Union[int, List[int], None]): 
            - A single token ID (int) that advances the constraint
            - A list of token IDs that could advance the constraint
            - None if the constraint is completed or cannot be advanced
    """
```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Just a docstring change.

### Expected behavior

Should be updated for accuracy, easy of implementing this abstract base class. 

Comments:
cc @gante 

@alex-calderwood thank you for opening this issue and, more importantly, for submitting a fix to it 🤗
Move layer_idx from a layer property to function argument.
Author: siddartha-RE
Pull request: no
Labels: name: Feature request

### Feature request

Currently the layer_idx is recorded in the attention module of each `LlamaDecoderLayer`. This has the unfortunate side effect that the layers cannot easily be moved around or reused within the layer list. It seems simple enough to pass in the layer index as part of loop over layers in the forward pass. That way the layers once again will be decouple from their position information.

Backward compatibility could be preserved by still accepting the argument in the constructor but defaulting it to None and then just ignoring it in favor of the passed forward argument.

### Motivation

The motivation is to allow for simple layer stacking (like we have been seeing with pass through merged models) at inference time without actually expanding the memory usage of the model.

### Your contribution

I am happy to send a PR. Seems simple enough.

Comments:
I don't mind taking a look at a PR, feel free to ping me! 🤗 

Would love to see this done!

@ciaran-regan-ie (pinging you as an extension of the interaction in #32843) The position of the layer is important to the model, to the point we need to load the right weights to the right layer to reproduce the original model implementations. As such, at first glance, it makes sense for the layer to hold its own index.

When moving/shuffling/stacking layers, can't you update the layer index on your end? Is it a technical limitation? 

Given what I wrote above and that the most common case it to not move layers, I'm struggling to see the benefits of adding another parameter to the signature of `forward` :)

@gante Thank you. 

I agree that this is a pretty niche use case and I have been able to work around it by updating the layer index on my end!
Model outputs are not passed to compute_metrics functions completely
Author: Johnny785
Pull request: no

### System Info

python version 3.8.5, transformers version 4.19.1

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Create a model class based on a base model class in transformers
2. Overwrite its forward method to return the logits, the scores, and another output that is a tensor and has a 0 dimension higher than the batch size(such as 3 times the batch size)
3. Initialize a Trainer class to train the model as usual, pass in a custom compute_metrics function
4. In the custom compute_metrics function, print out the custom output as outlined above from EvalPrediction.predictions, and check its dimensions

### Expected behavior

I customized my forward function to return an input field in the dictionary, which is of size 5BxC where B is the batch size. Other metrics such as loss and logits are BxC or B. However, in the compute_metrics function I customized, I received only BxC data entries from the EvalPrediction.predictions attribute for the input I returned in forward. Other fields that have the first dimension equal to B are passed in correctly. When I changed my input field shape to Bx5xC, it is also passed in correctly.

I read through the code that calls the compute_metrics function, however, I did not find a piece of code that trims the output returned from the model. After experimenting, I found that it's not taking mean, taking sum or taking max.

I would really appreciate an explanation as to why this happens, thank you for your assistance

Comments:
Hi @Johnny785, thanks for opening an issue! 

So that we can best help you, could you: 
* Provide more running environment information: run `transformers-cli env` in the terminal and copy-paste the output
* Provide a minimal code reproducer we can run to replicate the issue. We get many issues and PRs opened per day, so do not have the time to translate written descriptions into code. Making it as easy as possible for us to run and debug is the surest way of us being able to help. 

As this involves custom code and overriding standard methods, this is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
is_flash_atten_avaiable were wrong
Author: lucasjinreal
Pull request: no

I am on V100, but this function gives me True, it just detects if flash-atten installed or not but not considered the GPU support or not.

Consider add GPU identificate as well? (compute capcitiy)

Comments:
Hi @lucasjinreal, thanks for opening this feature request! 

This is outside the scope of `is_flash_attention_available`. The `is_xxx_available` functions are meant to provide boolean controls checking whether a package is installed in an environment, which is exactly what it does. 

Support for different hardwares is hard to track and this behaviour would likely be difficult to maintain (we'd have to monitor downstream changes and have many controls for all the different versions and hardware combinations and their compatibilities)

Is there any way to count hardware in? from transfomrers utils or something.

this is a common, since flashatten even installed can not be calculate on V100

@lucasjinreal Unfortunately there's so many different types of hardware, and with that some many different types of software e.g. cuda kernels, it's not feasible to reliably detect and monitor this - as well as keep up to date with any changes in their support or new hardware being release. In addition, checking hardware itself isn't as straightforward as install libraries and likely to be brittle. It's better to let each library e.g. flash-attention manage their own support and documentation, as this will be more robust than trying to do everything automatically. 

so, it is clear that is_flash_attn_avaiable only detect if flashattn installed or not?

Yes, it will detect if it is installed in the environment, but it won't tell you whether it's compatible with the hardware you're running on or other libraries 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
generate: missing `to` in DoLa body, causing exceptions in multi-gpu generation
Author: gante
Pull request: yes

# What does this PR do?

See title.

Fixes #31996 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32856). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

(unrelated CI failure, I'll inspect what's wrong with that test tomorrow)
Add Mamba2
Author: vasqu
Pull request: no
Labels: name: New model

name: Feature request

### Feature request

Support of a new version of the mamba architecture. See paper @ https://arxiv.org/abs/2405.21060. Base code has been released in their repo https://github.com/state-spaces/mamba.

### Motivation

It's still very competitive in NLP (according to the benchmarks) but also a good chunk faster due to new parallelism mechanisms.

### Your contribution

Not sure, I could look into it but it's definitely a bigger undertaking :) 

Comments:
cc @ArthurZucker 

https://github.com/huggingface/transformers/issues/29631

Hopefully that conversion script can be re-used to create the `mamba2*-hf` models

cc @3outeille do you want to have a go at this? 

@3outeille I would love to help out!

Glad to be @ when a PR is opened :eyes: 

Not sure if this counts as ad but I've created my own HF-compatible version of Mamba2: https://github.com/vasqu/mamba2-torch

Feel free to remove it if this is unwanted. 
- It's still very experimental 
- It doesn't support the hybrid variant with attention blocks. 
- I haven't tested too much so it might actually be crap. 
- It can also be used as a reference for an actual transformers implementation (?)

Hybrid variant will probably be here, as it's Jamba like

@vasqu Awesome work! Sorry for the late response, I was busy finishing up a paper. I will take a look tomorrow so we can work together and finalize it. Once Mamba2 is done, we can move on to Mamba2Hybrid.



@vasqu Can you make me a collaborator on the repo you shared so I can open up PRs?


@Adibvafa No worries and good luck on the paper! Let's move the discussion elsewhere: Do you have discord? Add me `_vasquez` or contact me via mail `a.vlasjuk@t-online.de`.

I think we should directly work on a fork. I want the repo above to stay as a standalone modular thing for now.

Hi everyone, 

Thanks so much for all the effort being put into porting mamba2 in transformers! I'm really excited about this. Just checking in to see if there's any update or estimated timeline on when the PR might be created? Thank you!

@pglorio Hey there, there's no specific timeline but @Adibvafa and I will start working on it soon, hopefully in the next couple of days :)

No idea how long this will take though. I can tag you when we open our PR (if you want to).

@vasqu Thank you for the reply, and yes, I'd be grateful if you can tag me when you guys will open your PR. Thanks again!

[Codestral Mamba](https://mistral.ai/news/codestral-mamba/) just got released and it uses the Mamba2 architecture, with untied embeddings unlike every other Mamba2 models, which use tied embeddings.

We are adding Codestral Mamba with @molbap, it's full mamba layers so keep #32027 open ! 🫂 

Closing this for now since it has been added recently with #32080 but if wanted hybrid variants can be worked on top now and maybe #32027 can be used as reference :)
model.generate slower than model forward call
Author: geekifan
Pull request: no
Labels: name: bug

### System Info

transformers=4.44.0
python=3.11
cuda=12.4

### Who can help?

@zucchini-nlp

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I need to get the hidden states when the model outputs the next token. I compare model.generate with model forward and find that the speed of model forward call is faster than model.generate.

Example:
Model: llava next
Prompt: \<image\>\n Summarize it in one word:
model forward call:
```
inputs = processor(input_prompts, images=batch['image'], return_tensors="pt", padding=True).to(device)
emb = model(**inputs, output_hidden_states=True, return_dict=True).hidden_states[-1][:, -1, :]
```
model.generate:
```
inputs = processor(input_prompts, images=batch['image'], return_tensors="pt", padding=True).to(device)
emb = model.generate(**inputs, max_new_tokens=60, output_hidden_states=True, return_dict_in_generate=True).hidden_states[0][-1][:, -1, :]
```
model forward call can process 1000 samples in 2 minutes but model.generate needs 20 minutes which is 10x slower than model forward call.

### Expected behavior

model.generate should have the same speed as model forward call

Comments:
`model.generate` generates up to 60 tokens when `max_new_tokens=60`, while `model(...)` only generates the next 1 token.
Setting `max_new_tokens=1` will make things about equal.

Right, calling generation with only one token should take equal time. Let me know if it doesn't

@zucchini-nlp @yonikremer Thanks for your reply!  Taking `max_new_tokens=1` solves the problem. The speed is roughly equal using both methods. Sorry for my misunderstanding of `max_new_tokens`. I thought it would always generate one word/token if I told it to do so no matter what `max_new_tokens` is.
Update encode function slowTokenizer vs FastTokenizer
Author: hackyhacka
Pull request: no

Hello,

I am trying to build a personalized tokenizer on top on GPT2 Tokenizer, so that's why I am modifying the code. I want to build encode words following a specific manner.

The code for the slow tokenizer is here:
https://github.com/huggingface/transformers/blob/a695c18649fc6ab4b1fb1d9c8cfa9258c5908e2a/src/transformers/models/gpt2/tokenization_gpt2.py

The code for the fast tokenizer is here:
https://github.com/huggingface/transformers/blob/a695c18649fc6ab4b1fb1d9c8cfa9258c5908e2a/src/transformers/models/gpt2/tokenization_gpt2_fast.py


To do pre-tokenization for the slowTokenizer is use "_tokenize()" method.

My question is how to do the same for the fast tokenizer? 


Comments:
Hi @hackyhacka, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
How to use 【examples/pytorch/contrastive-image-text】 to inter  inference
Author: rendaoyuan
Pull request: no
Labels: name: Feature request

### Feature request

I have reviewed the training code for CLIP and successfully executed it. Now, I want to use the obtained model for inference testing.


### Motivation

I would like to test the performance of the model I have trained.


### Your contribution

I hope I can get a example script to inference testing like below script :

python examples/pytorch/contrastive-image-text/run_clip.py \
    --output_dir ./clip-roberta-finetuned \
    --model_name_or_path ./clip-roberta \
    --data_dir $PWD/data \
    --dataset_name ydshieh/coco_dataset_script \
    --dataset_config_name=2017 \
    --image_column image_path \
    --caption_column caption \
    --remove_unused_columns=False \
    --do_train  --do_eval \
    --per_device_train_batch_size="64" \
    --per_device_eval_batch_size="64" \
    --learning_rate="5e-5" --warmup_steps="0" --weight_decay 0.1 \
    --overwrite_output_dir \
    --push_to_hub

Comments:
Hi,

Inference can be done as shown [here](https://huggingface.co/docs/transformers/en/model_doc/clip#usage-tips-and-example) (just pass the folder containing your weights and config.json to the `from_pretrained` method).
Revert "Add Descript-Audio-Codec model"
Author: kamilakesbi
Pull request: yes

Reverts huggingface/transformers#31494

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32876). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
bump tokenizers to support latest release
Author: winglian
Pull request: yes

# What does this PR do?

tokenizers is currently @ 0.20 https://github.com/huggingface/tokenizers/releases

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
cc @ArthurZucker
Q-GaLore Support
Author: fakerybakery
Pull request: no
Labels: name: Feature request

name: optimization

### Feature request

Add support for https://github.com/VITA-Group/Q-GaLore (https://arxiv.org/abs/2407.08296)

### Motivation

Q-GaLore allows more memory-efficient training

### Your contribution

M/A

Comments:
cc @SunMarc 

Hey @fakerybakery, thanks for the interest ! Ihave opened a [PR](https://github.com/huggingface/transformers/pull/31936) for that. I will try to finish it soon !
Add __repr__ for Conv1D
Author: AaronZLT
Pull request: yes

Add a simple representation for Conv1D. This allows the print (or the output) of the model's info has a better description for Conv1D.

Comments:
Hi~ @ArthurZucker
The CIs are green now with this new PR.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32425). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Author: twers1
Pull request: no

![image](https://github.com/huggingface/transformers/assets/102418053/f1f9b55c-ccad-438d-8add-40a86506e21f)

i have a python code: 

```
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

device = 'cpu'

model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
model.to('cpu')
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")

vegeterian_recipe_prompt = """### Instruction: Act as a gourmet chef. 
I have a friend coming over who is a vegetarian.
I want to impress my friend with a special vegetarian dish. 
What do you recommend?

Give me two options, along with the whole recipe for each.

 ### Answer:
 """

encoded_instruction = tokenizer(vegeterian_recipe_prompt,  return_tensors="pt", add_special_tokens=True)

model_inputs = encoded_instruction.to(device)

generated_ids = model.generate(**model_inputs, max_new_tokens=500, do_sample=True, pad_token_id=tokenizer.eos_token_id)
decoded = tokenizer.batch_decode(generated_ids)
print(decoded[0])
```

Comments:
Actually this should not be trigger as the tokens that are added are added at the beginning of the vocab. Thanks for reporting!

> Actually this should not be trigger as the tokens that are added are added at the beginning of the vocab. Thanks for reporting!

how can I get an answer from artificial intelligence then? he sends me this and doesn't finish the command "python openai.py"

It will be fixed by #26570 ! Otherwise it's just a warning should not impact your code 

I still have this issue when using the library on kaggle

![image](https://github.com/user-attachments/assets/ab7124e9-4be8-4e7e-b1f6-aed7d06c498a)


I think we recently removed thewarning: #32138