noamaanMulla-03/transformers-issues huggingface dataset:
🚨🚨🚨 [`Quantization`] Store the original dtype in the config as a private attribute 🚨🚨🚨
Author: younesbelkada
Pull request: yes

# What does this PR do?

First step of an alternative design of https://github.com/huggingface/transformers/pull/26560 

For quantized models, instead of introducing a complex logic of retrieving the original weights dtype, I propose to simply add a private attribute `_quantization_original_dtype` in the config object. 

`to` method does not need to be touched here as `to` cannot be called on quantized models (but for GPTQ models you can call `to` to perform device placement only - **not** for dtype casting)

that way we could adapt #26560 to simply check if the config has the attribute `_quantization_original_dtype` which is the case only for quantized models, else retrieve the dtype by retrieving the dtype of the linear layer weights in a classic manner.

cc @LysandreJik 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Problem with eos_token_id not being used when add_eos_token=True is used
Author: ziggygram
Pull request: no

### System Info

transformers==4.35.dev0

I have been playing around with the new Mistral models/finetunes. For training, I believe it is needed to add "add_eos_token=True" on the tokenizer thats being passed to the SFTrainer so that the model still predicts eos_token at the end of the sequence and it's better than including it in the "text" column. I didn't see a lot of docs on templates so want to check I am not inventing this right?

I have noticed that the behavior works well on the regular Mistral model but is completely off on some other models like the OpenOrca fine-tune. While the eos_token exists in the tokenizer and is correctly categorized for the prompt_template, the tokenizer always uses the unknown token instead when tokenizing and "add_eos_token=True" (see screenshot below) which is definitely not right and leads to problems with never-ending generation after finetunes.

<img width="889" alt="Screenshot 2023-10-11 at 9 28 22 PM" src="https://github.com/huggingface/transformers/assets/83364633/b43a774f-b652-4654-ade3-92dc890660d4">


### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

model_name = "Open-Orca/Mistral-7B-OpenOrca"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True,add_eos_token=True)
tokenizer("Test")

### Expected behavior

It should output `{'input_ids': [1, 3735, 32000], 'attention_mask': [1, 1, 1]} `instead of` {'input_ids': [1, 3735, 0], 'attention_mask': [1, 1, 1]}` since `tokenizer.eos_token_id` is 32000.

Comments:
That is most probably because `normalized = True` for this added token. Thus the actual token that is known to the underlying fast tokenizer is `'▁
'` which is why the `
`  is not recognized and encoded as an unk token. 

I think it was fixed !!
[`FA2`] Cast to correct dtype
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes: https://github.com/huggingface/transformers/issues/26451

Currently performing bf16 fine-tuning with FA-2 leads to hidden states silently being casted in float16
As it is challenging to retrieve the original dtype of the model in case the model is quantized, I propose to store that dtype in a private attribute to be able to retrieve it conveniently without having to perform any sort of hack that gets the correct dtype if the model is quantized




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26560). All of your documentation changes will be reflected on that endpoint.

I agree we should make all hacks / changes with respect to FA2 modules inside them. However this might introduce multiple patches and other hacks for quantized modules. I think for now this approach is fine, but I agree we should go for a better one, as this would unblock some users for the next release, I left it as a TODO! 

cc @hiyouga are you able to Fine-tune in bf16 with this branch?

I also thought that we could retrieve the data type from `self.config.torch_dtype` of `LlamaFlashAttention2`.

PS. Although it may fail in such an edge case as Lysandre said, we usually have a consistent data type in training.

Closing this PR in favor of https://github.com/huggingface/transformers/pull/26846
[docstring] Fix bert generation tokenizer
Author: przemL
Pull request: yes

# What does this PR do?

Fix docstring for bert generation tokenizer.

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@ydshieh

Comments:
@ydshieh Could you please review the PR?
Thank you.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26820). All of your documentation changes will be reflected on that endpoint.
Hey! The warning is a general warning. `Left` padding is the usual recommendation, but the original Llama codebase (and Code-Llama is part of the Llama codebase) use a default `right` padding. Our goal is to have similar results out of the box (so right padding) but still allow users to have the best results and we thus give recommendation on the padding side.
Author: rafa852
Pull request: no

              Hey! The warning is a general warning. `Left` padding is the usual recommendation, but the original Llama codebase (and Code-Llama is part of the Llama codebase) use a default `right` padding. Our goal is to have similar results out of the box (so right padding) but still allow users to have the best results and we thus give recommendation on the padding side. 
There is a guideline: CodeLlama is the same as Llama. Would it be clearer if the tip section is copied over to CodeLlama?

_Originally posted by @ArthurZucker in https://github.com/huggingface/transformers/issues/26072#issuecomment-1729231672_
            

Comments:
log interval
Author: KwangryeolPark
Pull request: no
Labels: name: Feature request

### Feature request

I want you to add "log_interval" argument for logging every N iteration not a epoch.

### Motivation

Some fine-tuning task/dataset does not require much epochs such as CoLA. It only requires 2~3 epochs, which means the log data(ex metric) is produced 2~3 times.

I want more log during training with 2~3 epochs, so I hope you add some feature which logs every N iteration.

I already add the argument in my project just adding below code, but it is borthing to add every project whenever I create new project based on your codes.

`
if completed_steps % args.log_interval == 0:
# presented evaluation code in original code.
`

### Your contribution

`
if completed_steps % args.log_interval == 0:
                model.eval()
                samples_seen = 0
                for step, batch in enumerate(eval_dataloader):
                    with torch.no_grad():
                        outputs = model(**batch)
                    predictions = outputs.logits.argmax(dim=-1) if not is_regression else outputs.logits.squeeze()
                    predictions, references = accelerator.gather((predictions, batch["labels"]))
                    # If we are in a multiprocess environment, the last batch has duplicates
                    if accelerator.num_processes > 1:
                        if step == len(eval_dataloader) - 1:
                            predictions = predictions[: len(eval_dataloader.dataset) - samples_seen]
                            references = references[: len(eval_dataloader.dataset) - samples_seen]
                        else:
                            samples_seen += references.shape[0]
                    metric.add_batch(
                        predictions=predictions,
                        references=references,
                    )

                eval_metric = metric.compute()
                logger.info(f"epoch {epoch}: {eval_metric}")

                if args.with_tracking:
                    accelerator.log(
                        {
                            "accuracy" if args.task_name is not None else "glue": eval_metric,
                            "train_loss": total_loss.item() / args.log_interval,
                            "train_mem": total_memory / args.log_interval,
                            "epoch": epoch,
                            "step": completed_steps,
                            "lr": optimizer.param_groups[0]["lr"],
                        },
                        step=completed_steps,
                    )
                total_loss = 0
`

Comments:
@KwangryeolPark Please assign this issue to me
Request support about the flash attention 2 for the gpt_bigcode models
Author: DongHande
Pull request: no

When I run the code,  
```
checkpoint = "bigcode/starcoder"
model = AutoModelForCausalLM.from_pretrained(checkpoint, use_flash_attention_2=True)
```
it shows that "ValueError: The current architecture does not support Flash Attention 2.0. Please open an issue on GitHub to request support for this architecture: https://github.com/huggingface/transformers/issues/new". 

The reason is that the gpt_bigcode (https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt_bigcode/modeling_gpt_bigcode.py) does not support flash attention 2 yet. 
I open this issue to request support about the flash attention 2 for the gpt_bigcode models. 

Comments:
Hey! Will be fixed by #26479

Hope it will be soon fixed and merged.
Update logits_process.py docstrings to clarify penalty and reward cases (attempt #2)
Author: larekrow
Pull request: yes

This PR fixes point 3 of https://github.com/huggingface/transformers/issues/25970 by clarifying the penalty and reward cases for RepetitionPenaltyLogitsProcessor and EncoderRepetitionPenaltyLogitsProcessor within the docstrings.

PR https://github.com/huggingface/transformers/pull/26129 was the original copy, but I have accidentally deleted my repo that submitted the PR, so I cannot reopen that PR 🙇 

@gante, for your review please.

Comments:
Could you run `make fixup` to pass the failing test

Hey @ArthurZucker, I tried to run `make fixup` but encountered an error in `repo-consistency`. 

```
Traceback (most recent call last):
  File "/home/users/user/temp/transformers/utils/update_metadata.py", line 337, in <module>
    check_pipeline_tags()
  File "/home/users/user/temp/transformers/utils/update_metadata.py", line 316, in check_pipeline_tags
    model = model[0]
IndexError: tuple index out of range
make: *** [Makefile:44: repo-consistency] Error 1
```

Anyway I thought I should not be encountering this in the first place since I only modified docstrings in a single file. So, I ran `black src/transformers/generation/logits_process.py` which gave:
```
All done! ✨ 🍰 ✨
1 file left unchanged.
```

Even `ruff src/transformers/generation/logits_process.py --fix` does nothing. Not sure what I should do to pass the failing test? I'm using `black==23.9.1` and `ruff==0.0.292` if that helps.

Hey! we have "ruff>=0.0.241,<=0.0.259"`  so the version you are using is probably too far! Try downgrading or doing something like `pip install -e ".[quality]"` and then run `make style`

Thanks for the guidance @ArthurZucker! The test has passed 😄 

Cool thanks for the contribution 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26784). All of your documentation changes will be reflected on that endpoint.
StoppingCriteria "scores" is always None
Author: jbmaxwell
Pull request: no
Labels: name: bug

### System Info

I've written a custom StoppingCriteria subclass and I'm trying to utilize the `scores` in my decision logic, but I'm finding that `scores` is always `None`. Is that intentional?

### Who can help?

@patrickvonplaten, @Narsil, @gante

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
class TopPredictionOutsideTargetSetStoppingCriteria(StoppingCriteria):
    def __init__(self, priority_tokens_ids: list):
        self.priority_token_ids = priority_tokens_ids

    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
        print(f"TopPred SCORES? {scores}, input_ids: {input_ids}")      # <--- "scores" is None but "input_ids" is correct
        top = torch.topk(scores, 1, dim=1).indices[0]
        if not top in self.priority_token_ids:
            return True
        return False
```

### Expected behavior

Since the function indicates `scores` as an input, I'd expect it to be a non-null value.

Comments:
Hey @jbmaxwell, 

Sadly we don't have the time to dive into problems related to customized code. For this could you maybe try to use the forum instead? 

https://discuss.huggingface.co/

Looking at this line: https://github.com/huggingface/transformers/blob/ac98a88fbc6377f93e8b7fbd244b0c3331bb82a0/src/transformers/generation_utils.py#L1738 `scores` should not be `None` though.

Thanks for the reply. I understand about custom code. I just wanted to report that `scores` is `None`, and to confirm that this _shouldn't_ be the case.  I'll see if I can figure out how that's happening. 

> Thanks for the reply. I understand about custom code. I just wanted to report that `scores` is `None`, and to confirm that this _shouldn't_ be the case. I'll see if I can figure out how that's happening.

hi,i have the same issue,how did you solve this problem？

same here. Any workaround?

Hey @CuteBadEgg @takacsg84 👋 

Have you tried adding `return_dict_in_generate=True, output_scores=True`? These two are needed to keep track of the scores, which then might be used in stopping criteria :)

(I'm updating the docs to reflect this need)

@gante this ended up to work. Initially I only had `output_scores=True` but `return_dict_in_generate` has to be set as well just as you mentioned.
🚨🚨 Generate: change order of ops in beam sample to avoid nans
Author: gante
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/26332 -- see [this](https://github.com/huggingface/transformers/issues/26332#issuecomment-1764736970) comment for a full explanation.

TL;DR: in `beam_sample`, `logits_warper` are now applied BEFORE adding the beam scores. We have been postponing this change to avoid introducing output differences, but the truth is that the order of operations causes issues (e.g. [1](https://github.com/huggingface/transformers/issues/26332) [2](https://github.com/huggingface/transformers/issues/22914))

This is technically a bug fix (`beam_sample` is unusable for long generations with `temperature < 1.0` before this change), hence the lack of a deprecation cycle. However, it may alter some generated outputs (short generations with low temperature), hence the 🚨🚨.

Please note that other common operators, like `top_p` and `top_k`, are unaltered by this change.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Batch Inference for Streaming generation strategy for transformer.generate()
Author: AyushVachaspati
Pull request: no

### Feature request

Hi,

I am working on implementing a service that receives inference requests form users and sends a stream of responses (Tokens as they are generated). The TextStreamer and TextIteratorStreamer  classes are good options to do that.
https://huggingface.co/docs/transformers/v4.31.0/en/internal/generation_utils#transformers.TextStreamer

The issue is that they only support one inference request at a time, which wastes a lot of GPU resources. Is there some way to run batches of requests using a stream interface, like "BatchTextIteratorStream" or something in that vain. This way the responses can be generated on a batch of requests. The generator would push list of tokens for each batch for each iteration of the model (in context of LLMs producing 1 token at a time) and the user can then process those to service their requests. 

### Motivation

Streaming responses are very useful for large and slow Chat LLMs. These models have large latencies. By batching the requests we can improve token throughout and improve GPU utilization, and also service more requests at the same time. 

### Your contribution

I can help write the Batch Streamer class to implement the functionality, but I'm not too familiar with the internals of the Transformers codebase, so I need someone to point me in the right direction. 
I'm motivated to contribute a substantial amount of time to implement this feature as it would be a great help for my project. 

Comments:
I found that I can Inherit the TextStreamer class and implement my own put() and end() methods for the functionality I need. But the general functionality for batch inference might still be useful for the community.  

cc @gante 

@AyushVachaspati Thank you for opening the issue 👍 

We want to add it eventually, but we are prioritizing a structural `generate` refactor before we go forward with this feature :)

BTW, since you're serving generation requests, have you looked at our [text-generation-inference](https://github.com/huggingface/text-generation-inference) server-grade library? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@AyushVachaspati @gante Has this issue gained any traction yet? Also interested in a streaming solution for batched inference! 

@ddl-avanitanna No bandwidth to expand it yet, I'm afraid :) The pointer to [TGI](https://github.com/huggingface/text-generation-inference) is still my suggested alternative
🚨 [Bug] Text-to-Audio pipeline is broken for Speech T5 TTS
Author: Vaibhavs10
Pull request: no

### System Info

- `transformers` version: 4.33.2
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.2
- Safetensors version: 0.3.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.1+cu118 (False)
- Tensorflow version (GPU?): 2.13.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.7.2 (cpu)
- Jax version: 0.4.14
- JaxLib version: 0.4.14
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@sanchit-gandhi @ylacombe

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Speech T5 TTS expects speaker embeddings to be passed for understandable speech to be produced. At the moment in the `Text-to-Audio` pipeline we just call the model (processor) + generate without any speaker embeddings. This results in garbage audio as an output.

Full-repro here: https://github.com/Vaibhavs10/scratchpad/blob/main/transformers_speecht5_tts_repro.ipynb

### Expected behavior

Text-to-Audio pipeline for SpeechT5 TTS checkpoints outputs audio that is understandable.

Comments:
cc: @Narsil for vis.

@ylacombe Maybe ?

Hey @Narsil and @Vaibhavs10,
At the moment, you have to pass the speaker embeddings as a forward argument:

```python
speaker_embeddings = torch.tensor(embeddings_dataset[7306]["xvector"]).unsqueeze(0)
forward_params = {
    "speaker_embeddings": speaker_embeddings,
}
speech = speecht5_tts("Hello, my dog is cute!", forward_params = forward_params)
```

For reference, see [this snippet](https://huggingface.co/suno/bark-small/discussions/4#64f5c7c11c2c6f58c91c8ad1) for Bark.

Adding a default speechT5 speaker embedding in the pipeline would imply model-specific code inside the pipeline code, which is [something](https://github.com/huggingface/transformers/pull/24952/#discussion_r1280336582) we'd like to avoid rn

Happy to hear about your thoughts on this, though.



Hey @ylacombe - I think we should explicitly raise an exception here asking the user to provide a speaker embedding. Currently, the pipeline would work without any exception or warning - outputting gibberish audio. This results in a significantly degraded Developer Experience.

I agree @Vaibhavs10! But instead of adding this to the pipeline, let's add this to [_generate_speech](https://github.com/huggingface/transformers/blob/6da93f5580e109fad5f7b523cf2b6e8a5bafb623/src/transformers/models/speecht5/modeling_speecht5.py#L2532) from `modeling_speecht5.py` since it is an issue specifically related to the model.

I'll make a PR this afternoon if that's okay with you

Agreed! and thanks for the PR 🤗