noamaanMulla-03/transformers-issues huggingface dataset:
🚨 🚨  Raise error when no speaker embeddings in speecht5._generate_speech
Author: ylacombe
Pull request: yes

# What does this PR do?

Following the discussion in #26401, this adds a warning when using `_generate_speech` without speaker embeddings. 
If necessary, and after discussion, we could throw an error instead of a warning!

<!-- Remove if not applicable -->

Fixes #26401



## Who can review?

cc @Vaibhavs10 @sanchit-gandhi and @ArthurZucker 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for the reviews @sanchit-gandhi and @Vaibhavs10 !

I'll request @ArthurZucker's reviews on this tiny PR ! :hugs: 

Hey @ArthurZucker, thanks for the review!

I'm not an expert on SpeechT5, but as @sanchit-gandhi said [here](https://github.com/huggingface/transformers/pull/26418#discussion_r1338953648):

> it's possible that people are using fine-tuned versions of SpeechT5 without speaker embeddings, so throwing an error here would block them.

Even more, even if it is fine-tuned with speaker embeddings, the default speaker embeddings for one version might be different for the default speaker embedding of another version.

In anyways, loading a default speaker embedding is done [here](https://huggingface.co/lysandre/text-to-speech-pipeline/blob/main/tts.py#L28-L30), with this snippet:


```python
from datasets import load_dataset

if speaker_embeddings is None:
  embeddings_dataset = load_dataset("Matthijs/cmu-arctic-xvectors", split="validation")
  speaker_embeddings = torch.tensor(embeddings_dataset[7305]["xvector"]).unsqueeze(0)
```

I'm not sure using `datasets` is compatible with `transformers`. How shoud we deal with this if we agree on a default speaker embedding ?

The fine-tuned models also require passing a `speaker_embedding` btw: https://huggingface.co/learn/audio-course/chapter6/fine-tuning#speaker-embeddings

So the expected behaviour would always be to pass a `speaker_embedding` right? @sanchit-gandhi 

You can register a non persistent buffer (only if the default is always the same) 

Using a default embedding makes sense if we can do it in a non-breaking way, such as with the non-persistent buffer

At the end of the day, I'll raise an error, for those reasons:
- Using a default embedding with a non-persistent buffer would be breaking
- The code leaves the impression that a speaker embedding is necessary anyways, even when finetuning: https://github.com/huggingface/transformers/blob/aa7a7b09a8afff1983c96c9a62ca8e5c2cd3b002/src/transformers/models/speecht5/modeling_speecht5.py#L733-L738
- we're moving away from warnings



Hey @sanchit-gandhi , @Vaibhavs10  and @ArthurZucker , feel free to quickly check the PR whenever you have time!

Nice, merging!
Fix wrong index for sequence length in Falcon
Author: fxmarty
Pull request: yes

As per title, please refer to https://github.com/huggingface/transformers/blob/db611aabee863cc5b1fdc22dcec5ce8e6c3e3b36/src/transformers/models/falcon/modeling_falcon.py#L1110

Comments:
Thank you @Rocketknight1 

Nevermind, I forgot about the reshaping happening in the modeling code itself. So the current index in `prepare_inputs_for_generation` is correct.
[docstring] Fix docstring for LukeConfig
Author: louietouie
Pull request: yes

Part of Docstring Sprint #26638 

One small bug I noticed, which can be seen in my first commit, was that check_docstrings.py added a duplicate default specification for use_entity_aware_attention, which I deleted in my second commit.

## Before submitting
- [Y] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [Y] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [Y] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [N] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [N] Did you write any new necessary tests?

Comments:
@ydshieh , thanks! I waited for the 7 CI tests to be green before tagging.
Fixed a typo in mistral.md
Author: DTennant
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->



## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Documentation: @stevhliu and @MKhalusova



Comments:
Translating `en/internal` folder docs to Japanese 🇯🇵
Author: rajveer43
Pull request: no
Labels: name: WIP

<!--
Note: Please search to see if an issue already exists for the language you are trying to translate.
-->

Hi!

Let's bring the documentation to all the Japanese-speaking community!

Who would want to translate? Please follow the 🤗 [TRANSLATING guide](https://github.com/huggingface/transformers/blob/main/docs/TRANSLATING.md). Here is a list of the files ready for translation. Let us know in this issue if you'd like to translate any, and we'll add your name to the list.

Some notes:

* Please translate using an informal tone (imagine you are talking with a friend about transformers 🤗).
* Please translate in a gender-neutral way.
* Add your translations to the folder called `ja` inside the [source folder](https://github.com/huggingface/transformers/tree/main/docs/source).
* Register your translation in `ja/_toctree.yml`; please follow the order of the [English version](https://github.com/huggingface/transformers/blob/main/docs/source/en/_toctree.yml).
* Once you're finished, open a pull request and tag this issue by including #issue-number in the description, where issue-number is the number of this issue. Please ping @stevhliu and @MKhalusova for review.
* 🙋 If you'd like others to help you with the translation, you can also post in the 🤗 [forums](https://discuss.huggingface.co/).


<!--
Keep on adding more as you go 🔥
-->


Comments:
Baichuan model is not working
Author: Fangzhou-Ai
Pull request: no

### System Info

using latest vllm by ```pip install vllm``` directly.

GPU: A100 @ 80 GB
System: Debian 10

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Simple test using current example
```bash
python benchmarks/benchmark_latency.py --trust-remote-code \
 --model baichuan-inc/Baichuan-13B-Base \
 --input-len 256 --output-len 256 \
 --batch-size 32 \
 --tensor-parallel 2
```

### Expected behavior

Error info : ```AttributeError: 'BaichuanTokenizer' object has no attribute 'sp_model'```

The error is due to the transformers 4.34.0, which is required or Mistral but imconpatiable with current vllm Baichuan model.


Comments:
Hey! This is because of a recent update to `transformers`, the model on the hub should be updated to move the call of `super()__init__` at the end of the `__init__()` of the model
Falcon: remove cache reformatting in the modeling code
Author: fxmarty
Pull request: yes

When Falcon has been ported to transformers, it appears that methods `_convert_to_rw_cache` and `_convert_cache_to_standard_format` were directly called in the model, which is NOT what is traditionally done in Transformers, with the precedent of Bloom: https://github.com/huggingface/transformers/blob/0a55d9f7376f72ad3ff296d4249840021b03bcc4/src/transformers/models/bloom/modeling_bloom.py#L853 & https://github.com/huggingface/transformers/blob/0a55d9f7376f72ad3ff296d4249840021b03bcc4/src/transformers/models/bloom/modeling_bloom.py#L949

@Rocketknight1 I am wondering if it is fine to put back the cache reordering in `prepare_inputs_for_generation` & `_reorder_cache`? Having it in the modeling is a bit bad for people who want to rewrite their own generation, export the model, etc. It is also inconsistent with the information in `FALCON_INPUTS_DOCSTRING`.

I noticed this working on the ONNX export, where it is not really meaningful to have those ops in the model itself. Another solution is to monkey patch transformers, but I believe this should be upstream.

Related: https://github.com/huggingface/transformers/issues/26097

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26199). All of your documentation changes will be reflected on that endpoint.

Hi @fxmarty, I think refactoring this out of the model code to put it in `prepare_inputs_for_generation` is fine!

I don't think that should break anything (cc @Narsil in case it affects TGI, but I don't think it should)

If you don't break `transformers`, you don't break `tgi` :).

Modifying repos themselves might break TGI though (because there's some running code relying on config.model_type at least).


This shouldn't need any modifications to repos/weights, thankfully!

Thank you @Rocketknight1, let me know if I should run any slow tests!

Left a couple of comments - I can run the slow tests afterwards, once they're resolved!

Slow tests look good to me, other test failures look like they're unrelated. Pinging @arthurzucker for core maintainer review, though - especially since there's some special-casing added to core code here!

WDYT @LysandreJik @ArthurZucker ?

@ArthurZucker well it is more the PR adding Falcon that was breaking in that regard with the common practice set by bloom. No other models than Falcon reformat the KV cache in the modeling itself. Why is that?

@LysandreJik @ArthurZucker Let me know if I should close this PR.

Yeah I don't think we should break this at this point and would rather way for the cache refactoring WDYT?
Generate: improve docstrings for custom stopping criteria
Author: gante
Pull request: yes

# What does this PR do?

Addresses https://github.com/huggingface/transformers/issues/18068, and a few other GH issues with the same problem

When a user defines a custom stopping criteria that makes use of the `scores` input, if `return_dict_in_generate=True, output_scores=True` are not passed, an exception will be thrown (`scores` is `None`). This is because we only keep track of `scores` when those two flags are passed.

I've added this requirement in several places in the docstrings, to reduce the odds of being missed.

For completeness: I've also considered always keeping track of scores. However, this adds a non-negligible overhead -- in BF16 for the 7B Mistral model, we would need an extra ~50MB per 1000 generated tokens (and batching would act as a multiplier here). Ditto for 4-bit BNB. Since custom stopping criteria are seldom used, this didn't look like a good tradeoff :)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Model training with torch_dtype=torch.bfloat16 is possible?
Author: cnut1648
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: Linux-5.10.184-174.730.amzn2.x86_64-x86_64-with-glibc2.26
- Python version: 3.9.16
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

From #23165 and specifically this comment https://github.com/huggingface/transformers/issues/23165#issuecomment-1536439098 of @sgugger , it seems that we should not set `torch_dtype` during training. However I think this is possible. See for example the following script
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
model_name = "gpt2"
input = "Hello world!"
tokenizer = AutoTokenizer.from_pretrained(model_name)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=5e-5)
input_ids = tokenizer.encode(input, return_tensors="pt").to(device)
output = model(input_ids, labels=input_ids)
output.loss.backward()
optimizer.step()
print(model.get_input_embeddings().weight.grad)
```
I think gpt2 is trained using fp32 but I can load it in bfloat16 and train it (or at least get a gradient with bf16).

Thus I wonder if I misunderstood something. I have another project running to train LLaMA using bfloat16 (essentially using [`run_clm.py`](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm.py) from the official repo with `--torch_dtype=bfloat16 --bf16` command line flag), so if it turns out that I should not use `torch_dtype` for training then it means that I need to stop the experiments and rerun a lot of things :(

Thank you.

### Expected behavior

I am not sure if we should use `torch_dtype` for training.

Comments:
You can try to train in full bfloat16 but it's not as stable as mixed precision bfloat16 training. When running `run_clm` with `--torch_dtype=bfloat16` you train in full bfloat16 so the flag `--bf16` (mixed precision training) is not really useful.

@sgugger thank you for the info!

@sgugger Hi, recently I use run_clm.py to train my LM. I am a little confused about  what you said, why `--bf` did nothing  when passed `--torch_dtype=bfloat16` ? `torch_dtype` seems to be used just when load a pretrained model,  `--bf ` is a training arg to mixed precision training . what is the connection?
Thanks
Knowledge distillation for vision guide
Author: merveenoyan
Pull request: yes

This is a draft PR that I opened in the past on KD guide for CV, but I accidentally removed my fork. I prioritized TGI docs so this PR might stay stale for a while, I will ask for a review after I iterate over comments left by @sayakpaul in my previous PR. (Mainly training MobileNet with random initial weights and not with pre-trained weights from transformers)

Comments:
@sayakpaul I changed the setup and didn't observe a lot of difference, but I felt like it would be still cool to show how to distill a model. WDYT?

cc @rafaelpadilla for reference

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25619). All of your documentation changes will be reflected on that endpoint.

@rafaelpadilla @NielsRogge can we merge this if this looks good? 

> @rafaelpadilla @NielsRogge can we merge this if this looks good?

Yes, it's OK to me. 
My comments were merely about writing style

@LysandreJik can you give a review or ask for another reviewer if needed?

Please resolve the merge conflicts and merge @merveenoyan
Benchmark whether we can get better performance out of ZeRO-3 vs. ZeRO-2 on a single GPU
Author: rajveer43
Pull request: no


Comments:
Hey @rajveer43, what is this issue? Please open issues relevant to what we're trying to do with this repository. These issues are not helpful and do not seem related to `transformers`. I appreciate your understanding.

Actually I was working on one of the documentation issues for a while and saw this thing marked as `todo`I understand your concern about it. I obey to follow the contributing guidelines strictly., @LysandreJik
Fix Seq2seqTrainer decoder attention mask 
Author: Rocketknight1
Pull request: yes

The Seq2SeqTrainer drops `decoder_input_ids` during the generation step for metrics that expect text generation (like `rouge`) when `labels` is present. However, it doesn't drop `decoder_attention_mask` when it does this, which means that in some cases, we pass `decoder_attention_mask` with no `decoder_input_ids`, resulting in the model getting very confused and throwing a shape error.

This PR fixes the issue.

Fixes #24567

Comments:
cc @gante and @ydshieh because I see you in the `git blame` near here - let me know if this is okay, or if I'm breaking anything with this fix!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26841). All of your documentation changes will be reflected on that endpoint.
bos_token does not persist through save_pretrained in 4.34
Author: dakinggg
Pull request: no

### System Info

- `transformers` version: 4.34.0
- Platform: macOS-13.5-arm64-arm-64bit
- Python version: 3.10.12
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.20.3
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction
```python
In [1]: import transformers

In [2]: t0tt = transformers.AutoTokenizer.from_pretrained('bigscience/T0pp')
You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565

In [3]: t0tt.add_special_tokens({'bos_token': '[NEWSPECIAL]'})
Out[3]: 1

In [4]: t0tt.save_pretrained('saved-tokenizer')
Out[4]: 
('saved-tokenizer/tokenizer_config.json',
 'saved-tokenizer/special_tokens_map.json',
 'saved-tokenizer/spiece.model',
 'saved-tokenizer/added_tokens.json',
 'saved-tokenizer/tokenizer.json')

In [5]: loaded_t0tt = transformers.AutoTokenizer.from_pretrained('saved-tokenizer')
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.

In [6]: t0tt.bos_token
Out[6]: '[NEWSPECIAL]'

In [7]: loaded_t0tt.bos_token
Using bos_token, but it is not set yet.
```
### Expected behavior

Expected that an added pad_token persists when saving and then reloading

Comments:
Thanks will be fixed by #26570
Saving and loading a tokenizer does not produce an identical tokenizer in 4.34
Author: dakinggg
Pull request: no

### System Info

- `transformers` version: 4.34.0
- Platform: macOS-13.5-arm64-arm-64bit
- Python version: 3.10.12
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.20.3
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction
```python
In [1]: import transformers

In [2]: t0tt = transformers.AutoTokenizer.from_pretrained('bigscience/T0pp')
You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565

In [3]: t0tt.save_pretrained('saved-tokenizer')
Out[3]: 
('saved-tokenizer/tokenizer_config.json',
 'saved-tokenizer/special_tokens_map.json',
 'saved-tokenizer/spiece.model',
 'saved-tokenizer/added_tokens.json',
 'saved-tokenizer/tokenizer.json')

In [4]: loaded_t0tt = transformers.AutoTokenizer.from_pretrained('saved-tokenizer')
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.

In [6]: t0tt._eos_token
Out[6]: AddedToken("
", rstrip=True, lstrip=True, single_word=False, normalized=True, special=True)

In [7]: loaded_t0tt._eos_token
Out[7]: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True)

In [8]: t0tt.eos_token
Out[8]: '
'

In [9]: t0tt('hello 
        goodbye')
Out[9]: {'input_ids': [21820, 1, 23281, 1], 'attention_mask': [1, 1, 1, 1]}

In [10]: loaded_t0tt('hello 
        goodbye')
Out[10]: {'input_ids': [21820, 3, 1, 23281, 1], 'attention_mask': [1, 1, 1, 1, 1]}
```
### Expected behavior

When saving and loading a tokenizer, it
(1) behaves the same
(2) has the same config details on the AddedToken

Comments:
@ArthurZucker 

Hey! Also fixed in #26570 THanks for reporting!