noamaanMulla-03/transformers-issues huggingface dataset:
Tests: upgrade `test_eager_matches_sdpa_generate`
Author: gante
Pull request: yes

# What does this PR do?

`test_eager_matches_sdpa_generate` has failed in our new failure reporting system ([here](https://huggingface.co/datasets/hf-internal-testing/transformers_daily_ci/raw/ccbc621b1791f7713e2d8e977a481844a4a97140/2024-10-24/ci_results_run_models_gpu/new_model_failures_with_bad_commit_grouped_by_authors.json), cc @ydshieh )

Having a look at the test, the cause for flakiness was clear: we are using random models with `generate`, and tiny perturbations can result in a different sampled token, causing generation to go in a different direction and ultimately failing the check (eager generate == sdpa generate).

This PR:
1. Moves the test to `GenerationTesterMixin`, as it calls `generate`
2. Adds logic to handle the flakiness and do the correct check. If the generation is different, check the logits in the first different token. Despite resulting in a different token, the logits should be nearly identical -- if they are, the test pass.
3. Remove most overwrites, which only existed to handle flakiness through `is_flaky()`.

________________________________

The following test commands were run:
1. `RUN_SLOW=1 py.test tests/models/ -k test_eager_matches_sdpa_generate` ✅ 
2. `RUN_SLOW=1 py.test tests/models/gpt2/test_modeling_gpt2.py::GPT2ModelTest::test_eager_matches_sdpa_generate --flake-finder --flake-runs 500` ✅ 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34386). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Check diff files in `check_copies`
Author: zucchini-nlp
Pull request: no
Labels: name: Feature request

### Feature request

As mentioned in [this comment](https://github.com/huggingface/transformers/pull/31252#discussion_r1646264467) we want to check that files generated by diff are not changed whenever we run `make fix-copies`. That way we can catch if there are any any changes in super classes that aren't being propagated to the diff child.

### Motivation

Repo consistency checks

### Your contribution

Tagging @ArthurZucker for diff and maybe @ydshieh for repo consistency check. I will be very slow if I start adding this 

Comments:
Closing as done, have no idea in which PR exactly
Avoid check expected exception when it is on CUDA
Author: ydshieh
Pull request: yes

# What does this PR do?

CUDA don't like this and will affect many subsequential tests.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34408). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix typos in agents_advanced.md
Author: rudydel
Pull request: yes

# What does this PR do?

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR, like @stevhliu in this case!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34405). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] Cache implementations
Author: stevhliu
Pull request: yes

Follow up to comment [here](https://github.com/huggingface/transformers/pull/34299#issuecomment-2428741017) by @pcuenca to explicitly list the cache methods.

Comments:
Clean! cc @gante @ArthurZucker let's not forget to also update here when new cache classes are added

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34325). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Improve EncoderDecoderModel docs
Author: Ryukijano
Pull request: yes

Fixes #16135

Improve the `EncoderDecoderModel` documentation.

* Add example code to create an `EncoderDecoderModel` using `from_encoder_decoder_pretrained`.
* Add instructions on how to save the model.
* Add instructions on how to fine-tune the model.
* Add a warning about correctly setting configuration values.

---

For more details, open the [Copilot Workspace session](https://copilot-workspace.githubnext.com/huggingface/transformers/issues/16135?shareId=57362139-2e86-4020-90f9-f15402ecf6e3).

Comments:
cc @stevhliu 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34323). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
How to get back the input time series after using PatchTSTForPretraining?
Author: nikhilajoshy
Pull request: no

### System Info

-

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

My model is PatchTSTForPretraining(
  (model): PatchTSTModel(
    (scaler): PatchTSTScaler(
      (scaler): PatchTSTStdScaler()
    )
    (patchifier): PatchTSTPatchify()
    (masking): PatchTSTMasking()
    (encoder): PatchTSTEncoder(
      (embedder): PatchTSTEmbedding(
        (input_embedding): Linear(in_features=5, out_features=768, bias=True)
      )
      (positional_encoder): PatchTSTPositionalEncoding(
        (positional_dropout): Identity()
      )
      (layers): ModuleList(
        (0-11): 12 x PatchTSTEncoderLayer(
          (self_attn): PatchTSTAttention(
            (k_proj): Linear(in_features=768, out_features=768, bias=True)
            (v_proj): Linear(in_features=768, out_features=768, bias=True)
            (q_proj): Linear(in_features=768, out_features=768, bias=True)
            (out_proj): Linear(in_features=768, out_features=768, bias=True)
          )
          (dropout_path1): Identity()
          (norm_sublayer1): PatchTSTBatchNorm(
            (batchnorm): BatchNorm1d(768, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
          (ff): Sequential(
            (0): Linear(in_features=768, out_features=3072, bias=True)
            (1): GELUActivation()
            (2): Identity()
            (3): Linear(in_features=3072, out_features=768, bias=True)
          )
          (dropout_path3): Identity()
          (norm_sublayer3): PatchTSTBatchNorm(
            (batchnorm): BatchNorm1d(768, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
      )
    )
  )
  (head): PatchTSTMaskPretrainHead(
    (dropout): Dropout(p=0.0, inplace=False)
    (linear): Linear(in_features=768, out_features=5, bias=True)
  )
)

prediction_output = model(time_series_data)

Output:

time_series_data = tensor([[[430.3000],
         [431.7600],
         [431.7600],
         [431.7600],
         [431.7600],
         [431.7600],
         [431.7600],
         [431.7600],
         [431.7600],
         [430.3000],
         [430.3000],
         [428.9600],
         [430.3000],
         [430.3000],
         [430.3000]]], device='cuda:0')
prediction_output = tensor([[[[-0.2321,  0.1897,  0.4731,  0.8893,  0.6723],
          [-0.5465, -0.9017,  0.0778,  0.0078,  1.3323],
          [ 0.4945,  0.5145, -0.5386, -0.7045, -1.5766],
          [ 0.2064,  0.6290, -0.8145,  1.0450, -0.2886]]]], device='cuda:0')

### Expected behavior

x_hat = self.head(model_output.last_hidden_state) produces output which is not consistent to the range of input time series values. I am trying to pretrain PatchTST for autoencoding. How do I get back the input time series?

Comments:
hello, have you solved the problem?
from_pretrained callback?
Author: lucasjinreal
Pull request: no
Labels: name: Core: Modeling

name: Usage

I have a model implement from PretrainModel, how can I set a from_pretrianed call back in my model class?

I want call some weigths re-initialize when MyModel.from_pretrained

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
'FastSpeech2ConformerConfig' object has no attribute 'model_config'
Author: spencerchubb
Pull request: no
Labels: name: Audio

### System Info

- `transformers` version: 4.41.2
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.11.1
- Huggingface_hub version: 0.23.3
- Safetensors version: 0.4.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.3.1+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Try to run the example code from the docs: https://huggingface.co/docs/transformers/en/model_doc/fastspeech2_conformer

```from transformers import pipeline, FastSpeech2ConformerHifiGan
import soundfile as sf

vocoder = FastSpeech2ConformerHifiGan.from_pretrained("espnet/fastspeech2_conformer_hifigan")
synthesiser = pipeline(model="espnet/fastspeech2_conformer", vocoder=vocoder)

speech = synthesiser("Hello, my dog is cooler than you!")

sf.write("speech.wav", speech["audio"].squeeze(), samplerate=speech["sampling_rate"])
```

### Expected behavior

Run text-to-speech inference

AttributeError: 'FastSpeech2ConformerConfig' object has no attribute 'model_config'. Did you mean: 'decoder_config'?

Comments:
cc @sanchit-gandhi @ylacombe 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Mamba 2 Multi-GPU errors out on generation with parallel beam search
Author: molbap
Pull request: no
Labels: name: Distributed Training / Models

name: Generation

### Observed issue
Found out when running multi-gpu slow tests in https://github.com/huggingface/transformers/pull/33560 .

Line 479 exactly of the mamba2 modeling file

https://github.com/huggingface/transformers/blob/8efc06ee1863bd6e34e8adb7b10901da87c66818/src/transformers/models/mamba2/modeling_mamba2.py#L472-L480

Will raise the following for the test `tests/models/mamba2/test_modeling_mamba2.py::Mamba2ModelTest::test_model_parallel_beam_search`
```bash
E           RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!
```

### Mitigation 

Move the smallest tensors of that operation to the device of the largest. Will do that soon, but if anyone wants to jump on the issue happy to review as well. 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
ValueError: FalconMambaForCausalLM does not support Flash Attention 2.0 yet
Author: Cshekar24
Pull request: no
Labels: name: Usage

name: Flash Attention

I'm facing issues while inferencing while using falcon LLM. The latency is around 20-30 minutes for a specific use case. I want to reduce the time and found that we can install Flash Attention 2 to significantly speedup inference. But I found that this is not supported in Falcon yet. 

Request the team/members to look into the issue & address it ASAP. 

Comments:
@Cshekar24 FalconMamba is a full mamba model. it doesn't use attention at all. 

As @avishaiElmakies mentions, Flash Attention can only be applied to models that use attention. That's not the case for FalconMamba, see: [Welcome FalconMamba: The first strong attention-free 7B model](https://huggingface.co/blog/falconmamba)

> Request the team/members to look into the issue & address it ASAP.

There is no issue to address.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
ValueError: invalid literal for int() with base 10: '<'
Author: 01Ashish
Pull request: no

While working on ASR and speaker diarization 
using - https://github.com/MahmoudAshraf97/ctc-forced-aligner/blob/main/ctc_forced_aligner/alignment_utils.py

get_spans function is getting used 
spans = get_spans(tokens_starred, segments, alignment_tokenizer.decode(blank_id))

check - _decode function in [alignment](https://github.com/huggingface/transformers/blob/main/src/transformers/models/wav2vec2/tokenization_wav2vec2.py)

tokenization_utils.py


error - 
ValueError: invalid literal for int() with base 10: '<'
Traceback:
File "/usr/local/lib/python3.10/dist-packages/streamlit/runtime/scriptrunner/exec_code.py", line 88, in exec_func_with_error_handling
    result = func()
File "/usr/local/lib/python3.10/dist-packages/streamlit/runtime/scriptrunner/script_runner.py", line 579, in code_to_exec
    exec(code, module.__dict__)
File "/content/app-streamlit.py", line 262, in <module>
    main()
File "/content/app-streamlit.py", line 239, in main
    transcript = transcribe(temp_file_path)
File "/content/app-streamlit.py", line 121, in transcribe
    spans = get_spans(tokens_starred, segments, alignment_tokenizer.decode(blank_id))
File "/usr/local/lib/python3.10/dist-packages/transformers/models/wav2vec2/tokenization_wav2vec2.py", line 637, in decode
    return self._decode(
File "/usr/local/lib/python3.10/dist-packages/transformers/models/wav2vec2/tokenization_wav2vec2.py", line 433, in _decode
    filtered_tokens = self.convert_ids_to_tokens(token_ids, skip_special_tokens=skip_special_tokens)
File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils.py", line 976, in convert_ids_to_tokens
    index = int(index)

Comments:
/assign

interesting issue, can you assign to me 

Feel free to try and solve it @rishabh1057d, but I believe we'll need more than what @01Ashish has provided to solve it.

Would it be possible to have a reproducible code snippet that returns the same error? Thank you

error is fixed 
if you want to understand refer : https://github.com/MahmoudAshraf97/whisper-diarization/blob/main/Whisper_Transcription_%2B_NeMo_Diarization.ipynb
doc: Trainer.hyperparameter_search docstring discrepancy solved
Author: GuillemGSubies
Pull request: yes

# What does this PR do?

I tried to explain what `kwargs` do for all 3 backends.

Fixes #34239


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. #34239 
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@stevhliu


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34311). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Suggestions added, thank you 😊 

Nice, if you can just rebase your branch on main to fix the failing `build_pr_documentation` test, that'd be great 🙂 

Ready

@LysandreJik the `run_tests` and `setup_and_quality` have been pending for quite some time now. Is there any way we can get these tests to run?

Can you maybe try closing this PR and opening a new one?
pin `tensorflow_probability<0.22` in docker files
Author: ydshieh
Pull request: yes

# What does this PR do?

After #34314, `tensorflow_probability==0.24` was selected and installed (under the pip dependency) and some tests (in particular, TF tapas) failed with `tensorflow==2.13`.

Let's pin `tensorflow_probability<0.22`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34381). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Tiny update after #34383
Author: ydshieh
Pull request: yes

# What does this PR do?

I forgot a place that needs an update too.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34404). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
exclude fsdp from delay_optimizer_creation
Author: eljandoubi
Pull request: yes

# What does this PR do?

It passes the model and  the optimizer to accelerate.prepare in order to enable fp8 mixed precision, if any.

Fixes #34024 

## Who can review?

Library:

- trainer: @muellerzr and @SunMarc 

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34140). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

The required tests are distributed tests. We need to verify FSDP functionality with and without FP8 mixed precision. The appropriate test file might be `tests/trainer/test_trainer_fsdp.py`.

Is the `test/trainer` folder included in the CI tests? Where can I check the results for `test_trainer_fsdp.py`? @muellerzr @SunMarc 

@eljandoubi we can't run them on the normal CI since GPU runners are not part of PR's. 

Instead when ready I'll pull the PR down and run it myself

@muellerzr Thank you for the information. I have tested the branch in my code on a multi-node, multi-GPU setup using FSDP mode, both with and without FP8 mixed precision, and it worked as expected. Please let me know if you encounter any issues on your end.

 @muellerzr Any updates regarding this PR?

@muellerzr  I have done `make fixup`.
HF Trainer do not support Pytorch FSDP with FP8; ValueError: You must pass a model and an optimizer together to `accelerate.prepare()` when using TransformerEngine.
Author: eljandoubi
Pull request: no
Labels: name: trainer

name: bug

name: PyTorch FSDP

name: Accelerate

### System Info

![image](https://github.com/user-attachments/assets/3bd519a7-6559-4922-8417-dec4402e3bbe)
acc_cfg.yml:

compute_environment: LOCAL_MACHINE
debug: false
distributed_type: FSDP
downcast_bf16: 'no'
enable_cpu_affinity: false
fsdp_config:
  fsdp_activation_checkpointing: true
  fsdp_auto_wrap_policy: NO_WRAP
  fsdp_backward_prefetch: NO_PREFETCH
  fsdp_cpu_ram_efficient_loading: true
  fsdp_forward_prefetch: true
  fsdp_offload_params: true
  fsdp_sharding_strategy: FULL_SHARD
  fsdp_state_dict_type: SHARDED_STATE_DICT
  fsdp_sync_module_states: true
  fsdp_use_orig_params: true
machine_rank: 0
main_process_ip: 0.0.0.0
main_process_port: 0
main_training_function: main
mixed_precision: fp8
fp8_config:
  amax_compute_algorithm: max
  amax_history_length: 1024
  backend: TE
  fp8_format: HYBRID
  interval: 1
  margin: 0
  override_linear_precision: false
  use_autocast_during_eval: true
num_machines: 3
num_processes: 24
rdzv_backend: etcd-v2
same_network: false
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

* accelerate launch --config_file acc_cfg.yml train.py $TRAINING_ARGS
* the train.py is any training script that train using transformers.Trainer
* $TRAINING_ARGS are the TrainingArguments plus some path to data
![image](https://github.com/user-attachments/assets/15e2af5b-f6af-4e5c-ae78-7ce6ea22ce3b)


### Expected behavior

Train Paligemma model with FSDP and FP8.

Comments: