noamaanMulla-03/transformers-issues huggingface dataset:
fix_mbart_tied_weights
Author: SunMarc
Pull request: yes

# What does this PR do ? 
Fixes #26266. This PR fixes the tied weights for mbart model. Before this PR, only `lm_head` was tied to `model.shared`. Now, we also make sure to tie `model.encoder.embed_tokens` and `model.decoder.embed_tokens` to `model.shared` by defining the `_tie_weights` method which will be called when we do `model.tie_weights()`. I've checked that we get the same weights at the end. This issue only happens when we load with `safetensors` + `device_map` because we don't save the shared tensors and the weights are on the meta device. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hmm I'm not sure if all mBART weights share all weight matrices with each other. 

We should make sure that at least for all the following models:
https://huggingface.co/models?other=mbart&sort=trending&search=facebook
all three embedding matrices are identical (I'm not sure this is always the case e.g. for multi-lingual ones) 

Yes I think we should look at the `config.tie_word_embeddings` value and adapt accordingly. See recent PR on FSMT: https://github.com/huggingface/transformers/pull/26292

Thanks for the link @LysandreJik. I've updated the code and added a test. 

As @patrickvonplaten was saying could you also quickly verify that it works with the most downloaded mbart models on the Hub? When doing the FSMT change I ended up breaking a few FSMT models on the Hub, let's try to prevent this here :grin: 

Thanks for your help @SunMarc 

Hi @LysandreJik , I confirm that for the most downloaded mbart models on the hub, all three embedding matrices are identical. Here's the snippet that I used: 
```py
from transformers import AutoModelForSeq2SeqLM
models = ["facebook/mbart-large-50-many-to-many-mmt", "facebook/mbart-large-50-many-to-one-mmt", "facebook/mbart-large-50-one-to-many-mmt","facebook/mbart-large-50","facebook/mbart-large-cc25","facebook/mbart-large-en-ro","facebook/mgenre-wiki"]
for model_id in models:
    for safetensors in [True, False]:
        for device_map in ["auto", None]:
            try:
                model = AutoModelForSeq2SeqLM.from_pretrained(model_id, use_safetensors=safetensors, device_map=device_map)
            except:
                print(f"{model_id} failed to load with safetensors={safetensors} and device_map={device_map}")
            assert len(
                {
                    model.get_output_embeddings().weight.data_ptr(),
                    model.get_input_embeddings().weight.data_ptr(),
                    model.base_model.decoder.embed_tokens.weight.data_ptr(),
                    model.base_model.encoder.embed_tokens.weight.data_ptr(),
                }
            ) == 1, "Embeddings are not tied in {}".format(model_id)
```

Thanks a lot @SunMarc !

Yay, that works. Thanks a lot everyone!
[Whisper Tokenizer] Make decoding faster after adding timestamps
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Following the update to the Whisper tokenizer to handle encoding/decoding timestamps (#26054), there is one line in the decoding which takes extremely long: https://github.com/huggingface/transformers/blob/f94c9b3d863c1a95b44b5b3ea9ce3cbd27fc7609/src/transformers/models/whisper/tokenization_whisper.py#L614

Here we do an order `N * M`  operation to filter out all the timestamp tokens, where `N` is the length of the token ids, and `M` the number of timestamp tokens (for each token, check whether it’s in the timestamp token list).

In practice, this is causing decoding to take **extremely** long for typical validation sets, e.g. LibriSpeech test clean took ~30 mins for the tokenizer to decode on a TPU v3 (which has lots of CPU power to run this operation).

This PR switches the timestamp filtering to a regex string operation, which in a toy benchmark was a factor of > 2000 faster. Would love to hear from @ArthurZucker whether we're happy to sacrifice a bit of readability for this speed-up!

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Yep it was significantly faster once scaled to large datasets and long sequence lengths (>5k samples with 256 seq len)
[docs] navigation improvement between text gen pipelines and text gen params
Author: MKhalusova
Pull request: yes

# What does this PR do?

This PR addresses issues brought up in  #26280. 
It adds a note about text generation parameters to the `TextGenerationPipeline` and `Text2TextGenerationPipeline` docs to improve navigation and discoverability. Also adds a note on stopping criteria to the `generation_strategies.md` guide.   




Comments:
_The documentation is not available anymore as the PR was closed or merged._
blip itm task implementation
Author: MasKong
Pull request: no

### System Info

In the official implementation of blip as well as the paper, the multi-modal embedding that is used to perform the itm task comes from a special token named enc_token_id. But the transformer implementation still employs [CLS] token.


```
text.input_ids[:,0] = self.tokenizer.enc_token_id
output = self.text_encoder(text.input_ids,
                           attention_mask = text.attention_mask,
                           encoder_hidden_states = image_embeds,
                           encoder_attention_mask = image_atts,      
                           return_dict = True,
                          )              
return output.last_hidden_state
```

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://github.com/huggingface/transformers/blob/v4.31.0/src/transformers/models/blip/modeling_blip.py#L1418

### Expected behavior

https://github.com/salesforce/BLIP/blob/main/models/blip.py#L67

Comments:
This is a duplicate of #22732. cc @younesbelkada as it seems to be confusing 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Avoid all-zeor attnetion mask used in testing
Author: ydshieh
Pull request: yes

# What does this PR do?

The method `random_attention_mask` used in testing makes sure the last token is non-zero. However, this property will be changed if a causal mask is applied.

This causes some issues in CI, see issue reported 

https://github.com/pytorch/pytorch/issues/110213

In general, a sequence with all zero as attention mask is bad. Let's avoid testing with such case.

(However, we probably need to do some processing in the modeling code - if torch decide this is undefined behavior and won't make change to have previous behavior). 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
AttributeError: 'FullyShardedDataParallelPlugin' object has no attribute 'activation_checkpointing'
Author: scissorstail
Pull request: no

### System Info


```
Traceback (most recent call last):
  File "/workspace/run/run_llm.py", line 717, in <module>
    main()
  File "/workspace/run/run_llm.py", line 644, in main
    trainer = Trainer(
  File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 342, in __init__
    self.create_accelerator_and_postprocess()
  File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 3900, in create_accelerator_and_postprocess
    "activation_checkpointing", fsdp_plugin.activation_checkpointing
AttributeError: 'FullyShardedDataParallelPlugin' object has no attribute 'activation_checkpointing'
```

https://github.com/huggingface/transformers/blob/aea761499f4b1193f2706f471442da6f9df65d65/src/transformers/trainer.py#L3893-L3907

The 'FullyShardedDataParallelPlugin' class in [accelerate](https://github.com/huggingface/accelerate) version **v0.22.0** does not have 'activation_checkpointing'. but the **main** branch does.

**v0.22.0**
https://github.com/huggingface/accelerate/blob/6b3e559926afc4b9a127eb7762fc523ea0ea656a/src/accelerate/utils/dataclasses.py#L778

**main**
https://github.com/huggingface/accelerate/blob/739b135f8367becb67ffaada12fe76e3aa60fefd/src/accelerate/utils/dataclasses.py#L783



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

-

### Expected behavior

-

Comments:
cc @muellerzr @pacman100 

Hi! This has been fixed on main, please install transformers via `pip install git+https://github.com/huggingface/transformers`
`resize_token_embeddings` sets `vocab_size` to 0 in Deepspeed mode.
Author: zhouku92
Pull request: no

### System Info

**System Info**
* transformers==4.33.2
* accelerate==0.23.0
* deepspeed==0.9.2

I was running a customized script to train a LLaMA model via Deepspeed and encountered the following exception:
```
File "/opt/conda/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1736, in forward
    shift_logits = shift_logits.view(-1, self.config.vocab_size)
RuntimeError: shape '[-1, 0]' is invalid for input of size 32705022

```

A quick check directs me to the following block ([link](https://github.com/huggingface/transformers/blob/v4.33.2/src/transformers/modeling_utils.py#L1439)):
```
    def resize_token_embeddings(
        self, new_num_tokens: Optional[int] = None, pad_to_multiple_of: Optional[int] = None
    ) -> nn.Embedding:
        model_embeds = self._resize_token_embeddings(new_num_tokens, pad_to_multiple_of)
        if new_num_tokens is None and pad_to_multiple_of is None:
            return model_embeds

        # Update base model and current model config
        self.config.vocab_size = model_embeds.weight.shape[0]
        self.vocab_size = model_embeds.weight.shape[0]

        # Tie weights again if needed
        self.tie_weights()

        return model_embeds
```
In Deepspeed, `model_embeds` is `Embedding(32001, 4096)` but `model_embeds.weight` is `torch.Size([0])` (per my understanding, the model hasn't been initialized yet). Thus, `self.config.vocab_size` is mistakenly set to `0`.

**Who can help?**
@pacman100

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```py
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("llama-path")
model.resize_token_embeddings(new_num_tokens=32001, pad_to_multiple_of=None)
```

Run with  deepspeeed


### Expected behavior

`self.config.vocab_size` and `self.vocab_size` should  be set to the correct number. Personally `model_embeds.num_embeddings` is better than `model_embeds.weight.shape[0]`.

Comments:
Same issue here with same package versions except deepspeed==0.10.3. Downgrading transformers to 4.30.2 temporarily resolved the issue. 

Indeed, it seems there is an issue here! Would you mind opening a PR with your proposed fix?

Ah actually it's already being fixed here: https://github.com/huggingface/transformers/pull/26387


Hello @weixi-feng, this merged PR https://github.com/huggingface/transformers/pull/26259 should have fixed the issue . Could you retry with the main branch?

Yes, the main branch works for me. Thanks for the quick follow-up.

Thank you, it works for me.
[Flax Examples] Seq2Seq ASR Fine-Tuning Script
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Can be used to fine-tune Flax Whisper for speech recognition.

Tested and verified as working with the following (dummy) config:
```
run_flax_speech_recognition_seq2seq.py \
            --model_name_or_path openai/whisper-tiny.en \
            --dataset_name hf-internal-testing/librispeech_asr_dummy \
            --dataset_config clean \
            --train_split_name validation \
            --eval_split_name validation \
            --output_dir whisper-tiny-ft-dummy \
            --overwrite_output_dir \
            --num_train_epochs=2 \
            --max_train_samples 10 \
            --max_eval_samples 10 \
            --warmup_steps=8 \
            --do_train \
            --do_eval \
            --learning_rate=2e-4 \
            --per_device_train_batch_size=2 \
            --per_device_eval_batch_size=1 \
            --predict_with_generate
```

Will add a README with preliminary training configs / results later this week after doing a full fine-tuning run.

cc @peregilk @andyehrenberg for interest

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sanchit-gandhi @andyehrenberg 

We have made a version of this script will support streaming and training on the TPU pods.

The current version of the script is available here:
[https://github.com/NbAiLab/nb-whisper/blob/main/run_flax_speech_recognition_seq2seq_streaming.py](https://github.com/NbAiLab/nb-whisper/blob/main/run_flax_speech_recognition_seq2seq_streaming.py)

We are however struggling with a bug at the moment. The script seems to work for training the Tiny models on multiple pod sizes. Both for scaling for speed and for increasing the batch size. All the other model sizes (small, base, medium, large) also works on the single TPU v4-8. However, training on the non-Tiny-model sizes runs for a few steps then freezes. 

If anyone have any idea about this could be happening, I really appreciate it.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Given the popularity of the PyTorch fine-tuning script and Whisper JAX, it's a pretty easy addition adding a Whisper fine-tuning script in JAX/Flax.

Note: this is largely based off the distil-whisper training script, but simplified to run offline, with just 1 training dataset and the cross-entropy objective https://github.com/huggingface/distil-whisper#training
Output of a model (e.g. logits for the first token) depends on input length
Author: tomasz-kielbasa
Pull request: no

### System Info

- `transformers` version: 4.33.3
- Platform: Linux-5.19.0-45-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.3.3
- Accelerate version: 0.23.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): 2.9.2 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.3 (gpu)
- Jax version: 0.4.1
- JaxLib version: 0.4.1
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf", device_map='auto')

text = 'This is a'
input_ids = tokenizer(text, return_tensors='pt').input_ids.to('cuda')

for i in range(input_ids.shape[1]):
    full_text_logits = model(input_ids).logits[0][0]
    single_token_logits = model(input_ids[:, :i + 1]).logits[0][0]
    
    error = (full_text_logits - single_token_logits).abs()
    print(error.sum().item(), error.max().item())
```
Output:
```
0.5521924495697021 8.106231689453125e-05
0.10465995222330093 2.002716064453125e-05
0.06035400182008743 8.58306884765625e-06
0.0 0.0
```

### Expected behavior

The above numbers should be zero or closer to zero. A total error of 0.55 looks suspicious. Tried also with a different model (gpt2) with similar results. If this is the desired behavior, is there a workaround to achieve deterministic results?

Comments:
cc @ArthurZucker 

It seems this is a PyTorch/CUDA thing.
```python
import torch
from torch import nn
torch.manual_seed(0)

linear = nn.Linear(4, 4).to('cuda')
dummy_input = torch.normal(mean=0, std=1, size=(2, 4)).to('cuda')
full = linear(dummy_input)[0]
single = linear(dummy_input[:1])[0]
full == single
```
This code results in the following output.
```
tensor([ True, False,  True,  True], device='cuda:0')
```
Note that if we run `linear` multiple times on the exact same input, we get the same output, so this problem isn't about determinism.
[ASR Pipe] Improve docs and error messages
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Fixes #26420 with three primary changes:
1. Renaming of `generator` -> `transcriber` in the pipeline tutorial
2. More descriptive error message when `ffmpeg_read` fails, in a move to guide the user towards fixing likely problems
3. Update docs for asr pipe inputs


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Use `F.multi_head_attention_forward()` to take advantage of PyTorch's Flash attention
Author: gau-nernst
Pull request: no

### Feature request

Currently I'm using Wav2Vec 2.0 models. Digging into the code, I can see that it manually computes multi-head attention (and actually it copies from Bart). Using `F.multi_head_attention_forward()` would enjoy the benefits of any new improvements PyTorch brings (e.g. Flash attention) without installing extra libraries to do the hacking (i.e. optimum). The current solution is to use HF optimum to convert the model, which calls a private PyTorch's method.

https://github.com/huggingface/optimum/blob/05d20df3e6602e26d01cf3994a108de5b097a719/optimum/bettertransformer/models/encoder_models.py#L1415

### Motivation

To take advantage of Flash attention, optimum is required to convert the model. I was quite surprised that using Flash attention is not the default behavior of HF models. By using `F.multi_head_attention_forward()`, the users can enjoy the best attention speedup by default. For advanced users, who will be able to dig into the code to figure out why Flash attention is not used, and figure out to use optimum, it will save debugging time. For beginner users, this provides the best speed without any prior knowledge. It will also save the trouble of installing an extra library and perform the conversion.

Some considerations:
- In terms of availability, `F.multi_head_attention_forward()` has existed for a long time (it goes back to at least PyTorch 1.8, I haven't checked before that). I see from latest main branch that minimum PyTorch version is 1.9 https://github.com/huggingface/transformers/blob/4d40109c3a93c9b8bbca204cb046ed510f1c72e8/setup.py#L176, so this function is definitely available.
- In terms of weight-compatibility, `F.multi_head_attention_forward()` supports passing in separate q, k, v projection weights, but input projection weight must be packed together. We can keep the `nn.Linear` modules, and pass the parameters directly to `F.multi_head_attention_forward()`. q, k, v biases will need to be packed into a single tensor (probably with `torch.cat()`).

For more details, check https://github.com/pytorch/pytorch/blob/v1.9.0/torch/nn/functional.py#L4836

### Your contribution

This seems like a big change, but I think it should be straight-forward. I'm happy to submit a PR if there are people to help me land this.

Do let me know other considerations from HF side that I'm not aware of. Thank you!

Comments:
Something is being cooked up in #25598 😉 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add FAST model
Author: raghavanone
Pull request: no
Labels: name: New model

### Model description

FAST model does efficient scene text detection and will be good addition to repo.

Paper : https://arxiv.org/pdf/2111.02394
Code : https://github.com/czczup/FAST

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
I will be working on this .

Hello @raghavanone! 

I find this model and idea more interesting...
I would like to work along with you
Pegasus Documentation May Conflict With Seq2Seq ReadMe
Author: kingpalethe
Pull request: no
Labels: name: wontfix

Here, under `tips and tricks`.....
https://github.com/huggingface/transformers/blob/master/examples/seq2seq/README.md#tips-and-tricks
`Both finetuning and eval are 30% faster with --fp16. For that you need to install apex.`

But in the documentation...
https://huggingface.co/transformers/master/model_doc/pegasus.html#examples
`FP16 is not supported (help/ideas on this appreciated!).`

Also in the documentation
https://huggingface.co/transformers/master/model_doc/pegasus.html#examples
`Script to fine-tune pegasus on the XSUM dataset.`
leads to a 404: https://github.com/huggingface/transformers/blob/master/examples/seq2seq/finetune_pegasus_xsum.sh



Comments:
Hi @kingpalethe,

In general, for BART and Marian models, training and eval is faster with fp16,  except Pegasus and T5 which currently don't work well with fp16

Yes, the fine-tuning script is now moved under `examples/research_projects/seq2seq-distillation` dir,
https://github.com/huggingface/transformers/tree/master/examples/research_projects/seq2seq-distillation

Thanks for reporting,

Also please note that this script is not maintained anymore and is provided as-is. We only maintain the `finetune_trainer.py` script now.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.

#26521 @lotaflor989@gmail.com[#v0](url)

sio
[Falcon] Fix AutoConfig Tests
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Fixes the auto-config Falcon tests that were failing on `main` after merging #26476. Note that the latest version of the Falcon model on the Hub specifies a model type of `falcon`, following @Rocketknight1's Hub commit [898df13](https://huggingface.co/tiiuae/falcon-7b/commit/898df1396f35e447d5fe44e0a3ccaaaa69f30d36). Thus, we update the config tests in `transformers` to reflect this.

There are still config tests to check that `trust_remote_code` gives the correct configs for older versions (with specific commit ids): https://github.com/huggingface/transformers/blob/0b192de1f353b0e04dad4813e02e2c672de077be/tests/models/falcon/test_modeling_falcon.py#L603

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26507). All of your documentation changes will be reflected on that endpoint.

Thanks for your PR @sanchit-gandhi!

Closing in favor of https://github.com/huggingface/transformers/pull/26472