noamaanMulla-03/transformers-issues huggingface dataset:
Fix a few typos in `GenerationMixin`'s docstring
Author: sadra-barikbin
Pull request: yes

Hi there!

To fix a few typos in `GenerationMixin`'s docstring.

@gante 

Comments:
Shouldn't we change the parameter shape in 

https://github.com/huggingface/transformers/blob/c8d98405a8f7b0e5d07391b671dcc61bb9d7bad5/src/transformers/generation/utils.py#L146

to `(batch_size*num_return_sequences, sequence_length)` to account for `SampleOutput` and its variants which have `do_sample=True, num_return_sequences>1`?

> Shouldn't we change the parameter shape in
> 
> https://github.com/huggingface/transformers/blob/c8d98405a8f7b0e5d07391b671dcc61bb9d7bad5/src/transformers/generation/utils.py#L146
> 
> to `(batch_size*num_return_sequences, sequence_length)` to account for `SampleOutput` and its variants which have `do_sample=True, num_return_sequences>1`?

yes, absolutely!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29277). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Token level timestamps for long-form generation in Whisper
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Continuation of PR #28984. Adds token level timestamps for long-form generation. The previous PR had a quite different of way to add timestamps, specifically by calling `extract_timestamps` for each segment and each batch separately. I believe, it can be done in one batch, and then divided into segments the same way sequences are divided.  

The final timestamps are already aligned with the total length, so there is not need to add start_time for each segment. Although, I am not sure if that is what we want to have, so I can remove this "total duration alignment" is needed.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sanchit-gandhi 
@patrickvonplaten 
@gante ?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29148). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Also cc @ArthurZucker for Whisper-related timestamps

> @amyeroberts ready for review here

Sorry, a mistag. Got a review from Arthur, so we merged it :)
[`core` / `modeling`] Fix training bug with PEFT + GC
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/28023

4.36.0 led to a bug when users are in the case of GC + training which should force-set `use_cache` to `False` here: https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L1008 - which is force-set to `True` during the backward pass for some reason, only in the case where one uses PEFT + GC.

The fix is to force-set `use_cache` to `False` before computing `past_key_value_length` here: https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L1042 

cc @amyeroberts 

Comments:
I am having this issue with Kohya Dreammaker as well. Downgrading to 4.27.1 has seemed to help. How do I go about actually fixing this issue? I have all the pre reqs installed and my models work, but I have no idea how to use python to fix this issue?

@DailyCasual Could you open a new issue, detailing the error encountered, what you've experimented e.g. versions and their behaviour? This helps us better track what bugs are new and have been resolved.
Starcoder2 model - bis
Author: RaymondLi0
Pull request: yes

The Starcoder2 model, adapted from Mistral. ~~All changes are done through options, so Mistral itself is still supported.~~ 
Main changes:

- Use layer norm
- Use standard MLP
- Add biases (optional)
- Change (default?) tokenizer class
*Embedding and residual dropout

It does not support absolute embeddings, so can't support Santacoder or Starcoder

Starcoder2-3B model: https://huggingface.co/bigcode/starcoder2-3b

Todo:

- Forward changes from https://github.com/huggingface/transformers/pull/27931, https://github.com/huggingface/transformers/pull/29027 (and future changes from Feb. 19) (in a future PR?)
- Documentation
- Copyright
- Point to starcoder2 checkpoint
- Other minor things (see todos)

@younesbelkada @ArthurZucker @jlamypoirier 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29215). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thank you for the PR @RaymondLi0. cc @loubnabnl and @lvwerra, next time let's make sure doc and paper links are fully completed before we merge, we require this for every model / organisation regardless of the release date 😉 let's make this an exception
[i18n-zh] Translate fsdp.md into Chinese
Author: windsonsea
Pull request: yes

Add zh text to this page:

```
docs/source/zh/fsdp.md
```
cc @stevhliu @Fan-Lin @jiaqiw09 

Comments:
/lgtm

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29305). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[i18n-zh] Translated task/asr.md into Chinese
Author: windsonsea
Pull request: yes

Add zh text to this page

cc @stevhliu

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29233). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Great work, thanks for the translation! I don't read any Chinese so I'm unable to review the content of the doc itself. Would you happen to know other Chinese readers who would be interested in reviewing your work?

OK, cc @jiaqiw09 @Fan-Lin @mengjiao-liu, please take a look if you are available.

/lgtm
[i18n-zh] Translated torchscript.md into Chinese
Author: windsonsea
Pull request: yes

Add zh text to this page.

```
docs/source/zh/torchscript.md
```

cc @stevhliu

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29234). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> LGTM, thanks!
> 
> As with your other translation PR, it would be super nice if you could find a Chinese reader to review the content 🙂

OK, cc @jiaqiw09 @Fan-Lin @mengjiao-liu, please take a look if you are available.

/lgtm
FIX [`Gemma` / `CI`] Make sure our runners have access to the model
Author: younesbelkada
Pull request: yes

# What does this PR do?

As discussed offline cc @ydshieh @sanchit-gandhi @ArthurZucker


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29242). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Anyone wants to run the corresponding tests have to do

> export HF_HUB_READ_TOKEN= ...(their own token, not the token stored on github as secret) ...

For CI, it will work fine as the secret is stored on Github Actions.
Weird behavior with mBART-50 and Spanish
Author: ArbinTimilsina
Pull request: no
Labels: name: WIP

## Environment info
- `transformers` version: 4.9.1
- Platform: Linux-5.4.0-1054-aws-x86_64-with-debian-buster-sid
- Python version: 3.7.10
- PyTorch version (GPU?): 1.9.0+cu102 (True)

## Who can help
@patrickvonplaten

## Information
I am seeing weird behavior with mBART-50 and Spanish. Please look at the code below:
```
from transformers import MBartForConditionalGeneration, MBart50TokenizerFast

text = "http://www.ted.com/talks/stephen_palumbi_following_the_mercury_trail.html"

model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50-many-to-one-mmt")
tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50-many-to-one-mmt")
tokenizer.src_lang = "es_XX"

encoded = tokenizer(text, return_tensors="pt")
generated_tokens = model.generate(**encoded, forced_bos_token_id=tokenizer.lang_code_to_id["en_XX"])
tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
```
The output is:
```
['(b) To continue to cooperate closely with the Special Rapporteur on extrajudicial, summary or arbitrary executions, the Special Rapporteur on torture and other cruel, inhuman or degrading treatment or punishment, the Special Rapporteur on the sale of children, child prostitution and child pornography, the Special Rapporteur on torture and other cruel, inhuman or degrading treatment or punishment, the Special Rapporteur on the sale of children, child prostitution and child pornography, the Special Rapporteur on the sale of children, child prostitution and child pornography, the Special Rapporteur on the sale of children, child prostitution and child pornography, the Special Rapporteur on violence against women, its causes and consequences, the Special Rapporteur on the sale of children, child prostitution and child pornography, the Special Rapporteur on the sale of children, child prostitution and child pornography, the Special']
```

However if I change the source language to french `tokenizer.src_lang = "fr_XX"` or any other language, I get the following output (which is what you expect):
```
['http://www.ted.com/talks/stephen_palumbi_following_the_mercury_trail.html']
```

This behavior is similar with other texts as well (e.g., "888"). Do you know why this behavior is unique to Spanish? Also, do you have any idea how to correct this behavior?

Thanks!


Comments:
Pinging @patil-suraj too, and @mrm8488 might have played with that model in the past.

Any progress here? I've faced the exact same problem when attempting to translate from Spanish, although slightly different output:

```
The Committee recommends that the State party take all necessary measures to ensure that the right to adequate housing is guaranteed in the State party's next periodic report, and that the State party take all necessary measures to ensure that the right to adequate housing is guaranteed in its next periodic report.
```

@patil-suraj - could you take a look here?

+1 I've been having the same issue translating from Spanish to English. Could someone take a look?
[`EfficientSAM`] Add EfficientSAM to the library
Author: younesbelkada
Pull request: yes

# What does this PR do?

As per title, this PR adds EfficientSAM, a new architecture from https://github.com/yformer/EfficientSAM that is similar than SAM architecture but with the benefit of being much smaller. 

Draft for now 

@xenova @yformer

Comments:
Hey when can we expect this to merge ?

Hi @rishabh063 
Thanks very much for your interest in this PR - hopefully quite soon, I need some time to make sure logits match then we can merge 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

hey @younesbelkada any updates on this ?

hi @rishabh063 unfortunately I had to work on more urgent things now, I will come back to this ASAP. If you have bandwith you can also take over the PR 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Adding FA2 support for MusicGen
Author: staghado
Pull request: yes

# What does this PR do?

This PR adds Flash Attention 2 support for MusicGen model. It is based on Bart example and it is a WIP for now.
I could not test the model because FA2 is not supported yet for T4 GPUs.

Fixes #27552

@sanchit-gandhi @ylacombe 

Comments:
Hey @staghado, thanks for taking care of this, let us know when it's ready to be reviewed!

I have conducted some tests on an A10 GPU : 
    - The code seems to work without errors when `_supports_flash_attn_2` is set to `True` for `MusicgenForConditionalGeneration` but does not load the model with FA2 if not specified by hand. Maybe it needs to be added at the class level in MusicgenForConditionalGeneration?
    - There is no difference in generation speed between eager attention and FA2 : 
    
![Screenshot from 2024-01-06 21-50-49](https://github.com/huggingface/transformers/assets/84044788/d93f298d-32e5-4345-b054-e5335ac266fe)



cc @ylacombe could you possibly circle back here when you get the chance!

Hi @ylacombe,

I confirm that the model was instantiated as described [here](https://huggingface.co/docs/transformers/v4.36.1/en/perf_infer_gpu_one#flashattention-2) with the exception of `torch_dtype=torch.float16` instead of `torch_dtype=torch.bfloat16` because some operations did not seem to implement bfloat16.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`require_read_token`] fix typo
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes a typo

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29345). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Weird warning when calling LlamaTokenizer.__call__
Author: m-Just
Pull request: no

When I run the following code:
```python
from transformers import LlamaTokenizer

model_path_or_name = "lmsys/vicuna-7b-v1.5"
tokenizer = LlamaTokenizer.from_pretrained(model_path_or_name)

prompt = 'hi'
tokenizer(prompt, return_tensors="pt")
```
it produces a weird warning that says:
`Keyword arguments {'add_special_tokens': False} not recognized.`

I traced the warning to this [line](https://github.com/huggingface/transformers/blame/c8d98405a8f7b0e5d07391b671dcc61bb9d7bad5/src/transformers/models/llama/tokenization_llama.py#L258) which calls `PreTrainedTokenizer.tokenize` with the keyword argument `add_special_tokens`. It seems that the argument should be handled by `self.prepare_for_tokenization` inside `PreTrainedTokenizer.tokenize`, but it is not; leading to the warning.

Transformers version: 4.38.1. (BTW, 4.37.2 does not have this issue)

Comments:
[`Llama ROPE`] Fix torch export but also slow downs in forward
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Reverts some of the breaking changes introduce in #29109 
The release mentions that we have a breaking change.
This makes it truly BC in the way you access `sin_cache` without memory / tracing / forward issue. 
fixes #29173

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29198). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

+1 @gante there should be tests for torch.compile / torch.compile with `fullgraph=True`

Yes yes!
[`T5 and Llama Tokenizer`] remove warning
Author: ArthurZucker
Pull request: yes

# What does this PR do?
The call to `super().tokenize(text, **kwargs)` included `add_special_tokens` which doe snot exist. Small regression from #28010. Superseeds #29278 as it is time sensitive.

Fixes #29273

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29346). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Better SDPA unmasking implementation
Author: fxmarty
Pull request: yes

As @ArthurZucker improved the unmasking for SDPA for mem-efficient code path let's do so for all archs using SDPA https://github.com/huggingface/transformers/pull/27931

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29318). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

`RUN_SLOW=1 CUDA_VISIBLE_DEVICES=3 pytest tests/ -k "test_eager_matches_sdpa_inference" -s -vvvvv` passes except for qwen2 (but it is unrelated, see https://github.com/huggingface/transformers/pull/28436#issuecomment-1966626609)
[i18n-zh] Sync source/zh/index.md
Author: windsonsea
Pull request: yes

Sync with the [recent en history](https://github.com/huggingface/transformers/commits/main/docs/source/en/index.md).

```
docs/source/zh/index.md
```

For references to model_doc , use the en url currently. It shall be updated after translation in the future.

cc @stevhliu @Fan-Lin @jiaqiw09

Comments:
/lgtm

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29331). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Idefics: generate fix
Author: gante
Pull request: yes

# What does this PR do?

`idefics` generate calls were broken as a result of https://github.com/huggingface/transformers/pull/29114 (changed interface in `_update_model_kwargs_for_generation`, which `idefics` overwrite). An example of failing CI, in another PR: https://app.circleci.com/pipelines/github/huggingface/transformers/85633/workflows/d9db77d3-abb4-46cc-948b-9aa9f80b1dfd/jobs/1108730

This PR fixes it 🤗 

👉 cc @amyeroberts, as you're working on updating the test fetcher. #29114 was an example of failure for the test fetcher. Happy to help on generate test fetching if an extra hand or pointers are needed

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29320). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@gante sorry about that, should have checked for redefinitions

@fxmarty no worries, actually it was our tests' job to catch that 😉 The test fetched failed here
FIX [`CI`]: Fix failing tests for peft integration
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes failing tests in https://github.com/huggingface/transformers/actions/runs/8058719347/job/22012098393 
Some PEFT integration tests do require bitsandbytes, we just need to add `require_bitsandbytes` on the tests that require bnb

cc @ydshieh 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29330). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
FIX [`CI` / `starcoder2`] Change starcoder2 path to correct one for slow tests
Author: younesbelkada
Pull request: yes

# What does this PR do?

As per title cc @ydshieh I can confirm the test pass on my T4 VM



Comments:
ahahah

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29359). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
FIX [`CI`] `require_read_token` in the llama FA2 test
Author: younesbelkada
Pull request: yes

# What does this PR do?

Makes https://github.com/huggingface/transformers/actions/runs/8090459143/job/22107983121 happy

cc @ArthurZucker 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29361). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
check if position_ids exists before using it
Author: jiqing-feng
Pull request: yes

Hi @ArthurZucker  and @younesbelkada 

I think we should check if `position_ids` is `None` or not before we use it.


Comments:
The CI is weird, I can pass the CI locally, could you please help to re-run the CI? Thx!

feel free to merge from main and run `make fixup` to make sure CIs go green

The CI still failed, and it is my local CI result:
![image](https://github.com/huggingface/transformers/assets/107918818/73721238-2fe2-4916-98ab-bac2e911f7ed)


@jiqing-feng I hope you don't mind, I took the liberty to fix the failing test case 🤗 TL;DR when `inputs_embeds` is passed, we can't rely on `input_ids` to get the correct input length

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29306). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> @jiqing-feng I hope you don't mind, I took the liberty to fix the failing test case 🤗 TL;DR when `inputs_embeds` is passed, we can't rely on `input_ids` to get the correct input length

Thanks for your fix!
Port MPT to Flax
Author: kanpuriyanawab
Pull request: yes

# What does this PR do?

This PR adds flax implementation of MPT to transformers

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@sanchit-gandhi @ArthurZucker

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Remove numpy usage from owlvit
Author: fxmarty
Pull request: yes

As per title.

This breaks ONNX export cc @regisss

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29326). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.