noamaanMulla-03/transformers-issues huggingface dataset:
[i18n-fr] Translate autoclass tutorial to French
Author: NoB0
Pull request: yes

# What does this PR do?

Translated the autoclass_tutorial.md file of the documentation to French.

Part of #21456

Thank you in advance for your review.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

French speaking contributors.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27659). All of your documentation changes will be reflected on that endpoint.

Thanks for the translation!
fix: fix gradient accumulate step for learning rate
Author: pphuc25
Pull request: yes

Hi, 
I think this is the mistake seen the learning rate should take the param `num_train_steps` as `total_train_steps`, instead of `len(vectorized_datasets["train"])`
So I fix them to the right suitable that learning rate will go down to zero as init
I would like to cc @sanchit-gandhi to review my PR

Comments:
Getting token probabilities of a caption given an image from BLIP2
Author: snpushpi
Pull request: no

I was considering the BLIP2 model for getting the probability distribution of each token in the caption given an image. So basically if the words in a caption are w1,w2,w3,…wt then I want to get these estimates P(w1|image), P(w2|image,w1),P(w3|image,w1,w2) etc. So this is the approach I took -

`
from PIL import Image
import requests
from transformers import Blip2Processor, Blip2ForConditionalGeneration
import torch

device = "cuda" 

processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
model = Blip2Model.from_pretrained(
    "Salesforce/blip2-opt-2.7b", torch_dtype=torch.float16
)
model.to(device)

url = "http://images.cocodataset.org/val2017/000000485895.jpg"
image = Image.open(requests.get(url, stream=True).raw)
pixel_values = processor(images=image, return_tensors="pt").pixel_values
pixel_values = pixel_values.to(device, torch.float16)

sentence = 'A giraffe stares into the camera while standing on green grass in front of a shade tree.'
input_ids = processor.tokenizer(sentence, return_tensors = 'pt').input_ids.to(device)

output = model(pixel_values,input_ids = input_ids).logits.detach()

considered_logits = output[:,32:,:]
`
So the considered logits is the probability distribution of each token in the caption, am I doing it right?

Comments:
Hey! I would  like to work on this issue.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @snpushpi, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Deprecate `LegacyIndex` used in `RagRetriever`
Author: ydshieh
Pull request: yes

# What does this PR do?

A first step to reduce the security issue of pickle for `RAG`.

In general, we can also ask users to set some environment variable to enable using the code block that contains `pickle.load` to have even better security. But that could be done in a follow up PR (which also applies to `TransfoXLTokenizer`)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27748). All of your documentation changes will be reflected on that endpoint.

Hi @LysandreJik 

After #27776, this PR doesn't add much value. Let me know if we would like to close this or still keep this deprecation.
update `create_model_card` to properly save peft details when using Trainer with PEFT
Author: pacman100
Pull request: yes

# What does this PR do?
1. When using Trainer with PEFT, `model.save_pretrained` in PEFT adds PEFT-specific details to the existing model card or creates a new model card and adds these details. However, `trainer.create_model_card` is called after model is saved and it overwrites the entire file thereby nullifies the addition of details related to PEFT such as library name, quantization used and PEFT version.
2. This PR fixes the above issue and thereby adds backs the PEFT details to the model card. This will help in better organization and understanding usage of PEFT on Hub. 
3. Example of the repo with the PR usage: https://huggingface.co/smangrul/mistral_lora_clm_with_added_tokens

Comments:
> From my understanding, this seems a little bit hacky to me, because we first create the model card with PEFT, then completely overwrite it with Trainer, then re-add PEFT-related content. It feels like the proper solution would be for the Trainer to update the model card if it already exists. But I understand that this would be more work, so I'd be okay with this more hacky approach.

Yes, Trainer should ideally update if README is already there, but it rewrites everything from the TrainerState and as such appending/updating would be trickier. Open to ideas on making this cleaner.

I can't really see a better solution currently either, so what we have here works for now
Fix device of masks in tests
Author: fxmarty
Pull request: yes

As per title, fix a bug introduced in https://github.com/huggingface/transformers/pull/24587 in the GPU tests.

Comments:
Strange that we don't see the issue on our CI. Do you know why @fxmarty?

@ydshieh Can you try `RUN_SLOW=1 CUDA_VISIBLE_DEVICES=0 pytest tests/models/llama -s -vvvvv -k "test_resize_tokens_embeddings"` on main?

I am not sure it's weird. The `push` tests should run on GPU so those should have failed...
https://github.com/huggingface/transformers/blob/fc71e815f6ff5fc8b743786e2146cff9644ec598/.github/workflows/self-push.yml#L133

As discussed offline it did actually fail (https://github.com/huggingface/transformers/actions/runs/7124821720) so all good!
Fix beam score calculation issue for JAX version
Author: VsonicV
Pull request: yes

# What does this PR do?
This PR fixes the same beam score calculation issue following #27351 , for JAX version.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@gante 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27816). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix id_tensor_storage in case the tensor is a view of an other tensor
Author: fxmarty
Pull request: yes

As per title and discussed offline @LysandreJik 

Currently `safe_serialization=True` may unexpectedly erase tensors from the state_dict due to this issue.

A release or patch release including https://github.com/huggingface/safetensors/pull/379 will be needed to work with this PR, to avoid this error with the current latest release of safetensors:

```
E           RuntimeError: 
E                       Some tensors share memory, this will lead to duplicate memory on disk and potential differences when loading them again:
```
here: https://github.com/huggingface/safetensors/blob/96061e97bb7fc4ea6cdd1f79f58701efc4710d22/bindings/python/py_src/safetensors/torch.py#L467

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27314). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Show new failing tests in a more clear way in slack report
Author: ydshieh
Pull request: yes

# What does this PR do?

It's now difficult to see the **new** failing tests on our Slack CI report, as they are > 100 failures. Also, scrolling over the many slack reply responses is time consuming to get the right place.

This PR add the new failing tests in 2 places: 

  - the end of post
  - the end of the replies to the post

Links are provided too.

With this, it's easy and fast to access the new failing tests.

Screenshot

<img width="1016" alt="Screenshot 2023-12-07 113217" src="https://github.com/huggingface/transformers/assets/2521628/9b0c85c9-589c-46f9-af7e-04c168294b88">


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27881). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Generate: All logits processors are documented and have examples
Author: gante
Pull request: yes

# What does this PR do?

This PR retouches the docstrings of the logits processors such that:
a) they all have a short description of what they do
b) there is a clear example for each class, ideally showcasing its usefulness (note: this will also means the class is doctested)

Moreover, to ensure this information is found, cross references to these classes were added.

Related issue #24575 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27796). All of your documentation changes will be reflected on that endpoint.
Generate: skip tests on unsupported models instead of passing
Author: gante
Pull request: yes

# What does this PR do?

Skip tests on unsupported models instead of passing, as discussed in #25086 

NOTE: I've skipped to whole test in the loop over different heads as soon as it encountered a problem because a) the reason for the skips in generate is always due to some core property of the architecture (AFAIK) b) simplicity


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Translate `model_doc` files from `clip` to `cpm`  to JP
Author: rajveer43
Pull request: yes

# What does this PR do?

translation of model docs

<!-- Remove if not applicable -->

Fixes #27773


## Who can review?

Documentation: @stevhliu and @MKhalusova




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27774). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[i18n-<Jp>] Translating docs to JP
Author: rajveer43
Pull request: no
Labels: name: WIP

<!--
Note: Please search to see if an issue already exists for the language you are trying to translate.
-->

Hi!

Let's bring the documentation to all the jp speaking community 🌐
Who would want to translate? Please follow the 🤗 [TRANSLATING guide](https://github.com/huggingface/transformers/blob/main/docs/TRANSLATING.md). Here is a list of the files ready for translation. Let us know in this issue if you'd like to translate any, and we'll add your name to the list.

Some notes:

* Please translate using an informal tone (imagine you are talking with a friend about transformers 🤗).
* Please translate in a gender-neutral way.
* Add your translations to the folder called `jp` inside the [source folder](https://github.com/huggingface/transformers/tree/main/docs/source).
* Register your translation in `jp/_toctree.yml`; please follow the order of the [English version](https://github.com/huggingface/transformers/blob/main/docs/source/en/_toctree.yml).
* Once you're finished, open a pull request and tag this issue by including #issue-number in the description, where issue-number is the number of this issue. Please ping @stevhliu and @MKhalusova for review.
* 🙋 If you'd like others to help you with the translation, you can also post in the 🤗 [forums](https://discuss.huggingface.co/).

## Get Started section


Comments:
[⚠️ removed a default argument] Make `AttentionMaskConverter` compatible with `torch.compile(..., fullgraph=True)`
Author: fxmarty
Pull request: yes

As per title, fixes https://github.com/huggingface/transformers/issues/27789.

This issue is only for PyTorch 2.1 and has been fixed in torch nightly.

Comments:
Also, `AttentionMaskConverter` is not in the documentation so not really user-facing.
[Seamless v2] Add FE to auto mapping
Author: sanchit-gandhi
Pull request: yes

As reported by @Vaibhavs10, Seamless M4T v2 is not compatible with the pipeline class. This is because the feature extractor is not present in the auto mapping. This PR adds the FE to the auto mapping, and implements a slow test to check Seamless M4T v2 works as expected with the pipeline.

### Example Usage

No chunking:
```python
from transformers import pipeline
import torch
from datasets import load_dataset

pipe = pipeline(
    "automatic-speech-recognition",
    model="facebook/seamless-m4t-v2-large",
    device="cuda:0",
    torch_dtype=torch.float16,
)

dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
sample = dataset[0]["audio"]

# eng -> fra speech translation
result = pipe(sample, generate_kwargs={"tgt_lang": "fra"})
```

Chunking (for long audio files):

```python
from transformers import pipeline
import torch
from datasets import load_dataset

pipe = pipeline(
    "automatic-speech-recognition",
    model="facebook/seamless-m4t-v2-large",
    device="cuda:0",
    chunk_length_s=30,
    batch_size=16,
    torch_dtype=torch.float16,
)

dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
sample = dataset[0]["audio"]

# eng -> fra speech translation
result = pipe(sample, generate_kwargs={"tgt_lang": "fra"})
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27829). All of your documentation changes will be reflected on that endpoint.

Thanks for taking care of this @sanchit-gandhi !
Fix: Raise informative exception when `prefix_allowed_tokens_fn` return empty set of tokens
Author: Saibo-creator
Pull request: yes

# What does this PR do?

Fixes #27676  #22890 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? https://github.com/huggingface/transformers/issues/22890 and  https://github.com/huggingface/transformers/issues/27676
- [x] Did you make sure to update the documentation with your changes?  I don't think there is anything to update in the doc
- [x] Did you write any new necessary tests?


## Who can review?
@gante @ArthurZucker 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The failing tests seem to be irrelevant 

Rebase on main will help, It will get fix there 

@ArthurZucker  Rebased 

(I'd commit Arthur's suggestion, as it saves a redundant computation :) )

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27797). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> (I'd commit Arthur's suggestion, as it saves a redundant computation :) )

@gante 
I somehow missed the suggestion from Arthur. Thanks for reminding me! It's now committed :)
_prepare_4d_causal_attention_mask doesn't work with torch.compile
Author: PhilJd
Pull request: no

### System Info

Transformers version: 4.35.2
Pytorch version: 2.1.1+cu118
Python version: 3.11

### Who can help?

Tagging @ArthurZucker as the original author

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Code to reproduce
```python
import torch
import torch.nn as nn
from transformers.modeling_attn_mask_utils import _prepare_4d_causal_attention_mask

# LLama models use this wrapper but this does not fix the compilation issue.
# if is_torch_fx_available():
#    _prepare_4d_causal_attention_mask = torch.fx.wrap(_prepare_4d_causal_attention_mask)

class Model(nn.Module):

    def forward(self, inputs_embeds):
        batch_size, seq_length, _ = inputs_embeds.shape
        past_key_values_length = 10
        attention_mask = _prepare_4d_causal_attention_mask(
            None, (batch_size, seq_length), inputs_embeds, past_key_values_length
        )
        return attention_mask

model = Model()
model = torch.compile(model, fullgraph=True)
model(torch.ones([1,5, 32]))
```
Gives a torch.compile error:
```
[...]
    return SourcelessBuilder()(tx, val).add_options(options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.virtualenvs/pytorch311/lib/python3.11/site-packages/torch/_dynamo/variables/builder.py", line 1650, in __call__
    unimplemented(f"Unexpected type in sourceless builder {type(value)}")
  File "/home/ubuntu/.virtualenvs/pytorch311/lib/python3.11/site-packages/torch/_dynamo/exc.py", line 172, in unimplemented
    raise Unsupported(msg)
torch._dynamo.exc.Unsupported: Unexpected type in sourceless builder <class 'torch.dtype'>

from user code:
   File "<stdin>", line 6, in forward
  File "/home/ubuntu/.virtualenvs/pytorch311/lib/python3.11/site-packages/transformers/modeling_attn_mask_utils.py", line 197, in _prepare_4d_causal_attention_mask
    attention_mask = attn_mask_converter.to_causal_4d(
```

Note that I could reproduced this by e.g. trying to compile a Llama2 model.

Removing the dtype argument and just setting dtype=torch.float32 in the function body compiles.
I'm not sure whether this is a bug in the torch compiler or unintended usage in transformers, so happy to open an issue there as well if required :)


### Expected behavior

No errors during torch.compile stge.
Let me know if there's something I can do to help debug!

Comments:
Ow nice catch and nice reproducer! 🤗 
I'll try to have a look, otherwise pinging @fxmarty 

Hi @PhilJd the issue is fixed in https://github.com/huggingface/transformers/pull/27868. This a bug in PyTorch 2.0/2.1, that is solved in pytorch nightly.

Thanks a lot! :)
Fix beam score calculation issue for Tensorflow version
Author: VsonicV
Pull request: yes

# What does this PR do?
This PR fixes the same beam score calculation issue following #27351 , for Tensorflow version.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@gante 

Comments:
@gante I have updated the PR following your suggestions. Now the `cur_len` represents the length of the entire sequence including the decoder prompt, which is consistent with the codebase.
Fix beam score calculation issue for Tensorflow
Author: VsonicV
Pull request: yes

# What does this PR do?
This PR fixes the same beam score calculation issue following #27351 , for Tensorflow version.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@gante 

Comments:
This PR is mixed with another PR. Close it for now. Will reopen another PR only with the relevant commits:  #27814 .