noamaanMulla-03/transformers-issues huggingface dataset:
Failure to produce exact input sequence from output logits
Author: hxiaoyang
Pull request: no

### System Info

- `transformers` version: 4.35.2
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.20.1
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): 2.15.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.5 (cpu)
- Jax version: 0.4.23
- JaxLib version: 0.4.23
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
prompt = "Please provide a code sample that reproduces the"
input_ids = torch.tensor([tokenizer.encode(prompt)])
with torch.no_grad():
    outputs = model(input_ids)
string = tokenizer.decode(torch.argmax(outputs.logits[0], dim=1).tolist(),skip_special_tokens=True)
print(string)
```
Output: `note the detailed snippet of demonstrces the issue`

### Expected behavior

I was expecting `string` to be `Please provide a code sample that reproduces the issue`, supposing that `issue` is the generated token here. What's the right way to produce the exact input sequence from output logits?

Comments:
You should use the `generate` function. Otherwise I have no idea what model you are using, but this is expected: the model does not necessarily predict the token you give as an input. Assisted decoding relies on this property.  

```python
prompt = "Please provide a code sample that reproduces the"
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("gpt2")
input_ids = tokenizer(prompt, return_tensors = "pt")
string = tokenizer.batch_decode(model.generate(**input_ids).tolist(),skip_special_tokens=True)
print(string)
['Please provide a code sample that reproduces the code.\n\nIf you are interested in contributing to']
```
use `max_new_tokens` if needed. 

Thank you for the timely response! I've been using `llama-2-13b-chat-hf` FYI.

A followup question: I'm building a conversational / recursive LM. For my use case, it's important that the previous conversation is not reprocessed. In other words, only previous hidden states and new context should be used. Is this the default behavior of `model.generate()`? If not, what would be the best way to reuse previous hidden states.

Thanks! @ArthurZucker

By default the `generate` function will use the `past_key_values` instead of re-computing everything which is probably what you are looking for 😉 the past key values still take into account the previous context in an efficient manner. See an answer [here](https://discuss.huggingface.co/t/what-is-the-purpose-of-use-cache-in-decoder/958)

Thank you!
Update VITS modeling to enable ONNX export
Author: echarlaix
Pull request: yes

This PR enables the ONNX export of VITS models in Optimum (https://github.com/huggingface/optimum/pull/1607), currently the export is failing due to [a cast operator added before the pow operator](https://github.com/pytorch/pytorch/blob/v2.1.2/torch/onnx/symbolic_opset9.py#L3382) in the model graph, resulting in an issue during the concatenation of two values of different data type


cc @xenova 







Comments:
> For futur reference best practice is this only when it power of 2 ? does pow(x,2) work better (but is slower I think)

In all cases it'll be [casted to fp32](https://github.com/pytorch/pytorch/blob/v2.1.2/torch/onnx/symbolic_opset9.py#L3386) (no matter the exponent), using a [multiplication](https://github.com/onnx/onnx/blob/main/docs/Operators.md#mul) instead removes this constraint, and should be "faster" (not that it'll have any impact here)
Fixed spelling error in T5 tokenizer warning message (s/thouroughly/t…
Author: jeddobson
Pull request: yes

# Spelling correction 

This is a simple word change for a warning message generated by the T5 tokenizer ('src/transformers/models/t5/tokenization_t5.py') that appeared when converting LLAMA weights to HF format. 



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28014). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add SeamlessM4T v2
Author: ylacombe
Pull request: yes

# What does this PR do?

Meta just released [Seamless M4T v2](https://ai.meta.com/resources/models-and-libraries/seamless-communication-models/#seamlessm4t), this model has enough difference to justify a new version: differences listed [here](https://github.com/ylacombe/transformers/blob/add-S2S-2/docs/source/en/model_doc/seamless_m4t_v2.md#difference-with-seamlessm4t-v1) in the .md

@ArthurZucker reviewed most of the files internally but a few last changes might require a final approval!

cc @ArthurZucker @LysandreJik @sanchit-gandhi 

Comments:
I'm astonished you got this added so quickly! Do you have plans to support seamless streaming as well?

I am unable to see it on the transformers library yet. How can i test this out ?

We'll do a release this week and it will be part of it! You have to install from source for now `pip install -e git+https://github.com/huggingface/transformers`

No plans yet for seamless streaming since the architecture diverges somewhat from Seamless M4T. The license is non-commercial unfortunately, which is also a barrier to usage. But if anyone in the community is interested in adding the model I'd sure be happy to lend a hand with the integration by answering any questions/queries and reviewing PRs! WDYT @ylacombe @yclicc - should we open a feature request up to the community for this?
chore: Fix typo s/exclusivelly/exclusively/
Author: hugo-syn
Pull request: yes

# What does this PR do?

Just fix some typos in the code/doc

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Sorry if you are not the right person, but I think that _documentation_ should be appropriate @stevhliu  @MKhalusova 



Comments:
RuntimeError: Could not infer dtype of JpegImageFile
Author: realbigi
Pull request: no

### System Info

transformers, windows 11, python 3.11.6

I have been trying to replicate this https://huggingface.co/docs/transformers/tasks/image_classification image classification with PyTorch. But when I run the skript I get this error:

RuntimeError: Could not infer dtype of JpegImageFile

Does anyone have an idea what the problem could be?

### Who can help?

@amyeroberts

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

https://github.com/realbigi/Classifier_2_Pytorch_Image_Classification

### Expected behavior

It should train on the data without a mistake

Comments:
Hi @realbigi, thanks for raising an issue! 

The error itself is arising because torch can't decode jpeg files. From the notebook, it's not immediately obvious what's happening, as the images should be opened as PIL.Image.Image. I can see there was an error reading in the dataset and some of the cells have been run out of order. Because of the statefulness of notebooks, it is therefore difficult to debug wrt to the error message. Could you update the notebook so that each cell has been executed in order? 

Hi @amyeroberts 

I just executed the notebook in order and reuploaded it. But you should be able to run it anyways as I uploaded the data as well to GitHub. Could the problem arise because there are webp images instead of jpeg's? Altough I don't think that is the problem because they were converted to PIL objects right?

@amyeroberts 

I must have made a mistake while copying the code. I downloaded the original notebook and loaded my data and it seems to work. 

Thanks for clarifying ! 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
`hub_strategy`'s documentation for `checkpoint` option is wrong and misleading
Author: omermazig
Pull request: no

### System Info

The documentation for [hub_strategy](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.hub_strategy) seems to be mistaken, because it stated:


> "checkpoint": like "every_save" but the latest checkpoint is also pushed in a subfolder named last-checkpoint, allowing you to resume training easily with trainer.train(resume_from_checkpoint="last-checkpoint").

But when I did that, I got:

> ValueError: 'videomae-finetuned/checkpoint-3000' is not in list


Where checkpoint 3000 was my **best** checkpoint (according to my `metric_for_best_model`)

So for `resume_from_checkpoint` to work, I found that I need to have both the last checkpoint AND the best checkpoint available in the output dir of the `Trainer`, and pass resume_from_checkpoint=True.

### Who can help?

@muellerzr and @pacman100

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

On any training process using the trainer, pass `hub_strategy="checkpoint"`, get the saved checkpoint from the hub, and run:

`trainer.train(resume_from_checkpoint="local/path/to/checkpoint"`

### Expected behavior

Other than fixing the docs, I think there should be a "best_and_last" option, which updates both the best (if changed) and last checkpoints in the hub, and that way really support resuming from a checkpoint.

A more complex solution would be to incorporate it with the `save_total_limit` mechanism - since both are "keeping" checkpoints - but that's a bigger change.

Comments:
A PR to the docs would be great, I also believe that if we use `save_total_limit`, it should take into account *not* deleting the one that has the best metric, and looking at the oldest scale to delete before that. I can look into this if it's a bit too complex on your end 😄 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Bin format tests
Author: LysandreJik
Pull request: yes

As requested in https://github.com/huggingface/transformers/pull/27231#issuecomment-1790681029 and https://github.com/huggingface/transformers/pull/27231#pullrequestreview-1710357492

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`tests`] Extend training tests to include half-precision (bf16) and automatic mixed precision training
Author: younesbelkada
Pull request: yes

# What does this PR do?

As per title and to specifically address: https://github.com/huggingface/transformers/pull/27220#pullrequestreview-1709756019

This PR aims to extend the current testing suite to include pure bfloat16 training (I think it is not possible to perform pure float16 training) + automatic mixed precision training through the `torch.amp` API: https://pytorch.org/tutorials/recipes/recipes/amp_recipe.html 

That way we can easily check which model supports mixed precision training and support that from day 0 of the model integration

cc @amyeroberts @ydshieh 

Draft for now as many tests are failing


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27235). All of your documentation changes will be reflected on that endpoint.

I did not had time to properly finish this PR, right now the blocker is that it appears for some architectures that some gradients are set to `None` when using GC even if I make sure that the forward pass has been correctly called. Except skipping the test for these archs, I am not sure how we should proceed here

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
itm_score output for BLIP2
Author: fferroni
Pull request: no

### Feature request

Would it be possible to add outputting ITM/ITC scores for BLIP2? It is currently supported for BLIP v1.

### Motivation

LAVIS already contains the image-text matching capability here
https://github.com/salesforce/LAVIS/blob/3446bac20c5646d35ae383ebe6d13cec4f8b00cb/lavis/models/blip2_models/blip2_image_text_matching.py


### Your contribution

Given some pointers on which objects/layers correspond to the LAVIS version vs `transformers` version, I would also try adding this.

Comments:
Hi @fferroni , maybe this PR https://github.com/huggingface/transformers/pull/25612 could be related

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add links to the Hub
Author: LysandreJik
Pull request: yes

Adds a few additional links to the Hub. If there are ideas to do this even deeper, would love to incorporate them in this PR.
I would love loved to share initial collections as well but I have to check if I can embed collection visualisation in the doc page + it's not relevant for quite a few architecures.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26694). All of your documentation changes will be reflected on that endpoint.

I think with the badges (but rename it to All model _checkpoints_ instead of All model pages) it is more visible than the text which doesn't stand out as much. 

Great idea, however is this currently hardcoded? Would be great to add it to the CookieCutter so that new models have this as well. Also more a fan of the badge for visibility.

Begone bot, I still want to get to this

I updated everything to use the tags.

If this looks good to you all I'll add it to cookie cutter + add a script to verify it's indeed in the docs.

That's the script I used to add it everywhere:

```py
import os
from pathlib import Path


def get_div_text(model_type):
    return f"""<div class="flex flex-wrap space-x-1">
<a href="https://huggingface.co/models?filter={model_type}">
<img alt="Models" src="https://img.shields.io/badge/All_model_pages-{model_type}-blueviolet">
</div>

"""


path_to_docs = Path('/path_t_transformers/transformers/docs/source/en/model_doc')
model_docs = os.listdir(path_to_docs)
print(model_docs)

for model_doc in model_docs:
    with open(path_to_docs / model_doc, 'r+') as f:
        doc = f.read()

    div = doc.split('#')[1]
    if '<div class="flex flex-wrap space-x-1">' in div:
        print('✅', model_doc)
    else:
        print('❌', model_doc)
        new_doc = doc.split('#')
        new_doc[1] = new_doc[1] + get_div_text(model_doc[:-3])

        with open(path_to_docs / model_doc, 'w') as f:
            f.write('#'.join(new_doc))
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Trainer subclasses should be able to replace Accelerator / Optimizer
Author: frankier
Pull request: no

### Feature request

It's possible that some in-depth customization of the `Trainer` class may require changes to Accelerator. There is already a pattern with the optimizer that the class and arguments are obtained from:

```python
@staticmethod
def get_optimizer_cls_and_kwargs(args: TrainingArguments) -> Tuple[Any, Any]:
```

So it would be nice to have a corresponding

```python
@staticmethod
def get_accelerator_cls_and_kwargs(args: TrainingArguments) -> Tuple[Any, Any]:
```

### Motivation

Customisability

### Your contribution

I am going to submit a PR

Comments:
Okay, I've now realised that this won't work unless it's `@classmethod`, so perhaps `get_optimizer_cls_and_kwargs` should be changed for this too? Delaying working on a PR until this has been discussed.

cc @pacman100 @muellerzr 

@frankier (sorry for the delay) why does this require it being a class method? It should just be static I believe unless you're seeing something I'm not

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
BERT model is slow in Pytorch
Author: akote123
Pull request: no

### System Info

transformers                 4.35.2
tensorflow                    2.15.0
torch                            2.1.1

- `transformers` version: 4.35.2
- Platform: Linux-6.2.0-1016-aws-aarch64-with-glibc2.35
- Python version: 3.11.0
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.0
- Accelerate version: 0.24.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.1 (False)
- Tensorflow version (GPU?): 2.15.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction


TF code is as follows:
[tf_bert.txt](https://github.com/huggingface/transformers/files/13499939/tf_bert.txt)
Pytorch code is : 
[pytorch_bert.txt](https://github.com/huggingface/transformers/files/13499940/pytorch_bert.txt)



### Expected behavior

I have tried BERT uncased model with Tensorflow and Pytorch but Pyt is slower . In pytorch if I increase the batch size to 512 the system gets hang.

Comments:
Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
How to log custom value?
Author: xmy0916
Pull request: no

I want to log some info to `{'loss': 2.5234, 'learning_rate': 1.0344827586206896e-06, 'epoch': 0.0}`
how can i do that?
like: {'loss': 2.5234, 'learning_rate': 1.0344827586206896e-06, 'epoch': 0.0, 'version': 'v1'}  

Comments:
Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!

Thanks!
Add support for PyTorch/XLA 2.1+ PJRT (Neuron SDK)
Author: jeffhataws
Pull request: yes

# What does this PR do?

In PyTorch/XLA 2.1+, `init_method="xla://"` is now needed in `dist.init_process_group` to properly initialize PJRT.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@muellerzr and @pacman100


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27800). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

We don't need this anymore as we have overwritten init_process_group within torch-neuronx to have pjrt as default. You might need this for torch-xla for other platforms.

@jeffhataws are we okay to close this then? Or are you saying this needs to be merged. 

We can close this. Thanks!
Is there any difference between hf-implemented llama2 and facebook-implemented llama2 ?
Author: artetaout
Pull request: no

In my view, hf-implemented llama2, the "intermediate size" is 11008, but in facebook-implemeted llama2, it's 4 * hidden_size ?
why?


Comments:
Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Support setting multiple adapters
Author: pbarker
Pull request: no
Labels: name: Feature request

### Feature request

The underlying peft library supports setting multiple adapters:

```python
model.set_adapters(["adapter_a", "adapter_b"])
```

It would be nice if the pipeline supported the same, from looking at https://github.com/huggingface/transformers/pull/25077 it appears it only supports a single adapter

### Motivation

This is useful functionality in the peft library

### Your contribution

Happy to make the changes here!

Comments:
cc @younesbelkada