noamaanMulla-03/transformers-issues huggingface dataset:
A question about the implementation of Sinkcache.
Author: kisseternity
Pull request: no

### System Info

Ubuntu

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Review the code in the cache_utils.py, for the implementation of SinkCache.

### Expected behavior

In the original paper, it indicates the position ids related to k matrix should be recomputed to focus on positions within the cache rather than those in the original text.
![image](https://github.com/huggingface/transformers/assets/15059072/2e0c2835-2389-4bc7-aaf5-2c1b46609c9b)

However in the implementation code, it seems that the last key_states value uses the original position id instead of recomputation, as the following code shows:
![image](https://github.com/huggingface/transformers/assets/15059072/53fbb31f-4a33-4006-8add-b134fbe44f73)

Please take a look if I am right, thanks.


Comments:
Hey!

Yes, Sink Cache should use position ids inside the cache. For that we firstly rerotate the prev key/values that will remain in cache here. In other words if the cache size is 10 and it's full, we will have to discard the oldest token (except sink tokens) and add a new one at the end. That causes shift by one for all prev tokens except for sink tokens
- https://github.com/huggingface/transformers/blob/74b92c62560b7ade42d35a49f9063adc8b805c4a/src/transformers/cache_utils.py#L659-L679 

The new key/value that we are appending at the end already has a correct position applied to it. If we use Llama model as an example, RoPE is applied inside Attention module just before sending kv to the cache. 

- https://github.com/huggingface/transformers/blob/74b92c62560b7ade42d35a49f9063adc8b805c4a/src/transformers/models/llama/modeling_llama.py#L333-L339

The position ids we used in RoPE are prepared here, because when we use `generate()` we always prepare inputs first by creating position ids from scratch (unless `position ids` were passed by user into generate(), but that's a known limitation which will be fixed soon). So we can be sure that position ids will not get more than cache max length. 

- https://github.com/huggingface/transformers/blob/74b92c62560b7ade42d35a49f9063adc8b805c4a/src/transformers/models/llama/modeling_llama.py#L1253-L1259


Let me know if that makes sense. Also, please note that this kind of questions are better placed in [the forum](https://discuss.huggingface.co/latest) :)

Thanks for your detailed explanation. I ignore the code in _prepare_inputs_for_generation_ func.
Error converting from PyTorch to HuggingFace  - Mistral / Mixtral
Author: efenocchi
Pull request: no

### System Info

A100 40GB
RAM 32GB

### Who can help?
@ArthurZucker, @younesbelkada

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

run the `transformers\src\transformers\models\mistral\convert_mistral_weights_to_hf.py` file.

### Expected behavior

Hi @ArthurZucker, @younesbelkada, 

I fine-tuned Mistral with Torchtune and I was trying to convert the weights from PyTorch to HF but I got an error and saw a couple of other things that could lead to other errors. (file src\transformers\models\mistral\convert_mistral_weights_to_hf.py)

- The first is in line 137 
```py
  state_dict = {
              f"model.layers.{layer_i}.input_layernorm.weight": loaded[0][
                  f"layers.{layer_i}.attention_norm.weight"
              ].clone(),
              f"model.layers.{layer_i}.post_attention_layernorm.weight": loaded[0][
                  f"layers.{layer_i}.ffn_norm.weight"
              ].clone(),
```
the problem is that `post_attention_norm.weight` doesn't exist in the fine-tuned model_*.pt file (not even in the original one) , I think it could be changed with `post_attention_layernorm.weight`.

- I also saw that in the configuration_mistral.py file 2 old repositories are suggested, which are no longer available ([here](https://huggingface.co/mistralai/Mistral-7B-v0.1) instead of [here](https://huggingface.co/mistralai/Mistral-7B-v0.1)).

```
    [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)
    [mistralai/Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1)

```
The `NUM_SHARDS` defaults to 1 for the 7B model and is the only one available `NUM_SHARDS = {"7B": 1} `, I think it would be correct to set the loading to two checkpoints by default, which the user can change to 3 when using the instruct version.

I think there are the same errors in the mixtral folder.

I'm sorry but not having adequate GPUs available I struggle to debug and fix these problems and via notebook it's quite frustrating.


Thank you for your time and efforts.

Emanuele

Comments:
There is probably a missunderstanding, `post_attention_layernorm` is for the HF format: https://github.com/NTHU-ML-2023-team19/transformers/blob/1b61f800fc0e56ffee8428dd65e451279526b534/src/transformers/models/mistral/modeling_mistral.py#L683

Can you share a traceback, of your issue? Here I don't see any problem and the script has been around for quite a while. It was not designed for post torch-tune tho.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add support to the MPS device on the Pipeline class
Author: marlonimg
Pull request: yes

# What does this PR do?

Implements automatic MPS device detection for the Pipeline class.

This PR extends the current functionality of the Pipeline class to automatically detect and leverage MPS devices for inference. This aligns with the behavior of existing accelerator support, streamlining the experience for users running on Apple devices.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@ArthurZucker or @amyeroberts as you each have an mps device, do you mind testing this change?

Sure

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
auto-detect device when no device is passed to pipeline
Author: faaany
Pull request: yes

# What does this PR do?
Currently, if no `device` is passed to `pipeline`, the model will stay on CPU. This PR makes it possible to auto-detect the underlying hardware environment and move the model to the corresponding accelerator. 
```python
import torch 
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline

model_id = "meta-llama/Llama-2-7b-chat-hf"
tokenizer = AutoTokenizer.from_pretrained(model_id)
 
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16)
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, do_sample=False, max_new_tokens=300, num_beams=1)
print(pipe.model.device)
### before fix: cpu
### after fix: cuda
```

@amyeroberts and @muellerzr 


Comments:
@yao-matrix

Hi @ArthurZucker , any thoughts on this PR?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31398). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

How about we emit a warning if we detect there's a device but none is set for the pipeline? 

A warning sounds good, and at this level I feel users would pay attention to it (rather than it just being bloat on the warning logs etc)

Thx for the suggestion! I will update the PR. 

 I think we need to re-trigger the CI

@faaany Could you try rebasing to include upstream changes from main? 

> @faaany Could you try rebasing to include upstream changes from main?

sure, done.
add preprocessing_num_workers to run_classification.py
Author: jiahuanluo
Pull request: yes

preprocessing_num_workers option to speed up preprocess

# What does this PR do?
Add preprocessing_num_workers to speedup tokenizer. 

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@ArthurZucker
<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc 

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Improve error message for mismatched copies in code blocks 
Author: molbap
Pull request: yes

# What does this PR do?

Very tiny PR to improve error message and include the `filename` of the culprit in the traceback. Stumbled upon a case where attributes set before `__init__` would raise this error.

cc @ydshieh @ArthurZucker 

```python
# Copied from transformers...
class MyNewModel():
    attribute = "foo"  # will break check-copies but not point to this class
    def __init__(self...
```


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31535). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Correct way to output attentions in BERT Model
Author: JonathanBhimani-Burrows
Pull request: no

### System Info

- `transformers` version: 4.26.1
- Platform: Linux-6.5.0-17-generic-x86_64-with-glibc2.35
- Python version: 3.9.18
- Huggingface_hub version: 0.13.0
- PyTorch version (GPU?): 1.13.1+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

.

### Expected behavior

Hey everyone,
Thanks in advance for the help! 
I'm running into an issue when I want to output the attentions from a BERT model. The model itself has multiple BERT encoders, one which deals with regular tabular data, and one that deals with tabular-timeseries data 
When I set output_attentions = True, I get a cuda OOM error once the evaluation loop is run (presumably because the attention tensor is still attached to the grad). 
As a result, I then attempted to detach the grad from the tensor before returning to avoid this. This fixed the memory error, but now, when the eval loop is run, it starts at the same iterations/s as train, but progressively slows down, and ends up at 1 it/s around 30% of the way through validation, meaning that the val loop takes 3x longer than a train epoch. 
To speed the process up, I don't return all the attentions, but grab only the first and last layer. 
        o1 = outputs[1][0].detach().cpu()
        o2 = outputs[1][-1].detach().cpu()
        attns = (o1, o2)
        output = (logits,) + attns + outputs[2:]
        return ((loss,) + output) if loss is not None else output

However, even with this, I have the same problem 
Is this, in theory, the correct way to do this? Or is there another way, such as using a callback, that is recommended/preferred by hugging face? 
Thanks,
Jonathan 

Comments:
Hi @JonathanBhimani-Burrows, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

The best way to get help (here and on the forums) is to share as minimal example as possible which enables someones to replicate the issue, as well as any other relevant information. For example, knowing things such as the size of the model, the hardware you're running on; how the training and evaluation loop are set up etc.
fix output data type of image classification
Author: jiqing-feng
Pull request: yes

Hi @Narsil . I found an issue when running the low-precision pipeline of image-classification:
```python
import torch
import requests
from transformers import pipeline
import PIL.Image

IMG_URL = "http://images.cocodataset.org/val2017/000000039769.jpg"
model_id = "google/vit-base-patch16-224"
classifier = pipeline("image-classification", model=model_id, torch_dtype=torch.bfloat16)
image = PIL.Image.open(requests.get(IMG_URL, stream=True, timeout=3000).raw)

output = classifier(image)
```

Error log:
```
Traceback (most recent call last):
  File "test_image_pipeline.py", line 11, in <module>
    output = classifier(image)
  File "/home/jiqingfe/miniconda3/envs/ipex/lib/python3.8/site-packages/transformers/pipelines/image_classification.py", line 157, in __call__
    return super().__call__(images, **kwargs)
  File "/home/jiqingfe/miniconda3/envs/ipex/lib/python3.8/site-packages/transformers/pipelines/base.py", line 1243, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
  File "/home/jiqingfe/miniconda3/envs/ipex/lib/python3.8/site-packages/transformers/pipelines/base.py", line 1251, in run_single
    outputs = self.postprocess(model_outputs, **postprocess_params)
  File "/home/jiqingfe/miniconda3/envs/ipex/lib/python3.8/site-packages/transformers/pipelines/image_classification.py", line 183, in postprocess
    outputs = outputs.numpy()
TypeError: Got unsupported ScalarType BFloat16
```

We need to convert the output tensor to `float32` so it can be converted to `numpy`.


Comments:
Also, for the failed CI, it seems like not all tensors can be converted to `fp32`?

Hi @amyeroberts . Thanks for your review. I have fixed your comments and added a test for the low-precision pipeline, but the CI seems to be going wrong. Can you help check the test? Thx!

And other task pipelines also have this issue. I would like to hear your opinion and then change the rest of pipelines :)

Hi @amyeroberts. Can you take a look at the new changes? If it is okay for you, I will update all other pipelines to support low precision.

Hi @amyeroberts . Could you please take a review? The failed CIs are not related to my changes :)

I am waiting for your comments to change the rest of the pipelines to support it. Thx!

Hi @amyeroberts . I have rebased and passed all CIs. Would you please review them?

Hi @amyeroberts . I have added `fp16` and `bf16` tests for the image classification pipeline; could you please review them? Thx!

HI @amyeroberts . I have fixed the import issue, please take a review, thx!
`TokenizerConverters` make sure to add based on type
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Update gemma

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30973). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

fyi @itazap a bug from sentencepiece 🫠
gh: consider `i18n`
Author: SauravMaheshkar
Pull request: no
Labels: name: Feature request

### Feature request

Perhaps we could move the translated versions of the README to a `i18n/` repository.

### Motivation

AFAIK this is considered to be the usual place for translations, viz. https://github.com/supabase/supabase/blob/master/i18n/languages.md

### Your contribution

Happy to submit a PR if interested

Comments:
cc @stevhliu 

Good idea! Feel free to open a PR for it 🙂
docs: move translations to `i18n`
Author: SauravMaheshkar
Pull request: yes

# What does this PR do?

Fixes #31542 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Request for Review: @stevhliu


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31584). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
`self.projection` is unnecessarily called twice in `VivitTubeletEmbeddings`
Author: v-iashin
Pull request: no

### System Info

the problem is on `main` and has been there since ViViT was added.

### Who can help?

@amyeroberts

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://github.com/huggingface/transformers/blob/0f67ba1d741d65b07d549daf4ee157609ce4f9c1/src/transformers/models/vivit/modeling_vivit.py#L75-L80

### Expected behavior

```python
# permute to (batch_size, num_channels, num_frames, height, width)
pixel_values = pixel_values.permute(0, 2, 1, 3, 4)

# out_batch_size, out_num_channels, out_num_frames, out_height, out_width = x.shape
x = self.projection(pixel_values).flatten(2).transpose(1, 2)
```
should be enough.


i can make a minor pr to fix it.

Comments:
Eek - indeed - thanks for raising! If you open a PR to fix, you can ping me for a quick review and you'll get the github contribution for the fix :)
ImportError: cannot import name 'AutoTokenizer' from 'transformers'
Author: Fushier
Pull request: no

transformers: 4.6.1
tokenizers: 0.10.3


I installed transformers with 
`conda install -c huggingface transformers`
but when I `from transformers import AutoTokenizer`

Traceback (most recent call last):
  File "D:/IIE/WorkSpace/Pycharm WorkSpace/HuggingfaceNER/tokenizers.py", line 1, in <module>
    from transformers import AutoTokenizer
  File "D:\Environment\Anaconda3\envs\huggingface\lib\site-packages\transformers\__init__.py", line 48, in <module>
    from .data import (
  File "D:\Environment\Anaconda3\envs\huggingface\lib\site-packages\transformers\data\__init__.py", line 6, in <module>
    from .processors import (
  File "D:\Environment\Anaconda3\envs\huggingface\lib\site-packages\transformers\data\processors\__init__.py", line 5, in <module>
    from .glue import glue_convert_examples_to_features, glue_output_modes, glue_processors, glue_tasks_num_labels
  File "D:\Environment\Anaconda3\envs\huggingface\lib\site-packages\transformers\data\processors\glue.py", line 25, in <module>
    from ...tokenization_utils import PreTrainedTokenizer
  File "D:\Environment\Anaconda3\envs\huggingface\lib\site-packages\transformers\tokenization_utils.py", line 26, in <module>
    from .tokenization_utils_base import (
  File "D:\Environment\Anaconda3\envs\huggingface\lib\site-packages\transformers\tokenization_utils_base.py", line 31, in <module>
    from tokenizers import AddedToken
  File "D:\IIE\WorkSpace\Pycharm WorkSpace\HuggingfaceNER\tokenizers.py", line 1, in <module>
    from transformers import AutoTokenizer
ImportError: cannot import name 'AutoTokenizer' from 'transformers' (D:\Environment\Anaconda3\envs\huggingface\lib\site-packages\transformers\__init__.py)

It even worked well yesterday, and I didn't upgrade anything...


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

same issue

你好 我们是一样的问题 你解决了吗？

Same Issue :(

@ZaidArman Could you confirm your running environment, how transformers was installed and a code snippet to reproduce? I'm unable to replicate this on `main`
Fix llama gguf converter
Author: SunMarc
Pull request: yes

# What does this PR do?
This PR fixes the failing gguf tests caused by this [PR](https://github.com/huggingface/transformers/pull/31305). Instead of using the converter methods from `SpmConverter` class for `GGUFLlamaConverter` , we copy it + remove the parts that are using `pieces` and `trainer_spec.control_symbols` attributes from `self.proto` that comes from the sentencepiece (https://github.com/google/sentencepiece/blob/6225e08edb2577757163b3f5dbba4c0b670ef445/src/sentencepiece_model.proto#L299C29-L299C33). `GGUFTokenizerSkeleton`  don't have these attributes and I don't think we should add them.

Fixes https://github.com/huggingface/transformers/issues/31553 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31575). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update RT-DETR code snippet
Author: qubvel
Pull request: yes

# What does this PR do?

Fix RT-DETR code snippet in docs + add detection output 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31631). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.