noamaanMulla-03/transformers-issues huggingface dataset:
Inexplicable shape error
Author: Murasame-sway
Pull request: no
Labels: name: bug

### System Info

python 3.10
torch 2.0.1+cu118
transformers 4.38.0

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Because of some special needs, I needed to pass inputs_embed directly to the llm's generate method, but I encountered some unexplained errors, so I made sure that the shape of inputs_embed was [batch*seq_len*hidden_dim]

here is my test code：
```
from transformers import GemmaForCausalLM
import torch

model = GemmaForCausalLM.from_pretrained(
    "google/gemma-2-9b-it",
    device_map="auto",
    torch_dtype=torch.bfloat16,
)
input=torch.zeros([1,625,3584],dtype=torch.bfloat16).cuda()
model.generate(inputs_embeds=input,max_new_tokens=32)
```

and the error info is:
![image](https://github.com/user-attachments/assets/377d88cd-4a77-45f4-8d58-32d91e94c5fc)







### Expected behavior

output normally

Comments:
It seems like the right thing to do is update transformers and use Gemma2ForCausalLM(with the accompanying issue of updating torch)
NER workflow improvement
Author: ain-soph
Pull request: no
Labels: name: Core: Pipeline

name: Ex: Named Entity Recognition

name: Feature request

### Feature request

1. [run_ner.py in examples](https://github.com/huggingface/transformers/blob/main/examples/pytorch/token-classification/run_ner.py) are requiring data of pre-tokenized words, like [conll2003 dataset](https://huggingface.co/datasets/eriktks/conll2003), where texts have been splitted into words. However, the [TokenClassificationPipeline](https://huggingface.co/docs/transformers/v4.44.0/en/main_classes/pipelines#transformers.TokenClassificationPipeline) has no such pre-tokenization but simply aggregate pre-entities by checking [is_subword](https://github.com/huggingface/transformers/blob/984bc11b0882ff1e5b34ba717ea357e069ceced9/src/transformers/pipelines/token_classification.py#L396) using whitespace separator.
This might work somehow on English words, but would fail when it is inconsistent with pre-tokenization used in training data (which would raise multiple `B-` for all sub-tokens in one pre-token)
**My solution:** In the `TokenClassificationPipeline`, use the pre-tokenizer to mark `is_subword` for all non-first sub-tokens to aggregate them in `gather_pre_entities`.
2. `run_ner.py` shall support chunks during training. For example, XLM-Roberta-Base supports 512 as the maximum sequence length, and we might suffer with memory concern to limit the sequence length during training for large models. In this case, it might be better to split long training data into chunks (with certain overlapping) to fully utilize the information instead of truncation.
3. For `label_all_tokens` argument in `run_ner.py`, it means we currently have 2 options for sub-token labelling: `first` and `all`. It might be better to provide a `last` option to better support for Decoder-only models. See https://github.com/huggingface/transformers/issues/15389#issuecomment-1387305312

### Motivation

Included in Feature request Section.

### Your contribution

I currently implement some parts of them in some private confidential codes during work. I would help if repo maintainers think it worth doing.

Comments:
cc @Rocketknight1 

Yes, I'd definitely support a PR to update the pipelines/examples for non-English languages!

Sadly I cannot submit such PR due to some reasons, maybe years later...
ISSUES in #TRANSLATING.md
Author: anshumangahlot
Pull request: no

Some potential issues and documentation improvements in #TRANSLATING.md are:

1. **Clarity in Steps**: Ensure each step is distinct and clearly numbered or bulleted to enhance readability. For example, consider using a numbered list for the steps (open an issue, fork the repository, etc.).


2. **Link Validation:** Confirm that all provided links are current and lead to the correct sections. If the links change, they may lead to confusion.


3. **Language Code Explanation**: Expand on the importance of using the correct language code. It may help to provide a brief context about why standard codes are necessary.


4. **File Paths:** When mentioning paths (like ~/path/to/transformers/docs), clarify that users should replace this with their actual path. It might be helpful to use an example.


5. **Translation Guidance**: Provide additional tips or resources for effective translation, especially if the target language has specific idiomatic expressions or technical terms.


6. **Documentation on _toctree.yml:** Consider giving more details about the structure of the _toctree.yml file for users who may not be familiar with YAML.


7. **Markdown and MDX**: Some users may not know the difference between Markdown and MDX. A brief explanation of what MDX is and why it’s used might be beneficial.


8. **Collaborative Work:** Emphasize the collaborative aspect of translations—encourage users to communicate and share resources or glossaries for consistent terminology.


9. **Formatting Consistency**: Ensure consistent formatting for headings, lists, and code snippets throughout the document for better visual appeal.


10. **Call to Action:** Add a closing remark encouraging contributions or sharing completed translations, fostering a sense of community.

Additionally , I am submitting a Pull Request (#34226) regarding the same .

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
ImportError: cannot import name 'AutoModelForImageTextToText' from 'transformers' (/usr/local/lib/python3.10/dist-packages/transformers/__init__.py)
Author: psologub
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.44.2
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.24.7
- Safetensors version: 0.4.5
- Accelerate version: 0.34.2
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.4.1+cu121 (False)
- Tensorflow version (GPU?): 2.17.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.8.5 (cpu)
- Jax version: 0.4.33
- JaxLib version: 0.4.33
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker @amyeroberts @qubv

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
!pip install torch
!pip install transformers

import requests
from PIL import Image
import torch
from transformers import AutoProcessor, AutoModelForImageTextToText
```

The error message:

```
ImportError                               Traceback (most recent call last)
[<ipython-input-5-af84153c3b01>](https://localhost:8080/#) in <cell line: 4>()
      2 from PIL import Image
      3 import torch
----> 4 from transformers import AutoProcessor, AutoModelForImageTextToText

ImportError: cannot import name 'AutoModelForImageTextToText' from 'transformers' (/usr/local/lib/python3.10/dist-packages/transformers/__init__.py)
```

### Expected behavior

I've tried to follow this [example](https://huggingface.co/docs/transformers/main/en/model_doc/llava_next#multi-image-inference) but I get stuck importing AutoModelForImageTextToText. Could you help me out please?

Comments:
Hey! That is because you are looking at the `main` documentation! This feature was shipped in #29572 which is not part of the latest release! 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Onnx validation got max absolute difference of: nan
Author: NakedEinstein
Pull request: no
Labels: name: ONNX

name: Deployment

Hi i got this error when i am trying to convert huggingface bert-based torch model to onnx. 

`ValueError: Outputs values doesn't match between reference model and ONNX exported model: Got max absolute difference of: nan`

The model i am trying to convert is training checkpoint model based on indobenchmark/indobert-large-p2.


### Steps done
I already tried both feature versions AutoModel & AutoModelForSequenceClassification
I already ensure file path(s) were correct


### System info
onnx == 1.13.0
transformers[onnx] == 4.25.1
onnxruntime == 1.14.0
torch == 1.13.0


Comments:
Hey @NakedEinstein, could you share the entire code and stacktrace that you have? Thanks

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
BUG : Modeling nemotron file does not cache key values even though 
Author: jeongin601
Pull request: no
Labels: name: Usage

name: bug

name: Cache

### System Info

huggingface-hub-0.26.2 
tokenizers-0.20.3 
transformers-4.47.0.dev0
Python 3.10.12
Driver Version: 535.129.03   
CUDA Version: 12.3 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

## Description

You can run the below code to reproduce prefill key value caching problem of minitron models.
I used "nvidia/Minitron-8B-Base" model.

## Code 
```
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load Minitron model and tokenizer from Hugging Face
model_name = "your-minitron-model-name"  # Replace with the actual Minitron model name
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Set the model to evaluation mode
model.eval()

# Sample input text
input_text = "Hello, how are you?"

# Tokenize the input
input_ids = tokenizer(input_text, return_tensors="pt").input_ids

# First forward pass (prefill phase)
with torch.no_grad():
    outputs = model(input_ids, use_cache=True)  # Set use_cache=True
    logits = outputs.logits
    past_key_values = outputs.past_key_values

# Check the output
print("Logits shape:", logits.shape)
print("Number of layers in past_key_values:", len(past_key_values))
print("Shape of keys and values in the first layer:")
print("Key shape:", past_key_values[0][0].shape)
print("Value shape:", past_key_values[0][1].shape)

# Add new input to test cache utilization
new_input_text = " What about you?"
new_input_ids = tokenizer(new_input_text, return_tensors="pt").input_ids

# Pass the new input along with the previous key-value cache
with torch.no_grad():
    outputs_with_cache = model(new_input_ids, past_key_values=past_key_values, use_cache=True)

# Check results after caching
new_logits = outputs_with_cache.logits
new_past_key_values = outputs_with_cache.past_key_values

print("New logits shape:", new_logits.shape)
print("Number of layers in new past_key_values:", len(new_past_key_values))

```

### Expected behavior

**As-Is**
![스크린샷 2024-11-15 오후 1 15 33](https://github.com/user-attachments/assets/bdb17a16-6777-4d7d-9b3b-be6d25295b9e)

Past key value is 'Nonetype', which means the key value caches are not cached.

Comments:
cc @ArthurZucker @gante @zucchini-nlp
Add safe_globals to resume training on PyTorch 2.6
Author: dvrogozh
Pull request: yes

Starting from version 2.4 PyTorch introduces a stricter check for the objects which can be loaded with `torch.load()`. Starting from version 2.6 loading with `weights_only=True` requires allowlisting of such objects.

This commit adds allowlist of some `numpy` objects used to load model checkpoints. Usage is restricted by context manager. User can still call `torch.serialization.add_safe_globals()` to add other objects into the safe globals list.

Accelerate library also stepped into same problem and addressed it with PR-3036.

Fixes: #34631
See: https://github.com/pytorch/pytorch/pull/137602
See: https://pytorch.org/docs/stable/notes/serialization.html#torch.serialization.add_safe_globals
See: https://github.com/huggingface/accelerate/pull/3036

CC: @muellerzr @SunMarc

Comments:
@muellerzr, @SunMarc, @ArthurZucker : can you, please, help comment on this PR? see issue #34631 on details.

I am getting 

> FAILED tests/trainer/test_trainer.py::TrainerIntegrationTest::test_can_resume_training - AttributeError: module 'numpy' has no attribute 'dtypes'. Did you mean: 'dtype'?

when running

> python3 -m pytest  tests/trainer/test_trainer.py::TrainerIntegrationTest::test_can_resume_training

against this PR.



@ydshieh : this might be due to numpy version. dtypes was added in 1.25 according to https://numpy.org/doc/2.1/reference/routines.dtypes.html#module-numpy.dtypes. Locally I have 1.26.4. Which version do you have?

I will work on using context manager since there is an alignment on that and also tune a list per versioning of numpy.

On our CI runner , I get `numpy=1.24.3`

The numpy GLOBALs for dtypes that need to be allowlisted might need an if statement depending on whether version < 1.25 or not, there's some documentation on this here https://pytorch.org/docs/main/notes/serialization.html#troubleshooting-weights-only

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34632). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Thanks! Just a documentation suggestion but this all looks correct

@muellerz : done, added a link to Accelerate PR. 

> LGTM ! Just a nit

@SunMarc : addressed, reused approach from accelerate on `numpy.core` deprecation.

Thanks for fixing 🤗
Cache: init empty cache when `use_cache`
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/34206. As per title we would have to initialize empty cache whenever `use_cache=True`. Additionally `MllamaForCausalLM` was not loading correctly for me, so I modified the base model prefix

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34274). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
MllamaForCausalLM not returning past_key_values even with use_cache=True
Author: Zhangyanbo
Pull request: no
Labels: name: WIP

name: bug

name: Vision

name: Multimodal

### System Info

- `transformers` version: 4.45.2
- Platform: Linux-5.4.0-187-generic-x86_64-with-glibc2.31
- Python version: 3.11.5
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: 0.33.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.2 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA RTX 6000 Ada Generation

### Who can help?

@amyeroberts @ArthurZucker 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
import torch.nn as nn
import os
import matplotlib.pyplot as plt
from promptmix import generate_probmix
from transformers import MllamaForConditionalGeneration, AutoProcessor


model_id = "meta-llama/Llama-3.2-11B-Vision-Instruct" # https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct


tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    cache_dir=cache_directory,
    torch_dtype=torch.float16,
    device_map='auto',
    )

processor = AutoProcessor.from_pretrained(model_id, cache_dir=cache_directory)

input_ids = tokenizer.encode('Hi, tell me a story of frog.', add_special_tokens=False, return_tensors='pt').to(model.device)

with torch.no_grad():
    output = model.forward(input_ids=input_ids, use_cache=True)
output
```

### Expected behavior

I expect to see a `past_key_values` in the `output`. However, I got `None`.

Comments:
Hmm, actually we did Mllama quite similar to Idefics so the cache is not initialized by default when "use-cache=True". And yes, I think makes sense to init an empty cache if those are not special models like Gemma with special cache

Until the fix is there you can get pask-kv by passing `model(**inputs, past_key_values=DynamicCache(), use_cache=True)` but I see that the model weights will not be loaded proper way for CausalModel. In fact the ConditionalModel can deal with text-only input so for proper logits computation i'd recommend to use the `ConditionalModel` :)

With `use_cache` we should probably just init a default cache for the user, or we opt for forcing users to pass a cache object

Yes, exactly. I can make a PR for that

not stale, PR under progress
BLIP: fix generation after hub update
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Some tests started failing after updating the hub configs and weights so that we can remove the legacy math on BLIP models. This PR fixes the tests, mainly the reason was that we didn't think about generation from image without text which failed to slice the `input_ids` and merged with `image_embeddings`



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34876). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add DEBERTA-base model for usage in EncoderDecoderModel.
Author: avacaondata
Pull request: no
Labels: name: Feature request

# 🚀 Feature request

Add DEBERTA-base model as an option for creating an EncoderDecoderModel.

## Motivation

Currently only BERT and RoBERTa models can be transformed to a Seq2Seq model via EncoderDecoder class, and for those of use developing DeBERTa models from scratch it would be wonderful to be able to generate a Seq2Seq model from them. Also, the Deberta-base model works much better than BERT and RoBERTa.

## Your contribution




Comments:
Great idea! Do you want to take a stab at it?

Hi @alexvaca0,
This is an interesting feature, 
But I was curious that Deberta, Bert, and Roberta are encoder-based models so there is no decoder part right? I checked their model class and I could not find the Decoder / EncoderDecoder class!
Can you please give more insight into it? 

That's right, there's no decoder in those models, but there is a class in Transformers, EncoderDecoderModel, that enables to create encoder-decoder architectures from encoder-only architectures :)

Perfect, let me have a look at it and see if I can code that adaptation  @LysandreJik

Great! If you run into any blockers, feel free to ping us. If you want to add the possibility for DeBERTa to be a decoder, you'll probably need to add the cross attention layers. 

cc @patrickvonplaten and @patil-suraj which have extensive experience with enc-dec models.

Hey, is this feature being worked on by someone? If not then I can pick it up! @LysandreJik 

Would be great if you could pick it up @manish-p-gupta :-) 

Great!. Any specific things I should go through before taking it up? I'm familiar with the Code of conduct and contributing guidelines. I'll also open a draft PR to carry on the discussions there. Let me know if you think I need to look at anything else. @patrickvonplaten 

@ArthurZucker has been working with DeBERTa models recently and can likely help and give advice!

Yes! Feel free to ping me for an early review if you have any doubts
Fixing strided perplexity calculation for fixed-length models
Author: forrestdavis
Pull request: yes

# What does this PR do?

Fixes an issue with perplexity of fixed length models with a strided window. The perplexity calculation assumes all batches are the same size. This calculates the average with different context sizes. The update is just to perplexity.md in the docs. 

Fixes  https://github.com/huggingface/transformers/issues/34138


## Before submitting

- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


@ArthurZucker





Comments:
Could you resolve conflicts with main? 🤗 

Thanks! I see that there were changes to address this. If those are preferred that's fine with me. I think the proposal here is simpler, but that's subjective. 

Got it! I think we can stay with what we have on main, especially given that the history of this PR is a bit messed up ! 
Thanks for your contribution 🤗
Memory optimization for gpt_bigcode
Author: ugolowic
Pull request: yes

# What does this PR do?

This PR optimizes memory usage for inference with GPT BigCode on devices other than CPU.

There is a spike in memory usage when generating the 1st token and the identified culprit is `torch.baddbm`. Using `torch.matmul` instead reduces the usage and allows for processing significantly larger batch sizes.

Formula for `torch.baddbmm`:
```
out = beta * attn_weights + scale_factor * (query ⋅ key)
```
For beta = 0, this becomes
```
out = scale_factor * (query ⋅ key)
```
and thus can be implemented as `torch.matmul(query, key) * scale_factor`.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

cc @ArthurZucker

Comments: