noamaanMulla-03/transformers-issues huggingface dataset:
🔴 Mllama: fix base prefix
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/34689. Separated into a new PR from https://github.com/huggingface/transformers/pull/34274

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34874). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Added! Well it is more fixing than breaking I hope hehe
Add more configurations to the class LlavaOnevisionConfig.
Author: litianjian
Pull request: no
Labels: name: Feature request

### Feature request

I would like to add more flags to configuration class `LlavaOnevisionConfig`, including `spatial_pool_mode`, `newline_position `

### Motivation

 [LlavaOnevisionModel](https://huggingface.co/llava-hf/llava-onevision-qwen2-7b-ov-hf) uses the fixed value of `spatial_pool_mode="bilinear"` and `newline_position="one_token"`,  so the defined class [LlavaOnevisionConfig](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llava_onevision/configuration_llava_onevision.py#L27C7-L27C27)  has omitted the corresponding configurations.

Many of models are trained with the LLaVA-Onevision model using the different flags mentioned above. Incorporating these flags into the configuration file would be highly helpful for developers.

### Your contribution

I can submit a PR with the modifications if you agree with the incorporation of this change.

Comments:
We also discussed this a bit in https://github.com/huggingface/transformers/pull/34195#discussion_r1804938575 (for more background info)

cc core maintainer @ArthurZucker , since I have no objections against adding new params and update modeling code

I am not sure what the changes would look like, but if it does not change / adds if elses to the forward pass sure! 
The main issue is that for us, different codepathes <==> a different model
Problems importing LlavaForConditionalGeneration
Author: ppsmk388
Pull request: no

### System Info


transformers version: 4.35.2
Platform: Linux-5.10.0-26-cloud-amd64-x86_64-with-glibc2.31
Python version: 3.9.2
Huggingface_hub version: 0.19.4
Safetensors version: 0.4.1
Accelerate version: 0.25.0
PyTorch version (GPU?): 2.1.1+cu121 (True)

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction



from transformers import  LlavaForConditionalGeneration








### Expected behavior

An error message will be displayed:
ImportError: cannot import name 'LlavaForConditionalGeneration' from 'transformers' (/data/kkk/anaconda3/envs/va/lib/python3.9/site-packages/transformers/__init__.py)




Comments:
Hey! 👋 llava is no part of the latest release! You need to install from source or wait until tomorrow! 🤗

ok, thank you for your reply




> ### System Info
> transformers version: 4.35.2 Platform: Linux-5.10.0-26-cloud-amd64-x86_64-with-glibc2.31 Python version: 3.9.2 Huggingface_hub version: 0.19.4 Safetensors version: 0.4.1 Accelerate version: 0.25.0 PyTorch version (GPU?): 2.1.1+cu121 (True)
> 
> ### Who can help?
> _No response_
> 
> ### Information
> * [x]  The official example scripts
> * [ ]  My own modified scripts
> 
> ### Tasks
> * [x]  An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
> * [ ]  My own task or dataset (give details below)
> 
> ### Reproduction
> from transformers import LlavaForConditionalGeneration
> 
> ### Expected behavior
> An error message will be displayed: ImportError: cannot import name 'LlavaForConditionalGeneration' from 'transformers' (/data/kkk/anaconda3/envs/va/lib/python3.9/site-packages/transformers/**init**.py)

Have you solved the problem? i met the error no matter i reinstalled the bag version of transformers or install it from source.
<img width="576" alt="image" src="https://github.com/user-attachments/assets/0c195dc4-1f8b-45ae-9e43-8a28c5fe2eac">


you have a typo: 
```diff
- LLava
+ Llava
```

Llava works.

but i think why people has this problem is because in this page:
https://huggingface.co/docs/transformers/model_doc/llava#overview

Batched inference part show two wrong examples, 

Batched inference
LLaVa also supports batched inference. Here is how you can do it:

Copied
```python
import requests
from PIL import Image
import torch
from transformers import AutoProcessor, LLavaForConditionalGeneration

# Load the model in half-precision
model = LLavaForConditionalGeneration.from_pretrained("llava-hf/llava-1.5-7b-hf", torch_dtype=torch.float16, device_map="auto")
processor = AutoProcessor.from_pretrained("llava-hf/llava-1.5-7b-hf")
```
you can see there is LLavaForConditionalGeneration not Llava

Indeed! Do you want to open a PR for a fix?
ViTImageproc handle pil hw images
Author: alem-147
Pull request: yes
Labels: name: Vision

name: Processing

# What does this PR do?

fixes #34820 

Modifies image transformations and ViTImageProcessor such that HW PIL images can be have __call__ applied given the right arguments.

To supply an HW image to the ViTImageProcessor use the `input_data_format='none' or `input_data_format=ChannelDimension.NONE`. Further it is required that this is converted to RGB via the `do_convert_rgb=True` as the first option

```
    dataset = load_dataset("ylecun/mnist")
    processor = AutoImageProcessor.from_pretrained("farleyknight-org-username/vit-base-mnist")

    def process(examples):
        processed_inputs = processor(examples["image"],
                                             input_data_format="none",
                                             do_convert_rgb=True)
        return processed_inputs


    processed_dataset = dataset.map(process, batched=True)
``` 
or to leave it as a single channeled np.array by setting the mean and std of the image processor with `image_mean=[mean_value]` and `image_std=[std_value]`
```
...
    def process(examples):
        processed_inputs = teacher_processor(examples["image"],
                                             input_data_format="none",
                                             do_convert_rgb=False,
                                             image_mean=[.5],
                                             image_std=[.5])
        return processed_inputs
...
```



## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
 @amyeroberts, @qubvel


Comments:
prepare_fa2_from_position_ids function bugfix
Author: meliksahturker
Pull request: yes

# What does this PR do?
It resolves the bug reported in [issue 33268](https://github.com/huggingface/transformers/issues/33268) by calling contiguous() before view() for key and value within **prepare_fa2_from_position_ids** function.
Related error stems from that the tensor is not contiguous in the memory, hence is fixed by simply calling contiguous() function.

The same error is encountered and discussed in:
[PyTorch forum](https://discuss.pytorch.org/t/view-size-is-not-compatible-with-input-tensors-size-and-stride/121488/2)
[Huggingface forum](https://discuss.huggingface.co/t/albertformaskedlm-error-view-size-is-not-compatible/1261)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@RhuiDih
@ArthurZucker

Comments:
+1 I am seeing the same problem. Looking forward to the fix!

Hi, a few months has passed and I would appreciate it if this can be merged! /cc @ArthurZucker 

cc @meliksahturker - it looks like the PR is good, but a test to ensure the issue doesn't regress would be great! Let us know if you need any help with that.

Thanks and looking forward to it! This is blocking my usage of position_ids + fa2 :(

> cc @meliksahturker - it looks like the PR is good, but a test to ensure the issue doesn't regress would be great! Let us know if you need any help with that.

I took a look to the existing [test](https://github.com/huggingface/transformers/pull/31629/files#diff-a6ebb0bccfe159ca855e9d9b5d6fbbeb62fced0c0296de0993ba39ef5c862cf0R4330) of the function, however I haven't been able to find the time to work on it.
I think it's best and the fastest if @RhuiDih would take a look and perhaps add the batched test since he implemented the function and the existing tests.
add norm_topk_prob argument on MixtralConfig
Author: bzantium
Pull request: yes

add `norm_topk_prob` config which is to decide whether to normalize router weights or not.
ref: [qwen2_moe](https://github.com/huggingface/transformers/blob/eb811449a2389e48930c45f84c88fd041735cf92/src/transformers/models/qwen2_moe/modeling_qwen2_moe.py#L681), [deepseek_moe](https://huggingface.co/deepseek-ai/deepseek-moe-16b-base/blob/521d2bc4fb69a3f3ae565310fcc3b65f97af2580/modeling_deepseek.py#L315)

resolved: #34608 

@ArthurZucker @gante

Comments:
Hey! Sorry for not answering, this is usually something we would label as feature request as it serves a custom usecase! 🤗
Finetuning with Distributed Training
Author: curious-ai-developer
Pull request: no
Labels: name: Feature request

### Feature request

I have two PCs each with 4090 GPU. I want to finetune Paligemma by google with these two GPUs. It is a multi node distributed training. Can you guide me how is it? Which one is supported Model parallelization or Data parallelization? How is it? 

### Motivation

Enabling Distributed Training

### Your contribution

I wrote some code using Jax but it doesn't do the job, it stops without error. I can share code.

Comments:
Hi @curious-ai-developer, our library for distributing training like this is called [Accelerate](https://huggingface.co/docs/transformers/en/accelerate). You should be able to set up multi-node training using it. If you have specific questions on how to do it, though, it's probably better to take it to the [forums](https://discuss.huggingface.co/) or the [Discord](http://hf.co/join/discord) than to GitHub issues!
[Modular Transformers] Request for comments
Author: LysandreJik
Pull request: no
Labels: name: WIP

name: Modular

Hello :wave: 

Over the past few weeks, we've been merging PRs related to "Modular Transformers".

We explain why we chose to go that way in this [tweet](https://x.com/LysandreJik/status/1841505287879958730) and this [YouTube video](https://youtu.be/P-asaQVmA3o?si=aZIsaRCpGQYdS05L) at the PyTorch Conference by @ArthurZucker.

We detail how to use it, and have started putting in examples, in this [documentation page](https://huggingface.co/docs/transformers/modular_transformers). Finally, some models, like [GLM, are being contributed](https://github.com/huggingface/transformers/pull/33823) (by @Cyrilvallez) using the modular file.

We'd be eager to hear about your experience using the tool. This is very much experimental, it is brittle, but we'll work on having it be more flexible, and usable in a myriad of situations. Please do share your experience (positive or negative), thoughts, comments, we'd be eager to hear what could be improved in order to lower even more the bar to contributing new models.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add JinaBert Model
Author: Jackmin801
Pull request: no
Labels: name: New model

### Model description

Jina AI has just released an open source embedding model that can handle 8k sequence on huggingface:
- https://huggingface.co/jinaai/jina-embeddings-v2-base-en
- https://huggingface.co/jinaai/jina-embeddings-v2-small-en

These models however, currently require the `trust_remote_code` flag as they reference a custom model implementation specified at https://huggingface.co/jinaai/jina-embedding-v2/tree/main. It should be relatively simple to upstream the model implementation as it is already implemented to work when `trust_remote_code` is passed.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

https://huggingface.co/jinaai/jina-embedding-v2/tree/main

Comments:
any update on this? 

Hey! I think the model authors like it as is, and the model is available in [TEI](https://github.com/huggingface/text-embeddings-inference) 🥳 

Has this issue been resolved？

Is there any chance this will happen? Now that [Jina Embeddings v3](https://jina.ai/news/jina-embeddings-v3-a-frontier-multilingual-embedding-model) is available and superior it might be alright for the authors?

TBH if anyone wants to add it to transformers, feel free to open a PR using `modular` transformers would be awesome https://huggingface.co/docs/transformers/en/modular_transformers
possible llama rope implementation issue
Author: ilml
Pull request: no
Labels: name: Discussion

name: bug

### System Info

their might be a bug in the llama rope code.

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

just look at the code

### Expected behavior

https://github.com/huggingface/transformers/blob/a3d69a8994d673899608a7c17fbf4f953f50474e/src/transformers/models/llama/modeling_llama.py#L199

heyyy 🤗,

i was learning rope recently and was looking at the reference implementation above. and i'm a little confused here.
in the original rope paper, we should pair each adjacent numbers and apply the roration. below is a screenshot from the paper.
![image](https://github.com/user-attachments/assets/e259db56-b1d4-48cd-b3a6-4b362e3de5c9)

notice that the index for the second X should goes like 2, 1, 4, 3, 6, 5 ....
but in the code above it uses  x1 = x[..., : x.shape[-1] // 2] , which rotate all the second half of X with the all the first half, so the index will goes like 4, 5, 6, 1, 2, 3...

so this seems not aligning with what rope needs, it is effectively trying to pair xi with xi+(d/2), but what we need is to pair xi with xi+1.

would love to hear anything from you guys 🤗 

tom


Comments:
@ArthurZucker ptal 🤗 

Hey! This is a duplicate of #31859 and #25199 closing!
Convert new SigLIP checkpoints
Author: merveenoyan
Pull request: yes

@NielsRogge as requested, I have modified the conversion script to convert new SigLIP ckpts.

I have modified model max length and pos embed shape because normally it was 64 and for the so-400m-patch14-224 it is now 16. 
My problem is that the expected output and the converted model outputs aren't close enough, see below. Did you come across the same issue?

```
outputs.logits_per_image[:3, :3]: tensor([[-1.0865,  1.1704],
        [-0.7178,  1.4354]])
expected_slice: tensor([[-1.0837,  1.1743],
        [-0.7196,  1.4371]])
```

Skipping the other model for now, the vocab dim of text embeddings is a bit of an issue, it is said to be 250k but is 256k, waiting for response from google folks (I think they used a different tokenizer, the demo notebook of big-vision also doesn't work for this reason)

Update: I confirmed with folks that there seems to be a mistake with tokenizer.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32938). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sorry for the delay, I was busy shipping Llama 3.2 release and then ECCV and then I took time off. I just got back to this PR. 

The logits are validated so except for cosmetic changes we are good to merge✨ the model is here https://huggingface.co/merve/siglip-so400m-patch16-256-i18n

I keep the model repo in sync with the changes here 

@NielsRogge I confirmed image/text output embeds with atol of 1e-4 against JAX ones and all seems well, if you don't have any other comments we could merge 🤗

I uploaded tokenizers and currently uploading the models

@ArthurZucker I addressed your request, does it work? ☺️

the latest ckpts I download from google cloud (in their official script) happen to have the same setup as the last one (no params/ prefix and different setup in encoder), I asked google folks to confirm. I'll have to change the script again if so. I have addressed Arthur's prev comments but haven't pushed yet since I want to confirm it works for older checkpoints.

Sounds good

@NielsRogge I have pushed the final version with the changes Arthur asked, both models I converted match atol and are compatible with transformers so I moved them to google org.

apparently there are some changes to keys and for some reason old models are not compatible with the code anymore (basically I downloaded and uploaded them using big_vision code). 
I am a bit too busy to debug and since we already have old models in place this doesn't seem urgent, can you check what's up?
Fix for TypeError in train_new_from_iterator() in tokenization_utils_fast.py
Author: sindhuvahinis
Pull request: yes

# What does this PR do?

Fixes # (issue)

When calling train_new_from_iterator() on a tokenizer initialized using AutoTokenizer.from_pretrained(), the following error is raised:
```
tokenizer = AutoTokenizer.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0")
new_tokenizer = tokenizer.train_new_from_iterator(
    training_corpus, vocab_size=vocab_keep_items
)
```

Error traceback:
```
    new_tokenizer = tokenizer.train_new_from_iterator(
  File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_fast.py", line 818, in train_new_from_iterator
    or tokenizer_json["pre_tokenizer"]["type"] == "Sequence"
    TypeError: 'NoneType' object is not subscriptable
```

Root Cause

The issue is due to improper handling of the conditional check for pre_tokenizer in tokenization_utils_fast.py. The current implementation does not correctly account for the case where tokenizer_json does not contain the "pre_tokenizer" key. This results in attempting to access "type" from a NoneType object, causing a TypeError. This bug seems to be existing from transformers version 4.45.2 onwards.


Manually tested with the above code. 

@ArthurZucker


Comments:
[New Model] X-ALMA
Author: fe1ixxu
Pull request: yes
Labels: name: New model

# What does this PR do?
This PR is trying to add a new model [X-ALMA](https://arxiv.org/abs/2410.03115).  X-ALMA is a high-quality LLM-based translation model that supports 50 languages, ensuring their high-performance in translation, regardless of their resource level.

@ArthurZucker



Comments:
Hey @fe1ixxu, sorry for the mistake with the earlier message! We're actually quite interested in adding the model to `transformers`, but would you be willing to make the PR in the new [Modular style](https://huggingface.co/docs/transformers/en/modular_transformers)? Modular PRs are much smaller and easier to review when a lot of the architecture is similar to existing models.
Fix Qwen2 failing tests
Author: jla524
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #34818 (issue)

- replace `Qwen/Qwen2-450m-beta` which no long exists
- update expected tensor values and text completions
- tested on a 3090 system

## Who can review?

@ArthurZucker 

Comments:
LGTM !  Thanks for the fix 😊
Qwen2 tests failing due to invalid model identifier
Author: jla524
Pull request: no
Labels: name: Tests

name: bug

### System Info

- `transformers` version: 4.47.0.dev0
- Platform: Linux-6.5.0-1025-oem-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.26.2
- Safetensors version: 0.4.5
- Accelerate version: 1.1.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.5.1+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce RTX 3090

### Who can help?

@arthur

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Install dependencies with `pip install -e [.[dev-torch]`
2. Run tests with `RUN_SLOW=1 pytest tests/models/qwen2/test_modeling_qwen2.py`

### Expected behavior

Tests should pass

Comments:
Here's the error message:
```
OSError: Qwen/Qwen2-450m-beta is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>`
```