noamaanMulla-03/transformers-issues huggingface dataset:
Customized position_ids not working
Author: huchanwei123
Pull request: no
Labels: name: bug

### System Info

Hello,

I am trying to feed a customized position IDs to Llama model.
If I fed a customized position_ids vector, for example, [[0, 0, 1, 2, 2, 2]] means batch size = 1, 1st and 2nd tokens share the same position, and 3rd-5th tokens share the same position 2, this will cause an error.

The error seems to be located in the function `prepare_inputs_for_generation` in `src/transformers/models/llama/modeling_llama.py`, where the `position_ids` does not change as the `cache_position` increase, so the shape inconsistency occurs.

Is there any way to successfully feed a customized position ids to the model?
Thanks!

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Give a customized `position_ids` as one of the input in `model.generate()`

### Expected behavior

Size mismatch

Comments:
Cc @gante 

You can directly modify how position_ids are computed within your code before passing them to the model. Ensure that your custom position_ids are aligned with the expected shape and values.


> You can directly modify how position_ids are computed within your code before passing them to the model. Ensure that your custom position_ids are aligned with the expected shape and values.

Thanks for the reply.
Yes, I am aware that I can pass a customized position_ids to the model, and I believe the shape and values are correct.

Since the model generates token-by-token, feeding a customized position_ids causes an size mismatch error after the first token is generated.
After a little bit digging, I found that in the function `prepare_inputs_for_generation`, there is no implementation handling when `position_ids is not None`. As a result, during the generation of the second token, the shape of `position_ids` is still the same, but it is expected to match the shape of `attention_mask`.

I am not sure if I missed or misunderstood anything.
Thanks!

It has a tracker here (https://github.com/huggingface/transformers/issues/29149) and we had a PR a few months ago. Unfortunately the PR was too big and needed to be decomposed into parts, after which it went low in priority :(

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Error on TPU: Invalid --2a886c8_slice_builder_worker_addresses specified. Expected 4 worker addresses, got 1.
Author: plaggy
Pull request: no
Labels: name: TPU

### System Info

```
transformers==4.39.3
diffusers==0.28.0.dev0
accelerate==0.30.0.dev0
```

### Who can help?

@muellerzr 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

In a colab notebook with the TPUv2 runtime:
```
!pip install --upgrade transformers
!pip install --upgrade git+https://github.com/huggingface/diffusers
!pip install --upgrade accelerate@git+https://github.com/huggingface/accelerate.git
```

```
import transformers
import diffusers
import accelerate
```

```
!accelerate launch --tpu --num_processes 8 --main_training_function main train_dreambooth.py \
  --pretrained_model_name_or_path=runwayml/stable-diffusion-v1-5  \
  --instance_data_dir=./images \
  --output_dir=./out \
  --instance_prompt="a photo of sks dog" \
  --resolution=512 \
  --train_batch_size=1 \
  --learning_rate=5e-6 \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --max_train_steps=400
```

Throws an error `Invalid --2a886c8_slice_builder_worker_addresses specified. Expected 4 worker addresses, got 1.`

The dreambooth script is [here](https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth.py) and the images are [here](https://huggingface.co/datasets/diffusers/dog-example)


### Expected behavior

no error

Comments:
cc @muellerzr @pacman100 

Gentle ping @muellerzr @pacman100 

cc @muellerzr @SunMarc 

Hey @plaggy, did you manage to fix your issue ? This issue seems the same as this one https://github.com/huggingface/transformers/issues/28204 . Maybe try to run this with a more recent version of TPU runtime. Sorry for the wait ! 

Yes, iirc we now require TPUv3 Colab is running behind. Kaggle TPUs would be a better option

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add no_trainer scripts to train CLIP
Author: thisisiron
Pull request: yes

# What does this PR do?

I have added a script to train the CLIP model without using the Trainer class.


PR implemented with reference to [code](https://github.com/huggingface/transformers/blob/main/examples/pytorch/image-pretraining/run_mim_no_trainer.py).

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@Rocketknight1 


Comments:
Mergekit support for GPT2 error 
Author: NamburiSrinath
Pull request: no

### Feature request

Hi,

I am working on model merging using mergekit (https://huggingface.co/blog/mlabonne/merge-models) and tried to merge finetuned GPT2. But I was getting the following errors - 

**SLERP:**
```
slices:
  - sources:
      - model: /hdd4/srinath2/Trading_Agent/model_merging/model1_training/checkpoint-3750
        layer_range: [0, 12]
      - model: /hdd4/srinath2/Trading_Agent/model_merging/model2_training/checkpoint-3750
        layer_range: [0, 12]
merge_method: slerp
base_model: openai-community/gpt2
parameters:
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5
dtype: bfloat16
```

Error - `RuntimeError: Tensor ln_f.bias required but not present in model /hdd4/srinath2/Trading_Agent/model_merging/model2_training/checkpoint-3750`

**TIES**
```
models:
  - model: openai-community/gpt2
    # no parameters necessary for base model
  - model: /hdd4/srinath2/Trading_Agent/model_merging/model1_training/checkpoint-3750
    parameters:
      density: 0.5
      weight: 0.5
  - model: /hdd4/srinath2/Trading_Agent/model_merging/model2_training/checkpoint-3750
    parameters:
      density: 0.5
      weight: 0.3
merge_method: ties
base_model: openai-community/gpt2
parameters:
  normalize: true
dtype: float16
```

Error - `RuntimeError: Tensor score.weight required but not present in model openai-community/gpt2`
Same error for **DARE**

**Note:** The moment I change the base model to one of the finetuned models, _I am able to merge the finetuned GPT2 models_. I am not sure if this is expected because the blogpost has a base model that's different from the other 2 models

### Motivation

I finetuned GPT2 on 2 tasks (model1 and model2) and wanted to merge them to understand if the merged model is better than base model on these tasks

### Your contribution

If there's any inconsistencies, I can contribute to blog editing/PR if needed!

Comments:
Hi @NamburiSrinath, thanks for raising an issue.

It's hard to tell from the information here - but this looks like an error relating to the mergekit library rather than anything to do with transformers.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Have you solve the problem yet? I meet the same problem as yours.

Nope, unfortunately I didn't.
Add Kannada language documentation to PEGASUS tokenizer class
Author: pratheeshkumar99
Pull request: yes

# What does this PR do?

This PR introduces Kannada language documentation alongside the existing English comments in the PEGASUS tokenizer class. By adding bilingual docstrings and comments, this update makes the code more accessible to Kannada-speaking developers and enhances inclusivity in the open-source community.

### Key Changes:
- Added Kannada translations for all major docstrings and function explanations within the PEGASUS tokenizer class.
- Ensured Kannada comments accurately mirror English documentation to maintain clarity and consistency.
- Encourages a broader range of contributors and users by supporting regional language accessibility.

This contribution aligns with Hugging Face’s mission to make open-source projects more inclusive and user-friendly, especially for non-English-speaking communities.

## Before submitting
- [x] This PR improves documentation with added multilingual support.
- [x] I have read and followed the [contributor guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request).
- [x] No additional dependencies or tests are required, as this change is purely documentation-based.
  
## Who can review?
Documentation: @stevhliu 
Tokenizers: @ArthurZucker 

Comments:
Hi, thanks for your contribution! 🤗 

To add a translation, please follow the instructions [here](https://github.com/huggingface/transformers/blob/main/docs/TRANSLATING.md). At a very high-level, you'll need to:

1. create a folder with your language code in `transformers/docs/source`
2. add a `toctree.yml` and add your translated doc to it
3. add the language code [here](https://github.com/huggingface/transformers/blob/be8748a53ce8364f3e584035e267030e7bfad06c/.github/workflows/build_documentation.yml#L19) and [here](https://github.com/huggingface/transformers/blob/be8748a53ce8364f3e584035e267030e7bfad06c/.github/workflows/build_pr_documentation.yml#L17) to trigger the doc-builder to build the docs in your language
4. find someone who can read and review your translated content because I can't read/review it for accuracy :)
[docs] Broken link in generation_strategies
Author: pcuenca
Pull request: yes

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@stevhliu, cc @gante 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34717). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
🌐 [i18n-KO] Translated marian.md to Korean
Author: maximizemaxwell
Pull request: yes

# What does this PR do?

Translated the marian.md file of the documentation to Korean.
Thank you in advance for your review.

Part of https://github.com/huggingface/transformers/issues/20179
## Before reviewing

  - [x]  Check for missing / redundant translations (번역 누락/중복 검사)
  - [x] Grammar Check (맞춤법 검사)
  - [x] Review or Add new terms to glossary (용어 확인 및 추가)
  - [x]  Check Inline TOC (e.g. [[lowercased-header]])
  - [ ]  Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

@junejae, @Jwaminju, @010kim, @4N3MONE, @boyunJang
## Before submitting

   - [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
   - [x]  Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
    Pull Request section?
   - [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
    to it if that's the case.
   - [x]  Did you make sure to update the documentation with your changes? Here are the
    [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
    [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
   - [ ]  Did you write any new necessary tests?

## Who can review? (Final)

@stevhliu May you please review this PR?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34698). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

LGTM! 번역 고생하셨습니다. 😊
Missing weights in model checkpoint
Author: uililo
Pull request: no
Labels: name: bug

name: Audio

### System Info

When trying to load a pretrained wav2vec2 checkpoint, I got the following warning:

>   warnings.warn(
> Some weights of Wav2Vec2Model were not initialized from the model checkpoint at facebook/wav2vec2-base and are newly initialized: ['wav2vec2.encoder.pos_conv_embed.conv.parametrizations.weight.original0', 'wav2vec2.encoder.pos_conv_embed.conv.parametrizations.weight.original1']
> You should probably TRAIN this model on a downstream task to be able to use it for predictions and inference.

Theoretically it should be possible to extract embeddings from this pre-trained model without fine-tuning on downstream tasks, however it seems that some weights are missing. I'm not sure whether the problem is with the Wav2Vec2Model class or the checkpoint.
I got the same warning when I tried to load some other checkpoints of wav2vec2 models published by facebook.

Thanks

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
model_name = "facebook/wav2vec2-base"
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(model_name)
model = Wav2Vec2Model.from_pretrained(model_name)

```

### Expected behavior

The pretrained model is successfully loaded into "model" without needing to initialize some of the weights

Comments:
Hey @uililo ,
i want to work on this issue.

-can you tell the file location


location of which file?

Hello @uililo, 

Don't worry about it. This is a harmless warning. It's because of the migration of parameter names between different versions of the transformers library. 
You can see a similar issue here: https://github.com/huggingface/transformers/issues/26796#issuecomment-1781551163

Take care


hey @uililo can i work on this issue?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Very slow process when `torch_dtype` is passed.
Author: realSAH
Pull request: no


When I use `from_pretrained`, the model loads up from my azure cache at almost 1gbps, but when I specify the `torch_dtype` the process is slowed to a crawling 1 tenth of original speed. When looking at resources it appears that its a single core type of process that is doing the disservice of being bottleneck.

Can we have that parallelized?

Comments:
There is nothing we can do without a clear reproducer.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add freq time masking to ast
Author: adrianfagerland
Pull request: yes

Supersedes #33493 because I was stupid enough to delete the forked repository.

# What does this PR do?

Adds frequency and time masking, as well as random noise data augmentations to AST feature extractor. This is very useful for training as AST has a tendency to overfit on practical data, at least in my own personal experience. This is also what was used in the original paper and [code](https://github.com/YuanGongND/ast/blob/master/src/dataloader.py#L187).


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request) Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)?
- [x] Did you make sure to update the documentation with your changes?
- [X] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed.

@ylacombe

This is my first time contributing to `transformers`, so please let me know if I need to make any changes :)

Comments:
I also found out that the error described [here](https://huggingface.co/MIT/ast-finetuned-audioset-10-10-0.4593/discussions/12) always occurs when the audio that gets sent into the feature extractor has length under 400. I could implement a check in the feature extractor that makes it easier to debug for others.

@ylacombe 
Thanks for the great review! I changed everything you suggested, and added the warning for when the audio is too short.

@ylacombe
Amazing, thank you! The issues you highlighted have all been addressed, so it should be ready for a final review.

@ylacombe How do I request core maintainer review?
Plans to Integrate LongRoPE into LLaMA?
Author: ryan-minato
Pull request: no
Labels: name: Feature request

### Feature request

Microsoft has introduced their [microsoft/LongRoPE](https://github.com/microsoft/LongRoPE) implementation. Unlike plug-and-play solutions, LongRoPE requires hyperparameter tuning via a genetic algorithm. This implementation is likely the same as described in the `Su` on Phi-3. Are there any plans to incorporate LongRoPE into LLaMA?

### Motivation

In my research on long content, I have managed to integrate LongRoPE into LLaMA with some minor code adjustments. 
I am curious if Huggingface is also working on integrating this feature.

### Your contribution

If necessary.

Comments:
cc @ArthurZucker @gante 

Hey @ryan-minato 👋 

Thank you for opening this issue! We're pausing new rope scaling contributions for a week or so, while we refactor the code. Past that, we'd love to get a contribution 🤗 

Will be fixed by #31999

@ryan-minato #31999 will include longrope 🤗 

Hi @ryan-minato, is there a chance to make the implementation of your llama-longrope public? :)

> 你好@ryan-minato，是否有机会公开你的 llama-longrope 的实现？:)

Sorry, as I work for a company, I don't hold the copyright for the previous code and therefore can’t share it. I’ll rewrite it from scratch when I have time.
potential rope implementation issue in llama model
Author: ilml
Pull request: no
Labels: name: Discussion

https://github.com/huggingface/transformers/blob/a3d69a8994d673899608a7c17fbf4f953f50474e/src/transformers/models/llama/modeling_llama.py#L199

heyyy 🤗,

i was learning rope recently and was looking at the reference implementation above. and i'm a little confused here.
in the original rope paper, we should pair each adjacent numbers and apply the roration. below is a screenshot from the paper.
![image](https://github.com/user-attachments/assets/e259db56-b1d4-48cd-b3a6-4b362e3de5c9)

notice that the index for the second X should goes like 2, 1, 4, 3, 6, 5 ....
but in the code above it uses  x1 = x[..., : x.shape[-1] // 2] , which rotate all the second half of X with the all the first half, so the index will goes like 4, 5, 6, 1, 2, 3...

so this seems not aligning with what rope needs, it is effectively trying to pair xi with xi+(d/2), but what we need is to pair xi with xi+1.

would love to hear anything from you guys 🤗 

tom


Comments:
@ArthurZucker ptal 🤗 

Closing as duplicate of the other issue you opened here https://github.com/huggingface/transformers/issues/34741
LLaVA: latency issues
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes latency issues in some llava models caused by the recent processing logic refactoring. It was reported by one of the users that the new release is twice slower than the prev release. The problem was in indentation where images were embedded in every run instead of only the pre-fill stage

I believe it is worth fixing and merging until https://github.com/huggingface/transformers/pull/33424 is merged to unblock e from updating hub configs and enabling the new logic

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34460). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @zucchini-nlp @ArthurZucker . This PR could lead the issue #34696 . Do you mind take a look? Thanks.

@jiqing-feng I'll be updating all model configs starting next week to enable the new logic for processing. So we stop seeing bugs related to the old logic maintenance. For the linked issue, will take a look soon , thanks