noamaanMulla-03/transformers-issues huggingface dataset:
add uniform processors for altclip + chinese_clip
Author: molbap
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Adds two models for https://github.com/huggingface/transformers/pull/30511 , see parent PR for more details

Comments:
Tests will succeed when #31197 is merged. See #30511 tests to confirm. 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31198). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks! I'm currently moving the kwargs merging in a common method actually, and explaining order of operations wrt to kwargs priorities. Will be in https://github.com/huggingface/transformers/pull/31197 in a minute

I simplified the addition/testing of uniformized processors on that one. It's pulling changes from #31197 so the file diff will be reduced once that first one is merged! shoud be ready for review after that. 

This one should be ready for review @amyeroberts after #31197 is reviewed&merged! 

@amyeroberts to help with your pings, this is PR 2 out of N to review for processors! (PR 1 was #31197 and is merged) 

@amyeroberts I forgot to launch slow tests, they are a bit slow, but should be good for review now!
- removed try/except block for processors attributes
- formatting
- removed unused teardown method

@amyeroberts I think this one is ok for a review!

Re-pinging @amyeroberts on that one and @qubvel and @zucchini-nlp optionally if you want to take another look - not much changed, just was between sprints and didn't merge that one :grimacing: should be good to go now (I know you approved, Amy, but that was a while ago!)

Tests that were working are broken since passing from `crop_size` to `size` in common tests I believe  - will look into this soon
4D attention masks not compatible with generate
Author: alex-hh
Pull request: no
Labels: name: bug

name: Generation

### System Info

Latest transformers

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Call generate with a 4d attention mask on a llama model. Currently prepare_inputs_for_generation assumes a 2d mask (though does not check the shape, leading to a hard-to-understand error).


```
    from transformers import LlamaForCausalLM, LlamaConfig
    config = LlamaConfig(hidden_dim=64, num_attention_heads=4, num_hidden_layers=1, vocab_size=20, intermediate_size=128)
    model = LlamaForCausalLM(config)

    model.eval()
    input_ids = torch.ones(1, 10).long()
    attention_mask = torch.zeros((
        1, 1, 10, 10
    ))
    # attention_mask = torch.ones(1, 10)
    model.generate(
        input_ids=input_ids,
        max_new_tokens=2,
        do_sample=True,
        num_return_sequences=2,
        attention_mask=attention_mask,
    )
```

### Expected behavior

Generate should probably raise an informative error when passing a 4d attention mask, telling the user they should implement the required logic in e.g. _update_model_kwargs_for_generate or prepare_inputs_for_generation.

currently the errors are hard to understand, and the lack of support for this is unexpected.


Comments:
cc @gante as well

Hi @alex-hh 👋 

Good point, we don't check that case in `generate`! Added a PR to throw an informative exception -- thank you for flagging 🤗
Generate: check that `attention_mask` is 2D
Author: gante
Pull request: yes

# What does this PR do?

Fixes #33549 

In `generate`, `attention_mask` must be in its 2D format. This PR adds a check for that.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33575). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Cache: don't throw warnings on `gemma2` when instantiating a new cache
Author: gante
Pull request: yes

# What does this PR do?

Related to #33541 

The warning in question should only be thrown in the case we are converting from a legacy cache, which will be deprecated soon. Gemma 2 doesn't support the legacy cache format, so no warning should ever be thrown :)

In the process, updates a few related inconsistencies.

____________________________

✅ slow `gemma2` tests ran locally. There are a few failures (also present on main). Some failures were fixed in this PR.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33595). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I'm not qualified to review this but thanks for addressing this so quickly.
Reproducibility issue with Trainer
Author: dkalpakchi
Pull request: no

Hi,
I am trying to fine-tune a model using PyTorch backend and Trainer class, but when I train 2 times with exactly the same parameters, I still get different results. I have supplied the seed and also used PyTorch-specific commands:
```py
torch.use_deterministic_algorithms(True)
torch.backends.cudnn.benchmark = False
```

However, after all of these failed I went to Trainer's source code and noticed these lines (https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L566)
```
if self.args.world_size <= 1 and _is_torch_generator_available:
    generator = torch.Generator()
    generator.manual_seed(int(torch.empty((), dtype=torch.int64).random_().item()))
```
Is there any reason to actually sample the seed here? Wouldn't it hurt reproducibility between the runs? I'm not sure if this is a bug or not, which is why I didn't open this issue as a bug.
Also do you have more suggestions on where to look for sources of randomness using PyTorch backend?

Thank you very much in advance!

Best regards,
Dmytro

Comments:
Hi,

Googling "reproducibility Huggingface trainer" returns this helpful post on our forum: https://discuss.huggingface.co/t/fixing-the-random-seed-in-the-trainer-does-not-produce-the-same-results-across-runs/3442

Hi Niels!

Thanks for your swift response! I've seen the post, but it seemed people still couldn't make it work judging from the last message in the thread. I've tested setting the seed before instantiating a model now and fixed a couple of other bugs of my own and now I managed to get reproducible results.

However, I still can't grasp why the original Trainer sets the seed to a random value in the code snippet provided in the original post of this thread, since this generator is used to instantiate a sampler, which is passed to a DataLoader, in turn. Won't that hurt reproducibility?

Best,
Dmytro

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Since the question here went without reply and the issue is a websearch result, I'm _guessing_ that the pasted snippet is to create a deterministic generator that won't have its state upset by other random calls, but that is indeed still seeded by a constant value based off the global state and seed. However, I don't know the APIs well enough to confirm this for sure.
Fix Llama 3 TikToken conversion
Author: pcuenca
Pull request: yes

# What does this PR do?

Fixes a tokenizer conversion problem for Llama 3, possibly introduced in https://github.com/huggingface/transformers/pull/31656

## How to reproduce:

The following fails in `main` and works with this PR:

```bash
python src/transformers/models/llama/convert_llama_weights_to_hf.py \
--input_dir path_to_original_checkpoint \
--model_size tokenizer_only \
--output_dir converted_tokenizer \
--llama_version 3.1
```

The same issue may happen with Gemma models as well. I didn't test yet, but can verify once we agree on a suitable approach.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@itazap @ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33538). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
🌐 [i18n-KO] Translated `main_classes/model.md` to Korean
Author: fabxoe
Pull request: yes

# What does this PR do?

Translated the `main_classes/model.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [ ] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [ ] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 N조 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! 차례대로 1 ~ 5조 -->
<!-- May you please review this PR? -->
@chhaewxn, @ahnjj, @jun048098, @fabxoe, @nuatmochoi, @heuristicwave

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. N조 팀원들과 리뷰가 끝난 후에 아래 주석을 노출해주세요! -->
<!-- @stevhliu May you please review this PR? -->


Comments:
Docs: add the ability to manually trigger jobs
Author: gante
Pull request: yes

# What does this PR do?

Our doc-builder had a bug that was fixed recently (https://github.com/huggingface/doc-builder/pull/516). This bug has an impact on our docs since v4.41 -- we need to manually trigger the job to push the fixed docs.

This PR adds the ability to manually trigger the doc builder :)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33598). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix CircleCI nightly run
Author: ydshieh
Pull request: yes

# What does this PR do?

Currently on `main`, it always gives `empty` as jobs.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33558). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Allow CI could be run on private forked repositories (e.g. new model additions)
Author: ydshieh
Pull request: yes

# What does this PR do?

As discussed offline with @ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33594). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Jamba model: 'AttributeError: 'HybridMambaAttentionDynamicCache' object has no attribute '_modules''
Author: graysonrie
Pull request: no
Labels: name: bug

name: Generation

name: Cache

### System Info

Transformers 4.44
PyTorch version: 2.3.1
Ubuntu 24.04
A100 GPU

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Almost the exact same sample code found on https://huggingface.co/ai21labs/Jamba-v0.1 The only modification is `device_map`
```
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("ai21labs/Jamba-v0.1",device_map="cuda")
tokenizer = AutoTokenizer.from_pretrained("ai21labs/Jamba-v0.1")

input_ids = tokenizer("In the recent Super Bowl LVIII,", return_tensors='pt').to(model.device)["input_ids"]

outputs = model.generate(input_ids, max_new_tokens=216)

print(tokenizer.batch_decode(outputs)
```

### Expected behavior

The error is thrown upon trying to perform inference. The easiest fix I have seen is by going to 'modeling_jamba.py' and making `HybridMambaAttentionDynamicCache` inherit from 'nn.Module' as such:

`class HybridMambaAttentionDynamicCache(DynamicCache, nn.Module):`

Then adding the superclass intitialization to the '\_\_init\_\_' method:

```
def __init__(self, config, batch_size, dtype=torch.float16, device=None):
    super(HybridMambaAttentionDynamicCache, self).__init__()
```



Comments:
cc @ArthurZucker @gante 

Hi @graysonrie 👋 Thank you for opening this issue!

I can't reproduce the issue on my end (running on `main`, `torch==2.4.0`, RTX4090). 

It is possible we have already sorted the issue, have you tried running your example on `main`? We have indeed updated the cache classes recently, to inherit from `nn.Module` :)

Hi @gante 👋  I'm still getting the same error on A100 GPU.
Packages & versions:
`python==3.10.0`
`transformers==4.45.0.dev0`
`torch==2.4.0`
`mamba-ssm==2.2.2`
`causal-conv1d==1.4.0`

`Cuda Driver Version: 550.90.07`
`CUDA Version: 12.4`

Code Snippet:
As mention in the issue [Code](https://github.com/huggingface/transformers/issues/32853#issue-2470710266)

Error: `AttributeError: 'HybridMambaAttentionDynamicCache' object has no attribute '_modules'`

Closing as this was fixed by #32894

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
No InternVLChatModel和InternVisionModel class in the transformers source code
Author: AmazDeng
Pull request: no

Why can the InternVLChatModel(https://huggingface.co/OpenGVLab/InternVL2-2B) and InternVisionModel(https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) models be loaded in the following way, but these two classes, InternVLChatModel and InternVisionModel, cannot be found in the transformers code? What is the reason for this?

Can you help me look at this issue? @zucchini-nlp @sgugger @LysandreJik @ydshieh

InternVLChatModel
```
import torch
from transformers import AutoTokenizer, AutoModel
path = "OpenGVLab/InternVL2-2B"
model = AutoModel.from_pretrained(
    path,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    use_flash_attn=True,
    trust_remote_code=True).eval().cuda()

```

InternVisionModel
```
import torch
from PIL import Image
from transformers import AutoModel, CLIPImageProcessor

model = AutoModel.from_pretrained(
    'OpenGVLab/InternViT-6B-448px-V1-5',
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True).cuda().eval()

image = Image.open('./examples/image1.jpg').convert('RGB')

image_processor = CLIPImageProcessor.from_pretrained('OpenGVLab/InternViT-6B-448px-V1-5')

pixel_values = image_processor(images=image, return_tensors='pt').pixel_values
pixel_values = pixel_values.to(torch.bfloat16).cuda()

outputs = model(pixel_values)

```

Comments:
Hi @AmazDeng, the reason is `trust_remote_code=True` and the modeling code is present under the files of the hub repositories you linked. If you look there for instance https://huggingface.co/OpenGVLab/InternVL2-2B/blob/main/modeling_internvl_chat.py, you'll find the modeling code corresponding to these models, uploaded by the original authors directly. 

Since there is no issue here, I'm closing, feel free to open if an issue arises or check out our [forums](https://discuss.huggingface.co/)!
Remove unnecessary CPM model tests
Author: amyeroberts
Pull request: yes

# What does this PR do?

Noticed that the CPM tokenization test were running XLNet model tests, which are already run elsewhere, so removing this inheritance. 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33621). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Generate: remove flakyness in `test_generate_from_inputs_embeds_decoder_only`
Author: gante
Pull request: yes

# What does this PR do?

`test_generate_from_inputs_embeds_decoder_only` had low probability of failure... but not zero (see #33593 and its CI runs)

Instead of assuming that calling `generate` with random `inputs_embeds` will result in a different text, let's check the scores instead. Even if the output text is the same (unlikely), the token scores are almost guaranteed to be different.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33602). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

hah, a PR to fight flakiness got stuck in another flaky test 🙃 

@ydshieh I don't have merge permissions, and I agree with you -- can you merge this PR for me? :)
Fix for slow the bug tokenizer adding spaces to single id decodes
Author: DuyguA
Pull request: yes

# What does this PR do?
Quick fix for a bug with the tokenizer, slow tokenizers add spaces in between when the input is a single id.

Fixes #29489 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@ArthurZucker 




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32564). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @itazap as well!

> Thanks for the quick update 🤗 Thanks for merging the tests! Left a few comments about the single special token case, let me know what you think!

No worries, I'll do the changes :wink: 

@ArthurZucker  and @LysandreJik  merge time please :wink: 

Gentle ping @itazap  , can we do the merge? Some commits from the main was failing this branch but looks like all fixed  , can we do the merge before any more breaking changes come :grin: :grin: :grimacing: 

@DuyguA Sorry for the delay! Merged !! 🚀 Thanks for working on this 🤗 

🎉Congrats @DuyguA !This issue has really been a long journey.
added quick clarification
Author: DuyguA
Pull request: yes

# What does this PR do?

Closes #28882.
Minor addition to docs for clarifying `BatchEncoding`'s `batch_axis` param. 


## Before submitting
- [ x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker 




Comments:
removed redundant creation of causal mask when attention mask is already 4D
Author: p4arth
Pull request: yes

# What does this PR do?

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments: