noamaanMulla-03/transformers-issues huggingface dataset:
[tests] make `test_model_parallelism` device-agnostic  
Author: faaany
Pull request: yes

## What does this PR do?
As the title suggests, `test_model_parallelism` is not GPU-specific and could also work on other accelerators, e.g. XPU device. Below are the before and after the fix for `pytest -rA tests/models/bart/test_modeling_bart.py::BartModelTest::test_model_parallelism`:
BEFORE
```bash
================================================ short test summary info ================================================
FAILED tests/models/bart/test_modeling_bart.py::BartModelTest::test_model_parallelism - AssertionError: device(type='xpu', index=0) != device(type='cuda', index=0)
============================================= 1 failed, 4 warnings in 3.75s =============================================
```
AFTER
```bash
================================================ short test summary info ================================================
PASSED tests/models/bart/test_modeling_bart.py::BartModelTest::test_model_parallelism
============================================= 1 passed, 4 warnings in 5.96s =============================================
```

Pls help revie this PR, thx! @amyeroberts and @ydshieh 



Comments:
Good from my side, thanks. Let's maybe wait a remark the our `accelerate` experts before merge.
FIX / TST: Fix expected results on Mistral AWQ test 
Author: SunMarc
Pull request: yes

# What does this PR do?
This PR fixes the Mistral AWQ slow test. This failing was caused by the torch.compile [PR](https://github.com/huggingface/transformers/pull/30642). Since the output is coherent, I think the best is to change the expected output, just like what we did [here](https://github.com/huggingface/transformers/pull/30909). 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30971). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker You know better than me about if such changes of expected outputs are reasonable.

@younesbelkada For PR like #30909, it would be nice to mention why such expected values are changed by which PR. If we can't track down the reason, at least mention that fact.

Yes, here's the answer: 

```
You're standing on the surface of the Earth. You walk one mile south, one mile west and one mile north. You end up exactly where you started. Where are you?

This is a classic puzzle that has been around for a while. The solution is that you are at the center of a square with sides of length 3 miles.

Let's
```


LGTM
allow multi-gpu
Author: ydshieh
Pull request: yes

# What does this PR do?

Allow `SSH into our runners` workflow to use `multi-gpu`

Comments:
@SunMarc let me know if it works for you. Merge now.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31011). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add Nomic Embed Code to Transformers
Author: zanussbaum
Pull request: no
Labels: name: New model

### Model description

[Nomic Embed](https://arxiv.org/abs/2402.01613) embedding models are high performing long-context embedding models. 

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

https://huggingface.co/nomic-ai/nomic-embed-text-v1
https://huggingface.co/nomic-ai/nomic-embed-text-v1.5
https://github.com/nomic-ai/contrastors

Comments:
The modeling code itself can be found here: https://huggingface.co/nomic-ai/nomic-bert-2048

Oops thanks for adding that, I forgot to add it!
Quantization / TST: Fix remaining quantization tests
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR fixes the [remaining quantization tests](https://github.com/huggingface/transformers/actions/runs/9217736706/job/25360291113)( quanto quantized cache) + makes sure the GGUF tests are run (they all pass on the T4 VM) by removing it from the transformers-all-latest-gpu docker image to the quantization docker image

```bash
========================================================================================================= 11 passed, 3 warnings in 506.17s (0:08:26) ==========================================================================================================
```

cc @SunMarc @ydshieh @zucchini-nlp 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31000). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

For GGUF it is just that the tests were currently compltely skipped, these tests are independent from the quanto cache tests !

OK. But my question is still there: we don't need any `require_xxx` for `QuantoKVCacheQuantizationTest`? It looks like it requires some quantize libraries being installed, no?
Fix resume_download future warning
Author: Wauplin
Pull request: yes

Fix https://github.com/huggingface/transformers/issues/31002 and follow-up https://github.com/huggingface/transformers/issues/30618.

Looks like I forgot a `resume_download` in https://github.com/huggingface/transformers/issues/30618. This PR fixes it to avoid a FutureWarning for which the user can't do much. 

cc @albertvillanova 

Note: this is "simply" as warning but should not trigger any errors. It triggers one in datasets' CI but that's only because of pytest settings.

Comments:
@ArthurZucker I added a regression test in https://github.com/huggingface/transformers/pull/31007/commits/6921a7f5fc02915c8761352e8bd0254c8328b85a. It is not related to local_files_only or snapshot_download problems, just a FutureWarning raised when `resume_download` is passed to latest version of `huggingface_hub`.

(feel free to merge whenever it's approved/CI is happy)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31007). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
save the list of new model failures
Author: ydshieh
Pull request: yes

# What does this PR do?

We finally get back the missing new model failure table and list on slack channels, but those are truncated version.
We have the full **table** saved as artifact, but not the list. This PR saves the full **list** of new model failures as artifact too, so we can access the information outside slack.

This is particularly useful when a CI run has many new failures.

**table**: (just some numbers)

```bash
Changed model modules failures:

Single PT |  Multi PT | Single TF |  Multi TF |     Other | Category
        0 |         0 |         0 |         0 |        +7 | fsdp
        0 |         0 |         0 |         0 |        +1 | models_cohere
        0 |       +22 |         0 |        +1 |         0 | models_deit
```

**list**: (has the test names and job links)

```bash
<https://github.com/huggingface/transformers/actions/runs/9194245573/job/25287588257|multi> gpu
tests/fsdp/test_fsdp.py::TrainerIntegrationFSDP::test_basic_run_full_shard_bf16

<https://github.com/huggingface/transformers/actions/runs/9194245573/job/25287588257|multi> gpu
tests/fsdp/test_fsdp.py::TrainerIntegrationFSDP::test_basic_run_shard_grad_op_bf16
```


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31013). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Perceiver interpolate position embedding
Author: justyx404
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Add interpolate position embedding to Perceiver #30579. Not really sure how to test the correctness of output value since it does change the entire forward pass. However, correctness of output shape is pretty straight forward.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts 

Comments:
Thanks for the review, I have addressed the comments. Turns out after working on the no longer needed `DETR`, it makes sorting through the code path of a different model much faster.
added interpolation for vitmae model in pytorch as well as tf.
Author: bhuvanmdev
Pull request: yes

# What does this PR do?

Adds interpolation capabilities for the tf and pytorch versions of vit_mae along with the test cases.

Addresses #30579  (vit_mae) 
as discussed in #30657

## Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested.

@amyeroberts


Comments:
Apologies, My previous PR got polluted with commits due to my negligence and lack of knowledge :( . I will make sure that is not repeated ever again.

> Apologies, My previous PR got polluted with commits due to my negligence and lack of knowledge :( . I will make sure that is not repeated ever again.

@bhuvanmdev It's OK, we're all learning :) If you hit tricky issues with the PR or git, feel free to ping on the PR and we can try to help (can't guarantee we'll know the answer though!)

Done @amyeroberts

@bhuvanmdev Should be good to rebase! 

@amyeroberts should I sync this branch wrt to upstream main branch?

@bhuvanmdev Yes please
`PreTrainedTokenizerFast._batch_encode_plus()` got an unexpected keyword argument `'split_special_tokens'`
Author: fahadh4ilyas
Pull request: no
Labels: name: Core: Tokenization

### System Info

Transformer version: 4.38.1
Platform: Ubuntu
Python version: 3.10.13

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer

access_token = 'YOUR_ACCESS_TOKEN'

tokenizer = AutoTokenizer.from_pretrained('meta-llama/Meta-Llama-3-8B', token=access_token)

print(tokenizer('Here is an example of bos token: 
', split_special_tokens=True)
```

### Expected behavior

Must return this:
```python
{'input_ids': [128000, 8586, 374, 459, 3187, 315, 43746, 4037, 25, 83739, 7413, 3659, 4424, 91, 29], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
```

Comments:
Hi,

this flag is only supported for slow tokenizers. See: https://github.com/huggingface/transformers/blob/f26e4073707189c93915227779a4f6ea3c40d43b/src/transformers/tokenization_utils_base.py#L1539-L1543

#28648 should fix this.

cc @itazap if you can take it over! 🤗
Paligemma- fix devices and dtype assignments
Author: molbap
Pull request: yes
Labels: name: run-slow

# What does this PR do?
Moves tensors to correct devices in case of multi-gpu training on accelerate and device_map = auto. 
Additionally ensures bf16 training works as well.

Fixes #30997 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31008). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @ArthurZucker wdyt?

fwiw most of the lines in here are nearly identical to the changes i have done locally as well besides the `final_embedding` related one which i believe can be done with only 1 cast but didnt think too deeply about it

@grahamannett , good to know. For `final_embedding` it's also to fix the bf16 dtype mismatch.
loss calculation for PaliGemmaForConditionalGeneration potentially not cast to correct device
Author: grahamannett
Pull request: no

### System Info

```
Name: transformers
Version: 4.42.0.dev0
```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Use `PaliGemmaForConditionalGeneration.forward` with device_map='auto' and labels for 4+ devices.

### Expected behavior

I believe which device logits/labels/attention_mask will be on for `PaliGemmaForConditionalGeneration` can potentially cause an error for others when using `device_map='auto'`.

https://github.com/huggingface/transformers/blob/main/src/transformers/models/paligemma/modeling_paligemma.py#L487

```
                shift_logits = shift_logits[shift_attention_mask.to(logits.device) != 0].contiguous()
                shift_labels = shift_labels[shift_attention_mask.to(logits.device) != 0].contiguous()
```

labels and attention_mask are going to be on the lowest index device by default.  logits likely to be a later device if the model is sharded with `device_map='auto'`.  The shift_logits line is right since the shift_logits come from the logits but the shift_labels line can cause issues since shift_labels should likely be moved to logits device.




Comments:
Hi @grahamannett , thanks for the catch - will open a PR to fix!

@grahamannett can you share the traceback as well? To identify where this failed
Missing "config.json" when loading Llama-2-7b-chat-hf
Author: stefanomezza
Pull request: no

### System Info

Google Colab T4 GPU

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction
```
hf_token = "<<token>>"  #(real token censored, but I have it in my local version )
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf", trust_remote_code=True, token=hf_token)
```
This gives the following error:
```
OSError: meta-llama/Llama-2-7b-chat-hf does not appear to have a file named config.json. Checkout 'https://huggingface.co/meta-llama/Llama-2-7b-chat-hf/tree/None' for available files.
```

Comments:
Hi @stefanomezza, thanks for raising an issue! 

It shouldn't be necessary to pass in `trust_remote_code=True` when calling `AutoTokenizer`. All of the code the llama model checkpoint uses is defined in transformers. 

Which version of transformers are you using? I'm able to run the following without issue on `main`: 

```py
In [1]: from transformers import AutoTokenizer

In [2]: tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf
   ...: ")
```

I was logged into HF in my python session, you may still need to pass in your hf token

I can also run when passing in `trust_remote_code`:

```py
In [3]: tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf
   ...: ", trust_remote_code=True)
```

cc @ArthurZucker 

Dear Amy,
Thank you for your prompt response. 

I double checked and the reason why I was getting that issue is that I had an empty folder called `meta-llama/Llama-2-7b-chat-hf` which was created in an except block by mistake 😅 this is what happens when you program after bedtime hahah. 

Thank you for your support, I guess I can close this one!
Is there a source code installation method available? For example: from Test_transformers import AutoModelForCausalLM, AutoTokenizer 
Author: gg22mm
Pull request: no

### Feature request

Is there a source code installation method available? For example: from Test_transformers import AutoModelForCausalLM, AutoTokenizer 
I see that the models are now written in the huggingface and packaged and called through: import transformers. If you want to debug, how do you debug it? 
If you want to modify the code or other operations inside, it's not too difficult~~

### Motivation

有没有源码方式安装？ 比如： from Test_transformers import AutoModelForCausalLM, AutoTokenizer 
﻿ 
我看现在模型都写在huggingface中，通过： import transformers 打包调用，如果想调试怎么调试？ 
如果想修改一下里面的代码 或是 其它一些操作 那不太难了 ！~~

### Your contribution

no

Comments:
Hi @gg22mm, thanks for raising an issue! 

You can install from source using this command: `pip install git+https://github.com/huggingface/transformers`. This will install the development version of transformers i.e. with the most recent commits on `main`.

I'm not sure what you mean "packaged and called through: import transformers". `import foo from bar` is just the module structure defined in the library through e.g. folders, imports and `__init__.py` files. You can see all of the code here: https://github.com/huggingface/transformers

If you want to modify the code, you can [make a fork of the library](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo)

Thank you~~

I think: One of the dumbest ways is to reinstall the library every time you make changes~:  
pip install git+ https://github.com/my_transformers 
------------------------------------ 
我想：有一种最笨的办法就是每改一次重新安装一下库 ： 
pip install git+https://github.com/my_transformers    

@gg22mm You asked about installing from source, which I gave you the command to do.

If you want to use the installed library and include any local changes, you can clone and install the library as an editable using `pip install -e .`, in which case you work from the local version and the install isn't repeated. In this case, all you need to do is fetch upstream changes using git, there is no need to re-install. 

That's really great. Thank you~
[Flax] from_pretrained does not consider the passed dtype
Author: patil-suraj
Pull request: no
Labels: name: WIP

## Environment info

When loading a flax model with `from_pretrained` the type argument is not used. The weights are initialized with the dtype of saved weights.

So if you do 
```python
model = FlaxGPT2ForCausalLM.from_pretrained("gpt2", dtype=jnp.dtype("bfloat16"))

# check the dtype of one of the params 
model.params["transformer"]["wpe"]["embedding"].dtype
=> dtype("float32")
```

We should probably cast the weights to `self.dtype`.

As a workaround for `bf16`,  one could manually cast the weighs with
```
def to_bf16(t):
    return jax.tree_map(lambda x: x.astype(jnp.bfloat16) if x.dtype == jnp.float32 else x, t)

model.params = to_bf16(model.params)
```

cc @patrickvonplaten 

Comments:
I wonder whether this might be problematic for layer norm weights since those should usually always be of type `float32`, no?

Would love to hear what @avital @marcvanzee think here

I think it's fine to manually port weights to bfloat16 if you want to. In general all Flax layers accept a dtype attribute when it's safe to do intermediate computation in bloat16 and you can set dtype=bfloat16 for those layers. Keeping parameters as bfloat16 should only be necessary if the model is huge and the parameters can't fit on device memory, from what I know. I think it's tricky to get that right and requires careful attention to which parameters are safe to keep in bfloat16, but I don't have too much personal context here. I can ask others if that helps.

So I'm first curious whether indeed it's necessary to keep parameters as bfloat16 in this case, and if so, why

hello so 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

will soon be taken care of by @patil-suraj :-) 

This issue still exists. I was trying to make a float64 GPT2 instance ([using this](https://huggingface.co/docs/transformers/v4.41.2/en/model_doc/gpt2#transformers.FlaxGPT2LMHeadModel)) and noticed the initiated LayerNorm parameters having float32 dtype.
Can't load tokenizer
Author: wb-zr707791
Pull request: no

OSError: Can't load tokenizer for 'togethercomputer/LLaMA-2-7B-32K'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'togethercomputer/LLaMA-2-7B-32K' is the correct path to a directory containing all relevant files for a LlamaTokenizer tokenizer.


请问这个问题该如何解决？

我已经尝试了通过access token的方式，不起作用
def cl100k(text, tokenizer):
    return tokenizer.encode(text, disallowed_special=())


def llama(text, tokenizer):
    return tokenizer(text, add_special_tokens=True, return_attention_mask=False)[
        "input_ids"
    ]


TOKENIZER_FUNCS = {
    "cl100k": (tiktoken.get_encoding("cl100k_base"), cl100k),
    "llama": (LlamaTokenizer.from_pretrained("togethercomputer/LLaMA-2-7B-32K", token=""), llama),
}

Comments:
Hi @wb-zr707791, please follow the [issue template](https://github.com/huggingface/transformers/blob/main/.github/ISSUE_TEMPLATE/bug-report.yml) nd provide information about the running environment. 

It's possible this was a connection issue. I'm unable to replicate it on my end and can successfully load the tokenizer:

```py
In [1]: from transformers import LlamaTokenizer

In [2]: tokenizer = LlamaTokenizer.from_pretrained("togethercomputer/LLaMA-2-7B-32K")
tokenizer_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████| 942/942 [00:00<00:00, 926kB/s]
tokenizer.model: 100%|████████████████████████████████████████████████████████████████████████████████████████| 500k/500k [00:00<00:00, 17.0MB/s]
added_tokens.json: 100%|██████████████████████████████████████████████████████████████████████████████████████| 4.00/4.00 [00:00<00:00, 9.15kB/s]
special_tokens_map.json: 100%|█████████████████████████████████████████████████████████████████████████████████| 96.0/96.0 [00:00<00:00, 290kB/s]
tokenizer.json: 100%|███████████████████████████████████████████████████████████████████████████████████████| 1.84M/1.84M [00:00<00:00, 4.93MB/s]
``` 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.