noamaanMulla-03/transformers-issues huggingface dataset:
Improve `PreTrainedTokenizerFast` loading time when there are many added tokens
Author: ydshieh
Pull request: yes

# What does this PR do?

The condition check

> token not in self.added_tokens_decoder

is very slow, especially when there are more added tokens.


## Loading time (see code snippet below): 2048 added tokens

### before this PR:
3.909789 seconds

### after this PR: 
0.23008 seconds

```
from transformers import AutoProcessor, XLMRobertaTokenizerFast, XLMRobertaTokenizer, AutoTokenizer, PreTrainedTokenizerFast

import datetime

ckpt = "ydshieh/dummy_tok"
s = datetime.datetime.now()
p = XLMRobertaTokenizerFast.from_pretrained("my_p")
e = datetime.datetime.now()
print((e-s).total_seconds())


```

Comments:
Super nice BTW 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31404). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> is it on par to have the init at the AddedTokens class level? (make sure to not import it from tokenizers)

sorry, I don't understand this. Could you elaborate a bit more?

What I mean is implement the __hash__ for this class

@ArthurZucker Everything is addressed.

https://github.com/huggingface/tokenizers/pull/1521 a related PR
Fix documentation typos
Author: qgallouedec
Pull request: yes

# What does this PR do?

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@stevhliu @MKhalusova

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31476). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Give more useful `metric_for_best_model` errors
Author: tomaarsen
Pull request: yes

# What does this PR do?
When you're using `load_best_model_at_end`, you may accidentally incorrectly specify the `metric_for_best_model` to something different than is returned by the `evaluate` method. This is especially true in Sentence Transformers, where evaluators return many different metrics with different and not-immediately-obvious metric names.

All this does is override the original error with a newer one that tells people that they should fix this by updating `metric_for_best_model` - it also wasn't clear what the possible `metrics` are. This wasn't exactly obvious before, e.g. https://github.com/UKPLab/sentence-transformers/issues/2749

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@muellerzr, @SunMarc

- Tom Aarsen

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31450). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Alright, awesome. I'm assuming the TF test failures are unrelated? I can't even inspect them 😄

Merging as the failures are unrelated to this PR.
Jamba: revert skipped test 
Author: gante
Pull request: yes

# What does this PR do?

Fixes #30828 🤗 

Jamba: reverts skipped test, it is now passing on `main`
Mamba: `all_generative_model_classes` is not set on Mamba's tests, as it requests the attention outputs in the `generate` calls (which mamba doesn't have). Added a comment regarding that on `MambaModelTest`

(feel free to merge if you approve)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31094). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Just so I understand, are mamba and jamba generative models? I thought so. Is there a reason why we can't have custom generative tests for these models which don't require attention? 

@amyeroberts both are generative. 

Jamba has the generative tests enabled (it has attention outputs): https://github.com/huggingface/transformers/blob/5c88253556b7f15cf7d7e9793d7b2a39b4aa588a/tests/models/jamba/test_modeling_jamba.py#L289

Mamba doesn't have. I agree it would be nice to have some tests -- or better yet, for the tests to be more flexible, don't check attention if the model doesn't have attentions. Would you be okay with a) adding a TODO in the code; b) opening an issue to make the tests more flexible?

> Would you be okay with a) adding a TODO in the code; b) opening an issue to make the tests more flexible?

@gante In this case no. As mamba is pretty popular, has very unusual logic, this isn't just something flaky e.g. hardware, and generates we should be testing its generation logic properly
ValueError: The checkpoint you are trying to load has model type `zoedepth` but Transformers does not recognize this architecture
Author: justinmanley
Pull request: no

### System Info

- `transformers` version: 4.41.2 (also true of github rep head @ 4.42.0.dev0, installed via `pip install git+https://github.com/huggingface/transformers.git`)
- Platform: macOS-11.5.1-x86_64-i386-64bit
- Python version: 3.10.13
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.2.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: ?? not sure

### Who can help?

@amyeroberts? (for vision -- depth estimation)

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run the example code from https://huggingface.co/Intel/zoedepth-nyu-kitti:

```python
from transformers import pipeline

depth_estimator = pipeline(task="depth-estimation", model="Intel/zoedepth-nyu-kitti")
```

### Expected behavior

I expect the `depth_estimator` to be fully initialized without throwing any errors.

Comments:
Hi @justinmanley, 

ZoeDepth hasn't been merged into transformers yet -- #30136 -- and so can't be used with the pipeline. It should be ready to go soon! @NielsRogge is off for a bit, so I'd expect it to be available within maybe a week? 

ahhh ok got it! that makes sense. thanks for explaining @amyeroberts -- I'll keep an eye on https://github.com/huggingface/transformers/pull/30136!
Fix autocast incompatibility in RecurrentGemma
Author: xplip
Pull request: yes

# What does this PR do?

Fixes #30830

In `modeling_recurrent_gemma.py`, cast the `key_states` and `value_states` in `RecurrentGemmaSdpaAttention._update_cache` to the dtype of the cache tensors they are written to. This way, a RuntimeError is prevented when autocasting to bfloat16/float16.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
cc @ArthurZucker 

Hey, could you share a reproducer? 🤗 

> Hey, could you share a reproducer? 🤗

Sure, I added one in the issue https://github.com/huggingface/transformers/issues/30830 :)
RecurrentGemma not compatible with autocast / AMP training
Author: xplip
Pull request: no

### System Info

- `transformers` version: 4.40.2
- Platform: Linux-5.15.0-92-generic-x86_64-with-glibc2.31
- Python version: 3.10.13
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.3
- Accelerate version: 0.30.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
import torch
from transformers import RecurrentGemmaConfig, RecurrentGemmaForCausalLM


def main():
    V = 288
    B = 16
    T = 300
    device = "cuda"

    config = RecurrentGemmaConfig(
        vocab_size=V,
        num_hidden_layers=12,
        hidden_size=1024,
        num_attention_heads=8,
        intermediate_size=6144,
        attention_window_size=T,
    )
    model = RecurrentGemmaForCausalLM._from_config(config, torch_dtype=torch.float32).to(device)

    autocast_settings = [
        {"dtype": torch.float16, "enabled": True},
        {"dtype": torch.bfloat16, "enabled": True},
        {"enabled": False},
    ]

    for autocast_setting in autocast_settings:
        print(f"\nRunning with autocast setting: {autocast_setting}:")
        try:
            with torch.cuda.amp.autocast(**autocast_setting):
                outputs = model(input_ids=torch.randint(0, V, (B, T), device=device))
                print(outputs.logits.shape)
        except RuntimeError as e:
            print(e)
    

if __name__ == "__main__":
    main()
```

### Expected behavior

The script should run without errors with autocast enabled, as otherwise training with AMP is not available.

Output of the script above:
```markdown
Running with autocast setting: {'dtype': torch.float16, 'enabled': True}
Index put requires the source and destination dtypes match, got Float for the destination and Half for the source.

Running with autocast setting: {'dtype': torch.bfloat16, 'enabled': True}
Index put requires the source and destination dtypes match, got Float for the destination and BFloat16 for the source.

Running with autocast setting: {'enabled': False}
torch.Size([16, 300, 288])
```

Expected output:
```markdown
Running with autocast setting: {'dtype': torch.float16, 'enabled': True}
torch.Size([16, 300, 288])

Running with autocast setting: {'dtype': torch.bfloat16, 'enabled': True}
torch.Size([16, 300, 288])

Running with autocast setting: {'enabled': False}
torch.Size([16, 300, 288])
```

Comments:
Will merge the PR!
Failed to import transformers.models.vit.feature_extraction_vit because of the following error (look up to see its traceback): No module named 'ml_dtypes._custom_floats'
Author: JJLee2910
Pull request: no

Please help me with the following errors, as I have no clue on what does this happen.
![image](https://github.com/huggingface/transformers/assets/83653118/272bc2eb-da5d-4c4d-bc6d-06a3b708f534)


Comments:
Hi @JJLee2910, thanks for raising an issue! 

So that we can help you, please make sure to follow the [bug report issue template](https://github.com/huggingface/transformers/blob/main/.github/ISSUE_TEMPLATE/bug-report.yml) and provide: 

* A minimal code reproducer we can run to replicate the issue
* The running environment: run `transformers-cli env` in the terminal and copy-paste the output
* All relevant information about the error encountered including: expected behaviour; experience behaviour and full error traceback. For example, at the moment it's not possible from the screenshot to see which line of code is triggering this. 

As a general note, it's better to copy-paste the terminal output than take a screen shot. This both makes it easier for us to debug (we can copy-paste the text for searches) and ensures that anyone else looking for this issue will find it and it will hit text matches from github or google search 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Use self.config_tester.run_common_tests()
Author: amyeroberts
Pull request: yes

# What does this PR do?

Remove overrides of `test_config` to use `self.config_test.run_common_tests()`. This makes sure all of the config tests in `test_configuration_common.py` are being run

In some cases, `common_properties` is set when creating the config tester. This is a bit hacky, but as `common_properties` isn't that well defined in the original test I think this is OK until we split into different testing modalities. 

c.f. https://github.com/huggingface/transformers/pull/31359#pullrequestreview-2118861632


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31431). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[tests] rename `test_config_object` to `test_ds_config_object`
Author: faaany
Pull request: yes

# What does this PR do?
`test_config` is a common test name for many models and is identified as `NOT_DEVICE_TESTS` in the [conftest.py](https://github.com/faaany/transformers/blob/main/conftest.py#L44)). So when users use the test marker `not_device_test` to run the test suit, `tests/deepspeed/test_deepspeed.py::TrainerIntegrationDeepSpeed::test_config_object` will be skipped. But `test_config_object` is not a CPU-only UT and should not be skipped. Can we rename it to `test_ds_config_object`? 

@amyeroberts and @ydshieh 


Comments:
Hi @ydshieh , could you take a look at this PR? Thx a lot!  

Hi @faaany . Thank you. Good from my side to change the names. But looking at `conftest.py`

```python
def pytest_collection_modifyitems(items):
    for item in items:
        if any(test_name in item.nodeid for test_name in NOT_DEVICE_TESTS):
            item.add_marker(pytest.mark.not_device_test)
```

I think the mentioned issue could be avoided by changing 

> test_name in item.nodeid

to 

> test_name == item.nodeid

WDYT, does it makes sense? If so could you give it a try. We can keep the both current change with what I suggest (if that works well)

Hi @ydshieh , I am not sure whether it is a good idea because there are also `test_name` like `ModelTest::test_pipeline_` or `/utils/` in the NOT_DEVICE_TESTS variable. So I thought maybe it was designed to use `in`. WDYT?

> Hi @ydshieh , I am not sure whether it is a good idea because there are also `test_name` like `ModelTest::test_pipeline_` or `/utils/` in the NOT_DEVICE_TESTS variable. So I thought maybe it was designed to use `in`. WDYT?

Wow, thank you for checking this. I think so (I'm not the author of this part of code 😅 )



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31403). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Docs  / AQLM: Clarify `torch.compile` support for AQLM
Author: younesbelkada
Pull request: yes

# What does this PR do?

As discussed offline with @justheuristic AQLM supports `torch.compile` which should be clearly stated in the docs

![image](https://github.com/huggingface/transformers/assets/49240599/286e377d-44e6-466c-bee1-2bc87dc6bfd5)

cc @SunMarc @amyeroberts 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31473). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Unable to run generation tests for Mamba & Jamba models
Author: amyeroberts
Pull request: no

### System Info

- `transformers` version: 4.41.0.dev0
- Platform: Linux-5.15.0-1045-aws-x86_64-with-glibc2.31
- Python version: 3.10.9
- Huggingface_hub version: 0.23.0
- Safetensors version: 0.4.2
- Accelerate version: 0.29.2
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.2.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): 0.7.0 (cpu)
- Jax version: 0.4.13
- JaxLib version: 0.4.13
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@gante @zucchini-nlp 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

See #30826

`test_assisted_decoding_matches_greedy_search_0_random` is forcibly skipped for Jamba because it's necessary to unset `_supports_cache_class` to resolve failing tests on main. 

`test_assisted_decoding_matches_greedy_search_0_random` appears to pass for Mamba, but this is because `all_generative_models` is not set in the model tester

### Expected behavior

Either `test_assisted_decoding_matches_greedy_search_0_random` can be run for both models with `_supports_cache_class` unset or it's not necessary to have `_supports_cache_class` unset for Jamba

Comments:
Might be related to #30800
Mamba: add generative tests
Author: gante
Pull request: yes

Supercedes #31094

Fixes #30828 🤗

This PR:
1. Enables the generative tests to `mamba`
2. Removes redundant/needless manual skips through the `has_attentions` tester flag (TIL about this flag)
3. To enable (1.), adds a new flag to `PreTrainedModel` -- `_is_stateful` (self explanatory) [note: AFAIK we only have 3 stateful models, `mamba`, `jamba`, and `rwkv`, lmk if there are more!]
4. Uses `_is_stateful` to set a few appropriate exceptions and test skips

Note: with `has_attentions` and `_is_stateful`, I suspect I can remove a few more skips in a follow up PR :)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31478). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Trouble with AutoModelForSequenceClassification + Lora + Deepspeed_zero3
Author: liuyu666-thu
Pull request: no

Need to finetune a seq-classifier based on Llama-70B using Lora, but get stuck when integrating deepspeed_zero3.

Deps version:
- transformers              4.31.0
- peft                             0.5.0
- deepspeed                 0.10.1

My code is like:
1. initialize training args: `train_args = transformers.TrainingArguments(..., deepspeed=ds_config.json)`
2. create model: `model = LlamaForSequenceClassification.from_pretrained(...)`
3. wrap it with peft: `model = get_peft_model(model, lora_config)`
4. enable gradient checkpointing: `model.gradient_checkpointing_enable()`
5. train it with HF-trainer: `trainer = Trainer(..., args=train_args)`
6. launch the training using `torchrun xxx.py`

The error is like:
`{'id': 547, 'status': 'NOT_AVAILABLE', 'numel': 0, 'ds_numel': 0, 'shape': (0,), 'ds_shape': (0,), 'requires_grad': True, 'grad_shape': None, 'persist': True, 'active_sub_modules': {1482}, 'ds_tensor.shape': torch.Size([0])}`

Seems like the classification layer is sharded and something incompetible happens. 

Can anyone show the standard way to use deepspeed under hf-trainer in this scene?

Comments:
Pretty sure you should be asking this on [the forum](https://discuss.huggingface.co/), include a full reproducer with a full traceback.

> Pretty sure you should be asking this on [the forum](https://discuss.huggingface.co/), include a full reproducer with a full traceback.

Sure. I'll post one there.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@liuyu666-thu Hi, did you manage to resolve this issue? I am dealing with the same problem in the same settings