modeling_phi3 errors with AttributeError: 'DynamicCache' object has no attribute 'get_max_length'
Author: doctorpangloss
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.49.0.dev0 (315a9f494e0e00d8652722ce950be590852a4727~1)
- Platform: Windows-10-10.0.20348-SP0
- Python version: 3.11.7
- Huggingface_hub version: 0.28.1
- Safetensors version: 0.5.2
- Accelerate version: 1.3.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): 0.10.2 (cpu)
- Jax version: 0.5.0
- JaxLib version: 0.5.0
- Using distributed or parallel set-up in script?: no
- Using GPU in script?: no
- GPU type: NVIDIA RTX A5000

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Use Phi3 with any cache configuration, including default (DynamicCache)

I think get_max_length is probably declared on a mixin that isn't on the cache classes yet?

```
comfy_extras\nodes\nodes_language.py:361: in execute
    return model.generate(tokens, max_new_tokens, repetition_penalty, seed, sampler),
comfy\language\transformers_model_management.py:228: in generate
    output_ids = transformers_model.generate(
..\..\.venv\Lib\site-packages\torch\utils\_contextlib.py:116: in decorate_context
    return func(*args, **kwargs)
..\..\.venv\Lib\site-packages\transformers\generation\utils.py:2224: in generate
    result = self._sample(
..\..\.venv\Lib\site-packages\transformers\generation\utils.py:3198: in _sample
    model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
C:\Users\bberman\.cache\huggingface\modules\transformers_modules\c1358f8a35e6d2af81890deffbbfa575b978c62f\modeling_phi3.py:1292: in prepare_inputs_for_generation
    max_cache_length = past_key_values.get_max_length()
```

### Expected behavior

related to #35168?

I'm not sure why this is only coming up with phi-3 so far

Comments:
Hey @doctorpangloss !

The `get_max_length()` has been deprecated for several releases and is removed in the last v4.48. It should be `cache.get_max_cache_shape()`.

Since the source of error is the remote code stored on hub, you need to open an issue in model repo and ask authors to fix that. Also you can downgrade transformers version a a workaround

I need to upgrade to v4.49 to use GRPO, so downgrading the transformers version isn't an option. Since the issue is with the [HF hub's modeling_phi3.py](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct/blob/main/modeling_phi3.py), is there an easy way to get it to use the [transformers' modeling_phi3.py](https://github.com/huggingface/transformers/blob/main/src/transformers/models/phi3/modeling_phi3.py) instead (where it looks like the issue is fixed)? My current workaround is to download the transformers version and modify it to use absolute imports instead of relative imports. 

@doctorpangloss you can set `trust_remote_code=False` if you are using one of the supported checkpoints. Check out https://github.com/huggingface/transformers/issues/32365#issuecomment-2270587554

Unfortunately the same workaround doesn't seem to exist for [phi3_v](https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/blob/main/modeling_phi3_v.py#L1352). @zucchini-nlp Are you aware of any limitations that would prevent a HF-maintained version of this model as well? I'd normally open a PR on HF, but the project hasn't been updated in 7 months.

@kylesayrs yeah, we didn't ship Phi3-Vision to transformers and it is supported only by hub code. You can still run it by cloning the repo and changing the problematic line. Not sure if authors are actively updating the model code

Hey I have the same error, with Phi2 backbone, did you manage to fix it ?

@kylesayrs limitation is just time 🤣 happy to have it in transformers

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

We currently have a user PR to add Phi-3.5-vision to the main library, which should resolve this: https://github.com/huggingface/transformers/pull/36036

> ### System Info
> * `transformers` version: 4.49.0.dev0 ([315a9f4](https://github.com/huggingface/transformers/commit/315a9f494e0e00d8652722ce950be590852a4727)~1)
> * Platform: Windows-10-10.0.20348-SP0
> * Python version: 3.11.7
> * Huggingface_hub version: 0.28.1
> * Safetensors version: 0.5.2
> * Accelerate version: 1.3.0
> * Accelerate config:    not found
> * PyTorch version (GPU?): 2.6.0+cu124 (True)
> * Tensorflow version (GPU?): not installed (NA)
> * Flax version (CPU?/GPU?/TPU?): 0.10.2 (cpu)
> * Jax version: 0.5.0
> * JaxLib version: 0.5.0
> * Using distributed or parallel set-up in script?: no
> * Using GPU in script?: no
> * GPU type: NVIDIA RTX A5000
> 
> ### Who can help?
> [@ArthurZucker](https://github.com/ArthurZucker)
> 
> ### Information
> * [ ]  The official example scripts[ ]  My own modified scripts
> 
> ### Tasks
> * [ ]  An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)[x]  My own task or dataset (give details below)
> 
> ### Reproduction
> Use Phi3 with any cache configuration, including default (DynamicCache)
> 
> I think get_max_length is probably declared on a mixin that isn't on the cache classes yet?
> 
> ```
> comfy_extras\nodes\nodes_language.py:361: in execute
>     return model.generate(tokens, max_new_tokens, repetition_penalty, seed, sampler),
> comfy\language\transformers_model_management.py:228: in generate
>     output_ids = transformers_model.generate(
> ..\..\.venv\Lib\site-packages\torch\utils\_contextlib.py:116: in decorate_context
>     return func(*args, **kwargs)
> ..\..\.venv\Lib\site-packages\transformers\generation\utils.py:2224: in generate
>     result = self._sample(
> ..\..\.venv\Lib\site-packages\transformers\generation\utils.py:3198: in _sample
>     model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
> C:\Users\bberman\.cache\huggingface\modules\transformers_modules\c1358f8a35e6d2af81890deffbbfa575b978c62f\modeling_phi3.py:1292: in prepare_inputs_for_generation
>     max_cache_length = past_key_values.get_max_length()
> ```
> 
> ### Expected behavior
> related to [#35168](https://github.com/huggingface/transformers/issues/35168)?
> 
> I'm not sure why this is only coming up with phi-3 so far

Updating the line 1292 of `modeling_phi3.py` from `max_cache_length = past_key_values.get_max_length()` to `max_cache_length = past_key_values.get_max_cache_shape()` seems to be fixing the issue. 

Submitted a fix to Phi-3.5-Vision-Instruct huggingface repo: https://huggingface.co/microsoft/Phi-3.5-vision-instruct/discussions/38

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Just use:
pip install transformers==4.47.0

Setting `use_cache=False `in the` generation_args` fixes the issue for me. It slows everything down, however.

> Just use: pip install transformers==4.47.0

I tried `pip install transformers==4.47.0` and restarted the kernel, but it did not fix the error for me, but adding `use_cache=False` to my Pipeline args did fix it.

> Setting `use_cache=False `in the` generation_args` fixes the issue for me. It slows everything down, however.

hey did, you find the workaround for this issue, i am running in same problem and currently doing same in generation_args to fix but latency is my concern