noamaanMulla-03/transformers-issues huggingface dataset:
Fix get_usable_length for StaticCache
Author: guangy10
Pull request: no
Labels: name: bug

name: ExecuTorch

### System Info

transformers: 4.45.0.dev0
torch: 2.5.0.dev20240716+cpu

### Who can help?

@ArthurZucker
@gante
@zucchini-nlp

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

It's a bug only affect the new workflow ["Export to ExecuTorch"](https://github.com/huggingface/transformers/issues/32253) that we're trying to enable. The method [`get_usable_length`](https://github.com/huggingface/transformers/blob/main/src/transformers/cache_utils.py#L69-L78) should be override for `StaticCache` where recompile, resizing or evicting the existing cache entry won't be applicable. This is because unlike eager or `torch.compile`d model, exported model will be running in a non-python env where recompiling from the eager source isn't available. 

### Expected behavior

The `generation` length using the exported artifact should not exceed the maximal cache length because the model and the size of the its cache are exported statically. When `get_usable_length` returns 0, it should terminate the generation.

Comments:
The `get_usable_length` is something that we are deprecating anyways, but good to note. StaticCache should not even have it! 

@ArthurZucker Because @helunwencser in our team is working on Phi-3-mini, and in Phi-3's modeling code, `get_usable_length` is used in several places like [here](https://github.com/huggingface/transformers/blob/044281605f5fc037e1048d4360da971fdfbca370/src/transformers/models/phi3/modeling_phi3.py#L441) or [here](https://github.com/huggingface/transformers/blob/044281605f5fc037e1048d4360da971fdfbca370/src/transformers/models/phi3/modeling_phi3.py#L536). The modeling code itself is fine because doesn't make assumption about the type of cache being used. However, when comes to trace the model via `torch.export`, the default `get_usable_length` will trigger evicting old cache entries hence new attentions from that point will be incorrect. I think it would make sense to override it for `StaticCache` to avoid misbehavior in the short-term before it can be fully deprecated.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
convert_megatron_gpt2_checkpoint.py
Author: syx11237744
Pull request: no
Labels: name: bug

### System Info

transformers 4.40.0
python 3.10

### Who can help?

@ArthurZucker
@Narsil
@SunMarc

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction
```python
when i use convert_megatron_gpt2_checkpoint,i got 
Traceback (most recent call last):
  File "/home/code/max_cp/convert_m_g.py", line 356, in <module>
    main()
  File "/home/code/max_cp/convert_m_g.py", line 316, in main
    output_state_dict = convert_megatron_checkpoint(args, input_state_dict, config)
  File "/home/code/max_cp/convert_m_g.py", line 215, in convert_megatron_checkpoint
    out_name = megatron_to_transformers[op_name]
KeyError: 'input_norm'
```

If I use another version of Megatron, I encounter another problem：
```python
Traceback (most recent call last):
  File "/home/sunyuanxu/code/max_cp/convert_m_g.py", line 356, in <module>
    main()
  File "/home/sunyuanxu/code/max_cp/convert_m_g.py", line 316, in main
    output_state_dict = convert_megatron_checkpoint(args, input_state_dict, config)
  File "/home/sunyuanxu/code/max_cp/convert_m_g.py", line 215, in convert_megatron_checkpoint
    out_name = megatron_to_transformers[op_name]
KeyError: 'self_attention.layernorm_qkv'
```
### Expected behavior

Is it due to the version of Megatron? Which version of Megatron should I use to train in order to use this script?

Comments:
Hey! Sorry we might need a bit more context, like which checkpoint are you converting? 
The conversion script here seems to say that the model you are trying to convert does not have that layer. I am also not sure `transformers` is used as I can't see that in the traceback

I face this problem recently. I found this caused by latest megatron replacing `layernorm` to `norm`. When i fix it, the code run successful.
```
if op_name.endswith("norm"):
  ln_name = "attention.ln" if op_name.startswith("input") else "ln"
    output_state_dict[layer_name + "." + ln_name + "." + weight_or_bias] = val

output_state_dict["bert.encoder.ln.weight"] = transformer["final_norm.weight"]
output_state_dict["bert.encoder.ln.bias"] = transformer["final_norm.bias"]

output_state_dict["cls.predictions.transform.LayerNorm.weight"] = lm_head["norm.weight"]
output_state_dict["cls.predictions.transform.LayerNorm.bias"] = lm_head["norm.bias"]

```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
FutureWarning: `torch._dynamo.external_utils.is_compiling` is deprecated. Use `torch.compiler.is_compiling` instead.
Author: xwang233
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.43.3
- PyTorch version (GPU?): 2.5.0a0+git522fa03 (True) https://github.com/pytorch/pytorch/commit/522fa03e911d822ced8b956e862f3875cb78dd84

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

transformers code: 
https://github.com/huggingface/transformers/blob/13dc6b0853c3cb54e79b18105c0528bc9e84881c/src/transformers/modeling_attn_mask_utils.py#L445C43-L445C69

failed test error message:
```python
/usr/local/lib/python3.10/dist-packages/transformers/models/bert/modeling_bert.py:1107: in forward
    extended_attention_mask = _prepare_4d_attention_mask_for_sdpa(
/usr/local/lib/python3.10/dist-packages/transformers/modeling_attn_mask_utils.py:445: in _prepare_4d_attention_mask_for_sdpa
    or (hasattr(torch, "_dynamo") and torch._dynamo.is_compiling())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (), kwargs = {}

    @functools.wraps(arg)
    def wrapper(*args, **kwargs):
>       warnings.warn(msg, category=category, stacklevel=stacklevel + 1)
E       FutureWarning: `torch._dynamo.external_utils.is_compiling` is deprecated. Use `torch.compiler.is_compiling` instead.
```

pytorch code:
https://github.com/pytorch/pytorch/blob/1add8c5f1c4909ca2a57d110807684aaa230b7b2/torch/_dynamo/external_utils.py#L18-L22

### Expected behavior

no future warning on deprecated API usages

Comments:
Hi @xwang233, thanks for raising this issue! 

The deprecation warning is coming from a nightly release or torch, which we don't officially support in the library. That being said, we should prepare for this when it's part of a stable release. It seems `torch.compiler.is_compiling` has been available for a few months, so can move to using this instead. As we still officially support older versions of pytorch, this would need to be version guarded and fall back to the dynamo import if not available. Would you like to open a PR to add this?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Unused variable in test_pipelines_text_generation.py function
Author: jacorobe
Pull request: no

https://github.com/huggingface/transformers/blob/c1aa0edb48217f416f4bbe6e3a9db1500284513b/tests/pipelines/test_pipelines_text_generation.py#L323-L325

TextGenerationPipeline parameter, processor, is unused within this scope of the function. Uncertain if this test is incomplete or if this variable is optional and potentially removeable.

Comments:
Seems like a lot of places don't use it cc @ydshieh 
Do you want to open a PR  and fix it ? 🤗 

I believe it's just a general pattern: we want to have `get_test_pipeline` having a uniform argument list. But inside the body, only parts of them are used, depends on the pipeline class.

I don't feel strong the need to change it. But if we decide to do it, I would prefer after #32514 being done and merged

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
must be real number, not SymFloat for Qwen/Qwen 7B Model with torch.compile inductor backend. 
Author: Vasanta1
Pull request: no
Labels: name: bug

### System Info

Transformers : 4.38.2 
Torch : 2.3.0 +cpu
Python : 3.8

Faced "must be real number, not SymFloat" error when running Qwen/Qwen-7B Model with auto tokenizer and auto model class from pretrained with torch.compile API. 
Let me know if any other experiments need to be run. 
![image](https://github.com/user-attachments/assets/c575f3b1-ff3a-428e-bb1b-aa043e351e91)



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Load Qwen/Qwen 7B with auto pretrained API. 
2. Run with torch.compile(model)

### Expected behavior

Error : must be real number, not SymFloat

Comments:
Hey! Could you try with the latest releases of `transformers`? 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
index is out of bounds for dimension with size 0 (after  4.41.2)
Author: geekodour
Pull request: no
Labels: name: bug

### System Info

see this issue for more details: https://github.com/kijai/ComfyUI-moondream/issues/19

Getting this error post this version which is preventing usage of the latest version. I'll try digging for a fix for this but otherwise if seems like an easy pick might be worth fixing.

### Who can help?

@amyeroberts  @ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

 https://github.com/kijai/ComfyUI-moondream/issues/19

### Expected behavior

 https://github.com/kijai/ComfyUI-moondream/issues/19

Comments:
Hey @geekodour ! From what I see the Phi-model code is totally defined within moondream, while they are calling to the `transformers` implementation of `generate()`. I recommend you to open an issue in moondream since they would have to update Phi modeling code, we have been actively modifying it lately.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
get_logits_warper_patch
Author: zhangyong114
Pull request: no
Labels: name: bug

### System Info

jetson-agix-orin


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction


Traceback (most recent call last):
  File "/opt/text-generation-webui/modules/callbacks.py", line 61, in gentask
    ret = self.mfunc(callback=_callback, *args, **self.kwargs)
  File "/opt/text-generation-webui/modules/text_generation.py", line 382, in generate_with_callback
    shared.model.generate(**kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 1975, in generate
    self._get_logits_warper(generation_config, device=input_ids.device)
TypeError: get_logits_warper_patch() got an unexpected keyword argument 'device'


### Expected behavior

Answer my question

Comments:
Hey @JokerZhY, I'm not sure how that relates to `transformers`, do you have a code sample I could run to see where the issue lies?

Hey @LysandreJik, I encountered an error while using text-generation-webui
.3, specifically when I was importing the llama3.1 model. At that time, I had just updated transformers to version 4.43.3 and now suspect there might be a conflict between libraries. When I output input_ids.device, it correctly shows cuda:0.

@oobabooga, please let us know if we can help with anything here

This may be helpful for oobabooga. I successfully loaded the model "hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4" with AutoAWQ in text-generation-webui (not latest version of webui). But it does not generate output and gives the same error. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
'weight' must be 2-D
Author: bahriwissal
Pull request: no
Labels: name: bug

### System Info


    transformers version == 4.42.4
    dp-transformers version == 1.0.1
    accelerate version == 0.29.3
    llm == Distil-gpt2
    torch version == 2.0.0
    The model is runned under ddp with 2 GPUs


### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Hello, I am trying to train a Tabula (https://github.com/zhao-zilong/Tabula) model using FSDP. I rewrote my Tabula trainer without middle padding to use the FSDP model. I added the parameters for training arguments fsdp=full_shard, and then I used setup and multiprocessing.spawn to run my code. When I run my code using DDP, it works, but when I run it using FSDP, the training works, but the sampling does not. I get the error 'weight' must be 2-D. I know that FSDP works because when I print the memory usage, the memory reduces while using FSDP instead of DDP.

### Expected behavior

The sampling function should generate data.


Comments:
Hey @bahriwissal we'd love to help you with `transformers` but distilgpt is a very heavily tested model we know should work; so the problem likely comes from your script. You can share it here to get opinions/help from the community but it'd be awesome if you could reduce it to a minimal reproduction script so that we may help the best.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Gemma-2 9B query_pre_attn_scalar value
Author: danielkeysers
Pull request: no

### System Info

(Sorry, I'm relatively new to github, please let me knowif this is not the right route to discuss this.)

[I think system info should not be relevant to this issue.]

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

No specific "reproduction", but I think the value for "query_pre_attn_scalar" at
https://github.com/huggingface/transformers/blob/cffa2b9c1dd825df1c0e949b99aaef1655c28625/src/transformers/models/gemma2/convert_gemma2_weights_to_hf.py#L64
should probably be changed from 224 to 256 to mirror the change in gemma_pytorch (https://github.com/google/gemma_pytorch/commit/03e657582d17cb5a8617ebf333c1c16f3694670e) that happened recently.


### Expected behavior

The differences in observed behavior are likely small given the small change in this scaling factor.

Comments:
Hey! A PR is being opened on the hub to fix it! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

In the meantime, the google-deepmind/gemma repository was also updated for gemma2 and also uses head_dim (i.e. 256) for the query_pre_attn_[scalar/norm]:
https://github.com/google-deepmind/gemma/blob/a0504162f99a1c238efb37b8197e711c0f3808fd/gemma/transformer.py#L195

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
about `mixed_precision` in acclerate config
Author: AIR-hl
Pull request: no
Labels: name: trainer

name: Accelerate

when I already set `mixed_precision: bf16` in acclerate config, should I also set `bf16=True` in training args？

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.



cc @muellerzr @SunMarc 

No, we'll pick it up from the environment

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Segmentation fault when trying to load models
Author: michaelcapizzi
Pull request: no

We are using `Azure ML` pipelines to train our `transformers` models.  We have had it working for a few weeks, and then recently (just noticed it a few days ago), when trying to initialize a model, we are getting `Segmentation fault`.

I tried just loading the models locally this morning and have the same issues.  See snippet below.

```
config = config_class.from_pretrained(model_name, num_labels=10)
tokenizer = tokenizer_class.from_pretrained(model_name, do_lower_case=False)
model = model_class.from_pretrained("distilroberta-base", from_tf=False, config=config)
```

I also tried to download the `*_model.bin` and pass a *local path* instead of the model *name* and also got a `Segmentation fault`.  I also tried to use `bert-base-uncased` instead of `distilroberta-base` and had the same issue.

I am running on Ubuntu, with the following package versions:

```
torch==1.3.0
tokenizers=0.0.11
transformers==2.4.1
```

**UPDATE**:

I hacked some example scripts and had success, so I *think* the issue is that our code uses...

```
    "roberta": (RobertaConfig, RobertaForTokenClassification, RobertaTokenizer),
    "mroberta": (RobertaConfig, RobertaForMultiLabelTokenClassification, RobertaTokenizer),    # our custom multilabel class
```

instead of what the example scripts use...

```
    AutoConfig,
    AutoModelForTokenClassification,
    AutoTokenizer,
```

Was there a breaking change to model files recently that would mean that our use of the "non-auto" classes are no longer usable?


**UPDATE 2**:

Our original code does *not* cause a `Segmentation fault` on Windows.

Comments:
Bumping to `torch==1.5.1` fixes this issue.  But it's still unclear why.

I have also met the same issue and upgrading to torch1.5.1 also solves my problem.

Possibly related to https://github.com/huggingface/transformers/issues/4857

**Downgrade to sentencepiece==0.1.91 solve it.** 
I am using PyTorch 1.2.0 + transformers3.0.0

> **Downgrade to sentencepiece==0.1.91 solve it.**
> I am using PyTorch 1.2.0 + transformers3.0.0

Also PyTorch 1.4.0 + transformers 3.0.2

Closing this as solved by #5418. Feel free to re-open if you still face an issue.

For me either adding `sentencepiece==0.1.91 + torch==1.3.1 + transformers==2.4.1`
or `torch==1.5.1 + transformers==2.4.1` worked.

I come across the same problem too. 
My solution is just to import torch before import the transformers

> I come across the same problem too. My solution is just to import torch before import the transformers

I followed your solution and it worked🤣. Before doing in this way, I downloaded almost every version of sentencepiece from 0.1.91 to 0.1.97. Although I do not know why, but it's something to happy about.

> I come across the same problem too. My solution is just to import torch before import the transformers

OMG, this worked for me. Why though :O