[exaone4] ZeroDivisionError/TypeError when sliding_window_pattern is None/"LLLG" and _attn_implementation stays None (4.54.0 & main)
Author: wheeze01
Pull request: no
Labels: name: bug

### System Info

```
- `transformers` version: 4.54.0
- Platform: Linux-5.15.0-124-generic-x86_64-with-glibc2.35
- Python version: 3.11.11
- Huggingface_hub version: 0.34.1
- Safetensors version: 0.5.3
- Accelerate version: 1.8.1
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: FSDP
        - mixed_precision: bf16
        - use_cpu: False
        - debug: False
        - num_processes: 4
        - machine_rank: 0
        - num_machines: 1
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - enable_cpu_affinity: False
        - fsdp_config: {'fsdp_activation_checkpointing': False, 'fsdp_auto_wrap_policy': 'TRANSFORMER_BASED_WRAP', 'fsdp_backward_prefetch': 'BACKWARD_PRE', 'fsdp_cpu_ram_efficient_loading': True, 'fsdp_forward_prefetch': False, 'fsdp_offload_params': False, 'fsdp_reshard_after_forward': 'FULL_SHARD', 'fsdp_state_dict_type': 'SHARDED_STATE_DICT', 'fsdp_sync_module_states': True, 'fsdp_use_orig_params': True, 'fsdp_version': 1}
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.7.0+cu126 (CUDA)
- Tensorflow version (GPU?): 2.18.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA RTX A6000
```

### Who can help?

@lgai-exaone 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "LGAI-EXAONE/EXAONE-4.0-1.2B"
# model_name = "LGAI-EXAONE/EXAONE-4.0-32B"

model = AutoModelForCausalLM.from_pretrained(
    model_name, torch_dtype="bfloat16", device_map=None
).to("cuda")
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = "너가 얼마나 대단한지 설명해 봐"
messages = [{"role": "user", "content": prompt}]
input_ids = tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_tensors="pt"
)

output = model.generate(
    input_ids.to(model.device),
    max_new_tokens=128,
    do_sample=False,
)
print(tokenizer.decode(output[0]))
```

### Expected behavior

Some routes are my personal, so I processed [MASKED]. The [MASKED] part is my account name.

# EXAONE-4.0-1.2B
```
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/[MASKED]/braille_translator/trash/test_exaone4.py", line 6, in <module>
    model = AutoModelForCausalLM.from_pretrained(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/[MASKED]/miniconda3/envs/py11/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 547, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/[MASKED]/miniconda3/envs/py11/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1277, in from_pretrained
    return config_class.from_dict(config_dict, **unused_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/[MASKED]/miniconda3/envs/py11/lib/python3.11/site-packages/transformers/configuration_utils.py", line 789, in from_dict
    config = cls(**config_dict)
             ^^^^^^^^^^^^^^^^^^
  File "/home/[MASKED]/miniconda3/envs/py11/lib/python3.11/site-packages/transformers/models/exaone4/configuration_exaone4.py", line 208, in __init__
    self.layer_types = [
                       ^
  File "/home/[MASKED]/miniconda3/envs/py11/lib/python3.11/site-packages/transformers/models/exaone4/configuration_exaone4.py", line 210, in <listcomp>
    if ((i + 1) % (sliding_window_pattern) != 0 and i < self.num_hidden_layers)
        ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~
ZeroDivisionError: integer modulo by zero
```
# EXAONE-4.0-32B
```
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/[MASKED]/braille_translator/trash/test_exaone4.py", line 6, in <module>
    model = AutoModelForCausalLM.from_pretrained(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/[MASKED]/miniconda3/envs/py11/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 547, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/[MASKED]/miniconda3/envs/py11/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1277, in from_pretrained
    return config_class.from_dict(config_dict, **unused_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/[MASKED]/miniconda3/envs/py11/lib/python3.11/site-packages/transformers/configuration_utils.py", line 789, in from_dict
    config = cls(**config_dict)
             ^^^^^^^^^^^^^^^^^^
  File "/home/[MASKED]/miniconda3/envs/py11/lib/python3.11/site-packages/transformers/models/exaone4/configuration_exaone4.py", line 208, in __init__
    self.layer_types = [
                       ^
  File "/home/[MASKED]/miniconda3/envs/py11/lib/python3.11/site-packages/transformers/models/exaone4/configuration_exaone4.py", line 210, in <listcomp>
    if ((i + 1) % (sliding_window_pattern) != 0 and i < self.num_hidden_layers)
        ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~
TypeError: unsupported operand type(s) for %: 'int' and 'str'
```


Comments:
## 1. Summary

In `Exaone4Config` initialization, when `sliding_window_pattern` is provided as `None` (for the 1.2B model) or the string `"LLLG"` (for the 32B model), a modulo operation (`%`) is performed, assuming it's an integer. This leads to `ZeroDivisionError` or `TypeError`. Additionally, an incorrect check for the `"sliding_window"` string can leave `_attn_implementation` as `None`.

**Affected Versions:** Reproduced in both Transformers 4.54.0 (release) and `main` (as of 4.55.0.dev0).

**Models:** `LGAI-EXAONE/EXAONE-4.0-1.2B`, `LGAI-EXAONE/EXAONE-4.0-32B`

---
## 2. Root Cause Analysis

* The `sliding_window_pattern` is expected to be an integer but can be `None` or a string. A modulo operation (`%`) is performed on it, assuming it's an integer, leading to a `ZeroDivisionError` or `TypeError`.
* Specifically, for the `EXAONE-4.0-1.2B` model hosted on Hugging Face, `"sliding_window_pattern"` is set to `None`.
* For the `EXAONE-4.0-32B` model, `"sliding_window_pattern"` is `"LLLG"`.
* The token `"sliding_window"` does not appear in `layer_types`, which results in `_attn_implementation` not being set.

---
## 3. Expected Behavior

- Referencing #39129, the intention appears to be:
    - **`EXAONE-4.0-1.2B`**: All layers should use `full_attention`.
    - **`EXAONE-4.0-32B`**: A hybrid approach of `sliding_attention` and `full_attention` should be applied according to the "LLLG" pattern.
- We expect `_attn_implementation` to rely on the default values set by Hugging Face's internal logic.

---
Here's the English translation of the remaining sections:

-----

## 4. Proposed Fix (PR Planned)

  - **Safe branching during `layer_types` automatic generation:**
      - If `sliding_window_pattern` is `None` or `0` → All layers become `"full_attention"`.
      - If `sliding_window_pattern` is a string (e.g., `"LLLG"`) → `'L'` means `"sliding_attention"`, otherwise `"full_attention"`, and the last layer is always `"full_attention"`.
      - If `sliding_window_pattern` is an integer (e.g., `4`) → `"full_attention"` is placed periodically, the rest are `"sliding_attention"`, and the last layer is `"full_attention"`.
  - Remove the incorrect `"sliding_window"` string check (it should be `"sliding_attention"`).
  - Do not forcefully set `_attn_implementation` to a custom string (`"hybrid"`). (Leave it to HF's internal `_check_and_adjust_attn_implementation` logic.)
  - Add a test script: For the 32B model, due to OOM prevention, test only on CPU with short tokens.

(Detailed diff and test logs will be included in the PR body)

-----

## 5. Local Test Results

```
=== Checking LGAI-EXAONE/EXAONE-4.0-1.2B ===
 - first 8 layer_types: ['full_attention', 'full_attention', 'full_attention', 'full_attention', 'full_attention', 'full_attention', 'full_attention', 'full_attention']
 - total layers: 30
 - has sliding_attention? False
 - generate OK, logits shape: torch.Size([1, 37])
 - sample output: [|user|]\n너가 얼마나 대단한지 설명해 봐[|endofturn|]\n[|assistant|]\n<think>\n\n</think>\n\n저는 EXAONE으로, LG AI Research에서 개발된 대규모 언어 모델 ...
 - full_mask: None
=== Done: LGAI-EXAONE/EXAONE-4.0-1.2B ===

=== Checking LGAI-EXAONE/EXAONE-4.0-32B ===
 - first 8 layer_types: ['sliding_attention', 'sliding_attention', 'sliding_attention', 'full_attention', 'sliding_attention', 'sliding_attention', 'sliding_attention', 'full_attention']
 - total layers: 64
 - has sliding_attention? True
 - generate OK, logits shape: torch.Size([1, 37])
 - sample output: [|user|]\n너가 얼마나 대단한지 설명해 봐[|endofturn|]\n[|assistant|]\n<think>\n\n</think>\n\n저는 LG AI 연구원에서 개발한 EXAONE입니다. 저는 다양한 ...
 - full_mask: None
 - sliding_mask: None
=== Done: LGAI-EXAONE/EXAONE-4.0-32B ===
```

-----

## 6. Impact / Compatibility

  - No impact on models other than the EXAONE 4.0 series.
  - No public API changes. (Backward compatibility maintained)
  - No performance impact (logic branching only during initialization).

-----

## 7. Discussion Points

  - Is it necessary to explicitly set `_attn_implementation`?
  - Should `sliding_window_pattern` have a standardized notation (string vs. integer)?

LG-EXAONE 4.0's config.json file is modified and this issue has been resolved!