Qwen3.5 Does not work with variable length inputs using position_ids
Author: kolehma8
Pull request: no
Labels: name: bug

### System Info

The current implementation of Qwen3.5 linear attention assumes that user passes cu_seqlen arguments explicitly as opposed to just position_ids like older models. While the problem can be circumvented by passing these inputs, it becomes a problem in downstream packages (such as Verl) that assume the same interface between HuggingFace based language models. The problem is also silent and does not raise any errors but simply performs incorrect computation when cu_seqlen(s) are omitted with variable length inputs.

To fix this, the variable sequence length inputs should be treated the same way in Qwen3.5 as all other models. Also, the current torch fallback silently fails even when the cu_seqlen inputs are passed as it never uses them.

Relevant code paths:
*https://github.com/huggingface/transformers/blob/v5.12.0/src/transformers/models/qwen3_5/modeling_qwen3_5.py#L545
*https://github.com/huggingface/transformers/blob/v5.12.0/src/transformers/models/qwen3_5/modeling_qwen3_5.py#L494
*https://github.com/huggingface/transformers/blob/v5.12.0/src/transformers/models/qwen3_5/modeling_qwen3_5.py#L246

Standard variable length attention handling:
*https://github.com/huggingface/transformers/blob/v5.12.0/src/transformers/modeling_flash_attention_utils.py#L790

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run forward pass of Qwen3.5 with packed inputs with and without the cu_seqlen arguments.

### Expected behavior

Same packed sequence length behavior as Qwen3 (or any other model).

Comments:
I confirmed and reproduced it and with a tiny Qwen3.5 text model (2 linear-attention layers), comparing a packed batch (`batch_size=1`, two sequences concatenated, `position_ids `reset per sequence) against each sequence run independently:
```
packed (position_ids only)   max|Δ| seqA=4.768e-07  seqB=3.818e-01  WRONG
packed (explicit cu_seq_lens) max|Δ| seqA=4.768e-07  seqB=3.818e-01  WRONG
```
Sequence A is fine (can't see B), but B leaks across the packing boundary in both cases:
1. Interface: `Qwen3_5GatedDeltaNet only reads boundaries from `kwargs["cu_seq_lens_q"]` and never derives them from `position_ids` like the standard FA path, so passing only `position_ids` silently gives wrong results.
2. Torch fallback: `torch_chunk_gated_delta_rule` ignores `cu_seqlens `entirely, so even passing them explicitly computes across boundaries. `Qwen3-Next` shares this fallback.

**Proposed fix:** derive `cu_seq_lens`/`max_length` from `position_ids` when packing is detected, and make `torch_chunk_gated_delta_rule` honor `cu_seqlens` by processing each sub-sequence independently. Both live in the modular files, so it covers Qwen3-Next and Qwen3.5-MoE too.

Happy to open a PR!

## Reproduction Script
```python3
import torch
from transformers import Qwen3_5TextConfig
from transformers.models.qwen3_5.modeling_qwen3_5 import Qwen3_5TextModel

torch.manual_seed(0)

def build_model():
    config = Qwen3_5TextConfig(
        vocab_size=256,
        hidden_size=128,
        intermediate_size=256,
        num_hidden_layers=2,
        num_attention_heads=4,
        num_key_value_heads=2,
        head_dim=32,
        linear_key_head_dim=16,
        linear_value_head_dim=16,
        linear_num_key_heads=2,
        linear_num_value_heads=4,
        linear_conv_kernel_dim=4,
    )
    model = Qwen3_5TextModel(config).eval()
    print("layer_types:", config.layer_types)
    return model

@torch.no_grad()
def run(model, input_ids, position_ids=None, **kwargs):
    out = model(
        input_ids=input_ids,
        position_ids=position_ids,
        use_cache=False,
        **kwargs,
    )
    return out.last_hidden_state

def main():
    model = build_model()

    seq_a = torch.tensor([[5, 9, 2, 7, 1, 8]])      # length 6
    seq_b = torch.tensor([[3, 6, 4, 10, 11]])        # length 5
    la, lb = seq_a.shape[1], seq_b.shape[1]

    # reference: run each sequence on its own
    ref_a = run(model, seq_a)[0]
    ref_b = run(model, seq_b)[0]

    # packed: concat into one sequence, position_ids reset per sequence
    packed_ids = torch.cat([seq_a, seq_b], dim=1)
    position_ids = torch.cat(
        [torch.arange(la), torch.arange(lb)]
    ).unsqueeze(0)  # [[0..5, 0..4]]

    # cu_seq_lens describing the two packed sub-sequences: [0, 6, 11]
    cu = torch.tensor([0, la, la + lb], dtype=torch.int32)

    packed_posids_only = run(model, packed_ids, position_ids=position_ids)[0]
    packed_with_cu = run(
        model,
        packed_ids,
        position_ids=position_ids,
        cu_seq_lens_q=cu,
        cu_seq_lens_k=cu,
        max_length_q=la,
        max_length_k=la,
    )[0]

    def report(name, packed):
        out_a = packed[:la]
        out_b = packed[la:]
        da = (out_a - ref_a).abs().max().item()
        db = (out_b - ref_b).abs().max().item()
        ok = da < 1e-4 and db < 1e-4
        print(f"  {name:28s} max|Δ| seqA={da:.3e}  seqB={db:.3e}  {'OK' if ok else 'WRONG'}")
        return ok

    print("\nComparison of packed forward vs independent forward:")
    report("packed (position_ids only)", packed_posids_only)
    report("packed (explicit cu_seq_lens)", packed_with_cu)

if __name__ == "__main__":
    main()
```

It is not supported for a reason, see https://github.com/huggingface/transformers/issues/46093#issuecomment-4535723384 for details

@vasqu , too much confusion around this issue, should we make it clear by raising an error/warning if possible?

The problem is to create an appropriate warning we would need to do a data dependent branching (i.e. check that position ids is indeed packing free format), not sure if this is really viable 🤔

I would really like https://huggingface.co/docs/transformers/v5.12.0/en/padding_free#padding-free-training to be the entry point for users to reference and check out

Thanks @zucchini-nlp , for the added context. Going forward, what is the standard way to use variable length inputs in any model? I am working on downstream side and looking for a consistent interface to do this for HuggingFace models. Currently it works for most and then fails silently for Qwen3.5. I don't think having unique interfaces to all models is really sustainable for applications using HuggingFace.

@kolehma8 check out https://huggingface.co/docs/transformers/v5.12.0/en/padding_free#linear-attention-and-convolution-models the data collator can take of everything you need in this case

on the docs side, we can try adding more entry points from places where users are likely to encounter this such as:

- add a note to the Qwen3.5 model docs with a link to `padding_free.md`, and mention users must use `DataCollatorWithFlattening`
- add wording to the `DataCollatorWithFlattening` API docs for affected models

Thanks @vasqu and @stevhliu , I will close the issue as there seems to a standard solution to the problem already.