Paged attention does not work
Author: yuyijiong
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.57.0
- Platform: Linux-5.15.0-107-generic-x86_64-with-glibc2.35
- Python version: 3.12.11
- Huggingface_hub version: 0.35.3
- Safetensors version: 0.6.2
- Accelerate version: 1.10.1
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: bf16
        - use_cpu: False
        - debug: False
        - num_processes: 8
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - enable_cpu_affinity: False
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.8.0+cu128 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no
- Using GPU in script?: yes
- GPU type: NVIDIA H20


### Who can help?

@vasqu @ArthurZucker @CyrilVallez

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When set attn_implementation to "paged_attention", I meet the error even with very simple inference code.
This error is similar to #39525

```python
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
import pandas as pd
import torch
from transformers import AutoTokenizer,Qwen3ForCausalLM,AutoConfig,AutoModelForCausalLM

if __name__ == '__main__':
    model_path="/share/models/Qwen3-4B-Thinking-2507"

    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True, add_bos_token=False,
                                              add_eos_token=False)

    model = AutoModelForCausalLM.from_pretrained(model_path,
                                             dtype=torch.bfloat16,
                                             trust_remote_code=True,
                                             #config=config,
                                             attn_implementation="paged_attention",#"flash_attention_2",#"sdpa",#
                                             device_map="cuda"  
                                             ).eval()

    prompt="How are you today?"
    chat_prompt=tokenizer.apply_chat_template([{"role":"user","content":prompt}],tokenize=False,add_generation_prompt=True)

    chat_prompt_ids=tokenizer(chat_prompt,return_tensors="pt")["input_ids"].to(model.device)
    output=model.generate(input_ids=chat_prompt_ids,max_new_tokens=500,num_beams=1,do_sample=False,temperature=1.0,use_cache=True,return_dict_in_generate=True,output_logits=True)
    output_text=tokenizer.decode(output['sequences'][0][chat_prompt_ids.size(1):])
    print(output_text)

```

Error:
```
  File "/root/miniconda3/envs/yyj/lib/python3.12/site-packages/transformers/models/qwen3/modeling_qwen3.py", line 260, in forward
    hidden_states, _ = self.self_attn(
                       ^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/yyj/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/yyj/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/yyj/lib/python3.12/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/yyj/lib/python3.12/site-packages/transformers/models/qwen3/modeling_qwen3.py", line 216, in forward
    attn_output, attn_weights = attention_interface(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/yyj/lib/python3.12/site-packages/transformers/integrations/flash_paged.py", line 85, in paged_attention_forward
    cu_seq_lens_q.to(torch.int32),
    ^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'to'
```

### Expected behavior

fix the bug

Comments:
Hey! that's expected! Sorry we should probably make that API obvious.... 
The way to use it: 
```python 
prompt="How are you today?"
chat_prompt=tokenizer.apply_chat_template([{"role":"user","content":prompt}],tokenize=False, add_generation_prompt=True)
batch_outputs = model.generate_batch(inputs=[chat_prompt])
```



@McPatate setting `cache_implementation` to `paged_cache` should probably make `generate` use `generate_batch` ! 

@ArthurZucker 
Hey! I saw the earlier discussion about generate_batch and paged_cache. Just wanted to confirm — is this bug still open or already resolved with the suggested fix? I’d be happy to help with this if it’s still unresolved. Can you confirm whether this issue still needs attention?

@ArthurZucker  What's more, ``num_return_sequences`` seems to have no effect when using ``generate_batch()``. Even when it is set to 8, I only get 1 output.

Example code:
```python
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
import pandas as pd
import torch
from transformers import AutoTokenizer,Qwen3ForCausalLM,AutoConfig,AutoModelForCausalLM,GenerationConfig

if __name__ == '__main__':
    model_path="/share/models/Qwen3-0.6B"

    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True, add_bos_token=False,
                                              add_eos_token=False)

    model = AutoModelForCausalLM.from_pretrained(model_path,
                                             dtype=torch.bfloat16,
                                             trust_remote_code=True,
                                             #config=config,
                                             attn_implementation="paged_attention",#"flash_attention_2",#"sdpa",#
                                             device_map="cuda" 
                                             ).eval()

    prompt="How are you today?"
    chat_prompt=tokenizer.apply_chat_template([{"role":"user","content":prompt}],tokenize=False,add_generation_prompt=True,enable_thinking=False)

    generation_config=GenerationConfig(
        do_sample=True,
        temperature=1.0,
        top_p=1.0,
        num_return_sequences=8,
        max_new_tokens=100,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
        use_cache=True,
        return_dict_in_generate=True,
        output_logits=True,
    )


    if "paged" not in model.config._attn_implementation:
        chat_prompt_ids = tokenizer(chat_prompt, return_tensors="pt")["input_ids"].to(model.device)
        output = model.generate(input_ids=chat_prompt_ids,
                                        generation_config=generation_config,)
        output_text = tokenizer.batch_decode(output[:, chat_prompt_ids.shape[1]:], skip_special_tokens=True)
        print(output_text)

    else:
        chat_prompt_ids = tokenizer([chat_prompt])["input_ids"]
        output = model.generate_batch(inputs=chat_prompt_ids,
                                        generation_config=generation_config,)
        for output_each_request in output.values():
            output_text = tokenizer.decode(output_each_request.generated_tokens)
            print(output_text)
```

That is correct, cc @remi-or and @McPatate that's one more generate arg that we should support along with beam search!

@rohaanmd you can have a go if you want, otherwise we'll open a PR this week ! 

Hey @yuyijiong , we are going to adress the issue with `cache_implementation` in #41655
As for anything related to beam search, sorry it is not implemented yet in continuous batching. 
It is the next new feature on the roadmap, thank you for your understanding! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.