Reproducibility Issue of Siglip2 with Blackwell Architecture GPUs (RTX 5090)
Author: silverstar0727
Pull request: no
Labels: name: bug

### System Info

- transformers version: `4.52.4`
- Platform: Linux
- Python version: 3.12
- PyTorch version: 2.7.0+cu128
- CUDA version: 12.8
- GPU: NVIDIA GeForce RTX 5090 (Blackwell architecture)
- Driver Version: 570.144

### Who can help?

## Who can help?

@amyeroberts  @qubvel  (vision models - SigLIP2 related issue)

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction


### Problem Description
**Non-deterministic behavior specifically on Blackwell architecture GPUs (RTX 5090)** when running identical inference code. The same SigLIP2 model produces different outputs across runs on RTX 5090, while maintaining perfect reproducibility on other GPU architectures (A6000, RTX 4090).

### Environment Comparison
- ✅ **NVIDIA A6000 (Ampere)**: Perfect reproducibility without any deterministic flags
- ✅ **NVIDIA RTX 4090 (Ada Lovelace)**: Perfect reproducibility without any deterministic flags  
- ❌ **NVIDIA RTX 5090 (Blackwell)**: Non-deterministic behavior even with all deterministic settings enabled

### Steps to reproduce the behavior:

1. **Set all deterministic flags** (this still doesn't fix the issue on RTX 5090):
```python
import os
os.environ['CUBLAS_WORKSPACE_CONFIG'] = ':4096:8'

import torch
torch.use_deterministic_algorithms(True)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.manual_seed(42)
torch.cuda.manual_seed_all(42)
```

2. **Run the reproduction script twice**:

```python
import torch
import pandas as pd
from datasets import load_dataset
from transformers import pipeline
from transformers.pipelines.pt_utils import KeyDataset
from tqdm import tqdm
import os
from accelerate import Accelerator
from accelerate.utils import gather_object


def set_deterministic():
    """Set deterministic behavior"""
    os.environ['CUBLAS_WORKSPACE_CONFIG'] = ':4096:8'
    torch.use_deterministic_algorithms(True)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    torch.manual_seed(42)
    torch.cuda.manual_seed_all(42)


def run_test(run_id):
    """Run reproducibility test"""
    set_deterministic()
    accelerator = Accelerator()
    
    # Load CIFAR-10 dataset (100 images)
    ds = load_dataset("cifar10", split="test[:100]")
    
    # Create pipeline with SigLIP2
    classifier = pipeline(
        "image-classification",
        model="google/siglip2-giant-opt-patch16-384",
        device_map={"": accelerator.process_index},
        trust_remote_code=True,
    )
    
    if hasattr(classifier.model, 'eval'):
        classifier.model.eval()
    
    accelerator.wait_for_everyone()
    
    # Process images
    with accelerator.split_between_processes(ds) as subset:
        img_ds = KeyDataset(subset, "img")
        rows = []
        
        for idx, out in enumerate(
            tqdm(
                classifier(img_ds, batch_size=32, top_k=50),
                total=len(img_ds),
                disable=not accelerator.is_main_process,
            )
        ):
            sorted_out = sorted(out, key=lambda x: x["label"])
            result_dict = {"ID": idx}
            for r in sorted_out:
                result_dict[r["label"]] = r["score"]
            rows.append(result_dict)
    
    rows = gather_object(rows)
    
    if accelerator.is_main_process:
        df = pd.DataFrame(rows)
        columns = ["ID"] + sorted([c for c in df.columns if c != "ID"])
        df = df.reindex(columns, axis=1)
        
        filename = f"siglip2_test_run{run_id}.csv"
        df.to_csv(filename, index=False)
        return filename
    return None


# Run twice and compare
file1 = run_test(1)
file2 = run_test(2)

# Compare results
df1 = pd.read_csv(file1)
df2 = pd.read_csv(file2)
numeric_cols = [col for col in df1.columns if col != "ID"]

for col in numeric_cols[:2]:
    diff = (df1[col] - df2[col]).abs()
    print(f"{col}: max_diff = {diff.max():.2e}")
```

3. **Run on RTX 5090**

### Actual Results on RTX 5090

```
🔍 COMPARING RESULTS:
❌ LABEL_0: max_diff = 4.63e-03
❌ LABEL_1: max_diff = 4.63e-03

📋 SUMMARY:
Maximum difference: 4.63e-03
⚠️ REPRODUCIBILITY ISSUES DETECTED!
🔍 Worst case in 'LABEL_1':
   Row 82: 0.4096104801 vs 0.4049816132
```

**Error magnitude**: Up to `4.63e-03` difference between identical runs - this is a **significant reproducibility issue**.

### Investigation Results

- ✅ All deterministic PyTorch settings properly configured
- ✅ Same code works reproducibly on Ampere (A6000) and Ada Lovelace (4090) architectures  
- ✅ Tested with Accelerate library (identical to production usage)
- ✅ Model set to eval mode explicitly
- ❌ Issue persists specifically on Blackwell architecture (RTX 5090)

### Expected behavior

Identical inference runs should produce identical outputs when deterministic flags are set, regardless of GPU architecture. This behavior works correctly on:
- NVIDIA A6000 (Ampere architecture)
- NVIDIA RTX 4090 (Ada Lovelace architecture)

## Additional Context

This appears to be a **Blackwell architecture-specific issue** where new CUDA kernels or hardware optimizations don't fully respect PyTorch's deterministic settings. The issue has significant practical implications:

1. **Model evaluation inconsistency**: Results vary between runs on the same hardware
2. **Research reproducibility**: Experiments cannot be reliably reproduced
3. **Production reliability**: Model outputs are unpredictable

### Hardware Specifications
- **RTX 5090**: Compute Capability (12, 0), CUDA 12.8, Driver 570.144
- **Memory**: 32GB GDDR7
- **Architecture**: Blackwell with 5th gen RT cores, 4th gen Tensor cores

### Potential Root Causes
- New Blackwell CUDA kernels may not implement deterministic algorithms
- Updated attention mechanisms (Flash Attention 3.0) might be non-deterministic
- Memory coalescing patterns could be different from previous architectures


Comments:
Hey @silverstar0727, thanks for sharing the issue. Unfortunately, I don't have an RTX 5090 to reproduce the problem, but if you have the bandwidth to narrow down the root cause, it would be helpful. Have you tried using another model / no accelerate / no pipelines (image processor + model)?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.