Issue Report: Abnormal GPU Utilization Pattern - DDP Training CLIP Model from Scratch
Author: xiehuanyi
Pull request: no

## Problem Description

Encountering abnormal GPU utilization patterns when training a CLIP model from scratch using DDP (Distributed Data Parallel). The monitoring charts show:

- **GPU Memory Allocation**: All 8 GPUs maintain a stable ~10% memory allocation
- **GPU Utilization**: Highly irregular fluctuation pattern, oscillating between 0-100% with significant idle periods

This pattern suggests potential issues with:
1. GPU synchronization/waiting problems
2. Data loading bottleneck
3. Gradient accumulation/communication issues
4. Torch compile or Flash Attention compatibility problems

## Environment Information

```yaml
Hardware:
  - GPU: 8x A100
  - Node: Single node

Software:
  - Framework: PyTorch + Transformers
  - Distributed: DDP (NCCL backend)
  - Mixed Precision: BF16
  - Optimizer: AdamW Fused
  - Special Features: 
    - torch_compile=True
    - flash_attention_2
    - gradient_checkpointing=False
```

## Training Configuration

```python
# Key Parameters
per_device_batch_size: 128
gradient_accumulation_steps: 1 (calculated)
target_global_batch_size: 1024
learning_rate: 1e-3
dataloader_num_workers: 8
dataloader_prefetch_factor: 4
dataloader_persistent_workers: True
```

## Reproduction Code

<details>
<summary>Complete Training Script (Click to expand)</summary>

```python
# -*- coding: utf-8 -*-
import os
import torch
from datasets import load_dataset
from transformers import (
    CLIPProcessor,
    CLIPModel,
    CLIPConfig,
    TrainingArguments,
    Trainer
)
from utils import (
    count_parameters,
    create_cc_dataset,
    parse_args,
    create_laion_dataset
)

def is_main_process():
    """Check if current process is main process"""
    return int(os.environ.get('LOCAL_RANK', 0)) == 0

def collate_fn(batch, processor):
    if 'text' in batch[0]:
        texts = [b['text'] for b in batch]
    elif 'conversations' in batch[0]:
        texts = [b['conversations'][1]['value'] for b in batch]
    images = [b['image'] for b in batch]
    return processor(
        text=texts,
        images=images,
        return_tensors="pt",
        padding=True,
        truncation=True,
        max_length=77
    )

def collate_fn_laion(batch, processor):
    texts = [b['txt'] for b in batch]
    images = [b['jpg'] for b in batch]
    out = processor(
        text=texts,
        images=images,
        return_tensors="pt",
        padding="max_length",
        truncation=True,
        max_length=77
    )
    return out

def train(args):
    world_size = int(os.environ.get('WORLD_SIZE', 1))
    rank = int(os.environ.get('RANK', 0))
    local_rank = int(os.environ.get('LOCAL_RANK', 0))
    
    print(f"[Rank {rank}] World size: {world_size}, Local rank: {local_rank}")
    print(f"[Rank {rank}] CUDA devices: {torch.cuda.device_count()}")
    
    # Set random seed
    torch.manual_seed(args.seed)
    if is_main_process():
        print("==== Config ====")
        for k, v in vars(args).items():
            print(f"{k}: {v}")
    
    # Performance optimization: Enable TF32
    torch.backends.cuda.matmul.allow_tf32 = True
    torch.backends.cudnn.allow_tf32 = True
    
    # Model initialization
    cfg = CLIPConfig.from_pretrained(
        args.target_model, 
        _attn_implementation='flash_attention_2', 
        torch_dtype=torch.bfloat16
    )
    model = CLIPModel(config=cfg)
    nparams = count_parameters(model)
    if is_main_process():
        print(f"Model Parameters: {nparams}")
    
    # Initialize processor
    processor = CLIPProcessor.from_pretrained(args.target_model, use_fast=False)
    
    # Load dataset
    if is_main_process():
        print("[info] Loading dataset ...")
    if "cc12m" in args.data_path:
        train_dataset = create_cc_dataset(
            "data/cc8m_metadata.jsonl",
            os.path.join(args.data_path, 'cc8m'),
            num_data=args.num_data
        )
        dname = 'cc12m'
    elif 'mscoco' in args.data_path:
        train_dataset = load_dataset(args.data_path)['train']
        dname = 'mscoco'
    elif 'laion' in args.data_path.lower():
        train_dataset = create_laion_dataset(args.data_path, num_data=args.num_data)
        dname = 'laion'
    
    # Calculate parameters based on configuration
    # Target: BS=1024, LR=1e-3, Warmup=10K, Samples=3B
    total_gpus = world_size
    target_global_bs = 1024
    per_device_bs = 128
    gradient_accumulation = target_global_bs // (per_device_bs * total_gpus)
    
    learning_rate = 1e-3
    warmup_steps = 10000
    total_samples = 3_000_000_000
    total_steps = total_samples // target_global_bs
    
    if is_main_process():
        print(f"[info] Target global batch size: {target_global_bs}")
        print(f"[info] Per device batch size: {per_device_bs}")
        print(f"[info] Gradient accumulation steps: {gradient_accumulation}")
        print(f"[info] Total training steps: {total_steps}")
        print(f"[info] Warmup steps: {warmup_steps}")
        print(f"[info] Learning rate: {learning_rate}")
    
    # Run name for WandB
    run_name = f"{dname}-bs{target_global_bs}-lr{learning_rate}-{args.width}-{args.tower}-{args.backbone}"
    
    # Training arguments
    training_args = TrainingArguments(
        output_dir=args.save_dir,
        run_name=run_name,
        # Training configuration
        per_device_train_batch_size=per_device_bs,
        gradient_accumulation_steps=gradient_accumulation,
        max_steps=total_steps,
        # Optimizer configuration
        learning_rate=learning_rate,
        weight_decay=0.01,
        max_grad_norm=4.0,
        # Learning rate scheduler
        optim="adamw_torch_fused",
        lr_scheduler_type="cosine",
        warmup_steps=warmup_steps,
        # Mixed precision - BF16 for A100
        fp16=False,
        bf16=True,
        bf16_full_eval=True,
        # Logging and saving
        logging_steps=100,
        logging_first_step=True,
        save_strategy="steps",
        save_steps=5000,
        save_total_limit=3,
        # Evaluation
        eval_strategy="no",
        load_best_model_at_end=False,
        # Data loading - Performance optimization
        dataloader_num_workers=8,
        dataloader_pin_memory=True,
        remove_unused_columns=False,
        dataloader_drop_last=True,
        dataloader_prefetch_factor=4,
        dataloader_persistent_workers=True,
        # Distributed training - Critical configuration
        ddp_backend='nccl',
        ddp_find_unused_parameters=False,
        ddp_bucket_cap_mb=25,
        local_rank=local_rank,
        # Other performance optimizations
        seed=args.seed,
        report_to="wandb" if is_main_process() else "none",
        gradient_checkpointing=False,
        save_on_each_node=False,
        torch_compile=True,  # Suspicious point 1
        tf32=True,
        ddp_broadcast_buffers=False,
    )
    
    collate_fn_to_use = collate_fn_laion if dname == 'laion' else collate_fn
    
    # Create Trainer
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        data_collator=lambda x: collate_fn_to_use(x, processor=processor)
    )
    
    model.train()
    trainer.train(resume_from_checkpoint=args.resume if args.resume else None)
    
    if rank == 0:
        final_path = os.path.join(args.save_dir, args.save_name)
        trainer.save_model(final_path)
        print(f"[info] Training finished. Final model saved to: {final_path}")

def main():
    args = parse_args()
    train(args)

if __name__ == "__main__":
    main()
```

</details>

<details>
<summary>Launch Script (Click to expand)</summary>

```bash
#!/bin/bash

# Find a free port safely
find_free_port() {
  for i in {1..50}; do
    port=$(( (RANDOM % 40000) + 20000 ))  # 20000-59999
    if ! ss -ltn | awk '{print $4}' | grep -q ":$port$"; then
      echo $port
      return 0
    fi
  done
  echo "No free port found after 50 tries" >&2
  return 1
}

# Environment variables
export OMP_NUM_THREADS=1
export PYTHONWARNINGS="ignore::FutureWarning"
export TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1

echo "Training with learning rate: $lr"

MASTER_ADDR=$(hostname)
FREE_PORT=$(find_free_port)
if [ $? -ne 0 ]; then
  echo "Error: Could not find a free port. Exiting."
  exit 1
fi

echo "Using rendezvous: ${MASTER_ADDR}:${FREE_PORT}"

torchrun \
  --nproc_per_node=8 \
  --rdzv_backend=c10d \
  --rdzv_endpoint=${MASTER_ADDR}:${FREE_PORT} \
  clip_hf.py \
    --num_workers 32 \
    --save_name main_t-cc12m-mup-lr${lr}.pt \
    --save_dir $SAVE_DIR \
    --data_path $DATA_PATH
```

</details>

## Observed Symptoms

### 1. Abnormally Low GPU Memory Usage
- All 8 GPUs show only ~10% memory allocation
- This is unusually low for CLIP training with batch_size=128
- May indicate data is not being properly distributed to GPUs

### 2. Severe GPU Utilization Fluctuations
- Frequent oscillation between 0-100%
- Significant idle cycles present
- GPUs don't appear to be working synchronously

### 3. Potential Bottleneck Points
- **Data Loading**: Despite 8 workers + prefetch, may still have I/O bottleneck
- **DDP Synchronization**: Gradient synchronization between GPUs may have waiting issues
- **Torch Compile**: Potential compatibility issues with DDP or Flash Attention
- **Gradient Accumulation**: Although set to 1, may have abnormal interactions with other components


## Questions Needing Help

1. **What is the root cause of this GPU utilization pattern?**
   - Is it a data loading bottleneck?
   - Is it a DDP communication issue?
   - Is it a torch.compile compatibility issue with DDP/FA2?

2. **How to diagnose the specific bottleneck?**
   - What profiling code should be added?
   - What metrics should be monitored?

3. **Suggested optimization directions?**
   - Should `torch_compile` be disabled?
   - Should we switch back to standard attention?
   - Do dataloader parameters need adjustment?

## Expected Behavior

- GPU utilization should maintain a stable 80-95% level
- Memory usage should be higher (at least 30-50%)
- All GPUs should work synchronously without obvious idle cycles

## Additional Information

- Dataset: CC12M / LAION
- Model: CLIP (training from scratch)
- Goal: Reproduce OpenAI CLIP-like training configuration

---

**Monitoring Screenshots:**

<img width="479" height="653" alt="Image" src="https://github.com/user-attachments/assets/5c9caa60-1bb7-4eac-b321-73659c72e3a9" />

)

- Top chart: GPU Memory Allocation (~10% across all GPUs)
- Bottom chart: GPU Utilization (severe fluctuation 0-100%)

## Possible Related Issues

- Interaction between `torch.compile` and DDP
- Flash Attention 2 compatibility with distributed training
- Dataloader performance in multi-GPU setup
- NCCL communication overhead

Any insights or suggestions would be greatly appreciated!

Comments:
- Is each process pinned to one GPU and seeing the model parameters on device? In each rank process run: `print(next(model.parameters()).device)` inside training startup. If CPU or `cuda:0` only for all ranks, that's a bug.

- Check DDP setup and see if you are using `DistributedSampler` and calling `sampler.set_epoch(epoch)`? .

- Run  with synthetic data (same shape, random tensors) to see if utilization stabilizes. If synthetic data saturates GPUs, it's a data/I/O issue.

- Switch off `torch.compile` and `flash_attention_2` temporarily and see the utilization. If its not stable, they’re the culprit.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.