Reproducibility with Multiple GPUs not working

Tanay_Arora: <p>Hello,</p>
<p>I am currently trying to implement a relatively large distributed environment as deterministic as part of research, however, at my very first step, I am failing.</p>
<p>When running the same code twice with a single torch.rand(… device = “cuda”) call, it fails to reproduce.</p>
<p>Does anyone have any suggestions?</p>
<p>Notes:</p>
<p>The environment variable CUBLAS_WORKSPACE_CONFIG=:4096:8 is set.<br>
The dynamo configs have been set through debugging previous parts of the project.<br>
The code and environment is below. I am running on a server with 4 2080 tis with updated drivers.</p>
<p>_______________________________________________<br>
Environment:<br>
numpy 1.26.4<br>
torch 2.5.0.dev20240703+cu124 {nightly}<br>
cuda 12.4<br>
python 3.12.3</p>
<p>_______________________________________________</p>
<p>main.py:</p>
<pre><code class="lang-auto">import torch
import torch.multiprocessing as mp
import torch._dynamo as dynamo
import torch.distributed as dist

import os
import sys

import matplotlib
matplotlib.use('Agg') 

def setup_distribute(rank, world_size): 
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'
    dist.init_process_group("nccl", rank = rank, world_size = world_size)

def cleanup_distribute():
    dist.destroy_process_group()

def dist_wrapper(rank, world_size, func, name: str):
    setup_distribute(rank, world_size)
    torch.cuda.set_device(rank)
    try: 
        func(rank, world_size, name)
    finally:
        cleanup_distribute()

def set_dynamo_cfg():
    dynamo.config.capture_scalar_outputs = True
    dynamo.config.cache_size_limit = 256
    dynamo.config.guard_nn_modules = True
    dynamo.config.suppress_errors = True

def set_non_deterministic():
    torch.backends.cudnn.benchmark = True

def set_deterministic():
    torch.manual_seed(42)
    import random
    import numpy as np
    random.seed(42)
    np.random.seed(42)
    torch.use_deterministic_algorithms(True)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

def main(func, name: str):
    world_size = torch.cuda.device_count()
    mp.spawn(dist_wrapper, args=(world_size, func, name), nprocs=world_size, join=True)

if __name__ == "__main__":

    if len(sys.argv) != 2:
        raise ValueError("Must pass exactly one name. Usage should be python main.py $NAME")

    name = sys.argv[1]

    set_dynamo_cfg()
    set_deterministic()

    import tmp
    main(tmp.main, name)
</code></pre>
<p>tmp.py:</p>
<pre><code class="lang-auto">import torch
from torch.nn.parallel import DistributedDataParallel as DDP

import gc

def main(rank, world_size, name: str):

    x = torch.rand((5, 4), device = "cuda", dtype = torch.float32)

    print(f"[RANK {rank}] ---- ", x.detach().cpu())
</code></pre>

Tanay_Arora: <p>How do I delete this post? I figured it out.</p>

ptrblck: <p>Could you post the solution to your issue as I guess other users would be interested in learning more about the issue in the posted code snippet?</p>

Tanay_Arora: <p>Upon looking at a little documentation, I found that the above method only set the seed for a single process (the launching process). So, I simply added the set_deterministic to the distributed wrapper.</p>