Windows & WSL2: zeroed CUDA tensors in spawned processes

taras-janea: <p>Dear PyTorch Team,</p>
<p>Please help to find the root cause of the issue described below.</p>
<p><strong>Issue description</strong></p>
<p>Executing the following script using the PyTorch 2.7, CUDA 12.6:</p>
<pre><code class="lang-auto">import torch
import torch.multiprocessing as mp

def producer(shared_queue, producer_done, consumer_done):
    for i in range(1, 10):
        t = torch.tensor([i, i*2], device="cuda:0")
        print(f"#{i}: Tensor to send: {t}")
        shared_queue.put(t)
    producer_done.set()
    consumer_done.wait()

def consumer(shared_queue, producer_done, consumer_done):
    producer_done.wait()
    for i in range(1, 10):
        t = shared_queue.get()
        print(f"#{i}: Tensor received: {t}")

    consumer_done.set()

if __name__ == "__main__":

    mp.set_start_method("spawn", True)

    shared_queue = torch.multiprocessing.Queue()
    consumer_done = mp.Event()
    producer_done = mp.Event()

    producer_process = mp.Process(target=producer, args=(shared_queue, producer_done, consumer_done))
    consumer_process = mp.Process(target=consumer, args=(shared_queue, producer_done, consumer_done))
    
    producer_process.start()
    consumer_process.start()   
    producer_process.join()
    consumer_process.join()
</code></pre>
<p>constantly results in the same output on both Windows and WSL2 (Ubuntu 24.04):</p>
<pre><code class="lang-auto">#1: Tensor to send: tensor([1, 2], device='cuda:0')
#2: Tensor to send: tensor([0, 0], device='cuda:0')
#3: Tensor to send: tensor([3, 6], device='cuda:0')
#4: Tensor to send: tensor([4, 8], device='cuda:0')
#5: Tensor to send: tensor([ 5, 10], device='cuda:0')
#6: Tensor to send: tensor([ 6, 12], device='cuda:0')
#7: Tensor to send: tensor([ 7, 14], device='cuda:0')
#8: Tensor to send: tensor([ 8, 16], device='cuda:0')
#9: Tensor to send: tensor([ 9, 18], device='cuda:0')
#1: Tensor received: tensor([0, 0], device='cuda:0')
#2: Tensor received: tensor([0, 0], device='cuda:0')
#3: Tensor received: tensor([3, 6], device='cuda:0')
#4: Tensor received: tensor([4, 8], device='cuda:0')
#5: Tensor received: tensor([ 5, 10], device='cuda:0')
#6: Tensor received: tensor([ 6, 12], device='cuda:0')
#7: Tensor received: tensor([ 7, 14], device='cuda:0')
#8: Tensor received: tensor([ 8, 16], device='cuda:0')
#9: Tensor received: tensor([ 9, 18], device='cuda:0')
</code></pre>
<p>The first two received tensors are zeroed, but not only received - also the second one that’s sent is zeroed as well. No issues on native Ubuntu setup.</p>
<p><strong>Some findings</strong></p>
<ul>
<li>
<p>Documentation says that <a href="https://docs.pytorch.org/docs/stable/notes/windows.html#cuda-ipc-operations" class="inline-onebox" rel="noopener nofollow ugc">Windows FAQ — PyTorch 2.7 documentation</a> sharing CUDA tensors is not supported;</p>
</li>
<li>
<p>There is a <a href="https://discuss.pytorch.org/t/use-torch-multiprocessing-queue-with-cuda-tensor/21648/2">comment</a> from <a href="https://discuss.pytorch.org/u/peterjc123">peterjc123</a> that sharing CUDA tensors is not supported on Windows, but that was back 2018.</p>
</li>
<li>
<p>Unit tests for multiprocessing are disabled on Windows: <a href="https://github.com/pytorch/pytorch/blob/56b03df6ac5b4185a2b7b92f253565500a5b51ca/test/test_multiprocessing.py#L42" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test/test_multiprocessing.py at 56b03df6ac5b4185a2b7b92f253565500a5b51ca · pytorch/pytorch · GitHub</a></p>
</li>
<li>
<p>Open topics that refer this issue:<br>
<a href="https://discuss.pytorch.org/t/torch-multiprocessing-on-cuda-turns-tensors-to-zeros/186455" class="inline-onebox">Torch.multiprocessing on CUDA turns tensors to zeros</a><br>
<a href="https://discuss.pytorch.org/t/pytorch-multiprocessing-with-cuda-sets-tensors-to-0/179117/11" class="inline-onebox">PyTorch multiprocessing with CUDA sets tensors to 0 - #11 by Skirlax</a><br>
<a href="https://discuss.pytorch.org/t/issue-with-cuda-tensors-shared-between-processes/176925" class="inline-onebox">Issue with CUDA tensors shared between processes</a><br>
<a href="https://discuss.pytorch.org/t/use-torch-multiprocessing-queue-with-cuda-tensor/21648" class="inline-onebox">Use torch.multiprocessing.queue with cuda tensor</a><br>
<a href="https://discuss.pytorch.org/t/pytorch-multiprocessing-not-sending-cuda-tensors-properly-on-windows/192611" class="inline-onebox">PyTorch multiprocessing not sending CUDA tensors properly on Windows</a></p>
</li>
<li>
<p>Open similar topics:<br>
<a href="https://discuss.pytorch.org/t/best-practice-to-share-cuda-tensors-across-multiprocess/45778" class="inline-onebox">Best practice to share CUDA tensors across multiprocess</a><br>
<a href="https://discuss.pytorch.org/t/cuda-tensors-on-multiprocessing-queue/28626" class="inline-onebox">CUDA tensors on multiprocessing queue</a><br>
<a href="https://discuss.pytorch.org/t/dataloader-multiprocessing-with-dataset-returning-a-cuda-tensor/151022" class="inline-onebox">DataLoader multiprocessing with Dataset returning a CUDA tensor</a><br>
<a href="https://discuss.pytorch.org/t/sharing-cuda-tensor/148764" class="inline-onebox">Sharing CUDA tensor</a><br>
<a href="https://discuss.pytorch.org/t/dataloader-is-returning-cuda-tensors-always-bad-in-distributed-training/145233" class="inline-onebox">DataLoader: is returning CUDA tensors always bad in distributed training?</a><br>
<a href="https://discuss.pytorch.org/t/synchronization-of-cuda-operations-between-multiprocess-processes/193585" class="inline-onebox">Synchronization of CUDA operations between `multiprocess` processes</a><br>
<a href="https://discuss.pytorch.org/t/allocate-cuda-tensor-in-subprocess/545/5" class="inline-onebox">Allocate cuda tensor in subprocess - #5 by florin</a><br>
<a href="https://discuss.pytorch.org/t/multiprocessing-cuda-memory/18951/12" class="inline-onebox">Multiprocessing CUDA memory - #12 by PatrickNercessian</a><br>
<a href="https://discuss.pytorch.org/t/using-cuda-ipc-memory-handles-in-pytorch/17548/2" class="inline-onebox">Using CUDA IPC memory handles in pytorch - #2 by colesbury</a><br>
<a href="https://discuss.pytorch.org/t/a-call-to-torch-cuda-is-available-makes-an-unrelated-multi-processing-computation-crash/4075" class="inline-onebox">A call to torch.cuda.is_available makes an unrelated multi-processing computation crash?</a></p>
</li>
<li>
<p>Relevant GitHub issues:<br>
<a href="https://github.com/pytorch/examples/issues/1015" class="inline-onebox" rel="noopener nofollow ugc">torch.multiprocessing subprocess receives tensor with zeros rather than actual data · Issue #1015 · pytorch/examples · GitHub</a><br>
<a href="https://github.com/pytorch/pytorch/issues/84994" class="inline-onebox" rel="noopener nofollow ugc">Cuda tensor is zero when passed through multiprocessing queue · Issue #84994 · pytorch/pytorch · GitHub</a><br>
<a href="https://github.com/pytorch/pytorch/issues/56771" class="inline-onebox" rel="noopener nofollow ugc">Problems with initial communication between GPUs · Issue #56771 · pytorch/pytorch · GitHub</a><br>
<a href="https://github.com/pytorch/pytorch/issues/109094" class="inline-onebox" rel="noopener nofollow ugc">Parameters of cuda module zero out when used in multiprocessing · Issue #109094 · pytorch/pytorch · GitHub</a><br>
<a href="https://github.com/pytorch/pytorch/issues/134273" class="inline-onebox" rel="noopener nofollow ugc">Data corruption when reading data as CUDA tensor from a different process · Issue #134273 · pytorch/pytorch · GitHub</a><br>
<a href="https://github.com/pytorch/pytorch/issues/112340" class="inline-onebox" rel="noopener nofollow ugc">Unexpected behaviour with shared modules in multiprocessing on WSL2 · Issue #112340 · pytorch/pytorch · GitHub</a></p>
</li>
</ul>
<p>Thank you in advance for your assistance</p>