Some confusion about torch.multiprocessing.spawn in pytorch

zbh2047: <p>I really get confused when I use the function torch.multiprocessing.spawn. Consider the following code:</p>
<pre><code class="lang-auto">import torch
import torch.multiprocessing as mp


x = [1, 2]

def f(id, a):
    print(x)
    print(a)

if __name__ == '__main__':
    x.append(3)
    mp.spawn(f, nprocs=2, args=(x, ))
</code></pre>
<p>For any process the <strong>main</strong> function spwans, it outputs the following:</p>
<pre><code class="lang-auto">[1, 2]
[1, 2, 3]
</code></pre>
<p>I have the following questions:<br>
(1) Why is the first line of output [1, 2]? I think x is a global variable, and fork new process will share the memory in linux, which follows this page: <a>https://stackoverflow.com/questions/5983159/python-multiprocessing-arguments-deep-copy</a><br>
(2) Are the parameters in spawn deep copied to the new processes? Or just pass a reference?</p>
<p>Thank you very much!</p>

kazem: <p>I have the exact same issue with torch.multiprocessing.spawn (mp.spawn) used for distributed parallel training.</p>
<p>Since I have a large dataset of csv files which i convert to a shared multiprocessing numpy array object to avoid memory leak outside of my main. but mp.spwan It makes multiple copies of it anyways.</p>

rvarm1: <p>It looks like python’s <code>multiprocessing</code> module also copies the data if we use a spawn start_method:</p>
<pre><code class="lang-auto">import multiprocessing as mp
x = [1, 2]
def foo(a):
    print(x)
    print(a)

if __name__ == '__main__':
   mp.set_start_method("spawn")
    x.append(3)
    p = mp.Process(target=foo, args=(x,))
    p.start()
    p.join()
</code></pre>
<p>To answer your question, there is a deepcopy, though shared memory will be used for <code>Tensor</code> data.</p>
<p>From the pytorch multiprocessing “best practices” page (<a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/multiprocessing.html</a>):</p>
<pre><code class="lang-auto">We recommend using python:multiprocessing.Queue for passing all kinds of PyTorch objects between processes. It is possible to e.g. inherit the tensors and storages already in shared memory, when using the fork start method, however it is very bug prone and should be used with care, and only by advanced users. Queues, even though they’re sometimes a less elegant solution, will work properly in all cases.

</code></pre>
<p>You could thus use the <code>fork</code> start method with pytorch multiprocessing to avoid the copy, though as the docs mention this is not supported.</p>

stevenwjy: <blockquote>
<p>To answer your question, there is a deepcopy, though shared memory will be used for <code>Tensor</code> data.</p>
</blockquote>
<p>Is this documented somewhere? Couldn’t find it on [1] and [2], but observed this behavior by writing some simple script</p>
<p>[1] <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" class="inline-onebox" rel="noopener nofollow ugc">Multiprocessing best practices — PyTorch 2.1 documentation</a><br>
[2] <a href="https://pytorch.org/docs/stable/multiprocessing.html" class="inline-onebox" rel="noopener nofollow ugc">Multiprocessing package - torch.multiprocessing — PyTorch 2.1 documentation</a></p>