Forking recommendations for MPS

dfalbel: <p>I know that forking is not supported when using CUDA eg:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/pytorch/pytorch/wiki/Autograd-and-Fork">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/pytorch/pytorch/wiki/Autograd-and-Fork" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/0b3a95f2c79d4cf53ddf1949a177da21364fbc15a6a3931716d8c406c35d4635/pytorch/pytorch" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/pytorch/pytorch/wiki/Autograd-and-Fork" target="_blank" rel="noopener nofollow ugc">Autograd and Fork · pytorch/pytorch Wiki</a></h3>

  <p>Tensors and Dynamic neural networks in Python with strong GPU acceleration - Autograd and Fork · pytorch/pytorch Wiki</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>But there are some constrained scenarios where forking is possible, eg:</p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/69839">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/69839" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/69839" target="_blank" rel="noopener nofollow ugc">Using Autograd before and after forking should be possible</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2021-12-13" data-time="14:37:47" data-timezone="UTC">02:37PM - 13 Dec 21 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2022-02-12" data-time="01:53:02" data-timezone="UTC">01:53AM - 12 Feb 22 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/ChristopherBrix" target="_blank" rel="noopener nofollow ugc">
          <img alt="ChristopherBrix" src="https://avatars.githubusercontent.com/u/5114411?v=4" class="onebox-avatar-inline" width="20" height="20">
          ChristopherBrix
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: multiprocessing
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: autograd
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">### 🚀 The feature, motivation and pitch

I'm aware that [autograd cannot be us<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">ed in combination with forking, because it uses thread pools](https://github.com/pytorch/pytorch/wiki/Autograd-and-Fork).
However, it would be nice if autograd could be used both before and after forking, as long as they don't share any information.

I've written a minimal example:
```
import multiprocessing as mp
import torch


def train():
    print("Training started")
    x = torch.Tensor(0)
    x.requires_grad = True
    x.sum().backward()
    print("Training ended")


if __name__ == "__main__":
    print("Entered main")
    train()
    worker = mp.Process(target=train)
    worker.start()
```

Expected output:
```
Entered main
Training started
Training ended
Training started
Training ended
```

Real output:
```
Entered main
Training started
Training ended
Training started
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "mwe.py", line 9, in train
    x.sum().backward()
  File "/home/christopher/.local/share/virtualenvs/src-i_X_I5Sj/lib/python3.8/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/christopher/.local/share/virtualenvs/src-i_X_I5Sj/lib/python3.8/site-packages/torch/autograd/__init__.py", line 98, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Unable to handle autograd's threading in combination with fork-based multiprocessing. See https://github.com/pytorch/pytorch/wiki/Autograd-and-Fork
```

The above example did work in PyTorch 1.4 but stopped working in 1.5.

### Alternatives

If this feature won't be added, I recommend to extend the documentation and to explicitly state that autograd will fail even if there is no sharing of information across threads.

### Additional context

This unsupported behavior seems to be a bit confusing, raising at least two questions on StackOverflow (second one by me):
 - https://stackoverflow.com/questions/63081486/pytorch-multiprocessing-error-with-hogwild
 - https://stackoverflow.com/questions/70308084/using-pytorch-in-multiple-independent-forked-threads

The error message is raised in https://github.com/pytorch/pytorch/blob/065018d8120567976b5f9296e56c3ece5db51e12/torch/csrc/autograd/engine.cpp#L1047-L1053

cc @VitalyFedyunin @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I wonder if there are some recommendations for using fork with MPS enabled builds of pytorch. Or when using MPS tensors.</p>
<p>FWIW I have tried forking with 3 simple different scenarios:</p>
<ol>
<li>
<p>Creating an MPS tensor:</p>
<pre><code class="lang-auto">def mps_tensor ():
  torch.randn(100, 100, device = "mps")```
</code></pre>
</li>
<li>
<p>Creating a tensor on the CPU</p>
<pre><code class="lang-auto">def cpu_tensor ():
  torch.randn(100, 100, device = "cpu")
</code></pre>
</li>
<li>
<p>Cpu but calling backward:</p>
<pre><code class="lang-auto">def cpu_but_backward():
    x = torch.randn(100,100)
    y = torch.randn(100,100, requires_grad=True)
    loss = torch.mm(x, y).sum()
    loss.backward()
</code></pre>
</li>
</ol>
<p>I then tried forking the process and running those functions with:</p>
<pre><code class="lang-auto">import multiprocessing
from multiprocessing import Process
multiprocessing.set_start_method("fork")
p = Process(target=mps_tensor)
p.start()
</code></pre>
<p>The 1 and 3 functions failed with:</p>
<pre><code class="lang-auto">The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.
objc[60703]: +[NSPlaceholderMutableString initialize] may have been in progress in another thread when fork() was called.
objc[60703]: +[NSPlaceholderMutableString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
</code></pre>
<p>while 2 works normally.</p>