Getting RuntimeError when running the parameter server tutorial

username: akrisilias21
cooked: <p>Hi there,</p>
<p>I’m trying to run <a href="https://pytorch.org/tutorials/intermediate/rpc_param_server_tutorial.html" rel="nofollow noopener">this</a> tutorial locally for one parameter server and two workers.</p>
<p>The problem is I’m getting the below error:<br>
Traceback (most recent call last):<br>
File “/usr/lib/python3.6/multiprocessing/process.py”, line 258, in _bootstrap<br>
self.run()<br>
File “/usr/lib/python3.6/multiprocessing/process.py”, line 93, in run<br>
self._target(*self._args, **self._kwargs)<br>
File “rpc_parameter_server.py”, line 228, in run_worker<br>
run_training_loop(rank, num_gpus, train_loader, test_loader)<br>
File “rpc_parameter_server.py”, line 187, in run_training_loop<br>
dist_autograd.backward(cid, [loss])<br>
RuntimeError: Error on Node 0: one of the variables needed for gradient computation has been modified by an inplace operation: [CPUFloatType [32, 1, 3, 3]] is at version 5; expected version 4 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
<p>Here’s my torch version if needed:<br>
pip3 freeze | grep torch<br>
torch==1.5.1+cpu<br>
torchtext==0.6.0<br>
torchvision==0.6.1+cpu</p>
<p>Thanks in advance for any advice!</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/rvarm1">@rvarm1</a>, I wonder if we need a lock in <code>ParameterServer.forward</code>, otherwise if the execution of <code>forward</code> got sliced into multiple pieces, interleaving execution from different RPC threads could mess up the autograd graph state?</p>

username: ollien
cooked: <p>I’m getting this, too, but interestingly only when I have &gt; 1 worker node.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "rpc_parameter_server.py", line 224, in run_worker
    run_training_loop(rank, num_gpus, train_loader, test_loader)
  File "rpc_parameter_server.py", line 183, in run_training_loop
    dist_autograd.backward(cid, [loss])
RuntimeError: Error on Node 0: one of the variables needed for gradient computation has been modified by an inplace operation: [CUDAFloatType [128, 10]], which is output 0 of TBackward, is at version 3; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>

username: ollien
cooked: <p>I did some looking into this. Adding a lock on <code>ParameterServer.forward</code> has no effect.</p>
<pre><code class="lang-auto">    def forward(self, inp):
        # forward_lock defined globally
        with forward_lock:
            inp = inp.to(self.input_device)
            out = self.model(inp)
            # This output is forwarded over RPC, which as of 1.5.0 only accepts CPU tensors.
            # Tensors must be moved in and out of GPU memory due to this.
            out = out.to("cpu")
            return out
</code></pre>

username: rvarm1
cooked: <p>Thanks for looking into this! I confirm that I am able to repro the issue, working on root causing it now.</p>

username: aonrdbit
cooked: <p>Hello , I have met the same problem, may I ask how this problem was finally solved?<a class="mention" href="/u/rvarm1">@rvarm1</a><br>
Sincerely thank you if you can give me any suggestions</p>