Multiprocessing cause model's parameters all become to 0.0

username: martin_xiao
cooked: <p>Hello,</p>
<p>In order to improve the sampling efficiency I decide to use multiprocessing to do the sampling with CPU(model prediction also with cpu) and train the model with single a process by GPU.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad87a50cdc27e821c8f9f3a78f47eead165bbbd4.png" data-download-href="https://discuss.pytorch.org/uploads/default/ad87a50cdc27e821c8f9f3a78f47eead165bbbd4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad87a50cdc27e821c8f9f3a78f47eead165bbbd4_2_690x241.png" alt="image" data-base62-sha1="oL7dDcByUi00aryBc4HmbwutuaU" width="690" height="241" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad87a50cdc27e821c8f9f3a78f47eead165bbbd4_2_690x241.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad87a50cdc27e821c8f9f3a78f47eead165bbbd4_2_1035x361.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad87a50cdc27e821c8f9f3a78f47eead165bbbd4.png 2x" data-dominant-color="BED3EC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1168×409 41 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The multiple rollout workers code as followed:</p>
<pre><code class="lang-auto">def multi_sampling(_env):
    # _env.actor's and _env.critic's named_parameters() became all 0 **after training**. 
    # The parameters are correct before training

    _env.env.reset()
    agent = Agent(_env)
    sample_res = agent.sampling()
    return sample_res


env = CartPole()       # _env has my actor model and critic model
process_pool = Pool(env.n_WORKER)
for sample_index in range(env.n_SAMPLES_PER_EPISODE):
    res = process_pool.apply_async(multi_sampling, args=(env,))   
    res_dict[sample_index] = res
</code></pre>
<p><strong>After</strong> training the models(update the weights of actor and critic), the model’s weights and biases becomes 0.0 in function multi_sampling. But the weights and biases are correct before my first training.</p>
<p>If I only use CPU to do the <strong>sampling and training</strong>, the results will just perfect which I could finish the cartpole task in 10 seconds (avg 195 rewards of continuous 100 episodes)</p>
<p>My question is, how do I sampling with CPU and training with GPU in a efficient and correct way?</p>

username: martin_xiao
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/2/322db8a2a3f33cbacfc02d2cf6395a01e51c254f.png" data-download-href="https://discuss.pytorch.org/uploads/default/322db8a2a3f33cbacfc02d2cf6395a01e51c254f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/322db8a2a3f33cbacfc02d2cf6395a01e51c254f_2_690x306.png" alt="image" data-base62-sha1="79TPvDbBYxeiYj5FRYcJlEg4CgT" width="690" height="306" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/322db8a2a3f33cbacfc02d2cf6395a01e51c254f_2_690x306.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/2/322db8a2a3f33cbacfc02d2cf6395a01e51c254f_2_1035x459.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/2/322db8a2a3f33cbacfc02d2cf6395a01e51c254f.png 2x" data-dominant-color="C7D8EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1142×508 50.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>use cpu to sample and train will be fine.</p>

username: martin_xiao
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/2/e2137179b1afa981c027a2b03f96317b824f8691.png" data-download-href="https://discuss.pytorch.org/uploads/default/e2137179b1afa981c027a2b03f96317b824f8691" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2137179b1afa981c027a2b03f96317b824f8691_2_690x345.png" alt="image" data-base62-sha1="wfXz35KegxWfDe8v7dB36nYzK0h" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2137179b1afa981c027a2b03f96317b824f8691_2_690x345.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2137179b1afa981c027a2b03f96317b824f8691_2_1035x517.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/2/e2137179b1afa981c027a2b03f96317b824f8691.png 2x" data-dominant-color="C9D9ED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1175×588 55.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: martin_xiao
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hi Ptrblck, could you please help me with this problem? Or could you tell me what should I do if I want to sample on CPU and train on GPU.</p>

username: ptrblck
cooked: <p>Sorry I don’t fully understand the use case and e.g. don’t know what the <code>to()</code> operation in “Update Weights” refers to.<br>
The usual use case would be to use a <code>DataLoader</code> with multiple workers for the data loading and push the data to the GPU once it’s ready.<br>
Based on your issue I guess you might be running into multiprocessing issues e.g. as described <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html">here</a>.</p>

username: martin_xiao
cooked: <p>Thanks for the response, I will try dataloader.</p>