Copying weights and biases

jchmack: <p>I am trying to convert this tutorial to libtorch:<br>
<a href="https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html</a></p>
<p>My problem is with this line(in python):<br>
target_net.load_state_dict(policy_net.state_dict())</p>
<p>Because there isn’t a libtorch equivalent I used this workaround:</p>
<pre><code class="lang-auto">void loadstatedict(torch::nn::Module&amp; model,torch::nn::Module&amp; target_model) 
{
	torch::autograd::GradMode::set_enabled(false);  // make parameters copying possible
	auto new_params = target_model.named_parameters(); // implement this
	auto params = model.named_parameters(true /*recurse*/);
	auto buffers = model.named_buffers(true /*recurse*/);

	for (auto&amp; val : new_params)
	{
		auto name = val.key();
		auto* t = params.find(name);
		if (t != nullptr)
			t-&gt;copy_(val.value());
		else
		{
			t = buffers.find(name);
			if (t != nullptr)
				t-&gt;copy_(val.value());
		}
	}
	
	torch::autograd::GradMode::set_enabled(true); //Set back
}
</code></pre>
<p>Everything seems to work but after long runs I usually get these from cuda during runtime:</p>
<pre><code class="lang-auto">C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\ScatterGatherKernel.cu:145: block: [0,0,0], thread: [0,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.

CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
</code></pre>
<p>I have tried:<br>
CUDA_LAUNCH_BLOCKING=1<br>
torch::cuda::synchronize() before the copy<br>
Moving both models to the cpu before copying.<br>
Running only on CPU(It doesn’t SEEM to happen in when running only on the cpu.)</p>
<p>It feels like some problem syncing data between cpu/gpu.  Everything runs fine for long runs until I start to do the copy(loadstatedict) and even then it seems like it works 99% of the time.  If I disable the copy it never crashes.</p>
<p>using Libtorch 1.13.1+cu117</p>
<p>Thanks for any info.</p>

ptrblck: <p>The error points to an invalid index in:</p>
<pre><code class="lang-python">C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\ScatterGatherKernel.cu:145: block: [0,0,0], thread: [0,0,0] Assertion `idx_dim &gt;= 0 &amp;&amp; idx_dim &lt; index_size &amp;&amp; "index out of bounds"` failed.
</code></pre>
<p>Running the code with <code>CUDA_LAUNCH_BLOCKING=1</code> should show you the line of code causing this issue. Once isolated you could print the indices and check why they contain wrong values.</p>

jchmack: <p>At crash time:<br>
next_target_q_values:<br>
0.01 *<br>
-1.0695<br>
-5.1743<br>
-2.5427<br>
3.3140<br>
4.4763<br>
0.1929<br>
3.3601<br>
3.3892<br>
1.8363<br>
6.2410<br>
[ CUDAFloatType{10} ]</p>
<p>maximum:<br>
10.1474<br>
[ CUDAFloatType{} ]</p>
<p>Crash on this line:<br>
torch::Tensor next_q_value = next_target_q_values.gather(0, maximum.to(torch::kInt64).unsqueeze(0)).squeeze(0);</p>
<p>So maximum evaluates to 10 which is outside the range</p>
<p>This is what I have:</p>
<pre><code class="lang-auto">torch::Tensor Learn(torch::Tensor state, torch::Tensor action, torch::Tensor reward, torch::Tensor next_state, torch::Tensor done)
	{

		torch::Device mTorchDevice = torch::Device(mDeviceType);
		bool Display = true;

		MyQnet-&gt;zero_grad();
		TargetQnet-&gt;zero_grad();

		torch::Tensor q_values = MyQnet-&gt;forward(state);																							
		torch::Tensor next_target_q_values = TargetQnet-&gt;forward(next_state);
		torch::Tensor next_q_values = MyQnet-&gt;forward(next_state);
		torch::Tensor q_value = q_values.gather(0, action.unsqueeze(0)).squeeze(0);
		torch::Tensor maximum = std::get&lt;0(next_q_values.max(0));																		
		torch::Tensor next_q_value = next_target_q_values.gather(0, maximum.to(torch::kInt64).unsqueeze(0)).squeeze(0);
		torch::Tensor expected_q_value = reward + FutureRewardRate * next_q_value * (1 - done.to(torch::kFloat32));				
		torch::Tensor loss = torch::mse_loss(q_value, expected_q_value);

		MyQnet_optimizer.zero_grad();
		loss.backward();
		MyQnet_optimizer.step();

		static int Copy_Count = 0;
		Copy_Count++;
		if (Copy_Count == 1000)
		{
			Copy_Count = 0;
			loadstatedict(*MyQnet, *TargetQnet);
		}

		return q_values;
	}
</code></pre>
<p>I think i should be taking the maximum of next_target_q_values instead of next_q_values<br>
or that max should be an argmax</p>
<p>I’m trying to follow this and I don’t understand why he does it that way:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/navneet-nmk/Pytorch-RL-CPP/blob/master/Trainer.cpp">
  <header class="source">

      <a href="https://github.com/navneet-nmk/Pytorch-RL-CPP/blob/master/Trainer.cpp" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/navneet-nmk/Pytorch-RL-CPP/blob/master/Trainer.cpp" target="_blank" rel="noopener nofollow ugc">navneet-nmk/Pytorch-RL-CPP/blob/master/Trainer.cpp</a></h4>


      <pre><code class="lang-cpp">//
// Created by Navneet Madhu Kumar on 2019-07-10.
//
#include "Trainer.h"
#include "dqn.h"
#include "ExperienceReplay.h"
#include "/Users/navneetmadhukumar/Downloads/Arcade-Learning-Environment-master/src/ale_interface.hpp"
#include &lt;math.h&gt;
#include &lt;chrono&gt;


Trainer::Trainer(int64_t input_channels, int64_t num_actions, int64_t capacity):
        buffer(capacity),
        network(input_channels, num_actions),
        target_network(input_channels, num_actions),
        dqn_optimizer(
            network.parameters(), torch::optim::AdamOptions(0.0001).beta1(0.5)){}

    torch::Tensor Trainer::compute_td_loss(int64_t batch_size, float gamma){
        std::vector&lt;std::tuple&lt;torch::Tensor, torch::Tensor, torch::Tensor, torch::Tensor, torch::Tensor&gt;&gt; batch =
</code></pre>



  This file has been truncated. <a href="https://github.com/navneet-nmk/Pytorch-RL-CPP/blob/master/Trainer.cpp" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


ptrblck: <p>Good to hear you were able to isolate the invalid indexing!<br>
I’m not familiar enough with the code so don’t know which value should be used and if <code>next_q_values</code> is wrong or just needs clipping.</p>

jchmack: <p>For anyone who might be searching about this tutorial in the future:</p>
<p>torch::Tensor next_q_value = next_target_q_values.gather(0, maximum.to(torch::kInt64).unsqueeze(0)).squeeze(0);</p>
<p>should be:</p>
<p>torch::Tensor next_q_value = next_target_q_values.gather(0, next_q_values.argmax(0).unsqueeze(0)).squeeze(0);</p>
<p>Thank you for the tips on how to debug data on gpu+cuda.  Knowing where to look is everything.</p>