How to deep clone a network (module) by C++ API?

username: fulltopic
cooked: <p>How to deep clone a network (module) by C++ API?</p>
<p>I’ve read topic <a href="https://discuss.pytorch.org/t/are-there-any-recommended-methods-to-clone-a-model/483">Are there any recommended methods to clone a model?</a>. Seemed that there are off-the-shelf methods for Python applications. Then what’s the corresponding method for C++ application?  Any extra work (e.g. interface implementation) required for customized module?</p>
<p>Thank you very much.</p>

username: tom
cooked: <p>I think <code>module::clone()</code> works either on Cloneable-subclasses or AnyModule (<a href="https://pytorch.org/cppdocs/api/classtorch_1_1nn_1_1_cloneable.html#_CPPv4I0EN5torch2nn9CloneableE" rel="nofollow noopener">there is some trickyness going on that makes it difficult for <code>Module</code> itself</a></p>
<p>Best regards</p>
<p>Thomas</p>

username: fulltopic
cooked: <p>Thank you for the instruction. Following example could be a demo of Double DQN?</p>
<pre><code class="lang-auto">struct TestModule: public torch::nn::Cloneable&lt;TestModule&gt; {
	torch::nn::GRU gru;
	torch::nn::Linear fc;
	int testMember;

	TestModule(int testValue):
		gru(torch::nn::GRUOptions(128, 1024).num_layers(2).batch_first(true)),
			fc(1024,16),
			testMember(testValue) {
		register_module("gru", gru);
		register_module("fc", fc);
	}

	TestModule(const TestModule&amp; other) = default;
	TestModule&amp; operator=(TestModule&amp; other) = default;
	TestModule(TestModule&amp;&amp; other) = default;
	TestModule&amp; operator=(TestModule&amp;&amp; other) = default;

	~TestModule() = default;

	void reset() override {
		register_module("gru", gru);
		register_module("fc", fc);
	}

	torch::Tensor forward(torch::Tensor input) {
		return input;
		//do sth.
	}
};

void testCloneable() {
	std::shared_ptr&lt;TestModule&gt; net(new TestModule(27));
	torch::optim::RMSprop optimizer(net-&gt;parameters(), torch::optim::RMSpropOptions(1e-3).eps(1e-8).alpha(0.99));
	//forward and backward of net


	auto copy = net-&gt;clone();
	std::shared_ptr&lt;TestModule&gt; cpyNet = std::dynamic_pointer_cast&lt;TestModule&gt;(copy);
	torch::optim::RMSprop cpyOptimizer(cpyNet-&gt;parameters(), torch::optim::RMSpropOptions(1e-3).eps(1e-8).alpha(0.99));
	//forward and backward of cpyNet
}
</code></pre>
<p>Thank you very much.</p>

username: fulltopic
cooked: <p>I’ve tried by following function, and the nets are the same before and after updating(loss.backward + optimizer.step) of original network. Seemed it is not a deep clone?</p>
<pre><code class="lang-auto">template&lt;typename NetType&gt;
static bool compNet(std::shared_ptr&lt;NetType&gt; net0, std::shared_ptr&lt;NetType&gt; net1) {
	cout &lt;&lt; endl &lt;&lt; endl;
	cout &lt;&lt; "Compare nets " &lt;&lt; endl;

	auto params0 = net0-&gt;named_parameters(true);
	auto params1 = net1-&gt;named_parameters(true);
	for (auto ite = params0.begin(); ite != params0.end(); ite ++) {
		auto key = ite-&gt;key();
		cout &lt;&lt; "Test param " &lt;&lt; key &lt;&lt; ": ---------------------------&gt; " &lt;&lt; endl;

		Tensor v0 = ite-&gt;value();
		Tensor* v1 = params1.find(key);
		if (v1-&gt;is_same(v0)) {
			cout &lt;&lt; "Not clone, just imp pointer copied" &lt;&lt; endl;
//			return false;
		}

		if (v1 == nullptr) {
			cout &lt;&lt; "Could not find " &lt;&lt; key &lt;&lt; " in net1 " &lt;&lt; endl;
			return false;
		}

		if (v0.dim() != v1-&gt;dim()) {
			cout &lt;&lt; "Param " &lt;&lt; key &lt;&lt; " have different dim " &lt;&lt; v0.dim() &lt;&lt; " != " &lt;&lt; v1-&gt;dim() &lt;&lt; endl;
			return false;
		}
		auto numel0 = v0.numel();
		auto numel1 = v1-&gt;numel();
		if (numel0 != numel1) {
			cout &lt;&lt; "Different cell number: " &lt;&lt; numel0 &lt;&lt; " != " &lt;&lt; numel1 &lt;&lt; endl;
			return false;
		}

		auto size0 = v0.sizes();
		auto size1 = v1-&gt;sizes();
		for (int i = 0; i &lt; v0.dim(); i ++) {
			if (v0.size(i) != v1-&gt;size(i)) {
				cout &lt;&lt; "Size does match at dim " &lt;&lt; i &lt;&lt; " " &lt;&lt; v0.size(i) &lt;&lt; " != " &lt;&lt; v1-&gt;size(i) &lt;&lt; endl;
				return false;
			}
		}

		auto data0 = v0.data_ptr&lt;float&gt;();
		auto data1 = v1-&gt;data_ptr&lt;float&gt;();
		for (int i = 0; i &lt; numel0; i ++) {
			if (data0[i] != data1[i]) {
				cout &lt;&lt; "Different value of element " &lt;&lt; i &lt;&lt; ": " &lt;&lt; data0[i] &lt;&lt; " != " &lt;&lt; data1[i] &lt;&lt; endl;
				return false;
			}
		}
	}

	vector&lt;Tensor&gt; buffers0 = net0-&gt;buffers(true);
	vector&lt;Tensor&gt; buffers1 = net1-&gt;buffers(true);
	for (int i = 0; i &lt; buffers0.size(); i ++) {
		if (!buffers0[i].equal(buffers1[i])) {
			cout &lt;&lt; "Buffer at " &lt;&lt; i &lt;&lt; " does not match " &lt;&lt; endl;
			return false;
		}
	}

	auto children0 = net0-&gt;children();
	auto children1 = net1-&gt;children();
	if (children0.size() != children1.size()) {
		cout &lt;&lt; "Different size of children: " &lt;&lt; children0.size() &lt;&lt; " != " &lt;&lt; children1.size() &lt;&lt; endl;
		return false;
	}
	for (int i = 0; i &lt; children0.size(); i ++) {
		cout &lt;&lt; "Name" &lt;&lt; i &lt;&lt; ": " &lt;&lt; children0[i]-&gt;name() &lt;&lt; ", " &lt;&lt; children1[i]-&gt;name() &lt;&lt; endl;
	}


	return true;
}
</code></pre>

username: fulltopic
cooked: <p>The reasons that caused above experiment result:</p>
<ol>
<li>Default copy constructor just copy the sub-module pionters.</li>
<li>cloneable.clone() just cloned members with recursive=false</li>
</ol>

username: fatvlad
cooked: <p>I had similar situation with torch::jit::Module. <code>Module::deepcopy</code> didn’t work for me - gradients were undefined for the new module instance. However serializing-deserializing seems to work here.</p>

username: Lin_Jia
cooked: <p>At first, it seems after cloning something, you need to invoke detach:<br>
</p><aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/55266154/pytorch-preferred-way-to-copy-a-tensor" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/2593878/dkv" target="_blank" rel="nofollow noopener">
    <img alt="dkv" src="https://www.gravatar.com/avatar/d48b3fc093f704b646e75703dea29107?s=128&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/55266154/pytorch-preferred-way-to-copy-a-tensor" target="_blank" rel="nofollow noopener">Pytorch preferred way to copy a tensor</a>
</h4>

<div class="tags">
  <strong>copy, pytorch, tensor</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/2593878/dkv" target="_blank" rel="nofollow noopener">
    dkv
  </a>
  on <a href="https://stackoverflow.com/questions/55266154/pytorch-preferred-way-to-copy-a-tensor" target="_blank" rel="nofollow noopener">04:51PM - 20 Mar 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I have not tried the clone method yet, but “first serialize class to disk, then load it to a new class instantiation” guarantee a safe and clean deep clone. The disadvantage is that it is not high performance compared to a clone API call.</p>

username: FenixFromTheShadows
cooked: <p>OK if anyone is still interested, I had the tread author’s mystery solved, at least for the most part.</p>
<p>The problem lies in the implementation of the pure virtual function “reset”, which is called by the clone function. The key issue is that, it is not enough to re-register the submodules there, but it should also re-construct the submodules (or network layers; maybe I am not using the correct terminology), i.e. the “gru”, “fc” in the author’s example (I’m ignoring the testMember here as it is probably not a network parameter and the copy of this integer member, if needed, should be relatively easy), and the re-construction needs to be done before re-registering. To be more concrete, the reset function in the author’s example should be implemented something like below. (Disclaimer: I did not test this exact example but tested on my own example, but the issue is similar enough for having confidence.)</p>
<pre><code class="lang-auto">void reset() override {
    gru = torch::nn::GRUOptions(128, 1024).num_layers(2).batch_first(true);
    fc = torch::nn::Linear(1024, 16);
    register_module("gru", gru);
    register_module("fc", fc);
}
</code></pre>
<p>The reason behind this, if I am not understanding it wrong, is that if we don’t re-construct the submodules and only re-register them, the parameters are going to be registered to the same memory space as those in the old submodules, thus causing the clone being not clean.</p>
<p>This is really an annoying issue and took me quite some time to figure it out. Thank the author for providing me some inspirations on it. I hope the documentation can be clearer on this. Hope this can help whoever running into this issue later.</p>