Why same model in CUDA and CPU got different result?

NickKao: <p>I tested a model with different results in the case of CUDA and CPU.<br>
The condition of the CPU can produce results normally.<br>
But in the case of CUDA, I have to convert the output to the CPU before I can execute it, but the result of the execution is incorrect.</p>
<pre><code class="lang-auto">		torch::Tensor score;
		if (isCUDA) {
			torch::Tensor new_out_tensor = out_tensor.to(torch::kCPU).detach();
			score = new_out_tensor.squeeze(0);// [288, 384, 2]
		}
		else {
			score = out_tensor.squeeze(0).detach();// [288, 384, 2]
		}
</code></pre>
<p>If there is no conversion to the CPU, a memory exception will occur.</p>
<p>Can I do something wrong there?</p>

NickKao: <p>CPU result:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/a/3acd2f6a77abee496c1daa41e11f7891dd7b2b1e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3acd2f6a77abee496c1daa41e11f7891dd7b2b1e" title="cpu"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3acd2f6a77abee496c1daa41e11f7891dd7b2b1e_2_617x499.jpeg" alt="cpu" data-base62-sha1="8obiDsJEwYHxWrsgmVfPV4FQMOi" width="617" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3acd2f6a77abee496c1daa41e11f7891dd7b2b1e_2_617x499.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/3/a/3acd2f6a77abee496c1daa41e11f7891dd7b2b1e.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/a/3acd2f6a77abee496c1daa41e11f7891dd7b2b1e.jpeg 2x" data-dominant-color="8E92AC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cpu</span><span class="informations">765×619 43.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>CUDA result:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/f/af6d4aac44ddbdce9f4bbc98f3a87e24fc925a8a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/af6d4aac44ddbdce9f4bbc98f3a87e24fc925a8a" title="cuda"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af6d4aac44ddbdce9f4bbc98f3a87e24fc925a8a_2_618x500.jpeg" alt="cuda" data-base62-sha1="p1TIfrWXfcXxBLRvcRKBqVnOrGO" width="618" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af6d4aac44ddbdce9f4bbc98f3a87e24fc925a8a_2_618x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/a/f/af6d4aac44ddbdce9f4bbc98f3a87e24fc925a8a.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/f/af6d4aac44ddbdce9f4bbc98f3a87e24fc925a8a.jpeg 2x" data-dominant-color="9090AE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cuda</span><span class="informations">761×615 44 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

albanD: <p>Hi,</p>
<p>Why do you detach() when you compute your scores? That would prevent gradients from flowing if you use this during training.</p>

NickKao: <p>Haha!<br>
Actually, I don’t know why I want to use it. It’s just from the Internet.<br>
Because I am not using it for training, I just use it to predict, so the result of the implementation is no problem.<br>
The part with detach() is that the CPU can execute normally without problems.</p>

albanD: <p>If you just use this for evaluation, you should use the <code>NoGradGuard</code> (not sure about the exact name, but something along those lines) to completely disable the autograd (that will make you code faster and use less memory) !</p>
<p>I guess there is something wrong in the way you send the model on the gpu. Are you sure that the weights / inputs to the network are properly sent to the gpu?</p>

NickKao: <p>This is load model code:</p>
<pre><code class="lang-auto">void CRAFT::loadModel(string&amp; model_path,bool&amp; isRuntimeIn) {
	isRuntime = isRuntimeIn;
	//at::init_num_threads();
	if (isCUDA) {
		module = torch::jit::load(model_path, torch::kCUDA);
	}
	else {
		module = torch::jit::load(model_path);
	}

	assert(module != nullptr);
	std::cout &lt;&lt; "ok\n";
}
</code></pre>
<p>As you can see, I switched the CUDA to the CPU in the same program, so all the method and process is the same, I don’t know why the result is different?</p>

NickKao: <p>Added <code>NoGradGuard</code> then same CUDA result is different.</p>
<pre><code class="lang-auto">torch::NoGradGuard no_grad_guard;
</code></pre>
<p>My forward code:</p>
<pre><code class="lang-auto">	if (isCUDA) {
		output = module.forward({ tensor_image.to(torch::kCUDA)});
	}
	else {
		output = module.forward({ tensor_image });
	}
</code></pre>

albanD: <p>You should be able to print to stdout the values of the different tensors (both cpu and gpu), can you print the inputs, weights and outputs to see where the difference appears?</p>

Nick_Kao: <p>謝謝。Thanks.<br>
I will back next Thus.<br>
Then continues to discuss…</p>

NickKao: <p>I try to write output to file.<br>
Store on my google drive.<br>
link to:<br>
<a href="https://drive.google.com/drive/folders/1vK3ixC792Kkv-OOD0ckX1yF_pzoDprJ1?usp=sharing" class="onebox" target="_blank" rel="nofollow noopener">https://drive.google.com/drive/folders/1vK3ixC792Kkv-OOD0ckX1yF_pzoDprJ1?usp=sharing</a><br>
GPU and CPU got different result. Really…</p>

albanD: <p>As I said above, you should inspect when the discrepancy appears. It is when you load the weights? Or on the inputs or when you apply the forward pass?</p>

NickKao: <p>Ok, I will try it out.<br>
But I want to put my model and code on github, if you can, please try it out (please pay attention to the operation instructions, choose 1 for CUDA, 2 for CPU).<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/kaonick/libtorch-gpu" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/8186823?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/kaonick/libtorch-gpu" target="_blank" rel="nofollow noopener">kaonick/libtorch-gpu</a></h3>

<p>https://discuss.pytorch.org/t/why-same-model-in-cuda-and-cpu-got-different-result/56241/10 - kaonick/libtorch-gpu</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
You should know it, it seems very strange.</p>
<p>Because the model is not small, please note that if you use the GPU, the GPU RAM should not be too small, otherwise it will be destroyed.</p>

laomaup: <p>Hello, the classifier trained by Resnet50 works well in python. The same test data has poor classification accuracy in C++（libtorch） calls. I don’t know why. Have you met it? thank you！</p>

NickKao: <p>Haha . not yet.<br>
May be later…<br>
Hope libtorch get better…</p>

ptrblck: <p>Did you make sure to use the same preprocessing etc.?<br>
I would recommend to check the inputs first and get the same values in the Python API and libtorch and then try to narrow down the discrepancy.</p>

NickKao: <p>I have upload my code and model here:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/kaonick/libtorch-gpu" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/8186823?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/kaonick/libtorch-gpu" target="_blank" rel="nofollow noopener">kaonick/libtorch-gpu</a></h3>

<p>https://discuss.pytorch.org/t/why-same-model-in-cuda-and-cpu-got-different-result/56241/10 - kaonick/libtorch-gpu</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>that is in same process in one code just switch gpu and cpu.</p>

martinr: <p>You are not the first one to have such problem. I have a similar one <a href="https://discuss.pytorch.org/t/gpu-inference-goes-way-off-after-first-iteration-possibly-tensor-transpose-async-issue/57319">here</a> and there is another unanswered one <a href="https://discuss.pytorch.org/t/getting-different-results-when-running-inference-using-gpu-versus-cpu/43215">here</a>.</p>
<p>I suggest you try to locate the source of divergence by yourself first, it makes it easier to help you. I had a Python code, not C++, but I’ll share it here so you get the idea on how to locate the problem.</p>
<ol>
<li>Save off the intermediate variables on CPU and GPU inference:<br>
<code>torch.save(variable, "/path/to/varfile")</code>
</li>
<li>then afterwards load both for analysis:</li>
</ol>
<pre><code class="lang-auto">cpuvar = torch.load("/path/to/varfile_cpu", map_location="cpu")
gpuvar = torch.load("/path/to/varfile_gpu", map_location="cpu")
</code></pre>
<ol start="3">
<li>compare:</li>
</ol>
<pre><code class="lang-auto">close = torch.isclose(cpuvar, gpuvar, rtol=1e-04, atol=1e-04)
print("SIMILAR", close[close==True].shape)
print("FAR", close[close==False].shape)
</code></pre>
<p>Perfect case is where CPU and GPU will have similar results for the same input. Compare all variables until you will find the divergence.</p>

NickKao: <p>Thanks.<br>
I will try it.<br>
But It’s very strange, I can use python pytorch on the GPU (cuda 10) to perform normally.</p>

laomaup: <p>Thank you, I will try it first.</p>

FantasyJXF: <p>Why CRAFT behaves so strange, I didn’t met this problem with other models, Have you figured it out now?</p>