Having problems in segmenting image when using libtorch

username: hwztable
cooked: <p>I build a U-Net for segmenting medical images in pytorch 1.3. There’s no problems in training and testing stages when using pytorch. Here is my codes for converting ‘model.pth’ to ‘model.pt’, and ‘model.pt’ will apply in libtorch for testing with C++ later.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a839ba617bb40b254d12b6c95e8584a6fd4a06b3.png" alt="微信截图_20200413012531" data-base62-sha1="o0bTBlqUOQQTH6FQugEOAcRPd9p" width="566" height="281"></p>
<p>and here is my test code for evaluate my model.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b25a76bc4fc5f4741d9f0157922d94af30d430d9.png" data-download-href="https://discuss.pytorch.org/uploads/default/b25a76bc4fc5f4741d9f0157922d94af30d430d9" title="微信截图_20200413013018"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b25a76bc4fc5f4741d9f0157922d94af30d430d9.png" alt="微信截图_20200413013018" data-base62-sha1="prMO4gnwnJ2F7wfI13u1TUsCJyp" width="690" height="481" data-dominant-color="444443"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">微信截图_20200413013018</span><span class="informations">829×579 12.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>everything goes fine in pytorch, and i can get the correct result in python code.<br>
However, when i code in C++ with using libtorch, i can’t get the same result in python code. Here is my C++ code for testing.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/f/6fcb538835dd9b3f9ba991e06f33616a9d73484b.png" data-download-href="https://discuss.pytorch.org/uploads/default/6fcb538835dd9b3f9ba991e06f33616a9d73484b" title="微信截图_20200413013654"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6fcb538835dd9b3f9ba991e06f33616a9d73484b_2_616x500.png" alt="微信截图_20200413013654" data-base62-sha1="fWYCFTAbVcap9qHu6PWBXFPNghB" width="616" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6fcb538835dd9b3f9ba991e06f33616a9d73484b_2_616x500.png, https://discuss.pytorch.org/uploads/default/original/3X/6/f/6fcb538835dd9b3f9ba991e06f33616a9d73484b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/f/6fcb538835dd9b3f9ba991e06f33616a9d73484b.png 2x" data-dominant-color="272827"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">微信截图_20200413013654</span><span class="informations">629×510 163 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I check every stage output in libtorch and pytorch. I find that in pytorch, after ‘predict = net(test_data)’, the tensor is:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e91c0499f033b75519a6c559c331538ed20a3ee4.png" data-download-href="https://discuss.pytorch.org/uploads/default/e91c0499f033b75519a6c559c331538ed20a3ee4" title="微信截图_20200413013931"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e91c0499f033b75519a6c559c331538ed20a3ee4.png" alt="微信截图_20200413013931" data-base62-sha1="xgbhhT1jsDThWcWT8O2RSpTAwWE" width="484" height="500" data-dominant-color="4A4A48"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">微信截图_20200413013931</span><span class="informations">579×597 15.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
But in C++, there’re werid values appear after ‘auto result = module.forward(inputs).toTensor();’, these values change into NaN, here is the result output:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/2/62880c05f8592b7de018da2e941fb1939f7afc42.png" data-download-href="https://discuss.pytorch.org/uploads/default/62880c05f8592b7de018da2e941fb1939f7afc42" title="微信截图_20200413014426"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/2/62880c05f8592b7de018da2e941fb1939f7afc42_2_515x499.png" alt="微信截图_20200413014426" data-base62-sha1="e3EgPcxKZdgfZjFJVI5kLUFNB3I" width="515" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/2/62880c05f8592b7de018da2e941fb1939f7afc42_2_515x499.png, https://discuss.pytorch.org/uploads/default/original/3X/6/2/62880c05f8592b7de018da2e941fb1939f7afc42.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/2/62880c05f8592b7de018da2e941fb1939f7afc42.png 2x" data-dominant-color="100F10"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">微信截图_20200413014426</span><span class="informations">581×564 76.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>so i go back to check my image input, i also find in pytorch, data input is:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2a4235c6a6138ca8b279a5dc0dcb404a65fbd62a.png" alt="image" data-base62-sha1="61PUiYK79mZHaznRBKcb19mq8oa" width="334" height="118"><br>
there is another value such as 255.0/128.0 in tensor. But in C++, there are manys of huge values:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/9/296fc33225a9f5317ea941b4387e43b876c03006.png" data-download-href="https://discuss.pytorch.org/uploads/default/296fc33225a9f5317ea941b4387e43b876c03006" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/296fc33225a9f5317ea941b4387e43b876c03006_2_678x500.png" alt="image" data-base62-sha1="5Uz1D7qWTrjXG14yr0dRMyrog2q" width="678" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/296fc33225a9f5317ea941b4387e43b876c03006_2_678x500.png, https://discuss.pytorch.org/uploads/default/original/3X/2/9/296fc33225a9f5317ea941b4387e43b876c03006.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/9/296fc33225a9f5317ea941b4387e43b876c03006.png 2x" data-dominant-color="201F20"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">805×593 184 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>therefore i get a full black image output in C++ while a nice result in python.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/3/4333ef14f5c78c14362d7a68de1a52490ec75a94.png" alt="image" data-base62-sha1="9Avg1lWRXgwh1hm91kliDVBZpSk" width="197" height="193"></p>
<p>I wonder how to do segmentation in libtorch with my case. Thanks!</p>

username: theycallmefm
cooked: <p>I had similar issues before. Turns out I was passing CPU data into loaded model’s forward function. Make sure your inputs are in CUDA.</p>
<p>Edit: Nevermind I just realized you also defined your network in CPU. However most of my nan output cases occured because I gave the wrong input type. It would be good to check the data type and which device it is in both Python and C++.</p>

username: hwztable
cooked: <p>Thanks for your attention! I defined my network in CPU which is in both Python and C++.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88b9d5f9dca0d83950f477b4d888d58dbc2a8eef.png" alt="image" data-base62-sha1="jvx5q7snU1cymEUOZFr0Gs321UX" width="592" height="140"><br>
<code>torch::jit::script::Module module = torch::jit::load(model_path,torch::kCPU);</code></p>
<p>I still get nan output. So i agree that maybe input data type is wrong:</p>
<pre><code class="lang-auto">    test_data = torch.from_numpy(test_data)
    test_data = torch.tensor(test_data,dtype=torch.float32)
</code></pre>
<pre><code class="lang-auto">	torch::Tensor tImage = torch::from_blob(src1.data, { 240,240,1 }, torch::kF32);
	tImage = tImage.permute({ 2,0,1 });
	tImage = tImage.div(255);
	tImage = tImage.unsqueeze(0);
</code></pre>
<p>Annoyingly, nan output still occures. <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=12" title=":sob:" class="emoji" alt=":sob:" loading="lazy" width="20" height="20"></p>

username: hwztable
cooked: <p>I may find where cause these problems.<br>
In pytorch, after transforming numpy to tensor, the image value doesn’t change expect int16 to float32:</p>
<pre><code class="lang-auto">    test_data = np.asarray(nda[110])        # int16
    test_data = torch.from_numpy(test_data)
    test_data = torch.tensor(test_data,dtype=torch.float32) #to float32
    test_data=torch.unsqueeze(test_data,0)
    test_data=torch.unsqueeze(test_data,1)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4c08bd3538600b01fb8b446b40d3849ce51862e4.png" alt="image" data-base62-sha1="aQD0GXmUSKQb7UkcXNYFKKxINtG" width="492" height="340"></p>
<p>But in libtorch, i implement the same progress with code:</p>
<pre><code class="lang-auto">        torch::Tensor tImage = torch::from_blob(src.data, { 240,240,1 }, torch::kFloat32);
	tImage = tImage.permute({ 2,0,1 });
	tImage = tImage.div(255);
	tImage = tImage.unsqueeze(0);
	cout &lt;&lt; "prob:" &lt;&lt; tImage.data() &lt;&lt; endl;
	vector&lt;torch::jit::IValue&gt; inputs;  //def an input

	inputs.push_back(torch::ones({ 1,1,240,240 }).toType(torch::kFloat32));
</code></pre>
<p>and i get different values:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6aa175b354a6e6b2ab145c2addea3114976d575b.png" data-download-href="https://discuss.pytorch.org/uploads/default/6aa175b354a6e6b2ab145c2addea3114976d575b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6aa175b354a6e6b2ab145c2addea3114976d575b.png" alt="image" data-base62-sha1="fdixloB0h6xIBH6wIqby3a47t5F" width="690" height="479" data-dominant-color="111111"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1330×925 45.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>i don’t exactly know whether is the key point or not. i need help.</p>
<p>Edit: those C++ output vaules are the result of ‘torch.div(255)’. i think it is not the key point.</p>