The result of libtorch is different to pytorch

username: Mr.Crab
cooked: <p>I want to deploy the model in C++, but i found the result i got in libtorch is different to pytorch. I can got 90% accuracy in pytorch but 20% in C++. How could lead to the difference?<br>
I just give the model the same data input size. Here is my code for input:</p>
<blockquote>
<p>torch::Tensor tensor_image = torch::from_blob(img.data, {  300, 800, 3 }, torch::kByte);<br>
tensor_image = tensor_image.permute({ 2,0,1});<br>
tensor_image = tensor_image.toType(torch::kFloat);<br>
tensor_image = tensor_image.to(device);</p>
<p>torch::Tensor tensor_asc = torch::from_blob(img.data, { 300, 800}, torch::kByte);<br>
tensor_asc = tensor_asc.toType(torch::kFloat);<br>
tensor_asc = tensor_asc.to(device);</p>
<p>cout &lt;&lt; “The data has been transformed to tensor\n”;</p>
<p>tensor_asc = torch::unsqueeze(tensor_asc, 0);<br>
torch::Tensor inputs = torch::cat({ tensor_image, tensor_asc }, 0);<br>
inputs = torch::stack({ inputs, inputs }, 0);<br>
inputs = torch::unsqueeze(inputs, 0);</p>
</blockquote>
<p>I need to concat two img. and i did the same in python for pytorch. The inference accuracy is 90%.</p>

username: ptrblck
cooked: <p>How did you load the image in C++?<br>
If you’ve used OpenCV, note that the default color channel format is BGR, while <code>torchvision</code> uses <code>PIL</code> by default, which is using RGB.<br>
If that might be the case, then you would have to convert the color channels in your C++ implementation.</p>

username: Mr.Crab
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>，  in notice the img channel has been changed  in the opencv and i  have add the code <code>cvtColor(img, img, COLOR_RGB2BGR);</code>  to make the channel right. And the results is also bad in the C++. did the model of “.pt” transformed from “.pth” would inference the accuracy?</p>

username: ptrblck
cooked: <p>No, the file suffix shouldn’t change anything.<br>
Did you make sure to apply exactly the same preprocessing pipeline in C++ as was used in Python (normalization etc.)?</p>

username: Mr.Crab
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks, I would check my code carefully.  And what you mean is that the results of libtorch should almost be same as pytorch?</p>

username: ptrblck
cooked: <p>If you are using the same preprocessing and load the same scripted or traced model, then the results should be the same up the the limited floating point precision.</p>

username: Mr.Crab
cooked: <p>I mistake the pointer in C++, the real accuray is 79.8%, which is same as 82.05% when i uese the code:</p>
<pre><code class="lang-auto">output = scripted_model(torch.from_numpy(input_data).cuda())
</code></pre>
<p>to do the infrence. However, the those accuracy is lower than 96.7%  when is ues the pytorch code to do the inference.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Mr.Crab" data-post="7" data-topic="78757">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ba9def/48.png" class="avatar"> Mr.Crab:</div>
<blockquote>
<p>However, the those accuracy is lower than 96.7% when is ues the pytorch code to do the inference.</p>
</blockquote>
</aside>
<p>What is the difference between the posted code snippet and the other code you are using to achieve the 96% accuracy?</p>

username: Mr.Crab
cooked: <p>The whole code is as follows:</p>
<pre><code class="lang-auto">import cv2
import time
import torch
import numpy as np
from ptsemseg.models import get_model
from ptsemseg.utils import convert_state_dict

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = get_model({'arch': 'hardnet'}, 3).to(device)
the_model = torch.load("./weights/hardnet_2D_best_model.pkl")
state = convert_state_dict(the_model['model_state'])
model.load_state_dict(state)
model = model.eval()
input_shape = [1, 2, 4, 300, 800]
input_data = torch.randn(input_shape, device='cuda')
scripted_model = torch.jit.trace(model, input_data)
start_time = time.perf_counter()
///The input data was dealt as the same way.
output = scripted_model(torch.from_numpy(input_data).cuda())
elapsed_time = time.perf_counter() - start_time
scripted_model.save("model.pt")

</code></pre>
<p>I calculate the output, and the accruracy is only 82.05%.</p>
<p>And the 96.7% is obtained from the pytorch inference with another code.</p>

username: ptrblck
cooked: <p>In that case I would recommend to compare the posted code to the other code, which yields the higher accuracy, and try to narrow down potential discrepancies between these codes.</p>

username: Mr.Crab
cooked: <p>Thanks for you advice! i would try as you said.</p>

username: Mr.Crab
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>， i check the code which is the same as what in pytorch about the data pre-procession. And i found there are two ways to transform pytorch model to torch script, which is trace and annotation. And i use trace to do that. The official said that the trace would limit the control flow such as “if else”, which several exist in my code. i wonder is real the control flow was limited that the trace-torch script can’t get the same accuracy as pytorch did?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Mr.Crab" data-post="12" data-topic="78757">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ba9def/48.png" class="avatar"> Mr.Crab:</div>
<blockquote>
<p>i wonder is real the control flow was limited that the trace-torch script can’t get the same accuracy as pytorch did?</p>
</blockquote>
</aside>
<p>That might be the case, as the executed forward pass with be traced and other paths through the model won’t be used.<br>
Therefore you should use a scripted model, which will collect the conditions.</p>

username: Mr.Crab
cooked: <p>Did the annotation model would deal with the proble and allow the controal flow works well?</p>

username: ptrblck
cooked: <p>Yes, scripting will record the control flow (<code>torch.jit.script</code>).</p>

username: Mr.Crab
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thankyou for your patient reply. I have ideas to realize now!</p>

username: Shisho_Sama
cooked: <p>Make sure you divide by 255 to normalize your input before you run any processing.<br>
that is make sure you have done sth like this:</p>
<pre><code class="lang-cpp">tensor_image = tensor_image.toType(torch::kFloat).div(255);
</code></pre>
<p>This is most likeley the case ! tell us if this was it!</p>