After torch::load model and predict, then got NaN

sudri: <p>When I was training and validating the model, the output was all normal. After training, I called <code>torch::save()</code> to save the model to a <code>.pt</code> file, and then called <code>torch::load()</code> to load the model from the file to make predictions. At this time, <strong>the predicted value becomes NaN</strong>. I have checked the predicted network structure and network data type(Double), input data and its data type(Double), they are all correct. I have been troubled by this problem for a few days, please help me.</p>
<p><em><strong>Network</strong></em></p>
<pre data-code-wrap="cpp"><code class="lang-cpp">struct NetImpl : torch::nn::Module {
    NetImpl(int in_feature, int out_feature)
        : fc1(in_feature, 10),
        fc2(10, 100),
        fc3(100, 500),
        fc4(500, 250),
        fc5(250, 125),
        fc6(125, 70),
        fc7(70, out_feature)
    {
        register_module("fc1", fc1);
        register_module("fc2", fc2);
        register_module("fc3", fc3);
        register_module("fc4", fc4);
        register_module("fc5", fc5);
        register_module("fc6", fc6);
        register_module("fc7", fc7);
    }

    torch::Tensor forward(torch::Tensor x) {
        x = torch::leaky_relu(fc1-&gt;forward(x));
        x = torch::leaky_relu(fc2-&gt;forward(x));
        x = torch::leaky_relu(fc3-&gt;forward(x));
        x = torch::leaky_relu(fc4-&gt;forward(x));
        x = torch::leaky_relu(fc5-&gt;forward(x));
        x = torch::leaky_relu(fc6-&gt;forward(x));
        x = fc7-&gt;forward(x);
        return x;
    }
    torch::nn::Linear fc1, fc2, fc3, fc4, fc5, fc6, fc7;
};
TORCH_MODULE(Net);
</code></pre>
<p><strong>Save and load then got output as below</strong></p>
<pre data-code-wrap="cpp"><code class="lang-cpp">torch::save(model, model_path);
torch::load(model, model_path);
std::cout &lt;&lt; model &lt;&lt; std::endl;
</code></pre>
<blockquote>
<p>NetImpl(<br>
(fc1): torch::nn::Linear(in_features=5, out_features=10, bias=true)<br>
(fc2): torch::nn::Linear(in_features=10, out_features=100, bias=true)<br>
(fc3): torch::nn::Linear(in_features=100, out_features=500, bias=true)<br>
(fc4): torch::nn::Linear(in_features=500, out_features=250, bias=true)<br>
(fc5): torch::nn::Linear(in_features=250, out_features=125, bias=true)<br>
(fc6): torch::nn::Linear(in_features=125, out_features=70, bias=true)<br>
(fc7): torch::nn::Linear(in_features=70, out_features=2, bias=true)<br>
)</p>
</blockquote>
<p><strong>Predict Method</strong></p>
<pre data-code-wrap="cpp"><code class="lang-cpp">template &lt;typename DataLoader&gt;
void predict(Net&amp; model,
    DataLoader&amp; data_loader,
    size_t dataset_size) {
    //torch::NoGradGuard no_grad;
    model-&gt;eval();
    model-&gt;zero_grad();
    auto loss_val = 0.0;
    for (auto&amp; batch : *data_loader) {
        auto data = batch.data, target = batch.target;
        auto output = model-&gt;forward(data);
        std::cout &lt;&lt; output &lt;&lt; std::endl;
        loss_val += torch::mse_loss(output, target).template item&lt;float&gt;();
    }
    loss_val /= dataset_size;
    std::printf("\nTest set: Average loss: %.8f\n", loss_val);
}
</code></pre>
<p><strong>Result</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/d/bdef6f11ae924509bc4687527168b7192d388615.png" data-download-href="https://discuss.pytorch.org/uploads/default/bdef6f11ae924509bc4687527168b7192d388615" title="20210930101752"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/d/bdef6f11ae924509bc4687527168b7192d388615_2_690x331.png" alt="20210930101752" data-base62-sha1="r6fe33wvZPrQ5OFu7KQWkMbotQp" width="690" height="331" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/d/bdef6f11ae924509bc4687527168b7192d388615_2_690x331.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/d/bdef6f11ae924509bc4687527168b7192d388615_2_1035x496.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/d/bdef6f11ae924509bc4687527168b7192d388615_2_1380x662.png 2x" data-dominant-color="8E8E8D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">20210930101752</span><span class="informations">1429×687 147 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Model File</strong><br>
If necessary, I will upload it to Google Drive.</p>
<p><strong>Environment</strong><br>
libtorch stable 1.9.1<br>
libtorch preview(nightly)<br>
Microsoft Visual Studio 2019<br>
Windows 10</p>
<p>Any suggestions and help are greatly appreciated!</p>
<hr>
<p><strong>UPDATE</strong><br>
When the tensor performs the first few forward operations, the output value produces several NaNs.</p>
<pre data-code-wrap="cpp"><code class="lang-cpp"> x = torch::leaky_relu(fc2-&gt;forward(x));
std::cout &lt;&lt; x &lt;&lt; std::endl;
</code></pre>
<p>If tensor continue to forward, all subsequent outputs will be NaN. I think this may be the cause of the problem, but how should it be solved?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/2/12b6547258a8058700ddba4a05e1646c50d9aa30.png" data-download-href="https://discuss.pytorch.org/uploads/default/12b6547258a8058700ddba4a05e1646c50d9aa30" title="20210930130037"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/2/12b6547258a8058700ddba4a05e1646c50d9aa30_2_690x396.png" alt="20210930130037" data-base62-sha1="2Fxex53GsFxPzanc1ralFzrXqLK" width="690" height="396" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/2/12b6547258a8058700ddba4a05e1646c50d9aa30_2_690x396.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/2/12b6547258a8058700ddba4a05e1646c50d9aa30_2_1035x594.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/2/12b6547258a8058700ddba4a05e1646c50d9aa30.png 2x" data-dominant-color="27191C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">20210930130037</span><span class="informations">1200×690 226 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you check the output of <code>fc1</code> and its values?<br>
The printed values of <code>x = torch::leaky_relu(fc2-&gt;forward(x));</code> have a huge range so it would be interesting to see what ranges the input to the model has (min. and max. in particular) as well as the parameters of <code>fc1</code>.</p>

sudri: <p>Hi, Thanks for help.</p>
<p>I made a small change in order to observe the change in value. The code and output results are shown below.</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">std::cout &lt;&lt; x &lt;&lt; std::endl;  // print normalized input data

x = fc1-&gt;forward(x);
std::cout &lt;&lt; x &lt;&lt; std::endl;

x = torch::leaky_relu(x);
std::cout &lt;&lt; x &lt;&lt; std::endl;

x = fc2-&gt;forward(x);
std::cout &lt;&lt; x &lt;&lt; std::endl;
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ead1f2026a846d04b6d0196e1eb9ea4c7bd1dc10.png" data-download-href="https://discuss.pytorch.org/uploads/default/ead1f2026a846d04b6d0196e1eb9ea4c7bd1dc10" title="20210930172440"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ead1f2026a846d04b6d0196e1eb9ea4c7bd1dc10.png" alt="20210930172440" data-base62-sha1="xvjx6ota9UJMfPm6nRFJbygLmG4" width="528" height="500" data-dominant-color="2C2D2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">20210930172440</span><span class="informations">1058×1001 69 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I built the same network on PyTorch, but his output was normal, with no large or small values. It is worth mentioning that in LibTorch, I initialized the network to <strong>Double type</strong> by using code below, maybe this is a problem?</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">model-&gt;to(device, torch::kDouble);
</code></pre>
<p>Looking forward to your reply, Thanks!</p>

ptrblck: <p>Thanks the the update! No, <code>double</code> shouldn’t be a problem.<br>
So it seems that the <code>fc1</code> layer already creates these large values. Could you check the <code>weight</code> and <code>bias</code> of this layer and post it here, please?</p>

sudri: <p>Thanks for your suggestion, I checked the weight and bias of fc1. As shown below, none of them are 0, but their range of values is indeed very large. For this situation, I have some confusion and don’t know what to do next.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d85b327b5776360432f47d30efaa28f9e18c1575.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d85b327b5776360432f47d30efaa28f9e18c1575" title="Snipaste_2021-09-30_20-03-26"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d85b327b5776360432f47d30efaa28f9e18c1575_2_335x500.jpeg" alt="Snipaste_2021-09-30_20-03-26" data-base62-sha1="uRYwphhofvMM1mrbrMWqTiLdkcR" width="335" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d85b327b5776360432f47d30efaa28f9e18c1575_2_335x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d85b327b5776360432f47d30efaa28f9e18c1575_2_502x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/8/d85b327b5776360432f47d30efaa28f9e18c1575_2_670x1000.jpeg 2x" data-dominant-color="1C1F24"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Snipaste_2021-09-30_20-03-26</span><span class="informations">1375×2050 308 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I save and load the model by using the following code.</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">Net model = Net(in_feature, out_feature);
// some train and test code
torch::save(model, model_path);  // model_path is like "xxx.pt"

// initialize new model
Net new_model = Net(in_feature, out_feature);
torch::load(new_model, model_path);
new_model-&gt;to(device, torch::kDouble);
// and predict...
</code></pre>
<p>Thank you very much and look forward to your reply!</p>

ptrblck: <p>So apparently something goes wrong during the saving or loading.<br>
Is it possible to load the model and/or parameters in Python? If so, are you seeing different parameters?</p>
<p>Could you also just run this check in a single code snippet:</p>
<ul>
<li>create the model</li>
<li>check parameter stats</li>
<li>save the model</li>
<li>load the model right afterwards</li>
<li>check parameters again</li>
</ul>

sudri: <p>After trying again and again, I finally found the problem. Let me briefly describe the prerequisites and solutions.</p>
<p>Prerequisites</p>
<ol>
<li>The model parameter type is Double</li>
<li>The input parameter is also of type Double</li>
<li>Therefore, the model saved by torch::save is also of Double type</li>
</ol>
<p>Solutions<br>
After LibTorch saves the model, before loading the model, the network needs to be initialized first. <strong>If you want to read an offline network file of type Double, you must first initialize the network to type Double, then load the model from file, which is very, very important.</strong> The sample code is shown below.</p>
<pre><code class="lang-cpp">// Initialize the network
Net model = Net(in_feature, out_feature);
model-&gt;to(device, torch::kDouble);
// some code to train and validate and more...
torch::save(model, "test.pt");

// Load the model
Net new_model = Net(in_feature, out_feature);
new_model-&gt;to(device, torch::kDouble); // very very important before load
torch::load(new_model, "test.pt");
</code></pre>
<p><strong>In summary, whether it is to save the model or load the model, the data type must be consistent throughout the process.</strong></p>
<p>Thank you very much for all your suggestions and help!<br>
Have a good day!</p>

ptrblck: <p>Thanks a lot for the detailed explanation!<br>
I would consider this a nasty bug, as it’s causing silently wrong answers so thanks again for the debugging!<br>
I’ll create an issue to track and fix this.</p>