Libtorch returns NaN on Arm64

Gorgo: <p>Hi!<br>
I’m using Pytorch’s torch C++ frontend on a Nvidia Orin NX (Arm64). The Jetpack is 5.1.2 and I’m using torch installed in <code>/home/nvidia/.local/lib/python3.8/site-packages/torch/</code> from the python wheel provided by Nvidia. (torch-2.1.0a0+41361538.nv23.06-cp38-cp38-linux_aarch64.whl)</p>
<p>Basically, after a few jit inferences on an IValue vector inputs,<br>
<code>at::Tensor action_mean_tensor = policy_.forward(inputs).toTensor();</code> starts returning NaN even with small values.</p>
<p>I tried to create a minimal application to replicate the issue:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://pastebin.com/kaXsY6K5">
  <header class="source">
      <img src="https://pastebin.com/favicon.ico" class="site-icon" width="16" height="16">

      <a href="https://pastebin.com/kaXsY6K5" target="_blank" rel="noopener nofollow ugc">Pastebin</a>
  </header>

  <article class="onebox-body">
    <img width="" height="" src="https://pastebin.com/i/facebook.png" class="thumbnail">

<h3><a href="https://pastebin.com/kaXsY6K5" target="_blank" rel="noopener nofollow ugc">#include #include #include  - Pastebin.com</a></h3>

  <p>Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>In my original bigger ROS2 node, I used to generate new observations every time, doing an <code>env.step(obs)</code> but I got easily NaNs during a few iterations.<br>
On my OrinNX, in the example above, I always get NaNs at the third iteration.</p>
<p>For some reason, removing <code>#include &lt;torch/torch.h&gt;</code> seems to remove the NaNs. Unfortunately, the same workaround on the bigger ROS2 node guarantees only good 20 iterations (instead of 2) but it eventually restarts with NaN and good values alternatively.</p>
<p>So far I found only 3 <strong>workaround</strong> that are giving always good results:</p>
<ol>
<li>
<p>appending <code>clone()</code> to<br>
<code> torch::Tensor obs_tensor = torch::from_blob(obs.data(), {1, 28}, torch::kFloat32);</code></p>
</li>
<li>
<p>Compiling with -O1, -O2 or -O3. Somehow g++ is optimizing the memory and fixing/hiding the issue.</p>
</li>
<li>
<p>Printing to stdout the obs before passing them to from_blob. It’s funny but it stops getting NaN if I show the obs values <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>
</li>
</ol>
<p>I hope somebody can give an insight into this strange issue.</p>
<p>While I was trying to debug, I found similar errors:</p><aside class="quote" data-post="3" data-topic="80129">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/klaus_voltmer/48/23890_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/different-inference-results-when-running-on-arm64-and-x86-64/80129/3">Different inference results when running on arm64 and x86_64</a> <a class="badge-category__wrapper " href="/c/c/11"><span data-category-id="11" style="--category-badge-color: #9EB83B; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="Topics related to the C++ Frontend, C++ API or C++ Extensions"><span class="badge-category__name">C++</span></span></a>
  </div>
  <blockquote>
    sure: 


load the model: 
module = torch::jit::load(fileName); 


process: 
torch::Tensor result; 
std::vector<a>torch::jit::IValue</a> inputs; 
at::Tensor tensor = torch::from_blob(data.data(), {1, 1, 11, 144}); 
inputs.push_back(tensor); 
torch::autograd::AutoGradMode guard( false ); 
result = module.forward(inputs).toTensor(); 


i have written unit tests and integrations tests that show that the results are equal to python. but once i run this on the iphone the results are different.
  </blockquote>
</aside>

<aside class="onebox discoursetopic" data-onebox-src="https://forums.developer.nvidia.com/t/nvidia-torch-cuda-produces-only-nan-on-cpu/232798/7">
  <header class="source">
      <img src="https://global.discourse-cdn.com/nvidia/optimized/2X/0/0372ccc95874f71d7fbff64bbbff6f8c69dd850b_2_32x32.png" class="site-icon" width="" height="">

      <a href="https://forums.developer.nvidia.com/t/nvidia-torch-cuda-produces-only-nan-on-cpu/232798/7" target="_blank" rel="noopener nofollow ugc" title="05:01PM - 10 November 2022">NVIDIA Developer Forums – 10 Nov 22</a>
  </header>

  <article class="onebox-body">
    <img src="https://global.discourse-cdn.com/nvidia/original/3X/e/4/e41d27491ee72562cf68c340597ca95f587879bd.png" class="thumbnail" width="" height="">

<div class="title-wrapper">
  <h3><a href="https://forums.developer.nvidia.com/t/nvidia-torch-cuda-produces-only-nan-on-cpu/232798/7" target="_blank" rel="noopener nofollow ugc">Nvidia torch + cuda produces only NAN on CPU</a></h3>
  <div class="topic-category">
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #76B900;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">Autonomous Machines</span>
        </span>
      </span>
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #76B900;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">Jetson &amp;amp; Embedded Systems</span>
        </span>
      </span>
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #76B900;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">Jetson AGX Orin</span>
        </span>
      </span>
    <div class="topic-header-extra">
      <div class="list-tags">
        <div class="discourse-tags">
          <svg class="fa d-icon d-icon-tag svg-icon svg-string"><use href="#tag"></use></svg>
            <span class="discourse-tag simple">cuda</span>
            <span class="discourse-tag simple">nvbugs</span>
            <span class="discourse-tag simple">pytorch</span>
        </div>
      </div>
    </div>
  </div>
</div>

  <p>Sure! The model is generated using:  from torchvision.models import mobilenet_v2 import torch  net = mobilenet_v2() torch.save(net,"net.pth") script_module = torch.jit.script(net) torch.jit.save(script_module,"mobilenetv2.pt")  And I’m running it...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<aside class="onebox discoursetopic" data-onebox-src="https://forums.developer.nvidia.com/t/pytorch-error-on-the-jetson-platform/221046/5">
  <header class="source">
      <img src="https://global.discourse-cdn.com/nvidia/optimized/2X/0/0372ccc95874f71d7fbff64bbbff6f8c69dd850b_2_32x32.png" class="site-icon" width="" height="">

      <a href="https://forums.developer.nvidia.com/t/pytorch-error-on-the-jetson-platform/221046/5" target="_blank" rel="noopener nofollow ugc" title="07:43AM - 28 July 2022">NVIDIA Developer Forums – 28 Jul 22</a>
  </header>

  <article class="onebox-body">
    <img src="https://global.discourse-cdn.com/nvidia/original/3X/e/4/e41d27491ee72562cf68c340597ca95f587879bd.png" class="thumbnail" width="" height="">

<div class="title-wrapper">
  <h3><a href="https://forums.developer.nvidia.com/t/pytorch-error-on-the-jetson-platform/221046/5" target="_blank" rel="noopener nofollow ugc">Pytorch Error on the Jetson Platform</a></h3>
  <div class="topic-category">
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #76B900;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">Autonomous Machines</span>
        </span>
      </span>
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #76B900;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">Jetson &amp;amp; Embedded Systems</span>
        </span>
      </span>
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #76B900;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">Jetson TX2</span>
        </span>
      </span>
    <div class="topic-header-extra">
      <div class="list-tags">
        <div class="discourse-tags">
          <svg class="fa d-icon d-icon-tag svg-icon svg-string"><use href="#tag"></use></svg>
            <span class="discourse-tag simple">pytorch</span>
        </div>
      </div>
    </div>
  </div>
</div>

  <p>Hi,  Confirmed that we can reproduce this issue with the l4t-pytorch:r34.1.1-pth1.12-py3 container.  Would you mind sharing the working version so we can compare?  Thanks.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>and the recent memory issues related to the C++ frontend:</p>
<aside class="quote quote-modified" data-post="1" data-topic="201026">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/byrdman32/48/69298_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/libtorch-crashes-docker-when-included-in-header-file/201026">Libtorch crashes docker when included in header file</a> <a class="badge-category__wrapper " href="/c/c/11"><span data-category-id="11" style="--category-badge-color: #9EB83B; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="Topics related to the C++ Frontend, C++ API or C++ Extensions"><span class="badge-category__name">C++</span></span></a>
  </div>
  <blockquote>
    I’ve been working on integrating libtorch C++ into my project for a few weeks and while it works fine when I include the 
#include &lt;torch/script.h&gt;
#include &lt;torch/torch.h&gt;

headers in a cpp source file it works fine. But when I include them in a cpp header file it results in my docker environment crashing and the following error log. 
Log Screenshot 
 <a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/0/7002210217c1e1592a78447e5eb980c56f8e6049.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/7002210217c1e1592a78447e5eb980c56f8e6049" title="image">[image]</a> 
Log File 
-- The CXX compiler identification is GNU 10.5.0
-- The CUDA compiler identification is NVIDIA 12.2.140
-- Detecting CXX compi…
  </blockquote>
</aside>


ptrblck: <p>Your workarounds suggest that <code>obs.data()</code> is invalid and goes out of scope. <code>torch::from_blob</code> will use the underlying memory directly and it’s your responsibility to guarantee the data is valid and won’t be freed.</p>

Gorgo: <p>I agree with you <a class="mention" href="/u/ptrblck">@ptrblck</a> but I don’t see why obs is not valid in my pastebin above.</p>
<p>This is the output from my x86_64 laptop with Ubuntu 22.04 and (Py)Torch 2.3.0 cxx11 ABI installed standalone and compiled with colcon:</p>
<blockquote>
<p>inputs: Columns 1 to 10 0.0000  0.0000  0.0000  1.0000  0.0000  0.0000  0.0000  1.0000  0.0000  0.0000</p>
<p>Columns 11 to 20 0.0000  1.0000  0.0000  0.0000  0.0000  0.0000  0.0000 -3.9500  2.3000  1.6500</p>
<p>Columns 21 to 28 0.6500  0.9000  1.6500  0.0000  0.0000  0.0000  0.0000  0.0000<br>
[ CPUFloatType{1,28} ]<br>
stds:  0.3192<br>
0.1498<br>
0.1619<br>
0.4095<br>
[ CPUFloatType{4} ]<br>
random_sample:  1.8895<br>
0.4808<br>
-0.1694<br>
0.9750<br>
[ CPUFloatType{4} ]<br>
action_mean_tensor:  0.8503 -0.7974  0.4450 -0.8844<br>
[ CPUFloatType{1,4} ]<br>
Action: 1.606528 -0.383409 -0.075384 -0.862261<br>
inputs: Columns 1 to 10-0.0076  0.0016  0.0101  0.8693  0.1434 -0.4731 -0.1096  0.9891  0.0984  0.4821</p>
<p>Columns 11 to 20-0.0337  0.8755 -0.3785  0.0787  0.5042 -3.4581 -25.0000 -6.6201 -3.9424  2.2984</p>
<p>Columns 21 to 28 1.6399  0.6576  0.8984  1.6399  1.4953 -0.1383 -1.2073 -0.4413<br>
[ CPUFloatType{1,28} ]<br>
stds:  0.3192<br>
0.1498<br>
0.1619<br>
0.4095<br>
[ CPUFloatType{4} ]<br>
random_sample: -0.7742<br>
-2.2891<br>
-0.8089<br>
0.3901<br>
[ CPUFloatType{4} ]<br>
action_mean_tensor:  1.7387 -0.1351  0.0408 -0.4169<br>
[ CPUFloatType{1,4} ]<br>
Action: -1.346089 0.309363 -0.033036 -0.162666<br>
inputs: Columns 1 to 10-0.0216  0.0067  0.0304  0.7917  0.4566 -0.4058 -0.3988  0.8895  0.2228  0.4627</p>
<p>Columns 11 to 20-0.0146  0.8864 -0.7031  0.2570  1.0170 -6.9673  1.4978 -15.0000 -3.9284  2.2933</p>
<p>Columns 21 to 28 1.6196  0.6716  0.8933  1.6196  2.1924 -0.2787  0.0599 -1.1601<br>
[ CPUFloatType{1,28} ]<br>
stds:  0.3192<br>
0.1498<br>
0.1619<br>
0.4095<br>
[ CPUFloatType{4} ]<br>
random_sample:  1.5993<br>
-1.9500<br>
-0.9119<br>
-1.6646<br>
[ CPUFloatType{4} ]<br>
action_mean_tensor:  1.8872  0.0295 -0.7745  0.0846<br>
[ CPUFloatType{1,4} ]<br>
Action: 3.018301 -0.057587 0.706246 -0.140806</p>
</blockquote>
<p>This is the output from the Orin NX that uses (Py)Torch 2.1.0 wheel (Ubuntu 20.04) and compiled with colcon:</p>
<blockquote>
<p>PyTorch version: 2.1.0<br>
inputs: Columns 1 to 10 0.0000  0.0000  0.0000  1.0000  0.0000  0.0000  0.0000  1.0000  0.0000  0.0000</p>
<p>Columns 11 to 20 0.0000  1.0000  0.0000  0.0000  0.0000  0.0000  0.0000 -3.9500  2.3000  1.6500</p>
<p>Columns 21 to 28 0.6500  0.9000  1.6500  0.0000  0.0000  0.0000  0.0000  0.0000<br>
[ CPUFloatType{1,28} ]<br>
stds:  0.3192<br>
0.1498<br>
0.1619<br>
0.4095<br>
[ CPUFloatType{4} ]<br>
random_sample: -0.7111<br>
0.0454<br>
-0.2371<br>
-0.6546<br>
[ CPUFloatType{4} ]<br>
action_mean_tensor:  0.8503 -0.7974  0.4450 -0.8844<br>
[ CPUFloatType{1,4} ]<br>
Action: -0.604600 -0.036172 -0.105509 0.578925<br>
inputs: Columns 1 to 10-0.0076  0.0016  0.0101  0.8693  0.1434 -0.4731 -0.1096  0.9891  0.0984  0.4821</p>
<p>Columns 11 to 20-0.0337  0.8755 -0.3785  0.0787  0.5042 -3.4581 -25.0000 -6.6201 -3.9424  2.2984</p>
<p>Columns 21 to 28 1.6399  0.6576  0.8984  1.6399  1.4953 -0.1383 -1.2073 -0.4413<br>
[ CPUFloatType{1,28} ]<br>
stds:  0.3192<br>
0.1498<br>
0.1619<br>
0.4095<br>
[ CPUFloatType{4} ]<br>
random_sample:  2.1585<br>
-1.5461<br>
-0.8058<br>
0.0587<br>
[ CPUFloatType{4} ]<br>
action_mean_tensor:  1.7387 -0.1351  0.0408 -0.4169<br>
[ CPUFloatType{1,4} ]<br>
Action: 3.752915 0.208950 -0.032909 -0.024473<br>
inputs: Columns 1 to 10-0.0216  0.0067  0.0304  0.7917  0.4566 -0.4058 -0.3988  0.8895  0.2228  0.4627</p>
<p>Columns 11 to 20-0.0146  0.8864 -0.7031  0.2570  1.0170 -6.9673  1.4978 -15.0000 -3.9284  2.2933</p>
<p>Columns 21 to 28 1.6196  0.6716  0.8933  1.6196  2.1924 -0.2787  0.0599 -1.1601<br>
[ CPUFloatType{1,28} ]<br>
stds:  0.3192<br>
0.1498<br>
0.1619<br>
0.4095<br>
[ CPUFloatType{4} ]<br>
random_sample:  0.5744<br>
-0.5015<br>
-0.2751<br>
-1.0906<br>
[ CPUFloatType{4} ]<br>
action_mean_tensor: -nan -nan -nan -nan<br>
[ CPUFloatType{1,4} ]<br>
Action: -nan -nan -nan -nan</p>
</blockquote>