Crashes when using torch.max() in torch==1.6.0

username: Stephane_Bersier
cooked: <p>I’m using Linux, Ubuntu 18.04 with python3.6.9 and have a problem with the torch.max() function when using torch==1.6.0.</p>
<p>With torch==1.5.0. It works correctly:</p>
<blockquote>
<p>steph@steph-desktop:~$ python3<br>
Python 3.6.9 (default, Jul 17 2020, 12:50:27)<br>
[GCC 8.4.0] on linux<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
torch.<strong>version</strong><br>
‘1.5.0’<br>
a=torch.randn(1,3,5)<br>
a.max(1)<br>
torch.return_types.max(<br>
values=tensor([[ 1.4852,  1.0638, -0.7425, -0.4036,  0.8044]]),<br>
indices=tensor([[1, 2, 0, 0, 2]]))<br>
quit()</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<p>Then I pip3 uninstall torch==1.5.0 and pip3 install torch==1.6.0 --user<br>
Then I do the same thing as above:</p>
<blockquote>
<p>steph@steph-desktop:~$ python3<br>
Python 3.6.9 (default, Jul 17 2020, 12:50:27)<br>
[GCC 8.4.0] on linux<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
torch.<strong>version</strong><br>
‘1.6.0’<br>
a=torch.randn(1,3,5)<br>
a.max(1)<br>
Illegal instruction (core dumped)</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<p>Then I pip3 unstall torch==1.6.0 and pip3 install torch==1.5.0 --user again.</p>
<blockquote>
<p>steph@steph-desktop:~$ python3<br>
Python 3.6.9 (default, Jul 17 2020, 12:50:27)<br>
[GCC 8.4.0] on linux<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
torch.<strong>version</strong><br>
‘1.5.0’<br>
a=torch.randn(1,3,5)<br>
a.max(1)<br>
torch.return_types.max(<br>
values=tensor([[ 0.2041,  1.7543,  0.4384,  0.7029, -0.3107]]),<br>
indices=tensor([[1, 1, 1, 0, 0]]))<br>
quit()<br>
steph@steph-desktop:~$</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<p>That is, we are back to normal.</p>
<p>Then I went on Google colab. Created a jupyter notebook. Imported torch. The imported version is torch==1.6.0. And there… it works.<br>
So, is it a problem on my side or is it a problem with the torch==1.6.0 version for linux?</p>

username: albanD
cooked: <p>Hi,</p>
<p>It does work fine on my side as well…</p>
<p>Do you have any specific configurations on your machine for libraries like openMP, blas, libc?<br>
Also what kind of CPU do you have?</p>
<p>Can you try with a different version of python to see if it changes anything as well?</p>

username: Stephane_Bersier
cooked: <p>Moving from python3.6.9 to python3.7.5 doesn’t help.</p>
<p>With torch==1.6.0 and python3.7.5:</p>
<p>steph@steph-desktop:~$ python3.7<br>
Python 3.7.5 (default, Nov  7 2019, 10:50:52)<br>
[GCC 8.3.0] on linux<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
torch.<strong>version</strong><br>
‘1.6.0’<br>
a=torch.randn(1,3,5)<br>
a.max(1)<br>
Illegal instruction (core dumped)</p>
</blockquote>
</blockquote>
</blockquote>
<p>While with torch==1.5.0 and python3.7.5:</p>
<p>steph@steph-desktop:~$ python3.7<br>
Python 3.7.5 (default, Nov  7 2019, 10:50:52)<br>
[GCC 8.3.0] on linux<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
torch.<strong>version</strong><br>
‘1.5.0’<br>
a=torch.randn(1,3,5)<br>
a.max(1)<br>
torch.return_types.max(<br>
values=tensor([[ 1.0653,  1.4195,  2.4267, -0.1173,  1.0128]]),<br>
indices=tensor([[1, 0, 1, 0, 1]]))<br>
quit()<br>
steph@steph-desktop:~$</p>
</blockquote>
</blockquote>
</blockquote>
<p>LIBRARIES, KERNEL AND PROCESSOR:</p>
<p>Libraries:<br>
(No “fancy” installs for these libraries)<br>
libc6: version 2.27-3ubuntu1.2<br>
libopenmp: version 2.1.1-8<br>
liblas3: version 3.7.1-4ubuntu1</p>
<p>Kernel:<br>
steph@steph-desktop:~$ uname -a<br>
Linux steph-desktop 5.4.0-42-generic #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux</p>
<p>For the cpu:<br>
Indeed, the problem might well be my (a bit old) processor which doesn’t support the SSE4 instruction set but only SSE2.<br>
Does the processor have to support the SSE4 instruction set for torch.max() function to work properly?</p>
<p>I know that I can’t install tensorflow because of incompatibility of my cpu with SSE4.</p>
<p>steph@steph-desktop:~$ lscpu<br>
Architecture:        x86_64<br>
CPU op-mode(s):      32-bit, 64-bit<br>
Byte Order:          Little Endian<br>
CPU(s):              6<br>
On-line CPU(s) list: 0-5<br>
Thread(s) per core:  1<br>
Core(s) per socket:  6<br>
Socket(s):           1<br>
NUMA node(s):        1<br>
Vendor ID:           AuthenticAMD<br>
CPU family:          16<br>
Model:               10<br>
Model name:          AMD Phenom™ II X6 1065T Processor<br>
Stepping:            0<br>
CPU MHz:             1560.909<br>
CPU max MHz:         2900.0000<br>
CPU min MHz:         800.0000<br>
BogoMIPS:            5827.05<br>
Virtualization:      AMD-V<br>
L1d cache:           64K<br>
L1i cache:           64K<br>
L2 cache:            512K<br>
L3 cache:            6144K<br>
NUMA node0 CPU(s):   0-5<br>
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt cpb hw_pstate vmmcall npt lbrv svm_lock nrip_save pausefilter</p>

username: malfet
cooked: <p>Hi,<br>
Can you please try running your code with <code>ATEN_CPU_CAPABILITY=default</code>?<br>
Also, if you have gdb installed on your machine, can you please run:</p>
<pre><code class="lang-auto">gdb /usr/bin/python -ex "set args -c 'import torch;print(torch.rand(1,2,5).max(1))'" -ex "run" -ex "bt"
</code></pre>
<p>And share the output here?</p>

username: Stephane_Bersier
cooked: <p>Hi,<br>
Regarding ATEN_CPU_CAPABILITY:</p>
<p>steph@steph-desktop:~$ export ATEN_CPU_CAPABILITY=default<br>
steph@steph-desktop:~$ echo $ATEN_CPU_CAPABILITY<br>
default<br>
steph@steph-desktop:~$ python3.7<br>
Python 3.7.5 (default, Nov  7 2019, 10:50:52)<br>
[GCC 8.3.0] on linux<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
torch.<strong>version</strong><br>
‘1.6.0’<br>
a=torch.randn(1,3,5)<br>
a.max(1)<br>
Illegal instruction (core dumped)</p>
</blockquote>
</blockquote>
</blockquote>
<p>Regarding gdb output:<br>
For python3.7.5 and torch==1.6.0 the gdb output is:</p>
<p>steph@steph-desktop:~$ gdb python3.7 -ex “set args -c ‘import torch;print(torch.rand(1,2,5).max(1))’” -ex “run” -ex “bt”<br>
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git<br>
Copyright (C) 2018 Free Software Foundation, Inc.<br>
License GPLv3+: GNU GPL version 3 or later …<br>
This is free software: you are free to change and redistribute it.<br>
There is NO WARRANTY, to the extent permitted by law.  Type “show copying”<br>
and “show warranty” for details.<br>
This GDB was configured as “x86_64-linux-gnu”.<br>
Type “show configuration” for configuration details.<br>
For bug reporting instructions, please see:</p>
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Registered pretty printers for UE4 classes
Reading symbols from python3.7...(no debugging symbols found)...done.
Starting program: /usr/bin/python3.7 -c 'import torch;print(torch.rand(1,2,5).max(1))'
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffa5778700 (LWP 3552)]
[New Thread 0x7fffa4f77700 (LWP 3553)]
[New Thread 0x7fffa2776700 (LWP 3554)]
[New Thread 0x7fff9df75700 (LWP 3555)]
[New Thread 0x7fff9b774700 (LWP 3556)]
<p>Thread 1 “python3.7” received signal SIGILL, Illegal instruction.<br>
0x00007fffe4c04409 in at::TensorIteratorConfig::declare_static_shape(c10::ArrayRef, long) ()<br>
from /home/steph/.local/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so<br>
<span class="hashtag-raw">#0</span>  0x00007fffe4c04409 in at::TensorIteratorConfig::declare_static_shape(c10::ArrayRef, long) ()<br>
from /home/steph/.local/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so<br>
<span class="hashtag-raw">#1</span>  0x00007fffe54e66f7 in at::native::(anonymous namespace)::max_kernel_impl ()<br>
from /home/steph/.local/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so<br>
<span class="hashtag-raw">#2</span>  0x00007fffe449d83e in void at::native::DispatchStub&lt;void (<em>)(at::Tensor&amp;, at::Tensor&amp;, at::Tensor const&amp;, long, bool), at::native::max_stub&gt;::operator()&lt;at::Tensor&amp;, at::Tensor&amp;, at::Tensor const&amp;, long&amp;, bool&amp;&gt;(c10::DeviceType, at::Tensor&amp;, at::Tensor&amp;, at::Tensor const&amp;, long&amp;, bool&amp;) () from /home/steph/.local/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so<br>
<span class="hashtag-raw">#3</span>  0x00007fffe4499bd2 in at::native::max_out(at::Tensor&amp;, at::Tensor&amp;, at::Tensor const&amp;, long, bool) ()<br>
from /home/steph/.local/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so<br>
<span class="hashtag-raw">#4</span>  0x00007fffe449a77f in at::native::max(at::Tensor const&amp;, long, bool) ()<br>
from /home/steph/.local/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so<br>
<span class="hashtag-raw">#5</span>  0x00007fffe48d3047 in at::TypeDefault::max_dim(at::Tensor const&amp;, long, bool) ()<br>
from /home/steph/.local/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so<br>
<span class="hashtag-raw">#6</span>  0x00007fffe4751544 in c10::impl::wrap_kernel_functor_unboxed_&lt;c10::impl::detail::WrapFunctionIntoRuntimeFunctor_&lt;std::tuple&lt;at::Tensor, at::Tensor&gt; (</em>)(at::Tensor const&amp;, long, bool), std::tuple&lt;at::Tensor, at::Tensor&gt;, c10::guts::typelist::typelist&lt;at::Tensor const&amp;, long, bool&gt; &gt;, std::tuple&lt;at::Tensor, at::Tensor&gt; (at::Tensor const&amp;, long, bool)&gt;::call(c10::OperatorKernel*, at::Tensor const&amp;, long, bool) () from /home/steph/.local/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so<br>
<span class="hashtag-raw">#7</span>  0x00007fffe47ebf54 in at::max(at::Tensor const&amp;, long, bool) ()<br>
from /home/steph/.local/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so<br>
<span class="hashtag-raw">#8</span>  0x00007fffe648b9e2 in torch::autograd::VariableType::max_dim(at::Tensor const&amp;, long, bool) ()<br>
from /home/steph/.local/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so<br>
<span class="hashtag-raw">#9</span>  0x00007fffe4751544 in c10::impl::wrap_kernel_functor_unboxed_&lt;c10::impl::detail::WrapFunctionIntoRuntimeFunctor_&lt;std::tup—Type  to continue, or q  to quit—<br>
le&lt;at::Tensor, at::Tensor&gt; (<em>)(at::Tensor const&amp;, long, bool), std::tuple&lt;at::Tensor, at::Tensor&gt;, c10::guts::typelist::typelist&lt;at::Tensor const&amp;, long, bool&gt; &gt;, std::tuple&lt;at::Tensor, at::Tensor&gt; (at::Tensor const&amp;, long, bool)&gt;::call(c10::OperatorKernel</em>, at::Tensor const&amp;, long, bool) () from /home/steph/.local/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so<br>
<span class="hashtag-raw">#10</span> 0x00007fffe4974944 in at::Tensor::max(long, bool) const ()<br>
from /home/steph/.local/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so<br>
<span class="hashtag-raw">#11</span> 0x00007ffff3de9d33 in torch::autograd::THPVariable_max(_object*, _object*, _object*) ()<br>
from /home/steph/.local/lib/python3.7/site-packages/torch/lib/libtorch_python.so<br>
<span class="hashtag-raw">#12</span> 0x00000000004d6194 in _PyMethodDescr_FastCallKeywords ()<br>
<span class="hashtag-raw">#13</span> 0x0000000000551d89 in _PyEval_EvalFrameDefault ()<br>
<span class="hashtag-raw">#14</span> 0x000000000054b302 in _PyEval_EvalCodeWithName ()<br>
<span class="hashtag-raw">#15</span> 0x0000000000530aef in PyRun_StringFlags ()<br>
<span class="hashtag-raw">#16</span> 0x000000000063138d in PyRun_SimpleStringFlags ()<br>
<span class="hashtag-raw">#17</span> 0x000000000065473d in ?? ()<br>
<span class="hashtag-raw">#18</span> 0x000000000065486e in _Py_UnixMain ()<br>
<span class="hashtag-raw">#19</span> 0x00007ffff7a05b97 in __libc_start_main (main=0x4b84d0 , argc=3, argv=0x7fffffffdba8, init=,<br>
fini=, rtld_fini=, stack_end=0x7fffffffdb98) at …/csu/libc-start.c:310<br>
<span class="hashtag-raw">#20</span> 0x00000000005df80a in _start ()<br>
(gdb) quit<br>
A debugging session is active.</p>
<pre><code>Inferior 1 [process 3542] will be killed.
</code></pre>
<p>Quit anyway? (y or n) y<br>
steph@steph-desktop:~$</p>
<p>While with python3.7.5 and torch==1.5.0 installed, the gdb output is:</p>
<p>steph@steph-desktop:~$ gdb python3.7 -ex “set args -c ‘import torch;print(torch.rand(1,2,5).max(1))’” -ex “run” -ex “bt”<br>
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git<br>
Copyright (C) 2018 Free Software Foundation, Inc.<br>
License GPLv3+: GNU GPL version 3 or later (link)<br>
This is free software: you are free to change and redistribute it.<br>
There is NO WARRANTY, to the extent permitted by law.  Type “show copying”<br>
and “show warranty” for details.<br>
This GDB was configured as “x86_64-linux-gnu”.<br>
Type “show configuration” for configuration details.<br>
For bug reporting instructions, please see:<br>
(link).<br>
Find the GDB manual and other documentation resources online at: (link)<br>
For help, type “help”.<br>
Type “apropos word” to search for commands related to “word”…<br>
Registered pretty printers for UE4 classes<br>
Reading symbols from python3.7…(no debugging symbols found)…done.<br>
Starting program: /usr/bin/python3.7 -c ‘import torch;print(torch.rand(1,2,5).max(1))’<br>
[Thread debugging using libthread_db enabled]<br>
Using host libthread_db library “/lib/x86_64-linux-gnu/libthread_db.so.1”.<br>
[New Thread 0x7fffa59f8700 (LWP 3197)]<br>
[New Thread 0x7fffa51f7700 (LWP 3198)]<br>
[New Thread 0x7fffa09f6700 (LWP 3199)]<br>
[New Thread 0x7fff9e1f5700 (LWP 3200)]<br>
[New Thread 0x7fff9b9f4700 (LWP 3201)]<br>
[New Thread 0x7fff92dcd700 (LWP 3203)]<br>
[New Thread 0x7fff925cc700 (LWP 3204)]<br>
[New Thread 0x7fff91dcb700 (LWP 3205)]<br>
[New Thread 0x7fff915ca700 (LWP 3206)]<br>
[New Thread 0x7fff90dc9700 (LWP 3207)]<br>
torch.return_types.max(<br>
values=tensor([[0.3803, 0.9634, 0.5299, 0.9652, 0.6461]]),<br>
indices=tensor([[1, 1, 0, 0, 0]]))<br>
[Thread 0x7fff9b9f4700 (LWP 3201) exited]<br>
[Thread 0x7fff90dc9700 (LWP 3207) exited]<br>
[Thread 0x7fff915ca700 (LWP 3206) exited]<br>
[Thread 0x7fff91dcb700 (LWP 3205) exited]<br>
[Thread 0x7fff925cc700 (LWP 3204) exited]<br>
[Thread 0x7fff92dcd700 (LWP 3203) exited]<br>
[Thread 0x7fff9e1f5700 (LWP 3200) exited]<br>
[Thread 0x7fffa09f6700 (LWP 3199) exited]javascript:;<br>
[Thread 0x7fffa51f7700 (LWP 3198) exited]<br>
[Thread 0x7fffa59f8700 (LWP 3197) exited]<br>
[Inferior 1 (process 3187) exited normally]<br>
No stack.<br>
(gdb) quit</p>

username: malfet
cooked: <p>Thank you for the quick reply. Can you please also issue <code>disassemble</code> command when exception is hit in gdb, by executing:<br>
gdb python3.7 -ex “set args -c ‘import torch;print(torch.rand(1,2,5).max(1))’” -ex “run” -ex “bt” -ex “disassemble”</p>

username: Stephane_Bersier
cooked: <p>Sorry, the file was incomplete…<br>
So, here it is again:<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://cfl.dropboxstatic.com/static/images/favicon-vflUeLeeY.ico" class="site-icon" width="16" height="16">
      <a href="https://www.dropbox.com/s/44wnrpwby56l2nn/dump_2.txt?dl=0" target="_blank" rel="nofollow noopener">Dropbox</a>
  </header>
  <article class="onebox-body">
    <img src="https://www.dropbox.com/static/images/spectrum-icons/generated/content/content-txt-large.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://www.dropbox.com/s/44wnrpwby56l2nn/dump_2.txt?dl=0" target="_blank" rel="nofollow noopener">dump_2.txt</a></h3>

<p>Shared with Dropbox</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: malfet
cooked: <p>Thank you for the detailed repro instructions,  filed <a href="https://github.com/pytorch/pytorch/issues/43300" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/43300</a></p>

username: Stephane_Bersier
cooked: <p>I’ll keep an eye on github about this issue.<br>
Thank you very much for having spent some time on this.<br>
Best regards,<br>
Stéphane</p>

username: Andre_Godinho
cooked: <p>Did anyone fix this?</p>
<p>Have the same problem, torch==1.6.0 and core dumps when using torch.max()</p>

username: Stephane_Bersier
cooked: <p>The issue should be fixed in version 1.7<br>
See <a href="https://github.com/pytorch/pytorch/issues/43300" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/issues/43300</a></p>