Error when installing from source - Makefile:159: recipe for target 'all' failed

mfazio: <p>Hi,</p>
<p>I am working my way into the source installation of pytorch. I am using ubuntu 18.04, CUDA 10.2 and cuDNN 7. The configuration goes well and I get to 86% when I get this error:</p>
<p>Makefile:159: recipe for target ‘all’ failed<br>
make: *** [all] Error 2</p>
<p>Any idea why this is happening and how to fix it?</p>
<p>Thanks!</p>

ptrblck: <p>Could you check the install log for the reported error and post it here, please?</p>

mfazio: <p>Thanks for your reply!</p>
<p>I am a newbie here, so I am not too sure what the install logs are. I did check the CMakeError.log file and this is what I see there regarding the error:</p>
<pre><code class="lang-auto">Run Build Command(s):/usr/bin/make cmTC_f7d0a/fast &amp;&amp; /usr/bin/make  -f CMakeFiles/cmTC_f7d0a.dir/build.make CMakeFiles/cmTC_f7d0a.dir/build
make[1]: Entering directory '/home/simulations/pytorch/build/CMakeFiles/CMakeTmp'
Building CXX object CMakeFiles/cmTC_f7d0a.dir/src.cxx.o
/usr/bin/c++ -DHAS_WERROR_CAST_FUNCTION_TYPE -I/home/simulations/anaconda3/envs/pytorch-build/include -I/usr/local/cuda-10.2/include -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format  -fPIE   -Werror=cast-function-type -std=gnu++14 -o CMakeFiles/cmTC_f7d0a.dir/src.cxx.o -c /home/simulations/pytorch/build/CMakeFiles/CMakeTmp/src.cxx
cc1plus: error: -Werror=cast-function-type: no option -Wcast-function-type
CMakeFiles/cmTC_f7d0a.dir/build.make:84: recipe for target 'CMakeFiles/cmTC_f7d0a.dir/src.cxx.o' failed
make[1]: *** [CMakeFiles/cmTC_f7d0a.dir/src.cxx.o] Error 1
make[1]: Leaving directory '/home/simulations/pytorch/build/CMakeFiles/CMakeTmp'
Makefile:140: recipe for target 'cmTC_f7d0a/fast' failed
make: *** [cmTC_f7d0a/fast] Error 2
</code></pre>
<p>Is that helpful? Should I check any other log files?</p>
<p>Thanks again! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>The output in the terminal should give you an error message and pointing towards the root cause of the issue. Based on your current output I don’t see what exactly failed.</p>

mfazio: <p>Hi again,</p>
<p>I tried the installation one more time, just to check if the error repeats and it does. From the terminal, I can see the following:</p>
<pre><code class="lang-auto">[ 83%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/CrossKernel.cpp.AVX2.cpp.o
[ 83%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/CopyKernel.cpp.AVX2.cpp.o
[ 84%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/ComplexKernel.cpp.AVX2.cpp.o
[ 84%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/CatKernel.cpp.AVX2.cpp.o
[ 84%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/BlasKernel.cpp.AVX2.cpp.o
[ 84%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/BinaryOpsKernel.cpp.AVX2.cpp.o
[ 84%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/AvgPoolKernel.cpp.AVX2.cpp.o
[ 84%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/AdaptiveMaxPoolKernel.cpp.AVX2.cpp.o
[ 84%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/AdaptiveAvgPoolKernel.cpp.AVX2.cpp.o
[ 84%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/native/cpu/Activation.cpp.AVX2.cpp.o
[ 84%] Linking CXX shared library ../lib/libtorch_cpu.so
[ 84%] Built target torch_cpu
Makefile:159: recipe for target 'all' failed
make: *** [all] Error 2

</code></pre>
<p>Thanks for the help!!</p>

ptrblck: <p>The error should be shown before the posted lines containing the <code>error</code> keyword, so you could search for it.</p>

mfazio: <p>Got it! I looked into the terminal output and found when the error is mentioned. It happens at 69% progress, so not sure why it keeps going until it reaches 84% and displays the error I showed above.</p>
<p>In any case, this is what I found:</p>
<pre><code class="lang-auto">[ 69%] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/core/adaption.cpp.o
prims_ll.h(198): error: identifier "__funnelshift_r" is undefined
          detected during:
            instantiation of "uint64_t Primitives&lt;T, RedOp, Fan, Direct, ProtoLL&gt;::DataLoader::loadFinish() [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Fan=FanSymmetric&lt;2&gt;, Direct=1]" 
(251): here
            instantiation of "void Primitives&lt;T, RedOp, Fan, Direct, ProtoLL&gt;::LLGenericOp&lt;RECV,SEND,SrcBuf,DstBuf&gt;(intptr_t, intptr_t, int, __nv_bool) [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Fan=FanSymmetric&lt;2&gt;, Direct=1, RECV=1, SEND=1, SrcBuf=0, DstBuf=1]" 
(384): here
            instantiation of "void Primitives&lt;T, RedOp, Fan, Direct, ProtoLL&gt;::recvReduceCopySend(intptr_t, intptr_t, int, __nv_bool) [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Fan=FanSymmetric&lt;2&gt;, Direct=1]" 
primitives.h(135): here
            instantiation of "void PrimitivesWithoutDirect&lt;RealPrimitives&gt;::directRecvReduceCopySend(intptr_t, intptr_t, intptr_t, int, __nv_bool) [with RealPrimitives=Primitives&lt;int8_t, FuncSum&lt;int8_t&gt;, FanSymmetric&lt;2&gt;, 1, ProtoLL&gt;]" 
all_reduce.h(206): here
            instantiation of "void &lt;unnamed&gt;::runTreeSplit&lt;T,RedOp,Proto&gt;(ncclWorkElem *) [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Proto=ProtoLL]" 
all_reduce.h(372): here
            instantiation of "void RunWorkElement&lt;(ncclFunc_t)4, T, RedOp, 0, 0&gt;::run(ncclWorkElem *) [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;]" 
common.h(77): here
            instantiation of "void RunWork&lt;Fn, T, RedOp, Algo, Proto&gt;::run(ncclWork *) [with Fn=(ncclFunc_t)4, T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Algo=0, Proto=0]" 
all_reduce.cu(11): here

prims_ll.h(198): error: identifier "__funnelshift_r" is undefined
          detected during:
            instantiation of "uint64_t Primitives&lt;T, RedOp, Fan, Direct, ProtoLL&gt;::DataLoader::loadFinish() [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Fan=FanAsymmetric&lt;2, 1&gt;, Direct=1]" 
(251): here
            instantiation of "void Primitives&lt;T, RedOp, Fan, Direct, ProtoLL&gt;::LLGenericOp&lt;RECV,SEND,SrcBuf,DstBuf&gt;(intptr_t, intptr_t, int, __nv_bool) [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Fan=FanAsymmetric&lt;2, 1&gt;, Direct=1, RECV=0, SEND=1, SrcBuf=0, DstBuf=-1]" 
(363): here
            instantiation of "void Primitives&lt;T, RedOp, Fan, Direct, ProtoLL&gt;::send(intptr_t, int) [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Fan=FanAsymmetric&lt;2, 1&gt;, Direct=1]" 
all_reduce.h(224): here
            instantiation of "void &lt;unnamed&gt;::runTreeSplit&lt;T,RedOp,Proto&gt;(ncclWorkElem *) [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Proto=ProtoLL]" 
all_reduce.h(372): here
            instantiation of "void RunWorkElement&lt;(ncclFunc_t)4, T, RedOp, 0, 0&gt;::run(ncclWorkElem *) [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;]" 
common.h(77): here
            instantiation of "void RunWork&lt;Fn, T, RedOp, Algo, Proto&gt;::run(ncclWork *) [with Fn=(ncclFunc_t)4, T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Algo=0, Proto=0]" 
all_reduce.cu(11): here

prims_ll.h(198): error: identifier "__funnelshift_r" is undefined
          detected during:
            instantiation of "uint64_t Primitives&lt;T, RedOp, Fan, Direct, ProtoLL&gt;::DataLoader::loadFinish() [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Fan=FanAsymmetric&lt;1, 2&gt;, Direct=1]" 
(251): here
            instantiation of "void Primitives&lt;T, RedOp, Fan, Direct, ProtoLL&gt;::LLGenericOp&lt;RECV,SEND,SrcBuf,DstBuf&gt;(intptr_t, intptr_t, int, __nv_bool) [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Fan=FanAsymmetric&lt;1, 2&gt;, Direct=1, RECV=1, SEND=0, SrcBuf=-1, DstBuf=1]" 
(369): here
            instantiation of "void Primitives&lt;T, RedOp, Fan, Direct, ProtoLL&gt;::recv(intptr_t, int, __nv_bool) [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Fan=FanAsymmetric&lt;1, 2&gt;, Direct=1]" 
primitives.h(125): here
            instantiation of "void PrimitivesWithoutDirect&lt;RealPrimitives&gt;::directRecv(intptr_t, int) [with RealPrimitives=Primitives&lt;int8_t, FuncSum&lt;int8_t&gt;, FanAsymmetric&lt;1, 2&gt;, 1, ProtoLL&gt;]" 
all_reduce.h(243): here
            instantiation of "void &lt;unnamed&gt;::runTreeSplit&lt;T,RedOp,Proto&gt;(ncclWorkElem *) [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Proto=ProtoLL]" 
all_reduce.h(372): here
            instantiation of "void RunWorkElement&lt;(ncclFunc_t)4, T, RedOp, 0, 0&gt;::run(ncclWorkElem *) [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;]" 
common.h(77): here
            instantiation of "void RunWork&lt;Fn, T, RedOp, Algo, Proto&gt;::run(ncclWork *) [with Fn=(ncclFunc_t)4, T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Algo=0, Proto=0]" 
all_reduce.cu(11): here

op128.h(51): error: identifier "__funnelshift_r" is undefined
          detected during:
            instantiation of "void Primitives&lt;T, RedOp, Fan, Direct, ProtoLL128&gt;::loadRegsBegin(uint64_t (&amp;)[WordPerThread], const T *, int) [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Fan=FanSymmetric&lt;2&gt;, Direct=1, WordPerThread=8]" 
prims_ll128.h(305): here
            instantiation of "void Primitives&lt;T, RedOp, Fan, Direct, ProtoLL128&gt;::GenericOp&lt;RECV,SEND,SrcBuf,DstBuf&gt;(intptr_t, intptr_t, int, __nv_bool) [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Fan=FanSymmetric&lt;2&gt;, Direct=1, RECV=1, SEND=1, SrcBuf=0, DstBuf=1]" 
prims_ll128.h(422): here
            instantiation of "void Primitives&lt;T, RedOp, Fan, Direct, ProtoLL128&gt;::recvReduceCopySend(intptr_t, intptr_t, int, __nv_bool) [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Fan=FanSymmetric&lt;2&gt;, Direct=1]" 
primitives.h(135): here
            instantiation of "void PrimitivesWithoutDirect&lt;RealPrimitives&gt;::directRecvReduceCopySend(intptr_t, intptr_t, intptr_t, int, __nv_bool) [with RealPrimitives=Primitives&lt;int8_t, FuncSum&lt;int8_t&gt;, FanSymmetric&lt;2&gt;, 1, ProtoLL128&gt;]" 
all_reduce.h(206): here
            instantiation of "void &lt;unnamed&gt;::runTreeSplit&lt;T,RedOp,Proto&gt;(ncclWorkElem *) [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Proto=ProtoLL128]" 
all_reduce.h(386): here
            instantiation of "void RunWorkElement&lt;(ncclFunc_t)4, T, RedOp, 0, 1&gt;::run(ncclWorkElem *) [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;]" 
common.h(77): here
            instantiation of "void RunWork&lt;Fn, T, RedOp, Algo, Proto&gt;::run(ncclWork *) [with Fn=(ncclFunc_t)4, T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Algo=0, Proto=1]" 
all_reduce.cu(11): here

prims_ll.h(198): error: identifier "__funnelshift_r" is undefined
          detected during:
            instantiation of "uint64_t Primitives&lt;T, RedOp, Fan, Direct, ProtoLL&gt;::DataLoader::loadFinish() [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Fan=FanSymmetric&lt;1&gt;, Direct=1]" 
(251): here
            instantiation of "void Primitives&lt;T, RedOp, Fan, Direct, ProtoLL&gt;::LLGenericOp&lt;RECV,SEND,SrcBuf,DstBuf&gt;(intptr_t, intptr_t, int, __nv_bool) [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Fan=FanSymmetric&lt;1&gt;, Direct=1, RECV=0, SEND=1, SrcBuf=0, DstBuf=-1]" 
(363): here
            instantiation of "void Primitives&lt;T, RedOp, Fan, Direct, ProtoLL&gt;::send(intptr_t, int) [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Fan=FanSymmetric&lt;1&gt;, Direct=1]" 
all_reduce.h(64): here
            instantiation of "void &lt;unnamed&gt;::runRing&lt;T,RedOp,Proto&gt;(ncclWorkElem *) [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Proto=ProtoLL]" 
all_reduce.h(365): here
            instantiation of "void RunWorkElement&lt;(ncclFunc_t)4, T, RedOp, 1, 0&gt;::run(ncclWorkElem *) [with T=int8_t, RedOp=FuncSum&lt;int8_t&gt;]" 
common.h(77): here
            instantiation of "void RunWork&lt;Fn, T, RedOp, Algo, Proto&gt;::run(ncclWork *) [with Fn=(ncclFunc_t)4, T=int8_t, RedOp=FuncSum&lt;int8_t&gt;, Algo=1, Proto=0]" 
all_reduce.cu(11): here

5 errors detected in the compilation of "/tmp/tmpxft_000064a1_00000000-6_all_reduce.cpp1.ii".
/home/simulations/pytorch/build/nccl/obj/collectives/device/Makefile.rules:227: recipe for target '/home/simulations/pytorch/build/nccl/obj/collectives/device/all_reduce_sum_i8.o' failed
make[5]: *** [/home/simulations/pytorch/build/nccl/obj/collectives/device/all_reduce_sum_i8.o] Error 1
Makefile:50: recipe for target '/home/simulations/pytorch/build/nccl/obj/collectives/device/colldevice.a' failed
make[4]: *** [/home/simulations/pytorch/build/nccl/obj/collectives/device/colldevice.a] Error 2
Makefile:25: recipe for target 'src.build' failed
make[3]: *** [src.build] Error 2
CMakeFiles/nccl_external.dir/build.make:129: recipe for target 'nccl_external-prefix/src/nccl_external-stamp/nccl_external-build' failed
make[2]: *** [nccl_external-prefix/src/nccl_external-stamp/nccl_external-build] Error 2
CMakeFiles/Makefile2:1986: recipe for target 'CMakeFiles/nccl_external.dir/all' failed
make[1]: *** [CMakeFiles/nccl_external.dir/all] Error 2

</code></pre>
<p>Thanks again!!</p>

ptrblck: <p>The error is raised in:</p>
<pre data-code-wrap="python"><code class="lang-python">error: identifier "__funnelshift_r" is undefined
</code></pre>
<p>Which device are you using? Funnel shift ops should be available for compute capabilities &gt;= 3.2.</p>
<aside class="quote no-group" data-username="mfazio" data-post="7" data-topic="134892">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/c89c15/48.png" class="avatar"> mfazio:</div>
<blockquote>
<p>so not sure why it keeps going until it reaches 84% and displays the error I showed above.</p>
</blockquote>
</aside>
<p>The build uses multiple jobs so when one job fails, the others could continue for a while until they get stopped.</p>

mfazio: <p>I am using a GTX Geforce 650, which has a computing capability of 3.0. I guess the error makes sense now. Any way to work around it??</p>

ptrblck: <p>I don’t think you would be able to build the current PyTorch version for this device and might need to stick to an older release.</p>

mfazio: <p>I see! What release would you recommend I use?</p>
<p>Thank you so much for all the help!!</p>

ptrblck: <p>Unfortunately, I’m unsure which release supported sm&lt;3.5 <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=10" title=":confused:" class="emoji" alt=":confused:"></p>

mfazio: <p>No problem!! Thanks for all the help!</p>