Issue compiling based on order of -isystem include dirs in conda environment

username: Brian_Wiley
cooked: <p>I am trying to compile pytorch inside a conda environment using my system version headers of cuda/cuda-toolkit located at <code>/usr/local/cuda-12/include</code>.</p>
<p>I get all sorts of compilation issues since there are headers in my environment include, <code>/home/coyote/.conda/envs/deepchem/include</code> that mess up compiling with system nvcc and system gcc when I have this order with environment includes being AUTOMATICALLY listed first when running <code>python setup.py install</code></p>
<pre><code class="lang-auto">/usr/local/cuda-12/bin/nvcc ... -isystem /home/coyote/.conda/envs/deepchem/include &lt;more -isysyem paths&gt; -isystem /usr/local/cuda-12/include ... -c pytorch/aten/src/ATen/native/cuda/CumminmaxKernel.cu -o caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/CumminmaxKernel.cu.o
</code></pre>
<p>errors such as below <strong>without</strong> moving <code>/home/coyote/.conda/envs/deepchem/include/nv</code> to <code>/home/coyote/.conda/envs/deepchem/include/_nv</code></p>
<pre><code class="lang-auto">/usr/local/cuda-12/include/cuda_bf16.hpp(383): error: identifier "__NV_IS_DEVICE" is undefined
</code></pre>
<p>or later after moving  <code>/home/coyote/.conda/envs/deepchem/include/nv</code> to <code>/home/coyote/.conda/envs/deepchem/include/_nv</code> then getting issues with the <code>cub</code> folder in the environment include</p>
<pre><code class="lang-auto">/home/coyote/.conda/envs/deepchem/include/cub/iterator/tex_ref_input_iterator.cuh(99): error: texture is not a template

/home/coyote/.conda/envs/deepchem/include/cub/iterator/tex_ref_input_iterator.cuh(111): error: identifier "cudaBindTexture" is undefined
</code></pre>
<p>But I don’t get these compile issues if I manually put system cuda <code>-isystem /usr/local/cuda-12/include</code> somewhere before the environment include <code>-isystem /home/coyote/.conda/envs/deepchem/include</code></p>
<p>So my question is how to make this order automatically when running setup.py install.  How can I make <code>/usr/local/cuda-12/include</code> the first of all <code>-isystem</code> includes?</p>

username: ptrblck
cooked: <p>Could you explain which files are in <code>deepchem/include</code> as I cannot find these references in the <a href="https://github.com/deepchem/deepchem">repository</a>?</p>

username: Brian_Wiley
cooked: <p>Well there are a lot since I have included a lot of packages for chemical machine learning here including tensorflow with cuda.  There are 59 <code>*.h</code> header files that intersect both system include (<code>/usr/local/cuda-12/include</code>) and environment includes (<code>/home/coyote/.conda/envs/deepchem/include</code>) and 9 directories below that are in both as well.</p>
<pre><code class="lang-auto">$ grep -f &lt;(ls -d /usr/local/cuda-12/include/*/ | cut -d/ -f6) &lt;(ls -d /home/coyote/.conda/envs/deepchem/include/*/ | cut -d/ -f8)
CL
crt
cub
cuda
nv
nvtx3
Openacc
Openmp
thrust
</code></pre>
<p>Regardless with deepchem since they implement both tensorflow and pytorch it would be nice to install tensorflow with cuda and pytorch from source with cuda12.1, i.e. for instance adding a manual -isystem  include before the rest with some code resembling <a href="https://gist.github.com/svvitale/49e19bfbec3b0ca100b4f1762981d855" class="inline-onebox" rel="noopener nofollow ugc">setup.py with -isystem includes · GitHub</a> but having the “extra_compile_args” of this manual -isystem after all the CFLAGS included “-I…” but before the rest of the -isystem includes.  In pytorch’s “extra_compile_args” these all come <strong>after</strong> the -isystem includes" so it won’t be helpful to add it there.</p>

username: Brian_Wiley
cooked: <p>So far updating CMake variables such as CUDNN_INCLUDE_PATH, CUDNN_LIBRARY, CUDNN_LIBRARY_PATH, CUB_INCLUDE_DIR and temporarily moving <code>/home/coyote/.conda/envs/deepchem/include/nv</code> to <code>/home/coyote/.conda/envs/deepchem/include/_nv</code> works for compiling <strong>some</strong> caffe2 sources.</p>