Repository: spack/spack

Issue:
build issues with py-torch
There are a few issues with building py-torch. 

1. The package recipe sets `+mkldnn` by default and depends on `intel-mkl-dnn`. However, the build actually builds the in-tree mkldnn
```
[265/2458] Building CXX object third_party/ideep/mkl-dnn/src/CMakeFiles/mkldnn.dir/cpu/rnn/ref_postgemm_gru.cpp.o
[292/2458] Building CXX object third_party/ideep/mkl-dnn/src/CMakeFiles/mkldnn.dir/cpu/rnn/ref_postgemm_gru_lbr.cpp.o
[313/2458] Building CXX object third_party/ideep/mkl-dnn/src/CMakeFiles/mkldnn.dir/cpu/rnn/ref_postgemm_lstm.cpp.o
[524/2458] Building CXX object third_party/ideep/mkl-dnn/src/CMakeFiles/mkldnn.dir/cpu/rnn/ref_postgemm_rnn.cpp.o
[905/2458] Building CXX object third_party/ideep/mkl-dnn/src/CMakeFiles/mkldnn.dir/cpu/rnn/rnn_utils.cpp.o
[986/2458] Building CXX object third_party/ideep/mkl-dnn/src/CMakeFiles/mkldnn.dir/cpu/rnn/ref_rnn.cpp.o
[994/2458] Linking CXX static library lib/libmkldnn.a
```
2. When `~caffe2` is set, `BUILD_CAFFE2_MOBILE` is still set to ON. This may be okay but not sure if that was the intent of the caffe2 variant.
3. The build fails when `+caffe2` is set.
```console
3 errors found in build log:
     2312     #warning "host_defines.h is an internal header file and must not be used directly.  This file will be removed in a
             future CUDA release.  Please use cuda_runtime_api.h or cuda_runtime.h instead."
     2313      ^~~~~~~
     2314    /opt/packages/gpjohnsn/apps/linux-centos7-broadwell/gcc-8.3.0/eigen-3.3.7-byl22ahrj5dv4lmcxzh3iudtqor3q24k/include/e
             igen3/Eigen/src/Core/arch/CUDA/Half.h(302): warning: calling a __host__ function from a __host__ __device__ function
              is not allowed
     2315
     2316    /opt/packages/gpjohnsn/apps/linux-centos7-broadwell/gcc-8.3.0/eigen-3.3.7-byl22ahrj5dv4lmcxzh3iudtqor3q24k/include/e
             igen3/Eigen/src/Core/arch/CUDA/Half.h(305): warning: calling a __host__ function from a __host__ __device__ function
              is not allowed
     2317
  >> 2318    /opt/packages/gpjohnsn/apps/linux-centos7-broadwell/gcc-8.3.0/eigen-3.3.7-byl22ahrj5dv4lmcxzh3iudtqor3q24k/include/e
             igen3/Eigen/src/Core/arch/CUDA/PacketMathHalf.h(233): error: no suitable conversion function from "__half" to "unsig
             ned short" exists
     2319
  >> 2320    /opt/packages/gpjohnsn/apps/linux-centos7-broadwell/gcc-8.3.0/eigen-3.3.7-byl22ahrj5dv4lmcxzh3iudtqor3q24k/include/e
             igen3/Eigen/src/Core/arch/CUDA/PacketMathHalf.h(267): error: no suitable conversion function from "__half" to "unsig
             ned short" exists
     2321
     2322    2 errors detected in the compilation of "/tmp/tmpxft_0000c7f1_00000000-13_rsqrt_op.compute_35.cpp1.ii".
  >> 2323    CMake Error at torch_generated_rsqrt_op.cu.o.Release.cmake:281 (message):
     2324      Error generating file
     2325      /dev/shm/gpjohnsn/spack-stage-py-torch-1.4.0-23jvdwla5padn5pcam62k6rzovczeyhe/spack-src/build/caffe2/CMakeFiles/to
             rch.dir/operators/./torch_generated_rsqrt_op.cu.o
     2326
     2327
     2328    [1082/2458] Building NVCC (Device) object caffe2/CMakeFiles/torch.dir/operators/torch_generated_resize_3d_op.cu.o
     2329    [1083/2458] Building NVCC (Device) object caffe2/CMakeFiles/torch.dir/__/aten/src/THC/torch_generated_THCTensorIndex
             .cu.o 
```

---

### Platform and user environment

Please report your OS here:
```commandline
$ uname -a 
Linux argon-build.hpc 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -d
Description:    CentOS Linux release 7.5.1804 (Core)
``` 



Discussion:
1. Yes, this is a known issue with `py-torch`, we need to find a way to use external installations of packages instead of vendored copies or downloading dependencies
2. The `py-torch` package doesn't set that variant, should it?
3. This looks like a `py-torch` bug, not a Spack bug. You're probably better off reporting it to them.
1. Should we depend on intel-mkl-dnn if it is not actually used?
2. There is a `caffe2` variant for the operators
```
variant('caffe2', default=False, description='Enables Caffe2 operators build') 
```
It is not clear whether that should cover BUILD_CAFFE2_MOBILE as well. From your comment, I guess not.

For 3, you are probably right. I know this spack package was recently updated so thought it could be my environment. In any event, it likely makes sense to mark building with `caffe2` as broken as it takes a significant amount of time compiling before it fails.
1. In theory, no, but I would consider the fact that we aren't linking to an external `intel-mkl-dnn` to be the bug, not that we have an extra unneeded dependency
2. I'm not sure either, depends on if you want to add an extra variant
3. I've never built/needed `py-torch+caffe2` before so I'm fine with that
For mkldn, it does not look like a bug but rather intentional. From `.../spack-src/cmake/Modules/FindMKLDNN.cmake`
```
SET(IDEEP_ROOT "${PROJECT_SOURCE_DIR}/third_party/ideep")
SET(MKLDNN_ROOT "${IDEEP_ROOT}/mkl-dnn")
```
Also, MKLDNN_HOME is not a variable that is used since version 0.4.1.

For caffe2, the package needs to set TORCH_CUDA_ARCH_LIST with values from the cuda_arch variant. I am getting ready to work up a PR to fix building caffe2 but I am not sure what to do about mkldnn. That is still a variant but it does not depend on the spack package, I am guessing since version 0.4.1. 

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/py-torch/package.py b/var/spack/repos/builtin/packages/py-torch/package.py
--- a/var/spack/repos/builtin/packages/py-torch/package.py
+++ b/var/spack/repos/builtin/packages/py-torch/package.py
@@ -62,6 +62,7 @@ class PyTorch(PythonPackage, CudaPackage):
     version('0.4.0', tag='v0.4.0', submodules=True)
     version('0.3.1', tag='v0.3.1', submodules=True)
 
+    variant('cuda', default=True, description='Build with CUDA')
     variant('cudnn', default=True, description='Enables the cuDNN build')
     variant('magma', default=False, description='Enables the MAGMA build')
     variant('fbgemm', default=False, description='Enables the FBGEMM build')
@@ -100,6 +101,27 @@ class PyTorch(PythonPackage, CudaPackage):
     conflicts('+zstd', when='@:1.0')
     conflicts('+tbb', when='@:1.1')
 
+    cuda_arch_conflict = ('This version of Torch/Caffe2 only supports compute '
+                          'capabilities ')
+
+    conflicts('cuda_arch=none', when='+cuda+caffe2',
+              msg='Must specify CUDA compute capabilities of your GPU, see '
+              'https://developer.nvidia.com/cuda-gpus')
+    conflicts('cuda_arch=52', when='@1.3.0:+cuda+caffe2',
+              msg=cuda_arch_conflict + '>=5.3')
+    conflicts('cuda_arch=50', when='@1.3.0:+cuda+caffe2',
+              msg=cuda_arch_conflict + '>=5.3')
+    conflicts('cuda_arch=35', when='@1.3.0:+cuda+caffe2',
+              msg=cuda_arch_conflict + '>=5.3')
+    conflicts('cuda_arch=32', when='@1.3.0:+cuda+caffe2',
+              msg=cuda_arch_conflict + '>=5.3')
+    conflicts('cuda_arch=30', when='@1.3.0:+cuda+caffe2',
+              msg=cuda_arch_conflict + '>=5.3')
+    conflicts('cuda_arch=30', when='@1.2.0:+cuda+caffe2',
+              msg=cuda_arch_conflict + '>=3.2')
+    conflicts('cuda_arch=20', when='@1.0.0:+cuda+caffe2',
+              msg=cuda_arch_conflict + '>=3.0')
+
     # Required dependencies
     depends_on('cmake@3.5:', type='build')
     # Use Ninja generator to speed up build times
@@ -128,7 +150,10 @@ class PyTorch(PythonPackage, CudaPackage):
     # depends_on('fbgemm', when='+fbgemm')
     # TODO: add dependency: https://github.com/ROCmSoftwarePlatform/MIOpen
     # depends_on('miopen', when='+miopen')
-    depends_on('intel-mkl-dnn', when='+mkldnn')
+    # TODO: See if there is a way to use an external mkldnn installation.
+    # Currently, only older versions of py-torch use an external mkldnn
+    # library.
+    depends_on('intel-mkl-dnn', when='@0.4:0.4.1+mkldnn')
     # TODO: add dependency: https://github.com/Maratyszcza/NNPACK
     # depends_on('nnpack', when='+nnpack')
     depends_on('qnnpack', when='+qnnpack')
@@ -197,6 +222,10 @@ def enable_or_disable(variant, keyword='USE', var=None, newer=False):
         enable_or_disable('cuda')
         if '+cuda' in self.spec:
             env.set('CUDA_HOME', self.spec['cuda'].prefix)
+            torch_cuda_arch = ';'.join('{0:.1f}'.format(float(i) / 10.0) for i
+                                       in
+                                       self.spec.variants['cuda_arch'].value)
+            env.set('TORCH_CUDA_ARCH_LIST', torch_cuda_arch)
 
         enable_or_disable('cudnn')
         if '+cudnn' in self.spec:
@@ -213,7 +242,7 @@ def enable_or_disable(variant, keyword='USE', var=None, newer=False):
             env.set('MIOPEN_LIBRARY', self.spec['miopen'].libs[0])
 
         enable_or_disable('mkldnn')
-        if '+mkldnn' in self.spec:
+        if '@0.4:0.4.1+mkldnn' in self.spec:
             env.set('MKLDNN_HOME', self.spec['intel-mkl-dnn'].prefix)
 
         enable_or_disable('nnpack')


Test patch: