Repository: rwth-i6/returnn

Issue:
Can't compile custom ops (eg. NativeLSTM) using TF 1.5
Using TensorFlow 1.5, it's not possible to use a layer that requires a custom op (eg. the NativeLSTM layer).
Doing so results in the following error:

```
OpCodeCompiler call: /usr/local/cuda-9.0/bin/nvcc -shared -O2 -std=c++11 -I /home/christopher/anaconda3/lib/python3.6/site-packages/tensorflow/include -I /home/christopher/anaconda3/lib/python3.6/site-packages/tensorflow/include/external/nsync/public -I /usr/local/cuda-9.0/include -L /usr/local/cuda-9.0/lib64 -x cu -v -DGOOGLE_CUDA=1 -Xcompiler -fPIC -Xcompiler -v -arch compute_50 -D_GLIBCXX_USE_CXX11_ABI=0 -g /tmp/christopher/returnn_tf_cache/ops/NativeLstm2/2ec60a4d21/NativeLstm2.cc -o /tmp/christopher/returnn_tf_cache/ops/NativeLstm2/2ec60a4d21/NativeLstm2.so -L/home/christopher/anaconda3/lib/python3.6/site-packages/numpy/.libs -l:libopenblasp-r0-39a31c03.2.18.so -L/home/christopher/anaconda3/lib/python3.6/site-packages/tensorflow -ltensorflow_framework
OpCodeCompiler: /usr/local/cuda-9.0/bin/nvcc failed.
Original stdout/stderr:
#$ _SPACE_= 
#$ _CUDART_=cudart
#$ _HERE_=/usr/local/cuda-9.0/bin
#$ _THERE_=/usr/local/cuda-9.0/bin
#$ _TARGET_SIZE_=
#$ _TARGET_DIR_=
#$ _TARGET_DIR_=targets/x86_64-linux
#$ TOP=/usr/local/cuda-9.0/bin/..
#$ NVVMIR_LIBRARY_DIR=/usr/local/cuda-9.0/bin/../nvvm/libdevice
#$ LD_LIBRARY_PATH=/usr/local/cuda-9.0/bin/../lib:~/Documents/Master/Semester2/Praktikum/mxnet/lib/:/usr/local/cuda/lib64::/opt/gurobi752/linux64/lib
#$ PATH=/usr/local/cuda-9.0/bin/../open64/bin:/usr/local/cuda-9.0/bin/../nvvm/bin:/usr/local/cuda-9.0/bin:/home/christopher/anaconda3/envs/test_tf14/bin:/usr/local/cuda/bin/:/home/christopher/anaconda3/bin:/home/christopher/bin:/home/christopher/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/gurobi752/linux64/bin
#$ INCLUDES="-I/usr/local/cuda-9.0/bin/../targets/x86_64-linux/include"  
#$ LIBRARIES=  "-L/usr/local/cuda-9.0/bin/../targets/x86_64-linux/lib/stubs" "-L/usr/local/cuda-9.0/bin/../targets/x86_64-linux/lib"
#$ CUDAFE_FLAGS=
#$ PTXAS_FLAGS=
#$ gcc -std=c++11 -D__CUDA_ARCH__=500 -E -x c++  -DCUDA_DOUBLE_MATH_FUNCTIONS -D__CUDACC__ -D__NVCC__  -fPIC -v -O2 -I"/home/christopher/anaconda3/lib/python3.6/site-packages/tensorflow/include" -I"/home/christopher/anaconda3/lib/python3.6/site-packages/tensorflow/include/external/nsync/public" -I"/usr/local/cuda-9.0/include" "-I/usr/local/cuda-9.0/bin/../targets/x86_64-linux/include"   -D"__CUDACC_VER_BUILD__=176" -D"__CUDACC_VER_MINOR__=0" -D"__CUDACC_VER_MAJOR__=9" -D"GOOGLE_CUDA=1" -D"_GLIBCXX_USE_CXX11_ABI=0" -include "cuda_runtime.h" -m64 -g -gdwarf-2 "/tmp/christopher/returnn_tf_cache/ops/NativeLstm2/2ec60a4d21/NativeLstm2.cc" > "/tmp/tmpxft_00006a9a_00000000-8_NativeLstm2.cpp1.ii" 
Using built-in specs.
COLLECT_GCC=gcc
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu1~16.04.11' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.11) 
COLLECT_GCC_OPTIONS='-std=c++11' '-D' '__CUDA_ARCH__=500' '-E' '-D' 'CUDA_DOUBLE_MATH_FUNCTIONS' '-D' '__CUDACC__' '-D' '__NVCC__' '-fPIC' '-v' '-O2' '-I' '/home/christopher/anaconda3/lib/python3.6/site-packages/tensorflow/include' '-I' '/home/christopher/anaconda3/lib/python3.6/site-packages/tensorflow/include/external/nsync/public' '-I' '/usr/local/cuda-9.0/include' '-I' '/usr/local/cuda-9.0/bin/../targets/x86_64-linux/include' '-D' '__CUDACC_VER_BUILD__=176' '-D' '__CUDACC_VER_MINOR__=0' '-D' '__CUDACC_VER_MAJOR__=9' '-D' 'GOOGLE_CUDA=1' '-D' '_GLIBCXX_USE_CXX11_ABI=0' '-include' 'cuda_runtime.h' '-m64' '-g' '-gdwarf-2' '-mtune=generic' '-march=x86-64'
 /usr/lib/gcc/x86_64-linux-gnu/5/cc1plus -E -quiet -v -I /home/christopher/anaconda3/lib/python3.6/site-packages/tensorflow/include -I /home/christopher/anaconda3/lib/python3.6/site-packages/tensorflow/include/external/nsync/public -I /usr/local/cuda-9.0/include -I /usr/local/cuda-9.0/bin/../targets/x86_64-linux/include -imultiarch x86_64-linux-gnu -D_GNU_SOURCE -D __CUDA_ARCH__=500 -D CUDA_DOUBLE_MATH_FUNCTIONS -D __CUDACC__ -D __NVCC__ -D __CUDACC_VER_BUILD__=176 -D __CUDACC_VER_MINOR__=0 -D __CUDACC_VER_MAJOR__=9 -D GOOGLE_CUDA=1 -D _GLIBCXX_USE_CXX11_ABI=0 -include cuda_runtime.h /tmp/christopher/returnn_tf_cache/ops/NativeLstm2/2ec60a4d21/NativeLstm2.cc -m64 -mtune=generic -march=x86-64 -std=c++11 -fPIC -g -gdwarf-2 -fworking-directory -O2 -fstack-protector-strong -Wformat -Wformat-security
ignoring duplicate directory "/usr/include/x86_64-linux-gnu/c++/5"
ignoring nonexistent directory "/usr/local/include/x86_64-linux-gnu"
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/5/../../../../x86_64-linux-gnu/include"
ignoring duplicate directory "/usr/local/cuda-9.0/bin/../targets/x86_64-linux/include"
#include "..." search starts here:
#include <...> search starts here:
 /home/christopher/anaconda3/lib/python3.6/site-packages/tensorflow/include
 /home/christopher/anaconda3/lib/python3.6/site-packages/tensorflow/include/external/nsync/public
 /usr/local/cuda-9.0/include
 /usr/include/c++/5
 /usr/include/x86_64-linux-gnu/c++/5
 /usr/include/c++/5/backward
 /usr/lib/gcc/x86_64-linux-gnu/5/include
 /usr/local/include
 /usr/lib/gcc/x86_64-linux-gnu/5/include-fixed
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
In file included from /home/christopher/anaconda3/lib/python3.6/site-packages/tensorflow/include/tensorflow/core/platform/stream_executor.h:26:0,
                 from /tmp/christopher/returnn_tf_cache/ops/NativeLstm2/2ec60a4d21/NativeLstm2.cc:18:
/home/christopher/anaconda3/lib/python3.6/site-packages/tensorflow/include/tensorflow/stream_executor/dso_loader.h:32:30: fatal error: cuda/cuda_config.h: No such file or directory
compilation terminated.
# --error 0x1 --
```

This problem does not occur for TF 1.4, or 1.6 to 1.8
To clarify, this is independent of issue #115, it happens even if I checkout a commit before 6d59402fbe60814792e19c7a9216f2097edd90d7


Discussion:
Can you check what you have at NativeLstm2.cc:18? It should be some include, but which one?
Also, check the code for `tf_blas_gemm_workaround`, which we currently use in this case:

      tf_blas_gemm_workaround = TFUtil.tf_version_tuple() < (1, 5, 0)

Maybe the version there is wrong, and should be (1,6,0)?
Line 18 of NativeLstm2.cc is `#include "tensorflow/core/platform/stream_executor.h"`

If I change the check for `tf_blas_gemm_workaround` to `(1, 6, 0)`, as well as the check in NativeOp.cpp to 

```
#if (TF_MAJOR_VERSION == 1 && TF_MINOR_VERSION >= 6) || (TF_MAJOR_VERSION > 1)
#define TF_issue_6602_workaround 0
#else
#define TF_issue_6602_workaround 1
#endif
```

it works. 
So it turns out that this issue and #115 really are connected, and both arise from the usage of `perftools::gputools::DeviceMemory<T>` before this support was added to TF. 

Resolving patch:
diff --git a/TFNativeOp.py b/TFNativeOp.py
--- a/TFNativeOp.py
+++ b/TFNativeOp.py
@@ -50,7 +50,7 @@ class OpMaker(object):
   """
   with_cuda = None  # type: None|bool
   # https://github.com/tensorflow/tensorflow/issues/6602
-  tf_blas_gemm_workaround = TFUtil.tf_version_tuple() < (1, 5, 0)
+  tf_blas_gemm_workaround = TFUtil.tf_version_tuple() < (1, 6, 0)
   global_lock = RLock()
   mod_cache = {}  # cache_key -> mod
   op_cache = {}  # cache_key -> op


Test patch: