Issue 1:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):

No

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

Windows 10

TensorFlow installed from (source or binary):

source

TensorFlow version (use command below):

r1.5

Python version:

3.6

GCC/Compiler version (if compiling from source):

Visual Studio 2017

CUDA/cuDNN version:

9.1

GPU model and memory:

1080Ti

Exact command to reproduce:

Cmake Command:

cmake -G "Visual Studio 15 2017 Win64" -T host=x64 -DCMAKE_BUILD_TYPE="
Release" -DSWIG_EXECUTABLE='C:\ProgramData\Chocolatey\bin\swig.exe' -Dtensorflow_ENABLE_GPU=ON -Dtensorflow_CUDA_VERSION
=9.1 -Dtensorflow_CUDNN_VERSION=7 -Dtensorflow_WIN_CPU_SIMD_OPTIONS="/arch:AVX2" -DCUDA_CUDART_LIBRARY=D:\NVIDIA\CUDA\v9
.1 -DCUDNN_HOME='D:\NVIDIA\CUDA\v9.1' ..

Build command

"C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\amd64\MSBuild.exe" /m:4 /p:Configuration=Release .\tf_core_gpu_kernels.vcxproj

Describe the problem

Cmake creates a bad command to send to nvcc.exe

In tf_core_gpu_kernels_generated_adjust_contrast_op_gpu.cu.cc.obj.Release.cmake:202L

there is an error with the resulting command.

There is issues with spacing, ";" in between arguments and others.

Command Ran

C:/NVIDIA/CUDA/v9.1/bin/nvcc.exe -M -D__CUDACC__ D:/tensorflow/tensorflow/core/kernels/adjust_contrast_op_gpu.cu.cc -o D:/tensorflow/tensorflow/contrib/cmake/build/CMakeFiles/tf_core_gpu_kernels.dir/__/__/core/kernels/tf_core_gpu_kernels_generated_adjust_contrast_op_gpu.cu.cc.obj.NVCC-depend -ccbin;C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/bin -m64;-DSQLITE_OMIT_LOAD_EXTENSION;-DEIGEN_AVOID_STL_ARRAY;-DNOMINMAX;-D_WIN32_WINNT=0x0A00;-DLANG_CXX11;-DCOMPILER_MSVC;-DWIN32;-DOS_WIN;-D_MBCS;-DWIN64;-DWIN32_LEAN_AND_MEAN;-DNOGDI;-DPLATFORM_WINDOWS;-DTENSORFLOW_USE_EIGEN_THREADPOOL;-DEIGEN_HAS_C99_MATH;-DTF_COMPILE_LIBRARY;-DGRPC_ARES=0;-DTF_USE_SNAPPY;-DGOOGLE_CUDA=1;-DTF_EXTRA_CUDA_CAPABILITIES=6.1 -Xcompiler;,"/DWIN32","/D_WINDOWS","/W3","/GR","/EHsc","/MP","/arch:AVX2","/MD","/O2","/Ob2","/DNDEBUG","/D_ITERATOR_DEBUG_LEVEL=0" -gencode;arch=compute_61,code="sm_61,compute_61";--include-path;D:/tensorflow/tensorflow/contrib/cmake/build/Release;--expt-relaxed-constexpr;-ftz=true; -DNVCC -IC:/NVIDIA/CUDA/v9.1/include ;-ID:/tensorflow ;-ID:/tensorflow/tensorflow/contrib/cmake/build ;-ID:/tensorflow/tensorflow/contrib/cmake/build/external/zlib_archive ;-ID:/tensorflow/tensorflow/contrib/cmake/build/external/gif_archive/giflib-5.1.4 ;-ID:/tensorflow/tensorflow/contrib/cmake/build/external/png_archive ;-ID:/tensorflow/tensorflow/contrib/cmake/build/external/jpeg_archive ;-ID:/tensorflow/tensorflow/contrib/cmake/build/external/lmdb ;-ID:/tensorflow/tensorflow/contrib/cmake/build/external/eigen_archive ;-ID:/tensorflow/third_party/eigen3 ;-ID:/tensorflow/tensorflow/contrib/cmake/build/gemmlowp/src/gemmlowp ;-ID:/tensorflow/tensorflow/contrib/cmake/build/jsoncpp/src/jsoncpp ;-ID:/tensorflow/tensorflow/contrib/cmake/build/external/farmhash_archive ;-ID:/tensorflow/tensorflow/contrib/cmake/build/external/farmhash_archive/util ;-ID:/tensorflow/tensorflow/contrib/cmake/build/external/highwayhash ;-ID:/tensorflow/tensorflow/contrib/cmake/build/cub/src/cub ;-ID:/tensorflow/tensorflow/contrib/cmake/build/external/nsync/public ;-ID:/tensorflow/tensorflow/contrib/cmake/build/protobuf/src/protobuf/src ;-ID:/tensorflow/tensorflow/contrib/cmake/build/re2/install/include ;-ID:/tensorflow/tensorflow/contrib/cmake/build/external/sqlite ;-ID:/tensorflow/tensorflow/contrib/cmake/build/grpc/src/grpc/include ;-ID:/tensorflow/tensorflow/contrib/cmake/build/snappy/src/snappy ;-IC:/NVIDIA/CUDA/v9.1 ;-IC:/NVIDIA/CUDA/v9.1/extras/CUPTI/include ;-ID:/tensorflow/third_party/gpus

Multable invalid cmake varables

${CCBIN} = -ccbin;C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/bin
${nvcc_flags} = -m64;-DSQLITE_OMIT_LOAD_EXTENSION;-DEIGEN_AVOID_STL_ARRAY;-DNOMINMAX;-D_WIN32_WINNT=0x0A00;-DLANG_CXX11;-DCOMPILER_MSVC;-DWIN32;-DOS_WIN;-D_MBCS;-DWIN64;-DWIN32_LEAN_AND_MEAN;-DNOGDI;-DPLATFORM_WINDOWS;-DTENSORFLOW_USE_EIGEN_THREADPOOL;-DEIGEN_HAS_C99_MATH;-DTF_COMPILE_LIBRARY;-DGRPC_ARES=0;-DTF_USE_SNAPPY;-DGOOGLE_CUDA=1;-DTF_EXTRA_CUDA_CAPABILITIES=6.1
${nvcc_host_compiler_flags} = -Xcompiler;,"/DWIN32","/D_WINDOWS","/W3","/GR","/EHsc","/MP","/arch:AVX2","/MD","/O2","/Ob2","/DNDEBUG","/D_ITERATOR_DEBUG_LEVEL=0"
${depends_CUDA_NVCC_FLAGS} = -gencode;arch=compute_61,code="sm_61,compute_61";--include-path;D:/tensorflow/tensorflow/contrib/cmake/build/Release;--expt-relaxed-constexpr;-ftz=true
${CUDA_NVCC_INCLUDE_ARGS} = -IC:/NVIDIA/CUDA/v9.1/include ;-ID:/tensorflow ;-ID:/tensorflow/tensorflow/contrib/cmake/build ;-ID:/tensorflow/tensorflow/contrib/cmake/build/external/zlib_archive ;-ID:/tensorflow/tensorflow/contrib/cmake/build/external/gif_archive/giflib-5.1.4 ;-ID:/tensorflow/tensorflow/contrib/cmake/build/external/png_archive ;-ID:/tensorflow/tensorflow/contrib/cmake/build/external/jpeg_archive ;-ID:/tensorflow/tensorflow/contrib/cmake/build/external/lmdb ;-ID:/tensorflow/tensorflow/contrib/cmake/build/external/eigen_archive ;-ID:/tensorflow/third_party/eigen3 ;-ID:/tensorflow/tensorflow/contrib/cmake/build/gemmlowp/src/gemmlowp ;-ID:/tensorflow/tensorflow/contrib/cmake/build/jsoncpp/src/jsoncpp ;-ID:/tensorflow/tensorflow/contrib/cmake/build/external/farmhash_archive ;-ID:/tensorflow/tensorflow/contrib/cmake/build/external/farmhash_archive/util ;-ID:/tensorflow/tensorflow/contrib/cmake/build/external/highwayhash ;-ID:/tensorflow/tensorflow/contrib/cmake/build/cub/src/cub ;-ID:/tensorflow/tensorflow/contrib/cmake/build/external/nsync/public ;-ID:/tensorflow/tensorflow/contrib/cmake/build/protobuf/src/protobuf/src ;-ID:/tensorflow/tensorflow/contrib/cmake/build/re2/install/include ;-ID:/tensorflow/tensorflow/contrib/cmake/build/external/sqlite ;-ID:/tensorflow/tensorflow/contrib/cmake/build/grpc/src/grpc/include ;-ID:/tensorflow/tensorflow/contrib/cmake/build/snappy/src/snappy ;-IC:/NVIDIA/CUDA/v9.1 ;-IC:/NVIDIA/CUDA/v9.1/extras/CUPTI/include ;-ID:/tensorflow/third_party/gpus

Should be

${CCBIN} = -ccbin "C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/bin"
${nvcc_flags} = -m64 -DSQLITE_OMIT_LOAD_EXTENSION -DEIGEN_AVOID_STL_ARRAY -DNOMINMAX -D_WIN32_WINNT=0x0A00 -DLANG_CXX11 -DCOMPILER_MSVC -DWIN32 -DOS_WIN -D_MBCS -DWIN64 -DWIN32_LEAN_AND_MEAN -DNOGDI -DPLATFORM_WINDOWS -DTENSORFLOW_USE_EIGEN_THREADPOOL -DEIGEN_HAS_C99_MATH -DTF_COMPILE_LIBRARY -DGRPC_ARES=0 -DTF_USE_SNAPPY -DGOOGLE_CUDA=1 -DTF_EXTRA_CUDA_CAPABILITIES=6.1
${nvcc_host_compiler_flags} = -Xcompiler "/DWIN32,/D_WINDOWS,/W3,/GR,/EHsc,/MP,/arch:AVX2,/MD,/O2,/Ob2,/DNDEBUG,/D_ITERATOR_DEBUG_LEVEL=0"
${depends_CUDA_NVCC_FLAGS} = -gencode arch=compute_61,code=\"sm_61,compute_61\" --include-path D:/tensorflow/tensorflow/contrib/cmake/build/Release --expt-relaxed-constexpr -ftz=true
${CUDA_NVCC_INCLUDE_ARGS} = -IC:/NVIDIA/CUDA/v9.1/include -ID:/tensorflow -ID:/tensorflow/tensorflow/contrib/cmake/build -ID:/tensorflow/tensorflow/contrib/cmake/build/external/zlib_archive -ID:/tensorflow/tensorflow/contrib/cmake/build/external/gif_archive/giflib-5.1.4 -ID:/tensorflow/tensorflow/contrib/cmake/build/external/png_archive -ID:/tensorflow/tensorflow/contrib/cmake/build/external/jpeg_archive -ID:/tensorflow/tensorflow/contrib/cmake/build/external/lmdb -ID:/tensorflow/tensorflow/contrib/cmake/build/external/eigen_archive -ID:/tensorflow/third_party/eigen3 -ID:/tensorflow/tensorflow/contrib/cmake/build/gemmlowp/src/gemmlowp -ID:/tensorflow/tensorflow/contrib/cmake/build/jsoncpp/src/jsoncpp -ID:/tensorflow/tensorflow/contrib/cmake/build/external/farmhash_archive -ID:/tensorflow/tensorflow/contrib/cmake/build/external/farmhash_archive/util -ID:/tensorflow/tensorflow/contrib/cmake/build/external/highwayhash -ID:/tensorflow/tensorflow/contrib/cmake/build/cub/src/cub -ID:/tensorflow/tensorflow/contrib/cmake/build/external/nsync/public -ID:/tensorflow/tensorflow/contrib/cmake/build/protobuf/src/protobuf/src -ID:/tensorflow/tensorflow/contrib/cmake/build/re2/install/include -ID:/tensorflow/tensorflow/contrib/cmake/build/external/sqlite -ID:/tensorflow/tensorflow/contrib/cmake/build/grpc/src/grpc/include -ID:/tensorflow/tensorflow/contrib/cmake/build/snappy/src/snappy -IC:/NVIDIA/CUDA/v9.1 -IC:/NVIDIA/CUDA/v9.1/extras/CUPTI/include -ID:/tensorflow/third_party/gpus

Issue 2:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):

No

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

Windows 10

TensorFlow installed from (source or binary):

Source

TensorFlow version (use command below):

1.5.0-dev20180103

25d2752

Python version:

3.5.4

Bazel version (if compiling from source):

GCC/Compiler version (if compiling from source):

CUDA/cuDNN version:

9.0 with cuDNN 7

GPU model and memory:

GeForce GTX 1060 6GB

Exact command to reproduce:

Open Visual Studio x64 Native Tools Command Prompt with admin rights.

using MSBuild 15.5.180.51428,

cmake .. -A x64 -DCMAKE_BUILD_TYPE=Release ^
-DSWIG_EXECUTABLE=C:/tools/swigwin-3.0.12/swig.exe ^
-DPYTHON_EXECUTABLE=C:\Users\csemp\AppData\Local\Programs\Python\Python35\python.exe ^
-DPYTHON_LIBRARIES=C:\Users\csemp\AppData\Local\Programs\Python\Python35\libs\python35.lib ^
-Dtensorflow_ENABLE_GPU=ON ^
-DCUDNN_HOME="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0" ^
-Dtensorflow_WIN_CPU_SIMD_OPTIONS=/arch:AVX ^
-Dtensorflow_BUILD_CC_EXAMPLE=OFF

set PreferredToolArchitecture=x64
"C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\amd64\MSBuild.exe" /m:2 /p:Configuration=Release tf_python_build_pip_package.vcxproj /v:diag > diag.log

Describe the problem

The build fails with the error

133>CustomBuild: (TargetId:8893)
CMake Error at tf_core_gpu_kernels_generated_adjust_contrast_op_gpu.cu.cc.obj.Release.cmake:222 (message): (TaskId:3323)
Error generating (TaskId:3323)
C:/Users/csemp/dev/tensorflowbuild/tensorflow/tensorflow/contrib/cmake/build/CMakeFiles/tf_core_gpu_kernels.dir/__/__/core/kernels/Release/tf_core_gpu_kernels_generated_adjust_contrast_op_gpu.cu.cc.obj (TaskId:3323)
(TaskId:3323)
(TaskId:3323)

It also has the error

(ClCompile target) ->
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.12.25827\include\algorithm(2417): error C2678: binary '*': no operator found which takes a left-hand operand of type 'const tensorflow::boosted_trees::utils::`anonymous-namespace'::IndicesRowIterator' (or there is no acceptable conversion) (compiling source file C:\Users\csemp\dev\tensorflowbuild2\tensorflow\tensorflow\contrib\boosted_trees\lib\utils\sparse_column_iterable.cc) [C:\Users\csemp\dev\tensorflowbuild2\tensorflow\tensorflow\contrib\cmake\build\tf_core_kernels.vcxproj]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.12.25827\include\algorithm(2417): error C2100: illegal indirection (compiling source file C:\Users\csemp\dev\tensorflowbuild2\tensorflow\tensorflow\contrib\boosted_trees\lib\utils\sparse_column_iterable.cc) [C:\Users\csemp\dev\tensorflowbuild2\tensorflow\tensorflow\contrib\cmake\build\tf_core_kernels.vcxproj]

the latter of which can be fixed by a simple change, as noted in #15925 (comment) .

Creating this new issue per @gunan's request

#14691 (comment)

Source code / logs

The log file is way too big to post apparently. I'm open to suggestions