Issue 1:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Well, I added patches

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10, 64 bit

TensorFlow installed from (source or binary): Source

TensorFlow version (use command below): 1.8.0

Python version: 3.5, 3.6

Bazel version (if compiling from source): 1.12.0, 1.14.0

GCC/Compiler version (if compiling from source): MSVC 2015 Update v3

CUDA/cuDNN version: 9.0/7.1

GPU model and memory: GeForce GT 430

Exact command to reproduce: ;-)

Description

I wanted to compile tensorflow with GPU support on Windows and I went through the ordeal (I was finally able to build it, but I'll come to that later). I always followed the stuff done in: https://github.com/tensorflow/tensorflow/blob/v1.8.0/tensorflow/tools/ci_build/

This is how it went:

First, I attempted the CPU build.

Initial attempt was with CMake. It took ~3hours, all went fine upto the final linking phase, there link.exe died with:

(Lib target) ->
tf_core_kernels.dir\Release\tf_core_kernels.lib : fatal error LNK1248: image size (100039B2C) exceeds maximum allowable size (FFFFFFFF)

I was lost here, I tried tinkering with some CMake build options, but all resulted into the same error. So, I gave up CMake.

Then, I attempted to build with bazel 1.12.0.

I was using the mingw64 shell, so first, I had to patch tensorflow/tools/pip_package/build_pip_package.sh . I also sent a PR #18953

Surprisingly, the build went fine after that, and then I tried to run the tests. All were successful, except tensorflow/python/kernel_tests/boosted_trees:training_ops_test, probably because it was not being run serially. Even for running the tests, I had to patch tensorflow/contrib/tensorboard/plugins/trace/trace.py and I also sent a PR #18954

Now comes the real deal. The GPU build.

Again, I attempted the build with CMake. It took ~3hours and then link.exe failed with the same error as before. I spent quite some time reading about it and couldn't figure it out as most solutions said 'break your huge lib into smaller libs'. I am still baffled as to why the CI job at http://ci.tensorflow.org/job/tf-master-win-gpu-cmake/ doesn't die with the same problem. I am using the same compiler, same linker,

and my machine is powerful enough. So, I just gave up again on CMake.

Going back to bazel now. By this time, bazel 1.14.0 was was released. So I decided to use that instead. I had to apply the two patches as before, but life isn't so easy.

Next big hurdle: #17067 .

So I got @dtrebbien 's patch from https://github.com/dtrebbien/protobuf/commit/50f552646ba1de79e07562b41f3999fe036b4fd0 and made changes to tensorflow/workspace.bzl to apply it to all protobuf checkouts.

Next big hurdle: NCCL doesn't seem to be officially supported on Windows.

But bazel is still trying to build nccl related stuff. I had to patch stuff again, to disable all references to nccl in the BUILD files all over the place. 😿

Next big hurdle:

Shared libraries under tensorflow/contrib/rnn/python/ops/ failed at link phase with the error:

Creating library bazel-out/host/bin/tensorflow/contrib/rnn/python/ops/_gru_ops.ifso and object bazel-out/host/bin/tensorflow/contrib/rnn/python/ops/_gru_ops.exp
blas_gemm.o : error LNK2019: unresolved external symbol "public: class perftools::gputools::Stream & __cdecl perftools::gputools::Stream::ThenBlasGemm(enum perftools::gputools::blas::Transpose,enum perftools::gputools::blas::Transpose,unsigned __int64,unsigned __int64,unsigned __int64,float,class perftools::gputools::DeviceMemory<float> const &,int,class perftools::gputools::DeviceMemory<float> const &,int,float,class perftools::gputools::DeviceMemory<float> *,int)" (?ThenBlasGemm@Stream@gputools@perftools@@QEAAAEAV123@W4Transpose@blas@23@0_K11MAEBV?$DeviceMemory@M@23@H2HMPEAV623@H@Z) referenced in function "public: void __cdecl tensorflow::functor::TensorCuBlasGemm<float>::operator()(class tensorflow::OpKernelContext *,bool,bool,unsigned __int64,unsigned __int64,unsigned __int64,float,float const *,int,float const *,int,float,float *,int)" (??R?$TensorCuBlasGemm@M@functor@tensorflow@@QEAAXPEAVOpKernelContext@2@_N1_K22MPEBMH3HMPEAMH@Z)
blas_gemm.o : error LNK2019: unresolved external symbol "public: class perftools::gputools::Stream & __cdecl perftools::gputools::Stream::ThenBlasGemm(enum perftools::gputools::blas::Transpose,enum perftools::gputools::blas::Transpose,unsigned __int64,unsigned __int64,unsigned __int64,double,class perftools::gputools::DeviceMemory<double> const &,int,class perftools::gputools::DeviceMemory<double> const &,int,double,class perftools::gputools::DeviceMemory<double> *,int)" (?ThenBlasGemm@Stream@gputools@perftools@@QEAAAEAV123@W4Transpose@blas@23@0_K11NAEBV?$DeviceMemory@N@23@H2HNPEAV623@H@Z) referenced in function "public: void __cdecl tensorflow::functor::TensorCuBlasGemm<double>::operator()(class tensorflow::OpKernelContext *,bool,bool,unsigned __int64,unsigned __int64,unsigned __int64,double,double const *,int,double const *,int,double,double *,int)" (??R?$TensorCuBlasGemm@N@functor@tensorflow@@QEAAXPEAVOpKernelContext@2@_N1_K22NPEBNH3HNPEANH@Z)

Then I stumbled upon #15013 and saw that another person had faced the same issue quite some time ago and the OP's solution was add all the .lib from tensorflow and cuda . I can't do that. So, I went looking for where this symbol comes from, and where should it be. Took me a while to realize what's happening and that's when I came across the workaround

of using intermediate interface shared object file at https://github.com/tensorflow/tensorflow/blob/v1.8.0/tensorflow/tensorflow.bzl#L1228-L1237. Apparently, this didn't have enough symbols for the kernels in tensorflow/contrib/rnn/python/ops/. So I started looking for where I could find them and ended up adding:

clean_dep("//tensorflow/stream_executor:stream_executor_impl"), to that list and the link error went away.

Next hurdle: #20088

The compiler doesn't like

https://github.com/tensorflow/tensorflow/blob/v1.8.0/tensorflow/contrib/fused_conv/kernels/fused_conv2d_bias_activation_op.cc#L46 when it was already being done at https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/kernels/conv_ops_gpu.h#L189 . So I commented out the typdef in fused_conv2d_bias_activation_op.cc and tried building it. The error went away, but link.exe cried again at the end, because it didn't know where to get the symbol for GetCudnnWorkspaceLimit at https://github.com/tensorflow/tensorflow/blob/v1.8.0/tensorflow/contrib/fused_conv/kernels/fused_conv2d_bias_activation_op.cc#L519 . After some searching, I found out that this symbol comes from https://github.com/tensorflow/tensorflow/blob/v1.8.0/tensorflow/core/kernels/conv_ops.cc#L456-L471 . So I tried making it part of the intermediary additional_deps_impl.ifso import lib, but no matter what, it always ended up bloating it with symbols > 64k (it went upto ~72k, when I tried adding some deps which might have that symbol). So I gave up on this contrib kernel at this point and when I looked at the CMake build (I could be wrong here) I didn't find it building this kernel either. So I just ended up commenting out all references to fused_conv in the BUILD files.

Next hurdle: Bazel had made some breaking changes in 1.13.0

I came across at bazelbuild/bazel#4583 and then applied two other patches locally to get around that problem.

Next hurdle: Tests won't run.

Most of them want to import nccl. 😭 So I ended up faking it by creating a blank module and then the tests ran fine. I did have to tell bazel to ignore the contrib/lite subset -//${PY_TEST_DIR}/tensorflow/contrib/lite/....

For the lost souls who attempt at building tensorflow with GPU support, I hope this issue might be helpful. All the patches and the entire build can be found here: https://github.com/AnacondaRecipes/aggregate/blob/459dee0989e0c7cc4ab66c49d3d7605cddbb1bc3/tensorflow-gpu-base-feedstock/recipe/meta.yaml

Issue 2:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10 1803

TensorFlow installed from (source or binary): source

TensorFlow version (use command below): v1.11.0-rc1

Python version: 3.6.5

Bazel version (if compiling from source): 0.15.2

GCC/Compiler version (if compiling from source): vs2017 15.8 / cl.exe 19.15.26726

CUDA/cuDNN version: 9.2.148.1/7.2.1

GPU model and memory: 1080ti 11GB

Exact command to reproduce:

bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package
bazel-bin\tensorflow\tools\pip_package\build_pip_package C:/tmp/tensorflow_pkg

Describe the problem

If build artifact large than 4GB, it can't generate the python package.

A easy way to reproduce the issue

Please note that each additional compute capability significantly increases your build time and binary size. [Default is: 3.5,7.0]: 3.0,3.2,3.5,5.0,5.2,5.3

related link:

#20332 (comment)

#22382

https://github.com/bazelbuild/bazel/blob/0.17.1/third_party/ijar/zip.cc#L74

Source code / logs

Uncompressed input jar has size ???, which exceeds the maximum supported output size 4294967295.
Assuming that ijar will be smaller and hoping for the best.

Unzipping simple_console_for_windows.zip to create runfiles tree...
[./bazel-bin/tensorflow/tools/pip_package/simple_console_for_windows.zip]
End-of-central-directory signature not found. Either this file is not
a zipfile, or it constitutes one disk of a multi-part archive. In the
latter case the central directory and zipfile comment will be found on
the last disk(s) of this archive.
unzip: cannot find zipfile directory in one of ./bazel-bin/tensorflow/tools/pip_package/simple_console_for_windows.zip or
./bazel-bin/tensorflow/tools/pip_package/simple_console_for_windows.zip.zip, and cannot find ./bazel-bin/tensorflow/tools/pip_package/simple_console_for_windows.zip.ZIP, period.