Issue 1:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:

TensorFlow installed from (source or binary): binary

TensorFlow version (use command below): v1.14.0-rc1-22-gaf24dc91b5 1.14.0

Python version: 3.6.7

GCC/Compiler version (if compiling from source): 8.2.0

CUDA/cuDNN version: cuda-10.0

GPU model and memory:

Describe the current behavior

tf.matmul on tensorflow-gpu gave wrong results. Here is a simplified version of the code.

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

r = [[ 1.0, 0.0], [0.0, 1.0]]

x, y = np.meshgrid(list(range(400)), list(range(400)))
coords = np.stack([x,y],-1).reshape((400,400,2,1))
coords = tf.convert_to_tensor(coords,dtype=tf.float32)

r1 = tf.constant(r)

newCoords = tf.matmul(r1, coords)

sess = tf.Session()
ret = sess.run(newCoords,feed_dict={r1:r})

plt.matshow(ret[:,:,0,0])
plt.show()

When I ran it on my tensorflow-gpu, here is the result:

Looks like it stops computing halfway through and gave the rest 0 as a result.

Describe the expected behavior

Here is the result with CPU:

Below is my old post. Initially, I thought the problem was related to TFRecord, but seems like this problem occurs without even using tfrecord too. For completeness, I keep the old example code with tfrecord.

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

def parser(serialized_example):
fs = tf.io.parse_single_example(
serialized_example,
features={ "r": tf.FixedLenFeature([4], tf.float32) })
fs["r"] = tf.reshape(fs["r"], [2, 2])
return fs

r = [[ 1.0, 0.0],[0.0, 1.0]]

with tf.io.TFRecordWriter("cc.test") as tfrecord_writer:
feature = {"r": tf.train.Feature(float_list=tf.train.FloatList(value=np.array(r).flatten() ))}
example = tf.train.Example(features=tf.train.Features(feature=feature))
tfrecord_writer.write(example.SerializeToString())
dataset = tf.data.TFRecordDataset(["cc.test"])
dataset = dataset.map(parser).repeat().make_one_shot_iterator()
features = dataset.get_next()

x, y = tf.meshgrid(list(range(400)), list(range(400)))
coords = tf.stack([x, y], -1) #(h,w,2)
coords = tf.expand_dims(tf.cast(coords,tf.float32),-1) #(h,w,2,1)

r1 = features["r"]
r2 = tf.constant(r)

newCoords = tf.matmul(r1, coords)

sess = tf.Session()
ret = sess.run(newCoords[:,:,0,0])
plt.matshow(ret)
plt.show()

The code will create "cc.test" file to save the variable r and load it as r1. Then matmul r1 with some big varibles.

Issue 2:
Can't compile tensorflow on Ubuntu with Skylake CPU (i7-7820X). I get the following error:

ERROR: /home/Dmitry/Private/Kaggle/tensorflow/tensorflow/core/kernels/BUILD:1369:1: C++ compilation of rule '//tensorflow/core/kernels:range_sampler' failed (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command
(cd /home/Dmitry/.cache/bazel/_bazel_Dmitry/a3a733ca304216612b7ed32f720401f1/execroot/org_tensorflow && \
exec env - \
CUDA_TOOLKIT_PATH=/usr/local/cuda \
CUDNN_INSTALL_PATH=/usr/local/cuda-8.0 \
GCC_HOST_COMPILER_PATH=/usr/bin/gcc-4.9 \
PWD=/proc/self/cwd \
PYTHON_BIN_PATH=/usr/bin/python \
PYTHON_LIB_PATH=/usr/local/lib/python2.7/dist-packages \
TF_CUDA_CLANG=0 \
TF_CUDA_COMPUTE_CAPABILITIES=6.1 \
TF_CUDA_VERSION=8.0 \
TF_CUDNN_VERSION=6 \
TF_NEED_CUDA=1 \
TF_NEED_OPENCL=0 \
external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections '-march=native' '-std=c++11' '-march=native' '-fabi-version=0' '-D_GLIBCXX_USE_CXX11_ABI=0' -MD -MF bazel-out/local_linux-opt/bin/tensorflow/core/kernels/_objs/range_sampler/tensorflow/core/kernels/range_sampler.pic.d '-frandom-seed=bazel-out/local_linux-opt/bin/tensorflow/core/kernels/_objs/range_sampler/tensorflow/core/kernels/range_sampler.pic.o' -fPIC -DEIGEN_MPL2_ONLY -DTENSORFLOW_USE_JEMALLOC -DSNAPPY -iquote . -iquote bazel-out/local_linux-opt/genfiles -iquote external/nsync -iquote bazel-out/local_linux-opt/genfiles/external/nsync -iquote external/bazel_tools -iquote bazel-out/local_linux-opt/genfiles/external/bazel_tools -iquote external/jemalloc -iquote bazel-out/local_linux-opt/genfiles/external/jemalloc -iquote external/protobuf_archive -iquote bazel-out/local_linux-opt/genfiles/external/protobuf_archive -iquote external/eigen_archive -iquote bazel-out/local_linux-opt/genfiles/external/eigen_archive -iquote external/local_config_sycl -iquote bazel-out/local_linux-opt/genfiles/external/local_config_sycl -iquote external/gif_archive -iquote bazel-out/local_linux-opt/genfiles/external/gif_archive -iquote external/jpeg -iquote bazel-out/local_linux-opt/genfiles/external/jpeg -iquote external/com_googlesource_code_re2 -iquote bazel-out/local_linux-opt/genfiles/external/com_googlesource_code_re2 -iquote external/farmhash_archive -iquote bazel-out/local_linux-opt/genfiles/external/farmhash_archive -iquote external/fft2d -iquote bazel-out/local_linux-opt/genfiles/external/fft2d -iquote external/highwayhash -iquote bazel-out/local_linux-opt/genfiles/external/highwayhash -iquote external/png_archive -iquote bazel-out/local_linux-opt/genfiles/external/png_archive -iquote external/zlib_archive -iquote bazel-out/local_linux-opt/genfiles/external/zlib_archive -iquote external/snappy -iquote bazel-out/local_linux-opt/genfiles/external/snappy -isystem external/nsync/public -isystem bazel-out/local_linux-opt/genfiles/external/nsync/public -isystem external/bazel_tools/tools/cpp/gcc3 -isystem external/jemalloc/include -isystem bazel-out/local_linux-opt/genfiles/external/jemalloc/include -isystem external/protobuf_archive/src -isystem bazel-out/local_linux-opt/genfiles/external/protobuf_archive/src -isystem external/eigen_archive -isystem bazel-out/local_linux-opt/genfiles/external/eigen_archive -isystem external/gif_archive/lib -isystem bazel-out/local_linux-opt/genfiles/external/gif_archive/lib -isystem external/farmhash_archive/src -isystem bazel-out/local_linux-opt/genfiles/external/farmhash_archive/src -isystem external/png_archive -isystem bazel-out/local_linux-opt/genfiles/external/png_archive -isystem external/zlib_archive -isystem bazel-out/local_linux-opt/genfiles/external/zlib_archive -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fno-canonical-system-headers -c tensorflow/core/kernels/range_sampler.cc -o bazel-out/local_linux-opt/bin/tensorflow/core/kernels/_objs/range_sampler/tensorflow/core/kernels/range_sampler.pic.o)
In file included from external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/Core:378:0,
from external/eigen_archive/unsupported/Eigen/CXX11/Tensor:14,
from ./third_party/eigen3/unsupported/Eigen/CXX11/Tensor:1,
from ./tensorflow/core/lib/random/random_distributions.h:27,
from ./tensorflow/core/lib/random/simple_philox.h:24,
from ./tensorflow/core/lib/random/distribution_sampler.h:38,
from ./tensorflow/core/kernels/range_sampler.h:23,
from tensorflow/core/kernels/range_sampler.cc:16:
external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/Core/arch/AVX512/PacketMath.h: In function 'Packet Eigen::internal::ploaddup(const typename Eigen::internal::unpacket_traits<Packet>::type*) [with Packet = __vector(16) float; typename Eigen::internal::unpacket_traits<Packet>::type = float]':
external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/Core/arch/AVX512/PacketMath.h:469:77: error: '_mm512_castsi512_ps' was not declared in this scope
__m512 even_elements = _mm512_castsi512_ps(_mm512_cvtepu32_epi64(low_half));
^
external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/Core/arch/AVX512/PacketMath.h: In function 'Packet Eigen::internal::ploaddup(const typename Eigen::internal::unpacket_traits<Packet>::type*) [with Packet = __vector(8) double; typename Eigen::internal::unpacket_traits<Packet>::type = double]':
external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/Core/arch/AVX512/PacketMath.h:478:56: error: '_mm512_insertf64x2' was not declared in this scope
x = _mm512_insertf64x2(x, _mm_loaddup_pd(&from[0]), 0);
^
In file included from ./tensorflow/core/platform/default/logging.h:24:0,
from ./tensorflow/core/platform/logging.h:25,
from ./tensorflow/core/lib/core/status.h:25,
from ./tensorflow/core/kernels/range_sampler.h:21,
from tensorflow/core/kernels/range_sampler.cc:16:
./tensorflow/core/platform/default/logging.h: In instantiation of 'std::string* tensorflow::internal::Check_LEImpl(const T1&, const T2&, const char*) [with T1 = long unsigned int; T2 = long long int; std::string = std::basic_string<char>]':
tensorflow/core/kernels/range_sampler.cc:86:5: required from here
./tensorflow/core/platform/default/logging.h:230:35: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
TF_DEFINE_CHECK_OP_IMPL(Check_LE, <= )
^
./tensorflow/core/platform/macros.h:79:29: note: in definition of macro 'TF_PREDICT_TRUE'
#define TF_PREDICT_TRUE(x) (x)
^
./tensorflow/core/platform/default/logging.h:230:1: note: in expansion of macro 'TF_DEFINE_CHECK_OP_IMPL'
TF_DEFINE_CHECK_OP_IMPL(Check_LE, <= )
^
./tensorflow/core/platform/default/logging.h: In instantiation of 'std::string* tensorflow::internal::Check_EQImpl(const T1&, const T2&, const char*) [with T1 = long long int; T2 = long unsigned int; std::string = std::basic_string<char>]':
tensorflow/core/kernels/range_sampler.cc:244:3: required from here
./tensorflow/core/platform/default/logging.h:228:25: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
== ) // Compilation error with CHECK_EQ(NULL, x)?
^
./tensorflow/core/platform/macros.h:79:29: note: in definition of macro 'TF_PREDICT_TRUE'
#define TF_PREDICT_TRUE(x) (x)
^
./tensorflow/core/platform/default/logging.h:227:1: note: in expansion of macro 'TF_DEFINE_CHECK_OP_IMPL'
TF_DEFINE_CHECK_OP_IMPL(Check_EQ,
^
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 1.894s, Critical Path: 1.15s
FAILED: Build did NOT complete successfully

OR the following error:

ERROR: /home/Dmitry/Private/Kaggle/tensorflow/tensorflow/core/BUILD:1299:1: C++ compilation of rule '//tensorflow/core:lib_internal' failed (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command
(cd /home/Dmitry/.cache/bazel/_bazel_Dmitry/a3a733ca304216612b7ed32f720401f1/execroot/org_tensorflow && \
exec env - \
CUDA_TOOLKIT_PATH=/usr/local/cuda \
CUDNN_INSTALL_PATH=/usr/local/cuda-8.0 \
GCC_HOST_COMPILER_PATH=/usr/bin/gcc-4.9 \
PWD=/proc/self/cwd \
PYTHON_BIN_PATH=/usr/bin/python3 \
PYTHON_LIB_PATH=/usr/local/lib/python3.5/dist-packages \
TF_CUDA_CLANG=0 \
TF_CUDA_COMPUTE_CAPABILITIES=6.1 \
TF_CUDA_VERSION=8.0 \
TF_CUDNN_VERSION=6 \
TF_NEED_CUDA=1 \
TF_NEED_OPENCL=0 \
external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections '-march=native' '-std=c++11' '-march=native' '-fabi-version=0' '-D_GLIBCXX_USE_CXX11_ABI=0' -MD -MF bazel-out/local_linux-py3-opt/bin/tensorflow/core/_objs/lib_internal/tensorflow/core/lib/random/distribution_sampler.pic.d '-frandom-seed=bazel-out/local_linux-py3-opt/bin/tensorflow/core/_objs/lib_internal/tensorflow/core/lib/random/distribution_sampler.pic.o' -fPIC -DEIGEN_MPL2_ONLY -DTENSORFLOW_USE_JEMALLOC -DSNAPPY -iquote . -iquote bazel-out/local_linux-py3-opt/genfiles -iquote external/nsync -iquote bazel-out/local_linux-py3-opt/genfiles/external/nsync -iquote external/bazel_tools -iquote bazel-out/local_linux-py3-opt/genfiles/external/bazel_tools -iquote external/jemalloc -iquote bazel-out/local_linux-py3-opt/genfiles/external/jemalloc -iquote external/protobuf_archive -iquote bazel-out/local_linux-py3-opt/genfiles/external/protobuf_archive -iquote external/eigen_archive -iquote bazel-out/local_linux-py3-opt/genfiles/external/eigen_archive -iquote external/local_config_sycl -iquote bazel-out/local_linux-py3-opt/genfiles/external/local_config_sycl -iquote external/gif_archive -iquote bazel-out/local_linux-py3-opt/genfiles/external/gif_archive -iquote external/jpeg -iquote bazel-out/local_linux-py3-opt/genfiles/external/jpeg -iquote external/com_googlesource_code_re2 -iquote bazel-out/local_linux-py3-opt/genfiles/external/com_googlesource_code_re2 -iquote external/farmhash_archive -iquote bazel-out/local_linux-py3-opt/genfiles/external/farmhash_archive -iquote external/fft2d -iquote bazel-out/local_linux-py3-opt/genfiles/external/fft2d -iquote external/highwayhash -iquote bazel-out/local_linux-py3-opt/genfiles/external/highwayhash -iquote external/png_archive -iquote bazel-out/local_linux-py3-opt/genfiles/external/png_archive -iquote external/zlib_archive -iquote bazel-out/local_linux-py3-opt/genfiles/external/zlib_archive -iquote external/snappy -iquote bazel-out/local_linux-py3-opt/genfiles/external/snappy -isystem external/nsync/public -isystem bazel-out/local_linux-py3-opt/genfiles/external/nsync/public -isystem external/bazel_tools/tools/cpp/gcc3 -isystem external/jemalloc/include -isystem bazel-out/local_linux-py3-opt/genfiles/external/jemalloc/include -isystem external/protobuf_archive/src -isystem bazel-out/local_linux-py3-opt/genfiles/external/protobuf_archive/src -isystem external/eigen_archive -isystem bazel-out/local_linux-py3-opt/genfiles/external/eigen_archive -isystem external/gif_archive/lib -isystem bazel-out/local_linux-py3-opt/genfiles/external/gif_archive/lib -isystem external/farmhash_archive/src -isystem bazel-out/local_linux-py3-opt/genfiles/external/farmhash_archive/src -isystem external/png_archive -isystem bazel-out/local_linux-py3-opt/genfiles/external/png_archive -isystem external/zlib_archive -isystem bazel-out/local_linux-py3-opt/genfiles/external/zlib_archive -DEIGEN_AVOID_STL_ARRAY -Iexternal/gemmlowp -Wno-sign-compare -fno-exceptions '-ftemplate-depth=900' '-DGOOGLE_CUDA=1' -msse3 -pthread -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fno-canonical-system-headers -c tensorflow/core/lib/random/distribution_sampler.cc -o bazel-out/local_linux-py3-opt/bin/tensorflow/core/_objs/lib_internal/tensorflow/core/lib/random/distribution_sampler.pic.o)
In file included from external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/Core:378:0,
from external/eigen_archive/unsupported/Eigen/CXX11/Tensor:14,
from ./third_party/eigen3/unsupported/Eigen/CXX11/Tensor:1,
from ./tensorflow/core/lib/random/random_distributions.h:27,
from ./tensorflow/core/lib/random/simple_philox.h:24,
from ./tensorflow/core/lib/random/distribution_sampler.h:38,
from tensorflow/core/lib/random/distribution_sampler.cc:16:
external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/Core/arch/AVX512/PacketMath.h: In function 'Packet Eigen::internal::ploaddup(const typename Eigen::internal::unpacket_traits<Packet>::type*) [with Packet = __vector(16) float; typename Eigen::internal::unpacket_traits<Packet>::type = float]':
external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/Core/arch/AVX512/PacketMath.h:469:77: error: '_mm512_castsi512_ps' was not declared in this scope
__m512 even_elements = _mm512_castsi512_ps(_mm512_cvtepu32_epi64(low_half));
^
external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/Core/arch/AVX512/PacketMath.h: In function 'Packet Eigen::internal::ploaddup(const typename Eigen::internal::unpacket_traits<Packet>::type*) [with Packet = __vector(8) double; typename Eigen::internal::unpacket_traits<Packet>::type = double]':
external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/Core/arch/AVX512/PacketMath.h:478:56: error: '_mm512_insertf64x2' was not declared in this scope
x = _mm512_insertf64x2(x, _mm_loaddup_pd(&from[0]), 0);
^
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 183.759s, Critical Path: 16.18s
FAILED: Build did NOT complete successfully

Compilation command:

bazel build --config=opt --config=cuda --cxxopt="-fabi-version=0" //tensorflow/tools/pip_package:build_pip_package --verbose_failures

Is it possible to disable AVX512 but leave all other native optimization parameters?