Issue 1:
Have I written custom code: Yes

OS Platform and Distribution: Linux Ubuntu 17.04

TensorFlow installed from: source

TensorFlow version: 1.6.0-rc1

Python version: 3.6

Bazel version: 0.10

GCC/Compiler version: 6.0

CUDA/cuDNN version: CUDA 9.1 cuDNN 7.0.5

GPU model and memory: NVIDIA Titan Z 12GB

Exact command to reproduce: multi_gpu_model(model, gpus=2)

Just upgraded from rc0 to rc1 of release 1.6.0 and I'm now getting the following crash when running the multi_gpu_model function (was working fine with rc0):

parallel_model = multi_gpu_model(model, gpus=2)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/_impl/keras/utils/training_utils.py", line 207, in multi_gpu_model

return Model(model.inputs, merged)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/_impl/keras/engine/topology.py", line 694, in init

self._init_graph_network(*args, **kwargs)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/_impl/keras/engine/topology.py", line 733, in _init_graph_network

if layer.is_placeholder:

AttributeError: 'Lambda' object has no attribute 'is_placeholder'

I rolled back to the 1.5 branch and I'm not having any issues running multi_gpu_model there.

Issue 2:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

Linux Ubuntu 16.04

TensorFlow installed from (source or binary):

Built from source

TensorFlow version (use command below):

v1.6.0-rc0-19-gecec1d8 1.6.0-rc1

master

Python version: 3.5.2

Bazel version (if compiling from source): 0.10.0

GCC/Compiler version (if compiling from source): 5.4.0

CUDA/cuDNN version: 9.1.85/7.0.5.15

GPU model and memory:

GeForce GTX 1080 Ti

totalMemory: 10.91GiB freeMemory: 8.36GiB

Describe the problem

If tf1.6 is compiled with --march=native then running inference on a large model ends up with crash with cryptic stack trace, 100% reproducible, no matter with or without CUDA (whether CUDA_VISIBLE_DEVICES="" or absent).

If compiled without avx512 support (i.e. -O3 -msse4.2 -mavx2 -mfma only), then everything works fine.

TF emits a warning though:

2018-02-15 14:03:54.237530: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX512F

I checked another issue with 64 byte alignment (#15588), but it does not help.

Here is a crash trace, if it may help.

(gdb) bt
#0 0x00007fdea3c5c9d5 in Eigen::internal::gemm_pack_lhs<float, long, Eigen::internal::TensorContractionSubMapper<float, long, 1, Eigen::TensorEvaluator<Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::ThreadPoolDevice>, Eigen::array<long, 1ul>, Eigen::array<long, 1ul>, 16, true, false, 0, Eigen::MakePointer>, 48, 16, 0, false, false>::operator()(float*, Eigen::internal::TensorContractionSubMapper<float, long, 1, Eigen::TensorEvaluator<Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::ThreadPoolDevice>, Eigen::array<long, 1ul>, Eigen::array<long, 1ul>, 16, true, false, 0, Eigen::MakePointer> const&, long, long, long, long) () from /home/zbr/.local/lib/python3.5/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#1 0x00007fdea3cd0fc4 in Eigen::TensorEvaluator<Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const> const, Eigen::ThreadPoolDevice>::Context<Eigen::internal::gemm_pack_lhs<float, long, Eigen::internal::TensorContractionSubMapper<float, long, 1, Eigen::TensorEvaluator<Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::ThreadPoolDevice>, Eigen::array<long, 1ul>, Eigen::array<long, 1ul>, 16, true, false, 0, Eigen::MakePointer>, 48, 16, 0, false, false>, Eigen::internal::gemm_pack_rhs<float, long, Eigen::internal::TensorContractionSubMapper<float, long, 0, Eigen::TensorEvaluator<Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::ThreadPoolDevice>, Eigen::array<long, 1ul>, Eigen::array<long, 1ul>, 16, true, false, 0, Eigen::MakePointer>, 4, 0, false, false>, Eigen::internal::gebp_kernel<float, float, long, Eigen::internal::blas_data_mapper<float, long, 0, 0>, 48, 4, false, false>, Eigen::internal::TensorContractionInputMapper<float, long, 1, Eigen::TensorEvaluator<Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::ThreadPoolDevice>, Eigen::array<long, 1ul>, Eigen::array<long, 1ul>, 16, true, false, 0, Eigen::MakePointer>, Eigen::internal::TensorContractionInputMapper<float, long, 0, Eigen::TensorEvaluator<Eigen::TensorMap<Eigen::Tensor<float const, 2, 1, long>, 16, Eigen::MakePointer> const, Eigen::ThreadPoolDevice>, Eigen::array<long, 1ul>, Eigen::array<long, 1ul>, 16, true, false, 0, Eigen::MakePointer>, Eigen::internal::blas_data_mapper<float, long, 0, 0> >::enqueue_packing_helper(long, long, long, bool) ()
from /home/zbr/.local/lib/python3.5/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#2 0x00007fdea06232d1 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) ()
from /home/zbr/.local/lib/python3.5/site-packages/tensorflow/python/../libtensorflow_framework.so
#3 0x00007fdea06210e7 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) ()
from /home/zbr/.local/lib/python3.5/site-packages/tensorflow/python/../libtensorflow_framework.so
#4 0x00007fde96afbc80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007fded77d76ba in start_thread (arg=0x7fdeb87d8700) at pthread_create.c:333
#6 0x00007fded750d41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb)