Q:

Tensorflow model lm_1b segmentation fault on model initialization

I'm trying to run the tensorflow lm_1b model but unfortunately I'm getting a segmentation fault when trying to load the model.   Specifically I get the fault when running...
bazel-bin/lm_1b/lm_1b_eval --mode sample --prefix "I love that I" --pbtxt data/graph-2016-09-10.pbtxt --vocab_file data/vocab-2016-09-10.txt --ckpt 'data/ckpt-*'
The fault occurs after the line sess.run(t['states_init']) in lm_1b_eval.py is executed. 
The backtrace of the 19GB core files gives...
#0  0x00007fd13a12e648 in Eigen::internal::gemm_pack_lhs<float, long, Eigen::internal::TensorContractionSubMapper<float, long, 1, Eigen::TensorEvaluator<Eigen::TensorReshapingOp<Eigen::DSizes<long, 2> const, Eigen::TensorMap<Eigen::Tensor<float const, 4, 1, long>, 16, Eigen::MakePointer> const> const, Eigen::ThreadPoolDevice>, Eigen::array<long, 1ul>, Eigen::array<long, 1ul>, 16, true, false, 0, Eigen::MakePointer>, 48, 16, 0, false, false>::operator()(float*, Eigen::internal::TensorContractionSubMapper<float, long, 1, Eigen::TensorEvaluator<Eigen::TensorReshapingOp<Eigen::DSizes<long, 2> const, Eigen::TensorMap<Eigen::Tensor<float const, 4, 1, long>, 16, Eigen::MakePointer> const> const, Eigen::ThreadPoolDevice>, Eigen::array<long, 1ul>, Eigen::array<long, 1ul>, 16, true, false, 0, Eigen::MakePointer> const&, long, long, long, long) ()
   from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#1  0x00007fd13a2bc4f0 in void Eigen::TensorContractionEvaluatorBase<Eigen::TensorEvaluator<Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorReshapingOp<Eigen::DSizes<long, 2> const, Eigen::TensorImagePatchOp<-1l, -1l, Eigen::TensorMap<Eigen::Tensor<float const, 4, 1, long>, 16, Eigen::MakePointer> const> const> const, Eigen::TensorReshapingOp<Eigen::DSizes<long, 2> const, Eigen::TensorMap<Eigen::Tensor<float const, 4, 1, long>, 16, Eigen::MakePointer> const> const> const, Eigen::ThreadPoolDevice> >::evalGemm<true, true, false, 0>(float*) const () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#2  0x00007fd13a2cb970 in void Eigen::TensorEvaluator<Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorReshapingOp<Eigen::DSizes<long, 2> const, Eigen::TensorImagePatchOp<-1l, -1l, Eigen::TensorMap<Eigen::Tensor<float const, 4, 1, long>, 16, Eigen::MakePointer> const> const> const, Eigen::TensorReshapingOp<Eigen::DSizes<long, 2> const, Eigen::TensorMap<Eigen::Tensor<float const, 4, 1, long>, 16, Eigen::MakePointer> const> const> const, Eigen::ThreadPoolDevice>::evalProduct<true, true, false, 0>(float*) const () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#3  0x00007fd13a2cd2e0 in Eigen::internal::TensorExecutor<Eigen::TensorAssignOp<Eigen::TensorMap<Eigen::Tensor<float, 4, 1, long>, 16, Eigen::MakePointer>, Eigen::TensorReshapingOp<Eigen::DSizes<long, 4> const, Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorReshapingOp<Eigen::DSizes<long, 2> const, Eigen::TensorImagePatchOp<-1l, -1l, Eigen::TensorMap<Eigen::Tensor<float const, 4, 1, long>, 16, Eigen::MakePointer> const> const> const, Eigen::TensorReshapingOp<Eigen::DSizes<long, 2> const, Eigen::TensorMap<Eigen::Tensor<float const, 4, 1, long>, 16, Eigen::MakePointer> const> const> const> const> const, Eigen::ThreadPoolDevice, true>::run(Eigen::TensorAssignOp<Eigen::TensorMap<Eigen::Tensor<float, 4, 1, long>, 16, Eigen::MakePointer>, Eigen::TensorReshapingOp<Eigen::DSizes<long, 4> const, Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorReshapingOp<Eigen::DSizes<long, 2> const, Eigen::TensorImagePatchOp<-1l, -1l, Eigen::TensorMap<Eigen::Tensor<float const, 4, 1, long>, 16, Eigen::MakePointer> const> const> const, Eigen::TensorReshapingOp<Eigen::DSizes<long, 2> const, Eigen::TensorMap<Eigen::Tensor<float const, 4, 1, long>, 16, Eigen::MakePointer> const> const> const> const> const&, Eigen::ThreadPoolDevice const&) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#4  0x00007fd13a2cd8cc in void tensorflow::functor::SpatialConvolutionFunc<Eigen::ThreadPoolDevice, Eigen::TensorMap<Eigen::Tensor<float const, 4, 1, long>, 16, Eigen::MakePointer>, Eigen::TensorMap<Eigen::Tensor<float const, 4, 1, long>, 16, Eigen::MakePointer>, Eigen::TensorMap<Eigen::Tensor<float, 4, 1, long>, 16, Eigen::MakePointer> >(Eigen::ThreadPoolDevice const&, Eigen::TensorMap<Eigen::Tensor<float, 4, 1, long>, 16, Eigen::MakePointer>, Eigen::TensorMap<Eigen::Tensor<float const, 4, 1, long>, 16, Eigen::MakePointer>, Eigen::TensorMap<Eigen::Tensor<float const, 4, 1, long>, 16, Eigen::MakePointer>, int, int, Eigen::PaddingType const&) ()
   from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#5  0x00007fd13a2fc9e4 in tensorflow::(anonymous namespace)::LaunchGeneric<Eigen::ThreadPoolDevice, float>::operator()(tensorflow::OpKernelContext*, tensorflow::Tensor const&, tensorflow::Tensor const&, int, int, tensorflow::Padding const&, tensorflow::Tensor*, tensorflow::TensorFormat) [clone .isra.1016] ()
   from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#6  0x00007fd13a2fce7a in tensorflow::LaunchConv2DOp<Eigen::ThreadPoolDevice, float>::operator()(tensorflow::OpKernelContext*, bool, bool, tensorflow::Tensor const&, tensorflow::Tensor const&, int, int, int, int, tensorflow::Padding const&, tensorflow::Tensor*, tensorflow::TensorFormat) ()
   from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#7  0x00007fd13a3020f7 in tensorflow::Conv2DOp<Eigen::ThreadPoolDevice, float>::Compute(tensorflow::OpKernelContext*) ()
   from /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow_internal.so
#8  0x00007fd135bcf34c in tensorflow::ThreadPoolDevice::Compute(tensorflow::OpKernel*, tensorflow::OpKernelContext*) ()
   from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so
#9  0x00007fd135b98e8b in tensorflow::(anonymous namespace)::ExecutorState::Process(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long) ()
   from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so
#10 0x00007fd135b8a645 in std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (tensorflow::(anonymous namespace)::ExecutorState::*)(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long)> (tensorflow::(anonymous namespace)::ExecutorState*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long)> >::_M_invoke(std::_Any_data const&) () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so
#11 0x00007fd1357d7671 in Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) ()
   from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so
#12 0x00007fd1357d5437 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&)
    () from /usr/local/lib/python2.7/dist-packages/tensorflow/python/../libtensorflow_framework.so
#13 0x00007fd12c8e10ff in ?? ()  from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#14 0x00007fd167b6e7fc in start_thread (arg=0x7fd007fff700) at pthread_create.c:465
#15 0x00007fd16789bb5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone. S:95

I've setup the example verbatim as described in the Readme and downloaded the ckpt* files twice but I'm still getting the fault.   Originally I had Tensorflow 1.4 but I'm still getting the fault after upgrading to 1.6-rc1.   My build of TF is follows the default options with the exception that I've enabled Cuda (v9.0/cudnn 7) and disabled Hadoop/AmazonS3.   I've been using Keras/TF regularly without issue on my own Keras models. 
I'm running this on a i9-7940x with 64GB RAM and a TitanX (Maxwell) so resources shouldn't be an issue.   I'm running Ubuntu 17.10 (nvcc setup to use gcc 6). 
The core doesn't help me much.   Any ideas on how to proceed would be appreciated.   If this appears to be a bug with the model itself I'll submit it to TF model Issues but I'd like to debug it here as far as possible before I submit the bug. 

A:

After some additional investigation is appears that the problem is with my custom built version of Tensorflow.   Installing tensorflow-gpu 1.5 from pip resolved the issue. 
The issue is that Tensorflow is does not work properly with the AVX-512 instructions.   Using the flag -march=native on SkylakeX enables these extensions and creates issues with the code.   The issues still persists as of TF 1.12.   To fix the issue, compile for an earlier architecture or explicitly turn of avx512 with "-march=native -mno-avx512f".