Issue 1:
See below. Please let me know what I can do to provide more information for you. I am working on pulling out the offending code into a standalone py file to replicate the bug elsewhere, but it might take a few days.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): arch linux (LTS kernel 4.9.44-1)

TensorFlow installed from (source or binary): source

TensorFlow version (use command below): 1.3 (master branch as of commit 566d167c)

Python version: 3.6.2

Bazel version (if compiling from source): 0.5.2-2

CUDA/cuDNN version: 8.0.61-2/6.0.21-2

GPU model and memory: 1080 GTX Ti

Exact command to reproduce:

I don't have exact source for you yet of a trimmed down example. I will see if I can put one together. Generally, the steps are

Generate a few TFRecord files

Instantiate a few TFRecordDataset from the files.

Perform necessary pre-processing on TFRecordDataset entries for each dataset, then on the dataset itself (.cache, .repeat, .shuffle, .batch, etc) For example, we might have datasets train_dataset, validate_dataset, and test_dataset.

Initialize each dataset (.make_initializable_iterator)

Train a model using an input tensor x of

train_phase = tf.placeholder(dtype=tf.int32, shape=())
x = tf.case([(tf.equal(train_phase, 1), lambda: train_dataset),
(tf.equal(train_phase, 2), lambda: validate_dataset),
(tf.equal(train_phase,3), lambda: test_dataset)],
default=lambda: train_dataset)
# ... construct model_op using tensor x as input ...
# ... call the initializer ...

# train for a bunch
for i in ...
sess.run(main_op, feed_dict={train_phase: 1})
# validate
for i in ...
sess.run(main_op, feed_dict={train_phase: 2})
# test
for i in ...
sess.run(main_op, feed_dict={train_phase: 3})

Describe the problem

I am using a single tensor x to represent my input values for all three phases of my model training (training, validation, and testing). To alternate between them without duplicating entire graphs, I set the tensor to be conditional on the value of train_phase (again, train, validate, or test) by using tf.case.

If the above steps are followed, my system will eventually (non-deterministically) crash during the training/validating/testing of the model. This is independent of the dataset used (I am using well-scrubbed versions of NYUv2, CIFAR-10, KITTI, etc.) and does not occur in particular records. The error is always long the lines of

DataLossError (see above for traceback): Checksum does not match: stored 3054820559 vs. calculated on the restored bytes 2831969904
[[Node: input_pipeline/IteratorGetNext_1 = IteratorGetNext[output_shapes=[[?,?,64,64]], output_types=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](input_pipeline/Iterator_1)]]
[[Node: add_1/_69 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:1", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_69_add_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:1"]()]]
Checksum does not match: stored 3054820559 vs. calculated on the restored bytes 2831969904
[[Node: input_pipeline/IteratorGetNext_1 = IteratorGetNext[output_shapes=[[?,?,64,64]], output_types=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](input_pipeline/Iterator_1)]]
[[Node: add_1/_69 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:1", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_69_add_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:1"]()]]

It can occur during any phase (that is, on any branch of the tf.case). It occurs with multiple versions of CUDA 8 and TF (at least since TFRecordDataset came out). It occurs with single or multiple GPUS. It occurs with different Linux kernel versions (multiple variants of 4.10, 4.11, 4.12, including the 4.9 TLS branch).

However, it does NOT occur if I don't use tf.case (that is, I use a single TFRecordDataset tensor as the input, i.e., only do training and skip validate/test).

I would assume this is a complication with the CUDA drivers, as I occasionally (though not always) get kernel panics at the same time. I will also occasionally (though not always) get errors like malloc(): smallbin double linked list corrupted.

I should also add that I've already tested for hardware issues and seem to have ruled all of them out (memory failures, HD/SSD cables, motherboard, PSU spikes, etc.).

Source code / logs

The full trace/output is

Caught unexpected exception during training: Checksum does not match: stored 3054820559 vs. calculated on the restored bytes 2831969904
[[Node: input_pipeline/IteratorGetNext_1 = IteratorGetNext[output_shapes=[[?,?,64,64]], output_types=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](input_pipeline/Iterator_1)]]
[[Node: add_1/_69 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:1", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_69_add_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:1"]()]]

Caused by op 'input_pipeline/IteratorGetNext_1', defined at:
File "train.py", line 266, in <module>
datasets = hem.get_datasets(args)
File "/mnt/research/projects/autoencoders/hem/util/data.py", line 64, in get_datasets
x = iterator.get_next()
File "/usr/lib/python3.6/site-packages/tensorflow/contrib/data/python/ops/dataset_ops.py", line 311, in get_next
name=name))
File "/usr/lib/python3.6/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 698, in iterator_get_next
output_shapes=output_shapes, name=name)
File "/usr/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3046, in create_op
op_def=op_def)
File "/usr/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1604, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

DataLossError (see above for traceback): Checksum does not match: stored 3054820559 vs. calculated on the restored bytes 2831969904
[[Node: input_pipeline/IteratorGetNext_1 = IteratorGetNext[output_shapes=[[?,?,64,64]], output_types=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](input_pipeline/Iterator_1)]]
[[Node: add_1/_69 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:1", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_69_add_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:1"]()]]
Checksum does not match: stored 3054820559 vs. calculated on the restored bytes 2831969904
[[Node: input_pipeline/IteratorGetNext_1 = IteratorGetNext[output_shapes=[[?,?,64,64]], output_types=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](input_pipeline/Iterator_1)]]
[[Node: add_1/_69 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:1", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_69_add_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:1"]()]]

Issue 2:
I try to write my own op and I have installed TensorFlow 0.12.0 with GPU support on Linux.

This code fails:

#include "tensorflow/core/platform/stream_executor.h"

With error:

fatal error: tensorflow/stream_executor/lib/status.h: No such file or directory

That files does not exists.

Some more include files seem to be missing. When grepping for DeviceMemory in the include path, the only file it finds is include/tensorflow/core/util/stream_executor_util.h.