Issue 1:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): YES

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOS 7.2 (and also Windows 10)

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A

TensorFlow installed from (source or binary): binary (via pip install tensorflow-gpu)

TensorFlow version (use command below): v1.11.0-0-gc19e29306c 1.11.0

Python version: 3.6.6

Bazel version (if compiling from source): N/A

GCC/Compiler version (if compiling from source): N/A

CUDA/cuDNN version: 9.0/7.1.4

GPU model and memory: NVIDIA GeForce GTX 1070 (8 GiB)

Exact command to reproduce: python nmsstest.py; see below for the content of nmstest.py

Describe the problem

The code in the following section, which calls tf.image.non_max_suppression() in tf.while_loop() many times, crashes abnormally.

Crash reason (and loop count) varies from time to time, for example:

F tensorflow/core/common_runtime/bfc_allocator.cc:384] Check failed: h != kInvalidChunkHandle at loop i == 140

F tensorflow/core/common_runtime/bfc_allocator.cc:462] Check failed: c->in_use() && (c->bin_num == kInvalidBinNum) at loop i == 76

Bus error at loop i == 34

heap corruption reported by libc (see the following section) at loop i == 35

sometimes it crashes sliently without any logs (on Windows)

I notice that:

it's since TensorFlow 1.11.0rc0; TF 1.10.1 was okay

it also reproduces on Windows

it also reproduces on CPU version (pip install tensorflow)

it does not reproduce if num_threads=1; calling tf.image.non_max_suppression() in parallel seems the trigger

even when I gave a fixed seed to tf.random_uniform(), crash cause varied

Source code / logs

The code to reproduce the problem is as follows:

import tensorflow as tf

if __name__ == '__main__':
# crashes iff num_threads > 1 on TensorFlow >= 1.11.rc0
num_threads = 10

top_k = 1
batch_size = 32
num_boxes = 10000
boxes_op = tf.random_uniform((batch_size,num_boxes,4), 0, 1)
scores_op = tf.random_uniform((batch_size,num_boxes), 0, 1)
indices_op = tf.while_loop(
(lambda b, ta: True),
(lambda b, ta: (b+1, ta.write(b, tf.image.non_max_suppression(boxes_op[b], scores_op[b], top_k, iou_threshold=0.3)))),
(tf.constant(0),
tf.TensorArray(tf.int32, size=batch_size, infer_shape=False, element_shape=(top_k,))),
back_prop=False,
parallel_iterations=num_threads,
maximum_iterations=batch_size)[1].stack()

with tf.Session() as session:
for i in range(1000):
indices = session.run(indices_op)
print(f'#{i}: {indices.shape}')

$ python nmstest.py
2018-09-28 13:18:42.760545: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-09-28 13:18:43.123328: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-09-28 13:18:43.124208: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1411] Found device 0 with properties:
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.683
pciBusID: 0000:01:00.0
totalMemory: 7.93GiB freeMemory: 7.83GiB
2018-09-28 13:18:43.124232: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1490] Adding visible gpu devices: 0
2018-09-28 13:18:43.351813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-09-28 13:18:43.351847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] 0
2018-09-28 13:18:43.351859: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0: N
2018-09-28 13:18:43.352078: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1103] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7558 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
#0: (32, 1)
#1: (32, 1)
#2: (32, 1)
#3: (32, 1)
#4: (32, 1)
#5: (32, 1)
#6: (32, 1)
#7: (32, 1)
#8: (32, 1)
#9: (32, 1)
#10: (32, 1)
#11: (32, 1)
#12: (32, 1)
#13: (32, 1)
#14: (32, 1)
#15: (32, 1)
#16: (32, 1)
#17: (32, 1)
#18: (32, 1)
#19: (32, 1)
#20: (32, 1)
#21: (32, 1)
#22: (32, 1)
#23: (32, 1)
#24: (32, 1)
#25: (32, 1)
#26: (32, 1)
#27: (32, 1)
#28: (32, 1)
#29: (32, 1)
#30: (32, 1)
#31: (32, 1)
#32: (32, 1)
#33: (32, 1)
#34: (32, 1)
#35: (32, 1)
*** Error in `python': malloc(): smallbin double linked list corrupted: 0x00007fb6c001c960 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f5e4)[0x7fb9816b75e4]
/lib64/libc.so.6(+0x82d00)[0x7fb9816bad00]
/lib64/libc.so.6(__libc_malloc+0x4c)[0x7fb9816bd84c]
/home/hyabe/anaconda3/envs/work/bin/../lib/libstdc++.so.6(_Znwm+0x16)[0x7fb936087084]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZN10tensorflow15OpKernelContext15allocate_outputEiRKNS_11TensorShapeEPPNS_6TensorENS_19AllocatorAttributesE+0x48)[0x7fb93f0c0608]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZN10tensorflow15OpKernelContext15allocate_outputEiRKNS_11TensorShapeEPPNS_6TensorE+0xc5)[0x7fb93f0c0785]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so(_ZN10tensorflow7MergeOp7ComputeEPNS_15OpKernelContextE+0xa4)[0x7fb9423854b4]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZN10tensorflow13BaseGPUDevice13ComputeHelperEPNS_8OpKernelEPNS_15OpKernelContextE+0x37d)[0x7fb93f23ac9d]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZN10tensorflow13BaseGPUDevice7ComputeEPNS_8OpKernelEPNS_15OpKernelContextE+0x8d)[0x7fb93f23b1dd]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(+0x63a4bc)[0x7fb93f2844bc]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(+0x63ae2a)[0x7fb93f284e2a]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZN5Eigen26NonBlockingThreadPoolTemplIN10tensorflow6thread16EigenEnvironmentEE10WorkerLoopEi+0x21a)[0x7fb93f2f296a]
/home/hyabe/anaconda3/envs/work/lib/python3.6/site-packages/tensorflow/python/../libtensorflow_framework.so(_ZNSt17_Function_handlerIFvvEZN10tensorflow6thread16EigenEnvironment12CreateThreadESt8functionIS0_EEUlvE_E9_M_invokeERKSt9_Any_data+0x32)[0x7fb93f2f1a12]
/home/hyabe/anaconda3/envs/work/bin/../lib/libstdc++.so.6(+0xb8678)[0x7fb9360a2678]
/lib64/libpthread.so.0(+0x7e25)[0x7fb981a0ce25]
/lib64/libc.so.6(clone+0x6d)[0x7fb981736bad]
======= Memory map: ========
200000000-200200000 rw-s 00000000 00:05 25687 /dev/nvidiactl
200200000-200400000 ---p 00000000 00:00 0
200400000-200404000 rw-s 00000000 00:05 25687 /dev/nvidiactl
200404000-200600000 ---p 00000000 00:00 0
200600000-200a00000 rw-s 00000000 00:05 25687 /dev/nvidiactl
200a00000-201200000 ---p 00000000 00:00 0
201200000-201204000 rw-s 00000000 00:05 25687 /dev/nvidiactl
201204000-201400000 ---p 00000000 00:00 0
201400000-201800000 rw-s 00000000 00:05 25687 /dev/nvidiactl
201800000-201804000 rw-s 00000000 00:05 25687 /dev/nvidiactl
201804000-201a00000 ---p 00000000 00:00 0
201a00000-201e00000 rw-s 00000000 00:05 25687 /dev/nvidiactl
201e00000-201e04000 rw-s 00000000 00:05 25687 /dev/nvidiactl
201e04000-202000000 ---p 00000000 00:00 0
202000000-202400000 rw-s 00000000 00:05 25687 /dev/nvidiactl
202400000-202404000 rw-s 00000000 00:05 25687 /dev/nvidiactl
202404000-202600000 ---p 00000000 00:00 0
202600000-202a00000 rw-s 00000000 00:05 25687 /dev/nvidiactl
202a00000-202a04000 rw-s 00000000 00:05 25687 /dev/nvidiactl
202a04000-202c00000 ---p 00000000 00:00 0
202c00000-203000000 rw-s 00000000 00:05 25687 /dev/nvidiactl
203000000-203004000 rw-s 00000000 00:05 25687 /dev/nvidiactl
203004000-203200000 ---p 00000000 00:00 0
203200000-203600000 rw-s 00000000 00:05 25687 /dev/nvidiactl
203600000-203604000 rw-s 00000000 00:05 25687 /dev/nvidiactl
203604000-203800000 ---p 00000000 00:00 0
203800000-203c00000 rw-s 00000000 00:05 25687 /dev/nvidiactl
203c00000-203c04000 rw-s 00000000 00:05 25687 /dev/nvidiactl
203c04000-203e00000 ---p 00000000 00:00 0
203e00000-204200000 rw-s 00000000 00:05 25687 /dev/nvidiactl
204200000-204204000 rw-s 00000000 00:05 25687 /dev/nvidiactl
204204000-204400000 ---p 00000000 00:00 0
204400000-204800000 rw-s 00000000 00:05 25687 /dev/nvidiactl
204800000-204804000 rw-s 00000000 00:05 25687 /dev/nvidiactl
204804000-204a00000 ---p 00000000 00:00 0
204a00000-204e00000 rw-s 00000000 00:05 25687 /dev/nvidiactl
204e00000-204e04000 rw-s 00000000 00:05 25687 /dev/nvidiactl
204e04000-205000000 ---p 00000000 00:00 0
205000000-205400000 rw-s 00000000 00:05 25687 /dev/nvidiactl
205400000-205404000 rw-s 00000000 00:05 25687 /dev/nvidiactl
205404000-205600000 ---p 00000000 00:00 0
205600000-205a00000 rw-s 00000000 00:05 25687 /dev/nvidiactl
205a00000-205a04000 rw-s 00000000 00:05 25687 /dev/nvidiactl
205a04000-205c00000 ---p 00000000 00:00 0
205c00000-206000000 rw-s 00000000 00:05 25687 /dev/nvidiactl
206000000-206004000 rw-s 00000000 00:05 25687 /dev/nvidiactl
206004000-206200000 ---p 00000000 00:00 0
206200000-206600000 rw-s 00000000 00:05 25687 /dev/nvidiactl
206600000-206604000 rw-s 00000000 00:05 25687 /dev/nvidiactl
206604000-206800000 ---p 00000000 00:00 0
206800000-206c00000 rw-s 00000000 00:05 25687 /dev/nvidiactl
206c00000-206e00000 ---p 00000000 00:00 0
206e00000-207000000 rw-s 00000000 00:05 25687 /dev/nvidiactl
207000000-300200000 ---p 00000000 00:00 0
10000000000-10204000000 ---p 00000000 00:00 0
5596eb5dd000-5596eb89c000 r-xp 00000000 fd:02 86155766 /home/hyabe/anaconda3/envs/work/bin/python3.6
5596eba9c000-5596eba9f000 r--p 002bf000 fd:02 86155766 /home/hyabe/anaconda3/envs/work/bin/python3.6
5596eba9f000-5596ebb02000 rw-p 002c2000 fd:02 86155766 /home/hyabe/anaconda3/envs/work/bin/python3.6
5596ebb02000-5596ebb33000 rw-p 00000000 00:00 0
5596eda67000-5597016ee000 rw-p 00000000 00:00 0 [heap]
7fb698000000-7fb69803a000 rw-p 00000000 00:00 0
7fb69803a000-7fb69c000000 ---p 00000000 00:00 0
7fb69c000000-7fb69c021000 rw-p 00000000 00:00 0
7fb69c021000-7fb6a0000000 ---p 00000000 00:00 0
7fb6a0000000-7fb6a0036000 rw-p 00000000 00:00 0
7fb6a0036000-7fb6a4000000 ---p 00000000 00:00 0
7fb6a4000000-7fb6a4049000 rw-p 00000000 00:00 0
7fb6a4049000-7fb6a8000000 ---p 00000000 00:00 0
7fb6a8000000-7fb6a8039000 rw-p 00000000 00:00 0
7fb6a8039000-7fb6ac000000 ---p 00000000 00:00 0
7fb6b0000000-7fb6b0036000 rw-p 00000000 00:00 0
7fb6b0036000-7fb6b4000000 ---p 00000000 00:00 0
7fb6b4000000-7fb6b403b000 rw-p 00000000 00:00 0
7fb6b403b000-7fb6b8000000 ---p 00000000 00:00 0
7fb6b8000000-7fb6b8037000 rw-p 00000000 00:00 0
7fb6b8037000-7fb6bc000000 ---p 00000000 00:00 0
7fb6c0000000-7fb6c003c000 rw-p 00000000 00:00 0
7fb6c003c000-7fb6c4000000 ---p 00000000 00:00 0
7fb6c5400000-7fb6c5a00000 rw-p 00000000 00:00 0
7fb6c5ad4000-7fb6c5ad5000 ---p 00000000 00:00 0
7fb6c5ad5000-7fb6c6326000 rw-p 00000000 00:00 0
7fb6c6326000-7fb6c6327000 ---p 00000000 00:00 0
7fb6c6327000-7fb6c6b78000 rw-p 00000000 00:00 0
7fb6c6b78000-7fb6c6b79000 ---p 00000000 00:00 0
7fb6c6b79000-7fb6d6000000 rw-p 00000000 00:00 0
7fb6d6000000-7fb8ae800000 ---p 00000000 00:00 0
7fb8ae800000-7fb8aea00000 rw-s 00000000 00:04 31968 /dev/zero (deleted)
7fb8aea00000-7fb8aec00000 rw-s 00000000 00:04 29039 /dev/zero (deleted)
7fb8aec00000-7fb8b0000000 ---p 00000000 00:00 0
7fb8b0000000-7fb8b0021000 rw-p 00000000 00:00 0
7fb8b0021000-7fb8b4000000 ---p 00000000 00:00 0
7fb8b4000000-7fb8b4021000 rw-p 00000000 00:00 0
7fb8b4021000-7fb8b8000000 ---p 00000000 00:00 0
7fb8b8000000-7fb8bc000000 ---p 00000000 00:00 0
7fb8bc000000-7fb8bc021000 rw-p 00000000 00:00 0
7fb8bc021000-7fb8c0000000 ---p 00000000 00:00 0
7fb8c0000000-7fb8c0001000 rw-s 00000000 00:05 25688 /dev/nvidia0
7fb8c0001000-7fb8c0002000 rw-s 00000000 00:05 25688 /dev/nvidia0
7fb8c0002000-7fb8c0003000 rw-s 00000000 00:05 25688 /dev/nvidia0
7fb8c0003000-7fb8c0004000 rw-s 00000000 00:05 25688 /dev/nvidia0
7fb8c0004000-7fb8c0005000 rw-s 00000000 00:05 25688 /dev/nvidia0
7fb8c0005000-7fb8c0006000 rw-s 00000000 00:05 25688 /dev/nvidia0
7fb8c0006000-7fb8c0007000 rw-s 00000000 00:05 25688 /dev/nvidia0
7fb8c0007000-7fb8c0008000 rw-s 00000000 00:05 25688 /dev/nvidia0
7fb8c0008000-7fb8c0009000 rw-s 00000000 00:05 25688 /dev/nvidia0
7fb8c0009000-7fb8c000a000 rw-s 00000000 00:05 25688 /dev/nvidia0
7fb8c000a000-7fb8c000b000 rw-s 00000000 00:05 25688 /dev/nvidia0
7fb8c000b000-7fb8c000c000 rw-s 00000000 00:05 25688 /dev/nvidia0
7fb8c000c000-7fb8c000d000 rw-s 00000000 00:05 25688 /dev/nvidia0
7fb8c000d000-7fb8c000e000 rw-s 00000000 00:05 25688 /dev/nvidia0
7fb8c000e000-7fb8c000f000 rw-s 00000000 00:05 25688 /dev/nvidia0
7fb8c000f000-7fb8c0010000 rw-s 00000000 00:05 25688 /dev/nvidia0
7fb8c0010000-7fb8d0000000 ---p 00000000 00:00 0
7fb8d0000000-7fb8d0021000 rw-p 00000000 00:00 0
7fb8d0021000-7fb8d4000000 ---p 00000000 00:00 0
7fb8d4200000-7fb8d4600000 rw-p 00000000 00:00 0
7fb8d470a000-7fb8d470b000 ---p 00000000 00:00 0
7fb8d470b000-7fb8d4f5c000 rw-p 00000000 00:00 0
7fb8d4f5c000-7fb8d4f5d000 ---p 00000000 00:00 0
7fb8d4f5d000-7fb8d57ae000 rw-p 00000000 00:00 0
7fb8d57ae000-7fb8d57af000 ---p 00000000 00:00 0
7fb8d57af000-7fb8d6000000 rw-p 00000000 00:00 0
7fb8d6000000-7fb8d6200000 ---p 00000000 00:00 0
7fb8d6200000-7fb8d6400000 rw-s 00000000 00:04 31965 /dev/zero (deleted)
7fb8d6400000-7fb8d6600000 rw-s 00000000 00:05 25687 /dev/nvidiactl
7fb8d6600000-7fb8d6800000 rw-s 00000000 00:04 31966 /dev/zero (deleted)
7fb8d6800000-7fb8d6a00000 rw-s 00000000 00:05 25687 /dev/nvidiactl
7fb8d6a00000-7fb8d6c00000 ---p 00000000 00:00 0
7fb8d6c00000-7fb8d6ed6000 rw-s 00000000 00:05 25687 /dev/nvidiactl
7fb8d6ed6000-7fb8d8000000 ---p 00000000 00:00 0
7fb8d8000000-7fb8d8021000 rw-p 00000000 00:00 0
7fb8d8021000-7fb8dc000000 ---p 00000000 00:00 0
7fb8dc200000-7fb8dc600000 rw-p 00000000 00:00 0
7fb8dc70a000-7fb8dc70b000 ---p 00000000 00:00 0
7fb8dc70b000-7fb8dcf5c000 rw-p 00000000 00:00 0
7fb8dcf5c000-7fb8dcf5d000 ---p 00000000 00:00 0
7fb8dcf5d000-7fb8dd7ae000 rw-p 00000000 00:00 0
7fb8dd7ae000-7fb8dd7af000 ---p 00000000 00:00 0
7fb8dd7af000-7fb8de000000 rw-p 00000000 00:00 0
7fb8de000000-7fb8e4000000 ---p 00000000 00:00 0
7fb8e4000000-7fb8e4021000 rw-p 00000000 00:00 0
7fb8e4021000-7fb8e8000000 ---p 00000000 00:00 0
7fb8e8000000-7fb8e8021000 rw-p 00000000 00:00 0
7fb8e8021000-7fb8ec000000 ---p 00000000 00:00 0
7fb8ec000000-7fb8ec021000 rw-p 00000000 00:00 0
7fb8ec021000-7fb8f0000000 ---p 00000000 00:00 0
7fb8f0000000-7fb8f0021000 rw-p 00000000 00:00 0
7fb8f0021000-7fb8f4000000 ---p 00000000 00:00 0
7fb8f4200000-7fb8f4400000 rw-p 00000000 00:00 0
7fb8f45c2000-7fb8f45c3000 ---p 00000000 00:00 0
7fb8f45c3000-7fb8f4e14000 rw-p 00000000 00:00 0
7fb8f4e14000-7fb8f4e15000 ---p 00000000 00:00 0
7fb8f4e15000-7fb8f5666000 rw-p 00000000 00:00 0
7fb8f5666000-7fb8f5667000 ---p 00000000 00:00 0
7fb8f5667000-7fb8f5eb8000 rw-p 00000000 00:00 0
7fb8f5eb8000-7fb8f5eb9000 ---p 00000000 00:00 0
7fb8f5eb9000-7fb8f670a000 rw-p 00000000 00:00 0
7fb8f670a000-7fb8f670b000 ---p 00000000 00:00 0
7fb8f670b000-7fb8f6f5c000 rw-p 00000000 00:00 0
7fb8f6f5c000-7fb8f6f5d000 ---p 00000000 00:00 0
7fb8f6f5d000-7fb8f77ae000 rw-p 00000000 00:00 0
7fb8f77ae000-7fb8f77af00Aborted

Issue 2:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): archlinux

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:n/a

TensorFlow installed from (source or binary):binary

TensorFlow version (use command below):b'v1.9.0-rc2-5276-ge57874169f' 1.12.0-dev20181004

Python version:3.6

Bazel version (if compiling from source):n/a

GCC/Compiler version (if compiling from source):n/a

CUDA/cuDNN version:9.0

GPU model and memory:1080Ti

Exact command to reproduce:below

This code:

import tensorflow as tf
import numpy as np

def f(boxes, scores):
def f(X):
prob, box = X
output_shape = tf.shape(prob)
ids = tf.reshape(tf.where(prob > 0.05), [-1])
prob = tf.gather(prob, ids)
box = tf.gather(box, ids)
# prob = tf.Print(prob, [box, prob], summarize=100, message='boxandprob')
selection = tf.image.non_max_suppression(box, prob, 100, 0.5)
selection = tf.to_int32(tf.gather(ids, selection))
selection = tf.Print(selection, [ids, selection], summarize=100, message='ids_selection_2')
sorted_selection = -tf.nn.top_k(-selection, k=tf.size(selection))[0]
mask = tf.sparse_to_dense(
sparse_indices=sorted_selection,
output_shape=output_shape,
sparse_values=True,
default_value=False)
return mask

masks = tf.map_fn(f, (scores, boxes), dtype=tf.bool, parallel_iterations=10) # #cat x N
return masks

with tf.device('/gpu:0'):
boxes = tf.placeholder(tf.float32, (80, None, 4), name='boxes')
scores = tf.placeholder(tf.float32, (80, None), name='scores')
outs = f(boxes, scores)

config = tf.ConfigProto()
config.allow_soft_placement = True
sess = tf.Session(config=config)
data = dict(np.load('debug.npz'))
for k in range(1000):
sess.run(outs, feed_dict={boxes: data['boxes'].transpose(1, 0, 2)[1:, :, :], scores: data['scores'][:, 1:].T})
print(k)

causes segmentation fault on tf-nightly-gpu, as well as tensorflow-gpu==1.11.0. It works on 1.10.

It needs the data file debug.npz here:

debug.zip

Note:

I tested on two machines, an error happen in >90% runs.

The code was distilled from the bug report about MaskRCNN evaluation here. The original bug report does not always segfault, but occasionally crash with other different unreasonable TF internal errors, such as:

InvalidArgumentError (see above for traceback): scores has incompatible shape
[[node map/while/non_max_suppression/NonMaxSuppressionV3 (defined at bug.py:15) = NonMaxSuppressionV3[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](map/while/Gather
V2_1/_29, map/while/GatherV2/_31, map/while/non_max_suppression/NonMaxSuppressionV3/max_output_size/_33, map/while/non_max_suppression/iou_threshold/_35, map/while/non_max_suppression/score_thresh
old/_37)]]

2018-10-04 14:59:14.736180: F tensorflow/core/common_runtime/bfc_allocator.cc:458] Check failed: c->in_use() && (c->bin_num == kInvalidBinNum)

2018-10-04 14:59:49.523436: F tensorflow/core/common_runtime/bfc_allocator.cc:380] Check failed: h != kInvalidChunkHandle

2018-10-05 00:12:03.720295: F ./tensorflow/core/framework/tensor.h:643] Check failed: new_num_elements == NumElements() (39 vs. 0)

InvalidArgumentError (see above for traceback): indices[1] = [0] is repeated
[[{{node map/while/SparseToDense}} = SparseToDense[T=DT_BOOL, Tindices=DT_INT32, _class=["loc:@map/while/TensorArrayWrite/TensorArrayWriteV3"], validate_indices=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](map/while/Neg_1/_51, map/while/Shape/_53, map/while/SparseToDense/sparse_values/_55, map/while/SparseToDense/default_value/_57)]]
[[{{node map/while/SparseToDense/sparse_values/_54}} = _Send[T=DT_BOOL, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_111_map/while/SparseToDense/sparse_values", _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/SparseToDense/sparse_values)]]

After distilled to this small repro, it seems to mostly do segfault. But the above error messages might help. Seems like a memory corruption.