Issue 1:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):

Yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

Container derived from nvidia/cuda:10.0-cudnn7-runtime-ubuntu18.04 running on Ubuntu 18.04 via Kubernetes 1.13

TensorFlow installed from (source or binary):

pip3 install tensorflow-gpu==2.0.0-beta1

TensorFlow version (use command below):

v1.12.1-6250-g37eafe0e74 2.0.0-dev20190715

Python version:

Python 3.6.8

CUDA/cuDNN version:

NVIDIA-SMI 410.78 Driver Version: 410.78 CUDA Version: 10.0

libcudnn.so.7 -> libcudnn.so.7.6.0
libcudnn is installed

GPU model and memory:

name, pci.bus_id, vbios_version
Tesla V100-SXM2-32GB, 00000000:1B:00.0, 88.00.43.00.03

Describe the current behavior

Doing a training with tf.keras results in out of memory after some time when including the class_weight parameter. Also there is a long delay between the start of each epoch. If I omit the class_weight parameter, training proceeds normally with constant memory.

Describe the expected behavior

Training proceed normally when the class_weight parameter is included without running out of memory.

Code to reproduce the issue

import tensorflow as tf

################ Data
def _parse_fn2(fn, label):
img = tf.random.uniform([224, 224, 3])
return img, label

train_data2 = tf.data.Dataset.from_tensor_slices(
(tf.random.uniform([100]), tf.random.uniform([100], maxval=9, dtype=tf.dtypes.int32))
)

val_data2 = tf.data.Dataset.from_tensor_slices(
(tf.random.uniform([100]), tf.random.uniform([100], maxval=9, dtype=tf.dtypes.int32))
)
train_data2 = (train_data2.map(_parse_fn2)).batch(32)
val_data2 = (val_data2.map(_parse_fn2)).batch(32)

############### Model
IMG_SHAPE = (224, 224, 3)

base_model = tf.keras.applications.ResNet50(input_shape=IMG_SHAPE,include_top=False, weights=None)
base_model.trainable = True
maxpool_layer = tf.keras.layers.GlobalMaxPooling2D()
prediction_layer = tf.keras.layers.Dense(9, activation='softmax')

model = tf.keras.Sequential([
base_model,
maxpool_layer,
tf.keras.layers.Dropout(0.4),
prediction_layer
])

model.compile(optimizer=tf.keras.optimizers.Adam(lr=0.01), loss='categorical_crossentropy', metrics=['accuracy'])

model.summary()
history = model.fit(train_data2.repeat(),
epochs=100,
steps_per_epoch = 50,
validation_data=val_data2.repeat(),
validation_steps=10,
class_weight={0:1,1:1,2:1,3:1,4:1,5:1,6:1,7:1,8:1,9:1},
callbacks = [])

Other info / logs

Memory Usage at Each Epoch:

total used free shared buff/cache available
E1: 502G 11G 461G 140M 30G 487G
E2: 502G 16G 456G 140M 30G 483G
E3: 502G 22G 449G 140M 30G 476G
E4: 502G 30G 441G 141M 30G 468G
E5: 502G 40G 431G 141M 30G 458G
E6: 502G 52G 419G 141M 30G 446G

vmstat output:

vmstat_output.txt

Issue 2:
When creating a Reader op inside core/user_ops that inherits from ReaderBase, Bazel responds with

tensorflow/core/BUILD:325:1: undeclared inclusion(s) in rule '//tensorflow/core:user_ops_op_lib':
this rule is missing dependency declarations for the following files included by 'tensorflow/core/user_ops/user_reader_op.cc':
'bazel-out/local_darwin-opt/genfiles/tensorflow/core/kernels/reader_base.pb.h'.

Is this because the user_ops_op_lib depends only on framework? user_reader_op.cc includes reader_base.h which includes reader_base.pb.h. What dependency should I add to make this work? I'm still figuring out Bazel atm.

This is with latest source code.