Issue 1:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Fedora 22

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: NA

TensorFlow installed from (source or binary): pip

TensorFlow version (use command below): v1.12.0-0-ga6d8ffae09 1.12.0

Python version: 3.6.5

Bazel version (if compiling from source): NA

GCC/Compiler version (if compiling from source): NA

CUDA/cuDNN version: NA

GPU model and memory: NA

Describe the current behavior

The test program throws an exception tensorflow.python.framework.errors_impl.InvalidArgumentError: Retval[0] does not have value in session run. However, it completes normally if EPISODE_LENGTH is halved, the SimpleRNN layer is commented out or the batch_normalization layer is commented out. The problem also goes away if I replace the RandomDataset-derived dataset with one built with from_tensor_slices from constant data.

Describe the expected behavior

The test program should complete normally. I don't think my program has a bug that is responsible for the exception but even if it does, the throwing of the exception should be consistent and not depend on EPISODE_LENGTH being big enough.

Code to reproduce the issue

#!/usr/bin/env python3

import numpy as np
import tensorflow as tf
import tensorflow.keras.layers as kl

NUM_FEATURES = 2
EPISODE_LENGTH = 32
BATCH_SIZE = 2

class GraphBuilder:
def __init__(self):
g = tf.Graph()
with g.as_default():
tf.set_random_seed(0)

with tf.variable_scope('input'):
ds = tf.data.Dataset.from_tensors(
(np.zeros(shape=(BATCH_SIZE, EPISODE_LENGTH, NUM_FEATURES), dtype=np.float32),
np.zeros(shape=(BATCH_SIZE, EPISODE_LENGTH), dtype=np.float32)))
it = ds.make_one_shot_iterator()
batch = it.get_next(name='batch')
batch_x, batch_y = batch
with tf.variable_scope('nn'):
layer = batch_x
layer = kl.SimpleRNN(1, return_sequences=True)(layer, training=True)
layer = tf.layers.batch_normalization(layer, axis=2, training=True)
batch_pred = layer[:, :, 0]

with tf.variable_scope('loss'):
num_rows = tf.reduce_prod(tf.shape(batch_y))
e = tf.subtract(batch_y, batch_pred, name='error')
sse = tf.reduce_sum(tf.square(e), name='sse')
mse = tf.divide(sse, tf.cast(num_rows, tf.float32), name='mse')
rmse = tf.sqrt(mse, name='rmse')

with tf.variable_scope('train'):
optimizer = tf.train.AdamOptimizer(0.001)
update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(update_ops):
train_op = optimizer.minimize(mse)

self.rmse = rmse
self.train_op = train_op
self.graph = g

def main():
gb = GraphBuilder()
config = tf.ConfigProto(intra_op_parallelism_threads=2, inter_op_parallelism_threads=2)
config.device_count['GPU'] = 0
with tf.Session(graph=gb.graph, config=config) as sess:
sess.run(tf.global_variables_initializer())
sess.run([gb.rmse, gb.train_op])

main()

Other info / logs

2018-11-20 13:31:56.998664: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
Traceback (most recent call last):
File "/data/jchia/venv/tf/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
return fn(*args)
File "/data/jchia/venv/tf/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/data/jchia/venv/tf/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Retval[0] does not have value

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "py3/train2.py", line 56, in <module>
main()
File "py3/train2.py", line 54, in main
sess.run([gb.rmse, gb.train_op])
File "/data/jchia/venv/tf/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/data/jchia/venv/tf/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/data/jchia/venv/tf/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/data/jchia/venv/tf/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Retval[0] does not have value

Issue 2:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 17.10

TensorFlow installed from (source or binary): Binary (pip install tensorflow)

TensorFlow version (use command below): v1.4.0-rc1-11-g130a514 1.4.0

Python version: 3.6.3

CUDA/cuDNN version: N/A

GPU model and memory: N/A

Exact command to reproduce: (see the "Source code" section)

Describe the problem

I was following the instructions for updating moving_mean and moving_variance by using the code snippet provided in batch_normalization documentation and it resulted in an "tensorflow.python.framework.errors_impl.InvalidArgumentError: Retval[0] does not have value" exception.

Source code / logs

Full instructions to reproduce:

git clone -b control-dependencies-exc https://github.com/naktinis/language-id.git ctrl-dep-exc
cd ctrl-dep-exc/
python3 -m venv venv
. venv/bin/activate
pip install tensorflow==1.4.0 Pillow
python3 main.py --image-dir test_data/ --label-file test_data/labels.csv --model rnn

The specific code change that was enough to produce the exception (seems to match the snippet in the official documentation):

naktinis/language-id@50740f

Full exception:

Traceback (most recent call last):
File "main.py", line 173, in <module>
tf.app.run(main=run_experiment)
File "/home/naktinis/code/ctrl-dep-exc/venv/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "main.py", line 165, in run_experiment
hparams=params,
File "/home/naktinis/code/ctrl-dep-exc/venv/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/learn_runner.py", line 218, in run
return _execute_schedule(experiment, schedule)
File "/home/naktinis/code/ctrl-dep-exc/venv/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/learn_runner.py", line 46, in _execute_schedule
return task()
File "/home/naktinis/code/ctrl-dep-exc/venv/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 625, in train_and_evaluate
self.train(delay_secs=0)
File "/home/naktinis/code/ctrl-dep-exc/venv/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 367, in train
hooks=self._train_monitors + extra_hooks)
File "/home/naktinis/code/ctrl-dep-exc/venv/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 807, in _call_train
hooks=hooks)
File "/home/naktinis/code/ctrl-dep-exc/venv/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 302, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/home/naktinis/code/ctrl-dep-exc/venv/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 783, in _train_model
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/home/naktinis/code/ctrl-dep-exc/venv/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 521, in run
run_metadata=run_metadata)
File "/home/naktinis/code/ctrl-dep-exc/venv/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 892, in run
run_metadata=run_metadata)
File "/home/naktinis/code/ctrl-dep-exc/venv/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 967, in run
raise six.reraise(*original_exc_info)
File "/home/naktinis/code/ctrl-dep-exc/venv/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/home/naktinis/code/ctrl-dep-exc/venv/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 952, in run
return self._sess.run(*args, **kwargs)
File "/home/naktinis/code/ctrl-dep-exc/venv/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1024, in run
run_metadata=run_metadata)
File "/home/naktinis/code/ctrl-dep-exc/venv/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 827, in run
return self._sess.run(*args, **kwargs)
File "/home/naktinis/code/ctrl-dep-exc/venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/home/naktinis/code/ctrl-dep-exc/venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "/home/naktinis/code/ctrl-dep-exc/venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run
options, run_metadata)
File "/home/naktinis/code/ctrl-dep-exc/venv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Retval[0] does not have value