Issue 1:
I am facing similar issue but for GRU. I am using tensorflow 1.1.0 and I tried dumping the model in different ways:

a) saver = tf.train.Saver(tf.global_variables())

model_exporter = exporter.Exporter(saver)

# Restore variables from training checkpoint
# TODO: This restores the most recent checkpoint, but if we use validation to counterract
# over-fitting, we may want to restore an earlier checkpoint.
checkpoint = tf.train.get_checkpoint_state(FLAGS.checkpoint_dir)
checkpoint_path = checkpoint.model_checkpoint_path
saver.restore(session, checkpoint_path)
log_info('Restored checkpoint at training epoch %d' % (int(checkpoint_path.split('-')[-1]) + 1))

# Initialise the model exporter and export the model
model_exporter.init(session.graph.as_graph_def(),
named_graph_signatures = {
'inputs': exporter.generic_signature(
{ 'input': input_tensor,
'input_lengths': seq_length}),
'outputs': exporter.generic_signature(
{ 'outputs': decoded})})
if FLAGS.remove_export:
actual_export_dir = os.path.join(FLAGS.export_dir, '%08d' % FLAGS.export_version)
if os.path.isdir(actual_export_dir):
log_info('Removing old export')
shutil.rmtree(actual_FLAGS.export_dir)
try:
# Export serving model
model_exporter.export(FLAGS.export_dir, tf.constant(FLAGS.export_version), session)

# Export graph
input_graph_name = 'input_graph.pb'
tf.train.write_graph(session.graph, FLAGS.export_dir, input_graph_name, as_text=False)

# Freeze graph
input_graph_path = os.path.join(FLAGS.export_dir, input_graph_name)
input_saver_def_path = ''
input_binary = True
output_node_names = 'output_node'
restore_op_name = 'save/restore_all'
filename_tensor_name = 'save/Const:0'
output_graph_path = os.path.join(FLAGS.export_dir, 'output_graph.pb')
clear_devices = False
freeze_graph.freeze_graph(input_graph_path, input_saver_def_path,
input_binary, checkpoint_path, output_node_names,
restore_op_name, filename_tensor_name,
output_graph_path, clear_devices, '')

b) output_graph_def = graph_util.convert_variables_to_constants(session, session.graph.as_graph_def(), ['output_node'])

with gfile.FastGFile('./data/ldc93s1/output_graph2.pb', 'wb') as f:

f.write(output_graph_def.SerializeToString())

but for both the dump I get the following error: -

'rnn/while/multi_rnn_cell/cell_0/gru_cell/gates/r/cond/rnn/while/multi_rnn_cell/cell_0/gru_cell/gates/r/strided_slice/_assign/RefEnter': Input tensor 'rnn/multi_rnn_cell/cell_0/gru_cell/gates/r/pop_mean:0' Cannot convert a tensor of type float32 to an input of type float32_ref

Any solution so far?

I followed the similar bug but that is specifically related to Batch norm #3628 . And what is the reason behind this??

Issue 2:
Please go to Stack Overflow for help and support:

https://stackoverflow.com/questions/tagged/tensorflow

If you open a GitHub issue, here is our policy:

It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).

The form below must be filled out.

It shouldn't be a TensorBoard issue. Those go here.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

TensorFlow installed from (source or binary):

TensorFlow version (use command below):

Python version:

Bazel version (if compiling from source):

GCC/Compiler version (if compiling from source):

CUDA/cuDNN version:

GPU model and memory:

Exact command to reproduce:

You can collect some of this information using our environment capture script:

https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh

You can obtain the TensorFlow version with

python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

Describe the problem

Reporting issues raise in this stackoverflow question:

https://stackoverflow.com/questions/50758110/warm-start-with-distribute-mirroredstrategy-and-tf-estimator:

tf.train.init_from_checkpoint doesn't work well with MirroredStrategy

tf.estimator.WarmStartSettings with Estimator doesn't work with MirroredStrategy

Source code / logs

See code snippets and error messages in this stack overflow page:

https://stackoverflow.com/questions/50758110/warm-start-with-distribute-mirroredstrategy-and-tf-estimator

The root cause is that we haven't modified these 2 paths for restoring from checkpoints to work with mirrored variables. So we need to do 2 things:

Use the appropriate method to restore mirrored variables in these codepaths - either via distribution.update or by using Saver (see Allen's suggestion below).

if the restoring is happening in tower context(for e.g. when init_from_checkpoint is called in model_fn), then we may need to first enter cross tower context before restoring.

Suggestion for addressing #1 from @allenlavoie :

Do the same SaveableObject extraction as Saver in warm_start, then just call SaveableObject.restore() with the Tensor instead of using .assign. This will work for MirroredVariables and whatever else is checkpointable (and means that to work with both, new objects only need to do the checkpointable thing). Potentially we could just call SaveableObjectsForOp for everything including variables.