Issue 1:
This is a feature request.

Let's consider a scenario where one trains multiple models and uses them in combination (like it is done in GANs).

To simplify the process of saving and restoring variables that are partly shared across the models (Pretraining Model, Training Model, Evaluation Model, Infer Model) one could instantiate the whole graph, containing all operations and variables, and save it.

Then in order to do pre-training only the subset of graph elements that is required for pretraining is used.

This results in the overhead of having to build the whole model (which might consist of multiple sub models) the first time the model is run even though only a smaller part (e.g. Pretraining Model) is required.

Another issue arises when using different optimizers in different training stages (e.g. SGD first then Adam). As Adam creates additional variables Adam has to be instantiated during the first training stage so restoring from a checkpoint does not fail when restoring with Adam instead of SGD.

This restriction of having to build everything despite parts not being required results in more complicated code. If it would be possible to silently fail when a variable is not found in a checkpoint, so it can be initialized with tf.global_variables_initializer() instead, would allow better structuring of code.

I have looked through all current issues regarding this problem and I have found a couple that face a similar problem and where a QuietlyFailRestoringSaver could solve this problem:

#12032

#16781

I might consider building this if there is enough support for it. I am open for feedback.

Issue 2:
Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: NA

TensorFlow installed from (source or binary): 1.13.1 from conda

TensorFlow version (use command below): 1.13.1

Python version: 3.7.3

Bazel version (if compiling from source):

GCC/Compiler version (if compiling from source):

CUDA/cuDNN version:

GPU model and memory:

You can collect some of this information using our environment capture script

You can also obtain the TensorFlow version with

python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

Code to reproduce the issue

import tensorflow as tf

def foo(x):
array = tf.Variable(lambda: tf.zeros(20, dtype=tf.float32),
trainable=False, use_resource=True)
ind = tf.reshape(x, [-1, 1])
val = tf.reshape(tf.cast(x, tf.float32), [-1])

array_assign = array.assign(tf.zeros_like(array))
with tf.control_dependencies([array_assign]):
array = array.scatter_nd_update(ind, tf.cast(val, dtype=tf.float32))

return array

dataset = tf.data.Dataset.range(20)
# dataset = dataset.map(foo)
# dataset = dataset.batch(5)
dataset = dataset.apply(
tf.data.experimental.map_and_batch(foo, 5, num_parallel_batches=1))

iterator = dataset.make_initializable_iterator()

with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
sess.run(iterator.initializer)
next_element = iterator.get_next()
fetches = sess.run(next_element)
print(fetches)

Describe the current behavior

If using normal map and batch, one get

[[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 2. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 3. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 4. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]

Describe the expected behavior

If using map_and_batch, one get changing results like:

[[0. 1. 2. 3. 4. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 1. 0. 0. 4. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 1. 2. 3. 4. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 1. 2. 3. 4. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 1. 0. 0. 4. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]

There should be always only one and non-duplicate non-zeros per-line.

Other info / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

Not sure if this is related to #27507 . On a Colab with tf-nightly, #27507 seems fixed but this issue still reproduces.