Optimizer that implements the Adadelta algorithm.
Inherits From: Optimizer
ADADELTA - An Adaptive Learning Rate Method: Zeiler, 2012 (pdf)
View source
Apply gradients to variables.
This is the second part of minimize(). It returns an Operation that applies gradients.
View source
Compute gradients of loss for the variables in var_list.
This is the first part of minimize(). It returns a list of (gradient, variable) pairs where "gradient" is the gradient for "variable". Note that "gradient" can be a Tensor, an IndexedSlices, or None if there is no gradient for the given variable.
When eager execution is enabled, gate_gradients, aggregation_method, and colocate_gradients_with_ops are ignored.
View source
View source
Return a slot named name created for var by the Optimizer.
Some Optimizer subclasses use additional variables. For example Momentum and Adagrad use variables to accumulate updates. This method gives access to these Variable objects if for some reason you need them.
Use get_slot_names() to get the list of slot names created by the Optimizer.
View source
Return a list of the names of slots created by the Optimizer.
See get_slot().
View source
Add operations to minimize loss by updating var_list.
This method simply combines calls compute_gradients() and apply_gradients(). If you want to process the gradient before applying them call compute_gradients() and apply_gradients() explicitly instead of using this function.
When eager execution is enabled, loss should be a Python function that takes no arguments and computes the value to be minimized. Minimization (and gradient computation) is done with respect to the elements of var_list if not None, else with respect to any trainable variables created during the execution of the loss function. gate_gradients, aggregation_method, colocate_gradients_with_ops and grad_loss are ignored when eager execution is enabled.
View source
A list of variables which encode the current state of Optimizer.
Includes slot variables and additional global variables created by the optimizer in the current default graph.


Adagrad Dual Averaging algorithm for sparse linear models.
Inherits From: Optimizer
This optimizer takes care of regularization of unseen features in a mini batch by updating them when they are seen with a closed form update rule that is equivalent to having updated them on every mini-batch.
AdagradDA is typically used when there is a need for large sparsity in the trained model. This optimizer only guarantees sparsity for linear models. Be careful when using AdagradDA for deep networks as it will require careful initialization of the gradient accumulators for it to train.
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization :Duchi et al., 2011 (pdf)
View source
Apply gradients to variables.
This is the second part of minimize(). It returns an Operation that applies gradients.
View source
Compute gradients of loss for the variables in var_list.
This is the first part of minimize(). It returns a list of (gradient, variable) pairs where "gradient" is the gradient for "variable". Note that "gradient" can be a Tensor, an IndexedSlices, or None if there is no gradient for the given variable.
When eager execution is enabled, gate_gradients, aggregation_method, and colocate_gradients_with_ops are ignored.
View source
View source
Return a slot named name created for var by the Optimizer.
Some Optimizer subclasses use additional variables. For example Momentum and Adagrad use variables to accumulate updates. This method gives access to these Variable objects if for some reason you need them.
Use get_slot_names() to get the list of slot names created by the Optimizer.
View source
Return a list of the names of slots created by the Optimizer.
See get_slot().
View source
Add operations to minimize loss by updating var_list.
This method simply combines calls compute_gradients() and apply_gradients(). If you want to process the gradient before applying them call compute_gradients() and apply_gradients() explicitly instead of using this function.
When eager execution is enabled, loss should be a Python function that takes no arguments and computes the value to be minimized. Minimization (and gradient computation) is done with respect to the elements of var_list if not None, else with respect to any trainable variables created during the execution of the loss function. gate_gradients, aggregation_method, colocate_gradients_with_ops and grad_loss are ignored when eager execution is enabled.
View source
A list of variables which encode the current state of Optimizer.
Includes slot variables and additional global variables created by the optimizer in the current default graph.


Optimizer that implements the Adagrad algorithm.
Inherits From: Optimizer
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization :Duchi et al., 2011 (pdf)
View source
Apply gradients to variables.
This is the second part of minimize(). It returns an Operation that applies gradients.
View source
Compute gradients of loss for the variables in var_list.
This is the first part of minimize(). It returns a list of (gradient, variable) pairs where "gradient" is the gradient for "variable". Note that "gradient" can be a Tensor, an IndexedSlices, or None if there is no gradient for the given variable.
When eager execution is enabled, gate_gradients, aggregation_method, and colocate_gradients_with_ops are ignored.
View source
View source
Return a slot named name created for var by the Optimizer.
Some Optimizer subclasses use additional variables. For example Momentum and Adagrad use variables to accumulate updates. This method gives access to these Variable objects if for some reason you need them.
Use get_slot_names() to get the list of slot names created by the Optimizer.
View source
Return a list of the names of slots created by the Optimizer.
See get_slot().
View source
Add operations to minimize loss by updating var_list.
This method simply combines calls compute_gradients() and apply_gradients(). If you want to process the gradient before applying them call compute_gradients() and apply_gradients() explicitly instead of using this function.
When eager execution is enabled, loss should be a Python function that takes no arguments and computes the value to be minimized. Minimization (and gradient computation) is done with respect to the elements of var_list if not None, else with respect to any trainable variables created during the execution of the loss function. gate_gradients, aggregation_method, colocate_gradients_with_ops and grad_loss are ignored when eager execution is enabled.
View source
A list of variables which encode the current state of Optimizer.
Includes slot variables and additional global variables created by the optimizer in the current default graph.


Optimizer that implements the Adam algorithm.
Inherits From: Optimizer
Adam - A Method for Stochastic Optimization: Kingma et al., 2015 (pdf)
View source
Apply gradients to variables.
This is the second part of minimize(). It returns an Operation that applies gradients.
View source
Compute gradients of loss for the variables in var_list.
This is the first part of minimize(). It returns a list of (gradient, variable) pairs where "gradient" is the gradient for "variable". Note that "gradient" can be a Tensor, an IndexedSlices, or None if there is no gradient for the given variable.
When eager execution is enabled, gate_gradients, aggregation_method, and colocate_gradients_with_ops are ignored.
View source
View source
Return a slot named name created for var by the Optimizer.
Some Optimizer subclasses use additional variables. For example Momentum and Adagrad use variables to accumulate updates. This method gives access to these Variable objects if for some reason you need them.
Use get_slot_names() to get the list of slot names created by the Optimizer.
View source
Return a list of the names of slots created by the Optimizer.
See get_slot().
View source
Add operations to minimize loss by updating var_list.
This method simply combines calls compute_gradients() and apply_gradients(). If you want to process the gradient before applying them call compute_gradients() and apply_gradients() explicitly instead of using this function.
When eager execution is enabled, loss should be a Python function that takes no arguments and computes the value to be minimized. Minimization (and gradient computation) is done with respect to the elements of var_list if not None, else with respect to any trainable variables created during the execution of the loss function. gate_gradients, aggregation_method, colocate_gradients_with_ops and grad_loss are ignored when eager execution is enabled.
View source
A list of variables which encode the current state of Optimizer.
Includes slot variables and additional global variables created by the optimizer in the current default graph.


Adds a QueueRunner to a collection in the graph. (deprecated)
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.train.queue_runner.add_queue_runner
When building a complex model that uses many queues it is often difficult to gather all the queue runners that need to be run. This convenience function allows you to add a queue runner to a well known collection in the graph.
The companion method start_queue_runners() can be used to start threads for all the collected queue runners.


Asserts global_step_tensor is a scalar int Variable or Tensor.


Basic loop to train a model.
Calls train_step_fn in a loop to train a model. The function is called as:
It is passed a tf.compat.v1.Session in addition to args and kwargs. The function typically runs one training step in the session.


Creates batches of tensors in tensors. (deprecated)
The argument tensors can be a list or a dictionary of tensors. The value returned by the function will be of the same type as tensors.
This function is implemented using a queue. A QueueRunner for the queue is added to the current Graph's QUEUE_RUNNER collection.
If enqueue_many is False, tensors is assumed to represent a single example. An input tensor with shape [x, y, z] will be output as a tensor with shape [batch_size, x, y, z].
If enqueue_many is True, tensors is assumed to represent a batch of examples, where the first dimension is indexed by example, and all members of tensors should have the same size in the first dimension. If an input tensor has shape [*, x, y, z], the output will have shape [batch_size, x, y, z]. The capacity argument controls the how long the prefetching is allowed to grow the queues.
The returned operation is a dequeue operation and will throw tf.errors.OutOfRangeError if the input queue is exhausted. If this operation is feeding another input queue, its queue runner will catch this exception, however, if this operation is used in your main thread you are responsible for catching this yourself.
If dynamic_pad is True, it is sufficient that the rank of the tensors is known, but individual dimensions may have shape None. In this case, for each enqueue the dimensions with value None may have a variable length; upon dequeue, the output tensors will be padded on the right to the maximum shape of the tensors in the current minibatch. For numbers, this padding takes value 0. For strings, this padding is the empty string. See PaddingFIFOQueue for more info.
If allow_smaller_final_batch is True, a smaller batch value than batch_size is returned when the queue is closed and there are not enough elements to fill the batch, otherwise the pending elements are discarded. In addition, all output tensors' static shapes, as accessed via the shape property will have a first Dimension value of None, and operations that depend on fixed batch_size would fail.
Input pipelines based on Queues are not supported when eager execution is enabled. Please use the tf.data API to ingest data under eager execution.


Runs a list of tensors to fill a queue to create batches of examples. (deprecated)
The tensors_list argument is a list of tuples of tensors, or a list of dictionaries of tensors. Each element in the list is treated similarly to the tensors argument of tf.compat.v1.train.batch().
Enqueues a different list of tensors in different threads. Implemented using a queue -- a QueueRunner for the queue is added to the current Graph's QUEUE_RUNNER collection.
len(tensors_list) threads will be started, with thread i enqueuing the tensors from tensors_list[i]. tensors_list[i1][j] must match tensors_list[i2][j] in type and shape, except in the first dimension if enqueue_many is true.
If enqueue_many is False, each tensors_list[i] is assumed to represent a single example. An input tensor x will be output as a tensor with shape [batch_size] + x.shape.
If enqueue_many is True, tensors_list[i] is assumed to represent a batch of examples, where the first dimension is indexed by example, and all members of tensors_list[i] should have the same size in the first dimension. The slices of any input tensor x are treated as examples, and the output tensors will have shape [batch_size] + x.shape[1:].
The capacity argument controls the how long the prefetching is allowed to grow the queues.
The returned operation is a dequeue operation and will throw tf.errors.OutOfRangeError if the input queue is exhausted. If this operation is feeding another input queue, its queue runner will catch this exception, however, if this operation is used in your main thread you are responsible for catching this yourself.
If dynamic_pad is True, it is sufficient that the rank of the tensors is known, but individual dimensions may have value None. In this case, for each enqueue the dimensions with value None may have a variable length; upon dequeue, the output tensors will be padded on the right to the maximum shape of the tensors in the current minibatch. For numbers, this padding takes value 0. For strings, this padding is the empty string. See PaddingFIFOQueue for more info.
If allow_smaller_final_batch is True, a smaller batch value than batch_size is returned when the queue is closed and there are not enough elements to fill the batch, otherwise the pending elements are discarded. In addition, all output tensors' static shapes, as accessed via the shape property will have a first Dimension value of None, and operations that depend on fixed batch_size would fail.
Input pipelines based on Queues are not supported when eager execution is enabled. Please use the tf.data API to ingest data under eager execution.


Groups trackable objects, saving and restoring them.
Checkpoint's constructor accepts keyword arguments whose values are types that contain trackable state, such as tf.compat.v1.train.Optimizer implementations, tf.Variable, tf.keras.Layer implementations, or tf.keras.Model implementations. It saves these values with a checkpoint, and maintains a save_counter for numbering checkpoints.
Example usage when graph building:
Example usage with eager execution enabled:
Checkpoint.save and Checkpoint.restore write and read object-based checkpoints, in contrast to tf.compat.v1.train.Saver which writes and reads variable.name based checkpoints. Object-based checkpointing saves a graph of dependencies between Python objects (Layers, Optimizers, Variables, etc.) with named edges, and this graph is used to match variables when restoring a checkpoint. It can be more robust to changes in the Python program, and helps to support restore-on-create for variables when executing eagerly. Prefer tf.train.Checkpoint over tf.compat.v1.train.Saver for new code.
Checkpoint objects have dependencies on the objects passed as keyword arguments to their constructors, and each dependency is given a name that is identical to the name of the keyword argument for which it was created. TensorFlow classes like Layers and Optimizers will automatically add dependencies on their variables (e.g. "kernel" and "bias" for tf.keras.layers.Dense). Inheriting from tf.keras.Model makes managing dependencies easy in user-defined classes, since Model hooks into attribute assignment. For example:
This Model has a dependency named "input_transform" on its Dense layer, which in turn depends on its variables. As a result, saving an instance of Regress using tf.train.Checkpoint will also save all the variables created by the Dense layer.
When variables are assigned to multiple workers, each worker writes its own section of the checkpoint. These sections are then merged/re-indexed to behave as a single checkpoint. This avoids copying all variables to one worker, but does require that all workers see a common filesystem.
While tf.keras.Model.save_weights and tf.train.Checkpoint.save save in the same format, note that the root of the resulting checkpoint is the object the save method is attached to. This means saving a tf.keras.Model using save_weights and loading into a tf.train.Checkpoint with a Model attached (or vice versa) will not match the Model's variables. See the guide to training checkpoints for details. Prefer tf.train.Checkpoint over tf.keras.Model.save_weights for training checkpoints.
View source
Restore a training checkpoint.
Restores this Checkpoint and any objects it depends on.
When executing eagerly, either assigns values immediately if variables to restore have been created already, or defers restoration until the variables are created. Dependencies added after this call will be matched if they have a corresponding object in the checkpoint (the restore request will queue in any trackable object waiting for the expected dependency to be added).
When graph building, restoration ops are added to the graph but not run immediately.
To ensure that loading is complete and no more assignments will take place, use the assert_consumed() method of the status object returned by restore:
An exception will be raised if any Python objects in the dependency graph were not found in the checkpoint, or if any checkpointed values do not have a matching Python object.
When graph building, assert_consumed() indicates that all of the restore ops that will be created for this checkpoint have been created. They can be run via the run_restore_ops() method of the status object:
If the checkpoint has not been consumed completely, then the list of restore ops will grow as more objects are added to the dependency graph.
Name-based tf.compat.v1.train.Saver checkpoints can be loaded using this method. Names are used to match variables. No restore ops are created/run until run_restore_ops() or initialize_or_restore() are called on the returned status object when graph building, but there is restore-on-creation when executing eagerly. Re-encode name-based checkpoints using tf.train.Checkpoint.save as soon as possible.
The returned status object has the following methods:
assert_consumed(): Raises an exception if any variables are unmatched: either checkpointed values which don't have a matching Python object or Python objects in the dependency graph with no values in the checkpoint. This method returns the status object, and so may be chained with initialize_or_restore or run_restore_ops.
assert_existing_objects_matched(): Raises an exception if any existing Python objects in the dependency graph are unmatched. Unlike assert_consumed, this assertion will pass if values in the checkpoint have no corresponding Python objects. For example a tf.keras.Layer object which has not yet been built, and so has not created any variables, will pass this assertion but fail assert_consumed. Useful when loading part of a larger checkpoint into a new Python program, e.g. a training checkpoint with a tf.compat.v1.train.Optimizer was saved but only the state required for inference is being loaded. This method returns the status object, and so may be chained with initialize_or_restore or run_restore_ops.
assert_nontrivial_match(): Asserts that something aside from the root object was matched. This is a very weak assertion, but is useful for sanity checking in library code where objects may exist in the checkpoint which haven't been created in Python and some Python objects may not have a checkpointed value.
expect_partial(): Silence warnings about incomplete checkpoint restores. Warnings are otherwise printed for unused parts of the checkpoint file or object when the Checkpoint object is deleted (often at program shutdown).
initialize_or_restore(session=None): When graph building, runs variable initializers if save_path is None, but otherwise runs restore operations. If no session is explicitly specified, the default session is used. No effect when executing eagerly (variables are initialized or restored eagerly).
run_restore_ops(session=None): When graph building, runs restore operations. If no session is explicitly specified, the default session is used. No effect when executing eagerly (restore operations are run eagerly). May only be called when save_path is not None.
View source
Saves a training checkpoint and provides basic checkpoint management.
The saved checkpoint includes variables created by this object and any trackable objects it depends on at the time Checkpoint.save() is called.
save is a basic convenience wrapper around the write method, sequentially numbering checkpoints using save_counter and updating the metadata used by tf.train.latest_checkpoint. More advanced checkpoint management, for example garbage collection and custom numbering, may be provided by other utilities which also wrap write (tf.train.CheckpointManager for example).
View source
Writes a training checkpoint.
The checkpoint includes variables created by this object and any trackable objects it depends on at the time Checkpoint.write() is called.
write does not number checkpoints, increment save_counter, or update the metadata used by tf.train.latest_checkpoint. It is primarily intended for use by higher level checkpoint management utilities. save provides a very basic implementation of these features.


Checks whether a V1 or V2 checkpoint exists with the specified prefix. (deprecated)
This is the recommended way to check if a checkpoint exists, since it takes into account the naming difference between V1 and V2 formats.


Creates a tf.compat.v1.Session for a chief.
Inherits From: SessionCreator
View source


Applies cosine decay to the learning rate.
When training a model, it is often recommended to lower the learning rate as the training progresses. This function applies a cosine decay function to a provided initial learning rate. It requires a global_step value to compute the decayed learning rate. You can just pass a TensorFlow variable that you increment at each training step.
The function returns the decayed learning rate. It is computed as:
Stochastic Gradient Descent with Warm Restarts: Loshchilov et al., 2017 (pdf)
When eager execution is enabled, this function returns a function which in turn returns the decayed learning rate Tensor. This can be useful for changing the learning rate value across different invocations of optimizer functions.


Applies cosine decay with restarts to the learning rate.
When training a model, it is often recommended to lower the learning rate as the training progresses. This function applies a cosine decay function with restarts to a provided initial learning rate. It requires a global_step value to compute the decayed learning rate. You can just pass a TensorFlow variable that you increment at each training step.
The function returns the decayed learning rate while taking into account possible warm restarts. The learning rate multiplier first decays from 1 to alpha for first_decay_steps steps. Then, a warm restart is performed. Each new warm restart runs for t_mul times more steps and with m_mul times smaller initial learning rate.
Stochastic Gradient Descent with Warm Restarts: Loshchilov et al., 2017 (pdf)
When eager execution is enabled, this function returns a function which in turn returns the decayed learning rate Tensor. This can be useful for changing the learning rate value across different invocations of optimizer functions.


A general quantization scheme is being developed in tf.contrib.quantize. (deprecated)
Consider using that instead, though since it is in the tf.contrib namespace, it is not subject to backward compatibility guarantees.


Public API for tf.train.experimental namespace.
class DynamicLossScale: Loss scale that dynamically adjusts itself.
class FixedLossScale: Loss scale with a fixed value.
class LossScale: Base class for all TF1 loss scales.
class MixedPrecisionLossScaleOptimizer: An optimizer that applies loss scaling.
class PythonState: A mixin for putting Python state in an object-based checkpoint.
disable_mixed_precision_graph_rewrite(...): Disables the mixed precision graph rewrite.
enable_mixed_precision_graph_rewrite(...): Enable mixed precision via a graph rewrite.


Applies exponential decay to the learning rate.
When training a model, it is often recommended to lower the learning rate as the training progresses. This function applies an exponential decay function to a provided initial learning rate. It requires a global_step value to compute the decayed learning rate. You can just pass a TensorFlow variable that you increment at each training step.
The function returns the decayed learning rate. It is computed as:
If the argument staircase is True, then global_step / decay_steps is an integer division and the decayed learning rate follows a staircase function.
Example: decay every 100000 steps with a base of 0.96:
When eager execution is enabled, this function returns a function which in turn returns the decayed learning rate Tensor. This can be useful for changing the learning rate value across different invocations of optimizer functions.


Returns MetaGraphDef proto.
Optionally writes it to filename.
This function exports the graph, saver, and collection objects into MetaGraphDef protocol buffer with the intention of it being imported at a later time or location to restart training, run inference, or be a subgraph.
Exporting/importing meta graphs is not supported unless both graph_def and graph are provided. No graph exists when eager execution is enabled.


Optimizer that implements the FTRL algorithm.
Inherits From: Optimizer
This version has support for both online L2 (McMahan et al., 2013) and shrinkage-type L2, which is the addition of an L2 penalty to the loss function.
Ad-click prediction: McMahan et al., 2013 (pdf)
View source
Apply gradients to variables.
This is the second part of minimize(). It returns an Operation that applies gradients.
View source
Compute gradients of loss for the variables in var_list.
This is the first part of minimize(). It returns a list of (gradient, variable) pairs where "gradient" is the gradient for "variable". Note that "gradient" can be a Tensor, an IndexedSlices, or None if there is no gradient for the given variable.
When eager execution is enabled, gate_gradients, aggregation_method, and colocate_gradients_with_ops are ignored.
View source
View source
Return a slot named name created for var by the Optimizer.
Some Optimizer subclasses use additional variables. For example Momentum and Adagrad use variables to accumulate updates. This method gives access to these Variable objects if for some reason you need them.
Use get_slot_names() to get the list of slot names created by the Optimizer.
View source
Return a list of the names of slots created by the Optimizer.
See get_slot().
View source
Add operations to minimize loss by updating var_list.
This method simply combines calls compute_gradients() and apply_gradients(). If you want to process the gradient before applying them call compute_gradients() and apply_gradients() explicitly instead of using this function.
When eager execution is enabled, loss should be a Python function that takes no arguments and computes the value to be minimized. Minimization (and gradient computation) is done with respect to the elements of var_list if not None, else with respect to any trainable variables created during the execution of the loss function. gate_gradients, aggregation_method, colocate_gradients_with_ops and grad_loss are ignored when eager execution is enabled.
View source
A list of variables which encode the current state of Optimizer.
Includes slot variables and additional global variables created by the optimizer in the current default graph.


Returns the mtimes (modification timestamps) of the checkpoints. (deprecated)
Globs for the checkpoints pointed to by checkpoint_prefixes. If the files exist, collect their mtime. Both V2 and V1 checkpoints are considered, in that priority.
This is the recommended way to get the mtimes, since it takes into account the naming difference between V1 and V2 formats.


Get the global step tensor.
The global step tensor must be an integer variable. We first try to find it in the collection GLOBAL_STEP, or by name global_step:0.


Returns and create (if necessary) the global step tensor.


Optimizer that implements the gradient descent algorithm.
Inherits From: Optimizer
View source
Apply gradients to variables.
This is the second part of minimize(). It returns an Operation that applies gradients.
View source
Compute gradients of loss for the variables in var_list.
This is the first part of minimize(). It returns a list of (gradient, variable) pairs where "gradient" is the gradient for "variable". Note that "gradient" can be a Tensor, an IndexedSlices, or None if there is no gradient for the given variable.
When eager execution is enabled, gate_gradients, aggregation_method, and colocate_gradients_with_ops are ignored.
View source
View source
Return a slot named name created for var by the Optimizer.
Some Optimizer subclasses use additional variables. For example Momentum and Adagrad use variables to accumulate updates. This method gives access to these Variable objects if for some reason you need them.
Use get_slot_names() to get the list of slot names created by the Optimizer.
View source
Return a list of the names of slots created by the Optimizer.
See get_slot().
View source
Add operations to minimize loss by updating var_list.
This method simply combines calls compute_gradients() and apply_gradients(). If you want to process the gradient before applying them call compute_gradients() and apply_gradients() explicitly instead of using this function.
When eager execution is enabled, loss should be a Python function that takes no arguments and computes the value to be minimized. Minimization (and gradient computation) is done with respect to the elements of var_list if not None, else with respect to any trainable variables created during the execution of the loss function. gate_gradients, aggregation_method, colocate_gradients_with_ops and grad_loss are ignored when eager execution is enabled.
View source
A list of variables which encode the current state of Optimizer.
Includes slot variables and additional global variables created by the optimizer in the current default graph.


Recreates a Graph saved in a MetaGraphDef proto.
This function takes a MetaGraphDef protocol buffer as input. If the argument is a file containing a MetaGraphDef protocol buffer , it constructs a protocol buffer from the file content. The function then adds all the nodes from the graph_def field to the current graph, recreates all the collections, and returns a saver constructed from the saver_def field.
In combination with export_meta_graph(), this function can be used to
Serialize a graph along with other Python objects such as QueueRunner, Variable into a MetaGraphDef.
Restart training from a saved graph and checkpoints.
Run inference from a saved graph and checkpoints.
Later we can continue training from this saved meta_graph without building the model from scratch.
Variables, placeholders, and independent operations can also be stored, as shown in the following example.
Later this model can be restored and contents loaded.
A None value is returned if no variables exist in the MetaGraphDef (i.e., there are no variables to restore).
Exporting/importing meta graphs is not supported. No graph exists when eager execution is enabled.


Replaces tf.Variable initializers so they load from a checkpoint file.
Values are not loaded immediately, but when the initializer is run (typically by running a tf.compat.v1.global_variables_initializer op).
Assignment map supports following syntax:
Supports loading into partitioned variables, which are represented as '<variable>/part_<part #>'.


Output the rows of input_tensor to a queue for an input pipeline. (deprecated)
Input pipelines based on Queues are not supported when eager execution is enabled. Please use the tf.data API to ingest data under eager execution.


Applies inverse time decay to the initial learning rate.
When training a model, it is often recommended to lower the learning rate as the training progresses. This function applies an inverse decay function to a provided initial learning rate. It requires an global_step value to compute the decayed learning rate. You can just pass a TensorFlow variable that you increment at each training step.
The function returns the decayed learning rate. It is computed as:
or, if staircase is True, as:
Example: decay 1/t with a rate of 0.5:
When eager execution is enabled, this function returns a function which in turn returns the decayed learning rate Tensor. This can be useful for changing the learning rate value across different invocations of optimizer functions.


Returns tensor num_epochs times and then raises an OutOfRange error. (deprecated)


Applies linear cosine decay to the learning rate.
Note that linear cosine decay is more aggressive than cosine decay and larger initial learning rates can typically be used.
When training a model, it is often recommended to lower the learning rate as the training progresses. This function applies a linear cosine decay function to a provided initial learning rate. It requires a global_step value to compute the decayed learning rate. You can just pass a TensorFlow variable that you increment at each training step.
The function returns the decayed learning rate. It is computed as:
Neural Optimizer Search with Reinforcement Learning: Bello et al., 2017 (pdf) Stochastic Gradient Descent with Warm Restarts: Loshchilov et al., 2017 (pdf)
When eager execution is enabled, this function returns a function which in turn returns the decayed learning rate Tensor. This can be useful for changing the learning rate value across different invocations of optimizer functions.


A thread that runs code repeatedly, optionally on a timer.
This thread class is intended to be used with a Coordinator. It repeatedly runs code specified either as target and args or by the run_loop() method.
Before each run the thread checks if the coordinator has requested stop. In that case the looper thread terminates immediately.
If the code being run raises an exception, that exception is reported to the coordinator and the thread terminates. The coordinator will then request all the other threads it coordinates to stop.
You typically pass looper threads to the supervisor Join() method.
This must be set before start() is called, otherwise RuntimeError is raised. Its initial value is inherited from the creating thread; the main thread is not a daemon thread and therefore all threads created in the main thread default to daemon = False.
The entire Python program exits when only daemon threads are left.
This is a nonzero integer. See the get_ident() function. Thread identifiers may be recycled when a thread exits and another thread is created. The identifier is available even after the thread has exited.
It has no semantics. Multiple threads may be given the same name. The initial name is set by the constructor.
Return whether the thread is alive.
This method is deprecated, use is_alive() instead.
Return whether the thread is alive.
This method returns True just before the run() method starts until just after the run() method terminates. The module function enumerate() returns a list of all alive threads.
Wait until the thread terminates.
This blocks the calling thread until the thread whose join() method is called terminates -- either normally or through an unhandled exception or until the optional timeout occurs.
When the timeout argument is present and not None, it should be a floating point number specifying a timeout for the operation in seconds (or fractions thereof). As join() always returns None, you must call is_alive() after join() to decide whether a timeout happened -- if the thread is still alive, the join() call timed out.
When the timeout argument is not present or None, the operation will block until the thread terminates.
A thread can be join()ed many times.
join() raises a RuntimeError if an attempt is made to join the current thread as that would cause a deadlock. It is also an error to join() a thread before it has been started and attempts to do so raises the same exception.
View source
Start a LooperThread that calls a function periodically.
If timer_interval_secs is None the thread calls target(args) repeatedly. Otherwise target(args) is called every timer_interval_secs seconds. The thread terminates when a stop of the coordinator is requested.
View source
Method representing the thread's activity.
You may override this method in a subclass. The standard run() method invokes the callable object passed to the object's constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.
View source
Called at 'timer_interval_secs' boundaries.
Start the thread's activity.
It must be called at most once per thread object. It arranges for the object's run() method to be invoked in a separate thread of control.
This method will raise a RuntimeError if called more than once on the same thread object.
View source
Called when the thread starts.
View source
Called when the thread stops.


Conditionally creates batches of tensors based on keep_input. (deprecated)
See docstring in batch for more details.


Runs a list of tensors to conditionally fill a queue to create batches. (deprecated)
See docstring in batch_join for more details.


Creates batches by randomly shuffling conditionally-enqueued tensors. (deprecated)
See docstring in shuffle_batch for more details.
Input pipelines based on Queues are not supported when eager execution is enabled. Please use the tf.data API to ingest data under eager execution.


Create batches by randomly shuffling conditionally-enqueued tensors. (deprecated)
See docstring in shuffle_batch_join for more details.
Input pipelines based on Queues are not supported when eager execution is enabled. Please use the tf.data API to ingest data under eager execution.


Optimizer that implements the Momentum algorithm.
Inherits From: Optimizer
Computes (if use_nesterov = False):
Note that in the dense version of this algorithm, accumulation is updated and applied regardless of a gradient's value, whereas the sparse version (when the gradient is an IndexedSlices, typically because of tf.gather or an embedding) only updates variable slices and corresponding accumulation terms when that part of the variable was used in the forward pass.
View source
Apply gradients to variables.
This is the second part of minimize(). It returns an Operation that applies gradients.
View source
Compute gradients of loss for the variables in var_list.
This is the first part of minimize(). It returns a list of (gradient, variable) pairs where "gradient" is the gradient for "variable". Note that "gradient" can be a Tensor, an IndexedSlices, or None if there is no gradient for the given variable.
When eager execution is enabled, gate_gradients, aggregation_method, and colocate_gradients_with_ops are ignored.
View source
View source
Return a slot named name created for var by the Optimizer.
Some Optimizer subclasses use additional variables. For example Momentum and Adagrad use variables to accumulate updates. This method gives access to these Variable objects if for some reason you need them.
Use get_slot_names() to get the list of slot names created by the Optimizer.
View source
Return a list of the names of slots created by the Optimizer.
See get_slot().
View source
Add operations to minimize loss by updating var_list.
This method simply combines calls compute_gradients() and apply_gradients(). If you want to process the gradient before applying them call compute_gradients() and apply_gradients() explicitly instead of using this function.
When eager execution is enabled, loss should be a Python function that takes no arguments and computes the value to be minimized. Minimization (and gradient computation) is done with respect to the elements of var_list if not None, else with respect to any trainable variables created during the execution of the loss function. gate_gradients, aggregation_method, colocate_gradients_with_ops and grad_loss are ignored when eager execution is enabled.
View source
A list of variables which encode the current state of Optimizer.
Includes slot variables and additional global variables created by the optimizer in the current default graph.


Session-like object that handles initialization, recovery and hooks.
Initialization: At creation time the monitored session does following things in given order:
Run: When run() is called, the monitored session does following things:
Exit: At the close(), the monitored session does following things in order:
How to set tf.compat.v1.Session arguments:
See MonitoredTrainingSession for an example usage based on chief or worker.
class StepContext
View source
View source
Run ops in the monitored session.
This method is completely compatible with the tf.Session.run() method.
View source
Run ops using a step function.
Hooks interact with the run_with_hooks() call inside the step_fn as they do with a MonitoredSession.run call.
View source
View source
View source


Control flow instrument for the step_fn from run_step_fn().
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.train.SingularMonitoredSession.StepContext
Users of step_fn may perform run() calls without running hooks by accessing the session. A run() call with hooks may be performed using run_with_hooks(). Computation flow can be interrupted using request_stop().
View source
Exit the training loop by causing should_stop() to return True.
Causes step_fn to exit by raising an exception.
View source
Same as MonitoredSession.run. Accepts the same arguments.


Creates a MonitoredSession for training.
For a chief, this utility sets proper session initializer/restorer. It also creates hooks related to checkpoint and summary saving. For workers, this utility sets proper session creator which waits for the chief to initialize/restore. Please check tf.compat.v1.train.MonitoredSession for more information.


Applies natural exponential decay to the initial learning rate.
When training a model, it is often recommended to lower the learning rate as the training progresses. This function applies an exponential decay function to a provided initial learning rate. It requires an global_step value to compute the decayed learning rate. You can just pass a TensorFlow variable that you increment at each training step.
The function returns the decayed learning rate. It is computed as:
or, if staircase is True, as:
Example: decay exponentially with a base of 0.96:
When eager execution is enabled, this function returns a function which in turn returns the decayed learning rate Tensor. This can be useful for changing the learning rate value across different invocations of optimizer functions.


A function that returns a CheckPointReader.


Applies noisy linear cosine decay to the learning rate.
Note that linear cosine decay is more aggressive than cosine decay and larger initial learning rates can typically be used.
When training a model, it is often recommended to lower the learning rate as the training progresses. This function applies a noisy linear cosine decay function to a provided initial learning rate. It requires a global_step value to compute the decayed learning rate. You can just pass a TensorFlow variable that you increment at each training step.
The function returns the decayed learning rate. It is computed as:
where eps_t is 0-centered gaussian noise with variance initial_variance / (1 + global_step) ** variance_decay
Neural Optimizer Search with Reinforcement Learning: Bello et al., 2017 (pdf) Stochastic Gradient Descent with Warm Restarts: Loshchilov et al., 2017 (pdf)
When eager execution is enabled, this function returns a function which in turn returns the decayed learning rate Tensor. This can be useful for changing the learning rate value across different invocations of optimizer functions.


Base class for optimizers.
This class defines the API to add Ops to train a model. You never use this class directly, but instead instantiate one of its subclasses such as GradientDescentOptimizer, AdagradOptimizer, or MomentumOptimizer.
In the training program you will just have to run the returned Op.
Calling minimize() takes care of both computing the gradients and applying them to the variables. If you want to process the gradients before applying them you can instead use the optimizer in three steps:
Both minimize() and compute_gradients() accept a gate_gradients argument that controls the degree of parallelism during the application of the gradients.
The possible values are: GATE_NONE, GATE_OP, and GATE_GRAPH.
GATE_NONE: Compute and apply gradients in parallel. This provides the maximum parallelism in execution, at the cost of some non-reproducibility in the results. For example the two gradients of matmul depend on the input values: With GATE_NONE one of the gradients could be applied to one of the inputs before the other gradient is computed resulting in non-reproducible results.
GATE_OP: For each Op, make sure all gradients are computed before they are used. This prevents race conditions for Ops that generate gradients for multiple inputs where the gradients depend on the inputs.
GATE_GRAPH: Make sure all gradients for all variables are computed before any one of them is used. This provides the least parallelism but can be useful if you want to process all gradients before applying any of them.
Some optimizer subclasses, such as MomentumOptimizer and AdagradOptimizer allocate and manage additional variables associated with the variables to train. These are called Slots. Slots have names and you can ask the optimizer for the names of the slots that it uses. Once you have a slot name you can ask the optimizer for the variable it created to hold the slot value.
This can be useful if you want to log debug a training algorithm, report stats about the slots, etc.
View source
Apply gradients to variables.
This is the second part of minimize(). It returns an Operation that applies gradients.
View source
Compute gradients of loss for the variables in var_list.
This is the first part of minimize(). It returns a list of (gradient, variable) pairs where "gradient" is the gradient for "variable". Note that "gradient" can be a Tensor, an IndexedSlices, or None if there is no gradient for the given variable.
When eager execution is enabled, gate_gradients, aggregation_method, and colocate_gradients_with_ops are ignored.
View source
View source
Return a slot named name created for var by the Optimizer.
Some Optimizer subclasses use additional variables. For example Momentum and Adagrad use variables to accumulate updates. This method gives access to these Variable objects if for some reason you need them.
Use get_slot_names() to get the list of slot names created by the Optimizer.
View source
Return a list of the names of slots created by the Optimizer.
See get_slot().
View source
Add operations to minimize loss by updating var_list.
This method simply combines calls compute_gradients() and apply_gradients(). If you want to process the gradient before applying them call compute_gradients() and apply_gradients() explicitly instead of using this function.
When eager execution is enabled, loss should be a Python function that takes no arguments and computes the value to be minimized. Minimization (and gradient computation) is done with respect to the elements of var_list if not None, else with respect to any trainable variables created during the execution of the loss function. gate_gradients, aggregation_method, colocate_gradients_with_ops and grad_loss are ignored when eager execution is enabled.
View source
A list of variables which encode the current state of Optimizer.
Includes slot variables and additional global variables created by the optimizer in the current default graph.


Piecewise constant from boundaries and interval values.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.train.piecewise_constant_decay
Example: use a learning rate that's 1.0 for the first 100001 steps, 0.5 for the next 10000 steps, and 0.1 for any additional steps.
When eager execution is enabled, this function returns a function which in turn returns the decayed learning rate Tensor. This can be useful for changing the learning rate value across different invocations of optimizer functions.


Applies a polynomial decay to the learning rate.
It is commonly observed that a monotonically decreasing learning rate, whose degree of change is carefully chosen, results in a better performing model. This function applies a polynomial decay function to a provided initial learning_rate to reach an end_learning_rate in the given decay_steps.
It requires a global_step value to compute the decayed learning rate. You can just pass a TensorFlow variable that you increment at each training step.
The function returns the decayed learning rate. It is computed as:
If cycle is True then a multiple of decay_steps is used, the first one that is bigger than global_steps.
Example: decay from 0.1 to 0.01 in 10000 steps using sqrt (i.e. power=0.5):
When eager execution is enabled, this function returns a function which in turn returns the decayed learning rate Tensor. This can be useful for changing the learning rate value across different invocations of optimizer functions.


Optimizer that implements the Proximal Adagrad algorithm.
Inherits From: Optimizer
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization: Duchi et al., 2011 (pdf) Efficient Learning using Forward-Backward Splitting: Duchi et al., 2009 (pdf)
View source
Apply gradients to variables.
This is the second part of minimize(). It returns an Operation that applies gradients.
View source
Compute gradients of loss for the variables in var_list.
This is the first part of minimize(). It returns a list of (gradient, variable) pairs where "gradient" is the gradient for "variable". Note that "gradient" can be a Tensor, an IndexedSlices, or None if there is no gradient for the given variable.
When eager execution is enabled, gate_gradients, aggregation_method, and colocate_gradients_with_ops are ignored.
View source
View source
Return a slot named name created for var by the Optimizer.
Some Optimizer subclasses use additional variables. For example Momentum and Adagrad use variables to accumulate updates. This method gives access to these Variable objects if for some reason you need them.
Use get_slot_names() to get the list of slot names created by the Optimizer.
View source
Return a list of the names of slots created by the Optimizer.
See get_slot().
View source
Add operations to minimize loss by updating var_list.
This method simply combines calls compute_gradients() and apply_gradients(). If you want to process the gradient before applying them call compute_gradients() and apply_gradients() explicitly instead of using this function.
When eager execution is enabled, loss should be a Python function that takes no arguments and computes the value to be minimized. Minimization (and gradient computation) is done with respect to the elements of var_list if not None, else with respect to any trainable variables created during the execution of the loss function. gate_gradients, aggregation_method, colocate_gradients_with_ops and grad_loss are ignored when eager execution is enabled.
View source
A list of variables which encode the current state of Optimizer.
Includes slot variables and additional global variables created by the optimizer in the current default graph.


Optimizer that implements the proximal gradient descent algorithm.
Inherits From: Optimizer
Efficient Learning using Forward-Backward Splitting: Duchi et al., 2009 (pdf)
View source
Apply gradients to variables.
This is the second part of minimize(). It returns an Operation that applies gradients.
View source
Compute gradients of loss for the variables in var_list.
This is the first part of minimize(). It returns a list of (gradient, variable) pairs where "gradient" is the gradient for "variable". Note that "gradient" can be a Tensor, an IndexedSlices, or None if there is no gradient for the given variable.
When eager execution is enabled, gate_gradients, aggregation_method, and colocate_gradients_with_ops are ignored.
View source
View source
Return a slot named name created for var by the Optimizer.
Some Optimizer subclasses use additional variables. For example Momentum and Adagrad use variables to accumulate updates. This method gives access to these Variable objects if for some reason you need them.
Use get_slot_names() to get the list of slot names created by the Optimizer.
View source
Return a list of the names of slots created by the Optimizer.
See get_slot().
View source
Add operations to minimize loss by updating var_list.
This method simply combines calls compute_gradients() and apply_gradients(). If you want to process the gradient before applying them call compute_gradients() and apply_gradients() explicitly instead of using this function.
When eager execution is enabled, loss should be a Python function that takes no arguments and computes the value to be minimized. Minimization (and gradient computation) is done with respect to the elements of var_list if not None, else with respect to any trainable variables created during the execution of the loss function. gate_gradients, aggregation_method, colocate_gradients_with_ops and grad_loss are ignored when eager execution is enabled.
View source
A list of variables which encode the current state of Optimizer.
Includes slot variables and additional global variables created by the optimizer in the current default graph.


Holds a list of enqueue operations for a queue, each to be run in a thread.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.train.queue_runner.QueueRunner
Queues are a convenient TensorFlow mechanism to compute tensors asynchronously using multiple threads. For example in the canonical 'Input Reader' setup one set of threads generates filenames in a queue; a second set of threads read records from the files, processes them, and enqueues tensors on a second queue; a third set of threads dequeues these input records to construct batches and runs them through training operations.
There are several delicate issues when running multiple threads that way: closing the queues in sequence as the input is exhausted, correctly catching and reporting exceptions, etc.
The QueueRunner, combined with the Coordinator, helps handle these issues.
QueueRunners are not compatible with eager execution. Instead, please use tf.data to get data into your model.
Exceptions raised in queue runner threads are handled in one of two ways depending on whether or not a Coordinator was passed to create_threads():
View source
Create threads to run the enqueue ops for the given session.
This method requires a session in which the graph was launched. It creates a list of threads, optionally starting them. There is one thread for each op passed in enqueue_ops.
The coord argument is an optional coordinator that the threads will use to terminate together and report exceptions. If a coordinator is given, this method starts an additional thread to close the queue when the coordinator requests a stop.
If previously created threads for the given session are still running, no new threads will be created.
View source
Returns a QueueRunner object created from queue_runner_def.
View source
Converts this QueueRunner to a QueueRunnerDef protocol buffer.


Public API for tf.train.queue_runner namespace.
class QueueRunner: Holds a list of enqueue operations for a queue, each to be run in a thread.
add_queue_runner(...): Adds a QueueRunner to a collection in the graph. (deprecated)
start_queue_runners(...): Starts all queue runners collected in the graph. (deprecated)


Produces the integers from 0 to limit-1 in a queue. (deprecated)
Input pipelines based on Queues are not supported when eager execution is enabled. Please use the tf.data API to ingest data under eager execution.


Removes a checkpoint given by checkpoint_prefix. (deprecated)


Return a device function to use when building a Graph for replicas.
Device Functions are used in with tf.device(device_function): statement to automatically assign devices to Operation objects as they are constructed, Device constraints are added from the inner-most context first, working outwards. The merging behavior adds constraints to fields that are yet unset by a more inner context. Currently the fields are (job, task, cpu/gpu).
If cluster is None, and ps_tasks is 0, the returned function is a no-op. Otherwise, the value of ps_tasks is derived from cluster.
By default, only Variable ops are placed on ps tasks, and the placement strategy is round-robin over all ps tasks. A custom ps_strategy may be used to do more intelligent placement, such as tf.contrib.training.GreedyLoadBalancingStrategy.
For example,


Optimizer that implements the RMSProp algorithm (Tielemans et al.
Inherits From: Optimizer
2012).
Coursera slide 29: Hinton, 2012 (pdf)
View source
Apply gradients to variables.
This is the second part of minimize(). It returns an Operation that applies gradients.
View source
Compute gradients of loss for the variables in var_list.
This is the first part of minimize(). It returns a list of (gradient, variable) pairs where "gradient" is the gradient for "variable". Note that "gradient" can be a Tensor, an IndexedSlices, or None if there is no gradient for the given variable.
When eager execution is enabled, gate_gradients, aggregation_method, and colocate_gradients_with_ops are ignored.
View source
View source
Return a slot named name created for var by the Optimizer.
Some Optimizer subclasses use additional variables. For example Momentum and Adagrad use variables to accumulate updates. This method gives access to these Variable objects if for some reason you need them.
Use get_slot_names() to get the list of slot names created by the Optimizer.
View source
Return a list of the names of slots created by the Optimizer.
See get_slot().
View source
Add operations to minimize loss by updating var_list.
This method simply combines calls compute_gradients() and apply_gradients(). If you want to process the gradient before applying them call compute_gradients() and apply_gradients() explicitly instead of using this function.
When eager execution is enabled, loss should be a Python function that takes no arguments and computes the value to be minimized. Minimization (and gradient computation) is done with respect to the elements of var_list if not None, else with respect to any trainable variables created during the execution of the loss function. gate_gradients, aggregation_method, colocate_gradients_with_ops and grad_loss are ignored when eager execution is enabled.
View source
A list of variables which encode the current state of Optimizer.
Includes slot variables and additional global variables created by the optimizer in the current default graph.


Saves and restores variables.
See Variables for an overview of variables, saving and restoring.
The Saver class adds ops to save and restore variables to and from checkpoints. It also provides convenience methods to run these ops.
Checkpoints are binary files in a proprietary format which map variable names to tensor values. The best way to examine the contents of a checkpoint is to load it using a Saver.
Savers can automatically number checkpoint filenames with a provided counter. This lets you keep multiple checkpoints at different steps while training a model. For example you can number the checkpoint filenames with the training step number. To avoid filling up disks, savers manage checkpoint files automatically. For example, they can keep only the N most recent files, or one checkpoint for every N hours of training.
You number checkpoint filenames by passing a value to the optional global_step argument to save():
Additionally, optional arguments to the Saver() constructor let you control the proliferation of checkpoint files on disk:
max_to_keep indicates the maximum number of recent checkpoint files to keep. As new files are created, older files are deleted. If None or 0, no checkpoints are deleted from the filesystem but only the last one is kept in the checkpoint file. Defaults to 5 (that is, the 5 most recent checkpoint files are kept.)
keep_checkpoint_every_n_hours: In addition to keeping the most recent max_to_keep checkpoint files, you might want to keep one checkpoint file for every N hours of training. This can be useful if you want to later analyze how a model progressed during a long training session. For example, passing keep_checkpoint_every_n_hours=2 ensures that you keep one checkpoint file for every 2 hours of training. The default value of 10,000 hours effectively disables the feature.
Note that you still have to call the save() method to save the model. Passing these arguments to the constructor will not save variables automatically for you.
A training program that saves regularly looks like:
In addition to checkpoint files, savers keep a protocol buffer on disk with the list of recent checkpoints. This is used to manage numbered checkpoint files and by latest_checkpoint(), which makes it easy to discover the path to the most recent checkpoint. That protocol buffer is stored in a file named 'checkpoint' next to the checkpoint files.
If you create several savers, you can specify a different filename for the protocol buffer file in the call to save().
You can pass any of the returned values to restore().
View source
Generates a SaverDef representation of this saver.
View source
View source
Writes MetaGraphDef to save_path/filename.
View source
Returns a Saver object created from saver_def.
View source
Recovers the internal saver state after a crash.
This method is useful for recovering the "self._last_checkpoints" state.
Globs for the checkpoints pointed to by checkpoint_paths. If the files exist, use their mtime as the checkpoint timestamp.
View source
Restores previously saved variables.
This method runs the ops added by the constructor for restoring variables. It requires a session in which the graph was launched. The variables to restore do not have to have been initialized, as restoring is itself a way to initialize variables.
The save_path argument is typically a value previously returned from a save() call, or a call to latest_checkpoint().
View source
Saves variables.
This method runs the ops added by the constructor for saving variables. It requires a session in which the graph was launched. The variables to save must also have been initialized.
The method returns the path prefix of the newly created checkpoint files. This string can be passed directly to a call to restore().
View source
DEPRECATED: Use set_last_checkpoints_with_time.
Sets the list of old checkpoint filenames.
View source
Sets the list of old checkpoint filenames and timestamps.
View source
Converts this Saver to a SaverDef protocol buffer.


Structure to create or gather pieces commonly needed to train a model.
When you build a model for training you usually need ops to initialize variables, a Saver to checkpoint them, an op to collect summaries for the visualizer, and so on.
Various libraries built on top of the core TensorFlow library take care of creating some or all of these pieces and storing them in well known collections in the graph. The Scaffold class helps pick these pieces from the graph collections, creating and adding them to the collections if needed.
If you call the scaffold constructor without any arguments, it will pick pieces from the collections, creating default ones if needed when scaffold.finalize() is called. You can pass arguments to the constructor to provide your own pieces. Pieces that you pass to the constructor are not added to the graph collections.
The following pieces are directly accessible as attributes of the Scaffold object:
You can also pass the following additional pieces to the constructor:
View source
Returns an op that groups the default local init ops.
This op is used during session initialization when a Scaffold is initialized without specifying the local_init_op arg. It includes tf.compat.v1.local_variables_initializer, tf.compat.v1.tables_initializer, and also initializes local session resources.
View source
Creates operations if needed and finalizes the graph.
View source
Get from cache or create a default operation.


Computes fingerprints of the input strings.


Distributed version of Stochastic Dual Coordinate Ascent (SDCA) optimizer for
linear models with L1 + L2 regularization. As global optimization objective is strongly-convex, the optimizer optimizes the dual objective at each step. The optimizer applies each update one example at a time. Examples are sampled uniformly, and the optimizer is learning rate free and enjoys linear convergence rate.
Proximal Stochastic Dual Coordinate Ascent. Shai Shalev-Shwartz, Tong Zhang. 2012
Adding vs. Averaging in Distributed Primal-Dual Optimization. Chenxin Ma, Virginia Smith, Martin Jaggi, Michael I. Jordan, Peter Richtarik, Martin Takac. 2015
Stochastic Dual Coordinate Ascent with Adaptive Probabilities. Dominik Csiba, Zheng Qu, Peter Richtarik. 2015


Applies L1 regularization shrink step on the parameters.


Training helper that restores from checkpoint and creates session.
This class is a small wrapper that takes care of session creation and checkpoint recovery. It also provides functions that to facilitate coordination among multiple training threads or processes.
prepare_session() initializes or restores a model. It requires init_op and saver as an argument.
A second process could wait for the model to be ready by doing the following:
wait_for_session() waits for a model to be initialized by other processes.
View source
Creates a Session. Makes sure the model is ready to be used.
Creates a Session on 'master'. If a saver object is passed in, and checkpoint_dir points to a directory containing valid checkpoint files, then it will try to recover the model from checkpoint. If no checkpoint files are available, and wait_for_checkpoint is True, then the process would check every recovery_wait_secs, up to max_wait_secs, for recovery to succeed.
If the model cannot be recovered successfully then it is initialized by running the init_op and calling init_fn if they are provided. The local_init_op is also run after init_op and init_fn, regardless of whether the model was recovered successfully, but only if ready_for_local_init_op passes.
If the model is recovered from a checkpoint it is assumed that all global variables have been initialized, in particular neither init_op nor init_fn will be executed.
It is an error if the model cannot be recovered and no init_op or init_fn or local_init_op are passed.
View source
Creates a Session, recovering if possible.
Creates a new session on 'master'. If the session is not initialized and can be recovered from a checkpoint, recover it.
View source
Creates a new Session and waits for model to be ready.
Creates a new Session on 'master'. Waits for the model to be initialized or recovered from a checkpoint. It's expected that another thread or process will make the model ready, and that this is intended to be used by threads/processes that participate in a distributed training configuration where a different thread/process is responsible for initializing or recovering the model being trained.
NB: The amount of time this method waits for the session is bounded by max_wait_secs. By default, this function will wait indefinitely.


Creates batches by randomly shuffling tensors. (deprecated)
This function adds the following to the current Graph:
If enqueue_many is False, tensors is assumed to represent a single example. An input tensor with shape [x, y, z] will be output as a tensor with shape [batch_size, x, y, z].
If enqueue_many is True, tensors is assumed to represent a batch of examples, where the first dimension is indexed by example, and all members of tensors should have the same size in the first dimension. If an input tensor has shape [*, x, y, z], the output will have shape [batch_size, x, y, z].
The capacity argument controls the how long the prefetching is allowed to grow the queues.
The returned operation is a dequeue operation and will throw tf.errors.OutOfRangeError if the input queue is exhausted. If this operation is feeding another input queue, its queue runner will catch this exception, however, if this operation is used in your main thread you are responsible for catching this yourself.
If allow_smaller_final_batch is True, a smaller batch value than batch_size is returned when the queue is closed and there are not enough elements to fill the batch, otherwise the pending elements are discarded. In addition, all output tensors' static shapes, as accessed via the shape property will have a first Dimension value of None, and operations that depend on fixed batch_size would fail.
Input pipelines based on Queues are not supported when eager execution is enabled. Please use the tf.data API to ingest data under eager execution.


Create batches by randomly shuffling tensors. (deprecated)
The tensors_list argument is a list of tuples of tensors, or a list of dictionaries of tensors. Each element in the list is treated similarly to the tensors argument of tf.compat.v1.train.shuffle_batch().
This version enqueues a different list of tensors in different threads. It adds the following to the current Graph:
len(tensors_list) threads will be started, with thread i enqueuing the tensors from tensors_list[i]. tensors_list[i1][j] must match tensors_list[i2][j] in type and shape, except in the first dimension if enqueue_many is true.
If enqueue_many is False, each tensors_list[i] is assumed to represent a single example. An input tensor with shape [x, y, z] will be output as a tensor with shape [batch_size, x, y, z].
If enqueue_many is True, tensors_list[i] is assumed to represent a batch of examples, where the first dimension is indexed by example, and all members of tensors_list[i] should have the same size in the first dimension. If an input tensor has shape [*, x, y, z], the output will have shape [batch_size, x, y, z].
The capacity argument controls the how long the prefetching is allowed to grow the queues.
The returned operation is a dequeue operation and will throw tf.errors.OutOfRangeError if the input queue is exhausted. If this operation is feeding another input queue, its queue runner will catch this exception, however, if this operation is used in your main thread you are responsible for catching this yourself.
If allow_smaller_final_batch is True, a smaller batch value than batch_size is returned when the queue is closed and there are not enough elements to fill the batch, otherwise the pending elements are discarded. In addition, all output tensors' static shapes, as accessed via the shape property will have a first Dimension value of None, and operations that depend on fixed batch_size would fail.
Input pipelines based on Queues are not supported when eager execution is enabled. Please use the tf.data API to ingest data under eager execution.


Session-like object that handles initialization, restoring, and hooks.
Please note that this utility is not recommended for distributed settings. For distributed settings, please use tf.compat.v1.train.MonitoredSession. The differences between MonitoredSession and SingularMonitoredSession are:
Initialization: At creation time the hooked session does following things in given order:
Run: When run() is called, the hooked session does following things:
Exit: At the close(), the hooked session does following things in order:
class StepContext
View source
View source
Returns underlying TensorFlow.Session object.
View source
Run ops in the monitored session.
This method is completely compatible with the tf.Session.run() method.
View source
Run ops using a step function.
Hooks interact with the run_with_hooks() call inside the step_fn as they do with a MonitoredSession.run call.
View source
View source
View source


Produces a slice of each Tensor in tensor_list. (deprecated)
Implemented using a Queue -- a QueueRunner for the Queue is added to the current Graph's QUEUE_RUNNER collection.
Input pipelines based on Queues are not supported when eager execution is enabled. Please use the tf.data API to ingest data under eager execution.


Starts all queue runners collected in the graph. (deprecated)
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.train.queue_runner.start_queue_runners
This is a companion method to add_queue_runner(). It just starts threads for all queue runners collected in the graph. It returns the list of all threads.
Not compatible with eager execution. To ingest data under eager execution, use the tf.data API instead.


Output strings (e.g. filenames) to a queue for an input pipeline. (deprecated)
Input pipelines based on Queues are not supported when eager execution is enabled. Please use the tf.data API to ingest data under eager execution.


Returns a iterator for reading Event protocol buffers from an event file.
You can use this function to read events written to an event file. It returns a Python iterator that yields Event protocol buffers.
Example: Print the contents of an events file.
Example: Print selected summary values.
Example: Continuously check for new summary values.
See the protocol buffer definitions of Event and Summary for more information about their attributes.


A training helper that checkpoints models and computes summaries.
This class is deprecated. Please use tf.compat.v1.train.MonitoredTrainingSession instead.
The Supervisor is a small wrapper around a Coordinator, a Saver, and a SessionManager that takes care of common needs of TensorFlow training programs.
Within the with sv.managed_session() block all variables in the graph have been initialized. In addition, a few services have been started to checkpoint the model and add summaries to the event log.
If the program crashes and is restarted, the managed session automatically reinitialize variables from the most recent checkpoint.
The supervisor is notified of any exception raised by one of the services. After an exception is raised, should_stop() returns True. In that case the training loop should also stop. This is why the training loop has to check for sv.should_stop().
Exceptions that indicate that the training inputs have been exhausted, tf.errors.OutOfRangeError, also cause sv.should_stop() to return True but are not re-raised from the with block: they indicate a normal termination.
To train with replicas you deploy the same program in a Cluster. One of the tasks must be identified as the chief: the task that handles initialization, checkpoints, summaries, and recovery. The other tasks depend on the chief for these services.
The only change you have to do to the single program code is to indicate if the program is running as the chief.
In the chief task, the Supervisor works exactly as in the first example above. In the other tasks sv.managed_session() waits for the Model to have been initialized before returning a session to the training code. The non-chief tasks depend on the chief task for initializing the model.
If one of the tasks crashes and restarts, managed_session() checks if the Model is initialized. If yes, it just creates a session and returns it to the training code that proceeds normally. If the model needs to be initialized, the chief task takes care of reinitializing it; the other tasks just wait for the model to have been initialized.
Whether you are running on your machine or in the cluster you can use the following values for the --master flag:
Specifying '' requests an in-process session that does not use RPC.
Specifying 'local' requests a session that uses the RPC-based "Master interface" to run TensorFlow programs. See tf.train.Server.create_local_server for details.
Specifying 'grpc://hostname:port' requests a session that uses the RPC interface to a specific host, and also allows the in-process master to access remote tensorflow workers. Often, it is appropriate to pass server.target (for some tf.distribute.Server named `server).
managed_session() launches the Checkpoint and Summary services (threads). If you need more services to run you can simply launch them in the block controlled by managed_session().
Example: Start a thread to print losses. We want this thread to run every 60 seconds, so we launch it with sv.loop().
managed_session() launches the "summary" and "checkpoint" threads which use either the optionally summary_op and saver passed to the constructor, or default ones created automatically by the supervisor. If you want to run your own summary and checkpointing logic, disable these services by passing None to the summary_op and saver parameters.
Example: Create summaries manually every 100 steps in the chief.
managed_session() only supports initializing the model by running an init_op or restoring from the latest checkpoint. If you have special initialization needs, see how to specify a local_init_op when creating the supervisor. You can also use the SessionManager directly to create a session and check if it could be initialized automatically.
The Coordinator can be useful if you want to run multiple threads during your training.
View source
Start a LooperThread that calls a function periodically.
If timer_interval_secs is None the thread calls target(*args, **kwargs) repeatedly. Otherwise it calls it every timer_interval_secs seconds. The thread terminates when a stop is requested.
The started thread is added to the list of threads managed by the supervisor so it does not need to be passed to the stop() method.
View source
Make sure the model is ready to be used.
Create a session on 'master', recovering or initializing the model as needed, or wait for a session to be ready. If running as the chief and start_standard_service is set to True, also call the session manager to start the standard services.
View source
Request that the coordinator stop the threads.
See Coordinator.request_stop().
View source
Check if the coordinator was told to stop.
See Coordinator.should_stop().
View source
Start threads for QueueRunners.
Note that the queue runners collected in the graph key QUEUE_RUNNERS are already started automatically when you create a session with the supervisor, so unless you have non-collected queue runners to start you do not need to call this explicitly.
Queues are not compatible with eager execution. To ingest data when eager execution is enabled, use the tf.data API.
View source
Start the standard services for 'sess'.
This starts services in the background. The services started depend on the parameters to the constructor and may include:
View source
Stop the services and the coordinator.
This does not close the session.
View source
Context handler to stop the supervisor when an exception is raised.
See Coordinator.stop_on_exception().
View source
Indicate that a summary was computed.
View source
Block waiting for the coordinator to stop.
View source
Start a LooperThread that calls a function periodically.
If timer_interval_secs is None the thread calls target(*args, **kwargs) repeatedly. Otherwise it calls it every timer_interval_secs seconds. The thread terminates when a stop is requested.
The started thread is added to the list of threads managed by the supervisor so it does not need to be passed to the stop() method.
View source
Returns a context manager for a managed session.
This context manager creates and automatically recovers a session. It optionally starts the standard services that handle checkpoints and summaries. It monitors exceptions raised from the with block or from the services and stops the supervisor as needed.
The context manager is typically used as follows:
An exception raised from the with block or one of the service threads is raised again when the block exits. This is done after stopping all threads and closing the session. For example, an AbortedError exception, raised in case of preemption of one of the workers in a distributed model, is raised again when the block exits.
If you want to retry the training loop in case of preemption you can do it as follows:
As a special case, exceptions used for control flow, such as OutOfRangeError which reports that input queues are exhausted, are not raised again from the with block: they indicate a clean termination of the training loop and are considered normal termination.
View source
Make sure the model is ready to be used.
Create a session on 'master', recovering or initializing the model as needed, or wait for a session to be ready. If running as the chief and start_standard_service is set to True, also call the session manager to start the standard services.
View source
Request that the coordinator stop the threads.
See Coordinator.request_stop().
View source
Check if the coordinator was told to stop.
See Coordinator.should_stop().
View source
Start threads for QueueRunners.
Note that the queue runners collected in the graph key QUEUE_RUNNERS are already started automatically when you create a session with the supervisor, so unless you have non-collected queue runners to start you do not need to call this explicitly.
Queues are not compatible with eager execution. To ingest data when eager execution is enabled, use the tf.data API.
View source
Start the standard services for 'sess'.
This starts services in the background. The services started depend on the parameters to the constructor and may include:
View source
Stop the services and the coordinator.
This does not close the session.
View source
Context handler to stop the supervisor when an exception is raised.
See Coordinator.stop_on_exception().
View source
Indicate that a summary was computed.
View source
Block waiting for the coordinator to stop.


Class to synchronize, aggregate gradients and pass them to the optimizer.
Inherits From: Optimizer
This class is deprecated. For synchronous training, please use Distribution Strategies.
In a typical asynchronous training environment, it's common to have some stale gradients. For example, with a N-replica asynchronous training, gradients will be applied to the variables N times independently. Depending on each replica's training speed, some gradients might be calculated from copies of the variable from several steps back (N-1 steps on average). This optimizer avoids stale gradients by collecting gradients from all replicas, averaging them, then applying them to the variables in one shot, after which replicas can fetch the new variables and continue.
The following accumulators/queue are created:
The following local variable is created:
The optimizer adds nodes to the graph to collect gradients and pause the trainers until variables are updated. For the Parameter Server job:
In the training program, every worker will run the train_op as if not synchronized.
To use SyncReplicasOptimizer with an Estimator, you need to send sync_replicas_hook while calling the fit.
View source
Apply gradients to variables.
This contains most of the synchronization implementation and also wraps the apply_gradients() from the real optimizer.
View source
Compute gradients of "loss" for the variables in "var_list".
This simply wraps the compute_gradients() from the real optimizer. The gradients will be aggregated in the apply_gradients() so that user can modify the gradients like clipping with per replica global norm if needed. The global norm with aggregated gradients can be bad as one replica's huge gradients can hurt the gradients from other replicas.
View source
Returns the QueueRunner for the chief to execute.
This includes the operations to synchronize replicas: aggregate gradients, apply to variables, increment global step, insert tokens to token queue.
Note that this can only be called after calling apply_gradients() which actually generates this queuerunner.
View source
Returns the op to fill the sync_token_queue with the tokens.
This is supposed to be executed in the beginning of the chief/sync thread so that even if the total_num_replicas is less than replicas_to_aggregate, the model can still proceed as the replicas can compute multiple steps per variable update. Make sure: num_tokens >= replicas_to_aggregate - total_num_replicas.
View source
View source
Return a slot named "name" created for "var" by the Optimizer.
This simply wraps the get_slot() from the actual optimizer.
View source
Return a list of the names of slots created by the Optimizer.
This simply wraps the get_slot_names() from the actual optimizer.
View source
Creates a hook to handle SyncReplicasHook ops such as initialization.
View source
Add operations to minimize loss by updating var_list.
This method simply combines calls compute_gradients() and apply_gradients(). If you want to process the gradient before applying them call compute_gradients() and apply_gradients() explicitly instead of using this function.
When eager execution is enabled, loss should be a Python function that takes no arguments and computes the value to be minimized. Minimization (and gradient computation) is done with respect to the elements of var_list if not None, else with respect to any trainable variables created during the execution of the loss function. gate_gradients, aggregation_method, colocate_gradients_with_ops and grad_loss are ignored when eager execution is enabled.
View source
Fetches a list of optimizer variables in the default graph.
This wraps variables() from the actual optimizer. It does not include the SyncReplicasOptimizer's local step.


Updates the content of the 'checkpoint' file. (deprecated)
This updates the checkpoint file containing a CheckpointState proto.


Warm-starts a model using the given settings.
If you are using a tf.estimator.Estimator, this will automatically be called during training.
Defaults to '.*', which warm-starts all variables in the TRAINABLE_VARIABLES collection. Note that this excludes variables such as accumulators and moving statistics from batch norm.


Creates a tf.compat.v1.Session for a worker.
Inherits From: SessionCreator
View source


Returns all variables created with trainable=True.
When passed trainable=True, the Variable() constructor automatically adds new variables to the graph collection GraphKeys.TRAINABLE_VARIABLES. This convenience function returns the contents of that collection.


Transposes a.
Permutes the dimensions according to perm.
The returned tensor's dimension i will correspond to the input dimension perm[i]. If perm is not given, it is set to (n-1...0), where n is the rank of the input tensor. Hence by default, this operation performs a regular matrix transpose on 2-D input Tensors. If conjugate is True and a.dtype is either complex64 or complex128 then the values of a are conjugated and transposed.
In numpy transposes are memory-efficient constant time operations as they simply return a new view of the same data with adjusted strides.
TensorFlow does not support strides, so transpose returns a new tensor with the items permuted.


Initializer that generates a truncated normal distribution.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.initializers.truncated_normal
These values are similar to values from a random_normal_initializer except that values more than two standard deviations from the mean are discarded and re-drawn. This is the recommended initializer for neural network weights and filters.
View source
Instantiates an initializer from a configuration dictionary.
View source
Returns the configuration of the initializer as a JSON-serializable dict.
View source
Returns a tensor object initialized as specified by the initializer.


Group tensors together.
This creates a tuple of tensors with the same values as the tensors argument, except that the value of each tensor is only returned after the values of all tensors have been computed.
control_inputs contains additional ops that have to finish before this op finishes, but whose outputs are not returned.
This can be used as a "join" mechanism for parallel computations: all the argument tensors can be computed in parallel, but the values of any tensor returned by tuple are only available after all the parallel computations are done.
See also tf.group and tf.control_dependencies.


Public TensorFlow type definitions.
For details, see https://github.com/tensorflow/community/blob/master/rfcs/20200211-tf-types.md
experimental module: Public API for tf.types.experimental namespace.


Public API for tf.types.experimental namespace.
TensorLike: Union of all types that can be converted to a tf.Tensor by tf.convert_to_tensor.


Initializer that generates tensors without scaling variance.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.initializers.uniform_unit_scaling
When initializing a deep network, it is in principle advantageous to keep the scale of the input variance constant, so it does not explode or diminish by reaching the final layer. If the input is x and the operation x * W, and we want to initialize W uniformly at random, we need to pick W from
to keep the scale intact, where dim = W.shape[0] (the size of the input). A similar calculation for convolutional networks gives an analogous result with dim equal to the product of the first 3 dimensions. When nonlinearities are present, we need to multiply this by a constant factor. See (Sussillo et al., 2014) for deeper motivation, experiments and the calculation of constants. In section 2.3 there, the constants were numerically computed: for a linear layer it's 1.0, relu: ~1.43, tanh: ~1.15.
Sussillo et al., 2014 (pdf)
View source
Instantiates an initializer from a configuration dictionary.
View source
Returns the configuration of the initializer as a JSON-serializable dict.
View source
Returns a tensor object initialized as specified by the initializer.


Public API for tf.user_ops namespace.
my_fact(...): Example of overriding the generated code for an Op.


Example of overriding the generated code for an Op.