See the Variables Guide.
Inherits From: Variable
A variable maintains state in the graph across calls to run(). You add a variable to the graph by constructing an instance of the class Variable.
The Variable() constructor requires an initial value for the variable, which can be a Tensor of any type and shape. The initial value defines the type and shape of the variable. After construction, the type and shape of the variable are fixed. The value can be changed using one of the assign methods.
If you want to change the shape of a variable later you have to use an assign Op with validate_shape=False.
Just like any Tensor, variables created with Variable() can be used as inputs for other Ops in the graph. Additionally, all the operators overloaded for the Tensor class are carried over to variables, so you can also add nodes to the graph by just doing arithmetic on variables.
When you launch the graph, variables have to be explicitly initialized before you can run Ops that use their value. You can initialize a variable by running its initializer op, restoring the variable from a save file, or simply running an assign Op that assigns a value to the variable. In fact, the variable initializer op is just an assign Op that assigns the variable's initial value to the variable itself.
The most common initialization pattern is to use the convenience function global_variables_initializer() to add an Op to the graph that initializes all the variables. You then run that Op after launching the graph.
If you need to create a variable with an initial value dependent on another variable, use the other variable's initialized_value(). This ensures that variables are initialized in the right order.
All variables are automatically collected in the graph where they are created. By default, the constructor adds the new variable to the graph collection GraphKeys.GLOBAL_VARIABLES. The convenience function global_variables() returns the contents of that collection.
When building a machine learning model it is often convenient to distinguish between variables holding the trainable model parameters and other variables such as a global step variable used to count training steps. To make this easier, the variable constructor supports a trainable=<bool> parameter. If True, the new variable is also added to the graph collection GraphKeys.TRAINABLE_VARIABLES. The convenience function trainable_variables() returns the contents of this collection. The various Optimizer classes use this collection as the default list of variables to optimize.
Here, adding use_resource=True when constructing the variable will fix any nondeterminism issues:
To use the replacement for variables which does not have these issues:
Note that this is different from initialized_value() which runs the op that initializes the variable before returning its value. This method returns the tensor that is used by the op that initializes the variable.
class SaveSliceInfo
View source
Assigns a new value to the variable.
This is essentially a shortcut for assign(self, value).
View source
Adds a value to this variable.
This is essentially a shortcut for assign_add(self, delta).
View source
Subtracts a value from this variable.
This is essentially a shortcut for assign_sub(self, delta).
View source
Assigns tf.IndexedSlices to this variable batch-wise.
Analogous to batch_gather. This assumes that this variable and the sparse_delta IndexedSlices have a series of leading dimensions that are the same for all of them, and the updates are performed on the last dimension of indices. In other words, the dimensions should be the following:
num_prefix_dims = sparse_delta.indices.ndims - 1 batch_dim = num_prefix_dims + 1 sparse_delta.updates.shape = sparse_delta.indices.shape + var.shape[ batch_dim:]
where
sparse_delta.updates.shape[:num_prefix_dims] == sparse_delta.indices.shape[:num_prefix_dims] == var.shape[:num_prefix_dims]
And the operation performed can be expressed as:
var[i_1, ..., i_n, sparse_delta.indices[i_1, ..., i_n, j]] = sparse_delta.updates[ i_1, ..., i_n, j]
When sparse_delta.indices is a 1D tensor, this operation is equivalent to scatter_update.
To avoid this operation one can looping over the first ndims of the variable and using scatter_update on the subtensors that result of slicing the first dimension. This is a valid option for ndims = 1, but less efficient than this implementation.
View source
Increments this variable until it reaches limit. (deprecated)
When that Op is run it tries to increment the variable by 1. If incrementing the variable would bring it above limit then the Op raises the exception OutOfRangeError.
If no error is raised, the Op outputs the value of the variable before the increment.
This is essentially a shortcut for count_up_to(self, limit).
View source
In a session, computes and returns the value of this variable.
This is not a graph construction method, it does not add ops to the graph.
This convenience method requires a session where the graph containing this variable has been launched. If no session is passed, the default session is used. See tf.compat.v1.Session for more information on launching a graph and on sessions.
View source
DEPRECATED FUNCTION
View source
Returns a Variable object created from variable_def.
View source
Gather slices from params into a Tensor with shape specified by indices.
See tf.gather_nd for details.
View source
Alias of Variable.shape.
View source
Returns the value of the initialized variable. (deprecated)
You should use this instead of the variable itself to initialize another variable with a value that depends on the value of this variable.
View source
Load new value into this variable. (deprecated)
Writes new value to variable's memory. Doesn't add ops to the graph.
This convenience method requires a session where the graph containing this variable has been launched. If no session is passed, the default session is used. See tf.compat.v1.Session for more information on launching a graph and on sessions.
View source
Returns the value of this variable, read in the current context.
Can be different from value() if it's on another device, with control dependencies, etc.
View source
Returns a hashable reference object to this Variable.
The primary use case for this API is to put variables in a set/dictionary. We can't put variables in a set/dictionary as variable.__hash__() is no longer available starting Tensorflow 2.0.
The following will raise an exception starting 2.0
Instead, we can use variable.ref().
Also, the reference object provides .deref() function that returns the original Variable.
View source
Adds tf.IndexedSlices to this variable.
View source
Divide this variable by tf.IndexedSlices.
View source
Updates this variable with the max of tf.IndexedSlices and itself.
View source
Updates this variable with the min of tf.IndexedSlices and itself.
View source
Multiply this variable by tf.IndexedSlices.
View source
Applies sparse addition to individual values or slices in a Variable.
The Variable has rank P and indices is a Tensor of rank Q.
indices must be integer tensor, containing indices into self. It must be shape [d_0, ..., d_{Q-2}, K] where 0 < K <= P.
The innermost dimension of indices (with length K) corresponds to indices into elements (if K = P) or slices (if K < P) along the Kth dimension of self.
updates is Tensor of rank Q-1+P-K with shape:
For example, say we want to add 4 scattered elements to a rank-1 tensor to 8 elements. In Python, that update would look like this:
The resulting update to v would look like this:
See tf.scatter_nd for more details about how to make updates to slices.
View source
Applies sparse subtraction to individual values or slices in a Variable.
Assuming the variable has rank P and indices is a Tensor of rank Q.
indices must be integer tensor, containing indices into self. It must be shape [d_0, ..., d_{Q-2}, K] where 0 < K <= P.
The innermost dimension of indices (with length K) corresponds to indices into elements (if K = P) or slices (if K < P) along the Kth dimension of self.
updates is Tensor of rank Q-1+P-K with shape:
For example, say we want to add 4 scattered elements to a rank-1 tensor to 8 elements. In Python, that update would look like this:
The resulting update to v would look like this:
See tf.scatter_nd for more details about how to make updates to slices.
View source
Applies sparse assignment to individual values or slices in a Variable.
The Variable has rank P and indices is a Tensor of rank Q.
indices must be integer tensor, containing indices into self. It must be shape [d_0, ..., d_{Q-2}, K] where 0 < K <= P.
The innermost dimension of indices (with length K) corresponds to indices into elements (if K = P) or slices (if K < P) along the Kth dimension of self.
updates is Tensor of rank Q-1+P-K with shape:
For example, say we want to add 4 scattered elements to a rank-1 tensor to 8 elements. In Python, that update would look like this:
The resulting update to v would look like this:
See tf.scatter_nd for more details about how to make updates to slices.
View source
Subtracts tf.IndexedSlices from this variable.
View source
Assigns tf.IndexedSlices to this variable.
View source
Overrides the shape for this variable.
View source
Gather slices from params axis axis according to indices.
This function supports a subset of tf.gather, see tf.gather for details on usage.
View source
Converts a Variable to a VariableDef protocol buffer.
View source
Returns the last snapshot of this variable.
You usually do not need to call this method as all ops that need the value of the variable call it automatically through a convert_to_tensor() call.
Returns a Tensor which holds the value of the variable. You can not assign a new value to this tensor as it is not a reference to the variable.
To avoid copies, if the consumer of the returned value is on the same device as the variable, this actually returns the live value of the variable, not a copy. Updates to the variable are seen by the consumer. If the consumer is on a different device it will get a copy of the variable.
View source
Computes the absolute value of a tensor.
Given a tensor of integer or floating-point values, this operation returns a tensor of the same type, where each element contains the absolute value of the corresponding element in the input.
Given a tensor x of complex numbers, this operation returns a tensor of type float32 or float64 that is the absolute value of each element in x. For a complex number \(a + bj\), its absolute value is computed as \(\sqrt{a^2 + b^2}\).
View source
The operation invoked by the Tensor.add operator.
Purpose in the API:
View source
View source
Divides x / y elementwise (using Python 2 division operator semantics). (deprecated)
This function divides x and y, forcing Python 2 semantics. That is, if x and y are both integers then the result will be an integer. This is in contrast to Python 3, where division with / is always a float while division with // is always an integer.
View source
Compares two variables element-wise for equality.
View source
Divides x / y elementwise, rounding toward the most negative integer.
The same as tf.compat.v1.div(x,y) for integers, but uses tf.floor(tf.compat.v1.div(x,y)) for floating point arguments so that the result is always an integer (though possibly an integer represented as floating point). This op is generated by x // y floor division in Python 3 and in Python 2.7 with from __future__ import division.
x and y must have the same type, and the result will have the same type as well.
Returns the truth value of (x >= y) element-wise.
View source
Creates a slice helper object given a variable.
This allows creating a sub-tensor from part of the current contents of a variable. See tf.Tensor.getitem for detailed examples of slicing.
This function in addition also allows assignment to a sliced range. This is similar to __setitem__ functionality in Python. However, the syntax is different so that the user can capture the assignment operation for grouping or passing to sess.run(). For example,
Note that assignments currently do not support NumPy broadcasting semantics.
Returns the truth value of (x > y) element-wise.
View source
View source
Dummy method to prevent iteration.
Do not call.
NOTE(mrry): If we register getitem as an overloaded operator, Python will valiantly attempt to iterate over the variable's Tensor from 0 to infinity. Declaring this method prevents this unintended behavior.
Returns the truth value of (x <= y) element-wise.
Returns the truth value of (x < y) element-wise.
View source
Multiplies matrix a by matrix b, producing a * b.
The inputs must, following any transpositions, be tensors of rank >= 2 where the inner 2 dimensions specify valid matrix multiplication dimensions, and any further outer dimensions specify matching batch size.
Both matrices must be of the same type. The supported types are: float16, float32, float64, int32, complex64, complex128.
Either matrix can be transposed or adjointed (conjugated and transposed) on the fly by setting one of the corresponding flag to True. These are False by default.
If one or both of the matrices contain a lot of zeros, a more efficient multiplication algorithm can be used by setting the corresponding a_is_sparse or b_is_sparse flag to True. These are False by default. This optimization is only available for plain matrices (rank-2 tensors) with datatypes bfloat16 or float32.
A simple 2-D tensor matrix multiplication:
A batch matrix multiplication with batch shape [2]:
Since python >= 3.5 the @ operator is supported (see PEP 465). In TensorFlow, it simply calls the tf.matmul() function, so the following lines are equivalent:
output[..., i, j] = sum_k (a[..., i, k] * b[..., k, j]), for all indices i, j.
View source
Returns element-wise remainder of division. When x < 0 xor y < 0 is
true, this follows Python semantics in that the result here is consistent with a flooring divide. E.g. floor(x / y) * y + mod(x, y) = x.
View source
Dispatches cwise mul for "DenseDense" and "DenseSparse".
View source
Compares two variables element-wise for equality.
Computes numerical negative value element-wise.
I.e., \(y = -x\).
View source
View source
Computes the power of one value to another.
Given a tensor x and a tensor y, this operation computes \(x^y\) for corresponding elements in x and y. For example:
View source
The operation invoked by the Tensor.add operator.
Purpose in the API:
View source
View source
Divides x / y elementwise (using Python 2 division operator semantics). (deprecated)
This function divides x and y, forcing Python 2 semantics. That is, if x and y are both integers then the result will be an integer. This is in contrast to Python 3, where division with / is always a float while division with // is always an integer.
View source
Divides x / y elementwise, rounding toward the most negative integer.
The same as tf.compat.v1.div(x,y) for integers, but uses tf.floor(tf.compat.v1.div(x,y)) for floating point arguments so that the result is always an integer (though possibly an integer represented as floating point). This op is generated by x // y floor division in Python 3 and in Python 2.7 with from __future__ import division.
x and y must have the same type, and the result will have the same type as well.
View source
Multiplies matrix a by matrix b, producing a * b.
The inputs must, following any transpositions, be tensors of rank >= 2 where the inner 2 dimensions specify valid matrix multiplication dimensions, and any further outer dimensions specify matching batch size.
Both matrices must be of the same type. The supported types are: float16, float32, float64, int32, complex64, complex128.
Either matrix can be transposed or adjointed (conjugated and transposed) on the fly by setting one of the corresponding flag to True. These are False by default.
If one or both of the matrices contain a lot of zeros, a more efficient multiplication algorithm can be used by setting the corresponding a_is_sparse or b_is_sparse flag to True. These are False by default. This optimization is only available for plain matrices (rank-2 tensors) with datatypes bfloat16 or float32.
A simple 2-D tensor matrix multiplication:
A batch matrix multiplication with batch shape [2]:
Since python >= 3.5 the @ operator is supported (see PEP 465). In TensorFlow, it simply calls the tf.matmul() function, so the following lines are equivalent:
output[..., i, j] = sum_k (a[..., i, k] * b[..., k, j]), for all indices i, j.
View source
Returns element-wise remainder of division. When x < 0 xor y < 0 is
true, this follows Python semantics in that the result here is consistent with a flooring divide. E.g. floor(x / y) * y + mod(x, y) = x.
View source
Dispatches cwise mul for "DenseDense" and "DenseSparse".
View source
View source
Computes the power of one value to another.
Given a tensor x and a tensor y, this operation computes \(x^y\) for corresponding elements in x and y. For example:
View source
Returns x - y element-wise.
View source
Divides x / y elementwise (using Python 3 division operator semantics).
This function forces Python 3 division operator semantics where all integer arguments are cast to floating types first. This op is generated by normal x / y division in Python 3 and in Python 2.7 with from __future__ import division. If you want integer division that rounds down, use x // y or tf.math.floordiv.
x and y must have the same numeric type. If the inputs are floating point, the output will have the same type. If the inputs are integral, the inputs are cast to float32 for int8 and int16 and float64 for int32 and int64 (matching the behavior of Numpy).
View source
View source
Returns x - y element-wise.
View source
Divides x / y elementwise (using Python 3 division operator semantics).
This function forces Python 3 division operator semantics where all integer arguments are cast to floating types first. This op is generated by normal x / y division in Python 3 and in Python 2.7 with from __future__ import division. If you want integer division that rounds down, use x // y or tf.math.floordiv.
x and y must have the same numeric type. If the inputs are floating point, the output will have the same type. If the inputs are integral, the inputs are cast to float32 for int8 and int16 and float64 for int32 and int64 (matching the behavior of Numpy).
View source


Indicates how a distributed variable will be aggregated.
tf.distribute.Strategy distributes a model by making multiple copies (called "replicas") acting data-parallel on different elements of the input batch. When performing some variable-update operation, say var.assign_add(x), in a model, we need to resolve how to combine the different values for x computed in the different replicas.


Variable scope object to carry defaults to provide to get_variable.
Many of the arguments we need for get_variable in a variable store are most easily handled with a context. This object is used for the defaults.
View source
Get this scope's variables.
View source
Gets an existing variable with this name or create a new one.
View source
Get this scope's global variables.
View source
Get this scope's local variables.
View source
Reuse variables in this scope.
View source
Set caching_device for this scope.
View source
Set custom getter for this scope.
View source
Set data type for this scope.
View source
Set initializer for this scope.
View source
Set partitioner for this scope.
View source
Set regularizer for this scope.
View source
Sets whether to use ResourceVariables for this scope.
View source
Get this scope's trainable variables.


Returns an Op that initializes a list of variables.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.initializers.variables
After you launch the graph in a session, you can run the returned Op to initialize all the variables in var_list. This Op runs all the initializers of the variables in var_list in parallel.
Calling initialize_variables() is equivalent to passing the list of initializers to Group().
If var_list is empty, however, the function still returns an Op that can be run. That Op just has no effect.


Get a partitioner for VariableScope to keep shards below max_shard_bytes.
This partitioner will shard a Variable along one axis, attempting to keep the maximum shard size below max_shard_bytes. In practice, this is not always possible when sharding along only one axis. When this happens, this axis is sharded as much as possible (i.e., every dimension becomes a separate shard).
If the partitioner hits the max_shards limit, then each shard may end up larger than max_shard_bytes. By default max_shards equals None and no limit on the number of shards is enforced.
One reasonable value for max_shard_bytes is (64 << 20) - 1, or almost 64MB, to keep below the protobuf byte limit.


Scope which defines a variable creation function to be used by variable().
variable_creator is expected to be a function with the following signature:
The creator is supposed to eventually call the next_creator to create a variable if it does want to create a variable and not call Variable or ResourceVariable directly. This helps make creators composable. A creator may choose to create multiple variables, return already existing variables, or simply register that a variable was created and defer to the next creators in line. Creators can also modify the keyword arguments seen by the next creators.
Custom getters in the variable scope will eventually resolve down to these custom creators when they do create variables.
The valid keyword arguments in kwds are:
This set may grow over time, so it's important the signature of creators is as mentioned above.


Deprecated: context manager for defining an op that creates variables.


A context manager for defining ops that creates variables (layers).
This context manager validates that the (optional) values are from the same graph, ensures that graph is the default graph, and pushes a name scope and a variable scope.
If name_or_scope is not None, it is used as is. If name_or_scope is None, then default_name is used. In that case, if the same name has been previously used in the same scope, it will be made unique by appending _N to it.
Variable scope allows you to create new variables and to share already created ones while providing checks to not create or share by accident. For details, see the Variable Scope How To, here we present only a few basic examples.
Simple example of how to create a new variable:
Simple example of how to reenter a premade variable scope safely:
Keep in mind that the counters for default_name are discarded once the parent scope is exited. Therefore when the code re-enters the scope (for instance by saving it), all nested default_name counters will be restarted.
Basic example of sharing a variable AUTO_REUSE:
Basic example of sharing a variable with reuse=True:
Sharing a variable by capturing a scope and setting reuse:
To prevent accidental sharing of variables, we raise an exception when getting an existing variable in a non-reusing scope.
Similarly, we raise an exception when trying to get a variable that does not exist in reuse mode.
Note that the reuse flag is inherited: if we open a reusing scope, then all its sub-scopes become reusing as well.
A note about name scoping: Setting reuse does not impact the naming of other ops such as mult. See related discussion on github#6189
Note that up to and including version 1.0, it was allowed (though explicitly discouraged) to pass False to the reuse argument, yielding undocumented behaviour slightly different from None. Starting at 1.1.0 passing None and False as reuse has exactly the same effect.
A note about using variable scopes in multi-threaded environment: Variable scopes are thread local, so one thread will not see another thread's current scope. Also, when using default_name, unique scopes names are also generated only on a per thread basis. If the same name was used within a different thread, that doesn't prevent a new thread from creating the same scope. However, the underlying variable store is shared across threads (within the same graph). As such, if another thread tries to create a new variable with the same name as a variable created by a previous thread, it will fail unless reuse is True.
Further, each thread starts with an empty variable scope. So if you wish to preserve name prefixes from a scope from the main thread, you should capture the main thread's scope and re-enter it in each thread. For e.g.
View source
View source


Assert that the tensor does not contain any NaN's or Inf's.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.debugging.assert_all_finite


Return the elements, either from x or y, depending on the condition.
If both x and y are None, then this operation returns the coordinates of true elements of condition. The coordinates are returned in a 2-D tensor where the first dimension (rows) represents the number of true elements, and the second dimension (columns) represents the coordinates of the true elements. Keep in mind, the shape of the output tensor can vary depending on how many true values there are in input. Indices are output in row-major order.
If both non-None, x and y must have the same shape. The condition tensor must be a scalar if x and y are scalar. If x and y are tensors of higher rank, then condition must be either a vector with size matching the first dimension of x, or must have the same shape as x.
The condition tensor acts as a mask that chooses, based on the value at each element, whether the corresponding element / row in the output should be taken from x (if true) or y (if false).
If condition is a vector and x and y are higher rank matrices, then it chooses which row (outer dimension) to copy from x and y. If condition has the same shape as x and y, then it chooses which element to copy from x and y.


Repeat body while the condition cond is true.
cond is a callable returning a boolean scalar tensor. body is a callable returning a (possibly nested) tuple, namedtuple or list of tensors of the same arity (length and structure) and types as loop_vars. loop_vars is a (possibly nested) tuple, namedtuple or list of tensors that is passed to both cond and body. cond and body both take as many arguments as there are loop_vars.
In addition to regular Tensors or IndexedSlices, the body may accept and return TensorArray objects. The flows of the TensorArray objects will be appropriately forwarded between loops and during gradient calculations.
Note that while_loop calls cond and body exactly once (inside the call to while_loop, and not at all during Session.run()). while_loop stitches together the graph fragments created during the cond and body calls with some additional graph nodes to create the graph flow that repeats body until cond returns false.
For correctness, tf.while_loop() strictly enforces shape invariants for the loop variables. A shape invariant is a (possibly partial) shape that is unchanged across the iterations of the loop. An error will be raised if the shape of a loop variable after an iteration is determined to be more general than or incompatible with its shape invariant. For example, a shape of [11, None] is more general than a shape of [11, 17], and [11, 21] is not compatible with [11, 17]. By default (if the argument shape_invariants is not specified), it is assumed that the initial shape of each tensor in loop_vars is the same in every iteration. The shape_invariants argument allows the caller to specify a less specific shape invariant for each loop variable, which is needed if the shape varies between iterations. The tf.Tensor.set_shape function may also be used in the body function to indicate that the output loop variable has a particular shape. The shape invariant for SparseTensor and IndexedSlices are treated specially as follows:
a) If a loop variable is a SparseTensor, the shape invariant must be TensorShape([r]) where r is the rank of the dense tensor represented by the sparse tensor. It means the shapes of the three tensors of the SparseTensor are ([None], [None, r], [r]). NOTE: The shape invariant here is the shape of the SparseTensor.dense_shape property. It must be the shape of a vector.
b) If a loop variable is an IndexedSlices, the shape invariant must be a shape invariant of the values tensor of the IndexedSlices. It means the shapes of the three tensors of the IndexedSlices are (shape, [shape[0]], [shape.ndims]).
while_loop implements non-strict semantics, enabling multiple iterations to run in parallel. The maximum number of parallel iterations can be controlled by parallel_iterations, which gives users some control over memory consumption and execution order. For correct programs, while_loop should return the same result for any parallel_iterations > 0.
For training, TensorFlow stores the tensors that are produced in the forward inference and are needed in back propagation. These tensors are a main source of memory consumption and often cause OOM errors when training on GPUs. When the flag swap_memory is true, we swap out these tensors from GPU to CPU. This for example allows us to train RNN models with very long sequences and large batches.
Example with nesting and a namedtuple:
Example using shape_invariants:
Example which demonstrates non-strict semantics: In the following example, the final value of the counter i does not depend on x. So the while_loop can increment the counter parallel to updates of x. However, because the loop counter at one loop iteration depends on the value at the previous iteration, the loop counter itself cannot be incremented in parallel. Hence if we just want the final value of the counter (which we print on the line print(sess.run(i))), then x will never be incremented, but the counter will be updated on a single thread. Conversely, if we want the value of the output (which we print on the line print(sess.run(out).shape)), then the counter may be incremented on its own thread, while x can be incremented in parallel on a separate thread. In the extreme case, it is conceivable that the thread incrementing the counter runs until completion before x is incremented even a single time. The only thing that can never happen is that the thread updating x can never get ahead of the counter thread because the thread incrementing x depends on the value of the counter.


A Reader that outputs the entire contents of a file as a value.
Inherits From: ReaderBase
To use, enqueue filenames in a Queue. The output of Read will be a filename (key) and the contents of that file (value).
See ReaderBase for supported methods.
Readers are not compatible with eager execution. Instead, please use tf.data to get data into your model.
View source
Returns the number of records this reader has produced.
This is the same as the number of Read executions that have succeeded.
View source
Returns the number of work units this reader has finished processing.
View source
Returns the next record (key, value) pair produced by a reader.
Will dequeue a work unit from queue if necessary (e.g. when the Reader needs to start reading from a new file since it has finished with the previous file).
View source
Returns up to num_records (key, value) pairs produced by a reader.
Will dequeue a work unit from queue if necessary (e.g., when the Reader needs to start reading from a new file since it has finished with the previous file). It may return less than num_records even before the last batch.
View source
Restore a reader to its initial clean state.
View source
Restore a reader to a previously saved state.
Not all Readers support being restored, so this can produce an Unimplemented error.
View source
Produce a string tensor that encodes the state of a reader.
Not all Readers support being serialized, so this can produce an Unimplemented error.


Wraps the TF 1.x function fn into a graph function.
The python function fn will be called once with symbolic arguments specified in the signature, traced, and turned into a graph function. Any variables created by fn will be owned by the object returned by wrap_function. The resulting graph function can be called with tensors which match the signature.
Both tf.compat.v1.wrap_function and tf.function create a callable TensorFlow graph. But while tf.function runs all stateful operations (e.g. tf.print) and sequences operations to provide the same semantics as eager execution, wrap_function is closer to the behavior of session.run in TensorFlow 1.x. It will not run any operations unless they are required to compute the function's outputs, either through a data dependency or a control dependency. Nor will it sequence operations.
Unlike tf.function, wrap_function will only trace the Python function once. As with placeholders in TF 1.x, shapes and dtypes must be provided to wrap_function's signature argument.
Since it is only traced once, variables and state may be created inside the function and owned by the function wrapper object.


Public API for tf.xla namespace.
experimental module: Public API for tf.xla.experimental namespace.


Public API for tf.xla.experimental namespace.
compile(...): Builds an operator that compiles and runs computation with XLA. (deprecated)
jit_scope(...): Enable or disable JIT compilation of operators within the scope.


Creates a tensor with all elements set to zero.
See also tf.zeros.
Given a single tensor (tensor), this operation returns a tensor of the same type and shape as tensor with all elements set to zero. Optionally, you can use dtype to specify a new type for the returned tensor.


Concatenates tensors along one dimension.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.concat
See also tf.tile, tf.stack, tf.repeat.
Concatenates the list of tensors values along dimension axis. If values[i].shape = [D0, D1, ... Daxis(i), ...Dn], the concatenated result has shape
where
That is, the data from the input tensors is joined along the axis dimension.
The number of dimensions of the input tensors must match, and all dimensions except axis must be equal.
As in Python, the axis could also be negative numbers. Negative axis are interpreted as counting from the end of the rank, i.e., axis + rank(values)-th dimension.
can be rewritten as


Return true_fn() if the predicate pred is true else false_fn().
true_fn and false_fn both return lists of output tensors. true_fn and false_fn must have the same non-zero number and type of outputs.
Although this behavior is consistent with the dataflow model of TensorFlow, it has frequently surprised users who expected a lazier semantics. Consider the following simple program:
If x < y, the tf.add operation will be executed and tf.square operation will not be executed. Since z is needed for at least one branch of the cond, the tf.multiply operation is always executed, unconditionally.
Note that cond calls true_fn and false_fn exactly once (inside the call to cond, and not at all during Session.run()). cond stitches together the graph fragments created during the true_fn and false_fn calls with some additional graph nodes to ensure that the right branch gets executed depending on the value of pred.
tf.cond supports nested structures as implemented in tensorflow.python.util.nest. Both true_fn and false_fn must return the same (possibly nested) value structure of lists, tuples, and/or named tuples. Singleton lists and tuples form the only exceptions to this: when returned by true_fn and/or false_fn, they are implicitly unpacked to single values.


Public API for tf.config namespace.
experimental module: Public API for tf.config.experimental namespace.
optimizer module: Public API for tf.config.optimizer namespace.
threading module: Public API for tf.config.threading namespace.
class LogicalDevice: Abstraction for a logical device initialized by the runtime.
class LogicalDeviceConfiguration: Configuration class for a logical devices.
class PhysicalDevice: Abstraction for a locally visible physical device.
experimental_connect_to_cluster(...): Connects to the given cluster.
experimental_connect_to_host(...): Connects to a single machine to enable remote execution on it.
experimental_functions_run_eagerly(...): Returns the value of the experimental_run_functions_eagerly setting. (deprecated)
experimental_run_functions_eagerly(...): Enables / disables eager execution of tf.functions. (deprecated)
functions_run_eagerly(...): Returns the value of the run_functions_eagerly setting.
get_logical_device_configuration(...): Get the virtual device configuration for a tf.config.PhysicalDevice.
get_soft_device_placement(...): Get if soft device placement is enabled.
get_visible_devices(...): Get the list of visible physical devices.
list_logical_devices(...): Return a list of logical devices created by runtime.
list_physical_devices(...): Return a list of physical devices visible to the host runtime.
run_functions_eagerly(...): Enables / disables eager execution of tf.functions.
set_logical_device_configuration(...): Set the logical device configuration for a tf.config.PhysicalDevice.
set_soft_device_placement(...): Set if soft device placement is enabled.
set_visible_devices(...): Set the list of visible devices.


Public API for tf.config.experimental namespace.
class ClusterDeviceFilters: Represent a collection of device filters for the remote workers in cluster.
class VirtualDeviceConfiguration: Configuration class for a logical devices.
disable_mlir_bridge(...): Disables experimental MLIR-Based TensorFlow Compiler Bridge.
disable_mlir_graph_optimization(...): Disables experimental MLIR-Based TensorFlow Compiler Optimizations.
enable_mlir_bridge(...): Enables experimental MLIR-Based TensorFlow Compiler Bridge.
enable_mlir_graph_optimization(...): Enables experimental MLIR-Based TensorFlow Compiler Optimizations.
enable_tensor_float_32_execution(...): Enable or disable the use of TensorFloat-32 on supported hardware.
get_device_details(...): Returns details about a physical devices.
get_device_policy(...): Gets the current device policy.
get_memory_growth(...): Get if memory growth is enabled for a PhysicalDevice.
get_memory_usage(...): Get the memory usage, in bytes, for the chosen device.
get_synchronous_execution(...): Gets whether operations are executed synchronously or asynchronously.
get_virtual_device_configuration(...): Get the virtual device configuration for a tf.config.PhysicalDevice.
get_visible_devices(...): Get the list of visible physical devices.
list_logical_devices(...): Return a list of logical devices created by runtime.
list_physical_devices(...): Return a list of physical devices visible to the host runtime.
set_device_policy(...): Sets the current thread device policy.
set_memory_growth(...): Set if memory growth should be enabled for a PhysicalDevice.
set_synchronous_execution(...): Specifies whether operations are executed synchronously or asynchronously.
set_virtual_device_configuration(...): Set the logical device configuration for a tf.config.PhysicalDevice.
set_visible_devices(...): Set the list of visible devices.
tensor_float_32_execution_enabled(...): Returns whether TensorFloat-32 is enabled.


Represent a collection of device filters for the remote workers in cluster.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.experimental.ClusterDeviceFilters
Set device filters for selective jobs and tasks. For each remote worker, the device filters are a list of strings. When any filters are present, the remote worker will ignore all devices which do not match any of its filters. Each filter can be partially specified, e.g. "/job:ps", "/job:worker/replica:3", etc. Note that a device is always visible to the worker it is located on.
For example, to set the device filters for a parameter server cluster:
The device filters can be partically specified. For remote tasks that do not have device filters specified, all devices will be visible to them.
View source
Set the device filters for given job name and task id.


Disables experimental MLIR-Based TensorFlow Compiler Bridge.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.experimental.disable_mlir_bridge


Disables experimental MLIR-Based TensorFlow Compiler Optimizations.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.experimental.disable_mlir_graph_optimization


Enables experimental MLIR-Based TensorFlow Compiler Bridge.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.experimental.enable_mlir_bridge
DO NOT USE, DEV AND TESTING ONLY AT THE MOMENT.
TensorFlow Compiler Bridge (TF Bridge) is responsible for translating parts of TensorFlow graph into a form that can be accepted as an input by a backend compiler such as XLA.


Enables experimental MLIR-Based TensorFlow Compiler Optimizations.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.experimental.enable_mlir_graph_optimization
DO NOT USE, DEV AND TESTING ONLY AT THE MOMENT.
TensorFlow Compiler Optimizations are responsible general graph level optimizations that in the current stack mostly done by Grappler graph optimizers.


Enable or disable the use of TensorFloat-32 on supported hardware.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.experimental.enable_tensor_float_32_execution
TensorFloat-32, or TF32 for short, is a math mode for NVIDIA Ampere GPUs. TensorFloat-32 execution causes certain float32 ops, such as matrix multiplications and convolutions, to run much faster on Ampere GPUs but with reduced precision. This reduced precision should not impact convergence of deep learning models in practice.
TensorFloat-32 is enabled by default. TensorFloat-32 is only supported on Ampere GPUs, so all other hardware will use the full float32 precision regardless of whether TensorFloat-32 is enabled or not. If you want to use the full float32 precision on Ampere, you can disable TensorFloat-32 execution with this function. For example:
To check whether TensorFloat-32 execution is currently enabled, use tf.config.experimental.tensor_float_32_execution_enabled.
If TensorFloat-32 is enabled, float32 inputs of supported ops, such as tf.linalg.matmul, will be rounded from 23 bits of precision to 10 bits of precision in most cases. This allows the ops to execute much faster by utilizing the GPU's tensor cores. TensorFloat-32 has the same dynamic range as float32, meaning it is no more likely to underflow or overflow than float32. Ops still use float32 accumulation when TensorFloat-32 is enabled. Enabling or disabling TensorFloat-32 only affects Ampere GPUs and subsequent GPUs that support TensorFloat-32.
Note TensorFloat-32 is not always used in supported ops, as only inputs of certain shapes are supported. Support for more input shapes and more ops may be added in the future. As a result, precision of float32 ops may decrease in minor versions of TensorFlow.
TensorFloat-32 is also used for some complex64 ops. Currently, TensorFloat-32 is used in fewer cases for complex64 as it is for float32.


Returns details about a physical devices.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.experimental.get_device_details
This API takes in a tf.config.PhysicalDevice returned by tf.config.list_physical_devices. It returns a dict with string keys containing various details about the device. Each key is only supported by a subset of devices, so you should not assume the returned dict will have any particular key.
Currently, details are only returned for GPUs. This function returns an empty dict if passed a non-GPU device.
The returned dict may have the following keys:


Gets the current device policy.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.experimental.get_device_policy
The device policy controls how operations requiring inputs on a specific device (e.g., on GPU:0) handle inputs on a different device (e.g. GPU:1).
This function only gets the device policy for the current thread. Any subsequently started thread will again use the default policy.


Get if memory growth is enabled for a PhysicalDevice.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.experimental.get_memory_growth
If memory growth is enabled for a PhysicalDevice, the runtime initialization will not allocate all memory on the device.


Get the memory usage, in bytes, for the chosen device.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.experimental.get_memory_usage
See https://www.tensorflow.org/api_docs/python/tf/device for specifying device strings.
Does not work for CPU.


Gets whether operations are executed synchronously or asynchronously.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.experimental.get_synchronous_execution
TensorFlow can execute operations synchronously or asynchronously. If asynchronous execution is enabled, operations may return "non-ready" handles.


Sets the current thread device policy.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.experimental.set_device_policy
The device policy controls how operations requiring inputs on a specific device (e.g., on GPU:0) handle inputs on a different device (e.g. GPU:1).
When using the default, an appropriate policy will be picked automatically. The default policy may change over time.
This function only sets the device policy for the current thread. Any subsequently started thread will again use the default policy.


Set if memory growth should be enabled for a PhysicalDevice.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.experimental.set_memory_growth
If memory growth is enabled for a PhysicalDevice, the runtime initialization will not allocate all memory on the device. Memory growth cannot be configured on a PhysicalDevice with virtual devices configured.


Specifies whether operations are executed synchronously or asynchronously.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.experimental.set_synchronous_execution
TensorFlow can execute operations synchronously or asynchronously. If asynchronous execution is enabled, operations may return "non-ready" handles.
When enable is set to None, an appropriate value will be picked automatically. The value picked may change between TensorFlow releases.


Returns whether TensorFloat-32 is enabled.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.experimental.tensor_float_32_execution_enabled
By default, TensorFloat-32 is enabled, but this can be changed with tf.config.experimental.enable_tensor_float_32_execution.


Connects to the given cluster.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.experimental_connect_to_cluster
Will make devices on the cluster available to use. Note that calling this more than once will work, but will invalidate any tensor handles on the old remote devices.
If the given local job name is not present in the cluster specification, it will be automatically added, using an unused port on the localhost.
Device filters can be specified to isolate groups of remote tasks to avoid undesired accesses between workers. Workers accessing resources or launching ops / functions on filtered remote devices will result in errors (unknown devices). For any remote task, if no device filter is present, all cluster devices will be visible; if any device filter is specified, it can only see devices matching at least one filter. Devices on the task itself are always visible. Device filters can be particially specified.
For example, for a cluster set up for parameter server training, the following device filters might be specified:


Connects to a single machine to enable remote execution on it.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.experimental_connect_to_host
Will make devices on the remote host available to use. Note that calling this more than once will work, but will invalidate any tensor handles on the old remote devices.
Using the default job_name of worker, you can schedule ops to run remotely as follows:


Returns the value of the experimental_run_functions_eagerly setting. (deprecated)
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.experimental_functions_run_eagerly


Enables / disables eager execution of tf.functions. (deprecated)
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.experimental_run_functions_eagerly
Calling tf.config.experimental_run_functions_eagerly(True) will make all invocations of tf.function run eagerly instead of running as a traced graph function.
See tf.config.run_functions_eagerly for an example.


Returns the value of the run_functions_eagerly setting.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.functions_run_eagerly


Get the virtual device configuration for a tf.config.PhysicalDevice.
Main aliases
tf.config.experimental.get_virtual_device_configuration
See Migration guide for more details.
tf.compat.v1.config.experimental.get_virtual_device_configuration, tf.compat.v1.config.get_logical_device_configuration
Returns the list of tf.config.LogicalDeviceConfiguration objects previously configured by a call to tf.config.set_logical_device_configuration.


Get if soft device placement is enabled.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.get_soft_device_placement
If enabled, an op will be placed on CPU if any of the following are true


Get the list of visible physical devices.
Main aliases
tf.config.experimental.get_visible_devices
See Migration guide for more details.
tf.compat.v1.config.experimental.get_visible_devices, tf.compat.v1.config.get_visible_devices
Returns the list of PhysicalDevices currently marked as visible to the runtime. A visible device will have at least one LogicalDevice associated with it once the runtime is initialized.
The following example verifies all visible GPUs have been disabled:


Return a list of logical devices created by runtime.
Main aliases
tf.config.experimental.list_logical_devices
See Migration guide for more details.
tf.compat.v1.config.experimental.list_logical_devices, tf.compat.v1.config.list_logical_devices
Logical devices may correspond to physical devices or remote devices in the cluster. Operations and tensors may be placed on these devices by using the name of the tf.config.LogicalDevice.
Calling tf.config.list_logical_devices triggers the runtime to configure any tf.config.PhysicalDevice visible to the runtime, thereby preventing further configuration. To avoid runtime initialization, call tf.config.list_physical_devices instead.


Return a list of physical devices visible to the host runtime.
Main aliases
tf.config.experimental.list_physical_devices
See Migration guide for more details.
tf.compat.v1.config.experimental.list_physical_devices, tf.compat.v1.config.list_physical_devices
Physical devices are hardware devices present on the host machine. By default all discovered CPU and GPU devices are considered visible.
This API allows querying the physical hardware resources prior to runtime initialization. Thus, giving an opportunity to call any additional configuration APIs. This is in contrast to tf.config.list_logical_devices, which triggers runtime initialization in order to list the configured devices.
The following example lists the number of visible GPUs on the host.
However, the number of GPUs available to the runtime may change during runtime initialization due to marking certain devices as not visible or configuring multiple logical devices.


Abstraction for a logical device initialized by the runtime.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.LogicalDevice
A tf.config.LogicalDevice corresponds to an initialized logical device on a tf.config.PhysicalDevice or a remote device visible to the cluster. Tensors and operations can be placed on a specific logical device by calling tf.device with a specified tf.config.LogicalDevice.


Configuration class for a logical devices.
Main aliases
tf.config.experimental.VirtualDeviceConfiguration
See Migration guide for more details.
tf.compat.v1.config.LogicalDeviceConfiguration, tf.compat.v1.config.experimental.VirtualDeviceConfiguration
The class specifies the parameters to configure a tf.config.PhysicalDevice as it is initialized to a tf.config.LogicalDevice during runtime initialization. Not all fields are valid for all device types.
See tf.config.get_logical_device_configuration and tf.config.set_logical_device_configuration for usage examples.


Public API for tf.config.optimizer namespace.
get_experimental_options(...): Get experimental optimizer options.
get_jit(...): Get if JIT compilation is enabled.
set_experimental_options(...): Set experimental optimizer options.
set_jit(...): Set if JIT compilation is enabled.


Get experimental optimizer options.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.optimizer.get_experimental_options
Refer to tf.config.optimizer.set_experimental_options for a list of current options.
Note that optimizations are only applied in graph mode, (within tf.function). In addition, as these are experimental options, the list is subject to change.


Get if JIT compilation is enabled.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.optimizer.get_jit
Note that optimizations are only applied to code that is compiled into a graph. In eager mode, which is the TF2 API default, that means only code that is defined under a tf.function decorator.


Set experimental optimizer options.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.optimizer.set_experimental_options
Note that optimizations are only applied in graph mode, (within tf.function). In addition, as these are experimental options, the list is subject to change.


Set if JIT compilation is enabled.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.optimizer.set_jit
Note that optimizations are only applied to code that is compiled into a graph. In eager mode, which is the TF2 API default, that means only code that is defined under a tf.function decorator.


Abstraction for a locally visible physical device.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.PhysicalDevice
TensorFlow can utilize various devices such as the CPU or multiple GPUs for computation. Before initializing a local device for use, the user can customize certain properties of the device such as it's visibility or memory configuration.
Once a visible tf.config.PhysicalDevice is initialized one or more tf.config.LogicalDevice objects are created. Use tf.config.set_visible_devices to configure the visibility of a physical device and tf.config.set_logical_device_configuration to configure multiple tf.config.LogicalDevice objects for a tf.config.PhysicalDevice. This is useful when separation between models is needed or to simulate a multi-device environment.


Enables / disables eager execution of tf.functions.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.run_functions_eagerly
Calling tf.config.run_functions_eagerly(True) will make all invocations of tf.function run eagerly instead of running as a traced graph function.
This can be useful for debugging.


Set the logical device configuration for a tf.config.PhysicalDevice.
Main aliases
tf.config.experimental.set_virtual_device_configuration
See Migration guide for more details.
tf.compat.v1.config.experimental.set_virtual_device_configuration, tf.compat.v1.config.set_logical_device_configuration
A visible tf.config.PhysicalDevice will by default have a single tf.config.LogicalDevice associated with it once the runtime is initialized. Specifying a list of tf.config.LogicalDeviceConfiguration objects allows multiple devices to be created on the same tf.config.PhysicalDevice.
The following example splits the CPU into 2 logical devices:
The following example splits the GPU into 2 logical devices with 100 MB each:


Set if soft device placement is enabled.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.set_soft_device_placement
If enabled, an op will be placed on CPU if any of the following are true


Set the list of visible devices.
Main aliases
tf.config.experimental.set_visible_devices
See Migration guide for more details.
tf.compat.v1.config.experimental.set_visible_devices, tf.compat.v1.config.set_visible_devices
Specifies which PhysicalDevice objects are visible to the runtime. TensorFlow will only allocate memory and place operations on visible physical devices, as otherwise no LogicalDevice will be created on them. By default all discovered devices are marked as visible.
The following example demonstrates disabling the first GPU on the machine.


Public API for tf.config.threading namespace.
get_inter_op_parallelism_threads(...): Get number of threads used for parallelism between independent operations.
get_intra_op_parallelism_threads(...): Get number of threads used within an individual op for parallelism.
set_inter_op_parallelism_threads(...): Set number of threads used for parallelism between independent operations.
set_intra_op_parallelism_threads(...): Set number of threads used within an individual op for parallelism.


Get number of threads used for parallelism between independent operations.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.threading.get_inter_op_parallelism_threads
Determines the number of threads used by independent non-blocking operations. 0 means the system picks an appropriate number.


Get number of threads used within an individual op for parallelism.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.threading.get_intra_op_parallelism_threads
Certain operations like matrix multiplication and reductions can utilize parallel threads for speed ups. A value of 0 means the system picks an appropriate number.


Set number of threads used for parallelism between independent operations.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.threading.set_inter_op_parallelism_threads
Determines the number of threads used by independent non-blocking operations. 0 means the system picks an appropriate number.


Set number of threads used within an individual op for parallelism.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.config.threading.set_intra_op_parallelism_threads
Certain operations like matrix multiplication and reductions can utilize parallel threads for speed ups. A value of 0 means the system picks an appropriate number.


Creates a constant tensor from a tensor-like object.
If the argument dtype is not specified, then the type is inferred from the type of value.
If dtype is specified the resulting tensor values are cast to the requested dtype.
If shape is set, the value is reshaped to match. Scalars are expanded to fill the shape:
tf.constant has no effect if an eager Tensor is passed as the value, it even transmits gradients:
But, since tf.constant embeds the value in the tf.Graph this fails for symbolic tensors:
tf.constant will always create CPU (host) tensors. In order to create tensors on other devices, use tf.identity. (If the value is an eager Tensor, however, the tensor will be returned unmodified as mentioned above.)


Initializer that generates tensors with constant values.
Initializers allow you to pre-specify an initialization strategy, encoded in the Initializer object, without knowing the shape and dtype of the variable being initialized.
tf.constant_initializer returns an object which when called returns a tensor populated with the value specified in the constructor. This value must be convertible to the requested dtype.
The argument value can be a scalar constant value, or a list of values. Scalars broadcast to whichever shape is requested from the initializer.
If value is a list, then the length of the list must be equal to the number of elements implied by the desired shape of the tensor. If the total number of elements in value is not equal to the number of elements required by the tensor shape, the initializer will raise a TypeError.
View source
Instantiates an initializer from a configuration dictionary.
View source
Returns the configuration of the initializer as a JSON-serializable dict.
View source
Returns a tensor object initialized as specified by the initializer.


Wrapper for Graph.control_dependencies() using the default graph.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.control_dependencies
See tf.Graph.control_dependencies for more details.
When eager execution is enabled, any callable object in the control_inputs list will be called.


Converts the given value to a Tensor.
This function converts Python objects of various types to Tensor objects. It accepts Tensor objects, numpy arrays, Python lists, and Python scalars.
This function can be useful when composing a new operation in Python (such as my_func in the example above). All standard Python op constructors apply this function to each of their Tensor-valued inputs, which allows those ops to accept numpy arrays, Python lists, and scalars in addition to Tensor objects.


Critical section.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.CriticalSection
A CriticalSection object is a resource in the graph which executes subgraphs in serial order. A common example of a subgraph one may wish to run exclusively is the one given by the following function:
Here, a snapshot of v is captured in value; and then v is updated. The snapshot value is returned.
If multiple workers or threads all execute count in parallel, there is no guarantee that access to the variable v is atomic at any point within any thread's calculation of count. In fact, even implementing an atomic counter that guarantees that the user will see each value 0, 1, ..., is currently impossible.
The solution is to ensure any access to the underlying resource v is only processed through a critical section:
The functions f1 and f2 will be executed serially, and updates to v will be atomic.
NOTES
All resource objects, including the critical section and any captured variables of functions executed on that critical section, will be colocated to the same device (host and cpu/gpu).
When using multiple critical sections on the same resources, there is no guarantee of exclusive access to those resources. This behavior is disallowed by default (but see the kwarg exclusive_resource_access).
For example, running the same function in two separate critical sections will not ensure serial execution:
View source
Execute function fn() inside the critical section.
fn should not accept any arguments. To add extra arguments to when calling fn in the critical section, create a lambda:


Decorator to define a function with a custom gradient.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.custom_gradient
This decorator allows fine grained control over the gradients of a sequence for operations. This may be useful for multiple reasons, including providing a more efficient or numerically stable gradient for a sequence of operations.
For example, consider the following function that commonly occurs in the computation of cross entropy and log likelihoods:
Due to numerical instability, the gradient of this function evaluated at x=100 is NaN. For example:
The gradient expression can be analytically simplified to provide numerical stability:
With this definition, the gradient at x=100 will be correctly evaluated as 1.0.
The variable dy is defined as the upstream gradient. i.e. the gradient from all the layers or functions originating from this layer.
By chain rule we know that dy/dx = dy/x_0 * dx_0/dx_1 * ... * dx_i/dx_i+1 * ... * dx_n/dx
In this case the gradient of our current function defined as dx_i/dx_i+1 = (1 - 1 / (1 + e)). The upstream gradient dy would be dx_i+1/dx_i+2 * dx_i+2/dx_i+3 * ... * dx_n/dx. The upstream gradient multiplied by the current gradient is then passed downstream.
In case the function takes multiple variables as input, the grad function must also return the same number of variables. We take the function z = x * y as an example.
Nesting custom gradients can lead to unintuitive results. The default behavior does not correspond to n-th order derivatives. For example
The function grad_grad_fn will be calculating the first order gradient of grad_fn with respect to dy, which is used to generate forward-mode gradient graphs from backward-mode gradient graphs, but is not the same as the second order gradient of op with respect to x.
Instead, wrap nested @tf.custom_gradients in another function:
Additional arguments to the inner @tf.custom_gradient-decorated function control the expected return values of the innermost function.
See also tf.RegisterGradient which registers a gradient function for a primitive TensorFlow operation. tf.custom_gradient on the other hand allows for fine grained control over the gradient computation of a sequence of operations.
Note that if the decorated function uses Variables, the enclosing variable scope must be using ResourceVariables.
In a pure mathematical sense, a vector-argument vector-valued function f's derivatives should be its Jacobian matrix J. Here we are expressing the Jacobian J as a function grad_fn which defines how J will transform a vector grad_ys when left-multiplied with it (grad_ys * J, the vector-Jacobian product, or VJP). This functional representation of a matrix is convenient to use for chain-rule calculation (in e.g. the back-propagation algorithm).
If f uses Variables (that are not part of the inputs), i.e. through get_variable, then grad_fn should have signature g(*grad_ys, variables=None), where variables is a list of the Variables, and return a 2-tuple (grad_xs, grad_vars), where grad_xs is the same as above, and grad_vars is a list<Tensor> with the derivatives of Tensors in y with respect to the variables (that is, grad_vars has one Tensor per variable in variables).


tf.data.Dataset API for input pipelines.
See Importing Data for an overview.
experimental module: Experimental API for building input pipelines.
class Dataset: Represents a potentially large set of elements.
class DatasetSpec: Type specification for tf.data.Dataset.
class FixedLengthRecordDataset: A Dataset of fixed-length records from one or more binary files.
class Iterator: Represents an iterator of a tf.data.Dataset.
class IteratorSpec: Type specification for tf.data.Iterator.
class Options: Represents options for tf.data.Dataset.
class TFRecordDataset: A Dataset comprising records from one or more TFRecord files.
class TextLineDataset: A Dataset comprising lines from one or more text files.


Represents a potentially large set of elements.
The tf.data.Dataset API supports writing descriptive and efficient input pipelines. Dataset usage follows a common pattern:
Iteration happens in a streaming fashion, so the full dataset does not need to fit into memory.
The simplest way to create a dataset is to create it from a python list:
To process lines from files, use tf.data.TextLineDataset:
To process records written in the TFRecord format, use TFRecordDataset:
To create a dataset of all files matching a pattern, use tf.data.Dataset.list_files:
See tf.data.FixedLengthRecordDataset and tf.data.Dataset.from_generator for more ways to create datasets.
Once you have a dataset, you can apply transformations to prepare the data for your model:
Element: A single output from calling next() on a dataset iterator. Elements may be nested structures containing multiple components. For example, the element (1, (3, "apple")) has one tuple nested in another tuple. The components are 1, 3, and "apple".
Component: The leaf in the nested structure of an element.
Elements can be nested structures of tuples, named tuples, and dictionaries. Note that Python lists are not treated as nested structures of components. Instead, lists are converted to tensors and treated as components. For example, the element (1, [1, 2, 3]) has only two components; the tensor 1 and the tensor [1, 2, 3]. Element components can be of any type representable by tf.TypeSpec, including tf.Tensor, tf.data.Dataset, tf.sparse.SparseTensor, tf.RaggedTensor, and tf.TensorArray.
View source
Applies a transformation function to this dataset.
apply enables chaining of custom Dataset transformations, which are represented as functions that take one Dataset argument and return a transformed Dataset.
View source
Returns an iterator which converts all elements of the dataset to numpy.
Use as_numpy_iterator to inspect the content of your dataset. To see element shapes and types, print dataset elements directly instead of using as_numpy_iterator.
This method requires that you are running in eager mode and the dataset's element_spec contains only TensorSpec components.
as_numpy_iterator() will preserve the nested structure of dataset elements.
View source
Combines consecutive elements of this dataset into batches.
The components of the resulting element will have an additional outer dimension, which will be batch_size (or N % batch_size for the last element if batch_size does not divide the number of input elements N evenly and drop_remainder is False). If your program depends on the batches having the same outer dimension, you should set the drop_remainder argument to True to prevent the smaller batch from being produced.
View source
Caches the elements in this dataset.
The first time the dataset is iterated over, its elements will be cached either in the specified file or in memory. Subsequent iterations will use the cached data.
When caching to a file, the cached data will persist across runs. Even the first iteration through the data will read from the cache file. Changing the input pipeline before the call to .cache() will have no effect until the cache file is removed or the filename is changed.
View source
Returns the cardinality of the dataset, if known.
cardinality may return tf.data.INFINITE_CARDINALITY if the dataset contains an infinite number of elements or tf.data.UNKNOWN_CARDINALITY if the analysis fails to determine the number of elements in the dataset (e.g. when the dataset source is a file).
View source
Creates a Dataset by concatenating the given dataset with this dataset.
View source
Enumerates the elements of this dataset.
It is similar to python's enumerate.
View source
Filters this dataset according to predicate.
View source
Maps map_func across this dataset and flattens the result.
Use flat_map if you want to make sure that the order of your dataset stays the same. For example, to flatten a dataset of batches into a dataset of their elements:
tf.data.Dataset.interleave() is a generalization of flat_map, since flat_map produces the same output as tf.data.Dataset.interleave(cycle_length=1)
View source
Creates a Dataset whose elements are generated by generator. (deprecated arguments)
The generator argument must be a callable object that returns an object that supports the iter() protocol (e.g. a generator function).
The elements generated by generator must be compatible with either the given output_signature argument or with the given output_types and (optionally) output_shapes arguments, whichiver was specified.
The recommended way to call from_generator is to use the output_signature argument. In this case the output will be assumed to consist of objects with the classes, shapes and types defined by tf.TypeSpec objects from output_signature argument:
There is also a deprecated way to call from_generator by either with output_types argument alone or together with output_shapes argument. In this case the output of the function will be assumed to consist of tf.Tensor objects with with the types defined by output_types and with the shapes which are either unknown or defined by output_shapes.
View source
Creates a Dataset whose elements are slices of the given tensors.
The given tensors are sliced along their first dimension. This operation preserves the structure of the input tensors, removing the first dimension of each tensor and using it as the dataset dimension. All input tensors must have the same size in their first dimensions.
Note that if tensors contains a NumPy array, and eager execution is not enabled, the values will be embedded in the graph as one or more tf.constant operations. For large datasets (> 1 GB), this can waste memory and run into byte limits of graph serialization. If tensors contains one or more large NumPy arrays, consider the alternative described in this guide.
View source
Creates a Dataset with a single element, comprising the given tensors.
from_tensors produces a dataset containing only a single element. To slice the input tensor into multiple elements, use from_tensor_slices instead.
Note that if tensors contains a NumPy array, and eager execution is not enabled, the values will be embedded in the graph as one or more tf.constant operations. For large datasets (> 1 GB), this can waste memory and run into byte limits of graph serialization. If tensors contains one or more large NumPy arrays, consider the alternative described in this guide.
View source
Maps map_func across this dataset, and interleaves the results.
For example, you can use Dataset.interleave() to process many input files concurrently:
The cycle_length and block_length arguments control the order in which elements are produced. cycle_length controls the number of input elements that are processed concurrently. If you set cycle_length to 1, this transformation will handle one input element at a time, and will produce identical results to tf.data.Dataset.flat_map. In general, this transformation will apply map_func to cycle_length input elements, open iterators on the returned Dataset objects, and cycle through them producing block_length consecutive elements from each iterator, and consuming the next input element each time it reaches the end of an iterator.
Performance can often be improved by setting num_parallel_calls so that interleave will use multiple threads to fetch elements. If determinism isn't required, it can also improve performance to set deterministic=False.
View source
A dataset of all files matching one or more glob patterns.
The file_pattern argument should be a small number of glob patterns. If your filenames have already been globbed, use Dataset.from_tensor_slices(filenames) instead, as re-globbing every filename with list_files may result in poor performance with remote storage systems.
If we had the following files on our filesystem:
If we pass "/path/to/dir/*.py" as the directory, the dataset would produce:
View source
Maps map_func across the elements of this dataset.
This transformation applies map_func to each element of this dataset, and returns a new dataset containing the transformed elements, in the same order as they appeared in the input. map_func can be used to change both the values and the structure of a dataset's elements. For example, adding 1 to each element, or projecting a subset of element components.
The input signature of map_func is determined by the structure of each element in this dataset.
The value or values returned by map_func determine the structure of each element in the returned dataset.
map_func can accept as arguments and return any type of dataset element.
Note that irrespective of the context in which map_func is defined (eager vs. graph), tf.data traces the function and executes it as a graph. To use Python code inside of the function you have a few options:
1) Rely on AutoGraph to convert Python code into an equivalent graph computation. The downside of this approach is that AutoGraph can convert some but not all Python code.
2) Use tf.py_function, which allows you to write arbitrary Python code but will generally result in worse performance than 1). For example:
3) Use tf.numpy_function, which also allows you to write arbitrary Python code. Note that tf.py_function accepts tf.Tensor whereas tf.numpy_function accepts numpy arrays and returns only numpy arrays. For example:
Note that the use of tf.numpy_function and tf.py_function in general precludes the possibility of executing user-defined transformations in parallel (because of Python GIL).
Performance can often be improved by setting num_parallel_calls so that map will use multiple threads to process elements. If deterministic order isn't required, it can also improve performance to set deterministic=False.
View source
Returns the options for this dataset and its inputs.
View source
Combines consecutive elements of this dataset into padded batches.
This transformation combines multiple consecutive elements of the input dataset into a single element.
Like tf.data.Dataset.batch, the components of the resulting element will have an additional outer dimension, which will be batch_size (or N % batch_size for the last element if batch_size does not divide the number of input elements N evenly and drop_remainder is False). If your program depends on the batches having the same outer dimension, you should set the drop_remainder argument to True to prevent the smaller batch from being produced.
Unlike tf.data.Dataset.batch, the input elements to be batched may have different shapes, and this transformation will pad each component to the respective shape in padded_shapes. The padded_shapes argument determines the resulting shape for each dimension of each component in an output element:
See also tf.data.experimental.dense_to_sparse_batch, which combines elements that may have different shapes into a tf.sparse.SparseTensor.
View source
Creates a Dataset that prefetches elements from this dataset.
Most dataset input pipelines should end with a call to prefetch. This allows later elements to be prepared while the current element is being processed. This often improves latency and throughput, at the cost of using additional memory to store prefetched elements.
View source
Creates a Dataset of a step-separated range of values.
View source
Reduces the input dataset to a single element.
The transformation calls reduce_func successively on every element of the input dataset until the dataset is exhausted, aggregating information in its internal state. The initial_state argument is used for the initial state and the final state is returned as the result.
View source
Repeats this dataset so each original value is seen count times.
View source
Creates a Dataset that includes only 1/num_shards of this dataset.
shard is deterministic. The Dataset produced by A.shard(n, i) will contain all elements of A whose index mod n = i.
This dataset operator is very useful when running distributed training, as it allows each worker to read a unique subset.
When reading a single input file, you can shard elements as follows:
View source
Randomly shuffles the elements of this dataset.
This dataset fills a buffer with buffer_size elements, then randomly samples elements from this buffer, replacing the selected elements with new elements. For perfect shuffling, a buffer size greater than or equal to the full size of the dataset is required.
For instance, if your dataset contains 10,000 elements but buffer_size is set to 1,000, then shuffle will initially select a random element from only the first 1,000 elements in the buffer. Once an element is selected, its space in the buffer is replaced by the next (i.e. 1,001-st) element, maintaining the 1,000 element buffer.
reshuffle_each_iteration controls whether the shuffle order should be different for each epoch. In TF 1.X, the idiomatic way to create epochs was through the repeat transformation:
In TF 2.0, tf.data.Dataset objects are Python iterables which makes it possible to also create epochs through Python iteration:
View source
Creates a Dataset that skips count elements from this dataset.
View source
Creates a Dataset with at most count elements from this dataset.
View source
Splits elements of a dataset into multiple elements.
For example, if elements of the dataset are shaped [B, a0, a1, ...], where B may vary for each input element, then for each element in the dataset, the unbatched dataset will contain B consecutive elements of shape [a0, a1, ...].
View source
Combines (nests of) input elements into a dataset of (nests of) windows.
A "window" is a finite dataset of flat elements of size size (or possibly fewer if there are not enough input elements to fill the window and drop_remainder evaluates to False).
The shift argument determines the number of input elements by which the window moves on each iteration. If windows and elements are both numbered starting at 0, the first element in window k will be element k * shift of the input dataset. In particular, the first element of the first window will always be the first element of the input dataset.
The stride argument determines the stride of the input elements, and the shift argument determines the shift of the window.
Note that when the window transformation is applied to a dataset of nested elements, it produces a dataset of nested windows.
View source
Returns a new tf.data.Dataset with the given options set.
The options are "global" in the sense they apply to the entire dataset. If options are set multiple times, they are merged as long as different options do not use different non-default values.
View source
Creates a Dataset by zipping together the given datasets.
This method has similar semantics to the built-in zip() function in Python, with the main difference being that the datasets argument can be an arbitrary nested structure of Dataset objects.
View source
View source
Creates an iterator for elements of this dataset.
The returned iterator implements the Python Iterator protocol.
View source
Returns the length of the dataset if it is known and finite.
This method requires that you are running in eager mode, and that the length of the dataset is known and non-infinite. When the length may be unknown or infinite, or if you are running in graph mode, use tf.data.Dataset.cardinality instead.
View source


Type specification for tf.data.Dataset.
Inherits From: TypeSpec
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.DatasetSpec, tf.compat.v1.data.experimental.DatasetStructure
See tf.TypeSpec for more information about TensorFlow type specifications.
In particular, all values that are compatible with this TypeSpec must be an instance of this type.
View source
Creates a DatasetSpec for the given tf.data.Dataset value.
View source
Returns true if spec_or_value is compatible with this TypeSpec.
View source
Returns the most specific TypeSpec compatible with self and other.
View source
Return self==value.
View source
Return self!=value.


Experimental API for building input pipelines.
This module contains experimental Dataset sources and transformations that can be used in conjunction with the tf.data.Dataset API. Note that the tf.data.experimental API is not subject to the same backwards compatibility guarantees as tf.data, but we will provide deprecation advice in advance of removing existing functionality.
See Importing Data for an overview.
service module: API for using the tf.data service.
class AutoShardPolicy: Represents the type of auto-sharding we enable.
class CheckpointInputPipelineHook: Checkpoints input pipeline state every N steps or seconds.
class CsvDataset: A Dataset comprising lines from one or more CSV files.
class DistributeOptions: Represents options for distributed data processing.
class MapVectorizationOptions: Represents options for the MapVectorization optimization.
class OptimizationOptions: Represents options for dataset optimizations.
class Optional: Represents a value that may or may not be present.
class RandomDataset: A Dataset of pseudorandom values.
class Reducer: A reducer is used for reducing a set of elements.
class SqlDataset: A Dataset consisting of the results from a SQL query.
class StatsAggregator: A stateful resource that aggregates statistics from one or more iterators.
class StatsOptions: Represents options for collecting dataset stats using StatsAggregator.
class TFRecordWriter: Writes a dataset to a TFRecord file.
class ThreadingOptions: Represents options for dataset threading.
Counter(...): Creates a Dataset that counts from start in steps of size step.
assert_cardinality(...): Asserts the cardinality of the input dataset.
bucket_by_sequence_length(...): A transformation that buckets elements in a Dataset by length.
bytes_produced_stats(...): Records the number of bytes produced by each element of the input dataset.
cardinality(...): Returns the cardinality of dataset, if known.
choose_from_datasets(...): Creates a dataset that deterministically chooses elements from datasets.
copy_to_device(...): A transformation that copies dataset elements to the given target_device.
dense_to_ragged_batch(...): A transformation that batches ragged elements into tf.RaggedTensors.
dense_to_sparse_batch(...): A transformation that batches ragged elements into tf.sparse.SparseTensors.
enumerate_dataset(...): A transformation that enumerates the elements of a dataset. (deprecated)
from_variant(...): Constructs a dataset from the given variant and structure.
get_next_as_optional(...): Returns a tf.experimental.Optional with the next element of the iterator. (deprecated)
get_single_element(...): Returns the single element in dataset as a nested structure of tensors.
get_structure(...): Returns the type signature for elements of the input dataset / iterator.
group_by_reducer(...): A transformation that groups elements and performs a reduction.
group_by_window(...): A transformation that groups windows of elements by key and reduces them.
ignore_errors(...): Creates a Dataset from another Dataset and silently ignores any errors.
latency_stats(...): Records the latency of producing each element of the input dataset.
load(...): Loads a previously saved dataset.
make_batched_features_dataset(...): Returns a Dataset of feature dictionaries from Example protos.
make_csv_dataset(...): Reads CSV files into a dataset.
make_saveable_from_iterator(...): Returns a SaveableObject for saving/restoring iterator state using Saver. (deprecated)
map_and_batch(...): Fused implementation of map and batch. (deprecated)
parallel_interleave(...): A parallel version of the Dataset.interleave() transformation. (deprecated)
parse_example_dataset(...): A transformation that parses Example protos into a dict of tensors.
prefetch_to_device(...): A transformation that prefetches dataset values to the given device.
rejection_resample(...): A transformation that resamples a dataset to achieve a target distribution.
sample_from_datasets(...): Samples elements at random from the datasets in datasets.
save(...): Saves the content of the given dataset.
scan(...): A transformation that scans a function across an input dataset.
shuffle_and_repeat(...): Shuffles and repeats a Dataset, reshuffling with each repetition. (deprecated)
snapshot(...): API to persist the output of the input dataset.
take_while(...): A transformation that stops dataset iteration based on a predicate.
to_variant(...): Returns a variant representing the given dataset.
unbatch(...): Splits elements of a dataset into multiple elements on the batch dimension. (deprecated)
unique(...): Creates a Dataset from another Dataset, discarding duplicates.


Asserts the cardinality of the input dataset.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.assert_cardinality


Represents the type of auto-sharding we enable.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.AutoShardPolicy
Please see the DistributeOptions.auto_shard_policy documentation for more information on each type of autosharding.


A transformation that buckets elements in a Dataset by length.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.bucket_by_sequence_length
Elements of the Dataset are grouped together by length and then are padded and batched.
This is useful for sequence tasks in which the elements have variable length. Grouping together elements that have similar lengths reduces the total fraction of padding in a batch which increases training step efficiency.


Records the number of bytes produced by each element of the input dataset.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.bytes_produced_stats
To consume the statistics, associate a StatsAggregator with the output dataset.


Returns the cardinality of dataset, if known.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.cardinality
The operation returns the cardinality of dataset. The operation may return tf.data.experimental.INFINITE_CARDINALITY if dataset contains an infinite number of elements or tf.data.experimental.UNKNOWN_CARDINALITY if the analysis fails to determine the number of elements in dataset (e.g. when the dataset source is a file).


Checkpoints input pipeline state every N steps or seconds.
Inherits From: SessionRunHook
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.CheckpointInputPipelineHook
This hook saves the state of the iterators in the Graph so that when training is resumed the input pipeline continues from where it left off. This could potentially avoid overfitting in certain pipelines where the number of training steps per eval are small compared to the dataset size or if the training pipeline is pre-empted.
Differences from CheckpointSaverHook:
Example of checkpointing the training pipeline:
This hook should be used if the input pipeline state needs to be saved separate from the model checkpoint. Doing so may be useful for a few reasons:
For saving the input pipeline checkpoint alongside the model weights use tf.data.experimental.make_saveable_from_iterator directly to create a SaveableObject and add to the SAVEABLE_OBJECTS collection. Note, however, that you will need to be careful not to restore the training iterator during eval. You can do that by not adding the iterator to the SAVEABLE_OBJECTS collector when building the eval graph.
View source
Called when new TensorFlow session is created.
This is called to signal the hooks that a new session has been created. This has two essential differences with the situation in which begin is called:
View source
Called after each call to run().
The run_values argument contains results of requested ops/tensors by before_run().
The run_context argument is the same one send to before_run call. run_context.request_stop() can be called to stop the iteration.
If session.run() raises any exceptions then after_run() is not called.
View source
Called before each call to run().
You can return from this call a SessionRunArgs object indicating ops or tensors to add to the upcoming run() call. These ops/tensors will be run together with the ops/tensors originally passed to the original run() call. The run args you return can also contain feeds to be added to the run() call.
The run_context argument is a SessionRunContext that provides information about the upcoming run() call: the originally requested op/tensors, the TensorFlow Session.
At this point graph is finalized and you can not add ops.
View source
Called once before using the session.
When called, the default graph is the one that will be launched in the session. The hook can modify the graph by adding new operations to it. After the begin() call the graph will be finalized and the other callbacks can not modify the graph anymore. Second call of begin() on the same graph, should not change the graph.
View source
Called at the end of session.
The session argument can be used in case the hook wants to run final ops, such as saving a last checkpoint.
If session.run() raises exception other than OutOfRangeError or StopIteration then end() is not called. Note the difference between end() and after_run() behavior when session.run() raises OutOfRangeError or StopIteration. In that case end() is called but after_run() is not called.


Creates a dataset that deterministically chooses elements from datasets.
For example, given the following datasets:
The elements of result will be:


A transformation that copies dataset elements to the given target_device.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.copy_to_device


Creates a Dataset that counts from start in steps of size step.