Module: tf.debugging Public API for tf.debugging namespace. Modules experimental module: Public API for tf.debugging.experimental namespace. Functions Assert(...): Asserts that the given condition is true. assert_all_finite(...): Assert that the tensor does not contain any NaN's or Inf's. assert_equal(...): Assert the condition x == y holds element-wise. assert_greater(...): Assert the condition x > y holds element-wise. assert_greater_equal(...): Assert the condition x >= y holds element-wise. assert_integer(...): Assert that x is of integer dtype. assert_less(...): Assert the condition x < y holds element-wise. assert_less_equal(...): Assert the condition x <= y holds element-wise. assert_near(...): Assert the condition x and y are close element-wise. assert_negative(...): Assert the condition x < 0 holds element-wise. assert_non_negative(...): Assert the condition x >= 0 holds element-wise. assert_non_positive(...): Assert the condition x <= 0 holds element-wise. assert_none_equal(...): Assert the condition x != y holds for all elements. assert_positive(...): Assert the condition x > 0 holds element-wise. assert_proper_iterable(...): Static assert that values is a "proper" iterable. assert_rank(...): Assert that x has rank equal to rank. assert_rank_at_least(...): Assert that x has rank of at least rank. assert_rank_in(...): Assert that x has a rank in ranks. assert_same_float_dtype(...): Validate and return float type based on tensors and dtype. assert_scalar(...): Asserts that the given tensor is a scalar. assert_shapes(...): Assert tensor shapes and dimension size relationships between tensors. assert_type(...): Asserts that the given Tensor is of the specified type. check_numerics(...): Checks a tensor for NaN and Inf values. disable_check_numerics(...): Disable the eager/graph unified numerics checking mechanism. enable_check_numerics(...): Enable tensor numerics checking in an eager/graph unified fashion. get_log_device_placement(...): Get if device placements are logged. is_numeric_tensor(...): Returns True if the elements of tensor are numbers. set_log_device_placement(...): Set if device placements should be logged.


tf.debugging.Assert     View source on GitHub    Asserts that the given condition is true.  View aliases  Main aliases
tf.Assert Compat aliases for migration See Migration guide for more details. tf.compat.v1.Assert, tf.compat.v1.debugging.Assert
tf.debugging.Assert(
    condition, data, summarize=None, name=None
)
 If condition evaluates to false, print the list of tensors in data. summarize determines how many entries of the tensors to print.

 Args
  condition   The condition to evaluate.
  data   The tensors to print out when condition is false.
  summarize   Print this many entries of each tensor.
  name   A name for this operation (optional).

 Returns
  assert_op   An Operation that, when executed, raises a tf.errors.InvalidArgumentError if condition is not true.

 Raises

Note: The output of this function should be used. If it is not, a warning will be logged or an error may be raised. To mark the output as used, call its .mark_used() method.
 Tf1 Compatibility When in TF V1 mode (that is, outside tf.function) Assert needs a control dependency on the output to ensure the assertion executes: # Ensure maximum element of x is smaller or equal to 1
assert_op = tf.Assert(tf.less_equal(tf.reduce_max(x), 1.), [x])
with tf.control_dependencies([assert_op]):
  ... code using x ...
 Eager Compatibility returns None


tf.debugging.assert_all_finite     View source on GitHub    Assert that the tensor does not contain any NaN's or Inf's.
tf.debugging.assert_all_finite(
    x, message, name=None
)

 Args
  x   Tensor to check.
  message   Message to log on failure.
  name   A name for this operation (optional).

 Returns   Same tensor as x.


tf.debugging.assert_equal     View source on GitHub    Assert the condition x == y holds element-wise.  View aliases  Main aliases
tf.assert_equal
tf.debugging.assert_equal(
    x, y, message=None, summarize=None, name=None
)
 This Op checks that x[i] == y[i] holds for every pair of (possibly broadcast) elements of x and y. If both x and y are empty, this is trivially satisfied. If x and y are not equal, message, as well as the first summarize entries of x and y are printed, and InvalidArgumentError is raised.

 Args
  x   Numeric Tensor.
  y   Numeric Tensor, same dtype as and broadcastable to x.
  message   A string to prefix to the default message.
  summarize   Print this many entries of each tensor.
  name   A name for this operation (optional). Defaults to "assert_equal".

 Returns   Op that raises InvalidArgumentError if x == y is False. This can be used with tf.control_dependencies inside of tf.functions to block followup computation until the check has executed.

 Raises
  InvalidArgumentError   if the check can be performed immediately and x == y is False. The check can be performed immediately during eager execution or if x and y are statically known.    Eager Compatibility returns None


tf.debugging.assert_greater     View source on GitHub    Assert the condition x > y holds element-wise.  View aliases  Main aliases
tf.assert_greater
tf.debugging.assert_greater(
    x, y, message=None, summarize=None, name=None
)
 This Op checks that x[i] > y[i] holds for every pair of (possibly broadcast) elements of x and y. If both x and y are empty, this is trivially satisfied. If x is not greater than y element-wise, message, as well as the first summarize entries of x and y are printed, and InvalidArgumentError is raised.

 Args
  x   Numeric Tensor.
  y   Numeric Tensor, same dtype as and broadcastable to x.
  message   A string to prefix to the default message.
  summarize   Print this many entries of each tensor.
  name   A name for this operation (optional). Defaults to "assert_greater".

 Returns   Op that raises InvalidArgumentError if x > y is False. This can be used with tf.control_dependencies inside of tf.functions to block followup computation until the check has executed.

 Raises
  InvalidArgumentError   if the check can be performed immediately and x > y is False. The check can be performed immediately during eager execution or if x and y are statically known.    Eager Compatibility returns None


tf.debugging.assert_greater_equal     View source on GitHub    Assert the condition x >= y holds element-wise.
tf.debugging.assert_greater_equal(
    x, y, message=None, summarize=None, name=None
)
 This Op checks that x[i] >= y[i] holds for every pair of (possibly broadcast) elements of x and y. If both x and y are empty, this is trivially satisfied. If x is not greater or equal to y element-wise, message, as well as the first summarize entries of x and y are printed, and InvalidArgumentError is raised.

 Args
  x   Numeric Tensor.
  y   Numeric Tensor, same dtype as and broadcastable to x.
  message   A string to prefix to the default message.
  summarize   Print this many entries of each tensor.
  name   A name for this operation (optional). Defaults to "assert_greater_equal".

 Returns   Op that raises InvalidArgumentError if x >= y is False. This can be used with tf.control_dependencies inside of tf.functions to block followup computation until the check has executed.

 Raises
  InvalidArgumentError   if the check can be performed immediately and x >= y is False. The check can be performed immediately during eager execution or if x and y are statically known.    Eager Compatibility returns None


tf.debugging.assert_integer     View source on GitHub    Assert that x is of integer dtype.
tf.debugging.assert_integer(
    x, message=None, name=None
)
 If x has a non-integer type, message, as well as the dtype of x are printed, and InvalidArgumentError is raised. This can always be checked statically, so this method returns nothing.

 Args
  x   A Tensor.
  message   A string to prefix to the default message.
  name   A name for this operation (optional). Defaults to "assert_integer".

 Raises
  TypeError   If x.dtype is not a non-quantized integer type.


tf.debugging.assert_less     View source on GitHub    Assert the condition x < y holds element-wise.  View aliases  Main aliases
tf.assert_less
tf.debugging.assert_less(
    x, y, message=None, summarize=None, name=None
)
 This Op checks that x[i] < y[i] holds for every pair of (possibly broadcast) elements of x and y. If both x and y are empty, this is trivially satisfied. If x is not less than y element-wise, message, as well as the first summarize entries of x and y are printed, and InvalidArgumentError is raised.

 Args
  x   Numeric Tensor.
  y   Numeric Tensor, same dtype as and broadcastable to x.
  message   A string to prefix to the default message.
  summarize   Print this many entries of each tensor.
  name   A name for this operation (optional). Defaults to "assert_less".

 Returns   Op that raises InvalidArgumentError if x < y is False. This can be used with tf.control_dependencies inside of tf.functions to block followup computation until the check has executed.

 Raises
  InvalidArgumentError   if the check can be performed immediately and x < y is False. The check can be performed immediately during eager execution or if x and y are statically known.    Eager Compatibility returns None


tf.debugging.assert_less_equal     View source on GitHub    Assert the condition x <= y holds element-wise.
tf.debugging.assert_less_equal(
    x, y, message=None, summarize=None, name=None
)
 This Op checks that x[i] <= y[i] holds for every pair of (possibly broadcast) elements of x and y. If both x and y are empty, this is trivially satisfied. If x is not less or equal than y element-wise, message, as well as the first summarize entries of x and y are printed, and InvalidArgumentError is raised.

 Args
  x   Numeric Tensor.
  y   Numeric Tensor, same dtype as and broadcastable to x.
  message   A string to prefix to the default message.
  summarize   Print this many entries of each tensor.
  name   A name for this operation (optional). Defaults to "assert_less_equal".

 Returns   Op that raises InvalidArgumentError if x <= y is False. This can be used with tf.control_dependencies inside of tf.functions to block followup computation until the check has executed.

 Raises
  InvalidArgumentError   if the check can be performed immediately and x <= y is False. The check can be performed immediately during eager execution or if x and y are statically known.    Eager Compatibility returns None


tf.debugging.assert_near     View source on GitHub    Assert the condition x and y are close element-wise.
tf.debugging.assert_near(
    x, y, rtol=None, atol=None, message=None, summarize=None, name=None
)
 This Op checks that x[i] - y[i] < atol + rtol * tf.abs(y[i]) holds for every pair of (possibly broadcast) elements of x and y. If both x and y are empty, this is trivially satisfied. If any elements of x and y are not close, message, as well as the first summarize entries of x and y are printed, and InvalidArgumentError is raised. The default atol and rtol is 10 * eps, where eps is the smallest representable positive number such that 1 + eps != 1. This is about 1.2e-6 in 32bit, 2.22e-15 in 64bit, and 0.00977 in 16bit. See numpy.finfo.

 Args
  x   Float or complex Tensor.
  y   Float or complex Tensor, same dtype as and broadcastable to x.
  rtol   Tensor. Same dtype as, and broadcastable to, x. The relative tolerance. Default is 10 * eps.
  atol   Tensor. Same dtype as, and broadcastable to, x. The absolute tolerance. Default is 10 * eps.
  message   A string to prefix to the default message.
  summarize   Print this many entries of each tensor.
  name   A name for this operation (optional). Defaults to "assert_near".

 Returns   Op that raises InvalidArgumentError if x and y are not close enough. This can be used with tf.control_dependencies inside of tf.functions to block followup computation until the check has executed.

 Raises
  InvalidArgumentError   if the check can be performed immediately and x != y is False for any pair of elements in x and y. The check can be performed immediately during eager execution or if x and y are statically known.    Eager Compatibility returns None Numpy Compatibility Similar to numpy.testing.assert_allclose, except tolerance depends on data type. This is due to the fact that TensorFlow is often used with 32bit, 64bit, and even 16bit data.


tf.debugging.assert_negative     View source on GitHub    Assert the condition x < 0 holds element-wise.
tf.debugging.assert_negative(
    x, message=None, summarize=None, name=None
)
 This Op checks that x[i] < 0 holds for every element of x. If x is empty, this is trivially satisfied. If x is not negative everywhere, message, as well as the first summarize entries of x are printed, and InvalidArgumentError is raised.

 Args
  x   Numeric Tensor.
  message   A string to prefix to the default message.
  summarize   Print this many entries of each tensor.
  name   A name for this operation (optional). Defaults to "assert_negative".

 Returns   Op raising InvalidArgumentError unless x is all negative. This can be used with tf.control_dependencies inside of tf.functions to block followup computation until the check has executed.

 Raises
  InvalidArgumentError   if the check can be performed immediately and x[i] < 0 is False. The check can be performed immediately during eager execution or if x is statically known.    Eager Compatibility returns None


tf.debugging.assert_none_equal     View source on GitHub    Assert the condition x != y holds for all elements.
tf.debugging.assert_none_equal(
    x, y, summarize=None, message=None, name=None
)
 This Op checks that x[i] != y[i] holds for every pair of (possibly broadcast) elements of x and y. If both x and y are empty, this is trivially satisfied. If any elements of x and y are equal, message, as well as the first summarize entries of x and y are printed, and InvalidArgumentError is raised.

 Args
  x   Numeric Tensor.
  y   Numeric Tensor, same dtype as and broadcastable to x.
  summarize   Print this many entries of each tensor.
  message   A string to prefix to the default message.
  name   A name for this operation (optional). Defaults to "assert_none_equal".

 Returns   Op that raises InvalidArgumentError if x != y is ever False. This can be used with tf.control_dependencies inside of tf.functions to block followup computation until the check has executed.

 Raises
  InvalidArgumentError   if the check can be performed immediately and x != y is False for any pair of elements in x and y. The check can be performed immediately during eager execution or if x and y are statically known.    Eager Compatibility returns None


tf.debugging.assert_non_negative     View source on GitHub    Assert the condition x >= 0 holds element-wise.
tf.debugging.assert_non_negative(
    x, message=None, summarize=None, name=None
)
 This Op checks that x[i] >= 0 holds for every element of x. If x is empty, this is trivially satisfied. If x is not >= 0 everywhere, message, as well as the first summarize entries of x are printed, and InvalidArgumentError is raised.

 Args
  x   Numeric Tensor.
  message   A string to prefix to the default message.
  summarize   Print this many entries of each tensor.
  name   A name for this operation (optional). Defaults to "assert_non_negative".

 Returns   Op raising InvalidArgumentError unless x is all non-negative. This can be used with tf.control_dependencies inside of tf.functions to block followup computation until the check has executed.

 Raises
  InvalidArgumentError   if the check can be performed immediately and x[i] >= 0 is False. The check can be performed immediately during eager execution or if x is statically known.    Eager Compatibility returns None


tf.debugging.assert_non_positive     View source on GitHub    Assert the condition x <= 0 holds element-wise.
tf.debugging.assert_non_positive(
    x, message=None, summarize=None, name=None
)
 This Op checks that x[i] <= 0 holds for every element of x. If x is empty, this is trivially satisfied. If x is not <= 0 everywhere, message, as well as the first summarize entries of x are printed, and InvalidArgumentError is raised.

 Args
  x   Numeric Tensor.
  message   A string to prefix to the default message.
  summarize   Print this many entries of each tensor.
  name   A name for this operation (optional). Defaults to "assert_non_positive".

 Returns   Op raising InvalidArgumentError unless x is all non-positive. This can be used with tf.control_dependencies inside of tf.functions to block followup computation until the check has executed.

 Raises
  InvalidArgumentError   if the check can be performed immediately and x[i] <= 0 is False. The check can be performed immediately during eager execution or if x is statically known.    Eager Compatibility returns None


tf.debugging.assert_positive     View source on GitHub    Assert the condition x > 0 holds element-wise.
tf.debugging.assert_positive(
    x, message=None, summarize=None, name=None
)
 This Op checks that x[i] > 0 holds for every element of x. If x is empty, this is trivially satisfied. If x is not positive everywhere, message, as well as the first summarize entries of x are printed, and InvalidArgumentError is raised.

 Args
  x   Numeric Tensor.
  message   A string to prefix to the default message.
  summarize   Print this many entries of each tensor.
  name   A name for this operation (optional). Defaults to "assert_positive".

 Returns   Op raising InvalidArgumentError unless x is all positive. This can be used with tf.control_dependencies inside of tf.functions to block followup computation until the check has executed.

 Raises
  InvalidArgumentError   if the check can be performed immediately and x[i] > 0 is False. The check can be performed immediately during eager execution or if x is statically known.    Eager Compatibility returns None


tf.debugging.assert_proper_iterable     View source on GitHub    Static assert that values is a "proper" iterable.  View aliases  Compat aliases for migration
See Migration guide for more details. tf.compat.v1.assert_proper_iterable, tf.compat.v1.debugging.assert_proper_iterable
tf.debugging.assert_proper_iterable(
    values
)
 Ops that expect iterables of Tensor can call this to validate input. Useful since Tensor, ndarray, byte/text type are all iterables themselves.

 Args
  values   Object to be checked.

 Raises
  TypeError   If values is not iterable or is one of Tensor, SparseTensor, np.array, tf.compat.bytes_or_text_types.


tf.debugging.assert_rank     View source on GitHub    Assert that x has rank equal to rank.  View aliases  Main aliases
tf.assert_rank
tf.debugging.assert_rank(
    x, rank, message=None, name=None
)
 This Op checks that the rank of x is equal to rank. If x has a different rank, message, as well as the shape of x are printed, and InvalidArgumentError is raised.

 Args
  x   Tensor.
  rank   Scalar integer Tensor.
  message   A string to prefix to the default message.
  name   A name for this operation (optional). Defaults to "assert_rank".

 Returns   Op raising InvalidArgumentError unless x has specified rank. If static checks determine x has correct rank, a no_op is returned. This can be used with tf.control_dependencies inside of tf.functions to block followup computation until the check has executed.

 Raises
  InvalidArgumentError   if the check can be performed immediately and x does not have rank rank. The check can be performed immediately during eager execution or if the shape of x is statically known.    Eager Compatibility returns None


tf.debugging.assert_rank_at_least     View source on GitHub    Assert that x has rank of at least rank.
tf.debugging.assert_rank_at_least(
    x, rank, message=None, name=None
)
 This Op checks that the rank of x is greater or equal to rank. If x has a rank lower than rank, message, as well as the shape of x are printed, and InvalidArgumentError is raised.

 Args
  x   Tensor.
  rank   Scalar integer Tensor.
  message   A string to prefix to the default message.
  name   A name for this operation (optional). Defaults to "assert_rank_at_least".

 Returns   Op raising InvalidArgumentError unless x has specified rank or higher. If static checks determine x has correct rank, a no_op is returned. This can be used with tf.control_dependencies inside of tf.functions to block followup computation until the check has executed.

 Raises
  InvalidArgumentError   x does not have rank at least rank, but the rank cannot be statically determined.
  ValueError   If static checks determine x has mismatched rank.    Eager Compatibility returns None


tf.debugging.assert_rank_in     View source on GitHub    Assert that x has a rank in ranks.
tf.debugging.assert_rank_in(
    x, ranks, message=None, name=None
)
 This Op checks that the rank of x is in ranks. If x has a different rank, message, as well as the shape of x are printed, and InvalidArgumentError is raised.

 Args
  x   Tensor.
  ranks   Iterable of scalar Tensor objects.
  message   A string to prefix to the default message.
  name   A name for this operation (optional). Defaults to "assert_rank_in".

 Returns   Op raising InvalidArgumentError unless rank of x is in ranks. If static checks determine x has matching rank, a no_op is returned. This can be used with tf.control_dependencies inside of tf.functions to block followup computation until the check has executed.

 Raises
  InvalidArgumentError   x does not have rank in ranks, but the rank cannot be statically determined.
  ValueError   If static checks determine x has mismatched rank.    Eager Compatibility returns None


tf.debugging.assert_same_float_dtype     View source on GitHub    Validate and return float type based on tensors and dtype.  View aliases  Compat aliases for migration
See Migration guide for more details. tf.compat.v1.assert_same_float_dtype, tf.compat.v1.debugging.assert_same_float_dtype
tf.debugging.assert_same_float_dtype(
    tensors=None, dtype=None
)
 For ops such as matrix multiplication, inputs and weights must be of the same float type. This function validates that all tensors are the same type, validates that type is dtype (if supplied), and returns the type. Type must be a floating point type. If neither tensors nor dtype is supplied, the function will return dtypes.float32.

 Args
  tensors   Tensors of input values. Can include None elements, which will be ignored.
  dtype   Expected type.

 Returns   Validated type.

 Raises
  ValueError   if neither tensors nor dtype is supplied, or result is not float, or the common type of the inputs is not a floating point type.


tf.debugging.assert_scalar     View source on GitHub    Asserts that the given tensor is a scalar.
tf.debugging.assert_scalar(
    tensor, message=None, name=None
)
 This function raises ValueError unless it can be certain that the given tensor is a scalar. ValueError is also raised if the shape of tensor is unknown. This is always checked statically, so this method returns nothing.

 Args
  tensor   A Tensor.
  message   A string to prefix to the default message.
  name   A name for this operation. Defaults to "assert_scalar"

 Raises
  ValueError   If the tensor is not scalar (rank 0), or if its shape is unknown.


tf.debugging.assert_shapes     View source on GitHub    Assert tensor shapes and dimension size relationships between tensors.
tf.debugging.assert_shapes(
    shapes, data=None, summarize=None, message=None, name=None
)
 This Op checks that a collection of tensors shape relationships satisfies given constraints. Example:
n = 10
q = 3
d = 7
x = tf.zeros([n,q])
y = tf.ones([n,d])
param = tf.Variable([1.0, 2.0, 3.0])
scalar = 1.0
tf.debugging.assert_shapes([
 (x, ('N', 'Q')),
 (y, ('N', 'D')),
 (param, ('Q',)),
 (scalar, ()),
])

tf.debugging.assert_shapes([
  (x, ('N', 'D')),
  (y, ('N', 'D'))
])
Traceback (most recent call last):

ValueError: ...
 If x, y, param or scalar does not have a shape that satisfies all specified constraints, message, as well as the first summarize entries of the first encountered violating tensor are printed, and InvalidArgumentError is raised. Size entries in the specified shapes are checked against other entries by their hash, except:  a size entry is interpreted as an explicit size if it can be parsed as an integer primitive. a size entry is interpreted as any size if it is None or '.'.  If the first entry of a shape is ... (type Ellipsis) or '*' that indicates a variable number of outer dimensions of unspecified size, i.e. the constraint applies to the inner-most dimensions only. Scalar tensors and specified shapes of length zero (excluding the 'inner-most' prefix) are both treated as having a single dimension of size one.

 Args
  shapes   dictionary with (Tensor to shape) items, or a list of (Tensor, shape) tuples. A shape must be an iterable.
  data   The tensors to print out if the condition is False. Defaults to error message and first few entries of the violating tensor.
  summarize   Print this many entries of the tensor.
  message   A string to prefix to the default message.
  name   A name for this operation (optional). Defaults to "assert_shapes".

 Raises
  ValueError   If static checks determine any shape constraint is violated.


tf.debugging.assert_type     View source on GitHub    Asserts that the given Tensor is of the specified type.
tf.debugging.assert_type(
    tensor, tf_type, message=None, name=None
)
 This can always be checked statically, so this method returns nothing.

 Args
  tensor   A Tensor or SparseTensor.
  tf_type   A tensorflow type (dtypes.float32, tf.int64, dtypes.bool, etc).
  message   A string to prefix to the default message.
  name   A name for this operation. Defaults to "assert_type"

 Raises
  TypeError   If the tensor's data type doesn't match tf_type.


tf.debugging.check_numerics Checks a tensor for NaN and Inf values.  View aliases  Compat aliases for migration
See Migration guide for more details. tf.compat.v1.check_numerics, tf.compat.v1.debugging.check_numerics
tf.debugging.check_numerics(
    tensor, message, name=None
)
 When run, reports an InvalidArgument error if tensor has any values that are not a number (NaN) or infinity (Inf). Otherwise, passes tensor as-is.

 Args
  tensor   A Tensor. Must be one of the following types: bfloat16, half, float32, float64.
  message   A string. Prefix of the error message.
  name   A name for the operation (optional).

 Returns   A Tensor. Has the same type as tensor.


tf.debugging.disable_check_numerics Disable the eager/graph unified numerics checking mechanism.  View aliases  Compat aliases for migration
See Migration guide for more details. tf.compat.v1.debugging.disable_check_numerics
tf.debugging.disable_check_numerics()
 This method can be used after a call to tf.debugging.enable_check_numerics() to disable the numerics-checking mechanism that catches infinity and NaN values output by ops executed eagerly or in tf.function-compiled graphs. This method is idempotent. Calling it multiple times has the same effect as calling it once. This method takes effect only on the thread in which it is called.


tf.debugging.enable_check_numerics Enable tensor numerics checking in an eager/graph unified fashion.  View aliases  Compat aliases for migration
See Migration guide for more details. tf.compat.v1.debugging.enable_check_numerics
tf.debugging.enable_check_numerics(
    stack_height_limit=30, path_length_limit=50
)
 The numerics checking mechanism will cause any TensorFlow eager execution or graph execution to error out as soon as an op's output tensor contains infinity or NaN. This method is idempotent. Calling it multiple times has the same effect as calling it once. This method takes effect only on the thread in which it is called. When a op's float-type output tensor contains any Infinity or NaN, an tf.errors.InvalidArgumentError will be thrown, with an error message that reveals the following information:  The type of the op that generated the tensor with bad numerics. Data type (dtype) of the tensor. Shape of the tensor (to the extent known at the time of eager execution or graph construction). Name of the containing graph (if available). (Graph mode only): The stack trace of the intra-graph op's creation, with a stack-height limit and a path-length limit for visual clarity. The stack frames that belong to the user's code (as opposed to tensorflow's internal code) are highlighted with a text arrow ("->"). (Eager mode only): How many of the offending tensor's elements are Infinity and NaN, respectively.  Once enabled, the check-numerics mechanism can be disabled by using tf.debugging.disable_check_numerics(). Example usage:
Catching infinity during the execution of a tf.function graph: import tensorflow as tf

tf.debugging.enable_check_numerics()

@tf.function
def square_log_x_plus_1(x):
  v = tf.math.log(x + 1)
  return tf.math.square(v)

x = -1.0

# When the following line runs, a function graph will be compiled
# from the Python function `square_log_x_plus_1()`. Due to the
# `enable_check_numerics()` call above, the graph will contain
# numerics checking ops that will run during the function graph's
# execution. The function call generates an -infinity when the Log
# (logarithm) op operates on the output tensor of the Add op.
# The program errors out at this line, printing an error message.
y = square_log_x_plus_1(x)
z = -y

Catching NaN during eager execution: import numpy as np
import tensorflow as tf

tf.debugging.enable_check_numerics()

x = np.array([[0.0, -1.0], [4.0, 3.0]])

# The following line executes the Sqrt op eagerly. Due to the negative
# element in the input array, a NaN is generated. Due to the
# `enable_check_numerics()` call above, the program errors immediately
# at this line, printing an error message.
y = tf.math.sqrt(x)
z = tf.matmul(y, y)

Note: If your code is running on TPUs, be sure to call tf.config.set_soft_device_placement(True) before calling tf.debugging.enable_check_numerics() as this API uses automatic outside compilation on TPUs. For example:

tf.config.set_soft_device_placement(True)
tf.debugging.enable_check_numerics()

resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='')
strategy = tf.distribute.TPUStrategy(resolver)
with strategy.scope():
  # ...

 Args
  stack_height_limit   Limit to the height of the printed stack trace. Applicable only to ops in tf.functions (graphs).
  path_length_limit   Limit to the file path included in the printed stack trace. Applicable only to ops in tf.functions (graphs).


Module: tf.debugging.experimental Public API for tf.debugging.experimental namespace. Functions disable_dump_debug_info(...): Disable the currently-enabled debugging dumping. enable_dump_debug_info(...): Enable dumping debugging information from a TensorFlow program.


tf.debugging.experimental.disable_dump_debug_info Disable the currently-enabled debugging dumping.  View aliases  Compat aliases for migration
See Migration guide for more details. tf.compat.v1.debugging.experimental.disable_dump_debug_info
tf.debugging.experimental.disable_dump_debug_info()
 If the enable_dump_debug_info() method under the same Python namespace has been invoked before, calling this method disables it. If no call to enable_dump_debug_info() has been made, calling this method is a no-op. Calling this method more than once is idempotent.


tf.debugging.experimental.enable_dump_debug_info   Enable dumping debugging information from a TensorFlow program.  View aliases  Compat aliases for migration
See Migration guide for more details. tf.compat.v1.debugging.experimental.enable_dump_debug_info
tf.debugging.experimental.enable_dump_debug_info(
    dump_root, tensor_debug_mode=DEFAULT_TENSOR_DEBUG_MODE,
    circular_buffer_size=1000, op_regex=None, tensor_dtypes=None
)
 The debugging information is dumped to a directory on the file system specified as dump_root. The dumped debugging information can be ingested by debugger UIs. The files in the dump directory contain the following information:  TensorFlow Function construction (e.g., compilation of Python functions decorated with @tf.function), the op types, names (if available), context, the input and output tensors, and the associated stack traces. Execution of TensorFlow operations (ops) and Functions and their stack traces, op types, names (if available) and contexts. In addition, depending on the value of the tensor_debug_mode argument (see Args section below), the value(s) of the output tensors or more concise summaries of the tensor values will be dumped. A snapshot of Python source files involved in the execution of the TensorFlow program.  Once enabled, the dumping can be disabled with the corresponding disable_dump_debug_info() method under the same Python namespace. Calling this method more than once with the same dump_root is idempotent. Calling this method more than once with different tensor_debug_modes leads to a ValueError. Calling this method more than once with different circular_buffer_sizes leads to a ValueError. Calling this method with a different dump_root abolishes the previously-enabled dump_root. Usage example: tf.debugging.experimental.enable_dump_debug_info('/tmp/my-tfdbg-dumps')

# Code to build, train and run your TensorFlow model...

Note: If your code is running on TPUs, be sure to call tf.config.set_soft_device_placement(True) before calling tf.debugging.experimental.enable_dump_debug_info() as this API uses automatic outside compilation on TPUs. For example:

tf.config.set_soft_device_placement(True)
tf.debugging.experimental.enable_dump_debug_info(
    logdir, tensor_debug_mode="FULL_HEALTH")

resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='')
strategy = tf.distribute.TPUStrategy(resolver)
with strategy.scope():
  # ...

 Args
  dump_root   The directory path where the dumping information will be written.
  tensor_debug_mode   Debug mode for tensor values, as a string. The currently supported options are:  "NO_TENSOR": (Default) Only traces the output tensors of all executed ops (including those executed eagerly at the Python level or as a part of a TensorFlow graph) and functions, while not extracting any information from the values of the tensors. "CURT_HEALTH": For each floating-dtype tensor (e.g., tensors of dtypes such as float32, float64 and bfloat16), extracts a binary bit indicating whether it contains any -infinity, +infinity or NaN. "CONCISE_HEALTH": For each floating-dtype tensor, extract total element count, and counts of -infinity, +infinity and NaN elements. "FULL_HEALTH": For each floating-dtype tensor, extracts the dtype, rank (number of dimensions), total element count, and counts of -infinity, +infinity and NaN elements. "SHAPE": For each tensor (regardless of dtype), extracts its dtype, rank, total element count and shape.

  circular_buffer_size   Size of the circular buffers for execution events. These circular buffers are designed to reduce the overhead of debugging dumping. They hold the most recent debug events concerning eager execution of ops and tf.functions and traces of tensor values computed inside tf.functions. They are written to the file system only when the proper flushing method is called (see description of return values below). Expected to be an integer. If <= 0, the circular-buffer behavior will be disabled, i.e., the execution debug events will be written to the file writers in the same way as non-execution events such as op creations and source-file snapshots.
  op_regex   Dump data from only the tensors from op types that matches to the regular expression (through Python's re.match()). "Op type" refers to the names of the TensorFlow operations (e.g., "MatMul", "LogSoftmax"), which may repeat in a TensorFlow function. It does not refer to the names of nodes (e.g., "dense/MatMul", "dense_1/MatMul_1") which are unique within a function. Example 1: Dump tensor data from only MatMul and Relu ops op_regex="^(MatMul|Relu)$". Example 2: Dump tensors from all ops except Relu: op_regex="(?!^Relu$)". This filter operates in a logical AND relation with tensor_dtypes.

  tensor_dtypes   Dump data from only the tensors of which the specified dtypes. This optional argument can be in any of the following format: a list or tuple of DType objects or strings that can be converted to DType objects via tf.as_dtype(). Examples:
tensor_dtype=[tf.float32, tf.float64],
tensor_dtype=["float32", "float64"],
tensor_dtypes=(tf.int32, tf.bool), tensor_dtypes=("int32", "bool") a callable that takes a single DType argument and returns a Python boolean indicating whether the dtype is to be included in the data dumping. Examples:
tensor_dtype=lambda dtype: dtype.is_integer. This filter operates in a logical AND relation with op_regex.

 Returns   A DebugEventsWriter instance used by the dumping callback. The caller may use its flushing methods, including FlushNonExecutionFiles() and FlushExecutionFiles().


tf.debugging.get_log_device_placement     View source on GitHub    Get if device placements are logged.  View aliases  Compat aliases for migration
See Migration guide for more details. tf.compat.v1.debugging.get_log_device_placement
tf.debugging.get_log_device_placement()

 Returns   If device placements are logged.


tf.debugging.is_numeric_tensor     View source on GitHub    Returns True if the elements of tensor are numbers.  View aliases  Compat aliases for migration
See Migration guide for more details. tf.compat.v1.debugging.is_numeric_tensor, tf.compat.v1.is_numeric_tensor
tf.debugging.is_numeric_tensor(
    tensor
)
 Specifically, returns True if the dtype of tensor is one of the following:  tf.float32 tf.float64 tf.int8 tf.int16 tf.int32 tf.int64 tf.uint8 tf.qint8 tf.qint32 tf.quint8 tf.complex64  Returns False if tensor is of a non-numeric type or if tensor is not a tf.Tensor object.


tf.debugging.set_log_device_placement     View source on GitHub    Set if device placements should be logged.  View aliases  Compat aliases for migration
See Migration guide for more details. tf.compat.v1.debugging.set_log_device_placement
tf.debugging.set_log_device_placement(
    enabled
)

 Args
  enabled   Whether to enabled device placement logging.


tf.device     View source on GitHub    Specifies the device for ops created/executed in this context.
tf.device(
    device_name
)
 This function specifies the device to be used for ops created/executed in a particular context. Nested contexts will inherit and also create/execute their ops on the specified device. If a specific device is not required, consider not using this function so that a device can be automatically assigned. In general the use of this function is optional. device_name can be fully specified, as in "/job:worker/task:1/device:cpu:0", or partially specified, containing only a subset of the "/"-separated fields. Any fields which are specified will override device annotations from outer scopes. For example: with tf.device('/job:foo'):
  # ops created here have devices with /job:foo
  with tf.device('/job:bar/task:0/device:gpu:2'):
    # ops created here have the fully specified device above
  with tf.device('/device:gpu:1'):
    # ops created here have the device '/job:foo/device:gpu:1'

 Args
  device_name   The device name to use in the context.

 Returns   A context manager that specifies the default device to use for newly created ops.

 Raises
  RuntimeError   If a function is passed in.


tf.DeviceSpec     View source on GitHub    Represents a (possibly partial) specification for a TensorFlow device.
tf.DeviceSpec(
    job=None, replica=None, task=None, device_type=None, device_index=None
)
 DeviceSpecs are used throughout TensorFlow to describe where state is stored and computations occur. Using DeviceSpec allows you to parse device spec strings to verify their validity, merge them or compose them programmatically. Example: # Place the operations on device "GPU:0" in the "ps" job.
device_spec = DeviceSpec(job="ps", device_type="GPU", device_index=0)
with tf.device(device_spec.to_string()):
  # Both my_var and squared_var will be placed on /job:ps/device:GPU:0.
  my_var = tf.Variable(..., name="my_variable")
  squared_var = tf.square(my_var)
 With eager execution disabled (by default in TensorFlow 1.x and by calling disable_eager_execution() in TensorFlow 2.x), the following syntax can be used: tf.compat.v1.disable_eager_execution()

# Same as previous
device_spec = DeviceSpec(job="ps", device_type="GPU", device_index=0)
# No need of .to_string() method.
with tf.device(device_spec):
  my_var = tf.Variable(..., name="my_variable")
  squared_var = tf.square(my_var)
 ```

If a `DeviceSpec` is partially specified, it will be merged with other
`DeviceSpec`s according to the scope in which it is defined. `DeviceSpec`
components defined in inner scopes take precedence over those defined in
outer scopes.

```python
gpu0_spec = DeviceSpec(job="ps", device_type="GPU", device_index=0)
with tf.device(DeviceSpec(job="train").to_string()):
  with tf.device(gpu0_spec.to_string()):
    # Nodes created here will be assigned to /job:ps/device:GPU:0.
  with tf.device(DeviceSpec(device_type="GPU", device_index=1).to_string()):
    # Nodes created here will be assigned to /job:train/device:GPU:1.
 A DeviceSpec consists of 5 components -- each of which is optionally specified:  Job: The job name. Replica: The replica index. Task: The task index. Device type: The device type string (e.g. "CPU" or "GPU"). Device index: The device index.

 Args
  job   string. Optional job name.
  replica   int. Optional replica index.
  task   int. Optional task index.
  device_type   Optional device type string (e.g. "CPU" or "GPU")
  device_index   int. Optional device index. If left unspecified, device represents 'any' device_index.

 Attributes
  device_index

  device_type

  job

  replica

  task
   Methods from_string View source
@classmethod
from_string(
    spec
)
 Construct a DeviceSpec from a string.

 Args
  spec   a string of the form /job:/replica:/task:/device:CPU: or /job:/replica:/task:/device:GPU: as cpu and gpu are mutually exclusive. All entries are optional.

 Returns   A DeviceSpec.
 make_merged_spec View source
make_merged_spec(
    dev
)
 Returns a new DeviceSpec which incorporates dev. When combining specs, dev will take precedence over the current spec. So for instance: first_spec = tf.DeviceSpec(job=0, device_type="CPU")
second_spec = tf.DeviceSpec(device_type="GPU")
combined_spec = first_spec.make_merged_spec(second_spec)
 is equivalent to: combined_spec = tf.DeviceSpec(job=0, device_type="GPU")

 Args
  dev   a DeviceSpec

 Returns   A new DeviceSpec which combines self and dev
 parse_from_string View source
parse_from_string(
    spec
)
 Parse a DeviceSpec name into its components. 2.x behavior change: In TensorFlow 1.x, this function mutates its own state and returns itself. In 2.x, DeviceSpecs are immutable, and this function will return a DeviceSpec which contains the spec. Recommended: ```
# my_spec and my_updated_spec are unrelated.
my_spec = tf.DeviceSpec.from_string("/CPU:0")
my_updated_spec = tf.DeviceSpec.from_string("/GPU:0")
with tf.device(my_updated_spec):
  ...
```
 Will work in 1.x and 2.x (though deprecated in 2.x): ```
my_spec = tf.DeviceSpec.from_string("/CPU:0")
my_updated_spec = my_spec.parse_from_string("/GPU:0")
with tf.device(my_updated_spec):
  ...
```
 Will NOT work in 2.x: ```
my_spec = tf.DeviceSpec.from_string("/CPU:0")
my_spec.parse_from_string("/GPU:0")  # <== Will not update my_spec
with tf.device(my_spec):
  ...
```
 In general, DeviceSpec.from_string should completely replace DeviceSpec.parse_from_string, and DeviceSpec.replace should completely replace setting attributes directly.

 Args
  spec   an optional string of the form /job:/replica:/task:/device:CPU: or /job:/replica:/task:/device:GPU: as cpu and gpu are mutually exclusive. All entries are optional.

 Returns   The DeviceSpec.

 Raises
  ValueError   if the spec was not valid.    replace View source
replace(
    **kwargs
)
 Convenience method for making a new DeviceSpec by overriding fields. For instance: my_spec = DeviceSpec=(job="my_job", device="CPU")
my_updated_spec = my_spec.replace(device="GPU")
my_other_spec = my_spec.replace(device=None)

 Args
  **kwargs   This method takes the same args as the DeviceSpec constructor

 Returns   A DeviceSpec with the fields specified in kwargs overridden.
 to_string View source
to_string()
 Return a string representation of this DeviceSpec.

 Returns   a string of the form /job:/replica:/task:/device::.

 __eq__ View source
__eq__(
    other
)
 Checks if the other DeviceSpec is same as the current instance, eg have same value for all the internal fields.

 Args
  other   Another DeviceSpec

 Returns   Return True if other is also a DeviceSpec instance and has same value as the current instance. Return False otherwise.


Module: tf.distribute Library for running a computation across multiple devices. The intent of this library is that you can write an algorithm in a stylized way and it will be usable with a variety of different tf.distribute.Strategy implementations. Each descendant will implement a different strategy for distributing the algorithm across multiple devices/machines. Furthermore, these changes can be hidden inside the specific layers and other library classes that need special treatment to run in a distributed setting, so that most users' model definition code can run unchanged. The tf.distribute.Strategy API works the same way with eager and graph execution. Guides  TensorFlow v2.x TensorFlow v1.x  Tutorials
Distributed Training Tutorials The tutorials cover how to use tf.distribute.Strategy to do distributed training with native Keras APIs, custom training loops, and Esitmator APIs. They also cover how to save/load model when using tf.distribute.Strategy.
  Glossary
Data parallelism is where we run multiple copies of the model on different slices of the input data. This is in contrast to model parallelism where we divide up a single copy of a model across multiple devices. Note: we only support data parallelism for now, but hope to add support for model parallelism in the future. A device is a CPU or accelerator (e.g. GPUs, TPUs) on some machine that TensorFlow can run operations on (see e.g. tf.device). You may have multiple devices on a single machine, or be connected to devices on multiple machines. Devices used to run computations are called worker devices. Devices used to store variables are parameter devices. For some strategies, such as tf.distribute.MirroredStrategy, the worker and parameter devices will be the same (see mirrored variables below). For others they will be different. For example, tf.distribute.experimental.CentralStorageStrategy puts the variables on a single device (which may be a worker device or may be the CPU), and tf.distribute.experimental.ParameterServerStrategy puts the variables on separate machines called parameter servers (see below). A replica is one copy of the model, running on one slice of the input data. Right now each replica is executed on its own worker device, but once we add support for model parallelism a replica may span multiple worker devices. A host is the CPU device on a machine with worker devices, typically used for running input pipelines. A worker is defined to be the physical machine(s) containing the physical devices (e.g. GPUs, TPUs) on which the replicated computation is executed. A worker may contain one or more replicas, but contains at least one replica. Typically one worker will correspond to one machine, but in the case of very large models with model parallelism, one worker may span multiple machines. We typically run one input pipeline per worker, feeding all the replicas on that worker.
Synchronous, or more commonly sync, training is where the updates from each replica are aggregated together before updating the model variables. This is in contrast to asynchronous, or async training, where each replica updates the model variables independently. You may also have replicas partitioned into groups which are in sync within each group but async between groups. Parameter servers: These are machines that hold a single copy of parameters/variables, used by some strategies (right now just tf.distribute.experimental.ParameterServerStrategy). All replicas that want to operate on a variable retrieve it at the beginning of a step and send an update to be applied at the end of the step. These can in priniciple support either sync or async training, but right now we only have support for async training with parameter servers. Compare to tf.distribute.experimental.CentralStorageStrategy, which puts all variables on a single device on the same machine (and does sync training), and tf.distribute.MirroredStrategy, which mirrors variables to multiple devices (see below).
Replica context vs. Cross-replica context vs Update context A replica context applies when you execute the computation function that was called with strategy.run. Conceptually, you're in replica context when executing the computation function that is being replicated. An update context is entered in a tf.distribute.StrategyExtended.update call. An cross-replica context is entered when you enter a strategy.scope. This is useful for calling tf.distribute.Strategy methods which operate across the replicas (like reduce_to()). By default you start in a replica context (the "default single replica context") and then some methods can switch you back and forth.

Distributed value: Distributed value is represented by the base class tf.distribute.DistributedValues. tf.distribute.DistributedValues is useful to represent values on multiple devices, and it contains a map from replica id to values. Two representative kinds of tf.distribute.DistributedValues are "PerReplica" and "Mirrored" values. "PerReplica" values exist on the worker devices, with a different value for each replica. They are produced by iterating through a distributed dataset returned by tf.distribute.Strategy.experimental_distribute_dataset and tf.distribute.Strategy.distribute_datasets_from_function. They are also the typical result returned by tf.distribute.Strategy.run. "Mirrored" values are like "PerReplica" values, except we know that the value on all replicas are the same. We can safely read a "Mirrored" value in a cross-replica context by using the value on any replica.
 Unwrapping and merging: Consider calling a function fn on multiple replicas, like strategy.run(fn, args=[w]) with an argument w that is a tf.distribute.DistributedValues. This means w will have a map taking replica id 0 to w0, replica id 1 to w1, etc. strategy.run() unwraps w before calling fn, so it calls fn(w0) on device d0, fn(w1) on device d1, etc. It then merges the return values from fn(), which leads to one common object if the returned values are the same object from every replica, or a DistributedValues object otherwise. Reductions and all-reduce: A reduction is a method of aggregating multiple values into one value, like "sum" or "mean". If a strategy is doing sync training, we will perform a reduction on the gradients to a parameter from all replicas before applying the update. All-reduce is an algorithm for performing a reduction on values from multiple devices and making the result available on all of those devices. Mirrored variables: These are variables that are created on multiple devices, where we keep the variables in sync by applying the same updates to every copy. Mirrored variables are created with tf.Variable(...synchronization=tf.VariableSynchronization.ON_WRITE...). Normally they are only used in synchronous training.
SyncOnRead variables SyncOnRead variables are created by tf.Variable(...synchronization=tf.VariableSynchronization.ON_READ...), and they are created on multiple devices. In replica context, each component variable on the local replica can perform reads and writes without synchronization with each other. When the SyncOnRead variable is read in cross-replica context, the values from component variables are aggregated and returned. SyncOnRead variables bring a lot of custom configuration difficulty to the underlying logic, so we do not encourage users to instantiate and use SyncOnRead variable on their own. We have mainly used SyncOnRead variables for use cases such as batch norm and metrics. For performance reasons, we often don't need to keep these statistics in sync every step and they can be accumulated on each replica independently. The only time we want to sync them is reporting or checkpointing, which typically happens in cross-replica context. SyncOnRead variables are also often used by advanced users who want to control when variable values are aggregated. For example, users sometimes want to maintain gradients independently on each replica for a couple of steps without aggregation.

Distribute-aware layers Layers are generally called in a replica context, except when defining a Keras functional model. tf.distribute.in_cross_replica_context will let you determine which case you are in. If in a replica context, the tf.distribute.get_replica_context function will return the default replica context outside a strategy scope, None within a strategy scope, and a tf.distribute.ReplicaContext object inside a strategy scope and within a tf.distribute.Strategy.run function. The ReplicaContext object has an all_reduce method for aggregating across all replicas.
  Note that we provide a default version of tf.distribute.Strategy that is used when no other strategy is in scope, that provides the same API with reasonable default behavior. Modules cluster_resolver module: Library imports for ClusterResolvers. experimental module: Public API for tf.distribute.experimental namespace. Classes class CrossDeviceOps: Base class for cross-device reduction and broadcasting algorithms. class DistributedDataset: Represents a dataset distributed among devices and machines. class DistributedIterator: An iterator over tf.distribute.DistributedDataset. class DistributedValues: Base class for representing distributed values. class HierarchicalCopyAllReduce: Hierarchical copy all-reduce implementation of CrossDeviceOps. class InputContext: A class wrapping information needed by an input function. class InputOptions: Run options for experimental_distribute_dataset(s_from_function). class InputReplicationMode: Replication mode for input function. class MirroredStrategy: Synchronous training across multiple replicas on one machine. class MultiWorkerMirroredStrategy: A distribution strategy for synchronous training on multiple workers. class NcclAllReduce: NCCL all-reduce implementation of CrossDeviceOps. class OneDeviceStrategy: A distribution strategy for running on a single device. class ReduceOp: Indicates how a set of values should be reduced. class ReductionToOneDevice: A CrossDeviceOps implementation that copies values to one device to reduce. class ReplicaContext: A class with a collection of APIs that can be called in a replica context. class RunOptions: Run options for strategy.run. class Server: An in-process TensorFlow server, for use in distributed training. class Strategy: A state & compute distribution policy on a list of devices. class StrategyExtended: Additional APIs for algorithms that need to be distribution-aware. class TPUStrategy: Synchronous training on TPUs and TPU Pods. Functions experimental_set_strategy(...): Set a tf.distribute.Strategy as current without with strategy.scope(). get_replica_context(...): Returns the current tf.distribute.ReplicaContext or None. get_strategy(...): Returns the current tf.distribute.Strategy object. has_strategy(...): Return if there is a current non-default tf.distribute.Strategy. in_cross_replica_context(...): Returns True if in a cross-replica context.


Module: tf.distribute.cluster_resolver Library imports for ClusterResolvers. This library contains all implementations of ClusterResolvers. ClusterResolvers are a way of specifying cluster information for distributed execution. Built on top of existing ClusterSpec framework, ClusterResolvers are a way for TensorFlow to communicate with various cluster management systems (e.g. GCE, AWS, etc...). Classes class ClusterResolver: Abstract class for all implementations of ClusterResolvers. class GCEClusterResolver: ClusterResolver for Google Compute Engine. class KubernetesClusterResolver: ClusterResolver for Kubernetes. class SimpleClusterResolver: Simple implementation of ClusterResolver that accepts all attributes. class SlurmClusterResolver: ClusterResolver for system with Slurm workload manager. class TFConfigClusterResolver: Implementation of a ClusterResolver which reads the TF_CONFIG EnvVar. class TPUClusterResolver: Cluster Resolver for Google Cloud TPUs. class UnionResolver: Performs a union on underlying ClusterResolvers.


tf.distribute.cluster_resolver.ClusterResolver     View source on GitHub    Abstract class for all implementations of ClusterResolvers.  View aliases  Compat aliases for migration
See Migration guide for more details. tf.compat.v1.distribute.cluster_resolver.ClusterResolver  This defines the skeleton for all implementations of ClusterResolvers. ClusterResolvers are a way for TensorFlow to communicate with various cluster management systems (e.g. GCE, AWS, etc...) and gives TensorFlow necessary information to set up distributed training. By letting TensorFlow communicate with these systems, we will be able to automatically discover and resolve IP addresses for various TensorFlow workers. This will eventually allow us to automatically recover from underlying machine failures and scale TensorFlow worker clusters up and down. Note to Implementors of tf.distribute.cluster_resolver.ClusterResolver subclass: In addition to these abstract methods, when task_type, task_id, and rpc_layer attributes are applicable, you should also implement them either as properties with getters or setters, or directly set the attributes self._task_type, self._task_id, or self._rpc_layer so the base class' getters and setters are used. See tf.distribute.clusterresolver.SimpleClusterResolver.init_ for an example. In general, multi-client tf.distribute strategies such as tf.distribute.experimental.MultiWorkerMirroredStrategy require task_type and task_id properties to be available in the ClusterResolver they are using. On the other hand, these concepts are not applicable in single-client strategies, such as tf.distribute.experimental.TPUStrategy, because the program is only expected to be run on one task, so there should not be a need to have code branches according to task type and task id.  task_type is the name of the server's current named job (e.g. 'worker', 'ps' in a distributed parameterized training job). task_id is the ordinal index of the server within the task type. rpc_layer is the protocol used by TensorFlow to communicate with other TensorFlow servers in a distributed environment.

 Attributes
  environment   Returns the current environment which TensorFlow is running in. There are two possible return values, "google" (when TensorFlow is running in a Google-internal environment) or an empty string (when TensorFlow is running elsewhere). If you are implementing a ClusterResolver that works in both the Google environment and the open-source world (for instance, a TPU ClusterResolver or similar), you will have to return the appropriate string depending on the environment, which you will have to detect. Otherwise, if you are implementing a ClusterResolver that will only work in open-source TensorFlow, you do not need to implement this property.

  task_id   Returns the task id this ClusterResolver indicates. In TensorFlow distributed environment, each job may have an applicable task id, which is the index of the instance within its task type. This is useful when user needs to run specific code according to task index. For example, cluster_spec = tf.train.ClusterSpec({
"ps": ["localhost:2222", "localhost:2223"],
"worker": ["localhost:2224", "localhost:2225", "localhost:2226"]
})

# SimpleClusterResolver is used here for illustration; other cluster
# resolvers may be used for other source of task type/id.
simple_resolver = SimpleClusterResolver(cluster_spec, task_type="worker",
task_id=0)

...

if cluster_resolver.task_type == 'worker' and cluster_resolver.task_id == 0:
# Perform something that's only applicable on 'worker' type, id 0. This
# block will run on this particular instance since we've specified this
# task to be a 'worker', id 0 in above cluster resolver.
else:
# Perform something that's only applicable on other ids. This block will
# not run on this particular instance.
 Returns None if such information is not available or is not applicable in the current distributed environment, such as training with tf.distribute.cluster_resolver.TPUClusterResolver. For more information, please see tf.distribute.cluster_resolver.ClusterResolver's class docstring.

  task_type   Returns the task type this ClusterResolver indicates. In TensorFlow distributed environment, each job may have an applicable task type. Valid task types in TensorFlow include 'chief': a worker that is designated with more responsibility, 'worker': a regular worker for training/evaluation, 'ps': a parameter server, or 'evaluator': an evaluator that evaluates the checkpoints for metrics. See Multi-worker configuration for more information about 'chief' and 'worker' task type, which are most commonly used. Having access to such information is useful when user needs to run specific code according to task types. For example, cluster_spec = tf.train.ClusterSpec({
"ps": ["localhost:2222", "localhost:2223"],
"worker": ["localhost:2224", "localhost:2225", "localhost:2226"]
})

# SimpleClusterResolver is used here for illustration; other cluster
# resolvers may be used for other source of task type/id.
simple_resolver = SimpleClusterResolver(cluster_spec, task_type="worker",
task_id=1)

...

if cluster_resolver.task_type == 'worker':
# Perform something that's only applicable on workers. This block
# will run on this particular instance since we've specified this task to
# be a worker in above cluster resolver.
elif cluster_resolver.task_type == 'ps':
# Perform something that's only applicable on parameter servers. This
# block will not run on this particular instance.
 Returns None if such information is not available or is not applicable in the current distributed environment, such as training with tf.distribute.experimental.TPUStrategy. For more information, please see tf.distribute.cluster_resolver.ClusterResolver's class doc.
   Methods cluster_spec View source
@abc.abstractmethod
cluster_spec()
 Retrieve the current state of the cluster and return a tf.train.ClusterSpec.

 Returns   A tf.train.ClusterSpec representing the state of the cluster at the moment this function is called.
 Implementors of this function must take care in ensuring that the ClusterSpec returned is up-to-date at the time of calling this function. This usually means retrieving the information from the underlying cluster management system every time this function is invoked and reconstructing a cluster_spec, rather than attempting to cache anything. master View source
@abc.abstractmethod
master(
    task_type=None, task_id=None, rpc_layer=None
)
 Retrieves the name or URL of the session master.
Note: this is only useful for TensorFlow 1.x.

 Args
  task_type   (Optional) The type of the TensorFlow task of the master.
  task_id   (Optional) The index of the TensorFlow task of the master.
  rpc_layer   (Optional) The RPC protocol for the given cluster.

 Returns   The name or URL of the session master.
 Implementors of this function must take care in ensuring that the master returned is up-to-date at the time to calling this function. This usually means retrieving the master every time this function is invoked. num_accelerators View source
num_accelerators(
    task_type=None, task_id=None, config_proto=None
)
 Returns the number of accelerator cores per worker. This returns the number of accelerator cores (such as GPUs and TPUs) available per worker. Optionally, we allow callers to specify the task_type, and task_id, for if they want to target a specific TensorFlow task to query the number of accelerators. This is to support heterogenous environments, where the number of accelerators cores per host is different.

 Args
  task_type   (Optional) The type of the TensorFlow task of the machine we want to query.
  task_id   (Optional) The index of the TensorFlow task of the machine we want to query.
  config_proto   (Optional) Configuration for starting a new session to query how many accelerator cores it has.

 Returns   A map of accelerator types to number of cores.


tf.distribute.cluster_resolver.GCEClusterResolver     View source on GitHub    ClusterResolver for Google Compute Engine. Inherits From: ClusterResolver  View aliases  Compat aliases for migration
See Migration guide for more details. tf.compat.v1.distribute.cluster_resolver.GCEClusterResolver
tf.distribute.cluster_resolver.GCEClusterResolver(
    project, zone, instance_group, port, task_type='worker', task_id=0,
    rpc_layer='grpc', credentials='default', service=None
)
 This is an implementation of cluster resolvers for the Google Compute Engine instance group platform. By specifying a project, zone, and instance group, this will retrieve the IP address of all the instances within the instance group and return a ClusterResolver object suitable for use for distributed TensorFlow.
Note: this cluster resolver cannot retrieve task_type, task_id or rpc_layer. To use it with some distribution strategies like tf.distribute.experimental.MultiWorkerMirroredStrategy, you will need to specify task_type and task_id in the constructor.
 Usage example with tf.distribute.Strategy: # On worker 0
cluster_resolver = GCEClusterResolver("my-project", "us-west1",
                                      "my-instance-group",
                                      task_type="worker", task_id=0)
strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy(
    cluster_resolver=cluster_resolver)

# On worker 1
cluster_resolver = GCEClusterResolver("my-project", "us-west1",
                                      "my-instance-group",
                                      task_type="worker", task_id=1)
strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy(
    cluster_resolver=cluster_resolver)

 Args
  project   Name of the GCE project.
  zone   Zone of the GCE instance group.
  instance_group   Name of the GCE instance group.
  port   Port of the listening TensorFlow server (default: 8470)
  task_type   Name of the TensorFlow job this GCE instance group of VM instances belong to.
  task_id   The task index for this particular VM, within the GCE instance group. In particular, every single instance should be assigned a unique ordinal index within an instance group manually so that they can be distinguished from each other.
  rpc_layer   The RPC layer TensorFlow should use to communicate across instances.
  credentials   GCE Credentials. If nothing is specified, this defaults to GoogleCredentials.get_application_default().
  service   The GCE API object returned by the googleapiclient.discovery function. (Default: discovery.build('compute', 'v1')). If you specify a custom service object, then the credentials parameter will be ignored.

 Raises
  ImportError   If the googleapiclient is not installed.

 Attributes
  environment   Returns the current environment which TensorFlow is running in. There are two possible return values, "google" (when TensorFlow is running in a Google-internal environment) or an empty string (when TensorFlow is running elsewhere). If you are implementing a ClusterResolver that works in both the Google environment and the open-source world (for instance, a TPU ClusterResolver or similar), you will have to return the appropriate string depending on the environment, which you will have to detect. Otherwise, if you are implementing a ClusterResolver that will only work in open-source TensorFlow, you do not need to implement this property.

  rpc_layer

  task_id   Returns the task id this ClusterResolver indicates. In TensorFlow distributed environment, each job may have an applicable task id, which is the index of the instance within its task type. This is useful when user needs to run specific code according to task index. For example, cluster_spec = tf.train.ClusterSpec({
"ps": ["localhost:2222", "localhost:2223"],
"worker": ["localhost:2224", "localhost:2225", "localhost:2226"]
})

# SimpleClusterResolver is used here for illustration; other cluster
# resolvers may be used for other source of task type/id.
simple_resolver = SimpleClusterResolver(cluster_spec, task_type="worker",
task_id=0)

...

if cluster_resolver.task_type == 'worker' and cluster_resolver.task_id == 0:
# Perform something that's only applicable on 'worker' type, id 0. This
# block will run on this particular instance since we've specified this
# task to be a 'worker', id 0 in above cluster resolver.
else:
# Perform something that's only applicable on other ids. This block will
# not run on this particular instance.
 Returns None if such information is not available or is not applicable in the current distributed environment, such as training with tf.distribute.cluster_resolver.TPUClusterResolver. For more information, please see tf.distribute.cluster_resolver.ClusterResolver's class docstring.

  task_type   Returns the task type this ClusterResolver indicates. In TensorFlow distributed environment, each job may have an applicable task type. Valid task types in TensorFlow include 'chief': a worker that is designated with more responsibility, 'worker': a regular worker for training/evaluation, 'ps': a parameter server, or 'evaluator': an evaluator that evaluates the checkpoints for metrics. See Multi-worker configuration for more information about 'chief' and 'worker' task type, which are most commonly used. Having access to such information is useful when user needs to run specific code according to task types. For example, cluster_spec = tf.train.ClusterSpec({
"ps": ["localhost:2222", "localhost:2223"],
"worker": ["localhost:2224", "localhost:2225", "localhost:2226"]
})

# SimpleClusterResolver is used here for illustration; other cluster
# resolvers may be used for other source of task type/id.
simple_resolver = SimpleClusterResolver(cluster_spec, task_type="worker",
task_id=1)

...

if cluster_resolver.task_type == 'worker':
# Perform something that's only applicable on workers. This block
# will run on this particular instance since we've specified this task to
# be a worker in above cluster resolver.
elif cluster_resolver.task_type == 'ps':
# Perform something that's only applicable on parameter servers. This
# block will not run on this particular instance.
 Returns None if such information is not available or is not applicable in the current distributed environment, such as training with tf.distribute.experimental.TPUStrategy. For more information, please see tf.distribute.cluster_resolver.ClusterResolver's class doc.
   Methods cluster_spec View source
cluster_spec()
 Returns a ClusterSpec object based on the latest instance group info. This returns a ClusterSpec object for use based on information from the specified instance group. We will retrieve the information from the GCE APIs every time this method is called.

 Returns   A ClusterSpec containing host information retrieved from GCE.
 master View source
master(
    task_type=None, task_id=None, rpc_layer=None
)
 Retrieves the name or URL of the session master.
Note: this is only useful for TensorFlow 1.x.

 Args
  task_type   (Optional) The type of the TensorFlow task of the master.
  task_id   (Optional) The index of the TensorFlow task of the master.
  rpc_layer   (Optional) The RPC protocol for the given cluster.

 Returns   The name or URL of the session master.
 Implementors of this function must take care in ensuring that the master returned is up-to-date at the time to calling this function. This usually means retrieving the master every time this function is invoked. num_accelerators View source
num_accelerators(
    task_type=None, task_id=None, config_proto=None
)
 Returns the number of accelerator cores per worker. This returns the number of accelerator cores (such as GPUs and TPUs) available per worker. Optionally, we allow callers to specify the task_type, and task_id, for if they want to target a specific TensorFlow task to query the number of accelerators. This is to support heterogenous environments, where the number of accelerators cores per host is different.

 Args
  task_type   (Optional) The type of the TensorFlow task of the machine we want to query.
  task_id   (Optional) The index of the TensorFlow task of the machine we want to query.
  config_proto   (Optional) Configuration for starting a new session to query how many accelerator cores it has.

 Returns   A map of accelerator types to number of cores.


tf.distribute.cluster_resolver.KubernetesClusterResolver     View source on GitHub    ClusterResolver for Kubernetes. Inherits From: ClusterResolver  View aliases  Compat aliases for migration
See Migration guide for more details. tf.compat.v1.distribute.cluster_resolver.KubernetesClusterResolver
tf.distribute.cluster_resolver.KubernetesClusterResolver(
    job_to_label_mapping=None, tf_server_port=8470, rpc_layer='grpc',
    override_client=None
)
 This is an implementation of cluster resolvers for Kubernetes. When given the the Kubernetes namespace and label selector for pods, we will retrieve the pod IP addresses of all running pods matching the selector, and return a ClusterSpec based on that information.
Note: it cannot retrieve task_type, task_id or rpc_layer. To use it with some distribution strategies like tf.distribute.experimental.MultiWorkerMirroredStrategy, you will need to specify task_type and task_id by setting these attributes.
 Usage example with tf.distribute.Strategy: # On worker 0
cluster_resolver = KubernetesClusterResolver(
    {"worker": ["job-name=worker-cluster-a", "job-name=worker-cluster-b"]})
cluster_resolver.task_type = "worker"
cluster_resolver.task_id = 0
strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy(
    cluster_resolver=cluster_resolver)

# On worker 1
cluster_resolver = KubernetesClusterResolver(
    {"worker": ["job-name=worker-cluster-a", "job-name=worker-cluster-b"]})
cluster_resolver.task_type = "worker"
cluster_resolver.task_id = 1
strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy(
    cluster_resolver=cluster_resolver)

 Args
  job_to_label_mapping   A mapping of TensorFlow jobs to label selectors. This allows users to specify many TensorFlow jobs in one Cluster Resolver, and each job can have pods belong with different label selectors. For example, a sample mapping might be {'worker': ['job-name=worker-cluster-a', 'job-name=worker-cluster-b'],
'ps': ['job-name=ps-1', 'job-name=ps-2']}

  tf_server_port   The port the TensorFlow server is listening on.
  rpc_layer   (Optional) The RPC layer TensorFlow should use to communicate between tasks in Kubernetes. Defaults to 'grpc'.
  override_client   The Kubernetes client (usually automatically retrieved using from kubernetes import client as k8sclient). If you pass this in, you are responsible for setting Kubernetes credentials manually.

 Raises
  ImportError   If the Kubernetes Python client is not installed and no override_client is passed in.
  RuntimeError   If autoresolve_task is not a boolean or a callable.

 Attributes
  environment   Returns the current environment which TensorFlow is running in. There are two possible return values, "google" (when TensorFlow is running in a Google-internal environment) or an empty string (when TensorFlow is running elsewhere). If you are implementing a ClusterResolver that works in both the Google environment and the open-source world (for instance, a TPU ClusterResolver or similar), you will have to return the appropriate string depending on the environment, which you will have to detect. Otherwise, if you are implementing a ClusterResolver that will only work in open-source TensorFlow, you do not need to implement this property.

  task_id   Returns the task id this ClusterResolver indicates. In TensorFlow distributed environment, each job may have an applicable task id, which is the index of the instance within its task type. This is useful when user needs to run specific code according to task index. For example, cluster_spec = tf.train.ClusterSpec({
"ps": ["localhost:2222", "localhost:2223"],
"worker": ["localhost:2224", "localhost:2225", "localhost:2226"]
})

# SimpleClusterResolver is used here for illustration; other cluster
# resolvers may be used for other source of task type/id.
simple_resolver = SimpleClusterResolver(cluster_spec, task_type="worker",
task_id=0)

...

if cluster_resolver.task_type == 'worker' and cluster_resolver.task_id == 0:
# Perform something that's only applicable on 'worker' type, id 0. This
# block will run on this particular instance since we've specified this
# task to be a 'worker', id 0 in above cluster resolver.
else:
# Perform something that's only applicable on other ids. This block will
# not run on this particular instance.
 Returns None if such information is not available or is not applicable in the current distributed environment, such as training with tf.distribute.cluster_resolver.TPUClusterResolver. For more information, please see tf.distribute.cluster_resolver.ClusterResolver's class docstring.

  task_type   Returns the task type this ClusterResolver indicates. In TensorFlow distributed environment, each job may have an applicable task type. Valid task types in TensorFlow include 'chief': a worker that is designated with more responsibility, 'worker': a regular worker for training/evaluation, 'ps': a parameter server, or 'evaluator': an evaluator that evaluates the checkpoints for metrics. See Multi-worker configuration for more information about 'chief' and 'worker' task type, which are most commonly used. Having access to such information is useful when user needs to run specific code according to task types. For example, cluster_spec = tf.train.ClusterSpec({
"ps": ["localhost:2222", "localhost:2223"],
"worker": ["localhost:2224", "localhost:2225", "localhost:2226"]
})

# SimpleClusterResolver is used here for illustration; other cluster
# resolvers may be used for other source of task type/id.
simple_resolver = SimpleClusterResolver(cluster_spec, task_type="worker",
task_id=1)

...

if cluster_resolver.task_type == 'worker':
# Perform something that's only applicable on workers. This block
# will run on this particular instance since we've specified this task to
# be a worker in above cluster resolver.
elif cluster_resolver.task_type == 'ps':
# Perform something that's only applicable on parameter servers. This
# block will not run on this particular instance.
 Returns None if such information is not available or is not applicable in the current distributed environment, such as training with tf.distribute.experimental.TPUStrategy. For more information, please see tf.distribute.cluster_resolver.ClusterResolver's class doc.
   Methods cluster_spec View source
cluster_spec()
 Returns a ClusterSpec object based on the latest info from Kubernetes. We retrieve the information from the Kubernetes master every time this method is called.

 Returns   A ClusterSpec containing host information returned from Kubernetes.

 Raises
  RuntimeError   If any of the pods returned by the master is not in the Running phase.    master View source
master(
    task_type=None, task_id=None, rpc_layer=None
)
 Returns the master address to use when creating a session. You must have set the task_type and task_id object properties before calling this function, or pass in the task_type and task_id parameters when using this function. If you do both, the function parameters will override the object properties.
Note: this is only useful for TensorFlow 1.x.

 Args
  task_type   (Optional) The type of the TensorFlow task of the master.
  task_id   (Optional) The index of the TensorFlow task of the master.
  rpc_layer   (Optional) The RPC protocol for the given cluster.

 Returns   The name or URL of the session master.
 num_accelerators View source
num_accelerators(
    task_type=None, task_id=None, config_proto=None
)
 Returns the number of accelerator cores per worker. This returns the number of accelerator cores (such as GPUs and TPUs) available per worker. Optionally, we allow callers to specify the task_type, and task_id, for if they want to target a specific TensorFlow task to query the number of accelerators. This is to support heterogenous environments, where the number of accelerators cores per host is different.

 Args
  task_type   (Optional) The type of the TensorFlow task of the machine we want to query.
  task_id   (Optional) The index of the TensorFlow task of the machine we want to query.
  config_proto   (Optional) Configuration for starting a new session to query how many accelerator cores it has.

 Returns   A map of accelerator types to number of cores.


tf.distribute.cluster_resolver.SimpleClusterResolver     View source on GitHub    Simple implementation of ClusterResolver that accepts all attributes. Inherits From: ClusterResolver  View aliases  Compat aliases for migration
See Migration guide for more details. tf.compat.v1.distribute.cluster_resolver.SimpleClusterResolver
tf.distribute.cluster_resolver.SimpleClusterResolver(
    cluster_spec, master='', task_type=None, task_id=None,
    environment='', num_accelerators=None, rpc_layer=None
)
 Please see the base class for documentation of arguments of its constructor. It is useful if you want to specify some or all attributes. Usage example with tf.distribute.Strategy: cluster = tf.train.ClusterSpec({"worker": ["worker0.example.com:2222",
                                           "worker1.example.com:2222"]})

# On worker 0
cluster_resolver = SimpleClusterResolver(cluster, task_type="worker",
                                         task_id=0,
                                         num_accelerators={"GPU": 8},
                                         rpc_layer="grpc")
strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy(
    cluster_resolver=cluster_resolver)

# On worker 1
cluster_resolver = SimpleClusterResolver(cluster, task_type="worker",
                                         task_id=1,
                                         num_accelerators={"GPU": 8},
                                         rpc_layer="grpc")
strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy(
    cluster_resolver=cluster_resolver)

 Attributes
  environment   Returns the current environment which TensorFlow is running in. There are two possible return values, "google" (when TensorFlow is running in a Google-internal environment) or an empty string (when TensorFlow is running elsewhere). If you are implementing a ClusterResolver that works in both the Google environment and the open-source world (for instance, a TPU ClusterResolver or similar), you will have to return the appropriate string depending on the environment, which you will have to detect. Otherwise, if you are implementing a ClusterResolver that will only work in open-source TensorFlow, you do not need to implement this property.

  rpc_layer

  task_id   Returns the task id this ClusterResolver indicates. In TensorFlow distributed environment, each job may have an applicable task id, which is the index of the instance within its task type. This is useful when user needs to run specific code according to task index. For example, cluster_spec = tf.train.ClusterSpec({
"ps": ["localhost:2222", "localhost:2223"],
"worker": ["localhost:2224", "localhost:2225", "localhost:2226"]
})

# SimpleClusterResolver is used here for illustration; other cluster
# resolvers may be used for other source of task type/id.
simple_resolver = SimpleClusterResolver(cluster_spec, task_type="worker",
task_id=0)

...

if cluster_resolver.task_type == 'worker' and cluster_resolver.task_id == 0:
# Perform something that's only applicable on 'worker' type, id 0. This
# block will run on this particular instance since we've specified this
# task to be a 'worker', id 0 in above cluster resolver.
else:
# Perform something that's only applicable on other ids. This block will
# not run on this particular instance.
 Returns None if such information is not available or is not applicable in the current distributed environment, such as training with tf.distribute.cluster_resolver.TPUClusterResolver. For more information, please see tf.distribute.cluster_resolver.ClusterResolver's class docstring.

  task_type   Returns the task type this ClusterResolver indicates. In TensorFlow distributed environment, each job may have an applicable task type. Valid task types in TensorFlow include 'chief': a worker that is designated with more responsibility, 'worker': a regular worker for training/evaluation, 'ps': a parameter server, or 'evaluator': an evaluator that evaluates the checkpoints for metrics. See Multi-worker configuration for more information about 'chief' and 'worker' task type, which are most commonly used. Having access to such information is useful when user needs to run specific code according to task types. For example, cluster_spec = tf.train.ClusterSpec({
"ps": ["localhost:2222", "localhost:2223"],
"worker": ["localhost:2224", "localhost:2225", "localhost:2226"]
})

# SimpleClusterResolver is used here for illustration; other cluster
# resolvers may be used for other source of task type/id.
simple_resolver = SimpleClusterResolver(cluster_spec, task_type="worker",
task_id=1)

...

if cluster_resolver.task_type == 'worker':
# Perform something that's only applicable on workers. This block
# will run on this particular instance since we've specified this task to
# be a worker in above cluster resolver.
elif cluster_resolver.task_type == 'ps':
# Perform something that's only applicable on parameter servers. This
# block will not run on this particular instance.
 Returns None if such information is not available or is not applicable in the current distributed environment, such as training with tf.distribute.experimental.TPUStrategy. For more information, please see tf.distribute.cluster_resolver.ClusterResolver's class doc.
   Methods cluster_spec View source
cluster_spec()
 Returns the ClusterSpec passed into the constructor. master View source
master(
    task_type=None, task_id=None, rpc_layer=None
)
 Returns the master address to use when creating a session.
Note: this is only useful for TensorFlow 1.x.

 Args
  task_type   (Optional) The type of the TensorFlow task of the master.
  task_id   (Optional) The index of the TensorFlow task of the master.
  rpc_layer   (Optional) The RPC used by distributed TensorFlow.

 Returns   The name or URL of the session master.
 If a task_type and task_id is given, this will override the master string passed into the initialization function. num_accelerators View source
num_accelerators(
    task_type=None, task_id=None, config_proto=None
)
 Returns the number of accelerator cores per worker. The SimpleClusterResolver does not do automatic detection of accelerators, and thus all arguments are unused and we simply return the value provided in the constructor.

 Args
  task_type   Unused.
  task_id   Unused.
  config_proto   Unused.


tf.distribute.cluster_resolver.SlurmClusterResolver     View source on GitHub    ClusterResolver for system with Slurm workload manager. Inherits From: ClusterResolver  View aliases  Compat aliases for migration
See Migration guide for more details. tf.compat.v1.distribute.cluster_resolver.SlurmClusterResolver
tf.distribute.cluster_resolver.SlurmClusterResolver(
    jobs=None, port_base=8888, gpus_per_node=None, gpus_per_task=None,
    tasks_per_node=None, auto_set_gpu=True, rpc_layer='grpc'
)
 This is an implementation of ClusterResolver for Slurm clusters. This allows the specification of jobs and task counts, number of tasks per node, number of GPUs on each node and number of GPUs for each task. It retrieves system attributes by Slurm environment variables, resolves allocated computing node names, constructs a cluster and returns a ClusterResolver object which can be used for distributed TensorFlow.

 Args
  jobs   Dictionary with job names as key and number of tasks in the job as value. Defaults to as many 'worker's as there are (Slurm) tasks.
  port_base   The first port number to start with for processes on a node.
  gpus_per_node   Number of GPUs available on each node. Defaults to the number of GPUs reported by nvidia-smi
  gpus_per_task   Number of GPUs to be used for each task. Default is to evenly distribute the gpus_per_node to tasks_per_node.
  tasks_per_node   Number of tasks running on each node. Can be an integer if the number of tasks per node is constant or a dictionary mapping hostnames to number of tasks on that node. If not set the Slurm environment is queried for the correct mapping.
  auto_set_gpu   Set the visible CUDA devices automatically while resolving the cluster by setting CUDA_VISIBLE_DEVICES environment variable. Defaults to True.
  rpc_layer   The protocol TensorFlow used to communicate between nodes. Defaults to 'grpc'.

 Raises
  RuntimeError   If requested more GPUs per node then available or requested more tasks then assigned tasks or resolving missing values from the environment failed.

 Attributes
  environment   Returns the current environment which TensorFlow is running in. There are two possible return values, "google" (when TensorFlow is running in a Google-internal environment) or an empty string (when TensorFlow is running elsewhere). If you are implementing a ClusterResolver that works in both the Google environment and the open-source world (for instance, a TPU ClusterResolver or similar), you will have to return the appropriate string depending on the environment, which you will have to detect. Otherwise, if you are implementing a ClusterResolver that will only work in open-source TensorFlow, you do not need to implement this property.

  task_id   Returns the task id this ClusterResolver indicates. In TensorFlow distributed environment, each job may have an applicable task id, which is the index of the instance within its task type. This is useful when user needs to run specific code according to task index. For example, cluster_spec = tf.train.ClusterSpec({
"ps": ["localhost:2222", "localhost:2223"],
"worker": ["localhost:2224", "localhost:2225", "localhost:2226"]
})

# SimpleClusterResolver is used here for illustration; other cluster
# resolvers may be used for other source of task type/id.
simple_resolver = SimpleClusterResolver(cluster_spec, task_type="worker",
task_id=0)

...

if cluster_resolver.task_type == 'worker' and cluster_resolver.task_id == 0:
# Perform something that's only applicable on 'worker' type, id 0. This
# block will run on this particular instance since we've specified this
# task to be a 'worker', id 0 in above cluster resolver.
else:
# Perform something that's only applicable on other ids. This block will
# not run on this particular instance.
 Returns None if such information is not available or is not applicable in the current distributed environment, such as training with tf.distribute.cluster_resolver.TPUClusterResolver. For more information, please see tf.distribute.cluster_resolver.ClusterResolver's class docstring.

  task_type   Returns the task type this ClusterResolver indicates. In TensorFlow distributed environment, each job may have an applicable task type. Valid task types in TensorFlow include 'chief': a worker that is designated with more responsibility, 'worker': a regular worker for training/evaluation, 'ps': a parameter server, or 'evaluator': an evaluator that evaluates the checkpoints for metrics. See Multi-worker configuration for more information about 'chief' and 'worker' task type, which are most commonly used. Having access to such information is useful when user needs to run specific code according to task types. For example, cluster_spec = tf.train.ClusterSpec({
"ps": ["localhost:2222", "localhost:2223"],
"worker": ["localhost:2224", "localhost:2225", "localhost:2226"]
})

# SimpleClusterResolver is used here for illustration; other cluster
# resolvers may be used for other source of task type/id.
simple_resolver = SimpleClusterResolver(cluster_spec, task_type="worker",
task_id=1)

...

if cluster_resolver.task_type == 'worker':
# Perform something that's only applicable on workers. This block
# will run on this particular instance since we've specified this task to
# be a worker in above cluster resolver.
elif cluster_resolver.task_type == 'ps':
# Perform something that's only applicable on parameter servers. This
# block will not run on this particular instance.
 Returns None if such information is not available or is not applicable in the current distributed environment, such as training with tf.distribute.experimental.TPUStrategy. For more information, please see tf.distribute.cluster_resolver.ClusterResolver's class doc.
   Methods cluster_spec View source
cluster_spec()
 Returns a ClusterSpec object based on the latest instance group info. This returns a ClusterSpec object for use based on information from the specified initialization parameters and Slurm environment variables. The cluster specification is resolved each time this function is called. The resolver extract hostnames of nodes by scontrol and pack tasks in that order until a node a has number of tasks that is equal to specification. GPUs on nodes are allocated to tasks by specification through setting CUDA_VISIBLE_DEVICES environment variable.

 Returns   A ClusterSpec containing host information retrieved from Slurm's environment variables.
 get_task_info View source
get_task_info()
 Returns job name and task_id for the process which calls this. This returns the job name and task index for the process which calls this function according to its rank and cluster specification. The job name and task index are set after a cluster is constructed by cluster_spec otherwise defaults to None.

 Returns   A string specifying job name the process belongs to and an integer specifying the task index the process belongs to in that job.
 master View source
master(
    task_type=None, task_id=None, rpc_layer=None
)
 Returns the master string for connecting to a TensorFlow master.

 Args
  task_type   (Optional) Overrides the default auto-selected task type.
  task_id   (Optional) Overrides the default auto-selected task index.
  rpc_layer   (Optional) Overrides the default RPC protocol TensorFlow uses to communicate across nodes.

 Returns   A connection string for connecting to a TensorFlow master.
 num_accelerators View source
num_accelerators(
    task_type=None, task_id=None, config_proto=None
)
 Returns the number of accelerator cores per worker. This returns the number of accelerator cores (such as GPUs and TPUs) available per worker. Optionally, we allow callers to specify the task_type, and task_id, for if they want to target a specific TensorFlow task to query the number of accelerators. This is to support heterogenous environments, where the number of accelerators cores per host is different.

 Args
  task_type   (Optional) The type of the TensorFlow task of the machine we want to query.
  task_id   (Optional) The index of the TensorFlow task of the machine we want to query.
  config_proto   (Optional) Configuration for starting a new session to query how many accelerator cores it has.

 Returns   A map of accelerator types to number of cores.