A Dataset comprising lines from one or more CSV files.
Inherits From: Dataset
View source
Applies a transformation function to this dataset.
apply enables chaining of custom Dataset transformations, which are represented as functions that take one Dataset argument and return a transformed Dataset.
View source
Returns an iterator which converts all elements of the dataset to numpy.
Use as_numpy_iterator to inspect the content of your dataset. To see element shapes and types, print dataset elements directly instead of using as_numpy_iterator.
This method requires that you are running in eager mode and the dataset's element_spec contains only TensorSpec components.
as_numpy_iterator() will preserve the nested structure of dataset elements.
View source
Combines consecutive elements of this dataset into batches.
The components of the resulting element will have an additional outer dimension, which will be batch_size (or N % batch_size for the last element if batch_size does not divide the number of input elements N evenly and drop_remainder is False). If your program depends on the batches having the same outer dimension, you should set the drop_remainder argument to True to prevent the smaller batch from being produced.
View source
Caches the elements in this dataset.
The first time the dataset is iterated over, its elements will be cached either in the specified file or in memory. Subsequent iterations will use the cached data.
When caching to a file, the cached data will persist across runs. Even the first iteration through the data will read from the cache file. Changing the input pipeline before the call to .cache() will have no effect until the cache file is removed or the filename is changed.
View source
Returns the cardinality of the dataset, if known.
cardinality may return tf.data.INFINITE_CARDINALITY if the dataset contains an infinite number of elements or tf.data.UNKNOWN_CARDINALITY if the analysis fails to determine the number of elements in the dataset (e.g. when the dataset source is a file).
View source
Creates a Dataset by concatenating the given dataset with this dataset.
View source
Enumerates the elements of this dataset.
It is similar to python's enumerate.
View source
Filters this dataset according to predicate.
View source
Maps map_func across this dataset and flattens the result.
Use flat_map if you want to make sure that the order of your dataset stays the same. For example, to flatten a dataset of batches into a dataset of their elements:
tf.data.Dataset.interleave() is a generalization of flat_map, since flat_map produces the same output as tf.data.Dataset.interleave(cycle_length=1)
View source
Creates a Dataset whose elements are generated by generator. (deprecated arguments)
The generator argument must be a callable object that returns an object that supports the iter() protocol (e.g. a generator function).
The elements generated by generator must be compatible with either the given output_signature argument or with the given output_types and (optionally) output_shapes arguments, whichiver was specified.
The recommended way to call from_generator is to use the output_signature argument. In this case the output will be assumed to consist of objects with the classes, shapes and types defined by tf.TypeSpec objects from output_signature argument:
There is also a deprecated way to call from_generator by either with output_types argument alone or together with output_shapes argument. In this case the output of the function will be assumed to consist of tf.Tensor objects with with the types defined by output_types and with the shapes which are either unknown or defined by output_shapes.
View source
Creates a Dataset whose elements are slices of the given tensors.
The given tensors are sliced along their first dimension. This operation preserves the structure of the input tensors, removing the first dimension of each tensor and using it as the dataset dimension. All input tensors must have the same size in their first dimensions.
Note that if tensors contains a NumPy array, and eager execution is not enabled, the values will be embedded in the graph as one or more tf.constant operations. For large datasets (> 1 GB), this can waste memory and run into byte limits of graph serialization. If tensors contains one or more large NumPy arrays, consider the alternative described in this guide.
View source
Creates a Dataset with a single element, comprising the given tensors.
from_tensors produces a dataset containing only a single element. To slice the input tensor into multiple elements, use from_tensor_slices instead.
Note that if tensors contains a NumPy array, and eager execution is not enabled, the values will be embedded in the graph as one or more tf.constant operations. For large datasets (> 1 GB), this can waste memory and run into byte limits of graph serialization. If tensors contains one or more large NumPy arrays, consider the alternative described in this guide.
View source
Maps map_func across this dataset, and interleaves the results.
For example, you can use Dataset.interleave() to process many input files concurrently:
The cycle_length and block_length arguments control the order in which elements are produced. cycle_length controls the number of input elements that are processed concurrently. If you set cycle_length to 1, this transformation will handle one input element at a time, and will produce identical results to tf.data.Dataset.flat_map. In general, this transformation will apply map_func to cycle_length input elements, open iterators on the returned Dataset objects, and cycle through them producing block_length consecutive elements from each iterator, and consuming the next input element each time it reaches the end of an iterator.
Performance can often be improved by setting num_parallel_calls so that interleave will use multiple threads to fetch elements. If determinism isn't required, it can also improve performance to set deterministic=False.
View source
A dataset of all files matching one or more glob patterns.
The file_pattern argument should be a small number of glob patterns. If your filenames have already been globbed, use Dataset.from_tensor_slices(filenames) instead, as re-globbing every filename with list_files may result in poor performance with remote storage systems.
If we had the following files on our filesystem:
If we pass "/path/to/dir/*.py" as the directory, the dataset would produce:
View source
Maps map_func across the elements of this dataset.
This transformation applies map_func to each element of this dataset, and returns a new dataset containing the transformed elements, in the same order as they appeared in the input. map_func can be used to change both the values and the structure of a dataset's elements. For example, adding 1 to each element, or projecting a subset of element components.
The input signature of map_func is determined by the structure of each element in this dataset.
The value or values returned by map_func determine the structure of each element in the returned dataset.
map_func can accept as arguments and return any type of dataset element.
Note that irrespective of the context in which map_func is defined (eager vs. graph), tf.data traces the function and executes it as a graph. To use Python code inside of the function you have a few options:
1) Rely on AutoGraph to convert Python code into an equivalent graph computation. The downside of this approach is that AutoGraph can convert some but not all Python code.
2) Use tf.py_function, which allows you to write arbitrary Python code but will generally result in worse performance than 1). For example:
3) Use tf.numpy_function, which also allows you to write arbitrary Python code. Note that tf.py_function accepts tf.Tensor whereas tf.numpy_function accepts numpy arrays and returns only numpy arrays. For example:
Note that the use of tf.numpy_function and tf.py_function in general precludes the possibility of executing user-defined transformations in parallel (because of Python GIL).
Performance can often be improved by setting num_parallel_calls so that map will use multiple threads to process elements. If deterministic order isn't required, it can also improve performance to set deterministic=False.
View source
Returns the options for this dataset and its inputs.
View source
Combines consecutive elements of this dataset into padded batches.
This transformation combines multiple consecutive elements of the input dataset into a single element.
Like tf.data.Dataset.batch, the components of the resulting element will have an additional outer dimension, which will be batch_size (or N % batch_size for the last element if batch_size does not divide the number of input elements N evenly and drop_remainder is False). If your program depends on the batches having the same outer dimension, you should set the drop_remainder argument to True to prevent the smaller batch from being produced.
Unlike tf.data.Dataset.batch, the input elements to be batched may have different shapes, and this transformation will pad each component to the respective shape in padded_shapes. The padded_shapes argument determines the resulting shape for each dimension of each component in an output element:
See also tf.data.experimental.dense_to_sparse_batch, which combines elements that may have different shapes into a tf.sparse.SparseTensor.
View source
Creates a Dataset that prefetches elements from this dataset.
Most dataset input pipelines should end with a call to prefetch. This allows later elements to be prepared while the current element is being processed. This often improves latency and throughput, at the cost of using additional memory to store prefetched elements.
View source
Creates a Dataset of a step-separated range of values.
View source
Reduces the input dataset to a single element.
The transformation calls reduce_func successively on every element of the input dataset until the dataset is exhausted, aggregating information in its internal state. The initial_state argument is used for the initial state and the final state is returned as the result.
View source
Repeats this dataset so each original value is seen count times.
View source
Creates a Dataset that includes only 1/num_shards of this dataset.
shard is deterministic. The Dataset produced by A.shard(n, i) will contain all elements of A whose index mod n = i.
This dataset operator is very useful when running distributed training, as it allows each worker to read a unique subset.
When reading a single input file, you can shard elements as follows:
View source
Randomly shuffles the elements of this dataset.
This dataset fills a buffer with buffer_size elements, then randomly samples elements from this buffer, replacing the selected elements with new elements. For perfect shuffling, a buffer size greater than or equal to the full size of the dataset is required.
For instance, if your dataset contains 10,000 elements but buffer_size is set to 1,000, then shuffle will initially select a random element from only the first 1,000 elements in the buffer. Once an element is selected, its space in the buffer is replaced by the next (i.e. 1,001-st) element, maintaining the 1,000 element buffer.
reshuffle_each_iteration controls whether the shuffle order should be different for each epoch. In TF 1.X, the idiomatic way to create epochs was through the repeat transformation:
In TF 2.0, tf.data.Dataset objects are Python iterables which makes it possible to also create epochs through Python iteration:
View source
Creates a Dataset that skips count elements from this dataset.
View source
Creates a Dataset with at most count elements from this dataset.
View source
Splits elements of a dataset into multiple elements.
For example, if elements of the dataset are shaped [B, a0, a1, ...], where B may vary for each input element, then for each element in the dataset, the unbatched dataset will contain B consecutive elements of shape [a0, a1, ...].
View source
Combines (nests of) input elements into a dataset of (nests of) windows.
A "window" is a finite dataset of flat elements of size size (or possibly fewer if there are not enough input elements to fill the window and drop_remainder evaluates to False).
The shift argument determines the number of input elements by which the window moves on each iteration. If windows and elements are both numbered starting at 0, the first element in window k will be element k * shift of the input dataset. In particular, the first element of the first window will always be the first element of the input dataset.
The stride argument determines the stride of the input elements, and the shift argument determines the shift of the window.
Note that when the window transformation is applied to a dataset of nested elements, it produces a dataset of nested windows.
View source
Returns a new tf.data.Dataset with the given options set.
The options are "global" in the sense they apply to the entire dataset. If options are set multiple times, they are merged as long as different options do not use different non-default values.
View source
Creates a Dataset by zipping together the given datasets.
This method has similar semantics to the built-in zip() function in Python, with the main difference being that the datasets argument can be an arbitrary nested structure of Dataset objects.
View source
View source
Creates an iterator for elements of this dataset.
The returned iterator implements the Python Iterator protocol.
View source
Returns the length of the dataset if it is known and finite.
This method requires that you are running in eager mode, and that the length of the dataset is known and non-infinite. When the length may be unknown or infinite, or if you are running in graph mode, use tf.data.Dataset.cardinality instead.
View source


A transformation that batches ragged elements into tf.RaggedTensors.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.dense_to_ragged_batch
This transformation combines multiple consecutive elements of the input dataset into a single element.
Like tf.data.Dataset.batch, the components of the resulting element will have an additional outer dimension, which will be batch_size (or N % batch_size for the last element if batch_size does not divide the number of input elements N evenly and drop_remainder is False). If your program depends on the batches having the same outer dimension, you should set the drop_remainder argument to True to prevent the smaller batch from being produced.
Unlike tf.data.Dataset.batch, the input elements to be batched may have different shapes:


A transformation that batches ragged elements into tf.sparse.SparseTensors.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.dense_to_sparse_batch
Like Dataset.padded_batch(), this transformation combines multiple consecutive elements of the dataset, which might have different shapes, into a single element. The resulting element has three components (indices, values, and dense_shape), which comprise a tf.sparse.SparseTensor that represents the same data. The row_shape represents the dense shape of each row in the resulting tf.sparse.SparseTensor, to which the effective batch size is prepended. For example:


Represents options for distributed data processing.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.DistributeOptions
You can set the distribution options of a dataset through the experimental_distribute property of tf.data.Options; the property is an instance of tf.data.experimental.DistributeOptions.
View source
Return self==value.
View source
Return self!=value.


A transformation that enumerates the elements of a dataset. (deprecated)
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.enumerate_dataset
It is similar to python's enumerate. For example:


Constructs a dataset from the given variant and structure.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.from_variant


Returns a tf.experimental.Optional with the next element of the iterator. (deprecated)
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.get_next_as_optional
If the iterator has reached the end of the sequence, the returned tf.experimental.Optional will have no value.


Returns the single element in dataset as a nested structure of tensors.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.get_single_element
This function enables you to use a tf.data.Dataset in a stateless "tensor-in tensor-out" expression, without creating an iterator. This can be useful when your preprocessing transformations are expressed as a Dataset, and you want to use the transformation at serving time.


Returns the type signature for elements of the input dataset / iterator.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.get_structure


A transformation that groups elements and performs a reduction.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.group_by_reducer
This transformation maps element of a dataset to a key using key_func and groups the elements by key. The reducer is used to process each group; its init_func is used to initialize state for each group when it is created, the reduce_func is used to update the state every time an element is mapped to the matching group, and the finalize_func is used to map the final state to an output value.


A transformation that groups windows of elements by key and reduces them.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.group_by_window
This transformation maps each consecutive element in a dataset to a key using key_func and groups the elements by key. It then applies reduce_func to at most window_size_func(key) elements matching the same key. All except the final window for each key will contain window_size_func(key) elements; the final window may be smaller.
You may provide either a constant window_size or a window size determined by the key through window_size_func.


Creates a Dataset from another Dataset and silently ignores any errors.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.ignore_errors
Use this transformation to produce a dataset that contains the same elements as the input, but silently drops any elements that caused an error. For example:
Args: log_warning: (Optional.) A 'tf.bool' scalar indicating whether ignored errors should be logged to stderr. Defaults to 'False'.


Records the latency of producing each element of the input dataset.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.latency_stats
To consume the statistics, associate a StatsAggregator with the output dataset.


Loads a previously saved dataset.
Note that to load a previously saved dataset, you need to specify element_spec -- a type signature of the elements of the saved dataset, which can be obtained via tf.data.Dataset.element_spec. This requirement exists so that shape inference of the loaded dataset does not need to perform I/O.
If the default option of sharding the saved dataset was used, the element order of the saved dataset will be preserved when loading it.
The reader_func argument can be used to specify a custom order in which elements should be loaded from the individual shards. The reader_func is expected to take a single argument -- a dataset of datasets, each containing elements of one of the shards -- and return a dataset of elements. For example, the order of shards can be shuffled when loading them as follows:


Returns a Dataset of feature dictionaries from Example protos.
If label_key argument is provided, returns a Dataset of tuple comprising of feature dictionaries and label.
And the expected output is:


Reads CSV files into a dataset.
Reads CSV files into a dataset, where each element is a (features, labels) tuple that corresponds to a batch of CSV rows. The features dictionary maps feature column names to Tensors containing the corresponding feature data, and labels is a Tensor containing the batch's label data.


Returns a SaveableObject for saving/restoring iterator state using Saver. (deprecated)
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.make_saveable_from_iterator


Represents options for the MapVectorization optimization.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.MapVectorizationOptions
View source
Return self==value.
View source
Return self!=value.


Fused implementation of map and batch. (deprecated)
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.map_and_batch
Maps map_func across batch_size consecutive elements of this dataset and then combines them into a batch. Functionally, it is equivalent to map followed by batch. This API is temporary and deprecated since input pipeline optimization now fuses consecutive map and batch operations automatically.


Represents options for dataset optimizations.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.OptimizationOptions
You can set the optimization options of a dataset through the experimental_optimization property of tf.data.Options; the property is an instance of tf.data.experimental.OptimizationOptions.
View source
Return self==value.
View source
Return self!=value.


A parallel version of the Dataset.interleave() transformation. (deprecated)
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.parallel_interleave
parallel_interleave() maps map_func across its input to produce nested datasets, and outputs their elements interleaved. Unlike tf.data.Dataset.interleave, it gets elements from cycle_length nested datasets in parallel, which increases the throughput, especially in the presence of stragglers. Furthermore, the sloppy argument can be used to improve performance, by relaxing the requirement that the outputs are produced in a deterministic order, and allowing the implementation to skip over nested datasets whose elements are not readily available when requested.


A transformation that parses Example protos into a dict of tensors.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.parse_example_dataset
Parses a number of serialized Example protos given in serialized. We refer to serialized as a batch with batch_size many entries of individual Example protos.
This op parses serialized examples into a dictionary mapping keys to Tensor, SparseTensor, and RaggedTensor objects. features is a dict from keys to VarLenFeature, RaggedFeature, SparseFeature, and FixedLenFeature objects. Each VarLenFeature and SparseFeature is mapped to a SparseTensor; each RaggedFeature is mapped to a RaggedTensor; and each FixedLenFeature is mapped to a Tensor. See tf.io.parse_example for more details about feature dictionaries.


A transformation that prefetches dataset values to the given device.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.prefetch_to_device


A Dataset of pseudorandom values.
Inherits From: Dataset
View source
Applies a transformation function to this dataset.
apply enables chaining of custom Dataset transformations, which are represented as functions that take one Dataset argument and return a transformed Dataset.
View source
Returns an iterator which converts all elements of the dataset to numpy.
Use as_numpy_iterator to inspect the content of your dataset. To see element shapes and types, print dataset elements directly instead of using as_numpy_iterator.
This method requires that you are running in eager mode and the dataset's element_spec contains only TensorSpec components.
as_numpy_iterator() will preserve the nested structure of dataset elements.
View source
Combines consecutive elements of this dataset into batches.
The components of the resulting element will have an additional outer dimension, which will be batch_size (or N % batch_size for the last element if batch_size does not divide the number of input elements N evenly and drop_remainder is False). If your program depends on the batches having the same outer dimension, you should set the drop_remainder argument to True to prevent the smaller batch from being produced.
View source
Caches the elements in this dataset.
The first time the dataset is iterated over, its elements will be cached either in the specified file or in memory. Subsequent iterations will use the cached data.
When caching to a file, the cached data will persist across runs. Even the first iteration through the data will read from the cache file. Changing the input pipeline before the call to .cache() will have no effect until the cache file is removed or the filename is changed.
View source
Returns the cardinality of the dataset, if known.
cardinality may return tf.data.INFINITE_CARDINALITY if the dataset contains an infinite number of elements or tf.data.UNKNOWN_CARDINALITY if the analysis fails to determine the number of elements in the dataset (e.g. when the dataset source is a file).
View source
Creates a Dataset by concatenating the given dataset with this dataset.
View source
Enumerates the elements of this dataset.
It is similar to python's enumerate.
View source
Filters this dataset according to predicate.
View source
Maps map_func across this dataset and flattens the result.
Use flat_map if you want to make sure that the order of your dataset stays the same. For example, to flatten a dataset of batches into a dataset of their elements:
tf.data.Dataset.interleave() is a generalization of flat_map, since flat_map produces the same output as tf.data.Dataset.interleave(cycle_length=1)
View source
Creates a Dataset whose elements are generated by generator. (deprecated arguments)
The generator argument must be a callable object that returns an object that supports the iter() protocol (e.g. a generator function).
The elements generated by generator must be compatible with either the given output_signature argument or with the given output_types and (optionally) output_shapes arguments, whichiver was specified.
The recommended way to call from_generator is to use the output_signature argument. In this case the output will be assumed to consist of objects with the classes, shapes and types defined by tf.TypeSpec objects from output_signature argument:
There is also a deprecated way to call from_generator by either with output_types argument alone or together with output_shapes argument. In this case the output of the function will be assumed to consist of tf.Tensor objects with with the types defined by output_types and with the shapes which are either unknown or defined by output_shapes.
View source
Creates a Dataset whose elements are slices of the given tensors.
The given tensors are sliced along their first dimension. This operation preserves the structure of the input tensors, removing the first dimension of each tensor and using it as the dataset dimension. All input tensors must have the same size in their first dimensions.
Note that if tensors contains a NumPy array, and eager execution is not enabled, the values will be embedded in the graph as one or more tf.constant operations. For large datasets (> 1 GB), this can waste memory and run into byte limits of graph serialization. If tensors contains one or more large NumPy arrays, consider the alternative described in this guide.
View source
Creates a Dataset with a single element, comprising the given tensors.
from_tensors produces a dataset containing only a single element. To slice the input tensor into multiple elements, use from_tensor_slices instead.
Note that if tensors contains a NumPy array, and eager execution is not enabled, the values will be embedded in the graph as one or more tf.constant operations. For large datasets (> 1 GB), this can waste memory and run into byte limits of graph serialization. If tensors contains one or more large NumPy arrays, consider the alternative described in this guide.
View source
Maps map_func across this dataset, and interleaves the results.
For example, you can use Dataset.interleave() to process many input files concurrently:
The cycle_length and block_length arguments control the order in which elements are produced. cycle_length controls the number of input elements that are processed concurrently. If you set cycle_length to 1, this transformation will handle one input element at a time, and will produce identical results to tf.data.Dataset.flat_map. In general, this transformation will apply map_func to cycle_length input elements, open iterators on the returned Dataset objects, and cycle through them producing block_length consecutive elements from each iterator, and consuming the next input element each time it reaches the end of an iterator.
Performance can often be improved by setting num_parallel_calls so that interleave will use multiple threads to fetch elements. If determinism isn't required, it can also improve performance to set deterministic=False.
View source
A dataset of all files matching one or more glob patterns.
The file_pattern argument should be a small number of glob patterns. If your filenames have already been globbed, use Dataset.from_tensor_slices(filenames) instead, as re-globbing every filename with list_files may result in poor performance with remote storage systems.
If we had the following files on our filesystem:
If we pass "/path/to/dir/*.py" as the directory, the dataset would produce:
View source
Maps map_func across the elements of this dataset.
This transformation applies map_func to each element of this dataset, and returns a new dataset containing the transformed elements, in the same order as they appeared in the input. map_func can be used to change both the values and the structure of a dataset's elements. For example, adding 1 to each element, or projecting a subset of element components.
The input signature of map_func is determined by the structure of each element in this dataset.
The value or values returned by map_func determine the structure of each element in the returned dataset.
map_func can accept as arguments and return any type of dataset element.
Note that irrespective of the context in which map_func is defined (eager vs. graph), tf.data traces the function and executes it as a graph. To use Python code inside of the function you have a few options:
1) Rely on AutoGraph to convert Python code into an equivalent graph computation. The downside of this approach is that AutoGraph can convert some but not all Python code.
2) Use tf.py_function, which allows you to write arbitrary Python code but will generally result in worse performance than 1). For example:
3) Use tf.numpy_function, which also allows you to write arbitrary Python code. Note that tf.py_function accepts tf.Tensor whereas tf.numpy_function accepts numpy arrays and returns only numpy arrays. For example:
Note that the use of tf.numpy_function and tf.py_function in general precludes the possibility of executing user-defined transformations in parallel (because of Python GIL).
Performance can often be improved by setting num_parallel_calls so that map will use multiple threads to process elements. If deterministic order isn't required, it can also improve performance to set deterministic=False.
View source
Returns the options for this dataset and its inputs.
View source
Combines consecutive elements of this dataset into padded batches.
This transformation combines multiple consecutive elements of the input dataset into a single element.
Like tf.data.Dataset.batch, the components of the resulting element will have an additional outer dimension, which will be batch_size (or N % batch_size for the last element if batch_size does not divide the number of input elements N evenly and drop_remainder is False). If your program depends on the batches having the same outer dimension, you should set the drop_remainder argument to True to prevent the smaller batch from being produced.
Unlike tf.data.Dataset.batch, the input elements to be batched may have different shapes, and this transformation will pad each component to the respective shape in padded_shapes. The padded_shapes argument determines the resulting shape for each dimension of each component in an output element:
See also tf.data.experimental.dense_to_sparse_batch, which combines elements that may have different shapes into a tf.sparse.SparseTensor.
View source
Creates a Dataset that prefetches elements from this dataset.
Most dataset input pipelines should end with a call to prefetch. This allows later elements to be prepared while the current element is being processed. This often improves latency and throughput, at the cost of using additional memory to store prefetched elements.
View source
Creates a Dataset of a step-separated range of values.
View source
Reduces the input dataset to a single element.
The transformation calls reduce_func successively on every element of the input dataset until the dataset is exhausted, aggregating information in its internal state. The initial_state argument is used for the initial state and the final state is returned as the result.
View source
Repeats this dataset so each original value is seen count times.
View source
Creates a Dataset that includes only 1/num_shards of this dataset.
shard is deterministic. The Dataset produced by A.shard(n, i) will contain all elements of A whose index mod n = i.
This dataset operator is very useful when running distributed training, as it allows each worker to read a unique subset.
When reading a single input file, you can shard elements as follows:
View source
Randomly shuffles the elements of this dataset.
This dataset fills a buffer with buffer_size elements, then randomly samples elements from this buffer, replacing the selected elements with new elements. For perfect shuffling, a buffer size greater than or equal to the full size of the dataset is required.
For instance, if your dataset contains 10,000 elements but buffer_size is set to 1,000, then shuffle will initially select a random element from only the first 1,000 elements in the buffer. Once an element is selected, its space in the buffer is replaced by the next (i.e. 1,001-st) element, maintaining the 1,000 element buffer.
reshuffle_each_iteration controls whether the shuffle order should be different for each epoch. In TF 1.X, the idiomatic way to create epochs was through the repeat transformation:
In TF 2.0, tf.data.Dataset objects are Python iterables which makes it possible to also create epochs through Python iteration:
View source
Creates a Dataset that skips count elements from this dataset.
View source
Creates a Dataset with at most count elements from this dataset.
View source
Splits elements of a dataset into multiple elements.
For example, if elements of the dataset are shaped [B, a0, a1, ...], where B may vary for each input element, then for each element in the dataset, the unbatched dataset will contain B consecutive elements of shape [a0, a1, ...].
View source
Combines (nests of) input elements into a dataset of (nests of) windows.
A "window" is a finite dataset of flat elements of size size (or possibly fewer if there are not enough input elements to fill the window and drop_remainder evaluates to False).
The shift argument determines the number of input elements by which the window moves on each iteration. If windows and elements are both numbered starting at 0, the first element in window k will be element k * shift of the input dataset. In particular, the first element of the first window will always be the first element of the input dataset.
The stride argument determines the stride of the input elements, and the shift argument determines the shift of the window.
Note that when the window transformation is applied to a dataset of nested elements, it produces a dataset of nested windows.
View source
Returns a new tf.data.Dataset with the given options set.
The options are "global" in the sense they apply to the entire dataset. If options are set multiple times, they are merged as long as different options do not use different non-default values.
View source
Creates a Dataset by zipping together the given datasets.
This method has similar semantics to the built-in zip() function in Python, with the main difference being that the datasets argument can be an arbitrary nested structure of Dataset objects.
View source
View source
Creates an iterator for elements of this dataset.
The returned iterator implements the Python Iterator protocol.
View source
Returns the length of the dataset if it is known and finite.
This method requires that you are running in eager mode, and that the length of the dataset is known and non-infinite. When the length may be unknown or infinite, or if you are running in graph mode, use tf.data.Dataset.cardinality instead.
View source


A reducer is used for reducing a set of elements.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.Reducer
A reducer is represented as a tuple of the three functions:
1) initialization function: key => initial state 2) reduce function: (old state, input) => new state 3) finalization function: state => result


A transformation that resamples a dataset to achieve a target distribution.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.rejection_resample


Samples elements at random from the datasets in datasets.


Saves the content of the given dataset.
The saved dataset is saved in multiple file "shards". By default, the dataset output is divided to shards in a round-robin fashion but custom sharding can be specified via the shard_func function. For example, you can save the dataset to using a single shard as follows:


A transformation that scans a function across an input dataset.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.scan
This transformation is a stateful relative of tf.data.Dataset.map. In addition to mapping scan_func across the elements of the input dataset, scan() accumulates one or more state tensors, whose initial values are initial_state.


API for using the tf.data service.
The tf.data service offers a way to improve training speed when the host attached to a training device can't keep up with the data consumption of the model. For example, suppose a host can generate 100 examples/second, but the model can process 200 examples/second. Training speed could be doubled by using the tf.data service to generate 200 examples/second.
There are a few things to do before using the tf.data service to speed up training.
The tf.data service uses a cluster of workers to prepare data for training your model. The processing_mode argument to tf.data.experimental.service.distribute describes how to leverage multiple workers to process the input dataset. Currently, there are two processing modes to choose from: "distributed_epoch" and "parallel_epochs".
"distributed_epoch" means that the dataset will be split across all tf.data service workers. The dispatcher produces "splits" for the dataset and sends them to workers for further processing. For example, if a dataset begins with a list of filenames, the dispatcher will iterate through the filenames and send the filenames to tf.data workers, which will perform the rest of the dataset transformations on those files. "distributed_epoch" is useful when your model needs to see each element of the dataset exactly once, or if it needs to see the data in a generally-sequential order. "distributed_epoch" only works for datasets with splittable sources, such as Dataset.from_tensor_slices, Dataset.list_files, or Dataset.range.
"parallel_epochs" means that the entire input dataset will be processed independently by each of the tf.data service workers. For this reason, it is important to shuffle data (e.g. filenames) non-deterministically, so that each worker will process the elements of the dataset in a different order. "parallel_epochs" can be used to distribute datasets that aren't splittable.
Before using the tf.data service, it is useful to first measure the potential performance improvement. To do this, add
at the end of your dataset, and see how it affects your model's step time. take(1).cache().repeat() will cache the first element of your dataset and produce it repeatedly. This should make the dataset very fast, so that the model becomes the bottleneck and you can identify the ideal model speed. With enough workers, the tf.data service should be able to achieve similar speed.
tf.data servers should be brought up alongside your training jobs, and brought down when the jobs are finished. The tf.data service uses one DispatchServer and any number of WorkerServers. See https://github.com/tensorflow/ecosystem/tree/master/data_service for an example of using Google Kubernetes Engine (GKE) to manage the tf.data service. The server implementation in tf_std_data_server.py is not GKE-specific, and can be used to run the tf.data service in other contexts.
By default, the tf.data dispatch server stores its state in-memory, making it a single point of failure during training. To avoid this, pass fault_tolerant_mode=True when creating your DispatchServer. Dispatcher fault tolerance requires work_dir to be configured and accessible from the dispatcher both before and after restart (e.g. a GCS path). With fault tolerant mode enabled, the dispatcher will journal its state to the work directory so that no state is lost when the dispatcher is restarted.
WorkerServers may be freely restarted, added, or removed during training. At startup, workers will register with the dispatcher and begin processing all outstanding jobs from the beginning.
Once you have a tf.data service cluster running, take note of the dispatcher IP address and port. To connect to the service, you will use a string in the format "grpc://:".
Below is a toy example that you can run yourself.
See the documentation of tf.data.experimental.service.distribute for more details about using the distribute transformation.
class DispatchServer: An in-process tf.data service dispatch server.
class DispatcherConfig: Configuration class for tf.data service dispatchers.
class WorkerConfig: Configuration class for tf.data service dispatchers.
class WorkerServer: An in-process tf.data service worker server.
distribute(...): A transformation that moves dataset processing to the tf.data service.
from_dataset_id(...): Creates a dataset which reads data from the tf.data service.
register_dataset(...): Registers a dataset with the tf.data service.


Configuration class for tf.data service dispatchers.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.service.DispatcherConfig


An in-process tf.data service dispatch server.
A tf.data.experimental.service.DispatchServer coordinates a cluster of tf.data.experimental.service.WorkerServers. When the workers start, they register themselves with the dispatcher.
When starting a dedicated tf.data dispatch process, use join() to block indefinitely after starting up the server.
To start a DispatchServer in fault-tolerant mode, set work_dir and fault_tolerant_mode like below:
The returned string will be in the form protocol://address, e.g. "grpc://localhost:5050".
View source
Blocks until the server has shut down.
This is useful when starting a dedicated dispatch process.
View source
Starts this server.


A transformation that moves dataset processing to the tf.data service.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.service.distribute
When you iterate over a dataset containing the distribute transformation, the tf.data service creates a "job" which produces data for the dataset iteration.
The tf.data service uses a cluster of workers to prepare data for training your model. The processing_mode argument to tf.data.experimental.service.distribute describes how to leverage multiple workers to process the input dataset. Currently, there are two processing modes to choose from: "distributed_epoch" and "parallel_epochs".
"distributed_epoch" means that the dataset will be split across all tf.data service workers. The dispatcher produces "splits" for the dataset and sends them to workers for further processing. For example, if a dataset begins with a list of filenames, the dispatcher will iterate through the filenames and send the filenames to tf.data workers, which will perform the rest of the dataset transformations on those files. "distributed_epoch" is useful when your model needs to see each element of the dataset exactly once, or if it needs to see the data in a generally-sequential order. "distributed_epoch" only works for datasets with splittable sources, such as Dataset.from_tensor_slices, Dataset.list_files, or Dataset.range.
"parallel_epochs" means that the entire input dataset will be processed independently by each of the tf.data service workers. For this reason, it is important to shuffle data (e.g. filenames) non-deterministically, so that each worker will process the elements of the dataset in a different order. "parallel_epochs" can be used to distribute datasets that aren't splittable.
With two workers, "parallel_epochs" will produce every element of the dataset twice:
"distributed_epoch", on the other hand, will still produce each element once:
When using apply(tf.data.experimental.service.distribute(...)), the dataset before the apply transformation executes within the tf.data service, while the operations after apply happen within the local process.
In the above example, the dataset operations (before applying the distribute function on the elements) will be executed on the tf.data workers, and the elements are provided over RPC. The remaining transformations (after the call to distribute) will be executed locally. The dispatcher and the workers will bind to usused free ports (which are chosen at random), in order to communicate with each other. However, to bind them to specific ports, the port parameter can be passed.
The job_name argument allows jobs to be shared across multiple datasets. Instead of each dataset creating its own job, all datasets with the same job_name will consume from the same job. A new job will be created for each iteration of the dataset (with each repetition of Dataset.repeat counting as a new iteration). Suppose the DispatchServer is serving on localhost:5000 and two training workers (in either a single client or multi-client setup) iterate over the below dataset, and there is a single tf.data worker:
The elements of each job will be split between the two processes, with elements being consumed by the processes on a first-come first-served basis. One possible result is that process 1 prints
and process 2 prints
Job names must not be re-used across different training jobs within the lifetime of the tf.data service. In general, the tf.data service is expected to live for the duration of a single training job. To use the tf.data service with multiple training jobs, make sure to use different job names to avoid conflicts. For example, suppose a training job calls distribute with job_name="job" and reads until end of input. If another independent job connects to the same tf.data service and tries to read from job_name="job", it will immediately receive end of input, without getting any data.
Keras and Distribution Strategies
The dataset produced by the distribute transformation can be passed to Keras' Model.fit or Distribution Strategy's tf.distribute.Strategy.experimental_distribute_dataset like any other tf.data.Dataset. We recommend setting a job_name on the call to distribute so that if there are multiple workers, they read data from the same job. Note that the autosharding normally performed by experimental_distribute_dataset will be disabled when setting a job_name, since sharing the job already results in splitting data across the workers. When using a shared job, data will be dynamically balanced across workers, so that they reach end of input about the same time. This results in better worker utilization than with autosharding, where each worker processes an independent set of files, and some workers may run out of data earlier than others.


Creates a dataset which reads data from the tf.data service.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.service.from_dataset_id
This is useful when the dataset is registered by one process, then used in another process. When the same process is both registering and reading from the dataset, it is simpler to use tf.data.experimental.service.distribute instead.
Before using from_dataset_id, the dataset must have been registered with the tf.data service using tf.data.experimental.service.register_dataset. register_dataset returns a dataset id for the registered dataset. That is the dataset_id which should be passed to from_dataset_id.
The element_spec argument indicates the tf.TypeSpecs for the elements produced by the dataset. Currently element_spec must be explicitly specified, and match the dataset registered under dataset_id. element_spec defaults to None so that in the future we can support automatically discovering the element_spec by querying the tf.data service.
tf.data.experimental.service.distribute is a convenience method which combines register_dataset and from_dataset_id into a dataset transformation. See the documentation for tf.data.experimental.service.distribute for more detail about how from_dataset_id works.


Registers a dataset with the tf.data service.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.service.register_dataset
register_dataset registers a dataset with the tf.data service so that datasets can be created later with tf.data.experimental.service.from_dataset_id. This is useful when the dataset is registered by one process, then used in another process. When the same process is both registering and reading from the dataset, it is simpler to use tf.data.experimental.service.distribute instead.
If the dataset is already registered with the tf.data service, register_dataset returns the already-registered dataset's id.


Configuration class for tf.data service dispatchers.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.service.WorkerConfig


An in-process tf.data service worker server.
A tf.data.experimental.service.WorkerServer performs tf.data.Dataset processing for user-defined datasets, and provides the resulting elements over RPC. A worker is associated with a single tf.data.experimental.service.DispatchServer.
When starting a dedicated tf.data worker process, use join() to block indefinitely after starting up the server.
View source
Blocks until the server has shut down.
This is useful when starting a dedicated worker process.
This method currently blocks forever.
View source
Starts this server.


Shuffles and repeats a Dataset, reshuffling with each repetition. (deprecated)
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.shuffle_and_repeat
produces the same output as
In each repetition, this dataset fills a buffer with buffer_size elements, then randomly samples elements from this buffer, replacing the selected elements with new elements. For perfect shuffling, set the buffer size equal to the full size of the dataset.
For instance, if your dataset contains 10,000 elements but buffer_size is set to 1,000, then shuffle will initially select a random element from only the first 1,000 elements in the buffer. Once an element is selected, its space in the buffer is replaced by the next (i.e. 1,001-st) element, maintaining the 1,000 element buffer.


API to persist the output of the input dataset.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.snapshot
The snapshot API allows users to transparently persist the output of their preprocessing pipeline to disk, and materialize the pre-processed data on a different training run.
This API enables repeated preprocessing steps to be consolidated, and allows re-use of already processed data, trading off disk storage and network bandwidth for freeing up more valuable CPU resources and accelerator compute time.
https://github.com/tensorflow/community/blob/master/rfcs/20200107-tf-data-snapshot.md has detailed design documentation of this feature.
Users can specify various options to control the behavior of snapshot, including how snapshots are read from and written to by passing in user-defined functions to the reader_func and shard_func parameters.
shard_func is a user specified function that maps input elements to snapshot shards.
Users may want to specify this function to control how snapshot files should be written to disk. Below is an example of how a potential shard_func could be written.
reader_func is a user specified function that accepts a single argument: (1) a Dataset of Datasets, each representing a "split" of elements of the original dataset. The cardinality of the input dataset matches the number of the shards specified in the shard_func (see above). The function should return a Dataset of elements of the original dataset.
Users may want specify this function to control how snapshot files should be read from disk, including the amount of shuffling and parallelism.
Here is an example of a standard reader function a user can define. This function enables both dataset shuffling and parallel reading of datasets:
By default, snapshot parallelizes reads by the number of cores available on the system, but will not attempt to shuffle the data.


A Dataset consisting of the results from a SQL query.
Inherits From: Dataset
View source
Applies a transformation function to this dataset.
apply enables chaining of custom Dataset transformations, which are represented as functions that take one Dataset argument and return a transformed Dataset.
View source
Returns an iterator which converts all elements of the dataset to numpy.
Use as_numpy_iterator to inspect the content of your dataset. To see element shapes and types, print dataset elements directly instead of using as_numpy_iterator.
This method requires that you are running in eager mode and the dataset's element_spec contains only TensorSpec components.
as_numpy_iterator() will preserve the nested structure of dataset elements.
View source
Combines consecutive elements of this dataset into batches.
The components of the resulting element will have an additional outer dimension, which will be batch_size (or N % batch_size for the last element if batch_size does not divide the number of input elements N evenly and drop_remainder is False). If your program depends on the batches having the same outer dimension, you should set the drop_remainder argument to True to prevent the smaller batch from being produced.
View source
Caches the elements in this dataset.
The first time the dataset is iterated over, its elements will be cached either in the specified file or in memory. Subsequent iterations will use the cached data.
When caching to a file, the cached data will persist across runs. Even the first iteration through the data will read from the cache file. Changing the input pipeline before the call to .cache() will have no effect until the cache file is removed or the filename is changed.
View source
Returns the cardinality of the dataset, if known.
cardinality may return tf.data.INFINITE_CARDINALITY if the dataset contains an infinite number of elements or tf.data.UNKNOWN_CARDINALITY if the analysis fails to determine the number of elements in the dataset (e.g. when the dataset source is a file).
View source
Creates a Dataset by concatenating the given dataset with this dataset.
View source
Enumerates the elements of this dataset.
It is similar to python's enumerate.
View source
Filters this dataset according to predicate.
View source
Maps map_func across this dataset and flattens the result.
Use flat_map if you want to make sure that the order of your dataset stays the same. For example, to flatten a dataset of batches into a dataset of their elements:
tf.data.Dataset.interleave() is a generalization of flat_map, since flat_map produces the same output as tf.data.Dataset.interleave(cycle_length=1)
View source
Creates a Dataset whose elements are generated by generator. (deprecated arguments)
The generator argument must be a callable object that returns an object that supports the iter() protocol (e.g. a generator function).
The elements generated by generator must be compatible with either the given output_signature argument or with the given output_types and (optionally) output_shapes arguments, whichiver was specified.
The recommended way to call from_generator is to use the output_signature argument. In this case the output will be assumed to consist of objects with the classes, shapes and types defined by tf.TypeSpec objects from output_signature argument:
There is also a deprecated way to call from_generator by either with output_types argument alone or together with output_shapes argument. In this case the output of the function will be assumed to consist of tf.Tensor objects with with the types defined by output_types and with the shapes which are either unknown or defined by output_shapes.
View source
Creates a Dataset whose elements are slices of the given tensors.
The given tensors are sliced along their first dimension. This operation preserves the structure of the input tensors, removing the first dimension of each tensor and using it as the dataset dimension. All input tensors must have the same size in their first dimensions.
Note that if tensors contains a NumPy array, and eager execution is not enabled, the values will be embedded in the graph as one or more tf.constant operations. For large datasets (> 1 GB), this can waste memory and run into byte limits of graph serialization. If tensors contains one or more large NumPy arrays, consider the alternative described in this guide.
View source
Creates a Dataset with a single element, comprising the given tensors.
from_tensors produces a dataset containing only a single element. To slice the input tensor into multiple elements, use from_tensor_slices instead.
Note that if tensors contains a NumPy array, and eager execution is not enabled, the values will be embedded in the graph as one or more tf.constant operations. For large datasets (> 1 GB), this can waste memory and run into byte limits of graph serialization. If tensors contains one or more large NumPy arrays, consider the alternative described in this guide.
View source
Maps map_func across this dataset, and interleaves the results.
For example, you can use Dataset.interleave() to process many input files concurrently:
The cycle_length and block_length arguments control the order in which elements are produced. cycle_length controls the number of input elements that are processed concurrently. If you set cycle_length to 1, this transformation will handle one input element at a time, and will produce identical results to tf.data.Dataset.flat_map. In general, this transformation will apply map_func to cycle_length input elements, open iterators on the returned Dataset objects, and cycle through them producing block_length consecutive elements from each iterator, and consuming the next input element each time it reaches the end of an iterator.
Performance can often be improved by setting num_parallel_calls so that interleave will use multiple threads to fetch elements. If determinism isn't required, it can also improve performance to set deterministic=False.
View source
A dataset of all files matching one or more glob patterns.
The file_pattern argument should be a small number of glob patterns. If your filenames have already been globbed, use Dataset.from_tensor_slices(filenames) instead, as re-globbing every filename with list_files may result in poor performance with remote storage systems.
If we had the following files on our filesystem:
If we pass "/path/to/dir/*.py" as the directory, the dataset would produce:
View source
Maps map_func across the elements of this dataset.
This transformation applies map_func to each element of this dataset, and returns a new dataset containing the transformed elements, in the same order as they appeared in the input. map_func can be used to change both the values and the structure of a dataset's elements. For example, adding 1 to each element, or projecting a subset of element components.
The input signature of map_func is determined by the structure of each element in this dataset.
The value or values returned by map_func determine the structure of each element in the returned dataset.
map_func can accept as arguments and return any type of dataset element.
Note that irrespective of the context in which map_func is defined (eager vs. graph), tf.data traces the function and executes it as a graph. To use Python code inside of the function you have a few options:
1) Rely on AutoGraph to convert Python code into an equivalent graph computation. The downside of this approach is that AutoGraph can convert some but not all Python code.
2) Use tf.py_function, which allows you to write arbitrary Python code but will generally result in worse performance than 1). For example:
3) Use tf.numpy_function, which also allows you to write arbitrary Python code. Note that tf.py_function accepts tf.Tensor whereas tf.numpy_function accepts numpy arrays and returns only numpy arrays. For example:
Note that the use of tf.numpy_function and tf.py_function in general precludes the possibility of executing user-defined transformations in parallel (because of Python GIL).
Performance can often be improved by setting num_parallel_calls so that map will use multiple threads to process elements. If deterministic order isn't required, it can also improve performance to set deterministic=False.
View source
Returns the options for this dataset and its inputs.
View source
Combines consecutive elements of this dataset into padded batches.
This transformation combines multiple consecutive elements of the input dataset into a single element.
Like tf.data.Dataset.batch, the components of the resulting element will have an additional outer dimension, which will be batch_size (or N % batch_size for the last element if batch_size does not divide the number of input elements N evenly and drop_remainder is False). If your program depends on the batches having the same outer dimension, you should set the drop_remainder argument to True to prevent the smaller batch from being produced.
Unlike tf.data.Dataset.batch, the input elements to be batched may have different shapes, and this transformation will pad each component to the respective shape in padded_shapes. The padded_shapes argument determines the resulting shape for each dimension of each component in an output element:
See also tf.data.experimental.dense_to_sparse_batch, which combines elements that may have different shapes into a tf.sparse.SparseTensor.
View source
Creates a Dataset that prefetches elements from this dataset.
Most dataset input pipelines should end with a call to prefetch. This allows later elements to be prepared while the current element is being processed. This often improves latency and throughput, at the cost of using additional memory to store prefetched elements.
View source
Creates a Dataset of a step-separated range of values.
View source
Reduces the input dataset to a single element.
The transformation calls reduce_func successively on every element of the input dataset until the dataset is exhausted, aggregating information in its internal state. The initial_state argument is used for the initial state and the final state is returned as the result.
View source
Repeats this dataset so each original value is seen count times.
View source
Creates a Dataset that includes only 1/num_shards of this dataset.
shard is deterministic. The Dataset produced by A.shard(n, i) will contain all elements of A whose index mod n = i.
This dataset operator is very useful when running distributed training, as it allows each worker to read a unique subset.
When reading a single input file, you can shard elements as follows:
View source
Randomly shuffles the elements of this dataset.
This dataset fills a buffer with buffer_size elements, then randomly samples elements from this buffer, replacing the selected elements with new elements. For perfect shuffling, a buffer size greater than or equal to the full size of the dataset is required.
For instance, if your dataset contains 10,000 elements but buffer_size is set to 1,000, then shuffle will initially select a random element from only the first 1,000 elements in the buffer. Once an element is selected, its space in the buffer is replaced by the next (i.e. 1,001-st) element, maintaining the 1,000 element buffer.
reshuffle_each_iteration controls whether the shuffle order should be different for each epoch. In TF 1.X, the idiomatic way to create epochs was through the repeat transformation:
In TF 2.0, tf.data.Dataset objects are Python iterables which makes it possible to also create epochs through Python iteration:
View source
Creates a Dataset that skips count elements from this dataset.
View source
Creates a Dataset with at most count elements from this dataset.
View source
Splits elements of a dataset into multiple elements.
For example, if elements of the dataset are shaped [B, a0, a1, ...], where B may vary for each input element, then for each element in the dataset, the unbatched dataset will contain B consecutive elements of shape [a0, a1, ...].
View source
Combines (nests of) input elements into a dataset of (nests of) windows.
A "window" is a finite dataset of flat elements of size size (or possibly fewer if there are not enough input elements to fill the window and drop_remainder evaluates to False).
The shift argument determines the number of input elements by which the window moves on each iteration. If windows and elements are both numbered starting at 0, the first element in window k will be element k * shift of the input dataset. In particular, the first element of the first window will always be the first element of the input dataset.
The stride argument determines the stride of the input elements, and the shift argument determines the shift of the window.
Note that when the window transformation is applied to a dataset of nested elements, it produces a dataset of nested windows.
View source
Returns a new tf.data.Dataset with the given options set.
The options are "global" in the sense they apply to the entire dataset. If options are set multiple times, they are merged as long as different options do not use different non-default values.
View source
Creates a Dataset by zipping together the given datasets.
This method has similar semantics to the built-in zip() function in Python, with the main difference being that the datasets argument can be an arbitrary nested structure of Dataset objects.
View source
View source
Creates an iterator for elements of this dataset.
The returned iterator implements the Python Iterator protocol.
View source
Returns the length of the dataset if it is known and finite.
This method requires that you are running in eager mode, and that the length of the dataset is known and non-infinite. When the length may be unknown or infinite, or if you are running in graph mode, use tf.data.Dataset.cardinality instead.
View source


A stateful resource that aggregates statistics from one or more iterators.
To record statistics, use one of the custom transformation functions defined in this module when defining your tf.data.Dataset. All statistics will be aggregated by the StatsAggregator that is associated with a particular iterator (see below). For example, to record the latency of producing each element by iterating over a dataset:
To associate a StatsAggregator with a tf.data.Dataset object, use the following pattern:


Represents options for collecting dataset stats using StatsAggregator.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.StatsOptions
You can set the stats options of a dataset through the experimental_stats property of tf.data.Options; the property is an instance of tf.data.experimental.StatsOptions. For example, to collect latency stats on all dataset edges, use the following pattern:
View source
Return self==value.
View source
Return self!=value.


A transformation that stops dataset iteration based on a predicate.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.take_while


Writes a dataset to a TFRecord file.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.TFRecordWriter
The elements of the dataset must be scalar strings. To serialize dataset elements as strings, you can use the tf.io.serialize_tensor function.
To read back the elements, use TFRecordDataset.
To shard a dataset across multiple TFRecord files:
View source
Writes a dataset to a TFRecord file.
An operation that writes the content of the specified dataset to the file specified in the constructor.
If the file exists, it will be overwritten.
Raises TypeError: if dataset is not a tf.data.Dataset. TypeError: if the elements produced by the dataset are not scalar strings.


Represents options for dataset threading.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.ThreadingOptions
You can set the threading options of a dataset through the experimental_threading property of tf.data.Options; the property is an instance of tf.data.experimental.ThreadingOptions.
View source
Return self==value.
View source
Return self!=value.


Returns a variant representing the given dataset.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.to_variant


Splits elements of a dataset into multiple elements on the batch dimension. (deprecated)
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.unbatch
For example, if elements of the dataset are shaped [B, a0, a1, ...], where B may vary for each input element, then for each element in the dataset, the unbatched dataset will contain B consecutive elements of shape [a0, a1, ...].


Creates a Dataset from another Dataset, discarding duplicates.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.experimental.unique
Use this transformation to produce a dataset that contains one instance of each unique element in the input. For example:


A Dataset of fixed-length records from one or more binary files.
Inherits From: Dataset
View source
Applies a transformation function to this dataset.
apply enables chaining of custom Dataset transformations, which are represented as functions that take one Dataset argument and return a transformed Dataset.
View source
Returns an iterator which converts all elements of the dataset to numpy.
Use as_numpy_iterator to inspect the content of your dataset. To see element shapes and types, print dataset elements directly instead of using as_numpy_iterator.
This method requires that you are running in eager mode and the dataset's element_spec contains only TensorSpec components.
as_numpy_iterator() will preserve the nested structure of dataset elements.
View source
Combines consecutive elements of this dataset into batches.
The components of the resulting element will have an additional outer dimension, which will be batch_size (or N % batch_size for the last element if batch_size does not divide the number of input elements N evenly and drop_remainder is False). If your program depends on the batches having the same outer dimension, you should set the drop_remainder argument to True to prevent the smaller batch from being produced.
View source
Caches the elements in this dataset.
The first time the dataset is iterated over, its elements will be cached either in the specified file or in memory. Subsequent iterations will use the cached data.
When caching to a file, the cached data will persist across runs. Even the first iteration through the data will read from the cache file. Changing the input pipeline before the call to .cache() will have no effect until the cache file is removed or the filename is changed.
View source
Returns the cardinality of the dataset, if known.
cardinality may return tf.data.INFINITE_CARDINALITY if the dataset contains an infinite number of elements or tf.data.UNKNOWN_CARDINALITY if the analysis fails to determine the number of elements in the dataset (e.g. when the dataset source is a file).
View source
Creates a Dataset by concatenating the given dataset with this dataset.
View source
Enumerates the elements of this dataset.
It is similar to python's enumerate.
View source
Filters this dataset according to predicate.
View source
Maps map_func across this dataset and flattens the result.
Use flat_map if you want to make sure that the order of your dataset stays the same. For example, to flatten a dataset of batches into a dataset of their elements:
tf.data.Dataset.interleave() is a generalization of flat_map, since flat_map produces the same output as tf.data.Dataset.interleave(cycle_length=1)
View source
Creates a Dataset whose elements are generated by generator. (deprecated arguments)
The generator argument must be a callable object that returns an object that supports the iter() protocol (e.g. a generator function).
The elements generated by generator must be compatible with either the given output_signature argument or with the given output_types and (optionally) output_shapes arguments, whichiver was specified.
The recommended way to call from_generator is to use the output_signature argument. In this case the output will be assumed to consist of objects with the classes, shapes and types defined by tf.TypeSpec objects from output_signature argument:
There is also a deprecated way to call from_generator by either with output_types argument alone or together with output_shapes argument. In this case the output of the function will be assumed to consist of tf.Tensor objects with with the types defined by output_types and with the shapes which are either unknown or defined by output_shapes.
View source
Creates a Dataset whose elements are slices of the given tensors.
The given tensors are sliced along their first dimension. This operation preserves the structure of the input tensors, removing the first dimension of each tensor and using it as the dataset dimension. All input tensors must have the same size in their first dimensions.
Note that if tensors contains a NumPy array, and eager execution is not enabled, the values will be embedded in the graph as one or more tf.constant operations. For large datasets (> 1 GB), this can waste memory and run into byte limits of graph serialization. If tensors contains one or more large NumPy arrays, consider the alternative described in this guide.
View source
Creates a Dataset with a single element, comprising the given tensors.
from_tensors produces a dataset containing only a single element. To slice the input tensor into multiple elements, use from_tensor_slices instead.
Note that if tensors contains a NumPy array, and eager execution is not enabled, the values will be embedded in the graph as one or more tf.constant operations. For large datasets (> 1 GB), this can waste memory and run into byte limits of graph serialization. If tensors contains one or more large NumPy arrays, consider the alternative described in this guide.
View source
Maps map_func across this dataset, and interleaves the results.
For example, you can use Dataset.interleave() to process many input files concurrently:
The cycle_length and block_length arguments control the order in which elements are produced. cycle_length controls the number of input elements that are processed concurrently. If you set cycle_length to 1, this transformation will handle one input element at a time, and will produce identical results to tf.data.Dataset.flat_map. In general, this transformation will apply map_func to cycle_length input elements, open iterators on the returned Dataset objects, and cycle through them producing block_length consecutive elements from each iterator, and consuming the next input element each time it reaches the end of an iterator.
Performance can often be improved by setting num_parallel_calls so that interleave will use multiple threads to fetch elements. If determinism isn't required, it can also improve performance to set deterministic=False.
View source
A dataset of all files matching one or more glob patterns.
The file_pattern argument should be a small number of glob patterns. If your filenames have already been globbed, use Dataset.from_tensor_slices(filenames) instead, as re-globbing every filename with list_files may result in poor performance with remote storage systems.
If we had the following files on our filesystem:
If we pass "/path/to/dir/*.py" as the directory, the dataset would produce:
View source
Maps map_func across the elements of this dataset.
This transformation applies map_func to each element of this dataset, and returns a new dataset containing the transformed elements, in the same order as they appeared in the input. map_func can be used to change both the values and the structure of a dataset's elements. For example, adding 1 to each element, or projecting a subset of element components.
The input signature of map_func is determined by the structure of each element in this dataset.
The value or values returned by map_func determine the structure of each element in the returned dataset.
map_func can accept as arguments and return any type of dataset element.
Note that irrespective of the context in which map_func is defined (eager vs. graph), tf.data traces the function and executes it as a graph. To use Python code inside of the function you have a few options:
1) Rely on AutoGraph to convert Python code into an equivalent graph computation. The downside of this approach is that AutoGraph can convert some but not all Python code.
2) Use tf.py_function, which allows you to write arbitrary Python code but will generally result in worse performance than 1). For example:
3) Use tf.numpy_function, which also allows you to write arbitrary Python code. Note that tf.py_function accepts tf.Tensor whereas tf.numpy_function accepts numpy arrays and returns only numpy arrays. For example:
Note that the use of tf.numpy_function and tf.py_function in general precludes the possibility of executing user-defined transformations in parallel (because of Python GIL).
Performance can often be improved by setting num_parallel_calls so that map will use multiple threads to process elements. If deterministic order isn't required, it can also improve performance to set deterministic=False.
View source
Returns the options for this dataset and its inputs.
View source
Combines consecutive elements of this dataset into padded batches.
This transformation combines multiple consecutive elements of the input dataset into a single element.
Like tf.data.Dataset.batch, the components of the resulting element will have an additional outer dimension, which will be batch_size (or N % batch_size for the last element if batch_size does not divide the number of input elements N evenly and drop_remainder is False). If your program depends on the batches having the same outer dimension, you should set the drop_remainder argument to True to prevent the smaller batch from being produced.
Unlike tf.data.Dataset.batch, the input elements to be batched may have different shapes, and this transformation will pad each component to the respective shape in padded_shapes. The padded_shapes argument determines the resulting shape for each dimension of each component in an output element:
See also tf.data.experimental.dense_to_sparse_batch, which combines elements that may have different shapes into a tf.sparse.SparseTensor.
View source
Creates a Dataset that prefetches elements from this dataset.
Most dataset input pipelines should end with a call to prefetch. This allows later elements to be prepared while the current element is being processed. This often improves latency and throughput, at the cost of using additional memory to store prefetched elements.
View source
Creates a Dataset of a step-separated range of values.
View source
Reduces the input dataset to a single element.
The transformation calls reduce_func successively on every element of the input dataset until the dataset is exhausted, aggregating information in its internal state. The initial_state argument is used for the initial state and the final state is returned as the result.
View source
Repeats this dataset so each original value is seen count times.
View source
Creates a Dataset that includes only 1/num_shards of this dataset.
shard is deterministic. The Dataset produced by A.shard(n, i) will contain all elements of A whose index mod n = i.
This dataset operator is very useful when running distributed training, as it allows each worker to read a unique subset.
When reading a single input file, you can shard elements as follows:
View source
Randomly shuffles the elements of this dataset.
This dataset fills a buffer with buffer_size elements, then randomly samples elements from this buffer, replacing the selected elements with new elements. For perfect shuffling, a buffer size greater than or equal to the full size of the dataset is required.
For instance, if your dataset contains 10,000 elements but buffer_size is set to 1,000, then shuffle will initially select a random element from only the first 1,000 elements in the buffer. Once an element is selected, its space in the buffer is replaced by the next (i.e. 1,001-st) element, maintaining the 1,000 element buffer.
reshuffle_each_iteration controls whether the shuffle order should be different for each epoch. In TF 1.X, the idiomatic way to create epochs was through the repeat transformation:
In TF 2.0, tf.data.Dataset objects are Python iterables which makes it possible to also create epochs through Python iteration:
View source
Creates a Dataset that skips count elements from this dataset.
View source
Creates a Dataset with at most count elements from this dataset.
View source
Splits elements of a dataset into multiple elements.
For example, if elements of the dataset are shaped [B, a0, a1, ...], where B may vary for each input element, then for each element in the dataset, the unbatched dataset will contain B consecutive elements of shape [a0, a1, ...].
View source
Combines (nests of) input elements into a dataset of (nests of) windows.
A "window" is a finite dataset of flat elements of size size (or possibly fewer if there are not enough input elements to fill the window and drop_remainder evaluates to False).
The shift argument determines the number of input elements by which the window moves on each iteration. If windows and elements are both numbered starting at 0, the first element in window k will be element k * shift of the input dataset. In particular, the first element of the first window will always be the first element of the input dataset.
The stride argument determines the stride of the input elements, and the shift argument determines the shift of the window.
Note that when the window transformation is applied to a dataset of nested elements, it produces a dataset of nested windows.
View source
Returns a new tf.data.Dataset with the given options set.
The options are "global" in the sense they apply to the entire dataset. If options are set multiple times, they are merged as long as different options do not use different non-default values.
View source
Creates a Dataset by zipping together the given datasets.
This method has similar semantics to the built-in zip() function in Python, with the main difference being that the datasets argument can be an arbitrary nested structure of Dataset objects.
View source
View source
Creates an iterator for elements of this dataset.
The returned iterator implements the Python Iterator protocol.
View source
Returns the length of the dataset if it is known and finite.
This method requires that you are running in eager mode, and that the length of the dataset is known and non-infinite. When the length may be unknown or infinite, or if you are running in graph mode, use tf.data.Dataset.cardinality instead.
View source


Represents an iterator of a tf.data.Dataset.
tf.data.Iterator is the primary mechanism for enumerating elements of a tf.data.Dataset. It supports the Python Iterator protocol, which means it can be iterated over using a for-loop:
or by fetching individual elements explicitly via get_next():
In addition, non-raising iteration is supported via get_next_as_optional(), which returns the next element (if available) wrapped in a tf.experimental.Optional.
View source
Returns a nested structure of tf.Tensors containing the next element.
View source
Returns a tf.experimental.Optional which contains the next element.
If the iterator has reached the end of the sequence, the returned tf.experimental.Optional will have no value.


Type specification for tf.data.Iterator.
Inherits From: TypeSpec
For instance, tf.data.IteratorSpec can be used to define a tf.function that takes tf.data.Iterator as an input argument:
In particular, all values that are compatible with this TypeSpec must be an instance of this type.
View source
View source
Returns true if spec_or_value is compatible with this TypeSpec.
View source
Returns the most specific TypeSpec compatible with self and other.
View source
Return self==value.
View source
Return self!=value.


Represents options for tf.data.Dataset.
Compat aliases for migration
See Migration guide for more details.
tf.compat.v1.data.Options
A tf.data.Options object can be, for instance, used to control which static optimizations to apply to the input pipeline graph or whether to use performance modeling to dynamically tune the parallelism of operations such as tf.data.Dataset.map or tf.data.Dataset.interleave.
The options are set for the entire dataset and are carried over to datasets created through tf.data transformations.
The options can be set either by mutating the object returned by tf.data.Dataset.options() or by constructing an Options object and using the tf.data.Dataset.with_options(options) transformation, which returns a dataset with the options set.
View source
Merges itself with the given tf.data.Options.
If this object and the options to merge set an option differently, a warning is generated and this object's value is updated with the options object's value.
View source
Return self==value.
View source
Return self!=value.