neulab/docprompting-conala huggingface dataset:
tf.data.experimental.save Saves the content of the given dataset. 
tf.data.experimental.save(
    dataset, path, compression=None, shard_func=None
)
 Example usage: 
import tempfile
path = os.path.join(tempfile.gettempdir(), "saved_data")
# Save a dataset
dataset = tf.data.Dataset.range(2)
tf.data.experimental.save(dataset, path)
new_dataset = tf.data.experimental.load(path,
    tf.TensorSpec(shape=(), dtype=tf.int64))
for elem in new_dataset:
  print(elem)
tf.Tensor(0, shape=(), dtype=int64)
tf.Tensor(1, shape=(), dtype=int64)
 The saved dataset is saved in multiple file "shards". By default, the dataset output is divided to shards in a round-robin fashion but custom sharding can be specified via the shard_func function. For example, you can save the dataset to using a single shard as follows: dataset = make_dataset()
def custom_shard_func(element):
  return 0
dataset = tf.data.experimental.save(
    path="/path/to/data", ..., shard_func=custom_shard_func)


Note: The directory layout and file format used for saving the dataset is considered an implementation detail and may change. For this reason, datasets saved through tf.data.experimental.save should only be consumed through tf.data.experimental.load, which is guaranteed to be backwards compatible.

 


 Args
  dataset   The dataset to save.  
  path   Required. A directory to use for saving the dataset.  
  compression   Optional. The algorithm to use to compress data when writing it. Supported options are GZIP and NONE. Defaults to NONE.  
  shard_func   Optional. A function to control the mapping of dataset elements to file shards. The function is expected to map elements of the input dataset to int64 shard IDs. If present, the function will be traced and executed as graph computation.
tf.data.experimental.scan     View source on GitHub    A transformation that scans a function across an input dataset.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.scan  
tf.data.experimental.scan(
    initial_state, scan_func
)
 This transformation is a stateful relative of tf.data.Dataset.map. In addition to mapping scan_func across the elements of the input dataset, scan() accumulates one or more state tensors, whose initial values are initial_state.
 


 Args
  initial_state   A nested structure of tensors, representing the initial state of the accumulator.  
  scan_func   A function that maps (old_state, input_element) to (new_state, output_element). It must take two arguments and return a pair of nested structures of tensors. The new_state must match the structure of initial_state.   
 


 Returns   A Dataset transformation function, which can be passed to tf.data.Dataset.apply.
tf.data.experimental.service.DispatcherConfig Configuration class for tf.data service dispatchers.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.service.DispatcherConfig  
tf.data.experimental.service.DispatcherConfig(
    port=0, protocol='grpc', work_dir=None, fault_tolerant_mode=False,
    job_gc_check_interval_ms=None, job_gc_timeout_ms=None
)
 Fields:  
port: Specifies the port to bind to. A value of 0 indicates that the server may bind to any available port. 
protocol: The protocol to use for communicating with the tf.data service. Defaults to "grpc". 
work_dir: A directory to store dispatcher state in. This argument is required for the dispatcher to be able to recover from restarts. 
fault_tolerant_mode: Whether the dispatcher should write its state to a journal so that it can recover from restarts. Dispatcher state, including registered datasets and created jobs, is synchronously written to the journal before responding to RPCs. If True, work_dir must also be specified. 
job_gc_check_interval_ms: How often the dispatcher should scan through to delete old and unused jobs, in milliseconds. If not set, the runtime will select a reasonable default. A higher value will reduce load on the dispatcher, while a lower value will reduce the time it takes for the dispatcher to garbage collect expired jobs. 
job_gc_timeout_ms: How long a job needs to be unused before it becomes a candidate for garbage collection, in milliseconds. If not set, the runtime will select a reasonable default. A higher value will cause jobs to stay around longer with no consumers. This is useful if there is a large gap in time between when consumers read from the job. A lower value will reduce the time it takes to reclaim the resources from expired jobs. 
 


 Attributes
  port  
 
  protocol  
 
  work_dir  
 
  fault_tolerant_mode  
 
  job_gc_check_interval_ms  
 
  job_gc_timeout_ms
tf.data.experimental.service.DispatchServer An in-process tf.data service dispatch server. 
tf.data.experimental.service.DispatchServer(
    config=None, start=True
)
 A tf.data.experimental.service.DispatchServer coordinates a cluster of tf.data.experimental.service.WorkerServers. When the workers start, they register themselves with the dispatcher. 
dispatcher = tf.data.experimental.service.DispatchServer()
dispatcher_address = dispatcher.target.split("://")[1]
worker = tf.data.experimental.service.WorkerServer(WorkerConfig(
    dispatcher_address=dispatcher_address))
dataset = tf.data.Dataset.range(10)
dataset = dataset.apply(tf.data.experimental.service.distribute(
    processing_mode="parallel_epochs", service=dispatcher.target))
print(list(dataset.as_numpy_iterator()))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
 When starting a dedicated tf.data dispatch process, use join() to block indefinitely after starting up the server. dispatcher = tf.data.experimental.service.DispatchServer(
    tf.data.experimental.service.DispatcherConfig(port=5050))
dispatcher.join()
 To start a DispatchServer in fault-tolerant mode, set work_dir and fault_tolerant_mode like below: dispatcher = tf.data.experimental.service.DispatchServer(
    tf.data.experimental.service.DispatcherConfig(
        port=5050,
        work_dir="gs://my-bucket/dispatcher/work_dir",
        fault_tolerant_mode=True))

 


 Args
  config   (Optional.) A tf.data.experimental.service.DispatcherConfig configration. If None, the dispatcher will use default configuration values.  
  start   (Optional.) Boolean, indicating whether to start the server after creating it. Defaults to True.   
 


 Attributes
  target   Returns a target that can be used to connect to the server. 
dispatcher = tf.data.experimental.service.DispatchServer()
dataset = tf.data.Dataset.range(10)
dataset = dataset.apply(tf.data.experimental.service.distribute(
    processing_mode="parallel_epochs", service=dispatcher.target))
 The returned string will be in the form protocol://address, e.g. "grpc://localhost:5050". 
   Methods join View source 
join()
 Blocks until the server has shut down. This is useful when starting a dedicated dispatch process. dispatcher = tf.data.experimental.service.DispatchServer(
    tf.data.experimental.service.DispatcherConfig(port=5050))
dispatcher.join()

 


 Raises
  tf.errors.OpError   Or one of its subclasses if an error occurs while joining the server.    start View source 
start()
 Starts this server. 
dispatcher = tf.data.experimental.service.DispatchServer(start=False)
dispatcher.start()

 


 Raises
  tf.errors.OpError   Or one of its subclasses if an error occurs while starting the server.
tf.data.experimental.service.register_dataset Registers a dataset with the tf.data service.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.service.register_dataset  
tf.data.experimental.service.register_dataset(
    service, dataset
)
 register_dataset registers a dataset with the tf.data service so that datasets can be created later with tf.data.experimental.service.from_dataset_id. This is useful when the dataset is registered by one process, then used in another process. When the same process is both registering and reading from the dataset, it is simpler to use tf.data.experimental.service.distribute instead. If the dataset is already registered with the tf.data service, register_dataset returns the already-registered dataset's id. 
dispatcher = tf.data.experimental.service.DispatchServer()
dispatcher_address = dispatcher.target.split("://")[1]
worker = tf.data.experimental.service.WorkerServer(
    tf.data.experimental.service.WorkerConfig(
        dispatcher_address=dispatcher_address))
dataset = tf.data.Dataset.range(10)
dataset_id = tf.data.experimental.service.register_dataset(
    dispatcher.target, dataset)
dataset = tf.data.experimental.service.from_dataset_id(
    processing_mode="parallel_epochs",
    service=dispatcher.target,
    dataset_id=dataset_id,
    element_spec=dataset.element_spec)
print(list(dataset.as_numpy_iterator()))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

 


 Args
  service   A string indicating how to connect to the tf.data service. The string should be in the format "protocol://address", e.g. "grpc://localhost:5000".  
  dataset   A tf.data.Dataset to register with the tf.data service.   
 


 Returns   A scalar int64 tensor of the registered dataset's id.
tf.data.experimental.service.WorkerConfig Configuration class for tf.data service dispatchers.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.service.WorkerConfig  
tf.data.experimental.service.WorkerConfig(
    dispatcher_address, worker_address=None, port=0, protocol='grpc',
    heartbeat_interval_ms=None, dispatcher_timeout_ms=None
)
 Fields:  
dispatcher_address: Specifies the address of the dispatcher. 
worker_address: Specifies the address of the worker server. This address is passed to the dispatcher so that the dispatcher can tell clients how to connect to this worker. 
port: Specifies the port to bind to. A value of 0 indicates that the worker can bind to any available port. 
protocol: (Optional.) Specifies the protocol to be used by the server. Defaults to "grpc". 
heartbeat_interval_ms: How often the worker should heartbeat to the dispatcher, in milliseconds. If not set, the runtime will select a reasonable default. A higher value will reduce the load on the dispatcher, while a lower value will reduce the time it takes to reclaim resources from finished jobs. 
dispatcher_timeout_ms: How long, in milliseconds, to retry requests to the dispatcher before giving up and reporting an error. Defaults to 1 hour. 
 


 Attributes
  dispatcher_address  
 
  worker_address  
 
  port  
 
  protocol  
 
  heartbeat_interval_ms  
 
  dispatcher_timeout_ms
tf.data.experimental.service.WorkerServer An in-process tf.data service worker server. 
tf.data.experimental.service.WorkerServer(
    config, start=True
)
 A tf.data.experimental.service.WorkerServer performs tf.data.Dataset processing for user-defined datasets, and provides the resulting elements over RPC. A worker is associated with a single tf.data.experimental.service.DispatchServer. 
dispatcher = tf.data.experimental.service.DispatchServer()
dispatcher_address = dispatcher.target.split("://")[1]
worker = tf.data.experimental.service.WorkerServer(
    tf.data.experimental.service.WorkerConfig(
        dispatcher_address=dispatcher_address))
dataset = tf.data.Dataset.range(10)
dataset = dataset.apply(tf.data.experimental.service.distribute(
    processing_mode="parallel_epochs", service=dispatcher.target))
print(list(dataset.as_numpy_iterator()))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
 When starting a dedicated tf.data worker process, use join() to block indefinitely after starting up the server. worker = tf.data.experimental.service.WorkerServer(
    port=5051, dispatcher_address="grpc://localhost:5050")
worker.join()

 


 Args
  config   A tf.data.experimental.service.WorkerConfig configration.  
  start   (Optional.) Boolean, indicating whether to start the server after creating it. Defaults to True.    Methods join View source 
join()
 Blocks until the server has shut down. This is useful when starting a dedicated worker process. worker_server = tf.data.experimental.service.WorkerServer(
    port=5051, dispatcher_address="grpc://localhost:5050")
worker_server.join()
 This method currently blocks forever.
 


 Raises
  tf.errors.OpError   Or one of its subclasses if an error occurs while joining the server.    start View source 
start()
 Starts this server.
 


 Raises
  tf.errors.OpError   Or one of its subclasses if an error occurs while starting the server.
tf.data.experimental.shuffle_and_repeat     View source on GitHub    Shuffles and repeats a Dataset, reshuffling with each repetition. (deprecated)  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.shuffle_and_repeat  
tf.data.experimental.shuffle_and_repeat(
    buffer_size, count=None, seed=None
)
 Warning: THIS FUNCTION IS DEPRECATED. It will be removed in a future version. Instructions for updating: Use tf.data.Dataset.shuffle(buffer_size, seed) followed by tf.data.Dataset.repeat(count). Static tf.data optimizations will take care of using the fused implementation. 
d = tf.data.Dataset.from_tensor_slices([1, 2, 3])
d = d.apply(tf.data.experimental.shuffle_and_repeat(2, count=2))
[elem.numpy() for elem in d] # doctest: +SKIP
[2, 3, 1, 1, 3, 2]
 dataset.apply(
  tf.data.experimental.shuffle_and_repeat(buffer_size, count, seed))
 produces the same output as dataset.shuffle(
  buffer_size, seed=seed, reshuffle_each_iteration=True).repeat(count)
 In each repetition, this dataset fills a buffer with buffer_size elements, then randomly samples elements from this buffer, replacing the selected elements with new elements. For perfect shuffling, set the buffer size equal to the full size of the dataset. For instance, if your dataset contains 10,000 elements but buffer_size is set to 1,000, then shuffle will initially select a random element from only the first 1,000 elements in the buffer. Once an element is selected, its space in the buffer is replaced by the next (i.e. 1,001-st) element, maintaining the 1,000 element buffer.
 


 Args
  buffer_size   A tf.int64 scalar tf.Tensor, representing the maximum number elements that will be buffered when prefetching.  
  count   (Optional.) A tf.int64 scalar tf.Tensor, representing the number of times the dataset should be repeated. The default behavior (if count is None or -1) is for the dataset be repeated indefinitely.  
  seed   (Optional.) A tf.int64 scalar tf.Tensor, representing the random seed that will be used to create the distribution. See tf.random.set_seed for behavior.   
 


 Returns   A Dataset transformation function, which can be passed to tf.data.Dataset.apply.
tf.data.experimental.StatsAggregator     View source on GitHub    A stateful resource that aggregates statistics from one or more iterators. 
tf.data.experimental.StatsAggregator()
 To record statistics, use one of the custom transformation functions defined in this module when defining your tf.data.Dataset. All statistics will be aggregated by the StatsAggregator that is associated with a particular iterator (see below). For example, to record the latency of producing each element by iterating over a dataset: dataset = ...
dataset = dataset.apply(tf.data.experimental.latency_stats("total_bytes"))
 To associate a StatsAggregator with a tf.data.Dataset object, use the following pattern: aggregator = tf.data.experimental.StatsAggregator()
dataset = ...

# Apply `StatsOptions` to associate `dataset` with `aggregator`.
options = tf.data.Options()
options.experimental_stats.aggregator = aggregator
dataset = dataset.with_options(options)


Note: This interface is experimental and expected to change. In particular, we expect to add other implementations of StatsAggregator that provide different ways of exporting statistics, and add more types of statistics.
tf.data.experimental.StatsOptions     View source on GitHub    Represents options for collecting dataset stats using StatsAggregator.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.StatsOptions  
tf.data.experimental.StatsOptions()
 You can set the stats options of a dataset through the experimental_stats property of tf.data.Options; the property is an instance of tf.data.experimental.StatsOptions. For example, to collect latency stats on all dataset edges, use the following pattern: aggregator = tf.data.experimental.StatsAggregator()

options = tf.data.Options()
options.experimental_stats.aggregator = aggregator
options.experimental_stats.latency_all_edges = True
dataset = dataset.with_options(options)

 


 Attributes
  aggregator   Associates the given statistics aggregator with the dataset pipeline.  
  counter_prefix   Prefix for the statistics recorded as counter.  
  latency_all_edges   Whether to add latency measurements on all edges. Defaults to False.  
  prefix   Prefix to prepend all statistics recorded for the input dataset with.    Methods __eq__ View source 
__eq__(
    other
)
 Return self==value. __ne__ View source 
__ne__(
    other
)
 Return self!=value.
tf.data.experimental.take_while     View source on GitHub    A transformation that stops dataset iteration based on a predicate.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.take_while  
tf.data.experimental.take_while(
    predicate
)

 


 Args
  predicate   A function that maps a nested structure of tensors (having shapes and types defined by self.output_shapes and self.output_types) to a scalar tf.bool tensor.   
 


 Returns   A Dataset transformation function, which can be passed to tf.data.Dataset.apply.
tf.data.experimental.TFRecordWriter     View source on GitHub    Writes a dataset to a TFRecord file.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.TFRecordWriter  
tf.data.experimental.TFRecordWriter(
    filename, compression_type=None
)
 The elements of the dataset must be scalar strings. To serialize dataset elements as strings, you can use the tf.io.serialize_tensor function. dataset = tf.data.Dataset.range(3)
dataset = dataset.map(tf.io.serialize_tensor)
writer = tf.data.experimental.TFRecordWriter("/path/to/file.tfrecord")
writer.write(dataset)
 To read back the elements, use TFRecordDataset. dataset = tf.data.TFRecordDataset("/path/to/file.tfrecord")
dataset = dataset.map(lambda x: tf.io.parse_tensor(x, tf.int64))
 To shard a dataset across multiple TFRecord files: dataset = ... # dataset to be written

def reduce_func(key, dataset):
  filename = tf.strings.join([PATH_PREFIX, tf.strings.as_string(key)])
  writer = tf.data.experimental.TFRecordWriter(filename)
  writer.write(dataset.map(lambda _, x: x))
  return tf.data.Dataset.from_tensors(filename)

dataset = dataset.enumerate()
dataset = dataset.apply(tf.data.experimental.group_by_window(
  lambda i, _: i % NUM_SHARDS, reduce_func, tf.int64.max
))

 


 Args
  filename   a string path indicating where to write the TFRecord data.  
  compression_type   (Optional.) a string indicating what type of compression to use when writing the file. See tf.io.TFRecordCompressionType for what types of compression are available. Defaults to None.    Methods write View source 
write(
    dataset
)
 Writes a dataset to a TFRecord file. An operation that writes the content of the specified dataset to the file specified in the constructor. If the file exists, it will be overwritten.
 


 Args
  dataset   a tf.data.Dataset whose elements are to be written to a file   
 


 Returns   In graph mode, this returns an operation which when executed performs the write. In eager mode, the write is performed by the method itself and there is no return value.  
 Raises TypeError: if dataset is not a tf.data.Dataset. TypeError: if the elements produced by the dataset are not scalar strings.
tf.data.experimental.ThreadingOptions     View source on GitHub    Represents options for dataset threading.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.ThreadingOptions  
tf.data.experimental.ThreadingOptions()
 You can set the threading options of a dataset through the experimental_threading property of tf.data.Options; the property is an instance of tf.data.experimental.ThreadingOptions. options = tf.data.Options()
options.experimental_threading.private_threadpool_size = 10
dataset = dataset.with_options(options)

 


 Attributes
  max_intra_op_parallelism   If set, it overrides the maximum degree of intra-op parallelism.  
  private_threadpool_size   If set, the dataset will use a private threadpool of the given size.    Methods __eq__ View source 
__eq__(
    other
)
 Return self==value. __ne__ View source 
__ne__(
    other
)
 Return self!=value.
tf.data.experimental.to_variant     View source on GitHub    Returns a variant representing the given dataset.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.to_variant  
tf.data.experimental.to_variant(
    dataset
)

 


 Args
  dataset   A tf.data.Dataset.   
 


 Returns   A scalar tf.variant tensor representing the given dataset.
tf.data.experimental.unbatch     View source on GitHub    Splits elements of a dataset into multiple elements on the batch dimension. (deprecated)  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.unbatch  
tf.data.experimental.unbatch()
 Warning: THIS FUNCTION IS DEPRECATED. It will be removed in a future version. Instructions for updating: Use tf.data.Dataset.unbatch(). For example, if elements of the dataset are shaped [B, a0, a1, ...], where B may vary for each input element, then for each element in the dataset, the unbatched dataset will contain B consecutive elements of shape [a0, a1, ...]. # NOTE: The following example uses `{ ... }` to represent the contents
# of a dataset.
a = { ['a', 'b', 'c'], ['a', 'b'], ['a', 'b', 'c', 'd'] }

a.unbatch() == {
    'a', 'b', 'c', 'a', 'b', 'a', 'b', 'c', 'd'}

 


 Returns   A Dataset transformation function, which can be passed to tf.data.Dataset.apply.
tf.data.experimental.unique     View source on GitHub    Creates a Dataset from another Dataset, discarding duplicates.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.unique  
tf.data.experimental.unique()
 Use this transformation to produce a dataset that contains one instance of each unique element in the input. For example: dataset = tf.data.Dataset.from_tensor_slices([1, 37, 2, 37, 2, 1])

# Using `unique()` will drop the duplicate elements.
dataset = dataset.apply(tf.data.experimental.unique())  # ==> { 1, 37, 2 }

 


 Returns   A Dataset transformation function, which can be passed to tf.data.Dataset.apply.