neulab/docprompting-conala huggingface dataset:
tf.data.experimental.from_variant     View source on GitHub    Constructs a dataset from the given variant and structure.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.from_variant  
tf.data.experimental.from_variant(
    variant, structure
)

 


 Args
  variant   A scalar tf.variant tensor representing a dataset.  
  structure   A tf.data.experimental.Structure object representing the structure of each element in the dataset.   
 


 Returns   A tf.data.Dataset instance.
tf.data.experimental.get_next_as_optional     View source on GitHub    Returns a tf.experimental.Optional with the next element of the iterator. (deprecated)  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.get_next_as_optional  
tf.data.experimental.get_next_as_optional(
    iterator
)
 Warning: THIS FUNCTION IS DEPRECATED. It will be removed in a future version. Instructions for updating: Use tf.data.Iterator.get_next_as_optional() instead. If the iterator has reached the end of the sequence, the returned tf.experimental.Optional will have no value.
 


 Args
  iterator   A tf.data.Iterator.   
 


 Returns   A tf.experimental.Optional object which either contains the next element of the iterator (if it exists) or no value.
tf.data.experimental.get_single_element     View source on GitHub    Returns the single element in dataset as a nested structure of tensors.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.get_single_element  
tf.data.experimental.get_single_element(
    dataset
)
 This function enables you to use a tf.data.Dataset in a stateless "tensor-in tensor-out" expression, without creating an iterator. This can be useful when your preprocessing transformations are expressed as a Dataset, and you want to use the transformation at serving time. For example: def preprocessing_fn(input_str):
  # ...
  return image, label

input_batch = ...  # input batch of BATCH_SIZE elements
dataset = (tf.data.Dataset.from_tensor_slices(input_batch)
           .map(preprocessing_fn, num_parallel_calls=BATCH_SIZE)
           .batch(BATCH_SIZE))

image_batch, label_batch = tf.data.experimental.get_single_element(dataset)

 


 Args
  dataset   A tf.data.Dataset object containing a single element.   
 


 Returns   A nested structure of tf.Tensor objects, corresponding to the single element of dataset.  

 


 Raises
  TypeError   if dataset is not a tf.data.Dataset object. InvalidArgumentError (at runtime): if dataset does not contain exactly one element.
tf.data.experimental.get_structure     View source on GitHub    Returns the type signature for elements of the input dataset / iterator.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.get_structure  
tf.data.experimental.get_structure(
    dataset_or_iterator
)

 


 Args
  dataset_or_iterator   A tf.data.Dataset or an tf.data.Iterator.   
 


 Returns   A nested structure of tf.TypeSpec objects matching the structure of an element of dataset_or_iterator and specifying the type of individual components.  

 


 Raises
  TypeError   If input is not a tf.data.Dataset or an tf.data.Iterator object.
tf.data.experimental.group_by_reducer     View source on GitHub    A transformation that groups elements and performs a reduction.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.group_by_reducer  
tf.data.experimental.group_by_reducer(
    key_func, reducer
)
 This transformation maps element of a dataset to a key using key_func and groups the elements by key. The reducer is used to process each group; its init_func is used to initialize state for each group when it is created, the reduce_func is used to update the state every time an element is mapped to the matching group, and the finalize_func is used to map the final state to an output value.
 


 Args
  key_func   A function mapping a nested structure of tensors (having shapes and types defined by self.output_shapes and self.output_types) to a scalar tf.int64 tensor.  
  reducer   An instance of Reducer, which captures the reduction logic using the init_func, reduce_func, and finalize_func functions.   
 


 Returns   A Dataset transformation function, which can be passed to tf.data.Dataset.apply.
tf.data.experimental.group_by_window     View source on GitHub    A transformation that groups windows of elements by key and reduces them.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.group_by_window  
tf.data.experimental.group_by_window(
    key_func, reduce_func, window_size=None, window_size_func=None
)
 This transformation maps each consecutive element in a dataset to a key using key_func and groups the elements by key. It then applies reduce_func to at most window_size_func(key) elements matching the same key. All except the final window for each key will contain window_size_func(key) elements; the final window may be smaller. You may provide either a constant window_size or a window size determined by the key through window_size_func.
 


 Args
  key_func   A function mapping a nested structure of tensors (having shapes and types defined by self.output_shapes and self.output_types) to a scalar tf.int64 tensor.  
  reduce_func   A function mapping a key and a dataset of up to window_size consecutive elements matching that key to another dataset.  
  window_size   A tf.int64 scalar tf.Tensor, representing the number of consecutive elements matching the same key to combine in a single batch, which will be passed to reduce_func. Mutually exclusive with window_size_func.  
  window_size_func   A function mapping a key to a tf.int64 scalar tf.Tensor, representing the number of consecutive elements matching the same key to combine in a single batch, which will be passed to reduce_func. Mutually exclusive with window_size.   
 


 Returns   A Dataset transformation function, which can be passed to tf.data.Dataset.apply.  

 


 Raises
  ValueError   if neither or both of {window_size, window_size_func} are passed.
tf.data.experimental.ignore_errors     View source on GitHub    Creates a Dataset from another Dataset and silently ignores any errors.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.ignore_errors  
tf.data.experimental.ignore_errors(
    log_warning=False
)
 Use this transformation to produce a dataset that contains the same elements as the input, but silently drops any elements that caused an error. For example: dataset = tf.data.Dataset.from_tensor_slices([1., 2., 0., 4.])

# Computing `tf.debugging.check_numerics(1. / 0.)` will raise an
InvalidArgumentError.
dataset = dataset.map(lambda x: tf.debugging.check_numerics(1. / x, "error"))

# Using `ignore_errors()` will drop the element that causes an error.
dataset =
    dataset.apply(tf.data.experimental.ignore_errors())  # ==> {1., 0.5, 0.2}
 Args: log_warning: (Optional.) A 'tf.bool' scalar indicating whether ignored errors should be logged to stderr. Defaults to 'False'.
 


 Returns   A Dataset transformation function, which can be passed to tf.data.Dataset.apply.
tf.data.experimental.latency_stats     View source on GitHub    Records the latency of producing each element of the input dataset.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.latency_stats  
tf.data.experimental.latency_stats(
    tag
)
 To consume the statistics, associate a StatsAggregator with the output dataset.
 


 Args
  tag   String. All statistics recorded by the returned transformation will be associated with the given tag.   
 


 Returns   A Dataset transformation function, which can be passed to tf.data.Dataset.apply.
tf.data.experimental.load Loads a previously saved dataset. 
tf.data.experimental.load(
    path, element_spec, compression=None, reader_func=None
)
 Example usage: 
import tempfile
path = os.path.join(tempfile.gettempdir(), "saved_data")
# Save a dataset
dataset = tf.data.Dataset.range(2)
tf.data.experimental.save(dataset, path)
new_dataset = tf.data.experimental.load(path,
    tf.TensorSpec(shape=(), dtype=tf.int64))
for elem in new_dataset:
  print(elem)
tf.Tensor(0, shape=(), dtype=int64)
tf.Tensor(1, shape=(), dtype=int64)
 Note that to load a previously saved dataset, you need to specify element_spec -- a type signature of the elements of the saved dataset, which can be obtained via tf.data.Dataset.element_spec. This requirement exists so that shape inference of the loaded dataset does not need to perform I/O. If the default option of sharding the saved dataset was used, the element order of the saved dataset will be preserved when loading it. The reader_func argument can be used to specify a custom order in which elements should be loaded from the individual shards. The reader_func is expected to take a single argument -- a dataset of datasets, each containing elements of one of the shards -- and return a dataset of elements. For example, the order of shards can be shuffled when loading them as follows: def custom_reader_func(datasets):
  datasets = datasets.shuffle(NUM_SHARDS)
  return datasets.interleave(lambda x: x, num_parallel_calls=AUTOTUNE)

dataset = tf.data.experimental.load(
    path="/path/to/data", ..., reader_func=custom_reader_func)

 


 Args
  path   Required. A path pointing to a previously saved dataset.  
  element_spec   Required. A nested structure of tf.TypeSpec objects matching the structure of an element of the saved dataset and specifying the type of individual element components.  
  compression   Optional. The algorithm to use to decompress the data when reading it. Supported options are GZIP and NONE. Defaults to NONE.  
  reader_func   Optional. A function to control how to read data from shards. If present, the function will be traced and executed as graph computation.   
 


 Returns   A tf.data.Dataset instance.
tf.data.experimental.make_saveable_from_iterator     View source on GitHub    Returns a SaveableObject for saving/restoring iterator state using Saver. (deprecated)  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.make_saveable_from_iterator  
tf.data.experimental.make_saveable_from_iterator(
    iterator, external_state_policy='fail'
)
 Warning: THIS FUNCTION IS DEPRECATED. It will be removed in a future version. Instructions for updating: make_saveable_from_iterator is intended for use in TF1 with tf.compat.v1.Saver. In TF2, use tf.train.Checkpoint instead.
 


 Args
  iterator   Iterator.  
  external_state_policy   A string that identifies how to handle input pipelines that depend on external state. Possible values are 'ignore': The external state is silently ignored. 'warn': The external state is ignored, logging a warning. 'fail': The operation fails upon encountering external state. By default we set it to 'fail'.   
 


 Returns   A SaveableObject for saving/restoring iterator state using Saver.  

 


 Raises
  ValueError   If iterator does not support checkpointing.  
  ValueError   If external_state_policy is not one of 'warn', 'ignore' or 'fail'.    For example: with tf.Graph().as_default():
  ds = tf.data.Dataset.range(10)
  iterator = ds.make_initializable_iterator()
  # Build the iterator SaveableObject.
  saveable_obj = tf.data.experimental.make_saveable_from_iterator(iterator)
  # Add the SaveableObject to the SAVEABLE_OBJECTS collection so
  # it can be automatically saved using Saver.
  tf.compat.v1.add_to_collection(tf.GraphKeys.SAVEABLE_OBJECTS, saveable_obj)
  saver = tf.compat.v1.train.Saver()

  while continue_training:
    ... Perform training ...
    if should_save_checkpoint:
      saver.save()


Note: When restoring the iterator, the existing iterator state is completely discarded. This means that any changes you may have made to the Dataset graph will be discarded as well! This includes the new Dataset graph that you may have built during validation. So, while running validation, make sure to run the initializer for the validation input pipeline after restoring the checkpoint.


Note: Not all iterators support checkpointing yet. Attempting to save the state of an unsupported iterator will throw an error.
tf.data.experimental.MapVectorizationOptions     View source on GitHub    Represents options for the MapVectorization optimization.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.MapVectorizationOptions  
tf.data.experimental.MapVectorizationOptions()

 


 Attributes
  enabled   Whether to vectorize map transformations. If None, defaults to False.  
  use_choose_fastest   Whether to use ChooseFastestBranchDataset with this transformation. If True, the pipeline picks between the vectorized and original segment at runtime based on their iterations speed. If None, defaults to False.    Methods __eq__ View source 
__eq__(
    other
)
 Return self==value. __ne__ View source 
__ne__(
    other
)
 Return self!=value.
tf.data.experimental.map_and_batch     View source on GitHub    Fused implementation of map and batch. (deprecated)  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.map_and_batch  
tf.data.experimental.map_and_batch(
    map_func, batch_size, num_parallel_batches=None, drop_remainder=False,
    num_parallel_calls=None
)
 Warning: THIS FUNCTION IS DEPRECATED. It will be removed in a future version. Instructions for updating: Use tf.data.Dataset.map(map_func, num_parallel_calls) followed by tf.data.Dataset.batch(batch_size, drop_remainder). Static tf.data optimizations will take care of using the fused implementation. Maps map_func across batch_size consecutive elements of this dataset and then combines them into a batch. Functionally, it is equivalent to map followed by batch. This API is temporary and deprecated since input pipeline optimization now fuses consecutive map and batch operations automatically.
 


 Args
  map_func   A function mapping a nested structure of tensors to another nested structure of tensors.  
  batch_size   A tf.int64 scalar tf.Tensor, representing the number of consecutive elements of this dataset to combine in a single batch.  
  num_parallel_batches   (Optional.) A tf.int64 scalar tf.Tensor, representing the number of batches to create in parallel. On one hand, higher values can help mitigate the effect of stragglers. On the other hand, higher values can increase contention if CPU is scarce.  
  drop_remainder   (Optional.) A tf.bool scalar tf.Tensor, representing whether the last batch should be dropped in case its size is smaller than desired; the default behavior is not to drop the smaller batch.  
  num_parallel_calls   (Optional.) A tf.int32 scalar tf.Tensor, representing the number of elements to process in parallel. If not specified, batch_size * num_parallel_batches elements will be processed in parallel. If the value tf.data.AUTOTUNE is used, then the number of parallel calls is set dynamically based on available CPU.   
 


 Returns   A Dataset transformation function, which can be passed to tf.data.Dataset.apply.  

 


 Raises
  ValueError   If both num_parallel_batches and num_parallel_calls are specified.
tf.data.experimental.parallel_interleave     View source on GitHub    A parallel version of the Dataset.interleave() transformation. (deprecated)  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.parallel_interleave  
tf.data.experimental.parallel_interleave(
    map_func, cycle_length, block_length=1, sloppy=False,
    buffer_output_elements=None, prefetch_input_elements=None
)
 Warning: THIS FUNCTION IS DEPRECATED. It will be removed in a future version. Instructions for updating: Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_deterministic. parallel_interleave() maps map_func across its input to produce nested datasets, and outputs their elements interleaved. Unlike tf.data.Dataset.interleave, it gets elements from cycle_length nested datasets in parallel, which increases the throughput, especially in the presence of stragglers. Furthermore, the sloppy argument can be used to improve performance, by relaxing the requirement that the outputs are produced in a deterministic order, and allowing the implementation to skip over nested datasets whose elements are not readily available when requested. Example usage: # Preprocess 4 files concurrently.
filenames = tf.data.Dataset.list_files("/path/to/data/train*.tfrecords")
dataset = filenames.apply(
    tf.data.experimental.parallel_interleave(
        lambda filename: tf.data.TFRecordDataset(filename),
        cycle_length=4))

Warning: If sloppy is True, the order of produced elements is not deterministic.
 


 Args
  map_func   A function mapping a nested structure of tensors to a Dataset.  
  cycle_length   The number of input Datasets to interleave from in parallel.  
  block_length   The number of consecutive elements to pull from an input Dataset before advancing to the next input Dataset.  
  sloppy   A boolean controlling whether determinism should be traded for performance by allowing elements to be produced out of order. If sloppy is None, the tf.data.Options.experimental_deterministic dataset option (True by default) is used to decide whether to enforce a deterministic order.  
  buffer_output_elements   The number of elements each iterator being interleaved should buffer (similar to the .prefetch() transformation for each interleaved iterator).  
  prefetch_input_elements   The number of input elements to transform to iterators before they are needed for interleaving.   
 


 Returns   A Dataset transformation function, which can be passed to tf.data.Dataset.apply.
tf.data.experimental.parse_example_dataset     View source on GitHub    A transformation that parses Example protos into a dict of tensors.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.parse_example_dataset  
tf.data.experimental.parse_example_dataset(
    features, num_parallel_calls=1, deterministic=None
)
 Parses a number of serialized Example protos given in serialized. We refer to serialized as a batch with batch_size many entries of individual Example protos. This op parses serialized examples into a dictionary mapping keys to Tensor, SparseTensor, and RaggedTensor objects. features is a dict from keys to VarLenFeature, RaggedFeature, SparseFeature, and FixedLenFeature objects. Each VarLenFeature and SparseFeature is mapped to a SparseTensor; each RaggedFeature is mapped to a RaggedTensor; and each FixedLenFeature is mapped to a Tensor. See tf.io.parse_example for more details about feature dictionaries.
 


 Args
  features   A dict mapping feature keys to FixedLenFeature, VarLenFeature, RaggedFeature, and SparseFeature values.  
  num_parallel_calls   (Optional.) A tf.int32 scalar tf.Tensor, representing the number of parsing processes to call in parallel.  
  deterministic   (Optional.) A boolean controlling whether determinism should be traded for performance by allowing elements to be produced out of order if some parsing calls complete faster than others. If deterministic is None, the tf.data.Options.experimental_deterministic dataset option (True by default) is used to decide whether to produce elements deterministically.   
 


 Returns   A dataset transformation function, which can be passed to tf.data.Dataset.apply.  

 


 Raises
  ValueError   if features argument is None.
tf.data.experimental.prefetch_to_device     View source on GitHub    A transformation that prefetches dataset values to the given device.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.prefetch_to_device  
tf.data.experimental.prefetch_to_device(
    device, buffer_size=None
)
 
Note: Although the transformation creates a tf.data.Dataset, the transformation must be the final Dataset in the input pipeline.

 


 Args
  device   A string. The name of a device to which elements will be prefetched.  
  buffer_size   (Optional.) The number of elements to buffer on device. Defaults to an automatically chosen value.   
 


 Returns   A Dataset transformation function, which can be passed to tf.data.Dataset.apply.
tf.data.experimental.Reducer     View source on GitHub    A reducer is used for reducing a set of elements.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.Reducer  
tf.data.experimental.Reducer(
    init_func, reduce_func, finalize_func
)
 A reducer is represented as a tuple of the three functions: 1) initialization function: key => initial state 2) reduce function: (old state, input) => new state 3) finalization function: state => result
 


 Attributes
  finalize_func  
 
  init_func  
 
  reduce_func
tf.data.experimental.rejection_resample     View source on GitHub    A transformation that resamples a dataset to achieve a target distribution.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.rejection_resample  
tf.data.experimental.rejection_resample(
    class_func, target_dist, initial_dist=None, seed=None
)
 
Note: Resampling is performed via rejection sampling; some fraction of the input values will be dropped.

 


 Args
  class_func   A function mapping an element of the input dataset to a scalar tf.int32 tensor. Values should be in [0, num_classes).  
  target_dist   A floating point type tensor, shaped [num_classes].  
  initial_dist   (Optional.) A floating point type tensor, shaped [num_classes]. If not provided, the true class distribution is estimated live in a streaming fashion.  
  seed   (Optional.) Python integer seed for the resampler.   
 


 Returns   A Dataset transformation function, which can be passed to tf.data.Dataset.apply.
tf.data.experimental.sample_from_datasets     View source on GitHub    Samples elements at random from the datasets in datasets. 
tf.data.experimental.sample_from_datasets(
    datasets, weights=None, seed=None
)

 


 Args
  datasets   A list of tf.data.Dataset objects with compatible structure.  
  weights   (Optional.) A list of len(datasets) floating-point values where weights[i] represents the probability with which an element should be sampled from datasets[i], or a tf.data.Dataset object where each element is such a list. Defaults to a uniform distribution across datasets.  
  seed   (Optional.) A tf.int64 scalar tf.Tensor, representing the random seed that will be used to create the distribution. See tf.random.set_seed for behavior.   
 


 Returns   A dataset that interleaves elements from datasets at random, according to weights if provided, otherwise with uniform probability.  

 


 Raises
  TypeError   If the datasets or weights arguments have the wrong type.  
  ValueError   If the weights argument is specified and does not match the length of the datasets element.