neulab/docprompting-conala huggingface dataset:
Module: tf.config.threading Public API for tf.config.threading namespace. Functions get_inter_op_parallelism_threads(...): Get number of threads used for parallelism between independent operations. get_intra_op_parallelism_threads(...): Get number of threads used within an individual op for parallelism. set_inter_op_parallelism_threads(...): Set number of threads used for parallelism between independent operations. set_intra_op_parallelism_threads(...): Set number of threads used within an individual op for parallelism.
tf.config.threading.get_inter_op_parallelism_threads     View source on GitHub    Get number of threads used for parallelism between independent operations.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.config.threading.get_inter_op_parallelism_threads  
tf.config.threading.get_inter_op_parallelism_threads()
 Determines the number of threads used by independent non-blocking operations. 0 means the system picks an appropriate number.
 


 Returns   Number of parallel threads
tf.config.threading.get_intra_op_parallelism_threads     View source on GitHub    Get number of threads used within an individual op for parallelism.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.config.threading.get_intra_op_parallelism_threads  
tf.config.threading.get_intra_op_parallelism_threads()
 Certain operations like matrix multiplication and reductions can utilize parallel threads for speed ups. A value of 0 means the system picks an appropriate number.
 


 Returns   Number of parallel threads
tf.config.threading.set_inter_op_parallelism_threads     View source on GitHub    Set number of threads used for parallelism between independent operations.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.config.threading.set_inter_op_parallelism_threads  
tf.config.threading.set_inter_op_parallelism_threads(
    num_threads
)
 Determines the number of threads used by independent non-blocking operations. 0 means the system picks an appropriate number.
 


 Args
  num_threads   Number of parallel threads
tf.config.threading.set_intra_op_parallelism_threads     View source on GitHub    Set number of threads used within an individual op for parallelism.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.config.threading.set_intra_op_parallelism_threads  
tf.config.threading.set_intra_op_parallelism_threads(
    num_threads
)
 Certain operations like matrix multiplication and reductions can utilize parallel threads for speed ups. A value of 0 means the system picks an appropriate number.
 


 Args
  num_threads   Number of parallel threads
tf.control_dependencies     View source on GitHub    Wrapper for Graph.control_dependencies() using the default graph.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.control_dependencies  
tf.control_dependencies(
    control_inputs
)
 See tf.Graph.control_dependencies for more details. 
Note: In TensorFlow 2 with eager and/or Autograph, you should not require this method, as code executes in the expected order. Only use tf.control_dependencies when working with v1-style code or in a graph context such as inside Dataset.map.
 When eager execution is enabled, any callable object in the control_inputs list will be called.
 


 Args
  control_inputs   A list of Operation or Tensor objects which must be executed or computed before running the operations defined in the context. Can also be None to clear the control dependencies. If eager execution is enabled, any callable object in the control_inputs list will be called.   
 


 Returns   A context manager that specifies control dependencies for all operations constructed within the context.
tf.convert_to_tensor     View source on GitHub    Converts the given value to a Tensor. 
tf.convert_to_tensor(
    value, dtype=None, dtype_hint=None, name=None
)
 This function converts Python objects of various types to Tensor objects. It accepts Tensor objects, numpy arrays, Python lists, and Python scalars. For example: 
import numpy as np
def my_func(arg):
  arg = tf.convert_to_tensor(arg, dtype=tf.float32)
  return arg
 
# The following calls are equivalent.

value_1 = my_func(tf.constant([[1.0, 2.0], [3.0, 4.0]]))
print(value_1)
tf.Tensor(
  [[1. 2.]
   [3. 4.]], shape=(2, 2), dtype=float32)
value_2 = my_func([[1.0, 2.0], [3.0, 4.0]])
print(value_2)
tf.Tensor(
  [[1. 2.]
   [3. 4.]], shape=(2, 2), dtype=float32)
value_3 = my_func(np.array([[1.0, 2.0], [3.0, 4.0]], dtype=np.float32))
print(value_3)
tf.Tensor(
  [[1. 2.]
   [3. 4.]], shape=(2, 2), dtype=float32)
 This function can be useful when composing a new operation in Python (such as my_func in the example above). All standard Python op constructors apply this function to each of their Tensor-valued inputs, which allows those ops to accept numpy arrays, Python lists, and scalars in addition to Tensor objects. 
Note: This function diverges from default Numpy behavior for float and string types when None is present in a Python list or scalar. Rather than silently converting None values, an error will be thrown.

 


 Args
  value   An object whose type has a registered Tensor conversion function.  
  dtype   Optional element type for the returned tensor. If missing, the type is inferred from the type of value.  
  dtype_hint   Optional element type for the returned tensor, used when dtype is None. In some cases, a caller may not have a dtype in mind when converting to a tensor, so dtype_hint can be used as a soft preference. If the conversion to dtype_hint is not possible, this argument has no effect.  
  name   Optional name to use if a new Tensor is created.   
 


 Returns   A Tensor based on value.  

 


 Raises
  TypeError   If no conversion function is registered for value to dtype.  
  RuntimeError   If a registered conversion function returns an invalid value.  
  ValueError   If the value is a tensor not of given dtype in graph mode.
Module: tf.data tf.data.Dataset API for input pipelines. See Importing Data for an overview. Modules experimental module: Experimental API for building input pipelines. Classes class Dataset: Represents a potentially large set of elements. class DatasetSpec: Type specification for tf.data.Dataset. class FixedLengthRecordDataset: A Dataset of fixed-length records from one or more binary files. class Iterator: Represents an iterator of a tf.data.Dataset. class IteratorSpec: Type specification for tf.data.Iterator. class Options: Represents options for tf.data.Dataset. class TFRecordDataset: A Dataset comprising records from one or more TFRecord files. class TextLineDataset: A Dataset comprising lines from one or more text files.
 


 Other Members
  AUTOTUNE   -1  
  INFINITE_CARDINALITY   -1  
  UNKNOWN_CARDINALITY   -2
tf.data.DatasetSpec     View source on GitHub    Type specification for tf.data.Dataset. Inherits From: TypeSpec  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.DatasetSpec, tf.compat.v1.data.experimental.DatasetStructure  
tf.data.DatasetSpec(
    element_spec, dataset_shape=()
)
 See tf.TypeSpec for more information about TensorFlow type specifications. 
dataset = tf.data.Dataset.range(3)
tf.data.DatasetSpec.from_value(dataset)
DatasetSpec(TensorSpec(shape=(), dtype=tf.int64, name=None), TensorShape([]))

 


 Attributes
  value_type   The Python type for values that are compatible with this TypeSpec. In particular, all values that are compatible with this TypeSpec must be an instance of this type. 
   Methods from_value View source 
@staticmethod
from_value(
    value
)
 Creates a DatasetSpec for the given tf.data.Dataset value. is_compatible_with View source 
is_compatible_with(
    spec_or_value
)
 Returns true if spec_or_value is compatible with this TypeSpec. most_specific_compatible_type View source 
most_specific_compatible_type(
    other
)
 Returns the most specific TypeSpec compatible with self and other.
 


 Args
  other   A TypeSpec.   
 


 Raises
  ValueError   If there is no TypeSpec that is compatible with both self and other.    __eq__ View source 
__eq__(
    other
)
 Return self==value. __ne__ View source 
__ne__(
    other
)
 Return self!=value.
tf.data.experimental.assert_cardinality Asserts the cardinality of the input dataset.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.assert_cardinality  
tf.data.experimental.assert_cardinality(
    expected_cardinality
)
 
Note: The following assumes that "examples.tfrecord" contains 42 records.
 
dataset = tf.data.TFRecordDataset("examples.tfrecord")
cardinality = tf.data.experimental.cardinality(dataset)
print((cardinality == tf.data.experimental.UNKNOWN_CARDINALITY).numpy())
True
dataset = dataset.apply(tf.data.experimental.assert_cardinality(42))
print(tf.data.experimental.cardinality(dataset).numpy())
42

 


 Args
  expected_cardinality   The expected cardinality of the input dataset.   
 


 Returns   A Dataset transformation function, which can be passed to tf.data.Dataset.apply.  

 


 Raises
  FailedPreconditionError   The assertion is checked at runtime (when iterating the dataset) and an error is raised if the actual and expected cardinality differ.
tf.data.experimental.AutoShardPolicy Represents the type of auto-sharding we enable.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.AutoShardPolicy  Please see the DistributeOptions.auto_shard_policy documentation for more information on each type of autosharding.
 


 Class Variables
  AUTO   <AutoShardPolicy.AUTO: 0>  
  DATA   <AutoShardPolicy.DATA: 2>  
  FILE   <AutoShardPolicy.FILE: 1>  
  OFF   <AutoShardPolicy.OFF: -1>
tf.data.experimental.bucket_by_sequence_length     View source on GitHub    A transformation that buckets elements in a Dataset by length.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.bucket_by_sequence_length  
tf.data.experimental.bucket_by_sequence_length(
    element_length_func, bucket_boundaries, bucket_batch_sizes, padded_shapes=None,
    padding_values=None, pad_to_bucket_boundary=False, no_padding=False,
    drop_remainder=False
)
 Elements of the Dataset are grouped together by length and then are padded and batched. This is useful for sequence tasks in which the elements have variable length. Grouping together elements that have similar lengths reduces the total fraction of padding in a batch which increases training step efficiency.
 


 Args
  element_length_func   function from element in Dataset to tf.int32, determines the length of the element, which will determine the bucket it goes into.  
  bucket_boundaries   list<int>, upper length boundaries of the buckets.  
  bucket_batch_sizes   list<int>, batch size per bucket. Length should be len(bucket_boundaries) + 1.  
  padded_shapes   Nested structure of tf.TensorShape to pass to tf.data.Dataset.padded_batch. If not provided, will use dataset.output_shapes, which will result in variable length dimensions being padded out to the maximum length in each batch.  
  padding_values   Values to pad with, passed to tf.data.Dataset.padded_batch. Defaults to padding with 0.  
  pad_to_bucket_boundary   bool, if False, will pad dimensions with unknown size to maximum length in batch. If True, will pad dimensions with unknown size to bucket boundary minus 1 (i.e., the maximum length in each bucket), and caller must ensure that the source Dataset does not contain any elements with length longer than max(bucket_boundaries).  
  no_padding   bool, indicates whether to pad the batch features (features need to be either of type tf.sparse.SparseTensor or of same shape).  
  drop_remainder   (Optional.) A tf.bool scalar tf.Tensor, representing whether the last batch should be dropped in the case it has fewer than batch_size elements; the default behavior is not to drop the smaller batch.   
 


 Returns   A Dataset transformation function, which can be passed to tf.data.Dataset.apply.  

 


 Raises
  ValueError   if len(bucket_batch_sizes) != len(bucket_boundaries) + 1.
tf.data.experimental.bytes_produced_stats     View source on GitHub    Records the number of bytes produced by each element of the input dataset.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.bytes_produced_stats  
tf.data.experimental.bytes_produced_stats(
    tag
)
 To consume the statistics, associate a StatsAggregator with the output dataset.
 


 Args
  tag   String. All statistics recorded by the returned transformation will be associated with the given tag.   
 


 Returns   A Dataset transformation function, which can be passed to tf.data.Dataset.apply.
tf.data.experimental.cardinality     View source on GitHub    Returns the cardinality of dataset, if known.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.cardinality  
tf.data.experimental.cardinality(
    dataset
)
 The operation returns the cardinality of dataset. The operation may return tf.data.experimental.INFINITE_CARDINALITY if dataset contains an infinite number of elements or tf.data.experimental.UNKNOWN_CARDINALITY if the analysis fails to determine the number of elements in dataset (e.g. when the dataset source is a file). 
dataset = tf.data.Dataset.range(42)
print(tf.data.experimental.cardinality(dataset).numpy())
42
dataset = dataset.repeat()
cardinality = tf.data.experimental.cardinality(dataset)
print((cardinality == tf.data.experimental.INFINITE_CARDINALITY).numpy())
True
dataset = dataset.filter(lambda x: True)
cardinality = tf.data.experimental.cardinality(dataset)
print((cardinality == tf.data.experimental.UNKNOWN_CARDINALITY).numpy())
True

 


 Args
  dataset   A tf.data.Dataset for which to determine cardinality.   
 


 Returns   A scalar tf.int64 Tensor representing the cardinality of dataset. If the cardinality is infinite or unknown, the operation returns the named constant INFINITE_CARDINALITY and UNKNOWN_CARDINALITY respectively.
tf.data.experimental.choose_from_datasets     View source on GitHub    Creates a dataset that deterministically chooses elements from datasets. 
tf.data.experimental.choose_from_datasets(
    datasets, choice_dataset
)
 For example, given the following datasets: datasets = [tf.data.Dataset.from_tensors("foo").repeat(),
            tf.data.Dataset.from_tensors("bar").repeat(),
            tf.data.Dataset.from_tensors("baz").repeat()]

# Define a dataset containing `[0, 1, 2, 0, 1, 2, 0, 1, 2]`.
choice_dataset = tf.data.Dataset.range(3).repeat(3)

result = tf.data.experimental.choose_from_datasets(datasets, choice_dataset)
 The elements of result will be: "foo", "bar", "baz", "foo", "bar", "baz", "foo", "bar", "baz"

 


 Args
  datasets   A list of tf.data.Dataset objects with compatible structure.  
  choice_dataset   A tf.data.Dataset of scalar tf.int64 tensors between 0 and len(datasets) - 1.   
 


 Returns   A dataset that interleaves elements from datasets according to the values of choice_dataset.  

 


 Raises
  TypeError   If the datasets or choice_dataset arguments have the wrong type.
tf.data.experimental.copy_to_device     View source on GitHub    A transformation that copies dataset elements to the given target_device.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.copy_to_device  
tf.data.experimental.copy_to_device(
    target_device, source_device='/cpu:0'
)

 


 Args
  target_device   The name of a device to which elements will be copied.  
  source_device   The original device on which input_dataset will be placed.   
 


 Returns   A Dataset transformation function, which can be passed to tf.data.Dataset.apply.
tf.data.experimental.Counter     View source on GitHub    Creates a Dataset that counts from start in steps of size step. 
tf.data.experimental.Counter(
    start=0, step=1, dtype=tf.dtypes.int64
)
 For example: Dataset.count() == [0, 1, 2, ...)
Dataset.count(2) == [2, 3, ...)
Dataset.count(2, 5) == [2, 7, 12, ...)
Dataset.count(0, -1) == [0, -1, -2, ...)
Dataset.count(10, -1) == [10, 9, ...)

 


 Args
  start   (Optional.) The starting value for the counter. Defaults to 0.  
  step   (Optional.) The step size for the counter. Defaults to 1.  
  dtype   (Optional.) The data type for counter elements. Defaults to tf.int64.   
 


 Returns   A Dataset of scalar dtype elements.
tf.data.experimental.dense_to_ragged_batch A transformation that batches ragged elements into tf.RaggedTensors.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.dense_to_ragged_batch  
tf.data.experimental.dense_to_ragged_batch(
    batch_size, drop_remainder=False, row_splits_dtype=tf.dtypes.int64
)
 This transformation combines multiple consecutive elements of the input dataset into a single element. Like tf.data.Dataset.batch, the components of the resulting element will have an additional outer dimension, which will be batch_size (or N % batch_size for the last element if batch_size does not divide the number of input elements N evenly and drop_remainder is False). If your program depends on the batches having the same outer dimension, you should set the drop_remainder argument to True to prevent the smaller batch from being produced. Unlike tf.data.Dataset.batch, the input elements to be batched may have different shapes:  If an input element is a tf.Tensor whose static tf.TensorShape is fully defined, then it is batched as normal. If an input element is a tf.Tensor whose static tf.TensorShape contains one or more axes with unknown size (i.e., shape[i]=None), then the output will contain a tf.RaggedTensor that is ragged up to any of such dimensions. If an input element is a tf.RaggedTensor or any other type, then it is batched as normal.  Example: 
dataset = tf.data.Dataset.from_tensor_slices(np.arange(6))
dataset = dataset.map(lambda x: tf.range(x))
dataset.element_spec.shape
TensorShape([None])
dataset = dataset.apply(
    tf.data.experimental.dense_to_ragged_batch(batch_size=2))
for batch in dataset:
  print(batch)
<tf.RaggedTensor [[], [0]]>
<tf.RaggedTensor [[0, 1], [0, 1, 2]]>
<tf.RaggedTensor [[0, 1, 2, 3], [0, 1, 2, 3, 4]]>

 


 Args
  batch_size   A tf.int64 scalar tf.Tensor, representing the number of consecutive elements of this dataset to combine in a single batch.  
  drop_remainder   (Optional.) A tf.bool scalar tf.Tensor, representing whether the last batch should be dropped in the case it has fewer than batch_size elements; the default behavior is not to drop the smaller batch.  
  row_splits_dtype   The dtype that should be used for the row_splits of any new ragged tensors. Existing tf.RaggedTensor elements do not have their row_splits dtype changed.   
 


 Returns
  Dataset   A Dataset.
tf.data.experimental.dense_to_sparse_batch     View source on GitHub    A transformation that batches ragged elements into tf.sparse.SparseTensors.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.dense_to_sparse_batch  
tf.data.experimental.dense_to_sparse_batch(
    batch_size, row_shape
)
 Like Dataset.padded_batch(), this transformation combines multiple consecutive elements of the dataset, which might have different shapes, into a single element. The resulting element has three components (indices, values, and dense_shape), which comprise a tf.sparse.SparseTensor that represents the same data. The row_shape represents the dense shape of each row in the resulting tf.sparse.SparseTensor, to which the effective batch size is prepended. For example: # NOTE: The following examples use `{ ... }` to represent the
# contents of a dataset.
a = { ['a', 'b', 'c'], ['a', 'b'], ['a', 'b', 'c', 'd'] }

a.apply(tf.data.experimental.dense_to_sparse_batch(
    batch_size=2, row_shape=[6])) ==
{
    ([[0, 0], [0, 1], [0, 2], [1, 0], [1, 1]],  # indices
     ['a', 'b', 'c', 'a', 'b'],                 # values
     [2, 6]),                                   # dense_shape
    ([[0, 0], [0, 1], [0, 2], [0, 3]],
     ['a', 'b', 'c', 'd'],
     [1, 6])
}

 


 Args
  batch_size   A tf.int64 scalar tf.Tensor, representing the number of consecutive elements of this dataset to combine in a single batch.  
  row_shape   A tf.TensorShape or tf.int64 vector tensor-like object representing the equivalent dense shape of a row in the resulting tf.sparse.SparseTensor. Each element of this dataset must have the same rank as row_shape, and must have size less than or equal to row_shape in each dimension.   
 


 Returns   A Dataset transformation function, which can be passed to tf.data.Dataset.apply.
tf.data.experimental.DistributeOptions     View source on GitHub    Represents options for distributed data processing.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.DistributeOptions  
tf.data.experimental.DistributeOptions()
 You can set the distribution options of a dataset through the experimental_distribute property of tf.data.Options; the property is an instance of tf.data.experimental.DistributeOptions. options = tf.data.Options()
options.experimental_distribute.auto_shard_policy = AutoShardPolicy.OFF
dataset = dataset.with_options(options)

 


 Attributes
  auto_shard_policy   The type of sharding that auto-shard should attempt. If this is set to FILE, then we will attempt to shard by files (each worker will get a set of files to process). If we cannot find a set of files to shard for at least one file per worker, we will error out. When this option is selected, make sure that you have enough files so that each worker gets at least one file. There will be a runtime error thrown if there are insufficient files. If this is set to DATA, then we will shard by elements produced by the dataset, and each worker will process the whole dataset and discard the portion that is not for itself. If this is set to OFF, then we will not autoshard, and each worker will receive a copy of the full dataset. This option is set to AUTO by default, AUTO will attempt to first shard by FILE, and fall back to sharding by DATA if we cannot find a set of files to shard.  
  num_devices   The number of devices attached to this input pipeline. This will be automatically set by MultiDeviceIterator.    Methods __eq__ View source 
__eq__(
    other
)
 Return self==value. __ne__ View source 
__ne__(
    other
)
 Return self!=value.
tf.data.experimental.enumerate_dataset     View source on GitHub    A transformation that enumerates the elements of a dataset. (deprecated)  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.data.experimental.enumerate_dataset  
tf.data.experimental.enumerate_dataset(
    start=0
)
 Warning: THIS FUNCTION IS DEPRECATED. It will be removed in a future version. Instructions for updating: Use tf.data.Dataset.enumerate(). It is similar to python's enumerate. For example: # NOTE: The following examples use `{ ... }` to represent the
# contents of a dataset.
a = { 1, 2, 3 }
b = { (7, 8), (9, 10) }

# The nested structure of the `datasets` argument determines the
# structure of elements in the resulting dataset.
a.apply(tf.data.experimental.enumerate_dataset(start=5))
=> { (5, 1), (6, 2), (7, 3) }
b.apply(tf.data.experimental.enumerate_dataset())
=> { (0, (7, 8)), (1, (9, 10)) }

 


 Args
  start   A tf.int64 scalar tf.Tensor, representing the start value for enumeration.   
 


 Returns   A Dataset transformation function, which can be passed to tf.data.Dataset.apply.