neulab/docprompting-conala huggingface dataset:
tf.strings.length     View source on GitHub    String lengths of input. 
tf.strings.length(
    input, unit='BYTE', name=None
)
 Computes the length of each string given in the input tensor. 
strings = tf.constant(['Hello','TensorFlow', '\U0001F642'])
tf.strings.length(strings).numpy() # default counts bytes
array([ 5, 10, 4], dtype=int32)
tf.strings.length(strings, unit="UTF8_CHAR").numpy()
array([ 5, 10, 1], dtype=int32)

 


 Args
  input   A Tensor of type string. The strings for which to compute the length for each element.  
  unit   An optional string from: "BYTE", "UTF8_CHAR". Defaults to "BYTE". The unit that is counted to compute string length. One of: "BYTE" (for the number of bytes in each string) or "UTF8_CHAR" (for the number of UTF-8 encoded Unicode code points in each string). Results are undefined if unit=UTF8_CHAR and the input strings do not contain structurally valid UTF-8.  
  name   A name for the operation (optional).   
 


 Returns   A Tensor of type int32.
tf.strings.lower Converts all uppercase characters into their respective lowercase replacements.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.strings.lower  
tf.strings.lower(
    input, encoding='', name=None
)
 Example: 
tf.strings.lower("CamelCase string and ALL CAPS")
<tf.Tensor: shape=(), dtype=string, numpy=b'camelcase string and all caps'>

 


 Args
  input   A Tensor of type string.  
  encoding   An optional string. Defaults to "".  
  name   A name for the operation (optional).   
 


 Returns   A Tensor of type string.
tf.strings.ngrams     View source on GitHub    Create a tensor of n-grams based on data.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.strings.ngrams  
tf.strings.ngrams(
    data, ngram_width, separator=' ', pad_values=None, padding_width=None,
    preserve_short_sequences=False, name=None
)
 Creates a tensor of n-grams based on data. The n-grams are created by joining windows of width adjacent strings from the inner axis of data using separator. The input data can be padded on both the start and end of the sequence, if desired, using the pad_values argument. If set, pad_values should contain either a tuple of strings or a single string; the 0th element of the tuple will be used to pad the left side of the sequence and the 1st element of the tuple will be used to pad the right side of the sequence. The padding_width arg controls how many padding values are added to each side; it defaults to ngram_width-1. If this op is configured to not have padding, or if it is configured to add padding with padding_width set to less than ngram_width-1, it is possible that a sequence, or a sequence plus padding, is smaller than the ngram width. In that case, no ngrams will be generated for that sequence. This can be prevented by setting preserve_short_sequences, which will cause the op to always generate at least one ngram per non-empty sequence. Examples: 
tf.strings.ngrams(["A", "B", "C", "D"], 2).numpy()
array([b'A B', b'B C', b'C D'], dtype=object)
tf.strings.ngrams(["TF", "and", "keras"], 1).numpy()
array([b'TF', b'and', b'keras'], dtype=object)

 


 Args
  data   A Tensor or RaggedTensor containing the source data for the ngrams.  
  ngram_width   The width(s) of the ngrams to create. If this is a list or tuple, the op will return ngrams of all specified arities in list order. Values must be non-Tensor integers greater than 0.  
  separator   The separator string used between ngram elements. Must be a string constant, not a Tensor.  
  pad_values   A tuple of (left_pad_value, right_pad_value), a single string, or None. If None, no padding will be added; if a single string, then that string will be used for both left and right padding. Values must be Python strings.  
  padding_width   If set, padding_width pad values will be added to both sides of each sequence. Defaults to ngram_width-1. Must be greater than  (Note that 1-grams are never padded, regardless of this value.) 

 
  preserve_short_sequences   If true, then ensure that at least one ngram is generated for each input sequence. In particular, if an input sequence is shorter than min(ngram_width) + 2*pad_width, then generate a single ngram containing the entire sequence. If false, then no ngrams are generated for these short input sequences.  
  name   The op name.   
 


 Returns   A RaggedTensor of ngrams. If data.shape=[D1...DN, S], then output.shape=[D1...DN, NUM_NGRAMS], where NUM_NGRAMS=S-ngram_width+1+2*padding_width.  

 


 Raises
  TypeError   if pad_values is set to an invalid type.  
  ValueError   if pad_values, padding_width, or ngram_width is set to an invalid value.
tf.strings.reduce_join     View source on GitHub    Joins all strings into a single string, or joins along an axis. 
tf.strings.reduce_join(
    inputs, axis=None, keepdims=False, separator='', name=None
)
 
tf.strings.reduce_join([['abc','123'],
                        ['def','456']]).numpy()
b'abc123def456'
tf.strings.reduce_join([['abc','123'],
                        ['def','456']], axis=-1).numpy()
array([b'abc123', b'def456'], dtype=object)
tf.strings.reduce_join([['abc','123'],
                        ['def','456']],
                       axis=-1,
                       separator=" ").numpy()
array([b'abc 123', b'def 456'], dtype=object)

 


 Args
  inputs   A tf.string tensor.  
  axis   Which axis to join along. The default behavior is to join all elements, producing a scalar.  
  keepdims   If true, retains reduced dimensions with length 1.  
  separator   a string added between each string being joined.  
  name   A name for the operation (optional).   
 


 Returns   A tf.string tensor.
tf.strings.regex_full_match       View source on GitHub    Check if the input matches the regex pattern.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.strings.regex_full_match  
tf.strings.regex_full_match(
    input, pattern, name=None
)
 The input is a string tensor of any shape. The pattern is a scalar string tensor which is applied to every element of the input tensor. The boolean values (True or False) of the output tensor indicate if the input matches the regex pattern provided. The pattern follows the re2 syntax (https://github.com/google/re2/wiki/Syntax) Examples: 
tf.strings.regex_full_match(["TF lib", "lib TF"], ".*lib$")
<tf.Tensor: shape=(2,), dtype=bool, numpy=array([ True, False])>
tf.strings.regex_full_match(["TF lib", "lib TF"], ".*TF$")
<tf.Tensor: shape=(2,), dtype=bool, numpy=array([False,  True])>

 


 Args
  input   A Tensor of type string. A string tensor of the text to be processed.  
  pattern   A Tensor of type string. A scalar string tensor containing the regular expression to match the input.  
  name   A name for the operation (optional).   
 


 Returns   A Tensor of type bool.
tf.strings.regex_replace     View source on GitHub    Replace elements of input matching regex pattern with rewrite.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.regex_replace, tf.compat.v1.strings.regex_replace  
tf.strings.regex_replace(
    input, pattern, rewrite, replace_global=True, name=None
)
 
tf.strings.regex_replace("Text with tags.<br /><b>contains html</b>",
                         "<[^>]+>", " ")
<tf.Tensor: shape=(), dtype=string, numpy=b'Text with tags.  contains html '>

 


 Args
  input   string Tensor, the source strings to process.  
  pattern   string or scalar string Tensor, regular expression to use, see more details at https://github.com/google/re2/wiki/Syntax  
  rewrite   string or scalar string Tensor, value to use in match replacement, supports backslash-escaped digits (\1 to \9) can be to insert text matching corresponding parenthesized group.  
  replace_global   bool, if True replace all non-overlapping matches, else replace only the first match.  
  name   A name for the operation (optional).   
 


 Returns   string Tensor of the same shape as input with specified replacements.
tf.strings.split     View source on GitHub    Split elements of input based on sep into a RaggedTensor. 
tf.strings.split(
    input, sep=None, maxsplit=-1, name=None
)
 Let N be the size of input (typically N will be the batch size). Split each element of input based on sep and return a RaggedTensor containing the split tokens. Empty tokens are ignored. Example: 
tf.strings.split('hello world').numpy()
 array([b'hello', b'world'], dtype=object)
tf.strings.split(['hello world', 'a b c'])
<tf.RaggedTensor [[b'hello', b'world'], [b'a', b'b', b'c']]>
 If sep is given, consecutive delimiters are not grouped together and are deemed to delimit empty strings. For example, input of "1<>2<><>3" and sep of "<>" returns ["1", "2", "", "3"]. If sep is None or an empty string, consecutive whitespace are regarded as a single separator, and the result will contain no empty strings at the start or end if the string has leading or trailing whitespace. Note that the above mentioned behavior matches python's str.split.
 


 Args
  input   A string Tensor of rank N, the strings to split. If rank(input) is not known statically, then it is assumed to be 1.  
  sep   0-D string Tensor, the delimiter string.  
  maxsplit   An int. If maxsplit > 0, limit of the split of the result.  
  name   A name for the operation (optional).   
 


 Raises
  ValueError   If sep is not a string.   
 


 Returns   A RaggedTensor of rank N+1, the strings split according to the delimiter.
tf.strings.strip Strip leading and trailing whitespaces from the Tensor.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.string_strip, tf.compat.v1.strings.strip  
tf.strings.strip(
    input, name=None
)

 


 Args
  input   A Tensor of type string. A string Tensor of any shape.  
  name   A name for the operation (optional).   
 


 Returns   A Tensor of type string.
tf.strings.substr     View source on GitHub    Return substrings from Tensor of strings. 
tf.strings.substr(
    input, pos, len, unit='BYTE', name=None
)
 For each string in the input Tensor, creates a substring starting at index pos with a total length of len. If len defines a substring that would extend beyond the length of the input string, or if len is negative, then as many characters as possible are used. A negative pos indicates distance within the string backwards from the end. If pos specifies an index which is out of range for any of the input strings, then an InvalidArgumentError is thrown. pos and len must have the same shape, otherwise a ValueError is thrown on Op creation. 
Note: Substr supports broadcasting up to two dimensions. More about broadcasting here
 Examples Using scalar pos and len: input = [b'Hello', b'World']
position = 1
length = 3

output = [b'ell', b'orl']
 Using pos and len with same shape as input: input = [[b'ten', b'eleven', b'twelve'],
         [b'thirteen', b'fourteen', b'fifteen'],
         [b'sixteen', b'seventeen', b'eighteen']]
position = [[1, 2, 3],
            [1, 2, 3],
            [1, 2, 3]]
length =   [[2, 3, 4],
            [4, 3, 2],
            [5, 5, 5]]

output = [[b'en', b'eve', b'lve'],
          [b'hirt', b'urt', b'te'],
          [b'ixtee', b'vente', b'hteen']]
 Broadcasting pos and len onto input: input = [[b'ten', b'eleven', b'twelve'],
         [b'thirteen', b'fourteen', b'fifteen'],
         [b'sixteen', b'seventeen', b'eighteen'],
         [b'nineteen', b'twenty', b'twentyone']]
position = [1, 2, 3]
length =   [1, 2, 3]

output = [[b'e', b'ev', b'lve'],
          [b'h', b'ur', b'tee'],
          [b'i', b've', b'hte'],
          [b'i', b'en', b'nty']]
 Broadcasting input onto pos and len: input = b'thirteen'
position = [1, 5, 7]
length =   [3, 2, 1]

output = [b'hir', b'ee', b'n']

 


 Raises    
ValueError: If the first argument cannot be converted to a Tensor of dtype string. 
InvalidArgumentError: If indices are out of range. 
ValueError: If pos and len are not the same shape. 

 

 


 Args
  input   A Tensor of type string. Tensor of strings  
  pos   A Tensor. Must be one of the following types: int32, int64. Scalar defining the position of first character in each substring  
  len   A Tensor. Must have the same type as pos. Scalar defining the number of characters to include in each substring  
  unit   An optional string from: "BYTE", "UTF8_CHAR". Defaults to "BYTE". The unit that is used to create the substring. One of: "BYTE" (for defining position and length by bytes) or "UTF8_CHAR" (for the UTF-8 encoded Unicode code points). The default is "BYTE". Results are undefined if unit=UTF8_CHAR and the input strings do not contain structurally valid UTF-8.  
  name   A name for the operation (optional).   
 


 Returns   A Tensor of type string.
tf.strings.to_hash_bucket     View source on GitHub    Converts each string in the input Tensor to its hash mod by a number of buckets. 
tf.strings.to_hash_bucket(
    input, num_buckets, name=None
)
 The hash function is deterministic on the content of the string within the process. Note that the hash function may change from time to time. This functionality will be deprecated and it's recommended to use tf.strings.to_hash_bucket_fast() or tf.strings.to_hash_bucket_strong(). Examples: 
tf.strings.to_hash_bucket(["Hello", "TensorFlow", "2.x"], 3)
<tf.Tensor: shape=(3,), dtype=int64, numpy=array([2, 0, 1])>

 


 Args
  input   A Tensor of type string.  
  num_buckets   An int that is >= 1. The number of buckets.  
  name   A name for the operation (optional).   
 


 Returns   A Tensor of type int64.
tf.strings.to_hash_bucket_fast Converts each string in the input Tensor to its hash mod by a number of buckets.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.string_to_hash_bucket_fast, tf.compat.v1.strings.to_hash_bucket_fast  
tf.strings.to_hash_bucket_fast(
    input, num_buckets, name=None
)
 The hash function is deterministic on the content of the string within the process and will never change. However, it is not suitable for cryptography. This function may be used when CPU time is scarce and inputs are trusted or unimportant. There is a risk of adversaries constructing inputs that all hash to the same bucket. To prevent this problem, use a strong hash function with tf.string_to_hash_bucket_strong. Examples: 
tf.strings.to_hash_bucket_fast(["Hello", "TensorFlow", "2.x"], 3).numpy()
array([0, 2, 2])

 


 Args
  input   A Tensor of type string. The strings to assign a hash bucket.  
  num_buckets   An int that is >= 1. The number of buckets.  
  name   A name for the operation (optional).   
 


 Returns   A Tensor of type int64.
tf.strings.to_hash_bucket_strong Converts each string in the input Tensor to its hash mod by a number of buckets.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.string_to_hash_bucket_strong, tf.compat.v1.strings.to_hash_bucket_strong  
tf.strings.to_hash_bucket_strong(
    input, num_buckets, key, name=None
)
 The hash function is deterministic on the content of the string within the process. The hash function is a keyed hash function, where attribute key defines the key of the hash function. key is an array of 2 elements. A strong hash is important when inputs may be malicious, e.g. URLs with additional components. Adversaries could try to make their inputs hash to the same bucket for a denial-of-service attack or to skew the results. A strong hash can be used to make it difficult to find inputs with a skewed hash value distribution over buckets. This requires that the hash function is seeded by a high-entropy (random) "key" unknown to the adversary. The additional robustness comes at a cost of roughly 4x higher compute time than tf.string_to_hash_bucket_fast. Examples: 
tf.strings.to_hash_bucket_strong(["Hello", "TF"], 3, [1, 2]).numpy()
array([2, 0])

 


 Args
  input   A Tensor of type string. The strings to assign a hash bucket.  
  num_buckets   An int that is >= 1. The number of buckets.  
  key   A list of ints. The key used to seed the hash function, passed as a list of two uint64 elements.  
  name   A name for the operation (optional).   
 


 Returns   A Tensor of type int64.
tf.strings.to_number     View source on GitHub    Converts each string in the input Tensor to the specified numeric type. 
tf.strings.to_number(
    input, out_type=tf.dtypes.float32, name=None
)
 (Note that int32 overflow results in an error while float overflow results in a rounded value.) Examples: 
tf.strings.to_number("1.55")
<tf.Tensor: shape=(), dtype=float32, numpy=1.55>
tf.strings.to_number("3", tf.int32)
<tf.Tensor: shape=(), dtype=int32, numpy=3>

 


 Args
  input   A Tensor of type string.  
  out_type   An optional tf.DType from: tf.float32, tf.float64, tf.int32, tf.int64. Defaults to tf.float32. The numeric type to interpret each string in string_tensor as.  
  name   A name for the operation (optional).   
 


 Returns   A Tensor of type out_type.
tf.strings.unicode_decode     View source on GitHub    Decodes each string in input into a sequence of Unicode code points.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.strings.unicode_decode  
tf.strings.unicode_decode(
    input, input_encoding, errors='replace', replacement_char=65533,
    replace_control_characters=False, name=None
)
 result[i1...iN, j] is the Unicode codepoint for the jth character in input[i1...iN], when decoded using input_encoding.
 


 Args
  input   An N dimensional potentially ragged string tensor with shape [D1...DN]. N must be statically known.  
  input_encoding   String name for the unicode encoding that should be used to decode each string.  
  errors   Specifies the response when an input string can't be converted using the indicated encoding. One of:  
'strict': Raise an exception for any illegal substrings. 
'replace': Replace illegal substrings with replacement_char. 
'ignore': Skip illegal substrings. 

 
  replacement_char   The replacement codepoint to be used in place of invalid substrings in input when errors='replace'; and in place of C0 control characters in input when replace_control_characters=True.  
  replace_control_characters   Whether to replace the C0 control characters (U+0000 - U+001F) with the replacement_char.  
  name   A name for the operation (optional).   
 


 Returns   A N+1 dimensional int32 tensor with shape [D1...DN, (num_chars)]. The returned tensor is a tf.Tensor if input is a scalar, or a tf.RaggedTensor otherwise.  
 Example: 
input = [s.encode('utf8') for s in (u'G\xf6\xf6dnight', u'\U0001f60a')]
tf.strings.unicode_decode(input, 'UTF-8').to_list()
[[71, 246, 246, 100, 110, 105, 103, 104, 116], [128522]]
tf.strings.unicode_decode_with_offsets     View source on GitHub    Decodes each string into a sequence of code points with start offsets.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.strings.unicode_decode_with_offsets  
tf.strings.unicode_decode_with_offsets(
    input, input_encoding, errors='replace', replacement_char=65533,
    replace_control_characters=False, name=None
)
 This op is similar to tf.strings.decode(...), but it also returns the start offset for each character in its respective string. This information can be used to align the characters with the original byte sequence. Returns a tuple (codepoints, start_offsets) where:  
codepoints[i1...iN, j] is the Unicode codepoint for the jth character in input[i1...iN], when decoded using input_encoding. 
start_offsets[i1...iN, j] is the start byte offset for the jth character in input[i1...iN], when decoded using input_encoding. 
 


 Args
  input   An N dimensional potentially ragged string tensor with shape [D1...DN]. N must be statically known.  
  input_encoding   String name for the unicode encoding that should be used to decode each string.  
  errors   Specifies the response when an input string can't be converted using the indicated encoding. One of:  
'strict': Raise an exception for any illegal substrings. 
'replace': Replace illegal substrings with replacement_char. 
'ignore': Skip illegal substrings. 

 
  replacement_char   The replacement codepoint to be used in place of invalid substrings in input when errors='replace'; and in place of C0 control characters in input when replace_control_characters=True.  
  replace_control_characters   Whether to replace the C0 control characters (U+0000 - U+001F) with the replacement_char.  
  name   A name for the operation (optional).   
 


 Returns   A tuple of N+1 dimensional tensors (codepoints, start_offsets).  
codepoints is an int32 tensor with shape [D1...DN, (num_chars)]. 
offsets is an int64 tensor with shape [D1...DN, (num_chars)].  The returned tensors are tf.Tensors if input is a scalar, or tf.RaggedTensors otherwise. 
 
 Example: 
input = [s.encode('utf8') for s in (u'G\xf6\xf6dnight', u'\U0001f60a')]
result = tf.strings.unicode_decode_with_offsets(input, 'UTF-8')
result[0].to_list()  # codepoints
[[71, 246, 246, 100, 110, 105, 103, 104, 116], [128522]]
result[1].to_list()  # offsets
[[0, 1, 3, 5, 6, 7, 8, 9, 10], [0]]
tf.strings.unicode_encode     View source on GitHub    Encodes each sequence of Unicode code points in input into a string.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.strings.unicode_encode  
tf.strings.unicode_encode(
    input, output_encoding, errors='replace', replacement_char=65533,
    name=None
)
 result[i1...iN] is the string formed by concatenating the Unicode codepoints input[1...iN, :], encoded using output_encoding.
 


 Args
  input   An N+1 dimensional potentially ragged integer tensor with shape [D1...DN, num_chars].  
  output_encoding   Unicode encoding that should be used to encode each codepoint sequence. Can be "UTF-8", "UTF-16-BE", or "UTF-32-BE".  
  errors   Specifies the response when an invalid codepoint is encountered (optional). One of:  
'replace': Replace invalid codepoint with the replacement_char. (default) 
'ignore': Skip invalid codepoints. 
'strict': Raise an exception for any invalid codepoint. 

 
  replacement_char   The replacement character codepoint to be used in place of any invalid input when errors='replace'. Any valid unicode codepoint may be used. The default value is the default unicode replacement character which is 0xFFFD (U+65533).  
  name   A name for the operation (optional).   
 


 Returns   A N dimensional string tensor with shape [D1...DN].  
 Example: 
input = tf.ragged.constant(
    [[71, 246, 246, 100, 110, 105, 103, 104, 116], [128522]])
print(unicode_encode(input, 'UTF-8'))
tf.Tensor([b'G\xc3\xb6\xc3\xb6dnight' b'\xf0\x9f\x98\x8a'],
          shape=(2,), dtype=string)
tf.strings.unicode_script Determine the script codes of a given tensor of Unicode integer code points.  View aliases  Compat aliases for migration 
See Migration guide for more details. tf.compat.v1.strings.unicode_script  
tf.strings.unicode_script(
    input, name=None
)
 This operation converts Unicode code points to script codes corresponding to each code point. Script codes correspond to International Components for Unicode (ICU) UScriptCode values. See ICU project docs for more details on script codes. For an example, see the unicode strings guide on unicode scripts. Returns -1 (USCRIPT_INVALID_CODE) for invalid codepoints. Output shape will match input shape. Examples: 
tf.strings.unicode_script([1, 31, 38])
<tf.Tensor: shape=(3,), dtype=int32, numpy=array([0, 0, 0], dtype=int32)>

 


 Args
  input   A Tensor of type int32. A Tensor of int32 Unicode code points.  
  name   A name for the operation (optional).   
 


 Returns   A Tensor of type int32.