Package: seqio
SeqIOTask-based datasets, preprocessing, and evaluation for sequence modelsGo toSeqIO ReadTheDocs Documentation Page.OverviewSeqIOis a library for processing sequential data to be fed into downstream
sequence models. It usestf.data.Datasetto create scalable data pipelines but requires minimal use of TensorFlow. In
particular, with one line of code, the returned dataset can be transformed to a
numpy iterator and hence it is fully compatible with other frameworks such asJAXorPyTorch.SeqIO assumes that the dataset is a sequence. Modalities such as text or audio
are naturally supported. Images are supported as long as they are represented as
sequences (e.g.,Image GPT).SeqIO is a refactor of thet5.datalibrary used (in conjunction with theMesh TensorflowTransformer
implementation) to train the T5 models introduced inExploring the Limits of
Transfer Learning with a Unified Text-to-Text
Transformer.If you have usedt5.datain the past and want to know how SeqIO differs,
please readthis section.InstallationFrom PypipipinstallseqioFrom Sourcegitclonehttps://github.com/google/seqio.gitcdseqio
pipinstall-e.Usage TutorialAt a high level, we use SeqIO with the following steps.Define aTask(and optionally aMixture).Define (or use an existing) aFeatureConverterbased on the model
architecture.Use the top-level functionseqio.get_datasetto obtain thetf.data.Datasetinstance.We will look at each of these steps in detail.Defining aTaskThe most important class in SeqIO is theTask. It is an abstraction that
combines:a rawdata sourceone or morepreprocessingstepsavocabularyto tokenize/detokenize each preprocessed feature for the
modelapostprocessorto convert detokenized model outputs into a format for
evaluationone or moremetricsto evaluate withOftentimes aTasklines up with a common benchmark. In this tutorial, we useWMT 19 English-Germanmachine translation task. In the end, ourTaskwill look like this:seqio.TaskRegistry.add("wmt19_ende",seqio.TfdsDataSource(tfds_name="wmt19_translate/de-en:1.0.0"),preprocessors=[functools.partial(translate,source_language='en',target_language='de'),seqio.preprocessors.tokenize,seqio.preprocessors.append_eos],output_features={'inputs':seqio.Feature(seqio.SentencePieceVocabulary('/path/to/inputs/vocab'),add_eos=False,dtype=tf.int32),'targets':seqio.Feature(seqio.SentencePieceVocabulary('/path/to/targets/vocab'),add_eos=True,dtype=tf.int32),},metric_fns=[bleu])We typically add theTaskto the global registry when we define it (as shown
above) to make it easier to use with model configs and flags. Thus, it  must
have a unique string name ("wmt19_ende"in this case). Note, however, that
you may also instantiate aseqio.Taskdirectly without adding it to the
registry, if desired.We'll now break down each part of the task definition.Data SourceData sources are the first step in your pipeline, providing a way to load raw
data in many formats as atf.data.Dataset.
All data sources are subclasses of theDataSourcebase class and are defined
indataset_providers.Existing implementations include:TfdsDataSourcefor loading examples fromTensorFlow Datasets.TextLineDataSourcefor loading examples from text files (e.g., tsv).TFExampleDataSourcefor loadingtf.train.Exampleprotos from a file (e.g. aTFRecordfile.)FunctionDataSourcefor providing an custom function that returns atf.data.Dataset.In our example, we are using theTfdsDataSource. We specify the name of the
WMT dataset in TFDS
("wmt19_translate"),
the specific config for the language pair that excludes the context for the open
domain setting ("de-en"), and the version number ("1.0.0").Output FeaturesTheoutput_featuresfield expects a dictionary that maps string feature names
toseqio.Featureobjects. This defines what theTaskis expected to produce
in its output examples. The output examplesmaycontain additional fields, but
theymustcontain these fields in the specified format or exceptions will be
raised.EachFeatureincludes:Avocabulary, which must subclassseqio.Vocabulary,
to specify how the feature can be tokenized and detokenized. You may useseqio.PassThroughVocabularyif tokenization is not necessary.add_eos, which specifies whether the feature should end with the
vocabulary's EOS token.The outputdtypewhich must be atf.dtypes.DType.Note:specifying these options onFeaturedoes not by itself ensure the
proper transformations are applied -- you must also include the necessary
preprocessors.Thetasks used in T5all produce "inputs" and "targets" features to be
consumed by the text-to-text model. For a decoder-only language model, only a
single feature (e.g., "targets") would be necessary. Nevertheless, SeqIO is
flexible enough to generate arbitrary output features what will be converted
into model features by theFeatureConverterlater in the
pipeline.PreprocessorsPreprocessors are functions that transform onetf.data.Datasetinto a newtf.data.Dataset. Typically this involves executing amapover the given
dataset. The preprocessors provided to theTaskwill be executed sequentially.As an example, let's look at the previously undefinedtranslatefrom the
"wmt19_ende" example above.deftranslate(dataset:tf.data.Dataset,source_language:str,target_language:str)->tf.data.Dataset:def_translate(ex:Mapping[str,tf.Tensor])->Mapping[str,tf.Tensor]:"""Convert a translation example to a text2text pair.For example, say the dataset returns examples of this format:{'de': 'Das ist gut.', 'en': 'That is good.'}If source_language = 'de', target_language = 'en', then the outputs willhave the format:{'inputs': 'translate de to en: Das ist gut.','targets': 'That is good.'}Args:ex: an example to process.source_language: source language code (e.g. 'en') to translate from.target_language: target language code (e.g. 'de') to translate to.Returns:A preprocessed example with the format listed above."""src_str=f'translate{source_language}'tgt_str=f' to{target_language}: 'return{'inputs':tf.strings.join([src_str,tgt_str,ex[source_language]]),'targets':ex[target_language],}returndataset.map(_translate,num_parallel_calls=tf.data.experimental.AUTOTUNE)The TFDS dataset provides the dataset where each example has the form:{'de': 'Das ist gut.', 'en': 'That is good.'}. We convert this to "inputs" and
"targets" with the appropriate prompt to inform the model of the task.A fewimportantnotes:When instantiating aTask, the preprocessor functions can have the
following arguments:dataset,output_features, andsequence_length.
The first (positional) dataset argument is always required. If an argument
namedoutput_featuresis provided, theoutput feature mappingwill be passed to the
preprocessor. Ifsequence_lengthis provided, a mapping from feature name
to itsmaximumfinal sequence length
(provided by the caller) will be
passed -- any sequences that are too long after preprocessing will be
automatically truncated. If a preprocessor function does have other
arguments, they must have default values or be bound (e.g., withfunctools.partialas used intranslate) before instantiating theTask.Mapping functions operate on and returntf.Tensors using TensorFlow
operations. This is more flexible than it may sound:AutomaticAutoGraphconversion allow you to write python control flow in your
transformations.tf.experimental.numpyprovides a numpy interface.tf.py_functionallows you to wrap arbitrary Python code. Note:tf.datapipelines
using this function can only be run in the python process where they
were defined, and performance is limited by the python GIL.Seetf.data.Datasetdocumentationfor more details.When callingmap, it is important toalwayssetnum_parallel_calls=tf.data.experimental.AUTOTUNEto avoid creating a
bottleneck. Theseqio.map_over_datasetdecorator helps enforce this as
follows.@seqio.map_over_datasetdeftranslate(ex:Mapping[str,tf.Tensor],source_language:str,target_language:str)->Mapping[str,tf.Tensor]:"""Convert a translation dataset to a text2text pair.For example, say the dataset returns examples of this format:{'de': 'Das ist gut.', 'en': 'That is good.'}If source_language = 'de', target_language = 'en', then the outputs willhave the format:{'inputs': 'translate German to English: Das ist gut.','targets': 'That is good.'}Args:ex: an example to process.source_language: source language code (e.g. 'en') to translate from.target_language: target language code (e.g. 'de') to translate to.Returns:A preprocessed example with the format listed above."""src_str=f'translate{source_language}'tgt_str=f' to{target_language}: 'return{'inputs':tf.strings.join([src_str,tgt_str,ex[source_language]]),'targets':ex[target_language],}Note thattranslatetakes as input an individual example. Thenseqio.map_over_datasetdecorates it to a function that takes in atf.data.Datasetinstance.Stochastic operations must bestatelessif deterministic pipelines are needed. To get (optionally deterministic)
seeds for these operations, use theseqio.map_over_dataset(num_seeds=n)decorator. For example:defrandom_chunk(dataset:tf.data.Dataset,sequence_length:Mapping[str,int])->tf.data.Dataset:"""Takes a random chunk out of each feature with size `sequence_length`."""@seqio.map_over_dataset(num_seeds=1)deftake_chunk(ex:Mapping[str,tf.Tensor],seed)->Mapping[str,tf.Tensor]:new_ex={}fork,vinex.items():ifkinsequence_length:length=sequence_length[k]start_idx=tf.random.stateless_uniform((),seed,0,tf.size(v)-(length+1))new_ex[k]=v[start_idx:start_idx+length]else:new_ex[k]=vreturnnew_exreturntake_chunk(dataset)Ifnum_seeds > 1, the arg will instead be calledseedsand will contain
a sequence of seeds.In our "wmt_19_ende" task, we also use the predefined preprocessorsseqio.preprocessors.tokenizeandseqio.preprocessors.append_eos. The former
uses eachFeature.vocabularyto tokenize it, and the the latter appendsFeature.vocabulary.eos_idto the feature if theFeature.add_eosis True. Seepreprocessors.pyfor
their implementations and other useful preprocessors.PostprocessorDuring evaluation, the model outputs are first detokenized using the output
feature vocabulary. Before passing these predictions to the metric functions,
they can be run through a Python postprocessing function, alongside the full
input example. Similarly, the raw targets are run through this function before
being passed to the metrics. Since the postprocess function is used on both the
model output and the targets, it is passed anis_targetboolean in case the
behavior should be different. It is also passed the fully preprocessed example,
including fields that were excluded fromoutput_features.For the "wmt19_ende", we don't need any postprocessors. See "trivia_qa_open"
task in theAdvanced PostprocessingTaskfor
an example postprocessor.MetricsMetrics are functions that are passed (by theEvaluator) the
fully-materialized list of postprocessed model outputs (or scores) and targets
and return a mapping from string names toMetricValueobjects containing their
values. These are most commonly floating-point scalars, but may also be text,
images, audio, histograms, etc (seemetrics.pyfor the full list).The first argument of a metric function must always be calledtargets. If the
second argument of a metric function is calledpredictions, it will be passed
the decoded and detokenized model prediction. If it is calledscores, it will
be passed a list of log-likelihood scores for each example.If multiple metric functions are provided, they will all be used and their
returned mappings merged.Prediction MetricsPrediction metrics are computed using the postprocessed targets and model
outputs (predictions). The args must be namedtargetsandpredictions.Let's look at the metric function used for "wmt19_ende" task. A standard metric
for the translation task is BLEU and we usesacrebleuimplementation.defbleu(targets:Sequence[str],predictions:Sequence[str]):"""Computes BLEU score.Args:targets: list of strings or list of list of strings if multiple referencesare present.predictions: list of stringsReturns:bleu_score across all targets and predictions"""ifisinstance(targets[0],list):targets=[[xforxintarget]fortargetintargets]else:# Need to wrap targets in another list for corpus_bleu.targets=[targets]bleu_score=sacrebleu.corpus_bleu(predictions,targets,smooth_method="exp",smooth_value=0.0,force=False,lowercase=False,tokenize="intl",use_effective_order=False)return{"bleu":bleu_score.score}Score MetricsScore metrics are computed using the postprocessed targets and their
log-likelihood scores according to the model. The args must be namedtargetsandscores.defperplexity(targets:Sequence[str],scores:Sequence[int]):return{"perplexity":seqio.metrics.Scalar(np.exp(np.mean(scores)))}Defining aMixtureOnce you have multipleTasks added to theTaskRegistry, you can defineMixtures that will combine the examples from them according to some specified
rate. Examples will then be sampled from each task in proportion to its rate.As an example,Multilingual T5uses aMixtureof
per-languageTasks with tail languages up-weighted in the mixture.There are 3 ways to specify the tasks and their rates:Provide a rate along with each task's name (rates are normalized before
sampling). In this example, the rates provided are units of the final
mixture that come from the component tasks. Here, 1/(1+7) of the final
mixture will come from "task1".seqio.MixtureRegistry.add("mix1",[("task1",1),("task2",7)])Provide a constant default rate for some or all tasks, which will be used
when only the name is provided. The example below will produce identical
mixing rates as the previous one.seqio.MixtureRegistry.add("mix1",[("task1",0.5),"task2"],default_rate=3.5)Provide a function that generates the rate for each task at runtime. The
example below uses the providedseqio.mixing_rate_num_examples,
which uses the number of examples (computed duringoffline caching) as the rate for each task.seqio.MixtureRegistry.add("mix2",["task1","task2"],default_rate=seqio.mixing_rate_num_examples)You can also includeMixtures in yourMixture! For example, the following
task would contain 1/24 (from "mix1") + 1/3 "task1", 7/24 (from "mix1") of
"task2", and 1/3 "task3".seqio.MixtureRegistry.add("mix3",["mix1","task1","task3"],default_rate=1)If sampling without replacement is important for your task, you can achieve that
by using either deterministic tasks or using dataset checkpointing (and not
running more than an epoch) for a non-deterministic task. Otherwise, the mixture
may sample with replacement.Getting a Preprocessed DatasetNow that yourTask(and/orMixture) is defined, its primary functionality is
to use it to generate a dataset.You may first need to useseqio.get_mixture_or_task(mixture_or_task_name)to
access your dataset provider from the registry.After that, you can callget_datasetto build thetf.data.Dataset. For
example:dataset=seqio.get_mixture_or_task("mix1").get_dataset(sequence_length={"inputs":256,"targets":128},split="train",shuffle=True,num_epochs=1,shard_info=seqio.ShardInfo(index=0,num_shards=10),use_cached=False,seed=42)# Print the first 5 examples.for_,exinzip(range(5),dataset.as_numpy_iterator()):print(ex)Some notes on a few of the arguments:sequence_length: Anoptionalmapping from feature name tomaximumlength. Will be passed to the preprocessors with asequence_lengthargument. If notNone, the final example features will be truncated if
they exceed the specified length. Note that this value may be required to be
set if any of the preprocessors use thesequence_lengthargument and do
not handle theNonecase.num_epochs: The number of times to repeat the source dataset.
Preprocessing will be re-applied with new seeds to enable new samples from
stochastic steps. Note that if theCacheDatasetPlaceholderis included
(see below) preprocessing is only re-applied after that step.shard_info: An optional sharding specification for loading a deterministic
subset of the dataset. Loading will be most efficient if the number of
shards evenly divides the number of shards in the raw data source.use_cached: Specifies whether to load from a pre-cached task for increased
performance or to do the preprocessing on-the-fly. See thefollowing sectionfor details on how to cache
your task, which must be done before this can be set toTrue.seed: An optional seed to use for deterministic shuffling and (stateless)
stochastic ops. These operations will still be pseudorandom but will be
reproducible with the same seed. Set toNoneif determinism is not
desired.(Optional) Offline CachingFor improved performance at load time and to avoid redundant computations for
commonly used tasks, you can pre-cache yourTaskwith all or part of the
preprocessing done in advance of training; this partial preprocessing is
especially useful if the Task is stochastic and one wishes to cache the
deterministic operations while running the stochastic ones on the fly. Caching
stochastic SeqIO Mixtures in this way is not supported.The first step to doing so is to add aseqio.CacheDatasetPlaceholder(required=False)as one of the steps in your
preprocessing pipeline. All steps before the placeholder will be cached offline
and all steps after will be executed on the fly at load time. You may setrequired=Trueif you wantget_datasetto fail unlessuse_cached=True.Caveats:Any stochastic operations that you wish to be re-run whennum_epochs > 1or with a differentseedshouldgo after the placeholder since only a
single sample will be cached.Any preprocessing steps that use thesequence_lengthargumentmustcome
after theseqio.CacheDatasetPlaceholderpreprocessor since this is only
known at runtime, or an exception will be raised. If you wish to cache for a
specific sequence length, you can useseqio.experimental.add_fully_cached_task.Once yourTaskis registered, you can runcache_tasks_mainto execute the offline preprocessing, providing it with the module containing
your task definitions via the--module_importflag. For very large datasets,
it's recommended you run thisApache Beamscript on
a distributed framework likeGoogle Cloud DataFlow.Finally, you are ready to load the cached version of yourTask(orMixture)
containing it. You will need to add the path to the directory you passed to--output_cache_dirviaseqio.add_global_cache_dirs(["/my/cache/dir"]). Now
when you calltask_or_mixture.get_dataset(..., use_cached=True), the data will
be loaded from the cache directory instead of the raw data source.Feature ConvertersThe role ofTaskis to provide the dataset object with as little
model-specific features (e.g., generic "inputs" and "targets") while the Feature
Converters transform the model-agnostic features to model-specific features
(e.g., "encoder_input_tokens"). We refer to the former as "task features" and
the latter as "model features".Let's use machine translation (English to German) as a running example.The raw data consists of sentence pairs such as"That is good\tDas ist gut."A task registered toTask(e.g.,wmt_t2t_ende_v003)
reads these sentence pairs from the data source and applies a series ofpreprocessors.
One of the internal representations looks like{"inputs":"translate English to German: That is good.","targets":"Das ist gut."}The final output from theTaskis a tokenized version of the parallel
sentences. In the following toy example (the token ids do not correspond to the
above string example), the dataset consists of 2 examples.dataset=[{"inputs":[7,8,5],"targets":[3,9]},{"inputs":[8,4,9,3],"targets":[4]}]The format is in thetf.data.Dataset(i.e., each example is a dictionary with
"inputs" and "targets" fields.TheFeatureConverterthen takes this as an input and converts to the
model-specific features. In addition, the feature converter performs padding and
optionally packing (for model implementations that support it) for efficiency.
For example, let's assume that we are using the standard Transformer
architecture with an encoder and a decoder. The output of the feature converter
isconverted_dataset=[{"encoder_input_tokens":[7,8,5,1,8,4,9,3,1,0],"encoder_segment_ids":[1,1,1,1,2,2,2,2,2,0],"encoder_positions":[0,1,2,3,0,1,2,3,4,0],"decoder_target_tokens":[3,9,1,4,1,0,0],"decoder_input_tokens":[0,3,9,0,4,0,0],"decoder_loss_weights":[1,1,1,1,1,0,0],"decoder_positions":[0,1,2,0,1,0,0],"decoder_segment_ids":[1,1,1,2,2,0,0],}]In this case, two task examples are packed into one.*_segment_idand*_positionare the fields used to denote the membership and position of packed
token in the original sequence. The EOS ids (i.e., 1) are appended. In addition,
each fields is padded to the specified length.We will look at the details of this example in Encoder-decoder architecture:seqio.EncDecFeatureConvertersection.Feature converters provided out of the boxWe provide feature converters for three common architectures: encoder-decoder,
decoder-only and encoder-only. Here we describe how users can use the feature
converters for each of these architectures out of the box as a part of the SeqIO
library.In the SeqIO library, each architecture has a class defining how the task
features are converted to model features. Since these feature converters are
already implemented, it is straightforward to use them by providing the class as
afeature_converterargument of theseqio.get_datasetfunction. The
following sections show example usage ofseqio.get_dataset.Encoder-decoder architecture:seqio.EncDecFeatureConverterThis is the architecture of the original Transformer paper. For the
English-to-German translation task, the following function call retrieves thetf.data.Datasetobject with the model features.dataset:tf.data.Dataset=seqio.get_dataset(mixture_or_task_name="wmt_t2t_ende_v003",task_feature_lengths={"inputs":32,"targets":32},dataset_split="train",shuffle=True,feature_converter=seqio.EncDecFeatureConverter(pack=True))The resulting dataset object has the following 7 fieldsFeature nameExplanationencoder_input_tokensInput tokens to the encoder.encoder_positionsPosition index in the sequence before packing.encoder_segment_idsSequence membership before packing. Two positions withthe same positive integer mean that they belong to the same sequence beforepacking.decoder_input_tokensInput tokens to the decoder.decoder_target_tokensOutput tokens from the decoder.decoder_loss_weightsA weight on each position that can be used as a mask.decoder_positionsPosition index in the sequence before packing.decoder_segment_idsSame asencoder_segment_idsbut for decoder.Decoder-only architectureThis architecture consists of a single autoregressive stack, which we denote as
a "decoder".A decoder autoregressively produces an output sequence.
Therefore, it can be used as a standard language model if the task dataset has
only "targets" features, i.e., self-supervised. If the task dataset also has an
"inputs" field, e.g., supervised machine translation, the decoder can still be
used by concatenating the inputs and targets fields. SeeRaffel et al.
(2020), Section 3.2.1 for more detailed take
on this topic.We support both uses cases and refer to the former asstandard language modeland the latter asprefix language model. Each of these models is described
separately below.Note that we do not provide special features to denote how the dataset should be
consumed. For example, a Transformer-based fully autoregressive decoder has a
fully-causal self-attention layer. Since there are many ways of implementing the
masking pattern for such attention layer and, more importantly, SeqIO is not
limited to attention-based models, we leave it up to the model implementations
to apply the masking pattern. There is one exception, and we cover this in
the Prefix LM section below.A common use pattern is to pretrain a decoder model with the left-to-right
language modeling objective (unsupervised) usingseqio.LMFeatureConverterand
then fine-tune (supervised) usingseqio.PrefixLMFeatureConverter.Standard LMFor the standard language model, the task dataset only has "targets" field.
Therefore, the sequence length specification only needs to specify targets.dataset:tf.data.Dataset=seqio.get_dataset(mixture_or_task_name="standard_lm",task_feature_lengths={"targets":32},dataset_split="train",shuffle=True,feature_converter=seqio.LMFeatureConverter(pack=True))Note that "standard_lm" is not a registered task in the codebase. It is the
left-to-right language modeling task, i.e., predict the next token given the
previous tokens on some language corpus (e.g.,C4).The output dataset has the following model features.Feature nameExplanationdecoder_target_tokensOutput tokens from the decoderdecoder_input_tokensInput tokens to the decoderdecoder_loss_weightsBinary mask to indicate where the loss should be takendecoder_positionsPosition index in the sequence before packingdecoder_segment_idsSequence membership before packing. Two positions withthe same positive integer mean that they belong to the same sequence beforepacking.Thedecoder_target_tokensis a shifted version ofdecoder_input_tokensfor
the standard teacher-forced autoregressive training.Prefix LM:seqio.PrefixLMFeatureConverterIf the input dataset has a notion of "inputs" and "targets", we can concatenate
them so that we can still use a single stack decoder. Therefore, the output only
contains "targets" just like standard LM case.We use the same toy example for English-to-German translation task as a running
example:{"inputs": "translate English to German: That is good.",
 "targets": "Das ist gut."}To be consumed by the decoder-only stack,seqio.PrefixLMFeatureConverterconcatenates them form the new "targets". Consider 2-layer decoder architecture
whose activations are shown belowThat  is  good <EOS> Das ist gut <EOS>
 |    |    |    |    |   |    |   |
 u1   u2   u3   u4   u5  u6   u7  u8
 |    |    |    |    |   |    |   |
 v1   v2   v3   v4   v5  v6   v7  v8
 |    |    |    |    |   |    |   |
<BOS> That is  good <EOS> Das ist gutLet us denote the first layer's activation in theith position asvi.
Similarly, letuidenote the activation of the second layer in theith
position.For attention-based sequence models such as Transformer decoders, the
self-attention layer is used to encode contextualized representation of the
sequence. At a given layer, each position's representation is computed as a
function of the representations of the tokensbeforeits position in the
previous layer.Referring to the toy example, when computingu2with fully-causal masking, we
do not usev3. This results in a representationu2of the word "is" that
does not take into account the word "good", which is unnecessarily limiting.For Prefix LM, this issue is resolved by having the fully visible masking
pattern for the inputs portion only. For example, when computingu2,v1,v2,v3,v4andv5are all visible and taken into account. For the tokens
in the "targets" of theTaskdataset, we use the causal masking. For example,
when computingu6, allvifori <= 6are taken into account but notv7.Why is `v5` included in the inputs attention pattern?In the same translation example, we note that when computing `u2`, the
  activation corresponding to the position where \ token was input (i.e.,
  `v5`) was visible. This doesn't count as "cheating" because the model doesn't
  see the next word "Das". This can provide additional context in building the
  representation for "good". In this case, `u4` has the context that "good" is
  the last word in the sentence.seqio.PrefixLMFeatureConverterprovides a featuredecoder_causal_attentionto encode this information. For the above example, we havedecoder_causal_attention = [1, 1, 1, 1, 1, 0, 0, 0]indicating that the non-causal attention can be applied to the first five
positions. Note that this feature seems trivial, but for a packed dataset
the inputs and targets boundary are more nuanced.A final consideration for the prefix LM is that because we concatenate "inputs"
and "targets", which tokens are used for the loss computation is a modeling
decision. For example, we can penalize the models only for the "targets" tokens
or we may choose to penalize building the representation for "inputs" tokens.
This is controlled byloss_on_targets_onlyargument (defaults toTrue) toseqio.PrefixLMFeatureConverterconstructor. In the above example, we would getdecoder_loss_weights = [0, 0, 0, 0, 1, 1, 1, 1]This indicates that the last 4 positions are used for the loss computation.To get the dataset with prefix LM features, we can usedataset:tf.data.Dataset=seqio.get_dataset(mixture_or_task_name="wmt_t2t_ende_v003",task_feature_lengths={"inputs":32,"targets":32},dataset_split="train",shuffle=True,feature_converter=seqio.PrefixLMFeatureConverter(pack=True,loss_on_targets_only=True))The resulting features have length 64 because it concatenates inputs and targets
each with length 32.The output dataset has the following model features. Note that the only
additional feature isdecoder_causal_attention.Feature nameExplanationdecoder_target_tokensOutput tokens from the decoderdecoder_input_tokensInput tokens to the decoderdecoder_loss_weightsBinary mask to indicate where the loss should betakendecoder_positionsPosition index in the sequence before packingdecoder_segment_idsSequence membership before packing. Two positions withthe ` same positive integer mean that they belong to the same sequence beforepacking.decoder_causal_attentionBinary mask denoting which tokens are in thenon-causal masking region.Encoder-only architectureLike decoder-only architecture, this one is a single stack, but not
autoregressive.One notable assumption is that the inputs and targets arealigned, i.e., they
have the same sequence length andith position in the targets correspond to
the output representation of theith token in the inputs.A common model using encoder-only architecture isBERT. We provideEncoderfeature converter
class to support the Masked Language Modeling (MLM) objective from BERT.We assume that a unique sentinel such as[MASK]token is used to mask some
fraction of the input text and the task is to recover the original text.
Therefore, the "targets" is naturally defined as the original text whereas
"inputs" are the masked text.Encoder-only models are often used for classification tasks. In BERT, a special
token[CLS]is prepended to the input sequence. The last layer's activation
corresponding to this sentinel token is the contextualized representation of the
sequence. We assume that such "classification" sentinel is prepended.Consider the following example for the MLM task. The input dataset has two
examples, which is packed to one example. We assume thatmask_id = 9and the[CLS]token has id of 8.dataset=[{"inputs":[8,9,9,3,4],"targets":[8,7,4,3,4]},{"inputs":[8,3,9],"targets":[8,3,6]}]converted_dataset={"encoder_input_tokens":[8,9,9,3,4,1,8,3,9,1,0],"encoder_target_tokens":[8,7,4,3,4,1,8,3,6,1,0],"encoder_segment_ids":[1,1,1,1,1,1,2,2,2,2,0],"encoder_positions":[0,1,2,3,4,5,0,1,2,3,0],"encoder_loss_weights":[0,1,1,0,0,0,0,0,1,0,0],}Note that the packed sequence has[CLS]token at the beginning of each
sequences. Also note that the loss is taken only on the masked position.To use the pre-definedEncoderFeatureConverter, providemask_idas an
argument.dataset:tf.data.Dataset=seqio.get_dataset(mixture_or_task_name="some mlm task",task_feature_lengths={"inputs":32,"targets":32},dataset_split="train",shuffle=True,feature_converter=seqio.EncoderFeatureConverter(pack=True,mask_id=9))The resulting dataset object has the following 5 fieldsFeature nameExplanationencoder_input_tokensInput tokens to the encoderencoder_positionsPosition index in the sequence before packingencoder_segment_idsSequence membership before packing. Two positions withthe ` same positive integer mean that they belong to the same sequence beforepacking.encoder_target_tokensOutput tokens from the encoderencoder_loss_weightsBinary mask to indicate where the loss should be takenCustom architecturesFor a custom model architecture, you need to create a subclass ofFeatureConverterand override two methods_convert_featuresandget_model_feature_lengthsto define how task features are mapped to the model
features, including the length relationships. The existing feature converters
(e.g.,seqio.EncDecFeatureConverter) follow the same pattern, which can be a
useful starting point.EvaluationThe SeqIOEvaluatorclass provides a way to evaluate models on SeqIO Tasks
and Mixtures. For an interactive walkthrough of SeqIO evaluation, see theEvaluation Notebook.
The following is a deep-dive into theEvaluatorclass.An Evaluator instance can be created by passing a SeqIO Task or
Mixture, and additional eval params like feature converter, split, sequence
lengths, seed, etc. The Evaluator init callsget_datasetfor each Task to be
evaluated with the appropriate params, creating thetask_dataset, and invokes
the model-specific feature converter on thetask_datasetto create features
that can be passed to a model, calledmodel_dataset. Bothtask_datasetandmodel_datasetare stored in-memory so that the dataset can be reused across
multiple evaluations (e.g. on checkpoints from a training run). Both datasets
are enumerated so that even if the order of examples is changed during model
inference, the enumeration can be used to match model outputs to examples from
thetask_dataset.For Mixtures, each sub-Task is evaluated separately, regardless of mixing
rates, because in the context of eval benchmarks, Mixtures commonly refer to a
collection of Tasks belonging to that benchmark, each of which is evaluated
separately, e.g. SuperGLUE mixture.Once anEvaluatorinstance is created with a SeqIO Task or Mixture, a model
can be evaluated by callingevaluator.evaluate(...)and passing apredict_fnand/or apredict_with_aux_fnand/or ascore_fnto interact with the model.predict_fntakes themodel_datasetas input and outputs aSequence[(index, token_ids)]wheretoken_idsis the sequence of token ids generated by the
model for the input example whose index matchesindex. Therefore, even ifpredict_fnmixes the order of the examples during prediction, the order can be
corrected as long as the correct index for each example is maintained. A common
example is the multi-host setup where the evaluation dataset is split amongst
multiple hosts that independently make predictions and combine the results
during which the ordering can be mixed.predict_with_aux_fnis similar topredict_fn, except that it can also return a dictionary of auxiliary values
along with each sequence oftoken_ids, e.g. scores from the generated tokens.
Thescore_fntakes themodel_datasetas input and returns aSequence[(index, score)]wherescoreis the sequence of log likelihood
scores for the targets in the dataset. This simple interface allows users to
easily integrate the SeqIO evaluation flow with popular training frameworks in
TF and Jax.Corresponding to the model fns, users can configure three kinds of metric fns in
their Tasks, which are differentiated by their function signature. Metrics
computed on the outputs ofpredict_fn(andpredict_with_aux_fn) have the
signaturetargetsandpredictions(and optionallyaux_values), while
metrics computed on the outputs ofscore_fnhave the signaturetargetsandpredictions. TheEvaluatortakes care of calling the correct model fns and
metric fns during evaluation. Here is an example of a metric of each type.def sequence_accuracy(targets, predictions):
 seq_acc = 100 * np.mean([p == t for p, t in zip(predictions, targets)])
 return {"sequence_accuracy": seq_acc}

def log_likelihood(targets, scores):
 log_likelihood = np.mean([scipy.special.logsumexp(el) for el in scores])
 return {"log_likelihood": log_likelihood}There are 4 steps involved in the evaluation using predicted tokens:thepredict_fnorpredict_with_aux_fnreturns indices and output_tokens:Sequence[Tuple[int, Sequence[int]]], potentially with some auxiliary
values.output tokens are decoded byvocab.decodepostprocessors configured in Tasks are applied to the decoded output. These
are denoted as predictions.metric fns configured in Tasks are applied to the predictions and the cached
targets.There are 2 steps involved in the evaluation using scores:thescore_fnreturns indices and scores:Sequence[Tuple[int, Sequence[float]]]metric fns configured in Tasks is applied to the scores and the cached
targets.Training codebases like T5X provide integration with SeqIO evaluation to allow
evaluating checkpoints on SeqIO Tasks and Mixtures. SeeT5X Evalfor instructions.Differences fromt5.dataThe originalt5library introduced and implemented thet5.data.Taskabstraction for specifying preprocessing and evaluation metrics for text-to-text
tasks. When creating a task, users specify a source dataset of raw text, some
preprocessing steps, a vocabulary for tokenization, and evaluation metrics. The
fully-specified Task can then be used to pre-train or fine-tune a
encoder-decoder transformer model. However, the design included many baked-in
assumptions about the types of tasks users could specify.SeqIO removes some of the constraints of this abstraction:Inputs and outputs are no longer required to be strings (e.g., it may be
images or audio).Architectures other than the original encoder-decoder are supported (e.g.,
decoder-only language models like GPT or encoder-only models like BERT).Users can control at which stage of the pipeline offline caching occurs.Users can control when and where EOS tokens are added.Furthermore, SeqIO has been made more modular with respect to the Mesh
TensorFlow Transformer. This allows it to be used with other model
implementations with more consistency and much less code duplication.Advanced PostprocessingTaskTriviaQA (Closed-book, open-domain version)This version of TriviaQA was introduced inRoberts et al.
2020.seqio.TaskRegistry.add("trivia_qa_open",source=seqio.TfdsDataSource(tfds_name="trivia_qa/unfiltered.nocontext:1.1.0",splits={"train":"train[:90%]","validation":"train[90%:]","test":"validation"}),preprocessors=[tqa_open_preprocessor,seqio.preprocessors.tokenize,seqio.preprocessors.append_eos,],output_features={"inputs":seqio.Feature(seqio.SentencePieceVocabulary("/path/to/inputs/vocab"),add_eos=False,dtype=tf.int32),"targets":seqio.Feature(seqio.SentencePieceVocabulary("/path/to/targets/vocab"),add_eos=True,dtype=tf.int32),},postprocess_fn=tqa_open_postprocessor,metric_fns=[tqa_metric])In this example, we are using theTfdsDataSource. We specify the name of the
TriviaQA dataset in TFDS
("trivia_qa"), the
specific config that excludes the context for the open domain setting
("unfiltered.nocontext"), and the version number ("1.1.0"). We also override
the default splits to match what is commonly used for the open domain setting.
Specifically, we set our "test" split to be the TFDS "validation" split, and
create a small pseudo-"validation" set by taking examples out of the TFDS
"train" split.The preprocessortqa_open_preprocessoris defined as follows.deftqa_open_preprocessor(dataset:tf.data.Dataset,prefix:str="trivia_qa question: ")->tf.data.Dataset:"""Convert TriviaQA dataset to open domain qa examples.The function takes the trivia_qa TFDS dataset and emits examples of theform:{"inputs": "trivia_qa question: What are the names of the Olsen Twins?""targets": "Mary-Kate and Ashley","answers": ["Mary-Kate and Ashley", "Ashley and Mary-Kate"]}Args:dataset: a tf.data.Dataset to process.prefix: str, prefix to prepend to the inputs.Returns:a tf.data.Dataset"""deftqa_map(ex):"""Map TriviaQA example to text-to-text example."""return{"inputs":prefix+ex["question"],"targets":ex["answer"]["value"],"answers":ex["answer"]["aliases"],}returndataset.map(tqa_map,num_parallel_calls=tf.data.experimental.AUTOTUNE)Or with theseqio.map_overdatasetdecorator, we havedeftqa_open_preprocessor(dataset:tf.data.Dataset,prefix:str="trivia_qa question: ")->tf.data.Dataset:@seqio.map_over_datasetdeftqa_map(ex:Mapping[str,tf.Tensor])->Mapping[str,tf.Tensor]:"""Map TriviaQA example to text-to-text example."""return{"inputs":prefix+ex["question"],"targets":ex["answer"]["value"],"answers":ex["answer"]["aliases"],}returntqa_map(dataset)Here we made a thin wrapper to emphasize that the function decorated byseqio.map_over_datasettakes in an instance oftf.data.Dataset. In practice,
this wrapper is not necessary.The postprocessor for this example istqa_open_postprocessor, which is defined
as follows:deftqa_open_postprocessor(output_or_target,example=None,is_target=False):"""Returns output as answer, or all answers if the full example is provided."""ifis_target:return[a.decode("utf-8")forainexample["answers"]]else:returnoutput_or_target.decode("utf-8")When processing the target, we ignoreoutput_or_target(equivalent toexample["targets"]) since it is just selecting a single answer intrivia_qa_open. Instead, we extract the full list of answers from the example
and convert them from bytes to text. When handling the model output, we simply
convert it to text from detokenized bytes.The metric functiontqa_metricis defined as:deftqa_metric(targets:Sequence[Sequence[str]],predictions:Sequence[str])->Mapping[str,seqio.metrics.MetricValueValue]:"""Computes official TriviaQA metrics.Args:targets: list of lists of stringspredictions: list of stringsReturns:dict with score_key: squad score across all targets and predictions"""iflen(targets)!=len(predictions):raiseValueError("Number of targets and predictions must match.")def_normalize_answer(text):"""Lower text and remove punctuation, articles and extra whitespace."""# Remove articles.text=re.sub(r"\b(a|an|the)\b"," ",s)# Remove punctuation.forpuncinstring.punctuation:text=text.replace(punc,'')# Normalize white spacetext=" ".join(s.split())returntext# Normalize answers before comparing.targets=[[_normalize_answer(t)fortinu]foruintargets]predictions=[_normalize_answer(p)forpinpredictions]em=np.mean([max(pred==gtforgtinground_truths)forpred,ground_truthsinzip(predictions,targets)])return{"exact_match":seqio.metrics.Scalar(em),}Citing SeqIOPlease use the following bibtex entry to cite SeqIO.@article{roberts2022t5x,
  url = {https://arxiv.org/abs/2203.17189},
  author = {Roberts, Adam and Chung, Hyung Won and Levskaya, Anselm and Mishra,
  Gaurav and Bradbury, James and Andor, Daniel and Narang, Sharan and Lester,
  Brian and Gaffney, Colin and Mohiuddin, Afroz and Hawthorne, Curtis and
  Lewkowycz, Aitor and Salcianu, Alex and van Zee, Marc and Austin, Jacob and
  Goodman, Sebastian and Soares, Livio Baldini and Hu, Haitang and
  Tsvyashchenko, Sasha and Chowdhery, Aakanksha and Bastings, Jasmijn and
  Bulian, Jannis and Garcia, Xavier and Ni, Jianmo and Chen, Andrew and Kenealy,
  Kathleen and Clark, Jonathan H. and Lee, Stephan and Garrette, Dan and
  Lee-Thorp, James and Raffel, Colin and Shazeer, Noam and Ritter, Marvin and
  Bosma, Maarten and Passos, Alexandre and Maitin-Shepard, Jeremy and Fiedel,
  Noah and Omernick, Mark and Saeta, Brennan and Sepassi, Ryan and Spiridonov,
  Alexander and Newlan, Joshua and Gesmundo, Andrea},
  title = {Scaling Up Models and Data with $\texttt{t5x}$ and $\texttt{seqio}$},
  journal={arXiv preprint arXiv:2203.17189},
  year = {2022},
}