Repository: google-research/text-to-text-transfer-transformer

Issue:
Clarification about the splits used in the training time 
Hi there, 

Wanted get some clarification about what splits are used during fine-tuning. 
In particular, when calling the following command: 

```python 
    print(" >>>>>>> fine-tuning  . . . ")
    model.finetune(
        mixture_or_task_name="my_questions",
        pretrained_model_dir=PRETRAINED_DIR,
        finetune_steps=FINETUNE_STEPS
    )
```
and given the following task registry definition: 
```python 
t5.data.TaskRegistry.add(
    "my_task",
    # Supply a function which returns a tf.data.Dataset.
    dataset_fn=dataset_fn,
    splits=["split1", "split2", "split3"],
    # Supply a function which preprocesses text from the tf.data.Dataset.
    text_preprocessor=[dataset_preprocessor],
    # Use the same vocabulary that we used for pre-training.
    sentencepiece_model_path=t5.data.DEFAULT_SPM_PATH,
    # Lowercase targets before computing metrics.
    postprocess_fn=t5.data.postprocessors.lower_text,
    # We'll use accuracy as our evaluation metric.
    metric_fns=[t5.evaluation.metrics.accuracy],
    # Not required, but helps for mixing and auto-caching.
    num_input_examples=num_examples
)
```
does the code use *all* the splits specified in `splits` to fine-tune (regardless of what the names are)? 

Complete code here: https://gist.github.com/danyaljj/288b58427c50f9ef844a767276d430f7 



Discussion:
Thanks for the question as we may need to document this better. Only
'train' is used for training and 'validation' is used for evaluation. This
can be changed in the gin config if necessary however.

On Sat, Feb 22, 2020, 11:01 AM Daniel Khashabi <notifications@github.com>
wrote:

> Hi there,
>
> Wanted get some clarification about what splits are used during
> fine-tuning.
> In particular, when calling the following command:
>
>     print(" >>>>>>> fine-tuning  . . . ")
>     model.finetune(
>         mixture_or_task_name="my_questions",
>         pretrained_model_dir=PRETRAINED_DIR,
>         finetune_steps=FINETUNE_STEPS
>     )
>
> and given the following task registry definition:
>
> t5.data.TaskRegistry.add(
>     "my_task",
>     # Supply a function which returns a tf.data.Dataset.
>     dataset_fn=dataset_fn,
>     splits=["split1", "split2", "split3"],
>     # Supply a function which preprocesses text from the tf.data.Dataset.
>     text_preprocessor=[dataset_preprocessor],
>     # Use the same vocabulary that we used for pre-training.
>     sentencepiece_model_path=t5.data.DEFAULT_SPM_PATH,
>     # Lowercase targets before computing metrics.
>     postprocess_fn=t5.data.postprocessors.lower_text,
>     # We'll use accuracy as our evaluation metric.
>     metric_fns=[t5.evaluation.metrics.accuracy],
>     # Not required, but helps for mixing and auto-caching.
>     num_input_examples=num_examples
> )
>
> does the code use *all* the splits specified in splits to fine-tune
> (regardless of what the names are)?
>
> Complete code here:
> https://gist.github.com/danyaljj/288b58427c50f9ef844a767276d430f7
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/google-research/text-to-text-transfer-transformer/issues/102?email_source=notifications&email_token=AAIJV2B4VXH3FF6HMNGKDADREGHDPA5CNFSM4KZUSTNKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IPQH2UQ>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAIJV2CXBM6ZSGZIAPOP4OLREGHDPANCNFSM4KZUSTNA>
> .
>

Leaving this open so we can add optional kwargs to MtfModel

On Sat, Feb 22, 2020, 11:55 AM Adam Roberts <adarob@google.com> wrote:

> Thanks for the question as we may need to document this better. Only
> 'train' is used for training and 'validation' is used for evaluation. This
> can be changed in the gin config if necessary however.
>
> On Sat, Feb 22, 2020, 11:01 AM Daniel Khashabi <notifications@github.com>
> wrote:
>
>> Hi there,
>>
>> Wanted get some clarification about what splits are used during
>> fine-tuning.
>> In particular, when calling the following command:
>>
>>     print(" >>>>>>> fine-tuning  . . . ")
>>     model.finetune(
>>         mixture_or_task_name="my_questions",
>>         pretrained_model_dir=PRETRAINED_DIR,
>>         finetune_steps=FINETUNE_STEPS
>>     )
>>
>> and given the following task registry definition:
>>
>> t5.data.TaskRegistry.add(
>>     "my_task",
>>     # Supply a function which returns a tf.data.Dataset.
>>     dataset_fn=dataset_fn,
>>     splits=["split1", "split2", "split3"],
>>     # Supply a function which preprocesses text from the tf.data.Dataset.
>>     text_preprocessor=[dataset_preprocessor],
>>     # Use the same vocabulary that we used for pre-training.
>>     sentencepiece_model_path=t5.data.DEFAULT_SPM_PATH,
>>     # Lowercase targets before computing metrics.
>>     postprocess_fn=t5.data.postprocessors.lower_text,
>>     # We'll use accuracy as our evaluation metric.
>>     metric_fns=[t5.evaluation.metrics.accuracy],
>>     # Not required, but helps for mixing and auto-caching.
>>     num_input_examples=num_examples
>> )
>>
>> does the code use *all* the splits specified in splits to fine-tune
>> (regardless of what the names are)?
>>
>> Complete code here:
>> https://gist.github.com/danyaljj/288b58427c50f9ef844a767276d430f7
>>
>> —
>> You are receiving this because you are subscribed to this thread.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/google-research/text-to-text-transfer-transformer/issues/102?email_source=notifications&email_token=AAIJV2B4VXH3FF6HMNGKDADREGHDPA5CNFSM4KZUSTNKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IPQH2UQ>,
>> or unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/AAIJV2CXBM6ZSGZIAPOP4OLREGHDPANCNFSM4KZUSTNA>
>> .
>>
>

Thanks for the clarification (and the efforts to improve the code/documentation)! 

Just a thought about the thread, it's odd that for evaluation you have the option to specify a `split`: 
```
    model.eval(
        mixture_or_task_name="my_questions",
        checkpoint_steps="all",
        split="split3"
    )
```
, while I didn't find such an option for `finetune`. 

> Just a thought about the thread, it's odd that for evaluation you have the option to specify a split

This is because you might want to eval on different splits - validation or test, for example.

> while I didn't find such an option for finetune.

This is because we assume you always want to train and finetune on the "train" split.
This should have been auto-closed by #119 .
#119 isn't merged yet!

Resolving patch:
diff --git a/t5/data/utils.py b/t5/data/utils.py
--- a/t5/data/utils.py
+++ b/t5/data/utils.py
@@ -626,7 +626,7 @@ def get_dataset(
     if use_cached and not self.supports_caching:
       logging.warning(
           "Task '%s' does not support caching. Switching to on-the-fly "
-          "preprocessing.")
+          "preprocessing.", self.name)
       use_cached = False
     if use_cached:
       ds = self._get_cached_dataset(split, shuffle)
diff --git a/t5/models/mtf_model.py b/t5/models/mtf_model.py
--- a/t5/models/mtf_model.py
+++ b/t5/models/mtf_model.py
@@ -199,7 +199,8 @@ def estimator(self, vocabulary, init_checkpoint=None, disable_tpu=False):
         cluster=self._cluster,
         init_checkpoint=init_checkpoint)
 
-  def train(self, mixture_or_task_name, steps, init_checkpoint=None):
+  def train(self, mixture_or_task_name, steps, init_checkpoint=None,
+            split="train"):
     """Train the model on the given Mixture or Task.
 
     Args:
@@ -210,6 +211,7 @@ def train(self, mixture_or_task_name, steps, init_checkpoint=None):
       init_checkpoint: a string, if not None then read in variables from this
         checkpoint path when initializing variables. Will only initialize
         variables that appear both in the current graph and the checkpoint.
+      split: str, the mixture/task split to train on.
     """
     vocabulary = t5.data.get_mixture_or_task(
         mixture_or_task_name).get_vocabulary()
@@ -217,7 +219,7 @@ def train(self, mixture_or_task_name, steps, init_checkpoint=None):
         mesh_train_dataset_fn, mixture_or_task_name=mixture_or_task_name)
     utils.train_model(self.estimator(vocabulary, init_checkpoint), vocabulary,
                       self._sequence_length, self.batch_size, dataset_fn,
-                      steps, self._ensemble_inputs)
+                      steps, self._ensemble_inputs, dataset_split=split)
 
   def eval(self, mixture_or_task_name, checkpoint_steps=None, summary_dir=None,
            split="validation"):
@@ -234,7 +236,7 @@ def eval(self, mixture_or_task_name, checkpoint_steps=None, summary_dir=None,
         checkpoint from the model directory.
       summary_dir: str, path to write TensorBoard events file summaries for
         eval. If None, use model_dir/eval_{split}.
-      split: str, the split to evaluate on.
+      split: str, the mixture/task split to evaluate on.
     """
     if checkpoint_steps == -1:
       checkpoint_steps = _get_latest_checkpoint_from_dir(self._model_dir)
@@ -249,7 +251,7 @@ def eval(self, mixture_or_task_name, checkpoint_steps=None, summary_dir=None,
                      self._model_dir, dataset_fn, summary_dir, checkpoint_steps)
 
   def finetune(self, mixture_or_task_name, finetune_steps, pretrained_model_dir,
-               pretrained_checkpoint_step=-1):
+               pretrained_checkpoint_step=-1, split="train"):
     """Finetunes a model from an existing checkpoint.
 
     Args:
@@ -262,6 +264,7 @@ def finetune(self, mixture_or_task_name, finetune_steps, pretrained_model_dir,
       pretrained_checkpoint_step: int, checkpoint to initialize weights from. If
         -1 (default), use the latest checkpoint from the pretrained model
         directory.
+      split: str, the mixture/task split to finetune on.
     """
     if pretrained_checkpoint_step == -1:
       checkpoint_step = _get_latest_checkpoint_from_dir(pretrained_model_dir)
@@ -272,7 +275,8 @@ def finetune(self, mixture_or_task_name, finetune_steps, pretrained_model_dir,
 
     model_ckpt = "model.ckpt-" + str(checkpoint_step)
     self.train(mixture_or_task_name, checkpoint_step + finetune_steps,
-               init_checkpoint=os.path.join(pretrained_model_dir, model_ckpt))
+               init_checkpoint=os.path.join(pretrained_model_dir, model_ckpt),
+               split=split)
 
   def predict(self, input_file, output_file, checkpoint_steps=-1,
               beam_size=1, temperature=1.0,


Test patch: