Repository: google-research/text-to-text-transfer-transformer

Issue:
train multiple datasets using gpu
I tried to train using multiple datasets (my own tsv datasets).

If I use the following way:
```
t5_mesh_transformer  \
  --model_dir="${MODEL_DIR}" \
  --t5_tfds_data_dir="${DATA_DIR}" \
  --gin_file="dataset.gin" \
  --gin_param="utils.run.mesh_shape = 'model:1,batch:1'" \
  --gin_param="utils.run.mesh_devices = ['gpu:0']" \
  --gin_param="utils.run.train_dataset_fn = @t5.models.mesh_transformer.tsv_dataset_fn"
  --gin_param="tsv_dataset_fn.filename = 'gs:/path/to/tsv'"
  --gin_file="gs://t5-data/pretrained_models/small/operative_config.gin"
```
I can only train with **one single** tsv-file, not **multiple** files using **GPU.**

So I use this way to set the model:
```
model = t5.models.MtfModel(
    model_dir=MODEL_DIR,
    tpu=None,
    tpu_topology=None,
    model_parallelism=model_parallelism,
    batch_size=train_batch_size,
    sequence_length={"inputs": 128, "targets": 32},
    mesh_shape="model:1,batch:1",
    mesh_devices=["GPU:0"],
    learning_rate_schedule=0.003,
    save_checkpoints_steps=5000,
    keep_checkpoint_max=None,
    iterations_per_loop=100,
)
```
The second method worked last week. But when I tried to train again today, I got the following error:

> ValueError: Please provide a TPU Name to connect to.
  In call to configurable 'MtfModel' (<class 't5.models.mtf_model.MtfModel'>)

How can I solve this problem for training multiple tsvs using GPU?


Discussion:
Full stacktrace as follows:
> ---------------------------------------------------------------------------
> 
> ValueError                                Traceback (most recent call last)
> 
> <ipython-input-43-59584a2c7b99> in <module>()
>      17     save_checkpoints_steps=5000,
>      18     keep_checkpoint_max=None,
> ---> 19     iterations_per_loop=100,
>      20 )
> 
> 6 frames
> 
> /usr/local/lib/python3.6/dist-packages/gin/config.py in gin_wrapper(*args, **kwargs)
>    1076       scope_info = " in scope '{}'".format(scope_str) if scope_str else ''
>    1077       err_str = err_str.format(name, fn_or_cls, scope_info)
> -> 1078       utils.augment_exception_message_and_reraise(e, err_str)
>    1079 
>    1080   return gin_wrapper
> 
> /usr/local/lib/python3.6/dist-packages/gin/utils.py in augment_exception_message_and_reraise(exception, message)
>      47   if six.PY3:
>      48     ExceptionProxy.__qualname__ = type(exception).__qualname__
> ---> 49     six.raise_from(proxy.with_traceback(exception.__traceback__), None)
>      50   else:
>      51     six.reraise(proxy, None, sys.exc_info()[2])
> 
> /usr/local/lib/python3.6/dist-packages/six.py in raise_from(value, from_value)
> 
> /usr/local/lib/python3.6/dist-packages/gin/config.py in gin_wrapper(*args, **kwargs)
>    1053 
>    1054     try:
> -> 1055       return fn(*new_args, **new_kwargs)
>    1056     except Exception as e:  # pylint: disable=broad-except
>    1057       err_str = ''
> 
> /usr/local/lib/python3.6/dist-packages/t5/models/mtf_model.py in __init__(self, model_dir, tpu, tpu_job_name, tpu_zone, gcp_project, tpu_topology, model_parallelism, batch_size, sequence_length, model_type, layout_rules, mesh_shape, mesh_devices, autostack, learning_rate_schedule, keep_checkpoint_max, save_checkpoints_steps, optimizer, predict_fn, variable_filter, ensemble_inputs, iterations_per_loop)
>     162 
>     163     self._cluster = tf.distribute.cluster_resolver.TPUClusterResolver(
> --> 164         tpu if (tpu) else "", zone=tpu_zone, project=gcp_project)
>     165     self._tpu = tpu
>     166     self._tpu_job_name = tpu_job_name
> 
> /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/cluster_resolver/tpu_cluster_resolver.py in __init__(self, tpu, zone, project, job_name, coordinator_name, coordinator_address, credentials, service, discovery_url)
>     154         credentials=credentials,
>     155         service=service,
> --> 156         discovery_url=discovery_url)
>     157 
>     158     self._tpu = self._cloud_tpu_client.name()
> 
> /usr/local/lib/python3.6/dist-packages/tensorflow/python/tpu/client/client.py in __init__(self, tpu, zone, project, credentials, service, discovery_url)
>     115 
>     116     if tpu is None:
> --> 117       raise ValueError('Please provide a TPU Name to connect to.')
>     118 
>     119     self._tpu = _as_text(tpu)
> 
> ValueError: Please provide a TPU Name to connect to.
>   In call to configurable 'MtfModel' (<class 't5.models.mtf_model.MtfModel'>)
I'm running on the colab. I downgraded the version to 0.3.2 and cannot reproduce the above error anymore. But I have another error: 

> AlreadyExistsError: Resource __per_step_39/while_loop/while/ArithmeticOptimizer/AddOpsRewrite_add_73/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
> 	 [[{{node while_loop/while/ArithmeticOptimizer/AddOpsRewrite_add_73/tmp_var}}]]

Full stacktrace:

> /usr/local/lib/python3.6/dist-packages/mesh_tensorflow/transformer/utils.py in train_model(estimator, vocabulary, sequence_length, batch_size, train_dataset_fn, train_steps, ensemble_inputs, dataset_split)
>    1130     return dataset
>    1131 
> -> 1132   estimator.train(input_fn=input_fn, max_steps=train_steps)
>    1133 
>    1134 
> 
> /usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py in train(self, input_fn, hooks, steps, max_steps, saving_listeners)
>    3033     finally:
>    3034       rendezvous.record_done('training_loop')
> -> 3035       rendezvous.raise_errors()
>    3036 
>    3037   def evaluate(self,
> 
> /usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/error_handling.py in raise_errors(self, timeout_sec)
>     134       else:
>     135         logging.warn('Reraising captured error')
> --> 136         six.reraise(typ, value, traceback)
>     137 
>     138     for k, (typ, value, traceback) in kept_errors:
> 
> /usr/local/lib/python3.6/dist-packages/six.py in reraise(tp, value, tb)
>     701             if value.__traceback__ is not tb:
>     702                 raise value.with_traceback(tb)
> --> 703             raise value
>     704         finally:
>     705             value = None
> 
> /usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py in train(self, input_fn, hooks, steps, max_steps, saving_listeners)
>    3028           steps=steps,
>    3029           max_steps=max_steps,
> -> 3030           saving_listeners=saving_listeners)
>    3031     except Exception:  # pylint: disable=broad-except
>    3032       rendezvous.record_error('training_loop', sys.exc_info())
> 
> /usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py in train(self, input_fn, hooks, steps, max_steps, saving_listeners)
>     368 
>     369       saving_listeners = _check_listeners_type(saving_listeners)
> --> 370       loss = self._train_model(input_fn, hooks, saving_listeners)
>     371       logging.info('Loss for final step: %s.', loss)
>     372       return self
> 
> /usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py in _train_model(self, input_fn, hooks, saving_listeners)
>    1159       return self._train_model_distributed(input_fn, hooks, saving_listeners)
>    1160     else:
> -> 1161       return self._train_model_default(input_fn, hooks, saving_listeners)
>    1162 
>    1163   def _train_model_default(self, input_fn, hooks, saving_listeners):
> 
> /usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py in _train_model_default(self, input_fn, hooks, saving_listeners)
>    1193       return self._train_with_estimator_spec(estimator_spec, worker_hooks,
>    1194                                              hooks, global_step_tensor,
> -> 1195                                              saving_listeners)
>    1196 
>    1197   def _train_model_distributed(self, input_fn, hooks, saving_listeners):
> 
> /usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py in _train_with_estimator_spec(self, estimator_spec, worker_hooks, hooks, global_step_tensor, saving_listeners)
>    1492       any_step_done = False
>    1493       while not mon_sess.should_stop():
> -> 1494         _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
>    1495         any_step_done = True
>    1496     if not any_step_done:
> 
> /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py in run(self, fetches, feed_dict, options, run_metadata)
>     752         feed_dict=feed_dict,
>     753         options=options,
> --> 754         run_metadata=run_metadata)
>     755 
>     756   def run_step_fn(self, step_fn):
> 
> /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py in run(self, fetches, feed_dict, options, run_metadata)
>    1257             feed_dict=feed_dict,
>    1258             options=options,
> -> 1259             run_metadata=run_metadata)
>    1260       except _PREEMPTION_ERRORS as e:
>    1261         logging.info(
> 
> /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py in run(self, *args, **kwargs)
>    1358         raise six.reraise(*original_exc_info)
>    1359       else:
> -> 1360         raise six.reraise(*original_exc_info)
>    1361 
>    1362 
> 
> /usr/local/lib/python3.6/dist-packages/six.py in reraise(tp, value, tb)
>     701             if value.__traceback__ is not tb:
>     702                 raise value.with_traceback(tb)
> --> 703             raise value
>     704         finally:
>     705             value = None
> 
> /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py in run(self, *args, **kwargs)
>    1343   def run(self, *args, **kwargs):
>    1344     try:
> -> 1345       return self._sess.run(*args, **kwargs)
>    1346     except _PREEMPTION_ERRORS:
>    1347       raise
> 
> /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py in run(self, fetches, feed_dict, options, run_metadata)
>    1416         feed_dict=feed_dict,
>    1417         options=options,
> -> 1418         run_metadata=run_metadata)
>    1419 
>    1420     for hook in self._hooks:
> 
> /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py in run(self, *args, **kwargs)
>    1174 
>    1175   def run(self, *args, **kwargs):
> -> 1176     return self._sess.run(*args, **kwargs)
>    1177 
>    1178   def run_step_fn(self, step_fn, raw_session, run_with_hooks):
> 
> /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
>     954     try:
>     955       result = self._run(None, fetches, feed_dict, options_ptr,
> --> 956                          run_metadata_ptr)
>     957       if run_metadata:
>     958         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)
> 
> /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
>    1178     if final_fetches or final_targets or (handle and feed_dict_tensor):
>    1179       results = self._do_run(handle, final_targets, final_fetches,
> -> 1180                              feed_dict_tensor, options, run_metadata)
>    1181     else:
>    1182       results = []
> 
> /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
>    1357     if handle is None:
>    1358       return self._do_call(_run_fn, feeds, fetches, targets, options,
> -> 1359                            run_metadata)
>    1360     else:
>    1361       return self._do_call(_prun_fn, handle, feeds, fetches)
> 
> /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py in _do_call(self, fn, *args)
>    1382                     '\nsession_config.graph_options.rewrite_options.'
>    1383                     'disable_meta_optimizer = True')
> -> 1384       raise type(e)(node_def, op, message)
>    1385 
>    1386   def _extend_graph(self):
> 
> AlreadyExistsError: Resource __per_step_39/while_loop/while/ArithmeticOptimizer/AddOpsRewrite_add_73/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
> 	 [[{{node while_loop/while/ArithmeticOptimizer/AddOpsRewrite_add_73/tmp_var}}]]

Can you share your colab for me to lookat?
https://colab.research.google.com/drive/1PG2ulIu1QiHhYmex19K1poLy01DRnbmg?usp=sharing
here is the link
Why did you downgrade? I'm not sure I expect GPU to work on 0.3.2.
I changed the version and ran again (version 0.5.0 now). Then I had this error: 

> ValueError: Please provide a TPU Name to connect to.
>   In call to configurable 'MtfModel' (<class 't5.models.mtf_model.MtfModel'>)


I just looked at the code and I think I have a fix. I will make a PR in a bit for you to test against.
Please try installing this PR with `!pip install git+https://github.com/google-research/text-to-text-transfer-transformer.git@refs/pull/223/merge`
My code stopped so far when I tried to add the data to TaskRegister:

> ValueError: No Task or Mixture found with name: review_context_free
>   In call to configurable 'get_sentencepiece_model_path' (<function get_sentencepiece_model_path at 0x7f3cd1a4f620>)
How can I pass the mixture or task vocabulary directly to the Mesh TensorFlow Transformer?
I tried to add in my operative_config.gin:
`mesh_train_dataset_fn.vocabulary = "gs://t5-data/vocabs/cc_all.32000/sentencepiece.model"`

But still have this error.

> ValueError: No Task or Mixture found with name: review_context_free
>   In call to configurable 'get_sentencepiece_model_path' (<function get_sentencepiece_model_path at 0x7faa5ff0f620>)
I solve the above error by manually adding the output_features as follows:

```
t5.data.TaskRegistry.add(
    name = "review_context_free",
    dataset_fn = review_dataset_fn,
    splits=["train","validation"],
    text_preprocessor = [trivia_preprocessor],
    #sentencepiece_model_path=t5.data.DEFAULT_SPM_PATH,
    postprocess_fn=t5.data.postprocessors.lower_text,
    metric_fns=[t5.evaluation.metrics.accuracy],
    output_features={
        "inputs": t5.data.utils.Feature(vocabulary = t5.data.SentencePieceVocabulary(t5.data.DEFAULT_SPM_PATH)),
        "targets": t5.data.utils.Feature(vocabulary = t5.data.SentencePieceVocabulary(t5.data.DEFAULT_SPM_PATH)),
    },
    num_input_examples=num_review_examples
)
```

And my model begins to train now. Thanks @adarob !
Great!

Resolving patch:
diff --git a/t5/models/mtf_model.py b/t5/models/mtf_model.py
--- a/t5/models/mtf_model.py
+++ b/t5/models/mtf_model.py
@@ -160,7 +160,7 @@ def __init__(
     self._iterations_per_loop = iterations_per_loop
 
     self._cluster = tf.distribute.cluster_resolver.TPUClusterResolver(
-        tpu if (tpu) else "", zone=tpu_zone, project=gcp_project)
+        tpu, zone=tpu_zone, project=gcp_project) if tpu else None
     self._tpu = tpu
     self._tpu_job_name = tpu_job_name
     self._estimator = None
diff --git a/t5/version.py b/t5/version.py
--- a/t5/version.py
+++ b/t5/version.py
@@ -18,4 +18,4 @@
 Stored in a separate file so that setup.py can reference the version without
 pulling in all the dependencies in __init__.py.
 """
-__version__ = '0.5.0'
+__version__ = '0.5.1'


Test patch: