Repository: google-research/text-to-text-transfer-transformer

Issue:
Training T5 from scratch on a new language?
Hi I was wondering if there are any guidelines or documentation as to pre-training T5 from scratch (not just to any particular downstream task) in a new language?

Also is it possible to do the same with PyTorch under the current framework?

Please let me know if this is not the right place to discuss this, thank you!


Discussion:
I did refer to issue #172 for this, but that just seems to be initializing it for fine-tuning on a specific task?
Check this:
https://github.com/google-research/google-research/tree/master/t5_closed_book_qa

I'm sorry if I am missing something, but isn't this training specifically for the QA task?
Check also this:
https://github.com/google-research/text-to-text-transfer-transformer/issues/253

Hi, thanks for the link. Did you guys perfork unsupervised pre-training these models from scratch on this dataset?

Any ideas on how this would shift to a new language? (Also if there is any chance I could make it work with Pytorch)

Sorry about the series of questions, but thanks for the help!

I pretrained t5 base and small on Malay language (Malaysia), all steps in here, https://github.com/huseinzol05/Malaya/tree/master/pretrained-model/t5

Step to generate sentencepiece for this T5 model, https://github.com/huseinzol05/Malaya/tree/master/pretrained-model/preprocess, no 4.

I increased input and output length to 1024 because our use case to summarize long texts (https://malaya.readthedocs.io/en/latest/Abstractive.html#load-t5) and generate long texts given important contexts (https://malaya.readthedocs.io/en/latest/Generator.html#load-t5).

You can found our t5 model in huggingface, https://huggingface.co/huseinzol05/t5-base-bahasa-cased

I never seen a powerful seq2seq as T5.
@huseinzol05 I understand it is in tensorflow, but still is extremely helpful and very close to what I am looking for, thanks for taking the time to share the details!

@ritvik1512  I was able to get the pytorch model going and here's my team's notebook: 

https://colab.research.google.com/github/jameschartouni/arabic_translation/blob/google-t5/Model_1.ipynb#scrollTo=UL5yLXs4YJw7

but I'm still trying to figure out how the operative_config.gin should be adjusted when pre-training the  pytorch model.

I know that you guys have only put out the API for fine-tuning purposes only but is there a way to correctly set up operative_config.gin for pretraining the HfPyTorch model?
Hey folks. I'm going to work on setting up the unsupervised task to not require the use of gin.
PTAL at #274 and see if it helps.
@traumasv thanks for sharing the notebook! If I get this correctly, you guys are trying to implement translation for Arabic from scratch?
Thanks @adarob! If it works well with @traumasv's task, I will be trying to implement the same

Resolving patch:
diff --git a/t5/data/tasks.py b/t5/data/tasks.py
--- a/t5/data/tasks.py
+++ b/t5/data/tasks.py
@@ -34,6 +34,7 @@
 }
 
 # ==================================== C4 ======================================
+# Configurable tasks used for comparisons in Raffel et al., 2019.
 _c4_config_suffixes = ["", ".noclean", ".realnewslike", ".webtextlike"]
 for config_suffix in _c4_config_suffixes:
   TaskRegistry.add(
@@ -47,6 +48,80 @@
       output_features=DEFAULT_OUTPUT_FEATURES,
       metric_fns=[])
 
+# Final pretraining task used in Raffel et al., 2019.
+TaskRegistry.add(
+    "c4_v220_span_corruption",
+    TfdsTask,
+    tfds_name="c4/en:2.2.0".format(config=config_suffix),
+    text_preprocessor=functools.partial(
+        preprocessors.rekey, key_map={"inputs": None, "targets": "text"}),
+    token_preprocessor=[
+        functools.partial(
+            preprocessors.select_random_chunk,
+            feature_key="targets",
+            max_length=65536
+        ),
+        functools.partial(
+            preprocessors.reduce_concat_tokens,
+            feature_key="targets",
+            batch_size=128
+        ),
+        functools.partial(
+            preprocessors.split_tokens,
+            feature_key="targets",
+            min_tokens_per_segment=None,
+            max_tokens_per_segment=preprocessors.random_spans_helper(
+                extra_tokens_per_span_inputs=1,
+                extra_tokens_per_span_targets=1,
+                inputs_length=512,
+                mean_noise_span_length=3.0,
+                noise_density=0.15
+            )[0]
+        ),
+        functools.partial(
+            preprocessors.denoise,
+            inputs_fn=preprocessors.noise_span_to_unique_sentinel,
+            targets_fn=preprocessors.nonnoise_span_to_unique_sentinel,
+            noise_density=0.15,
+            noise_function=functools.partial(
+                preprocessors.random_spans_noise_mask,
+                mean_noise_span_length=3.0
+            )
+        )
+    ],
+    output_features=DEFAULT_OUTPUT_FEATURES,
+    metric_fns=[])
+
+# Baseline pretraining task used in Raffel et al., 2019.
+TaskRegistry.add(
+    "c4_v220_iid_denoising",
+    TfdsTask,
+    tfds_name="c4/en:2.2.0".format(config=config_suffix),
+    text_preprocessor=functools.partial(
+        preprocessors.rekey, key_map={"inputs": None, "targets": "text"}),
+    token_preprocessor=[
+        functools.partial(
+            preprocessors.select_random_chunk,
+            feature_key="targets",
+            max_length=65536
+        ),
+        functools.partial(
+            preprocessors.reduce_concat_tokens,
+            feature_key="targets",
+            batch_size=128
+        ),
+        preprocessors.split_tokens_to_inputs_length,
+        functools.partial(
+            preprocessors.denoise,
+            inputs_fn=preprocessors.noise_span_to_unique_sentinel,
+            targets_fn=preprocessors.nonnoise_span_to_unique_sentinel,
+            noise_density=0.15,
+            noise_function=preprocessors.iid_noise_mask
+        )
+    ],
+    output_features=DEFAULT_OUTPUT_FEATURES,
+    metric_fns=[])
+
 # ================================ Wikipedia ===================================
 TaskRegistry.add(
     "wikipedia_20190301.en_v003_unsupervised",


Test patch: