Repository: autogluon/autogluon
Pull request: [CI] Split multimodal tutorial build into multiple usecases
*Issue #, if available:*

[Issue-2428](https://github.com/autogluon/autogluon/issues/2428)

*Description of changes:*
Split multimodal tutorial build into multiple use cases so that they can run in parallel

Sample [run](https://github.com/tonyhoo/ci-test/actions/runs/3571586899) after split in forked repo

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.


Problem statement:
[Post 0.6] Speed up AutoMM tutorial build
Currently as of v0.6 release, AutoMM tutorials take >90 minutes to run, accounting for ~75% of the wall-clock time of the CI. This is too long and makes development cumbersome.

It should be shorted to ideally wall-clock 30 minutes, and 45 minutes at most.

Hints from issue discussion:
Can we separate the tutorial build pipeline to build tutorial for each folder under "multimodal"?

- [text_prediction](https://github.com/awslabs/autogluon/tree/master/docs/tutorials/multimodal/text_prediction)
- [object_detection](https://github.com/awslabs/autogluon/tree/master/docs/tutorials/multimodal/object_detection)
- [multimodal_prediction](https://github.com/awslabs/autogluon/tree/master/docs/tutorials/multimodal/multimodal_prediction)
- [matching](https://github.com/awslabs/autogluon/tree/master/docs/tutorials/multimodal/matching)
- [image_prediction](https://github.com/awslabs/autogluon/tree/master/docs/tutorials/multimodal/image_prediction)
- [advanced_topics](https://github.com/awslabs/autogluon/tree/master/docs/tutorials/multimodal/advanced_topics)

Commit message: added index.rst on the root folder level when sub folder build enabled

Patch to review:
diff --git a/.github/workflow_scripts/build_doc.sh b/.github/workflow_scripts/build_doc.sh
index b203c086af98..7935493a25b5 100644
--- a/.github/workflow_scripts/build_doc.sh
+++ b/.github/workflow_scripts/build_doc.sh
@@ -4,6 +4,7 @@ function build_doc {
     GIT_REPO="$3"
     COMMIT_SHA="$4"
     PR_NUMBER="$5"  # For push events, PR_NUMBER will be empty
+    SUB_DOC=$6  # Can be empty
 
     source $(dirname "$0")/write_to_s3.sh
     source $(dirname "$0")/setup_mmcv.sh
@@ -13,6 +14,7 @@ function build_doc {
     setup_mmcv
     # only build for docs/$DOC
     rm -rf ./docs/tutorials/!($DOC)
+    if [[ -n $SUB_DOC ]]; then rm -rf ./docs/tutorials/"$DOC"/!(index.rst|$SUB_DOC); fi
     cd docs && rm -rf _build && d2lbook build rst
 
     COMMAND_EXIT_CODE=$?
diff --git a/.github/workflow_scripts/build_multimodal_tutorial.sh b/.github/workflow_scripts/build_multimodal_tutorial.sh
index 5d7dbbf8e684..c1f13d5b33a8 100644
--- a/.github/workflow_scripts/build_multimodal_tutorial.sh
+++ b/.github/workflow_scripts/build_multimodal_tutorial.sh
@@ -3,12 +3,13 @@
 set -ex
 shopt -s extglob
 
-BRANCH=$(basename $1)
-GIT_REPO=$2
-COMMIT_SHA=$3
-PR_NUMBER=$4  # For push events, PR_NUMBER will be empty
+SUB_DOC=$1
+BRANCH=$(basename $2)
+GIT_REPO=$3
+COMMIT_SHA=$4
+PR_NUMBER=$5  # For push events, PR_NUMBER will be empty
 
 source $(dirname "$0")/env_setup.sh
 source $(dirname "$0")/build_doc.sh
 
-build_doc multimodal $BRANCH $GIT_REPO $COMMIT_SHA $PR_NUMBER
\ No newline at end of file
+build_doc multimodal $BRANCH $GIT_REPO $COMMIT_SHA ${PR_NUMBER:-""} $SUB_DOC
\ No newline at end of file
diff --git a/.github/workflows/continuous_integration.yml b/.github/workflows/continuous_integration.yml
index 3e54225a28fa..3a7dfe3281e0 100644
--- a/.github/workflows/continuous_integration.yml
+++ b/.github/workflows/continuous_integration.yml
@@ -322,9 +322,11 @@ jobs:
           job-type: CI-GPU
           job-name: AutoGluon-BuildTabularPrediction
           command: chmod +x ./.github/workflow_scripts/build_tabular_prediction_tutorial.sh && ./.github/workflow_scripts/build_tabular_prediction_tutorial.sh '${{ env.BRANCH }}' '${{ env.GIT_REPO }}' '${{ env.SHORT_SHA }}' '${{ env.PR_NUMBER }}'
-  build_multimodal_tutorial:
+  build_multimodal_advanced_topics_tutorial:
     needs: [test_common, test_core, test_features, test_tabular, test_eda, test_fair, test_text, test_multimodal_predictor, test_multimodal_others, test_vision, test_timeseries, test_install]
     runs-on: ubuntu-latest
+    env:
+      SUB_DOC: advanced_topics
     steps:
       - name: Checkout repository
         uses: actions/checkout@v2
@@ -335,7 +337,88 @@ jobs:
         with:
           job-type: CI-GPU
           job-name: AutoGluon-BuildMultimodal
-          command: chmod +x ./.github/workflow_scripts/build_multimodal_tutorial.sh && ./.github/workflow_scripts/build_multimodal_tutorial.sh '${{ env.BRANCH }}' '${{ env.GIT_REPO }}' '${{ env.SHORT_SHA }}' '${{ env.PR_NUMBER }}'
+          command: chmod +x ./.github/workflow_scripts/build_multimodal_tutorial.sh && ./.github/workflow_scripts/build_multimodal_tutorial.sh $SUB_DOC '${{ env.BRANCH }}' '${{ env.GIT_REPO }}' '${{ env.SHORT_SHA }}' '${{ env.PR_NUMBER }}'
+  build_multimodal_image_prediction_tutorial:
+    needs: [ test_common, test_core, test_features, test_tabular, test_eda, test_fair, test_text, test_multimodal_predictor, test_multimodal_others, test_vision, test_timeseries, test_install ]
+    runs-on: ubuntu-latest
+    env:
+      SUB_DOC: image_prediction
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v2
+      - name: Setup Env Vars
+        uses: ./.github/actions/setup-env-vars
+      - name: Build Multimodal Tutorial on AWS Batch
+        uses: ./.github/actions/submit-job
+        with:
+          job-type: CI-GPU
+          job-name: AutoGluon-BuildMultimodal
+          command: chmod +x ./.github/workflow_scripts/build_multimodal_tutorial.sh && ./.github/workflow_scripts/build_multimodal_tutorial.sh $SUB_DOC '${{ env.BRANCH }}' '${{ env.GIT_REPO }}' '${{ env.SHORT_SHA }}' '${{ env.PR_NUMBER }}'
+  build_multimodal_matching_tutorial:
+    needs: [ test_common, test_core, test_features, test_tabular, test_eda, test_fair, test_text, test_multimodal_predictor, test_multimodal_others, test_vision, test_timeseries, test_install ]
+    runs-on: ubuntu-latest
+    env:
+      SUB_DOC: matching
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v2
+      - name: Setup Env Vars
+        uses: ./.github/actions/setup-env-vars
+      - name: Build Multimodal Tutorial on AWS Batch
+        uses: ./.github/actions/submit-job
+        with:
+          job-type: CI-GPU
+          job-name: AutoGluon-BuildMultimodal
+          command: chmod +x ./.github/workflow_scripts/build_multimodal_tutorial.sh && ./.github/workflow_scripts/build_multimodal_tutorial.sh $SUB_DOC '${{ env.BRANCH }}' '${{ env.GIT_REPO }}' '${{ env.SHORT_SHA }}' '${{ env.PR_NUMBER }}'
+  build_multimodal_prediction_tutorial:
+    needs: [ test_common, test_core, test_features, test_tabular, test_eda, test_fair, test_text, test_multimodal_predictor, test_multimodal_others, test_vision, test_timeseries, test_install ]
+    runs-on: ubuntu-latest
+    env:
+      SUB_DOC: multimodal_prediction
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v2
+      - name: Setup Env Vars
+        uses: ./.github/actions/setup-env-vars
+      - name: Build Multimodal Tutorial on AWS Batch
+        uses: ./.github/actions/submit-job
+        with:
+          job-type: CI-GPU
+          job-name: AutoGluon-BuildMultimodal
+          command: chmod +x ./.github/workflow_scripts/build_multimodal_tutorial.sh && ./.github/workflow_scripts/build_multimodal_tutorial.sh $SUB_DOC '${{ env.BRANCH }}' '${{ env.GIT_REPO }}' '${{ env.SHORT_SHA }}' '${{ env.PR_NUMBER }}'
+
+  build_multimodal_object_detection_tutorial:
+    needs: [ test_common, test_core, test_features, test_tabular, test_eda, test_fair, test_text, test_multimodal_predictor, test_multimodal_others, test_vision, test_timeseries, test_install ]
+    runs-on: ubuntu-latest
+    env:
+      SUB_DOC: object_detection
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v2
+      - name: Setup Env Vars
+        uses: ./.github/actions/setup-env-vars
+      - name: Build Multimodal Tutorial on AWS Batch
+        uses: ./.github/actions/submit-job
+        with:
+          job-type: CI-GPU
+          job-name: AutoGluon-BuildMultimodal
+          command: chmod +x ./.github/workflow_scripts/build_multimodal_tutorial.sh && ./.github/workflow_scripts/build_multimodal_tutorial.sh $SUB_DOC '${{ env.BRANCH }}' '${{ env.GIT_REPO }}' '${{ env.SHORT_SHA }}' '${{ env.PR_NUMBER }}'
+  build_multimodal_text_prediction_tutorial:
+    needs: [ test_common, test_core, test_features, test_tabular, test_eda, test_fair, test_text, test_multimodal_predictor, test_multimodal_others, test_vision, test_timeseries, test_install ]
+    runs-on: ubuntu-latest
+    env:
+      SUB_DOC: text_prediction
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v2
+      - name: Setup Env Vars
+        uses: ./.github/actions/setup-env-vars
+      - name: Build Multimodal Tutorial on AWS Batch
+        uses: ./.github/actions/submit-job
+        with:
+          job-type: CI-GPU
+          job-name: AutoGluon-BuildMultimodal
+          command: chmod +x ./.github/workflow_scripts/build_multimodal_tutorial.sh && ./.github/workflow_scripts/build_multimodal_tutorial.sh $SUB_DOC '${{ env.BRANCH }}' '${{ env.GIT_REPO }}' '${{ env.SHORT_SHA }}' '${{ env.PR_NUMBER }}'
   build_text_prediction_tutorial:
     needs: [test_common, test_core, test_features, test_tabular, test_eda, test_fair, test_text, test_multimodal_predictor, test_multimodal_others, test_vision, test_timeseries, test_install]
     runs-on: ubuntu-latest
@@ -379,7 +462,7 @@ jobs:
           job-name: AutoGluon-BuildTimeseries
           command: chmod +x ./.github/workflow_scripts/build_timeseries_tutorial.sh && ./.github/workflow_scripts/build_timeseries_tutorial.sh '${{ env.BRANCH }}' '${{ env.GIT_REPO }}' '${{ env.SHORT_SHA }}' '${{ env.PR_NUMBER }}'
   build_all_docs:
-    needs: [build_image_prediction_tutorial, build_object_detection_tutorial, build_tabular_prediction_tutorial, build_multimodal_tutorial, build_text_prediction_tutorial, build_cloud_fit_deploy_tutorial, build_timeseries_tutorial]
+    needs: [build_image_prediction_tutorial, build_object_detection_tutorial, build_tabular_prediction_tutorial, build_multimodal_advanced_topics_tutorial, build_multimodal_image_prediction_tutorial, build_multimodal_matching_tutorial, build_multimodal_prediction_tutorial, build_multimodal_object_detection_tutorial, build_multimodal_text_prediction_tutorial, build_text_prediction_tutorial, build_cloud_fit_deploy_tutorial, build_timeseries_tutorial]
     runs-on: ubuntu-latest
     steps:
       - name: Checkout repository


Code review comments:
diff_hunk: @@ -379,7 +462,7 @@ jobs:
           job-name: AutoGluon-BuildTimeseries
           command: chmod +x ./.github/workflow_scripts/build_timeseries_tutorial.sh && ./.github/workflow_scripts/build_timeseries_tutorial.sh '${{ env.BRANCH }}' '${{ env.GIT_REPO }}' '${{ env.SHORT_SHA }}' '${{ env.PR_NUMBER }}'
   build_all_docs:
-    needs: [build_image_prediction_tutorial, build_object_detection_tutorial, build_tabular_prediction_tutorial, build_multimodal_tutorial, build_text_prediction_tutorial, build_cloud_fit_deploy_tutorial, build_timeseries_tutorial]
+    needs: [build_image_prediction_tutorial, build_object_detection_tutorial, build_tabular_prediction_tutorial, build_multimodal_advanced_topics_tutorial, build_multimodal_image_prediction_tutorial, build_multimodal_matching_tutorial, build_multimodal_prediction_tutorial, build_multimodal_object_detection_tutorial, build_multimodal_text_prediction_tutorial, build_text_prediction_tutorial, build_cloud_fit_deploy_tutorial, build_timeseries_tutorial]
path: .github/workflows/continuous_integration.yml
text: @user1:
Would it be possible to use a [matrix](https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs) for this since we're getting to the point where there is a lot of duplicated yaml?

@author:
Good point and thanks for the link. Will add it