Repository: kubeflow/training-operator (language: python)

Issue:
Add E2E test for gang-scheduling
Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

<!--  Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, check our contributor guidelines: https://www.kubeflow.org/docs/about/contributing
2. To know more about Training Operator, check the developer guide:
    https://github.com/kubeflow/training-operator/blob/master/docs/development/developer_guide.md
3. If you want *faster* PR reviews, check how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
-->

**What this PR does / why we need it**:
I added the E2E test for gang-scheduling.
Also, I added functions to apply patches to CustomJob resources to Python SDK.

**Which issue(s) this PR fixes** _(optional, in `Fixes #<issue number>, #<issue number>, ...` format, will close the issue(s) when PR gets merged)_:
Follow up on #1724.

**Checklist:**

- [ ] [Docs](https://www.kubeflow.org/docs/components/training/) included if any changes are user facing



Discussion / hints:
## Pull Request Test Coverage Report for [Build 3994952641](https://coveralls.io/builds/56228075)

* **0** of **0**   changed or added relevant lines in **0** files are covered.
* **9** unchanged lines in **1** file lost coverage.
* Overall coverage increased (+**0.06%**) to **39.095%**

---


|  Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [pkg/controller.v1/pytorch/pytorchjob_controller.go](https://coveralls.io/builds/56228075/source?filename=pkg%2Fcontroller.v1%2Fpytorch%2Fpytorchjob_controller.go#L162) | 9 | 58.16% |
<!-- | **Total:** | **9** |  | -->

|  Totals | [![Coverage Status](https://coveralls.io/builds/56228075/badge)](https://coveralls.io/builds/56228075) |
| :-- | --: |
| Change from base [Build 3980850357](https://coveralls.io/builds/56171581): |  0.06% |
| Covered Lines: | 2687 |
| Relevant Lines: | 6873 |

---
##### 💛  - [Coveralls](https://coveralls.io)

Blocked by #1737.

/hold

The mxjob-controller watches Pods created by other FrameworkJob resources and updates MXJob Status :(
> The mxjob-controller watches Pods created by other FrameworkJob resources and updates MXJob Status :(

Ah, this is my misunderstanding.
@johnugeorge This PR is ready for review, and all tests were passed. Please take a look.

/hold cancel
Great work @tenzen-y for adding this e2e-test

/cc @andreyvelich PTAL at SDK changes 
@johnugeorge: GitHub didn't allow me to request PR reviews from the following users: PTAL, at, SDK, changes.

Note that only [kubeflow members](https://github.com/orgs/kubeflow/people) and repo collaborators can review this PR, and authors cannot review their own PRs.

<details>

In response to [this](https://github.com/kubeflow/training-operator/pull/1736#issuecomment-1399823435):

>Great work @tenzen-y for adding this e2e-test
>
>/cc @andreyvelich PTAL at SDK changes 


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
Thanks for the updates @tenzen-y!
/lgtm
/assign @johnugeorge @terrytangyuan 
Thanks @tenzen-y 
/approve
[APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubeflow/training-operator/pull/1736#issuecomment-1401807605" title="Approved">johnugeorge</a>*, *<a href="https://github.com/kubeflow/training-operator/pull/1736#" title="Author self-approved">tenzen-y</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubeflow%2Ftraining-operator).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubeflow/training-operator/blob/master/OWNERS)~~ [johnugeorge]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->

Fix patch:
diff --git a/scripts/gha/setup-training-operator.sh b/scripts/gha/setup-training-operator.sh
index aa9c98cd74..ee0131a763 100755
--- a/scripts/gha/setup-training-operator.sh
+++ b/scripts/gha/setup-training-operator.sh
@@ -42,6 +42,16 @@ if [ "${GANG_SCHEDULER_NAME}" = "scheduler-plugins" ]; then
   echo "Configure gang-scheduling using scheduler-plugins to training-operator"
   kubectl patch -n kubeflow deployments training-operator --type='json' \
     -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command/1", "value": "--gang-scheduler-name=scheduler-plugins"}]'
+elif  [ "${GANG_SCHEDULER_NAME}" = "volcano" ]; then
+  VOLCANO_SCHEDULER_VERSION=$(go list -m -f "{{.Version}}" volcano.sh/apis)
+
+  # patch scheduler first so that it is ready when scheduler-deployment installing finished
+  echo "Configure gang-scheduling using volcano to training-operator"
+  kubectl patch -n kubeflow deployments training-operator --type='json' \
+    -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command/1", "value": "--gang-scheduler-name=volcano"}]'
+
+  echo "Installing volcano scheduler ${VOLCANO_SCHEDULER_VERSION}..."
+  kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/${VOLCANO_SCHEDULER_VERSION}/installer/volcano-development.yaml
 fi
 
 TIMEOUT=30
@@ -54,6 +64,14 @@ if [ "${GANG_SCHEDULER_NAME}" = "scheduler-plugins" ]; then
     (kubectl get pods -n scheduler-plugins && kubectl describe pods -n scheduler-plugins; exit 1)
 fi
 
+# wait for volcano up
+if [ "${GANG_SCHEDULER_NAME}" = "volcano" ]; then
+  kubectl rollout status deployment -n volcano-system volcano-admission --timeout "${TIMEOUT}s" && \
+  kubectl rollout status deployment -n volcano-system volcano-scheduler --timeout "${TIMEOUT}s" && \
+  kubectl rollout status deployment -n volcano-system volcano-controllers --timeout "${TIMEOUT}s" || \
+    (kubectl get pods -n volcano-system && kubectl describe pods -n volcano-system; exit 1)
+fi
+
 kubectl version
 kubectl cluster-info
 kubectl get nodes


Test patch:
diff --git a/.github/workflows/integration-tests.yaml b/.github/workflows/integration-tests.yaml
index 810be72b83..a6889dc0ce 100644
--- a/.github/workflows/integration-tests.yaml
+++ b/.github/workflows/integration-tests.yaml
@@ -44,6 +44,15 @@ jobs:
           - kubernetes-version: v1.25.2
             gang-scheduler-name: "scheduler-plugins"
             python-version: "3.10"
+          - kubernetes-version: v1.26.4
+            gang-scheduler-name: "volcano"
+            python-version: "3.9"
+          - kubernetes-version: v1.24.6
+            gang-scheduler-name: "volcano"
+            python-version: "3.10"
+          - kubernetes-version: v1.25.2
+            gang-scheduler-name: "volcano"
+            python-version: "3.10"
 
     steps:
       - name: Checkout