Repository: GoogleCloudPlatform/spark-on-k8s-operator
Pull request #1240 (2021-04-23 21:50:48)

[#1239] Custom service port name for spark application UI
A proposal for solving https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/1239 .
This PR makes it possible to specify the service port name for exposing the spark UI.

google-cla[bot]: We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors.  If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA ([login here](https://cla.developers.google.com/) to double check)?  If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment `@googlebot I fixed it.`. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2FGoogleCloudPlatform%2Fspark-on-k8s-operator%2Fpull%2F1240) for more info**.

<!-- need_author_cla -->
google-cla[bot]: We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors.  If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA ([login here](https://cla.developers.google.com/) to double check)?  If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment `@googlebot I fixed it.`. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2FGoogleCloudPlatform%2Fspark-on-k8s-operator%2Fpull%2F1240) for more info**.

<!-- need_author_cla -->
google-cla[bot]: We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors.  If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA ([login here](https://cla.developers.google.com/) to double check)?  If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment `@googlebot I fixed it.`. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2FGoogleCloudPlatform%2Fspark-on-k8s-operator%2Fpull%2F1240) for more info**.

<!-- need_author_cla -->

Patch:
diff --git a/.github/workflows/main.yaml b/.github/workflows/main.yaml
new file mode 100644
index 0000000..301ea68
--- /dev/null
+++ b/.github/workflows/main.yaml
@@ -0,0 +1,75 @@
+name: Pre-commit checks
+
+on:
+  pull_request:
+    branches:
+      - master
+
+  push:
+    branches:
+      - master
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Set up Go
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.15
+
+      - name: Checkout source code
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 2
+
+      - name: fmt check
+        run: make fmt-check
+
+      - name: unit tests
+        run: make test
+
+      - name: integration tests
+        run: make test
+
+  it:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+
+      - name: Set up Go
+        uses: actions/setup-go@v2
+        with:
+          go-version: 1.15
+
+      - name: Checkout source code
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 2
+
+      - name: Install Helm
+        uses: azure/setup-helm@v1
+        with:
+          version: v3.4.0
+
+      - uses: actions/setup-python@v2
+        with:
+          python-version: 3.7
+
+      - name: Set up chart-testing
+        uses: helm/chart-testing-action@v2.0.1
+
+      - name: Print ct version information and List files
+        run: ct version && ls -lh
+
+      - name: Run chart-testing (lint)
+        run: ct lint
+
+      - name: Create kind cluster
+        uses: helm/kind-action@v1.0.0
+
+      - name: Run chart-testing (install)
+        run: ct install
diff --git a/.travis.gofmt.sh b/.travis.gofmt.sh
index bd66cf9..d56c12f 100755
--- a/.travis.gofmt.sh
+++ b/.travis.gofmt.sh
@@ -15,7 +15,10 @@
 # limitations under the License.
 
 cd "$(dirname $0)"
-if [ -n "$(go fmt ./...)" ]; then
+if [ -n "$(go fmt ./...)" ];
+then
   echo "Go code is not formatted, please run 'go fmt ./...'." >&2
   exit 1
+else
+  echo "Go code is formatted"
 fi
diff --git a/Dockerfile b/Dockerfile
index 866152d..ad89a5b 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -14,7 +14,7 @@
 # limitations under the License.
 #
 
-ARG SPARK_IMAGE=gcr.io/spark-operator/spark:v3.0.0
+ARG SPARK_IMAGE=gcr.io/spark-operator/spark:v3.1.1
 
 FROM golang:1.15.2-alpine as builder
 
diff --git a/Dockerfile.rh b/Dockerfile.rh
index 42880b3..4fe0110 100644
--- a/Dockerfile.rh
+++ b/Dockerfile.rh
@@ -20,7 +20,7 @@
 # 1. Your Docker version is >= 18.09.3
 # 2. export DOCKER_BUILDKIT=1
 
-ARG SPARK_IMAGE=gcr.io/spark-operator/spark:v3.0.0
+ARG SPARK_IMAGE=gcr.io/spark-operator/spark:v3.1.1
 
 FROM golang:1.14.0-alpine as builder
 
diff --git a/Makefile b/Makefile
index 12dd810..92cc18d 100644
--- a/Makefile
+++ b/Makefile
@@ -6,6 +6,7 @@ SPARK_OPERATOR_GOPATH=/go/src/github.com/GoogleCloudPlatform/spark-on-k8s-operat
 DEP_VERSION:=`grep DEP_VERSION= Dockerfile | awk -F\" '{print $$2}'`
 BUILDER=`grep "FROM golang:" Dockerfile | awk '{print $$2}'`
 UNAME:=`uname | tr '[:upper:]' '[:lower:]'`
+REPO=github.com/GoogleCloudPlatform/spark-on-k8s-operator/pkg
 
 all: clean-sparkctl build-sparkctl install-sparkctl
 
@@ -43,3 +44,27 @@ build-api-docs:
 
 helm-docs:
 	helm-docs -c ./charts
+
+fmt-check: clean
+	@echo "running fmt check"
+	./.travis.gofmt.sh
+
+clean:
+	@echo "cleaning up caches and output"
+	go clean -cache -testcache -r -x ./... 2>&1 >/dev/null
+	-rm -rf _output
+
+test: clean
+	@echo "running unit tests"
+	go test -v ./... -covermode=atomic
+
+
+it-test: clean all
+	@echo "running unit tests"
+	go test -v ./test/e2e/ --kubeconfig "$HOME/.kube/config" --operator-image=gcr.io/spark-operator/spark-operator:v1beta2-1.2.3-3.1.1
+
+vet:
+	@echo "running go vet"
+	# echo "Building using $(BUILDER)"
+	# go vet ./...
+	go vet $(REPO)...
diff --git a/README.md b/README.md
index c2ba75f..00e3169 100644
--- a/README.md
+++ b/README.md
@@ -47,6 +47,7 @@ The following table lists the most recent few versions of the operator.
 | Operator Version | API Version | Kubernetes Version | Base Spark Version | Operator Image Tag |
 | ------------- | ------------- | ------------- | ------------- | ------------- |
 | `latest` (master HEAD) | `v1beta2` | 1.13+ | `3.0.0` | `latest` |
+| `v1beta2-1.2.3-3.1.1` | `v1beta2` | 1.13+ | `3.1.1` | `v1beta2-1.2.3-3.1.1` |
 | `v1beta2-1.2.0-3.0.0` | `v1beta2` | 1.13+ | `3.0.0` | `v1beta2-1.2.0-3.0.0` |
 | `v1beta2-1.1.2-2.4.5` | `v1beta2` | 1.13+ | `2.4.5` | `v1beta2-1.1.2-2.4.5` |
 | `v1beta2-1.0.1-2.4.4` | `v1beta2` | 1.13+ | `2.4.4` | `v1beta2-1.0.1-2.4.4` |
diff --git a/charts/spark-operator-chart/Chart.yaml b/charts/spark-operator-chart/Chart.yaml
index 2541565..21548cf 100644
--- a/charts/spark-operator-chart/Chart.yaml
+++ b/charts/spark-operator-chart/Chart.yaml
@@ -1,8 +1,8 @@
 apiVersion: v2
 name: spark-operator
 description: A Helm chart for Spark on Kubernetes operator
-version: 1.0.9
-appVersion: v1beta2-1.2.0-3.0.0
+version: 1.1.0
+appVersion: v1beta2-1.2.3-3.1.1
 keywords:
   - spark
 home: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator
diff --git a/charts/spark-operator-chart/README.md b/charts/spark-operator-chart/README.md
index 26112a6..16c26c8 100644
--- a/charts/spark-operator-chart/README.md
+++ b/charts/spark-operator-chart/README.md
@@ -46,6 +46,35 @@ $ helm uninstall my-release
 
 The command removes all the Kubernetes components associated with the chart and deletes the release, except for the `crds`, those will have to be removed manually.
 
+## Test the chart
+
+Install [chart-testing cli](https://github.com/helm/chart-testing#installation)
+
+In Mac OS, you can just:
+
+```bash
+pip install yamale
+pip install yamllint
+brew install chart-testing
+```
+
+Run ct lint and Verify `All charts linted successfully`
+
+```bash
+Chart version ok.
+Validating /Users/chethanuk/Work/Github/Personal/spark-on-k8s-operator-1/charts/spark-operator-chart/Chart.yaml...
+Validation success! 👍
+Validating maintainers...
+==> Linting charts/spark-operator-chart
+[INFO] Chart.yaml: icon is recommended
+
+1 chart(s) linted, 0 chart(s) failed
+------------------------------------------------------------------------------------------------------------------------
+ ✔︎ spark-operator => (version: "1.1.0", path: "charts/spark-operator-chart")
+------------------------------------------------------------------------------------------------------------------------
+All charts linted successfully
+```
+
 ## Values
 
 | Key | Type | Default | Description |
@@ -60,24 +89,27 @@ The command removes all the Kubernetes components associated with the chart and
 | imagePullSecrets | list | `[]` | Image pull secrets |
 | ingressUrlFormat | string | `""` | Ingress URL format |
 | istio.enabled | bool | `false` | When using `istio`, spark jobs need to run without a sidecar to properly terminate |
+| labelSelectorFilter | string | `""` | A comma-separated list of key=value, or key labels to filter resources during watch and list based on the specified labels. |
 | leaderElection.lockName | string | `"spark-operator-lock"` | Leader election lock name. Ref: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#enabling-leader-election-for-high-availability. |
 | leaderElection.lockNamespace | string | `""` | Optionally store the lock in another namespace. Defaults to operator's namespace |
 | logLevel | int | `2` | Set higher levels for more verbose logging |
 | metrics.enable | bool | `true` | Enable prometheus metric scraping |
 | metrics.endpoint | string | `"/metrics"` | Metrics serving endpoint |
 | metrics.port | int | `10254` | Metrics port |
-| metrics.portName | string | `metrics` | Metrics port name |
+| metrics.portName | string | `"metrics"` | Metrics port name |
 | metrics.prefix | string | `""` | Metric prefix, will be added to all exported metrics |
 | nameOverride | string | `""` | String to partially override `spark-operator.fullname` template (will maintain the release name) |
 | nodeSelector | object | `{}` | Node labels for pod assignment |
 | podAnnotations | object | `{}` | Additional annotations to add to the pod |
-| podMonitor.enable | bool| `false` | Submit a prometheus pod monitor for operator's pod. Note that prometheus metrics should be enabled as well.|
+| podMonitor | object | `{"enable":false,"jobLabel":"spark-operator-podmonitor","labels":{},"podMetricsEndpoint":{"interval":"5s","scheme":"http"}}` | Prometheus pod monitor for operator's pod. |
+| podMonitor.enable | bool | `false` | If enabled, a pod monitor for operator's pod will be submitted. Note that prometheus metrics should be enabled as well. |
+| podMonitor.jobLabel | string | `"spark-operator-podmonitor"` | The label to use to retrieve the job name from |
 | podMonitor.labels | object | `{}` | Pod monitor labels |
-| podMonitor.jobLabel | string | `spark-operator-podmonitor` | The label to use to retrieve the job name from |
-| podMonitor.podMetricsEndpoint.scheme | string | `http` | Prometheus metrics endpoint scheme |
-| podMonitor.podMetricsEndpoint.interval | string | `5s` | Interval at which metrics should be scraped |
+| podMonitor.podMetricsEndpoint | object | `{"interval":"5s","scheme":"http"}` | Prometheus metrics endpoint properties. `metrics.portName` will be used as a port |
 | podSecurityContext | object | `{}` | Pod security context |
-| rbac.create | bool | `true` | Create and use `rbac` resources |
+| rbac.create | bool | `false` | **DEPRECATED** use `createRole` and `createClusterRole` |
+| rbac.createClusterRole | bool | `true` | Create and use RBAC `ClusterRole` resources |
+| rbac.createRole | bool | `true` | Create and use RBAC `Role` resources |
 | replicaCount | int | `1` | Desired number of pods, leaderElection will be enabled if this is greater than 1 |
 | resourceQuotaEnforcement.enable | bool | `false` | Whether to enable the ResourceQuota enforcement for SparkApplication resources. Requires the webhook to be enabled by setting `webhook.enable` to true. Ref: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#enabling-resource-quota-enforcement. |
 | resources | object | `{}` | Pod resource requests and limits |
@@ -89,10 +121,10 @@ The command removes all the Kubernetes components associated with the chart and
 | serviceAccounts.sparkoperator.name | string | `""` | Optional name for the operator service account |
 | sparkJobNamespace | string | `""` | Set this if running spark jobs in a different namespace than the operator |
 | tolerations | list | `[]` | List of node taints to tolerate |
+| webhook.cleanupAnnotations | object | `{"helm.sh/hook":"pre-delete, pre-upgrade","helm.sh/hook-delete-policy":"hook-succeeded"}` | The annotations applied to the cleanup job, required for helm lifecycle hooks |
 | webhook.enable | bool | `false` | Enable webhook server |
 | webhook.namespaceSelector | string | `""` | The webhook server will only operate on namespaces with this label, specified in the form key1=value1,key2=value2. Empty string (default) will operate on all namespaces |
 | webhook.port | int | `8080` | Webhook service port |
-| labelSelectorFilter | string | `""` | Set this if only operator watches spark jobs with certain labels are allowed |
 
 ## Maintainers
 
diff --git a/charts/spark-operator-chart/templates/deployment.yaml b/charts/spark-operator-chart/templates/deployment.yaml
index 120a465..952874c 100644
--- a/charts/spark-operator-chart/templates/deployment.yaml
+++ b/charts/spark-operator-chart/templates/deployment.yaml
@@ -90,7 +90,7 @@ spec:
       volumes:
         - name: webhook-certs
           secret:
-            secretName: spark-webhook-certs
+            secretName: {{ include "spark-operator.fullname" . }}-webhook-certs
       {{- end }}
       {{- with .Values.nodeSelector }}
       nodeSelector:
diff --git a/charts/spark-operator-chart/templates/rbac.yaml b/charts/spark-operator-chart/templates/rbac.yaml
index 63ab288..8fd06e5 100644
--- a/charts/spark-operator-chart/templates/rbac.yaml
+++ b/charts/spark-operator-chart/templates/rbac.yaml
@@ -1,4 +1,4 @@
-{{- if .Values.rbac.create }}
+{{- if or .Values.rbac.create .Values.rbac.createClusterRole }}
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole
 metadata:
diff --git a/charts/spark-operator-chart/templates/spark-rbac.yaml b/charts/spark-operator-chart/templates/spark-rbac.yaml
index 5f36333..af507c8 100644
--- a/charts/spark-operator-chart/templates/spark-rbac.yaml
+++ b/charts/spark-operator-chart/templates/spark-rbac.yaml
@@ -1,4 +1,4 @@
-{{- if .Values.rbac.create }}
+{{- if or .Values.rbac.create .Values.rbac.createRole }}
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
@@ -19,7 +19,12 @@ rules:
   - services
   verbs:
   - "*"
-
+- apiGroups:
+  - ""
+  resources:
+  - configmaps
+  verbs:
+  - "*"
 ---
 
 apiVersion: rbac.authorization.k8s.io/v1
diff --git a/charts/spark-operator-chart/templates/webhook-cleanup-job.yaml b/charts/spark-operator-chart/templates/webhook-cleanup-job.yaml
index 2f54a12..93bb31e 100644
--- a/charts/spark-operator-chart/templates/webhook-cleanup-job.yaml
+++ b/charts/spark-operator-chart/templates/webhook-cleanup-job.yaml
@@ -36,7 +36,7 @@ spec:
           -H \"Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" \
           -H \"Accept: application/json\" \
           -H \"Content-Type: application/json\" \
-          https://kubernetes.default.svc/api/v1/namespaces/{{ .Release.Namespace }}/secrets/spark-webhook-certs \
+          https://kubernetes.default.svc/api/v1/namespaces/{{ .Release.Namespace }}/secrets/{{ include "spark-operator.fullname" . }}-webhook-certs \
           && \
           curl -ik \
           -X DELETE \
diff --git a/charts/spark-operator-chart/templates/webhook-init-job.yaml b/charts/spark-operator-chart/templates/webhook-init-job.yaml
index 8c6c371..b6fd11e 100644
--- a/charts/spark-operator-chart/templates/webhook-init-job.yaml
+++ b/charts/spark-operator-chart/templates/webhook-init-job.yaml
@@ -26,5 +26,11 @@ spec:
         imagePullPolicy: {{ .Values.image.pullPolicy }}
         securityContext:
           {{- toYaml .Values.securityContext | nindent 10 }}
-        command: ["/usr/bin/gencerts.sh", "-n", "{{ .Release.Namespace }}", "-s", "{{ include "spark-operator.fullname" . }}-webhook", "-p"]
+        command: [
+            "/usr/bin/gencerts.sh",
+            "-n", "{{ .Release.Namespace }}",
+            "-s", "{{ include "spark-operator.fullname" . }}-webhook",
+            "-r", "{{ include "spark-operator.fullname" . }}-webhook-certs",
+            "-p"
+          ]
 {{ end }}
diff --git a/charts/spark-operator-chart/values.yaml b/charts/spark-operator-chart/values.yaml
index 2718c8a..2a735fc 100644
--- a/charts/spark-operator-chart/values.yaml
+++ b/charts/spark-operator-chart/values.yaml
@@ -12,7 +12,7 @@ image:
   # -- Image pull policy
   pullPolicy: IfNotPresent
   # -- Overrides the image tag whose default is the chart appVersion.
-  tag: ""
+  tag: "latest"
 
 # -- Image pull secrets
 imagePullSecrets: []
@@ -24,8 +24,12 @@ nameOverride: ""
 fullnameOverride: ""
 
 rbac:
-  # -- Create and use `rbac` resources
-  create: true
+  # -- **DEPRECATED** use `createRole` and `createClusterRole`
+  create: false
+  # -- Create and use RBAC `Role` resources
+  createRole: true
+  # -- Create and use RBAC `ClusterRole` resources
+  createClusterRole: true
 
 serviceAccounts:
   spark:
diff --git a/docs/api-docs.md b/docs/api-docs.md
index 9fec91e..5a25a6e 100644
--- a/docs/api-docs.md
+++ b/docs/api-docs.md
@@ -2918,6 +2918,20 @@ TargetPort should be the same as the one defined in spark.ui.port</p>
 </tr>
 <tr>
 <td>
+<code>ServicePortName</code></br>
+<em>
+string
+</em>
+</td>
+<td>
+<em>(Optional)</em>
+<p>ServicePortName allows configuring the name of the service port.
+This may be useful for sidecar proxies like Envoy injected by Istio which require specific ports names to treat traffic as proper HTTP.
+Defaults to spark-driver-ui-port.</p>
+</td>
+</tr>
+<tr>
+<td>
 <code>serviceType</code></br>
 <em>
 <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/#servicetype-v1-core">
@@ -2961,5 +2975,5 @@ map[string]string
 <hr/>
 <p><em>
 Generated with <code>gen-crd-api-reference-docs</code>
-on git commit <code>2f6a3f9</code>.
+on git commit <code>eb07ee0</code>.
 </em></p>
diff --git a/docs/developer-guide.md b/docs/developer-guide.md
index 1308a7c..aefcfa7 100644
--- a/docs/developer-guide.md
+++ b/docs/developer-guide.md
@@ -10,7 +10,7 @@ The easiest way to build the operator without worrying about its dependencies is
 $ docker build -t <image-tag> .
 ```
 
-The operator image is built upon a base Spark image that defaults to `gcr.io/spark-operator/spark:v3.0.0`. If you want to use your own Spark image (e.g., an image with a different version of Spark or some custom dependencies), specify the argument `SPARK_IMAGE` as the following example shows: 
+The operator image is built upon a base Spark image that defaults to `gcr.io/spark-operator/spark:v3.1.1`. If you want to use your own Spark image (e.g., an image with a different version of Spark or some custom dependencies), specify the argument `SPARK_IMAGE` as the following example shows: 
 
 ```bash
 $ docker build --build-arg SPARK_IMAGE=<your Spark image> -t <image-tag> .
diff --git a/docs/quick-start-guide.md b/docs/quick-start-guide.md
index a3edb0a..58e62bc 100644
--- a/docs/quick-start-guide.md
+++ b/docs/quick-start-guide.md
@@ -4,16 +4,20 @@ For a more detailed guide on how to use, compose, and work with `SparkApplicatio
 [User Guide](user-guide.md). If you are running the Kubernetes Operator for Apache Spark on Google Kubernetes Engine and want to use Google Cloud Storage (GCS) and/or BigQuery for reading/writing data, also refer to the [GCP guide](gcp.md). The Kubernetes Operator for Apache Spark will simply be referred to as the operator for the rest of this guide.
 
 ## Table of Contents
-* [Installation](#installation)
-* [Running the Examples](#running-the-examples)
-* [Configuration](#configuration)
-* [Upgrade](#upgrade)
-* [About the Spark Job Namespace](#about-the-spark-job-namespace)
-* [About the Service Account for Driver Pods](#about-the-service-account-for-driver-pods)
-* [Enable Metric Exporting to Prometheus](#enable-metric-exporting-to-prometheus)
-* [Driver UI Access and Ingress](#driver-ui-access-and-ingress)
-* [About the Mutating Admission Webhook](#about-the-mutating-admission-webhook)
-* [Mutating Admission Webhooks on a private GKE cluster](#mutating-admission-webhooks-on-a-private-gke-cluster)
+- [Quick Start Guide](#quick-start-guide)
+  - [Table of Contents](#table-of-contents)
+  - [Installation](#installation)
+  - [Running the Examples](#running-the-examples)
+  - [Configuration](#configuration)
+  - [Upgrade](#upgrade)
+  - [About the Spark Job Namespace](#about-the-spark-job-namespace)
+  - [About the Service Account for Driver Pods](#about-the-service-account-for-driver-pods)
+  - [Enable Metric Exporting to Prometheus](#enable-metric-exporting-to-prometheus)
+      - [Spark Application Metrics](#spark-application-metrics)
+      - [Work Queue Metrics](#work-queue-metrics)
+  - [Driver UI Access and Ingress](#driver-ui-access-and-ingress)
+  - [About the Mutating Admission Webhook](#about-the-mutating-admission-webhook)
+    - [Mutating Admission Webhooks on a private GKE cluster](#mutating-admission-webhooks-on-a-private-gke-cluster)
 
 ## Installation
 
@@ -89,8 +93,8 @@ spec:
     labels:
       version: 2.3.0
     memory: 512m
-  image: gcr.io/ynli-k8s/spark:v3.0.0
-  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-2.3.0.jar
+  image: gcr.io/ynli-k8s/spark:v3.1.1
+  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.1.1.jar
   mainClass: org.apache.spark.examples.SparkPi
   mode: cluster
   restartPolicy:
diff --git a/docs/user-guide.md b/docs/user-guide.md
index de2ebff..2c9aea0 100644
--- a/docs/user-guide.md
+++ b/docs/user-guide.md
@@ -2,52 +2,55 @@
 
 For a quick introduction on how to build and install the Kubernetes Operator for Apache Spark, and how to run some example applications, please refer to the [Quick Start Guide](quick-start-guide.md). For a complete reference of the API definition of the `SparkApplication` and `ScheduledSparkApplication` custom resources, please refer to the [API Specification](api-docs.md).
 
-The Kubernetes Operator for Apache Spark ships with a command-line tool called `sparkctl` that offers additional features beyond what `kubectl` is able to do. Documentation on `sparkctl` can be found in [README](../sparkctl/README.md). If you are running the Spark Operator on Google Kubernetes Engine and want to use Google Cloud Storage (GCS) and/or BigQuery for reading/writing data, also refer to the [GCP guide](gcp.md). The Kubernetes Operator for Apache Spark will simply be referred to as the operator for the rest of this guide.  
+The Kubernetes Operator for Apache Spark ships with a command-line tool called `sparkctl` that offers additional features beyond what `kubectl` is able to do. Documentation on `sparkctl` can be found in [README](../sparkctl/README.md). If you are running the Spark Operator on Google Kubernetes Engine and want to use Google Cloud Storage (GCS) and/or BigQuery for reading/writing data, also refer to the [GCP guide](gcp.md). The Kubernetes Operator for Apache Spark will simply be referred to as the operator for the rest of this guide.
 
 ## Table of Contents
 
-* [Using a SparkApplication](#using-a-sparkapplication)
-* [Writing a SparkApplication Spec](#writing-a-sparkapplication-spec)
-    * [Specifying Deployment Mode](#specifying-deployment-mode)
-    * [Specifying Application Dependencies](#specifying-application-dependencies)
-    * [Specifying Spark Configuration](#specifying-spark-configuration)
-    * [Specifying Hadoop Configuration](#specifying-hadoop-configuration)
-    * [Writing Driver Specification](#writing-driver-specification)
-    * [Writing Executor Specification](#writing-executor-specification)
-    * [Specifying Extra Java Options](#specifying-extra-java-options)
-    * [Specifying Environment Variables](#specifying-environment-variables)
-    * [Requesting GPU Resources](#requesting-gpu-resources)
-    * [Host Network](#host-network)    
-    * [Mounting Secrets](#mounting-secrets)
-    * [Mounting ConfigMaps](#mounting-configmaps)
-        * [Mounting a ConfigMap storing Spark Configuration Files](#mounting-a-configmap-storing-spark-configuration-files)
-        * [Mounting a ConfigMap storing Hadoop Configuration Files](#mounting-a-configmap-storing-hadoop-configuration-files)
-    * [Mounting Volumes](#mounting-volumes)
-    * [Using Secrets As Environment Variables](#using-secrets-as-environment-variables)
-    * [Using Image Pull Secrets](#using-image-pull-secrets)
-    * [Using Pod Affinity](#using-pod-affinity)
-    * [Using Tolerations](#using-tolerations)
-    * [Using Pod Security Context](#using-pod-security-context)
-    * [Using Sidecar Containers](#using-sidecar-containers)
-    * [Using Init-Containers](#using-init-containers)
-    * [Using Volume For Scratch Space](#using-volume-for-scratch-space)
-    * [Using Termination Grace Period](#using-termination-grace-period)
-    * [Using Container LifeCycle Hooks](#using-container-lifecycle-hooks)
-    * [Python Support](#python-support)
-    * [Monitoring](#monitoring)
-    * [Dynamic Allocation](#dynamic-allocation)
-* [Working with SparkApplications](#working-with-sparkapplications)
-    * [Creating a New SparkApplication](#creating-a-new-sparkapplication)
-    * [Deleting a SparkApplication](#deleting-a-sparkapplication)
-    * [Updating a SparkApplication](#updating-a-sparkapplication)
-    * [Checking a SparkApplication](#checking-a-sparkapplication)
-    * [Configuring Automatic Application Restart and Failure Handling](#configuring-automatic-application-restart-and-failure-handling)
-    * [Setting TTL for a SparkApplication](#setting-ttl-for-a-sparkapplication)
-* [Running Spark Applications on a Schedule using a ScheduledSparkApplication](#running-spark-applications-on-a-schedule-using-a-scheduledsparkapplication)
-* [Enabling Leader Election for High Availability](#enabling-leader-election-for-high-availability)
-* [Enabling Resource Quota Enforcement](#enabling-resource-quota-enforcement)
-* [Running Multiple Instances Of The Operator Within The Same K8s Cluster](#running-multiple-instances-of-the-operator-within-the-same-k8s-cluster)
-* [Customizing the Operator](#customizing-the-operator)
+- [User Guide](#user-guide)
+  - [Table of Contents](#table-of-contents)
+  - [Using a SparkApplication](#using-a-sparkapplication)
+  - [Writing a SparkApplication Spec](#writing-a-sparkapplication-spec)
+    - [Specifying Deployment Mode](#specifying-deployment-mode)
+    - [Specifying Application Dependencies](#specifying-application-dependencies)
+    - [Specifying Spark Configuration](#specifying-spark-configuration)
+    - [Specifying Hadoop Configuration](#specifying-hadoop-configuration)
+    - [Writing Driver Specification](#writing-driver-specification)
+    - [Writing Executor Specification](#writing-executor-specification)
+    - [Specifying Extra Java Options](#specifying-extra-java-options)
+    - [Specifying Environment Variables](#specifying-environment-variables)
+    - [Requesting GPU Resources](#requesting-gpu-resources)
+    - [Host Network](#host-network)
+    - [Mounting Secrets](#mounting-secrets)
+    - [Mounting ConfigMaps](#mounting-configmaps)
+      - [Mounting a ConfigMap storing Spark Configuration Files](#mounting-a-configmap-storing-spark-configuration-files)
+      - [Mounting a ConfigMap storing Hadoop Configuration Files](#mounting-a-configmap-storing-hadoop-configuration-files)
+    - [Mounting Volumes](#mounting-volumes)
+    - [Using Secrets As Environment Variables](#using-secrets-as-environment-variables)
+    - [Using Image Pull Secrets](#using-image-pull-secrets)
+    - [Using Pod Affinity](#using-pod-affinity)
+    - [Using Tolerations](#using-tolerations)
+    - [Using Security Context](#using-security-context)
+    - [Using Sidecar Containers](#using-sidecar-containers)
+    - [Using Init-Containers](#using-init-containers)
+    - [Using DNS Settings](#using-dns-settings)
+    - [Using Volume For Scratch Space](#using-volume-for-scratch-space)
+    - [Using Termination Grace Period](#using-termination-grace-period)
+    - [Using Container LifeCycle Hooks](#using-container-lifecycle-hooks)
+    - [Python Support](#python-support)
+    - [Monitoring](#monitoring)
+    - [Dynamic Allocation](#dynamic-allocation)
+  - [Working with SparkApplications](#working-with-sparkapplications)
+    - [Creating a New SparkApplication](#creating-a-new-sparkapplication)
+    - [Deleting a SparkApplication](#deleting-a-sparkapplication)
+    - [Updating a SparkApplication](#updating-a-sparkapplication)
+    - [Checking a SparkApplication](#checking-a-sparkapplication)
+    - [Configuring Automatic Application Restart and Failure Handling](#configuring-automatic-application-restart-and-failure-handling)
+    - [Setting TTL for a SparkApplication](#setting-ttl-for-a-sparkapplication)
+  - [Running Spark Applications on a Schedule using a ScheduledSparkApplication](#running-spark-applications-on-a-schedule-using-a-scheduledsparkapplication)
+  - [Enabling Leader Election for High Availability](#enabling-leader-election-for-high-availability)
+  - [Enabling Resource Quota Enforcement](#enabling-resource-quota-enforcement)
+  - [Running Multiple Instances Of The Operator Within The Same K8s Cluster](#running-multiple-instances-of-the-operator-within-the-same-k8s-cluster)
+  - [Customizing the Operator](#customizing-the-operator)
 
 ## Using a SparkApplication
 The operator runs Spark applications specified in Kubernetes objects of the `SparkApplication` custom resource type. The most common way of using a `SparkApplication` is store the `SparkApplication` specification in a YAML file and use the `kubectl` command or alternatively the `sparkctl` command to work with the `SparkApplication`. The operator automatically submits the application as configured in a `SparkApplication` to run on the Kubernetes cluster and uses the `SparkApplication` to collect and surface the status of the driver and executors to the user.
@@ -71,9 +74,9 @@ metadata:
 spec:
   type: Scala
   mode: cluster
-  image: gcr.io/spark/spark:v3.0.0
+  image: gcr.io/spark/spark:v3.1.1
   mainClass: org.apache.spark.examples.SparkPi
-  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0.jar
+  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.1.1.jar
 ```
 
 ### Specifying Deployment Mode
@@ -161,7 +164,7 @@ spec:
     coreLimit: 200m
     memory: 512m
     labels:
-      version: 3.0.0
+      version: 3.1.1
     serviceAccount: spark
 ```
 
@@ -181,7 +184,7 @@ spec:
     instances: 1
     memory: 512m
     labels:
-      version: 3.0.0
+      version: 3.1.1
 ```
 
 ### Specifying Extra Java Options
@@ -238,7 +241,7 @@ spec:
         valueFrom:
           secretKeyRef:
             name: some-secret
-            key: auth-key  
+            key: auth-key
     envFrom:
       - configMapRef:
           name: my-env-config-map
@@ -262,7 +265,7 @@ spec:
       name: "amd.com/gpu"   # GPU resource name
       quantity: 1           # number of GPUs to request
     labels:
-      version: 3.0.0
+      version: 3.1.1
     serviceAccount: spark
   executor:
     cores: 1
@@ -286,7 +289,7 @@ spec:
     memory: "512m"
     hostNetwork: true
     labels:
-      version: 3.0.0
+      version: 3.1.1
     serviceAccount: spark
   executor:
     cores: 1
@@ -411,7 +414,7 @@ spec:
     affinity:
       podAffinity:
         requiredDuringSchedulingIgnoredDuringExecution:
-          ...   
+          ...
   executor:
     affinity:
       podAntiAffinity:
@@ -438,7 +441,7 @@ spec:
     - key: Key
       operator: Equal
       value: Value
-      effect: NoSchedule    
+      effect: NoSchedule
 ```
 
 Note that the mutating admission webhook is needed to use this feature. Please refer to the
@@ -456,7 +459,7 @@ spec:
       runAsUser: 1000
     securityContext:
       allowPrivilegeEscalation: false
-      runAsUser: 2000    
+      runAsUser: 2000
   executor:
     podSecurityContext:
       runAsUser: 1000
@@ -478,7 +481,7 @@ spec:
     sidecars:
     - name: "sidecar1"
       image: "sidecar1:latest"
-      ...  
+      ...
   executor:
     sidecars:
     - name: "sidecar1"
@@ -496,7 +499,7 @@ spec:
     initContainers:
     - name: "init-container1"
       image: "init-container1:latest"
-      ...  
+      ...
   executor:
     initContainers:
     - name: "init-container1"
@@ -654,7 +657,7 @@ Note that Python binding for PySpark is available in Apache Spark 2.4.
 
 The operator supports using the Spark metric system to expose metrics to a variety of sinks. Particularly, it is able to automatically configure the metric system to expose metrics to [Prometheus](https://prometheus.io/). Specifically, the field `.spec.monitoring` specifies how application monitoring is handled and particularly how metrics are to be reported. The metric system is configured through the configuration file `metrics.properties`, which gets its content from the field `.spec.monitoring.metricsProperties`. The content of [metrics.properties](../spark-docker/conf/metrics.properties) will be used by default if `.spec.monitoring.metricsProperties` is not specified. `.spec.monitoring.metricsPropertiesFile` overwrite the value `spark.metrics.conf` in spark.properties, and will not use content from `.spec.monitoring.metricsProperties`. You can choose to enable or disable reporting driver and executor metrics using the fields `.spec.monitoring.exposeDriverMetrics` and `.spec.monitoring.exposeExecutorMetrics`, respectively.
 
-Further, the field `.spec.monitoring.prometheus` specifies how metrics are exposed to Prometheus using the [Prometheus JMX exporter](https://github.com/prometheus/jmx_exporter). When `.spec.monitoring.prometheus` is specified, the operator automatically configures the JMX exporter to run as a Java agent. The only required field of `.spec.monitoring.prometheus` is `jmxExporterJar`, which specified the path to the Prometheus JMX exporter Java agent jar in the container. If you use the image `gcr.io/spark-operator/spark:v3.0.0-gcs-prometheus`, the jar is located at `/prometheus/jmx_prometheus_javaagent-0.11.0.jar`. The field `.spec.monitoring.prometheus.port` specifies the port the JMX exporter Java agent binds to and defaults to `8090` if not specified. The field `.spec.monitoring.prometheus.configuration` specifies the content of the configuration to be used with the JMX exporter. The content of [prometheus.yaml](../spark-docker/conf/prometheus.yaml) will be used by default if `.spec.monitoring.prometheus.configuration` is not specified.    
+Further, the field `.spec.monitoring.prometheus` specifies how metrics are exposed to Prometheus using the [Prometheus JMX exporter](https://github.com/prometheus/jmx_exporter). When `.spec.monitoring.prometheus` is specified, the operator automatically configures the JMX exporter to run as a Java agent. The only required field of `.spec.monitoring.prometheus` is `jmxExporterJar`, which specified the path to the Prometheus JMX exporter Java agent jar in the container. If you use the image `gcr.io/spark-operator/spark:v3.1.1-gcs-prometheus`, the jar is located at `/prometheus/jmx_prometheus_javaagent-0.11.0.jar`. The field `.spec.monitoring.prometheus.port` specifies the port the JMX exporter Java agent binds to and defaults to `8090` if not specified. The field `.spec.monitoring.prometheus.configuration` specifies the content of the configuration to be used with the JMX exporter. The content of [prometheus.yaml](../spark-docker/conf/prometheus.yaml) will be used by default if `.spec.monitoring.prometheus.configuration` is not specified.
 
 Below is an example that shows how to configure the metric system to expose metrics to Prometheus using the Prometheus JMX exporter. Note that the JMX exporter Java agent jar is listed as a dependency and will be downloaded to where `.spec.dep.jarsDownloadDir` points to in Spark 2.3.x, which is `/var/spark-data/spark-jars` by default. Things are different in Spark 2.4 as dependencies will be downloaded to the local working directory instead in Spark 2.4. A complete example can be found in [examples/spark-pi-prometheus.yaml](../examples/spark-pi-prometheus.yaml).
 
@@ -666,7 +669,7 @@ spec:
   monitoring:
     exposeDriverMetrics: true
     prometheus:
-      jmxExporterJar: "/var/spark-data/spark-jars/jmx_prometheus_javaagent-0.11.0.jar"    
+      jmxExporterJar: "/var/spark-data/spark-jars/jmx_prometheus_javaagent-0.11.0.jar"
 ```
 
 The operator automatically adds the annotations such as `prometheus.io/scrape=true` on the driver and/or executor pods (depending on the values of  `.spec.monitoring.exposeDriverMetrics` and `.spec.monitoring.exposeExecutorMetrics`) so the metrics exposed on the pods can be scraped by the Prometheus server in the same cluster.
@@ -757,9 +760,9 @@ spec:
   template:
     type: Scala
     mode: cluster
-    image: gcr.io/spark/spark:v3.0.0
+    image: gcr.io/spark/spark:v3.1.1
     mainClass: org.apache.spark.examples.SparkPi
-    mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-2.3.0.jar
+    mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.1.1.jar
     driver:
       cores: 1
       memory: 512m
@@ -812,7 +815,7 @@ Or if you want your operator to watch specific resources that may exist in diffe
 
 * You need to add custom labels on resources by defining for each instance of the operator a different set of labels in `-label-selector-filter (e.g. env=dev,app-type=spark)`.
 * Run different `webhook` instances by specifying different `-webhook-config-name` flag for each deployment of the operator.
-* Specify different `webhook-svc-name` and/or `webhook-svc-namespace` for each instance of the operator. 
+* Specify different `webhook-svc-name` and/or `webhook-svc-namespace` for each instance of the operator.
 * Edit the job that generates the certificates `webhook-init` by specifying the namespace and the service name of each instance of the operator, `e.g. command: ["/usr/bin/gencerts.sh", "-n", "ns-op1", "-s", "spark-op1-webhook", "-p"]`. Where `spark-op1-webhook` should match what you have specified in `webhook-svc-name`. For instance, if you use the following [helm chart](https://github.com/helm/charts/tree/master/incubator/sparkoperator) to deploy the operator you may specify for each instance of the operator a different `--namespace` and `--name-template` arguments to make sure you generate a different certificate for each instance, e.g:
 ```
 helm install spark-op1 incubator/sparkoperator --namespace ns-op1
diff --git a/docs/volcano-integration.md b/docs/volcano-integration.md
index 783ad55..d013167 100644
--- a/docs/volcano-integration.md
+++ b/docs/volcano-integration.md
@@ -31,11 +31,11 @@ metadata:
 spec:
   type: Scala
   mode: cluster
-  image: "gcr.io/spark-operator/spark:v3.0.0"
+  image: "gcr.io/spark-operator/spark:v3.1.1"
   imagePullPolicy: Always
   mainClass: org.apache.spark.examples.SparkPi
-  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0.jar"
-  sparkVersion: "3.0.0"
+  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-v3.1.1.jar"
+  sparkVersion: "3.1.1"
   batchScheduler: "volcano"   #Note: the batch scheduler name must be specified with `volcano`
   restartPolicy:
     type: Never
@@ -49,7 +49,7 @@ spec:
     coreLimit: "1200m"
     memory: "512m"        
     labels:
-      version: 3.0.0
+      version: 3.1.1
     serviceAccount: spark
     volumeMounts:
       - name: "test-volume"
@@ -59,7 +59,7 @@ spec:
     instances: 1
     memory: "512m"    
     labels:
-      version: 3.0.0
+      version: 3.1.1
     volumeMounts:
       - name: "test-volume"
         mountPath: "/tmp"
diff --git a/docs/who-is-using.md b/docs/who-is-using.md
index 4cb6187..ecc1021 100644
--- a/docs/who-is-using.md
+++ b/docs/who-is-using.md
@@ -30,6 +30,6 @@
 | [MongoDB](https://www.mongodb.com) | @chickenpopcorn | Production | Data Infrastructure |
 | [MavenCode](https://www.mavencode.com) | @charlesa101 | Production | MLOps & Data Infrastructure |
 | [Gojek](https://www.gojek.io/) | @pradithya | Production | Machine Learning Platform |
-| Fossil | @duyet | Evaluation | Data Platform |
+| Fossil | @duyet | Production | Data Platform |
 | Carrefour | @AliGouta | Production | Data Platform |
 | Scaling Smart | @tarek-izemrane | Evaluation | Data Platform |
diff --git a/examples/spark-pi-configmap.yaml b/examples/spark-pi-configmap.yaml
index 07200fc..ce0c7cf 100644
--- a/examples/spark-pi-configmap.yaml
+++ b/examples/spark-pi-configmap.yaml
@@ -21,11 +21,11 @@ metadata:
 spec:
   type: Scala
   mode: cluster
-  image: "gcr.io/spark-operator/spark:v3.0.0"
+  image: "gcr.io/spark-operator/spark:v3.1.1"
   imagePullPolicy: Always
   mainClass: org.apache.spark.examples.SparkPi
-  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0.jar"
-  sparkVersion: "3.0.0"
+  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.1.1.jar"
+  sparkVersion: "3.1.1"
   restartPolicy:
     type: Never
   volumes:
@@ -37,7 +37,7 @@ spec:
     coreLimit: "1200m"
     memory: "512m"
     labels:
-      version: 3.0.0
+      version: 3.1.1
     serviceAccount: spark
     volumeMounts:
       - name: config-vol
@@ -47,7 +47,7 @@ spec:
     instances: 1
     memory: "512m"
     labels:
-      version: 3.0.0
+      version: 3.1.1
     volumeMounts:
       - name: config-vol
         mountPath: /opt/spark/mycm
diff --git a/examples/spark-pi-custom-resource.yaml b/examples/spark-pi-custom-resource.yaml
index 3f27bd7..bae7cc4 100644
--- a/examples/spark-pi-custom-resource.yaml
+++ b/examples/spark-pi-custom-resource.yaml
@@ -21,11 +21,11 @@ metadata:
 spec:
   type: Scala
   mode: cluster
-  image: "gcr.io/spark-operator/spark:v2.4.5"
+  image: "gcr.io/spark-operator/spark:v3.1.1"
   imagePullPolicy: Always
   mainClass: org.apache.spark.examples.SparkPi
-  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar"
-  sparkVersion: "2.4.5"
+  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.1.1.jar"
+  sparkVersion: "3.1.1"
   restartPolicy:
     type: Never
   volumes:
@@ -38,7 +38,7 @@ spec:
     coreLimit: "1200m"
     memory: "512m"
     labels:
-      version: 2.4.5
+      version: 3.1.1
     serviceAccount: spark
     volumeMounts:
       - name: "test-volume"
@@ -48,11 +48,11 @@ spec:
     instances: 1
     memory: "512m"
     labels:
-      version: 2.4.5
+      version: 3.1.1
     volumeMounts:
       - name: "test-volume"
         mountPath: "/tmp"
   batchSchedulerOptions:
     resources:
         cpu: "2"
-        memory: "4096m"
\ No newline at end of file
+        memory: "4096m"
diff --git a/examples/spark-pi-prometheus.yaml b/examples/spark-pi-prometheus.yaml
index bd936aa..98298c1 100644
--- a/examples/spark-pi-prometheus.yaml
+++ b/examples/spark-pi-prometheus.yaml
@@ -22,13 +22,13 @@ metadata:
 spec:
   type: Scala
   mode: cluster
-  image: "gcr.io/spark-operator/spark:v3.0.0-gcs-prometheus"
+  image: "gcr.io/spark-operator/spark:v3.1.1-gcs-prometheus"
   imagePullPolicy: Always
   mainClass: org.apache.spark.examples.SparkPi
-  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0.jar"
+  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.1.1.jar"
   arguments:
     - "100000"
-  sparkVersion: "3.0.0"
+  sparkVersion: "3.1.1"
   restartPolicy:
     type: Never
   driver:
@@ -36,14 +36,14 @@ spec:
     coreLimit: "1200m"
     memory: "512m"
     labels:
-      version: 3.0.0
+      version: 3.1.1
     serviceAccount: spark
   executor:
     cores: 1
     instances: 1
     memory: "512m"
     labels:
-      version: 3.0.0
+      version: 3.1.1
   monitoring:
     exposeDriverMetrics: true
     exposeExecutorMetrics: true
diff --git a/examples/spark-pi-schedule.yaml b/examples/spark-pi-schedule.yaml
index f508c59..4af6b73 100644
--- a/examples/spark-pi-schedule.yaml
+++ b/examples/spark-pi-schedule.yaml
@@ -25,11 +25,11 @@ spec:
   template:
     type: Scala
     mode: cluster
-    image: "gcr.io/spark-operator/spark:v3.0.0"
+    image: "gcr.io/spark-operator/spark:v3.1.1"
     imagePullPolicy: Always
     mainClass: org.apache.spark.examples.SparkPi
-    mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0.jar"
-    sparkVersion: "3.0.0"
+    mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.1.1.jar"
+    sparkVersion: "3.1.1"
     restartPolicy:
       type: Never
     driver:
@@ -37,11 +37,11 @@ spec:
       coreLimit: "1200m"
       memory: "512m"
       labels:
-        version: 3.0.0
+        version: 3.1.1
       serviceAccount: spark
     executor:
       cores: 1
       instances: 1
       memory: "512m"
       labels:
-        version: 3.0.0 
+        version: 3.1.1
diff --git a/examples/spark-pi.yaml b/examples/spark-pi.yaml
index e0ad2d1..1f7fafa 100644
--- a/examples/spark-pi.yaml
+++ b/examples/spark-pi.yaml
@@ -21,11 +21,11 @@ metadata:
 spec:
   type: Scala
   mode: cluster
-  image: "gcr.io/spark-operator/spark:v3.0.0"
+  image: "gcr.io/spark-operator/spark:v3.1.1"
   imagePullPolicy: Always
   mainClass: org.apache.spark.examples.SparkPi
-  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0.jar"
-  sparkVersion: "3.0.0"
+  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.1.1.jar"
+  sparkVersion: "3.1.1"
   restartPolicy:
     type: Never
   volumes:
@@ -38,7 +38,7 @@ spec:
     coreLimit: "1200m"
     memory: "512m"
     labels:
-      version: 3.0.0
+      version: 3.1.1
     serviceAccount: spark
     volumeMounts:
       - name: "test-volume"
@@ -48,7 +48,7 @@ spec:
     instances: 1
     memory: "512m"
     labels:
-      version: 3.0.0
+      version: 3.1.1
     volumeMounts:
       - name: "test-volume"
         mountPath: "/tmp"
diff --git a/examples/spark-py-pi.yaml b/examples/spark-py-pi.yaml
index ee8639a..d601262 100644
--- a/examples/spark-py-pi.yaml
+++ b/examples/spark-py-pi.yaml
@@ -25,10 +25,10 @@ spec:
   type: Python
   pythonVersion: "2"
   mode: cluster
-  image: "gcr.io/spark-operator/spark-py:v3.0.0"
+  image: "gcr.io/spark-operator/spark-py:v3.1.1"
   imagePullPolicy: Always
   mainApplicationFile: local:///opt/spark/examples/src/main/python/pi.py
-  sparkVersion: "3.0.0"
+  sparkVersion: "3.1.1"
   restartPolicy:
     type: OnFailure
     onFailureRetries: 3
@@ -40,11 +40,11 @@ spec:
     coreLimit: "1200m"
     memory: "512m"
     labels:
-      version: 3.0.0
+      version: 3.1.1
     serviceAccount: spark
   executor:
     cores: 1
     instances: 1
     memory: "512m"
     labels:
-      version: 3.0.0
+      version: 3.1.1
diff --git a/hack/gencerts.sh b/hack/gencerts.sh
index 6387079..14d3619 100755
--- a/hack/gencerts.sh
+++ b/hack/gencerts.sh
@@ -18,6 +18,7 @@
 
 set -e
 SCRIPT=`basename ${BASH_SOURCE[0]}`
+RESOURCE_NAME="spark-webhook-certs"
 
 function usage {
   cat<< EOF
@@ -27,6 +28,7 @@ function usage {
   -n | --namespace <namespace>   The namespace where the Spark operator is installed.			
   -s | --service <service>       The name of the webhook service.
   -p | --in-pod                  Whether the script is running inside a pod or not.
+  -r | --resource-name           The spark resource name that will hold the secret [default: $RESOURCE_NAME]
 EOF
 }
 
@@ -59,6 +61,16 @@ function parse_arguments {
       shift 1
       continue
       ;;
+      -r|--resource-name)
+      if [[ -n "$2" ]]; then
+        RESOURCE_NAME="$2"
+      else
+        echo "-r or --resource-name requires a value."
+        exit 1
+      fi
+      shift 2
+      continue
+      ;;
       -h|--help)
       usage
       exit 0
@@ -135,7 +147,7 @@ if [[ "$IN_POD" == "true" ]];  then
 	  "kind": "Secret",
 	  "apiVersion": "v1",
 	  "metadata": {
-	    "name": "spark-webhook-certs",
+	    "name": "'"$RESOURCE_NAME"'",
 	    "namespace": "'"$NAMESPACE"'"
 	  },
 	  "data": {
@@ -162,7 +174,7 @@ if [[ "$IN_POD" == "true" ]];  then
      ;;
   esac
 else
-  kubectl create secret --namespace=${NAMESPACE} generic spark-webhook-certs --from-file=${TMP_DIR}/ca-key.pem --from-file=${TMP_DIR}/ca-cert.pem --from-file=${TMP_DIR}/server-key.pem --from-file=${TMP_DIR}/server-cert.pem
+  kubectl create secret --namespace=${NAMESPACE} generic ${RESOURCE_NAME} --from-file=${TMP_DIR}/ca-key.pem --from-file=${TMP_DIR}/ca-cert.pem --from-file=${TMP_DIR}/server-key.pem --from-file=${TMP_DIR}/server-cert.pem
 fi
 
 # Clean up after we're done.
diff --git a/manifest/crds/sparkoperator.k8s.io_sparkapplications.yaml b/manifest/crds/sparkoperator.k8s.io_sparkapplications.yaml
index 709d44d..8032f79 100644
--- a/manifest/crds/sparkoperator.k8s.io_sparkapplications.yaml
+++ b/manifest/crds/sparkoperator.k8s.io_sparkapplications.yaml
@@ -3672,6 +3672,8 @@ spec:
                 servicePort:
                   format: int32
                   type: integer
+                servicePortName:
+                  type: string
                 serviceType:
                   type: string
               type: object
diff --git a/manifest/spark-operator-with-metrics.yaml b/manifest/spark-operator-with-metrics.yaml
index 8130805..1a0e4cc 100644
--- a/manifest/spark-operator-with-metrics.yaml
+++ b/manifest/spark-operator-with-metrics.yaml
@@ -21,13 +21,13 @@ metadata:
   namespace: spark-operator
   labels:
     app.kubernetes.io/name: sparkoperator
-    app.kubernetes.io/version: v1beta2-1.2.0-3.0.0
+    app.kubernetes.io/version: v1beta2-1.2.3-3.1.1
 spec:
   replicas: 1
   selector:
     matchLabels:
       app.kubernetes.io/name: sparkoperator
-      app.kubernetes.io/version: v1beta2-1.2.0-3.0.0
+      app.kubernetes.io/version: v1beta2-1.2.3-3.1.1
   strategy:
     type: Recreate
   template:
@@ -38,12 +38,12 @@ spec:
         prometheus.io/path: "/metrics"
       labels:
         app.kubernetes.io/name: sparkoperator
-        app.kubernetes.io/version: v1beta2-1.2.0-3.0.0
+        app.kubernetes.io/version: v1beta2-1.2.3-3.1.1
     spec:
       serviceAccountName: sparkoperator
       containers:
       - name: sparkoperator
-        image: gcr.io/spark-operator/spark-operator:v1beta2-1.2.0-3.0.0
+        image: gcr.io/spark-operator/spark-operator:v1beta2-1.2.3-3.1.1
         imagePullPolicy: Always
         ports:
           - containerPort: 10254
diff --git a/manifest/spark-operator-with-webhook.yaml b/manifest/spark-operator-with-webhook.yaml
index c206243..fa90b12 100644
--- a/manifest/spark-operator-with-webhook.yaml
+++ b/manifest/spark-operator-with-webhook.yaml
@@ -21,20 +21,20 @@ metadata:
   namespace: spark-operator
   labels:
     app.kubernetes.io/name: sparkoperator
-    app.kubernetes.io/version: v1beta2-1.2.0-3.0.0
+    app.kubernetes.io/version: v1beta2-1.2.3-3.1.1
 spec:
   replicas: 1
   selector:
     matchLabels:
       app.kubernetes.io/name: sparkoperator
-      app.kubernetes.io/version: v1beta2-1.2.0-3.0.0
+      app.kubernetes.io/version: v1beta2-1.2.3-3.1.1
   strategy:
     type: Recreate
   template:
     metadata:
       labels:
         app.kubernetes.io/name: sparkoperator
-        app.kubernetes.io/version: v1beta2-1.2.0-3.0.0
+        app.kubernetes.io/version: v1beta2-1.2.3-3.1.1
     spec:
       serviceAccountName: sparkoperator
       volumes:
@@ -43,7 +43,7 @@ spec:
           secretName: spark-webhook-certs
       containers:
       - name: sparkoperator
-        image: gcr.io/spark-operator/spark-operator:v1beta2-1.2.0-3.0.0
+        image: gcr.io/spark-operator/spark-operator:v1beta2-1.2.3-3.1.1
         imagePullPolicy: Always
         volumeMounts:
         - name: webhook-certs
@@ -62,20 +62,20 @@ metadata:
   namespace: spark-operator
   labels:
     app.kubernetes.io/name: sparkoperator
-    app.kubernetes.io/version: v1beta2-1.2.0-3.0.0
+    app.kubernetes.io/version: v1beta2-1.2.3-3.1.1
 spec:
   backoffLimit: 3
   template:
     metadata:
       labels:
         app.kubernetes.io/name: sparkoperator
-        app.kubernetes.io/version: v1beta2-1.2.0-3.0.0
+        app.kubernetes.io/version: v1beta2-1.2.3-3.1.1
     spec:
       serviceAccountName: sparkoperator
       restartPolicy: Never
       containers:
       - name: main
-        image: gcr.io/spark-operator/spark-operator:v1beta2-1.2.0-3.0.0
+        image: gcr.io/spark-operator/spark-operator:v1beta2-1.2.3-3.1.1
         imagePullPolicy: IfNotPresent
         command: ["/usr/bin/gencerts.sh", "-p"]
 ---
@@ -91,4 +91,4 @@ spec:
       name: webhook
   selector:
     app.kubernetes.io/name: sparkoperator
-    app.kubernetes.io/version: v1beta2-1.2.0-3.0.0
+    app.kubernetes.io/version: v1beta2-1.2.3-3.1.1
diff --git a/manifest/spark-operator.yaml b/manifest/spark-operator.yaml
index 8b827a2..41f53b9 100644
--- a/manifest/spark-operator.yaml
+++ b/manifest/spark-operator.yaml
@@ -21,25 +21,25 @@ metadata:
   namespace: spark-operator
   labels:
     app.kubernetes.io/name: sparkoperator
-    app.kubernetes.io/version: v1beta2-1.2.0-3.0.0
+    app.kubernetes.io/version: v1beta2-1.2.3-3.1.1
 spec:
   replicas: 1
   selector:
     matchLabels:
       app.kubernetes.io/name: sparkoperator
-      app.kubernetes.io/version: v1beta2-1.2.0-3.0.0
+      app.kubernetes.io/version: v1beta2-1.2.3-3.1.1
   strategy:
     type: Recreate
   template:
     metadata:
       labels:
         app.kubernetes.io/name: sparkoperator
-        app.kubernetes.io/version: v1beta2-1.2.0-3.0.0
+        app.kubernetes.io/version: v1beta2-1.2.3-3.1.1
     spec:
       serviceAccountName: sparkoperator
       containers:
       - name: sparkoperator
-        image: gcr.io/spark-operator/spark-operator:v1beta2-1.2.0-3.0.0
+        image: gcr.io/spark-operator/spark-operator:v1beta2-1.2.3-3.1.1
         imagePullPolicy: Always
         args:
         - -logtostderr
diff --git a/pkg/apis/sparkoperator.k8s.io/v1beta2/types.go b/pkg/apis/sparkoperator.k8s.io/v1beta2/types.go
index ada9c40..83376b2 100644
--- a/pkg/apis/sparkoperator.k8s.io/v1beta2/types.go
+++ b/pkg/apis/sparkoperator.k8s.io/v1beta2/types.go
@@ -306,6 +306,11 @@ type SparkUIConfiguration struct {
 	// TargetPort should be the same as the one defined in spark.ui.port
 	// +optional
 	ServicePort *int32 `json:"servicePort"`
+	// ServicePortName allows configuring the name of the service port.
+	// This may be useful for sidecar proxies like Envoy injected by Istio which require specific ports names to treat traffic as proper HTTP.
+	// Defaults to spark-driver-ui-port.
+	// +optional
+	ServicePortName *string `json:"servicePortName"`
 	// ServiceType allows configuring the type of the service. Defaults to ClusterIP.
 	// +optional
 	ServiceType *apiv1.ServiceType `json:"serviceType"`
diff --git a/pkg/controller/sparkapplication/sparkui.go b/pkg/controller/sparkapplication/sparkui.go
index f9f15e3..80e1537 100644
--- a/pkg/controller/sparkapplication/sparkui.go
+++ b/pkg/controller/sparkapplication/sparkui.go
@@ -36,8 +36,9 @@ import (
 )
 
 const (
-	sparkUIPortConfigurationKey       = "spark.ui.port"
-	defaultSparkWebUIPort       int32 = 4040
+	sparkUIPortConfigurationKey        = "spark.ui.port"
+	defaultSparkWebUIPort       int32  = 4040
+	defaultSparkWebUIPortName   string = "spark-driver-ui-port"
 )
 
 var ingressAppNameURLRegex = regexp.MustCompile("{{\\s*[$]appName\\s*}}")
@@ -49,11 +50,12 @@ func getSparkUIingressURL(ingressURLFormat string, appName string, appNamespace
 
 // SparkService encapsulates information about the driver UI service.
 type SparkService struct {
-	serviceName string
-	serviceType apiv1.ServiceType
-	servicePort int32
-	targetPort  intstr.IntOrString
-	serviceIP   string
+	serviceName     string
+	serviceType     apiv1.ServiceType
+	servicePort     int32
+	servicePortName string
+	targetPort      intstr.IntOrString
+	serviceIP       string
 }
 
 // SparkIngress encapsulates information about the driver UI ingress.
@@ -132,6 +134,7 @@ func createSparkUIIngress(app *v1beta2.SparkApplication, service SparkService, i
 func createSparkUIService(
 	app *v1beta2.SparkApplication,
 	kubeClient clientset.Interface) (*SparkService, error) {
+	portName := getUIServicePortName(app)
 	port, err := getUIServicePort(app)
 	if err != nil {
 		return nil, fmt.Errorf("invalid Spark UI servicePort: %d", port)
@@ -150,7 +153,7 @@ func createSparkUIService(
 		Spec: apiv1.ServiceSpec{
 			Ports: []apiv1.ServicePort{
 				{
-					Name: "spark-driver-ui-port",
+					Name: portName,
 					Port: port,
 					TargetPort: intstr.IntOrString{
 						Type:   intstr.Int,
@@ -173,11 +176,12 @@ func createSparkUIService(
 	}
 
 	return &SparkService{
-		serviceName: service.Name,
-		serviceType: service.Spec.Type,
-		servicePort: service.Spec.Ports[0].Port,
-		targetPort:  service.Spec.Ports[0].TargetPort,
-		serviceIP:   service.Spec.ClusterIP,
+		serviceName:     service.Name,
+		serviceType:     service.Spec.Type,
+		servicePort:     service.Spec.Ports[0].Port,
+		servicePortName: service.Spec.Ports[0].Name,
+		targetPort:      service.Spec.Ports[0].TargetPort,
+		serviceIP:       service.Spec.ClusterIP,
 	}, nil
 }
 
@@ -194,7 +198,6 @@ func getUITargetPort(app *v1beta2.SparkApplication) (int32, error) {
 }
 
 func getUIServicePort(app *v1beta2.SparkApplication) (int32, error) {
-
 	if app.Spec.SparkUIOptions == nil {
 		return getUITargetPort(app)
 	}
@@ -204,3 +207,14 @@ func getUIServicePort(app *v1beta2.SparkApplication) (int32, error) {
 	}
 	return defaultSparkWebUIPort, nil
 }
+
+func getUIServicePortName(app *v1beta2.SparkApplication) string {
+	if app.Spec.SparkUIOptions == nil {
+		return defaultSparkWebUIPortName
+	}
+	portName := app.Spec.SparkUIOptions.ServicePortName
+	if portName != nil {
+		return *portName
+	}
+	return defaultSparkWebUIPortName
+}
diff --git a/pkg/webhook/patch.go b/pkg/webhook/patch.go
index a6f54f6..870122e 100644
--- a/pkg/webhook/patch.go
+++ b/pkg/webhook/patch.go
@@ -560,6 +560,8 @@ func addPriorityClassName(pod *corev1.Pod, app *v1beta2.SparkApplication) []patc
 
 		if pod.Spec.Priority != nil {
 			ops = append(ops, patchOperation{Op: "remove", Path: "/spec/priority"})
+		}
+		if pod.Spec.PreemptionPolicy != nil {
 			ops = append(ops, patchOperation{Op: "remove", Path: "/spec/preemptionPolicy"})
 		}
 	}
diff --git a/spark-docker/Dockerfile b/spark-docker/Dockerfile
index edf21ec..cbc7223 100644
--- a/spark-docker/Dockerfile
+++ b/spark-docker/Dockerfile
@@ -14,7 +14,7 @@
 # limitations under the License.
 #
 
-ARG SPARK_IMAGE=gcr.io/spark-operator/spark:v3.0.0
+ARG SPARK_IMAGE=gcr.io/spark-operator/spark:v3.1.1
 FROM ${SPARK_IMAGE}
 
 # Switch to user root so we can add additional jars and configuration files.


Test patch:
diff --git a/pkg/controller/sparkapplication/sparkui_test.go b/pkg/controller/sparkapplication/sparkui_test.go
index c182479..81b92a8 100644
--- a/pkg/controller/sparkapplication/sparkui_test.go
+++ b/pkg/controller/sparkapplication/sparkui_test.go
@@ -77,11 +77,15 @@ func TestCreateSparkUIService(t *testing.T) {
 		if port.Port != test.expectedService.servicePort {
 			t.Errorf("%s: unexpected port wanted %d got %d", test.name, test.expectedService.servicePort, port.Port)
 		}
+		if port.Name != test.expectedService.servicePortName {
+			t.Errorf("%s: unexpected port name wanted %s got %s", test.name, test.expectedService.servicePortName, port.Name)
+		}
 	}
 	defaultPort := defaultSparkWebUIPort
+	defaultPortName := defaultSparkWebUIPortName
 	app1 := &v1beta2.SparkApplication{
 		ObjectMeta: metav1.ObjectMeta{
-			Name:      "foo",
+			Name:      "foo1",
 			Namespace: "default",
 			UID:       "foo-123",
 		},
@@ -97,7 +101,7 @@ func TestCreateSparkUIService(t *testing.T) {
 	}
 	app2 := &v1beta2.SparkApplication{
 		ObjectMeta: metav1.ObjectMeta{
-			Name:      "foo",
+			Name:      "foo2",
 			Namespace: "default",
 			UID:       "foo-123",
 		},
@@ -108,7 +112,7 @@ func TestCreateSparkUIService(t *testing.T) {
 	}
 	app3 := &v1beta2.SparkApplication{
 		ObjectMeta: metav1.ObjectMeta{
-			Name:      "foo",
+			Name:      "foo3",
 			Namespace: "default",
 			UID:       "foo-123",
 		},
@@ -124,7 +128,7 @@ func TestCreateSparkUIService(t *testing.T) {
 	var appPort int32 = 80
 	app4 := &v1beta2.SparkApplication{
 		ObjectMeta: metav1.ObjectMeta{
-			Name:      "foo",
+			Name:      "foo4",
 			Namespace: "default",
 			UID:       "foo-123",
 		},
@@ -145,7 +149,7 @@ func TestCreateSparkUIService(t *testing.T) {
 	var serviceTypeNodePort apiv1.ServiceType = apiv1.ServiceTypeNodePort
 	app5 := &v1beta2.SparkApplication{
 		ObjectMeta: metav1.ObjectMeta{
-			Name:      "foo",
+			Name:      "foo5",
 			Namespace: "default",
 			UID:       "foo-123",
 		},
@@ -159,21 +163,39 @@ func TestCreateSparkUIService(t *testing.T) {
 			ExecutionAttempts:  2,
 		},
 	}
+	appPortName := "http-spark-test"
+	app6 := &v1beta2.SparkApplication{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "foo6",
+			Namespace: "default",
+			UID:       "foo-123",
+		},
+		Spec: v1beta2.SparkApplicationSpec{
+			SparkUIOptions: &v1beta2.SparkUIConfiguration{
+				ServicePort:     &appPort,
+				ServicePortName: &appPortName,
+			},
+		},
+		Status: v1beta2.SparkApplicationStatus{
+			SparkApplicationID: "foo-6",
+		},
+	}
 	testcases := []testcase{
 		{
 			name: "service with custom serviceport and serviceport and target port are same",
 			app:  app1,
 			expectedService: SparkService{
-				serviceName: fmt.Sprintf("%s-ui-svc", app1.GetName()),
-				serviceType: apiv1.ServiceTypeClusterIP,
-				servicePort: 4041,
+				serviceName:     fmt.Sprintf("%s-ui-svc", app1.GetName()),
+				serviceType:     apiv1.ServiceTypeClusterIP,
+				servicePortName: defaultPortName,
+				servicePort:     4041,
 				targetPort: intstr.IntOrString{
 					Type:   intstr.Int,
 					IntVal: int32(4041),
 				},
 			},
 			expectedSelector: map[string]string{
-				config.SparkAppNameLabel: "foo",
+				config.SparkAppNameLabel: "foo1",
 				config.SparkRoleLabel:    config.SparkDriverRole,
 			},
 			expectError: false,
@@ -182,12 +204,13 @@ func TestCreateSparkUIService(t *testing.T) {
 			name: "service with default port",
 			app:  app2,
 			expectedService: SparkService{
-				serviceName: fmt.Sprintf("%s-ui-svc", app2.GetName()),
-				serviceType: apiv1.ServiceTypeClusterIP,
-				servicePort: int32(defaultPort),
+				serviceName:     fmt.Sprintf("%s-ui-svc", app2.GetName()),
+				serviceType:     apiv1.ServiceTypeClusterIP,
+				servicePortName: defaultPortName,
+				servicePort:     int32(defaultPort),
 			},
 			expectedSelector: map[string]string{
-				config.SparkAppNameLabel: "foo",
+				config.SparkAppNameLabel: "foo2",
 				config.SparkRoleLabel:    config.SparkDriverRole,
 			},
 			expectError: false,
@@ -196,16 +219,17 @@ func TestCreateSparkUIService(t *testing.T) {
 			name: "service with custom serviceport and serviceport and target port are different",
 			app:  app4,
 			expectedService: SparkService{
-				serviceName: fmt.Sprintf("%s-ui-svc", app4.GetName()),
-				serviceType: apiv1.ServiceTypeClusterIP,
-				servicePort: 80,
+				serviceName:     fmt.Sprintf("%s-ui-svc", app4.GetName()),
+				serviceType:     apiv1.ServiceTypeClusterIP,
+				servicePortName: defaultPortName,
+				servicePort:     80,
 				targetPort: intstr.IntOrString{
 					Type:   intstr.Int,
 					IntVal: int32(4041),
 				},
 			},
 			expectedSelector: map[string]string{
-				config.SparkAppNameLabel: "foo",
+				config.SparkAppNameLabel: "foo4",
 				config.SparkRoleLabel:    config.SparkDriverRole,
 			},
 			expectError: false,
@@ -214,12 +238,28 @@ func TestCreateSparkUIService(t *testing.T) {
 			name: "service with custom servicetype",
 			app:  app5,
 			expectedService: SparkService{
-				serviceName: fmt.Sprintf("%s-ui-svc", app4.GetName()),
-				serviceType: apiv1.ServiceTypeNodePort,
-				servicePort: int32(defaultPort),
+				serviceName:     fmt.Sprintf("%s-ui-svc", app5.GetName()),
+				serviceType:     apiv1.ServiceTypeNodePort,
+				servicePortName: defaultPortName,
+				servicePort:     int32(defaultPort),
+			},
+			expectedSelector: map[string]string{
+				config.SparkAppNameLabel: "foo5",
+				config.SparkRoleLabel:    config.SparkDriverRole,
+			},
+			expectError: false,
+		},
+		{
+			name: "service with custom serviceportname",
+			app:  app6,
+			expectedService: SparkService{
+				serviceName:     fmt.Sprintf("%s-ui-svc", app6.GetName()),
+				serviceType:     apiv1.ServiceTypeClusterIP,
+				servicePortName: "http-spark-test",
+				servicePort:     int32(80),
 			},
 			expectedSelector: map[string]string{
-				config.SparkAppNameLabel: "foo",
+				config.SparkAppNameLabel: "foo6",
 				config.SparkRoleLabel:    config.SparkDriverRole,
 			},
 			expectError: false,
diff --git a/test/e2e/README.md b/test/e2e/README.md
index 64859d2..9c4f6e7 100644
--- a/test/e2e/README.md
+++ b/test/e2e/README.md
@@ -12,7 +12,7 @@ Prerequisites:
 e2e tests are written as Go test. All go test techniques apply (e.g. picking what to run, timeout length). Let's say I want to run all tests in "test/e2e/":
 
 ```bash
-$ go test -v ./test/e2e/ --kubeconfig "$HOME/.kube/config" --operator-image=gcr.io/spark-operator/spark-operator:v1beta2-1.2.0-3.0.0
+$ go test -v ./test/e2e/ --kubeconfig "$HOME/.kube/config" --operator-image=gcr.io/spark-operator/spark-operator:v1beta2-1.2.3-3.1.1
 ```
 
 ### Available Tests