Repository: volcano-sh/volcano (Go)

Issue:
Volcano multi-copy deployment does not enable master election by default
<!-- This form is for bug reports ONLY!

If you're looking for a help then check our [Slack Channel](https://cloud-native.slack.com/messages/volcano) or have a look at our [dev mailing](https://groups.google.com/forum/#!forum/volcano-sh) 
-->

**What happened**:
Deploying Volcano in the form of multiple copies, the vc-controller and vc-scheduler components do not enable the master selection function by default, and multiple vc-schedulers work at the same time, there may be a problem that the scheduled pod fails to pull up on the node node, such as encountering ' ouf of cpu' and other errors.

**What you expected to happen**:
The master election function is enabled by default, and only one scheduler works in the case of multi-copy deployment. 
**How to reproduce it (as minimally and precisely as possible)**:

- The default value is empty (false)
https://github.com/volcano-sh/volcano/blob/0d3fd2b3a22a32b0fee75e6d816f3545f3426499/cmd/scheduler/app/options/options.go#L102

- Scheduling will be enabled in the case of multiple copies
https://github.com/volcano-sh/volcano/blob/0d3fd2b3a22a32b0fee75e6d816f3545f3426499/cmd/scheduler/app/server.go#L106-L115

**Anything else we need to know?**:

**Environment**:
- Volcano Version: master
- Kubernetes version (use `kubectl version`):
- Cloud provider or hardware configuration:
- OS (e.g. from /etc/os-release):
- Kernel (e.g. `uname -a`):
- Install tools:
- Others:

service account has no permit to access lease when enable leader elect

**What happened**:

```sh
E0527 06:15:36.041286       1 leaderelection.go:330] error retrieving resource lock volcano-system/volcano: leases.coordination.k8s.io "volcano" is forbidden: User "system:serviceaccount:volcano-system:volcano-scheduler" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "volcano-system"
E0527 06:15:41.445378       1 leaderelection.go:330] error retrieving resource lock volcano-system/volcano: leases.coordination.k8s.io "volcano" is forbidden: User "system:serviceaccount:volcano-system:volcano-scheduler" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "volcano-system"
E0527 06:15:47.393144       1 leaderelection.go:330] error retrieving resource lock volcano-system/volcano: leases.coordination.k8s.io "volcano" is forbidden: User "system:serviceaccount:volcano-system:volcano-scheduler" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "volcano-system"
E0527 06:15:52.987215       1 leaderelection.go:330] error retrieving resource lock volcano-system/volcano: leases.coordination.k8s.io "volcano" is forbidden: User "system:serviceaccount:volcano-system:volcano-scheduler" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "volcano-system"
```

**What you expected to happen**:

no error and get lease successfully

**How to reproduce it (as minimally and precisely as possible)**:

start up with leader-elect enabled 
```yaml
     - --leader-elect=true
     - --lock-object-namespace=volcano-system
```

**Anything else we need to know?**:

both scheduler and controller

**Environment**:
- Volcano Version: master branch
- Kubernetes version (use `kubectl version`):
- Cloud provider or hardware configuration:
- OS (e.g. from /etc/os-release):
- Kernel (e.g. `uname -a`):
- Install tools:
- Others:



Discussion:
/assign

/assign

fix by pr #2828



Patch:
diff --git a/cmd/controller-manager/app/options/options.go b/cmd/controller-manager/app/options/options.go
index ab0fd92847..31b59554d7 100644
--- a/cmd/controller-manager/app/options/options.go
+++ b/cmd/controller-manager/app/options/options.go
@@ -34,6 +34,7 @@ const (
 	defaultSchedulerName                  = "volcano"
 	defaultHealthzAddress                 = ":11251"
 	defaultDetectionPeriodOfDependsOntask = 100 * time.Millisecond
+	defaultLockObjectNamespace            = "volcano-system"
 )
 
 // ServerOption is the main context object for the controllers.
@@ -81,9 +82,9 @@ func (s *ServerOption) AddFlags(fs *pflag.FlagSet) {
 		"File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated "+
 		"after server cert).")
 	fs.StringVar(&s.KeyFile, "tls-private-key-file", s.KeyFile, "File containing the default x509 private key matching --tls-cert-file.")
-	fs.BoolVar(&s.EnableLeaderElection, "leader-elect", s.EnableLeaderElection, "Start a leader election client and gain leadership before "+
-		"executing the main loop. Enable this when running replicated vc-controller-manager for high availability.")
-	fs.StringVar(&s.LockObjectNamespace, "lock-object-namespace", s.LockObjectNamespace, "Define the namespace of the lock object.")
+	fs.BoolVar(&s.EnableLeaderElection, "leader-elect", true, "Start a leader election client and gain leadership before "+
+		"executing the main loop. Enable this when running replicated vc-controller-manager for high availability; it is enabled by default")
+	fs.StringVar(&s.LockObjectNamespace, "lock-object-namespace", defaultLockObjectNamespace, "Define the namespace of the lock object; it is volcano-system by default")
 	fs.Float32Var(&s.KubeClientOptions.QPS, "kube-api-qps", defaultQPS, "QPS to use while talking with kubernetes apiserver")
 	fs.IntVar(&s.KubeClientOptions.Burst, "kube-api-burst", defaultBurst, "Burst to use while talking with kubernetes apiserver")
 	fs.BoolVar(&s.PrintVersion, "version", false, "Show version and quit")
diff --git a/cmd/scheduler/app/options/options.go b/cmd/scheduler/app/options/options.go
index 5f134aba21..7d91e203ab 100644
--- a/cmd/scheduler/app/options/options.go
+++ b/cmd/scheduler/app/options/options.go
@@ -41,6 +41,7 @@ const (
 	defaultMinPercentageOfNodesToFind = 5
 	defaultMinNodesToFind             = 100
 	defaultPercentageOfNodesToFind    = 100
+	defaultLockObjectNamespace        = "volcano-system"
 )
 
 // ServerOption is the main context object for the controller manager.
@@ -99,11 +100,11 @@ func (s *ServerOption) AddFlags(fs *pflag.FlagSet) {
 	fs.StringVar(&s.SchedulerConf, "scheduler-conf", "", "The absolute path of scheduler configuration file")
 	fs.DurationVar(&s.SchedulePeriod, "schedule-period", defaultSchedulerPeriod, "The period between each scheduling cycle")
 	fs.StringVar(&s.DefaultQueue, "default-queue", defaultQueue, "The default queue name of the job")
-	fs.BoolVar(&s.EnableLeaderElection, "leader-elect", s.EnableLeaderElection,
+	fs.BoolVar(&s.EnableLeaderElection, "leader-elect", true,
 		"Start a leader election client and gain leadership before "+
-			"executing the main loop. Enable this when running replicated vc-scheduler for high availability")
+			"executing the main loop. Enable this when running replicated vc-scheduler for high availability; it is enabled by default")
 	fs.BoolVar(&s.PrintVersion, "version", false, "Show version and quit")
-	fs.StringVar(&s.LockObjectNamespace, "lock-object-namespace", s.LockObjectNamespace, "Define the namespace of the lock object that is used for leader election")
+	fs.StringVar(&s.LockObjectNamespace, "lock-object-namespace", defaultLockObjectNamespace, "Define the namespace of the lock object that is used for leader election; it is volcano-system by default")
 	fs.StringVar(&s.ListenAddress, "listen-address", defaultListenAddress, "The address to listen on for HTTP requests.")
 	fs.StringVar(&s.HealthzBindAddress, "healthz-address", defaultHealthzAddress, "The address to listen on for the health check server.")
 	fs.BoolVar(&s.EnablePriorityClass, "priority-class", true,
diff --git a/installer/README.md b/installer/README.md
index 93172b1e43..63dd8c1ea3 100644
--- a/installer/README.md
+++ b/installer/README.md
@@ -14,10 +14,13 @@ Kubernetes that are commonly required by many classes of batch & elastic workloa
 ## Installing volcano via yaml file
 
 All-in-one yaml has been generated for quick deployment. Try command:
+
 ```$xslt
 kubectl apply -f volcano-v0.0.x.yaml
 ```
+
 Check the status in namespace `volcano-system`
+
 ```$xslt
 $kubectl get all -n volcano-system
 NAME                                       READY   STATUS      RESTARTS   AGE
@@ -47,14 +50,22 @@ replicaset.apps/volcano-scheduler-94998fc64      1         1         1       36s
 To install the volcano with chart:
 
 ```bash
-helm install helm/chart/volcano --namespace <namespace> --name <specified-name>
+helm install <specified-name> helm/chart/volcano --namespace <namespace> --create-namespace
 
 e.g :
-helm install helm/chart/volcano --namespace volcano-trial --name volcano-trial
+helm install volcano-trial helm/chart/volcano --namespace volcano-trial --create-namespace
 ```
 
 This command deploys volcano in kubernetes cluster with default configuration.  The [configuration](#configuration) section lists the parameters that can be configured during installation.
 
+To list the volcano chart:
+
+```bash
+helm list -n <namespace>
+
+e.g:
+ helm list -n volcano-trial
+```
 
 ## Uninstalling the Chart
 
@@ -75,7 +86,7 @@ The following are the list configurable parameters of Volcano Chart and their de
 |`basic.admission_secret_name`|Volcano Admission Secret Name|`volcano-admission-secret`|
 |`basic.scheduler_config_file`|Configuration File name for Scheduler|`config/volcano-scheduler.conf`|
 |`basic.image_pull_secret`|Image Pull Secret|`""`|
-|`basic.image_pull_policy`|Image Pull Policy|`IfNotPresent`|
+|`basic.image_pull_policy`|Image Pull Policy|`Always`|
 |`basic.admission_app_name`|Admission Controller App Name|`volcano-admission`|
 |`basic.controller_app_name`|Controller App Name|`volcano-controller`|
 |`basic.scheduler_app_name`|Scheduler App Name|`volcano-scheduler`|
@@ -86,6 +97,7 @@ The following are the list configurable parameters of Volcano Chart and their de
 |`custom.controller_replicas`|The number of Controller pods to run|`1`|
 |`custom.scheduler_enable`|Whether to Enable Scheduler|`true`|
 |`custom.scheduler_replicas`|The number of Scheduler pods to run|`1`|
+|`custom.leader_elect_enable`|Whether to Enable leader elect|`false`|
 
 Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example,
 
@@ -95,7 +107,6 @@ $ helm install --name volcano-release --set basic.image_pull_policy=Always volca
 
 The above command set image pull policy to `Always`, so docker image will be pulled each time.
 
-
 Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,
 
 ```bash
diff --git a/installer/helm/chart/volcano/templates/controllers.yaml b/installer/helm/chart/volcano/templates/controllers.yaml
index 6869731c5b..f38d4ff527 100644
--- a/installer/helm/chart/volcano/templates/controllers.yaml
+++ b/installer/helm/chart/volcano/templates/controllers.yaml
@@ -62,6 +62,9 @@ rules:
   - apiGroups: ["batch"]
     resources: ["jobs"]
     verbs: ["get"]
+  - apiGroups: ["coordination.k8s.io"]
+    resources: ["leases"]
+    verbs: ["get", "create", "update", "watch"]
 ---
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
@@ -106,6 +109,10 @@ spec:
             args:
               - --logtostderr
               - --enable-healthz=true
+              - --leader-elect={{ .Values.custom.leader_elect_enable }}
+              {{- if .Values.custom.leader_elect_enable }}
+              - --lock-object-namespace={{ .Release.Namespace }}
+              {{- end }}
               - -v=4
               - 2>&1
             imagePullPolicy: {{ .Values.basic.image_pull_policy }}
diff --git a/installer/helm/chart/volcano/templates/scheduler.yaml b/installer/helm/chart/volcano/templates/scheduler.yaml
index 1bc1763234..a224e6e1e7 100644
--- a/installer/helm/chart/volcano/templates/scheduler.yaml
+++ b/installer/helm/chart/volcano/templates/scheduler.yaml
@@ -78,7 +78,9 @@ rules:
   - apiGroups: ["apps"]
     resources: ["daemonsets", "replicasets", "statefulsets"]
     verbs: ["list", "watch", "get"]
-
+  - apiGroups: ["coordination.k8s.io"]
+    resources: ["leases"]
+    verbs: ["get", "create", "update", "watch"]
 ---
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
@@ -125,6 +127,10 @@ spec:
             - --scheduler-conf=/volcano.scheduler/{{base .Values.basic.scheduler_config_file}}
             - --enable-healthz=true
             - --enable-metrics=true
+            - --leader-elect={{ .Values.custom.leader_elect_enable }}
+            {{- if .Values.custom.leader_elect_enable }}
+            - --lock-object-namespace={{ .Release.Namespace }}
+            {{- end }}
             - -v=3
             - 2>&1
           imagePullPolicy: {{ .Values.basic.image_pull_policy }}
diff --git a/installer/helm/chart/volcano/values.yaml b/installer/helm/chart/volcano/values.yaml
index 6dfe9ef234..0b29d9780a 100644
--- a/installer/helm/chart/volcano/values.yaml
+++ b/installer/helm/chart/volcano/values.yaml
@@ -17,6 +17,7 @@ custom:
   controller_replicas: 1
   scheduler_enable: true
   scheduler_replicas: 1
+  leader_elect_enable: false
   enabled_admissions: "/jobs/mutate,/jobs/validate,/podgroups/mutate,/pods/validate,/pods/mutate,/queues/mutate,/queues/validate"
 nodeSelector: {}
 affinity: []
diff --git a/installer/volcano-development.yaml b/installer/volcano-development.yaml
index 8436249ae6..40ec79d4b6 100644
--- a/installer/volcano-development.yaml
+++ b/installer/volcano-development.yaml
@@ -8573,6 +8573,9 @@ rules:
   - apiGroups: ["batch"]
     resources: ["jobs"]
     verbs: ["get"]
+  - apiGroups: ["coordination.k8s.io"]
+    resources: ["leases"]
+    verbs: ["get", "create", "update", "watch"]
 ---
 # Source: volcano/templates/controllers.yaml
 kind: ClusterRoleBinding
@@ -8614,6 +8617,7 @@ spec:
             args:
               - --logtostderr
               - --enable-healthz=true
+              - --leader-elect=false
               - -v=4
               - 2>&1
             imagePullPolicy: Always
@@ -8715,6 +8719,9 @@ rules:
   - apiGroups: ["apps"]
     resources: ["daemonsets", "replicasets", "statefulsets"]
     verbs: ["list", "watch", "get"]
+  - apiGroups: ["coordination.k8s.io"]
+    resources: ["leases"]
+    verbs: ["get", "create", "update", "watch"]
 ---
 # Source: volcano/templates/scheduler.yaml
 kind: ClusterRoleBinding
@@ -8780,6 +8787,7 @@ spec:
             - --scheduler-conf=/volcano.scheduler/volcano-scheduler.conf
             - --enable-healthz=true
             - --enable-metrics=true
+            - --leader-elect=false
             - -v=3
             - 2>&1
           imagePullPolicy: Always


Test patch:
diff --git a/cmd/controller-manager/app/options/options_test.go b/cmd/controller-manager/app/options/options_test.go
index 120718bfeb..e3a7f56b82 100644
--- a/cmd/controller-manager/app/options/options_test.go
+++ b/cmd/controller-manager/app/options/options_test.go
@@ -53,6 +53,8 @@ func TestAddFlags(t *testing.T) {
 		HealthzBindAddress:             ":11251",
 		DetectionPeriodOfDependsOntask: defaultDetectionPeriodOfDependsOntask,
 		InheritOwnerAnnotations:        true,
+		EnableLeaderElection:           true,
+		LockObjectNamespace:            defaultLockObjectNamespace,
 	}
 
 	if !reflect.DeepEqual(expected, s) {
diff --git a/cmd/scheduler/app/options/options_test.go b/cmd/scheduler/app/options/options_test.go
index 9d7e76b527..e815e6ff3e 100644
--- a/cmd/scheduler/app/options/options_test.go
+++ b/cmd/scheduler/app/options/options_test.go
@@ -55,6 +55,8 @@ func TestAddFlags(t *testing.T) {
 		MinNodesToFind:             defaultMinNodesToFind,
 		MinPercentageOfNodesToFind: defaultMinPercentageOfNodesToFind,
 		PercentageOfNodesToFind:    defaultPercentageOfNodesToFind,
+		EnableLeaderElection:       true,
+		LockObjectNamespace:        defaultLockObjectNamespace,
 	}
 
 	if !reflect.DeepEqual(expected, s) {