Repository: elastic/cloud-on-k8s

Issue:
Use a command for Kibana readiness probe due to security constraints with httpGet
<!--

** Please read the guidelines below. **

1.   GitHub is reserved for bug reports and feature requests. The best place to
     ask a general question is at the Elastic [forums](https://discuss.elastic.co).
     GitHub is not the place for general questions.
    
2.  Please fill out EITHER the feature request block or the bug report block
    below, and delete the other block.

-->

<!-- Feature request -->

## Proposal

**Use case. Why is this important?**

It is required for reliably checking for kibana pod containers readiness.

The current readiness seems very basic and not sure when it is expect to get the actual readiness.

Current readiness:
```
readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /login
        port: 5601
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
```

The above readiness probe just probes on the /login but may hide few other problems with the kibana container readiness itself. 

I feel it could be something to reliably to check the readiness, as done for elasticsearch containers.

Discussion:
@sarjeet2013: I just tried to see if this can be similar to elasticsearch container readiness and it seems to work fine.

curl -o /dev/null -w "%{http_code}" --max-time 3 -XGET -s -k -u default-kibana-1node-kibana-user:zx42xz5fwzqxdw5vtr8hzf7p  https://127.0.0.1:5601/login
200

This works as expected, but require the user and password which I retrieved from `<cr>-kibana-user` secret. I am not sure if this secret user can be used or it may require a new kibana probe user for readiness check.@sebgl: @sarjeet2013 I'm curious about errors cases you may have found that an authenticated curl against `/login` returned but not an unauthenticated one?@sarjeet2013: @sebgl There are 2 reasons for this issue.

1. Currently, our k8s env blocks traffic from host to container (due to security reasons) which blocks the readinessProbe with httpGet.

2. Regarding authenticated vs unauthenticated, AFAICT both returns 200 fine.
```
➜  curl -o /dev/null -w "%{http_code}" --max-time 3 -XGET -s -k -u default-kibana-1node-kibana-user:xxxx https://x.x.x.x:5601/login
200                                                                                                                                                                                                   

➜  curl -o /dev/null -w "%{http_code}" https://x.x.x.x:5601/login -k -s
200
```
But, It would be preferable if the readinessProbe request could be authenticated, similar to ES using exec based readinessProbe.@sebgl: > Currently, our k8s env blocks traffic from host to container (due to security reasons) which blocks the readinessProbe with httpGet.

Indeed, that sounds like a good reason.

As a workaround, you can set your own readiness probe, by overriding the podTemplate section of the Kibana resource.
@sarjeet2013: @sebgl Yes. Using this workaround already :)

```
podTemplate:
    spec:
      containers:
        - name: kibana
          readinessProbe:
            failureThreshold: 3
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            exec:
              command:
                - /bin/bash
                - -c
                - curl -o /dev/null -w "%{http_code}" https://127.0.0.1:5601/login -k -s
```

Please let me know if the above suggestion of using command is the (preferred) workable solution. I am happy to work on it and make a PR for it.@sebgl: This may break existing ingresses configuration, especially on [GKE where the default ingress healthcheck binds to the same path as the `httpGet` readiness probe](https://cloud.google.com/kubernetes-engine/docs/concepts/ingress?hl=en#health_checks).
IMO this is a flaw in the ingress itself - which already has some other flaws such as the lack of TLS support in some cases. So I would consider it's still ok to remove the `httpGet` readiness probe and default to a command.@dongdong0421: > [@sebgl](https://github.com/sebgl)是的。已经在用这个解决方法了 :)
> 
> ```
> podTemplate:
>     spec:
>       containers:
>         - name: kibana
>           readinessProbe:
>             failureThreshold: 3
>             initialDelaySeconds: 10
>             periodSeconds: 10
>             successThreshold: 1
>             exec:
>               command:
>                 - /bin/bash
>                 - -c
>                 - curl -o /dev/null -w "%{http_code}" https://127.0.0.1:5601/login -k -s
> ```
> 
> 请让我知道上述使用命令的建议是否是（首选的）可行解决方案。我很乐意为此努力并为其创建 PR。

I don't seem to be able to follow the configuration settings above

Solution patch:
diff --git a/config/e2e/filebeat.yaml b/config/e2e/filebeat.yaml
index ba978a32c..871c1c24d 100644
--- a/config/e2e/filebeat.yaml
+++ b/config/e2e/filebeat.yaml
@@ -16,7 +16,7 @@ data:
           hints.default_config:
             type: container
             paths:
-              - /var/lib/docker/containers/${data.kubernetes.container.id}/*.log
+              - /var/log/containers/*${data.kubernetes.container.id}.log
 
     processors:
       - add_cloud_metadata:
@@ -100,6 +100,12 @@ spec:
             - name: varlibdockercontainers
               mountPath: /var/lib/docker/containers
               readOnly: true
+            - name: varlogpods
+              mountPath: /var/log/pods
+              readOnly: true
+            - name: varlogcontainers
+              mountPath: /var/log/containers
+              readOnly: true
             - name: monitoring-ca
               mountPath: /mnt/elastic/monitoring-ca.crt
               readOnly: true
@@ -112,6 +118,12 @@ spec:
         - name: varlibdockercontainers
           hostPath:
             path: /var/lib/docker/containers
+        - name: varlogcontainers
+          hostPath:
+            path: /var/log/containers
+        - name: varlogpods
+          hostPath:
+            path: /var/log/pods
         # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
         - name: data
           hostPath:
diff --git a/docs/eck-attributes.asciidoc b/docs/eck-attributes.asciidoc
index 5aafa8c63..b16c58c8f 100644
--- a/docs/eck-attributes.asciidoc
+++ b/docs/eck-attributes.asciidoc
@@ -1,2 +1,2 @@
-:eck_version: 1.0.0
+:eck_version: 1.0.1
 :eck_crd_version: v1
diff --git a/docs/release-notes.asciidoc b/docs/release-notes.asciidoc
index f559e76d1..b7ffc0137 100644
--- a/docs/release-notes.asciidoc
+++ b/docs/release-notes.asciidoc
@@ -6,10 +6,12 @@
 
 This section summarizes the changes in each release.
 
+* <<release-notes-1.0.1>>
 * <<release-notes-1.0.0>>
 * <<release-notes-1.0.0-beta1>>
 
 --
 
+include::release-notes/1.0.1.asciidoc[]
 include::release-notes/1.0.0.asciidoc[]
 include::release-notes/1.0.0-beta1.asciidoc[]
diff --git a/docs/release-notes/1.0.1.asciidoc b/docs/release-notes/1.0.1.asciidoc
new file mode 100644
index 000000000..2c8527e7c
--- /dev/null
+++ b/docs/release-notes/1.0.1.asciidoc
@@ -0,0 +1,18 @@
+:issue: https://github.com/elastic/cloud-on-k8s/issues/
+:pull: https://github.com/elastic/cloud-on-k8s/pull/
+
+[[release-notes-1.0.1]]
+== {n} version 1.0.1
+
+[[enhancement-1.0.1]]
+[float]
+=== Enhancements
+
+* Increase memory of prepare-fs init container to 50MB {pull}2492[#2492] (issue: {issue}2306[#2306])
+* Add log for cert timeout {pull}2477[#2477]
+
+[[bug-1.0.1]]
+[float]
+=== Bug fixes
+
+* Fix License model in Elasticsearch client for 7.6 {pull}2441[#2441]
\ No newline at end of file
diff --git a/docs/release-notes/highlights-1.0.1.asciidoc b/docs/release-notes/highlights-1.0.1.asciidoc
new file mode 100644
index 000000000..7475b542e
--- /dev/null
+++ b/docs/release-notes/highlights-1.0.1.asciidoc
@@ -0,0 +1,6 @@
+[[release-highlights-1.0.1]]
+== 1.0.1 release highlights
+
+This release addresses an issue when using an Enterprise licence with the version 7.6 of the Elastic stack.
+
+For a more complete list of the bug fixes and enhancements included in this release, see <<release-notes-1.0.1>>
\ No newline at end of file
diff --git a/docs/release-notes/highlights.asciidoc b/docs/release-notes/highlights.asciidoc
index c421c830c..184649aef 100644
--- a/docs/release-notes/highlights.asciidoc
+++ b/docs/release-notes/highlights.asciidoc
@@ -6,10 +6,12 @@
 This section summarizes the most important changes in each release. For the
 full list, see <<eck-release-notes>>.
 
+* <<release-highlights-1.0.1>>
 * <<release-highlights-1.0.0>>
 * <<release-highlights-1.0.0-beta1>>
 
 --
 
+include::highlights-1.0.1.asciidoc[]
 include::highlights-1.0.0.asciidoc[]
 include::highlights-1.0.0-beta1.asciidoc[]
diff --git a/hack/check-requisites.sh b/hack/check-requisites.sh
index d9d8f40a1..90b37e719 100755
--- a/hack/check-requisites.sh
+++ b/hack/check-requisites.sh
@@ -8,30 +8,36 @@
 
 set -eu
 
+MIN_GO_VERSION=13
+MIN_KUBECTL_VERSION=14
+
+green="\e[32m"
+red="\e[31m"
+reset="\e[39m"
 all_found=true
 
-function check {
+check() {
     local exec_name="$@"
     printf "Checking for $exec_name... "
     if ! command -v $exec_name >/dev/null 2>&1; then
-        printf "missing!"
+        printf "${red}not found${reset}"
         all_found=false
     else
-        printf "found."
+        printf "${green}found${reset}"
     fi
     printf "\n"
 }
 
-function check_oneof {
+check_oneof() {
     local found_one=false
 
     for exec_name in $@
     do
         printf "Checking for (optional) $exec_name... "
         if ! command -v $exec_name >/dev/null 2>&1; then
-            printf "not found."
+            printf "${red}not found${reset}"
         else
-            printf "found."
+            printf "${green}found${reset}"
             found_one=true
         fi
         printf "\n"
@@ -43,16 +49,46 @@ function check_oneof {
     fi
 }
 
+check_go_version() {
+    local major=$(go version | sed -E "s|.* go([1-9]).[0-9]*[0-9.]* .*|\1|")
+    local minor=$(go version | sed -E "s|.* go[1-9].([0-9]*)[0-9.]* .*|\1|")
+
+    printf "Checking for go >= 1.$MIN_GO_VERSION... "
+    if [[ "$major" -gt 1 ]] || [[ "$minor" -ge $MIN_GO_VERSION ]]; then
+        printf "${green}ok${reset} ($major.$minor)"
+    else
+        printf "${red}ko${reset} ($major.$minor)"
+        all_found=false
+    fi
+    printf "\n"
+}
+
+check_kubectl_version() {
+    local major=$(kubectl --client=true version | grep -Eo 'Major:"[0-9]*' | grep -Eo '[0-9]+')
+    local minor=$(kubectl --client=true version | grep -Eo 'Minor:"[0-9]*' | grep -Eo '[0-9]+')
+
+    printf "Checking for kubectl >= 1.$MIN_KUBECTL_VERSION... "
+    if [[ "$major" -gt 1 ]] || [[ "$minor" -ge $MIN_KUBECTL_VERSION ]]; then
+        printf "${green}ok${reset} ($major.$minor)"
+    else
+        printf "${red}ko${reset} ($major.$minor)"
+        all_found=false
+    fi
+    printf "\n"
+}
+
 check go
 check golangci-lint
 check kubectl
 check kubebuilder
 check_oneof gcloud minikube kind
+check_go_version
+check_kubectl_version
 
 echo
 if [[ "$all_found" != "true" ]]; then
-    echo "Some tools are missing."
+    printf "${red}Error${reset}: some requirements not satified.\n" >&2
     exit 1
 else
-    echo "All tools are present."
+    printf "${green}OK${reset}: all requirements met.\n"
 fi
diff --git a/pkg/apis/common/v1/validations.go b/pkg/apis/common/v1/validations.go
index 3f741525a..4e26d417e 100644
--- a/pkg/apis/common/v1/validations.go
+++ b/pkg/apis/common/v1/validations.go
@@ -11,16 +11,18 @@ import (
 
 	v1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/apimachinery/pkg/util/validation/field"
 )
 
 // NoUnknownFields checks whether the last applied config annotation contains json with unknown fields.
-func NoUnknownFields(dest interface{}, meta metav1.ObjectMeta) field.ErrorList {
+func NoUnknownFields(dest runtime.Object, meta metav1.ObjectMeta) field.ErrorList {
 	var errs field.ErrorList
 	if cfg, ok := meta.Annotations[v1.LastAppliedConfigAnnotation]; ok {
 		d := json.NewDecoder(strings.NewReader(cfg))
 		d.DisallowUnknownFields()
-		if err := d.Decode(&dest); err != nil {
+		// decode in a copy of the resource to be validated to avoid mutation if the object in the annotation is different
+		if err := d.Decode(dest.DeepCopyObject()); err != nil {
 			errString := err.Error()
 			unknownPrefix := "json: unknown field "
 			if strings.HasPrefix(errString, unknownPrefix) {
diff --git a/pkg/controller/apmserver/pod.go b/pkg/controller/apmserver/pod.go
index b6e8a8f60..ea51b0b13 100644
--- a/pkg/controller/apmserver/pod.go
+++ b/pkg/controller/apmserver/pod.go
@@ -5,6 +5,7 @@
 package apmserver
 
 import (
+	"fmt"
 	"path/filepath"
 	"strings"
 
@@ -16,7 +17,6 @@ import (
 	"github.com/elastic/cloud-on-k8s/pkg/utils/stringsutil"
 	corev1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/api/resource"
-	"k8s.io/apimachinery/pkg/util/intstr"
 )
 
 const (
@@ -43,6 +43,7 @@ var (
 	}
 )
 
+// readinessProbe is the readiness probe for the APM Server container
 func readinessProbe(tls bool) corev1.Probe {
 	scheme := corev1.URISchemeHTTP
 	if tls {
@@ -55,10 +56,10 @@ func readinessProbe(tls bool) corev1.Probe {
 		SuccessThreshold:    1,
 		TimeoutSeconds:      5,
 		Handler: corev1.Handler{
-			HTTPGet: &corev1.HTTPGetAction{
-				Port:   intstr.FromInt(HTTPPort),
-				Path:   "/",
-				Scheme: scheme,
+			Exec: &corev1.ExecAction{
+				Command: []string{"bash", "-c",
+					fmt.Sprintf(`curl -o /dev/null -w "%%{http_code}" %s://127.0.0.1:%d/ -k -s`, scheme, HTTPPort),
+				},
 			},
 		},
 	}
diff --git a/pkg/controller/common/user/external.go b/pkg/controller/common/user/external.go
index 9eea56373..e6b4156a9 100644
--- a/pkg/controller/common/user/external.go
+++ b/pkg/controller/common/user/external.go
@@ -26,9 +26,9 @@ const (
 // ExternalUser represents a user that is not created or managed by Elasticsearch.
 // For example in the case of Kibana a user with the right role is provided by the Kibana association controller.
 type ExternalUser struct {
-	name     string
-	password []byte
-	roles    []string
+	name  string
+	hash  []byte
+	roles []string
 }
 
 // NewExternalUserFromSecret reads an external user from a secret.
@@ -44,8 +44,8 @@ func NewExternalUserFromSecret(secret v1.Secret) (ExternalUser, error) {
 		return user, pkgerrors.Errorf(fieldNotFound, UserName, secret.Namespace, secret.Name)
 	}
 
-	if password, ok := secret.Data[PasswordHash]; ok && len(password) > 0 {
-		user.password = password
+	if hash, ok := secret.Data[PasswordHash]; ok && len(hash) > 0 {
+		user.hash = hash
 	} else {
 		return user, pkgerrors.Errorf(fieldNotFound, PasswordHash, secret.Namespace, secret.Name)
 	}
@@ -62,14 +62,14 @@ func (u ExternalUser) Id() string {
 	return u.name
 }
 
-// PasswordHash is the password hash and returns it or error.
+// PasswordHash returns the password hash.
 func (u ExternalUser) PasswordHash() ([]byte, error) {
-	return u.password, nil
+	return u.hash, nil
 }
 
-// PasswordMatches compares the given hash with the password of this user.
+// PasswordMatches compares the user password hash with the given one.
 func (u *ExternalUser) PasswordMatches(hash []byte) bool {
-	return bytes.Equal(u.password, hash)
+	return bytes.Equal(u.hash, hash)
 }
 
 // Roles are any Elasticsearch roles associated with this user
diff --git a/pkg/controller/kibana/pod/pod.go b/pkg/controller/kibana/pod/pod.go
index 33cce41f6..6c4c92f00 100644
--- a/pkg/controller/kibana/pod/pod.go
+++ b/pkg/controller/kibana/pod/pod.go
@@ -5,6 +5,8 @@
 package pod
 
 import (
+	"fmt"
+
 	"github.com/elastic/cloud-on-k8s/pkg/controller/common/annotation"
 	"k8s.io/apimachinery/pkg/api/resource"
 
@@ -17,7 +19,6 @@ import (
 	"github.com/elastic/cloud-on-k8s/pkg/utils/stringsutil"
 
 	corev1 "k8s.io/api/core/v1"
-	"k8s.io/apimachinery/pkg/util/intstr"
 )
 
 const (
@@ -56,10 +57,10 @@ func readinessProbe(useTLS bool) corev1.Probe {
 		SuccessThreshold:    1,
 		TimeoutSeconds:      5,
 		Handler: corev1.Handler{
-			HTTPGet: &corev1.HTTPGetAction{
-				Port:   intstr.FromInt(HTTPPort),
-				Path:   "/login",
-				Scheme: scheme,
+			Exec: &corev1.ExecAction{
+				Command: []string{"bash", "-c",
+					fmt.Sprintf(`curl -o /dev/null -w "%%{http_code}" %s://127.0.0.1:%d/login -k -s`, scheme, HTTPPort),
+				},
 			},
 		},
 	}



Test patch:
diff --git a/pkg/controller/apmserver/apmserver_controller_test.go b/pkg/controller/apmserver/apmserver_controller_test.go
index 6d29a6aeb..a1f138af0 100644
--- a/pkg/controller/apmserver/apmserver_controller_test.go
+++ b/pkg/controller/apmserver/apmserver_controller_test.go
@@ -24,7 +24,6 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/runtime"
-	"k8s.io/apimachinery/pkg/util/intstr"
 	"k8s.io/client-go/kubernetes/scheme"
 	"k8s.io/client-go/tools/record"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
@@ -174,10 +173,10 @@ func expectedDeploymentParams() testParams {
 							SuccessThreshold:    1,
 							TimeoutSeconds:      5,
 							Handler: corev1.Handler{
-								HTTPGet: &corev1.HTTPGetAction{
-									Port:   intstr.FromInt(8200),
-									Path:   "/",
-									Scheme: corev1.URISchemeHTTPS,
+								Exec: &corev1.ExecAction{
+									Command: []string{"bash", "-c",
+										`curl -o /dev/null -w "%{http_code}" HTTPS://127.0.0.1:8200/ -k -s`,
+									},
 								},
 							},
 						},
diff --git a/pkg/controller/common/user/external_test.go b/pkg/controller/common/user/external_test.go
index 8361e5e61..2643bd19a 100644
--- a/pkg/controller/common/user/external_test.go
+++ b/pkg/controller/common/user/external_test.go
@@ -39,9 +39,9 @@ func TestNewExternalUserFromSecret(t *testing.T) {
 				},
 			},
 			want: ExternalUser{
-				name:     "ns2-kibana-sample-kibana-user",
-				password: []byte("$2a$10$D6q/zdYfGJsJxipsZ4Jioul8tWIcL.o.Mhx/as1nlNdOX6EgqRRRS"),
-				roles:    []string{"kibana_system"},
+				name:  "ns2-kibana-sample-kibana-user",
+				hash:  []byte("$2a$10$D6q/zdYfGJsJxipsZ4Jioul8tWIcL.o.Mhx/as1nlNdOX6EgqRRRS"),
+				roles: []string{"kibana_system"},
 			},
 		},
 		{
@@ -61,9 +61,9 @@ func TestNewExternalUserFromSecret(t *testing.T) {
 				},
 			},
 			want: ExternalUser{
-				name:     "ns2-kibana-sample-kibana-user",
-				password: []byte("$2a$10$D6q/zdYfGJsJxipsZ4Jioul8tWIcL.o.Mhx/as1nlNdOX6EgqRRRS"),
-				roles:    []string{"kibana_system1", "kibana_system2", "kibana_system3"},
+				name:  "ns2-kibana-sample-kibana-user",
+				hash:  []byte("$2a$10$D6q/zdYfGJsJxipsZ4Jioul8tWIcL.o.Mhx/as1nlNdOX6EgqRRRS"),
+				roles: []string{"kibana_system1", "kibana_system2", "kibana_system3"},
 			},
 		},
 		{
diff --git a/pkg/controller/kibana/driver_test.go b/pkg/controller/kibana/driver_test.go
index 30a86e3c9..e6a3e893b 100644
--- a/pkg/controller/kibana/driver_test.go
+++ b/pkg/controller/kibana/driver_test.go
@@ -17,7 +17,6 @@ import (
 	"k8s.io/apimachinery/pkg/api/resource"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/runtime"
-	"k8s.io/apimachinery/pkg/util/intstr"
 	"k8s.io/client-go/kubernetes/scheme"
 	"k8s.io/client-go/tools/record"
 	"sigs.k8s.io/controller-runtime/pkg/client"
@@ -252,7 +251,7 @@ func TestDriverDeploymentParams(t *testing.T) {
 				params := expectedDeploymentParams()
 				params.PodTemplateSpec.Spec.Volumes = params.PodTemplateSpec.Spec.Volumes[:3]
 				params.PodTemplateSpec.Spec.Containers[0].VolumeMounts = params.PodTemplateSpec.Spec.Containers[0].VolumeMounts[:3]
-				params.PodTemplateSpec.Spec.Containers[0].ReadinessProbe.Handler.HTTPGet.Scheme = corev1.URISchemeHTTP
+				params.PodTemplateSpec.Spec.Containers[0].ReadinessProbe.Handler.Exec.Command[2] = `curl -o /dev/null -w "%{http_code}" HTTP://127.0.0.1:5601/login -k -s`
 				params.PodTemplateSpec.Spec.Containers[0].Ports[0].Name = "http"
 				return params
 			}(),
@@ -523,10 +522,10 @@ func expectedDeploymentParams() deployment.Params {
 						SuccessThreshold:    1,
 						TimeoutSeconds:      5,
 						Handler: corev1.Handler{
-							HTTPGet: &corev1.HTTPGetAction{
-								Port:   intstr.FromInt(5601),
-								Path:   "/login",
-								Scheme: corev1.URISchemeHTTPS,
+							Exec: &corev1.ExecAction{
+								Command: []string{"bash", "-c",
+									`curl -o /dev/null -w "%{http_code}" HTTPS://127.0.0.1:5601/login -k -s`,
+								},
 							},
 						},
 					},