Repository: kumahq/kuma
Pull request #1228 (2020-11-27 14:55:40)

fix(kuma-cp) generate stable EDS config
### Summary

We did not sort the EDS config which resulted in unnecessary XDS traffic.

### Documentation

- [X] No docs



Patch:
diff --git a/UPGRADE.md b/UPGRADE.md
index 80f5193f3..a565b7d73 100644
--- a/UPGRADE.md
+++ b/UPGRADE.md
@@ -6,14 +6,79 @@ with `x.y.z` being the version you are planning to upgrade to.
 If such a section does not exist, the upgrade you want to perform
 does not have any particular instructions.
 
-## Upgrade to `1.0.0-rc1`
+## Upgrade to `1.0.0`
 
-This release introduces a new dataplane lifecycle, which is mostly relevant to Universal deployments.
+This release introduces a number of breaking changes. If Kuma is being deployed in production we strongly suggest to backup the current configuration, tear down the whole cluster and zones, and install in a clean setup. However, we enumerate the details of these changes below.
 
-### Suggested Upgrade Path on Universal
+### Suggested Upgrade Path on Kubernetes
+ * Drop k8s 1.13 support
+
+    Take this into account if you run Kuma on an old Kubernetes version.
+
+ * `kumactl` merged `install ingress` into `install control-plane`
+
+    This change impacts any deployment pipelines that are based on `kumactl` and are used for multi-zone deployments.
+
+ * Change policies on K8S to scope global
+
+    All the CRDs are now in the global scope, therefore all policies need to be backed up. The relevant CRDs need to be deleted, which will clear all the policies. After the upgrade, you can apply the policies again. We do recommend to keep all the Kuma Control Planes down while doing these operations.
+
+ * Autoconfigure single cert for all services
+
+    Deployment flags for providing TLS certificates in Helm and `kumactl` have changed, refer to the relevant [documentation](https://github.com/kumahq/kuma/blob/release-1.0/deployments/charts/kuma/README.md#values) to verify the new naming.
+
+ * Create default resources for Mesh
 
-Refer to the official documentation for details on how to use the new Dataplane creating lifecycle, including the new token generation and DP resource templating.
+    The following default resources will be created upon the first start of Kuma Control Plane
+        - default signing key
+        - default [Allow All traffic permission](https://kuma.io/docs/1.0.0/policies/traffic-permissions/#traffic-permissions) policy `allow-all-<mesh name>`
+        - Default [Allow All traffic route](https://kuma.io/docs/1.0.0/policies/traffic-route/#default-trafficroute) policy `allow-all-<mesh name>`
+    
+    Please verify if this conflicts with your deployment and expected policies.
 
+ * New Multizone deployment flow
+
+    Deploying Multizone clusters is now simplified, please refer to the deployment [documentation](https://kuma.io/docs/1.0.0/documentation/deployments/#multi-zone-mode) of the updated procedure.
+   
+ * Improved control plane communication security
+   
+    Kuma Control Plane exposed ports are reduced, please revise the [documentation](https://kuma.io/docs/1.0.0/documentation/networking/#kuma-cp-ports) for detailed list.
+    Consider reinstalling the metrics due to the port changes in Kuma Prometheus SD.
+ 
+ * Traffic route format
+ 
+    The format of the [TrafficRoute](https://kuma.io/docs/1.0.0/policies/traffic-route) has changed. Please check the documentation and adapt your resources. 
+
+### Suggested Upgrade Path on Universal
+ * Get rid of advertised hostname
+    `KUMA_GENERAL_ADVERTISED_HOSTNAME` was removed and not needed now.
+ 
+ * Autoconfigure single cert for all services
+    Deployment flags for providing TLS certificates in Helm and `kumactl` have changed, refer to the [documentation](https://github.com/kumahq/kuma/blob/release-1.0/pkg/config/app/kuma-cp/kuma-cp.defaults.yaml) to verify the new naming.
+
+ * Create default resources for Mesh
+    
+    The following default resources will be created upon the first start of Kuma Control Plane
+        - default signing key
+        - default [Allow All traffic permission](https://kuma.io/docs/1.0.0/policies/traffic-permissions/#traffic-permissions) policy `allow-all-<mesh name>`
+        - Default [Allow All traffic route](https://kuma.io/docs/1.0.0/policies/traffic-route/#default-trafficroute) policy `allow-all-<mesh name>`
+    
+    Please verify if this conflicts with your deployment and expected policies.
+
+* New Multizone deployment flow
+
+    Deploying Multizone clusters is now simplified, please refer to the deployment [documentation](https://kuma.io/docs/1.0.0/documentation/deployments/#multi-zone-mode) of the updated procedure.
+   
+ * Improved control plane communication security
+   
+    `kuma-dp` invocation has changed and now [allows](https://kuma.io/docs/1.0.1/documentation/dps-and-data-model/#dataplane-entity) for a more flexible usage leveraging automated, template based Dataplane resource creation, customizable data-plane token boundaries and additional CA ceritficate validation for the Kuma Control plane boostrap server.
+    Kuma Control Plane exposed ports are reduced, please revise the [documentation](https://kuma.io/docs/1.0.0/documentation/networking/#kuma-cp-ports) for detailed list.
+ 
+  * Traffic route format
+  
+     The format of the [TrafficRoute](https://kuma.io/docs/1.0.0/policies/traffic-route) has changed. Please check the documentation and adapt your resources. 
+
+ 
 ## Upgrade to `0.7.0`
 Support for `kuma.io/sidecar-injection` annotation. On Kubernetes change the namespace resources that host Kuma mesh services with the aforementioned annotation and delete the label. 
 
diff --git a/app/kuma-dp/pkg/config/validate.go b/app/kuma-dp/pkg/config/validate.go
index 5b7e16a09..57513d95c 100644
--- a/app/kuma-dp/pkg/config/validate.go
+++ b/app/kuma-dp/pkg/config/validate.go
@@ -2,6 +2,7 @@ package config
 
 import (
 	"io/ioutil"
+	"unicode"
 
 	"github.com/dgrijalva/jwt-go"
 	"github.com/pkg/errors"
@@ -26,10 +27,32 @@ func ValidateTokenPath(path string) error {
 		return errors.Wrapf(err, "could not read the token in the file %s", path)
 	}
 
-	_, _, err = new(jwt.Parser).ParseUnverified(string(rawToken), &jwt.MapClaims{})
+	token, parts, err := new(jwt.Parser).ParseUnverified(string(rawToken), &jwt.MapClaims{})
 	if err != nil {
-		return errors.Wrapf(err, "token in the file %s is not valid JWT token. Double check blank characters in the file", path)
+		return errors.Wrap(err, "not valid JWT token. Can't parse it.")
+	}
+
+	if token.Method.Alg() == "" {
+		return errors.New("not valid JWT token. No Alg.")
+	}
+
+	if token.Header == nil {
+		return errors.New("not valid JWT token. No Header.")
+	}
+	for _, part := range parts {
+		if !isASCII(part) {
+			return errors.New("The file cannot have blank characters like empty lines. Example how to get rid of non-printable characters: sed -i '' '/^$/d' token.file")
+		}
 	}
 
 	return nil
 }
+
+func isASCII(s string) bool {
+	for i := 0; i < len(s); i++ {
+		if !unicode.IsPrint(rune(s[i])) {
+			return false
+		}
+	}
+	return true
+}
diff --git a/pkg/core/managers/apis/mesh/mesh_manager.go b/pkg/core/managers/apis/mesh/mesh_manager.go
index 886232c3f..a2120f710 100644
--- a/pkg/core/managers/apis/mesh/mesh_manager.go
+++ b/pkg/core/managers/apis/mesh/mesh_manager.go
@@ -103,7 +103,7 @@ func (m *meshManager) Delete(ctx context.Context, resource core_model.Resource,
 	}
 	opts := core_store.NewDeleteOptions(fs...)
 	// delete all secrets
-	if err := m.otherManagers.DeleteAll(ctx, &system.SecretResourceList{}, core_store.DeleteAllByMesh(opts.Mesh)); err != nil {
+	if err := m.otherManagers.DeleteAll(ctx, &system.SecretResourceList{}, core_store.DeleteAllByMesh(opts.Name)); err != nil {
 		return errors.Wrap(err, "could not delete associated secrets")
 	}
 	return notFoundErr
diff --git a/pkg/core/secrets/manager/validator.go b/pkg/core/secrets/manager/validator.go
index d5b5a1ae1..97a687040 100644
--- a/pkg/core/secrets/manager/validator.go
+++ b/pkg/core/secrets/manager/validator.go
@@ -43,6 +43,9 @@ func (s *secretValidator) ValidateDelete(ctx context.Context, name string, mesh
 	meshRes := &mesh_core.MeshResource{}
 	err := s.store.Get(ctx, meshRes, core_store.GetByKey(mesh, model.NoMesh))
 	if err != nil {
+		if core_store.IsResourceNotFound(err) {
+			return nil // when Mesh no longer exist we should be able to safely delete a secret because it's not referenced anywhere
+		}
 		return err
 	}
 
diff --git a/pkg/plugins/bootstrap/k8s/plugin.go b/pkg/plugins/bootstrap/k8s/plugin.go
index 3cd5bb75c..2a15b43e3 100644
--- a/pkg/plugins/bootstrap/k8s/plugin.go
+++ b/pkg/plugins/bootstrap/k8s/plugin.go
@@ -3,6 +3,7 @@ package k8s
 import (
 	kube_runtime "k8s.io/apimachinery/pkg/runtime"
 	kube_ctrl "sigs.k8s.io/controller-runtime"
+	kube_client "sigs.k8s.io/controller-runtime/pkg/client"
 	kube_manager "sigs.k8s.io/controller-runtime/pkg/manager"
 
 	kuma_kube_cache "github.com/kumahq/kuma/pkg/plugins/bootstrap/k8s/cache"
@@ -25,8 +26,9 @@ func init() {
 
 func (p *plugin) BeforeBootstrap(b *core_runtime.Builder, _ core_plugins.PluginConfig) error {
 	scheme := kube_runtime.NewScheme()
+	config := kube_ctrl.GetConfigOrDie()
 	mgr, err := kube_ctrl.NewManager(
-		kube_ctrl.GetConfigOrDie(),
+		config,
 		kube_ctrl.Options{
 			Scheme:   scheme,
 			NewCache: kuma_kube_cache.New,
@@ -42,8 +44,22 @@ func (p *plugin) BeforeBootstrap(b *core_runtime.Builder, _ core_plugins.PluginC
 	if err != nil {
 		return err
 	}
+
+	// We need non cached client for resources that we don't have (get/list/watch) RBAC for all namespaces / cluster scope. Right now the only such resource is Secret
+	// Kubernetes cache lists resources under the hood from all Namespace unless we specify the "Namespace" in Options.
+	// If we don't do this the result is the following error: E1126 10:42:52.097662       1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.9/tools/cache/reflector.go:125: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:kuma-system:kuma-control-plane" cannot list resource "secrets" in API group "" at the cluster scope
+	// We cannot specify this Namespace parameter because it affect all the resources, therefore we need separate client for Secrets.
+	nonCachedClient, err := kube_client.New(config, kube_client.Options{
+		Scheme: scheme,
+		Mapper: mgr.GetRESTMapper(),
+	})
+	if err != nil {
+		return err
+	}
+
 	b.WithComponentManager(&kubeComponentManager{mgr})
 	b.WithExtensions(k8s_extensions.NewManagerContext(b.Extensions(), mgr))
+	b.WithExtensions(k8s_extensions.NewNonCachedClientContext(b.Extensions(), nonCachedClient))
 	if expTime := b.Config().Runtime.Kubernetes.MarshalingCacheExpirationTime; expTime > 0 {
 		b.WithExtensions(k8s_extensions.NewResourceConverterContext(b.Extensions(), k8s.NewCachingConverter(expTime)))
 	} else {
diff --git a/pkg/plugins/extensions/k8s/context.go b/pkg/plugins/extensions/k8s/context.go
index 1e1f002c1..cb2335dd1 100644
--- a/pkg/plugins/extensions/k8s/context.go
+++ b/pkg/plugins/extensions/k8s/context.go
@@ -4,6 +4,7 @@ import (
 	"context"
 
 	kube_ctrl "sigs.k8s.io/controller-runtime"
+	kube_client "sigs.k8s.io/controller-runtime/pkg/client"
 
 	k8s_common "github.com/kumahq/kuma/pkg/plugins/common/k8s"
 )
@@ -33,3 +34,14 @@ func FromResourceConverterContext(ctx context.Context) (converter k8s_common.Con
 	converter, ok = ctx.Value(converterKey{}).(k8s_common.Converter)
 	return
 }
+
+type nonCachedClient struct{}
+
+func NewNonCachedClientContext(ctx context.Context, client kube_client.Client) context.Context {
+	return context.WithValue(ctx, nonCachedClient{}, client)
+}
+
+func FromNonCachedClientContext(ctx context.Context) (client kube_client.Client, ok bool) {
+	client, ok = ctx.Value(nonCachedClient{}).(kube_client.Client)
+	return
+}
diff --git a/pkg/plugins/runtime/k8s/plugin.go b/pkg/plugins/runtime/k8s/plugin.go
index 8c11d7526..3961d87da 100644
--- a/pkg/plugins/runtime/k8s/plugin.go
+++ b/pkg/plugins/runtime/k8s/plugin.go
@@ -222,8 +222,12 @@ func addValidators(mgr kube_ctrl.Manager, rt core_runtime.Runtime, converter k8s
 	mgr.GetWebhookServer().Register("/validate-v1-service", &kube_webhook.Admission{Handler: &k8s_webhooks.ServiceValidator{}})
 	log.Info("Registering a validation webhook for v1/Service", "path", "/validate-v1-service")
 
+	client, ok := k8s_extensions.FromNonCachedClientContext(rt.Extensions())
+	if !ok {
+		return errors.Errorf("non cached client hasn't been configured")
+	}
 	secretValidator := &k8s_webhooks.SecretValidator{
-		Client:    mgr.GetClient(),
+		Client:    client,
 		Validator: manager.NewSecretValidator(rt.CaManagers(), rt.ResourceStore()),
 	}
 	mgr.GetWebhookServer().Register("/validate-v1-secret", &kube_webhook.Admission{Handler: secretValidator})
diff --git a/pkg/plugins/runtime/k8s/webhooks/injector/probes.go b/pkg/plugins/runtime/k8s/webhooks/injector/probes.go
index b5eb56a40..ead7e5ca4 100644
--- a/pkg/plugins/runtime/k8s/webhooks/injector/probes.go
+++ b/pkg/plugins/runtime/k8s/webhooks/injector/probes.go
@@ -31,12 +31,14 @@ func (i *KumaInjector) overrideHTTPProbes(pod *kube_core.Pod) error {
 	for _, c := range pod.Spec.Containers {
 		if c.LivenessProbe != nil && c.LivenessProbe.HTTPGet != nil {
 			log.V(1).Info("overriding liveness probe", "container", c.Name)
+			resolveNamedPort(c, c.LivenessProbe)
 			if err := overrideHTTPProbe(c.LivenessProbe, port); err != nil {
 				return err
 			}
 		}
 		if c.ReadinessProbe != nil && c.ReadinessProbe.HTTPGet != nil {
 			log.V(1).Info("overriding readiness probe", "container", c.Name)
+			resolveNamedPort(c, c.ReadinessProbe)
 			if err := overrideHTTPProbe(c.ReadinessProbe, port); err != nil {
 				return err
 			}
@@ -45,6 +47,18 @@ func (i *KumaInjector) overrideHTTPProbes(pod *kube_core.Pod) error {
 	return nil
 }
 
+func resolveNamedPort(container kube_core.Container, probe *kube_core.Probe) {
+	port := probe.HTTPGet.Port
+	if port.IntValue() != 0 {
+		return
+	}
+	for _, containerPort := range container.Ports {
+		if containerPort.Name != "" && containerPort.Name == port.String() {
+			probe.HTTPGet.Port = intstr.FromInt(int(containerPort.ContainerPort))
+		}
+	}
+}
+
 func overrideHTTPProbe(probe *kube_core.Probe, virtualPort uint32) error {
 	virtual, err := probes.KumaProbe(*probe).ToVirtual(virtualPort)
 	if err != nil {
diff --git a/pkg/plugins/secrets/k8s/plugin.go b/pkg/plugins/secrets/k8s/plugin.go
index 444efed31..14ffdcb1c 100644
--- a/pkg/plugins/secrets/k8s/plugin.go
+++ b/pkg/plugins/secrets/k8s/plugin.go
@@ -26,5 +26,9 @@ func (p *plugin) NewSecretStore(pc core_plugins.PluginContext, _ core_plugins.Pl
 	if err := kube_core.AddToScheme(mgr.GetScheme()); err != nil {
 		return nil, errors.Wrapf(err, "could not add %q to scheme", kube_core.SchemeGroupVersion)
 	}
-	return NewStore(mgr.GetAPIReader(), mgr.GetClient(), pc.Config().Store.Kubernetes.SystemNamespace)
+	client, ok := k8s_extensions.FromNonCachedClientContext(pc.Extensions())
+	if !ok {
+		return nil, errors.Errorf("non cached client hasn't been configured")
+	}
+	return NewStore(client, client, pc.Config().Store.Kubernetes.SystemNamespace)
 }
diff --git a/pkg/xds/envoy/endpoints/endpoints.go b/pkg/xds/envoy/endpoints/endpoints.go
index d2fc380f8..5f9652c96 100644
--- a/pkg/xds/envoy/endpoints/endpoints.go
+++ b/pkg/xds/envoy/endpoints/endpoints.go
@@ -64,12 +64,29 @@ func CreateClusterLoadAssignment(clusterName string, endpoints []core_xds.Endpoi
 		})
 	}
 
+	for _, lbEndpoints := range localityLbEndpoints {
+		// sort the slice to ensure stable Envoy configuration
+		sortLbEndpoints(lbEndpoints.LbEndpoints)
+	}
+
 	return &envoy_api.ClusterLoadAssignment{
 		ClusterName: clusterName,
 		Endpoints:   localityLbEndpoints.AsSlice(),
 	}
 }
 
+func sortLbEndpoints(lbEndpoints []*envoy_endpoint.LbEndpoint) {
+	sort.Slice(lbEndpoints, func(i, j int) bool {
+		left, right := lbEndpoints[i], lbEndpoints[j]
+		leftAddr := left.GetEndpoint().GetAddress().GetSocketAddress().GetAddress()
+		rightAddr := right.GetEndpoint().GetAddress().GetSocketAddress().GetAddress()
+		if leftAddr == rightAddr {
+			return left.GetEndpoint().GetAddress().GetSocketAddress().GetPortValue() < right.GetEndpoint().GetAddress().GetSocketAddress().GetPortValue()
+		}
+		return leftAddr < rightAddr
+	})
+}
+
 type LocalityLbEndpointsMap map[string]*envoy_endpoint.LocalityLbEndpoints
 
 func (l LocalityLbEndpointsMap) Get(ep core_xds.Endpoint) *envoy_endpoint.LocalityLbEndpoints {


Test patch:
diff --git a/app/kuma-dp/pkg/config/validate_test.go b/app/kuma-dp/pkg/config/validate_test.go
index 904ab1d06..8d70d0e78 100644
--- a/app/kuma-dp/pkg/config/validate_test.go
+++ b/app/kuma-dp/pkg/config/validate_test.go
@@ -6,6 +6,7 @@ import (
 	"os"
 
 	. "github.com/onsi/ginkgo"
+	. "github.com/onsi/ginkgo/extensions/table"
 	. "github.com/onsi/gomega"
 
 	"github.com/kumahq/kuma/app/kuma-dp/pkg/config"
@@ -57,18 +58,40 @@ var _ = Describe("ValidateTokenPath", func() {
 		Expect(err).To(MatchError(fmt.Sprintf("token under file %s is empty", tokenFile.Name())))
 	})
 
-	It("should fail for invalid token", func() {
-		// setup
-		invalidTokenFile, err := ioutil.TempFile("", "")
-		Expect(err).ToNot(HaveOccurred())
-
-		_, err = invalidTokenFile.Write([]byte("invalid.token.file"))
-		Expect(err).ToNot(HaveOccurred())
+	Context("should valicate token", func() {
+		type testCase struct {
+			token    string
+			expected string
+		}
+		DescribeTable("should fail with invalid token",
+			func(given testCase) {
+				// setup
+				invalidTokenFile, err := ioutil.TempFile("", "")
+				Expect(err).ToNot(HaveOccurred())
+
+				_, err = invalidTokenFile.Write([]byte(given.token))
+				Expect(err).ToNot(HaveOccurred())
+
+				// when
+				err = config.ValidateTokenPath(invalidTokenFile.Name())
+
+				// then
+				Expect(err).To(HaveOccurred())
+				Expect(err.Error()).To(Equal(given.expected))
+			},
+			Entry("can't parse token", testCase{
+				token:    "yJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJOYW1lIjoidGVzdCIsIk1lc2giOiJkZWZhdWx0IiwiVGFncyI6e30sIlR5cGUiOiIifQ.rdQ6l_6hzT93Kbk9kO-kZYY7BaexUH8QknvbdRy_f6s",
+				expected: "not valid JWT token. Can't parse it.: invalid character 'È' looking for beginning of value",
+			}),
+			Entry("need 3 segments", testCase{
+				token:    "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJOYW1lIjoidGVzdCIsIk1lc2giOiJkZWZhdWx0IiwiVGFncyI6e30sIlR5cGUiOiIifQ",
+				expected: "not valid JWT token. Can't parse it.: token contains an invalid number of segments",
+			}),
+			Entry("new line in the end ", testCase{
+				token:    "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJOYW1lIjoidGVzdCIsIk1lc2giOiJkZWZhdWx0IiwiVGFncyI6e30sIlR5cGUiOiIifQ.rdQ6l_6hzT93Kbk9kO-kZYY7BaexUH8QknvbdRy_f6s\n",
+				expected: "The file cannot have blank characters like empty lines. Example how to get rid of non-printable characters: sed -i '' '/^$/d' token.file",
+			}),
+		)
 
-		// when
-		err = config.ValidateTokenPath(invalidTokenFile.Name())
-
-		// then
-		Expect(err).To(HaveOccurred())
 	})
 })
diff --git a/pkg/core/managers/apis/mesh/mesh_manager_test.go b/pkg/core/managers/apis/mesh/mesh_manager_test.go
index 2f11c175d..42d91d8d5 100644
--- a/pkg/core/managers/apis/mesh/mesh_manager_test.go
+++ b/pkg/core/managers/apis/mesh/mesh_manager_test.go
@@ -212,6 +212,34 @@ var _ = Describe("Mesh Manager", func() {
 		})
 	})
 
+	Describe("Delete()", func() {
+		It("should delete secrets within one mesh", func() {
+			// given two meshes
+			err := resManager.Create(context.Background(), &core_mesh.MeshResource{}, store.CreateByKey("demo-1", model.NoMesh))
+			Expect(err).ToNot(HaveOccurred())
+			err = resManager.Create(context.Background(), &core_mesh.MeshResource{}, store.CreateByKey("demo-2", model.NoMesh))
+			Expect(err).ToNot(HaveOccurred())
+
+			// when demo-1 is deleted
+			err = resManager.Delete(context.Background(), &core_mesh.MeshResource{}, store.DeleteByKey("demo-1", model.NoMesh))
+
+			// then
+			Expect(err).ToNot(HaveOccurred())
+
+			// and all secrets are deleted
+			secrets := &system.SecretResourceList{}
+			err = secretManager.List(context.Background(), secrets, store.ListByMesh("demo-1"))
+			Expect(err).ToNot(HaveOccurred())
+			Expect(secrets.Items).To(BeEmpty())
+
+			// and all secrets from other mesh are preserved
+			secrets = &system.SecretResourceList{}
+			err = secretManager.List(context.Background(), secrets, store.ListByMesh("demo-2"))
+			Expect(err).ToNot(HaveOccurred())
+			Expect(secrets.Items).To(HaveLen(1)) // default signing key
+		})
+	})
+
 	Describe("Update()", func() {
 		It("should not allow to change CA when mTLS is enabled", func() {
 			// given
diff --git a/pkg/core/secrets/manager/validator_test.go b/pkg/core/secrets/manager/validator_test.go
index 2d3cfa1a0..e91202257 100644
--- a/pkg/core/secrets/manager/validator_test.go
+++ b/pkg/core/secrets/manager/validator_test.go
@@ -109,11 +109,19 @@ var _ = Describe("Secret Validator", func() {
 		}),
 	)
 
-	It("should validate that secret is not in use", func() {
+	It("should pass validation of secrets that are not in use", func() {
 		// when
 		err := validator.ValidateDelete(context.Background(), "some-not-used-secret", "default")
 
 		// then
 		Expect(err).ToNot(HaveOccurred())
 	})
+
+	It("should pass validation of secrets in mesh that non exist", func() {
+		// when
+		err := validator.ValidateDelete(context.Background(), "some-not-used-secret", "non-existing-mesh")
+
+		// then
+		Expect(err).ToNot(HaveOccurred())
+	})
 })
diff --git a/pkg/plugins/runtime/k8s/webhooks/injector/injector_test.go b/pkg/plugins/runtime/k8s/webhooks/injector/injector_test.go
index e8cbd642a..f66296d98 100644
--- a/pkg/plugins/runtime/k8s/webhooks/injector/injector_test.go
+++ b/pkg/plugins/runtime/k8s/webhooks/injector/injector_test.go
@@ -474,5 +474,22 @@ var _ = Describe("Injector", func() {
                   kuma.io/sidecar-injection: enabled`,
 			cfgFile: "inject.vp-disabled.config.yaml",
 		}),
+		Entry("23. Adjust Pod's probes, named port", testCase{
+			num: "23",
+			mesh: `
+              apiVersion: kuma.io/v1alpha1
+              kind: Mesh
+              metadata:
+                name: default
+              spec: {}`,
+			namespace: `
+              apiVersion: v1
+              kind: Namespace
+              metadata:
+                name: default
+                annotations:
+                  kuma.io/sidecar-injection: enabled`,
+			cfgFile: "inject.config.yaml",
+		}),
 	)
 })
diff --git a/pkg/plugins/runtime/k8s/webhooks/injector/testdata/inject.23.golden.yaml b/pkg/plugins/runtime/k8s/webhooks/injector/testdata/inject.23.golden.yaml
new file mode 100644
index 000000000..ff54cc444
--- /dev/null
+++ b/pkg/plugins/runtime/k8s/webhooks/injector/testdata/inject.23.golden.yaml
@@ -0,0 +1,173 @@
+apiVersion: v1
+kind: Pod
+metadata:
+  annotations:
+    kuma.io/mesh: default
+    kuma.io/sidecar-injected: "true"
+    kuma.io/transparent-proxying: enabled
+    kuma.io/transparent-proxying-inbound-port: "15006"
+    kuma.io/transparent-proxying-outbound-port: "15001"
+    kuma.io/virtual-probes: enabled
+    kuma.io/virtual-probes-port: "9000"
+  creationTimestamp: null
+  labels:
+    run: busybox
+  name: busybox
+spec:
+  containers:
+    - image: busybox
+      ports:
+        - name: readiness-port
+          containerPort: 3001
+        - name: liveness-port
+          containerPort: 8080
+      livenessProbe:
+        httpGet:
+          path: /8080/metrics
+          port: 9000
+        initialDelaySeconds: 3
+        periodSeconds: 3
+      name: busybox
+      readinessProbe:
+        httpGet:
+          path: /3001/metrics
+          port: 9000
+        initialDelaySeconds: 3
+        periodSeconds: 3
+      resources: {}
+      volumeMounts:
+        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
+          name: default-token-w7dxf
+          readOnly: true
+    - args:
+        - run
+        - --log-level=info
+      env:
+        - name: POD_NAME
+          valueFrom:
+            fieldRef:
+              apiVersion: v1
+              fieldPath: metadata.name
+        - name: POD_NAMESPACE
+          valueFrom:
+            fieldRef:
+              apiVersion: v1
+              fieldPath: metadata.namespace
+        - name: INSTANCE_IP
+          valueFrom:
+            fieldRef:
+              apiVersion: v1
+              fieldPath: status.podIP
+        - name: KUMA_CONTROL_PLANE_URL
+          value: http://kuma-control-plane.kuma-system:5681
+        - name: KUMA_DATAPLANE_MESH
+          value: default
+        - name: KUMA_DATAPLANE_NAME
+          value: $(POD_NAME).$(POD_NAMESPACE)
+        - name: KUMA_DATAPLANE_ADMIN_PORT
+          value: "9901"
+        - name: KUMA_DATAPLANE_DRAIN_TIME
+          value: 31s
+        - name: KUMA_DATAPLANE_RUNTIME_TOKEN_PATH
+          value: /var/run/secrets/kubernetes.io/serviceaccount/token
+        - name: KUMA_CONTROL_PLANE_CA_CERT
+          value: |
+            -----BEGIN CERTIFICATE-----
+            MIIDLDCCAhSgAwIBAgIQHdPhxOfXgWuNxoFlV/EwqTANBgkqhkiG9w0BAQsFADAP
+            MQ0wCwYDVQQDEwRrdW1hMB4XDTIwMDkxNjEyMjg0NFoXDTMwMDkxNDEyMjg0NFow
+            DzENMAsGA1UEAxMEa3VtYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
+            AOZGbWhSlQSRxFNt5p/2WCKFyHZ3CuwNgyLEP7nS4ZXykxsFbYSuV3bIgF7bT/uq
+            a5Qire+C60guhFbpLcPh2Z6UfgId69GlQzHMVYcmLGjVQuyAt4FMMkTfVEl5I4Oa
+            +2it3BvihVkKhUz8y5RR5KbqJfGp4Z20Fh6fttoCFbeODmvBsYJFmUQS+ifoyMY/
+            P3R03Su7g5iIvnz7tmkydoNC8nGRDzdD5C8fJvrVI1UX6JRGyLKt45oQXt1mxK10
+            5KaN2zNV2WtHsaJp9bwrPH+JiZGeZyvuh5UwrLdHCmqK7sm9TodGztUZY0VzAc4q
+            kYViXY8gUjfNm+cQrPO1kN8CAwEAAaOBgzCBgDAOBgNVHQ8BAf8EBAMCAqQwHQYD
+            VR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMBMA8GA1UdEwEB/wQFMAMBAf8wHQYD
+            VR0OBBYEFGMDBPPiBFJ3mv2oA9CTqjemFTV2MB8GA1UdEQQYMBaCCWxvY2FsaG9z
+            dIIJbG9jYWxob3N0MA0GCSqGSIb3DQEBCwUAA4IBAQC/17QweOpGdb1MEBJ8XPG7
+            3sK/utoWLX1tjf8Su1Dga6CDT/eTWHZrWRf81KOVY07dle5SRIDK1QhfbGGtFP+T
+            vZkroousI9US2aCWlkeCZWGTnqvLmDoOujqagDoKRRuk4mQdtNNonxiL/wZtTFKi
+            +1iNjUVbLWiDXdBLxoRIVdLOzqb/MNxwElUyaTDAkopQyOWaTDkYPrGmaWjcsfPG
+            aOKow0ze+zHVFqTHbjnCqEV3hnsUyRUwsBln9+jDJXgwZM/tMlVJrZCh0Sle9Y5Z
+            MOB0fCf6sTMNRTGg5Lpl6uIYM/5INpmHVMo3n7MBSnpEDAUS2bf/uo5gIiq6XCdp
+            -----END CERTIFICATE-----
+      image: kuma/kuma-sidecar:latest
+      imagePullPolicy: IfNotPresent
+      livenessProbe:
+        exec:
+          command:
+            - wget
+            - -qO-
+            - http://127.0.0.1:9901/ready
+        failureThreshold: 212
+        initialDelaySeconds: 260
+        periodSeconds: 25
+        successThreshold: 1
+        timeoutSeconds: 23
+      name: kuma-sidecar
+      readinessProbe:
+        exec:
+          command:
+            - wget
+            - -qO-
+            - http://127.0.0.1:9901/ready
+        failureThreshold: 112
+        initialDelaySeconds: 11
+        periodSeconds: 15
+        successThreshold: 11
+        timeoutSeconds: 13
+      resources:
+        limits:
+          cpu: 1100m
+          memory: 1512Mi
+        requests:
+          cpu: 150m
+          memory: 164Mi
+      securityContext:
+        runAsGroup: 5678
+        runAsUser: 5678
+      volumeMounts:
+        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
+          name: default-token-w7dxf
+          readOnly: true
+  initContainers:
+    - args:
+        - -p
+        - "15001"
+        - -z
+        - "15006"
+        - -u
+        - "5678"
+        - -g
+        - "5678"
+        - -d
+        - ""
+        - -o
+        - ""
+        - -m
+        - REDIRECT
+        - -i
+        - '*'
+        - -b
+        - '*'
+      image: kuma/kuma-init:latest
+      imagePullPolicy: IfNotPresent
+      name: kuma-init
+      resources:
+        limits:
+          cpu: 100m
+          memory: 50M
+        requests:
+          cpu: 10m
+          memory: 10M
+      securityContext:
+        capabilities:
+          add:
+            - NET_ADMIN
+        runAsGroup: 0
+        runAsUser: 0
+  volumes:
+    - name: default-token-w7dxf
+      secret:
+        secretName: default-token-w7dxf
+status: {}
diff --git a/pkg/plugins/runtime/k8s/webhooks/injector/testdata/inject.23.input.yaml b/pkg/plugins/runtime/k8s/webhooks/injector/testdata/inject.23.input.yaml
new file mode 100644
index 000000000..c578b6855
--- /dev/null
+++ b/pkg/plugins/runtime/k8s/webhooks/injector/testdata/inject.23.input.yaml
@@ -0,0 +1,36 @@
+apiVersion: v1
+kind: Pod
+metadata:
+  name: busybox
+  labels:
+    run: busybox
+spec:
+  volumes:
+    - name: default-token-w7dxf
+      secret:
+        secretName: default-token-w7dxf
+  containers:
+    - name: busybox
+      image: busybox
+      resources: {}
+      ports:
+        - name: readiness-port
+          containerPort: 3001
+        - name: liveness-port
+          containerPort: 8080
+      readinessProbe:
+        httpGet:
+          path: /metrics
+          port: readiness-port
+        initialDelaySeconds: 3
+        periodSeconds: 3
+      livenessProbe:
+        httpGet:
+          path: /metrics
+          port: liveness-port
+        initialDelaySeconds: 3
+        periodSeconds: 3
+      volumeMounts:
+        - name: default-token-w7dxf
+          readOnly: true
+          mountPath: "/var/run/secrets/kubernetes.io/serviceaccount"
diff --git a/pkg/xds/cache/cla/testdata/cla.get.1.json b/pkg/xds/cache/cla/testdata/cla.get.1.json
index 138297c94..e10be7e25 100644
--- a/pkg/xds/cache/cla/testdata/cla.get.1.json
+++ b/pkg/xds/cache/cla/testdata/cla.get.1.json
@@ -9,9 +9,9 @@
               "address": {
                 "Address": {
                   "SocketAddress": {
-                    "address": "192.168.0.1",
+                    "address": "1.1.1.1",
                     "PortSpecifier": {
-                      "PortValue": 1010
+                      "PortValue": 1011
                     }
                   }
                 }
@@ -28,9 +28,9 @@
               "address": {
                 "Address": {
                   "SocketAddress": {
-                    "address": "1.1.1.1",
+                    "address": "192.168.0.1",
                     "PortSpecifier": {
-                      "PortValue": 1011
+                      "PortValue": 1010
                     }
                   }
                 }
diff --git a/pkg/xds/envoy/endpoints/endpoints_test.go b/pkg/xds/envoy/endpoints/endpoints_test.go
index c3ceabcba..1af754d6d 100644
--- a/pkg/xds/envoy/endpoints/endpoints_test.go
+++ b/pkg/xds/envoy/endpoints/endpoints_test.go
@@ -85,6 +85,38 @@ var _ = Describe("Endpoints", func() {
                           address: 192.168.0.2
                           portValue: 8082
                     loadBalancingWeight: 1
+`,
+			}),
+			Entry("stable output", testCase{
+				cluster: "127.0.0.1:8080",
+				endpoints: []core_xds.Endpoint{ // endpoints in different order, but result is the same as in previous test
+					{
+						Target: "192.168.0.2",
+						Port:   8082,
+						Weight: 1,
+					},
+					{
+						Target: "192.168.0.1",
+						Port:   8081,
+						Weight: 2,
+					},
+				},
+				expected: `
+                clusterName: 127.0.0.1:8080
+                endpoints:
+                - lbEndpoints:
+                  - endpoint:
+                      address:
+                        socketAddress:
+                          address: 192.168.0.1
+                          portValue: 8081
+                    loadBalancingWeight: 2
+                  - endpoint:
+                      address:
+                        socketAddress:
+                          address: 192.168.0.2
+                          portValue: 8082
+                    loadBalancingWeight: 1
 `,
 			}),
 			Entry("with tags", testCase{