opened
rolandkool
username_0
Hi,

We've found that Elasticsearch clusters won't start if Istio injects a sidecar into statefulsets that get created.

There seem to be two issues:
* Not all of the Kubernetes services that get created have the proper port naming as required by Istio. For example the service for the master (on port 9300) has a portname of 'transport', while 'tcp-transport' would be required for Istio (see https://istio.io/docs/setup/additional-setup/requirements/)
* With the services fixed, the data nodes will not start and are unable to reach the master. There's a related issue in the Istio project: https://github.com/istio/istio/issues/14662

Other somewhat related issue: https://github.com/kubedb/project/issues/544
Elasticsearch doesn't work in combination with Istio
issue
created
BufferUnderflower
username_1
The main reason i couldn't use KubeDB with Istio was the fact that KubeDB relies heavily on initContainers to setup connectivity for db clusters (discovery, policies), while `istio-proxy` (container with envoy) starts up right after all initContainers are finished. [This means **you can't rely on any Istio networking features in initContainers** ](https://discuss.istio.io/t/k8s-istio-sidecar-injection-with-other-init-containers/845). What i did was moving init logic to main container and delayed its execution until proxy is up and running.

If you're trying to setup mTLS - keep in mind initContainer doesn't yet have any keys, certs, ServiceEntries (for headless), etc that are normally provided by Istio sidecar.

My piece of advice - If you're going with Istio - prefer raw manifests over KubeDB , because you'll find yourself very limited in advanced Istio features (mTLS, multicluster), using this operator for db clusters/replicasets.
comment
created
rolandkool
username_0
We're actually using Istio CNI in stead of init containers, but the issue remains. Even with 1.4 where some improvements for dealing with statefulsets are included, it still won't start when the sidecar is active.
Our approach for now is to just leave it out of the mesh, but it would be nice if at least the improvements with regards to the service portnames could be addressed in a future version.
comment
created
BufferUnderflower
username_1
Problem is not Istio's init container, but kubedb's. Even though you use CNI. KubeDB does it's initializing in initContaner and assumes there is network connectivity at this stage, but there is none, because istio-proxy is a normal container that will start after all initContainers done. What i managed to do  was moving that logic to run inside main container.
comment