Repository: openshift/openshift-ansible

Issue #5987: Openshift cluster recovering needs a very long time if first master goes down
Created: 2017-11-02T12:39:25Z
Description
Using a multi master installation with a host set as load-balancer, if the first master goes down it takes a very long time for recovering the whole cluster services (login on the management UI, listing the deployments, scaling up or down ...). Only after about 1 hour and 30 minutes the cluster is fully usable.
Version
oc v3.6.0+c4dd4cf
kubernetes v1.6.1+5115d708d7
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://...
openshift v3.6.0+c4dd4cf
kubernetes v1.6.1+5115d708d7

ansible2.3.1.0
  config file = /usr/share/ansible/openshift-ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.5 (default, Nov  6 2016, 00:28:07) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

Installation description:
We are setting the OpenShift cluster with

MULTIPLE MASTERS WITH MASTER AND ETCD ON THE SAME HOST
https://docs.openshift.org/latest/install_config/install/advanced_install.html#multiple-masters

All our hosts run on Linux CentOS 7.3 64-bit:
3 masters (schedulable = false, etcd on them): 16 GB RAM, CPU 4 x 2198 MHz, Disk 50 GB
3 nodes (one on the infra region for serving the registry and router services): 8 GB RAM, CPU 4 x 2198 MHz and Disk 50 GB
1 host as load-balancer: 4 GB RAM, CPU 2 x 2198 MHz and Disk 50 GB
1 host as NFS server: 8 GB RAM, CPU 4 x 2198 MHz and Disk 50 GB
The openshift_master_identity_providers is HTPASSWD_AUTH
We set openshift_clock_enabled=true also.
Nothing special else.
All is running well we can create few projects, deploy a service based on wildfly getting data from a postgresql service (map to a mounted volume). We don't have a lot of running pods (one project with 6 pods, one with 3 pods).
So we want to test the way the Openshift Cluster can handle a master failure, and we stop the first master host (shutdown the host).
Expected Results
The running Services should still be accessible. This part is working as expected.
The cluster should be manageable after a "reasonable delay". I would expect something like between 2 and 5 minutes. This is not working: the cluster needs up to 1 hour and a half for becoming fully manageable again.
Observed Results
I wait 2 Minutes and I still cannot login on the Management web Console (Error 500), only after something like 10 to 15 minutes I can login again.
I open a project and the overview is not working... We get a lot of errors about the different cluster services (all messages with (504) error codes)...
After about 30 - 40 minutes I could list the deployments in the overview.
Then I tried to scale one service up: "Failed to get imagestreamimages/v1/country-service@sha256:034d1191c50d39ffeccdec8614b04256a012378706730850044dd5f04f47998c (504)"
Scaling-up was only possible after one hour the first master has been stopped.
Then I went to list the storage persistent volumes with the storage menu: I had an error: "Failed to list persistentvolumeclaims/v1 (504)"
etc ...
So it seems that the different cluster services are recovering one after one and very slowly.
In our case the Openshift-cluster is fully manageable only after 1 hour and 30 minutes....
Why is that so slow? Did we miss some extra settings not mentioned in the documentation? Or are we to short in hardware resources?
Thanks a lot in advance
Emmanuel

Duplicate issue #8359: oc replace
Created: 2018-05-13T03:38:20Z
Description
Provide a brief description of your issue here. For example:

On a multi master install, if the first master goes down we can no
longer scaleup the cluster with new nodes or masters.

Version
Please put the following version information in the code block
indicated below.

Your ansible version per ansible --version

If you're operating from a git clone:

The output of git describe

If you're running from playbooks installed via RPM

The output of rpm -q openshift-ansible

Place the output between the code block below:
VERSION INFORMATION HERE PLEASE

Steps To Reproduce

[step 1]
[step 2]

Expected Results
Describe what you expected to happen.
Example command and output or error messages

Observed Results
Describe what is actually happening.
Example command and output or error messages

For long output or logs, consider using a gist
Additional Information
Provide any additional information which may help us diagnose the
issue.

Your operating system and version, ie: RHEL 7.2, Fedora 23 ($ cat /etc/redhat-release)
Your inventory file (especially any non-standard configuration parameters)
Sample code, etc

EXTRA INFORMATION GOES HERE

Closing comment (2018-05-14T09:02:42Z): Duplicate of #5987