Repository: openshift/openshift-ansible

Issue:
KeyError: 'openshift' when ssh key verification failed or ssh timeout during run
I am using ansible 1.9.4 for installing 3.1 cluster(problem happening for 3.2 cluster too)
for internal reason we are using puppet to launch ansible. and because we are installing on gce and using preemptible vms (a preemptible vm can be reclaimed by gce and shutdown at any moment and has max duration of 24h), because of thoses special vms we are cleaning known_hosts file on the ansible admin machine 
we are facing this error, happening always on the same play 

```
PLAY [Configure master instances]
*********************************************  
GATHERING FACTS
*************************************************************** ok: [plop-master-556a8]

TASK: [Ensure certificate directory exists]
*********************************** skipping: [plop-master-556a8]

TASK: [Unarchive the tarball on the master]
*********************************** skipping: [plop-master-556a8]

TASK: [openshift_facts | Verify Ansible version is greater than or
equal to 1.9.4] *** 
skipping: [plop-master-556a8]

TASK: [openshift_facts | Detecting Operating System]
************************** ok: [plop-master-556a8]

TASK: [openshift_facts | set_fact ]
******************************************* ok: [plop-master-556a8]

TASK: [openshift_facts | set_fact ]
******************************************* ok: [plop-master-556a8]

TASK: [openshift_facts | Ensure PyYaml is installed]
************************** ok: [plop-master-556a8]

TASK: [openshift_facts | Ensure yum-utils is installed]
*********************** ok: [plop-master-556a8]

TASK: [openshift_facts | Gather Cluster facts and set is_containerized
if needed] *** 
fatal: [plop-master-556a8] => Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/runner/__init__.py",
line 589, in _executor
    exec_rc = self._executor_internal(host, new_stdin)
  File "/usr/lib/python2.7/site-packages/ansible/runner/__init__.py",
line 793, in _executor_internal
    return self._executor_internal_inner(host, self.module_name,
self.module_args, inject, port, complex_args=complex_args)
  File "/usr/lib/python2.7/site-packages/ansible/runner/__init__.py",
line 1017, in _executor_internal_inner
    complex_args = template.template(self.basedir, complex_args,
inject, fail_on_undefined=self.error_on_undefined_vars)
  File "/usr/lib/python2.7/site-packages/ansible/utils/template.py",
line 140, in template
    d[k] = template(basedir, v, templatevars, lookup_fatal, depth,
expand_lists, convert_bare, fail_on_undefined, filter_fatal)
  File "/usr/lib/python2.7/site-packages/ansible/utils/template.py",
line 140, in template
    d[k] = template(basedir, v, templatevars, lookup_fatal, depth,
expand_lists, convert_bare, fail_on_undefined, filter_fatal)
  File "/usr/lib/python2.7/site-packages/ansible/utils/template.py",
line 124, in template
    varname = template_from_string(basedir, varname, templatevars,
fail_on_undefined)
  File "/usr/lib/python2.7/site-packages/ansible/utils/template.py",
line 382, in template_from_string
    res = jinja2.utils.concat(rf)
  File "<template>", line 6, in root
  File "/usr/lib/python2.7/site-packages/jinja2/runtime.py", line 156,
in resolve
    return self.parent[key]
  File "/usr/lib/python2.7/site-packages/ansible/utils/template.py",
line 205, in __getitem__
    return template(self.basedir, var, self.vars,
fail_on_undefined=self.fail_on_undefined)
  File "/usr/lib/python2.7/site-packages/ansible/utils/template.py",
line 124, in template
    varname = template_from_string(basedir, varname, templatevars,
fail_on_undefined)
  File "/usr/lib/python2.7/site-packages/ansible/utils/template.py",
line 382, in template_from_string
    res = jinja2.utils.concat(rf)
  File "<template>", line 14, in root
  File
"/home/jawed/poc/openshift-ansible/filter_plugins/oo_filters.py", line
139, in oo_collect
    retval = [FilterModule.get_attr(d, attribute) for d in data]
  File
"/home/jawed/poc/openshift-ansible/filter_plugins/oo_filters.py", line
49, in get_attr
    ptr = ptr[attr]
KeyError: 'openshift'
```

it is reproducible by lanching ansible on byo mode with this inventory

```
[[OSEv3:children]
masters
nodes



[OSEv3:vars]
ansible_ssh_user=openshift
ansible_sudo=true
deployment_type=openshift-enterprise
openshift_additional_repos=[]

[masters]
plop-master-556a8 openshift_hostname=plop-master-556a8
[nodes]
plop-node-compute-1a65e openshift_hostname=plop-node-compute-1a65e
plop-node-compute-d7368 openshift_hostname=plop-node-compute-d7368
plop-node-infra-60bde openshift_hostname=plop-node-infra-60bde
```

and after passing the first phases of ssh all nodes, try to lean the matser entry in known_hosts 



Discussion:
Hey @jkhelil, it looks like this error may be related to a missing `openshift.common.hostname` fact.

This particular failing `oo_collect` should be from play vars for the "Configure masters" play [here](https://github.com/openshift/openshift-ansible/blob/36a875d58fd6832851fad11dccd1245758950523/playbooks/common/openshift-master/config.yml#L317-L321).

Can add a task prior to gathering cluster facts like this to test.

``` diff
diff --git a/roles/openshift_facts/tasks/main.yml b/roles/openshift_facts/tasks/main.yml
index c372367..414db38 100644
--- a/roles/openshift_facts/tasks/main.yml
+++ b/roles/openshift_facts/tasks/main.yml
@@ -23,6 +23,9 @@
   action: "{{ ansible_pkg_mgr }} name=yum-utils state=present"
   when: not l_is_atomic | bool

+- debug:
+    var: openshift.common.hostname
+
 - name: Gather Cluster facts and set is_containerized if needed
   openshift_facts:
     role: common
```

Are you calling into the playbooks in a custom way? Just curious since we don't have a "Configure master instances" play. Is there an `oo_collect` filter being used in vars for that play?

I added the debug line but seems that the hostname is well set for the master, 
here is the ansible log 
https://gist.github.com/jkhelil/aaef219c0f700dbe69a07e1c450e7771

I was running old version thats why the play did not has the same name, I updated to master branch  and get the last plays
I Still have the error, but seems  not related to hostname of the master
to reproduce, youh should have host key verification error for a node (not specialy for the master as I put in last comment) so it is related to the hostname of the unjoignable node
How can  we secure it in thi statement here https://github.com/openshift/openshift-ansible/blob/36a875d58fd6832851fad11dccd1245758950523/playbooks/common/openshift-master/config.yml#L317-L321

One collegue gets the error also when he notice an ssh timeout during the run of playbooks 

! fatal: [openshift-node-ccm6.xxxxxx] => SSH Error:! Connection timed out during banner exchange!   while connecting to 10.55.0.88:22 
Then, same exception with "KeyError: openshift is encountered"
!TASK: [openshift_facts | Gather Cluster facts and set is_containerized!if needed] ***![...]!   ptr = ptr[attr]!KeyError: 'openshift'

I add a print data and print attr in the filter get_attr method and I can confirm that it fails on this 

{'inventory_hostname': 'plop-node-compute-314e5', 'g_new_node_hosts': [], 'g_node_hosts': ['plop-node-compute-314e5', 'plop-node-compute-e9bbf', 'plop-node-infra-ff972'], 'g_master_hosts': ['plop-master-f1943'], 'inventory_hostname_short': 'plop-node-compute-314e5', 'g_etcd_hosts': [], 'g_lb_hosts': [], 'g_new_master_hosts': [], 'openshift_additional_repos': [], 'ansible_ssh_user': 'openshift', 'group_names': ['OSEv3', 'l_oo_all_hosts', 'nodes', 'oo_all_hosts', 'oo_nodes_to_config'], 'g_all_hosts': u'{{ g_master_hosts | union(g_node_hosts) | union(g_etcd_hosts) | union(g_lb_hosts) | union(g_nfs_hosts) | union(g_new_node_hosts)| union(g_new_master_hosts) | default([]) }}', 'openshift_hostname': 'plop-node-compute-314e5', 'ansible_sudo': 'true', 'g_nfs_hosts': [], 'deployment_type': 'openshift-enterprise'}

openshift.common.hostname

It is like at some point the node is selected into node_to_config but facts were not gathered for it because it was not joignable and then it fails because it can not find the openshift.common.hostname entry for the node

I think this happen because we are not able to initialise host facts here playbooks/common/openshift-cluster/initialize_facts.yml
I can find in logs 
PLAY [Initialize host facts] *************************************************\* 

GATHERING FACTS **************************************************************\* 
ok: [plop-master-f1943]
ok: [plop-node-compute-e9bbf]
fatal: [plop-node-compute-314e5] => SSH Error: Host key verification failed.
    while connecting to 192.168.55.239:22
It is sometimes useful to re-run the command using -vvvv, which prints SSH debug output to help diagnose the issue.
fatal: [plop-node-infra-ff972] => SSH Error: Host key verification failed.
    while connecting to 192.168.55.83:22
It is sometimes useful to re-run the command using -vvvv, which prints SSH debug output to help diagnose the issue.

by the way any_fatal_errors is not working for ansible 1.9, there is a bug fixed here https://github.com/ansible/ansible/issues/14100

@jkhelil In that case we should probably add `any_errors_fatal` to the initialize facts play.

@detiber Maybe we should add `any_errors_fatal` to all or most of our plays... I doubt we'd ever want to continue with a subset of our hosts. We could also add extra checking around tasks that require facts from other hosts so that we fail before they are accessed.

there is already any_errors_fatal in initialize facts play. I think you added it in june, it is recent. 
My understanding is that openshift-ansible is designed to abord if we fail to initialise facts for some of nodes. 
that means in our case, we should fail before, but we did not because of the ansible bug not handling any_fatal_errors in ansible 1.9.
that means also for us that if we want to resolve our issue with preemptible, we should use the scaleup playbook to add new node because config.yml is not designed to add nodes, @abutcher @detiber  can you confirm plz?

@abutcher @detiber  what do you think about finxing get attribute method in filters plugin to return empty thing if the key is not present, and not getting the runtime error 

Nodes can be added with config.yml with the understanding that ansible will be ran against all hosts. Some users modify configuration after config.yml has been ran so the scaleup playbook is a way to add nodes without interfering with any other configs.

I have a commit in an upcoming change that returns an empty list if the key wasn't present. https://github.com/openshift/openshift-ansible/pull/2049 

@abutcher I'm a bit hesitant to rely on any_errors_fatal, since environments with many nodes can/will have some of those nodes unavailable at any given time. 

@abutcher I agree with @detiber , if one node of 255 is unavailable, we expect to not abort plays

Agreed, in that case this should be resolved by the changes in https://github.com/openshift/openshift-ansible/pull/2049/commits/d3ba0624faf1119eb01604d574363dfa256a2949. There are some other places in the installer where we try to pull facts from all hosts such as https://github.com/openshift/openshift-ansible/blob/master/roles/etcd/templates/etcd.conf.j2#L2-L9 that will need to be addressed.

@abutcher so, I think we may be able to draw a distinction between services like etcd and the master services and the nodes here. It would be a nice-to-have to be able to continue to configure the etcd and master services when one or more hosts are down, but i think it is a necessity to do so with the nodes.


Resolving patch:
diff --git a/filter_plugins/oo_filters.py b/filter_plugins/oo_filters.py
--- a/filter_plugins/oo_filters.py
+++ b/filter_plugins/oo_filters.py
@@ -39,7 +39,6 @@ def get_attr(data, attribute=None):
             the value.
 
             If the key isn't present, None is returned.
-
             Ex: data = {'a': {'b': {'c': 5}}}
                 attribute = "a.b.c"
                 returns 5
@@ -57,6 +56,7 @@ def get_attr(data, attribute=None):
 
         return ptr
 
+
     @staticmethod
     def oo_flatten(data):
         """ This filter plugin will flatten a list of lists
@@ -146,6 +146,7 @@ def oo_collect(data, attribute=None, filters=None):
             retval = [FilterModule.get_attr(d, attribute) for d in data]
 
         retval = [val for val in retval if val != None]
+
         return retval
 
     @staticmethod


Test patch: