Repository: openshift/openshift-ansible

Issue:
PV creation fails when trying to install openshift_metrics with external nfs storage
#### Description

When trying to install openshift 3.7 with metrics enabled with external nfs, PV creation fails with the following error:

```
TASK [openshift_persistent_volumes : Create temp directory for volume definitions] ***
fatal: [REDACTED]: FAILED! => {"failed": true, "msg": "The conditional check 'persistent_volumes | length > 0 or persistent_volume_claims | length > 0' failed. The error was: {{ hostvars[groups.oo_first_master.0] | oo_persistent_volumes(groups) }}: |failed no storage host detected\n\nThe error appears to have been in '/var/lib/jenkins/workspace/ocp-ansible-jenkins-3.7-ext_pvs/23/openshift-ansible/roles/openshift_persistent_volumes/tasks/main.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n---\n- name: Create temp directory for volume definitions\n  ^ here\n"}
	to retry, use: --limit @/var/lib/jenkins/workspace/ocp-ansible-jenkins-3.7-ext_pvs/23/openshift-ansible/playbooks/byo/config.retry
```

I've investigated this problem, and it seems that most of the metrics facts from the inventory get discarded and replaced with defaults, apart from `openshift_metrics_install_metrics` and `openshift_metrics_storage_kind`. This means the `host` is set to `None`, which causes the failure above.

#### More details

Further debugging reveals that the problem is in `openshift_facts`, around [roles/openshift_facts/library/openshift_facts.py#L2264 ](https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_facts/library/openshift_facts.py#L2264)

when this code runs for the `openshift_metrics_install_metrics=true` directive in the inventory file,
after the loop in [lines 2256 - 2263](https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_facts/library/openshift_facts.py#L2256-L2263), ``oo_env_facts`` looks like this:

``{"install": { "metrics": "true" }, "metrics": "true"}``

and `facts_to_set` (before `merge_facts()` is run) is a dict containing all the other values from the inventory.
After `merge_facts()` runs, `facts_to_set` becomes identical to `oo_env_facts` mentioned above, with all values overriden, 

So it goes from looking like this (before the merge)
````json
"metrics": {
    "storage": {
        "host": "[redacted]",
        "labels": "{'storage': 'metrics'}",
        "nfs": { "directory": "/exports/elad-37-t4" },
        "volume": { "name": "metrics" }
    }
}
````

to be identical to the line above, discarding all information. This is because `merge_facts()` sees the "new" parameter has a `metrics` key which is not a dictionary, but rather a boolean, so instead of merging contents (which isn't really possible for a boolean and a dict) it just discards the "old" dict, and replaces it with the boolean.

It looks like it only causes an error to metrics by pure chance - it's down to the order in which it evaluates the inventory directives,
and because the inventory directives are first loaded by ansible into a dict, which is not ordered, the order is pretty much arbitrary (and might differ between Python versions).

For logging, for example, the bug causes `openshift_logging_image_version` to be discarded. For other components it might cause other directives to be discarded too.
metrics is the only one which caused an actual failure because it evaluated `openshift_metrics_install_metrics` relatively late, discarding most other directives.

##### Additional Information

You'd need a snippet like this one in your inventory file to reproduce the issue:

```
openshift_metrics_install_metrics=true
openshift_metrics_image_prefix=openshift3/
openshift_metrics_storage_kind=nfs
openshift_metrics_storage_host=nfs.example.com
openshift_metrics_storage_nfs_directory=/exports/elad-37-t4
openshift_metrics_storage_volume_name=metrics
openshift_metrics_storage_labels={'storage': 'metrics'}
```



Discussion:

Resolving patch:
diff --git a/roles/openshift_facts/library/openshift_facts.py b/roles/openshift_facts/library/openshift_facts.py
--- a/roles/openshift_facts/library/openshift_facts.py
+++ b/roles/openshift_facts/library/openshift_facts.py
@@ -2253,14 +2253,27 @@ def init_local_facts(self, facts=None,
                 oo_env_facts = dict()
                 current_level = oo_env_facts
                 keys = self.split_openshift_env_fact_keys(fact, openshift_env_structures)[1:]
+
                 if len(keys) > 0 and keys[0] != self.role:
                     continue
-                for key in keys:
-                    if key == keys[-1]:
-                        current_level[key] = value
-                    elif key not in current_level:
-                        current_level[key] = dict()
-                        current_level = current_level[key]
+
+                # Build a dictionary from the split fact keys.
+                # After this loop oo_env_facts is the resultant dictionary.
+                # For example:
+                # fact = "openshift_metrics_install_metrics"
+                # value = 'true'
+                # keys = ['metrics', 'install', 'metrics']
+                # result = {'metrics': {'install': {'metrics': 'true'}}}
+                for i, _ in enumerate(keys):
+                    # This is the last key. Set the value.
+                    if i == (len(keys) - 1):
+                        current_level[keys[i]] = value
+                    # This is a key other than the last key. Set as
+                    # dictionary and continue.
+                    else:
+                        current_level[keys[i]] = dict()
+                        current_level = current_level[keys[i]]
+
                 facts_to_set = merge_facts(orig=facts_to_set,
                                            new=oo_env_facts,
                                            additive_facts_to_overwrite=[],


Test patch: