Repository: openshift/openshift-ansible

Issue:
Health check callback plugin causes warning on unhashable value
#### Description

Traceback when there was a random failure in logging:

```
roles/openshift_health_checker/callback_plugins/zz_failure_summary.py", line 147, in deduplicate_failures
    groups[group_key].append(failure)
TypeError: unhashable type: 'dict'
```

This is from the failure de-duplication code in the callback plugin. The result came back with a dict for a value and it won't hash.

##### Version

master

##### Observed Results

```
$ ansible-playbook playbooks/byo/config.yml -vvv
...
TASK [openshift_logging_elasticsearch : Set ES configmap] *********************************************************************************************************************************************************
task path: /home/lmeyer/go/src/github.com/openshift/openshift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:147
Wednesday 06 September 2017  09:52:01 -0400 (0:00:00.031)       0:16:19.770 *** 
Using module file /home/lmeyer/go/src/github.com/openshift/openshift-ansible/roles/lib_openshift/library/oc_configmap.py
<ec2-54-172-162-93.compute-1.amazonaws.com> (1, '\n{"msg": {"returncode": 1, "cmd": "/bin/oc create configmap logging-elasticsearch --from-file=elasticsearch.yml=/tmp/openshift-logging-ansible-C6rjd2/elasticsearch.yml --from-file=logging.yml=/tmp/openshift-logging-ansible-C6rjd2/elasticsearch-logging.yml -n logging", "results": {}, "stderr": "Error from server: timeout\\n", "stdout": ""}, "failed": true, "invocation": {"module_args": {"from_file": {"elasticsearch.yml": "/tmp/openshift-logging-ansible-C6rjd2/elasticsearch.yml", "logging.yml": "/tmp/openshift-logging-ansible-C6rjd2/elasticsearch-logging.yml"}, "name": "logging-elasticsearch", "from_literal": null, "namespace": "logging", "kubeconfig": "/etc/origin/master/admin.kubeconfig", "state": "present", "debug": false}}}\n', '')
fatal: [ec2-54-172-162-93.compute-1.amazonaws.com]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_args": {
            "debug": false, 
            "from_file": {
                "elasticsearch.yml": "/tmp/openshift-logging-ansible-C6rjd2/elasticsearch.yml", 
                "logging.yml": "/tmp/openshift-logging-ansible-C6rjd2/elasticsearch-logging.yml"
            }, 
            "from_literal": null, 
            "kubeconfig": "/etc/origin/master/admin.kubeconfig", 
            "name": "logging-elasticsearch", 
            "namespace": "logging", 
            "state": "present"
        }
    }
}

MSG:

{u'returncode': 1, u'cmd': u'/bin/oc create configmap logging-elasticsearch --from-file=elasticsearch.yml=/tmp/openshift-logging-ansible-C6rjd2/elasticsearch.yml --from-file=logging.yml=/tmp/openshift-logging-ansible-C6rjd2/elasticsearch-logging.yml -n logging', u'results': {}, u'stderr': u'Error from server: timeout\n', u'stdout': u''}

...
An error happened while generating a summary of failures:
Traceback (most recent call last):
  File "/home/lmeyer/go/src/github.com/openshift/openshift-ansible/roles/openshift_health_checker/callback_plugins/zz_failure_summary.py", line 48, in v2_playbook_on_stats
    self._display.display(failure_summary(self.__failures, self.__playbook_file))
  File "/home/lmeyer/go/src/github.com/openshift/openshift-ansible/roles/openshift_health_checker/callback_plugins/zz_failure_summary.py", line 74, in failure_summary
    failures = deduplicate_failures(failures)
  File "/home/lmeyer/go/src/github.com/openshift/openshift-ansible/roles/openshift_health_checker/callback_plugins/zz_failure_summary.py", line 147, in deduplicate_failures
    groups[group_key].append(failure)
TypeError: unhashable type: 'dict'

```

##### Additional Information
This isn't the cause of whatever failed, it's just adding to the confusion with a warning at the end.

I think it's the "msg" being returned in the task failure that it's trying to hash and failing. Callback plugin code needs to be more paranoid :)


Discussion:
@sosiouxme, ack. I've hit this when implementing the dedup logic. We could capture the error and abort the dedup -- print the regular output.

Do you have the other part of the log to show what was the value of `failure`?
I have the full log but I'm not sure what you mean. Nothing shows exactly what the dedup code was looking at. I'm guessing the result that got sent off to the failure handler included an element with some of the result shown above. I don't see anything else that's of interest. I'll attach the log though...

[ansible.20170906.1.txt](https://github.com/openshift/openshift-ansible/files/1284589/ansible.20170906.1.txt)

Just to be sure, are you observing this issue on the master branch?
This code here is responsible for transforming a failure into a `dict` such that all *keys* are strings and all *values* should be hashable.

https://github.com/openshift/openshift-ansible/blob/aad4ccce7ce6641d3c720507568f6da779a66995/roles/openshift_health_checker/callback_plugins/zz_failure_summary.py#L101-L120

`host`, `play`, `task`, and `msg` should all be strings. For `checks` we should have a tuple of tuples of strings... now if for some reason check's `msg` is not a string... then... boom.
I can slowly type a PR to make it more robust
On Thu, Sep 7, 2017 at 8:34 AM, Rodolfo Carvalho <notifications@github.com>
wrote:

> For checks we should have a tuple of tuples of strings... now if for some
> reason check's msg is not a string... then... boom.
>
>
Yeah, but this wasn't a check failure, it was a regular task failing. And I
can't tell for sure but I think its `msg` was a hash.

This was... pretty close to master branch, if not on it (was testing an
unrelated PR at the time).

yes, the most likely in having a `dict` as the message... hmm, so yeah, will try to make it more robust

Resolving patch:
diff --git a/roles/openshift_health_checker/callback_plugins/zz_failure_summary.py b/roles/openshift_health_checker/callback_plugins/zz_failure_summary.py
--- a/roles/openshift_health_checker/callback_plugins/zz_failure_summary.py
+++ b/roles/openshift_health_checker/callback_plugins/zz_failure_summary.py
@@ -10,6 +10,7 @@
 from ansible.plugins.callback import CallbackBase
 from ansible import constants as C
 from ansible.utils.color import stringc
+from ansible.module_utils.six import string_types
 
 
 FAILED_NO_MSG = u'Failed without returning a message.'
@@ -140,11 +141,19 @@ def deduplicate_failures(failures):
     Returns a new list of failures such that identical failures from different
     hosts are grouped together in a single entry. The relative order of failures
     is preserved.
+
+    If failures is unhashable, the original list of failures is returned.
     """
     groups = defaultdict(list)
     for failure in failures:
         group_key = tuple(sorted((key, value) for key, value in failure.items() if key != 'host'))
-        groups[group_key].append(failure)
+        try:
+            groups[group_key].append(failure)
+        except TypeError:
+            # abort and return original list of failures when failures has an
+            # unhashable type.
+            return failures
+
     result = []
     for failure in failures:
         group_key = tuple(sorted((key, value) for key, value in failure.items() if key != 'host'))
@@ -159,7 +168,10 @@ def format_failure(failure):
     """Return a list of pretty-formatted text entries describing a failure, including
     relevant information about it. Expect that the list of text entries will be joined
     by a newline separator when output to the user."""
-    host = u', '.join(failure['host'])
+    if isinstance(failure['host'], string_types):
+        host = failure['host']
+    else:
+        host = u', '.join(failure['host'])
     play = failure['play']
     task = failure['task']
     msg = failure['msg']


Test patch:
diff --git a/roles/openshift_health_checker/test/zz_failure_summary_test.py b/roles/openshift_health_checker/test/zz_failure_summary_test.py
--- a/roles/openshift_health_checker/test/zz_failure_summary_test.py
+++ b/roles/openshift_health_checker/test/zz_failure_summary_test.py
@@ -65,6 +65,21 @@
             },
         ],
     ),
+    # if a failure contain an unhashable value, it will not be deduplicated
+    (
+        [
+            {
+                'host': 'master1',
+                'msg': {'unhashable': 'value'},
+            },
+        ],
+        [
+            {
+                'host': 'master1',
+                'msg': {'unhashable': 'value'},
+            },
+        ],
+    ),
 ])
 def test_deduplicate_failures(failures, deduplicated):
     assert deduplicate_failures(failures) == deduplicated