Repository: ansible/ansible

Issue:
RuntimeError: maximum recursion depth exceeded in cmp when handling large amounts of data in module_utils/basic.py remove_values()
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
basic

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.3.1.0 (stable-2.3 cdd6ab000b) last updated 2017/05/12 08:12:54 (GMT +000)
  config file = /root/bachelor/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]

```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->
N/A

##### SUMMARY
<!--- Explain the problem briefly -->
When dealing with large amount of output, in my case junos netconf routing tables and bgp peer data, I've hit an issue with what seems to be `remove_values()` in `module_utils/basic.py` which causes python to error out with a `RuntimeError: maximum recursion depth exceeded in cmp`

An abbreviated log can be seen further down.

Given I have no testcase that does not involve a specific router that returns large amounts of data, I'm not entirely sure this can easily be resolved. The error could also be misleading, but it's all I have to go by at present.
##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
Not 100% sure how to reproduce this issue as it depends on a very large amount of output for ansible run trough `remove_values()`. In my test case the XML output from the routing table alone is close to `600k` lines, so somewhere between `600k` and `1m` entries to sanitize.


<!--- Paste example playbooks or commands between quotes below -->


<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
Play finishing successfully

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
Please note that the junos_facts module has been modified to pull extra data which accounts for the unusual line number for `main()`
```
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: RuntimeError: maximum recursion depth exceeded in cmp
fatal: [trd-gw]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Traceback (most recent call last):
  File \"/tmp/ansible_4P2WOB/ansible_module_junos_facts.py\", line 757, in <module>
    main()
  File \"/tmp/ansible_4P2WOB/ansible_module_junos_facts.py\", line 753, in main
    module.exit_json(ansible_facts=ansible_facts, warnings=warnings)
  File \"/tmp/ansible_4P2WOB/ansible_modlib.zip/ansible/module_utils/basic.py\", line 1983, in exit_json
  File \"/tmp/ansible_4P2WOB/ansible_modlib.zip/ansible/module_utils/basic.py\", line 1973, in _return_formatted
  File \"/tmp/ansible_4P2WOB/ansible_modlib.zip/ansible/module_utils/basic.py\", line 414, in remove_values
  File \"/tmp/ansible_4P2WOB/ansible_modlib.zip/ansible/module_utils/basic.py\", line 414, in <genexpr>
  File \"/tmp/ansible_4P2WOB/ansible_modlib.zip/ansible/module_utils/basic.py\", line 414, in remove_values
  File \"/tmp/ansible_4P2WOB/ansible_modlib.zip/ansible/module_utils/basic.py\", line 414, in <genexpr>
  File \"/tmp/ansible_4P2WOB/ansible_modlib.zip/ansible/module_utils/basic.py\", line 414, in remove_values
  File \"/tmp/ansible_4P2WOB/ansible_modlib.zip/ansible/module_utils/basic.py\", line 414, in <genexpr>
  [---- SNIP ----]
  File \"/tmp/ansible_4P2WOB/ansible_modlib.zip/ansible/module_utils/basic.py\", line 411, in remove_values
  File \"/usr/lib/python2.7/abc.py\", line 141, in __instancecheck__
    subtype in cls._abc_negative_cache):
  File \"/usr/lib/python2.7/_weakrefset.py\", line 75, in __contains__
    return wr in self.data
RuntimeError: maximum recursion depth exceeded in cmp
", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 0}
```



Discussion:

Resolving patch:
diff --git a/lib/ansible/module_utils/basic.py b/lib/ansible/module_utils/basic.py
--- a/lib/ansible/module_utils/basic.py
+++ b/lib/ansible/module_utils/basic.py
@@ -89,6 +89,8 @@
 import platform
 import errno
 import datetime
+from collections import deque
+from collections import Mapping, MutableMapping, Sequence, MutableSequence, Set, MutableSet
 from itertools import repeat, chain
 
 try:
@@ -113,13 +115,6 @@
 # Python2 & 3 way to get NoneType
 NoneType = type(None)
 
-try:
-    from collections import Sequence, Mapping
-except ImportError:
-    # python2.5
-    Sequence = (list, tuple)
-    Mapping = (dict,)
-
 # Note: When getting Sequence from collections, it matches with strings.  If
 # this matters, make sure to check for strings before checking for sequencetype
 try:
@@ -405,20 +400,44 @@ def return_values(obj):
         raise TypeError('Unknown parameter type: %s, %s' % (type(obj), obj))
 
 
-def remove_values(value, no_log_strings):
-    """ Remove strings in no_log_strings from value.  If value is a container
-    type, then remove a lot more"""
+def _remove_values_conditions(value, no_log_strings, deferred_removals):
+    """
+    Helper function for :meth:`remove_values`.
+
+    :arg value: The value to check for strings that need to be stripped
+    :arg no_log_strings: set of strings which must be stripped out of any values
+    :arg deferred_removals: List which holds information about nested
+        containers that have to be iterated for removals.  It is passed into
+        this function so that more entries can be added to it if value is
+        a container type.  The format of each entry is a 2-tuple where the first
+        element is the ``value`` parameter and the second value is a new
+        container to copy the elements of ``value`` into once iterated.
+    :returns: if ``value`` is a scalar, returns ``value`` with two exceptions:
+        1. :class:`~datetime.datetime` objects which are changed into a string representation.
+        2. objects which are in no_log_strings are replaced with a placeholder
+            so that no sensitive data is leaked.
+        If ``value`` is a container type, returns a new empty container.
+
+    ``deferred_removals`` is added to as a side-effect of this function.
+
+    .. warning:: It is up to the caller to make sure the order in which value
+        is passed in is correct.  For instance, higher level containers need
+        to be passed in before lower level containers. For example, given
+        ``{'level1': {'level2': 'level3': [True]} }`` first pass in the
+        dictionary for ``level1``, then the dict for ``level2``, and finally
+        the list for ``level3``.
+    """
     if isinstance(value, (text_type, binary_type)):
         # Need native str type
         native_str_value = value
         if isinstance(value, text_type):
             value_is_text = True
             if PY2:
-                native_str_value = to_bytes(value, encoding='utf-8', errors='surrogate_or_strict')
+                native_str_value = to_bytes(value, errors='surrogate_or_strict')
         elif isinstance(value, binary_type):
             value_is_text = False
             if PY3:
-                native_str_value = to_text(value, encoding='utf-8', errors='surrogate_or_strict')
+                native_str_value = to_text(value, errors='surrogate_or_strict')
 
         if native_str_value in no_log_strings:
             return 'VALUE_SPECIFIED_IN_NO_LOG_PARAMETER'
@@ -431,10 +450,31 @@ def remove_values(value, no_log_strings):
             value = to_bytes(native_str_value, encoding='utf-8', errors='surrogate_then_replace')
         else:
             value = native_str_value
-    elif isinstance(value, SEQUENCETYPE):
-        return [remove_values(elem, no_log_strings) for elem in value]
+
+    elif isinstance(value, Sequence):
+        if isinstance(value, MutableSequence):
+            new_value = type(value)()
+        else:
+            new_value = []  # Need a mutable value
+        deferred_removals.append((value, new_value))
+        value = new_value
+
+    elif isinstance(value, Set):
+        if isinstance(value, MutableSet):
+            new_value = type(value)()
+        else:
+            new_value = set()  # Need a mutable value
+        deferred_removals.append((value, new_value))
+        value = new_value
+
     elif isinstance(value, Mapping):
-        return dict((k, remove_values(v, no_log_strings)) for k, v in value.items())
+        if isinstance(value, MutableMapping):
+            new_value = type(value)()
+        else:
+            new_value = {}  # Need a mutable value
+        deferred_removals.append((value, new_value))
+        value = new_value
+
     elif isinstance(value, tuple(chain(NUMBERTYPES, (bool, NoneType)))):
         stringy_value = to_native(value, encoding='utf-8', errors='surrogate_or_strict')
         if stringy_value in no_log_strings:
@@ -442,13 +482,42 @@ def remove_values(value, no_log_strings):
         for omit_me in no_log_strings:
             if omit_me in stringy_value:
                 return 'VALUE_SPECIFIED_IN_NO_LOG_PARAMETER'
+
     elif isinstance(value, datetime.datetime):
         value = value.isoformat()
     else:
         raise TypeError('Value of unknown type: %s, %s' % (type(value), value))
+
     return value
 
 
+def remove_values(value, no_log_strings):
+    """ Remove strings in no_log_strings from value.  If value is a container
+    type, then remove a lot more"""
+    deferred_removals = deque()
+
+    no_log_strings = [to_native(s, errors='surrogate_or_strict') for s in no_log_strings]
+    new_value = _remove_values_conditions(value, no_log_strings, deferred_removals)
+
+    while deferred_removals:
+        old_data, new_data = deferred_removals.popleft()
+        if isinstance(new_data, Mapping):
+            for old_key, old_elem in old_data.items():
+                new_elem = _remove_values_conditions(old_elem, no_log_strings, deferred_removals)
+                new_data[old_key] = new_elem
+        else:
+            for elem in old_data:
+                new_elem = _remove_values_conditions(elem, no_log_strings, deferred_removals)
+                if isinstance(new_data, MutableSequence):
+                    new_data.append(new_elem)
+                elif isinstance(new_data, MutableSet):
+                    new_data.add(new_elem)
+                else:
+                    raise TypeError('Unknown container type encountered when removing private values from output')
+
+    return new_value
+
+
 def heuristic_log_sanitize(data, no_log_values=None):
     ''' Remove strings that look like passwords from log messages '''
     # Currently filters:
@@ -695,6 +764,13 @@ class AnsibleFallbackNotFound(Exception):
     pass
 
 
+class _SetEncoder(json.JSONEncoder):
+    def default(self, obj):
+        if isinstance(obj, Set):
+            return list(obj)
+        return super(_SetEncoder, self).default(obj)
+
+
 class AnsibleModule(object):
     def __init__(self, argument_spec, bypass_checks=False, no_log=False,
                  check_invalid_arguments=True, mutually_exclusive=None, required_together=None,
@@ -1733,7 +1809,7 @@ def _check_type_jsonarg(self, value):
             return value.strip()
         else:
             if isinstance(value, (list, tuple, dict)):
-                return json.dumps(value)
+                return self.jsonify(value)
         raise TypeError('%s cannot be converted to a json string' % type(value))
 
     def _check_type_raw(self, value):
@@ -1983,14 +2059,14 @@ def boolean(self, arg):
     def jsonify(self, data):
         for encoding in ("utf-8", "latin-1"):
             try:
-                return json.dumps(data, encoding=encoding)
+                return json.dumps(data, encoding=encoding, cls=_SetEncoder)
             # Old systems using old simplejson module does not support encoding keyword.
             except TypeError:
                 try:
                     new_data = json_dict_bytes_to_unicode(data, encoding=encoding)
                 except UnicodeDecodeError:
                     continue
-                return json.dumps(new_data)
+                return json.dumps(new_data, cls=_SetEncoder)
             except UnicodeDecodeError:
                 continue
         self.fail_json(msg='Invalid unicode encoding encountered')


Test patch:
diff --git a/test/units/module_utils/basic/test_no_log.py b/test/units/module_utils/basic/test_no_log.py
--- a/test/units/module_utils/basic/test_no_log.py
+++ b/test/units/module_utils/basic/test_no_log.py
@@ -90,10 +90,9 @@ class TestRemoveValues(unittest.TestCase):
             },
             frozenset(['nope'])
         ),
-        ('Toshio くら', frozenset(['とみ'])),
-        (u'Toshio くら', frozenset(['とみ'])),
+        (u'Toshio くら'.encode('utf-8'), frozenset([u'とみ'.encode('utf-8')])),
+        (u'Toshio くら', frozenset([u'とみ'])),
     )
-
     dataset_remove = (
         ('string', frozenset(['string']), OMIT),
         (1234, frozenset(['1234']), OMIT),
@@ -138,8 +137,8 @@ class TestRemoveValues(unittest.TestCase):
             frozenset(['enigma', 'mystery', 'secret']),
             'This sentence has an ******** wrapped in a ******** inside of a ********. - mr ********'
         ),
-        ('Toshio くらとみ', frozenset(['くらとみ']), 'Toshio ********'),
-        (u'Toshio くらとみ', frozenset(['くらとみ']), u'Toshio ********'),
+        (u'Toshio くらとみ'.encode('utf-8'), frozenset([u'くらとみ'.encode('utf-8')]), u'Toshio ********'.encode('utf-8')),
+        (u'Toshio くらとみ', frozenset([u'くらとみ']), u'Toshio ********'),
     )
 
     def test_no_removal(self):
@@ -152,3 +151,30 @@ def test_strings_to_remove(self):
 
     def test_unknown_type(self):
         self.assertRaises(TypeError, remove_values, object(), frozenset())
+
+    def test_hit_recursion_limit(self):
+        """ Check that we do not hit a recursion limit"""
+        data_list = []
+        inner_list = data_list
+        for i in range(0, 10000):
+            new_list = []
+            inner_list.append(new_list)
+            inner_list = new_list
+        inner_list.append('secret')
+
+        # Check that this does not hit a recursion limit
+        actual_data_list = remove_values(data_list, frozenset(('secret',)))
+
+        levels = 0
+        inner_list = actual_data_list
+        while inner_list:
+            if isinstance(inner_list, list):
+                self.assertEquals(len(inner_list), 1)
+            else:
+                levels -= 1
+                break
+            inner_list = inner_list[0]
+            levels += 1
+
+        self.assertEquals(inner_list, self.OMIT)
+        self.assertEquals(levels, 10000)