Repository: ansible/ansible

Issue:
python3 - Intersect filter change in behavior (docs + code)
##### ISSUE TYPE
 - Bug Report
 - Documentation Report

##### COMPONENT NAME
 intersect filter

##### ANSIBLE VERSION
```
ansible 2.3.1.0
  config file =
  configured module search path = Default w/o overrides
  python version = 3.6.1 (default, Mar 27 2017, 00:27:06) [GCC 6.3.1 20170306]
```

##### CONFIGURATION
Nothing special

##### OS / ENVIRONMENT
I am currently running ArchLinux, but same behavior caught with Ubuntu Xenial.
Different behavior with python3 and python2.

##### SUMMARY
First, the documentation about filters says ([1]) ``All these functions return a unique set from sets or lists.``
You can read, on the same page, ``To get the intersection of 2 lists (unique list of all items in both)``. 

I don't know if I should expect a set or a list if I intersect with a set. There is an expectation issue here.

However, fun stuff happens when you realize it doesn't behave the same way under python2 and python3.

I get an intersection under python2 and a warning (it doesn't work) in python3.

[1]: http://docs.ansible.com/ansible/playbooks_filters.html#set-theory-filters

##### STEPS TO REPRODUCE

```
---
- hosts: nova_all
  connection: local
  gather_facts: no
  tasks:
    - debug:
        msg: "{{ groups['nova_all'] | intersect(ansible_play_hosts) }}"
```

Here is my inventory:

```
[all]
host1
localhost

[nova_all]
localhost
```

##### EXPECTED RESULTS
Output: "localhost"

##### ACTUAL RESULTS

```
TASK [debug] ************************************************************************************************************************************************************************
task path: /home/jean-philippe/evrardjp/pocs/intersect.yml:6
 11311 1499361217.87448: sending task start callback
 11311 1499361217.87454: entering _queue_task() for localhost/debug
 11311 1499361217.87462: Creating lock for debug
 11311 1499361217.87657: worker is 1 (out of 1 available)
 11311 1499361217.87697: exiting _queue_task() for localhost/debug
 11311 1499361217.87712: done queuing things up, now waiting for results queue to drain
 11311 1499361217.87720: waiting for pending results...
 11319 1499361217.90017: running TaskExecutor() for localhost/TASK: debug
 11319 1499361217.90069: in run()
 11319 1499361217.90111: calling self._execute()
 11319 1499361217.90326: Loading FilterModule 'core' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/filter/core.py (found_in_cache=True, class_only=False)
 11319 1499361217.90348: Loading FilterModule 'ipaddr' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/filter/ipaddr.py (found_in_cache=True, class_only=False)
 11319 1499361217.90362: Loading FilterModule 'json_query' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/filter/json_query.py (found_in_cache=True, class_only=False)
 11319 1499361217.90374: Loading FilterModule 'mathstuff' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/filter/mathstuff.py (found_in_cache=True, class_only=False)
 11319 1499361217.90420: Loading TestModule 'core' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/test/core.py (found_in_cache=True, class_only=False)
 11319 1499361217.90434: Loading TestModule 'files' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/test/files.py (found_in_cache=True, class_only=False)
 11319 1499361217.90446: Loading TestModule 'mathstuff' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/test/mathstuff.py (found_in_cache=True, class_only=False)
 11319 1499361217.90804: Loading FilterModule 'core' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/filter/core.py (found_in_cache=True, class_only=False)
 11319 1499361217.90823: Loading FilterModule 'ipaddr' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/filter/ipaddr.py (found_in_cache=True, class_only=False)
 11319 1499361217.90842: Loading FilterModule 'json_query' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/filter/json_query.py (found_in_cache=True, class_only=False)
 11319 1499361217.90855: Loading FilterModule 'mathstuff' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/filter/mathstuff.py (found_in_cache=True, class_only=False)
 11319 1499361217.90893: Loading TestModule 'core' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/test/core.py (found_in_cache=True, class_only=False)
 11319 1499361217.90906: Loading TestModule 'files' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/test/files.py (found_in_cache=True, class_only=False)
 11319 1499361217.90920: Loading TestModule 'mathstuff' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/test/mathstuff.py (found_in_cache=True, class_only=False)
 11319 1499361217.91143: Loading Connection 'local' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/connection/local.py
 11319 1499361217.91231: Loading ShellModule 'csh' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/shell/csh.py
 11319 1499361217.91306: Loading ShellModule 'fish' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/shell/fish.py
 11319 1499361217.91378: Loading ShellModule 'powershell' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/shell/powershell.py
 11319 1499361217.91420: Loading ShellModule 'sh' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/shell/sh.py
 11319 1499361217.91448: Loading ShellModule 'sh' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/shell/sh.py (found_in_cache=True, class_only=False)
 11319 1499361217.91468: Loading ActionModule 'debug' from /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/action/debug.py (searched paths: /home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/action:/home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/action/__pycache__) (found_in_cache=True, class_only=False)
 11319 1499361217.91483: starting attempt loop
 11319 1499361217.91494: running the handler
 11319 1499361217.91506: handler run complete
 11319 1499361217.91526: attempt loop complete, returning result
 11319 1499361217.91537: _execute() done
 11319 1499361217.91546: dumping result to json
 11319 1499361217.91557: done dumping result, returning
 11319 1499361217.91568: done running TaskExecutor() for localhost/TASK: debug
 11319 1499361217.91584: sending task result
 11319 1499361217.91627: done sending task result
 11319 1499361217.91683: WORKER PROCESS EXITING
 [WARNING]: Failure using method (v2_runner_on_ok) in callback plugin (<ansible.plugins.callback.default.CallbackModule object at 0x7ffad0014a20>): Object of type 'set' is not JSON
serializable

 11311 1499361217.91893: Callback Exception:
  File "/home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/executor/task_queue_manager.py", line 373, in send_callback
    method(*args, **kwargs)
   File "/home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/callback/default.py", line 95, in v2_runner_on_ok
    msg += " => %s" % (self._dump_results(result._result),)
   File "/home/jean-philippe/.pip-apps/virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/callback/__init__.py", line 98, in _dump_results
    return json.dumps(abridged_result, indent=indent, ensure_ascii=False, sort_keys=sort_keys)
   File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
   File "/usr/lib64/python3.6/json/encoder.py", line 201, in encode
    chunks = list(chunks)
   File "/usr/lib64/python3.6/json/encoder.py", line 430, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
   File "/usr/lib64/python3.6/json/encoder.py", line 404, in _iterencode_dict
    yield from chunks
   File "/usr/lib64/python3.6/json/encoder.py", line 437, in _iterencode
    o = _default(o)
   File "/usr/lib64/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)

 11311 1499361217.91910: no more pending results, returning what we have
 11311 1499361217.91917: results queue empty
 11311 1499361217.91923: checking for any_errors_fatal
 11311 1499361217.91930: done checking for any_errors_fatal
 11311 1499361217.91935: checking for max_fail_percentage
 11311 1499361217.91941: done checking for max_fail_percentage
 11311 1499361217.91947: checking to see if all hosts have failed and the running result is not ok
 11311 1499361217.91952: done checking to see if all hosts have failed
 11311 1499361217.91957: getting the remaining hosts for this loop
 11311 1499361217.91964: done getting the remaining hosts for this loop
 11311 1499361217.91972: building list of next tasks for hosts
 11311 1499361217.91979: getting the next task for host localhost
 11311 1499361217.91987: done getting next task for host localhost
 11311 1499361217.91997:  ^ task is: TASK: meta (flush_handlers)
 11311 1499361217.92004:  ^ state is: HOST STATE: block=3, task=1, rescue=0, always=0, run_state=ITERATING_TASKS, fail_state=FAILED_NONE, pending_setup=False, tasks child state? (None), rescue child state? (None), always child state? (None), did rescue? False, did start at task? False
 11311 1499361217.92010: done building task lists
```





Discussion:
It appears to me that the same happens for the difference filter.
Probably all math ones?

Resolving patch:
diff --git a/lib/ansible/plugins/filter/mathstuff.py b/lib/ansible/plugins/filter/mathstuff.py
--- a/lib/ansible/plugins/filter/mathstuff.py
+++ b/lib/ansible/plugins/filter/mathstuff.py
@@ -44,7 +44,7 @@ def intersect(a, b):
     if isinstance(a, collections.Hashable) and isinstance(b, collections.Hashable):
         c = set(a) & set(b)
     else:
-        c = unique(filter(lambda x: x in b, a))
+        c = unique([x for x in a if x in b])
     return c
 
 
@@ -52,7 +52,7 @@ def difference(a, b):
     if isinstance(a, collections.Hashable) and isinstance(b, collections.Hashable):
         c = set(a) - set(b)
     else:
-        c = unique(filter(lambda x: x not in b, a))
+        c = unique([x for x in a if x not in b])
     return c
 
 
@@ -60,7 +60,7 @@ def symmetric_difference(a, b):
     if isinstance(a, collections.Hashable) and isinstance(b, collections.Hashable):
         c = set(a) ^ set(b)
     else:
-        c = unique(filter(lambda x: x not in intersect(a, b), union(a, b)))
+        c = unique([x for x in union(a, b) if x not in intersect(a, b)])
     return c
 
 
@@ -105,7 +105,7 @@ def inversepower(x, base=2):
             return math.sqrt(x)
         else:
             return math.pow(x, 1.0 / float(base))
-    except TypeError as e:
+    except (ValueError, TypeError) as e:
         raise errors.AnsibleFilterError('root() can only be used on numbers: %s' % str(e))
 
 


Test patch:
diff --git a/test/units/plugins/filter/test_mathstuff.py b/test/units/plugins/filter/test_mathstuff.py
new file mode 100644
--- /dev/null
+++ b/test/units/plugins/filter/test_mathstuff.py
@@ -0,0 +1,122 @@
+# Copyright: (c) 2017, Ansible Project
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+
+# Make coding more python3-ish
+from __future__ import (absolute_import, division, print_function)
+__metaclass__ = type
+
+import pytest
+
+import ansible.plugins.filter.mathstuff as ms
+from ansible.errors import AnsibleFilterError
+
+
+UNIQUE_DATA = (([1, 3, 4, 2], sorted([1, 2, 3, 4])),
+               ([1, 3, 2, 4, 2, 3], sorted([1, 2, 3, 4])),
+               (['a', 'b', 'c', 'd'], sorted(['a', 'b', 'c', 'd'])),
+               (['a', 'a', 'd', 'b', 'a', 'd', 'c', 'b'], sorted(['a', 'b', 'c', 'd'])),
+               )
+
+TWO_SETS_DATA = (([1, 2], [3, 4], ([], sorted([1, 2]), sorted([1, 2, 3, 4]), sorted([1, 2, 3, 4]))),
+                 ([1, 2, 3], [5, 3, 4], ([3], sorted([1, 2]), sorted([1, 2, 5, 4]), sorted([1, 2, 3, 4, 5]))),
+                 (['a', 'b', 'c'], ['d', 'c', 'e'], (['c'], sorted(['a', 'b']), sorted(['a', 'b', 'd', 'e']), sorted(['a', 'b', 'c', 'e', 'd']))),
+                 )
+
+
+@pytest.mark.parametrize('data, expected', UNIQUE_DATA)
+class TestUnique:
+    def test_unhashable(self, data, expected):
+        assert sorted(ms.unique(list(data))) == expected
+
+    def test_hashable(self, data, expected):
+        assert sorted(ms.unique(tuple(data))) == expected
+
+
+@pytest.mark.parametrize('dataset1, dataset2, expected', TWO_SETS_DATA)
+class TestIntersect:
+    def test_unhashable(self, dataset1, dataset2, expected):
+        assert sorted(ms.intersect(list(dataset1), list(dataset2))) == expected[0]
+
+    def test_hashable(self, dataset1, dataset2, expected):
+        assert sorted(ms.intersect(tuple(dataset1), tuple(dataset2))) == expected[0]
+
+
+@pytest.mark.parametrize('dataset1, dataset2, expected', TWO_SETS_DATA)
+class TestDifference:
+    def test_unhashable(self, dataset1, dataset2, expected):
+        assert sorted(ms.difference(list(dataset1), list(dataset2))) == expected[1]
+
+    def test_hashable(self, dataset1, dataset2, expected):
+        assert sorted(ms.difference(tuple(dataset1), tuple(dataset2))) == expected[1]
+
+
+@pytest.mark.parametrize('dataset1, dataset2, expected', TWO_SETS_DATA)
+class TestSymmetricDifference:
+    def test_unhashable(self, dataset1, dataset2, expected):
+        assert sorted(ms.symmetric_difference(list(dataset1), list(dataset2))) == expected[2]
+
+    def test_hashable(self, dataset1, dataset2, expected):
+        assert sorted(ms.symmetric_difference(tuple(dataset1), tuple(dataset2))) == expected[2]
+
+
+class TestMin:
+    def test_min(self):
+        assert ms.min((1, 2)) == 1
+        assert ms.min((2, 1)) == 1
+        assert ms.min(('p', 'a', 'w', 'b', 'p')) == 'a'
+
+
+class TestMax:
+    def test_max(self):
+        assert ms.max((1, 2)) == 2
+        assert ms.max((2, 1)) == 2
+        assert ms.max(('p', 'a', 'w', 'b', 'p')) == 'w'
+
+
+class TestLogarithm:
+    def test_log_non_number(self):
+        with pytest.raises(AnsibleFilterError, message='log() can only be used on numbers: a float is required'):
+            ms.logarithm('a')
+        with pytest.raises(AnsibleFilterError, message='log() can only be used on numbers: a float is required'):
+            ms.logarithm(10, base='a')
+
+    def test_log_ten(self):
+        assert ms.logarithm(10, 10) == 1.0
+        assert ms.logarithm(69, 10) * 1000 // 1 == 1838
+
+    def test_log_natural(self):
+        assert ms.logarithm(69) * 1000 // 1 == 4234
+
+    def test_log_two(self):
+        assert ms.logarithm(69, 2) * 1000 // 1 == 6108
+
+
+class TestPower:
+    def test_power_non_number(self):
+        with pytest.raises(AnsibleFilterError, message='pow() can only be used on numbers: a float is required'):
+            ms.power('a', 10)
+
+        with pytest.raises(AnsibleFilterError, message='pow() can only be used on numbers: a float is required'):
+            ms.power(10, 'a')
+
+    def test_power_squared(self):
+        assert ms.power(10, 2) == 100
+
+    def test_power_cubed(self):
+        assert ms.power(10, 3) == 1000
+
+
+class TestInversePower:
+    def test_root_non_number(self):
+        with pytest.raises(AnsibleFilterError, message='root() can only be used on numbers: a float is required'):
+            ms.inversepower(10, 'a')
+
+        with pytest.raises(AnsibleFilterError, message='root() can only be used on numbers: a float is required'):
+            ms.inversepower('a', 10)
+
+    def test_square_root(self):
+        assert ms.inversepower(100) == 10
+        assert ms.inversepower(100, 2) == 10
+
+    def test_cube_root(self):
+        assert ms.inversepower(27, 3) == 3