Repository: ansible/ansible

Issue:
When using filter in setup module returned variables may be wrong
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
setup

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.4.2.0
```

##### CONFIGURATION
<!---
If using Ansible 2.4 or above, paste the results of "ansible-config dump --only-changed"

Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).

-->
does not matter

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->
N/A

##### SUMMARY
<!--- Explain the problem briefly -->
Similar to bugs #30753, #31095 ansible_service_mgr fact may be wrong but only when used
together with the setup filter! Omiting filter the value of ansible_service_mgr is correct.

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
Broken:
`ansible hostname -bK -m setup -a filter=ansible_service_mgr`

vs:

This one works:
`ansible hostname -bK -m setup | grep ansible_service_mgr`

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
A valid service_mgr like `upstart`, `sysvinit` or `systemd`

##### ACTUAL RESULTS

Fallback to `service`
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
host | SUCCESS => {
    "ansible_facts": {
        "ansible_service_mgr": "service"
    }
}
```



Discussion:
Some discussion was done in #32086 
/cc @alikins
cc @david_obrien
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
I've just downloaded and tried 2.4.2-rc1, and this bug is still in there. Any chance there will be a fix before 2.4.2 is released?
Finally have a reproducer for this, needed ubuntu1404 vm on remote not a 16 (or a docker image). 

```
[newswoop:F27:ansible (devel % u=)]$ ansible -i hosts_vagrant 192.168.121.125 -m setup| grep service
        "ansible_service_mgr": "upstart", 
[newswoop:F27:ansible (devel % u=)]$ ansible -i hosts_vagrant 192.168.121.125 -m setup -a filter=ansible_service_mgr
192.168.121.125 | SUCCESS => {
    "ansible_facts": {
        "ansible_service_mgr": "service"
    }, 
    "changed": false
}
```

For non filtered case, the distro/version facts seem correct. So filter must be effecting the fact deps somehow even though filter should be getting applied afterwards. Digging...

Interesting. Even when using a fine grained gather_subset ('gather_subset=service_mgr') adding the filter still returns wrong answer. With the gather_subset and without the filter does the right thing though.

(To some degree using fine grained gather_subset is a better/faster approach than to 'filter=ansible_service_mgr. That does correctly collect all the fact deps and return correct ansible_service_mgr fact ('upstart').)

ie, instead of:

`ansible localhost -m setup -a filter=ansible_service_mgr`

use gather_setup instead:

`ansible localhost -m setup gather_subset=service_mgr'

```
[newswoop:F27:ansible (devel % u=)]$ time ansible -i hosts_vagrant 192.168.121.125 -m setup -a filter=ansible_service_mgr
192.168.121.125 | SUCCESS => {
    "ansible_facts": {
        "ansible_service_mgr": "service"
    }, 
    "changed": false
}

real	0m2.862s
user	0m1.006s
sys	0m0.139s
[newswoop:F27:ansible (devel % u=)]$ time ansible -i hosts_vagrant 192.168.121.125 -m setup -a gather_subset=service_mgr | grep ansible_service_mgr
        "ansible_service_mgr": "upstart", 

real	0m1.835s
user	0m0.999s
sys	0m0.124s
```

The gather_subset is faster since it only collects the service_mgr facts and it's fact deps. 'filter' collects the normal set of facts and then filters post collection.

(not to say the filter behavior isnt busted, just that some of the 2.4 features mean it's not required now)
Ah! The code is filtering the facts before adding it to the 'collected_facts' that is passed along. So with 'filter=ansible_service_mgr' the distro/platform facts were never getting added to collected_facts so the service_mgr collector would do the wrong thing. 

Looks like just changing the collected_facts to use the unfiltered facts fixes it.

```
[newswoop:F27:ansible (devel % u=)]$ ansible -i hosts_vagrant 192.168.121.125 -m setup -a 'filter=ansible_service_mgr'
192.168.121.125 | SUCCESS => {
    "ansible_facts": {
        "ansible_service_mgr": "upstart"
    }, 
    "changed": false
}

```


Resolving patch:
diff --git a/lib/ansible/module_utils/facts/ansible_collector.py b/lib/ansible/module_utils/facts/ansible_collector.py
--- a/lib/ansible/module_utils/facts/ansible_collector.py
+++ b/lib/ansible/module_utils/facts/ansible_collector.py
@@ -67,10 +67,6 @@ def collect(self, module=None, collected_facts=None):
         for collector_obj in self.collectors:
             info_dict = {}
 
-            # shallow copy of the accumulated collected facts to pass to each collector
-            # for reference.
-            collected_facts.update(facts_dict.copy())
-
             try:
 
                 # Note: this collects with namespaces, so collected_facts also includes namespaces
@@ -80,6 +76,10 @@ def collect(self, module=None, collected_facts=None):
                 sys.stderr.write(repr(e))
                 sys.stderr.write('\n')
 
+            # shallow copy of the new facts to pass to each collector in collected_facts so facts
+            # can reference other facts they depend on.
+            collected_facts.update(info_dict.copy())
+
             # NOTE: If we want complicated fact dict merging, this is where it would hook in
             facts_dict.update(self._filter(info_dict, self.filter_spec))
 


Test patch:
diff --git a/test/units/module_utils/facts/test_ansible_collector.py b/test/units/module_utils/facts/test_ansible_collector.py
--- a/test/units/module_utils/facts/test_ansible_collector.py
+++ b/test/units/module_utils/facts/test_ansible_collector.py
@@ -75,13 +75,16 @@
      FacterFactCollector]
 
 
-def mock_module(gather_subset=None):
+def mock_module(gather_subset=None,
+                filter=None):
     if gather_subset is None:
         gather_subset = ['all', '!facter', '!ohai']
+    if filter is None:
+        filter = '*'
     mock_module = Mock()
     mock_module.params = {'gather_subset': gather_subset,
                           'gather_timeout': 5,
-                          'filter': '*'}
+                          'filter': filter}
     mock_module.get_bin_path = Mock(return_value=None)
     return mock_module
 
@@ -258,6 +261,153 @@ def _assert_not_expected_facts(self, facts):
             self.assertNotIn(not_expected_fact, facts_keys)
 
 
+class ProvidesOtherFactCollector(collector.BaseFactCollector):
+    name = 'provides_something'
+    _fact_ids = set(['needed_fact'])
+
+    def collect(self, module=None, collected_facts=None):
+        return {'needed_fact': 'THE_NEEDED_FACT_VALUE'}
+
+
+class RequiresOtherFactCollector(collector.BaseFactCollector):
+    name = 'requires_something'
+
+    def collect(self, module=None, collected_facts=None):
+        collected_facts = collected_facts or {}
+        fact_dict = {}
+        fact_dict['needed_fact'] = collected_facts['needed_fact']
+        fact_dict['compound_fact'] = "compound-%s" % collected_facts['needed_fact']
+        return fact_dict
+
+
+class ConCatFactCollector(collector.BaseFactCollector):
+    name = 'concat_collected'
+
+    def collect(self, module=None, collected_facts=None):
+        collected_facts = collected_facts or {}
+        fact_dict = {}
+        con_cat_list = []
+        for key, value in collected_facts.items():
+            con_cat_list.append(value)
+
+        fact_dict['concat_fact'] = '-'.join(con_cat_list)
+        return fact_dict
+
+
+class TestCollectorDepsWithFilter(unittest.TestCase):
+    gather_subset = ['all', '!facter', '!ohai']
+
+    def _mock_module(self, gather_subset=None, filter=None):
+        return mock_module(gather_subset=self.gather_subset,
+                           filter=filter)
+
+    def setUp(self):
+        self.mock_module = self._mock_module()
+        self.collectors = self._collectors(mock_module)
+
+    def _collectors(self, module,
+                    all_collector_classes=None,
+                    minimal_gather_subset=None):
+        return [ProvidesOtherFactCollector(),
+                RequiresOtherFactCollector()]
+
+    def test_no_filter(self):
+        _mock_module = mock_module(gather_subset=['all', '!facter', '!ohai'])
+
+        facts_dict = self._collect(_mock_module)
+
+        expected = {'needed_fact': 'THE_NEEDED_FACT_VALUE',
+                    'compound_fact': 'compound-THE_NEEDED_FACT_VALUE'}
+
+        self.assertEquals(expected, facts_dict)
+
+    def test_with_filter_on_compound_fact(self):
+        _mock_module = mock_module(gather_subset=['all', '!facter', '!ohai'],
+                                   filter='compound_fact')
+
+        facts_dict = self._collect(_mock_module)
+
+        expected = {'compound_fact': 'compound-THE_NEEDED_FACT_VALUE'}
+
+        self.assertEquals(expected, facts_dict)
+
+    def test_with_filter_on_needed_fact(self):
+        _mock_module = mock_module(gather_subset=['all', '!facter', '!ohai'],
+                                   filter='needed_fact')
+
+        facts_dict = self._collect(_mock_module)
+
+        expected = {'needed_fact': 'THE_NEEDED_FACT_VALUE'}
+
+        self.assertEquals(expected, facts_dict)
+
+    def test_with_filter_on_compound_gather_compound(self):
+        _mock_module = mock_module(gather_subset=['!all', '!any', 'compound_fact'],
+                                   filter='compound_fact')
+
+        facts_dict = self._collect(_mock_module)
+
+        expected = {'compound_fact': 'compound-THE_NEEDED_FACT_VALUE'}
+
+        self.assertEquals(expected, facts_dict)
+
+    def test_with_filter_no_match(self):
+        _mock_module = mock_module(gather_subset=['all', '!facter', '!ohai'],
+                                   filter='ansible_this_doesnt_exist')
+
+        facts_dict = self._collect(_mock_module)
+
+        expected = {}
+        self.assertEquals(expected, facts_dict)
+
+    def test_concat_collector(self):
+        _mock_module = mock_module(gather_subset=['all', '!facter', '!ohai'])
+
+        _collectors = self._collectors(_mock_module)
+        _collectors.append(ConCatFactCollector())
+
+        fact_collector = \
+            ansible_collector.AnsibleFactCollector(collectors=_collectors,
+                                                   namespace=ns,
+                                                   filter_spec=_mock_module.params['filter'])
+
+        collected_facts = {}
+        facts_dict = fact_collector.collect(module=_mock_module,
+                                            collected_facts=collected_facts)
+        self.assertIn('concat_fact', facts_dict)
+        self.assertTrue('THE_NEEDED_FACT_VALUE' in facts_dict['concat_fact'])
+
+    def test_concat_collector_with_filter_on_concat(self):
+        _mock_module = mock_module(gather_subset=['all', '!facter', '!ohai'],
+                                   filter='concat_fact')
+
+        _collectors = self._collectors(_mock_module)
+        _collectors.append(ConCatFactCollector())
+
+        fact_collector = \
+            ansible_collector.AnsibleFactCollector(collectors=_collectors,
+                                                   namespace=ns,
+                                                   filter_spec=_mock_module.params['filter'])
+
+        collected_facts = {}
+        facts_dict = fact_collector.collect(module=_mock_module,
+                                            collected_facts=collected_facts)
+        self.assertIn('concat_fact', facts_dict)
+        self.assertTrue('THE_NEEDED_FACT_VALUE' in facts_dict['concat_fact'])
+        self.assertTrue('compound' in facts_dict['concat_fact'])
+
+    def _collect(self, _mock_module, collected_facts=None):
+        _collectors = self._collectors(_mock_module)
+
+        fact_collector = \
+            ansible_collector.AnsibleFactCollector(collectors=_collectors,
+                                                   namespace=ns,
+                                                   filter_spec=_mock_module.params['filter'])
+        facts_dict = fact_collector.collect(module=_mock_module,
+                                            collected_facts=collected_facts)
+        return facts_dict
+
+
 class ExceptionThrowingCollector(collector.BaseFactCollector):
     def collect(self, module=None, collected_facts=None):
         raise Exception('A collector failed')