Repository: ansible/ansible

Issue:
Callback plugins are called twice if custom plugin has the same name as standard one
##### ISSUE TYPE

 - Bug Report

##### COMPONENT NAME

callback plugins

##### ANSIBLE VERSION

```
ansible-3 2.4.1.0
  config file = /home/ericl/Documents/Technik/Ansible/ansible.cfg
  configured module search path = ['/home/ericl/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible-3
  python version = 3.6.3 (default, Oct  9 2017, 12:07:10) [GCC 7.2.1 20170915 (Red Hat 7.2.1-2)]
```

##### CONFIGURATION

DEFAULT_CALLBACK_WHITELIST(/home/ericl/Documents/Technik/Ansible/ansible.cfg) = ['junit']

##### OS / ENVIRONMENT

Fedora 27 (but I don't think that it's relevant)

##### SUMMARY

While developing an extension of the junit callback plug-in and testing it under Python 2 and 3, I noticed that under Python 3, there were 2 log files created while only one under Python 2, even though I hadn't changed anything to the actual process of writing the junit log file.

My interpretation is that it's also happening under Python 2 but because the time stamp is under Python 2 only the hundredth of second whereas Python 3 is much more precise, the file name remains the same and one overwrites the other, e.g.:

* Under Python 2: tryjunit-1512199631.75.xml
* Under Python 3: tryjunit-1512199663.8626602.xml and tryjunit-1512199663.864464.xml

##### STEPS TO REPRODUCE

```
mkdir callback_plugins
cp /usr/lib/python3.6/site-packages/ansible/plugins/callback/junit.py callback_plugins
echo -e '[defaults]\ncallback_whitelist = junit' > ansible.cfg
```
Playbook used doesn't matter...

##### EXPECTED RESULTS

I would expect that the custom junit callback plugin _replaces_ the standard one and there would be only one logfile created.

##### ACTUAL RESULTS

The junit XML log is created twice with slightly different times, telling me that the junit plug-in was called twice.


Discussion:
Files identified in the description:
* [lib/ansible/plugins/callback](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
I have confirmed that this is not specific to python3.  It happens on both python2 and python3.  In my tests I used the `profile_tasks` callback for verification.

Resolving patch:
diff --git a/lib/ansible/plugins/loader.py b/lib/ansible/plugins/loader.py
--- a/lib/ansible/plugins/loader.py
+++ b/lib/ansible/plugins/loader.py
@@ -424,7 +424,14 @@ def all(self, *args, **kwargs):
                 continue
 
             if path not in self._module_cache:
-                self._module_cache[path] = self._load_module_source(name, path)
+                module = self._load_module_source(name, path)
+                if module in self._module_cache.values():
+                    # In ``_load_module_source`` if a plugin has a duplicate name, we just return the
+                    # previously matched plugin from sys.modules, which means you are never getting both,
+                    # just one, but cached for both paths, this isn't normally a problem, except with callbacks
+                    # where it will run that single callback twice. This rejects duplicates.
+                    continue
+                self._module_cache[path] = module
                 found_in_cache = False
 
             try:


Test patch:
diff --git a/test/units/plugins/loader_fixtures/__init__.py b/test/units/plugins/loader_fixtures/__init__.py
new file mode 100644
diff --git a/test/units/plugins/loader_fixtures/import_fixture.py b/test/units/plugins/loader_fixtures/import_fixture.py
new file mode 100644
--- /dev/null
+++ b/test/units/plugins/loader_fixtures/import_fixture.py
@@ -0,0 +1,8 @@
+# Nothing to see here, this file is just empty to support a imp.load_source
+# without doing anything
+__metaclass__ = type
+
+
+class test:
+    def __init__(self, *args, **kwargs):
+        pass
diff --git a/test/units/plugins/test_plugins.py b/test/units/plugins/test_plugins.py
--- a/test/units/plugins/test_plugins.py
+++ b/test/units/plugins/test_plugins.py
@@ -91,3 +91,48 @@ def test_plugin__init_config_str(self):
 
     def test_plugin__init_config_none(self):
         self.assertPluginLoaderConfigBecomes(None, [])
+
+    def test__load_module_source_no_duplicate_names(self):
+        '''
+        This test simulates importing 2 plugins with the same name,
+        and validating that the import is shortcirtuited if a file with the same name
+        has already been imported
+        '''
+
+        fixture_path = os.path.join(os.path.dirname(__file__), 'loader_fixtures')
+
+        pl = PluginLoader('test', '', 'test', 'test_plugin')
+        one = pl._load_module_source('import_fixture', os.path.join(fixture_path, 'import_fixture.py'))
+        # This line wouldn't even succeed if we didn't short cirtuit on finding a duplicate name
+        two = pl._load_module_source('import_fixture', '/path/to/import_fixture.py')
+
+        self.assertEqual(one, two)
+
+    @patch('ansible.plugins.loader.glob')
+    @patch.object(PluginLoader, '_get_paths')
+    def test_all_no_duplicate_names(self, gp_mock, glob_mock):
+        '''
+        This test goes along with ``test__load_module_source_no_duplicate_names``
+        and ensures that we ignore duplicate imports on multiple paths
+        '''
+
+        fixture_path = os.path.join(os.path.dirname(__file__), 'loader_fixtures')
+
+        gp_mock.return_value = [
+            fixture_path,
+            '/path/to'
+        ]
+
+        glob_mock.glob.side_effect = [
+            [os.path.join(fixture_path, 'import_fixture.py')],
+            ['/path/to/import_fixture.py']
+        ]
+
+        pl = PluginLoader('test', '', 'test', 'test_plugin')
+        # Aside from needing ``list()`` so we can do a len, ``PluginLoader.all`` returns a generator
+        # so ``list()`` actually causes ``PluginLoader.all`` to run.
+        plugins = list(pl.all())
+        self.assertEqual(len(plugins), 1)
+
+        self.assertIn(os.path.join(fixture_path, 'import_fixture.py'), pl._module_cache)
+        self.assertNotIn('/path/to/import_fixture.py', pl._module_cache)