Repository: ansible/ansible

Issue:
Error with pip-docker (3.0.1): "Error connecting: load_config() got an unexpected keyword argument 'config_dict'" (Moved from AWX to Ansible)
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and devel branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest -->
 - Bug Report

##### COMPONENT NAME
lib/ansible/module_utils/docker_common.py

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
AWX version: master
AWX install method: docker on linux
Ansible version: 2.4.3.0-1
```

##### CONFIGURATION
<!---
If using Ansible 2.4 or above, paste the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
The output of `ansible-config dump --only-changed` was `VOID`

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->
Both `Arch Linux` and `Ubuntu` tried

##### SUMMARY
<!--- Explain the problem briefly -->
`AWX` fails to execute its demo example. It was working fine before updating `pip-docker` from `3.0.0` to `3.0.1`

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
1. cd awx/installer
2. Edit inventory to add the ssh credentials of the target machine (Ubuntu)
3. ansible-playbook -i inventory install.yml


<!--- Paste example playbooks or commands between quotes below -->
The error first appear in the task is `Activate postgres container`


<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
Successful installation

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
Error: `"Error connecting: load_config() got an unexpected keyword argument 'config_dict'"`

After downgrading `pip-docker` from `3.0.1` to `3.0.0`, the error disappeared.

I reported this first to `ansible/awx`, however it was suggested there to move the issue to `ansible/ansible`: https://github.com/ansible/awx/issues/1197


Discussion:
Files identified in the description:
None

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
Should This Be Filed Elsewhere?
===============================

Hi!

Thanks very much for your interest in Ansible.  It sincerely means a lot to us. 

This appears to be something that should be filed against another project or bug tracker:

   * https://github.com/ansible/awx

If you can stop by the tracker or forum for one of those projects, we'd appreciate it.  

Should you still wish to discuss things further, or if you disagree with our thought process, please stop by one of our two mailing lists:

  * https://groups.google.com/forum/#!forum/ansible-project
  * https://groups.google.com/forum/#!forum/ansible-devel

We'd be happy to discuss things.

Thank you once again!


@sivel Thanks. Please note that this was first reported to `ansible/awx` (https://github.com/ansible/awx/issues/1197), and they asked me to report it here because the error comes from the `docker` module not their scripts. 
@sivel 

This should be filed here. We had similar issue reported in Galaxy: https://github.com/ansible/galaxy/issues/412

It appears there's a breaking change introduced by docker-py 3.0.1.
Files identified in the description:
* [lib/ansible/module_utils/docker_common.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/docker_common.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
@chouseknecht One thing I notice in your linked galaxy issue is that both `docker` and `docker-py` are both installed at the same time which is not functional.  They both occupy the same namespace and depending on install order overwrite each others files.

Only one of `docker` or `docker-py` can be installed at a time to have a functional library.

Uninstalling both, and re-installing `docker` should work in 2.5 at this point with work that was done in https://github.com/ansible/ansible/pull/36973
Another person had this problem in #38840, again, by having both (docker-py) and (docker) installed at the same time. If Ansible can detect the presence of 'both at once' and it's possible to do so, some sort of error should probably be thrown rather than give this, an indirect error that doesn't explain the actual problem.
Additionally, if the documentation can be clearer about this it'd be good. I faced issue #38840 while trying to use 'auto_remove' but that was not happening before. The docs mention that "docker-py" is needed, not "docker", and that's confusing. I was getting different errors by installing different versions of those packages simultaneously.
It comes from the fact that the name of the Docker SDK for Python changed its name from 'docker-py' to 'docker' at version 2.0.0, and the presence of 'both at once' isn't accounted for. Ansible errors mention 'docker-py' because Docker SDK for Python by that name is a least-common-denominator that works with versions of Docker Engine/ee/ce/toolbox as old as [1.8.0,](https://github.com/ansible/ansible/pull/37513) while Docker SDK for Python >= 2.0.0 ('docker') supports only Docker Engine/ee/ce/toolbox 1.9.0+.

I'd call it an imperfect solution for a difficult versioning problem. But if it's possible to detect the presence of both libraries installed at the same time, that would be good.
I see no way to reliably detect both at the same time.  It would require a bunch of assumptions that could be false in either situation depending on installation order.
Actually, I may be wrong.  I just took an in depth look to see if there are files/dirs that exist only in 1 version or the other.  This may work:

```
try:
    # docker
    import docker.models
    HAS_DOCKER_MODELS = True
except ImportError:
    HAS_DOCKER_MODELS = False

try:
    # docker-py
    import docker.ssladapter
    HAS_DOCKER_SSLADAPTER = True
except ImportError:
    HAS_DOCKER_SSLADAPTER = False

if HAS_DOCKER_MODELS and HAS_DOCKER_SSLADAPTER:
    raise Exception('Cannot have both docker-py and docker installed together')
```

@skylerbunny do you want to play with that and see if it works?  We of course wouldn't raise an exception, we would instead want to utilize `module.fail_json`.
Checking as @sivel suggested does throw an exception when both are installed. That said, I carefully looked at what pip does in the situation where both are installed, and it's a mess because I believe it'll overwrite some files based on which one is installed last - ones common to both.

In any case, uninstalling 'one or the other' when both are installed removes files common to both while only leaving the files unique to the other.

This means that if and when we throw an error, this should be the resolution:
'Cannot have both the docker-py and docker libraries installed together. Uninstall BOTH completely, then install EITHER docker-py or docker.'
Note on above: Fortunately, the exception is thrown regardless of which package is installed first, so either case is caught by it.
I do agree the message needs special attention. In order to fix it, you must uninstall both and install the appropriate one.
I went ahead and submitted a PR for this at https://github.com/ansible/ansible/pull/38884

Resolving patch:
diff --git a/lib/ansible/module_utils/docker_common.py b/lib/ansible/module_utils/docker_common.py
--- a/lib/ansible/module_utils/docker_common.py
+++ b/lib/ansible/module_utils/docker_common.py
@@ -56,6 +56,25 @@
     HAS_DOCKER_ERROR = str(exc)
     HAS_DOCKER_PY = False
 
+
+# The next 2 imports ``docker.models`` and ``docker.ssladapter`` are used
+# to ensure the user does not have both ``docker`` and ``docker-py`` modules
+# installed, as they utilize the same namespace are are incompatible
+try:
+    # docker
+    import docker.models
+    HAS_DOCKER_MODELS = True
+except ImportError:
+    HAS_DOCKER_MODELS = False
+
+try:
+    # docker-py
+    import docker.ssladapter
+    HAS_DOCKER_SSLADAPTER = True
+except ImportError:
+    HAS_DOCKER_SSLADAPTER = False
+
+
 DEFAULT_DOCKER_HOST = 'unix://var/run/docker.sock'
 DEFAULT_TLS = False
 DEFAULT_TLS_VERIFY = False
@@ -144,6 +163,10 @@ def __init__(self, argument_spec=None, supports_check_mode=False, mutually_exclu
             required_together=required_together_params,
             required_if=required_if)
 
+        if HAS_DOCKER_MODELS and HAS_DOCKER_SSLADAPTER:
+            self.fail("Cannot have both the docker-py and docker python modules installed together as they use the same namespace and "
+                      "cause a corrupt installation. Please uninstall both packages, and re-install only the docker-py or docker python module")
+
         if not HAS_DOCKER_PY:
             self.fail("Failed to import docker-py - %s. Try `pip install docker-py`" % HAS_DOCKER_ERROR)
 


Test patch: