Repository: ansible/ansible-modules-extras

Issue:
lxd_container module does not work on remote host
##### ISSUE TYPE
- Bug Report
##### COMPONENT NAME

lxd_container
##### ANSIBLE VERSION

```
ansible 2.2.0 (devel f4237b2151) last updated 2016/08/16 23:49:16 (GMT +200)
  lib/ansible/modules/core: (detached HEAD 45c1ae0ac1) last updated 2016/08/16 23:49:20 (GMT +200)
  lib/ansible/modules/extras: (detached HEAD a6b34973a8) last updated 2016/08/16 23:49:20 (GMT +200)
  config file = 
  configured module search path = Default w/o overrides
```
##### CONFIGURATION

default
##### OS / ENVIRONMENT

Ubuntu 16.04 both local and remote
##### SUMMARY

The lxd_container module does not work on remote machine with ssh connection
##### STEPS TO REPRODUCE

`ansible-playbook -i inventory test.yml`

<!--- Paste example playbooks or commands between quotes below -->
###### inventory

```
remote ansible_host=10.1.2.3
```
###### test.yml

```
- hosts: remote

  tasks:
    - name: remove container test1
      lxd_container:
        name: "test1"
        state: absent
```

<!--- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS

Expected OK, since no container named `test1` is running on `remote`.
##### ACTUAL RESULTS

<!--- What actually happened? If possible run with high verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->

```
No config file found; using defaults
Loaded callback default of type stdout, v2.0

PLAYBOOK: test.yml *************************************************************
1 plays in test.yml

PLAY [remote] ******************************************************************

TASK [setup] *******************************************************************
Using module file /home/magne/src/ansible/lib/ansible/modules/core/system/setup.py
<10.1.2.3> ESTABLISH SSH CONNECTION FOR USER: None
<10.1.2.3> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/magne/.ansible/cp/ansible-ssh-%h-%p-%r 10.1.2.3 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1471384712.01-12990656835615 `" && echo ansible-tmp-1471384712.01-12990656835615="` echo $HOME/.ansible/tmp/ansible-tmp-1471384712.01-12990656835615 `" ) && sleep 0'"'"''
<10.1.2.3> PUT /tmp/magne/tmp7N7O84 TO /home/magne/.ansible/tmp/ansible-tmp-1471384712.01-12990656835615/setup.py
<10.1.2.3> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/magne/.ansible/cp/ansible-ssh-%h-%p-%r '[10.1.2.3]'
<10.1.2.3> ESTABLISH SSH CONNECTION FOR USER: None
<10.1.2.3> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/magne/.ansible/cp/ansible-ssh-%h-%p-%r 10.1.2.3 '/bin/sh -c '"'"'chmod u+x /home/magne/.ansible/tmp/ansible-tmp-1471384712.01-12990656835615/ /home/magne/.ansible/tmp/ansible-tmp-1471384712.01-12990656835615/setup.py && sleep 0'"'"''
<10.1.2.3> ESTABLISH SSH CONNECTION FOR USER: None
<10.1.2.3> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/magne/.ansible/cp/ansible-ssh-%h-%p-%r -tt 10.1.2.3 '/bin/sh -c '"'"'/usr/bin/python /home/magne/.ansible/tmp/ansible-tmp-1471384712.01-12990656835615/setup.py; rm -rf "/home/magne/.ansible/tmp/ansible-tmp-1471384712.01-12990656835615/" > /dev/null 2>&1 && sleep 0'"'"''
ok: [remote]

TASK [remove container test1] **************************************************
task path: /home/magne/development/ansible/t/test.yml:4
Using module file /home/magne/src/ansible/lib/ansible/modules/extras/cloud/lxd/lxd_container.py
<10.1.2.3> ESTABLISH SSH CONNECTION FOR USER: None
<10.1.2.3> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/magne/.ansible/cp/ansible-ssh-%h-%p-%r 10.1.2.3 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1471384716.3-242133616298960 `" && echo ansible-tmp-1471384716.3-242133616298960="` echo $HOME/.ansible/tmp/ansible-tmp-1471384716.3-242133616298960 `" ) && sleep 0'"'"''
<10.1.2.3> PUT /tmp/magne/tmpVJ25yL TO /home/magne/.ansible/tmp/ansible-tmp-1471384716.3-242133616298960/lxd_container.py
<10.1.2.3> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/magne/.ansible/cp/ansible-ssh-%h-%p-%r '[10.1.2.3]'
<10.1.2.3> ESTABLISH SSH CONNECTION FOR USER: None
<10.1.2.3> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/magne/.ansible/cp/ansible-ssh-%h-%p-%r 10.1.2.3 '/bin/sh -c '"'"'chmod u+x /home/magne/.ansible/tmp/ansible-tmp-1471384716.3-242133616298960/ /home/magne/.ansible/tmp/ansible-tmp-1471384716.3-242133616298960/lxd_container.py && sleep 0'"'"''
<10.1.2.3> ESTABLISH SSH CONNECTION FOR USER: None
<10.1.2.3> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/magne/.ansible/cp/ansible-ssh-%h-%p-%r -tt 10.1.2.3 '/bin/sh -c '"'"'/usr/bin/python /home/magne/.ansible/tmp/ansible-tmp-1471384716.3-242133616298960/lxd_container.py; rm -rf "/home/magne/.ansible/tmp/ansible-tmp-1471384716.3-242133616298960/" > /dev/null 2>&1 && sleep 0'"'"''
fatal: [remote]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_name": "lxd_container"
    }, 
    "module_stderr": "OpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g-fips  1 Mar 2016\r\ndebug1: Reading configuration data /home/magne/.ssh/config\r\ndebug1: /home/magne/.ssh/config line 4: Applying options for *\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 29081\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to 10.1.2.3 closed.\r\n", 
    "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible_Ey11Ac/ansible_module_lxd_container.py\", line 249, in <module>\r\n    from ansible.modules.extras.cloud.lxd import LXDClient, LXDClientException\r\nImportError: No module named modules.extras.cloud.lxd\r\n", 
    "msg": "MODULE FAILURE"
}
    to retry, use: --limit @test.retry

PLAY RECAP *********************************************************************
remote                     : ok=1    changed=0    unreachable=0    failed=1   
```



Discussion:
<!-- @magne: Thanks for filing this bug report! -->

@hnakamur ping, this issue is waiting for your response.
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)

<!--- boilerplate: issue_bug_new --->

@hnakamur your module incorrectly uses local `__init__.py`, this is not how we share code, this should be moved to module_utils in the main ansible  repo.


Resolving patch:
diff --git a/cloud/lxd/__init__.py b/cloud/lxd/__init__.py
--- a/cloud/lxd/__init__.py
+++ b/cloud/lxd/__init__.py
@@ -1,133 +0,0 @@
-#!/usr/bin/python
-# -*- coding: utf-8 -*-
-
-# (c) 2016, Hiroaki Nakamura <hnakamur@gmail.com>
-#
-# This file is part of Ansible
-#
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-
-try:
-    import json
-except ImportError:
-    import simplejson as json
-
-# httplib/http.client connection using unix domain socket
-import socket
-import ssl
-try:
-    from httplib import HTTPConnection, HTTPSConnection
-except ImportError:
-    # Python 3
-    from http.client import HTTPConnection, HTTPSConnection
-
-class UnixHTTPConnection(HTTPConnection):
-    def __init__(self, path):
-        HTTPConnection.__init__(self, 'localhost')
-        self.path = path
-
-    def connect(self):
-        sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
-        sock.connect(self.path)
-        self.sock = sock
-
-from ansible.module_utils.urls import generic_urlparse
-try:
-    from urlparse import urlparse
-except ImportError:
-    # Python 3
-    from url.parse import urlparse
-
-class LXDClientException(Exception):
-    def __init__(self, msg, **kwargs):
-        self.msg = msg
-        self.kwargs = kwargs
-
-class LXDClient(object):
-    def __init__(self, url, key_file=None, cert_file=None, debug=False):
-        """LXD Client.
-
-        :param url: The URL of the LXD server. (e.g. unix:/var/lib/lxd/unix.socket or https://127.0.0.1)
-        :type url: ``str``
-        :param key_file: The path of the client certificate key file.
-        :type key_file: ``str``
-        :param cert_file: The path of the client certificate file.
-        :type cert_file: ``str``
-        :param debug: The debug flag. The request and response are stored in logs when debug is true.
-        :type debug: ``bool``
-        """
-        self.url = url
-        self.debug = debug
-        self.logs = []
-        if url.startswith('https:'):
-            self.cert_file = cert_file
-            self.key_file = key_file
-            parts = generic_urlparse(urlparse(self.url))
-            ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
-            ctx.load_cert_chain(cert_file, keyfile=key_file)
-            self.connection = HTTPSConnection(parts.get('netloc'), context=ctx)
-        elif url.startswith('unix:'):
-            unix_socket_path = url[len('unix:'):]
-            self.connection = UnixHTTPConnection(unix_socket_path)
-        else:
-            raise LXDClientException('URL scheme must be unix: or https:')
-
-    def do(self, method, url, body_json=None, ok_error_codes=None, timeout=None):
-        resp_json = self._send_request(method, url, body_json=body_json, ok_error_codes=ok_error_codes, timeout=timeout)
-        if resp_json['type'] == 'async':
-            url = '{0}/wait'.format(resp_json['operation'])
-            resp_json = self._send_request('GET', url)
-            if resp_json['metadata']['status'] != 'Success':
-                self._raise_err_from_json(resp_json)
-        return resp_json
-
-    def authenticate(self, trust_password):
-        body_json = {'type': 'client', 'password': trust_password}
-        return self._send_request('POST', '/1.0/certificates', body_json=body_json)
-
-    def _send_request(self, method, url, body_json=None, ok_error_codes=None, timeout=None):
-        try:
-            body = json.dumps(body_json)
-            self.connection.request(method, url, body=body)
-            resp = self.connection.getresponse()
-            resp_json = json.loads(resp.read())
-            self.logs.append({
-                'type': 'sent request',
-                'request': {'method': method, 'url': url, 'json': body_json, 'timeout': timeout},
-                'response': {'json': resp_json}
-            })
-            resp_type = resp_json.get('type', None)
-            if resp_type == 'error':
-                if ok_error_codes is not None and resp_json['error_code'] in ok_error_codes:
-                    return resp_json
-                self._raise_err_from_json(resp_json)
-            return resp_json
-        except socket.error as e:
-            raise LXDClientException('cannot connect to the LXD server', err=e)
-
-    def _raise_err_from_json(self, resp_json):
-        err_params = {}
-        if self.debug:
-            err_params['logs'] = self.logs
-        raise LXDClientException(self._get_err_from_resp_json(resp_json), **err_params)
-
-    @staticmethod
-    def _get_err_from_resp_json(resp_json):
-        err = None
-        metadata = resp_json.get('metadata', None)
-        if metadata is not None:
-            err = metadata.get('err', None)
-        if err is None:
-            err = resp_json.get('error', None)
-        return err
diff --git a/cloud/lxd/lxd_container.py b/cloud/lxd/lxd_container.py
--- a/cloud/lxd/lxd_container.py
+++ b/cloud/lxd/lxd_container.py
@@ -246,7 +246,7 @@
 '''
 
 import os
-from ansible.modules.extras.cloud.lxd import LXDClient, LXDClientException
+from ansible.module_utils.lxd import LXDClient, LXDClientException
 
 # LXD_ANSIBLE_STATES is a map of states that contain values of methods used
 # when a particular state is evoked.
diff --git a/cloud/lxd/lxd_profile.py b/cloud/lxd/lxd_profile.py
--- a/cloud/lxd/lxd_profile.py
+++ b/cloud/lxd/lxd_profile.py
@@ -172,7 +172,7 @@
 '''
 
 import os
-from ansible.modules.extras.cloud.lxd import LXDClient, LXDClientException
+from ansible.module_utils.lxd import LXDClient, LXDClientException
 
 # PROFILE_STATES is a list for states supported
 PROFILES_STATES = [


Test patch: