Repository: ansible/ansible

Issue:
CloudStack wants http:// prefix when adding host with cs_host module
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
lib/ansible/modules/cloud/cloudstack/cs_host.py
##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.4.0
  config file = /home/john/.ansible.cfg
  configured module search path = [u'/home/john/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible-2.4.0-py2.7.egg/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.13 (default, May 10 2017, 20:04:28) [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)]

```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->
This bug has been confirmed when testing against both CloudStack 4.9.2.0 and 4.10 (snapshot).
##### SUMMARY
<!--- Explain the problem briefly -->
When adding a host to Cloudstack the API insists that the hostname or IP address is prefixed with a URI scheme. Specifically it wants a prefix of "http://". This is only the case when adding the host, and this prefix is apparently never used again, and does not for part of this host name in the database/API.

Adding the http:// prefix to the hostname is a workaround that will allow you to successfully add a host, but you must subsequently remove it from your playbook to manage the host afterward.

I have a fix for this and will open a pull request shortly.
##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
---
- name: Setup Cloudstack
  hosts: ceph-the1
  tasks:
  # Ensure a host is present but disabled
  - local_action:
      module: cs_host
      name: server12.example.com
      cluster: Xeon
      pod: Telehouse
      zone: London
      hypervisor: KVM
      allocation_state: disabled
      password: SUPERsecret
      username: root
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
The request would pass API validation and be accepted.
##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
The API rejects the request due to a lack of a uri scheme prefix.
<!--- Paste verbatim command output between quotes below -->
```
$ ansible-playbook test_cs_host.yml

PLAY [Setup Cloudstack] ***********************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************
ok: [ceph-the1]

TASK [cs_host] ********************************************************************************************************************************
fatal: [ceph-the1 -> localhost]: FAILED! => {"changed": false, "failed": true, "msg": "CloudStackException: ('HTTP 431 response from CloudStack', <Response [431]>, {u'errorcode': 431, u'uuidList': [], u'cserrorcode': 4350, u'errortext': u'uri.scheme is null server12.example.com, add nfs:// (or cifs://) as a prefix'})"}
        to retry, use: --limit @/home/john/ansible/theceph/test_cs_host.retry

PLAY RECAP ************************************************************************************************************************************
ceph-the1                  : ok=1    changed=0    unreachable=0    failed=1   

```

Add http:// prefix when adding new host.
##### SUMMARY
<!--- Describe the change, including rationale and design decisions -->
Add http:// prefix to host/url when adding a new host.
Fixes #25641 
<!---
If you are fixing an existing issue, please include "Fixes #nnn" in your
commit message and your description; but you should still explain what
the change does.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bugfix Pull Request

##### COMPONENT NAME
<!--- Name of the module/plugin/module/task -->
lib/ansible/modules/cloud/cloudstack/cs_host.py
##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.4.0
  config file = /home/john/.ansible.cfg
  configured module search path = [u'/home/john/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible-2.4.0-py2.7.egg/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.13 (default, May 10 2017, 20:04:28) [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)]
```


##### ADDITIONAL INFORMATION
<!---
Include additional information to help people understand the change here.
For bugs that don't have a linked bug report, a step-by-step reproduction
of the problem is helpful.
  -->

<!--- Paste verbatim command output below, e.g. before and after your change -->
Before change:
```
$ ansible-playbook test_cs_host.yml

PLAY [Setup Cloudstack] ***********************************************************************************************************************

TASK [Gathering Facts] ************************************************************************************************************************
ok: [ceph-the1]

TASK [cs_host] ********************************************************************************************************************************
fatal: [ceph-the1 -> localhost]: FAILED! => {"changed": false, "failed": true, "msg": "CloudStackException: ('HTTP 431 response from CloudStack', <Response [431]>, {u'errorcode': 431, u'uuidList': [], u'cserrorcode': 4350, u'errortext': u'uri.scheme is null server12.example.com, add nfs:// (or cifs://) as a prefix'})"}
        to retry, use: --limit @/home/john/ansible/theceph/test_cs_host.retry

PLAY RECAP ************************************************************************************************************************************
ceph-the1                  : ok=1    changed=0    unreachable=0    failed=1   

```
After change:
```
$ ansible-playbook test_cs_host.yml

PLAY [Setup Cloudstack] ****************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************
ok: [ceph-the1]

TASK [cs_host] *************************************************************************************************************************************
changed: [ceph-the1 -> localhost]

PLAY RECAP *****************************************************************************************************************************************
ceph-the1                  : ok=2    changed=1    unreachable=0    failed=0   

```



Discussion:

cc @resmo
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
See comments in #25645

Resolving patch:
diff --git a/lib/ansible/modules/cloud/cloudstack/cs_host.py b/lib/ansible/modules/cloud/cloudstack/cs_host.py
--- a/lib/ansible/modules/cloud/cloudstack/cs_host.py
+++ b/lib/ansible/modules/cloud/cloudstack/cs_host.py
@@ -36,7 +36,14 @@
     description:
       - Name of the host.
     required: true
-    aliases: [ 'url', 'ip_address' ]
+    aliases: [ 'ip_address' ]
+  url:
+    description:
+      - Url of the host used to create a host.
+      - If not provided, C(http://) and param C(name) is used as url.
+      - Only considered if C(state=present) and host does not yet exist.
+    required: false
+    default: null
   username:
     description:
       - Username for the host.
@@ -301,6 +308,11 @@
   returned: success
   type: string
   sample: Enabled
+allocation_state::
+  description: Allocation state of the host.
+  returned: success
+  type: string
+  sample: enabled
 state:
   description: State of the host.
   returned: success
@@ -335,7 +347,14 @@
 '''
 
 from ansible.module_utils.basic import AnsibleModule
-from ansible.module_utils.cloudstack import AnsibleCloudStack, CloudStackException, cs_argument_spec, cs_required_together, CS_HYPERVISORS
+from ansible.module_utils.cloudstack import (
+    AnsibleCloudStack,
+    CloudStackException,
+    cs_argument_spec,
+    cs_required_together,
+    CS_HYPERVISORS
+)
+import time
 
 
 class AnsibleCloudStackHost(AnsibleCloudStack):
@@ -360,7 +379,6 @@ def __init__(self, module):
             'events': 'events',
             'hahost': 'ha_host',
             'hasenoughcapacity': 'has_enough_capacity',
-            'hosttags': 'host_tags',
             'hypervisor': 'hypervisor',
             'hypervisorversion': 'hypervisor_version',
             'ipaddress': 'ip_address',
@@ -381,12 +399,12 @@ def __init__(self, module):
             'type': 'host_type',
             'version': 'host_version',
             'gpugroup': 'gpu_group',
-
         }
         self.allocation_states = {
-            'enabled': 'enable',
-            'disabled': 'disable',
+            'enabled': 'Enable',
+            'disabled': 'Disable',
         }
+        self.host = None
 
     def get_pod(self, key=None):
         pod_name = self.module.params.get('pod')
@@ -426,8 +444,10 @@ def get_allocation_state(self):
             return None
         return self.allocation_states[allocation_state]
 
-    def get_host(self):
-        host = None
+    def get_host(self, refresh=False):
+        if self.host is not None and not refresh:
+            return self.host
+
         name = self.module.params.get('name')
         args = {
             'zoneid': self.get_zone(key='id'),
@@ -436,8 +456,8 @@ def get_host(self):
         if res:
             for h in res['host']:
                 if name in [h['ipaddress'], h['name']]:
-                    host = h
-        return host
+                    self.host = h
+        return self.host
 
     def present_host(self):
         host = self.get_host()
@@ -447,6 +467,13 @@ def present_host(self):
             host = self._update_host(host)
         return host
 
+    def _get_url(self):
+        url = self.module.params.get('url')
+        if url:
+            return url
+        else:
+            return "http://%s" % self.module.params.get('name')
+
     def _create_host(self, host):
         required_params = [
             'password',
@@ -458,7 +485,7 @@ def _create_host(self, host):
         self.result['changed'] = True
         args = {
             'hypervisor': self.module.params.get('hypervisor'),
-            'url': self.module.params.get('name'),
+            'url': self._get_url(),
             'username': self.module.params.get('username'),
             'password': self.module.params.get('password'),
             'podid': self.get_pod(key='id'),
@@ -471,24 +498,24 @@ def _create_host(self, host):
             host = self.cs.addHost(**args)
             if 'errortext' in host:
                 self.module.fail_json(msg="Failed: '%s'" % host['errortext'])
-            host = host['host']
+            host = host['host'][0]
         return host
 
     def _update_host(self, host):
         args = {
             'id': host['id'],
             'hosttags': self.get_host_tags(),
-            'allocationstate': self.module.params.get('allocation_state'),
+            'allocationstate': self.get_allocation_state()
         }
-        host['allocationstate'] = host['resourcestate'].lower()
+        host['allocationstate'] = self.allocation_states[host['resourcestate'].lower()]
         if self.has_changed(args, host):
-            args['allocationstate'] = self.get_allocation_state()
             self.result['changed'] = True
             if not self.module.check_mode:
                 host = self.cs.updateHost(**args)
                 if 'errortext' in host:
                     self.module.fail_json(msg="Failed: '%s'" % host['errortext'])
                 host = host['host']
+
         return host
 
     def absent_host(self):
@@ -499,16 +526,51 @@ def absent_host(self):
                 'id': host['id'],
             }
             if not self.module.check_mode:
-                res = self.cs.deleteHost(**args)
+                res = self.enable_maintenance()
+                if res:
+                    res = self.cs.deleteHost(**args)
+                    if 'errortext' in res:
+                        self.module.fail_json(msg="Failed: '%s'" % res['errortext'])
+        return host
+
+    def enable_maintenance(self):
+        host = self.get_host()
+        if host['resourcestate'] not in ['PrepareForMaintenance', 'Maintenance']:
+            self.result['changed'] = True
+            args = {
+                'id': host['id'],
+            }
+            if not self.module.check_mode:
+                res = self.cs.prepareHostForMaintenance(**args)
                 if 'errortext' in res:
                     self.module.fail_json(msg="Failed: '%s'" % res['errortext'])
+                host = self.poll_job(res, 'host')
+                self._poll_for_maintenance()
         return host
 
+    def _poll_for_maintenance(self):
+        for i in range(0, 300):
+            time.sleep(2)
+            host = self.get_host(refresh=True)
+            if not host:
+                return None
+            elif host['resourcestate'] != 'PrepareForMaintenance':
+                return host
+        self.fail_json("Polling for maintenance timed out")
+
+    def get_result(self, host):
+        super(AnsibleCloudStackHost, self).get_result(host)
+        if host:
+            self.result['allocation_state'] = host['resourcestate'].lower()
+            self.result['host_tags'] = host['hosttags'].split(',') if host.get('hosttags') else []
+        return self.result
+
 
 def main():
     argument_spec = cs_argument_spec()
     argument_spec.update(dict(
-        name=dict(required=True, aliases=['url', 'ip_address']),
+        name=dict(required=True, aliases=['ip_address']),
+        url=dict(),
         password=dict(default=None, no_log=True),
         username=dict(default=None),
         hypervisor=dict(choices=CS_HYPERVISORS, default=None),


Test patch:
diff --git a/test/integration/targets/cs_host/aliases b/test/integration/targets/cs_host/aliases
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/cs_host/aliases
@@ -0,0 +1,2 @@
+cloud/cs
+posix/ci/cloud/cs
diff --git a/test/integration/targets/cs_host/meta/main.yml b/test/integration/targets/cs_host/meta/main.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/cs_host/meta/main.yml
@@ -0,0 +1,3 @@
+---
+dependencies:
+  - cs_common
diff --git a/test/integration/targets/cs_host/tasks/main.yml b/test/integration/targets/cs_host/tasks/main.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/cs_host/tasks/main.yml
@@ -0,0 +1,296 @@
+---
+- name: test fail missing params
+  cs_host:
+  register: host
+  ignore_errors: true
+- name: verify test fail missing url if host is not existent
+  assert:
+    that:
+      - host|failed
+      - 'host.msg == "missing required arguments: name"'
+
+- name: test fail missing params if host is not existent
+  cs_host:
+    name: sim
+  register: host
+  ignore_errors: true
+- name: verify test fail missing params if host is not existent
+  assert:
+    that:
+      - host|failed
+      - 'host.msg == "missing required arguments: password,username,hypervisor,pod"'
+
+- name: test create a host in check mode
+  cs_host:
+    name: sim
+    url: "http://sim/c0-basic/h2"
+    cluster: C0-basic
+    pod: POD0-basic
+    username: root
+    password: password
+    hypervisor: Simulator
+    allocation_state: enabled
+    host_tags:
+    - perf
+    - gpu
+  register: host
+  check_mode: true
+- name: verify test create a host in check mode
+  assert:
+    that:
+      - host|changed
+
+- name: test create a host
+  cs_host:
+    name: sim
+    url: "http://sim/c0-basic/h2"
+    cluster: C0-basic
+    pod: POD0-basic
+    username: root
+    password: password
+    hypervisor: Simulator
+    allocation_state: enabled
+    host_tags:
+    - perf
+    - gpu
+  register: host
+- name: verify test create a host
+  assert:
+    that:
+      - host|changed
+      - host.cluster == 'C0-basic'
+      - host.pod == 'POD0-basic'
+      - host.hypervisor == 'Simulator'
+      - host.allocation_state == 'enabled'
+      - host.zone == 'Sandbox-simulator-basic'
+      - host.state == 'Up'
+      - "host.name.startswith('SimulatedAgent.')"
+      - host.host_tags == ['perf', 'gpu']
+
+# This is special in simulator mode, we can not predict the full hostname.
+# That is why we gather the infos from the returns and use a fact.
+- name: assume the sim would resolve to the IP address
+  set_fact:
+    host_hostname: "{{ host.name }}"
+    host_ip_address: "{{ host.ip_address }}"
+
+- name: test create a host idempotence
+  cs_host:
+    name: "{{ host_hostname }}"
+    url: "http://sim/c0-basic/h2"
+    cluster: C0-basic
+    pod: POD0-basic
+    username: admin
+    password: password
+    hypervisor: Simulator
+    allocation_state: enabled
+    host_tags:
+    - perf
+    - gpu
+  register: host
+- name: verify test create a host idempotence
+  assert:
+    that:
+      - not host|changed
+      - host.cluster == 'C0-basic'
+      - host.pod == 'POD0-basic'
+      - host.hypervisor == 'Simulator'
+      - host.allocation_state == 'enabled'
+      - host.zone == 'Sandbox-simulator-basic'
+      - host.state == 'Up'
+      - host.name == '{{ host_hostname }}'
+      - host.host_tags == ['perf', 'gpu']
+
+- name: test update host in check mode
+  cs_host:
+    name: "{{ host_hostname }}"
+    url: "http://sim/c0-basic/h2"
+    cluster: C0-basic
+    pod: POD0-basic
+    username: admin
+    password: password
+    hypervisor: Simulator
+    allocation_state: enabled
+    host_tags:
+    - perf
+    - gpu
+    - x2
+  register: host
+  check_mode: true
+- name: verify test update a host in check mode
+  assert:
+    that:
+      - host|changed
+      - host.cluster == 'C0-basic'
+      - host.pod == 'POD0-basic'
+      - host.hypervisor == 'Simulator'
+      - host.allocation_state == 'enabled'
+      - host.zone == 'Sandbox-simulator-basic'
+      - host.state == 'Up'
+      - host.name == '{{ host_hostname }}'
+      - host.host_tags == ['perf', 'gpu']
+
+- name: test update host
+  cs_host:
+    name: "{{ host_hostname }}"
+    url: "http://sim/c0-basic/h2"
+    cluster: C0-basic
+    pod: POD0-basic
+    username: admin
+    password: password
+    hypervisor: Simulator
+    allocation_state: enabled
+    host_tags:
+    - perf
+    - gpu
+    - x2
+  register: host
+- name: verify test update a host in check mode
+  assert:
+    that:
+      - host|changed
+      - host.cluster == 'C0-basic'
+      - host.pod == 'POD0-basic'
+      - host.hypervisor == 'Simulator'
+      - host.allocation_state == 'enabled'
+      - host.zone == 'Sandbox-simulator-basic'
+      - host.state == 'Up'
+      - host.name == '{{ host_hostname }}'
+      - host.host_tags == ['perf', 'gpu', 'x2']
+
+- name: test update host idempotence
+  cs_host:
+    name: "{{ host_hostname }}"
+    url: "http://sim/c0-basic/h2"
+    cluster: C0-basic
+    pod: POD0-basic
+    username: admin
+    password: password
+    hypervisor: Simulator
+    allocation_state: enabled
+    host_tags:
+    - perf
+    - gpu
+    - x2
+  register: host
+- name: verify test update a host idempotence
+  assert:
+    that:
+      - not host|changed
+      - host.cluster == 'C0-basic'
+      - host.pod == 'POD0-basic'
+      - host.hypervisor == 'Simulator'
+      - host.allocation_state == 'enabled'
+      - host.zone == 'Sandbox-simulator-basic'
+      - host.state == 'Up'
+      - host.name == '{{ host_hostname }}'
+      - host.host_tags == ['perf', 'gpu', 'x2']
+
+# FIXME: Removing by empty list seems to be an issue in the used lib cs underneath, disabled
+- name: test update host remove host_tags
+  cs_host:
+    name: "{{ host_hostname }}"
+    url: "http://sim/c0-basic/h2"
+    cluster: C0-basic
+    pod: POD0-basic
+    username: admin
+    password: password
+    hypervisor: Simulator
+    allocation_state: enabled
+    host_tags: []
+  register: host
+  when: false
+- name: verify test update host remove host_tags
+  assert:
+    that:
+      - host|changed
+      - host.host_tags|length == 0
+      - host.cluster == 'C0-basic'
+      - host.pod == 'POD0-basic'
+      - host.hypervisor == 'Simulator'
+      - host.allocation_state == 'enabled'
+      - host.zone == 'Sandbox-simulator-basic'
+      - host.state == 'Up'
+      - host.name == '{{ host_hostname }}'
+  when: false
+
+# FIXME: Removing by empty list seems to be an issue in the used lib cs underneath, disabled
+- name: test update host remove host_tags idempotence
+  cs_host:
+    name: "{{ host_hostname }}"
+    url: "http://sim/c0-basic/h2"
+    cluster: C0-basic
+    pod: POD0-basic
+    username: admin
+    password: password
+    hypervisor: Simulator
+    allocation_state: enabled
+    host_tags: []
+  register: host
+  when: false
+- name: verify test update host remove host_tags idempotence
+  assert:
+    that:
+      - not host|changed
+      - len(host.host_tags) == 0
+      - host.cluster == 'C0-basic'
+      - host.pod == 'POD0-basic'
+      - host.hypervisor == 'Simulator'
+      - host.allocation_state == 'enabled'
+      - host.zone == 'Sandbox-simulator-basic'
+      - host.state == 'Up'
+      - host.name == '{{ host_hostname }}'
+  when: false
+
+- name: test remove host in check mode
+  cs_host:
+    name: "{{ host_hostname }}"
+    cluster: C0-basic
+    pod: POD0-basic
+    state: absent
+  check_mode: true
+  register: host
+- name: verify test remove a host in check mode
+  assert:
+    that:
+      - host|changed
+      - host.cluster == 'C0-basic'
+      - host.pod == 'POD0-basic'
+      - host.hypervisor == 'Simulator'
+      - host.allocation_state == 'enabled'
+      - host.zone == 'Sandbox-simulator-basic'
+      - host.state == 'Up'
+      - host.name == '{{ host_hostname }}'
+      - host.host_tags == ['perf', 'gpu', 'x2']
+
+- name: test remove host
+  cs_host:
+    name: "{{ host_hostname }}"
+    cluster: C0-basic
+    pod: POD0-basic
+    state: absent
+  register: host
+- name: verify test remove a host
+  assert:
+    that:
+      - host|changed
+      - host.cluster == 'C0-basic'
+      - host.pod == 'POD0-basic'
+      - host.hypervisor == 'Simulator'
+      - host.allocation_state == 'enabled'
+      - host.zone == 'Sandbox-simulator-basic'
+      - host.state == 'Up'
+      - host.name == '{{ host_hostname }}'
+      - host.host_tags == ['perf', 'gpu', 'x2']
+
+- name: test remove host idempotence
+  cs_host:
+    name: "{{ host_hostname }}"
+    cluster: C0-basic
+    pod: POD0-basic
+    state: absent
+  register: host
+- name: verify test remove a host idempotenc
+  assert:
+    that:
+      - not host|changed