Repository: ansible/ansible

Issue:
manageiq_provider fails while adding an OCP provider
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and devel branch are affected too.
Always add information AFTER of these html comments.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest -->
 - Bug Report


##### COMPONENT NAME
<!---
Insert below this comment the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path
-->
manageiq_provider

https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/remote_management/manageiq/manageiq_provider.py

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.5.0
  config file = None
  configured module search path = [u'/Users/victor/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.14 (default, Mar 22 2018, 15:04:47) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)]
```

##### CONFIGURATION
<!---
If using Ansible 2.4 or above, paste the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
Nothing changed
##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->
N/A
##### SUMMARY
<!--- Explain the problem briefly -->
While adding a provider with the type Openshift it fails with the message 
```
fatal: [localhost]: FAILED! => {"changed": false, "msg": "failed to create provider OpenShift: Api::BadRequestError: Could not create the new provider - OpenShift api_version cannot be modified"}
```

When you comment (or delete) the api_version from the code, everything works. "api_version" should only be used for OpenStack providers

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
Try to add a Openshift provider into your ManageIQ environment.
<!--- Paste example playbooks or commands between quotes below -->
```yaml
---
- hosts: localhost
  connection: local
  vars: 
    manageiq_user: admin
  vars_prompt:
    - name: "manageiq_url" 
    - name: "ocp_auth_key"
    - name: "manageiq_password"
  tasks:
  - name: Create a new provider in ManageIQ ('Hawkular' metrics)
    manageiq_provider:
      name: 'OpenShift'
      type: 'Openshift'
      provider:
        auth_key: "{{ ocp_auth_key }}"
        hostname: 'ose.example.com'
        port: 8443
        verify_ssl: False
      metrics:
        role: 'hawkular'
        hostname: 'hawkular-metrics.example.com'
        port: 443
        verify_ssl: False
      manageiq_connection:
        url: "{{ manageiq_url }}"
        username: "{{ manageiq_user }}"
        password: "{{ manageiq_password }}" 
        verify_ssl: False
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
Task OK

```
PLAY RECAP ********************************************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0
```
##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
task path: /Users/victor/summit-fy19/playbooks/createOCP.yml:11
Using module file /usr/local/lib/python2.7/site-packages/ansible/modules/remote_management/manageiq/manageiq_provider.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: victor
<127.0.0.1> EXEC /bin/sh -c 'echo ~ && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/victor/.ansible/tmp/ansible-tmp-1522927554.79-120799432528931 `" && echo ansible-tmp-1522927554.79-120799432528931="` echo /Users/victor/.ansible/tmp/ansible-tmp-1522927554.79-120799432528931 `" ) && sleep 0'
<127.0.0.1> PUT /Users/victor/.ansible/tmp/ansible-local-75002Ws618f/tmpPHFbIV TO /Users/victor/.ansible/tmp/ansible-tmp-1522927554.79-120799432528931/manageiq_provider.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/victor/.ansible/tmp/ansible-tmp-1522927554.79-120799432528931/ /Users/victor/.ansible/tmp/ansible-tmp-1522927554.79-120799432528931/manageiq_provider.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/local/opt/python@2/bin/python2.7 /Users/victor/.ansible/tmp/ansible-tmp-1522927554.79-120799432528931/manageiq_provider.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /Users/victor/.ansible/tmp/ansible-tmp-1522927554.79-120799432528931/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
  File "/var/folders/hv/zq6vmgxs7nn29_4ztjz2vggh0000gn/T/ansible_G6eG2I/ansible_module_manageiq_provider.py", line 826, in create_provider
    connection_configurations=endpoints,
  File "/usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/manageiq_client/api.py", line 119, in post
    return self._result_processor(data)
  File "/usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/manageiq_client/api.py", line 87, in _result_processor
    "{}: {}".format(result_json["error"]["klass"], result_json["error"]["message"]))

fatal: [localhost]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "alerts": null,
            "api_version": null,
            "azure_tenant_id": null,
            "host_default_vnc_port_end": null,
            "host_default_vnc_port_start": null,
            "manageiq_connection": {
                "ca_bundle_path": null,
                "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
                "token": null,
                "url": "https://1cfme46-summitrhpdsl1077l1-vwmoc4gz.srv.ravcloud.com",
                "username": "admin",
                "verify_ssl": false
            },
            "metrics": {
                "auth_key": null,
                "certificate_authority": null,
                "hostname": "hawkular-metrics.example.com",
                "password": null,
                "path": null,
                "port": 443,
                "project": null,
                "role": "hawkular",
                "security_protocol": null,
                "subscription": null,
                "uid_ems": null,
                "userid": null,
                "verify_ssl": false
            },
            "name": "OpenShift",
            "project": null,
            "provider": {
                "auth_key": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
                "certificate_authority": null,
                "hostname": "ose.example.com",
                "password": null,
                "path": null,
                "port": 8443,
                "project": null,
                "role": null,
                "security_protocol": null,
                "subscription": null,
                "uid_ems": null,
                "userid": null,
                "verify_ssl": false
            },
            "provider_region": null,
            "ssh_keypair": null,
            "state": "present",
            "subscription": null,
            "tenant_mapping_enabled": false,
            "type": "Openshift",
            "zone": "default"
        }
    },
    "msg": "failed to create provider OpenShift: Api::BadRequestError: Could not create the new provider - OpenShift api_version cannot be modified"
}
```



Discussion:
Files identified in the description:
* [lib/ansible/modules/remote_management/manageiq/manageiq_provider.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/remote_management/manageiq/manageiq_provider.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @abellotti @cben @dkorn @gtanzillo @yaacov @zgalor
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Thanks for detailed report!  I've seen this too recently.  It happens for creation because creation function doesnt drop nulls from whole payload.  I have a wip fix & refactoring somewhere, will finish early next week.

If you need an immediate workaround, commenting out api_version line 764 will let you create open shift provider (but break one of the other provider types)
Some background info:

We have a function "delete_nulls" [1] that should have prevented this.
In line 725 [2] we try to remove "api_version" but for some reason it remains :-(
The "delete_nulls" function does not work properly in this case.

[1] https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/remote_management/manageiq/manageiq_provider.py#L574
[2] https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/remote_management/manageiq/manageiq_provider.py#L724

Thanks @cben I had to also comment line 715

Resolving patch:
diff --git a/lib/ansible/modules/remote_management/manageiq/manageiq_provider.py b/lib/ansible/modules/remote_management/manageiq/manageiq_provider.py
--- a/lib/ansible/modules/remote_management/manageiq/manageiq_provider.py
+++ b/lib/ansible/modules/remote_management/manageiq/manageiq_provider.py
@@ -694,7 +694,7 @@ def delete_provider(self, provider):
     def edit_provider(self, provider, name, provider_type, endpoints, zone_id, provider_region,
                       host_default_vnc_port_start, host_default_vnc_port_end,
                       subscription, project, uid_ems, tenant_mapping_enabled, api_version):
-        """ Edit a user from manageiq.
+        """ Edit a provider from manageiq.
 
         Returns:
             a short message describing the operation executed.
@@ -737,33 +737,32 @@ def edit_provider(self, provider, name, provider_type, endpoints, zone_id, provi
     def create_provider(self, name, provider_type, endpoints, zone_id, provider_region,
                         host_default_vnc_port_start, host_default_vnc_port_end,
                         subscription, project, uid_ems, tenant_mapping_enabled, api_version):
-        """ Creates the user in manageiq.
+        """ Creates the provider in manageiq.
 
         Returns:
-            the created user id, name, created_on timestamp,
-            updated_on timestamp, userid and current_group_id.
+            a short message describing the operation executed.
         """
+        resource = dict(
+            name=name,
+            zone={'id': zone_id},
+            provider_region=provider_region,
+            host_default_vnc_port_start=host_default_vnc_port_start,
+            host_default_vnc_port_end=host_default_vnc_port_end,
+            subscription=subscription,
+            project=project,
+            uid_ems=uid_ems,
+            tenant_mapping_enabled=tenant_mapping_enabled,
+            api_version=api_version,
+            connection_configurations=endpoints,
+        )
+
         # clean nulls, we do not send nulls to the api
-        endpoints = delete_nulls(endpoints)
+        resource = delete_nulls(resource)
 
         # try to create a new provider
         try:
             url = '%s/providers' % (self.api_url)
-            result = self.client.post(
-                url,
-                name=name,
-                type=supported_providers()[provider_type]['class_name'],
-                zone={'id': zone_id},
-                provider_region=provider_region,
-                host_default_vnc_port_start=host_default_vnc_port_start,
-                host_default_vnc_port_end=host_default_vnc_port_end,
-                subscription=subscription,
-                project=project,
-                uid_ems=uid_ems,
-                tenant_mapping_enabled=tenant_mapping_enabled,
-                api_version=api_version,
-                connection_configurations=endpoints,
-            )
+            result = self.client.post(url, type=supported_providers()[provider_type]['class_name'], **resource)
         except Exception as e:
             self.module.fail_json(msg="failed to create provider %s: %s" % (name, str(e)))
 


Test patch: