Repository: ansible/ansible

Issue:
Cloudstack VPN modules raise KeyError: 'forceencap'
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->
cs_vpn_customer_gateway
cs_vpn_connection

##### ANSIBLE VERSION
<!--- Paste, BELOW THIS COMMENT, verbatim output from "ansible --version" between quotes below -->
```
ansible 2.5.2
  config file = /home/user/.ansible.cfg
  configured module search path = [u'/home/user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.15 (default, May  1 2018, 05:55:50) [GCC 7.3.0]
```

##### CONFIGURATION
<!--- If using Ansible 2.4 or above, paste, BELOW THIS COMMENT, the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).-->
```
RETRY_FILES_ENABLED(/home/user/.ansible.cfg) = False
ANSIBLE_PIPELINING(/home/user/.ansible.cfg) = True
DEFAULT_FORKS(/home/user/.ansible.cfg) = 2
DEFAULT_GATHERING(/home/user/.ansible.cfg) = smart
```

##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->
Control host: Debian buster (testing)
Target host: Not relevant, using local_action and cs_ modules only communicate via the CloudStack API (HTTP based) version 4.5

##### SUMMARY
<!--- Explain the problem briefly -->
When creating a VPN on CloudStack using the Ansible cs_ modules, both cs_vpn_customer_gateway and cs_vpn_connection raise a KeyError: 'force_encap'

##### STEPS TO REPRODUCE
<!--- For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used. -->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
    - name: connect VPC site VPN
      local_action:
        module: cs_vpn_connection
        vpc: "{{ cs_vpc }}"
        vpn_customer_gateway: "{{ cs_vpn }}"
        project: "{{ cs_project }}"
        domain: "{{ cs_domain }}"
        api_region: "{{ cs_api_region }}"
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
A VPN connection (or gateway) is created.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
TASK [connect VPC site VPN] *************************************************************************************************************************************************************************************************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'forceencap'
fatal: [test-01.vpc-01.local -> localhost]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_zwxu0i/ansible_module_cs_vpn_connection.py\", line 343, in <module>\n    main()\n  File \"/tmp/ansible_zwxu0i/ansible_module_cs_vpn_connection.py\", line 338, in main\n    result = acs_vpn_conn.get_result(vpn_conn)\n  File \"/tmp/ansible_zwxu0i/ansible_module_cs_vpn_connection.py\", line 299, in get_result\n    self.result['force_encap'] = True if vpn_conn['forceencap'] else False\nKeyError: 'forceencap'\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1}
```
Expanded exception:
```
Traceback (most recent call last):
  File "/tmp/ansible_zwxu0i/ansible_module_cs_vpn_connection.py", line 343, in <module>
    main()
  File "/tmp/ansible_zwxu0i/ansible_module_cs_vpn_connection.py", line 338, in main
    result = acs_vpn_conn.get_result(vpn_conn)
  File "/tmp/ansible_zwxu0i/ansible_module_cs_vpn_connection.py", line 299, in get_result
    self.result['force_encap'] = True if vpn_conn['forceencap'] else False
KeyError: 'forceencap'
```

##### ADDITIONAL INFORMATION
The exception can be worked around by adding a check around the use of the dictionary:
```
            self.result['force_encap'] = True if 'forceencap' in vpn_customer_gateway and vpn_conn['forceencap'] else False
```
instead of
```
            self.result['force_encap'] = True if vpn_conn['forceencap'] else False
```
But I'm not sure if this is the right approach.
It seems like the option is not passed along correctly.

Even when force_encap is set in the task, the exception still happens.
Setting forceencap doesn't work, Ansible complains that it's an invalid parameter.


Discussion:
Files identified in the description:
* [lib/ansible/modules/cloud/cloudstack/cs_vpn_connection.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/cloudstack/cs_vpn_connection.py)
* [lib/ansible/modules/cloud/cloudstack/cs_vpn_customer_gateway.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/cloudstack/cs_vpn_customer_gateway.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @dpassante @resmo
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Seems forceencap is not always returned by the api, this probably changed since cloudstack 4.5

~~~diff
-            self.result['force_encap'] = True if vpn_conn['forceencap'] else False
+            self.result['force_encap'] = True if vpn_conn.get('forceencap') else False
~~~

would do it.
Doesn't seem like it's working:
```
Traceback (most recent call last):
  File "/tmp/ansible_Qi5gLv/ansible_module_cs_vpn_customer_gateway.py", line 334, in <module>
    main()
  File "/tmp/ansible_Qi5gLv/ansible_module_cs_vpn_customer_gateway.py", line 329, in main
    result = acs_vpn_cgw.get_result(vpn_customer_gateway)
  File "/tmp/ansible_Qi5gLv/ansible_module_cs_vpn_customer_gateway.py", line 291, in get_result
    self.result['force_encap'] = True if vpn_conn.get('forceencap') else False
NameError: global name 'vpn_conn' is not defined
```

Resolving patch:
diff --git a/lib/ansible/modules/cloud/cloudstack/cs_vpn_connection.py b/lib/ansible/modules/cloud/cloudstack/cs_vpn_connection.py
--- a/lib/ansible/modules/cloud/cloudstack/cs_vpn_connection.py
+++ b/lib/ansible/modules/cloud/cloudstack/cs_vpn_connection.py
@@ -297,7 +297,7 @@ def get_result(self, vpn_conn):
             if 'cidrlist' in vpn_conn:
                 self.result['cidrs'] = vpn_conn['cidrlist'].split(',') or [vpn_conn['cidrlist']]
             # Ensure we return a bool
-            self.result['force_encap'] = True if vpn_conn['forceencap'] else False
+            self.result['force_encap'] = True if vpn_conn.get('forceencap') else False
             args = {
                 'key': 'name',
                 'identifier': vpn_conn['s2scustomergatewayid'],
diff --git a/lib/ansible/modules/cloud/cloudstack/cs_vpn_customer_gateway.py b/lib/ansible/modules/cloud/cloudstack/cs_vpn_customer_gateway.py
--- a/lib/ansible/modules/cloud/cloudstack/cs_vpn_customer_gateway.py
+++ b/lib/ansible/modules/cloud/cloudstack/cs_vpn_customer_gateway.py
@@ -289,7 +289,7 @@ def get_result(self, vpn_customer_gateway):
             if 'cidrlist' in vpn_customer_gateway:
                 self.result['cidrs'] = vpn_customer_gateway['cidrlist'].split(',') or [vpn_customer_gateway['cidrlist']]
             # Ensure we return a bool
-            self.result['force_encap'] = True if vpn_customer_gateway['forceencap'] else False
+            self.result['force_encap'] = True if vpn_customer_gateway.get('forceencap') else False
         return self.result
 
 


Test patch: