Repository: ansible/ansible

Issue:
Subspec options validation fails on non-subspec required_on_of 
With commit 97a34cf008501088ae012f24e4712e2bc6dd6d4c, a bug was introduced where a `required_on_of` is also being checked on a subspec

An example spec is that of the `nuage_vspk` module: https://github.com/ansible/ansible/blob/96784160b1d9b2233107a9135bcaab1c134330cd/lib/ansible/modules/network/nuage/nuage_vspk.py#L985
This has a subspec, and a `required_on_of` configuration. 

When running a simple task (details below), the output is:
```
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "one of the following is required: command,state found in auth"}
```

Code: https://github.com/ansible/ansible/blob/96784160b1d9b2233107a9135bcaab1c134330cd/lib/ansible/module_utils/basic.py#L1657

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
basic.py

##### ANSIBLE VERSION
```
ansible 2.4.0 (devel 96784160b1) last updated 2017/08/03 21:06:16 (GMT +200)
  config file = None
  configured module search path = [u'/home/pdellaer/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/pdellaer/GitHub/ansible/ansible/lib/ansible
  executable location = /home/pdellaer/GitHub/ansible/ansible/bin/ansible
  python version = 2.7.5 (default, Nov  6 2016, 00:28:07) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]
```

##### CONFIGURATION
N/A

##### OS / ENVIRONMENT
N/A

##### SUMMARY
With commit 97a34cf008501088ae012f24e4712e2bc6dd6d4c, a bug was introduced where a `required_on_of` is also being checked on a subspec

##### STEPS TO REPRODUCE
Run the following playbook (it doesn't matter you don't have a Nuage environment as this is pre-execution):
```yaml
- hosts: localhost

#
# Configure to match your environment
#
  vars:
    nuage_auth:
      api_username: csproot
      api_password: csproot
      api_enterprise: csp
      api_url: https://localhost:8443
      api_version: v5_0
    enterprise_name: Ansible-Test

  tasks:
  # Creating a new Enterprise, this should create a new Enterprise
  # Unless an Enterprise with the name already exists
  - name: Create Enterprise
    connection: local
    nuage_vspk:
      auth: "{{ nuage_auth }}"
      type: Enterprise
      state: present
      properties:
        name: "{{ enterprise_name }}"
        description: "Created by Ansible"
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
```
TASK [Create Enterprise] **************************************************
changed: [localhost]
```

##### ACTUAL RESULTS
```
TASK [Create Enterprise] **************************************************
task path: /home/pdellaer/Tools/advanced-domain-environment.yml:67
Using module_utils file /home/pdellaer/GitHub/ansible/ansible/lib/ansible/module_utils/basic.py
Using module_utils file /home/pdellaer/GitHub/ansible/ansible/lib/ansible/module_utils/_text.py
Using module_utils file /home/pdellaer/GitHub/ansible/ansible/lib/ansible/module_utils/parsing/convert_bool.py
Using module_utils file /home/pdellaer/GitHub/ansible/ansible/lib/ansible/module_utils/parsing/__init__.py
Using module_utils file /home/pdellaer/GitHub/ansible/ansible/lib/ansible/module_utils/pycompat24.py
Using module_utils file /home/pdellaer/GitHub/ansible/ansible/lib/ansible/module_utils/six/__init__.py
Using module file /home/pdellaer/GitHub/ansible/ansible/lib/ansible/modules/network/nuage/nuage_vspk.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: pdellaer
<127.0.0.1> EXEC /bin/sh -c 'echo ~ && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/pdellaer/.ansible/tmp/ansible-tmp-1501788842.66-83231368444955 `" && echo ansible-tmp-1501788842.66-83231368444955="` echo /home/pdellaer/.ansible/tmp/ansible-tmp-1501788842.66-83231368444955 `" ) && sleep 0'
<127.0.0.1> PUT /tmp/tmpCz37Ad TO /home/pdellaer/.ansible/tmp/ansible-tmp-1501788842.66-83231368444955/nuage_vspk.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/pdellaer/.ansible/tmp/ansible-tmp-1501788842.66-83231368444955/ /home/pdellaer/.ansible/tmp/ansible-tmp-1501788842.66-83231368444955/nuage_vspk.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/home/pdellaer/VirtualEnvs/ansible/bin/python /home/pdellaer/.ansible/tmp/ansible-tmp-1501788842.66-83231368444955/nuage_vspk.py; rm -rf "/home/pdellaer/.ansible/tmp/ansible-tmp-1501788842.66-83231368444955/" > /dev/null 2>&1 && sleep 0'
fatal: [localhost]: FAILED! => {
    "changed": false,
    "failed": true,
    "invocation": {
        "module_args": {
            "auth": {
                "api_enterprise": "csp",
                "api_password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
                "api_url": "https://vsd01.phd.plm.nuagedemo.net:8443",
                "api_username": "tester",
                "api_version": "v5_0"
            },
            "children": null,
            "command": null,
            "id": null,
            "match_filter": null,
            "parent_id": null,
            "parent_type": null,
            "properties": {
                "description": "Created by Ansible",
                "name": "Ansible-Test"
            },
            "state": "present",
            "type": "Enterprise"
        }
    },
    "msg": "one of the following is required: command,state found in auth"
}
        to retry, use: --limit @/home/pdellaer/Tools/advanced-domain-environment.retry

PLAY RECAP  **************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1
```



Discussion:
@gundalow @bcoca 
This seems to also go wider than just `required_on_of` if i understand the code correctly... It looks like it is checking the global `self.required_on_of` against a subspec, i'd suggest allowing the use of `required_on_of` and the other features in a subspec and doing a check to see if they are defined in the subspec, and use that in the check when verifying a subspec.
@ganeshrn @nitzmahone FYI 

Resolving patch:
diff --git a/lib/ansible/module_utils/basic.py b/lib/ansible/module_utils/basic.py
--- a/lib/ansible/module_utils/basic.py
+++ b/lib/ansible/module_utils/basic.py
@@ -1951,7 +1951,7 @@ def _handle_options(self, argument_spec=None, params=None):
 
                     # check exclusive early
                     if not self.bypass_checks:
-                        self._check_mutually_exclusive(self.mutually_exclusive, param)
+                        self._check_mutually_exclusive(v.get('mutually_exclusive', None), param)
 
                     self._set_defaults(pre=True, spec=spec, param=param)
 
@@ -1960,9 +1960,9 @@ def _handle_options(self, argument_spec=None, params=None):
                         self._check_argument_types(spec, param)
                         self._check_argument_values(spec, param)
 
-                        self._check_required_together(self.required_together, param)
-                        self._check_required_one_of(self.required_one_of, param)
-                        self._check_required_if(self.required_if, param)
+                        self._check_required_together(v.get('required_together', None), param)
+                        self._check_required_one_of(v.get('required_one_of', None), param)
+                        self._check_required_if(v.get('required_if', None), param)
 
                     self._set_defaults(pre=False, spec=spec, param=param)
 


Test patch:
diff --git a/test/units/module_utils/test_basic.py b/test/units/module_utils/test_basic.py
--- a/test/units/module_utils/test_basic.py
+++ b/test/units/module_utils/test_basic.py
@@ -350,20 +350,36 @@ def test_module_utils_basic_ansible_module_with_options_creation(self):
             bar=dict(),
             bam=dict(),
             baz=dict(),
-            bam1=dict(default='test')
+            bam1=dict(),
+            bam2=dict(default='test')
+        )
+        arg_spec = dict(
+            foobar=dict(
+                type='list',
+                elements='dict',
+                options=options_spec,
+                mutually_exclusive=[
+                    ['bam', 'bam1']
+                ],
+                required_if=[
+                    ['foo', 'hello', ['bam']],
+                    ['foo', 'bam2', ['bam2']]
+                ],
+                required_one_of=[
+                    ['bar', 'bam']
+                ],
+                required_together=[
+                    ['bam1', 'baz']
+                ]
+            )
         )
-        arg_spec = dict(foobar=dict(type='list', elements='dict', options=options_spec))
-        mut_ex = (('bar', 'bam'),)
-        req_to = (('bam', 'baz'),)
 
-        # should test ok
-        args = json.dumps(dict(ANSIBLE_MODULE_ARGS={'foobar': [{"foo": "hello"}, {"foo": "test"}]}))
+        # should test ok, tests basic foo requirement and required_if
+        args = json.dumps(dict(ANSIBLE_MODULE_ARGS={'foobar': [{"foo": "hello", "bam": "good"}, {"foo": "test", "bar": "good"}]}))
         with swap_stdin_and_argv(stdin_data=args):
             basic._ANSIBLE_ARGS = None
             am = basic.AnsibleModule(
                 argument_spec=arg_spec,
-                mutually_exclusive=mut_ex,
-                required_together=req_to,
                 no_log=True,
                 check_invalid_arguments=False,
                 add_file_common_args=True,
@@ -371,13 +387,11 @@ def test_module_utils_basic_ansible_module_with_options_creation(self):
             )
 
         # should test ok, handles aliases
-        args = json.dumps(dict(ANSIBLE_MODULE_ARGS={'foobar': [{"dup": "hello"}]}))
+        args = json.dumps(dict(ANSIBLE_MODULE_ARGS={'foobar': [{"dup": "test", "bar": "good"}]}))
         with swap_stdin_and_argv(stdin_data=args):
             basic._ANSIBLE_ARGS = None
             am = basic.AnsibleModule(
                 argument_spec=arg_spec,
-                mutually_exclusive=mut_ex,
-                required_together=req_to,
                 no_log=True,
                 check_invalid_arguments=False,
                 add_file_common_args=True,
@@ -392,86 +406,74 @@ def test_module_utils_basic_ansible_module_with_options_creation(self):
                 SystemExit,
                 basic.AnsibleModule,
                 argument_spec=arg_spec,
-                mutually_exclusive=mut_ex,
-                required_together=req_to,
                 no_log=True,
                 check_invalid_arguments=False,
                 add_file_common_args=True,
                 supports_check_mode=True
             )
 
-        # fail because of mutually exclusive parameters
-        args = json.dumps(dict(ANSIBLE_MODULE_ARGS={'foobar': [{"foo": "hello", "bar": "bad", "bam": "bad"}]}))
+        # fail because of mutually exclusive parameters (mutually_exclusive, baz is added as it is required_together with bam1)
+        args = json.dumps(dict(ANSIBLE_MODULE_ARGS={'foobar': [{"foo": "test", "bam": "bad", "bam1": "bad", "baz": "req_to"}]}))
         with swap_stdin_and_argv(stdin_data=args):
             basic._ANSIBLE_ARGS = None
             self.assertRaises(
                 SystemExit,
                 basic.AnsibleModule,
                 argument_spec=arg_spec,
-                mutually_exclusive=mut_ex,
-                required_together=req_to,
                 no_log=True,
                 check_invalid_arguments=False,
                 add_file_common_args=True,
                 supports_check_mode=True
             )
 
-        # fail because a param required due to another param was not specified
-        args = json.dumps(dict(ANSIBLE_MODULE_ARGS={'foobar': [{"bam": "bad"}]}))
+        # fail because a param required if for foo=hello is missing (required_if)
+        args = json.dumps(dict(ANSIBLE_MODULE_ARGS={'foobar': [{"foo": "hello", "bar": "bad"}]}))
         with swap_stdin_and_argv(stdin_data=args):
             basic._ANSIBLE_ARGS = None
             self.assertRaises(
                 SystemExit,
                 basic.AnsibleModule,
                 argument_spec=arg_spec,
-                mutually_exclusive=mut_ex,
-                required_together=req_to,
                 no_log=True,
                 check_invalid_arguments=False,
                 add_file_common_args=True,
                 supports_check_mode=True
             )
 
-        # fail because one of param is required
-        req_one_of = (('bar', 'bam'),)
-        args = json.dumps(dict(ANSIBLE_MODULE_ARGS={'foobar': [{"foo": "hello"}]}))
+        # fail because one of param is required (required_one_of)
+        args = json.dumps(dict(ANSIBLE_MODULE_ARGS={'foobar': [{"foo": "test"}]}))
         with swap_stdin_and_argv(stdin_data=args):
             basic._ANSIBLE_ARGS = None
             self.assertRaises(
                 SystemExit,
                 basic.AnsibleModule,
                 argument_spec=arg_spec,
-                required_one_of=req_one_of,
                 no_log=True,
                 check_invalid_arguments=False,
                 add_file_common_args=True,
                 supports_check_mode=True
             )
 
-        # fail because value of one param mandates presence of other param required
-        req_if = (('foo', 'hello', ('bam')),)
-        args = json.dumps(dict(ANSIBLE_MODULE_ARGS={'foobar': [{"foo": "hello"}]}))
+        # fail because one parameter requires another (required_together, bar is added for the required_one_of field)
+        args = json.dumps(dict(ANSIBLE_MODULE_ARGS={'foobar': [{"foo": "test", "bar": "required_one_of", "bam1": "bad"}]}))
         with swap_stdin_and_argv(stdin_data=args):
             basic._ANSIBLE_ARGS = None
             self.assertRaises(
                 SystemExit,
                 basic.AnsibleModule,
                 argument_spec=arg_spec,
-                required_if=req_if,
                 no_log=True,
                 check_invalid_arguments=False,
                 add_file_common_args=True,
                 supports_check_mode=True
             )
 
-        # should test ok, the required param is set by default from spec
-        req_if = [('foo', 'hello', ('bam1',))]
-        args = json.dumps(dict(ANSIBLE_MODULE_ARGS={'foobar': [{"foo": "hello"}]}))
+        # should test ok, the required param is set by default from spec (required_if together with default value, bar added for required_one_of
+        args = json.dumps(dict(ANSIBLE_MODULE_ARGS={'foobar': [{"foo": "bam2", "bar": "required_one_of"}]}))
         with swap_stdin_and_argv(stdin_data=args):
             basic._ANSIBLE_ARGS = None
             am = basic.AnsibleModule(
                 argument_spec=arg_spec,
-                required_if=req_if,
                 no_log=True,
                 check_invalid_arguments=False,
                 add_file_common_args=True,