Repository: ansible/ansible

Issue:
fallback argument_spec parameter is broken
##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
AnsibleModule

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.4.0
  config file = None
  configured module search path = [u'/Users/trupp/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/trupp/src/envs/ansible-dev/lib/python2.7/site-packages/ansible
  executable location = /Users/trupp/src/envs/ansible-dev/bin/ansible
  python version = 2.7.10 (default, Feb  7 2017, 00:08:15) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)]
```

##### CONFIGURATION
[defaults]
retry_files_enabled = False
host_key_checking=False
library=./library


##### OS / ENVIRONMENT
Mac OSX Sierra (10.12.6)

##### SUMMARY
<!--- Explain the problem briefly -->
fallback functionality for an argument spec is broken.

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

Download the following module and put it in `repro.py` in your `library` directory
* https://gist.github.com/caphrim007/8dd2f8ad3f7a58b6186ec53ba97cc96a

Next, download the example playbook and put it in `repro.yaml`
* https://gist.github.com/caphrim007/cfaa7b8d1edef984fb2e2c416980e804

Finally, run the following command

`ansible-playbook -i notahost, repro.yaml`

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
I expect that the playbook should run without error. "ok" should equal 2

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
The playbook fails with the error

`"msg": "missing required arguments: server"`

<!--- Paste verbatim command output between quotes below -->
```
(ansible-dev) SEA-ML-RUPP1:f5-ansible trupp$ ansible-playbook -i notahost, repro.yaml 

PLAY [Test the bigip_dns_record_facts module] ************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************
ok: [notahost]

TASK [Repro 27119] ***************************************************************************************************************************
task path: /Users/trupp/src/trupp/f5-ansible/repro.yaml:9
Using module_utils file /Users/trupp/src/envs/ansible-dev/lib/python2.7/site-packages/ansible/module_utils/basic.py
Using module_utils file /Users/trupp/src/envs/ansible-dev/lib/python2.7/site-packages/ansible/module_utils/_text.py
Using module_utils file /Users/trupp/src/envs/ansible-dev/lib/python2.7/site-packages/ansible/module_utils/parsing/convert_bool.py
Using module_utils file /Users/trupp/src/envs/ansible-dev/lib/python2.7/site-packages/ansible/module_utils/parsing/__init__.py
Using module_utils file /Users/trupp/src/envs/ansible-dev/lib/python2.7/site-packages/ansible/module_utils/pycompat24.py
Using module_utils file /Users/trupp/src/envs/ansible-dev/lib/python2.7/site-packages/ansible/module_utils/six/__init__.py
Using module file /Users/trupp/src/trupp/f5-ansible/library/repro.py
<notahost> ESTABLISH LOCAL CONNECTION FOR USER: trupp
<notahost> EXEC /bin/sh -c 'echo ~ && sleep 0'
<notahost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/trupp/.ansible/tmp/ansible-tmp-1502309149.77-186982986195915 `" && echo ansible-tmp-1502309149.77-186982986195915="` echo /Users/trupp/.ansible/tmp/ansible-tmp-1502309149.77-186982986195915 `" ) && sleep 0'
<notahost> PUT /var/folders/jc/9d1188j962931rhqrlm4173w5j5m45/T/tmpMVbdIY TO /Users/trupp/.ansible/tmp/ansible-tmp-1502309149.77-186982986195915/repro.py
<notahost> EXEC /bin/sh -c 'chmod u+x /Users/trupp/.ansible/tmp/ansible-tmp-1502309149.77-186982986195915/ /Users/trupp/.ansible/tmp/ansible-tmp-1502309149.77-186982986195915/repro.py && sleep 0'
<notahost> EXEC /bin/sh -c 'F5_SERVER=1.1.1.1 /usr/bin/python /Users/trupp/.ansible/tmp/ansible-tmp-1502309149.77-186982986195915/repro.py; rm -rf "/Users/trupp/.ansible/tmp/ansible-tmp-1502309149.77-186982986195915/" > /dev/null 2>&1 && sleep 0'
fatal: [notahost]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_args": {}
    }, 
    "msg": "missing required arguments: server"
}
        to retry, use: --limit @/Users/trupp/src/trupp/f5-ansible/repro.retry

PLAY RECAP ***********************************************************************************************************************************
notahost                   : ok=1    changed=0    unreachable=0    failed=1   
```



Discussion:
@nitzmahone @ganeshrn @Qalthos 
Here's another breaking change. A default of `None` used to be perfectly valid and there's no reason it shouldn't stay that way. consider when the user doesn't want to supply a value for a param and they are updating an existing config.

ansible module
* https://gist.github.com/caphrim007/e81df8de85b295535a80f35a30dda172

ansible playbook
* https://gist.github.com/caphrim007/43a80e89acdfed448c2878eaa297609e

The above examples effectively break every f5 module.
@bcoca @abadger 
correction, that last set of gists was because of the addition of this

https://github.com/ansible/ansible/blob/6d33e59ca19bc8db60281c1022db105d97b61958/lib/ansible/module_utils/basic.py#L1739

is the following syntax no longer valid?

```
            from ansible.module_utils.parsing.convert_bool import BOOLEANS
            full_sync=dict(
                type='bool',
                choices=BOOLEANS
            ),
```
It's redundant with `type='bool'` but if it's not still working, that's a bug, too.

Bummer that we don't have a test suite around this yet- I created the basic.py argument_spec unit test when I plugged in the callable validator stuff, but I only covered what I added... All these little dusty corners need to be in there. :(
Ah, I remember, @ganeshrn added a bunch of tests in a different place. We should definitely extend those to include regression tests for these things that are failing, along with the fixes...
@nitzmahone someone on the irc channel mentioned that fallback used to be used by the network modules, but no longer is. what do those modules use instead?
Fallback support hasn't been removed from the code, so yet another bug if that's broken. I'm not aware of a different way to do what it does, but someone on the networking team might be able to shed more light.

Resolving patch:
diff --git a/lib/ansible/module_utils/basic.py b/lib/ansible/module_utils/basic.py
--- a/lib/ansible/module_utils/basic.py
+++ b/lib/ansible/module_utils/basic.py
@@ -115,9 +115,9 @@
 # this matters, make sure to check for strings before checking for sequencetype
 try:
     from collections.abc import KeysView
-    SEQUENCETYPE = (Sequence, KeysView)
+    SEQUENCETYPE = (Sequence, frozenset, KeysView)
 except:
-    SEQUENCETYPE = Sequence
+    SEQUENCETYPE = (Sequence, frozenset)
 
 try:
     import json
@@ -2044,7 +2044,7 @@ def _set_fallbacks(self, spec=None, param=None):
                     else:
                         fallback_args = item
                 try:
-                    param = fallback_strategy(*fallback_args, **fallback_kwargs)
+                    param[k] = fallback_strategy(*fallback_args, **fallback_kwargs)
                 except AnsibleFallbackNotFound:
                     continue
 


Test patch:
diff --git a/test/units/module_utils/test_basic.py b/test/units/module_utils/test_basic.py
--- a/test/units/module_utils/test_basic.py
+++ b/test/units/module_utils/test_basic.py
@@ -260,6 +260,7 @@ def test_module_utils_basic_get_module_path(self):
 
     def test_module_utils_basic_ansible_module_creation(self):
         from ansible.module_utils import basic
+        from ansible.module_utils.parsing.convert_bool import BOOLEANS
 
         am = basic.AnsibleModule(
             argument_spec=dict(),
@@ -269,7 +270,8 @@ def test_module_utils_basic_ansible_module_creation(self):
             foo=dict(required=True),
             bar=dict(),
             bam=dict(),
-            baz=dict(),
+            baz=dict(fallback=(basic.env_fallback, ['BAZ'])),
+            bar1=dict(type='bool', choices=BOOLEANS)
         )
         mut_ex = (('bar', 'bam'),)
         req_to = (('bam', 'baz'),)
@@ -342,16 +344,50 @@ def test_module_utils_basic_ansible_module_creation(self):
                 supports_check_mode=True,
             )
 
+        # test fallback, should test ok as value of param required due to another param
+        # is set by environment variable
+        os.environ['BAZ'] = 'bad'
+        args = json.dumps(dict(ANSIBLE_MODULE_ARGS={"foo": "hello", "bam": "bad"}))
+
+        with swap_stdin_and_argv(stdin_data=args):
+            basic._ANSIBLE_ARGS = None
+            am = basic.AnsibleModule(
+                argument_spec=arg_spec,
+                mutually_exclusive=mut_ex,
+                required_together=req_to,
+                no_log=True,
+                check_invalid_arguments=False,
+                add_file_common_args=True,
+                supports_check_mode=True,
+            )
+        os.environ.pop('BAZ', None)
+
+        # should test ok, check for boolean values
+        args = json.dumps(dict(ANSIBLE_MODULE_ARGS={"foo": "hello", "bar1": "yes"}))
+        with swap_stdin_and_argv(stdin_data=args):
+            basic._ANSIBLE_ARGS = None
+            am = basic.AnsibleModule(
+                argument_spec=arg_spec,
+                mutually_exclusive=mut_ex,
+                required_together=req_to,
+                no_log=True,
+                check_invalid_arguments=False,
+                add_file_common_args=True,
+                supports_check_mode=True,
+            )
+
     def test_module_utils_basic_ansible_module_with_options_creation(self):
         from ansible.module_utils import basic
+        from ansible.module_utils.parsing.convert_bool import BOOLEANS
 
         options_spec = dict(
             foo=dict(required=True, aliases=['dup']),
             bar=dict(),
             bam=dict(),
-            baz=dict(),
+            baz=dict(fallback=(basic.env_fallback, ['BAZ'])),
             bam1=dict(),
-            bam2=dict(default='test')
+            bam2=dict(default='test'),
+            bam3=dict(type='bool', choices=BOOLEANS)
         )
         arg_spec = dict(
             foobar=dict(
@@ -509,6 +545,34 @@ def test_module_utils_basic_ansible_module_with_options_creation(self):
                 supports_check_mode=True
             )
 
+        # test fallback, should test ok as value of param required due to another param
+        # is set by environment variable
+        os.environ['BAZ'] = 'bad'
+        args = json.dumps(dict(ANSIBLE_MODULE_ARGS={"foo": "hello", "bam": "bad"}))
+
+        with swap_stdin_and_argv(stdin_data=args):
+            basic._ANSIBLE_ARGS = None
+            am = basic.AnsibleModule(
+                argument_spec=arg_spec,
+                no_log=True,
+                check_invalid_arguments=False,
+                add_file_common_args=True,
+                supports_check_mode=True,
+            )
+        os.environ.pop('BAZ', None)
+
+        # should test ok, check for boolean values
+        args = json.dumps(dict(ANSIBLE_MODULE_ARGS={"foo": "hello", "bam3": "yes"}))
+        with swap_stdin_and_argv(stdin_data=args):
+            basic._ANSIBLE_ARGS = None
+            am = basic.AnsibleModule(
+                argument_spec=arg_spec,
+                no_log=True,
+                check_invalid_arguments=False,
+                add_file_common_args=True,
+                supports_check_mode=True,
+            )
+
     def test_module_utils_basic_ansible_module_type_check(self):
         from ansible.module_utils import basic