Repository: ansible/ansible

Issue:
elasticache_parameter_group module docs are inaccurate
##### ISSUE TYPE
 - Documentation Report

##### COMPONENT NAME
elasticache_parameter_group module

##### ANSIBLE VERSION
```
ansible 2.3.0.0 (detached HEAD d56ba0945d) last updated 2017/04/14 10:38:39 (GMT -500)
  config file = /Users/mconlon/git/ansible/ansible.cfg
  configured module search path = [u'custom_modules']
  python version = 2.7.12 (default, Oct 11 2016, 05:20:59) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.38)]
```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
ansible.cfg:
```
[defaults]
roles_path = roles
lookup_plugins = lookup_plugins
filter_plugins = filter_plugins
hash_behaviour = merge
library = custom_modules
forks = 50

[ssh_connection]
scp_if_ssh = True
pipelining = True
# an attempt to fix ssh unreachable errors when machines are actually reachable
retries = 5
```

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->
running Ansible from OSX Sierra 10.12.3, managing elasticache parameter groups

##### SUMMARY
The values option for this module doesn't seem to accept the format specified in the docs. The docs are clearly outdated since they also aren't using the required option group_family.

From the docs:
```yaml
  - name: 'Modify a test parameter group'
      elasticache_parameter_group:
        name: 'test-param-group'
        values:
          - ['activerehashing', 'yes']
          - ['client-output-buffer-limit-normal-hard-limit', 4]
        state: 'present'
```

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
My code:
```yaml
    - name: create elasticache parameter groups
      elasticache_parameter_group:
        region: "{{ region }}"
        name: test-group
        group_family: memcached1.4
        values:
          - ['memcached_connections_overhead', 500]
        state: present
```

Since the output (below) indicates it wants a dictionary instead of a list, I also tried many other formats with no luck:

this gives the same error output as above:
```yaml
values:
  - memcached_connections_overhead: 500
```

both of these give a different error, listed immediately below
```yaml
values:
  memcached_connections_overhead: 500
```

```yaml
values:
  - key: memcached_connections_overhead
    value: 500
```

error output:
```
The full traceback is:
Traceback (most recent call last):
  File "/var/folders/dx/xqv6cb4n7ys7mrfpvy4k3f5h0000gn/T/ansible_BUBf8g/ansible_module_elasticache_parameter_group.py", line 330, in <module>
    main()
  File "/var/folders/dx/xqv6cb4n7ys7mrfpvy4k3f5h0000gn/T/ansible_BUBf8g/ansible_module_elasticache_parameter_group.py", line 310, in main
    response, changed = create(module, connection, parameter_group_name, parameter_group_family, group_description)
  File "/var/folders/dx/xqv6cb4n7ys7mrfpvy4k3f5h0000gn/T/ansible_BUBf8g/ansible_module_elasticache_parameter_group.py", line 131, in create
    except boto.exception.BotoServerError as e:
NameError: global name 'boto' is not defined

fatal: [localhost]: FAILED! => {
    "changed": false,
    "failed": true,
    "module_stderr": "Traceback (most recent call last):\n  File \"/var/folders/dx/xqv6cb4n7ys7mrfpvy4k3f5h0000gn/T/ansible_BUBf8g/ansible_module_elasticache_parameter_group.py\", line 330, in <module>\n    main()\n  File \"/var/folders/dx/xqv6cb4n7ys7mrfpvy4k3f5h0000gn/T/ansible_BUBf8g/ansible_module_elasticache_parameter_group.py\", line 310, in main\n    response, changed = create(module, connection, parameter_group_name, parameter_group_family, group_description)\n  File \"/var/folders/dx/xqv6cb4n7ys7mrfpvy4k3f5h0000gn/T/ansible_BUBf8g/ansible_module_elasticache_parameter_group.py\", line 131, in create\n    except boto.exception.BotoServerError as e:\nNameError: global name 'boto' is not defined\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE",
    "rc": 1
}
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
An elasticsearch parameter group should be created with one altered value

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
TASK [create elasticache parameter groups] ***************************************************************************************************************************************************
task path: /Users/mconlon/git/ansible/playbooks/scaffold/elasticache.yml:32
Using module file /Users/mconlon/git/ansible/ansible2300/src/ansible/lib/ansible/modules/cloud/amazon/elasticache_parameter_group.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: mconlon
<localhost> EXEC /bin/sh -c '/usr/bin/python && sleep 0'
fatal: [localhost]: FAILED! => {
    "changed": false,
    "failed": true,
    "invocation": {
        "module_args": {
            "group_family": "memcached1.4",
            "name": "test-group",
            "region": "us-east-1",
            "state": "present",
            "validate_certs": true,
            "values": [
                [
                    "memcached_connections_overhead",
                    500
                ]
            ]
        }
    },
    "msg": "argument values is of type <type 'list'> and we were unable to convert to dict: <type 'list'> cannot be converted to a dict"
}
	to retry, use: --limit @/Users/mconlon/git/ansible/playbooks/scaffold/elasticache.retry
```

elasticache_parameter_group module docs are inaccurate
##### ISSUE TYPE
 - Documentation Report

##### COMPONENT NAME
elasticache_parameter_group module

##### ANSIBLE VERSION
```
ansible 2.3.0.0 (detached HEAD d56ba0945d) last updated 2017/04/14 10:38:39 (GMT -500)
  config file = /Users/mconlon/git/ansible/ansible.cfg
  configured module search path = [u'custom_modules']
  python version = 2.7.12 (default, Oct 11 2016, 05:20:59) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.38)]
```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
ansible.cfg:
```
[defaults]
roles_path = roles
lookup_plugins = lookup_plugins
filter_plugins = filter_plugins
hash_behaviour = merge
library = custom_modules
forks = 50

[ssh_connection]
scp_if_ssh = True
pipelining = True
# an attempt to fix ssh unreachable errors when machines are actually reachable
retries = 5
```

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->
running Ansible from OSX Sierra 10.12.3, managing elasticache parameter groups

##### SUMMARY
The values option for this module doesn't seem to accept the format specified in the docs. The docs are clearly outdated since they also aren't using the required option group_family.

From the docs:
```yaml
  - name: 'Modify a test parameter group'
      elasticache_parameter_group:
        name: 'test-param-group'
        values:
          - ['activerehashing', 'yes']
          - ['client-output-buffer-limit-normal-hard-limit', 4]
        state: 'present'
```

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
My code:
```yaml
    - name: create elasticache parameter groups
      elasticache_parameter_group:
        region: "{{ region }}"
        name: test-group
        group_family: memcached1.4
        values:
          - ['memcached_connections_overhead', 500]
        state: present
```

Since the output (below) indicates it wants a dictionary instead of a list, I also tried many other formats with no luck:

this gives the same error output as above:
```yaml
values:
  - memcached_connections_overhead: 500
```

both of these give a different error, listed immediately below
```yaml
values:
  memcached_connections_overhead: 500
```

```yaml
values:
  - key: memcached_connections_overhead
    value: 500
```

error output:
```
The full traceback is:
Traceback (most recent call last):
  File "/var/folders/dx/xqv6cb4n7ys7mrfpvy4k3f5h0000gn/T/ansible_BUBf8g/ansible_module_elasticache_parameter_group.py", line 330, in <module>
    main()
  File "/var/folders/dx/xqv6cb4n7ys7mrfpvy4k3f5h0000gn/T/ansible_BUBf8g/ansible_module_elasticache_parameter_group.py", line 310, in main
    response, changed = create(module, connection, parameter_group_name, parameter_group_family, group_description)
  File "/var/folders/dx/xqv6cb4n7ys7mrfpvy4k3f5h0000gn/T/ansible_BUBf8g/ansible_module_elasticache_parameter_group.py", line 131, in create
    except boto.exception.BotoServerError as e:
NameError: global name 'boto' is not defined

fatal: [localhost]: FAILED! => {
    "changed": false,
    "failed": true,
    "module_stderr": "Traceback (most recent call last):\n  File \"/var/folders/dx/xqv6cb4n7ys7mrfpvy4k3f5h0000gn/T/ansible_BUBf8g/ansible_module_elasticache_parameter_group.py\", line 330, in <module>\n    main()\n  File \"/var/folders/dx/xqv6cb4n7ys7mrfpvy4k3f5h0000gn/T/ansible_BUBf8g/ansible_module_elasticache_parameter_group.py\", line 310, in main\n    response, changed = create(module, connection, parameter_group_name, parameter_group_family, group_description)\n  File \"/var/folders/dx/xqv6cb4n7ys7mrfpvy4k3f5h0000gn/T/ansible_BUBf8g/ansible_module_elasticache_parameter_group.py\", line 131, in create\n    except boto.exception.BotoServerError as e:\nNameError: global name 'boto' is not defined\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE",
    "rc": 1
}
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
An elasticsearch parameter group should be created with one altered value

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
TASK [create elasticache parameter groups] ***************************************************************************************************************************************************
task path: /Users/mconlon/git/ansible/playbooks/scaffold/elasticache.yml:32
Using module file /Users/mconlon/git/ansible/ansible2300/src/ansible/lib/ansible/modules/cloud/amazon/elasticache_parameter_group.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: mconlon
<localhost> EXEC /bin/sh -c '/usr/bin/python && sleep 0'
fatal: [localhost]: FAILED! => {
    "changed": false,
    "failed": true,
    "invocation": {
        "module_args": {
            "group_family": "memcached1.4",
            "name": "test-group",
            "region": "us-east-1",
            "state": "present",
            "validate_certs": true,
            "values": [
                [
                    "memcached_connections_overhead",
                    500
                ]
            ]
        }
    },
    "msg": "argument values is of type <type 'list'> and we were unable to convert to dict: <type 'list'> cannot be converted to a dict"
}
	to retry, use: --limit @/Users/mconlon/git/ansible/playbooks/scaffold/elasticache.retry
```



Discussion:
cc @s-hertel
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
At first glance, it looks like the params were changed as part of the PR (https://github.com/ansible/ansible/pull/21023) that added this module, but the docs weren't updated to match.
Hi, @conlon thanks for opening this issue. The module expects the format of values to be:
```
values:
  activerehashing: "no"
  client-output-buffer-limit-normal-hard-limit: 20
```
and I'll fix the docs for that promptly. 

About group_family not being in the [docs](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/elasticache_parameter_group.py#L32).

I'm sorry about the unfriendly error message you get for the correctly formatted values. I mistakenly used boto errors as opposed to boto3. Fixing right now!

Side note - you might not have elasticache permissions for your aws profile if you're getting that error.
cc @s-hertel
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
At first glance, it looks like the params were changed as part of the PR (https://github.com/ansible/ansible/pull/21023) that added this module, but the docs weren't updated to match.
Hi, @conlon thanks for opening this issue. The module expects the format of values to be:
```
values:
  activerehashing: "no"
  client-output-buffer-limit-normal-hard-limit: 20
```
and I'll fix the docs for that promptly. 

About group_family not being in the [docs](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/elasticache_parameter_group.py#L32).

I'm sorry about the unfriendly error message you get for the correctly formatted values. I mistakenly used boto errors as opposed to boto3. Fixing right now!

Side note - you might not have elasticache permissions for your aws profile if you're getting that error.

Resolving patch:
diff --git a/lib/ansible/modules/cloud/amazon/elasticache_parameter_group.py b/lib/ansible/modules/cloud/amazon/elasticache_parameter_group.py
--- a/lib/ansible/modules/cloud/amazon/elasticache_parameter_group.py
+++ b/lib/ansible/modules/cloud/amazon/elasticache_parameter_group.py
@@ -32,8 +32,9 @@
   group_family:
     description:
       - The name of the cache parameter group family that the cache parameter group can be used with.
+        Required when creating a cache parameter group.
     choices: ['memcached1.4', 'redis2.6', 'redis2.8', 'redis3.2']
-    required: yes
+    required: no
   name:
     description:
      - A user-specified name for the cache parameter group.
@@ -48,7 +49,7 @@
     required: true
   values:
     description:
-      - A user-specified list of parameters to reset or modify for the cache parameter group.
+      - A user-specified dictionary of parameters to reset or modify for the cache parameter group.
     required: no
     default: None
 """
@@ -70,8 +71,8 @@
       elasticache_parameter_group:
         name: 'test-param-group'
         values:
-          - ['activerehashing', 'yes']
-          - ['client-output-buffer-limit-normal-hard-limit', 4]
+          activerehashing: yes
+          client-output-buffer-limit-normal-hard-limit: 4
         state: 'present'
     - name: 'Reset all modifiable parameters for the test parameter group'
       elasticache_parameter_group:
@@ -123,25 +124,28 @@
 except ImportError:
     HAS_BOTO3 = False
 
+
 def create(module, conn, name, group_family, description):
     """ Create ElastiCache parameter group. """
     try:
         response = conn.create_cache_parameter_group(CacheParameterGroupName=name, CacheParameterGroupFamily=group_family, Description=description)
         changed = True
-    except boto.exception.BotoServerError as e:
-        module.fail_json(msg="Unable to create cache parameter group.", exception=traceback.format_exc())
+    except botocore.exceptions.ClientError as e:
+        module.fail_json(msg="Unable to create cache parameter group.", exception=traceback.format_exc(), **camel_dict_to_snake_dict(e.response))
     return response, changed
 
+
 def delete(module, conn, name):
     """ Delete ElastiCache parameter group. """
     try:
         conn.delete_cache_parameter_group(CacheParameterGroupName=name)
         response = {}
         changed = True
-    except boto.exception.BotoServerError as e:
-        module.fail_json(msg="Unable to delete cache parameter group.", exception=traceback.format_exc())
+    except botocore.exceptions.ClientError as e:
+        module.fail_json(msg="Unable to delete cache parameter group.", exception=traceback.format_exc(), **camel_dict_to_snake_dict(e.response))
     return response, changed
 
+
 def make_current_modifiable_param_dict(module, conn, name):
     """ Gets the current state of the cache parameter group and creates a dict with the format: {ParameterName: [Allowed_Values, DataType, ParameterValue]}"""
     current_info = get_info(conn, name)
@@ -156,6 +160,7 @@ def make_current_modifiable_param_dict(module, conn, name):
             modifiable_params[param["ParameterName"]] = [param["AllowedValues"], param["DataType"], param["ParameterValue"]]
     return modifiable_params
 
+
 def check_valid_modification(module, values, modifiable_params):
     """ Check if the parameters and values in values are valid.  """
     changed_with_update = False
@@ -184,6 +189,7 @@ def check_valid_modification(module, values, modifiable_params):
 
     return changed_with_update
 
+
 def check_changed_parameter_values(values, old_parameters, new_parameters):
     """ Checking if the new values are different than the old values.  """
     changed_with_update = False
@@ -203,6 +209,7 @@ def check_changed_parameter_values(values, old_parameters, new_parameters):
 
     return changed_with_update
 
+
 def modify(module, conn, name, values):
     """ Modify ElastiCache parameter group to reflect the new information if it differs from the current. """
     # compares current group parameters with the parameters we've specified to to a value to see if this will change the group
@@ -212,10 +219,11 @@ def modify(module, conn, name, values):
         format_parameters.append({'ParameterName': key, 'ParameterValue': value})
     try:
         response = conn.modify_cache_parameter_group(CacheParameterGroupName=name, ParameterNameValues=format_parameters)
-    except boto.exception.BotoServerError as e:
-        module.fail_json(msg="Unable to modify cache parameter group.", exception=traceback.format_exc())
+    except botocore.exceptions.ClientError as e:
+        module.fail_json(msg="Unable to modify cache parameter group.", exception=traceback.format_exc(), **camel_dict_to_snake_dict(e.response))
     return response
 
+
 def reset(module, conn, name, values):
     """ Reset ElastiCache parameter group if the current information is different from the new information. """
     # used to compare with the reset parameters' dict to see if there have been changes
@@ -235,8 +243,8 @@ def reset(module, conn, name, values):
 
     try:
         response = conn.reset_cache_parameter_group(CacheParameterGroupName=name, ParameterNameValues=format_parameters, ResetAllParameters=all_parameters)
-    except boto.exception.BotoServerError as e:
-        module.fail_json(msg="Unable to reset cache parameter group.", exception=traceback.format_exc())
+    except botocore.exceptions.ClientError as e:
+        module.fail_json(msg="Unable to reset cache parameter group.", exception=traceback.format_exc(), **camel_dict_to_snake_dict(e.response))
 
     # determine changed
     new_parameters_dict = make_current_modifiable_param_dict(module, conn, name)
@@ -244,6 +252,7 @@ def reset(module, conn, name, values):
 
     return response, changed
 
+
 def get_info(conn, name):
     """ Gets info about the ElastiCache parameter group. Returns false if it doesn't exist or we don't have access. """
     try:
@@ -287,7 +296,7 @@ def main():
     exists = get_info(connection, parameter_group_name)
 
     # check that the needed requirements are available
-    if state == 'present' and not exists and not (parameter_group_family or group_description):
+    if state == 'present' and not (exists and parameter_group_family and group_description):
         module.fail_json(msg="Creating a group requires a family group and a description.")
     elif state == 'reset' and not exists:
         module.fail_json(msg="No group %s to reset. Please create the group before using the state 'reset'." % parameter_group_name)


Test patch:
diff --git a/test/sanity/pep8/legacy-files.txt b/test/sanity/pep8/legacy-files.txt
--- a/test/sanity/pep8/legacy-files.txt
+++ b/test/sanity/pep8/legacy-files.txt
@@ -193,7 +193,6 @@ lib/ansible/modules/cloud/amazon/ecs_taskdefinition.py
 lib/ansible/modules/cloud/amazon/efs.py
 lib/ansible/modules/cloud/amazon/efs_facts.py
 lib/ansible/modules/cloud/amazon/elasticache.py
-lib/ansible/modules/cloud/amazon/elasticache_parameter_group.py
 lib/ansible/modules/cloud/amazon/elasticache_snapshot.py
 lib/ansible/modules/cloud/amazon/elasticache_subnet_group.py
 lib/ansible/modules/cloud/amazon/execute_lambda.py