Repository: ansible/ansible

Issue:
elb_application_lb module: ParamValidationError in botocore when specifying "Rules" on a new application load balancer
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
elb_application_lb

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.4.0 (devel 498bf4b5be) last updated 2017/07/26 15:27:07 (GMT +000)
  config file = None
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /ansible/.ansible-fork/lib/ansible
  executable location = /ansible/.ansible-fork/bin/ansible
  python version = 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]
```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
Default configuration

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->
ubuntu 16.04

##### SUMMARY
<!--- Explain the problem briefly -->
Parameter validation error in `boto` when running `elb_application_lb` task requiring updates to listeners.

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
Run a task definition like the following on to create an application load balancer:

```yaml
elb_application_lb:
  name: my-application-lb
  region: eu-west-1
  scheme: internet-facing
  subnets:
  - my-subnet
  security_groups:
  - my-sg
  listeners:
  - Protocol: HTTP
    Port: 80
    DefaultActions:
    - Type: forward
      TargetGroupName: my-target-group
    Rules:
    - Conditions:
      - Field: host-header
        Values:
        - www.example.com
      Priority: '1'
      Actions:
      - TargetGroupName: my-target-group
        Type: forward
  state: present
```

The following error is produced in boto:

```
Unknown parameter in input: "Rules", must be one of: LoadBalancerArn, Protocol, Port, SslPolicy, Certificates, DefaultActions
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_LtFMN2/ansible_module_elb_application_lb.py\", line 1011, in <module>\n    main()\n  File \"/tmp/ans
ible_LtFMN2/ansible_module_elb_application_lb.py\", line 1006, in main\n    create_or_update_elb(connection, connection_ec2, module)\n  File \"/tmp/ansible_LtFMN2/ansible_module_elb_application_lb.py\", line 901, in create_or_update_elb\n
listener_changed = create_or_update_elb_listeners(connection, module, elb)\n  File \"/tmp/ansible_LtFMN2/ansible_module_elb_application_lb.py\", line 722, in create_or_update_elb_listeners\n    connection.create_listener(**listener_to_add)\n
 File \"/usr/local/lib/python2.7/dist-packages/botocore/client.py\", line 310, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n  File \"/usr/local/lib/python2.7/dist-packages/botocore/client.py\", line 573, in _make_api_
call\n    api_params, operation_model, context=request_context)\n  File \"/usr/local/lib/python2.7/dist-packages/botocore/client.py\", line 628, in _convert_to_request_dict\n    api_params, operation_model)\n  File \"/usr/local/lib/python2.7/
dist-packages/botocore/validate.py\", line 291, in serialize_to_request\n    raise ParamValidationError(report=report.generate_report())\nbotocore.exceptions.ParamValidationError: Parameter validation failed:\nUnknown parameter in input: \"Ru
les\", must be one of: LoadBalancerArn, Protocol, Port, SslPolicy, Certificates, DefaultActions\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 0}
```

The fix is straightforward enough, and I'll submit a PR with a test.

<!--- Paste example playbooks or commands between quotes below -->
```yaml
elb_application_lb:
  name: my-application-lb
  region: eu-west-1
  scheme: internet-facing
  subnets:
  - my-subnet
  security_groups:
  - my-sg
  listeners:
  - Protocol: HTTP
    Port: 80
    DefaultActions:
    - Type: forward
      TargetGroupName: my-target-group
    Rules:
    - Conditions:
      - Field: host-header
        Values:
        - www.example.com
      Priority: '1'
      Actions:
      - TargetGroupName: my-target-group
        Type: forward
  state: present
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
No error from boto

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
Error displayed in boto

<!--- Paste verbatim command output between quotes below -->
```
ansible-playbook playbook.yml
```



Discussion:

Resolving patch:
diff --git a/lib/ansible/modules/cloud/amazon/elb_application_lb.py b/lib/ansible/modules/cloud/amazon/elb_application_lb.py
--- a/lib/ansible/modules/cloud/amazon/elb_application_lb.py
+++ b/lib/ansible/modules/cloud/amazon/elb_application_lb.py
@@ -688,8 +688,8 @@ def compare_rules(connection, module, current_listeners, listener):
                 if modified_rule:
                     modified_rule['Priority'] = int(current_rule['Priority'])
                     modified_rule['RuleArn'] = current_rule['RuleArn']
-                    modified_rule['Actions'] = current_rule['Actions']
-                    modified_rule['Conditions'] = current_rule['Conditions']
+                    modified_rule['Actions'] = new_rule['Actions']
+                    modified_rule['Conditions'] = new_rule['Conditions']
                     rules_to_modify.append(modified_rule)
                 break
 
@@ -719,7 +719,11 @@ def create_or_update_elb_listeners(connection, module, elb):
     for listener_to_add in listeners_to_add:
         try:
             listener_to_add['LoadBalancerArn'] = elb['LoadBalancerArn']
-            connection.create_listener(**listener_to_add)
+            # Rules is not a valid parameter for create_listener
+            listener_to_add.pop('Rules')
+            response = connection.create_listener(**listener_to_add)
+            # Add the new listener
+            current_listeners.append(response['Listeners'][0])
             listener_changed = True
         except ClientError as e:
             module.fail_json(msg=e.message, exception=traceback.format_exc(), **camel_dict_to_snake_dict(e.response))
@@ -727,6 +731,8 @@ def create_or_update_elb_listeners(connection, module, elb):
     # Modify listeners
     for listener_to_modify in listeners_to_modify:
         try:
+            # Rules is not a valid parameter for modify_listener
+            listener_to_modify.pop('Rules')
             connection.modify_listener(**listener_to_modify)
             listener_changed = True
         except ClientError as e:
@@ -741,7 +747,7 @@ def create_or_update_elb_listeners(connection, module, elb):
             module.fail_json(msg=e.message, exception=traceback.format_exc(), **camel_dict_to_snake_dict(e.response))
 
     # For each listener, check rules
-    for listener in listeners:
+    for listener in deepcopy(listeners):
         if 'Rules' in listener:
             # Ensure rules are using Target Group ARN not name
             listener['Rules'] = ensure_rules_action_has_arn(connection, module, listener['Rules'])


Test patch:
diff --git a/test/units/modules/cloud/amazon/test_elb_application_lb.py b/test/units/modules/cloud/amazon/test_elb_application_lb.py
new file mode 100644
--- /dev/null
+++ b/test/units/modules/cloud/amazon/test_elb_application_lb.py
@@ -0,0 +1,153 @@
+#
+# (c) 2017 Michael Tinning
+#
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+
+from __future__ import (absolute_import, division, print_function)
+
+from nose.plugins.skip import SkipTest
+import json
+import pytest
+from copy import deepcopy
+from ansible.module_utils._text import to_bytes
+from ansible.module_utils import basic
+from ansible.module_utils.ec2 import HAS_BOTO3
+
+if not HAS_BOTO3:
+    raise SkipTest("test_elb_application_lb.py requires the `boto3` and `botocore` modules")
+
+import ansible.modules.cloud.amazon.elb_application_lb as elb_module
+
+
+@pytest.fixture
+def listener():
+    return {
+        'Protocol': 'HTTP',
+        'Port': 80,
+        'DefaultActions': [{
+            'Type': 'forward',
+            'TargetGroupName': 'target-group'
+        }],
+        'Rules': [{
+            'Conditions': [{
+                'Field': 'host-header',
+                'Values': [
+                    'www.example.com'
+                ]
+            }],
+            'Priority': 1,
+            'Actions': [{
+                'TargetGroupName': 'other-target-group',
+                'Type': 'forward'
+            }]
+        }]
+    }
+
+
+@pytest.fixture
+def compare_listeners(mocker):
+    return mocker.Mock()
+
+
+@pytest.fixture
+def ensure_listeners(mocker):
+    ensure_listeners_mock = mocker.Mock()
+    ensure_listeners_mock.return_value = []
+    return ensure_listeners_mock
+
+
+@pytest.fixture
+def compare_rules(mocker):
+    compare_rules_mock = mocker.Mock()
+    compare_rules_mock.return_value = ([], [], [])
+    return compare_rules_mock
+
+
+@pytest.fixture
+def get_elb_listeners(mocker):
+    get_elb_listeners_mock = mocker.Mock()
+    get_elb_listeners_mock.return_value = []
+    return get_elb_listeners_mock
+
+
+@pytest.fixture
+def elb(mocker, monkeypatch, compare_listeners, ensure_listeners, compare_rules, get_elb_listeners):
+    monkeypatch.setattr(elb_module, "ensure_listeners_default_action_has_arn", ensure_listeners)
+    monkeypatch.setattr(elb_module, "get_elb_listeners", get_elb_listeners)
+    monkeypatch.setattr(elb_module, "ensure_rules_action_has_arn", mocker.Mock())
+    monkeypatch.setattr(elb_module, "get_listener", mocker.Mock())
+    monkeypatch.setattr(elb_module, "compare_rules", compare_rules)
+    monkeypatch.setattr(elb_module, "compare_listeners", compare_listeners)
+    return elb_module
+
+
+@pytest.fixture
+def created_listener(mocker, listener):
+    return {
+        'Port': listener['Port'],
+        'ListenerArn': 'new-listener-arn'
+    }
+
+
+@pytest.fixture
+def connection(mocker, created_listener):
+    connection_mock = mocker.Mock()
+    connection_mock.create_listener.return_value = {
+        'Listeners': [created_listener]
+    }
+    return connection_mock
+
+
+@pytest.fixture
+def existing_elb():
+    return {'LoadBalancerArn': 'fake'}
+
+
+def test_create_listeners_called_with_correct_args(mocker, connection, listener, elb, compare_listeners, existing_elb):
+    compare_listeners.return_value = ([listener], [], [])
+
+    elb.create_or_update_elb_listeners(connection, mocker.Mock(), existing_elb)
+
+    connection.create_listener.assert_called_once_with(
+        Protocol=listener['Protocol'],
+        Port=listener['Port'],
+        DefaultActions=listener['DefaultActions'],
+        LoadBalancerArn=existing_elb['LoadBalancerArn']
+    )
+
+
+def test_modify_listeners_called_with_correct_args(mocker, connection, listener, elb, compare_listeners, existing_elb):
+    # In the case of modify listener, LoadBalancerArn is set in compare_listeners
+    listener['LoadBalancerArn'] = existing_elb['LoadBalancerArn']
+    compare_listeners.return_value = ([], [listener], [])
+
+    elb.create_or_update_elb_listeners(connection, mocker.Mock(), existing_elb)
+
+    connection.modify_listener.assert_called_once_with(
+        Protocol=listener['Protocol'],
+        Port=listener['Port'],
+        DefaultActions=listener['DefaultActions'],
+        LoadBalancerArn=existing_elb['LoadBalancerArn']
+    )
+
+
+def test_compare_rules_called_with_new_listener(
+    mocker,
+    connection,
+    listener,
+    elb,
+    compare_listeners,
+    ensure_listeners,
+    compare_rules,
+    existing_elb,
+    created_listener
+):
+    compare_listeners.return_value = ([listener], [], [])
+    listener_from_ensure_listeners = deepcopy(listener)
+    ensure_listeners.return_value = [listener_from_ensure_listeners]
+
+    elb.create_or_update_elb_listeners(connection, mocker.Mock(), existing_elb)
+
+    (_conn, _module, current_listeners, _listener), _kwargs = compare_rules.call_args
+
+    assert created_listener in current_listeners