Repository: ansible/ansible

Issue:
lambda_event module fails to create SQS based events
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->

##### SUMMARY
<!--- Explain the problem briefly -->
lambda_events module cannot properly create SQS based events due to incompatibility with starting_position source parameter.
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->
lambda_event
##### ANSIBLE VERSION
<!--- Paste, BELOW THIS COMMENT, verbatim output from "ansible --version" between quotes below -->
2.5.4
```

```

##### CONFIGURATION
<!--- If using Ansible 2.4 or above, paste, BELOW THIS COMMENT, the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).-->

##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->

##### STEPS TO REPRODUCE
<!--- For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used. -->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
<these are the contents of bug_report.yml>
---

- hosts: localhost
  connection: local
  gather_facts: False

  vars:
    ansible_python_interpreter: /usr/local/bin/python
    bucket: <omitted from bug report>
    key: <omitted from bug report>
    role: <omitted from bug report>
    handler: <omitted from bug report>
    aws_account: <omitted from bug report>

  tasks:
  - name: create SQS topic testme
    sqs_queue:
      name: testme
      region: us-west-2
      default_visibility_timeout: 30
      message_retention_period: 345600
      maximum_message_size: 256000
      delivery_delay: 0
      receive_message_wait_time: 0

  - name: create test lambdas
    lambda:
      name: testHandler
      region: us-west-2
      state: present
      s3_bucket: "{{ bucket }}"
      s3_key: "{{ key }}" 
      runtime: java8
      role: "{{ role }}"
      handler: "{{ handler }}"

  - name: create lambda sqs event mapping
    lambda_event:
      state: present
      region: us-west-2
      event_source: stream
      lambda_function_arn: testHandler
      source_params:
        source_arn: "arn:aws:sqs:us-west-2:{{ aws_account }}:testme"
        enabled: True
        batch_size: 1

```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
create lambda sqs event mapping task to correctly create the lambda sqs mapping
##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
fatal: [localhost]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "alias": null,
            "aws_access_key": null,
            "aws_secret_key": null,
            "ec2_url": null,
            "event_source": "stream",
            "lambda_function_arn": "arn:aws:lambda:us-west-2:<omitted from bug report>:function:testHandler",
            "profile": null,
            "region": "us-west-2",
            "security_token": null,
            "source_params": {
                "batch_size": 1,
                "enabled": true,
                "source_arn": "arn:aws:sqs:us-west-2:<omitted from bug report>:testme"
            },
            "state": "present",
            "validate_certs": true,
            "version": 0
        }
    },
    "msg": "Source parameter 'starting_position' is required for stream event notification."
}
<!--- Paste verbatim command output between quotes below -->
```
ansible-playbook bug_report.yml -vvv
```



Discussion:
Files identified in the description:
* [lib/ansible/modules/cloud/amazon/lambda_event.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/lambda_event.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @pjodouin @ryansb
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
I tested bypassing ansible and writing a quick python script to test boto. This worked fine:
```
#!/usr/bin/env python

import boto3

client = boto3.client('lambda', region_name='us-west-2')

response = client.create_event_source_mapping(
    EventSourceArn='arn:aws:sqs:us-west-2:<omitted from bug report>:testme',
    FunctionName='testHandler,
    Enabled=True,
    BatchSize=1,
)
```

Also tested through Ansible by commenting out lines 324 & 325 from https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/lambda_event.py on my local install. This worked well and Ansible output the following:

```
changed: [localhost] => {
    "changed": true,
    "events": {
        "batch_size": 1,
        "event_source_arn": "arn:aws:sqs:us-west-2:<omitted from bug report>:testme",
        "function_arn": "arn:aws:lambda:us-west-2:<omitted from bug report>:function:testHandler",
        "last_modified": "2018-07-18T15:33:04.229000-07:00",
        "response_metadata": {
            "http_headers": {
                "connection": "keep-alive",
                "content-length": "317",
                "content-type": "application/json",
                "date": "Wed, 18 Jul 2018 22:33:04 GMT",
                "x-amzn-requestid": "<omitted from bug report>"
            },
            "http_status_code": 202,
            "request_id": "<omitted from bug report>",
            "retry_attempts": 0
        },
        "state": "Creating",
        "state_transition_reason": "USER_INITIATED",
        "uuid": "<omitted from bug report>"
    },
    "invocation": {
        "module_args": {
            "alias": null,
            "aws_access_key": null,
            "aws_secret_key": null,
            "ec2_url": null,
            "event_source": "stream",
            "lambda_function_arn": "arn:aws:lambda:us-west-2:<omitted from bug report>:function:testHandler",
            "profile": null,
            "region": "us-west-2",
            "security_token": null,
            "source_params": {
                "batch_size": 1,
                "enabled": true,
                "source_arn": "arn:aws:sqs:us-west-2:<omitted from bug report>:testme"
            },
            "state": "present",
            "validate_certs": true,
            "version": 0
        }
    }
}
```

I'm not sure if simply commenting that out is the right solution though, but works for this case. Please let me know if I can assist further. Thanks.

Resolving patch:
diff --git a/lib/ansible/modules/cloud/amazon/lambda_event.py b/lib/ansible/modules/cloud/amazon/lambda_event.py
--- a/lib/ansible/modules/cloud/amazon/lambda_event.py
+++ b/lib/ansible/modules/cloud/amazon/lambda_event.py
@@ -47,10 +47,12 @@
     required: false
   event_source:
     description:
-      -  Source of the event that triggers the lambda function.
+      - Source of the event that triggers the lambda function.
+      - For DynamoDB and Kinesis events, select 'stream'
+      - For SQS queues, select 'sqs'
     required: false
     default: stream
-    choices: ['stream']
+    choices: ['stream', 'sqs']
   source_params:
     description:
       -  Sub-parameters required for event source.
@@ -61,6 +63,11 @@
          time of invoking your function. Default is 100.
       -  C(starting_position) The position in the stream where AWS Lambda should start reading.
          Choices are TRIM_HORIZON or LATEST.
+      -  I(== sqs event source ==)
+      -  C(source_arn) The Amazon Resource Name (ARN) of the SQS queue to read events from.
+      -  C(enabled) Indicates whether AWS Lambda should begin reading from the event source. Default is True.
+      -  C(batch_size) The largest number of records that AWS Lambda will retrieve from your event source at the
+         time of invoking your function. Default is 100.
     required: true
 requirements:
     - boto3
@@ -321,6 +328,9 @@ def lambda_event_stream(module, aws):
             starting_position = source_params.get('starting_position')
             if starting_position:
                 api_params.update(StartingPosition=starting_position)
+            elif module.params.get('event_source') == 'sqs':
+                # starting position is not required for SQS
+                pass
             else:
                 module.fail_json(msg="Source parameter 'starting_position' is required for stream event notification.")
 
@@ -384,7 +394,7 @@ def lambda_event_stream(module, aws):
 def main():
     """Produce a list of function suffixes which handle lambda events."""
     this_module = sys.modules[__name__]
-    source_choices = ["stream"]
+    source_choices = ["stream", "sqs"]
 
     argument_spec = ec2_argument_spec()
     argument_spec.update(
@@ -413,9 +423,10 @@ def main():
 
     validate_params(module, aws)
 
-    this_module_function = getattr(this_module, 'lambda_event_{0}'.format(module.params['event_source'].lower()))
-
-    results = this_module_function(module, aws)
+    if module.params['event_source'].lower() in ('stream', 'sqs'):
+        results = lambda_event_stream(module, aws)
+    else:
+        module.fail_json('Please select `stream` or `sqs` as the event type')
 
     module.exit_json(**results)
 


Test patch: