Repository: Miserlou/Zappa

Issue:
DynamoDB streams as event source is broken  
I could identify two issues, first it seems like in https://github.com/Miserlou/Zappa/blob/master/zappa/util.py#L151 the name should be the Lambda ARN and not the function name (`funk.name = lambda_arn`). Otherwise I get the following error:

```
Traceback (most recent call last):
  File "/Users/cvessaz/.virtualenvs/tokenserviceenv/lib/python2.7/site-packages/zappa/cli.py", line 1251, in handle
    sys.exit(cli.handle())
  File "/Users/cvessaz/.virtualenvs/tokenserviceenv/lib/python2.7/site-packages/zappa/cli.py", line 219, in handle
    self.schedule()
  File "/Users/cvessaz/.virtualenvs/tokenserviceenv/lib/python2.7/site-packages/zappa/cli.py", line 563, in schedule
    events=events
  File "/Users/cvessaz/.virtualenvs/tokenserviceenv/lib/python2.7/site-packages/zappa/zappa.py", line 1212, in schedule_events
    self.unschedule_events(lambda_name=lambda_name, lambda_arn=lambda_arn, events=events)
  File "/Users/cvessaz/.virtualenvs/tokenserviceenv/lib/python2.7/site-packages/zappa/zappa.py", line 1358, in unschedule_events
    self.boto_session
  File "/Users/cvessaz/.virtualenvs/tokenserviceenv/lib/python2.7/site-packages/zappa/util.py", line 192, in remove_event_source
    rule_response = event_source_obj.remove(funk)
  File "/Users/cvessaz/.virtualenvs/tokenserviceenv/lib/python2.7/site-packages/kappa/event_source/kinesis.py", line 96, in remove
    uuid = self._get_uuid(function)
  File "/Users/cvessaz/.virtualenvs/tokenserviceenv/lib/python2.7/site-packages/kappa/event_source/kinesis.py", line 36, in _get_uuid
    EventSourceArn=self.arn)
  File "/Users/cvessaz/.virtualenvs/tokenserviceenv/lib/python2.7/site-packages/kappa/awsclient.py", line 83, in call
    data = results.build_full_result()
  File "/Users/cvessaz/.virtualenvs/tokenserviceenv/lib/python2.7/site-packages/botocore/paginate.py", line 290, in build_full_result
    for response in self:
  File "/Users/cvessaz/.virtualenvs/tokenserviceenv/lib/python2.7/site-packages/botocore/paginate.py", line 102, in __iter__
    response = self._make_request(current_kwargs)
  File "/Users/cvessaz/.virtualenvs/tokenserviceenv/lib/python2.7/site-packages/botocore/paginate.py", line 174, in _make_request
    return self._method(**current_kwargs)
  File "/Users/cvessaz/.virtualenvs/tokenserviceenv/lib/python2.7/site-packages/botocore/client.py", line 159, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/cvessaz/.virtualenvs/tokenserviceenv/lib/python2.7/site-packages/botocore/client.py", line 494, in _make_api_call
    raise ClientError(parsed_response, operation_name)
ClientError: An error occurred (ValidationException) when calling the ListEventSourceMappings operation: 1 validation error detected: Value 'replication.replicate_records' at 'functionName' failed to satisfy constraint: Member must satisfy regular expression pattern: (arn:aws:lambda:)?([a-z]{2}-[a-z]+-\d{1}:)?(\d{12}:)?(function:)?([a-zA-Z0-9-_]+)(:(\$LATEST|[a-zA-Z0-9-_]+))?
```

The second issue is related to the `create_event_permission` call in the `schedule_events` function (https://github.com/Miserlou/Zappa/blob/master/zappa/zappa.py#L1263). It seems like this is not necessary for DynamoDB streams events, it started working once I commented that section out.

Anyways, I do not fully understand yet how this exactly works, so before I open a pull request and break a whole bunch of other things, any idea what is wrong and what is the best way to fix it?



Discussion:
Annoying, I haven't used DDB as an event source myself yet. Thanks for reporting. The create_event_permissions is definitely needed for S3 events, maybe useless for other events, and apparently detrimental for DDB events.

This stuff is full of quirks that need sending through trial and error like this. Hopefully it'll be an easy fix and we can get it in the next release tomorrow.

Would you like to give me a try to fix it, or are you looking into it?

If you're blocking on this right now and you know what to expect, please by all means, be my guest and try to fix it.

Alright, I have a look.


Resolving patch:
diff --git a/zappa/handler.py b/zappa/handler.py
--- a/zappa/handler.py
+++ b/zappa/handler.py
@@ -345,14 +345,13 @@ def handler(self, event, context):
 
             records = event.get('Records')
             result = None
-            for record in records:
-                whole_function = self.get_function_for_aws_event(record)
-                if whole_function:
-                    app_function = self.import_module_and_get_function(whole_function)
-                    result = self.run_function(app_function, event, context)
-                    logger.debug(result)
-                else:
-                    logger.error("Cannot find a function to process the triggered event.")
+            whole_function = self.get_function_for_aws_event(records[0])
+            if whole_function:
+                app_function = self.import_module_and_get_function(whole_function)
+                result = self.run_function(app_function, event, context)
+                logger.debug(result)
+            else:
+                logger.error("Cannot find a function to process the triggered event.")
             return result
 
         # Normal web app flow
diff --git a/zappa/util.py b/zappa/util.py
--- a/zappa/util.py
+++ b/zappa/util.py
@@ -132,7 +132,6 @@ def __init__(self):
     }
 
     arn = event_source['arn']
-    events = event_source['events']
     _, _, svc, _ = arn.split(':', 3)
 
     event_source_func = event_source_map.get(svc, None)
@@ -148,7 +147,7 @@ def autoreturn(self, function_name):
     ctx.session = boto_session
 
     funk = PseudoFunction()
-    funk.name = target_function
+    funk.name = lambda_arn
     
     # Kappa 0.6.0 requires this nasty hacking,
     # hopefully we can remove at least some of this soon.
@@ -173,11 +172,18 @@ def add_event_source(event_source, lambda_arn, target_function, boto_session, dr
     """
 
     event_source_obj, ctx, funk = get_event_source(event_source, lambda_arn, target_function, boto_session, dry=False)
+    # TODO: Detect changes in config and refine exists algorithm
     if not dry:
-        rule_response = event_source_obj.add(funk)
-        return rule_response
-    else:
-        return event_source_obj
+        if not event_source_obj.status(funk):
+            event_source_obj.add(funk)
+            if event_source_obj.status(funk):
+                return 'successful'
+            else:
+                return 'failed'
+        else:
+            return 'exists'
+
+    return 'dryrun'
 
 def remove_event_source(event_source, lambda_arn, target_function, boto_session, dry=False):
     """
diff --git a/zappa/zappa.py b/zappa/zappa.py
--- a/zappa/zappa.py
+++ b/zappa/zappa.py
@@ -179,7 +179,7 @@
 
 ZIP_EXCLUDES = [
     '*.exe', '*.DS_Store', '*.Python', '*.git', '.git/*', '*.zip', '*.tar.gz',
-    '*.hg', '*.egg-info', 'botocore*', 'pip', 'docutils*', 'boto3*', 'setuputils*'
+    '*.hg', '*.egg-info', 'pip', 'docutils*', 'setuputils*'
 ]
 
 ##
@@ -1320,11 +1320,19 @@ def schedule_events(self, lambda_arn, lambda_name, events, default=True):
         Expressions can be in rate or cron format:
             http://docs.aws.amazon.com/lambda/latest/dg/tutorial-scheduled-events-schedule-expressions.html
         """
+
+        # The two stream sources - DynamoDB and Kinesis - are working differently than the other services (pull vs push)
+        # and do not require event permissions. They do require additional permissions on the Lambda roles though.
+        # http://docs.aws.amazon.com/lambda/latest/dg/lambda-api-permissions-ref.html
+        pull_services = ['dynamodb', 'kinesis']
+
+
         # XXX: Not available in Lambda yet.
         # We probably want to execute the latest code.
         # if default:
         #     lambda_arn = lambda_arn + ":$LATEST"
-        self.unschedule_events(lambda_name=lambda_name, lambda_arn=lambda_arn, events=events)
+
+        self.unschedule_events(lambda_name=lambda_name, lambda_arn=lambda_arn, events=events, excluded_source_services=pull_services)
         for event in events:
             function = event['function']
             expression = event.get('expression', None)
@@ -1370,16 +1378,18 @@ def schedule_events(self, lambda_arn, lambda_name, events, default=True):
                 else:
                     print("Problem scheduling {}.".format(name))
 
-            else:
-
-                svc = ','.join(event['event_source']['events'])
-                service = svc.split(':')[0]
+            elif event_source:
+                service = self.service_from_arn(event_source['arn'])
 
-                self.create_event_permission(
-                    lambda_name,
-                    service + '.amazonaws.com',
-                    event['event_source']['arn']
-                )
+                if service not in pull_services:
+                    svc = ','.join(event['event_source']['events'])
+                    self.create_event_permission(
+                        lambda_name,
+                        service + '.amazonaws.com',
+                        event['event_source']['arn']
+                    )
+                else:
+                    svc = service
 
                 rule_response = add_event_source(
                     event_source,
@@ -1387,9 +1397,18 @@ def schedule_events(self, lambda_arn, lambda_name, events, default=True):
                     function,
                     self.boto_session
                 )
-                # # if rule_response: # Kappa doesn't give us this yet.
-                # So, we print as if was sucessful.
-                print("Created %s event schedule for %s!" % (svc, function))
+
+                if rule_response == 'successful':
+                    print("Created {} event schedule for {}!".format(svc, function))
+                elif rule_response == 'failed':
+                    print("Problem creating {} event schedule for {}!".format(svc, function))
+                elif rule_response == 'exists':
+                    print("{} event schedule for {} already exists - Nothing to do here.".format(svc, function))
+                elif rule_response == 'dryrun':
+                    print("Dryrun for creating {} event schedule for {}!!".format(svc, function))
+            else:
+                print("Could not create event {} - Please define either an expression or an event source".format(name))
+
 
     @staticmethod
     def get_scheduled_event_name(event, function, lambda_name):
@@ -1443,7 +1462,8 @@ def get_event_rules_for_lambda(self, lambda_name):
         rules = [r['Name'] for r in self.events_client.list_rules(NamePrefix=lambda_name)['Rules']]
         return [self.events_client.describe_rule(Name=r) for r in rules]
 
-    def unschedule_events(self, events, lambda_arn=None, lambda_name=None):
+    def unschedule_events(self, events, lambda_arn=None, lambda_name=None, excluded_source_services=None):
+        excluded_source_services = excluded_source_services or []
         """
         Given a list of events, unschedule these CloudWatch Events.
 
@@ -1466,13 +1486,18 @@ def unschedule_events(self, events, lambda_arn=None, lambda_name=None):
             function = event['function']
             name = event.get('name', function)
             event_source = event.get('event_source', function)
-            remove_event_source(
-                event_source,
-                lambda_arn,
-                function,
-                self.boto_session
-            )
-            print("Removed event " + name + ".")
+            service = self.service_from_arn(event_source['arn'])
+            # DynamoDB and Kinesis streams take quite a while to setup after they are created and do not need to be
+            # re-scheduled when a new Lambda function is deployed. Therefore, they should not be removed during zappa
+            # update or zappa schedule.
+            if service not in excluded_source_services:
+                remove_event_source(
+                    event_source,
+                    lambda_arn,
+                    function,
+                    self.boto_session
+                )
+                print("Removed event " + name + ".")
 
     def _clear_policy(self, lambda_name):
         """
@@ -1663,3 +1688,7 @@ def human_size(self, num, suffix='B'):
                 return "{0:3.1f}{1!s}{2!s}".format(num, unit, suffix)
             num /= 1024.0
         return "{0:.1f}{1!s}{2!s}".format(num, 'Yi', suffix)
+
+    @staticmethod
+    def service_from_arn(arn):
+        return arn.split(':')[2]


Test patch: