Repository: localstack/localstack

Issue:
Lambda with self-managed kafka cluster
<!-- Love localstack? Please consider supporting our collective:
👉  https://opencollective.com/localstack/donate -->

# Type of request: This is a ...

[ ] bug report
[ x ] feature request

# Detailed description
I am trying to set up a localstack lambda function that has a locally-run kafka cluster as its event source. I am using aws cli v2. When I run the command to create event source mapping for my lambda function using aws cli, I get a "500 Internal Server Error".
...

## Expected behavior
A new lambda event source mapping should be created for my local kafka cluster.
...

## Actual behavior
Getting an Internal Server Error on aws cli output and getting a "KeyError: EventSourceArn" on localstack logs.
...

# Steps to reproduce
1. Created a lambda handler in python.
    ```python 3
    import json

    def handler(event, context):
        body = {
            "message": str(event)
        }
        response = {
            "statusCode": 200,
            "body": json.dumps(body)
        }
        return response

2. Created my lambda function.
    ```
    $  aws lambda create-function \
        --function-name fun1 \
        --handler lambda.handler \
        --runtime python3.8 \
        --role r1 \
        --zip-file fileb://l1.zip \
        --endpoint-url http://localhost:4566
    ```

3. Created a secret.
    ```
    aws secretsmanager create-secret --name localstack --endpoint-url http://localhost:4566
    ```

4. Created a kafka cluster and a topic called `t1` on my local system using the [official kafka binary](https://www.apache.org/dyn/closer.cgi?path=/kafka/2.7.0/kafka_2.13-2.7.0.tgz). It works without problems.

5. Tried to create event source mapping using the command mentioned [here](https://docs.aws.amazon.com/lambda/latest/dg/services-smaa-topic-add.html) in the official AWS website.
    ```
   $ aws lambda create-event-source-mapping \
      --topics t1 \
      --source-access-configuration Type=SASL_SCRAM_512_AUTH,URI=arn:aws:secretsmanager:us-east-1:000000000000:secret:localstack-TDIuI \
      --function-name arn:aws:lambda:us-east-1:000000000000:function:fun1 \
      --self-managed-event-source '{"Endpoints":{"KAFKA_BOOTSTRAP_SERVERS":["localhost:9092"]}}' \
      --endpoint-url http://localhost:4566
    ```
    
    Here, aws cli throws the following error message
    ```
    An error occurred (500) when calling the CreateEventSourceMapping operation (reached max retries: 2): <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
    <title>500 Internal Server Error</title>
    <h1>Internal Server Error</h1>
    <p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>
    ```

    And the localstack log gives the following error message
    ```
    localstack_main | 2021-01-30T13:32:54:ERROR:lambda_api: Exception on /2015-03-31/event-source-mappings/ [POST]
    localstack_main | Traceback (most recent call last):
    localstack_main |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    localstack_main |     response = self.full_dispatch_request()
    localstack_main |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    localstack_main |     rv = self.handle_user_exception(e)
    localstack_main |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    localstack_main |     return cors_after_request(app.make_response(f(*args, **kwargs)))
    localstack_main |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    localstack_main |     reraise(exc_type, exc_value, tb)
    localstack_main |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    localstack_main |     raise value
    localstack_main |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    localstack_main |     rv = self.dispatch_request()
    localstack_main |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    localstack_main |     return self.view_functions[rule.endpoint](**req.view_args)
    localstack_main |   File "/opt/code/localstack/localstack/services/awslambda/lambda_api.py", line 1457, in create_event_source_mapping
    localstack_main |     data['FunctionName'], data['EventSourceArn'], data.get('Enabled'), data.get('BatchSize')
    localstack_main | KeyError: 'EventSourceArn'
    ```

## Command used to start LocalStack
Copied the contents of docker-compose.yml from localstack repo as-is and ran the following command:
```
docker-compose up
```
...

## Client code (AWS SDK code snippet, or sequence of "awslocal" commands)
Mentioned above.
...



Discussion:

Resolving patch:
diff --git a/localstack/services/awslambda/lambda_api.py b/localstack/services/awslambda/lambda_api.py
--- a/localstack/services/awslambda/lambda_api.py
+++ b/localstack/services/awslambda/lambda_api.py
@@ -53,6 +53,7 @@
 FUNCTION_MAX_SIZE = 69905067
 
 BATCH_SIZE_RANGES = {
+    'kafka': (100, 10000),
     'kinesis': (100, 10000),
     'dynamodb': (100, 1000),
     'sqs': (10, 10)
@@ -161,7 +162,9 @@ def func_qualifier(function_name, qualifier=None):
 
 
 def check_batch_size_range(source_arn, batch_size=None):
-    batch_size_entry = BATCH_SIZE_RANGES.get(source_arn.split(':')[2].lower())
+    source = source_arn.split(':')[2].lower()
+    source = 'kafka' if 'secretsmanager' in source else source
+    batch_size_entry = BATCH_SIZE_RANGES.get(source)
     if not batch_size_entry:
         raise ValueError(
             INVALID_PARAMETER_VALUE_EXCEPTION, 'Unsupported event source type'
@@ -185,40 +188,60 @@ def add_function_mapping(lambda_name, lambda_handler, lambda_cwd=None):
     lambda_details.cwd = lambda_cwd or lambda_details.cwd
 
 
-def add_event_source(function_name, source_arn, enabled, batch_size=None):
+def build_mapping_obj(data):
+    mapping = {}
+    function_name = data['FunctionName']
+    enabled = data.get('Enabled', True)
+    batch_size = data.get('BatchSize')
+    mapping['UUID'] = str(uuid.uuid4())
+    mapping['FunctionArn'] = func_arn(function_name)
+    mapping['LastProcessingResult'] = 'OK'
+    mapping['StateTransitionReason'] = 'User action'
+    mapping['LastModified'] = float(time.mktime(datetime.utcnow().timetuple()))
+    mapping['State'] = 'Enabled' if enabled in [True, None] else 'Disabled'
+    if 'SelfManagedEventSource' in data:
+        source_arn = data['SourceAccessConfigurations'][0]['URI']
+        mapping['SelfManagedEventSource'] = data['SelfManagedEventSource']
+        mapping['Topics'] = data['Topics']
+        mapping['SourceAccessConfigurations'] = data['SourceAccessConfigurations']
+    else:
+        source_arn = data['EventSourceArn']
+        mapping['EventSourceArn'] = source_arn
+        mapping['StartingPosition'] = LAMBDA_DEFAULT_STARTING_POSITION
     batch_size = check_batch_size_range(source_arn, batch_size)
-    region = LambdaRegion.get()
+    mapping['BatchSize'] = batch_size
+    return mapping
 
-    mapping = {
-        'UUID': str(uuid.uuid4()),
-        'StateTransitionReason': 'User action',
-        'LastModified': float(time.mktime(datetime.utcnow().timetuple())),
-        'BatchSize': batch_size,
-        'State': 'Enabled' if enabled in [True, None] else 'Disabled',
-        'FunctionArn': func_arn(function_name),
-        'EventSourceArn': source_arn,
-        'LastProcessingResult': 'OK',
-        'StartingPosition': LAMBDA_DEFAULT_STARTING_POSITION
-    }
+
+def add_event_source(data):
+    region = LambdaRegion.get()
+    mapping = build_mapping_obj(data)
     region.event_source_mappings.append(mapping)
     return mapping
 
 
-def update_event_source(uuid_value, function_name, enabled, batch_size):
+def update_event_source(uuid_value, data):
     region = LambdaRegion.get()
-    for m in region.event_source_mappings:
-        if uuid_value == m['UUID']:
+    function_name = data.get('FunctionName') or ''
+    batch_size = None
+    enabled = data.get('Enabled', True)
+    for mapping in region.event_source_mappings:
+        if uuid_value == mapping['UUID']:
             if function_name:
-                m['FunctionArn'] = func_arn(function_name)
-
-            batch_size = check_batch_size_range(m['EventSourceArn'], batch_size or m['BatchSize'])
-
-            m['BatchSize'] = batch_size
-            m['State'] = 'Enabled' if enabled is True else 'Disabled'
-            m['LastModified'] = float(time.mktime(datetime.utcnow().timetuple()))
-
-            return m
-
+                mapping['FunctionArn'] = func_arn(function_name)
+            batch_size = data.get('BatchSize')
+            if 'SelfManagedEventSource' in mapping:
+                batch_size = check_batch_size_range(
+                    mapping['SourceAccessConfigurations'][0]['URI'],
+                    batch_size or mapping['BatchSize'])
+            else:
+                batch_size = check_batch_size_range(mapping['EventSourceArn'], batch_size or mapping['BatchSize'])
+            mapping['State'] = 'Enabled' if enabled in [True, None] else 'Disabled'
+            mapping['LastModified'] = float(time.mktime(datetime.utcnow().timetuple()))
+            mapping['BatchSize'] = batch_size
+            if 'SourceAccessConfigurations' in (mapping and data):
+                mapping['SourceAccessConfigurations'] = data['SourceAccessConfigurations']
+            return mapping
     return {}
 
 
@@ -1461,9 +1484,7 @@ def create_event_source_mapping():
     """
     data = json.loads(to_str(request.data))
     try:
-        mapping = add_event_source(
-            data['FunctionName'], data['EventSourceArn'], data.get('Enabled'), data.get('BatchSize')
-        )
+        mapping = add_event_source(data)
         return jsonify(mapping)
     except ValueError as error:
         error_type, message = error.args
@@ -1479,16 +1500,12 @@ def update_event_source_mapping(mapping_uuid):
             - name: 'request'
               in: body
     """
-    data = json.loads(request.data)
+    data = json.loads(to_str(request.data))
     if not mapping_uuid:
         return jsonify({})
 
-    function_name = data.get('FunctionName') or ''
-    enabled = data.get('Enabled', True)
-    batch_size = data.get('BatchSize')
-
     try:
-        mapping = update_event_source(mapping_uuid, function_name, enabled, batch_size)
+        mapping = update_event_source(mapping_uuid, data)
         return jsonify(mapping)
     except ValueError as error:
         error_type, message = error.args


Test patch:
diff --git a/tests/integration/test_api_gateway.py b/tests/integration/test_api_gateway.py
--- a/tests/integration/test_api_gateway.py
+++ b/tests/integration/test_api_gateway.py
@@ -134,11 +134,12 @@ def test_api_gateway_sqs_integration_with_event_source(self):
 
         # create event source for sqs lambda processor
         self.create_lambda_function(self.TEST_LAMBDA_SQS_HANDLER_NAME)
-        add_event_source(
-            self.TEST_LAMBDA_SQS_HANDLER_NAME,
-            aws_stack.sqs_queue_arn(queue_name),
-            True
-        )
+        event_source_data = {
+            'FunctionName': self.TEST_LAMBDA_SQS_HANDLER_NAME,
+            'EventSourceArn': aws_stack.sqs_queue_arn(queue_name),
+            'Enabled': True
+        }
+        add_event_source(event_source_data)
 
         # generate test data
         test_data = {'spam': 'eggs & beans'}
diff --git a/tests/unit/test_lambda.py b/tests/unit/test_lambda.py
--- a/tests/unit/test_lambda.py
+++ b/tests/unit/test_lambda.py
@@ -11,6 +11,7 @@
 from localstack.utils.aws.aws_models import LambdaFunction
 
 TEST_EVENT_SOURCE_ARN = 'arn:aws:sqs:eu-west-1:000000000000:testq'
+TEST_SECRETSMANANAGER_EVENT_SOURCE_ARN = 'arn:aws:secretsmanager:us-east-1:000000000000:secret:mysecret-kUBhE'
 
 
 class TestLambdaAPI(unittest.TestCase):
@@ -234,6 +235,23 @@ def test_create_event_source_mapping(self):
         self.assertEqual(1, len(eventSourceMappings))
         self.assertEqual('Enabled', eventSourceMappings[0]['State'])
 
+    def test_create_event_source_mapping_self_managed_event_source(self):
+        self.client.post('{0}/event-source-mappings/'.format(lambda_api.PATH_ROOT),
+            data=json.dumps({'FunctionName': 'test-lambda-function',
+                'Topics': ['test'],
+                'SourceAccessConfigurations': [
+                    {'Type': 'SASL_SCRAM_512_AUTH', 'URI': TEST_SECRETSMANANAGER_EVENT_SOURCE_ARN}
+                ],
+                'SelfManagedEventSource': {'Endpoints': {'KAFKA_BOOTSTRAP_SERVERS': ['127.0.0.1:9092']}}
+            }))
+        listResponse = self.client.get('{0}/event-source-mappings/'.format(lambda_api.PATH_ROOT))
+        listResult = json.loads(listResponse.get_data())
+
+        eventSourceMappings = listResult.get('EventSourceMappings')
+
+        self.assertEqual(1, len(eventSourceMappings))
+        self.assertEqual('Enabled', eventSourceMappings[0]['State'])
+
     def test_create_disabled_event_source_mapping(self):
         createResponse = self.client.post('{0}/event-source-mappings/'.format(lambda_api.PATH_ROOT),
                             data=json.dumps({'FunctionName': 'test-lambda-function',
@@ -268,6 +286,30 @@ def test_update_event_source_mapping(self):
 
         self.assertEqual('Disabled', getResult['State'])
 
+    def test_update_event_source_mapping_self_managed_event_source(self):
+        createResponse = self.client.post('{0}/event-source-mappings/'.format(lambda_api.PATH_ROOT),
+            data=json.dumps({'FunctionName': 'test-lambda-function',
+                'Topics': ['test'],
+                'SourceAccessConfigurations': [
+                    {'Type': 'SASL_SCRAM_512_AUTH', 'URI': TEST_SECRETSMANANAGER_EVENT_SOURCE_ARN}
+                ],
+                'SelfManagedEventSource': {'Endpoints': {'KAFKA_BOOTSTRAP_SERVERS': ['127.0.0.1:9092']}},
+                'Enabled': 'true'
+            }))
+        createResult = json.loads(createResponse.get_data())
+
+        putResponse = self.client.put('{0}/event-source-mappings/{1}'.format(lambda_api.PATH_ROOT,
+                        createResult.get('UUID')), data=json.dumps({'Enabled': 'false'}))
+        putResult = json.loads(putResponse.get_data())
+
+        self.assertEqual('Disabled', putResult['State'])
+
+        getResponse = self.client.get('{0}/event-source-mappings/{1}'.format(lambda_api.PATH_ROOT,
+                        createResult.get('UUID')))
+        getResult = json.loads(getResponse.get_data())
+
+        self.assertEqual('Disabled', getResult['State'])
+
     def test_publish_function_version(self):
         with self.app.test_request_context():
             self._create_function(self.FUNCTION_NAME)