Repository: Sage-Bionetworks/synapse-docker-registry
Pull request #13: PLFM-8947
Author: login: brucehoff
Created: 2025-05-04T21:42:25Z | Merged: 2025-05-06T22:04:24Z

Add logging of [registry event notifications](https://distribution.github.io/distribution/about/notifications/) to Cloudwatch Logs, since event notifications are not currently logged and we feel the information would be useful for sleuthing a problem which has recently occurred.

Since the registry requires a URL for its logging endpoint we create a (private) API Gateway.  We then define a Lambda to write the event content to a log group.


Diff:
diff --git a/README.md b/README.md
index 861fb83..46d9654 100644
--- a/README.md
+++ b/README.md
@@ -12,7 +12,7 @@ In dev the secret is named `registry-dev-DockerFargateStack/dev/ecs` and in the
 
 A secret is a collection of key-value pairs.  For this application there are two pairs.  The key for the first should be `notification_auth` and the value is the
 Base64 encoded "Basic auth" credentials which are a shared-secret with Synapse as the event notification recipient.
-The key for the second should be 'http_secret' and the value is a cryptogrphically generated string for use by the
+The key for the second should be 'http_secret' and the value is a cryptographically generated string for use by the
 server as described [here](https://distribution.github.io/distribution/about/configuration/).
 
 ### Registry container
diff --git a/app.py b/app.py
index c107e64..852a002 100644
--- a/app.py
+++ b/app.py
@@ -12,7 +12,7 @@
   raise SystemExit(err)
 
 vpc_stack = VpcStack(app, context, app_config)
-docker_fargate_stack = DockerFargateStack(app, context, app_config, vpc=vpc_stack.vpc)
+docker_fargate_stack = DockerFargateStack(app, context, app_config, vpc=vpc_stack.vpc, vpc_endpoint=vpc_stack.vpc_endpoint)
 docker_fargate_stack.add_dependency(vpc_stack)
 
 app.synth()
diff --git a/common/vpc_stack.py b/common/vpc_stack.py
index 78b8af8..e91e708 100644
--- a/common/vpc_stack.py
+++ b/common/vpc_stack.py
@@ -18,6 +18,13 @@ def __init__(self, scope: Construct, context: str, env: dict, **kwargs) -> None:
                            cidr=env.get(VPC_CIDR_CONTEXT),
                            max_azs=2)
 
+        # Create a VPE Endpoint to let the registry reach API Gateway
+        self.vpc_endpoint = ec2.InterfaceVpcEndpoint(self, f'{stack_id}-VpcEndpoint',
+            vpc=self.vpc,
+            service=ec2.InterfaceVpcEndpointService(f"com.amazonaws.{self.region}.execute-api"),
+            private_dns_enabled=True,
+            subnets=ec2.SubnetSelection())
+
         # Tag all resources in this Stack's scope with context tags
         for key, value in env.get(config.TAGS_CONTEXT).items():
             Tags.of(scope).add(key, value)
diff --git a/docker_fargate/docker_fargate_stack.py b/docker_fargate/docker_fargate_stack.py
index 70037a1..3c5fe2c 100644
--- a/docker_fargate/docker_fargate_stack.py
+++ b/docker_fargate/docker_fargate_stack.py
@@ -5,7 +5,10 @@
     aws_ecs_patterns as ecs_patterns,
     aws_elasticloadbalancingv2 as elbv2,
     aws_route53 as r53,
+    aws_apigateway as apigateway,
     aws_iam as iam,
+    aws_lambda,
+    aws_logs as logs,
     CfnOutput,
     Duration,
     Tags)
@@ -56,7 +59,7 @@ def get_port(env: dict) -> int:
 
 class DockerFargateStack(Stack):
 
-    def __init__(self, scope: Construct, context: str, env: dict, vpc: ec2.Vpc, **kwargs) -> None:
+    def __init__(self, scope: Construct, context: str, env: dict, vpc: ec2.Vpc, vpc_endpoint: ec2.InterfaceVpcEndpoint, **kwargs) -> None:
         stack_prefix = f'{env.get(config.STACK_NAME_PREFIX_CONTEXT)}'
         stack_id = f'{stack_prefix}-DockerFargateStack'
         super().__init__(scope, stack_id, **kwargs)
@@ -85,6 +88,9 @@ def __init__(self, scope: Construct, context: str, env: dict, vpc: ec2.Vpc, **kw
         # give the user S3 access
         bucket.grant_read_write(user)
 
+        # create an APIGateway that logs registry events to Cloudwatch Logs
+        api_url = create_logging_apigateway(self, stack_prefix, stack_id, vpc_endpoint)
+
         cluster = ecs.Cluster(
             self,
             f'{stack_id}-Cluster',
@@ -104,6 +110,7 @@ def __init__(self, scope: Construct, context: str, env: dict, vpc: ec2.Vpc, **kw
         env_vars = get_container_env(env)
         env_vars[BUCKET_NAME]=bucket_name
         env_vars["AWS_ACCESS_KEY_ID"]=access_key.access_key_id
+        env_vars["api_gateway_url"]=api_url
 
         # Build the container image for the registry
         # Need self-signed certificates to add to the image
@@ -196,3 +203,71 @@ def __init__(self, scope: Construct, context: str, env: dict, vpc: ec2.Vpc, **kw
         lb_dns_name = load_balanced_fargate_service.load_balancer.load_balancer_dns_name
         lb_dns_export_name = f'{stack_id}-LoadBalancerDNS'
         CfnOutput(self, 'LoadBalancerDNS', value=lb_dns_name, export_name=lb_dns_export_name)
+
+#
+# Create an API Gateway that the registry can call by its URL
+# to log events to CloudWatch Logs
+#
+def create_logging_apigateway(self, stack_prefix, stack_id, vpc_endpoint):
+    # Create a policy to allow invoking the API Gateway
+    # Note that the Gateway is only accessible within the VPC
+    gateway_resource_policy=iam.PolicyDocument(
+        statements=[
+        iam.PolicyStatement(
+            actions =['execute-api:Invoke'],
+            principals = [iam.StarPrincipal()],
+            resources = ['*']
+        )]
+    )
+
+    # Create the log group & stream to receive the event logs
+    log_group_name = f"{stack_id}-execution-logs"
+    log_group = logs.LogGroup(self, log_group_name, retention=logs.RetentionDays.SIX_MONTHS)
+    log_stream = logs.LogStream(self, f"{stack_id}-log-stream", log_group=log_group)
+    CfnOutput(self, 'LogGroup', value=log_group.log_group_name, export_name=log_group_name)
+
+    # Define the code for the lambda, in-line
+    # We simply log the event to Cloudwatch Logs
+    lambda_code = f"""
+import boto3, json, time
+client = boto3.client('logs')
+def handler(event, context):
+    headers = event.get('multiValueHeaders',{{}})
+    body = json.loads(event.get('body',{{}}))
+    content_to_log={{'headers':headers,'body':body}}
+    message = json.dumps(content_to_log)
+    milliseconds = int(round(time.time() * 1000))
+    client.put_log_events(
+        logGroupName='{log_group.log_group_name}',
+        logStreamName='{log_stream.log_stream_name}',
+        logEvents=[{{'timestamp':milliseconds,'message':message}}])
+    return {{'statusCode': 204}}
+"""
+
+    # Define the lambda function that runs the code
+    lambda_function = aws_lambda.Function(self, "Function",
+        runtime=aws_lambda.Runtime.PYTHON_3_9,
+        handler="index.handler",
+        code=aws_lambda.InlineCode(lambda_code)
+    )
+
+    # Create a policy to allow the lambda to put logs to Cloudwatch Logs
+    lambda_function.add_to_role_policy(
+        iam.PolicyStatement(
+            actions=["logs:*"],
+            resources=[log_group.log_group_arn]
+        )
+    )
+
+    # Create the Lambda-integrated API Gateway
+    api = apigateway.LambdaRestApi(self,
+        f'{stack_prefix}-events-collector',
+        handler=lambda_function,
+        endpoint_configuration=apigateway.EndpointConfiguration(
+            types=[apigateway.EndpointType.PRIVATE],
+            vpc_endpoints=[vpc_endpoint]),
+        policy=gateway_resource_policy
+    )
+    # the URL for this gateway is api.url
+
+    return api.url
diff --git a/resources/dev/config.yml b/resources/dev/config.yml
index b74b77e..5794f73 100644
--- a/resources/dev/config.yml
+++ b/resources/dev/config.yml
@@ -47,3 +47,9 @@ notifications:
       timeout: 1m
       threshold: 1
       backoff: 30s
+    - name: LoggingApiGateway
+      disabled: false
+      url: api_gateway_url
+      timeout: 1m
+      threshold: 1
+      backoff: 30s
diff --git a/resources/prod/config.yml b/resources/prod/config.yml
index ee57895..db425a5 100644
--- a/resources/prod/config.yml
+++ b/resources/prod/config.yml
@@ -47,3 +47,9 @@ notifications:
       timeout: 1m
       threshold: 1
       backoff: 30s
+    - name: LoggingApiGateway
+      disabled: false
+      url: api_gateway_url
+      timeout: 1m
+      threshold: 1
+      backoff: 30s
diff --git a/startup.sh b/startup.sh
index d784188..d2f499f 100755
--- a/startup.sh
+++ b/startup.sh
@@ -3,17 +3,24 @@
 # Inject http_secret into config.yml
 # The value is taken from the environment variable, `http_secret` which,
 # during ECS deployment comes from the AWS Secrets Manager.
-sed -i "s/http_secret/$http_secret/g" /etc/docker/registry/config.yml
+sed -i "s|http_secret|$http_secret|g" /etc/docker/registry/config.yml
 
 # Inject notification listener authorization credentials into config.yml
-# The value is taken from the environment variable, `notification_auth` which,
+# The value is taken from the environment variable, `notification_auth` which
 # during ECS deployment comes from the AWS Secrets Manager.
-sed -i "s/notification_auth/$notification_auth/g" /etc/docker/registry/config.yml
+sed -i "s|notification_auth|$notification_auth|g" /etc/docker/registry/config.yml
+
+# Inject api_gateway_url into config.yml
+# The value is taken from the environment variable, `api_gateway_url` which
+# is generated by CDK during ECS deployment.
+# Note we can't use the standard '/' for the sed separator because
+# the URL contains that character.
+sed -i "s|api_gateway_url|$api_gateway_url|g" /etc/docker/registry/config.yml
 
 # Inject bucket name into config.yml
 # The value is taken from the environment variable, `BUCKET_NAME` which,
 # during ECS deployment comes from the cdk.json file.
-sed -i "s/bucket_name/$BUCKET_NAME/g" /etc/docker/registry/config.yml
+sed -i "s|bucket_name|$BUCKET_NAME|g" /etc/docker/registry/config.yml
 
 # this assumed a particular start-up for the container registry
 # if the command changes in future versions, this will have to be updated too


Reviews:
user: login: marcomarasca
body: See comments
state: CHANGES_REQUESTED
author_association: NONE
submitted_at: 2025-05-05T21:02:11Z
user: login: marcomarasca
state: COMMENTED
author_association: NONE
submitted_at: 2025-05-05T21:03:43Z
user: login: brucehoff
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T15:29:18Z
user: login: brucehoff
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T15:43:34Z
user: login: marcomarasca
state: APPROVED
author_association: NONE
submitted_at: 2025-05-06T21:57:18Z

Review comments:
diff_hunk: @@ -196,3 +203,88 @@ def __init__(self, scope: Construct, context: str, env: dict, vpc: ec2.Vpc, **kw
         lb_dns_name = load_balanced_fargate_service.load_balancer.load_balancer_dns_name
         lb_dns_export_name = f'{stack_id}-LoadBalancerDNS'
         CfnOutput(self, 'LoadBalancerDNS', value=lb_dns_name, export_name=lb_dns_export_name)
+
+#
+# Create an API Gateway that the registry can call by its URL
+# to log events to CloudWatch Logs
+#
+def create_logging_apigateway(self, stack_prefix, stack_id, vpc):
+    # Create a VPE Endpoint to let the registry reach API Gateway
+    vpc_endpoint = ec2.InterfaceVpcEndpoint(self, f'{stack_id}-VpcEndpoint',
path: docker_fargate/docker_fargate_stack.py
user: login: marcomarasca
body: To double check: The VPC endpoint for a given service is unique for the VPC/region/subnet triple. So, it would more appropriate to create it where the VPC is created.
created_at: 2025-05-05T20:46:34Z
author_association: NONE
diff_hunk: @@ -196,3 +203,88 @@ def __init__(self, scope: Construct, context: str, env: dict, vpc: ec2.Vpc, **kw
         lb_dns_name = load_balanced_fargate_service.load_balancer.load_balancer_dns_name
         lb_dns_export_name = f'{stack_id}-LoadBalancerDNS'
         CfnOutput(self, 'LoadBalancerDNS', value=lb_dns_name, export_name=lb_dns_export_name)
+
+#
+# Create an API Gateway that the registry can call by its URL
+# to log events to CloudWatch Logs
+#
+def create_logging_apigateway(self, stack_prefix, stack_id, vpc):
+    # Create a VPE Endpoint to let the registry reach API Gateway
+    vpc_endpoint = ec2.InterfaceVpcEndpoint(self, f'{stack_id}-VpcEndpoint',
+        vpc=vpc,
+        service=ec2.InterfaceVpcEndpointService(f"com.amazonaws.{self.region}.execute-api"),
+        private_dns_enabled=True,
+        subnets=ec2.SubnetSelection())
+
+    # Create a policy to allow invoking the API Gateway
+    # Note that the Gateway is only accessible within the VPC
+    gateway_resource_policy=iam.PolicyDocument(
+        statements=[
+        iam.PolicyStatement(
+            actions =['execute-api:Invoke'],
+            principals = [iam.StarPrincipal()],
+            resources = ['*']
+        )]
+    )
+
+    # Create the API Gateway REST API
+    api = apigateway.RestApi(self,
+        f'{stack_prefix}-events-collector',
+        endpoint_configuration=apigateway.EndpointConfiguration(
+            types=[apigateway.EndpointType.PRIVATE],
+            vpc_endpoints=[vpc_endpoint]),
+        policy=gateway_resource_policy,
+        deploy=True,
+        deploy_options=apigateway.StageOptions(
+            logging_level=apigateway.MethodLoggingLevel.INFO,
+            data_trace_enabled=True
path: docker_fargate/docker_fargate_stack.py
user: login: marcomarasca
body: We do not need this
created_at: 2025-05-05T20:48:54Z
author_association: NONE
diff_hunk: @@ -196,3 +203,88 @@ def __init__(self, scope: Construct, context: str, env: dict, vpc: ec2.Vpc, **kw
         lb_dns_name = load_balanced_fargate_service.load_balancer.load_balancer_dns_name
         lb_dns_export_name = f'{stack_id}-LoadBalancerDNS'
         CfnOutput(self, 'LoadBalancerDNS', value=lb_dns_name, export_name=lb_dns_export_name)
+
+#
+# Create an API Gateway that the registry can call by its URL
+# to log events to CloudWatch Logs
+#
+def create_logging_apigateway(self, stack_prefix, stack_id, vpc):
+    # Create a VPE Endpoint to let the registry reach API Gateway
+    vpc_endpoint = ec2.InterfaceVpcEndpoint(self, f'{stack_id}-VpcEndpoint',
+        vpc=vpc,
+        service=ec2.InterfaceVpcEndpointService(f"com.amazonaws.{self.region}.execute-api"),
+        private_dns_enabled=True,
+        subnets=ec2.SubnetSelection())
+
+    # Create a policy to allow invoking the API Gateway
+    # Note that the Gateway is only accessible within the VPC
+    gateway_resource_policy=iam.PolicyDocument(
+        statements=[
+        iam.PolicyStatement(
+            actions =['execute-api:Invoke'],
+            principals = [iam.StarPrincipal()],
+            resources = ['*']
+        )]
+    )
+
+    # Create the API Gateway REST API
+    api = apigateway.RestApi(self,
path: docker_fargate/docker_fargate_stack.py
user: login: marcomarasca
body: Why the rest and not the http?
created_at: 2025-05-05T20:49:12Z
author_association: NONE
diff_hunk: @@ -196,3 +203,88 @@ def __init__(self, scope: Construct, context: str, env: dict, vpc: ec2.Vpc, **kw
         lb_dns_name = load_balanced_fargate_service.load_balancer.load_balancer_dns_name
         lb_dns_export_name = f'{stack_id}-LoadBalancerDNS'
         CfnOutput(self, 'LoadBalancerDNS', value=lb_dns_name, export_name=lb_dns_export_name)
+
+#
+# Create an API Gateway that the registry can call by its URL
+# to log events to CloudWatch Logs
+#
+def create_logging_apigateway(self, stack_prefix, stack_id, vpc):
+    # Create a VPE Endpoint to let the registry reach API Gateway
+    vpc_endpoint = ec2.InterfaceVpcEndpoint(self, f'{stack_id}-VpcEndpoint',
+        vpc=vpc,
+        service=ec2.InterfaceVpcEndpointService(f"com.amazonaws.{self.region}.execute-api"),
+        private_dns_enabled=True,
+        subnets=ec2.SubnetSelection())
+
+    # Create a policy to allow invoking the API Gateway
+    # Note that the Gateway is only accessible within the VPC
+    gateway_resource_policy=iam.PolicyDocument(
+        statements=[
+        iam.PolicyStatement(
+            actions =['execute-api:Invoke'],
+            principals = [iam.StarPrincipal()],
+            resources = ['*']
+        )]
+    )
+
+    # Create the API Gateway REST API
+    api = apigateway.RestApi(self,
+        f'{stack_prefix}-events-collector',
+        endpoint_configuration=apigateway.EndpointConfiguration(
+            types=[apigateway.EndpointType.PRIVATE],
+            vpc_endpoints=[vpc_endpoint]),
+        policy=gateway_resource_policy,
+        deploy=True,
+        deploy_options=apigateway.StageOptions(
+            logging_level=apigateway.MethodLoggingLevel.INFO,
+            data_trace_enabled=True
+        )
+    )
+    # the URL for this gateway is api.url
+
+    # Create the log group & stream to receive the event logs
+    log_group_name = f"{stack_id}-execution-logs"
+    log_group = logs.LogGroup(self, log_group_name)
+    log_stream = logs.LogStream(self, f"{stack_id}-log-stream", log_group=log_group)
+    CfnOutput(self, 'LogGroup', value=log_group.log_group_name, export_name=log_group_name)
+
+    # Define the code for the lambda, in-line
+    # We simply log the event to Cloudwatch Logs
+    lambda_code = f"""
+import boto3, json, time
+def handler(event, context):
+    client = boto3.client('logs')
path: docker_fargate/docker_fargate_stack.py
user: login: marcomarasca
body: Double check if this is ok to do in a lambda
created_at: 2025-05-05T20:56:07Z
author_association: NONE
diff_hunk: @@ -196,3 +203,88 @@ def __init__(self, scope: Construct, context: str, env: dict, vpc: ec2.Vpc, **kw
         lb_dns_name = load_balanced_fargate_service.load_balancer.load_balancer_dns_name
         lb_dns_export_name = f'{stack_id}-LoadBalancerDNS'
         CfnOutput(self, 'LoadBalancerDNS', value=lb_dns_name, export_name=lb_dns_export_name)
+
+#
+# Create an API Gateway that the registry can call by its URL
+# to log events to CloudWatch Logs
+#
+def create_logging_apigateway(self, stack_prefix, stack_id, vpc):
+    # Create a VPE Endpoint to let the registry reach API Gateway
+    vpc_endpoint = ec2.InterfaceVpcEndpoint(self, f'{stack_id}-VpcEndpoint',
+        vpc=vpc,
+        service=ec2.InterfaceVpcEndpointService(f"com.amazonaws.{self.region}.execute-api"),
+        private_dns_enabled=True,
+        subnets=ec2.SubnetSelection())
+
+    # Create a policy to allow invoking the API Gateway
+    # Note that the Gateway is only accessible within the VPC
+    gateway_resource_policy=iam.PolicyDocument(
+        statements=[
+        iam.PolicyStatement(
+            actions =['execute-api:Invoke'],
+            principals = [iam.StarPrincipal()],
+            resources = ['*']
+        )]
+    )
+
+    # Create the API Gateway REST API
+    api = apigateway.RestApi(self,
+        f'{stack_prefix}-events-collector',
+        endpoint_configuration=apigateway.EndpointConfiguration(
+            types=[apigateway.EndpointType.PRIVATE],
+            vpc_endpoints=[vpc_endpoint]),
+        policy=gateway_resource_policy,
+        deploy=True,
+        deploy_options=apigateway.StageOptions(
+            logging_level=apigateway.MethodLoggingLevel.INFO,
+            data_trace_enabled=True
+        )
+    )
+    # the URL for this gateway is api.url
+
+    # Create the log group & stream to receive the event logs
+    log_group_name = f"{stack_id}-execution-logs"
+    log_group = logs.LogGroup(self, log_group_name)
+    log_stream = logs.LogStream(self, f"{stack_id}-log-stream", log_group=log_group)
+    CfnOutput(self, 'LogGroup', value=log_group.log_group_name, export_name=log_group_name)
+
+    # Define the code for the lambda, in-line
+    # We simply log the event to Cloudwatch Logs
+    lambda_code = f"""
+import boto3, json, time
+def handler(event, context):
+    client = boto3.client('logs')
+    headers = event.get('multiValueHeaders',{{}})
+    body = json.loads(event.get('body',{{}}))
+    content_to_log={{'headers':headers,'body':body}}
+    message = json.dumps(content_to_log)
+    milliseconds = int(round(time.time() * 1000))
+    client.put_log_events(
+        logGroupName='{log_group.log_group_name}',
+        logStreamName='{log_stream.log_stream_name}',
+        logEvents=[{{'timestamp':milliseconds,'message':message}}])
+    return {{'statusCode': 204}}
+"""
+
+    # Define the lambda function that runs the code
+    lambda_function = aws_lambda.Function(self, "Function",
+        runtime=aws_lambda.Runtime.PYTHON_3_9,
path: docker_fargate/docker_fargate_stack.py
user: login: marcomarasca
body: Let us use the latest
created_at: 2025-05-05T20:57:22Z
author_association: NONE
diff_hunk: @@ -47,3 +47,12 @@ notifications:
       timeout: 1m
       threshold: 1
       backoff: 30s
+    - name: LoggingApiGateway
+      disabled: false
+      url: api_gateway_url
+      headers:
+          Content-Type:
+              - application/json
path: resources/dev/config.yml
user: login: marcomarasca
body: Might not be needed
created_at: 2025-05-05T21:01:43Z
author_association: NONE
diff_hunk: @@ -196,3 +203,88 @@ def __init__(self, scope: Construct, context: str, env: dict, vpc: ec2.Vpc, **kw
         lb_dns_name = load_balanced_fargate_service.load_balancer.load_balancer_dns_name
         lb_dns_export_name = f'{stack_id}-LoadBalancerDNS'
         CfnOutput(self, 'LoadBalancerDNS', value=lb_dns_name, export_name=lb_dns_export_name)
+
+#
+# Create an API Gateway that the registry can call by its URL
+# to log events to CloudWatch Logs
+#
+def create_logging_apigateway(self, stack_prefix, stack_id, vpc):
+    # Create a VPE Endpoint to let the registry reach API Gateway
+    vpc_endpoint = ec2.InterfaceVpcEndpoint(self, f'{stack_id}-VpcEndpoint',
+        vpc=vpc,
+        service=ec2.InterfaceVpcEndpointService(f"com.amazonaws.{self.region}.execute-api"),
+        private_dns_enabled=True,
+        subnets=ec2.SubnetSelection())
+
+    # Create a policy to allow invoking the API Gateway
+    # Note that the Gateway is only accessible within the VPC
+    gateway_resource_policy=iam.PolicyDocument(
+        statements=[
+        iam.PolicyStatement(
+            actions =['execute-api:Invoke'],
+            principals = [iam.StarPrincipal()],
+            resources = ['*']
+        )]
+    )
+
+    # Create the API Gateway REST API
+    api = apigateway.RestApi(self,
+        f'{stack_prefix}-events-collector',
+        endpoint_configuration=apigateway.EndpointConfiguration(
+            types=[apigateway.EndpointType.PRIVATE],
+            vpc_endpoints=[vpc_endpoint]),
+        policy=gateway_resource_policy,
+        deploy=True,
+        deploy_options=apigateway.StageOptions(
+            logging_level=apigateway.MethodLoggingLevel.INFO,
+            data_trace_enabled=True
+        )
+    )
+    # the URL for this gateway is api.url
+
+    # Create the log group & stream to receive the event logs
+    log_group_name = f"{stack_id}-execution-logs"
+    log_group = logs.LogGroup(self, log_group_name)
path: docker_fargate/docker_fargate_stack.py
user: login: marcomarasca
body: Might want to specify a retention period (maybe 30 days?)
created_at: 2025-05-05T21:03:36Z
author_association: NONE
diff_hunk: @@ -196,3 +203,88 @@ def __init__(self, scope: Construct, context: str, env: dict, vpc: ec2.Vpc, **kw
         lb_dns_name = load_balanced_fargate_service.load_balancer.load_balancer_dns_name
         lb_dns_export_name = f'{stack_id}-LoadBalancerDNS'
         CfnOutput(self, 'LoadBalancerDNS', value=lb_dns_name, export_name=lb_dns_export_name)
+
+#
+# Create an API Gateway that the registry can call by its URL
+# to log events to CloudWatch Logs
+#
+def create_logging_apigateway(self, stack_prefix, stack_id, vpc):
+    # Create a VPE Endpoint to let the registry reach API Gateway
+    vpc_endpoint = ec2.InterfaceVpcEndpoint(self, f'{stack_id}-VpcEndpoint',
+        vpc=vpc,
+        service=ec2.InterfaceVpcEndpointService(f"com.amazonaws.{self.region}.execute-api"),
+        private_dns_enabled=True,
+        subnets=ec2.SubnetSelection())
+
+    # Create a policy to allow invoking the API Gateway
+    # Note that the Gateway is only accessible within the VPC
+    gateway_resource_policy=iam.PolicyDocument(
+        statements=[
+        iam.PolicyStatement(
+            actions =['execute-api:Invoke'],
+            principals = [iam.StarPrincipal()],
+            resources = ['*']
+        )]
+    )
+
+    # Create the API Gateway REST API
+    api = apigateway.RestApi(self,
+        f'{stack_prefix}-events-collector',
+        endpoint_configuration=apigateway.EndpointConfiguration(
+            types=[apigateway.EndpointType.PRIVATE],
+            vpc_endpoints=[vpc_endpoint]),
+        policy=gateway_resource_policy,
+        deploy=True,
+        deploy_options=apigateway.StageOptions(
+            logging_level=apigateway.MethodLoggingLevel.INFO,
+            data_trace_enabled=True
+        )
+    )
+    # the URL for this gateway is api.url
+
+    # Create the log group & stream to receive the event logs
+    log_group_name = f"{stack_id}-execution-logs"
+    log_group = logs.LogGroup(self, log_group_name)
+    log_stream = logs.LogStream(self, f"{stack_id}-log-stream", log_group=log_group)
+    CfnOutput(self, 'LogGroup', value=log_group.log_group_name, export_name=log_group_name)
+
+    # Define the code for the lambda, in-line
+    # We simply log the event to Cloudwatch Logs
+    lambda_code = f"""
+import boto3, json, time
+def handler(event, context):
+    client = boto3.client('logs')
path: docker_fargate/docker_fargate_stack.py
user: login: brucehoff
body: You are right.  The [best practice](https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtime-environment.html#:~:text=Static%20initialization%20happens%20before%20the,up%20connections%20and%20other%20resources.) is to create the client statically, outside of the handler.
created_at: 2025-05-06T15:29:18Z
author_association: MEMBER
diff_hunk: @@ -196,3 +203,88 @@ def __init__(self, scope: Construct, context: str, env: dict, vpc: ec2.Vpc, **kw
         lb_dns_name = load_balanced_fargate_service.load_balancer.load_balancer_dns_name
         lb_dns_export_name = f'{stack_id}-LoadBalancerDNS'
         CfnOutput(self, 'LoadBalancerDNS', value=lb_dns_name, export_name=lb_dns_export_name)
+
+#
+# Create an API Gateway that the registry can call by its URL
+# to log events to CloudWatch Logs
+#
+def create_logging_apigateway(self, stack_prefix, stack_id, vpc):
+    # Create a VPE Endpoint to let the registry reach API Gateway
+    vpc_endpoint = ec2.InterfaceVpcEndpoint(self, f'{stack_id}-VpcEndpoint',
+        vpc=vpc,
+        service=ec2.InterfaceVpcEndpointService(f"com.amazonaws.{self.region}.execute-api"),
+        private_dns_enabled=True,
+        subnets=ec2.SubnetSelection())
+
+    # Create a policy to allow invoking the API Gateway
+    # Note that the Gateway is only accessible within the VPC
+    gateway_resource_policy=iam.PolicyDocument(
+        statements=[
+        iam.PolicyStatement(
+            actions =['execute-api:Invoke'],
+            principals = [iam.StarPrincipal()],
+            resources = ['*']
+        )]
+    )
+
+    # Create the API Gateway REST API
+    api = apigateway.RestApi(self,
path: docker_fargate/docker_fargate_stack.py
user: login: brucehoff
body: The advice [here](https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_apigateway/README.html#aws-lambda-backed-apis) is to use `LambdaRestApi`.  The original `HttpApi` [seems](https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_apigatewayv2/HttpApi.html) to be deprecated.  I will try switching to `LambdaRestApi`. 
created_at: 2025-05-06T15:43:34Z
author_association: MEMBER

PR comments:
user: login: zaro0508
created_at: 2025-05-05T19:17:43Z
body: > Add logging of registry events to Cloudwatch Logs.

I don't understand what you are trying to do in this PR.  Are the docker registry logs not going into cloudwatch now and this PR is making that happen?  or are you appending something extra logs to existing cloudwatch logs?  Can you please provide a little more explanation?
author_association: MEMBER
user: login: brucehoff
created_at: 2025-05-05T20:02:07Z
body: > > Add logging of registry events to Cloudwatch Logs.
> 
> I don't understand what you are trying to do in this PR. Are the docker registry logs not going into cloudwatch now and this PR is making that happen? or are you appending something extra logs to existing cloudwatch logs? Can you please provide a little more explanation?

Docker registry logs are not going into cloudwatch now, and this PR is making that happen.
author_association: MEMBER
user: login: zaro0508
created_at: 2025-05-05T22:52:10Z
body: > Since the registry requires a URL for its logging endpoint we create a (private) API Gateway. We then define a Lambda to write the event content to a log group.

This is one approach however there's a much simpler, ECS centric approach.  ECS already does most of the work of sending logs to cloudwatch.  In this specific case you want to send the container's application logs to cloudwatch.  Therefore I think all you need to do is tell ECS where the log files are and it should stream the contents of the file to cloudwatch.  For the docker distribution container it's gonna be at `/var/lib/docker/containers/XXXXXX-json.log`.  Take a look at these documentation: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html and https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-logs-collector.html
 
author_association: MEMBER
user: login: brucehoff
created_at: 2025-05-06T00:13:23Z
body: > > Since the registry requires a URL for its logging endpoint we create a (private) API Gateway. We then define a Lambda to write the event content to a log group.
> 
> This is one approach however there's a much simpler, ECS centric approach. ECS already does most of the work of sending logs to cloudwatch. In this specific case you want to send the container's application logs to cloudwatch. Therefore I think all you need to do is tell ECS where the log files are and it should stream the contents of the file to cloudwatch. For the docker distribution container it's gonna be at `/var/lib/docker/containers/XXXXXX-json.log`. Take a look at these documentation: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html and https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-logs-collector.html


Thanks for the suggestion, but I think there is a misunderstanding: The information being logged are not "application logs" (i.e., text written to stdio or stderr).  We do indeed log that information to CloudWatch logs already. This information is JSON formatted notifications indicating to registered listeners when key events occur, like `docker push` and `docker pull`.  The events are only sent via HTTP POST.  So I don't think ECS's logging can help with the problem.  Rather we need an HTTP endpoint to send the logs to.  For more details see https://distribution.github.io/distribution/about/notifications/
author_association: MEMBER