Serverless Monitoring Is No Longer “Finding a Needle in a Haystack”
Tags: AWS, Monitoring, Technology, Serverless, AI

At Human AI Labs (hu.man.ai), formerly Luther.ai, we use AWS Serverless Stack and Kubernetes for all the core real-time pipelines, and it is data-driven execution across all the AWS Services — ECS, Lambda, SQS, Fargate, etc.

With hundreds of services with thousands of invocations, every day comes the complexity to configure, execution monitoring, logs review, latency measurements, etc. For configuration and CI/CD, we use serverless for the packaging and deploying Lambda functions.

However, we tried multiple monitoring solutions, including just leveraging AWS native options; however, the scale brings various issues like -

Multiple programming languages are used for AWS Lambda development. Containers / Tasks used in ECS with EC2 and ECS with Fargate. External service access ( outbound API calls ) review, including unauthorized calls. Persistent Storage access review and latency measurements. Unified access to execution logs with options of searchable across with full text and time-based. Contextualization of the service-based view. Proactive notification to places where we work — slack, pagerduty, etc. History of the events, searchable, etc.

Along with the specifics above, the development team wants to focus on the serverless function rather than increase its monitoring footprint, causing lots of worries to on-call DevOps engineers.

After lots of review of various services to solve the issues listed above, Epsagon is the solution we implemented. Here is the journey of how it saved hours and hours for our serverless implementation. Let us break the journey into the installation, monitoring, latency measurements, and notifications.

Installation / Onboarding: If you as a reader and does AWS serverless development and not using lambda layers, you miss a core feature that will help a lot. Auto-tracing for your AWS region lambda functions is enabled with a simple workflow leveraging the lambda layers, thus solving multiple programming languages development dependencies to enable monitoring, including custom logic per serverless function.

Monitoring: Once you have the auto-tracing enabled, proactive monitoring will help you with alerts and notifications. We use native slack and email integration to receive notifications. Each of the notification has the contextual link to the alert where AWS CloudWatch log, the start time of the lambda execution along with the service map of all the services used ( external API calls, AWS Services inbound and outbound) is available as a quick action — handy link to AWS.

If you are like me who are interested in patterns, you can use the historical ( available for the last 7 days) view for all the issues to understand scenarios like if any of the last deployment, any specific user action, and or its scalability caused — I have an interesting issue which we uncovered using historical patterns but for a future blog not now.

Epsagon — Tabular view of all the lambda functions

Latency measurement: With 100’s of services and multi-thousands of executions every day, even a couple of milliseconds, etc., execution time added to one service in the real-time pipeline can result in a bad user experience. With Epsagon, it is efficient to isolate latency issues in multiple facets.

As in the picture above, a unified view across all the functions is available with the Average Execution duration — which is a great start. For each lambda execution, a contextual service map is available with the service calls' time duration. With the Service Map view, we can review all the lambda dependent services as a single landscape with the capability to go back in time and understand the subtleties.

Epsagon — Service Map View example

Notifications: With all the integrated features, we configured extensible alerts — of which we use PagerDuty integration for core functions both for high latency, errors. Also, native integration with Jira helps to document a bug from the tool itself for each of the issues. The contextualization of information captured in the bug is a key feature — no more to worry about, did you took the log capture, what time was the issue, etc.

Did I say that we self-configured from start to finish in a weekend? Yes, we did both for our dev, prod environments. Here are the resources that have proven handy

AWS Workshop — https://epsagon.awsworkshop.io/

Epsagon Docs — https://docs.epsagon.com/

and AWS Activate program — https://aws.amazom.com/activate/

To conclude, serverless deployments, monitoring of workloads with hundreds of services and millions of invocations is no longer a “needle in a haystack.”