Service observability
By: Priya Aggarwal
Tags: Metrics, Services, Service Design

Development for your service is complete. It’s been tested well for its functionality and all tests pass. It is ready for its first release in a production environment and then in sometime it is released. And, so, it is running in production and is serving requests and all is going well. Well, it seems all is going well… no customers have complained yet, so, everything should be alright…right?

It is such a popular and mistaken belief that “if customer isn’t complaining then our service must be very good”. You may not know how many loyal customers may have stopped using your service or some may be compromising with their system performance by using it just because your service is their only option.

You/your team are/is a service provider, you are the owner of it so you get to choose how good quality and performant you want your service to be for your customers. And, whatever is your service’s quality and performance objective, you should have enough data to know if your service is meeting those expectations. This service’s quality and performance objective is also known as Service Level Objective (SLO).

Service’s quality and performance objectives — what are they? These are metric values that indicate service non-functional behaviour goal like Availability, Reliability, Efficiency, etc. The service is a process which, just like all other processes, is often not running is isolation. And so, there are many things which can affect a service’s performance and quality. Some of those things are available memory, CPU usage, external dependencies, connections, threads, etc.

To have our service meet its quality & performance goal, keep a check on all things which can affect it becomes essential. We can keep a check on these things by monitoring the various metrics which show state of system, service, dependencies, etc and can create charts which record metrics state change with time. These metrics are also called Service Level Indicators (SLI)

There is an overwhelming number of available metrics..which ones should you record and monitor is very important to understand. Start this activity by thinking about whose point of interest are you exploring. There are many stakeholders in a service or system like — Product manager, architect, developer, etc and they like to have different view of metrics for the service. Let’s start by exploring few of those views.

Customer’s view

Just tell me when will it work again!

Both customers and product managers and all other kinds of managers are mostly interested in knowing two things — is it working? and is it working well?

For them, you need to have following two metric views available

Summary view

In this view, show numbers only, no graphs/charts, just numbers for performance in last 1 week, last 1 month, etc. Capture information about service uptime, success rate or requests served (critical and important APIs only) and top percentile(tp) 99 of API response (critical and important APIs only).

Detailed view

In this view, we need detailed view of each API performance within service. It is generally a view with information for last hour or so. For each API, have charts for following things

Success Rate (2xx. response rate) Response time — tp99, tp90 and tp50 Volume of requests per unit time Count of error response by error type

Architect’s view

Have you checked all dependencies and points of interactions?

Architects and system designers have tendency to want to look into following aspects about service

How many requests are coming? What is the average response time? Which service, database, cache or other dependencies this service has?

For them, you would need to cover all of the things external to service that can impact its performance.

Summary view

This should have charts for following things:

Incoming requests count (measured by using load balancer stats) Average response time (measured by using load balancer stats) Outgoing requests count Outgoing requests’ average response time

Detailed view

Once they have seen summary metrics, in case of anomalies, they would want to get deeper level metrics. For deep level metrics, dashboards should capture all service interactions.

For incoming request traffic metrics, the detailed view for Customers can be used. For outgoing request traffic metrics, capture following things

Success Rate Response time — tp99, tp90 and tp50 Volume of requests per unit time Timeouts — connection and read Count of error response by error type

Developer’s view

There is no change in incoming traffic, I don’t know what’s wrong. Let me look into some detailed metrics and logging to find out.

Developers need to know the service in and out to be able to troubleshoot issues. They need to have metrics available for so many different things. Let’s consider that this service is a JVM based service with few dependencies like services, database, caches (cache as a service) and few internal resource intensive things like in-memory cache. For an application like this, they would need to know details of following things

JVM

Create charts to know metrics like heap memory usage, jvm CPU usage, GC frequency and time, threads state

Tomcat/Jetty/Undertow

You can capture metrics for state of worker threads, request acceptor threads, request queue.

Host and process

Know the system metrics by plotting charts for host’s RAM usage, CPU usage, network sent, received bytes, number of cores available. Process metrics like number of files opened and number of files process is allowed to open.

Here having knowledge of sidecar processes is good practice too.

Http connection pool (and queues)

Using http connection pool is a common practice for http API calls. Knowing the number of connections by each downstream and number of connection requests waiting in queue can help identify and network/downstream issue.

Database connection pool (and queues)

Creating database connections is an expensive operation, all db driver libraries use database connection pools to improve performance. Create charts to know current number of connections and connection requests waiting in queue.

ThreadPool (and queues)

If your service have things executing asynchronously or in parallel, then it would most likely be using a threadPool with queue. Capture metrics for the thread pool current size, jobs in queue, etc.

Collection/ Map/In-memory cache

If there are any collections — list, set, map, etc in application, that are expected to grow or shrink over time, then keep an eye on them by having metrics for size, put and remove actions.

That’s all I had to say about service metrics. Let me know your comments and suggestions for improving service observability.