The Rise of Open Observability Standards and Predictions for the Future
By: Shivnath Babu
Tags: Metrics, Prometheus, Observability, Opentelemetry, DevOps

The Rise of Open Observability Standards and Predictions for the Future Shivnath Babu Jan 18·4 min read

The past year saw big movements in the evolution and adoption of standards like OpenTelemetry and OpenMetrics. In this article, I will briefly chronicle the rise of these standards and then give my predictions on the future of these standards over the next five years.

You can scroll back and forth in time through the above timeline which ranges over the past two decades. This walk through time looks at the major milestones in the rise of open standards in observability.

The timeline shows how we got here, but the question is: “why did we get here?”

Note that the above standards are primarily about defining common standards for communication of telemetry data. Such standards enable interoperability among the large number of independent systems and tools for observability that collect and/or transport and/or store and/or process telemetry data. Designing for interoperability brings two key advantages.

First, interoperability helps avoid vendor lock-in. When customers have the power of choice without the fear of lock-in, the chances of them adopting new solutions for observability increase significantly.

Second, the accelerated pace of adoption of observability solutions in the market facilitates innovation along many directions. These include: (i) intelligent analysis of telemetry data using AI/ML techniques to automatically identify root causes of problems and to fix them whenever possible; (ii) end-to-end visibility of data pipelines that span across multiple applications, services, and clouds, (iii) efficient and cost-effective storage of diverse types of metrics, traces, events, and logs; and (iv) real-time processing of the large volumes of telemetry data as they are being collected. Healthy competition to deliver this innovation is always welcome.

So what’s next?

Predictions for the Future

Prediction #1: OpenMetrics will have fast-growing adoption.

Rationale: OpenMetrics is based on the Prometheus wire exposition format. The adoption of Prometheus has been growing rapidly, with hundreds of thousands of installations, more than 700 exporters, and thousands of library integrations. This rapid proliferation of Prometheus will serve as the “carrier” that brings OpenMetrics into most enterprises.

Prediction #2: OpenTelemetry will have slow, but steady, growth over the next two years, and will experience accelerated growth afterwards.

Rationale: Distributed tracing across the modern stack is not yet a must-have in most companies; but it is getting closer to having a breakout year. Trends working in this direction are:

Proliferation of applications built using a combination of services on Kubernetes and/or cloud platforms.

Proliferation of mission-critical data pipelines that combine analytics with intelligent decision making in close to real time.

How one-click installs are now possible in containerized environments like Kubernetes to bring end-to-end visibility.

Prediction #3: Open standards for logs will not get very far.

Rationale: Logs are the natural next frontier for open observability standards after metrics and traces. Arguably, logs are more easily available and more widely used today in observability than either traces or metrics. Can this ubiquitous and sticky nature of logs be attributed to them not being shackled by standards? I think the answer is yes.

Logs enable a certain freedom of expression for developers that is critical to protect. Forcing more structure here may result in developers creating fewer log messages, and be counterproductive for observability; especially in the ability to drill down to the root cause of complex problems. Furthermore, trying to convert the trillions of existing log messages from current applications, platforms, and services into a common standard will be a nightmarish experience.

Instead of open standards for logs, I think two other efforts will have growing adoption and bigger impact in the next five years: (i) Using AI/ML techniques to automatically learn the structure and extract insights from logs; and (ii) Developing open standards for events. CloudEvents is a project to watch out for in this space.

Prediction #4: New open observability standards will emerge from the DataOps community.

Rationale: The push for open observability standards to date has come predominantly from the DevOps/SRE community. More and more companies that are now transforming themselves using data and technology — accelerated significantly due to the pandemic — have mushrooming DataOps teams. These teams foster a data-driven culture and manage the entire lifecycle around operationalizing data-driven processes. The current state of affairs of DataOps teams is similar to where DevOps teams were six to eight years ago:

There is growing demand for DataOps services from the business side

As discussed here and here, there is a large and diverse set of emerging data technologies to deal with

There are strong concerns about vendor lock-in and lack of observability

The emergence of open standards to facilitate interoperability and observability can eliminate the constant balancing act needed between concerns in adoption of new technologies and the need to innovate to meet growing business needs. This topic — including why current open standards for observability like OpenTelemetry, OpenMetrics, and the W3C Trace Context do not meet all needs of DataOps teams — deserves further discussion which I will cover it in a future blog.

Did someone say that the best way to predict the future is to invent it? Stay tuned!