Repository: jaegertracing/documentation
Pull request #881: chore(docs): stop re-fetching Medium feed on every Hugo rebuild
Author: login: abhayguptas
Created: 2025-05-17T12:15:11Z | Merged: 2025-05-18T18:31:21Z

Fixes #855 

### Summary

Fix repeated Medium RSS fetches during `make develop` and shift to a static, manually-controlled approach.

### Problem

Previously, the Jaeger documentation site **re-downloaded the Medium RSS feed on every rebuild**, even in development mode. This meant:

- Every file save triggered a network request to Medium.
- Local development became slow and noisy due to repeated warnings.
- Any Medium API issues (like rate limits or downtime) could break the build.

Example:
```
Change detected, rebuilding site...
WARN Downloaded RSS feed for: JaegerTracing - Medium
```

This happened **every single time**, slowing dev feedback loops unnecessarily.

### Solution

We've moved to a **static fetch approach**, committing the feed manually only when needed:

#### Manual Feed Fetching
Added a new Makefile step:
```makefile
fetch-blog-feed:
	curl -s https://medium.com/feed/jaegertracing | yq -p xml -o json -P > data/medium.json
```
---

> No more dev slowdowns


Linked issue #855: Avoid re-downloading blog feed when rebuilding site in dev
During the build we query Medium to get the list of recent blog posts. In the past it used to only be done when Hugo was starting in dev mode for the first time, but now it seems to happen on every change to the content, which makes editing & viewing the content much more time consuming.
```
Change detected, rebuilding site (#25).
2025-02-27 10:57:46.409 -0500
WARN  Downloaded RSS feed for: JaegerTracing - Medium
WARN  Post: Jaeger v2 released
WARN  Post: Towards Jaeger v2  Moar OpenTelemetry!
WARN  Post: Making design decisions for ClickHouse as a core storage backend in Jaeger
WARN  Post: Experiment: Migrating OpenTracing-based application in Go to use the OpenTelemetry SDK
WARN  Post: Better alignment with OpenTelemetry by focusing on OTLP
WARN  Post: Introducing native support for OpenTelemetry in Jaeger
WARN  Post: Jaeger Tracing: A Friendly Guide for Beginners
WARN  Post: Adaptive Sampling in Jaeger
WARN  Post: Migrating from Jaeger client to OpenTelemetry SDK
WARN  Post: Jaeger Persistent Storage With Elasticsearch, Cassandra & Kafka
```

Find a way to avoid repeated fetches of the blog posts once the dev version of Hugo is running (i.e. via `make develop`)
Linked issue comments: user: login: abhayguptas
created_at: 2025-05-15T04:25:49Z
body: Hi @yurishkuro,

I looked into the issue where the Medium blog feed is being re-downloaded on every rebuild when running the site in development mode. This is happening because the feed is being fetched dynamically using resources.GetRemote within themes/jaeger-docs/layouts/index.html. In Hugo's dev mode, this function does not cache responses, so it triggers a re-fetch every time the file is touched or the site rebuilds.

Root Cause - 
The current code in index.html looks like this:
`
{{ with try (resources.GetRemote "https://medium.com/feed/jaegertracing") }}
  ...
{{ end }}
`
This fetches and parses the RSS feed directly during every rebuild, leading to slower development cycles.

Solution - 
To prevent repeated downloads and improve dev experience:

- Pre-fetch the RSS feed during make develop and store it in data/medium.xml.
- Update the template to read from the local file using Hugo's resources.Get.

Example make develop update:
```
make
Copy
Edit
data/medium.xml:
	curl -s https://medium.com/feed/jaegertracing -o data/medium.xml

develop: data/medium.xml
	hugo server --disableFastRender
```

Updated Hugo template:
```
go
Copy
Edit
{{ $rssFile := resources.Get "data/medium.xml" }}
{{ $rss := dict }}
{{ with try (transform.Unmarshal $rssFile) }}
  {{ with .Err }}
    {{ errorf "%s" . }}
  {{ else with .Value }}
    {{ $rss = . }}
  {{ end }}
{{ end }}
```
This way, the feed is downloaded only once when the dev server starts, and Hugo uses the local cached file for subsequent rebuilds, significantly improving the development speed.

Let me know if you'd like me to help implement this or send a PR.

Best regards,
Abhay
author_association: CONTRIBUTOR
user: login: yurishkuro
created_at: 2025-05-15T04:53:39Z
body: this used to work in the past without hacks in the makefile, so what has changed?
author_association: MEMBER
user: login: abhayguptas
created_at: 2025-05-15T19:17:25Z
body: Seems weird but as per my research, The behavior is likely caused by Hugo's live reload logic re-evaluating the resources.GetRemote call each time it rebuilds. While Hugo may have previously cached the remote resource across reloads, that cache mechanism seems to have changed (or now behaves differently depending on environment or Hugo version).

I confirmed that resources.GetRemote is treated as a dynamic resource and re-fetched every time Hugo rebuilds, especially when the output is consumed in the template logic (transform.Unmarshal, etc.). So even if the file itself hasn’t changed, the rebuild triggers a new network call.
author_association: CONTRIBUTOR
user: login: abhayguptas
created_at: 2025-05-15T19:23:08Z
body: Also another temp solution for this is we can add ` {{ $cached := $remote | resources.ExecuteAsTemplate "cached-medium.xml" . }}`
author_association: CONTRIBUTOR
user: login: yurishkuro
created_at: 2025-05-15T21:38:46Z
body: Tbh, I think we should simply have a manual build step to download and commit to git the up to date rss file. The dynamic pulling is rather pointless, since it's not dynamic once the site is deployed, and we don't have such high frequency of blog posts to make the extra manual step too cumbersome. 

Another alternative would be to have the carousel be real dynamic by loading the feed from medium when the page is accessed. But the first option is simpler and gives us more control. It's a slight variation to your proposal. 
author_association: MEMBER
user: login: abhayguptas
created_at: 2025-05-16T05:48:17Z
body: I agree that committing a static copy of the RSS feed to the repo makes the most sense given our low update frequency.
So, Should I go ahead and make the changes?

Proposed changes a.k.a solution - 

1. Add a manual make step to fetch and save the Medium RSS feed:
This uses curl to fetch the XML feed and converts it to JSON using xq, storing it at data/medium.json.
```
fetch-blog-feed:
	curl -s https://medium.com/feed/jaegertracing | xq -j > data/medium.json
```
We can run it manually before deploying or when a new blog post is published:
```
make fetch-blog-feed
git add data/medium.json
git commit -m "chore: update Medium blog feed"
```
2. Update Hugo template to load from the static data/medium.json file instead of fetching it dynamically:
```{{ define "main" }}
  {{ $rss := site.Data.medium }}
  {{ range first 5 $rss.rss.channel.item }}
    <div class="blog-post">
      <a href="{{ .link }}">{{ .title }}</a>
      <p>{{ .pubDate }}</p>
    </div>
  {{ end }}
{{ end }}
```

This ensures:

- No more repeated fetches during make develop
- Full control over the content shown
- No broken builds due to network issues
author_association: CONTRIBUTOR
user: login: yurishkuro
created_at: 2025-05-16T16:43:17Z
body: yes, please open a PR. Also please add a section to the readme, similar to `## Generating Roadmap page`


author_association: MEMBER

Diff:
diff --git a/Makefile b/Makefile
index c2f03156..4ad6c1ff 100644
--- a/Makefile
+++ b/Makefile
@@ -56,6 +56,9 @@ spellcheck:
 	cat scripts/cspell/project-names.txt | grep -v '^#' | grep -v '^\s*$$' | tr ' ' '\n' > scripts/cspell/project-names-parsed.txt
 	cd scripts/cspell && ./spellcheck.sh
 
+fetch-blog-feed:
+	curl -s -o assets/data/medium.xml https://medium.com/feed/jaegertracing
+
 # only x.y.0 semver values are valid for kicking off a new release.
 SEMVER_REGEX := ^([0-9]+\.){2}0$$
 VALID_VERSION := $(shell echo "$(VERSION)" | grep -E "$(SEMVER_REGEX)")
diff --git a/README.md b/README.md
index 44217fb9..75dacea5 100644
--- a/README.md
+++ b/README.md
@@ -87,6 +87,44 @@ python3 scripts/generate_roadmap.py
 
 This script fetches issues from the [GitHub project board](https://github.com/orgs/jaegertracing/projects/4/views/1?layout=table), extracts the required information, and generates the roadmap document. Make sure to set the `GITHUB_TOKEN` environment variable with your GitHub API token before running the script, or save the token in `~/.github_token` file (protect the file so only you can read it: `chmod 0600 <file>`). Personal tokens can be created at https://github.com/settings/tokens/.
 
+## Updating Medium Blog Feed
+
+The homepage displays the latest blog posts from the [Jaeger Medium blog](https://medium.com/jaegertracing).  
+To avoid network calls during builds and to ensure fast, reliable local development, the Medium RSS feed is downloaded and stored as a static XML file.
+
+### Prerequisites
+
+Ensure you have [`curl`](https://curl.se/) installed on your system to download the RSS feed.
+
+Most Linux and macOS systems already have `curl` pre-installed.  
+You can verify installation by running:
+
+```bash
+curl --version
+```
+
+### To update the Medium blog feed
+Run the following command to fetch and save the feed locally as XML:
+
+```bash
+make fetch-blog-feed
+```
+
+This will download the RSS feed from Medium and save it to:
+```bash
+assets/data/medium.xml
+```
+
+After updating, commit the changes:
+```bash
+git add assets/data/medium.xml
+git commit -m "chore: update Medium blog feed"
+```
+
+### How it's used
+The Hugo site reads and parses data/medium.xml using resources.Get and transform.Unmarshal.
+This converts the XML into structured data at build time, allowing full control over the content without relying on live network requests.
+
 ## License
 
 [Apache 2.0 License](./LICENSE).
diff --git a/assets/data/medium.xml b/assets/data/medium.xml
new file mode 100644
index 00000000..bd182c93
--- /dev/null
+++ b/assets/data/medium.xml
@@ -0,0 +1,139 @@
+<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
+    <channel>
+        <title><![CDATA[JaegerTracing - Medium]]></title>
+        <description><![CDATA[Open source distributed tracing platform at Cloud Native Computing Foundation (graduated). https://jaegertracing.io - Medium]]></description>
+        <link>https://medium.com/jaegertracing?source=rss----99735986d50---4</link>
+        <image>
+            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
+            <title>JaegerTracing - Medium</title>
+            <link>https://medium.com/jaegertracing?source=rss----99735986d50---4</link>
+        </image>
+        <generator>Medium</generator>
+        <lastBuildDate>Sun, 18 May 2025 03:25:44 GMT</lastBuildDate>
+        <atom:link href="https://medium.com/feed/jaegertracing" rel="self" type="application/rss+xml"/>
+        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
+        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
+        <item>
+            <title><![CDATA[Jaeger v2 released ]]></title>
+            <link>https://medium.com/jaegertracing/jaeger-v2-released-09a6033d1b10?source=rss----99735986d50---4</link>
+            <guid isPermaLink="false">https://medium.com/p/09a6033d1b10</guid>
+            <category><![CDATA[major-release]]></category>
+            <category><![CDATA[distributed-tracing]]></category>
+            <category><![CDATA[jaeger]]></category>
+            <category><![CDATA[v2]]></category>
+            <category><![CDATA[opentelemetry]]></category>
+            <dc:creator><![CDATA[Yuri Shkuro]]></dc:creator>
+            <pubDate>Sat, 23 Nov 2024 18:33:07 GMT</pubDate>
+            <atom:updated>2025-01-02T23:56:04.707Z</atom:updated>
+            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ccqn-B5CfJhwNkjFjqAFLg.jpeg" /></figure><p><a href="https://www.jaegertracing.io/"><strong>Jaeger</strong></a>, the popular open-source distributed tracing platform, has had a successful 9 year history as being one of the first graduated projects in the Cloud Native Computing Foundation (CNCF). After over 60 releases, Jaeger is celebrating a major milestone with the release of Jaeger v2. This new version is a new architecture for Jaeger components that utilizes <a href="https://opentelemetry.io/docs/collector/"><strong>OpenTelemetry Collector</strong></a> framework as the base and extends it to implement Jaeger’s unique features. It brings significant improvements and changes, making Jaeger more flexible, extensible, and better aligned with the OpenTelemetry project.</p><p>In this blog post, we’ll dive into the details of Jaeger v2, exploring its design, features, and benefits. Sharing what users can expect from this exciting new release and what is next for the project.</p><h3>OpenTelemetry Foundation <strong>🧱</strong></h3><p>OpenTelemetry is the de-facto standard for application instrumentation providing the foundation for observability. Jaeger is now based on the cornerstone of this project, the OpenTelemetry Collector. Jaeger is a complete tracing platform that includes storage and the UI. OpenTelemetry Collector is usually an intermediate component in the collection pipelines that are used to receive, process, transform, and export different telemetry types. The two systems have some overlaps, for example, `jaeger-agent` and `jaeger-collector` play a role similar to what can be done with OpenTelemetry Collector, but only for traces.</p><p>Historically, both Jaeger and OpenTelemetry Collector reused each other’s code. Collector supports receivers for legacy Jaeger formats implemented by importing Jaeger packages. And Jaeger reuses Collector’s receivers and data model converters. Because of this synergy, it’s been our goal for a while to bring the two projects closer.</p><p>OpenTelemetry Collector has a very flexible and extensible design, which makes it easy to extend with additional components needed for Jaeger use cases.</p><h3>Features and Benefits 💡</h3><p>By aligning the Jaeger v2 architecture with OpenTelemetry Collector, we deliver several exciting features and benefits for users, including:</p><ul><li><strong>Native OpenTelemetry processing</strong>: Jaeger v2 natively supports OTLP data format, eliminating the translation step from OTLP to Jaeger’s internal data format and improving performance.</li><li><strong>Batched data processing</strong>: the OpenTelemetry Collector pipelines operate on batches of data, which can be especially important for storage backends like ClickHouse that are much more performant with batch inserts. Jaeger v2 will be able to utilize this batch-based internal pipeline design.</li><li><strong>Familiar developer experience</strong>: Jaeger v2 follows the same configuration and deployment model as OpenTelemetry Collector, providing a more consistent developer experience.</li><li><strong>Access to OpenTelemetry Collector features</strong>: Jaeger v2 inherits all the core features of the Collector, including auth, cert reloading, internal monitoring, health checks, z-pages, etc.</li><li><strong>Advanced sampling capabilities: </strong>Jaeger v2 introduces the ability to perform tail-based sampling, using the upstream<a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor"><strong> OpenTelemetry-contrib processor</strong></a>. It also fully supports Jaeger’s original<a href="https://www.jaegertracing.io/docs/1.62/sampling/#remote-sampling"><strong> Remote</strong></a> and<a href="https://www.jaegertracing.io/docs/1.62/sampling/#adaptive-sampling"><strong> Adaptive</strong></a> head-based sampling methods.</li><li><strong>Access to OpenTelemetry Collector ecosystem</strong>: Jaeger v2 can use a multitude of extensions available for OpenTelemetry Collector, such as span-to-metric connector, telemetry rewriting processors, PII filtering, and many others. For example, we are able to reuse Kafka exporter and receiver and replicate the Jaeger v1 collector-ingester deployment model without maintaining any extra code.</li></ul><p>The result is a leaner code base and more importantly the ability to future proof Jaeger as OpenTelemetry evolves to ensure Jaeger is always the first tracing system for open source users. Compatibility between OpenTelemetry and Jaeger will be supported on day 1 due to the tight integration between the projects. This will continue collaboration between both projects, getting more users to adopt open source technologies more quickly.</p><h3>Design and Architecture 🏛</h3><p>Overall, Jaeger v2 architecture is very similar to a standard OpenTelemetry Collector that has <em>pipelines</em> for receiving and processing telemetry (a pipeline encapsulates <em>receivers</em>, <em>processors</em>, and <em>exporters</em>), and <em>extensions</em> that perform functions not directly related to processing of telemetry. Jaeger v2 makes a few design decisions in how to use the Collector framework.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*erzuvdTb31Ff3oFI" /></figure><h3>Query Extension 🔎</h3><p>Since Collector is primarily designed for data ingestion, querying for traces and presenting them in the UI is an example of functionality that is not in its remit. Jaeger v2 implements it as a Collector extension.</p><h3>Single Binary 🧩</h3><p>Jaeger v1 provided multiple binaries for different purposes (agent, collector, ingester, query). Those binaries were hardwired to perform different functions and exposed different configuration options passed via command line. We realized that all that complexity was unnecessary in the v2 architecture as we achieve the same simply by enabling different components in the configuration file. We also did some benchmarking of executable size and noticed that if we bundle all possible Jaeger v2 components in a single binary, including ~3Mb (compressed) of UI assets, we end up with a container image of ~40Mb in size, versus ~30Mb in v1. As a result, Jaeger v2 ships just a single binary `jaeger`, and it will be configurable for different deployment roles via YAML configuration file, the same as the OpenTelemetry Collector.</p><h3>Storage Implementation 💾</h3><p>Both Jaeger and OpenTelemetry Collector process telemetry data, but they differ in how they handle it.</p><ul><li><strong>OpenTelemetry Collector:</strong> Focuses on one-way data processing (receive, process, export). To store data, it uses specific exporters for each database (e.g., Elasticsearch Exporter).</li><li><strong>Jaeger:</strong> Handles both writing and reading data, allowing queries and visualization through a UI. This requires a shared storage backend, unlike the Collector’s approach.</li></ul><p>Jaeger v2 implements a storage extension to abstract the storage layer. This allows both the query component (read path) and a generic exporter (write path) to interact with various storage backends without dedicated implementations for each. This approach provides flexibility and maintains compatibility with Jaeger v1’s multi-storage capabilities.</p><h3>What’s in v2</h3><ul><li>✅ A single binary &amp; <a href="https://hub.docker.com/r/jaegertracing/jaeger"><strong>Docker image</strong></a> configurable to run in different roles, such as collector, ingester, query, and a collection of <a href="https://github.com/jaegertracing/jaeger/tree/main/cmd/jaeger"><strong>configuration file templates</strong></a> for these roles.</li><li>✅ Support for all official storage backends with full backwards compatibility with Jaeger v1.</li><li>✅ Support for Kafka as intermediate queue, in both OTLP and Jaeger legacy data formats.</li><li>✅ Support for primary and archive storage.</li><li>✅ Support for remote and adaptive head sampling.</li><li>✅ Support for tail sampling .</li><li>✅ Service Performance Management (SPM).</li><li>✅ Internal observability integrated with OpenTelemetry Collector configuration.</li><li>✅ Refreshed documentation and <a href="https://www.jaegertracing.io/docs/next-release-v2/migration/"><strong>v1→v2 migration guide</strong></a>.</li></ul><p>▶️ Try Jaeger v2 today and check out the <a href="https://www.jaegertracing.io/docs/next-release-v2/getting-started/"><strong>Getting Started</strong></a> documentation for more options:</p><pre>docker run --rm --name jaeger \<br>  -p 5778:5778 \<br>  -p 16686:16686 \<br>  -p 4317:4317 \<br>  -p 4318:4318 \<br>  -p 14250:14250 \<br>  -p 14268:14268 \<br>  -p 9411:9411 \<br>  jaegertracing/jaeger:2.1.0</pre><h3>What’s Next</h3><p>Version 2.0.0 already supports all the core features of Jaeger, but the development will continue in 2025 to add remaining feature parity, improving performance, and enhancing the overall user experience.</p><p>The roadmap for Jaeger v2 includes the following milestones:</p><ul><li>🚧 Helm Chart for Jaeger v2.</li><li>🚧 Jaeger v2 support natively in the OpenTelemetry Operator.</li><li>📅 Upgrading storage backend implementations to Storage v2 interface to use OpenTelemetry data natively.</li><li>📅 Support ClickHouse as official storage backend.</li><li>🔮 Upgrading UI to use OpenTelemetry data natively.</li><li>🔮 Upgrading UI to normalize dependency views.</li></ul><h3>Leveraging Mentorship Programs 🎓</h3><p>The Jaeger v2 roadmap was designed to minimize the amount of changes we needed to make to the project by avoiding a big-bang approach in favor of incremental improvements to the existing code base. Yet it was still a significant amount of new development, which is often difficult to sustain for a volunteer-driven project. We were able to attract new contributors and drive the Jaeger v2 roadmap by participating in the <a href="https://www.jaegertracing.io/mentorship/"><strong>mentorship programs</strong></a> run by Linux Foundation, CNCF, and Google, such as LFX Mentorship and Google Summer of Code. This has been a rewarding and mutually beneficial engagement for both the project and the participating interns. Stay tuned for our next mentorships to build out the roadmap items.</p><h3>Conclusion 🏁</h3><p>Jaeger v2 represents a significant step forward for the Jaeger project, bringing improved flexibility, extensibility, and alignment with the OpenTelemetry project. With its native OTLP ingestion, simplified deployment model, and access to OpenTelemetry Collector features, Jaeger v2 promises to provide a more efficient and scalable distributed tracing solution.</p><p>As the development of Jaeger v2 continues, we can expect to see a more robust and feature-rich system emerge. Stay tuned for updates and get ready to experience the next generation of distributed tracing with Jaeger v2!</p><p><em>Originally posted on </em><a href="https://www.cncf.io/blog/2024/11/12/jaeger-v2-released-opentelemetry-in-the-core/"><em>CNCF Blog</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=09a6033d1b10" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jaegertracing/jaeger-v2-released-09a6033d1b10">Jaeger v2 released 🎉</a> was originally published in <a href="https://medium.com/jaegertracing">JaegerTracing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
+        </item>
+        <item>
+            <title><![CDATA[Towards Jaeger v2  Moar OpenTelemetry!]]></title>
+            <link>https://medium.com/jaegertracing/towards-jaeger-v2-moar-opentelemetry-2f8239bee48e?source=rss----99735986d50---4</link>
+            <guid isPermaLink="false">https://medium.com/p/2f8239bee48e</guid>
+            <category><![CDATA[distributed-tracing]]></category>
+            <category><![CDATA[opentelemetry]]></category>
+            <category><![CDATA[jaeger]]></category>
+            <dc:creator><![CDATA[Yuri Shkuro]]></dc:creator>
+            <pubDate>Thu, 25 Jul 2024 23:44:19 GMT</pubDate>
+            <atom:updated>2024-07-25T23:44:19.069Z</atom:updated>
+            <content:encoded><![CDATA[<p>by <a href="https://shkuro.com">Yuri Shkuro</a> and <a href="https://x.com/jkowall">Jonah Kowall</a></p><p>Jaeger, the popular open-source distributed tracing system, is getting a major upgrade with the upcoming release of <strong>Jaeger v2</strong>. This new version is a new architecture for Jaeger backend components that utilizes <a href="https://opentelemetry.io/docs/collector/">OpenTelemetry Collector</a> framework as the base and extends it with Jaeger’s unique features. It promises to bring significant improvements and changes, making Jaeger more flexible, extensible, and even better aligned with the OpenTelemetry project.</p><p>In this blog post, we’ll dive into the details of Jaeger v2, exploring its design, features, and benefits. We’ll also discuss the roadmap for development and what users can expect from this exciting new release.</p><h3>Why OpenTelemetry Collector</h3><p>Jaeger and OpenTelemetry Collector solve different problems. Jaeger is a complete tracing platform that includes storage and the UI. OpenTelemetry Collector is usually an intermediate component in the collection pipelines that is used to receive, process, transform, and export different telemetry types. The two systems have some overlaps, for example, jaeger-agent and jaeger-collector play roles similar to what can be done with OpenTelemetry Collector, but only for traces.</p><p>Historically, both Jaeger and OpenTelemetry Collector reused each other’s code. Collector supports receivers for legacy Jaeger formats implemented by importing Jaeger packages. And Jaeger reuses Collector’s OTLP receivers and OTLP-to-Jaeger data model converters. Because of this synergy, it’s been our goal for a while to bring the two projects closer.</p><p>OpenTelemetry Collector has a very flexible and extensible design, which makes it easy to extend with additional components needed for Jaeger use cases.</p><h3>Third Time a Charm</h3><p>This is actually our third attempt to utilize the OpenTelemetry Collector framework as a basis for Jaeger v2. In the first attempt we tried to keep the Jaeger v2 configuration compatible with v1, reusing the same CLI flags, which was ultimately a mistake as it was too difficult to maintain. In the second attempt we tried to use the OpenTelemetry Collector Builder (known as ocb) to compose Jaeger v2 binary in a different repository, which also proved to be difficult to maintain.</p><p>This third attempt is much further along due to several decisions (see the <a href="https://docs.google.com/document/d/1s4_6VgAS7qAVp6iEm5KYvpiGw3h2Ja5T5HpKo29iv00/edit#">RFC doc</a> for more details):</p><ul><li>We decided to break CLI configuration compatibility and embrace the OpenTelemetry Collector file configuration mechanism.</li><li>We build Jaeger v2 binary by directly importing OpenTelemetry Collector code as a library, which makes the development much easier than with ocb (although we do plan to support ocb in the future as an extension mechanism).</li><li>We’ve implemented adapter layers that allow us to reuse the existing Jaeger v1 code directly in Jaeger v2, which means we can continue evolving a single code base and do the upgrades in-place, instead of working on an incompatible fork for many months.</li></ul><h3>Features and Benefits</h3><p>By aligning Jaeger v2 architecture with the OpenTelemetry Collector, we can deliver several exciting features and benefits for users, including:</p><ul><li><strong>Native OpenTelemetry processing</strong>: Jaeger v2 will natively support the <a href="https://opentelemetry.io/docs/specs/otel/protocol/">OTLP data format</a>, eliminating the translation step from OTLP to Jaeger’s internal data format and improving performance.</li><li><strong>Batched data processing</strong>: the OpenTelemetry Collector pipelines operate on batches of data, which can be especially important when sending data to storage backends like ClickHouse that are much more performant with batch inserts. Jaeger v2 will be able to utilize this batch-based pipeline design, in contrast to v1’s own internal pipeline which was designed around individual spans.</li><li><strong>Familiar developer experience</strong>: Jaeger v2 will follow the same configuration and deployment model as the OpenTelemetry Collector, providing a more consistent developer experience.</li><li><strong>Access to OpenTelemetry Collector features</strong>: Jaeger v2 will inherit all the core features of the Collector, including auth, cert reloading, internal monitoring, health checks, z-pages, etc.</li><li><strong>Access to OpenTelemetry Collector ecosystem</strong>: Jaeger v2 will be able to use a multitude of extensions available for OpenTelemetry Collector, such as span-to-metric connector, tail-based sampling processor, telemetry rewriting processors, PII filtering, etc. For example, we are able to reuse Kafka exporter and receiver and replicate Jaeger v1&#39;s collector/ingester deployment model without maintaining any extra code.</li></ul><p>The result of v2 is less code to maintain in the Jaeger project and an assured alignment with OpenTelemetry, which is already the standard way to instrument applications and collect telemetry.</p><p>The other major benefit to users is the ability to future-proof Jaeger as OpenTelemetry evolves, to ensure Jaeger is always the first tracing system for open source users. This is likely to improve collaboration and evolution of both projects.</p><h3>Design and Architecture</h3><p>Overall, Jaeger v2 architecture is very similar to a standard OpenTelemetry Collector that has <em>pipelines</em> for receiving and processing telemetry (a pipeline encapsulates <em>receivers</em>, <em>processors</em>, and <em>exporters</em>), and <em>extensions </em>that perform functions not directly related to processing of telemetry. Jaeger v2 makes a few specific design decisions in how to use the Collector framework.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iu_PaG5VWUVR2vRz" /><figcaption>High level architecture of Jaeger v2 binary</figcaption></figure><h4>Query Extension 🔎</h4><p>Querying for traces and presenting them in the UI is an example of functionality “not directly related to telemetry processing”, so naturally the equivalent of v1 jaeger-query is implemented as a Collector extension in Jaeger v2.</p><h4>Single Binary</h4><p>Jaeger v1 provided multiple binaries for different purposes (agent, collector, ingester, query). Those binaries were hardwired to perform different functions and exposed different configuration options. We realized that all that complexity was unnecessary in v2 architecture because we can achieve the same simply by enabling different components in the configuration file. We also did some benchmarking of executable size and noticed that if we bundle all possible Jaeger v2 components in a single binary, including ~3Mb (compressed) of UI assets, we end up with all binaries being around 50Mb in size, so separating them does not bring any real benefits. As a result, Jaeger v2 will ship as just a single binary jaeger, and it will be configurable for different deployment roles via YAML configuration file, the same as the OpenTelemetry Collector.</p><h4>Storage Extension</h4><p>One significant difference between Jaeger and OpenTelemetry Collector is that the Collector is designed for one-directional data processing (receivers → processors → exporters), which we call the <em>write path</em>. When the write path needs to store data in a database, the traditional approach in the OpenTelemetry Collector is to implement different exporters for different backends, such as Elasticsearch Exporter. In contrast, Jaeger supports both the write path and the <em>read path</em>, via query and UI. When we combine those functions in a single binary, like the equivalent of Jaeger v1 all-in-one, the write and read paths must share the storage backend implementation, so exporter-per-storage approach does not work for us.</p><p>To accommodate that, and to support Jaeger v1 existing capability of utilizing different storage backends, we abstracted the notion of a “storage” into Jaeger Storage Extension. The Jaeger Query Extension locates that Jaeger Storage Extension on start-up and asks it for a TraceReader. For the write path, we implemented a generic Jaeger Storage Exporter, which also consults the Storage Extension to obtain a TraceWriter implementation.</p><p>Because Query and Exporter ask for storage by name, we are able to simultaneously support different storage implementations for different purposes, for example one could configure Elasticsearch as the main trace storage, Cassandra as the archive storage, and something else as sampling strategies storage. In Jaeger v1 all storage roles had to be done in the same backend.</p><h3>Roadmap and Development</h3><p>The development of Jaeger v2 is ongoing, with several milestones planned before its general availability (GA). The alpha version is already available and supports most of Jaeger v1 functions, such as ingestion of different formats, support for the same storage backends as v1, query/UI, and all-in-one deployment. The team is working on adding remaining feature parity, improving performance, and enhancing the overall user experience.</p><p>The roadmap for Jaeger v2 includes the following milestones:</p><ul><li>✅ Proof of concept: A single binary with memory storage and all-in-one functionality.</li><li>✅ Storage integration: support for the same storage backends as v1.</li><li>✅ New, more comprehensive end-to-end integration tests for all storage backends.</li><li>🚧 Feature parity: Kafka integration.</li><li>🚧 Feature parity: Service Performance Monitoring (SPM).</li><li>📅 Prepare for Beta: release pipeline, and documentation.</li><li>📅 Prepare for GA: creating a Helm chart, Kubernetes operator, and clarifying version compatibility guarantees.</li></ul><p>Once the GA is announced, there are a few more milestones to continue improving Jaeger v2:</p><ul><li>🚀 Upgrading UI to use OpenTelemetry data natively.</li><li>🚀 Upgrading storage backend implementations to Storage v2 interface to use OpenTelemetry data natively.</li><li>🚀 Support ClickHouse is official storage backend.</li></ul><p>▶▶▶️ You can try out Jaeger v2 today! We publish a Docker image (<a href="https://hub.docker.com/r/jaegertracing/jaeger">https://hub.docker.com/r/jaegertracing/jaeger</a>) and we provide a collection of <a href="https://github.com/jaegertracing/jaeger/tree/main/cmd/jaeger">configuration file templates</a> for different deployment modes of Jaeger.</p><h3>Leveraging Mentorship Programs 🎓</h3><p>The Jaeger v2 roadmap was designed to minimize the amount of changes we need to make to the project, by avoiding big-bang approach in favor of incremental improvements to the existing code base. Yet it was still a significant amount of new development, which is often difficult to sustain for a volunteer-driven project. We were able to attract new contributors and drive the Jaeger v2 roadmap by participating in the <a href="https://www.jaegertracing.io/mentorship/">mentorship programs</a> run by Linux Foundation and Google, such as LFX Mentorship and Google Summer of Code. This has been a rewarding and mutually beneficial engagement for both the project and the participating interns. We are currently proposing <a href="https://github.com/jaegertracing/jaeger/issues/5772">two projects for the Fall LFX term</a>.</p><h3>Conclusion</h3><p>Jaeger v2 represents a significant step forward for the Jaeger project, bringing improved flexibility, extensibility, and alignment with the OpenTelemetry project. With its native OTLP ingestion, simplified deployment model, and access to OpenTelemetry Collector features, Jaeger v2 promises to provide a more efficient and scalable distributed tracing solution.</p><p>As the development of Jaeger v2 continues, we can expect to see a more robust and feature-rich system emerge. Stay tuned for updates and get ready to experience the next generation of distributed tracing with Jaeger v2! 💥</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f8239bee48e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jaegertracing/towards-jaeger-v2-moar-opentelemetry-2f8239bee48e">Towards Jaeger v2 💥💥💥 Moar OpenTelemetry!</a> was originally published in <a href="https://medium.com/jaegertracing">JaegerTracing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
+        </item>
+        <item>
+            <title><![CDATA[Making design decisions for ClickHouse as a core storage backend in Jaeger]]></title>
+            <link>https://medium.com/jaegertracing/making-design-decisions-for-clickhouse-as-a-core-storage-backend-in-jaeger-62bf90a979d?source=rss----99735986d50---4</link>
+            <guid isPermaLink="false">https://medium.com/p/62bf90a979d</guid>
+            <dc:creator><![CDATA[Ha Anh Vu]]></dc:creator>
+            <pubDate>Sun, 24 Sep 2023 20:40:51 GMT</pubDate>
+            <atom:updated>2023-09-24T23:02:07.905Z</atom:updated>
+            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nrY-UQFxXoQEjKKXDzHwSw.png" /></figure><h3>Overview</h3><p>ClickHouse database has been used as a remote storage server for Jaeger traces for quite some time, thanks to a <a href="https://github.com/jaegertracing/jaeger-clickhouse">gRPC storage plugin built by the community</a>. Lately, we have decided to make ClickHouse one of the core storage backends for Jaeger, besides Cassandra and Elasticsearch. The first step for this integration was figuring out an optimal schema design. Also, since ClickHouse is designed for batch inserts, we also needed to consider how to support that in Jaeger.</p><p>There are different approaches for schema and batch insert support. So we decided to do performance benchmarks on these options, to figure out the optimal ones. In this blog, we will share our benchmark setups and results, from which we will recommend design choices.</p><p>For tracing data, we found two existing schema design approaches from the <a href="https://github.com/jaegertracing/jaeger-clickhouse">jaeger-clickhouse gRPC plugin</a> (or <em>plugin</em> in short) and <a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/clickhouseexporter">OpenTelemetry Collector’s ClickHouse exporter</a> (or <em>exporter</em>). Both of these allow batching a configurable number of spans internally and sending batches of spans to ClickHouse. To handle batch inserts, ClickHouse also provides another option of using <a href="https://clickhouse.com/docs/en/optimize/asynchronous-inserts">asynchronous insert feature</a> that we would like to try. So overall, our performance benchmarks were carried out in these steps:</p><ul><li>Step 1: Figure out which schema design approach performs better between the <em>plugin</em>’s and the <em>exporter</em>’s.</li><li>Step 2: For the higher-performance schema, figure out how it can be improved to perform even better.</li><li>Step 3: With the schema improved from step 2, try ClickHouse’s asynchronous insert feature and compare the performance with internal client-side batching approach.</li></ul><p>We ran performance benchmarks on different dimensions and configs. In this blog, we will share the setups and results of the benchmarks that showed the most representative results and led to our design choices.</p><p>The configs/scripts/code used to carry out the benchmarks can be found in <a href="https://github.com/haanhvu/clickhouse-trace-exporting-benchmark">this repo</a>.</p><h3>Step 1: Figure out the better schema design approach</h3><p>The <em>plugin</em> and the <em>exporter</em> use different database schemas. For full details of each schema, you can visit <a href="https://github.com/haanhvu/clickhouse-trace-exporting-benchmark/tree/main/schema">the benchmark repo</a>. Here, we will summarize the key differences between them:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9f00fedd3903b1fe0b19288499c935af/href">https://medium.com/media/9f00fedd3903b1fe0b19288499c935af/href</a></iframe><p>To evaluate these schemas, we used these criteria (ordered by priority):</p><ul><li>Search performance</li><li>Insert performance</li><li>Compression ratio</li></ul><p>The reasons for this order of priority are:</p><ul><li>Search performance is an important metric that clearly shows the difference between two schemas.</li><li>Insert performance is also an important metric. But it is affected by other factors outside of the schema or the database, such as the number of insert threads from the client side.</li><li>Compression ratio also shows the performance of each schema, that is, how well each schema compress data and saves storage space. However, compression ratio depends very much on the cardinality of data, which is different case by case in reality. Also, a high compression ratio may also indicate high decompression cost in search queries. So there are pros but also cons here.</li></ul><p>The ultimate purpose of the benchmarks was to compare two schema approaches, not to examine the performance of setups, so we just aimed at a simple setup. For full technical details of the setup, please visit the <a href="https://github.com/haanhvu/clickhouse-trace-exporting-benchmarks/tree/main/setup">benchmark setup directory</a>. Overall, our setup consisted of:</p><ul><li>1 instance of Jaeger Collector with jaeger-clickhouse plugin with batch size 100,000 for the <em>plugin</em> schema</li><li>1 instance of OpenTelemetry Collector with ClickHouse exporter with batch size 100,000 for the <em>exporter</em> schema</li><li>1 instance of ClickHouse single-node server for each collector</li><li>3 instances of <a href="https://hub.docker.com/r/jaegertracing/jaeger-tracegen">jaeger-tracegen</a> that generated 495 millions spans in 1.2 hours to each collector. To ensure that our data was representative enough, we configured generated traces to have 11 spans per trace (1 parent span and 10 child spans), 11 attributes per child span, 97 distinct attribute keys, and 1000 distinct attribute values. (There was one benchmark where we increased the number of distinct values in attribute keys/values to 1000/10000. But the benchmark results didn’t change noticeably).</li></ul><p>For benchmark results, we retrieved data from <a href="https://clickhouse.com/docs/en/operations/system-tables">ClickHouse system tables</a>. The SQL queries to retrieve benchmark results can be found in the <a href="https://github.com/haanhvu/clickhouse-trace-exporting-benchmark/tree/main/performance-retrieval-scripts">benchmark repo</a>.</p><p>Here’s the performance results of two benchmarked schemas (important metrics are <strong>highlighted</strong>):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e8acb1111de0a3f3f2b1177c3ce5cd79/href">https://medium.com/media/e8acb1111de0a3f3f2b1177c3ce5cd79/href</a></iframe><p>Regarding <em>inserted_spans_per_sec_single_thread</em> metric, this is the estimated inserted spans per second <strong><em>in a single insert thread</em></strong>, measured by dividing the number of spans saved by the sum of inserts’ durations. We needed to measure by single insert thread because the numbers of insert threads in the plugin and the exporter are different. The real inserted spans per second with multi-threaded inserts would be much higher.</p><p>With these results, we can see that:</p><ul><li>Search performance: the <em>exporter</em> schema performed better in most search queries except for searching all distinct services, searching all distinct operations, and retrieving all spans by trace ID. It seems that the search performance of the <em>plugin</em> schema was worse because it had to read more tables, hence took more time. However, the performance of searching all distinct services and searching all distinct operations of the <em>plugin</em> schema was much better, thanks to its <strong>jaeger_operations</strong> materialized view. This is one thing the <em>exporter</em> schema could learn from.</li><li>Insert performance: Both used roughly equal memory per span insert. And the <em>exporter</em> schema inserted more spans per second per single insert thread than the <em>plugin</em> schema. So the <em>exporter</em> schema performed better here.</li><li>Compression ratio: The <em>plugin</em> schema had better compression ratios.</li></ul><p>As stated above, we prioritized search performance and insert performance over compression ratio, so to us the exporter schema was the better approach. The next step was trying different improvement opportunities on this schema.</p><h3>Step 2: Try improvement opportunities on OpenTelemetry Collector’s ClickHouse exporter’s schema</h3><p>The first thing we tweaked was <em>ORDER BY (ServiceName, SpanName, toUnixTimestamp(Timestamp), TraceId)</em>. According to <a href="https://clickhouse.com/docs/en/optimize/sparse-primary-indexes">ClickHouse’s doc</a>, to ensure optimal columns’ compression ratios and search performance, the rule of thumb is to pick the columns from which data will be filtered and place lower-cardinality columns first in <em>ORDER BY.</em> So we tried adding two other filtering columns: <em>SpanAttributes</em>, <em>Duration</em>, sorting the cardinalities of the picked columns and had this new <em>ORDER BY (ServiceName, SpanName, toUnixTimestamp(Timestamp), Duration, SpanAttributes, TraceId)</em>.</p><p>With this new <em>ORDER BY</em>, the only noticeable change we observed was that the compression ratio of the Duration column doubled from 3.89 to 7.43, so adding <em>Duration</em> column to <em>ORDER BY</em> seemed like a reasonable choice. We didn’t see noticeable changes in the compression or search performance of the <em>SpanAttributes</em> column. So we’ll take more consideration about whether to include this column in <em>ORDER BY</em>. Also, since the cardinalities of filtering columns may be different case by case, we will allow users to configure their <em>ORDER BY</em> for optimal performance.</p><p>Next, we tried tweaking the data types of some columns since they showed better compression ratios in equivalent columns in the <em>plugin</em> schema:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/196b5c18a441822a9d2fed73a77c191d/href">https://medium.com/media/196b5c18a441822a9d2fed73a77c191d/href</a></iframe><p>With these changes in data types, we saw these changes in related metrics:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3d806fbe439e9ba9e42d48f9238c369e/href">https://medium.com/media/3d806fbe439e9ba9e42d48f9238c369e/href</a></iframe><p>From the metrics, we saw that the new data types performed better:</p><ul><li>The compression ratios were improved. Especially with <em>SpanAttributes</em>, both uncompressed size and compressed size were reduced and compression ratio was increased.</li><li>Searching all spans’ information by tag/attribute took less time and memory.</li></ul><p>We’ll include these changes into our design choices.</p><p>(The setups and modified codebase of the OpenTelemetry Collector’s ClickHouse exporter for these schema changes can be found in the <a href="https://github.com/haanhvu/clickhouse-trace-exporting-benchmark/tree/main/setup/opentelemetry-collector/step2-schemaimprovements">benchmark repo</a>).</p><h3>Step 3: Try ClickHouse’s asynchronous insert feature on the furthest improved schema</h3><p>In the previous benchmarks, we tried client-side batching in both Jaeger Collector and OpenTelemetry Collector. In this step, we tried server-side batching with ClickHouse’s asynchronous insert feature. Overall, we ran three benchmarks:</p><ul><li>Server-side batching alone</li><li>Server-side batching combined with small client-side batching of 10,000 spans per batch</li><li>Server-side batching combined with large client-side batching of 100,000 spans per batch</li></ul><p>The setups and modified codebase of the OpenTelemetry Collector’s ClickHouse exporter for these benchmarks can be found in the <a href="https://github.com/haanhvu/clickhouse-trace-exporting-benchmark/tree/main/setup/opentelemetry-collector/step3-asyncinsert">benchmark repo</a>.</p><p>Here’s the benchmark results:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ca7aad5d51b1e312d682883d8bf665d6/href">https://medium.com/media/ca7aad5d51b1e312d682883d8bf665d6/href</a></iframe><p>From the metrics, asynchronous insert didn’t improve the rate of inserted spans, even though it decreased the memory usage per span insert. In fact, asynchronous insert in ClickHouse has a <a href="https://clickhouse.com/docs/en/operations/settings/settings#asynchronous-insert-settings">wide range of settings</a> and we just tried <a href="https://github.com/haanhvu/opentelemetry-collector-contrib/blob/newtypes-asyncinsert/exporter/clickhouseexporter/exporter_traces.go#L241-L246">one example of settings</a> in our benchmark. We need more experiments and feedback from users to decide whether we should support this in our storage backend.</p><h3>Conclusion</h3><p>From the benchmark results, we concluded our design choices for ClickHouse as a core storage backend for Jaeger will notably consist of:</p><ul><li>A single table for all data fields</li><li>A materialized view for fast services and operations retrieval to Jaeger UI</li><li>A materialized view of trace ID’s time range for fast spans retrieval by trace ID</li><li><em>ORDER BY</em> lower-cardinality filtering columns first: We will provide a default <em>ORDER BY</em> and also allow users to configure since the cardinalities of filtering columns may be different case by case.</li><li>Span attributes stored with <em>Nested</em> type</li><li>Support for client-side batching</li></ul><p>The next step would be implementing all these choices into actual features.</p><p>Stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62bf90a979d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jaegertracing/making-design-decisions-for-clickhouse-as-a-core-storage-backend-in-jaeger-62bf90a979d">Making design decisions for ClickHouse as a core storage backend in Jaeger</a> was originally published in <a href="https://medium.com/jaegertracing">JaegerTracing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
+        </item>
+        <item>
+            <title><![CDATA[Experiment: Migrating OpenTracing-based application in Go to use the OpenTelemetry SDK]]></title>
+            <link>https://medium.com/jaegertracing/experiment-migrating-opentracing-based-application-in-go-to-use-the-opentelemetry-sdk-29b09fe2fbc4?source=rss----99735986d50---4</link>
+            <guid isPermaLink="false">https://medium.com/p/29b09fe2fbc4</guid>
+            <category><![CDATA[opentelemetry]]></category>
+            <category><![CDATA[migration]]></category>
+            <category><![CDATA[jaegertracing]]></category>
+            <category><![CDATA[opentracing]]></category>
+            <dc:creator><![CDATA[Yuri Shkuro]]></dc:creator>
+            <pubDate>Thu, 09 Feb 2023 05:44:06 GMT</pubDate>
+            <atom:updated>2023-02-09T05:46:24.711Z</atom:updated>
+            <content:encoded><![CDATA[<p>TL;DR: This post explains how Jaeger’s 🚗 HotROD 🚗 app was migrated to the OpenTelemetry SDK.</p><p>Jaeger’s <a href="https://www.jaegertracing.io/docs/1.42/getting-started/#sample-app-hotrod">HotROD demo</a> has been around for a few years. It was written with OpenTracing-based instrumentation, including a couple of OSS libraries for HTTP and gRPC middleware, and used Jaeger’s native SDK for Go, <a href="https://github.com/jaegertracing/jaeger-client-go">jaeger-client-go</a>. The latter was deprecated in 2022, so we had a choice to either convert all of the HotROD app’s instrumentation to OpenTelemetry, or try the OpenTracing-bridge, which is a required part of every OpenTelemetry API / SDK. The bridge is an adapter layer that wraps an OpenTelemetry Tracer in a facade to makes it look like the OpenTracing Tracer. This way we can use the OpenTelemetry SDK in an application like HotROD that only understands the OpenTracing API.</p><p>I wanted to try the bridge solution, to minimize the code changes in the application. It is not the most efficient way, since an adapter layer incurs some performance overhead, but for a demo app it seemed like a reasonable trade-off.</p><p>The code can be found in the <a href="https://github.com/jaegertracing/jaeger/tree/fedeb4cab75399e4672b77efe6a067a7bd148ddf/examples/hotrod">Jaeger repository</a> (at specific commit hash).</p><h3>Setup</h3><p>First, we need to initialize the OpenTelemetry SDK and create an OpenTracing Bridge. Fortunately, I did not have to start from scratch, because there was an earlier <a href="https://github.com/jaegertracing/jaeger/pull/3390">pull request #3390</a> by <a href="https://github.com/rbroggi">@rbroggi</a>, which I picked up to make certain improvements. Initialization happens in pkg/tracing/init.go :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ba947995089d22c1c24f68f2202ac1b7/href">https://medium.com/media/ba947995089d22c1c24f68f2202ac1b7/href</a></iframe><p>In the beginning, this is a pretty vanilla OpenTelemetry SDK initialization. We create an exporter using a helper function (more on it below), and build a TracerProvider. The compliant OpenTelemetry instrumentation would use this provider to create named Tracer objects as needed, usually with distinct names reflecting the instrumentation library or the application component. However, the OpenTracing API did not have the concept of named tracers, its Tracer as a singleton, so here we create a Tracer with a blank name (in line 23) and pass it to the bridge factory that wraps it and returns an OpenTracing Tracer.</p><p>Side note: in a better-organized code there would also be some sort of close/shutdown function returned so that the caller of tracing.Init could gracefully shutdown the tracer, e.g. to flush the span buffers when stopping the application.</p><p>The original PR used the Jaeger exporter that lets the SDK export data in the Jaeger’s native data format. However, last year we extended Jaeger to accept OpenTelemetry’s OTLP format directly, so I decided to add a bit of flexibility and make the choice of the exporter configurable:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/67ffc7651db87b6891d18b107029b92b/href">https://medium.com/media/67ffc7651db87b6891d18b107029b92b/href</a></iframe><h3>Broken Traces</h3><p>At this point things should have started to work. However, the resulting traces looked like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b-RsMFPGcTtpxKaL2oRMEg.png" /><figcaption>Trace with many spans, but all coming from a single service `frontend`.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0XHZYX0Xwe32mMnQLdbH3w.png" /><figcaption>Another part of the workflow captured as a different trace. It looks like there are two services here, but in fact the HotROD app simulates the `sql` and `redis` database services, it’s not actually making RPC calls to them.</figcaption></figure><p>Instead of one trace per request we are getting several disjoined traces. This is where <a href="https://github.com/rbroggi">@rbroggi</a>’s PR got stuck. After some debugging I came to realize that the SDK defaults to a no-op propagation, so no trace context was sent in RPC requests between services, resulting in multiple disjoined traces for the same workflow. It was easy to fix, but it felt like an unnecessary friction in using the OpenTelemetry SDK. I also added the Baggage propagator, which we will discuss later.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aaceccd029d20a9df3628341bfea94da/href">https://medium.com/media/aaceccd029d20a9df3628341bfea94da/href</a></iframe><p>Since the Init() function is called many times by different services in the HotROD app, I only set the propagator once using sync.Once.</p><p>After this change, the traces looked better, more colorful, so I committed the change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_DzcdeHiRp6JOG2FqA3bfg.png" /><figcaption>A better-looking trace after “fixing” the propagation.</figcaption></figure><h3>Traces Still Broken</h3><p>However, I should’ve paid better attention. Notice the lone span in the middle called /driver.DriverService/FindNearest. Let’s take a closer look:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XZ1VM84sCFU0kjFbkiI5ig.png" /><figcaption>A client span trying to make a gRPC request to service `driver`.</figcaption></figure><p>This span is a client-side of a gRPC request from frontend service to driver service. The latter is missing from the trace! This was a different issue with the context propagation. There was an error returned when trying to inject the context into the request headers. The instrumentation actually logged the error back into the client span, which we can see in the Logs section: Invalid Inject/Extract carrier. Unfortunately, it was difficult to spot this error without opening up the span, because the RPC itself was successful, and the instrumentation was correct in not setting the error=true span tag, which would’ve shown in the Jaeger UI as a red icon.</p><p>After a bit more digging I found the issue, which was due to a bug in the OpenTelemetry SDK’s bridge implementation. You can read about it in the following GitHub issue.</p><p><a href="https://github.com/open-telemetry/opentelemetry-go/issues/3678">[opentracing] OT Bridge does not work with OT gRPC instrumentation · Issue #3678 · open-telemetry/opentelemetry-go</a></p><p>As of this writing, the fix if still waiting to be merged, so as a workaround I made a branch of opentracing-contrib/go-grpc and changed it to use TextMap propagation instead of HTTPHeaders, which by chance happened to work with the bridge code.</p><p>With these fixes, we were back to the “classic” HotROD traces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s26BJwM-r_pP97TZ_4Nk-w.png" /><figcaption>Full HotROD trace, as the AI overlords intended.</figcaption></figure><h3>RPC Metrics</h3><p>I was ready to call it a day, but there was one piece missing. The original Jaeger SDK initialization code had one extra feature — it was enabling the collection of RPC metrics from spans (supported only by the Go SDK in Jaeger). My original blog post, <a href="https://medium.com/opentracing/take-opentracing-for-a-hotrod-ride-f6e3141f7941">Take OpenTracing for a HotROD ride</a>, had a discussion about it, so it was a shame to lose this during this upgrade. If I were upgrading to the OpenTelemetry instrumentation as well, it might have contained a metrics-oriented instrumentation, although it would somewhat miss the point of the blog post that tracing instrumentation is already sufficient in this case. Another possibility is to generate metrics from spans using a special processor in the OpenTelemetry Collector, but using the Collector is not part of the HotROD demo setup.</p><p>The OpenTelemetry SDK has the notion of span processors, an abstract API invoked on all finished spans. It is similar to how the RPCMetricsObserver was implemented in the jaeger-client-go, so I did what any scrappy engineer would do — copy &amp; paste the code from jaeger-client-go directly into the HotROD code and adopt it to implement otel.SpanProcessor. And voilà:</p><pre>$ curl http://127.0.0.1:8083/debug/vars | grep &#39;&quot;requests.endpoint_HTTP&#39;<br>&quot;requests.endpoint_HTTP_GET_/.error_false&quot;: 3,<br>&quot;requests.endpoint_HTTP_GET_/.error_true&quot;: 0,<br>&quot;requests.endpoint_HTTP_GET_/config.error_false&quot;: 4,<br>&quot;requests.endpoint_HTTP_GET_/config.error_true&quot;: 0,<br>&quot;requests.endpoint_HTTP_GET_/customer.error_false&quot;: 4,<br>&quot;requests.endpoint_HTTP_GET_/customer.error_true&quot;: 0,<br>&quot;requests.endpoint_HTTP_GET_/debug/vars.error_false&quot;: 5,<br>&quot;requests.endpoint_HTTP_GET_/debug/vars.error_true&quot;: 0,<br>&quot;requests.endpoint_HTTP_GET_/dispatch.error_false&quot;: 4,<br>&quot;requests.endpoint_HTTP_GET_/dispatch.error_true&quot;: 0,<br>&quot;requests.endpoint_HTTP_GET_/route.error_false&quot;: 40,<br>&quot;requests.endpoint_HTTP_GET_/route.error_true&quot;: 0,</pre><h3>Baggage</h3><p>As I was looking through the metrics in HotROD, I realized there was another area I neglected. These sections in the expvar output were not supposed to be empty:</p><pre>$ curl http://127.0.0.1:8083/debug/vars | grep route.calc.by<br>&quot;route.calc.by.customer.sec&quot;: {},<br>&quot;route.calc.by.session.sec&quot;: {}</pre><p>These measures require baggage to work. The term “baggage” was introduced in the academia (<a href="https://www.usenix.org/conference/atc16/technical-sessions/presentation/mace">Jonathan Mace <em>et al., </em>SOSP 2015 Best Paper Award</a>). It refers to a <a href="https://medium.com/jaegertracing/embracing-context-propagation-7100b9b6029a">general-purpose context propagation mechanism</a>, which can be used to carry both the tracing context and any other contextual metadata across the distributed workflow execution. The HotROD app demonstrates a number of capabilities that require baggage propagation, and they were all completely broken after upgrading to OpenTelemetry SDK 😭.</p><p>The first thing that broke was propagation of baggage from the web UI. HotROD does not start the trace in the browser, only in the backend. The Jaeger SDK had a feature that allowed it to accept baggage from the incoming request even when there was no incoming tracing context. Internally the Jaeger SDK achieved this by returning an “invalid” SpanContext from the Extract method where the trace ID / span ID were blank, but the baggage was present. Digging through the OpenTracing Bridge code I found that it returns an error in this case. This could probably be fixed there, but I decided to add a workaround directly to HotROD where I used the OpenTelemetry’s Baggage propagator to extract the baggage from the request manually and then copy it into the span.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ab6775e1bef5ed4e4561486174c6754b/href">https://medium.com/media/ab6775e1bef5ed4e4561486174c6754b/href</a></iframe><p>I trimmed down the code example above a bit to only show relevant parts. The otelBaggageExtractor function creates a middleware that manually extracts the baggage into the current Context. Then the instrumentation library nethttp is given a span observer (invoked after the server span is created) which copies the baggage from the context into the span. This functionality is only needed at the root span, because once the trace context is propagated through the workflow, the Bridge correctly propagates the baggage as well (remember that I registered Baggage propagator as a global propagator in the Init function, as shown in the earlier code snippet). I was actually pleasantly surprised that the maintainers were able to achieve that, because the OpenTracing API operates purely on Span objects, not on the Context, while in OpenTelemetry the baggage is carried in the Context, a lower logical level.</p><p>One other small change I had to make was to change the web UI to use the baggage header (per W3C standard), instead of the jaeger-baggage header that was recognized by the Jaeger SDK.</p><p>Strictly speaking, these were all the changes I had to make to the HotROD code to make the baggage work. Yet, it didn’t work. Some baggage values were correctly propagated, but others were missing. After more digging I found several places where it was silently dropped on the floor because of some (misplaced, in my opinion) validations in the baggage and the bridge/opentracing packages in the OpenTelemetry SDK. The ticket below explains the issue in more details.</p><p><a href="https://github.com/open-telemetry/opentelemetry-go/issues/3685">Baggage not working with OpenTracing Bridge · Issue #3685 · open-telemetry/opentelemetry-go</a></p><p>Running against a patched version of OpenTelemetry SDK yielded the desired behavior and the baggage-reliant functionality was restored. I was getting performance metrics grouped by baggage values:</p><pre>$ curl http://127.0.0.1:8083/debug/vars | grep route.calc.by<br>&quot;route.calc.by.customer.sec&quot;: {<br>  &quot;Amazing Coffee Roasters&quot;: 0.9080000000000004, <br>  &quot;Japanese Desserts&quot;: 1.0490000000000002, <br>  &quot;Rachel&#39;s Floral Designs&quot;: 1.0090000000000003, <br>  &quot;Trom Chocolatier&quot;: 1.0000000000000004<br>},<br>&quot;route.calc.by.session.sec&quot;: {<br>  &quot;2885&quot;: 1.4760000000000002, <br>  &quot;5161&quot;: 2.4899999999999993<br>}</pre><p>And the mutex instrumentation was able to capture IDs of multiple transactions in the queue (see the <a href="https://medium.com/opentracing/take-opentracing-for-a-hotrod-ride-f6e3141f7941">original blog post</a> for explanation of this one):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lxeVafTIXsC7PVzkaOLIhA.png" /><figcaption>Logs show a transactions blocked on three other transactions.</figcaption></figure><h3>Summary</h3><p>Overall, the migration required fairly minimal amount of changes to the code, mostly because I chose to reuse the existing OpenTracing instrumentation and only swap the SDK from Jaeger to OpenTelemetry. The most friction with the migration was due to a couple of bugs in the OpenTelemetry Bridge code (and likely one place in the baggage package). This only leads me to believe that the baggage functionality is not yet widely used, especially when someone uses the OpenTracing instrumentation with a bridge to OpenTelemetry, so it is likely I just ran into a bunch of the early adopter issues.</p><p>At this point I am interested in taking the next step and doing a full migration of HotROD to OpenTelemetry (or help reviewing if someone wants to volunteer!) It could make a complementary Part 2 to this post to describe how that goes.</p><p>There is also a possible Part 3 involving a no-less interesting migration to the OpenTelemetry Metrics. Right now all of the Jaeger code base is using an internal abstraction for metrics backed by the Prometheus SDK.</p><p>Stay tuned.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=29b09fe2fbc4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jaegertracing/experiment-migrating-opentracing-based-application-in-go-to-use-the-opentelemetry-sdk-29b09fe2fbc4">Experiment: Migrating OpenTracing-based application in Go to use the OpenTelemetry SDK</a> was originally published in <a href="https://medium.com/jaegertracing">JaegerTracing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
+        </item>
+        <item>
+            <title><![CDATA[Better alignment with OpenTelemetry by focusing on OTLP]]></title>
+            <link>https://medium.com/jaegertracing/better-alignment-with-opentelemetry-by-focusing-on-otlp-f3688939073f?source=rss----99735986d50---4</link>
+            <guid isPermaLink="false">https://medium.com/p/f3688939073f</guid>
+            <category><![CDATA[opentelemetry]]></category>
+            <dc:creator><![CDATA[Yuri Shkuro]]></dc:creator>
+            <pubDate>Thu, 03 Nov 2022 18:40:39 GMT</pubDate>
+            <atom:updated>2022-11-03T18:12:55.688Z</atom:updated>
+            <content:encoded><![CDATA[<p>TL;DR: proposal (and a <a href="https://forms.gle/aUuJg5DQwNzncJ4s8">survey</a>) to deprecate native Jaeger exporters in OpenTelemetry SDKs in favor of OTLP exporters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iFJFYZsdPvaFuwaAoZ1HRQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@miquel_parera_mila?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Miquel Parera</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>This is a re-post from the <a href="https://opentelemetry.io/blog/2022/jaeger-native-otlp/">OpenTelemetry blog article</a>.</p><p>By <a href="https://github.com/breedx-splk"><strong>Jason Plumb</strong></a><strong> (Splunk)</strong> | Thursday, November 03, 2022</p><p>Back in May of 2022, the Jaeger project <a href="https://medium.com/jaegertracing/introducing-native-support-for-opentelemetry-in-jaeger-eb661be8183c">announced native support for the OpenTelemetry Protocol</a> (OTLP). This followed a <a href="https://twitter.com/YuriShkuro/status/1455170693197402119">generous deprecation cycle</a> for the Jaeger client libraries across many languages. With these changes, OpenTelemetry users are now able to send traces into Jaeger with industry-standard OTLP, and the Jaeger client library repositories have been finally archived.</p><p>We intend to <strong>deprecate Jaeger exporters from OpenTelemetry</strong> in the near future, and are looking for your feedback to determine the length of the deprecation phase. The best way to provide feedback is by <a href="https://forms.gle/aUuJg5DQwNzncJ4s8">filling out a 4-question survey</a> or commenting on <a href="https://github.com/open-telemetry/opentelemetry-specification/pull/2858">the existing draft pull request</a>.</p><h3>OpenTelemetry Support</h3><p>This interoperability is a wonderful victory both for Jaeger users and for OpenTelemetry users. However, we’re not done yet. The OpenTelemetry specification still requires support for Jaeger client exporters across languages.</p><p>This causes challenges for both Jaeger users and OpenTelemetry maintainers:</p><ol><li><strong>Confusing Choices: </strong>Currently, users are faced with a choice of exporter (Jaeger or OTLP), and this can be a source of confusion. A user might be inclined, when exporting telemetry to Jaeger, to simply choose the Jaeger exporter because the name matches (even though Jaeger now actively encourages the use of OTLP).<br>If we can eliminate this potentially confusing choice, we can improve the user experience and continue standardizing on a single interoperable protocol. We love it when things “just work” out of the box!</li><li><strong>Maintenance and duplication: </strong>Because the Jaeger client libraries are now archived, they will not receive updates (including security patches). To continue properly supporting Jaeger client exporters, OpenTelemetry authors would be required to re-implement some of the functionality it had previously leveraged from the Jaeger clients.<br>Now that Jaeger supports OTLP, this feels like a step backwards: It results in an increased maintenance burden with very little benefit.</li></ol><h3>User Impact</h3><p>The proposal is to deprecate the following exporters from OpenTelemetry in favor of using native OTLP into Jaeger:</p><ul><li>Jaeger Thrift over HTTP</li><li>Jaeger Protobuf via gRPC</li><li>Jaeger Thrift over UDP</li></ul><p>In addition to application configuration changes, there could be other architectural considerations. HTTP and gRPC should be straightforward replacements, although it may require exposing ports 4317 and 4318 if they are not already accessible.</p><p>Thrift over UDP implies the use of the <a href="https://www.jaegertracing.io/docs/1.24/architecture/#agent">Jaeger Agent</a>. Users with this deployment configuration will need to make a slightly more complicated change, typically one of the following:</p><ol><li>Direct ingest. Applications will change from using Thrift+UDP to sending OTLP traces directly to their jaeger-collector instance. This may also have sampling implications.</li><li>Replacing the Jaeger Agent with a sidecar <a href="https://github.com/open-telemetry/opentelemetry-collector">OpenTelemetry Collector</a> instance. This could have sampling implications and requires changes to your infrastructure deployment code.</li></ol><h3>Intent to Deprecate — We’d Like Your Feedback!</h3><p>In order to better support users and the interop between OpenTelemetry and Jaeger, we intend to deprecate and eventually remove support for Jaeger client exporters / Jaeger native data format in OpenTelemetry.</p><p>We would like your feedback! We want to hear from users who could be impacted by this change. To better make a data-informed decision, <a href="https://forms.gle/aUuJg5DQwNzncJ4s8">we have put together a short 4-question survey</a>.</p><p>Your input will help us to choose how long to deprecate before removal.</p><p>A <a href="https://github.com/open-telemetry/opentelemetry-specification/pull/2858">draft PR has been created in the specification</a> to support this deprecation. If would like to contribute and provide feedback, visit the link above and add some comments. We want to hear from you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3688939073f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jaegertracing/better-alignment-with-opentelemetry-by-focusing-on-otlp-f3688939073f">Better alignment with OpenTelemetry by focusing on OTLP</a> was originally published in <a href="https://medium.com/jaegertracing">JaegerTracing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
+        </item>
+        <item>
+            <title><![CDATA[Introducing native support for OpenTelemetry in Jaeger]]></title>
+            <link>https://medium.com/jaegertracing/introducing-native-support-for-opentelemetry-in-jaeger-eb661be8183c?source=rss----99735986d50---4</link>
+            <guid isPermaLink="false">https://medium.com/p/eb661be8183c</guid>
+            <category><![CDATA[opentelemetry]]></category>
+            <category><![CDATA[jaeger]]></category>
+            <category><![CDATA[distributed-tracing]]></category>
+            <dc:creator><![CDATA[Yuri Shkuro]]></dc:creator>
+            <pubDate>Mon, 30 May 2022 22:45:00 GMT</pubDate>
+            <atom:updated>2022-05-31T18:17:15.393Z</atom:updated>
+            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*jJm7BHkMListmocakAMcqA.png" /></figure><p>The latest <a href="https://github.com/jaegertracing/jaeger/releases/tag/v1.35.0">Jaeger v1.35 release</a> introduced the ability to receive OpenTelemetry trace data via the <a href="https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md">OpenTelemetry Protocol (OTLP)</a>, which all OpenTelemetry SDKs are required to support. This is a follow-up to the previous <a href="https://twitter.com/YuriShkuro/status/1455170693197402119">announcement</a> to retire Jaeger’s “classic” client libraries.</p><p>With this new capability, it is no longer necessary to use the Jaeger exporters with the OpenTelemetry SDKs, or to run the OpenTelemetry Collector in front of the Jaeger backend. Using the OTLP exporter, the SDKs can be configured to send the data directly to the Jaeger backend. The OTLP receiver accepts data via gRPC and HTTP endpoints (gRPC mode had an issue that was patched in 1.35.1).</p><h3>Primer</h3><p>Let’s see this functionality in action. First, start the Jaeger all-in-one as described in the <a href="https://www.jaegertracing.io/docs/latest/getting-started/">Getting Started documentation</a>:</p><pre>docker run --name jaeger \<br>  -e COLLECTOR_OTLP_ENABLED=true \<br>  -p 16686:16686 \<br>  -p 4317:4317 \<br>  -p 4318:4318 \<br>  jaegertracing/all-in-one:1.35</pre><p>Notice that compared to the previous releases:</p><ol><li>There are two more ports added to the export list, 4317 and 4318, used by the OTLP receiver to listen for gRPC and HTTP connections.</li><li>The OTLP receiver must be enabled via COLLECTOR_OTLP_ENABLED=true.</li><li>We removed the other ports that are not relevant to this example.</li></ol><p>When the Jaeger backend is starting, you should see these two log lines:</p><pre>{... &quot;msg&quot;:&quot;Starting GRPC server on endpoint 0.0.0.0:4317&quot;}<br>{... &quot;msg&quot;:&quot;Starting HTTP server on endpoint 0.0.0.0:4318&quot;}</pre><p>As usual, the Jaeger UI can be accessed at <a href="http://localhost:16686/">http://localhost:16686/</a>.</p><p>Now let’s use a simple Python program that configures the OpenTelemetry SDK with OTLPSpanExporter and generates a single-span trace.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5288cb4b7a19d2a79483198694ff3960/href">https://medium.com/media/5288cb4b7a19d2a79483198694ff3960/href</a></iframe><p>Run this program as follows:</p><pre>pip install -r requirements.txt<br>OTEL_SERVICE_NAME=primer python3 basic_trace.py</pre><p>If we now refresh the Search screen in the Jaeger UI, the Services dropdown should contain the service primer(notice that we pass this service name to the SDK via an environment variable) and the trace from this service should look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fb8xit2l7SxYMyKhIRrBGg.png" /><figcaption>Sample trace submitted via OTLP.</figcaption></figure><p>The OTLP receiver can be further customized by a number of flags starting with --collector.otlp.*, available via the help command in the collector and all-in-one binaries. These flags allow changing the port numbers for the two OTLP servers, configuring TLS, and changing some other parameters like the max message size and keep-alive.</p><h3>Limitations</h3><p>There are a few caveats with the existing implementation:</p><ul><li>If your application exports both traces and metrics using OTLP, then you would still need to run the OpenTelemetry Collector, because the Jaeger collector can only accept the tracing portion of OTLP data. Alternatively, you may configure the SDK with two OTLP exporters pointing to different backends.</li><li>Not all the options supported by the OTLP receiver in the OpenTelemetry Collector are supported by the Jaeger backend.</li><li>Only the Jaeger collector supports the new OTLP receiver. The Jaeger agent only supports the “classic” Jaeger formats. If your deployment requires a local agent, we recommend running the OpenTelemetry Collector in that capacity.</li></ul><h3>Try it out</h3><p>As usual, we are interested in the community’s feedback about this new feature; please comment <a href="https://www.jaegertracing.io/get-in-touch/">in the chat room</a> or open an issue.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb661be8183c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jaegertracing/introducing-native-support-for-opentelemetry-in-jaeger-eb661be8183c">Introducing native support for OpenTelemetry in Jaeger</a> was originally published in <a href="https://medium.com/jaegertracing">JaegerTracing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
+        </item>
+        <item>
+            <title><![CDATA[Jaeger Tracing: A Friendly Guide for Beginners]]></title>
+            <link>https://medium.com/jaegertracing/jaeger-tracing-a-friendly-guide-for-beginners-7b53a4a568ca?source=rss----99735986d50---4</link>
+            <guid isPermaLink="false">https://medium.com/p/7b53a4a568ca</guid>
+            <category><![CDATA[opentelemetry]]></category>
+            <category><![CDATA[observability]]></category>
+            <category><![CDATA[distributed-tracing]]></category>
+            <category><![CDATA[jaeger]]></category>
+            <dc:creator><![CDATA[Team Aspecto]]></dc:creator>
+            <pubDate>Sun, 27 Feb 2022 20:39:38 GMT</pubDate>
+            <atom:updated>2022-02-27T20:39:37.929Z</atom:updated>
+            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T1FPdT_GcTo_2loOgrLyUA.png" /></figure><p>Written by <a href="https://twitter.com/thetomzach">@thetomzach</a> @ <a href="https://www.aspecto.io/?utm_source=jaeger-medium&amp;utm_medium=post&amp;utm_campaign=jaeger-tracing-the-ultimate-guide">Aspecto</a>.</p><p>In this guide, you’ll learn what Jaeger tracing is, what distributed tracing is, and how to set it up in your system. We’ll go over Jaeger’s UI and touch on advanced concepts such as sampling and deploying in production.</p><p>You’ll leave this guide knowing how to create spans with OpenTelemetry and send them to Jaeger tracing for visualization. All that, from scratch.</p><h3>What is Distributed Tracing? Introduction</h3><p>Before we dive into explaining everything you need to know from 0 to 100 about Jaeger tracing, it’s important to understand the umbrella term that Jaeger is part of — distributed tracing.</p><p>In the world of microservices, most issues occur due to networking issues and the relations between the different microservices. A distributed architecture (as opposed to a monolith) makes it a lot harder to get to the root of an issue. To resolve these issues, we need to see which service sent what parameters to another service or a component (a DB, queue, etc.).</p><p>Distributed tracing helps us achieve just that by enabling us to collect data from the different parts of our system, to enable this desired observability into our system. You can think of it as ‘call-stacks’ for distributed services. In addition, traces are a visual tool, allowing us to visualize our system to better understand the relationships between services, making it easier to investigate and pinpoint issues.</p><h3><strong>What is Jaeger Tracing?</strong></h3><p>Now that you know what distributed tracing is, we can safely talk about Jaeger. Jaeger is an open-source distributed tracing platform created by Uber back in 2015. It consists of instrumentation SDKs, a backend for data collection and storage, a UI for visualizing the data, and Spark/Flink framework for aggregate trace analysis.</p><p>The Jaeger data model is compatible with OpenTracing — which is a specification that defines how the collected tracing data would look, as well as libraries of implementations in different languages (more on OpenTracing and OpenTelemetry later).</p><p>As most other distributed tracing systems, Jaeger works with spans and traces, as defined in the OpenTracing specification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DzjXpBSuNiyCFcYq" /></figure><p>A span represents a unit of work in an application(HTTP request, call to a DB, etc) and is Jaeger’s most basic unit of work. A span must have an operation name, start time, and duration.</p><p>A trace is a collection/list of spans connected in a parent/child relationship (and can also be thought of as a directed acyclic graph of spans). Traces specify how requests are propagated through our services and other components.</p><h3>Jaeger Tracing Architecture</h3><p>Here’s what Jaeger architecture looks like</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xIdm2tN5PkOTJHy-" /></figure><p>It consists of a few parts, all of which I explain below:</p><ul><li><strong>Instrumentation SDKs: </strong>libraries that are integrated into applications and frameworks to capture tracing data. Historically the Jaeger project supported its own clients libraries written in various programming languages. They are now being deprecated in favor of OpenTelemetry (again, more on that later).</li><li><strong>Jaeger Agent:</strong> Jaeger agent is a network daemon that listens for spans received from the Jaeger client over UDP. It gathers batches of them and then sends them together to the collector. The agent is not required if the SDKs are configured to send the spans directly to the collector.</li><li><strong>Jaeger Collector:</strong> The Jaeger collector is responsible for receiving traces from the Jaeger agent, performing validations and transformations, and saving them to the selected storage backends.</li><li><strong>Storage Backends: </strong>Jaeger supports various storage backends to store the spans. Supported storage backends are In-Memory, Cassandra, Elasticsearch, and Badger (for single-instance collector deployments).</li><li><strong>Jaeger Query: </strong>This is a service responsible for retrieving traces from the Jaeger storage backend and making them accessible for the Jaeger UI.</li><li><strong>Jaeger UI:</strong> a React application that lets you visualize the traces and analyze them. Useful for debugging system issues.</li><li><strong>Ingester:</strong> The ingester is relevant only if we use Kafka as a buffer between the collector and the storage backend. It is responsible for receiving data from Kafka and ingesting it into the storage backend. More info can be found in the <a href="https://www.jaegertracing.io/docs/1.30/architecture/#ingester">official Jaeger Tracing docs</a>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6Pjtk8IgfVpfQp2F" /></figure><h3>Running Jaeger locally using Docker</h3><p>Jaeger comes with a ready-to-use <strong>all-in-one</strong> Docker image that contains all the components necessary for Jaeger to run.</p><p>It’s really simple to get it up and running on your local machine:</p><pre>docker run -d --name jaeger \<br>  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \<br>  -p 5775:5775/udp \<br>  -p 6831:6831/udp \<br>  -p 6832:6832/udp \<br>  -p 5778:5778 \<br>  -p 16686:16686 \<br>  -p 14250:14250 \<br>  -p 14268:14268 \<br>  -p 14269:14269 \<br>  -p 9411:9411 \<br>  jaegertracing/all-in-one:1.30</pre><p>Then you can simply open the jaeger UI on <a href="http://localhost:16686/">http://localhost:16686</a>.</p><h3>Jaeger Tracing and OpenTelemetry</h3><p>Yes, you’re right. I did mention before that Jaeger’s data model is compatible with the OpenTracing specification. You may already know that OpenTracing and OpenCensus have merged to form OpenTelemetry and are wondering why does Jaeger use OpenTracing and if you can use OpenTelemetry to report to Jaeger instead.</p><p>As to why Jaeger uses OpenTracing — well, the reason is that Jaeger existed from before the above-mentioned merger.</p><p>To get a full understanding of OpenTelemetry, what is it, its components, and how you can use with, <a href="https://www.aspecto.io/blog/what-is-opentelemetry-the-infinitive-guide/?utm_source=jaeger-medium&amp;utm_medium=post&amp;utm_campaign=jaeger-tracing-the-ultimate-guide"><strong>read this guide</strong></a>.</p><h3>Deprecation of Jaeger Client in favor of OpenTelemetry Distro:</h3><p>I also mentioned that Jaeger clients are now deprecating.</p><p>You can find more info about this deprecation <a href="https://github.com/jaegertracing/jaeger/issues/3362">here</a>, but essentially the idea is that you should now use the OpenTelemetry SDK in the programming language of your choice, alongside a Jaeger exporter.</p><p>This way created spans would be converted to a format Jaeger knows how to work with, passing all the way through to the Jaeger collector and then to the storage backend.</p><p>At the time of writing this, the OpenTelemetry collector is not considered a replacement for the Jaeger collector [<a href="https://github.com/jaegertracing/jaeger/milestone/12">1</a>]. In the future, the Jaeger collector will be able to receive OTLP, the OpenTelemetry native data format.</p><p>If you can’t wait and want to try using the OpenTelemetry collector with Jaeger now — see <a href="https://medium.com/jaegertracing/jaeger-embraces-opentelemetry-collector-90a545cbc24">this guide</a>.</p><h3>Jaeger Tracing Python Example</h3><p>Here is a Python example of creating spans and sending them to Jaeger. Note that you could also use automatic instrumentations and still use the Jaeger exporter (assuming you’re running Jaeger locally like shown above):</p><pre># jaeger_tracing.py<br>from opentelemetry import trace<br>from opentelemetry.exporter.jaeger.thrift import JaegerExporter<br>from opentelemetry.sdk.resources import SERVICE_NAME, Resource<br>from opentelemetry.sdk.trace import TracerProvider<br>from opentelemetry.sdk.trace.export import BatchSpanProcessor</pre><pre>trace.set_tracer_provider(<br>   TracerProvider(<br>       resource=Resource.create({SERVICE_NAME: &quot;my-hello-service&quot;})<br>   )<br>)</pre><pre>jaeger_exporter = JaegerExporter(<br>   agent_host_name=&quot;localhost&quot;,<br>   agent_port=6831,<br>)</pre><pre>trace.get_tracer_provider().add_span_processor(<br>   BatchSpanProcessor(jaeger_exporter)<br>)</pre><pre>tracer = trace.get_tracer(__name__)</pre><pre>with tracer.start_as_current_span(&quot;rootSpan&quot;):<br>   with tracer.start_as_current_span(&quot;childSpan&quot;):<br>           print(&quot;Hello world!&quot;)</pre><p>This is what they would look like in the Jaeger UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BYbo6cI12ePWQ_OF" /></figure><p>To learn how to hands-on use OpenTelemetry in Python from scratch, <a href="https://www.aspecto.io/blog/getting-started-with-opentelemetry-python/?utm_source=jaeger-medium&amp;utm_medium=post&amp;utm_campaign=jaeger-tracing-the-ultimate-guide"><strong>read this guide</strong></a><strong>.</strong></p><h3>Jaeger Tracing UI Review</h3><p>The Jaeger UI is a powerful tool for us to debug and understand our distributed services better.</p><p>Here’s what you need to know about it:</p><p>The search pane:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/0*TiHLXzFCPMCyXoHM" /></figure><p>You can use the search pane to search for traces with specific properties: which service they come from, what operation was made, specific tags that were included within the trace (for example, the http status code), how long in the past to look for and result amount limiting.</p><p>When you’re done defining your search in this pane, click on Find Traces.</p><p>The search results section:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E8Oe-476BxUPjmU2" /></figure><p>In this example, I chose to query the jaeger-query service. I can see my traces on a timeline or as a list. Click on the desired trace to drill down into it.</p><p>The specific trace view:</p><p>When you find a specific trace where you think there might be an issue and click on it, you’d see a screen that looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FlqtsSJIuUXSb8yz" /></figure><p>Here you can find specific information about execution times, which calls were made, their durations, specific properties like http status code, route path (in the case of an http call), and more.</p><p>Feel free to play around and investigate for yourself with your actual data.</p><h3><strong>Advanced Concepts: Sampling</strong></h3><p>Sampling is a complex topic by itself. in Jaeger the sampling decisions are always made in the SDK, via head-based sampling.</p><h4>Sampling Strategies in the SDK</h4><p>The (deprecating) Jaeger SDKs have 4 sampling modes:</p><ul><li>Remote: the default, and is used to tell the Jaeger SDK that sampling strategy is controlled by the Jaeger backend.</li><li>Constant: either take all traces or take none. Nothing in between. Receives 1 for all and 0 for none</li><li>Rate limiting: choose how many traces would be sampled per second.</li><li>Probabilistic: choose a percentage of traces that would be sampled, for example — choose 0.1 to have 1 of each 10 traces to be sampled.</li></ul><h4>Remote sampling</h4><p>If we choose to enable remote sampling, the Jaeger collector becomes responsible for figuring out which sampling strategy an SDK in each service should be using. The operators have two ways to configure the collector: with a sampling strategies configuration file, or with adaptive sampling.</p><p>Configuration file — you give the collector a path to a file that contains the per-service and pre-operation sampling configuration.</p><p>Adaptive sampling — let Jaeger learn the amount of traffic each endpoint receives and calculate the most appropriate rate for that endpoint. Note that at the time of writing only Memory and Cassandra backends support this.</p><p>More info on Jaeger sampling can be found here: <a href="https://www.jaegertracing.io/docs/1.30/sampling/">https://www.jaegertracing.io/docs/latest/sampling/</a></p><h3>Jaeger Tracing Production Deployment</h3><h4>All-in-one or separate containers?</h4><p>Jaeger all-in-one is a pre-built Docker image containing all the Jaeger components needed to get up and running quickly with Jaeger tracing by launching a single command.</p><p>A lot of people (including my past self) ask themselves what’s the correct way to launch Jaeger in production. If it’s safe to use Jaeger all-in-one in production, etc. While at the time of writing I could not find any official answer to use or not to use it, I think the right answer is — you could, but you probably shouldn’t. Using it as in production means you have a single source of failure which is not distributed. Theoretically, an issue even with the Jaeger UI might crush the entire container and you wouldn’t be able to receive critical spans created by your system.</p><p>The best way to go about this would be to run each Jaeger component separately, without the all-in-one.</p><h3>Mastering OpenTelemetry and Distributed Tracing</h3><p>Jaeger is a distributed tracing beast and the leading open source project for tracing visualization. OpenTelemetry is becoming the industry standard for tracing instrumentation, making it a good place to start learning and implementing traces. To get started with OpenTelemetry, check out this free, vendor-neutral <a href="https://www.aspecto.io/opentelemetry-bootcamp/?utm_source=jaeger-medium&amp;utm_medium=post&amp;utm_campaign=jaeger-tracing-the-ultimate-guide">OpenTelemetry Bootcamp.</a></p><p>The <a href="https://www.aspecto.io/opentelemetry-bootcamp/?utm_source=jaeger-medium&amp;utm_medium=post&amp;utm_campaign=jaeger-tracing-the-ultimate-guide">Bootcamp</a> includes</p><ul><li>Episode 1: OpenTelemetry Fundamentals</li><li>Episode 2: Integrate Your Code (logs, metrics, and traces)</li><li>Episode 3: Deploy to Production + Collector</li><li>Episode 4: Sampling and Dealing with High Volumes</li><li>Episode 5: Custom Instrumentation</li><li>Episode 6: Testing with OpenTelemetry</li></ul><p>If you have any questions, feel free to reach out to me on Twitter <a href="https://twitter.com/thetomzach">@thetomzach</a> and to join our <a href="https://cloud-native.slack.com/messages/opentelemetry-bootcamp">#OpenTelemetry-Bootcamp</a> slack channel to be on top of what’s happening in observability.</p><h3>Jaeger Tracing Glossary</h3><p><strong>Span</strong> — a representation of a unit of work (action/operation) that occurs in our system; an HTTP request or a database operation that spans over time (start at X and has a duration of Y milliseconds). Usually, it will be the parent and/or child of another span.</p><p><strong>Trace</strong> — a tree/list of spans representing the progression of requests as it is handled by the different services and components in our system. For example, sending an API call to user-service resulted in a DB query to users-db. They are ‘call-stacks’ for distributed services.</p><p><strong>Observability</strong> — a measure of how well we can understand the internal states of a system based on its external outputs. When you have logs, metrics, and traces you have the “3 pillars of observability”.</p><p><strong>OpenTelemetry</strong> — OpenTelemetry is an open-source project at CNCF (Cloud Native Computing Function) that provides a collection of tools, APIs, and SDKs. OpenTelemetry enables the automated collection and generation of traces, logs, and metrics with a single specification.</p><p><strong>OpenTracing</strong> — an open-source project for distributed tracing. It was deprecated and “merged” into OpenTelemetry. OpenTelemetry offers backward compatibility for OpenTracing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b53a4a568ca" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jaegertracing/jaeger-tracing-a-friendly-guide-for-beginners-7b53a4a568ca">Jaeger Tracing: A Friendly Guide for Beginners</a> was originally published in <a href="https://medium.com/jaegertracing">JaegerTracing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
+        </item>
+        <item>
+            <title><![CDATA[Adaptive Sampling in Jaeger]]></title>
+            <link>https://medium.com/jaegertracing/adaptive-sampling-in-jaeger-50f336f4334?source=rss----99735986d50---4</link>
+            <guid isPermaLink="false">https://medium.com/p/50f336f4334</guid>
+            <category><![CDATA[adaptive-sampling]]></category>
+            <category><![CDATA[sampling]]></category>
+            <category><![CDATA[jaeger]]></category>
+            <dc:creator><![CDATA[Yuri Shkuro]]></dc:creator>
+            <pubDate>Mon, 17 Jan 2022 15:44:58 GMT</pubDate>
+            <atom:updated>2022-01-17T15:44:58.783Z</atom:updated>
+            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NKMqp96KpCm1hyhDyGP7fg.jpeg" /><figcaption>Illustration © by Lev Polyakov</figcaption></figure><p>In collaboration with <a href="https://medium.com/@joe_19459">Joe Elliott</a>.</p><p>In distributed tracing, sampling is frequently used to reduce the number of traces that are collected and stored in the backend. This is often desirable because it is easy to produce more data than can be efficiently stored and queried. Sampling allows us to store only a subset of the total traces produced.</p><p>Traditionally Jaeger SDKs have supported a variety of <a href="https://www.jaegertracing.io/docs/latest/sampling/">sampling techniques</a>. Our favorite has always been so called <em>remote sampling</em>, a feature pioneered in open source by the Jaeger project. In this setup, the Jaeger SDKs would query the Jaeger backend to retrieve a configuration of sampling rules for the given service, up to the granularity of individual endpoints. This can be a very powerful method of sampling as it can give operators central control of sampling rates across an entire organization.</p><p>Until recently, the only way to control which sampling rules are returned by the backend in the remote sampling mode was with a configuration file provided to the collector via the --sampling.strategies-file flag. Usually, the operators must manually update this file to push out different sampling rules. <em>Adaptive sampling</em>, added in v1.27.0, allows the collectors to automatically adjust sampling rates to meet preconfigured goals, by observing the current traffic in the system and the number of traces collected. This feature has been in production at Uber for several years, and is finally available on the open source version of Jaeger. A special shout-out to <a href="https://medium.com/@joe_19459">Joe Elliott</a> for completing the adoption of the code contributed by Uber, which was sitting in the Jaeger repo for two years without being wired into the collector’s main.</p><h3>Why do we need remote &amp; adaptive sampling?</h3><p>It is always possible to configure the SDK to apply a very simple sampling strategy like a coin-flip decision, also known as probabilistic sampling. This might work fine in a small application, but when your architecture is measured in 100s or even 1000s of services, which all have different volumes of traffic, a single sampling probability for every service does not work that well, and configuring it individually for each service is a deployment nightmare. Remote sampling addresses this problem by centralizing all sampling configuration in the Jaeger collectors, where changes can be pushed out quickly to any service.</p><p>However, configuring sampling rules for every service manually, even if centrally, is still very tedious. Adaptive sampling takes this a step further and transforms this into a <em>declarative</em> configuration, where the operator only needs to set the target rate of trace collection, and the adaptive sampling engine dynamically adjusts the sampling rates individually for each service and each endpoint.</p><p>Another benefit of adaptive sampling is that it can automatically react to changes in the traffic. Many online services exhibit fluctuations in traffic during the day, e.g. Uber would have higher volume of requests during peak hours. Adaptive sampling engine would automatically adjust the sampling rates to keep the volume of trace data stable and within our sampling budget.</p><h3>How to set up adaptive sampling?</h3><p>First, adaptive sampling requires that the Jaeger SDKs reach out to the backend to request the remote sampling document. This can be configured using environment variables. Please refer to the <a href="https://www.jaegertracing.io/docs/1.30/client-features/">client features documentation</a> to confirm that these are supported by your Jaeger client.</p><pre>JAEGER_SAMPLER_TYPE=remote<br>JAEGER_SAMPLING_ENDPOINT=&lt;sampling endpoint on the jaeger agent&gt;</pre><p>The defaults are generally set up to work with a local Jaeger agent, running as a host agent or a sidecar, so it’s possible your setup is already close to working. Jaeger SDK configuration actually defaults to this:</p><pre>JAEGER_SAMPLER_TYPE=remote<br>JAEGER_SAMPLING_ENDPOINT=http://127.0.0.1:5778/sampling</pre><p>After your clients are configured you will need to make sure that your collectors are configured correctly to store adaptive sampling information. Currently, Jaeger uses the same storage for adaptive sampling as span storage and the only supported storage options for adaptive sampling are cassandra (since v.1.27) and memory (since v1.28). Using the environment variables to configure your collector may look something like:</p><pre>SPAN_STORAGE_TYPE=cassandra<br>SAMPLING_CONFIG_TYPE=adaptive</pre><p>If you’re just getting started, we encourage you to check out this simple <a href="https://github.com/jaegertracing/jaeger/blob/50764f9ee4ac2d897331565066064ad2a3865fc9/docker-compose/jaeger-docker-compose.yml">docker-compose example</a> which starts up Jaeger in a configuration that supports adaptive sampling.</p><p>The adaptive sampling algorithm can be tuned with a number of parameters that can be found in the <a href="https://www.jaegertracing.io/docs/1.30/cli/#jaeger-collector">documentation</a>, or form the help command:</p><pre>$ docker run --rm \<br>  -e SAMPLING_CONFIG_TYPE=adaptive \<br>  jaegertracing/jaeger-collector:1.30 \<br>  help | grep -e &#39;--sampling.&#39;</pre><pre>      --sampling.aggregation-buckets int                          Amount of historical data to keep in memory. (default 10)<br>      --sampling.buckets-for-calculation int                      This determines how much of the previous data is used in calculating the weighted QPS, ie. if BucketsForCalculation is 1, only the most recent data will be used in calculating the weighted QPS. (default 1)<br>      --sampling.calculation-interval duration                    How often new sampling probabilities are calculated. Recommended to be greater than the polling interval of your clients. (default 1m0s)<br>      --sampling.delay duration                                   Determines how far back the most recent state is. Use this if you want to add some buffer time for the aggregation to finish. (default 2m0s)<br>      --sampling.delta-tolerance float                            The acceptable amount of deviation between the observed samples-per-second and the desired (target) samples-per-second, expressed as a ratio. (default 0.3)<br>      --sampling.follower-lease-refresh-interval duration         The duration to sleep if this processor is a follower. (default 1m0s)<br>      --sampling.initial-sampling-probability float               The initial sampling probability for all new operations. (default 0.001)<br>      --sampling.leader-lease-refresh-interval duration           The duration to sleep if this processor is elected leader before attempting to renew the lease on the leader lock. This should be less than follower-lease-refresh-interval to reduce lock thrashing. (default 5s)<br>      --sampling.min-samples-per-second float                     The minimum number of traces that are sampled per second. (default 0.016666666666666666)<br>      --sampling.min-sampling-probability float                   The minimum sampling probability for all operations. (default 1e-05)<br>      --sampling.target-samples-per-second float                  The the global target rate of samples per operation. (default 1)</pre><h3>How does adaptive sampling work?</h3><p>We start with some default sampling probability p assigned to every endpoint and a target rate R of traces we want to collect, such as 1 trace per second per endpoint. The collectors monitor the spans passing through them, looking for root spans of the traces started with this sampling policy, and calculate the actual rate of traces R’ being collected. If R’ &gt; R then our current probability for this endpoint is too high and needs to be reduced. Conversely, if R’ &lt; R then we need to increase the probability. Since the actual traffic is always a bit noisy, the situation where R’ == R rarely occurs, so the collector uses a certain tolerance threshold k such that the above rules are actually R’ &gt; R + k and R’ &lt; R — k. Once the new probability p’ is calculated, the collector waits for a certain time interval to make sure it was retrieved by the SDKs and applied to new traces, then observes a new value of rate R’ and repeats the cycle. Yuri Shkuro’s book <a href="https://www.shkuro.com/books/2019-mastering-distributed-tracing/">Mastering Distributed Tracing</a> contains a more detailed description of the math involved in the adaptive probability calculations implemented in the Jaeger collectors.</p><p>We also need to talk about how this is all done given that Jaeger allows us to run multiple collectors simultaneously. The adaptive sampling module implements a simple leader election mechanism using compare-and-swap operations supported by the storage backends. Each collector receives a distinct stream of spans from the services (remember, we’re only interested in the root spans since that is where the sampling decision always happens), and maintains an in-memory aggregate of trace counts for each service / endpoint pair. Then after a certain time interval each collector writes this data (referred to as throughput in the code) to the storage backend. Then the collector that won the leader election reads all throughput data from storage for a given time range, aggregates it, performs the probability calculations, and writes the new probabilities summary for all services back to storage. The other collectors load that summary and use it to serve the requests for sampling strategies from the SDKs. Note that the leader election in this model is purely an optimization, because the sampling summary is written under a stable time-based key known to all collectors, so if more than one collector happens to perform the calculation of the probabilities, they would just override each other’s writes with the same data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qtt02KOzwajdyn1cFbMcfw.png" /><figcaption>High level architecture of Jaeger’s adaptive sampling engine.</figcaption></figure><h3>What’s missing?</h3><p>There are a few features on our wishlist that would make adaptive sampling better. One is the ability to account for the total number of spans instead of the total number of traces. Different endpoints can result in very different sizes of traces, even by several orders of magnitude. Yet the current implementation is only built around counting traces. It can be extended with additional heuristics, such as calculating the average trace size per endpoint offline and providing the adaptive sampling engine with a weights matrix to take into account when computing actual throughput.</p><p>Another nice-to-have feature, which actually requires changes in the remote sampling configuration, is to use other dimensions from trace data besides service name and endpoint name that are currently hardcoded in the schema.</p><p>And yet another useful extension would be a configuration mechanism to allow overriding the target throughput rateR for specific services / endpoints, instead of using a single global parameter, because some services may be more important to your business and you might want to collect more data for them, or perhaps this could be a temporary setting due to some investigation.</p><h3>Wrapping up</h3><p>We are pleased to release the first open source end-to-end implementation of adaptive sampling for the Jaeger community. Please give it a shot, provide feedback and let us continue iterating on this feature.</p><p>If you’re interested in contributing to the future development of this feature, there are a couple of areas where we could use some help immediately:</p><ol><li>Supporting ElasticSearch / OpenSearch as the backend for storing adaptive sampling data.</li><li>Decoupling Jaeger storage configuration so that different storage backends could be used for span storage and adaptive sampling.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50f336f4334" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jaegertracing/adaptive-sampling-in-jaeger-50f336f4334">Adaptive Sampling in Jaeger</a> was originally published in <a href="https://medium.com/jaegertracing">JaegerTracing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
+        </item>
+        <item>
+            <title><![CDATA[Migrating from Jaeger client to OpenTelemetry SDK]]></title>
+            <link>https://medium.com/jaegertracing/migrating-from-jaeger-client-to-opentelemetry-sdk-bd337d796759?source=rss----99735986d50---4</link>
+            <guid isPermaLink="false">https://medium.com/p/bd337d796759</guid>
+            <category><![CDATA[opentelemetry]]></category>
+            <category><![CDATA[opentracing]]></category>
+            <category><![CDATA[distributed-tracing]]></category>
+            <category><![CDATA[jaeger]]></category>
+            <dc:creator><![CDATA[Juraci Paixão Kröhling]]></dc:creator>
+            <pubDate>Tue, 26 Oct 2021 08:07:06 GMT</pubDate>
+            <atom:updated>2022-01-17T16:24:37.513Z</atom:updated>
+            <content:encoded><![CDATA[<h3>Migrating from Jaeger Java client to OpenTelemetry SDK</h3><p>A couple of years ago, the <a href="https://opentelemetry.io/">OpenTelemetry</a> project was founded by the merger of two similarly aimed projects: <a href="https://opentracing.io/">OpenTracing</a> and <a href="https://opencensus.io/">OpenCensus</a>. One of the goals of this new project was to create an <a href="https://medium.com/opentracing/merging-opentracing-and-opencensus-f0fe9c7ca6f0">initial version that would “just work” with existing applications</a> instrumented using OpenTracing and OpenCensus.</p><p>On the Jaeger community, we decided some time ago that we would start recommending users to migrate to OpenTelemetry SDK once there was feature-parity with our existing clients, and we believe this time has come.</p><p>In the next few months, we’ll be deprecating our own clients in favor of the OpenTelemetry SDK. With this, we believe we’ll remain focused on the backend side of our tracing solution, leaving a bigger community to provide and support clients in a myriad of languages.</p><p>This guide will help you with your first steps in migrating to OpenTelemetry SDK from Jaeger for your application instrumented using the OpenTracing API. For the longer term, we recommend that you get familiar with the OpenTelemetry API and start using it to instrument your applications. We also recommend that you get familiar with the OpenTelemetry SDK and understand its features and limitations.</p><h3>Our instrumented application</h3><p>For this guide, we’ll be using <a href="https://github.com/yurishkuro/opentracing-tutorial">Yuri Shkuro’s OpenTracing tutorial</a> as the starting point. More specifically, <a href="https://github.com/yurishkuro/opentracing-tutorial/tree/master/java/src/main/java/lesson04/solution">the solution for lesson 4</a>. Before we start the migration, let’s do a sanity check. Fork and clone that repository, and run each one of the following commands on its own console:</p><pre>$ ./run.sh lesson04.solution.Formatter server<br>$ ./run.sh lesson04.solution.Publisher server<br>$ podman run --rm --name jaeger -p 6831:6831/udp -p 14250:14250 -p 16686:16686 -p 14268:14268 jaegertracing/all-in-one:1.27</pre><p>If you don’t have podman, replace it with docker in the last command. Even though we are using a recent version of the Jaeger backend, any version bigger than v1.8.2 (2018-11-28) should work.</p><p>Once all the servers are running, execute the client:</p><pre>$ ./run.sh lesson04.solution.Hello Bryan Bonjour</pre><p>You should now see a trace in <a href="http://localhost:16686">your local Jaeger instance</a> similar to the image below. If this is the case, you’re good to go. Stop the Formatter and the Publisher servers, but leave Jaeger running.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kHD8sLyWsbEcR0lqMhFFNA.png" /><figcaption>A trace generated by the Jaeger Client for Java</figcaption></figure><h3>Adding the OpenTelemetry shim</h3><p>Before we can start using the shim, we need to add two BOMs (Bill of Materials) to our application. The first one contains the stable components for the OpenTelemetry Java SDK like the actual SDK, the API, and a few exporters, where the second has components that might not have the same stability or criticality, like the shim or the semantic conventions library.</p><pre>&lt;dependencyManagement&gt;<br>    &lt;dependencies&gt;<br>        &lt;dependency&gt;<br>            &lt;groupId&gt;io.opentelemetry&lt;/groupId&gt;<br>            &lt;artifactId&gt;opentelemetry-bom&lt;/artifactId&gt;<br>            &lt;version&gt;1.7.0&lt;/version&gt;<br>            &lt;type&gt;pom&lt;/type&gt;<br>            &lt;scope&gt;import&lt;/scope&gt;<br>        &lt;/dependency&gt;<br>        &lt;dependency&gt;<br>            &lt;groupId&gt;io.opentelemetry&lt;/groupId&gt;<br>            &lt;artifactId&gt;opentelemetry-bom-alpha&lt;/artifactId&gt;<br>            &lt;version&gt;1.7.0-alpha&lt;/version&gt;<br>            &lt;type&gt;pom&lt;/type&gt;<br>            &lt;scope&gt;import&lt;/scope&gt;<br>        &lt;/dependency&gt;<br>    &lt;/dependencies&gt;<br>&lt;/dependencyManagement&gt;</pre><p>Next, we add the dependencies related to the OpenTelemetry SDK to our project:</p><pre>&lt;dependency&gt;<br>    &lt;groupId&gt;io.opentelemetry&lt;/groupId&gt;<br>    &lt;artifactId&gt;opentelemetry-opentracing-shim&lt;/artifactId&gt;<br>&lt;/dependency&gt;<br><br>&lt;dependency&gt;<br>    &lt;groupId&gt;io.opentelemetry&lt;/groupId&gt;<br>    &lt;artifactId&gt;opentelemetry-semconv&lt;/artifactId&gt;<br>&lt;/dependency&gt;<br><br>&lt;dependency&gt;<br>    &lt;groupId&gt;io.opentelemetry&lt;/groupId&gt;<br>    &lt;artifactId&gt;opentelemetry-exporter-jaeger&lt;/artifactId&gt;<br>&lt;/dependency&gt;<br><br>&lt;dependency&gt;<br>    &lt;groupId&gt;io.opentelemetry&lt;/groupId&gt;<br>    &lt;artifactId&gt;opentelemetry-extension-trace-propagators&lt;/artifactId&gt;<br>&lt;/dependency&gt;</pre><ul><li>io.opentelemetry:opentelemetry-opentracing-shim is our shim, an OpenTracing Tracer implementation that will serve as the drop-in replacement for our Jaeger client.</li><li>io.opentelemetry:opentelemetry-semconv is a convenience library for adding attributes based on the semantic conventions.</li><li>io.opentelemetry:opentelemetry-exporter-jaeger is an exporter that sends data to our Jaeger instance. Note that there are <a href="https://github.com/open-telemetry/opentelemetry-java/issues/1387">no exporters sending Thrift data over UDP</a>, which was the default encoding and transport for most Jaeger clients. Even though there is a Thrift HTTP exporter, we recommend using the gRPC exporter. Take a moment also to review the <a href="https://opentelemetry.io/registry/?language=java&amp;component=exporter">available exporters</a>.</li><li>io.opentelemetry:opentelemetry-extension-trace-propagators contains the Jaeger propagator.</li></ul><p>We also need a couple of gRPC dependencies. We are using netty-shaded as the underlying transport, but make sure to learn about the alternatives and pick the appropriate one for your scenario.</p><pre>&lt;dependency&gt;<br>    &lt;groupId&gt;io.grpc&lt;/groupId&gt;<br>    &lt;artifactId&gt;grpc-protobuf&lt;/artifactId&gt;<br>    &lt;version&gt;1.41.0&lt;/version&gt;<br>&lt;/dependency&gt;<br>&lt;dependency&gt;<br>    &lt;groupId&gt;io.grpc&lt;/groupId&gt;<br>    &lt;artifactId&gt;grpc-netty-shaded&lt;/artifactId&gt;<br>    &lt;version&gt;1.41.0&lt;/version&gt;<br>&lt;/dependency&gt;</pre><p>At this point, we can remove the io.jaegertracing:jaeger-client dependency from our project.</p><h3>Replacing the tracer</h3><p>With the dependencies in place, we can now replace the Tracer in our application with the OpenTelemetry Tracer. Most applications should have used the Tracer interface from the OpenTracing API in the method signatures instead of referencing the JaegerTracer directly. This makes our job easier now, as our changes are localized to the lib.Tracing class, init method.</p><p>The first step in our refactoring will be to remove the entire implementation and just return null. This is located in the java/src/main/java/lib/Tracing.java file.</p><pre>public static Tracer init(String service) {<br>    return null;<br>}</pre><p>Take the opportunity to also remove the imports starting with io.jaegertracing.</p><p>We’ll now create the Resource attribute holding the service name for our application. In the init method, add the following:</p><pre>Resource serviceNameResource = Resource.create(Attributes.of(ResourceAttributes.SERVICE_NAME, service));</pre><p>We now initialize a gRPC channel with our Jaeger collector. In the init method, add the following:</p><pre>ManagedChannel jaegerChannel = ManagedChannelBuilder<br>    .forAddress(&quot;localhost&quot;, 14250)<br>    .usePlaintext()<br>    .build();</pre><p>We can now create the Jaeger exporter:</p><pre>JaegerGrpcSpanExporter jaegerExporter = JaegerGrpcSpanExporter.builder()<br>    .setChannel(jaegerChannel)<br>    .setTimeout(1, TimeUnit.SECONDS)<br>    .build();</pre><p>Creating the Tracer provider is the next step:</p><pre>SdkTracerProvider tracerProvider = SdkTracerProvider.builder()<br>    .addSpanProcessor(SimpleSpanProcessor.create(jaegerExporter))<br>    .setResource(Resource.getDefault().merge(serviceNameResource))<br>    .build();</pre><p>We create an OpenTelemetry SDK instance with our tracer provider and the context propagators. To keep backward compatibility with existing services, we added the JaegerPropagator. For the long run though, we might want to start using the W3CTraceContextPropagator instead. Given that not all services are going to be updated at once, it&#39;s a good idea to run with both propagators during the transition time. The side-effect is that we&#39;ll end up having a bigger HTTP request between our microservices, but hopefully that&#39;s not big enough to have a considerable impact.</p><pre>OpenTelemetrySdk openTelemetry = OpenTelemetrySdk.builder()<br>    .setPropagators(ContextPropagators.create(<br>        TextMapPropagator.composite(<br>            W3CTraceContextPropagator.getInstance(),<br>            JaegerPropagator.getInstance()<br>        )<br>    ))<br>    .setTracerProvider(tracerProvider)<br>    .build();</pre><p>As a good practice, we try to close our tracer provider when the JVM is shutting down:</p><pre>Runtime.getRuntime().addShutdownHook(new Thread(tracerProvider::close));</pre><p>And finally, we wrap our OpenTelemetry SDK instance in our shim, returning it to callers:</p><pre>return OpenTracingShim.createTracerShim(openTelemetry);</pre><h3>Trying it out</h3><p>Our migration should be ready by now, so, let’s try it out by running the same commands as before:</p><pre>./run.sh lesson04.solution.Formatter server<br>./run.sh lesson04.solution.Publisher server<br>./run.sh lesson04.solution.Hello Bryan Bonjour</pre><p>At this point, we should have a new trace in our Jaeger instance, created by the OpenTelemetry SDK: confirm this is the case by checking the otel.library.name tag and telemetry.sdk.name process attribute, like in the following image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P-ntq0qqfcy0qGcV2rkD3g.png" /><figcaption>A trace generated by the OpenTelemetry SDK for Java</figcaption></figure><h3>Remote sampling</h3><p>If you are using the remotely controlled sampling configuration in the Jaeger client, you should double-check if the language you are using supports it already. Even though the <a href="https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/sdk-environment-variables.md#general-sdk-configuration">jaeger_remote sampler is a valid value for the env var </a><a href="https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/sdk-environment-variables.md#general-sdk-configuration">OTEL_TRACES_SAMPLER</a> as per the OpenTelemetry SDK specification, it is not supported by most languages yet. At the moment of this writing, only the OpenTelemetry Java SDK supports it.</p><h3>Wrapping up</h3><p>This migration was simple and without a lot of work: a localized refactor of the init method was all it took. A bigger migration is certainly switching from the OpenTracing API to the OpenTelemetry API, but with the shim in place, you can take an opportunistic approach and migrate the APIs only for the services that are undergoing some maintenance or refactoring already.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd337d796759" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jaegertracing/migrating-from-jaeger-client-to-opentelemetry-sdk-bd337d796759">Migrating from Jaeger client to OpenTelemetry SDK</a> was originally published in <a href="https://medium.com/jaegertracing">JaegerTracing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
+        </item>
+        <item>
+            <title><![CDATA[Jaeger Persistent Storage With Elasticsearch, Cassandra & Kafka]]></title>
+            <link>https://medium.com/jaegertracing/jaeger-persistent-storage-with-elasticsearch-cassandra-kafka-f3c6c680a58c?source=rss----99735986d50---4</link>
+            <guid isPermaLink="false">https://medium.com/p/f3c6c680a58c</guid>
+            <category><![CDATA[jaegertracing]]></category>
+            <category><![CDATA[storage]]></category>
+            <category><![CDATA[distributed-tracing]]></category>
+            <category><![CDATA[high-availability]]></category>
+            <category><![CDATA[devops]]></category>
+            <dc:creator><![CDATA[Dotan Horovits (@horovits)]]></dc:creator>
+            <pubDate>Wed, 24 Feb 2021 16:29:22 GMT</pubDate>
+            <atom:updated>2021-02-28T16:26:15.774Z</atom:updated>
+            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6wjV2PGdDbt1z_sNHNVokQ.jpeg" /></figure><p>Running systems in production involves requirements for high availability, resilience and recovery from failure. When running cloud native applications this becomes even more critical, as the base assumption in such environments is that compute nodes will suffer outages, Kubernetes nodes will go down and microservices instances are likely to fail, yet the service is expected to remain up and running.</p><p>In a recent post, I presented the different Jaeger components and best practices for <a href="https://medium.com/jaegertracing/a-guide-to-deploying-jaeger-on-kubernetes-in-production-69afb9a7c8e5">deploying Jaeger in production</a>. In that post, I mentioned that Jaeger uses external services for ingesting and persisting the span data, such as Elasticsearch, Cassandra and Kafka. This is due to the fact that the Jaeger Collector is a stateless service and you need to point it to some sort of storage to which it will forward the span data.</p><p>In this post, I’d like to discuss how to ingest and persist Jaeger trace data in production to ensure resilience and high availability, and the external services you need to set up for that. I’ll cover:</p><ul><li>Standard persistent storage for Jaeger with Elasticsearch and Cassandra</li><li>Alternative persistent storage with gRPC plugin</li><li>Handling high load tracing data streams with Kafka</li><li>Jaeger persistence during development with<a href="https://www.jaegertracing.io/docs/1.18/cli/#jaeger-all-in-one"> jaegertracing all-in-one</a></li></ul><h3>Deploying Jaeger with Elasticsearch, Kafka or other External Services</h3><p>Jaeger deployments may involve additional services such as Elasticsearch, Cassandra and Kafka. But do these services come as part of Jaeger’s installation and how are these services deployed?</p><p>The Jaeger Operator and Jaeger’s Helm chart (see Jaeger’s deployment tools in <a href="https://logz.io/blog/jaeger-kubernetes-best-practices/?utm_source=jaegerblog&amp;utm_medium=social&amp;utm_campaign=community_outreach&amp;utm_content=jaeger">this post</a>) offer the option of a self-provisioned Elasticsearch/Cassandra/Kafka cluster (in which Jaeger deployment also deploys these clusters), as well as the option of connecting to an existing cluster.</p><p>The self-provisioned option offers a good starting point, but you may prefer to deploy these services independently for better flexibility and control over the way these clusters are deployed, managed, monitored, upgraded and secured, in accordance with your team’s DevOps practices. In particular, if you are already running a Kafka or Elasticsearch cluster, it may make more sense to re-use these infrastructure components rather than maintain a separate cluster.</p><h3>Elasticsearch vs. Cassandra as Jaeger Backend Storage</h3><p>For production deployments, Jaeger currently provides built-in support for two storage solutions, both of which are very popular open source NoSQL databases: Elasticsearch and Cassandra. The Jaeger collector and query service need to be configured with the storage solution of choice so they can write to it and query it. You can pass the desired storage type and the database endpoint via environment variables. For example, a basic Elasticsearch setup will define the following environment variables:</p><pre>SPAN_STORAGE_TYPE=elasticsearch<br>ES_SERVER_URLS=&lt;...&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*mC9p0H_o-JcEWWBiGubj8w.png" /><figcaption>Illustration of direct-to-storage architecture. Source: jaegertracing.io</figcaption></figure><p>So which storage backend should you use: Elasticsearch or Cassandra?</p><p>The Jaeger team provides a clear <strong>recommendation to use Elasticsearch as the storage backend</strong> over Cassandra. And they have very good <a href="https://www.jaegertracing.io/docs/1.18/faq/">reasons</a>:</p><ul><li>Cassandra is a key-value database, so it is more efficient for retrieving traces by trace ID, but it does not provide the same powerful search capabilities as Elasticsearch. Effectively, the Jaeger backend implements the search functionality on the client side, on top of k-v storage, which is limited and may produce inconsistent results (see <a href="https://github.com/jaegertracing/jaeger/issues/166">issue-166</a> for more details). Elasticsearch does not suffer from these issues, resulting in better usability. Elasticsearch can also be queried directly, e.g. from Kibana dashboards, and provide useful analytics and aggregations.</li><li>Based on past performance experiments we observed single writes to be much faster in Cassandra than Elasticsearch, which might suggest that it may sustain higher write throughput. However, because the Jaeger backend needs to implement search capability on top of k-v storage, writing spans to Cassandra is actually subject to large write amplification: in addition to writing a record for the span itself, Jaeger performs extra writes for service name and operation name indexing, as well as extra index writes for every tag. In contrast, saving a span to Elasticsearch is a single write, and all indexing takes place inside the ES node. As a result, the overall throughput to Cassandra is comparable with Elasticsearch.</li></ul><p>One benefit of Cassandra backend is simplified maintenance due to its native support for data TTL. In Elasticsearch the data expiration is managed through index rotation, which requires additional setup (see <a href="https://logz.io/blog/managing-elasticsearch-indices/#elasticsearch-rollover">Elasticsearch Rollover</a>).</p><h3>Alternative Persistent Storage for Jaeger</h3><p>In addition to Jaeger’s built-in support for Elasticsearch and Cassandra, Jaeger supports a <a href="https://github.com/jaegertracing/jaeger/tree/master/plugin/storage/grpc">gRPC plugin</a> (SPAN_STORAGE_TYPE=grpc-plugin) which enables developing custom plugins to other storage types. The Jaeger community currently offers integrations with several persistent storage types, four of which are defined as ‘available’ at present: ScyllaDB, InfluxDB, Couchbase and <a href="https://github.com/logzio/jaeger-logzio">Logz.io</a> (disclaimer: I work at Logz.io).</p><p>Other integrations, which are not yet available, include NoSQL data stores from the big cloud vendors such as Amazon DynamoDB, Azure CosmosDB and Google BigTable, as well as popular SQL databases MySQL and PostgreSQL. You can check out the list of additional storage backends and updated status on this Jaeger GitHub <a href="https://github.com/jaegertracing/jaeger/issues/638">issue</a>.</p><h3>Using Kafka to Ingest High-Load Jaeger Span Data</h3><p>If you monitor many microservices, if you have a high volume of span data, or if your system generates data bursts on occasions, then your external backend storage may not be able to handle the load and may become a bottleneck, impacting the overall performance. In such cases you should employ the streaming deployment strategy that I mentioned in the previous post which uses Kafka between the Collector and the storage to buffer the span data from the Jaeger Collector.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*QqyTQNP1GI4jQhNyGWF1pA.png" /><figcaption>Illustration of architecture with Kafka as intermediate buffer. Source: jaegertracing.io</figcaption></figure><p>In this case, you configure Kafka as the target for Jaeger Collector ( SPAN_STORAGE_TYPE=kafka ) as well as the relevant Kafka brokers, topic and other parameters.</p><p>I’d like to stress that Kafka is not an alternative backend storage (although the setting SPAN_STORAGE_TYPE=kafka may be confusing). Your Jaeger backend still needs a backend storage as described in the previous sections, with Kafka serving as a buffer to take off pressure.</p><p>To support the streaming deployment Jaeger project also offers the Jaeger Ingester service, which can asynchronously read from Kafka topic and write to the storage backend (Elasticsearch or Cassandra). Of course, you can choose to implement your own service to do the same, if you need a particular target storage or ingestion strategy.</p><h3>Jaeger Persistence During Development with jaegertracing All-in-One</h3><p>Up till now I discussed production deployment. However, if you are exploring Jaeger or are doing a small PoC or development, then you are probably using Jaeger’s All-in-One installation, and you may be wondering how this is applicable to you.</p><p>All-in-one is a single node installation, in which you don’t trouble yourself with non-functional requirements such as resilience or scalability. In an all-in-one deployment, Jaeger uses in-memory persistence by default. Alternatively, you can choose to use Badger, which provides a single-node storage based on a filesystem (similar to Prometheus model). You can find more details on using Badger <a href="https://www.jaegertracing.io/docs/latest/deployment/#badger---local-storage">here</a>.</p><p>Bear in mind that both in-memory and Badger are meant for all-in-one deployments only, and are not suitable for production deployments.</p><h3>Endnote</h3><p>When deploying <a href="https://logz.io/blog/jaeger-kubernetes-best-practices/?utm_source=jaegerblog&amp;utm_medium=social&amp;utm_campaign=community_outreach&amp;utm_content=jaeger">Jaeger in production</a>, you need to address data persistence, high availability and scalability concerns. In order to address these concerns you need to deploy additional services.</p><p>First of all, you should deploy and configure an external persistence storage for your span data. The recommended persistence storage for Jaeger in production is Elasticsearch.</p><p>Secondly, when dealing with high load of span data, you should deploy Kafka in front of the storage to handle the ingestion and provide backpressure.</p><p>Running in production entails many other considerations not covered in this post, such as upgrades to Jaeger components as well as Elasticsearch, Kafka or any additional service in the deployment; monitoring the different services, and securing access to these services.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3c6c680a58c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jaegertracing/jaeger-persistent-storage-with-elasticsearch-cassandra-kafka-f3c6c680a58c">Jaeger Persistent Storage With Elasticsearch, Cassandra &amp; Kafka</a> was originally published in <a href="https://medium.com/jaegertracing">JaegerTracing</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
+        </item>
+    </channel>
+</rss>
\ No newline at end of file
diff --git a/themes/jaeger-docs/layouts/index.html b/themes/jaeger-docs/layouts/index.html
index 1a202b1d..101abedf 100644
--- a/themes/jaeger-docs/layouts/index.html
+++ b/themes/jaeger-docs/layouts/index.html
@@ -1,18 +1,21 @@
 {{ define "main" }}
-{{ $rss := dict }}
-{{ with try (resources.GetRemote "https://medium.com/feed/jaegertracing") }}
-  {{ with .Err }}
-    {{ errorf "%s" . }}
-  {{ else with .Value }}
-    {{ $rss = . | transform.Unmarshal }}
+
+  {{ $rss := resources.Get "data/medium.xml" | transform.Unmarshal }}
+  {{ $channel := dict }}
+
+  {{ if not $rss }}
+    {{ warnf "Medium feed is nil or failed to load." }}
+    {{ $channel = dict "title" "No data" "item" (slice) }}
+  {{ else if not ($rss.channel) }}
+    {{ warnf "Medium feed found but missing 'channel'" }}
+    {{ $channel = dict "title" "No data" "item" (slice) }}
   {{ else }}
-    {{ warnf "Unable to fetch RSS feed" }}
+    {{ $channel = $rss.channel }}
+    {{ warnf "Loaded Medium feed for: %v" $channel.title }}
   {{ end }}
-{{ end }}
-
-{{/* Parse image links */}}
+  
 {{ $defaultImgLink := "/img/jaeger-icon-color.png" }}
-{{ warnf "Downloaded RSS feed for: %v" $rss.channel.title }}
+{{ warnf "Loaded RSS feed for: %v" $rss.channel.title }}
 {{ $posts := first 20 $rss.channel.item }}
 {{ $imgLinks := slice }}
 


Reviews:
user: login: yurishkuro
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-17T19:51:36Z
user: login: yurishkuro
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-17T19:52:08Z
user: login: yurishkuro
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-18T17:28:53Z
user: login: abhayguptas
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-18T17:34:00Z

Review comments:
diff_hunk: @@ -56,6 +56,9 @@ spellcheck:
 	cat scripts/cspell/project-names.txt | grep -v '^#' | grep -v '^\s*$$' | tr ' ' '\n' > scripts/cspell/project-names-parsed.txt
 	cd scripts/cspell && ./spellcheck.sh
 
+fetch-blog-feed:
+	curl -s https://medium.com/feed/jaegertracing | yq -p xml -o json -P > data/medium.json
path: Makefile
user: login: yurishkuro
body: Hugo can read XML file directly, there is no reason to convert to JSON (and JSON will introduce more escaping)
created_at: 2025-05-17T19:51:35Z
author_association: MEMBER
diff_hunk: @@ -56,6 +56,9 @@ spellcheck:
 	cat scripts/cspell/project-names.txt | grep -v '^#' | grep -v '^\s*$$' | tr ' ' '\n' > scripts/cspell/project-names-parsed.txt
 	cd scripts/cspell && ./spellcheck.sh
 
+fetch-blog-feed:
+	curl -s https://medium.com/feed/jaegertracing | yq -p xml -o json -P > data/medium.json
path: Makefile
user: login: yurishkuro
body: I prefer not to require an extra dependency on yq
created_at: 2025-05-17T19:52:08Z
author_association: MEMBER
diff_hunk: @@ -56,6 +56,9 @@ spellcheck:
 	cat scripts/cspell/project-names.txt | grep -v '^#' | grep -v '^\s*$$' | tr ' ' '\n' > scripts/cspell/project-names-parsed.txt
 	cd scripts/cspell && ./spellcheck.sh
 
+fetch-blog-feed:
+	curl -s https://medium.com/feed/jaegertracing > assets/data/medium.xml
path: Makefile
user: login: yurishkuro
body: I think there is a flag to specify output file instead of doing stream redirect
created_at: 2025-05-18T04:27:54Z
author_association: MEMBER
diff_hunk: @@ -56,6 +56,9 @@ spellcheck:
 	cat scripts/cspell/project-names.txt | grep -v '^#' | grep -v '^\s*$$' | tr ' ' '\n' > scripts/cspell/project-names-parsed.txt
 	cd scripts/cspell && ./spellcheck.sh
 
+fetch-blog-feed:
+	curl -s https://medium.com/feed/jaegertracing > assets/data/medium.xml
path: Makefile
user: login: abhayguptas
body: yes, -o with the location
created_at: 2025-05-18T17:34:00Z
author_association: CONTRIBUTOR

PR comments:
user: login: netlify[bot]
created_at: 2025-05-17T12:15:16Z
body: ### <span aria-hidden="true">✅</span> Deploy Preview for *romantic-neumann-1959d7* ready!


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | 7646787ef38a95c48a305ea4ce0f836327c961f0 |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/projects/romantic-neumann-1959d7/deploys/682a1a94bd1b540008c6c6a9 |
|<span aria-hidden="true">😎</span> Deploy Preview | [https://deploy-preview-881--romantic-neumann-1959d7.netlify.app](https://deploy-preview-881--romantic-neumann-1959d7.netlify.app) |
|<span aria-hidden="true">📱</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br />![QR Code](https://app.netlify.com/qr-code/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1cmwiOiJodHRwczovL2RlcGxveS1wcmV2aWV3LTg4MS0tcm9tYW50aWMtbmV1bWFubi0xOTU5ZDcubmV0bGlmeS5hcHAifQ.yGKv_rOMHSn_aiHdY4XClf6_KGJk3ZUxXofKyn9yO3w)<br /><br />_Use your smartphone camera to open QR code link._</details> |
---
<!-- [romantic-neumann-1959d7 Preview](https://deploy-preview-881--romantic-neumann-1959d7.netlify.app) -->
_To edit notification comments on pull requests, go to your [Netlify project configuration](https://app.netlify.com/projects/romantic-neumann-1959d7/configuration/notifications#deploy-notifications)._
author_association: NONE
user: login: abhayguptas
created_at: 2025-05-18T04:15:02Z
body: Requested Changes Done!

Also, we created an assets/data folder (instead of placing the XML feed directly in data/) because Hugo’s resources.Get only works with files located inside the assets/ directory. Since we're using resources.Get "data/medium.xml" along with transform.Unmarshal to load and parse the Medium RSS feed at build time, the file must be placed within the assets/ folder to be treated as a Hugo resource. The structure assets/data/medium.xml helps maintain clarity and mirrors the logical purpose of storing external feed data, while remaining compatible with Hugo's build pipeline.

----
## Screenshots - 

![image](https://github.com/user-attachments/assets/b9189598-881d-4142-b4e6-b51eba23a741)
![image](https://github.com/user-attachments/assets/1211cb89-b764-4f5b-ad94-74b1ee4627f9)


## Logs - 

```
abhayporwals@professor:~/documentation$ make develop
HUGO_PREVIEW=true hugo server \
        --buildDrafts \
        --buildFuture \
        --ignoreCache
Watching for changes in /home/abhayporwals/documentation/{assets,content,data,static,themes}
Watching for config changes in /home/abhayporwals/documentation/hugo.yaml
Start building sites … 
hugo v0.126.1-3d40aba512931031921463dafc172c0d124437b8+extended linux/amd64 BuildDate=2024-05-15T10:42:34Z VendorInfo=gohugoio

WARN  Loaded Medium feed for: JaegerTracing - Medium
WARN  Loaded RSS feed for: JaegerTracing - Medium
WARN  Post: Jaeger v2 released
WARN  Post: Towards Jaeger v2  Moar OpenTelemetry!
WARN  Post: Making design decisions for ClickHouse as a core storage backend in Jaeger
WARN  Post: Experiment: Migrating OpenTracing-based application in Go to use the OpenTelemetry SDK
WARN  Post: Better alignment with OpenTelemetry by focusing on OTLP
WARN  Post: Introducing native support for OpenTelemetry in Jaeger
WARN  Post: Jaeger Tracing: A Friendly Guide for Beginners
WARN  Post: Adaptive Sampling in Jaeger
WARN  Post: Migrating from Jaeger client to OpenTelemetry SDK
WARN  Post: Jaeger Persistent Storage With Elasticsearch, Cassandra & Kafka
WARN  Google Analytics 4 (GA4) replaced Google Universal Analytics (UA) effective 1 July 2023. See https://support.google.com/analytics/answer/11583528. Create a GA4 property and data stream, then replace the Google Analytics ID in your site configuration with the new value.

                   |  EN   
-------------------+-------
  Pages            | 1555  
  Paginator pages  |    0  
  Non-page files   |    0  
  Static files     |   34  
  Processed images |    0  
  Aliases          |    0  
  Cleaned          |    0  

Built in 74099 ms
Environment: "development"
Serving pages from disk
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at http://localhost:1313/ (bind address 127.0.0.1) 
Press Ctrl+C to stop

Change detected, rebuilding site (#1).
2025-05-18 04:05:23.885 +0000
Template changed /index.html
Web Server is available at http://localhost:1313/ (bind address 127.0.0.1)
Total in 767 ms

Change detected, rebuilding site (#2).
2025-05-18 04:05:31.884 +0000
Template changed /index.html
Web Server is available at http://localhost:1313/ (bind address 127.0.0.1)
Total in 1183 ms
```
author_association: CONTRIBUTOR
user: login: abhayguptas
created_at: 2025-05-18T17:36:38Z
body: updated
author_association: CONTRIBUTOR
user: login: yurishkuro
created_at: 2025-05-18T18:31:37Z
body: thanks
author_association: MEMBER