Hazardous Defaults: Managing Cardinality and Performance for Your Lo... Derek Cavanaugh & Sara Moore
Channel: CNCF [Cloud Native Computing Foundation]
Date: 2023-05-01

Hazardous Defaults: Managing Cardinality and Performance for Your Logging Stack - Derek Cavanaugh & Sara Moore, Recursion Pharma

Instrumented systems generate A LOT of data and we are fortunate to have performant open-source tools that help us spelunk through all that telemetry (logs, metrics, traces). Configuring these monitoring and observability tools - so that they themselves are performant and efficient - can be a challenge. For those new or unfamiliar to monitoring and observability, it can be appealing to just ‘roll the defaults’ from a configuration perspective. However, leaving those defaults unexamined can lead to unexpected performance issues; and worse, potential data loss. In this talk, we walk through the basic structure of the PLG-stack (Promtail, Loki and Grafana). We explore some unexpected cardinality (and associated performance) impacts that arise from the default configurations and how we made thoughtful adjustments to address those impacts. Finally, we will lay out a step-by-step guide to give your logging stack some ‘love’ and ensure that you are getting the most out of your tooling.