Project: Grafana Loki (Observability and Analysis / Observability)
File: recording-rules.md

title: Recording Rules menuTitle: description: Working with recording rules. weight: Recording rules are evaluated by the `ruler` component. Each `ruler` acts as its own `querier`, in the sense that it executes queries against the store without using the `query-frontend` or `querier` components. It will respect all query put in place for the `querier`. Loki's implementation of recording rules largely reuses Prometheus' code. Samples generated by recording rules are sent to Prometheus using Prometheus' remote-write feature. All samples generated by recording rules are written to a WAL. The WALs main benefit is that it persists the samples generated by recording rules to disk, which means that if your `ruler` crashes, you won't lose any data. We are trading off extra memory usage and slower start-up times for this functionality. A WAL is created per tenant; this is done to prevent cross-tenant interactions. If all samples were to be written to a single WAL, this would increase the chances that one tenant could cause data-loss for others. A typical scenario here is that Prometheus will, for example, reject a remote-write request with 100 samples if just 1 of those samples is invalid in some way. When the `ruler` starts up, it will load the WALs for the tenants who have recording rules. These WAL files are stored on disk and are loaded into memory. {{% admonition type="note" %}} WALs are loaded one at a time upon start-up. This is a current limitation of the Loki ruler. For this reason, it is adviseable that the number of rule groups serviced by a ruler be kept to a reasonable size, since no rule evaluation occurs while WAL replay is in progress (this includes alerting rules). {{% /admonition %}} WAL files are regularly truncated to reduce their size on disk. from one of the Prometheus maintainers (Ganesh Vernekar) gives an excellent overview of the truncation, checkpointing, and replaying of the WAL. <span style="background-color:#f3f973;">WAL Cleaner is an experimental feature.</span> The WAL Cleaner watches for abandoned WALs (tenants who no longer have recording rules associated) and deletes them. Enable this feature only if you are running into storage concerns with WALs that are too large. WALs should not grow excessively large due to truncation. See Mimir's guide for for scaling the ruler using a ring. {{% admonition type="note" %}} The `ruler` shards by rule group, not by individual rules. This is an artifact of the fact that Prometheus recording rules need to run in order since one recording rule can reuse another - but this is not possible in Loki. {{% /admonition %}} The `ruler` needs to persist its WAL files to disk, and it incurs a bit of a start-up cost by reading these WALs into memory. As such, it is recommended that you try to minimize churn of individual `ruler` instances since rule evaluation is blocked while the WALs are being read from disk. It is recommended that you run the `rulers` using `StatefulSets`. The `ruler` will write its WAL files to persistent storage, so a `Persistent Volume` should be

Remote-write can be configured at a global level in the base configuration, and certain parameters tuned specifically on a per-tenant basis. Most of the configuration options have (which can be also applied at runtime!). Remote-write can be tuned if the default configuration is insufficient (see below). There is a on the Prometheus website, all of which applies to Loki, too. Rules can be evenly distributed across available rulers by using `-ruler.enable-sharding=true` and `-ruler.sharding-strategy="by-rule"`. Rule groups execute in order; this is a feature inherited from Prometheus' rule engine (which Loki uses), but Loki has no need for this constraint because rules cannot depend on each other. The default sharding strategy will shard by rule groups, but this may be undesirable as some rule groups could contain more expensive rules, which can lead to subsequent rules missing evaluations. The `by-rule` sharding strategy creates one rule group for each rule the ruler instance "owns" (based on its hash ring), and these rings are all executed concurrently. Since Loki reuses the Prometheus code for recording rules and WALs, it also gains all of Prometheus' observability. Prometheus exposes a number of metrics for its WAL implementation, and these have all been prefixed with `lokirulerwal_`. For example: `prometheusremotestoragebytestotal` `lokirulerwalprometheusremotestoragebytes_total` Additional metrics are exposed, also with the prefix `lokirulerwal_`. All per-tenant metrics contain a `tenant` label, so be aware that cardinality could begin to be a concern if the number of tenants grows sufficiently large. Some key metrics to note are: `lokirulerwalappenderready`: whether a WAL appender is ready to accept samples (1) or not (0) `lokirulerwalprometheusremotestoragesamples_total`: number of samples sent per tenant to remote storage `lokirulerwalprometheusremotestoragesamples...` `lokirulerwalprometheusremotestoragesamplespendingtotal`: samples buffered in memory, waiting to be sent to remote storage `lokirulerwalprometheusremotestoragesamplesfailedtotal`: samples that failed when sent to remote storage `lokirulerwalprometheusremotestoragesamplesdroppedtotal`: samples dropped by relabel configurations `lokirulerwalprometheusremotestoragesamplesretriedtotal`: samples re-resent to remote storage `lokirulerwalprometheusremotestoragehighesttimestampin_seconds`: highest timestamp of sample appended to WAL `lokirulerwalprometheusremotestoragequeuehighestsenttimestampseconds`: highest timestamp of sample sent to remote storage. We've created a basic which you can use to administer recording rules. Remote-write can lag behind for many reasons: Remote-write storage (Prometheus) is temporarily unavailable A tenant is producing samples too quickly from a recording rule Remote-write is tuned too low, creating backpressure It can be determined by subtracting `lokirulerwalprometheusremotestoragequeuehighestsenttimestampseconds` from `lokirulerwalprometheusremotestoragehighesttimestampin_seconds`. In case 1, the `ruler` will continue to retry sending these samples until the remote storage becomes available again. Be aware that if the remote storage is down for longer than `ruler.wal.max-age`, data loss may occur after truncation occurs. In cases 2 and 3, you should consider remote-write appropriately. Further reading: see by Prometheus maintainer Callum Styan. Each tenant's WAL has an "appender" internally; this appender is used to append samples to the WAL. The appender is marked as not ready until the WAL replay is complete upon startup. If the WAL is corrupted for some reason, or is taking a long time to replay, you can determine this by alerting on `lokirulerwalappenderready < 1`. If a disk fails or the `ruler` does not terminate correctly, there's a chance one or more tenant WALs can become corrupted. A mechanism exists for automatically repairing the WAL, but this cannot handle every conceivable scenario. In this case, the `lokirulerwalcorruptionsrepairfailedtotal` metric will be incremented. Open an and tell us about it!