Project: Grafana Loki (Observability and Analysis / Observability)
File: observability.md

title: Observability menuTitle: description: Observing Grafana Loki weight: Both Grafana Loki and Promtail expose a `/metrics` endpoint that expose Prometheus metrics (the default port is 3100 for Loki and 80 for Promtail). You will need a local Prometheus and add Loki and Promtail as targets. See [configuring Prometheus](https://prometheus.io/docs/prometheus/latest/configuration/configuration) for more information. All components of Loki expose the following metrics: | Metric Name | Metric Type | Description | | - | -- | - | | `lokilogmessagestotal` | Counter | DEPRECATED. Use internallogmessagestotal for the same functionality. Total number of log messages created by Loki itself. | | `lokiinternallogmessagestotal` | Counter | Total number of log messages created by Loki itself. | | `lokirequestduration_seconds` | Histogram | Number of received HTTP requests. | The Loki Distributors expose the following metrics: | Metric Name | Metric Type | Description | | - | -- | | | `lokidistributoringesterappendstotal` | Counter | The total number of batch appends sent to ingesters. | | `lokidistributoringesterappendfailures_total` | Counter | The total number of failed batch appends sent to ingesters. | | `lokidistributorbytesreceivedtotal` | Counter | The total number of uncompressed bytes received per both tenant and retention hours. | | `lokidistributorlinesreceivedtotal` | Counter | The total number of log entries received per tenant (not necessarily of lines, as an entry can have more than one line of text). | The Loki Ingesters expose the following metrics: | Metric Name | Metric Type | Description | | -- | -- | | | `lokiingesterflushqueuelength` | Gauge | The total number of series pending in the flush queue. | | `lokichunkstoreindexentriesperchunk` | Histogram | Number of index entries written to storage per chunk. | | `lokiingestermemory_chunks` | Gauge | The total number of chunks in memory. | | `lokiingestermemory_streams` | Gauge | The total number of streams in memory. | | `lokiingesterchunkageseconds` | Histogram | Distribution of chunk ages when flushed. | | `lokiingesterchunkencodetime_seconds` | Histogram | Distribution of chunk encode times. | | `lokiingesterchunk_entries` | Histogram | Distribution of lines per-chunk when flushed. | | `lokiingesterchunksizebytes` | Histogram | Distribution of chunk sizes when flushed. | | `lokiingesterchunk_utilization` | Histogram | Distribution of chunk utilization (filled uncompressed bytes vs maximum uncompressed bytes) when flushed. | | `lokiingesterchunkcompressionratio` | Histogram | Distribution of chunk compression ratio when flushed. | | `lokiingesterchunkstoredbytes_total` | Counter | Total bytes stored in chunks per tenant. | | `lokiingesterchunkscreatedtotal` | Counter | The total number of chunks created in the ingester. | | `lokiingesterchunksstoredtotal` | Counter | Total stored chunks per tenant. | | `lokiingesterreceived_chunks` | Counter | The total number of chunks sent by this ingester whilst joining during the handoff process. | | `lokiingestersamplesperchunk` | Histogram | The number of samples in a chunk. | | `lokiingestersent_chunks` | Counter | The total number of chunks sent by this ingester whilst leaving during the handoff process. | | `lokiingesterstreamscreatedtotal` | Counter | The total number of streams created per tenant. | | `lokiingesterstreamsremovedtotal` | Counter | The total number of streams removed per

| The Loki compactor exposes the following metrics: | Metric Name | Metric Type | Description | | - | -- | - | | `lokicompactordeleterequestsprocessed_total` | Counter | Number of delete requests processed per user. | | `lokicompactordeleterequestschunksselectedtotal` | Counter | Number of chunks selected while building delete plans per user. | | `lokicompactordeleteprocessingfails_total` | Counter | Number of times the delete phase of compaction has failed. | | `lokicompactorloadpendingrequestsattemptstotal` | Counter | Number of attempts that were made to load pending requests with status. | | `lokicompactoroldestpendingdeleterequestage_seconds` | Gauge | Age of oldest pending delete request in seconds since they are over their cancellation period. | | `lokicompactorpendingdeleterequests_count` | Gauge | Count of delete requests which are over their cancellation period and have not finished processing yet. | | `lokicompactordeleted_lines` | Counter | Number of deleted lines per user. | Promtail exposes these metrics: | Metric Name | Metric Type | Description | | -- | -- | | | `promtailreadbytes_total` | Gauge | Number of bytes read. | | `promtailreadlines_total` | Counter | Number of lines read. | | `promtaildroppedbytes_total` | Counter | Number of bytes dropped because failed to be sent to the ingester after all retries. | | `promtaildroppedentries_total` | Counter | Number of log entries dropped because failed to be sent to the ingester after all retries. | | `promtailencodedbytes_total` | Counter | Number of bytes encoded and ready to send. | | `promtailfilebytes_total` | Gauge | Number of bytes read from files. | | `promtailfilesactive_total` | Gauge | Number of active files. | | `promtailrequestduration_seconds` | Histogram | Number of send requests. | | `promtailsentbytes_total` | Counter | Number of bytes sent. | | `promtailsententries_total` | Counter | Number of log entries sent to the ingester. | | `promtailtargetsactive_total` | Gauge | Number of total active targets. | | `promtailtargetsfailed_total` | Counter | Number of total failed targets. | Most of these metrics are counters and should continuously increase during normal operations: Your app emits a log line to a file that is tracked by Promtail. Promtail reads the new line and increases its counters. Promtail forwards the log line to a Loki distributor, where the received counters should increase. The Loki distributor forwards the log line to a Loki ingester, where the request duration counter should increase. If Promtail uses any pipelines with metrics stages, those metrics will also be exposed by Promtail at its `/metrics` endpoint. See Promtail's documentation on for more information. An example Grafana dashboard was built by the community and is available as dashboard . Some of the Loki observability metrics are emitted per tracked file (active), with the file path included in labels. This increases the quantity of label values across the environment, thereby increasing cardinality. Best practices with Prometheus discourage increasing cardinality in this way. Review your emitted metrics before scraping with Prometheus, and configure the scraping to avoid this issue. The Loki repository has a that includes a set of dashboards, recording rules, and alerts. Together, the mixin gives you a comprehensive package for monitoring Loki in production. For more information about mixins, take a look at the docs for the .