Project: Pravega (App Definition and Development / Streaming & Messaging)
File: metrics.md

<!-- Copyright Pravega Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> - - - - - - In the Pravega Metrics Framework, we use as the underlying library, and provide our own API to make it easier to use. `StatsProvider`: The Statistics Provider which provides the whole Metric service. `StatsLogger`: The Statistics Logger is where the required Metrics (///) are registered. `OpStatsLogger`: The Operation Statistics Logger is a sub-metric for the complex ones (/). It is included in `StatsLogger` and `DynamicLogger`. Pravega Metric Framework is initiated using the `StatsProvider` interface: it provides the start and stop methods for the Metric service. It also provides `startWithoutExporting()` for testing purpose, which only stores metrics in memory without exporting them to external systems. Currently we have support for , , and registries. `start()`: Initializes the and Reporters for our Metric service. `startWithoutExporting()`: Initializes `SimpleMeterRegistry` that holds the latest value of each Meter in memory and does not export the data anywhere, typically for unit tests. `close()`: Shuts down the Metric Service. `createStatsLogger()`: Create a `StatsLogger` instance which is used to register and return metric objects. Application code could then perform metric operations directly with the returned metric objects. `createDynamicLogger()`: Creates a Dynamic Logger. This interface can be used to register the required metrics for simple types like and and some complex statistics type of Metric like `OpStatsLogger`, through which we provide and . `createStats()`: Register and get a `OpStatsLogger`, which is used for complex type of metrics. Notice the optional metric tags. `createCounter()`: Register and get a Metric. `createMeter()`: Create and register a Metric. `registerGauge()`: Register a Metric. `createScopeLogger()`: Create the `StatsLogger` under the given scope name. `OpStatsLogger` can be used if the user is interested in measuring the latency of operations like `CreateSegment` and `ReadSegment`. Further, we could use it to record the number of operation and time/duration of each operation. `reportSuccessEvent()`: Used to track the of a successful operation and will record the latency in nanoseconds in the required metric. `reportFailEvent()`: Used to track the of a failed operation and will record the latency in nanoseconds in required metric. `reportSuccessValue()`: Used to track the of a success value. `reportFailValue()`: Used to track the of a failed value. `toOpStatsData()`: Used to support the Reporters and unit tests. `clear`: Used to clear the stats for this operation. The following is an example of a simple interface that exposes only the simple type metrics: (//). `incCounterValue()`: Increases the with the given value. Notice the optional metric tags. `updateCounterValue()`: Updates the with the given value. `freezeCounter()`: Notifies that, the will not be updated. `reportGaugeValue()`: Reports the value. `freezeGaugeValue()`: Notifies that, the value will not be

`recordMeterEvents()`: Records the occurrences of a given number of events in . This example is from `io.pravega.segmentstore.server.host.ServiceStarter`. The code for this example can be found . It starts Pravega Segment Store service and the Metrics Service is started as a sub service. This is an example from `io.pravega.segmentstore.server.host.stat.SegmentStatsRecorderImpl.java`. The code for this example can be found . In the class `PravegaRequestProcessor`, we have registered two metrics: one Timer (`createStreamSegment`) one dynamic counter (`dynamicLogger`) From the above example, we can see the required steps to register and use dynamic counter: Get a dynamic logger from MetricsProvider: ``` DynamicLogger dynamicLogger = MetricsProvider.getDynamicLogger(); ``` Increase the counter by providing metric base name and optional tags associated with the metric. ``` DynamicLogger dl = getDynamicLogger(); dl.incCounterValue(globalMetricName(SEGMENTWRITEBYTES), dataLength); ... dl.incCounterValue(SEGMENTWRITEBYTES, dataLength, segmentTags(streamSegmentName)); ``` Here `SEGMENTWRITEBYTES` is the base name of the metric. Below are the two metrics associated with it: The global Counter which has no tags associated. A Segment specific Counter which has a list of Segment tags associated. Note that, the `segmentTags` is a method to generate tags based on fully qualified Segment name. The following are the required steps to register and use `OpStatsLogger(Timer)`: Get a `StatsLogger` from `MetricsProvider`. ``` StatsLogger STATS_LOGGER = MetricsProvider.getStatsLogger("segmentstore"); ``` Register all the desired metrics through `StatsLogger`. ``` @Getter(AccessLevel.PROTECTED) final OpStatsLogger createStreamSegment = STATSLOGGER.createStats(SEGMENTCREATE_LATENCY); ``` Use these metrics within code at the appropriate places where the values should be collected and recorded. ``` getCreateStreamSegment().reportSuccessEvent(elapsed); ``` Here `SEGMENTCREATELATENCY` is the name of this metric, and `createStreamSegment` is the metric object, which tracks operations of `createSegment` and we will get the time (i.e. time taken by each operation and other numbers computed based on them) for each `createSegment` operation happened. This is an example from `io.pravega.controller.metrics.StreamMetrics`. In this class, we report a Dynamic Gauge which represents the open Transactions of a Stream. The code for this example can be found . This is an example from `io.pravega.segmentstore.server.SegmentStoreMetrics`. The code for this example can be found . In the class `SegmentStoreMetrics`, we report a Dynamic Meter which represents the Segments created with a particular container. With Micrometer, each meter registry is responsible for both storage and exporting of metrics objects. In order to have a unified interface, Micrometer provides the `CompositeMeterRegistry` for the application to interact with, `CompositeMeterRegistry` will forward metric operations to all the concrete registries bounded to it. Note that when metrics service `start()`, initially only a global registry (of type `CompositeMeterRegistry`) is provided, which will bind concrete registries (e.g. statsD, Influxdb) based on the configurations. If no registry is switched on in `config`, metrics service throws error to prevent the global registry runs into no-op mode. Mainly for testing purpose, metrics service can also `startWithoutExporting()`, where a `SimpleMeterRegistry` is bound to the global registry. `SimpleMeterRegistry` holds memory only storage but does not export metrics, makes it ideal for tests to verify metrics objects. Currently Pravega supports the following: StatsD registry in `Telegraf` flavor. Dimensional metrics data model (or metric tags). UDP as Communication protocol. Direct InfluxDB connection. The reporter could be configured using the `MetricsConfig`. Please refer to the . When starting a Segment Store/Controller Service, start a Metric Service as a sub

Please check ```java public class AddMetrics { MetricsProvider.initialize(Config.METRICS_CONFIG); statsProvider.start(metricsConfig); statsProvider = MetricsProvider.getMetricsProvider(); statsProvider.start(); ``` Create a new `StatsLogger` instance through the `MetricsProvider.createStatsLogger(String loggerName)`, and register metric using name, e.g. `STATS_LOGGER.createCounter(String name)`; and then update the metric object as appropriately in the code. ```java static final StatsLogger STATS_LOGGER = MetricsProvider.getStatsLogger(); // < 1 DynamicLogger dynamicLogger = MetricsProvider.getDynamicLogger(); static class Metrics { // < 2 //Using Stats Logger static final String CREATESTREAM = "streamcreated"; static final OpStatsLogger CREATESTREAM = STATSLOGGER.createStats(CREATE_STREAM); static final String SEGMENTCREATELATENCY = "segmentstore.segment.createlatencyms"; static final OpStatsLogger createStreamSegment = STATSLOGGER.createStats(SEGMENTCREATE_LATENCY); //Using Dynamic Logger static final String SEGMENTREADBYTES = "segmentstore.segment.read_bytes"; //Dynamic Counter static final String OPEN_TRANSACTIONS = "controller.transactions.opened"; //Dynamic Gauge ... } //to report success or increment Metrics.CREATE_STREAM.reportSuccessValue(1); // < 3 Metrics.createStreamSegment.reportSuccessEvent(timer.getElapsed()); dynamicLogger.incCounterValue(Metrics.SEGMENTREADBYTES, 1); dynamicLogger.reportGaugeValue(OPEN_TRANSACTIONS, 0); //in case of failure Metrics.CREATE_STREAM.reportFailValue(1); Metrics.createStreamSegment.reportFailEvent(timer.getElapsed()); //to freeze dynamicLogger.freezeCounter(Metrics.SEGMENTREADBYTES); dynamicLogger.freezeGaugeValue(OPEN_TRANSACTIONS); } ``` All metric names are in the following format: ``` Metrics Prefix + Component Origin + Sub-Component (or Abstraction) + Metric Base Name ``` Metric Prefix: By default `pravega` is configurable. Component Origin: Indicates which component generates the metric, such as `segmentstore`, `controller`. Sub-Component (or Abstraction): Indicates the second level component or abstraction, such as `cache`, `transaction`, `storage`. Metric Base Name: Indicates the `readlatencyms`, `create_count`. For example: ``` pravega.segmentstore.segment.createlatencyms ``` Following are some common combinations of component and sub-components (or abstractions) being used: `segmentstore.segment`: Metrics for individual Segments `segmentstore.storage`: Metrics related to long-term storage (Tier 2) `segmentstore.bookkeeper`: Metrics related to Bookkeeper (Tier 1) `segmentstore.container`: Metrics for Segment Containers `segmentstore.thread_pool`: Metrics for Segment Store thread pool `segmentstore.cache`: Cache-related metrics `controller.stream`: Metrics for operations on Streams (e.g., number of streams created) `controller.segments`: Metrics about Segments, per Stream (e.g., count, splits, merges) `controller.transactions`: Metrics related to Transactions (e.g., created, committed, aborted) `controller.retention`: Metrics related to data retention, per Stream (e.g., frequency, size of truncated data) `controller.hosts`: Metrics related to Pravega servers in the cluster (e.g., number of servers, failures) `controller.container`: Metrics related to Container lifecycle (e.g., failovers) Following are the two types of metrics: Global Metric: `_global` metrics are reporting global values per component (Segment Store or Controller) instance, and further aggregation logic is needed if looking for Pravega cluster globals. For instance, `STORAGEREADBYTES` can be classified as a Global metric. Object-based Metric: Sometimes, we need to report metrics only based on specific objects, such as Streams or Segments. This kind of metrics use metric name as a base name in the file and are "dynamically" created based on the objects to be measured. For instance, in `CONTAINERAPPENDCOUNT` we actually report multiple metrics, one per each `containerId` measured, with different container tag (e.g. `["containerId", "3"]`). There are cases in which we may want both a Global and Object-based versions for the same metric. For example, regarding `SEGMENTREADBYTES` we publish the Global version of it by adding `_global` suffix to the base name ``` segmentstore.segment.readbytesglobal ``` to track the globally total number of bytes read, as well as the per-segment version of it by using the same base name and also supplying additional Segment tags to report in a finer granularity the events read per Segment. ``` segmentstore.segment.read_bytes, ["scope", "...", "stream", "...", "segment", "...", "epoch",

``` ``` jvmgclivedatasize jvmgcmaxdatasize jvmgcmemory_allocated jvmgcmemory_prompted jvmgcpause jvmmemorycommitted jvmmemorymax jvmmemoryused jvmthreadsdaemon jvmthreadslive jvmthreadspeak jvmthreadsstates ``` Segment Store Read/Write latency of storage operations (): ``` segmentstore.segment.createlatencyms segmentstore.segment.readlatencyms segmentstore.segment.writelatencyms ``` Segment Store global and per-segment Read/Write Metrics (): ``` // Global counters segmentstore.segment.readbytesglobal segmentstore.segment.writebytesglobal segmentstore.segment.writeeventsglobal // Per segment counters - all with tags {"scope", $scope, "stream", $stream, "segment", $segment, "epoch", $epoch} segmentstore.segment.write_bytes segmentstore.segment.read_bytes segmentstore.segment.write_events ``` Segment Store cache Read/Write latency Metrics (): ``` segmentstore.cache.insertlatencyms segmentstore.cache.get_latency ``` Segment Store cache Read/Write Metrics (): ``` segmentstore.cache.write_bytes segmentstore.cache.read_bytes ``` Segment Store cache size () and generation spread () Metrics: ``` segmentstore.cache.size_bytes segmentstore.cache.gen ``` Tier 1 Storage `DurableDataLog` Read/Write latency and queuing Metrics (): ``` segmentstore.bookkeeper.totalwritelatency_ms segmentstore.bookkeeper.writelatencyms segmentstore.bookkeeper.writequeuesize segmentstore.bookkeeper.writequeuefill ``` Tier 1 Storage `DurableDataLog` Read/Write () and per-container ledger count Metrics (): ``` segmentstore.bookkeeper.write_bytes segmentstore.bookkeeper.bookkeeperledgercount - with tag {"container", $containerId} ``` Tier 2 Storage Read/Write latency Metrics (): ``` segmentstore.storage.readlatencyms segmentstore.storage.writelatencyms ``` Tier 2 Storage Read/Write data and file creation Metrics (): ``` segmentstore.storage.read_bytes segmentstore.storage.write_bytes segmentstore.storage.create_count ``` Segment Store container-specific operation Metrics: ``` // Histograms - all with tags {"container", $containerId} segmentstore.container.processoperations.latencyms segmentstore.container.processoperations.batchsize segmentstore.container.operation_queue.size segmentstore.container.operationprocessor.inflight segmentstore.container.operationqueue.waittime segmentstore.container.operationprocessor.delayms segmentstore.container.operationcommit.latencyms segmentstore.container.operation.latency_ms segmentstore.container.operationcommit.metadatatxn_count segmentstore.container.operationcommit.memorylatency_ms // Gauge segmentstore.container.operation.log_size ``` Segment Store operation processor () Metrics - all with tags {"container", $containerId}. ``` // Counters/Meters segmentstore.container.append_count segmentstore.container.appendoffsetcount segmentstore.container.updateattributescount segmentstore.container.getattributescount segmentstore.container.read_count segmentstore.container.getinfocount segmentstore.container.createsegmentcount segmentstore.container.deletesegmentcount segmentstore.container.mergesegmentcount segmentstore.container.seal_count segmentstore.container.truncate_count ``` Segment Store active Segments () and thread pool status () Metrics: ``` // Gauge - with tags {"container", $containerId} segmentstore.active_segments // Histograms segmentstore.threadpool.queuesize segmentstore.threadpool.activethreads ``` Controller Stream operation latency Metrics (): ``` controller.stream.createdlatencyms controller.stream.sealedlatencyms controller.stream.deletedlatencyms controller.stream.updatedlatencyms controller.stream.truncatedlatencyms ``` Controller global and per-Stream operation Metrics (): ``` controller.stream.created controller.stream.createfailedglobal controller.stream.create_failed - with tags {"scope", $scope, "stream", $stream} controller.stream.sealed controller.stream.sealfailedglobal controller.stream.seal_failed - with tags {"scope", $scope, "stream", $stream} controller.stream.deleted controller.stream.deletefailedglobal controller.stream.delete_failed - with tags {"scope", $scope, "stream", $stream} controller.stream.updated_global controller.stream.updated - with tags {"scope", $scope, "stream", $stream} controller.stream.updatefailedglobal controller.stream.update_failed - with tags {"scope", $scope, "stream", $stream} controller.stream.truncated_global controller.stream.truncated - with tags {"scope", $scope, "stream", $stream} controller.stream.truncatefailedglobal controller.stream.truncate_failed - with tags {"scope", $scope, "stream", $stream} ``` Controller Stream retention frequency () and truncated size () Metrics: ``` controller.retention.frequency - with tags {"scope", $scope, "stream", $stream} controller.retention.truncated_size - with tags {"scope", $scope, "stream", $stream} ``` Controller Stream Segment operations () and open/timed out Transactions on a Stream () Metrics - all with tags {"scope", $scope, "stream", $stream}: ``` controller.transactions.opened controller.transactions.timedout controller.segments.count controller.segment.splits controller.segment.merges ``` Controller Transaction operation latency Metrics: ``` controller.transactions.createdlatencyms controller.transactions.committedlatencyms controller.transactions.abortedlatencyms ``` Controller Transaction operation counter Metrics: ``` controller.transactions.created_global controller.transactions.created - with tags {"scope", $scope, "stream", $stream} controller.transactions.createfailedglobal controller.transactions.create_failed - with tags {"scope", $scope, "stream", $stream} controller.transactions.committed_global controller.transactions.committed - with tags {"scope", $scope, "stream", $stream} controller.transactions.commitfailedglobal controller.transactions.commit_failed - with tags {"scope", $scope, "stream", $stream} controller.transactions.commit_failed - with tags {"scope", $scope, "stream", $stream, "transaction", $txnId} controller.transactions.aborted_global controller.transactions.aborted - with tags {"scope", $scope, "stream", $stream} controller.transactions.abortfailedglobal controller.transactions.abort_failed - with tags {"scope", $scope, "stream", $stream} controller.transactions.abort_failed - with tags {"scope", $scope, "stream", $stream, "transaction", $txnId} ``` Controller hosts available () and host failure () Metrics: ``` controller.hosts.count controller.hosts.failures_global controller.hosts.failures - with tags {"host", $host} ``` Controller Container count per host () and failover () Metrics: ``` controller.hosts.container_count controller.container.failovers_global controller.container.failovers - with tags {"container", $containerId} ``` Controller Zookeeper session expiration () metrics: ``` controller.zookeeper.session_expiration ```