Project: EaseAgent (Observability and Analysis / Observability)
File: metric-api.md

In Easeagent, Metric is inspired by Dropwizard, but Dropwizard is just the basis of Metric. The use of time series indicators needs to be very rigorous. In order to prevent the random use of metrics and the wrong use of metrics as relational databases, we have a strict standard for business metrics. In order to be applicable to the business, we additionally encapsulate and define a dedicated interface. These interfaces are related to business. We have briefly categorized common business types. | name | Description | |:|:| | DEFAULT | Default,no metric with error | | ERROR | Metric with error | | CHANNEL | The metric of the connection channel, for rabbitmq. | | CONSUMER | The metric of the consumer of the data queue, for messaging kafka/rabbitmq consumer, etc. | | PRODUCER | The metric of the producer of the data queue, for messaging kafka/rabbitmq producer, etc. | | CONSUMER_ERROR | The error metric of the consumer of the data queue, for messaging kafka/rabbitmq consumer error, etc. | | PRODUCER_ERROR | The error metric of the producer of the data queue, for messaging kafka/rabbitmq producer error, etc. | | NONE | Unknown type | Different types will use different names depending on the business. For example, the Count value of Counter and Timer can be named TotalCount or ErrorCount according to success or failure. We have made certain specifications for the name: ```java public enum MetricField { MINEXECUTIONTIME("min", ConverterType.DURATION, 2), MAXEXECUTIONTIME("max", ConverterType.DURATION, 2), MEANEXECUTIONTIME("mean", ConverterType.DURATION, 2), P25EXECUTIONTIME("p25", ConverterType.DURATION, 2), P50EXECUTIONTIME("p50", ConverterType.DURATION, 2), P75EXECUTIONTIME("p75", ConverterType.DURATION, 2), P95EXECUTIONTIME("p95", ConverterType.DURATION, 2), P98EXECUTIONTIME("p98", ConverterType.DURATION, 2), P99EXECUTIONTIME("p99", ConverterType.DURATION, 2), P999EXECUTIONTIME("p999", ConverterType.DURATION, 2), STD("std"), EXECUTION_COUNT("cnt"), EXECUTIONERRORCOUNT("errcnt"), M1_RATE("m1", ConverterType.RATE, 5), M5_RATE("m5", ConverterType.RATE, 5), M15_RATE("m15", ConverterType.RATE, 5), RETRYM1RATE("retrym1", ConverterType.RATE, 5), RETRYM5RATE("retrym5", ConverterType.RATE, 5), RETRYM15RATE("retrym15", ConverterType.RATE, 5), RATELIMITERM1RATE("rlm1", ConverterType.RATE, 5), RATELIMITERM5RATE("rlm5", ConverterType.RATE, 5), RATELIMITERM15RATE("rlm15", ConverterType.RATE, 5), CIRCUITBREAKERM1RATE("cbm1", ConverterType.RATE, 5), CIRCUITBREAKERM5RATE("cbm5", ConverterType.RATE, 5), CIRCUITBREAKERM15RATE("cbm15", ConverterType.RATE, 5), MEANRATE("meanrate", ConverterType.RATE, 5), M1ERRORRATE("m1err", ConverterType.RATE, 5), M5ERRORRATE("m5err", ConverterType.RATE, 5), M15ERRORRATE("m15err", ConverterType.RATE, 5), M1_COUNT("m1cnt", ConverterType.RATE, 0), M5_COUNT("m5cnt", ConverterType.RATE, 0), M15_COUNT("m15cnt", ConverterType.RATE, 0), TIMESRATE("timerate", ConverterType.RATE, 5), TOTALCOLLECTIONTIME("totalcollectiontime", ConverterType.RATE, 0), TIMES("times", ConverterType.RATE, 0), / channel is for rabbitmq / CHANNELM1RATE("channelm1rate", ConverterType.RATE, 5), CHANNELM5RATE("channelm5rate", ConverterType.RATE, 5), CHANNELM15RATE("channelm15rate", ConverterType.RATE, 5), QUEUEM1RATE("queuem1rate", ConverterType.RATE, 5), QUEUEM5RATE("queuem5rate", ConverterType.RATE, 5), QUEUEM15RATE("queuem15rate", ConverterType.RATE, 5), QUEUEM1ERRORRATE("queuem1errorrate", ConverterType.RATE, 5), QUEUEM5ERRORRATE("queuem5errorrate", ConverterType.RATE, 5), QUEUEM15ERRORRATE("queuem15errorrate", ConverterType.RATE, 5), /producer and consumer is for message kafka rabbitmq service/ PRODUCERM1RATE("prodrm1", ConverterType.RATE, 5), PRODUCERM5RATE("prodrm5", ConverterType.RATE, 5), PRODUCERM15RATE("prodrm15", ConverterType.RATE, 5), PRODUCERM1ERROR_RATE("prodrm1err", ConverterType.RATE, 5), PRODUCERM5ERROR_RATE("prodrm5err", ConverterType.RATE, 5), PRODUCERM15ERROR_RATE("prodrm15err", ConverterType.RATE, 5), CONSUMERM1RATE("consrm1", ConverterType.RATE, 5), CONSUMERM5RATE("consrm5", ConverterType.RATE, 5), CONSUMERM15RATE("consrm15", ConverterType.RATE, 5), CONSUMERM1ERROR_RATE("consrm1err", ConverterType.RATE, 5), CONSUMERM5ERROR_RATE("consrm5err", ConverterType.RATE, 5), CONSUMERM15ERROR_RATE("consrm15err", ConverterType.RATE, 5), EXECUTIONPRODUCERERROR_COUNT("prodrerrcnt"), EXECUTIONCONSUMERERROR_COUNT("consrerrcnt"), EXECUTIONPRODUCERCOUNT("prodrcnt"), EXECUTIONCONSUMERCOUNT("consrcnt"), PRODUCERMINEXECUTION_TIME("prodrmin", ConverterType.DURATION, 2), PRODUCERMAXEXECUTION_TIME("prodrmax", ConverterType.DURATION, 2), PRODUCERMEANEXECUTION_TIME("prodrmean", ConverterType.DURATION, 2), PRODUCERP25EXECUTION_TIME("prodrp25", ConverterType.DURATION, 2), PRODUCERP50EXECUTION_TIME("prodrp50", ConverterType.DURATION, 2), PRODUCERP75EXECUTION_TIME("prodrp75", ConverterType.DURATION, 2), PRODUCERP95EXECUTION_TIME("prodrp95", ConverterType.DURATION, 2), PRODUCERP98EXECUTION_TIME("prodrp98", ConverterType.DURATION, 2), PRODUCERP99EXECUTION_TIME("prodrp99", ConverterType.DURATION, 2), PRODUCERP999EXECUTION_TIME("prodrp999", ConverterType.DURATION, 2), CONSUMERMINEXECUTION_TIME("consrmin", ConverterType.DURATION, 2), CONSUMERMAXEXECUTION_TIME("consrmax", ConverterType.DURATION, 2), CONSUMERMEANEXECUTION_TIME("consrmean", ConverterType.DURATION, 2), CONSUMERP25EXECUTION_TIME("consrp25", ConverterType.DURATION, 2), CONSUMERP50EXECUTION_TIME("consrp50", ConverterType.DURATION, 2), CONSUMERP75EXECUTION_TIME("consrp75", ConverterType.DURATION, 2), CONSUMERP95EXECUTION_TIME("consrp95", ConverterType.DURATION, 2), CONSUMERP98EXECUTION_TIME("consrp98", ConverterType.DURATION, 2), CONSUMERP99EXECUTION_TIME("consrp99", ConverterType.DURATION, 2), CONSUMERP999EXECUTION_TIME("consrp999", ConverterType.DURATION, 2), NONE("", ConverterType.RATE, 0); } ``` Metric has different types, and each type can calculate different data. For example: Counter can only get Count, Timer can get Max, Mean, Min, P95 ... Regarding the value that the type can calculate, we have encapsulated it: ```java public enum MetricValueFetcher { CountingCount(Counter::getCount, Counter.class), SnapshotMaxValue(Snapshot::getMax, Snapshot.class), SnapshotMeanValue(Snapshot::getMean, Snapshot.class), SnapshotMinValue(Snapshot::getMin, Snapshot.class), Snapshot25Percentile(s -> s.getValue(0.25), Snapshot.class), SnapshotMedianValue(Snapshot::getMedian,

Snapshot50PercentileValue(Snapshot::getMedian, Snapshot.class), Snapshot75PercentileValue(Snapshot::get75thPercentile, Snapshot.class), Snapshot95PercentileValue(Snapshot::get95thPercentile, Snapshot.class), Snapshot98PercentileValue(Snapshot::get98thPercentile, Snapshot.class), Snapshot99PercentileValue(Snapshot::get99thPercentile, Snapshot.class), Snapshot999PercentileValue(Snapshot::get999thPercentile, Snapshot.class), MeteredM1Rate(Meter::getOneMinuteRate, Meter.class), MeteredM1RateIgnoreZero(Meter::getOneMinuteRate, Meter.class, aDouble -> aDouble), MeteredM5Rate(Meter::getFiveMinuteRate, Meter.class), MeteredM15Rate(Meter::getFifteenMinuteRate, Meter.class), MeteredMeanRate(Meter::getMeanRate, Meter.class), MeteredCount(Meter::getCount, Meter.class); } ``` According to the value type and naming combination, we standardize a NameFactory. ```java class ServerMetric{ @Nonnull public static NameFactory nameFactory() { return NameFactory.createBuilder() .counterType(MetricSubType.DEFAULT, ImmutableMap.<MetricField, MetricValueFetcher>builder() .put(MetricField.EXECUTION_COUNT, MetricValueFetcher.CountingCount) .build()) .meterType(MetricSubType.DEFAULT, ImmutableMap.<MetricField, MetricValueFetcher>builder() .put(MetricField.M1_RATE, MetricValueFetcher.MeteredM1RateIgnoreZero) .put(MetricField.M5_RATE, MetricValueFetcher.MeteredM5Rate) .put(MetricField.M15_RATE, MetricValueFetcher.MeteredM15Rate) .put(MetricField.MEAN_RATE, MetricValueFetcher.MeteredMeanRate) .build()) .counterType(MetricSubType.ERROR, ImmutableMap.<MetricField, MetricValueFetcher>builder() .put(MetricField.EXECUTIONERRORCOUNT, MetricValueFetcher.CountingCount) .build()) .meterType(MetricSubType.ERROR, ImmutableMap.<MetricField, MetricValueFetcher>builder() .put(MetricField.M1ERRORRATE, MetricValueFetcher.MeteredM1Rate) .put(MetricField.M5ERRORRATE, MetricValueFetcher.MeteredM5Rate) .put(MetricField.M15ERRORRATE, MetricValueFetcher.MeteredM15Rate) .put(MetricField.MEAN_RATE, MetricValueFetcher.MeteredMeanRate) .build()) .gaugeType(MetricSubType.DEFAULT, new HashMap<>()) .timerType(MetricSubType.DEFAULT, ImmutableMap.<MetricField, MetricValueFetcher>builder() .put(MetricField.MINEXECUTIONTIME, MetricValueFetcher.SnapshotMinValue) .put(MetricField.MAXEXECUTIONTIME, MetricValueFetcher.SnapshotMaxValue) .put(MetricField.MEANEXECUTIONTIME, MetricValueFetcher.SnapshotMeanValue) .put(MetricField.P25EXECUTIONTIME, MetricValueFetcher.Snapshot25Percentile) .put(MetricField.P50EXECUTIONTIME, MetricValueFetcher.Snapshot50PercentileValue) .put(MetricField.P75EXECUTIONTIME, MetricValueFetcher.Snapshot75PercentileValue) .put(MetricField.P95EXECUTIONTIME, MetricValueFetcher.Snapshot95PercentileValue) .put(MetricField.P98EXECUTIONTIME, MetricValueFetcher.Snapshot98PercentileValue) .put(MetricField.P99EXECUTIONTIME, MetricValueFetcher.Snapshot99PercentileValue) .put(MetricField.P999EXECUTIONTIME, MetricValueFetcher.Snapshot999PercentileValue) .build()) .build(); } } ``` Inherit the ServiceMetric class in order to obtain the MetricRegistry. Use the ServiceMetricRegistry interface to register and create a singleton. Metric configuration follows plugin configuration rules Metrics have values as well as tags. EaseAgent supports custom tags, but these tags must be predictable and given in advance. To better support the business, three tags must be given: categorytype and key So it is fixed in Tags and must provide three pieces of information: categorytype and keyFieldName Its tag is copied as follows: ``` output.put("category", tags.category) output.put("type", tags.type) output.put(tags.keyFieldName, {@link NameFactory}.key[?]) tags.tags.forEach((k,v)->{ output.put(k,v) }) ``` The Key of the singleton is: `domain`, `namespace`, `id`, `tags` and the `type` of class. ```java public class ServerMetric extends ServiceMetric { public ServerMetric(@Nonnull MetricRegistry metricRegistry, @Nonnull NameFactory nameFactory) { super(metricRegistry, nameFactory); } public void collectMetric(String key, int statusCode, Throwable throwable, long startMillis, long endMillis) { Timer timer = metricRegistry.timer(nameFactory.timerName(key, MetricSubType.DEFAULT)); timer.update(Duration.ofMillis(endMillis - startMillis)); final Meter errorMeter = metricRegistry.meter(nameFactory.meterName(key, MetricSubType.ERROR)); final Meter meter = metricRegistry.meter(nameFactory.meterName(key, MetricSubType.DEFAULT)); Counter errorCounter = metricRegistry.counter(nameFactory.counterName(key, MetricSubType.ERROR)); Counter counter = metricRegistry.counter(nameFactory.counterName(key, MetricSubType.DEFAULT)); boolean hasException = throwable != null; if (statusCode >= 400 || hasException) { errorMeter.mark(); errorCounter.inc(); } counter.inc(); meter.mark(); metricRegistry.gauge(nameFactory.gaugeName(key, MetricSubType.DEFAULT), () -> () -> { BigDecimal m1ErrorPercent = BigDecimal.ZERO; BigDecimal m5ErrorPercent = BigDecimal.ZERO; BigDecimal m15ErrorPercent = BigDecimal.ZERO; BigDecimal error = BigDecimal.valueOf(errorMeter.getOneMinuteRate()).setScale(5, BigDecimal.ROUNDHALFDOWN); BigDecimal n = BigDecimal.valueOf(meter.getOneMinuteRate()); if (n.compareTo(BigDecimal.ZERO) != 0) { m1ErrorPercent = error.divide(n, 2, BigDecimal.ROUNDHALFUP); } error = BigDecimal.valueOf(errorMeter.getFiveMinuteRate()).setScale(5, BigDecimal.ROUNDHALFDOWN); n = BigDecimal.valueOf(meter.getFiveMinuteRate()); if (n.compareTo(BigDecimal.ZERO) != 0) { m5ErrorPercent = error.divide(n, 2, BigDecimal.ROUNDHALFUP); } error = BigDecimal.valueOf(errorMeter.getFifteenMinuteRate()).setScale(5, BigDecimal.ROUNDHALFDOWN); n = BigDecimal.valueOf(meter.getFifteenMinuteRate()); if (n.compareTo(BigDecimal.ZERO) != 0) { m15ErrorPercent = error.divide(n, 2, BigDecimal.ROUNDHALFUP); } return new ErrorPercentModelGauge(m1ErrorPercent, m5ErrorPercent, m15ErrorPercent); }); } @Nonnull public static NameFactory nameFactory() { return NameFactory.createBuilder() .counterType(MetricSubType.DEFAULT, ImmutableMap.<MetricField, MetricValueFetcher>builder() .put(MetricField.EXECUTION_COUNT, MetricValueFetcher.CountingCount) .build()) .meterType(MetricSubType.DEFAULT, ImmutableMap.<MetricField, MetricValueFetcher>builder() .put(MetricField.M1_RATE, MetricValueFetcher.MeteredM1RateIgnoreZero) .put(MetricField.M5_RATE, MetricValueFetcher.MeteredM5Rate) .put(MetricField.M15_RATE, MetricValueFetcher.MeteredM15Rate) .put(MetricField.MEAN_RATE, MetricValueFetcher.MeteredMeanRate) .build()) .counterType(MetricSubType.ERROR, ImmutableMap.<MetricField, MetricValueFetcher>builder() .put(MetricField.EXECUTIONERRORCOUNT, MetricValueFetcher.CountingCount) .build()) .meterType(MetricSubType.ERROR, ImmutableMap.<MetricField, MetricValueFetcher>builder() .put(MetricField.M1ERRORRATE, MetricValueFetcher.MeteredM1Rate) .put(MetricField.M5ERRORRATE, MetricValueFetcher.MeteredM5Rate) .put(MetricField.M15ERRORRATE, MetricValueFetcher.MeteredM15Rate) .put(MetricField.MEAN_RATE, MetricValueFetcher.MeteredMeanRate) .build()) .gaugeType(MetricSubType.DEFAULT, new HashMap<>()) .timerType(MetricSubType.DEFAULT, ImmutableMap.<MetricField, MetricValueFetcher>builder() .put(MetricField.MINEXECUTIONTIME, MetricValueFetcher.SnapshotMinValue) .put(MetricField.MAXEXECUTIONTIME, MetricValueFetcher.SnapshotMaxValue) .put(MetricField.MEANEXECUTIONTIME, MetricValueFetcher.SnapshotMeanValue) .put(MetricField.P25EXECUTIONTIME, MetricValueFetcher.Snapshot25Percentile) .put(MetricField.P50EXECUTIONTIME, MetricValueFetcher.Snapshot50PercentileValue) .put(MetricField.P75EXECUTIONTIME, MetricValueFetcher.Snapshot75PercentileValue) .put(MetricField.P95EXECUTIONTIME, MetricValueFetcher.Snapshot95PercentileValue) .put(MetricField.P98EXECUTIONTIME, MetricValueFetcher.Snapshot98PercentileValue) .put(MetricField.P99EXECUTIONTIME, MetricValueFetcher.Snapshot99PercentileValue) .put(MetricField.P999EXECUTIONTIME, MetricValueFetcher.Snapshot999PercentileValue) .build()) .build(); } } public class DoFilterMetricInterceptor extends Interceptor { private static volatile ServerMetric SERVER_METRIC = null; private static final Object START_MACK = new Object(); @Override public void init(Config config, String className, String methodName, String methodDescriptor) { SERVER_METRIC = ServiceMetricRegistry.getOrCreate(config, new Tags("application", "http-request", "url"), new ServiceMetricSupplier<ServerMetric>() { @Override public NameFactory newNameFactory() { return ServerMetric.nameFactory(); } @Override public ServerMetric newInstance(MetricRegistry metricRegistry, NameFactory nameFactory) { return new ServerMetric(metricRegistry, nameFactory); } }); } @Override public void before(MethodInfo methodInfo, Context context){ context.put(START_MACK,

} @Override public void after(MethodInfo methodInfo, Context context){ long start = context.remove(START_MACK); long end = System.currentTimeMillis(); String key = "GET /demo"; int statusCode = 200; SERVER_METRIC.collectMetric(key, statusCode, throwable, start, end); } } ``` Even though EaseAgent's Metric is inspired by Dropwizard, it still hopes to decouple its implementation from Dropwizard. So EaseAgent has its own API, although the implementation scheme currently used is Dropwizard. When you are sure that you want to implement your own metric, then you only need a metric output class, and finally use this output class to output to Kafka or backend. We provide such an output interface, this interface accepts string output. The standard of EaseMonitor is to use json format, if unnecessary, please use json as output. The obtained Reporter is a singleton, and the key of the singleton is `namespace`. Its output configuration complies with metric configuration rules: ```java public class MD5ReportConsumer { private final Config config; private final Reporter reporter; public MD5ReportConsumer() { this.config = AutoRefreshRegistry.getOrCreate("observability", "md5Dictionary", "metric"); this.reporter = EaseAgent.metricReporterFactory(config); } @Override public void accept(Map<String, String> map) { if (!this.config.enabled()) { return; } map.put("host_name", HostAddress.localhost()); map.put("host_ipv4", HostAddress.getHostIpv4()); map.put("category", "application"); String json = JsonUtil.toJson(item); this.reporter.report(json); } } ``` We have redefined Prometheus Exports according to the following naming standard and statistics rules. For better readability, we get the `category` and `type` from `Tags`, and add `MetricField.field` as the metric name Name format: `${category}${type}${MetricField.field}` Example: `applicationhttprequest_m1` Except `category`, `type`, `MetricField.field` and `value`, other fields will be exported in the form of labels. By default, there will be the following labels. | label name | label value | Description | |:|:--|:--| | MetricSubType | enum | The enum MetricSubType value: `DEFAULT,ERROR,CHANNEL,CONSUMER,PRODUCER,CONSUMERERROR,PRODUCERERROR,NONE` | | MetricType | enum | The Metric Type by metric calculate: `TimerType,HistogramType,MeterType,CounterType,GaugeType` | | host_ipv4 | String, xxx.xxx.xxx.xx | The ipv4 by host. | | host_name | String | host name. | | service | String | The `name` read from the configuration. for you service name. | | system | String | The `system` read from the configuration. for you system name. | | ${Tags.keyFieldName} | ${metricKey} | The metric label by metric key. | | ${Tags.tags().key} | ${Tags.tags().value} | Your custom label, which comes from `Tags` | Under normal circumstances, each metric name should correspond to a calculation method. Metric name is `applicationhttprequest_m1`, the value is `1.0`, which is the weighted average QPS of the last minute calculated. It is calculated using `Meter::getOneMinuteRate`. Its corresponding `MetricValueFetcher` is `MeteredM1RateIgnoreZero`. ```java public class M1MetricCollect { static final M1Metric m1Metric; static { IPluginConfig config = EaseAgent.getConfig("observability", "collectM1", ConfigConst.PluginID.METRIC); Tags tags = new Tags("application", "http-request", "url").put("city", "beijing"); ServiceMetricSupplier<M1Metric> m1MetricSupplier = new ServiceMetricSupplier<M1Metric>() { @Override public NameFactory newNameFactory() { return NameFactory.createBuilder() .meterType(MetricSubType.DEFAULT, ImmutableMap.<MetricField, MetricValueFetcher>builder() .put(MetricField.M1_RATE, MetricValueFetcher.MeteredM1RateIgnoreZero) .build()) .build(); } @Override public M1Metric newInstance(MetricRegistry metricRegistry, NameFactory nameFactory) { return new M1Metric(metricRegistry, nameFactory); } }; m1Metric = EaseAgent.getOrCreateServiceMetric(config, tags, m1MetricSupplier); } public void collectM1() { String url = "GET /web_client"; for (int i = 0; i < 100; i++) { m1Metric.collectMetric(url); } } public static class M1Metric extends ServiceMetric { public M1Metric(@Nonnull MetricRegistry metricRegistry, @Nonnull NameFactory nameFactory) { super(metricRegistry, nameFactory); } public void collectMetric(String key) { final Meter meter = meter(key, MetricSubType.DEFAULT); meter.mark(); } } } ``` The above example will export metrics like the following Name and Label: `applicationhttprequestm1{MetricSubType="DEFAULT",MetricType="MeterType",city="beijing",hostipv4="10.127.48.163",hostname="MacBook-Pro.local",service="demo-service",system="demo-system",url="GET /webclient"}` value : `90.0`