Package: mask-prometheus
MaskgRPC Interceptors for Prometheus monitoringInstallpip install mask-prometheusUsage# 3p
from mask import Mask
from mask_prometheus import Prometheus
# project
from examples.protos.hello_pb2 import HelloResponse


app = Mask(__name__)

app.config["PROMETHEUS_PORT"] = 18080

prometheus = Prometheus()
prometheus.init_app(app)



@app.route(method="SayHelloStream", service="Hello")
def say_hello_stream_handler(request, context):
    """ Handler stream SayHello request
    """
    for item in request:
        yield HelloResponse(message="Hello Reply: %s" % item.name)

if __name__ == "__main__":
    app.run(port=1020)MetricsLabelsAll metrics start withmaskas Prometheus subsystem name. Both of them have mirror-concepts. Similarly all methods
contain the same rich labels:grpc_service- thegRPC servicename, which is the combination of protobufpackageand
thegrpc_servicesection name. E.g. forpackage = mwitkow.testprotoandservice TestServicethe label will begrpc_service="mwitkow.testproto.TestService"grpc_method- the name of the method called on the gRPC service. E.g.grpc_method="Ping"grpc_type- the gRPCtype of request.
Differentiating between the two is important especially for latency measurements.unaryis single request, single response RPCclient_streamis a multi-request, single response RPCserver_streamis a single request, multi-response RPCbidi_streamis a multi-request, multi-response RPCAdditionally for completed RPCs, the following labels are used:grpc_code- the human-readablegRPC status code.
The list of all statuses is to long, but here are some common ones:StatusCode.OK- means the RPC was successfulStatusCode.IllegalArgument- RPC contained bad valuesStatusCode.Internal- server-side error not disclosed to the clientsCountersFor simplicity, let's assume we're tracking a single server-side RPC call ofmwitkow.testproto.TestService,
calling the methodPingList. The call succeeds and returns 20 messages in the stream.First, immediately after the server receives the call it will increment thegrpc_server_started_totaland start the handling time clock (if histograms are enabled).grpc_server_started_total{grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream"} 1Then the user logic gets invoked. It receives one message from the client containing the request
(it's aserver_stream):grpc_server_msg_received_total{grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream"} 1The user logic may return an error, or send multiple messages back to the client. In this case, on
each of the 20 messages sent back, a counter will be incremented:grpc_server_msg_sent_total{grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream"} 20After the call completes, its status (OKor othergRPC status code)
and the relevant call labels increment thegrpc_server_handled_totalcounter.grpc_server_handled_total{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream"} 1HistogramsPrometheus histogramsare a great way
to measure latency distributions of your RPCs. However, since it is bad practice to have metrics
ofhigh cardinalitythe latency monitoring metrics are disabled by default.After the call completes, its handling time will be recorded in aPrometheus histogramvariablegrpc_server_handling_seconds. The histogram variable contains three sub-metrics:grpc_server_handling_seconds_count- the count of all completed RPCs by status and methodgrpc_server_handling_seconds_sum- cumulative time of RPCs by status and method, useful for
calculating average handling timesgrpc_server_handling_seconds_bucket- contains the counts of RPCs by status and method in respective
handling-time buckets. These buckets can be used by Prometheus to estimate SLAs (seehere)The counter values will look as follows:grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="0.005"} 1
grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="0.01"} 1
grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="0.025"} 1
grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="0.05"} 1
grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="0.1"} 1
grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="0.25"} 1
grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="0.5"} 1
grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="1"} 1
grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="2.5"} 1
grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="5"} 1
grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="10"} 1
grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="+Inf"} 1
grpc_server_handling_seconds_sum{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream"} 0.0003866430000000001
grpc_server_handling_seconds_count{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream"} 1Useful query examplesPrometheus philosophy is to provide raw metrics to the monitoring system, and
let the aggregations be handled there. The verbosity of above metrics make it possible to have that
flexibility. Here's a couple of useful monitoring queries:request inbound ratesum(rate(grpc_server_started_total{job="foo"}[1m])) by (grpc_service)Forjob="foo"(common label to differentiate between Prometheus monitoring targets), calculate the
rate of requests per second (1 minute window) for each gRPCgrpc_servicethat the job has. Please note
how thegrpc_methodis being omitted here: all methods of a given gRPC service will be summed together.unary request error ratesum(rate(grpc_server_handled_total{job="foo",grpc_type="unary",grpc_code!="OK"}[1m])) by (grpc_service)Forjob="foo", calculate the per-grpc_servicerate ofunary(1:1) RPCs that failed, i.e. the
ones that didn't finish withOKcode.unary request error percentagesum(rate(grpc_server_handled_total{job="foo",grpc_type="unary",grpc_code!="OK"}[1m])) by (grpc_service)
 / 
sum(rate(grpc_server_started_total{job="foo",grpc_type="unary"}[1m])) by (grpc_service)
 * 100.0Forjob="foo", calculate the percentage of failed requests by service. It's easy to notice that
this is a combination of the two above examples. This is an example of a query you would like toalert onin your system for SLA violations, e.g.
"no more than 1% requests should fail".average response stream sizesum(rate(grpc_server_msg_sent_total{job="foo",grpc_type="server_stream"}[10m])) by (grpc_service)
 /
sum(rate(grpc_server_started_total{job="foo",grpc_type="server_stream"}[10m])) by (grpc_service)Forjob="foo"what is thegrpc_service-wide10maverage of messages returned for allserver_streamRPCs. This allows you to track the stream sizes returned by your system, e.g. allows
you to track when clients started to send "wide" queries that ret
Note the divisor is the number of started RPCs, in order to account for in-flight requests.99%-tile latency of unary requestshistogram_quantile(0.99, 
  sum(rate(grpc_server_handling_seconds_bucket{job="foo",grpc_type="unary"}[5m])) by (grpc_service,le)
)Forjob="foo", returns an 99%-tilequantile estimationof the handling time of RPCs per service. Please note the5mrate, this means that the quantile
estimation will take samples in a rolling5mwindow. When combined with other quantiles
(e.g. 50%, 90%), this query gives you tremendous insight into the responsiveness of your system
(e.g. impact of caching).percentage of slow unary queries (>250ms)100.0 - (
sum(rate(grpc_server_handling_seconds_bucket{job="foo",grpc_type="unary",le="0.25"}[5m])) by (grpc_service)
 / 
sum(rate(grpc_server_handling_seconds_count{job="foo",grpc_type="unary"}[5m])) by (grpc_service)
) * 100.0Forjob="foo"calculate the by-grpc_servicefraction of slow requests that took longer than0.25seconds. This query is relatively complex, since the Prometheus aggregations usele(less or equal)
buckets, meaning that counting "fast" requests fractions is easier. However, simple maths helps.
This is an example of a query you would like to alert on in your system for SLA violations,
e.g. "less than 1% of requests are slower than 250ms".