Custom metrics source in apache spark
By: Simhadri G
Tags: Metrics, Apache Spark, Big Data Analytics

Metrics provide a window through which we can analyze the behavior of various components in a running application. They provide an easy way to measure performance, identify bottlenecks or detect bugs in a running application . Therefore, most of the software frameworks have a wide variety of metrics exposed by default. But sometimes, we may need to set up additional custom metrics based on the needs of our application. Having a custom metric tailored to our application will greatly simplify the analysis of an application.

In this article we will take a brief look at how we can set up custom metric source.

Apache Spark

Spark exposes wide variety of metrics for external consumption. These

metrics include things like resource usage, scheduling delay, executor

time etc. These metrics can be consumed using wide variety of sinks

for further analysis. But there are many cases where we will need to implement our own custom metric which will allow for advanced monitoring and tracking which will in turn help us to tune spark better.

To set up a custom metric, spark provides us a configurable metrics system based on the Dropwizard Metrics Library. This allows users to report Spark metrics to a variety of sinks including HTTP, JMX, and CSV files. The metrics are generated by sources embedded in the Spark code base. The metrics system is configured via a configuration file that Spark expects to be present at $SPARK_HOME/conf/metrics.properties .These metrics are can also be accessed as JSON from the REST endpoints described by spark . This gives developers an easy way to create new visualizations and monitoring tools for Spark. The JSON is available for both running applications, and in the history server. The endpoints are mounted at /api/v1 . For example, for the history server, they would typically be accessible at http://<server-url>:18080/api/v1 , and for a running application, at http://localhost:4040/api/v1 . For more details on spark monitoring, metrics and rest api’s please refer: https://spark.apache.org/docs/latest/monitoring.html .

Next let us see how we can setup a simple metric source and sink.

Setting up a simple metric source

Here are the step to set up a simple spark metric source:

The first step is to write a class that extends the Source trait. org.apache.spark.metrics.source.Source is the top-level class for the

metric registries in Spark. Sources expose their internal status. Next, enable the sink. Here, the output is printed to console. Finally, instantiate the source and register it with SparkEnv. Now we are ready to collect the metrics.

Following code snippet demonstrates a simple metrics source:

Output after running the above code should contain the following output:

-- Counters --------------------------------------------------------------------

...

local-1813126621247.driver. CustomMetricSource . metricCounter

count = 10 -- Histograms ------------------------------------------------------------------

... local-1813126621247.driver. CustomMetricSource . metricHistory

count = 10

min = 1

max = 1

mean = 1.00

stddev = 0.00

median = 1.00

75% <= 1.00

95% <= 1.00

98% <= 1.00

99% <= 1.00

99.9% <= 1.00

References:

[1]https://spark.apache.org/docs/latest/monitoring.html