Bug: Prometheus metrics for Master and Worker isn't working 
Status: Reopened | Priority: Major | Resolution: 
Created: 25/Nov/20 19:52
Affects Version/s: 3.0.0

Following the [PR|https://github.com/apache/spark/pull/25769] that introduced the Prometheus sink, I downloaded the {{spark-3.0.1-bin-hadoop2.7.tgz}}  (also tested with 3.0.0), uncompressed the tgz and created a file called {{metrics.properties}} adding this content:
{quote}{{*.sink.prometheusServlet.class=org.apache.spark.metrics.sink.PrometheusServlet}}
 {{*.sink.prometheusServlet.path=/metrics/prometheus}}
 master.sink.prometheusServlet.path=/metrics/master/prometheus
 applications.sink.prometheusServlet.path=/metrics/applications/prometheus
{quote}
Then I ran: 
{quote}{{$ sbin/start-master.sh}}
 {{$ sbin/start-slave.sh spark://`hostname`:7077}}
 {{$ bin/spark-shell --master spark://`hostname`:7077 --files=./metrics.properties --conf spark.metrics.conf=./metrics.properties}}
{quote}
{{The Spark shell opens without problems:}}
{quote}{{20/11/25 17:36:07 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable}}

{{Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties}}

{{Setting default log level to "WARN".}}

{{To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).}}

{{Spark context Web UI available at [http://192.168.0.6:4040|http://192.168.0.6:4040/]}}

{{Spark context available as 'sc' (master = spark://MacBook-Pro-de-Paulo-2.local:7077, app id = app-20201125173618-0002).}}

{{Spark session available as 'spark'.}}

{{Welcome to}}

{{      ____              __}}

{{     / __/_  ____ _____/ /__}}

{{    _\ \/ _ \/ _ `/ __/  '_/}}

{{   /___/ .__/_,_/_/ /_/_\   version 3.0.0}}

{{      /_/}}

{{         }}

{{Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_212)}}

{{Type in expressions to have them evaluated.}}

{{Type :help for more information. }}

{{scala>}}
{quote}
{{And when I try to fetch prometheus metrics for driver, everything works fine:}}
{quote}$ curl -s [http://localhost:4040/metrics/prometheus/] | head -n 5

metrics_app_20201125173618_0002_driver_BlockManager_disk_diskSpaceUsed_MB_Number\{type="gauges"} 0

metrics_app_20201125173618_0002_driver_BlockManager_disk_diskSpaceUsed_MB_Value\{type="gauges"} 0

metrics_app_20201125173618_0002_driver_BlockManager_memory_maxMem_MB_Number\{type="gauges"} 732

metrics_app_20201125173618_0002_driver_BlockManager_memory_maxMem_MB_Value\{type="gauges"} 732

metrics_app_20201125173618_0002_driver_BlockManager_memory_maxOffHeapMem_MB_Number\{type="gauges"} 0
{quote}
*The problem appears when I try accessing master metrics*, and I get the following problem:
{quote}{{$ curl -s [http://localhost:8080/metrics/master/prometheus]}}

{{<!DOCTYPE html><html>}}

{{      <head>}}

{{        <meta http-equiv="Content-type" content="text/html; charset=utf-8"/><link rel="stylesheet" href="/static/bootstrap.min.css" type="text/css"/><link rel="stylesheet" href="/static/vis-timeline-graph2d.min.css" type="text/css"/><link rel="stylesheet" href="/static/webui.css" type="text/css"/><link rel="stylesheet" href="/static/timeline-view.css" type="text/css"/><script src="/static/sorttable.js"></script><script src="/static/jquery-3.4.1.min.js"></script><script src="/static/vis-timeline-graph2d.min.js"></script><script src="/static/bootstrap-tooltip.js"></script><script src="/static/initialize-tooltips.js"></script><script src="/static/table.js"></script><script src="/static/timeline-view.js"></script><script src="/static/log-view.js"></script><script src="/static/webui.js"></script><script>setUIRoot('')</script>}}

{{        }}

{{        <link rel="shortcut icon" href="/static/spark-logo-77x50px-hd.png"></link>}}

{{        <title>Spark Master at spark://MacBook-Pro-de-Paulo-2.local:7077</title>}}

{{      </head>}}

{{      <body>}}

{{        <div class="container-fluid">}}

{{          <div class="row-fluid">}}

{{            <div class="span12">}}

{{              <h3 style="vertical-align: middle; display: inline-block;">}}

{{                <a style="text-decoration: none" href="/">}}

{{                  <img src="/static/spark-logo-77x50px-hd.png"/>}}

{{                  <span class="version" style="margin-right: 15px;">3.0.0</span>}}

{{                </a>}}

{{                Spark Master at spark://MacBook-Pro-de-Paulo-2.local:7077}}

{{              </h3>}}

{{            </div>}}

{{          </div>}}

{{          <div class="row-fluid">}}

{{          <div class="span12">}}

{{            <ul class="unstyled">}}

{{              <li><strong>URL:</strong> spark://MacBook-Pro-de-Paulo-2.local:7077</li>}}
 ...
{quote}
Instead of the metrics I'm getting an HTML page.  The same happens for all of those here:
{quote}{{$ curl -s [http://localhost:8080/metrics/applications/prometheus/]}}
 {{$ curl -s [http://localhost:8081/metrics/prometheus/]}}
{quote}
Instead, *I expected metrics in prometheus metrics*. All related JSON endpoints seem to be working fine.