Repository: mlflow/mlflow

Issue:
[BUG] Service metrics endpoint excludes many important routes
### Willingness to contribute
The MLflow Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the MLflow code base?

- [x] Yes. I can contribute a fix for this bug independently.
- [ ] Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.
- [ ] No. I cannot contribute a bug fix at this time.

### System information
- **Have I written custom code (as opposed to using a stock example script provided in MLflow)**: no
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Debian GNU/Linux 11
- **MLflow installed from (source or binary)**: binary
- **MLflow version (run ``mlflow --version``)**: 1.21.0
- **Python version**: 3.8
- **npm version, if running the dev UI**:
- **Exact command to reproduce**: see section below

### Describe the problem

The mlflow server option to expose a Prometheus metrics endpoint is a great observability feature for MLflow. Unfortunately, the current implementation leaves an incomplete view of the server health/performance. Currently, mlflow only logs metrics to a [subset of endpoints](https://github.com/mlflow/mlflow/blob/master/mlflow/server/prometheus_exporter.py#L18).

As of MLflow version 1.21, the following routes are not being included in the service metrics:

```
['static', '_get_experiment_by_name', '_create_experiment', '_list_experiments', '_get_experiment',
'_delete_experiment', '_restore_experiment', '_update_experiment',  '_update_run', '_delete_run', 
'_restore_run', '_set_experiment_tag',  '_delete_tag', '_get_run',  '_list_artifacts', '_get_metric_history',
'_log_batch', '_log_model', '_create_registered_model', '_rename_registered_model', 
'_update_registered_model',  '_delete_registered_model', '_get_registered_model', '_search_registered_models', 
'_list_registered_models',  '_get_latest_versions', '_create_model_version', '_update_model_version', 
'_transition_stage', '_delete_model_version',  '_get_model_version', '_search_model_versions', 
'_get_model_version_download_uri', '_set_registered_model_tag', '_set_model_version_tag', 
'_delete_registered_model_tag', '_delete_model_version_tag', 'health', 'serve_artifacts', 
'serve_model_version_artifact', 'serve_static_file', 'serve']
```

(see full list of endpoints)
```
from mlflow.server import app

app.view_functions.keys()
```

Filtering the set of routes to be included in the metrics endpoint seems like a potentially fragile approach as new routes are added in later versions of mlflow. It's especially problematic that the list of filtered routes cannot be configured. We currently have no way to monitor the health of the overall service given that many key routes (e.g. `log_batch`) are not included in the service metrics.

### Code to reproduce issue

Dockerfile for mlflow server
```
FROM python:3.8
RUN pip install mlflow==1.21.0

ENTRYPOINT mlflow server \
    --backend-store-uri sqlite:///mlflow.sqlite \
    --default-artifact-root file:///artifacts \
    --host 0.0.0.0 \
    --port 5000 \
    --expose-prometheus /prometheus
```

Build and run the Docker container
```
docker build -t mlflow_example -f Dockerfile .
docker run -p 5000:5000 mlflow_example
```


Script with incomplete representation in metrics endpoint
```
import mlflow
import random

mlflow.set_tracking_uri("http://127.0.0.1:5000")
mlflow.set_experiment("service_metrics")

with mlflow.start_run(run_name="test"):

    for _ in range(100):
        mlflow.log_metrics({
            'loss_a': random.random(),
            'loss_b': random.random(),
            'loss_c': random.random(),
        })

    mlflow.log_params({'a': 1, 'b': 2, 'c': 3})
```
See how metrics for these endpoints **_do not_** appear at http://127.0.0.1:5000/metrics

---

Script with expected representation in metrics endpoint
```
import mlflow
import random

mlflow.set_tracking_uri("http://127.0.0.1:5000")
mlflow.set_experiment("service_metrics")

with mlflow.start_run(run_name="test"):
    for _ in range(100):
        mlflow.log_metric('loss', random.random())

    mlflow.log_param('param', 'test')
```
See how metrics for these endpoints appear at http://127.0.0.1:5000/metrics

### Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.


### What component(s), interfaces, languages, and integrations does this bug affect?
Components 
- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [x] `area/server-infra`: MLflow Tracking server backend
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

Interface 
- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

Language 
- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

Integrations
- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations



Discussion:
cc @t-henri 

Wanted to get this on your radar since you originally contributed the Prometheus metrics exporter feature (#2097) and related PRs/changes may affect you - do you have any opinions on how we resolve this issue? Could you provide some background as to why you had originally chosen to filter the list of routes to be instrumented?
Hello,

I think if you feel like that some routes should be monitored, you can just add them (to be confirmed by a maintainer of mlflow). 
Back then I only added these routes because they were the one that were always called in our different jobs, so it gave a pretty good idea of the server health. Having metrics on routes like health or create_experiment doesn't give us much information.
Also back then the registry didn't exist, so you might want to add some metrics for it.
Sounds good, thanks for the context @t-henri. I wasn't sure if you were cautious in filtering out specific routes to avoid having a high-cardinality label, which is a concern that I have if we have an open-ended path label included in the metric. Generally a good rule of thumb for Prometheus metrics is to keep the cardinality for any given label less than 10. Whereas there are currently 47 endpoints and that number can potentially grow.

I also have concern with maintainability if we keep an explicit list of paths to filter, as its likely that as new endpoints are added to the server people won't know to also update a list in `mlflow/server/prometheus_exporter.py`. 

The simplest approach, in my opinion, towards getting better operational visibility would be to remove the path label from the metric and instead have request counts/latencies grouped by HTTP verb and HTTP status (and we can choose to explicitly _exclude_ some endpoints like the health endpoint if we'd like). 
@dbczumar do you have any thoughts on the proposed change? I'm happy to open a PR with a fix once we land on an approved design.
Hi @jeremyjordan , your proposal sounds good! We'd be happy to review a PR for this.

Resolving patch:
diff --git a/mlflow/server/prometheus_exporter.py b/mlflow/server/prometheus_exporter.py
--- a/mlflow/server/prometheus_exporter.py
+++ b/mlflow/server/prometheus_exporter.py
@@ -1,32 +1,19 @@
 from prometheus_flask_exporter.multiprocess import GunicornInternalPrometheusMetrics
 from flask import request
 
+from mlflow.version import VERSION
+
 
 def activate_prometheus_exporter(app):
-    metrics = GunicornInternalPrometheusMetrics(app, export_defaults=False)
+    def mlflow_version(_: request):
+        return VERSION
 
-    endpoint = app.view_functions
-    histogram = metrics.histogram(
-        "mlflow_requests_by_status_and_path",
-        "Request latencies and count by status and path",
-        labels={
-            "status": lambda r: r.status_code,
-            "path": lambda: change_path_for_metric(request.path),
-        },
+    metrics = GunicornInternalPrometheusMetrics(
+        app,
+        export_defaults=True,
+        defaults_prefix="mlflow",
+        excluded_paths=["/health"],
+        group_by=mlflow_version,
     )
-    for func_name, func in endpoint.items():
-        if func_name in ["_search_runs", "_log_metric", "_log_param", "_set_tag", "_create_run"]:
-            app.view_functions[func_name] = histogram(func)
-
-    return app
-
 
-def change_path_for_metric(path):
-    """
-    Replace the '/' in the metric path by '_' so grafana can correctly use it.
-    :param path: path of the metric (example: runs/search)
-    :return: path with '_' instead of '/'
-    """
-    if "mlflow/" in path:
-        path = path.split("mlflow/")[-1]
-    return path.replace("/", "_")
+    return metrics


Test patch:
diff --git a/tests/server/test_prometheus_exporter.py b/tests/server/test_prometheus_exporter.py
new file mode 100644
--- /dev/null
+++ b/tests/server/test_prometheus_exporter.py
@@ -0,0 +1,69 @@
+import os
+from unittest import mock
+
+import pytest
+
+
+from mlflow.server.prometheus_exporter import activate_prometheus_exporter
+
+
+@pytest.fixture(autouse=True)
+def mock_settings_env_vars(tmpdir):
+    with mock.patch.dict(os.environ, {"PROMETHEUS_MULTIPROC_DIR": tmpdir.strpath}):
+        yield
+
+
+@pytest.fixture()
+def app():
+    from mlflow.server import app
+
+    with app.app_context():
+        yield app
+
+
+@pytest.fixture()
+def test_client(app):
+    with app.test_client() as c:
+        yield c
+
+
+def test_metrics(app, test_client):
+    metrics = activate_prometheus_exporter(app)
+
+    # test metrics for successful responses
+    success_labels = {"method": "GET", "status": "200"}
+    assert (
+        metrics.registry.get_sample_value("mlflow_http_request_total", labels=success_labels)
+        is None
+    )
+    resp = test_client.get("/")
+    assert resp.status_code == 200
+    assert (
+        metrics.registry.get_sample_value("mlflow_http_request_total", labels=success_labels) == 1
+    )
+
+    # calling the metrics endpoint should not increment the counter
+    resp = test_client.get("/metrics")
+    assert resp.status_code == 200
+    assert (
+        metrics.registry.get_sample_value("mlflow_http_request_total", labels=success_labels) == 1
+    )
+
+    # calling the health endpoint should not increment the counter
+    resp = test_client.get("/health")
+    assert resp.status_code == 200
+    assert (
+        metrics.registry.get_sample_value("mlflow_http_request_total", labels=success_labels) == 1
+    )
+
+    # test metrics for failed responses
+    failure_labels = {"method": "GET", "status": "404"}
+    assert (
+        metrics.registry.get_sample_value("mlflow_http_request_total", labels=failure_labels)
+        is None
+    )
+    resp = test_client.get("/non-existent-endpoint")
+    assert resp.status_code == 404
+    assert (
+        metrics.registry.get_sample_value("mlflow_http_request_total", labels=failure_labels) == 1
+    )