Repository: dask/distributed

Issue:
Instrumentation of SpillBuffer
The only way we currently have to observe disk access is `startstops` we measure whenever we load/store data.

However, with our `SpillBuffer` we have the possibility to introduce many instrumentation hooks to get much better insights into what's going on

For instance
- Total number of keys written / read
- Number of currently managed files
- Average and total size of data written to disk per key
- Total duration spent writing / reading
- Average / max time spent reading / writing per key
- Number of evicts (memory.spill)
- Number of writes without evict because buffer is full right away
- ...

These metrics should not be tracked on `TaskState` level.


Discussion:
> Number of writes without evict because buffer is full right away

Not sure if I understand this one. The only case where a key doesn't end up in fast when you write to the SpillBuffer is when it's individually larger than the target threshold.
> Not sure if I understand this one. The only case where a key doesn't end up in fast when you write to the SpillBuffer is when it's individually larger than the target threshold.

I thought there was some logic that would put a key into slow if this key would push us over the limit. If that's not the case, ignore this, I don't think data shards larger than the limit are a very common problem we need to build instrumentation for
> I thought there was some logic that would put a key into slow if this key would push us over the limit.

No, if inserting a key pushes us over the limit, the least recently used keys are pushed out. The latest inserted one is on top of the LRU pipe and is the only one guaranteed to be in fast at the end of the insertion.
The thing I was hoping to differentiate with this comment is the spilling that happens as a result of setitem vs the spilling that happens when the memory_manager is evicting.
If it's not the same key, that's fine but if there is an evict coupled to the setitem, that might be an interesting datapoint
> spilling that happens as a result of setitem vs the spilling that happens when the memory_manager is evicting.

Both will evict the same keys, in the same order. The only difference is that memory_manager kicks in when there's substantial unmanaged memory (but on the flip side it's less responsive).

Resolving patch:
diff --git a/docs/source/conf.py b/docs/source/conf.py
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -411,7 +411,6 @@
         "https://docs.dask.org/en/latest/setup/single-distributed.html",
     ),
     ("adaptive.html", "https://docs.dask.org/en/latest/setup/adaptive.html"),
-    ("prometheus.html", "https://docs.dask.org/en/latest/setup/prometheus.html"),
     ("web.html", "https://docs.dask.org/en/latest/diagnostics-distributed.html"),
 ]
 


Test patch:
diff --git a/distributed/http/tests/test_core.py b/distributed/http/tests/test_core.py
--- a/distributed/http/tests/test_core.py
+++ b/distributed/http/tests/test_core.py
@@ -1,8 +1,12 @@
 from __future__ import annotations
 
+import pathlib
+
+import pytest
 from tornado.httpclient import AsyncHTTPClient
 
-from distributed.utils_test import gen_cluster
+from distributed import Semaphore
+from distributed.utils_test import fetch_metrics_sample_names, gen_cluster, inc
 
 
 @gen_cluster(client=True)
@@ -10,3 +14,49 @@ async def test_scheduler(c, s, a, b):
     client = AsyncHTTPClient()
     response = await client.fetch(f"http://localhost:{s.http_server.port}/health")
     assert response.code == 200
+
+
+@gen_cluster(client=True, nthreads=[("", 1)])
+async def test_prometheus_api_doc(c, s, a):
+    """Test that the Sphinx documentation of Prometheus endpoints matches the
+    implementation.
+    """
+    pytest.importorskip("prometheus_client")
+
+    # Some metrics only appear after a task is executed
+    await c.submit(inc, 1)
+    # Semaphore metrics only appear after semaphores are used
+    sem = await Semaphore()
+    await sem.acquire()
+    await sem.release()
+
+    # Note: built-in Prometheus metrics are undocumented
+    scheduler_metrics = await fetch_metrics_sample_names(
+        s.http_server.port, prefix="dask_"
+    )
+    worker_metrics = await fetch_metrics_sample_names(
+        a.http_server.port, prefix="dask_"
+    )
+
+    try:
+        import crick  # noqa: F401
+
+        crick_metrics = set()  # Already in worker_metrics
+    except ImportError:
+        crick_metrics = {
+            "dask_worker_tick_duration_median_seconds",
+            "dask_worker_task_duration_median_seconds",
+            "dask_worker_transfer_bandwidth_median_bytes",
+        }
+
+    implemented = scheduler_metrics | worker_metrics | crick_metrics
+
+    documented = set()
+    root_dir = pathlib.Path(__file__).parent.parent.parent.parent
+    with open(root_dir / "docs" / "source" / "prometheus.rst") as fh:
+        for row in fh:
+            row = row.strip()
+            if row.startswith("dask_"):
+                documented.add(row)
+
+    assert documented == implemented