Repository: knowledgepixels/nanopub-registry
Pull request #61: Implement prometheus metrics collection and expose metrics through an endpoint
Author: login: nk2IsHere
Labels: name: released
Created: 2025-05-22T19:27:46Z | Merged: 2025-05-22T20:17:55Z

This PR implements the collection and exposure of some domain-specific metrics useful for monitoring registry's state.
The following metrics are added and collected at an interval of 1 second:
- Extended JVM and system metrics
- Current application state (exposed via `registry_server_status` with 5 labeled gauges)
- Load Counter (exposed via `registry_load_counter`) 
- Trust State Counter (exposed via `registry_trust_state_counter`)
- Current Agent Count (exposed via `registry_agent_count`)
- Current Account Count (exposed via `registry_account_count`)

The metrics are exposed and served ready to consume by prometheus through the `/metrics` endpoint:
![image](https://github.com/user-attachments/assets/bd3ab1ad-5771-4cc4-91b0-f19f68de6a94)

Technical notes:
- The PR replaces standard Vertx `Launcher` with a custom `ApplicationLauncher` extension due to a requirement to configure metrics collection before Vertx is initialized.
- All metrics are collected and managed by `MetricsCollector` class


Diff:
diff --git a/docker-compose.yml b/docker-compose.yml
index 37ba029..73fa58b 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -5,6 +5,8 @@ services:
     restart: unless-stopped
     ports:
       - 9292:9292
+      # :9293 is only accessible from localhost as it is used for the /metrics endpoint
+      - "127.0.0.1:9293:9293" 
     volumes:
       - ./setting.trig:/data/setting.trig
     logging:
diff --git a/pom.xml b/pom.xml
index f77a577..696c2b0 100644
--- a/pom.xml
+++ b/pom.xml
@@ -28,7 +28,7 @@
     <maven.build.timestamp.format>yyyy-MM-dd HH:mm</maven.build.timestamp.format>
 
     <main.verticle>com.knowledgepixels.registry.MainVerticle</main.verticle>
-    <launcher.class>io.vertx.core.Launcher</launcher.class>
+    <launcher.class>com.knowledgepixels.registry.ApplicationLauncher</launcher.class>
   </properties>
 
   <repositories>
@@ -66,6 +66,10 @@
       <groupId>io.vertx</groupId>
       <artifactId>vertx-web</artifactId>
     </dependency>
+    <dependency>
+      <groupId>io.vertx</groupId>
+      <artifactId>vertx-micrometer-metrics</artifactId>
+    </dependency>
     <dependency>
       <groupId>io.vertx</groupId>
       <artifactId>vertx-junit5</artifactId>
@@ -134,6 +138,11 @@
       <artifactId>jelly-rdf4j_3</artifactId>
       <version>2.9.1</version>
     </dependency>
+    <dependency>
+      <groupId>io.micrometer</groupId>
+      <artifactId>micrometer-registry-prometheus</artifactId>
+      <version>1.10.13</version>
+    </dependency>
   </dependencies>
 
   <build>
diff --git a/src/main/java/com/knowledgepixels/registry/ApplicationLauncher.java b/src/main/java/com/knowledgepixels/registry/ApplicationLauncher.java
new file mode 100644
index 0000000..0ce2f0f
--- /dev/null
+++ b/src/main/java/com/knowledgepixels/registry/ApplicationLauncher.java
@@ -0,0 +1,24 @@
+package com.knowledgepixels.registry;
+
+import io.vertx.core.Launcher;
+import io.vertx.core.VertxOptions;
+import io.vertx.micrometer.MicrometerMetricsOptions;
+import io.vertx.micrometer.VertxPrometheusOptions;
+
+public class ApplicationLauncher extends Launcher {
+
+    public static void main(String[] args) {
+        new ApplicationLauncher().dispatch(args);
+    }
+
+    @Override
+    public void beforeStartingVertx(VertxOptions options) {
+        options.setMetricsOptions(
+            // Enable Micrometer metrics
+            new MicrometerMetricsOptions()
+                .setPrometheusOptions(new VertxPrometheusOptions().setEnabled(true))
+                .setJvmMetricsEnabled(true)
+                .setEnabled(true)
+        );
+    }
+}
diff --git a/src/main/java/com/knowledgepixels/registry/MainVerticle.java b/src/main/java/com/knowledgepixels/registry/MainVerticle.java
index 1c86378..f03d398 100644
--- a/src/main/java/com/knowledgepixels/registry/MainVerticle.java
+++ b/src/main/java/com/knowledgepixels/registry/MainVerticle.java
@@ -8,6 +8,9 @@
 import java.util.Scanner;
 import java.util.concurrent.TimeUnit;
 
+import io.micrometer.prometheus.PrometheusMeterRegistry;
+import io.vertx.micrometer.PrometheusScrapingHandler;
+import io.vertx.micrometer.backends.BackendRegistries;
 import org.eclipse.rdf4j.rio.RDFFormat;
 import org.eclipse.rdf4j.rio.Rio;
 import org.nanopub.MalformedNanopubException;
@@ -34,7 +37,6 @@ public void start(Promise<Void> startPromise) throws Exception {
 		Router router = Router.router(vertx);
 		server.requestHandler(router);
 		server.listen(9292);
-
 		router.route(HttpMethod.GET, "/agent*").handler(c -> {
 			// /agent/... | /agents | /agentAccounts
 			ListPage.show(c);
@@ -57,6 +59,16 @@ public void start(Promise<Void> startPromise) throws Exception {
 		router.route(HttpMethod.GET, "/style.css").handler(c -> {
 			ResourcePage.show(c, "style.css", "text/css");
 		});
+
+		// Metrics
+		final var metricsHttpServer = vertx.createHttpServer();
+		final var metricsRouter = Router.router(vertx);
+		metricsHttpServer.requestHandler(metricsRouter).listen(9293);
+
+		final var metricsRegistry = (PrometheusMeterRegistry) BackendRegistries.getDefaultNow();
+		final var collector = new MetricsCollector(metricsRegistry);
+		metricsRouter.route("/metrics").handler(PrometheusScrapingHandler.create(metricsRegistry));
+
 		router.route(HttpMethod.GET, "/*").handler(c -> {
 			MainPage.show(c);
 		});
@@ -126,6 +138,9 @@ public void start(Promise<Void> startPromise) throws Exception {
 			System.err.println("DB initialization finished");
 		});
 
+		// Periodic metrics update
+		vertx.setPeriodic(1000, id -> collector.updateMetrics());
+
 		// SHUTDOWN
 		Runtime.getRuntime().addShutdownHook(new Thread(() -> {
 			try {
diff --git a/src/main/java/com/knowledgepixels/registry/MetricsCollector.java b/src/main/java/com/knowledgepixels/registry/MetricsCollector.java
new file mode 100644
index 0000000..594f14a
--- /dev/null
+++ b/src/main/java/com/knowledgepixels/registry/MetricsCollector.java
@@ -0,0 +1,103 @@
+package com.knowledgepixels.registry;
+
+import com.mongodb.client.ClientSession;
+import io.micrometer.core.instrument.Gauge;
+import io.micrometer.core.instrument.MeterRegistry;
+
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static com.knowledgepixels.registry.RegistryDB.*;
+
+public final class MetricsCollector {
+
+    private final AtomicInteger loadCounter = new AtomicInteger(0);
+    private final AtomicInteger trustStateCounter = new AtomicInteger(0);
+    private final AtomicInteger agentCount = new AtomicInteger(0);
+    private final AtomicInteger accountCount = new AtomicInteger(0);
+
+    private final Map<ServerStatus, AtomicInteger> statusStates = new ConcurrentHashMap<>();
+
+    public MetricsCollector(MeterRegistry meterRegistry) {
+        // Numeric metrics
+        Gauge.builder("registry.load.counter", loadCounter, AtomicInteger::get).register(meterRegistry);
+        Gauge.builder("registry.trust.state.counter", trustStateCounter, AtomicInteger::get).register(meterRegistry);
+        Gauge.builder("registry.agent.count", agentCount, AtomicInteger::get).register(meterRegistry);
+        Gauge.builder("registry.account.count", accountCount, AtomicInteger::get).register(meterRegistry);
+
+        // Status label metrics
+        for (final var status : ServerStatus.values()) {
+            AtomicInteger stateGauge = new AtomicInteger(0);
+            statusStates.put(status, stateGauge);
+            Gauge.builder("registry.server.status", stateGauge, AtomicInteger::get)
+                .description("Server status (1 if current)")
+                .tag("status", status.name())
+                .register(meterRegistry);
+        }
+    }
+
+    public void updateMetrics() {
+        try (final var session = RegistryDB.getClient().startSession()) {
+            // Update numeric metrics
+            extractMaximalIntegerValueFromField(session, "nanopubs", "counter")
+                .ifPresent(loadCounter::set);
+
+            extractIntegerValueFromField(session, "serverInfo", "trustStateCounter")
+                .ifPresent(trustStateCounter::set);
+
+            agentCount.set(countDocumentsInCollection(session, "agents"));
+            accountCount.set(countDocumentsInCollection(session, "accounts"));
+
+            // Update status gauge
+            final var currentStatus = extractStringValueFromField(session, "serverInfo", "status")
+                .map(ServerStatus::valueOf)
+                .orElse(null);
+            for (final var status : ServerStatus.values()) {
+                statusStates.get(status).set(status.equals(currentStatus) ? 1 : 0);
+            }
+        } catch (Exception e) {
+            System.err.printf("Error updating metrics: %s%n", e.getMessage());
+        }
+    }
+
+    private Optional<Integer> extractMaximalIntegerValueFromField(
+        ClientSession session,
+        String collectionName,
+        String fieldName
+    ) {
+        return Optional
+            .ofNullable(getMaxValue(session, collectionName, fieldName))
+            .filter(Number.class::isInstance)
+            .map(Number.class::cast)
+            .map(Number::intValue);
+    }
+
+    private Optional<Integer> extractIntegerValueFromField(
+        ClientSession session,
+        String collectionName,
+        String fieldName
+    ) {
+        return Optional
+            .ofNullable(getValue(session, collectionName, fieldName))
+            .filter(Number.class::isInstance)
+            .map(Number.class::cast)
+            .map(Number::intValue);
+    }
+
+    private Optional<String> extractStringValueFromField(
+        ClientSession session,
+        String collectionName,
+        String fieldName
+    ) {
+        return Optional
+            .ofNullable(getValue(session, collectionName, fieldName))
+            .filter(String.class::isInstance)
+            .map(String.class::cast);
+    }
+
+    private int countDocumentsInCollection(ClientSession session, String collectionName) {
+        return (int) collection(collectionName).countDocuments(session);
+    }
+}


Reviews:
user: login: Ostrzyciel
body: Looks good to me!

@tkuhn sending this over to you :)
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-22T19:32:11Z
user: login: Ostrzyciel
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-05-22T20:07:14Z
user: login: Ostrzyciel
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-22T20:13:50Z
user: login: Ostrzyciel
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-22T20:14:08Z

Review comments:
diff_hunk: @@ -57,6 +59,16 @@ public void start(Promise<Void> startPromise) throws Exception {
 		router.route(HttpMethod.GET, "/style.css").handler(c -> {
 			ResourcePage.show(c, "style.css", "text/css");
 		});
+
+		// Metrics
+		final var metricsHttpServer = vertx.createHttpServer();
+		final var metricsRouter = Router.router(vertx);
+		metricsHttpServer.requestHandler(metricsRouter).listen(9293);
path: src/main/java/com/knowledgepixels/registry/MainVerticle.java
user: login: Ostrzyciel
body: Please update docker-compose.yml to expose this to localhost only, along with a comment that it's for the metrics. This should make it easier for people to get started with the endpoint.
created_at: 2025-05-22T20:07:09Z
author_association: COLLABORATOR
diff_hunk: @@ -5,6 +5,7 @@ services:
     restart: unless-stopped
     ports:
       - 9292:9292
+      - "127.0.0.1:9293:9293" # this port should only be accessible from restricted environment as it is used for the metrics endpoint.
path: docker-compose.yml
user: login: Ostrzyciel
body: ```suggestion
      # :9293 is only accessible from localhost as it is used for the /metrics endpoint
      - "127.0.0.1:9293:9293" 
```
created_at: 2025-05-22T20:13:50Z
author_association: COLLABORATOR

PR comments:
user: login: Ostrzyciel
created_at: 2025-05-22T19:39:46Z
body: Ah, I just realized that these metrics probably should not be exposed publicly – it includes potentially sensitive statistics on the JVM which could be used as a tool in a DoS attack.

@tkuhn not sure what is your infrastructure in front of the Registry. We can either:

- Hide the /metrics endpoint from the public by simply not forwarding requests to it in the proxy (e.g., nginx), if there is a proxy in front of the service.
- Or we could make a completely new HTTP listener on a different point, dedicated to non-public endpoints (e.g., metrics, other diagnostic stuff).

The same applies to Query: https://github.com/knowledgepixels/nanopub-query/pull/36
author_association: COLLABORATOR
user: login: Ostrzyciel
created_at: 2025-05-22T19:40:40Z
body: Converting to draft to avoid merging this until we resolve this issue.

author_association: COLLABORATOR
user: login: tkuhn
created_at: 2025-05-22T19:53:56Z
body: OK, looking good!

We are using Nginx upfront on the servers, so it should be easy to block /metrics there.
author_association: CONTRIBUTOR
user: login: tkuhn
created_at: 2025-05-22T19:55:49Z
body: Though it would be a bit more complicated to tell people how to set it up themselves. If it can be done on a different port, which the docker-compose.yml doesn't expose by default, that'd be better, I think. Or just an environment var to turn it on/off with off as default?
author_association: CONTRIBUTOR
user: login: Ostrzyciel
created_at: 2025-05-22T19:56:05Z
body: > We are using Nginx upfront on the servers, so it should be easy to block /metrics there.

OK, unblocking it then, all good to merge from our side.


author_association: COLLABORATOR
user: login: Ostrzyciel
created_at: 2025-05-22T19:58:02Z
body: Ah, wrote my reply before you posted the second comment! :) I think a solution with two ports will be better, it's also how I've implemented it in some other projects.

@nk2IsHere could you apply this for Registry and Query alike? Please expose this new port only to 127.0.0.1
author_association: COLLABORATOR
user: login: nk2IsHere
created_at: 2025-05-22T20:03:29Z
body: Team, metrics are now exposed through port `9293`
author_association: CONTRIBUTOR
user: login: Ostrzyciel
created_at: 2025-05-22T20:14:27Z
body: @tkuhn should be good now – can you have a look?
author_association: COLLABORATOR
user: login: tkuhn
created_at: 2025-05-22T20:17:48Z
body: Great, thanks. Yes, looking good from what I can see, and will merge.
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-12-19T15:10:06Z
body: :tada: This PR is included in version 1.0.0 :tada:

The release is available on:
- `nanopub-registry-1.0.0`
- [GitHub release](https://github.com/knowledgepixels/nanopub-registry/releases/tag/nanopub-registry-1.0.0)

Your **[semantic-release](https://github.com/semantic-release/semantic-release)** bot :package::rocket:
author_association: NONE