Repository: DataDog/integrations-core
Pull request #19681: Add more FoundationDB metrics
Author: login: jon-signal
Labels: name: integration/foundationdb
name: team/agent-integrations
name: ecosystems/review-requested
name: product/review-requested
name: agent/approved
Created: 2025-02-21T17:01:27Z | Merged: 2025-03-18T15:45:23Z

### What does this PR do?

This pull request adds new metrics to the FoundationDB integration. In a small number of instances, it adds new dimensions to existing metrics in a manner that should not affect existing queries.

### Motivation

We're starting our journey with FoundationDB, and have found that the existing integration—while we're very happy to have it!—is missing some metrics we hope to track. Mostly, this change adds metrics mentioned in the ["Monitored Metrics"](https://apple.github.io/foundationdb/monitored-metrics.html) section of the FoundationDB documentation. There are a few deviations, though:

- We didn't add backup/disaster recovery metrics quite yet; we can follow up on those in a separate pull request after we get a little more hands-on experience with that part of FoundationDB
- We didn't add machine-related metrics where operators are likely to be getting equivalent metrics from other sources (e.g. the Datadog Agent is almost certainly reporting machine-level CPU load by other means)
- We added a couple metrics (like partition counts/sizes) that aren't called out by the docs, but that still seemed helpful to us for understanding overall database performance and behavior

~We broke this into a lot of very small commits, but are happy to squash as needed.~ Happy to revise the commit boundaries as you folks see fit!

Thanks!

### Review checklist (to be filled by reviewers)

- [ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
- [ ] Add the `qa/skip-qa` label if the PR doesn't need to be tested during QA.
- [ ] If you need to backport this PR to another branch, you can add the `backport/<branch-name>` label to the PR and it will automatically open a backport PR once this one is merged



Diff:
diff --git a/foundationdb/changelog.d/19681.added b/foundationdb/changelog.d/19681.added
new file mode 100644
index 0000000000000..d298efb43c229
--- /dev/null
+++ b/foundationdb/changelog.d/19681.added
@@ -0,0 +1 @@
+Add more FoundationDB metrics
diff --git a/foundationdb/datadog_checks/foundationdb/check.py b/foundationdb/datadog_checks/foundationdb/check.py
index dc81614705785..cf8c01a68787e 100644
--- a/foundationdb/datadog_checks/foundationdb/check.py
+++ b/foundationdb/datadog_checks/foundationdb/check.py
@@ -96,6 +96,7 @@ def report_process(self, process):
             self.maybe_gauge("process.cpu.usage_cores", process["cpu"], "usage_cores", tags)
         if "disk" in process:
             disk = process["disk"]
+            self.maybe_gauge("process.disk.busy", disk, "busy", tags)
             self.maybe_gauge("process.disk.free_bytes", disk, "free_bytes", tags)
             self.maybe_gauge("process.disk.total_bytes", disk, "total_bytes", tags)
             if "reads" in disk:
@@ -150,6 +151,8 @@ def report_role(self, role, process_tags):
         self.maybe_gauge("process.role.kvstore_total_nodes", role, "kvstore_total_nodes", tags)
         self.maybe_gauge("process.role.kvstore_total_size", role, "kvstore_total_size", tags)
         self.maybe_gauge("process.role.kvstore_used_bytes", role, "kvstore_used_bytes", tags)
+        self.maybe_gauge("process.role.queue_disk_available_bytes", role, "queue_disk_available_bytes", tags)
+        self.maybe_gauge("process.role.queue_disk_total_bytes", role, "queue_disk_total_bytes", tags)
 
         if "data_lag" in role:
             self.maybe_gauge("process.role.data_lag.seconds", role["data_lag"], "seconds", tags)
@@ -182,20 +185,43 @@ def check_metrics(self, status):
         if "cluster" not in status:
             raise ValueError("JSON Status data doesn't include cluster data")
 
+        if "client" in status:
+            client = status["client"]
+
+            if "coordinators" in client:
+                if "coordinators" in client["coordinators"]:
+                    coordinators = client["coordinators"]["coordinators"]
+
+                    reachable_coordinators = sum([1 for coordinator in coordinators if coordinator["reachable"]])
+                    unreachable_coordinators = len(coordinators) - reachable_coordinators
+
+                    self.gauge("coordinators", reachable_coordinators, ["reachable:true"])
+                    self.gauge("coordinators", unreachable_coordinators, ["reachable:false"])
+
         cluster = status["cluster"]
         if "machines" in cluster:
-            self.gauge("machines", len(cluster["machines"]))
+            machines = cluster["machines"]
+
+            self.gauge("machines", len(machines))
+            self.gauge("excluded_machines", sum([1 for machine_key in machines if machines[machine_key]["excluded"]]))
         if "processes" in cluster:
-            self.gauge("processes", len(cluster["processes"]))
+            processes = cluster["processes"]
+
+            self.gauge("processes", len(processes))
+            self.gauge(
+                "excluded_processes", sum([1 for process_key in processes if processes[process_key]["excluded"]])
+            )
 
             self.count(
                 "instances",
-                sum((len(p["roles"]) if "roles" in p else 0 for p in cluster["processes"].values())),
+                sum((len(p["roles"]) if "roles" in p else 0 for p in processes.values())),
             )
 
             role_counts = {}
-            for process_key in cluster["processes"]:
-                process = cluster["processes"][process_key]
+
+            for process_key in processes:
+                process = processes[process_key]
+
                 self.report_process(process)
                 if "roles" in process:
                     for role in process["roles"]:
@@ -220,6 +246,9 @@ def check_metrics(self, status):
                 "least_operating_space_bytes_log_server",
             )
 
+            self.maybe_gauge("data.average_partition_size_bytes", data, "average_partition_size_bytes")
+            self.maybe_gauge("data.partitions_count", data, "partitions_count")
+
             if "moving_data" in data:
                 self.maybe_gauge("data.moving_data.in_flight_bytes", data["moving_data"], "in_flight_bytes")
                 self.maybe_gauge("data.moving_data.in_queue_bytes", data["moving_data"], "in_queue_bytes")
@@ -240,10 +269,31 @@ def check_metrics(self, status):
                 for k in operations:
                     self.maybe_hz_counter("workload.operations." + k, operations, k)
 
+            if "keys" in workload:
+                keys = workload["keys"]
+                for k in keys:
+                    self.maybe_hz_counter("workload.keys." + k, keys, k)
+
+            if "bytes" in workload:
+                bytes = workload["bytes"]
+                for k in bytes:
+                    self.maybe_hz_counter("workload.bytes." + k, bytes, k)
+
         if "latency_probe" in cluster:
             for k, v in cluster["latency_probe"].items():
                 self.gauge("latency_probe." + k, v)
 
+        if "clients" in cluster:
+            clients = cluster["clients"]
+
+            if "supported_versions" in clients:
+                for supported_version in clients["supported_versions"]:
+                    if "count" in supported_version and "client_version" in supported_version:
+                        client_version_tag = ["fdb_client_version:" + supported_version["client_version"]]
+                        self.gauge("clients.connected", supported_version["count"], client_version_tag)
+            else:
+                self.maybe_gauge("clients.connected", clients, "count")
+
         degraded_processes = 0
         if "degraded_processes" in cluster:
             self.gauge("degraded_processes", cluster["degraded_processes"])
@@ -254,6 +304,33 @@ def check_metrics(self, status):
         else:
             self.service_check("can_connect", AgentCheck.OK)
 
+        if "fault_tolerance" in cluster:
+            fault_tolerance = cluster["fault_tolerance"]
+
+            self.maybe_gauge(
+                "fault_tolerance.max_zone_failures_without_losing_availability",
+                fault_tolerance,
+                "max_zone_failures_without_losing_availability",
+            )
+
+            self.maybe_gauge(
+                "fault_tolerance.max_zone_failures_without_losing_data",
+                fault_tolerance,
+                "max_zone_failures_without_losing_data",
+            )
+
+        self.maybe_gauge("maintenance_seconds_remaining", cluster, "maintenance_seconds_remaining")
+        self.maybe_gauge("cluster_generation", cluster, "generation")
+
+        if "qos" in cluster:
+            qos = cluster["qos"]
+
+            self.maybe_gauge("qos.transactions_per_second_limit", qos, "transactions_per_second_limit")
+            self.maybe_gauge("qos.batch_transactions_per_second_limit", qos, "batch_transactions_per_second_limit")
+            self.maybe_gauge("qos.released_transactions_per_second", qos, "released_transactions_per_second")
+            self.maybe_gauge("qos.worst_queue_bytes_storage_server", qos, "worst_queue_bytes_storage_server")
+            self.maybe_gauge("qos.worst_queue_bytes_log_server", qos, "worst_queue_bytes_log_server")
+
     def maybe_gauge(self, metric, obj, key, tags=None):
         if key in obj:
             self.gauge(metric, obj[key], tags=tags)
diff --git a/foundationdb/metadata.csv b/foundationdb/metadata.csv
index 15b073ae809ed..63a8693125392 100644
--- a/foundationdb/metadata.csv
+++ b/foundationdb/metadata.csv
@@ -1,13 +1,22 @@
 metric_name,metric_type,interval,unit_name,per_unit_name,description,orientation,integration,short_name,curated_metric
+foundationdb.clients.connected,gauge,,connection,,Number of connected clients tagged by client version,0,foundationdb,connected clients,
+foundationdb.cluster_generation,gauge,,,,,0,foundationdb,cluster generation,
+foundationdb.coordinators,gauge,,instance,,Number of coordinators tagged by reachability,0,foundationdb,coordinators,
+foundationdb.data.average_partition_size_bytes,gauge,,byte,,,0,foundationdb,average partition size,
 foundationdb.data.least_operating_space_bytes_log_server,gauge,,byte,,,0,foundationdb,,
 foundationdb.data.moving_data.in_flight_bytes,gauge,,byte,,,0,foundationdb,in flight bytes,
 foundationdb.data.moving_data.in_queue_bytes,gauge,,byte,,,0,foundationdb,in queue bytes,
 foundationdb.data.moving_data.total_written_bytes,gauge,,byte,,,0,foundationdb,total written bytes,
+foundationdb.data.partitions_count,gauge,,,,,0,foundationdb,number of database partitions,
 foundationdb.data.system_kv_size_bytes,gauge,,byte,,,0,foundationdb,system kv size,
 foundationdb.data.total_disk_used_bytes,gauge,,byte,,,0,foundationdb,disk used,
 foundationdb.data.total_kv_size_bytes,gauge,,byte,,,0,foundationdb,total kv size,
 foundationdb.datacenter_lag.seconds,gauge,,second,,,0,foundationdb,lag,
 foundationdb.degraded_processes,gauge,,process,,,0,foundationdb,degraded processes,
+foundationdb.excluded_machines,gauge,,host,,,0,foundationdb,excluded machines,
+foundationdb.excluded_processes,gauge,,process,,,0,foundationdb,excluded processes,
+foundationdb.fault_tolerance.max_zone_failures_without_losing_availability,gauge,,location,,,0,foundationdb,fault tolerance domains that can fail before loss of cluster availability,
+foundationdb.fault_tolerance.max_zone_failures_without_losing_data,gauge,,location,,,0,foundationdb,fault tolerance domains that can fail before data fault tolerance domains that can fail before loss of cluster availability,
 foundationdb.instances,count,,instance,,,0,foundationdb,instances,
 foundationdb.latency_probe.batch_priority_transaction_start_seconds,gauge,,second,,Batch priority transaction start seconds,0,foundationdb,probe batch transaction start,
 foundationdb.latency_probe.commit_seconds,gauge,,second,,,0,foundationdb,probe commit seconds,
@@ -15,7 +24,9 @@ foundationdb.latency_probe.immediate_priority_transaction_start_seconds,gauge,,s
 foundationdb.latency_probe.read_seconds,gauge,,second,,,0,foundationdb,probe read seconds,
 foundationdb.latency_probe.transaction_start_seconds,gauge,,second,,,0,foundationdb,probe transaction start,
 foundationdb.machines,gauge,,host,,,0,foundationdb,machines,
+foundationdb.maintenance_seconds_remaining,gauge,,second,,,0,foundationdb,maintenance seconds remaining,
 foundationdb.process.cpu.usage_cores,gauge,,core,,,0,foundationdb,cores usage,
+foundationdb.process.disk.busy,gauge,,fraction,,,0,foundationdb,disk busy fraction,
 foundationdb.process.disk.free_bytes,gauge,,byte,,,0,foundationdb,free bytes,
 foundationdb.process.disk.reads.hz,gauge,,read,,,0,foundationdb,disk reads,
 foundationdb.process.disk.total_bytes,gauge,,byte,,,0,foundationdb,disk total bytes,
@@ -70,6 +81,8 @@ foundationdb.process.role.mutation_bytes.hz,gauge,,byte,,,0,foundationdb,mutatio
 foundationdb.process.role.mutations.counter,count,,item,,,0,foundationdb,mutations,
 foundationdb.process.role.mutations.hz,gauge,,item,,,0,foundationdb,mutations,
 foundationdb.process.role.query_queue_max,gauge,,query,,,0,foundationdb,query queue max,
+foundationdb.process.role.queue_disk_available_bytes,gauge,,byte,,,0,foundationdb,space available for log queue,
+foundationdb.process.role.queue_disk_total_bytes,gauge,,byte,,,0,foundationdb,log queue total size,
 foundationdb.process.role.queue_length,gauge,,item,,,0,foundationdb,queue length,
 foundationdb.process.role.read_latency_statistics.count,count,,millisecond,,,0,foundationdb,read latency,
 foundationdb.process.role.read_latency_statistics.max,gauge,,millisecond,,,0,foundationdb,read latency,
@@ -90,6 +103,17 @@ foundationdb.processes_per_role.proxy,gauge,,process,,,0,foundationdb,processes
 foundationdb.processes_per_role.ratekeeper,gauge,,process,,,0,foundationdb,processes per ratekeeper,
 foundationdb.processes_per_role.resolver,gauge,,process,,,0,foundationdb,processes per resolver,
 foundationdb.processes_per_role.storage,gauge,,process,,,0,foundationdb,processes per storage,
+foundationdb.qos.batch_transactions_per_second_limit,gauge,,transaction,second,,0,foundationdb,batch-priority transaction rate limit,
+foundationdb.qos.released_transactions_per_second,gauge,,transaction,second,,0,foundationdb,released transaction rate,
+foundationdb.qos.transactions_per_second_limit,gauge,,transaction,second,,0,foundationdb,transaction rate limit,
+foundationdb.qos.worst_queue_bytes_log_server,gauge,,byte,,,0,foundationdb,largest write queue on a log server,
+foundationdb.qos.worst_queue_bytes_storage_server,gauge,,byte,,,0,foundationdb,largest write queue on a storage server,
+foundationdb.workload.bytes.read.counter,count,,byte,,,0,foundationdb,bytes read,
+foundationdb.workload.bytes.read.hz,gauge,,byte,,,0,foundationdb,bytes read,
+foundationdb.workload.bytes.written.counter,count,,byte,,,0,foundationdb,bytes written,
+foundationdb.workload.bytes.written.hz,gauge,,byte,,,0,foundationdb,bytes written,
+foundationdb.workload.keys.read.counter,count,,key,,,0,foundationdb,keys read,
+foundationdb.workload.keys.read.hz,gauge,,key,,,0,foundationdb,keys read,
 foundationdb.workload.operations.location_requests.counter,count,,operation,,,0,foundationdb,location requests,
 foundationdb.workload.operations.location_requests.hz,gauge,,operation,,,0,foundationdb,location requests,
 foundationdb.workload.operations.low_priority_reads.counter,count,,operation,,,0,foundationdb,low priority reads,
diff --git a/foundationdb/tests/common.py b/foundationdb/tests/common.py
index 471bf078342ad..cc039e75e8a20 100644
--- a/foundationdb/tests/common.py
+++ b/foundationdb/tests/common.py
@@ -63,6 +63,13 @@
 }
 
 METRICS = [
+    "foundationdb.clients.connected",
+    "foundationdb.cluster_generation",
+    "foundationdb.coordinators",
+    "foundationdb.excluded_machines",
+    "foundationdb.excluded_processes",
+    "foundationdb.fault_tolerance.max_zone_failures_without_losing_availability",
+    "foundationdb.fault_tolerance.max_zone_failures_without_losing_data",
     "foundationdb.latency_probe.batch_priority_transaction_start_seconds",
     "foundationdb.latency_probe.commit_seconds",
     "foundationdb.latency_probe.immediate_priority_transaction_start_seconds",
@@ -71,6 +78,12 @@
     "foundationdb.machines",
     "foundationdb.processes",
     "foundationdb.degraded_processes",
+    "foundationdb.workload.bytes.read.counter",
+    "foundationdb.workload.bytes.read.hz",
+    "foundationdb.workload.bytes.written.counter",
+    "foundationdb.workload.bytes.written.hz",
+    "foundationdb.workload.keys.read.counter",
+    "foundationdb.workload.keys.read.hz",
     "foundationdb.workload.transactions.committed.counter",
     "foundationdb.workload.transactions.committed.hz",
     "foundationdb.workload.transactions.conflicted.counter",
@@ -97,6 +110,7 @@
     "foundationdb.workload.operations.reads.hz",
     "foundationdb.workload.operations.writes.counter",
     "foundationdb.workload.operations.writes.hz",
+    "foundationdb.data.average_partition_size_bytes",
     "foundationdb.data.system_kv_size_bytes",
     "foundationdb.data.total_disk_used_bytes",
     "foundationdb.data.total_kv_size_bytes",
@@ -104,6 +118,7 @@
     "foundationdb.data.moving_data.in_flight_bytes",
     "foundationdb.data.moving_data.in_queue_bytes",
     "foundationdb.data.moving_data.total_written_bytes",
+    "foundationdb.data.partitions_count",
     "foundationdb.datacenter_lag.seconds",
     "foundationdb.instances",
     "foundationdb.process.role.input_bytes.hz",
@@ -126,6 +141,8 @@
     "foundationdb.process.role.mutations.counter",
     "foundationdb.process.role.stored_bytes",
     "foundationdb.process.role.query_queue_max",
+    "foundationdb.process.role.queue_disk_available_bytes",
+    "foundationdb.process.role.queue_disk_total_bytes",
     "foundationdb.process.role.local_rate",
     "foundationdb.process.role.kvstore_available_bytes",
     "foundationdb.process.role.kvstore_free_bytes",
@@ -135,6 +152,7 @@
     "foundationdb.process.role.kvstore_total_size",
     "foundationdb.process.role.kvstore_used_bytes",
     "foundationdb.process.cpu.usage_cores",
+    "foundationdb.process.disk.busy",
     "foundationdb.process.disk.free_bytes",
     "foundationdb.process.disk.reads.hz",
     "foundationdb.process.disk.total_bytes",
@@ -180,4 +198,9 @@
     "foundationdb.processes_per_role.ratekeeper",
     "foundationdb.processes_per_role.resolver",
     "foundationdb.processes_per_role.storage",
+    "foundationdb.qos.batch_transactions_per_second_limit",
+    "foundationdb.qos.released_transactions_per_second",
+    "foundationdb.qos.transactions_per_second_limit",
+    "foundationdb.qos.worst_queue_bytes_log_server",
+    "foundationdb.qos.worst_queue_bytes_storage_server",
 ]


Reviews:
user: login: steveny91
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-14T19:26:10Z
user: login: jon-signal
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-14T19:51:48Z
user: login: jon-signal
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T15:10:07Z
user: login: steveny91
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T19:39:18Z
user: login: steveny91
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T19:40:20Z

Review comments:
diff_hunk: @@ -182,20 +185,52 @@ def check_metrics(self, status):
         if "cluster" not in status:
             raise ValueError("JSON Status data doesn't include cluster data")
 
+        if "client" in status:
+            client = status["client"]
+
+            if "coordinators" in client:
+                if "coordinators" in client["coordinators"]:
+                    coordinators = client["coordinators"]["coordinators"]
+
+                    reachable_coordinators = sum([1 for coordinator in coordinators if coordinator["reachable"]])
+                    unreachable_coordinators = len(coordinators) - reachable_coordinators
+
+                    self.gauge("coordinators", reachable_coordinators, ["reachable:true"])
+                    self.gauge("coordinators", unreachable_coordinators, ["reachable:false"])
+
         cluster = status["cluster"]
         if "machines" in cluster:
-            self.gauge("machines", len(cluster["machines"]))
-        if "processes" in cluster:
-            self.gauge("processes", len(cluster["processes"]))
+            included_machines = 0
+            excluded_machines = 0
+
+            for machine_key in cluster["machines"]:
+                machine = cluster["machines"][machine_key]
+
+                if machine["excluded"]:
+                    excluded_machines += 1
+                else:
+                    included_machines += 1
 
+            self.gauge("machines", included_machines, ["excluded:false"])
+            self.gauge("machines", excluded_machines, ["excluded:true"])
path: foundationdb/datadog_checks/foundationdb/check.py
user: login: steveny91
body: I'm a bit worried about how these metrics are now changed. So essentially the original metric, was sending the total amount of machines. But now we're splitting these up and submitting them with different tag context. Here's what I believe will happen:

Assuming we have a cluster of 4 machines and 3 included 1 excluded.

Original behavior was that it would submit a gauge value of 4 and that will show up on graph as 4 since it's a single payload, there's no aggregation. 

With this change, we will have 2 distinct payload (excluded:true|false) with values of 3 and 1, and average is the default aggregation, this will change  the value of the same query to show a value of 2 ((3+1)/2) instead of the original 4. So if a user of this integration would upgrade the check to this version, there might be a change in their monitor or dashboard that could be a false alarm. To get back to the value of 4, we would have to change the default avg by aggregation to a sum by aggregation.

This is also applied to the processes metrics. 

I would suggest if possible to create a new metric to track this instead so that we don't run into the risk of triggering a monitor.
created_at: 2025-03-14T19:26:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -182,20 +185,52 @@ def check_metrics(self, status):
         if "cluster" not in status:
             raise ValueError("JSON Status data doesn't include cluster data")
 
+        if "client" in status:
+            client = status["client"]
+
+            if "coordinators" in client:
+                if "coordinators" in client["coordinators"]:
+                    coordinators = client["coordinators"]["coordinators"]
+
+                    reachable_coordinators = sum([1 for coordinator in coordinators if coordinator["reachable"]])
+                    unreachable_coordinators = len(coordinators) - reachable_coordinators
+
+                    self.gauge("coordinators", reachable_coordinators, ["reachable:true"])
+                    self.gauge("coordinators", unreachable_coordinators, ["reachable:false"])
+
         cluster = status["cluster"]
         if "machines" in cluster:
-            self.gauge("machines", len(cluster["machines"]))
-        if "processes" in cluster:
-            self.gauge("processes", len(cluster["processes"]))
+            included_machines = 0
+            excluded_machines = 0
+
+            for machine_key in cluster["machines"]:
+                machine = cluster["machines"][machine_key]
+
+                if machine["excluded"]:
+                    excluded_machines += 1
+                else:
+                    included_machines += 1
 
+            self.gauge("machines", included_machines, ["excluded:false"])
+            self.gauge("machines", excluded_machines, ["excluded:true"])
path: foundationdb/datadog_checks/foundationdb/check.py
user: login: jon-signal
body: Ah—point taken. I'm on board with that, but is that a thing we could address by adjusting the dashboard/monitors included in this integration? I think the answer there is "no" because users may have created their own monitors or dashboards, but figured I'd put it forward all the same.
created_at: 2025-03-14T19:51:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -182,20 +185,52 @@ def check_metrics(self, status):
         if "cluster" not in status:
             raise ValueError("JSON Status data doesn't include cluster data")
 
+        if "client" in status:
+            client = status["client"]
+
+            if "coordinators" in client:
+                if "coordinators" in client["coordinators"]:
+                    coordinators = client["coordinators"]["coordinators"]
+
+                    reachable_coordinators = sum([1 for coordinator in coordinators if coordinator["reachable"]])
+                    unreachable_coordinators = len(coordinators) - reachable_coordinators
+
+                    self.gauge("coordinators", reachable_coordinators, ["reachable:true"])
+                    self.gauge("coordinators", unreachable_coordinators, ["reachable:false"])
+
         cluster = status["cluster"]
         if "machines" in cluster:
-            self.gauge("machines", len(cluster["machines"]))
-        if "processes" in cluster:
-            self.gauge("processes", len(cluster["processes"]))
+            included_machines = 0
+            excluded_machines = 0
+
+            for machine_key in cluster["machines"]:
+                machine = cluster["machines"][machine_key]
+
+                if machine["excluded"]:
+                    excluded_machines += 1
+                else:
+                    included_machines += 1
 
+            self.gauge("machines", included_machines, ["excluded:false"])
+            self.gauge("machines", excluded_machines, ["excluded:true"])
path: foundationdb/datadog_checks/foundationdb/check.py
user: login: jon-signal
body: In any case, I've addressed this by removing the new tags and introducing new `excluded_processes`/`excluded_machines` gauges.
created_at: 2025-03-17T15:10:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -182,20 +185,52 @@ def check_metrics(self, status):
         if "cluster" not in status:
             raise ValueError("JSON Status data doesn't include cluster data")
 
+        if "client" in status:
+            client = status["client"]
+
+            if "coordinators" in client:
+                if "coordinators" in client["coordinators"]:
+                    coordinators = client["coordinators"]["coordinators"]
+
+                    reachable_coordinators = sum([1 for coordinator in coordinators if coordinator["reachable"]])
+                    unreachable_coordinators = len(coordinators) - reachable_coordinators
+
+                    self.gauge("coordinators", reachable_coordinators, ["reachable:true"])
+                    self.gauge("coordinators", unreachable_coordinators, ["reachable:false"])
+
         cluster = status["cluster"]
         if "machines" in cluster:
-            self.gauge("machines", len(cluster["machines"]))
-        if "processes" in cluster:
-            self.gauge("processes", len(cluster["processes"]))
+            included_machines = 0
+            excluded_machines = 0
+
+            for machine_key in cluster["machines"]:
+                machine = cluster["machines"][machine_key]
+
+                if machine["excluded"]:
+                    excluded_machines += 1
+                else:
+                    included_machines += 1
 
+            self.gauge("machines", included_machines, ["excluded:false"])
+            self.gauge("machines", excluded_machines, ["excluded:true"])
path: foundationdb/datadog_checks/foundationdb/check.py
user: login: steveny91
body: > I think the answer there is "no" because users may have created their own monitors or dashboards, but figured I'd put it forward all the same.

Yes, that is indeed the case. We can only edit the assets with Ootb stuff, but once a users clones, or creates their own then we can't alter it.
created_at: 2025-03-17T19:40:20Z
author_association: CONTRIBUTOR

PR comments:
user: login: jon-signal
created_at: 2025-02-21T17:04:36Z
body: Ah—and it looks like I have some learning to do about the tooling here. I'll address those failing checks as soon as I can!
author_association: CONTRIBUTOR
user: login: codecov[bot]
created_at: 2025-02-21T18:06:44Z
body: ## [Codecov](https://app.codecov.io/gh/DataDog/integrations-core/pull/19681?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) Report
Attention: Patch coverage is `74.50980%` with `13 lines` in your changes missing coverage. Please review.
> Project coverage is 88.71%. Comparing base [(`a19c5f3`)](https://app.codecov.io/gh/DataDog/integrations-core/commit/a19c5f3ae05a10543b12e937898d7e51cf204db2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) to head [(`72fdfcb`)](https://app.codecov.io/gh/DataDog/integrations-core/commit/72fdfcb8531c643313b4df46a175da4002cbfd54?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog).
> Report is 10 commits behind head on master.

<details><summary>Additional details and impacted files</summary>


| [Flag](https://app.codecov.io/gh/DataDog/integrations-core/pull/19681/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | Coverage Δ | |
|---|---|---|
| [activemq](https://app.codecov.io/gh/DataDog/integrations-core/pull/19681/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `?` | |
| [cassandra](https://app.codecov.io/gh/DataDog/integrations-core/pull/19681/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `?` | |
| [foundationdb](https://app.codecov.io/gh/DataDog/integrations-core/pull/19681/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `∅ <74.50%> (∅)` | |
| [hive](https://app.codecov.io/gh/DataDog/integrations-core/pull/19681/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `?` | |
| [hivemq](https://app.codecov.io/gh/DataDog/integrations-core/pull/19681/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `?` | |
| [hudi](https://app.codecov.io/gh/DataDog/integrations-core/pull/19681/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `?` | |
| [ignite](https://app.codecov.io/gh/DataDog/integrations-core/pull/19681/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `?` | |
| [jboss_wildfly](https://app.codecov.io/gh/DataDog/integrations-core/pull/19681/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `?` | |
| [kafka](https://app.codecov.io/gh/DataDog/integrations-core/pull/19681/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `?` | |
| [presto](https://app.codecov.io/gh/DataDog/integrations-core/pull/19681/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `?` | |
| [solr](https://app.codecov.io/gh/DataDog/integrations-core/pull/19681/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog) | `?` | |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=DataDog#carryforward-flags-in-the-pull-request-comment) to find out more.

</details>
<details><summary>🚀 New features to boost your workflow: </summary>

- ❄ [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- 📦 [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE
user: login: jon-signal
created_at: 2025-02-21T18:07:21Z
body: > I'll address those failing checks as soon as I can!

…and I think we're good!
author_association: CONTRIBUTOR
user: login: steveny91
created_at: 2025-03-17T19:41:42Z
body: @jon-signal Thanks for addressing and the contribution, this looks good to me. I'll merge this one a bit later today.
author_association: CONTRIBUTOR
user: login: jon-signal
created_at: 2025-03-17T19:44:17Z
body: Excellent. Thanks! Appreciate you working through this pile of FoundationDB pull requests!
author_association: CONTRIBUTOR