Repository: syncthing/syncthing (language: go)

Issue:
Device connection state metric
### Feature description

A new metric that exposes the connection state of each device as a gauge value (e.g., 0 = disconnected; 1 = connected, ...).

### Problem or use case

I have an android device where the operating system frequently kills the syncthing app.
I want to monitor, using prometheus, whether the device is connected with my central syncthing node and get alerted if it is disconnected for an extended period of time.

### Alternatives or workarounds

Create a custom exporter that uses the syncthing API.

---

If this is something you would be interested into having, I can try to open a PR for it.
Device connection state metric
### Feature description

A new metric that exposes the connection state of each device as a gauge value (e.g., 0 = disconnected; 1 = connected, ...).

### Problem or use case

I have an android device where the operating system frequently kills the syncthing app.
I want to monitor, using prometheus, whether the device is connected with my central syncthing node and get alerted if it is disconnected for an extended period of time.

### Alternatives or workarounds

Create a custom exporter that uses the syncthing API.

---

If this is something you would be interested into having, I can try to open a PR for it.


Discussion / hints:
Sounds reasonable. I'd probably make it a gauge on the number of connections, so zero is disconnected and then positive numbers for connected devices. 
> Sounds reasonable. I'd probably make it a gauge on the number of connections, so zero is disconnected and then positive numbers for connected devices.

Oh, then I probably didn't phrase it good enough :sweat_smile: 

My plan was to do it specific to the device ID, e.g.,
```text
# specific device connected:
syncthing_protocol_connection_state{device="DEVICE-A"} 1
# specific device not connected:
syncthing_protocol_connection_state{device="DEVICE-B"} 0
```
I agree, but a single device can have an arbitrary number of active connections nowadays. So, what you're proposing, but `1` can also become `3` or `28`.

```
syncthing_protocol_connections_active{device="DEVICE-A"} 28 # connected, large number of concurrent data connections
syncthing_protocol_connections_active{device="DEVICE-b"} 0 # disconnected
```

> I agree, but a single device can have an arbitrary number of active connections nowadays. So, what you're proposing, but `1` can also become `3` or `28`.
> 
> ```
> syncthing_protocol_connections_active{device="DEVICE-A"} 28 # connected, large number of concurrent data connections
> syncthing_protocol_connections_active{device="DEVICE-b"} 0 # disconnected
> ```

Got it, thanks for clarifying.

I'll try to implement it and open a PR :slightly_smiling_face: 
After looking around a bit I found `connections.accountAddedConnection` & `connections.accountRemovedConnection` as functions where I can add the `Inc` & `Dec` calls. The `connections` package currently doesn't have its own `metrics.go`, so I would add one.

I however don't know if this is the *best* location. I've also found `protocol.NewConnection` & `protocol.internalClose` and `model.AddConnection` & `model.Closed`.
It would be great if you could point me to the location you would consider to be suitable.
I think the accounting functions you found are a good place. There's probably some map or slice there that has the number of connections you can get the length of, or just pair the inc/dec as you say. 
Sounds reasonable. I'd probably make it a gauge on the number of connections, so zero is disconnected and then positive numbers for connected devices. 
> Sounds reasonable. I'd probably make it a gauge on the number of connections, so zero is disconnected and then positive numbers for connected devices.

Oh, then I probably didn't phrase it good enough :sweat_smile: 

My plan was to do it specific to the device ID, e.g.,
```text
# specific device connected:
syncthing_protocol_connection_state{device="DEVICE-A"} 1
# specific device not connected:
syncthing_protocol_connection_state{device="DEVICE-B"} 0
```
I agree, but a single device can have an arbitrary number of active connections nowadays. So, what you're proposing, but `1` can also become `3` or `28`.

```
syncthing_protocol_connections_active{device="DEVICE-A"} 28 # connected, large number of concurrent data connections
syncthing_protocol_connections_active{device="DEVICE-b"} 0 # disconnected
```

> I agree, but a single device can have an arbitrary number of active connections nowadays. So, what you're proposing, but `1` can also become `3` or `28`.
> 
> ```
> syncthing_protocol_connections_active{device="DEVICE-A"} 28 # connected, large number of concurrent data connections
> syncthing_protocol_connections_active{device="DEVICE-b"} 0 # disconnected
> ```

Got it, thanks for clarifying.

I'll try to implement it and open a PR :slightly_smiling_face: 
After looking around a bit I found `connections.accountAddedConnection` & `connections.accountRemovedConnection` as functions where I can add the `Inc` & `Dec` calls. The `connections` package currently doesn't have its own `metrics.go`, so I would add one.

I however don't know if this is the *best* location. I've also found `protocol.NewConnection` & `protocol.internalClose` and `model.AddConnection` & `model.Closed`.
It would be great if you could point me to the location you would consider to be suitable.
I think the accounting functions you found are a good place. There's probably some map or slice there that has the number of connections you can get the length of, or just pair the inc/dec as you say. 

Fix patch:
diff --git a/lib/connections/metrics.go b/lib/connections/metrics.go
new file mode 100644
index 00000000000..e002833915d
--- /dev/null
+++ b/lib/connections/metrics.go
@@ -0,0 +1,27 @@
+// Copyright (C) 2024 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.
+
+package connections
+
+import (
+	"github.com/prometheus/client_golang/prometheus"
+	"github.com/prometheus/client_golang/prometheus/promauto"
+)
+
+var (
+	metricDeviceActiveConnections = promauto.NewGaugeVec(prometheus.GaugeOpts{
+		Namespace: "syncthing",
+		Subsystem: "connections",
+		Name:      "active",
+		Help:      "Number of currently active connections, per device. If value is 0, the device is disconnected.",
+	}, []string{"device"})
+)
+
+func registerDeviceMetrics(deviceID string) {
+	// Register metrics for this device, so that counters & gauges are present even
+	// when zero.
+	metricDeviceActiveConnections.WithLabelValues(deviceID)
+}
diff --git a/lib/connections/service.go b/lib/connections/service.go
index 359dcff4d54..5b69f2f99a6 100644
--- a/lib/connections/service.go
+++ b/lib/connections/service.go
@@ -846,6 +846,7 @@ func (s *service) CommitConfiguration(from, to config.Configuration) bool {
 	newDevices := make(map[protocol.DeviceID]bool, len(to.Devices))
 	for _, dev := range to.Devices {
 		newDevices[dev.DeviceID] = true
+		registerDeviceMetrics(dev.DeviceID.String())
 	}
 
 	for _, dev := range from.Devices {
@@ -853,6 +854,7 @@ func (s *service) CommitConfiguration(from, to config.Configuration) bool {
 			warningLimitersMut.Lock()
 			delete(warningLimiters, dev.DeviceID)
 			warningLimitersMut.Unlock()
+			metricDeviceActiveConnections.DeleteLabelValues(dev.DeviceID.String())
 		}
 	}
 
@@ -1378,6 +1380,9 @@ func (c *deviceConnectionTracker) accountAddedConnection(conn protocol.Connectio
 	c.wantConnections[d] = int(h.NumConnections)
 	l.Debugf("Added connection for %s (now %d), they want %d connections", d.Short(), len(c.connections[d]), h.NumConnections)
 
+	// Update active connections metric
+	metricDeviceActiveConnections.WithLabelValues(d.String()).Inc()
+
 	// Close any connections we no longer want to retain.
 	c.closeWorsePriorityConnectionsLocked(d, conn.Priority()-upgradeThreshold)
 }
@@ -1399,6 +1404,10 @@ func (c *deviceConnectionTracker) accountRemovedConnection(conn protocol.Connect
 		delete(c.connections, d)
 		delete(c.wantConnections, d)
 	}
+
+	// Update active connections metric
+	metricDeviceActiveConnections.WithLabelValues(d.String()).Dec()
+
 	l.Debugf("Removed connection for %s (now %d)", d.Short(), c.connections[d])
 }
 
diff --git a/lib/protocol/metrics.go b/lib/protocol/metrics.go
index 1f57e51fd98..d1873b7d16b 100644
--- a/lib/protocol/metrics.go
+++ b/lib/protocol/metrics.go
@@ -58,5 +58,6 @@ func registerDeviceMetrics(deviceID string) {
 	metricDeviceSentUncompressedBytes.WithLabelValues(deviceID)
 	metricDeviceSentMessages.WithLabelValues(deviceID)
 	metricDeviceRecvBytes.WithLabelValues(deviceID)
+	metricDeviceRecvDecompressedBytes.WithLabelValues(deviceID)
 	metricDeviceRecvMessages.WithLabelValues(deviceID)
 }


Test patch: