Repository: matrix-org/synapse

Issue:
Inbound federation lag metrics should not report if no messages sent/received.
The metrics introduced in https://github.com/matrix-org/synapse/pull/8430 are not clear when no inbound federation is being received from a server:

An inspection of a graph zoomed in shows this being hard to use in action

![lag](https://user-images.githubusercontent.com/1917473/98104948-1831b700-1e8f-11eb-9b16-a67f0831f9b4.png)

This however doesn't show up as clearly when looking at a wider scale, and it's not obvious there is an issue - it just looks like it's successfully having a very small amount of lag and the problem has gone away.

![lag2](https://user-images.githubusercontent.com/1917473/98105043-42837480-1e8f-11eb-9658-421e527c8c2e.png)

Not emitting a metric while there aren't enough messages to provide any data seems like a better option here - at the moment we are emitting the out of date metric only, so any averages / minimums etc will not work correctly.


Discussion:
the metrics are frozen from 18:00ish to 04:00; the top graph above is the zoomed in small peak at 18:00
I guess the solution here is probably to convert the metric to a timestamp rather than an age, and convert back to an age on the prometheus side.

Resolving patch:
diff --git a/synapse/federation/federation_server.py b/synapse/federation/federation_server.py
--- a/synapse/federation/federation_server.py
+++ b/synapse/federation/federation_server.py
@@ -90,10 +90,9 @@
     "Time taken to process an event",
 )
 
-
-last_pdu_age_metric = Gauge(
-    "synapse_federation_last_received_pdu_age",
-    "The age (in seconds) of the last PDU successfully received from the given domain",
+last_pdu_ts_metric = Gauge(
+    "synapse_federation_last_received_pdu_time",
+    "The timestamp of the last PDU which was successfully received from the given domain",
     labelnames=("server_name",),
 )
 
@@ -369,8 +368,7 @@ async def process_pdus_for_room(room_id: str):
         )
 
         if newest_pdu_ts and origin in self._federation_metrics_domains:
-            newest_pdu_age = self._clock.time_msec() - newest_pdu_ts
-            last_pdu_age_metric.labels(server_name=origin).set(newest_pdu_age / 1000)
+            last_pdu_ts_metric.labels(server_name=origin).set(newest_pdu_ts / 1000)
 
         return pdu_results
 
diff --git a/synapse/federation/sender/transaction_manager.py b/synapse/federation/sender/transaction_manager.py
--- a/synapse/federation/sender/transaction_manager.py
+++ b/synapse/federation/sender/transaction_manager.py
@@ -36,9 +36,9 @@
 
 logger = logging.getLogger(__name__)
 
-last_pdu_age_metric = Gauge(
-    "synapse_federation_last_sent_pdu_age",
-    "The age (in seconds) of the last PDU successfully sent to the given domain",
+last_pdu_ts_metric = Gauge(
+    "synapse_federation_last_sent_pdu_time",
+    "The timestamp of the last PDU which was successfully sent to the given domain",
     labelnames=("server_name",),
 )
 
@@ -187,9 +187,8 @@ def json_data_cb():
 
             if success and pdus and destination in self._federation_metrics_domains:
                 last_pdu = pdus[-1]
-                last_pdu_age = self.clock.time_msec() - last_pdu.origin_server_ts
-                last_pdu_age_metric.labels(server_name=destination).set(
-                    last_pdu_age / 1000
+                last_pdu_ts_metric.labels(server_name=destination).set(
+                    last_pdu.origin_server_ts / 1000
                 )
 
             set_tag(tags.ERROR, not success)


Test patch: