Repository: matrix-org/synapse

Issue:
per-block cpu and db usage metrics are no longer reported accurately
this is a regression introduced by #7120.


Discussion:
The symptoms of this bug are basically this:

![image](https://user-images.githubusercontent.com/1389908/81014436-c2834000-8e54-11ea-978e-1d62fe08d304.png)

(no, we didn't magically make those processes 3 times as efficient. But we did deploy develop on 1st May.)

Those graphs chart the CPU usage used by `with Measure()` blocks, which are implemented via LoggingContexts. I think the basic problem is that, as of [this change](https://github.com/matrix-org/synapse/pull/7120/files#diff-984b645db421f3b1c753de557bed4ea5R380-R381), the CPU usage stats are moved to the parent LoggingContext each time we switch away from the LoggingContext for the Measure block (ie, when it starts a blocking operation such as a database transaction); previously this happened only when the Measure block was all done and dusted [here](https://github.com/matrix-org/synapse/pull/7120/files#diff-984b645db421f3b1c753de557bed4ea5L349) - ie, after Measure had added the CPU usage to its prometheus stats.

The problem is that I made that change for quite a good reason - which was that I was annoyed that LoggingContexts which did most of their work inside a child LoggingContext didn't report any CPU usage until the child finished (when it would suddenly report 5000% CPU usage).
Basically, I think we need to get rid of that `self._resource_usage.reset()`, and instead just add the delta to the parent context. But if we do that, we need to make sure that `add_database_transaction` and co are propagated correctly.
I'm going to see if I can figure this out.

Resolving patch:
diff --git a/synapse/logging/context.py b/synapse/logging/context.py
--- a/synapse/logging/context.py
+++ b/synapse/logging/context.py
@@ -431,15 +431,7 @@ def stop(self, rusage: "Optional[resource._RUsage]") -> None:
                 return
 
             utime_delta, stime_delta = self._get_cputime(rusage)
-            self._resource_usage.ru_utime += utime_delta
-            self._resource_usage.ru_stime += stime_delta
-
-            # if we have a parent, pass our CPU usage stats on
-            if self.parent_context:
-                self.parent_context._resource_usage += self._resource_usage
-
-                # reset them in case we get entered again
-                self._resource_usage.reset()
+            self.add_cputime(utime_delta, stime_delta)
         finally:
             self.usage_start = None
 
@@ -497,30 +489,52 @@ def _get_cputime(self, current: "resource._RUsage") -> Tuple[float, float]:
 
         return utime_delta, stime_delta
 
+    def add_cputime(self, utime_delta: float, stime_delta: float) -> None:
+        """Update the CPU time usage of this context (and any parents, recursively).
+
+        Args:
+            utime_delta: additional user time, in seconds, spent in this context.
+            stime_delta: additional system time, in seconds, spent in this context.
+        """
+        self._resource_usage.ru_utime += utime_delta
+        self._resource_usage.ru_stime += stime_delta
+        if self.parent_context:
+            self.parent_context.add_cputime(utime_delta, stime_delta)
+
     def add_database_transaction(self, duration_sec: float) -> None:
+        """Record the use of a database transaction and the length of time it took.
+
+        Args:
+            duration_sec: The number of seconds the database transaction took.
+        """
         if duration_sec < 0:
             raise ValueError("DB txn time can only be non-negative")
         self._resource_usage.db_txn_count += 1
         self._resource_usage.db_txn_duration_sec += duration_sec
+        if self.parent_context:
+            self.parent_context.add_database_transaction(duration_sec)
 
     def add_database_scheduled(self, sched_sec: float) -> None:
         """Record a use of the database pool
 
         Args:
-            sched_sec (float): number of seconds it took us to get a
-                connection
+            sched_sec: number of seconds it took us to get a connection
         """
         if sched_sec < 0:
             raise ValueError("DB scheduling time can only be non-negative")
         self._resource_usage.db_sched_duration_sec += sched_sec
+        if self.parent_context:
+            self.parent_context.add_database_scheduled(sched_sec)
 
     def record_event_fetch(self, event_count: int) -> None:
         """Record a number of events being fetched from the db
 
         Args:
-            event_count (int): number of events being fetched
+            event_count: number of events being fetched
         """
         self._resource_usage.evt_db_fetch_count += event_count
+        if self.parent_context:
+            self.parent_context.record_event_fetch(event_count)
 
 
 class LoggingContextFilter(logging.Filter):


Test patch: