Bug: Leaks when switching SERPTelemetry.sys.mjs uninit calls to use category manager mechanism when running browser_search_glean_serp_event_telemetry_categorization_enabled_by_pref.js
Status: RESOLVED
Priority: P2
Resolution: FIXED
Created: 2025-02-19 22:06:16+00:00
Resolved: 2025-03-04 09:41:48+00:00

This is pretty bemusing.

The patches in bug 1916424 move a bunch of init/uninit calls to happen via a generic mechanism using the category manager.

Before the patch, a loop like this:

```js
    for (let task of tasks) {
      try {
        task();
      } catch (ex) {
        console.error(`Error during quit-application-granted: ${ex}`);
        if (Cu.isInAutomation) {
          // This usually happens after the test harness is done collecting
          // test errors, thus we can't easily add a failure to it. The only
          // noticeable solution we have is crashing.
          Cc["@mozilla.org/xpcom/debug;1"]
            .getService(Ci.nsIDebug2)
            .abort(ex.filename, ex.lineNumber);
        }
      }
    }
```

runs tasks, and the relevant tasks look like this:

```js
      () => lazy.SearchSERPTelemetry.uninit(),
      () => lazy.SearchSERPCategorization.uninit(),
```

Post-patch, at the same time where we would normally run the loop over these tasks, we first call a bunch of consumers via the category manager. The loop for that looks like this:

```js
  callModulesFromCategory(
    {
      categoryName,
      profilerMarker = "",
      idleDispatch = false,
      failureHandler = null,
    },
    ...args
  ) {
    // Use an async function for profiler markers and error handling.
    // Note that we deliberately don't await at the top level, so we
    // can guarantee all consumers get run/queued.
    let callSingleListener = async fn => {
      let startTime = profilerMarker ? Cu.now() : 0;
      try {
        await fn(...args);
      } catch (ex) {
        console.error(
          `Error in processing ${categoryName} for ${fn._descriptiveName}`
        );
        console.error(ex);
        try {
          await failureHandler?.(ex);
        } catch (nestedEx) {
          console.error(`Error in handling failure: ${nestedEx}`);
          // Crash in automation:
          if (BrowserUtils._inAutomation) {
            Cc["@mozilla.org/xpcom/debug;1"]
              .getService(Ci.nsIDebug2)
              .abort(nestedEx.filename, nestedEx.lineNumber);
          }
        }
      }
      if (profilerMarker) {
        ChromeUtils.addProfilerMarker(
          profilerMarker,
          startTime,
          fn._descriptiveName
        );
      }
    };

    for (let listener of lazy.CatManListenerManager.getListeners(
      categoryName
    )) {
      if (idleDispatch) {
        ChromeUtils.idleDispatch(() => callSingleListener(listener));
      } else {
        callSingleListener(listener);
      }
    }
```

This is a bit more convoluted because it's more generic code that gets used in other places. But fundamentally it still ends up in an outer-non-async function that calls `callSingleListener` which eventually invokes and returns the result of calling the `uninit` method on `SearchSERPTelemetry` resp. `SearchSERPCategorization`. There is an `await` in there purely to ensure we log errors from async rejections/failures (but no errors appear in [failing logs](https://treeherder.mozilla.org/logviewer?job_id=495482744&repo=autoland&lineNumber=7701-7717), AFAICT, despite other logging appearing).

If I'm reading my own code right (lol) then I don't think this ultimately awaits on the inner task code (and neither did the original code), so there _shouldn't_ be any timing differences.

So I'm pretty lost as to why this is causing leaks.

Two things I did notice:

1. we do not appear to have a shutdown blocker for the async and sqlite related code in this module. That seems potentially problematic, and possibly fodder for a separate bug.
2. the test toggles the enabled state of this code, which runs a pref observer in the module, which will call init/uninit - but there is nothing outside the test guaranteeing the timing of those (async) calls. But then, the test does wait for some observer topics so I wouldn't have expected it to be an issue in this test, just something that is potentially fragile...

The one other random hypothesis I have is that somehow doing:

```
              if (!Object.hasOwn(this.cachedModules, module)) {
                this.cachedModules[module] = ChromeUtils.importESModule(module);
              }
```

(inside `lazy.CatManListenerManager.getListeners`) is not causing us to avoid importing the module twice (as both objects are from the same module) and that is somehow messing things up in a way that other modules' loads haven't. But I... can't see it right now? Plus `importESModule` is supposed to do the Right Thing anyway.

Both before and after the change, the uninit code logs in these test runs, so I don't _think_ there's any meaningful difference in if/when this runs?

I'm going to try to land the patches in bug 1916424 without these bits so I can unblock some other work, and then it'll also be easier to repro/check what is going on here, I guess. Andrew, sorry to bother you again with this, but I don't suppose with this narrowing down there's anything obvious that sticks out to you, or that you have a suggestion on where to start digging into this more? Your description of "You can also use CC/GC logs to brute force investigate this" sounded scary. 😅