Repository: Parsl/parsl

Issue:
monitoring vs serialization extremely poor performance with many apps
**Describe the bug**
PR #2468 moves the monitoring remote wrapper into its own module - `parsl.monitoring.remote` - because serialization of that wrapper was bringing along with it the entirely of the module it was defined in. By placing the remote wrapper in its own module, the size of the module brought along can be reduced.

However, subsequently, PR #2493 introduces a cache that is stored as a global in the `parsl.monitoring.remote` module, to try to work around a lack of caching in `parsl.serialize` except at the top level - prior to PR #2493, a new callable would be defined for each invocation when used with monitoring, bringing along with it the task and try ID in a closure. PR #2493 instead tries to make a callable that is re-used between multiple invocations of the same app, with the task and try ID added as additional parameters: attempting to be equivalent to a closure but unwrapped in a way that the naive parsl.serialize callable serializer can see that. To do that, it introduces its own cache, so that any particular app will be wrapped only once.

Because this cache is in the `parsl.monitoring.remote` module, it is serialized as part of serializing a monitoring-wrapped app invocation.

In workflows with a small number of apps, this is a small-ish (but undesired) overhead. In workflows with many apps (in my case, ctrl_parsl_bps which makes a new app for every task), this cache becomes large and the cache easily grows to hundreds of kilobytes, repeatedly serialized on each invocation.

While debugging this, I discovered that this serialization of caching only happens when the monitoring wrapper has the functools.wraps decorator applied -https://github.com/Parsl/parsl/blob/4b8b0b88e2af79ae0e5ada65eeae4940767a8028/parsl/monitoring/remote.py#L47

Removing that wraps decorator makes this problem go away.

More nuanced, having the wraps decorator copy a smaller subset of attributes, rather than its default of functools.WRAPPER_ASSIGNMENTS, also makes the problem go away:

```
        @functools.wraps(f, assigned = ('__name__', '__qualname__', '__doc__', '__annotations__'))
```

This copies the same attributes except for the `__module__` attribute.

It isn't clear to me why copying that attribute onto the wrapped function causes the module to be serialized, when that module is probably also the `__module__` attribute of the underlying function.

**To Reproduce**
Observe the size of serialized messages using a configuration such as `parsl/tests/configs/htex_local_alternate.py`.

**Expected behavior**
This cache should not be transmitted.

This whole caching feature in parsl.monitoring.remote only exists to cope with the lack of nested-function-away caching inside `parsl.serialize.concretes.DillCallableSerializer` and so perhaps some more work can happen there to make nested functions get cached, at which point PR #2493 could be reverted.

**Environment**
my laptop, parsl master 4b8b0b88e2af79ae0e5ada65eeae4940767a8028



Discussion:
As part of trying to quantify a potential fix to this, I wrote a script that monkeypatches `parsl.serialize.serialize` and reports both time and byte usage.

```
import time
import parsl.serialize

cumulative_serializer_time = 0
cumulative_serializer_bytes = 0

def s(*args, **kwargs):
  global cumulative_serializer_time
  global cumulative_serializer_bytes
  start = time.time()
  b = old_serializer(*args, **kwargs)
  end = time.time()
  cumulative_serializer_time += end-start
  cumulative_serializer_bytes += len(b)
  return b

# patch both symbols...
old_serializer = parsl.serialize.serialize
parsl.serialize.serialize = s
parsl.serialize.facade.serialize = s

import parsl

# configuration axes:
from parsl.tests.configs.htex_local_alternate import fresh_config
decorate_many = True

parsl.load(fresh_config())

def g():
  return 7


if not decorate_many:
  @parsl.python_app
  def f():
    return 7

count = 1000

for b in (1,2):
  print(f"Batch {b}")
  for _ in range(0,count):
    if decorate_many:
      f = parsl.python_app(g)
    r = f().result()
    assert r == 7

  print(f"Cumulative serializer time: {cumulative_serializer_time}")
  print(f"Cumulative serializer bytes: {cumulative_serializer_bytes}")

  print(f"Serializer time per task: {cumulative_serializer_time / (b*count)}")
  print(f"Serializer bytes per task: {cumulative_serializer_bytes / (b*count)}")
```

I ran this against current `master`, 4b8b0b88e2af79ae0e5ada65eeae4940767a8028, in 4 configurations, on 2 axes:

Monitoring on or off?
i) parsl/tests/config/htex_local.py - has no monitoring
ii) parsl/tests/config/htex_local_alternate.py - has monitoring

Decoration style:
a) decorate once, invoke 2000 times
b) decorate each invocation, 2000 times

This latter decoration may sound surprising but it's relatively common in power user cases where the callable to be executed is dynamically generated - for example, ctrl_bps_parsl in LSST generates and decorates a `partial` for every invocation.

The results against master are here. The values in cells are serialized bytes/invocation and serialization time/invocation.

|                             | No monitoring | With monitoring |
|---|---|---|
| Decorate once  | 2063b, 2x10^-5s | 5665b, 2.6x10^-5s | 
| Decorate many | 2063b, 2x10^-3s | 732kb, 0.37s | 

The bottom right corner is a demonstration of this current issue #2848

This test is run on my laptop with a custom-build python 3.11.

Resolving patch:
diff --git a/parsl/monitoring/remote.py b/parsl/monitoring/remote.py
--- a/parsl/monitoring/remote.py
+++ b/parsl/monitoring/remote.py
@@ -14,9 +14,6 @@
 
 logger = logging.getLogger(__name__)
 
-monitoring_wrapper_cache: Dict
-monitoring_wrapper_cache = {}
-
 
 def monitor_wrapper(f: Any,           # per app
                     args: Sequence,   # per invocation
@@ -33,76 +30,63 @@ def monitor_wrapper(f: Any,           # per app
     """Wrap the Parsl app with a function that will call the monitor function and point it at the correct pid when the task begins.
     """
 
-    # this makes assumptions that when subsequently executed with the same
-    # cache key, then the relevant parameters will not have changed from the
-    # first invocation with that cache key (otherwise, the resulting cached
-    # closure will be incorrectly cached)
-    cache_key = (run_id, f, radio_mode)
-
-    if cache_key in monitoring_wrapper_cache:
-        wrapped = monitoring_wrapper_cache[cache_key]
-
-    else:
+    @wraps(f)
+    def wrapped(*args: List[Any], **kwargs: Dict[str, Any]) -> Any:
+        task_id = kwargs.pop('_parsl_monitoring_task_id')
+        try_id = kwargs.pop('_parsl_monitoring_try_id')
+        terminate_event = Event()
+        # Send first message to monitoring router
+        send_first_message(try_id,
+                           task_id,
+                           monitoring_hub_url,
+                           run_id,
+                           radio_mode,
+                           run_dir)
+
+        if monitor_resources:
+            # create the monitor process and start
+            pp = ForkProcess(target=monitor,
+                             args=(os.getpid(),
+                                   try_id,
+                                   task_id,
+                                   monitoring_hub_url,
+                                   run_id,
+                                   radio_mode,
+                                   logging_level,
+                                   sleep_dur,
+                                   run_dir,
+                                   terminate_event),
+                             daemon=True,
+                             name="Monitor-Wrapper-{}".format(task_id))
+            pp.start()
+            p = pp
+            #  TODO: awkwardness because ForkProcess is not directly a constructor
+            # and type-checking is expecting p to be optional and cannot
+            # narrow down the type of p in this block.
+
+        else:
+            p = None
 
-        @wraps(f)
-        def wrapped(*args: List[Any], **kwargs: Dict[str, Any]) -> Any:
-            task_id = kwargs.pop('_parsl_monitoring_task_id')
-            try_id = kwargs.pop('_parsl_monitoring_try_id')
-            terminate_event = Event()
-            # Send first message to monitoring router
-            send_first_message(try_id,
-                               task_id,
-                               monitoring_hub_url,
-                               run_id,
-                               radio_mode,
-                               run_dir)
-
-            if monitor_resources:
-                # create the monitor process and start
-                pp = ForkProcess(target=monitor,
-                                 args=(os.getpid(),
-                                       try_id,
-                                       task_id,
-                                       monitoring_hub_url,
-                                       run_id,
-                                       radio_mode,
-                                       logging_level,
-                                       sleep_dur,
-                                       run_dir,
-                                       terminate_event),
-                                 daemon=True,
-                                 name="Monitor-Wrapper-{}".format(task_id))
-                pp.start()
-                p = pp
-                #  TODO: awkwardness because ForkProcess is not directly a constructor
-                # and type-checking is expecting p to be optional and cannot
-                # narrow down the type of p in this block.
-
-            else:
-                p = None
-
-            try:
-                return f(*args, **kwargs)
-            finally:
-                # There's a chance of zombification if the workers are killed by some signals (?)
-                if p:
-                    terminate_event.set()
-                    p.join(30)  # 30 second delay for this -- this timeout will be hit in the case of an unusually long end-of-loop
-                    if p.exitcode is None:
-                        logger.warn("Event-based termination of monitoring helper took too long. Using process-based termination.")
-                        p.terminate()
-                        # DANGER: this can corrupt shared queues according to docs.
-                        # So, better that the above termination event worked.
-                        # This is why this log message is a warning
-                        p.join()
-
-                send_last_message(try_id,
-                                  task_id,
-                                  monitoring_hub_url,
-                                  run_id,
-                                  radio_mode, run_dir)
-
-        monitoring_wrapper_cache[cache_key] = wrapped
+        try:
+            return f(*args, **kwargs)
+        finally:
+            # There's a chance of zombification if the workers are killed by some signals (?)
+            if p:
+                terminate_event.set()
+                p.join(30)  # 30 second delay for this -- this timeout will be hit in the case of an unusually long end-of-loop
+                if p.exitcode is None:
+                    logger.warn("Event-based termination of monitoring helper took too long. Using process-based termination.")
+                    p.terminate()
+                    # DANGER: this can corrupt shared queues according to docs.
+                    # So, better that the above termination event worked.
+                    # This is why this log message is a warning
+                    p.join()
+
+            send_last_message(try_id,
+                              task_id,
+                              monitoring_hub_url,
+                              run_id,
+                              radio_mode, run_dir)
 
     new_kwargs = kwargs.copy()
     new_kwargs['_parsl_monitoring_task_id'] = x_task_id


Test patch: