Repository: pulp/pulpcore

Issue:
Reduce memory usage of the pipeline
Author: @bmbouter (bmbouter)


Redmine Issue: 9635, https://pulp.plan.io/issues/9635

---

## Motivation

It would be nice if users could specify a desired maximum amount of RAM to be used during sync. For example, a user can say I only want 1500 MB of RAM to be used max.

## What is already in place

The stages pipeline restricts memory usage by only allowing 1000 declarative content objects between each stage (so for 8-9 stages that's 8000-9000 declarative content objects. This happens [here](https://github.com/pulp/pulpcore/blob/main/pulpcore/plugin/stages/api.py#L217).

Interestingly the docstring says this defaults to 100, but it seems to actually be 1000!

Also the stages perform batching, so they will only taking in a limited number of items (the batch size). That happens [with minsize](https://github.com/pulp/pulpcore/blob/main/pulpcore/plugin/stages/api.py#L84).

## Why this isn't enough

These are count-based mechnisms and don't correspond to actual MB or GB of memory used. Some content units vary a lot in how much memory each DeclarativeContent objects take up.

Another lesser problem is that it doesn't help plugin writers restrict their usage of memory in FirstStage.

## Idea

Add a new param called `max_mb` to base Remote, which defaults to None. If specified, the user will be specifying the desired maximum MB used by process syncing.

Have the queues between the stages, and the bather implementation, both check the total memory the current process is using and asyncio.sleep() polling until it goes down. This should keep the maximum amount used by all objects roughly to that number.

## Details

Introduce a new `MBSizeQueue` which is a wrapper around `asyncio.Queue` used today. It will have the same `put()` call, only wait if the amount of memory in use is greater than the remote is configured for.

Then introduce the same memory checking feature in the batcher. I'm not completely sure this second part is needed though.

We have to be very careful not to deadlock with this feature. For example, we have to account for the base case where even a single item is larger than the memory desired. Repos in pulp_rpm have had a single unit use more than 1.2G if I remember right, so if someone was syncing with 800 MB and we weren't careful to allow that unit to still flow through the pipeline we'd deadlock.....





Discussion:
From: @dralley (dalley)
Date: 2021-12-13T16:54:22Z

---

> Repos in pulp_rpm have had a single unit use more than 1.2G if I remember right, so if someone was syncing with 800 MB and we weren't careful to allow that unit to still flow through the pipeline we'd deadlock.....

This is true but:

1) the metadata is very messed up - 13 million duplicate "files" are listed for that package.
2) the postgresql maximum insert size is 1gb - so a single content unit exceeding that is a hard limitation regardless of anything else we do. Luckily I think that would be much much less frequent than an entire batch exceeding that limit, which I don't think we've ever seen happen either (but still a theoretical issue).
@bmbouter I finally have a PR for he pulp_rpm plugin that reduces the frontend memory consumption to ~50 megabytes.  Unfortunately the whole sync (for a large repo like RHEL7) is still using ~1.8gb

So this effort is necessary to make any further improvements.
@dralley great I'm hoping to PoC it this week. It was further down on my list of things to do, but now it's close to the top. I'll take as assigned.
[I implemented the version](https://github.com/pulp/pulpcore/compare/main...bmbouter:memory-reduction-PoC?expand=1) which would have a setting and if the memory in-use was over that setting it would disable batching on the queues and let the queue drain.

It did slow significantly limit memory usage, but the runtime slows down hugely. For example consider a pulp_rpm sync of EL7 with `on_demand=True` so no artifact downloading. Runtime-wise the regular one took  777 seconds, then 7671 seconds with the single-item pipeline. Here are two graphs of the memory usage. This was configured with `TASK_MEMORY_LIMIT = 128` which put it into effect almost immediately for all tests run.

Note the y-axis is MB of the process and the x-axis is from every call to put() after first-stage. So by the time the first call occurs first stage has already allocated a lot of memory.

# RPM Regular, No Memory Restriction
![rpm_no_memory_restriction](https://user-images.githubusercontent.com/180319/157891720-5bc2d44c-82b5-40a9-89ba-dff3506d0b57.png)


# RPM With Memory Restriction
![rpm_with_memory_restriction](https://user-images.githubusercontent.com/180319/157891753-3f28f291-e3f5-448f-be16-54271af4f8db.png)


Here are some results for pulp_file which also had a roughly 10x slowdown.

# pulp_file Regular, No Memory Restriction
![pulp_file_scalene_no_memory_restriction](https://user-images.githubusercontent.com/180319/157892049-2c238f93-7e49-45db-85b6-d696a3011e08.png)


# pulp_file With Memory Restriction
![pulp_file_scalene_with_memory_restriction](https://user-images.githubusercontent.com/180319/157892068-8bbf6db2-2098-454d-89e7-5eee67f848d7.png)


@bmbouter What about some intermediate value, like a limit of 1.2gb?
So I don't think we should merge or persue this experiment, mainly for two reasons.

1) It doesn't actually limit the memory to the desired amount and it's off by a lot. In all tests it was supposed to hold to 128 M, so if you had the OOM set to 140 in all cases your OOM would still occur.

2) It runs crazy crazy slow

Here are a few ideas though:

1) We should add a debugging feature to produce memory usage graphs for every task run and dump them into a directory. This way users and developers can accurate record how their memory performance is doing. An adaptation of this PR can add this feature.

2) The batch sizes are really what is important. In my case my memory restriction was reducing them to 0, and by default they are 1000, so what if we made this a configurable option on BaseRemote allowing users to specify a different default.

3) We should try to optimize first stage more using cpu and memory profilers, but this will have to be a plugin-by-plugin effort.
> @bmbouter What about some intermediate value, like a limit of 1.2gb?

It takes so long to run (like hours) with memory restriction I don't know what effective memory it would be using. What do you think about my next-steps proposals [here](https://github.com/pulp/pulpcore/issues/2069#issuecomment-1065199607)?
> It doesn't actually limit the memory to the desired amount and it's off by a lot. In all tests it was supposed to hold to 128 M, so if you had the OOM set to 140 in all cases your OOM would still occur.

That's expected with the current frontend, all the work to reduce the amount of memory used by the rpm plugin frontend isn't merged yet.  Probably 128mb would still be too low, but it could probably stay under 400mb.

> It runs crazy crazy slow

It sounds like you're disabling batching entirely though, what if we didn't go quite that far?  Dynamically sized batches I mean.  But perhaps that's not very easy with the current architecture.

Looking at the PR, all of the queues are swapped with memory restricting ones, when it seems like really the only one that needs to be done that way is the first stage.

>We should add a debugging feature to produce memory usage graphs for every task run and dump them into a directory. This way users and developers can accurate record how their memory performance is doing. An adaptation of this PR can add this feature.

How much overhead is there for the memory tracking alone?  Is that using the pympler method or just a general "how much memory is this process using" graph?

>The batch sizes are really what is important. In my case my memory restriction was reducing them to 0, and by default they are 1000, so what if we made this a configurable option on BaseRemote allowing users to specify a different default.

I'm uncomfortable making an implementation detail part of our public API like that.  I could agree with putting it in `settings.py`

Also could you write up how you were using scalene as a knowledge sharing exercise?  It's really useful to be able to reference things like that months from now :)
In the PR, only one queue is swapped with the memory restricting one, the first one just after first stage. It does disable batching of the entire pipeline though. I've kind of determined that if we memory throttle we need to also disable batching because throttling and using batching to wait for additional put() calls would likely deadlock. I think what this experiment does show is that changing the batch size does influence the memory usage numbers significantly, which is good.

I'm hesitant to try to dynamically change the batch size because very likely the algorithm is going to continue to reduce it because memory seems to still grow, just more slowly, and then you're in the super-slow no-batching situation again. I'd rather the user (or administrator) tell us and for all tasks or this specific task use this value and otherwise default to what it was doing before 1000.

The memory tracking was just periodic calls to the resource library, which I think just reads the values from the kernel. I ended up not even using pympler.

The scalene stuff was pretty easy, in my dev env I `pip install scalene` and the `pclean`. I `pstart` all the processes, but then I shutdown the workers with `sudo systemctl stop pulpcore-worker@1 pulpcore-worker@2`. I then use the script below as the scalene entry point by calling it with:  `/usr/local/lib/pulp/bin/scalene --html --outfile profiling_output.html scalene_pulpcore_worker.py`.

### scalene_pulpcore_worker.py
```
import logging
import os

import django

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "pulpcore.app.settings")

django.setup()

from pulpcore.tasking.pulpcore_worker import NewPulpWorker  # noqa: E402: module level not at top


_logger = logging.getLogger(__name__)

NewPulpWorker().run_forever()
```

Then run a task, but you'll see tracebacks after the tasking code completes, it's the RuntimeError [here](https://github.com/pulp/pulpcore/blob/e3421edb42d669d2f0cd91403803892f37710dd9/pulpcore/app/models/task.py#L217). Also you'll see scalene not provide any actual tasking code in its analysis. It does profile the startup of the worker nicely though.
@pulp/core FYI ^
Here's the story I have on recording the memory:  https://github.com/pulp/pulpcore/issues/2329
Here's the story I wrote on making batch_size a BaseRemote param https://github.com/pulp/pulpcore/issues/2332
Maybe we're looking at this wrong.

Right now the abridged version of what we do is:

* parse the full metadata, put it in the pipeline
* try to find any that already exist, swap the content models
* save the ones whose internal state says they aren't saved yet

But the entire pipeline basically holds these fat, completely populated models objects from beginning to end.

What if we merge the `QueryExistingContents` and `ContentSaver` stages, promoted them as far upwards in the pipeline as we can and then only hold on to the absolute minimum possible amount of data possible after those stages are finished.  So amount of memory would roughly scale with the contents of the first one or two queues, rather than the number of stages.  Then the whole pipeline could stay full.
The pipeline also handles downloading, creating of remote artifacts, resolving content futures, and the searching+reading of `Artifact` objects from the db too.

It does allow a lot of objects to be loaded into the pipeline, with 1000 in each queue, then maybe 500 as the batch iterator (holding objects within each stage) that's like 6K objects in the pipeline that I've seen.

Overall reducing the number of items will reduce the memory footprint, but also it will slow down the sync with more RTTs to the database and possibly less parallel downloading. I think ultimately the batcher is in real control of that (500) items each, so what we could consider doing is reducing the queue size from 1000 to 500 because there isn't performance gained from allowing more items in the queues between the stages than each stage could handle in one batch. This would reduce the number of items by like 40% (quick guess). That would be pareto efficient because runtime should be unaffected, yet the memory amount reduced.

Maybe we could benchmark a pre-post change after I finish my memory recording PR that needs revision.

Resolving patch:
diff --git a/pulpcore/plugin/stages/api.py b/pulpcore/plugin/stages/api.py
--- a/pulpcore/plugin/stages/api.py
+++ b/pulpcore/plugin/stages/api.py
@@ -179,12 +179,12 @@ def __str__(self):
         return "[{id}] {name}".format(id=id(self), name=self.__class__.__name__)
 
 
-async def create_pipeline(stages, maxsize=1000):
+async def create_pipeline(stages, maxsize=1):
     """
     A coroutine that builds a Stages API linear pipeline from the list `stages` and runs it.
 
     Each stage is an instance of a class derived from :class:`pulpcore.plugin.stages.Stage` that
-    implements the :meth:`run` coroutine. This coroutine reads asyncromously either from the
+    implements the :meth:`run` coroutine. This coroutine reads asynchronously either from the
     `items()` iterator or the `batches()` iterator and outputs the items with `put()`. Here is an
     example of the simplest stage that only passes data::
 
@@ -196,7 +196,7 @@ async def run(self):
     Args:
         stages (list of coroutines): A list of Stages API compatible coroutines.
         maxsize (int): The maximum amount of items a queue between two stages should hold. Optional
-            and defaults to 100.
+            and defaults to 1.
 
     Returns:
         A single coroutine that can be used to run, wait, or cancel the entire pipeline with.


Test patch: