Repository: dbt-labs/dbt-core
Pull request: Add more accurate RSS high water mark measurement for Linux
resolves #10120

### Problem

Memory measurement on Linux reflected the maximum RSS level of not just the current process, but all parent processes.

### Solution

Measure max RSS on Linux via a platform-specific mechanism that does not have this defect.

### Checklist

- [x] I have read [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md) and understand what's expected of me  
- [x] I have run this code in development and it appears to resolve the stated issue  
- [x] This PR includes tests, or tests are not required/relevant for this PR
- [x] This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX
- [x] This PR includes [type annotations](https://docs.python.org/3/library/typing.html) for new and modified functions


Problem statement:
Make Memory Usage Reporting More Reliable
At present, dbt uses python's resource.getrusage() function to record the max RSS upon termination. This is not reliable in all of our use cases. Unfortunately this value persists across fork and exec calls on linux, which means that the process which kicks off dbt may set a memory floor that dbt never reaches, obscuring its actual memory usage.

To improve the situation, we can use the /proc/self/status file which should be present on Linux systems, extracting the value of the VmHWM line from that file, which reprsents the high water mark (HWM) of the resident set size. When this file is missing we should fall back to the current implementation.

Hints from issue discussion:

Commit message: Add changelog entry.

Patch to review:
diff --git a/.changes/unreleased/Under the Hood-20240519-155946.yaml b/.changes/unreleased/Under the Hood-20240519-155946.yaml
new file mode 100644
index 00000000000..920c7ff860d
--- /dev/null
+++ b/.changes/unreleased/Under the Hood-20240519-155946.yaml	
@@ -0,0 +1,6 @@
+kind: Under the Hood
+body: Make RSS high water mark measurement more accurate on Linux
+time: 2024-05-19T15:59:46.700842315-04:00
+custom:
+    Author: peterallenwebb
+    Issue: "10177"
diff --git a/core/dbt/cli/requires.py b/core/dbt/cli/requires.py
index 0104515d601..855196faecf 100644
--- a/core/dbt/cli/requires.py
+++ b/core/dbt/cli/requires.py
@@ -32,6 +32,7 @@
 from dbt.plugins import set_up_plugin_manager
 from dbt.profiler import profiler
 from dbt.tracking import active_user, initialize_from_flags, track_run
+from dbt.utils import try_get_max_rss_kb
 from dbt.version import installed as installed_version
 from dbt_common.clients.system import get_env
 from dbt_common.context import get_invocation_context, set_invocation_context
@@ -148,6 +149,9 @@ def wrapper(*args, **kwargs):
                 import resource
 
                 rusage = resource.getrusage(resource.RUSAGE_SELF)
+                max_rss = try_get_max_rss_kb()
+                if max_rss is None:
+                    max_rss = rusage.ru_maxrss
                 fire_event(
                     ResourceReport(
                         command_name=ctx.command.name,
@@ -155,7 +159,7 @@ def wrapper(*args, **kwargs):
                         command_wall_clock_time=time.perf_counter() - start_func,
                         process_user_time=rusage.ru_utime,
                         process_kernel_time=rusage.ru_stime,
-                        process_mem_max_rss=rusage.ru_maxrss,
+                        process_mem_max_rss=max_rss,
                         process_in_blocks=rusage.ru_inblock,
                         process_out_blocks=rusage.ru_oublock,
                     ),
diff --git a/core/dbt/utils.py b/core/dbt/utils.py
index 065d9649479..d8599e67e25 100644
--- a/core/dbt/utils.py
+++ b/core/dbt/utils.py
@@ -386,3 +386,21 @@ def strtobool(val: str) -> bool:
         return False
     else:
         raise ValueError("invalid truth value %r" % (val,))
+
+
+def try_get_max_rss_kb() -> Optional[int]:
+    """Attempts to get the high water mark for this process's memory use via
+    the most reliable and accurate mechanism available through the host OS.
+    Currently only implemented for Linux."""
+    try:
+        # On Linux, the most reliable documented mechanism for getting the RSS
+        # high-water-mark comes from the line confusingly labeled VmHWM in the
+        # /proc/self/status virtual file.
+        with open("/proc/self/status") as f:
+            for line in f:
+                if line.startswith("VmHWM:"):
+                    return int(str.split(line)[1])
+    except Exception:
+        pass
+
+    return None


Code review comments:
diff_hunk: @@ -386,3 +386,21 @@ def strtobool(val: str) -> bool:
         return False
     else:
         raise ValueError("invalid truth value %r" % (val,))
+
+
+def try_get_max_rss_kb() -> Optional[int]:
+    """Attempts to get the high water mark for this process's memory use via
+    the most reliable and accurate mechanism available through the host OS.
+    Currently only implemented for Linux."""
+    try:
path: core/dbt/utils.py
text: @user1:
Maybe check for `sys.platform == 'linux'` in the first line to avoid the `try/catch`?

diff_hunk: @@ -148,14 +149,17 @@ def wrapper(*args, **kwargs):
                 import resource
 
                 rusage = resource.getrusage(resource.RUSAGE_SELF)
+                max_rss = try_get_max_rss_kb()
path: core/dbt/cli/requires.py
text: @user1:
Nit: `max_rss = try_get_max_rss_kb() or rusage.ru_maxrss`