Repository: bloomberg/memray
Pull request #556 (2024-03-02 19:40:48)

Add a warning to promote --aggregated if files are too big
Users tend to run memray over long running programs but they don't tend
to know about the --aggregated option, while most of the cases they
don't need the extra features that runnin with the default mode has.

*Issue number of the reported bug or feature request: #<number>*

**Describe your changes**
A clear and concise description of the changes you have made.

**Testing performed**
Describe the testing you have performed to ensure that the bug has been addressed, or that the new feature works as planned.

**Additional context**
Add any other context about your contribution here.


codecov-commenter: ## [Codecov](https://app.codecov.io/gh/bloomberg/memray/pull/556?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bloomberg) Report
Attention: Patch coverage is `92.30769%` with `1 lines` in your changes are missing coverage. Please review.
> Project coverage is 92.91%. Comparing base [(`41248ed`)](https://app.codecov.io/gh/bloomberg/memray/commit/41248eded6f4a94936b824299b5679f2ffa18c6e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bloomberg) to head [(`3556b3b`)](https://app.codecov.io/gh/bloomberg/memray/pull/556?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bloomberg).
> Report is 24 commits behind head on main.

| [Files](https://app.codecov.io/gh/bloomberg/memray/pull/556?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bloomberg) | Patch % | Lines |
|---|---|---|
| [src/memray/commands/common.py](https://app.codecov.io/gh/bloomberg/memray/pull/556?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bloomberg#diff-c3JjL21lbXJheS9jb21tYW5kcy9jb21tb24ucHk=) | 83.33% | [1 Missing :warning: ](https://app.codecov.io/gh/bloomberg/memray/pull/556?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bloomberg) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #556      +/-   ##
==========================================
+ Coverage   92.55%   92.91%   +0.35%     
==========================================
  Files          91       91              
  Lines       11304    11171     -133     
  Branches     1581     2039     +458     
==========================================
- Hits        10462    10379      -83     
+ Misses        837      792      -45     
+ Partials        5        0       -5     
```

| [Flag](https://app.codecov.io/gh/bloomberg/memray/pull/556/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bloomberg) | Coverage Δ | |
|---|---|---|
| [cpp](https://app.codecov.io/gh/bloomberg/memray/pull/556/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bloomberg) | `92.91% <92.30%> (+6.96%)` | :arrow_up: |
| [python_and_cython](https://app.codecov.io/gh/bloomberg/memray/pull/556/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bloomberg) | `92.91% <92.30%> (-2.82%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bloomberg#carryforward-flags-in-the-pull-request-comment) to find out more.


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/bloomberg/memray/pull/556?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bloomberg).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bloomberg).


Patch:
diff --git a/src/memray/_memray.pyx b/src/memray/_memray.pyx
index 3205338..0423215 100644
--- a/src/memray/_memray.pyx
+++ b/src/memray/_memray.pyx
@@ -757,6 +757,7 @@ cdef _create_metadata(header, peak_memory):
         python_allocator=allocator_id_to_name[header["python_allocator"]],
         has_native_traces=header["native_traces"],
         trace_python_allocators=header["trace_python_allocators"],
+        file_format=FileFormat(header["file_format"]),
     )
 
 
diff --git a/src/memray/_metadata.py b/src/memray/_metadata.py
index 64cb2d7..a44caa9 100644
--- a/src/memray/_metadata.py
+++ b/src/memray/_metadata.py
@@ -1,6 +1,8 @@
 from dataclasses import dataclass
 from datetime import datetime
 
+from ._memray import FileFormat
+
 
 @dataclass
 class Metadata:
@@ -14,3 +16,4 @@ class Metadata:
     python_allocator: str
     has_native_traces: bool
     trace_python_allocators: bool
+    file_format: FileFormat
diff --git a/src/memray/commands/common.py b/src/memray/commands/common.py
index 4986097..c01d043 100644
--- a/src/memray/commands/common.py
+++ b/src/memray/commands/common.py
@@ -20,6 +20,7 @@ from memray import AllocationRecord
 from memray import FileReader
 from memray import MemorySnapshot
 from memray._errors import MemrayCommandError
+from memray._memray import FileFormat
 from memray._memray import SymbolicSupport
 from memray._memray import TemporalAllocationRecord
 from memray._memray import get_symbolic_support
@@ -77,6 +78,24 @@ def warn_if_not_enough_symbols() -> None:
         return
 
 
+def warn_if_file_is_not_aggregated_and_is_too_big(
+    reader: FileReader, result_path: Path
+) -> None:
+    FILE_SIZE_LIMIT = 10 * 1000 * 1000
+    if (
+        reader.metadata.file_format == FileFormat.ALL_ALLOCATIONS
+        and result_path.stat().st_size > FILE_SIZE_LIMIT
+    ):
+        pprint(
+            ":warning: [bold yellow] This capture file is large and may take a long"
+            " time to process [/] :warning:\n\n"
+            "Next time, consider using the `--aggregate` option to `memray run` to"
+            " reduce the size of the file.\n"
+            "Check https://bloomberg.github.io/memray/run.html#aggregated-capture-files"
+            " for more information.\n"
+        )
+
+
 class HighWatermarkCommand:
     def __init__(
         self,
@@ -140,6 +159,9 @@ class HighWatermarkCommand:
             if reader.metadata.has_native_traces:
                 warn_if_not_enough_symbols()
 
+            if not temporal and not temporary_allocation_threshold >= 0:
+                warn_if_file_is_not_aggregated_and_is_too_big(reader, result_path)
+
             if temporal:
                 assert self.temporal_reporter_factory is not None
                 if show_memory_leaks:
diff --git a/src/memray/commands/summary.py b/src/memray/commands/summary.py
index ae56326..1320758 100644
--- a/src/memray/commands/summary.py
+++ b/src/memray/commands/summary.py
@@ -5,6 +5,7 @@ from textwrap import dedent
 
 from memray import FileReader
 from memray._errors import MemrayCommandError
+from memray.commands.common import warn_if_file_is_not_aggregated_and_is_too_big
 from memray.commands.common import warn_if_not_enough_symbols
 from memray.reporters.summary import SummaryReporter
 
@@ -68,6 +69,9 @@ class SummaryCommand:
             if reader.metadata.has_native_traces:
                 warn_if_not_enough_symbols()
 
+            if not args.temporary_allocation_threshold >= 0:
+                warn_if_file_is_not_aggregated_and_is_too_big(reader, result_path)
+
             if args.temporary_allocation_threshold >= 0:
                 snapshot = iter(
                     reader.get_temporary_allocation_records(
diff --git a/src/memray/commands/tree.py b/src/memray/commands/tree.py
index 0a1d9e1..c678f73 100644
--- a/src/memray/commands/tree.py
+++ b/src/memray/commands/tree.py
@@ -5,6 +5,7 @@ from textwrap import dedent
 
 from memray import FileReader
 from memray._errors import MemrayCommandError
+from memray.commands.common import warn_if_file_is_not_aggregated_and_is_too_big
 from memray.commands.common import warn_if_not_enough_symbols
 from memray.reporters.tree import TreeReporter
 
@@ -56,6 +57,10 @@ class TreeCommand:
             reader = FileReader(os.fspath(args.results), report_progress=True)
             if reader.metadata.has_native_traces:
                 warn_if_not_enough_symbols()
+
+            if not args.temporary_allocation_threshold >= 0:
+                warn_if_file_is_not_aggregated_and_is_too_big(reader, result_path)
+
             if args.temporary_allocation_threshold >= 0:
                 snapshot = iter(
                     reader.get_temporary_allocation_records(


Test patch:
diff --git a/tests/unit/test_highwatermark_command.py b/tests/unit/test_highwatermark_command.py
index 527aef2..57ff404 100644
--- a/tests/unit/test_highwatermark_command.py
+++ b/tests/unit/test_highwatermark_command.py
@@ -1,5 +1,7 @@
 import os
+import sys
 from pathlib import Path
+from unittest.mock import ANY
 from unittest.mock import MagicMock
 from unittest.mock import Mock
 from unittest.mock import call
@@ -8,6 +10,7 @@ from unittest.mock import patch
 import pytest
 
 from memray._errors import MemrayCommandError
+from memray._memray import FileFormat
 from memray.commands.common import HighWatermarkCommand
 
 
@@ -165,6 +168,9 @@ class TestReportGeneration:
         calls = [
             call(os.fspath(result_path), report_progress=True),
             call().metadata.has_native_traces.__bool__(),
+            call().metadata.file_format.__eq__(FileFormat.ALL_ALLOCATIONS)
+            if sys.version_info >= (3, 8, 0)
+            else ANY,
             call().get_high_watermark_allocation_records(merge_threads=merge_threads),
             call().get_memory_snapshots(),
         ]
@@ -195,6 +201,9 @@ class TestReportGeneration:
         calls = [
             call(os.fspath(result_path), report_progress=True),
             call().metadata.has_native_traces.__bool__(),
+            call().metadata.file_format.__eq__(FileFormat.ALL_ALLOCATIONS)
+            if sys.version_info >= (3, 8, 0)
+            else ANY,
             call().get_leaked_allocation_records(merge_threads=merge_threads),
             call().get_memory_snapshots(),
         ]
diff --git a/tests/unit/test_stats_reporter.py b/tests/unit/test_stats_reporter.py
index be25e2b..e688360 100644
--- a/tests/unit/test_stats_reporter.py
+++ b/tests/unit/test_stats_reporter.py
@@ -9,6 +9,7 @@ from unittest.mock import patch
 import pytest
 
 from memray import AllocatorType as AT
+from memray._memray import FileFormat
 from memray._metadata import Metadata
 from memray._stats import Stats
 from memray.reporters.stats import StatsReporter
@@ -97,6 +98,7 @@ def fake_stats():
             python_allocator="pymalloc",
             has_native_traces=False,
             trace_python_allocators=True,
+            file_format=FileFormat.ALL_ALLOCATIONS,
         ),
         total_num_allocations=20,
         total_memory_allocated=sum(mem_allocation_list),
@@ -435,6 +437,7 @@ def test_stats_output_json(fake_stats, tmp_path):
             "python_allocator": "pymalloc",
             "has_native_traces": False,
             "trace_python_allocators": True,
+            "file_format": 0,
         },
     }
     actual = json.loads(output_file.read_text())