Repository: python/mypy

Issue:
Increase in memory requirements for compiling mypy with MYPY_USE_MYPYC=1
This is technically not a bug/crash reported or a feature request. This is more of an FYI for package maintainers and an invitation to discuss the potential issue.

To give you some context, I'm one of the maintainers of the [mypy feedstock](https://github.com/conda-forge/mypy-feedstock) at conda-forge. During the `py310` build migration for `mypy`, our CI build container kept getting killed/OOMed on `linux-aarch64`. On investigation, I found that the memory requirements to build the `__native_<>.c module` has increased (significantly?) for `py310`.

```bash
/usr/bin/time -v gcc -pthread -B $CONDA_PREFIX/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -fPIC -O2 -isystem $CONDA_PREFIX/include -fPIC -O2 -isystem $CONDA_PREFIX/include -fPIC -I/path/to/mypy/mypyc/lib-rt -Ibuild -I$CONDA_PREFIX/include/python3.10 -c build/__native_a64f60bb4641b67cb20c.c -o build/temp.linux-x86_64-3.10/build/__native_a64f60bb4641b67cb20c.o -O3 -Werror -Wno-unused-function -Wno-unused-label -Wno-unreachable-code -Wno-unused-variable -Wno-unused-command-line-argument -Wno-unknown-warning-option -Wno-unused-but-set-variable
build/__native_a64f60bb4641b67cb20c.c: In function ‘CPyDef_fixup___TypeFixer___visit_instance’:
build/__native_a64f60bb4641b67cb20c.c:570654: note: ‘-Wmisleading-indentation’ is disabled from this point onwards, since column-tracking was disabled due to the size of the code/headers
570654 | CPyL13: ;
       | 
	Command being timed: "gcc -pthread -B $CONDA_PREFIX/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -fPIC -O2 -isystem $CONDA_PREFIX/include -fPIC -O2 -isystem $CONDA_PREFIX/include -fPIC -I/path/to/mypy/mypyc/lib-rt -Ibuild -I$CONDA_PREFIX/include/python3.10 -c build/__native_a64f60bb4641b67cb20c.c -o build/temp.linux-x86_64-3.10/build/__native_a64f60bb4641b67cb20c.o -O3 -Werror -Wno-unused-function -Wno-unused-label -Wno-unreachable-code -Wno-unused-variable -Wno-unused-command-line-argument -Wno-unknown-warning-option -Wno-unused-but-set-variable"
	User time (seconds): 624.63
	System time (seconds): 5.31
	Percent of CPU this job got: 99%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 10:30.21
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): **5037436** <--- !!!!
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 20
	Minor (reclaiming a frame) page faults: 2640454
	Voluntary context switches: 191
	Involuntary context switches: 1676
	Swaps: 0
	File system inputs: 2792
	File system outputs: 1818944
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
```

To compare, with `py38`, the requirements were:
```bash
/usr/bin/time -v gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/path/to/mypy/mypyc/lib-rt -Ibuild -I$CONDA_PREFIX/include/python3.8 -c build/__native_a64f60bb4641b67cb20c.c -o build/temp.linux-x86_64-3.8/build/__native_a64f60bb4641b67cb20c.o -O3 -Werror -Wno-unused-function -Wno-unused-label -Wno-unreachable-code -Wno-unused-variable -Wno-unused-command-line-argument -Wno-unknown-warning-option -Wno-unused-but-set-variable
build/__native_a64f60bb4641b67cb20c.c: In function ‘CPyDef_fixup___TypeFixer___visit_none_type__TypeVisitor_glue’:
build/__native_a64f60bb4641b67cb20c.c:570413: note: ‘-Wmisleading-indentation’ is disabled from this point onwards, since column-tracking was disabled due to the size of the code/headers
570413 | CPyL1: ;
       | 
	Command being timed: "gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/path/to/mypy/mypyc/lib-rt -Ibuild -I$CONDA_PREFIX/include/python3.8 -c build/__native_a64f60bb4641b67cb20c.c -o build/temp.linux-x86_64-3.8/build/__native_a64f60bb4641b67cb20c.o -O3 -Werror -Wno-unused-function -Wno-unused-label -Wno-unreachable-code -Wno-unused-variable -Wno-unused-command-line-argument -Wno-unknown-warning-option -Wno-unused-but-set-variable"
	User time (seconds): 595.81
	System time (seconds): 5.07
	Percent of CPU this job got: 99%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 10:01.20
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): **4817776** <--- !!!!
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 2593657
	Voluntary context switches: 153
	Involuntary context switches: 3692
	Swaps: 0
	File system inputs: 24
	File system outputs: 1723584
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
```
By [decreasing](https://github.com/conda-forge/mypy-feedstock/blob/master/recipe/patches/0001-Reduce-amount-of-debug-info-injected-in-extension-mo.patch) the amount of debug info being injected using `-g1`, I was able reduce the amount of maximum `RSS` by `~1GB` for `py310` and `~700MB` for `py38` and then CI was happy. Particularly for release builds, one may wish to simply use `-g0` as well.

It might be interesting to see why we see these increases with cpython version bumps. I would :heart: to get some insight by the mypy devs :slightly_smiling_face:


Discussion:
One reason for the difference might be switching from C macros to inline functions in recent Python versions (just a guess though). Perhaps we should use `-g1` by default. It may also produce smaller binaries.

Relevant gcc documentation: https://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html

Resolving patch:
diff --git a/mypyc/__main__.py b/mypyc/__main__.py
--- a/mypyc/__main__.py
+++ b/mypyc/__main__.py
@@ -22,7 +22,7 @@
 from mypyc.build import mypycify
 
 setup(name='mypyc_output',
-      ext_modules=mypycify({}, opt_level="{}"),
+      ext_modules=mypycify({}, opt_level="{}", debug_level="{}"),
 )
 """
 
@@ -35,10 +35,11 @@ def main() -> None:
         pass
 
     opt_level = os.getenv("MYPYC_OPT_LEVEL", '3')
+    debug_level = os.getenv("MYPYC_DEBUG_LEVEL", '1')
 
     setup_file = os.path.join(build_dir, 'setup.py')
     with open(setup_file, 'w') as f:
-        f.write(setup_format.format(sys.argv[1:], opt_level))
+        f.write(setup_format.format(sys.argv[1:], opt_level, debug_level))
 
     # We don't use run_setup (like we do in the test suite) because it throws
     # away the error code from distutils, and we don't care about the slight
diff --git a/mypyc/build.py b/mypyc/build.py
--- a/mypyc/build.py
+++ b/mypyc/build.py
@@ -431,7 +431,8 @@ def mypycify(
     *,
     only_compile_paths: Optional[Iterable[str]] = None,
     verbose: bool = False,
-    opt_level: str = '3',
+    opt_level: str = "3",
+    debug_level: str = "1",
     strip_asserts: bool = False,
     multi_file: bool = False,
     separate: Union[bool, List[Tuple[List[str], Optional[str]]]] = False,
@@ -454,6 +455,7 @@ def mypycify(
         verbose: Should mypyc be more verbose. Defaults to false.
 
         opt_level: The optimization level, as a string. Defaults to '3' (meaning '-O3').
+        debug_level: The debug level, as a string. Defaults to '1' (meaning '-g1').
         strip_asserts: Should asserts be stripped from the generated code.
 
         multi_file: Should each Python module be compiled into its own C source file.
@@ -511,7 +513,9 @@ def mypycify(
     cflags: List[str] = []
     if compiler.compiler_type == 'unix':
         cflags += [
-            '-O{}'.format(opt_level), '-Werror', '-Wno-unused-function', '-Wno-unused-label',
+            '-O{}'.format(opt_level),
+            '-g{}'.format(debug_level),
+            '-Werror', '-Wno-unused-function', '-Wno-unused-label',
             '-Wno-unreachable-code', '-Wno-unused-variable',
             '-Wno-unused-command-line-argument', '-Wno-unknown-warning-option',
         ]
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -144,10 +144,12 @@ def run(self):
 
     from mypyc.build import mypycify
     opt_level = os.getenv('MYPYC_OPT_LEVEL', '3')
+    debug_level = os.getenv('MYPYC_DEBUG_LEVEL', '1')
     force_multifile = os.getenv('MYPYC_MULTI_FILE', '') == '1'
     ext_modules = mypycify(
         mypyc_targets + ['--config-file=mypy_bootstrap.ini'],
         opt_level=opt_level,
+        debug_level=debug_level,
         # Use multi-file compilation mode on windows because without it
         # our Appveyor builds run out of memory sometimes.
         multi_file=sys.platform == 'win32' or force_multifile,


Test patch:
diff --git a/mypyc/test/test_run.py b/mypyc/test/test_run.py
--- a/mypyc/test/test_run.py
+++ b/mypyc/test/test_run.py
@@ -242,6 +242,7 @@ def run_case_step(self, testcase: DataDrivenTestCase, incremental_step: int) ->
             check_serialization_roundtrip(ir)
 
         opt_level = int(os.environ.get('MYPYC_OPT_LEVEL', 0))
+        debug_level = int(os.environ.get('MYPYC_DEBUG_LEVEL', 0))
 
         setup_file = os.path.abspath(os.path.join(WORKDIR, 'setup.py'))
         # We pass the C file information to the build script via setup.py unfortunately
@@ -250,7 +251,8 @@ def run_case_step(self, testcase: DataDrivenTestCase, incremental_step: int) ->
                                         separate,
                                         cfiles,
                                         self.multi_file,
-                                        opt_level))
+                                        opt_level,
+                                        debug_level))
 
         if not run_setup(setup_file, ['build_ext', '--inplace']):
             if testcase.config.getoption('--mypyc-showc'):