Package: pymemtrace
pymemtraceprovides tools for tracking and understanding Python memory usage at different levels, at different
granularities and with different runtime costs.Full documentation:https://pymemtrace.readthedocs.iopymemtrace ToolsThe tools provided bypymemtrace:processis a very lightweight way of logging the total memory usage at regular time intervals.
It can plot memory over time with plotting programs such asgnuplot.
Seesome process examplescPyMemTraceis a memory tracer written in C that can report total memory usage for every function call/return for
both C and Python sections.
See somecPyMemTrace examplesand atechnical note on cPyMemTrace.DTrace: Here are a number of D scripts that can trace the low levelmalloc()andfree()system calls and
report how much memory was allocated and by whom.
See someDTrace examplesand atechnical note on DTrace.trace_mallocis a convenience wrapper around the Python standard librarytracemallocmodule.
This can report Python memory usage by module and line compensating for the cost oftracemalloc.
This can take memory snapshots before and after code blocks and show the change on memory caused by that code.
See sometrace_malloc examplesdebug_malloc_statsis a wrapper around thesys._debugmallocstatsfunction that can take snapshots of
memory before and after code execution and report the significant differences of the Python small object allocator.
See somedebug_malloc_stats examplesTool CharacteristicsEach tool can be characterised by:Memory Granularity: In how much detail is a memory change is observed.
An example ofcoarsememory granularity is measuring theResident Set Sizewhich is normally in chunks of 4096 bytes.
An example offinememory granularity is recording everymalloc()andfree().Execution Granularity: In how much code detail is the memory change observed.
An example ofcoarseexecution granularity is measuring the memory usage every second.
An example offineexecution granularity is recording the memory usage every Python line.Memory Cost: How much extra memory the tool needs.Execution Cost: How much the execution time is increased.Clearly there are trade-offs between these depending on the problem you are trying to solve.Tool CharacteristicsToolMemory GranularityExecution GranularityMemory CostExecution CostprocessRSS (total Python and C memory).Regular time intervals.Near zero.Near zero.cPyMemTraceRSS (total Python and C memory).Per Python line, Python function and C function call.Near zero.x10 to x20.DTraceEverymalloc()andfree().Per function call and return.Minimal.x90 to x100.trace_mallocEvery Python object.Per Python line, per function call.Significant but compensated.x900 for small objects, x6 for large objects.debug_malloc_statsPython memory pool.Snapshots the CPython memory pool either side of a block of code.Minimal.x2000+ for small objects, x12 for large objects.Package MetadataLicencePython memory tracing.Free software: MIT licenseDocumentation:https://pymemtrace.readthedocs.io.Project:https://github.com/paulross/pymemtrace.CreditsPhil Smith (AHL) with whom a casual lunch time chat lead to the creation of an earlier, but quite different
implementation, ofcPyMemTracein pure Python.This package was created withCookiecutterand theaudreyr/cookiecutter-pypackageproject template.History0.1.5 (2023-06-21)Add support for Python versions 3.10, 3.11. Now supports Python versions 3.7, 3.8, 3.9, 3.10, 3.11.0.1.4 (2022-03-19)Fix Linux build.0.1.3 (2022-03-17)Fix some tests.0.1.2 (2022-03-17)Fix source distribution that had missing headers.0.1.1 (2020-11-17)Add cPyMemTrace the C level profiler.Add DTrace scripts for low level tracing.Add debug_malloc_stats the wrapper around sys._debugmallocstats.Add process from the TotalDepth project.Add redirect_stdout for debug_malloc_stats.Add trace_malloc, a wrapper around the tracemalloc module.Includes extensive documentation and performance measurement.First release on PyPI.0.1.0 (2017-12-04)Initial idea and implementation, never released.