Package: memory-analyzer
Python3 Memory Analyzer For Running ProcessesChasing down a memory leak? Don't want to add debugging code, or even stop your process?You've come to the right place.memory_analyzerRunning the memory analyzer does not require you to stop your process or add any
special debugging code or flags. Running the analysis will not (coughshould not) break
your process, although your process (and all of its threads!) will be paused while
the memory analyzer gathers information about the objects in memory.You will need to install objgraph and pympler in the target binary's library
path, in addition to the deps inrequirements*.txtfor running the frontend.LicenseThis source code is licensed under the MIT license. Please see the LICENSE in the root directory for more information.Things you can find out:How many of each object you have in memoryThe total size of all of these objectsThe forward references to the objectsThe backwards references to the objectsThe difference in size/number of objects over time (through snapshot comparisons)Example Output:How To RunFind the PID of your process:ps aux | grep <your process>Run the memory analyzer against it:memory_analyzer run $PIDThis will put you into an ncurses UI and make a binary output file located inmemory_analyzer_out/memory_analyzer_snapshot-{TIMESTAMP}If you are analyzing a process that was run as root, you need to run the analyzer as root as well.Running on a modern ptrace-limited systemModern versions of at least Ubuntu and Arch have a patch that disallows ptracing
of non-child processes by non-root users.You can disable this usingecho 0 | sudo tee /proc/sys/kernel/yama/ptrace_scopeor runmemory_analyzeras root.View the Output without Re-AnalyzingIf you exit out of the UI but want to view the snapshot data again without re-running the analysis, you can use the 'view' command.memory_analyzer view <snapshot output file>Analyze Multiple ProcessesYou can analyze multiple processes at once by simply providing a list of the PIDs separated by spaces.memory_analyzer run 1234 4567 890If--snapshotis used, it tries to pair up the listed PIDs with the PIDs in the snapshot file. If they do pair, a new page is created for each like PID comparing the old and the new version. If it can't find any PIDs that pair up it just compares the first new and first old object.If the references flags are used the references are found for all of the PIDs listed.Viewing Forwards and Backwards ReferencesYou can either view the top N number of objects (sorted by size of object), or you
can look for specific objects. The memory analyzer will generate PNG files displaying the graph, and will upload them for you to phabricator for easy viewing. If you don't want them uploaded you can use the—no-uploadflag.WARNING: Getting the references is a costly operation! The more references you grab
the longer your process will remain paused for.View the Top 2 Objectsmemory_analyzer run -s 2 $PIDView specific objectsTo view a specific object you must know it's fully qualified domain name. If you are unsure, I'd recommend running the analyzer first with no flags and identifying the names of the objects you are most interested in there.memory_analyzer run $PID -ss __main__.Foo -ss builtins.strThis will get the forwards and backwards references of the Foo and str objects.Comparing SnapshotsComparing snapshots can be useful to show you how objects grow over time. To create an initial snapshot, just run the analyzer normally:memory_analyzer run $PID --snapshot <previous snapshot file>By default your snapshot files will all be saved in memory_analyzer_out/.
The snapshot analysis will be located on the second page of the ncurses UI, so hit the arrow key to the right to scroll to the snapshot page once you are in the UI.Specify the executableThe memory analyzer launches GDB with the executable found insys.executable. This might not be the executable you want to use to analyze your binary. For example, you may need to use the debuginfo binary. You can specify the executable with the-eflag:memory_analyzer run $PID -e /usr/local/bin/python3.7-debuginfoCustom output fileIf you don't want the file to be the default name, pass in your own custom name.memory_analyzer run $PID -f test_output1Navigating the UINavigating the UI should be relatively intuitive. You can use the arrow keys or wasd to scroll up and down.Scroll Up: Up arrow key or 'w'Scroll Down: Down arrow key or 's'Switch to next page: Right arrow key or 'd'Switch to previous page: Left arrow key or 'a'Page Up: Page Up keyPage Down: Page Down KeyJump to Bottom of page: GJump to Top of page: HExit: 'q'Dealing with MultithreadingRight now the program does NOT run in non-stop mode, meaning that if you have multiple threads they will all be paused during analysis. This could be a problem for some services that require threads to be completing live in the background. It might be possible to support non-stop mode (where only one thread gets paused), but it will only work if the process is stopped and started again by the memory analyzer (this is a limitation of GDB). Please contact me (lisroach) if you are interested in this feature.Importantly, this means debugging multithreaded processes goesmuchslower than single threaded- because it takes the time to find and pause each individual thread. If your process has timeouts that could be triggered by a paused thread (e.g. something that talks over the network), use the memory_analyzer with caution.FAQNo data returnedTry running the process with the--debugflag enabled for more information. If there are no obvious errors occurring, your program is likely hung and cannot be analyzed.No symbol table is loaded.  Use the "file" command.This is an error from GDB itself, and it could mean a couple of things:You don't have the correct debuginfo installed, orYou are using the wrong Python executable.To fix the incorrect debuginfo install the debuginfo associated with the python runtime the analyzed process is using. For example:sudo yum install python3-debuginfoor
sudo apt-get install python3-dbgTo solve the wrong Python executable, figure out what executable you need (which python version the analyzed process is running with), and use the --exec flag to specify it:memory_analyzer run $PID -e /usr/local/bin/python3.6It is hangingTry running the memory_analyzer with the--debugflag. This is going to show you what GDB is currently doing.You should not see:$5 = <some number>orThese are problems. Please file a bug. The memory_analyzer is a new tool, so not all scenarios have been tested and you may have run into something new. Keep reading for more information on understanding the debug output.If you see:That means your process has a lot of threads, and it's probably taking GDB a long time to pause them all.After this you will see:That means GDB is still working on doing GDB things, loading symbols and what not. This is usually what takes the longest, and as of right now I am not sure how to fix it. If you were to rungdbdirectly and attach to your process, this would take just as long :(It is not until you see:That the analyzer itself is running. Generally this runs very fast and you should barely see these messages flicker by.If you don't see the above, but you do see:This usually means you are not analyzing the correct process. Perhaps your process is not Python (maybe you analyzed while buck is building)?Does it run on WindowsNo. If you are interested in having this available on Windows it should be possible to add this, letlisroachknow.Does it work with Python 2No, unfortunately this is Python 3 exclusive and is likely to stay that way.DevelopmentAny help is welcome and appreciated!