Identify Variables High On Memory Consumption | by Ujjwal Dalmia | Towards Data Science

Sometimes, when executing Python scripts, we encounter memory errors. These errors are primarily due to some variables that have high memory consumption.
In this tutorial, we will focus on profiling Python codes to optimize memory consumption. Memory profiling is a process using which we can dissect our code and identify variables that lead to memory errors.
The Memory Profiler is a python package that evaluates each line of Python code written within a function and correspondingly checks the usage of internal memory.
We can either use pip or conda package managers to install this package. If using Anaconda distribution, I would recommend using conda install as it resolves dependencies and environment issues automatically.
#### Install Packagesconda install memory_profiler
Once installed, the profiler can be loaded within the iPython environment using the following line of code:
#### Loading Memory Profiler Package%load_ext memory_profiler
After loading, use the following syntax to memory profile any predefined function:
#### Memory Profiling a Function%mprun -f function_name_only Call_to_function_with_arguments
Note: To profile a function using a Memory Profiler, import it as a module which means, writing it in a separate .py file and then importing it into the main program like any other package.
For this tutorial, I have used the same function as available on Memory Profiler’s PyPi webpage. It creates two list variables with a large number of elements and then deletes one of them (refer to the function definition in the code below). We then save and import this function into the main program for profiling.
#### Function Definition (Saved as example.py file)def my_func():     a = [1] * (10 ** 6)     b = [2] * (2 * 10 ** 7)     del b     return a#### Loading the line profiler within ipython/jupyter environment%load_ext memory_profilerfrom example import my_func#### Profiling the function using line_profiler%mprun -f my_func my_func()#### Memory Profile OutputLine #    Mem usage    Increment   Line Contents================================================     1     50.8 MiB     50.8 MiB   def my_func():     2     58.4 MiB      7.6 MiB    a = [1] * (10 ** 6)     3    211.0 MiB    152.6 MiB    b = [2] * (2 * 10 ** 7)     4     58.4 MiB      0.0 MiB    del b     5     58.4 MiB      0.0 MiB    return a
From the output of the Memory Profiler, one can see that the output table has four columns:
Column 1 (Line #) — Line number of code
Column 2 (Mem Usage) — Total memory used by function till the line number
Column 3 (Increment) — Memory used by a specific line of the program
Column 4(Content) — Actual code content
Note: The memory consumption dropped significantly after dropping the variable “b” from the program.
From the table, it is clear that the creation of variable “b” has led to a sudden spike in memory usage. In real-time scenarios, once we are aware of such findings, we can always look at alternative ways to program our code such that the memory consumption stays within acceptable limits.
In this short tutorial, we learned about memory profile our Python codes. To learn about time profiling your Python codes, please refer to this tutorial.
I hope this tutorial of the Did You Know series was informative and resourceful.
We will try and bring more engaging topics in future tutorials. Till then:
HAPPY LEARNING ! ! ! !