Optimize Memory Tips in Python. Tracking, managing, and optimizing... | by Nhu Hoang | Towards Data Science

Memory management in Python is not a simple issue to solve, it requires a decent understanding of Python objects and data structures. Unlike in C/C++, users have no control over memory management. It is taken over by Python itself. However, with some insights about how Python works and memory support modules, we can somehow seed some light on how to control this issue.
There are several ways to get the size of an object in Python. You can use sys.getsizeof() to get the exact size of the object, objgraph.show_refs() to visualize the structure of an object or psutil.Process().memory_info().rss to get all memory is allocated at the moment.
tracemalloc is also another choice. It is included in the Python standard library and provides block-level traces of memory allocation, statistics for the overall memory behavior of a program.
The most used file is the arr object which takes up 2 memory blocks with a total size of 2637 MiB. Other objects are minimal.
Another important technique is to estimate how much memory is needed for the process to run. This can be guessed through monitoring the peak memory usage of the process. To measure peak memory, you can use the below code at the end of the process.
### For Linux (in KiB) and MacOS (in bytes)from resource import getrusage, RUSAGE_SELFprint(getrusage(RUSAGE_SELF).ru_maxrss)### For Windowsimport psutilprint(psutil.Process().memory_info().peak_wset)
Having the peak number and the amount of data put in the process, you can, by some means, judge the amount of memory to be consumed for your next process.
Training a large dataset is a bottleneck for your memory and you will never be able to train a complete model given the whole dataset never fits in your memory at the same time, especially for unstructured data such as image, text, voice,... However, with Pytorch DataLoader, you manage to set up various mini-batches for the whole dataset and each is loaded uninterruptedly into your model (the number of samples depends on your memory capability). You can see here the tutorial on using Pytorch DataLoader.
However, if you want to train a Machine Learning model on tabular data without using Deep learning (hence, not using Pytorch) or you don’t have access to a Database and have to work solely on the memory, what will be the choice for memory optimization?
By understanding how data is stored and manipulated and using the optimal data type for the tasks, it will save you huge space in memory and computation time. In Numpy, there are multiple types, including boolean (bool), integer (int), Unsigned integer (uint), float, complex, datetime64, timedelta64, object_, etc...
### Check numpy integer>>> import numpy as np>>> ii16 = np.iinfo(np.int16)>>> ii16iinfo(min=-32768, max=32767, dtype=int16)### Access min value>>> ii16.min-32768
I narrow them down to uint, int, and float given these are the most common when training models, handling data in Python. Depending on different needs and objectives, using sufficient data types becomes vital know-how. To check type minimum and maximum values, you can use function numpy.iinfo(), and numpy.finfo() for float.
Below is the summary information for each type.
The CSV file size doubles if the data type is converted to numpy.float64, which is the default type of numpy.array, compared to numpy.float32. Therefore, float32 is one of the optimal ones to use (Pytorch datatype is also float32).
As the default data type numpy.float() is float64 and numpy.int() is int64, remember to define the dtype when creating numpy array will save a huge amount of memory space.
When working with DataFrame, there will be another usual type which is “object”. Converting from object to category for feature having various repetitions will help computation time faster.
Below is an example function to optimize the pd.DataFrame data type for scalars and strings.
Another way to easily and efficiently reduce pd.DataFrame memory footprint is to import data with specific columns using usercols parameters in pd.read_csv()
Python is faster at retrieving a local variable than a global one. Moreover, declaring too many variables as global leads to Out of Memory issue as these remain in the memory till program execution is completed while local variables are deleted as soon as the function is over and release the memory space which it occupies. Read more at The real-life skill set that data scientists must master
towardsdatascience.com
Python yield returns a generator object, which converts the expression given into a generator function. To get the values of the object, it has to be iterated to read the values given to the yield. To read the generator’s values, you can use list(), for loop, or next().
>>> def say_hello():>>>    yield "HELLO!">>> SENTENCE = say_hello()>>> print(next(SENTENCE))HELLO!
However, generators are for one-time use objects. If you access it the second time, it returns empty.
>>> def say_hello():>>>    yield "HELLO!">>> SENTENCE = say_hello()>>> print(next(SENTENCE))HELLO!>>> print("calling the generator again: ", list(SENTENCE))calling the generator again: []
As there is no value returned unless the generator object is iterated, no memory is used when the Yield function is defined, while calling Return in a function leads to the allocation in memory.
Hence, Yield is suitable for large datasets, or when you don’t need to store all the output values but just one value for each iteration of the main function.
>>> import sys>>> my_generator_list = (i*2 for i in range(100000))>>> print(f"My generator is {sys.getsizeof(my_generator_list)} bytes")My generator is 128 bytes>>> timeit(my_generator_list)10000000 loops, best of 5: 32 ns per loop  >>> my_list = [i*2 for i in range(1000000)]>>> print(f"My list is {sys.getsizeof(my_list)} bytes")My list is 824472 bytes>>> timeit(my_list)10000000 loops, best of 5: 34.5 ns per loop
Looking at the above code, the list comprehension is 6441 times heavier than the generator and runs slower than the other generator.
Use Python Built-in Functions to improve code performance, list of functions.
Python class objects’ attributes are stored in the form of a dictionary. Thus, defining thousands of objects is the same as allocating thousands of dictionaries to the memory space. And adding __slots__ (which reduces the wastage of space and speeds up the program by allocating space for a fixed amount of attributes.)
Regarding memory usage, given there is no longer __dict__ in a class object, the memory space reduces noticeably from (64+16+120)=200 to 56 bytes.
As strings are immutable, every time you add an element to a string by the “+” operator, a new string will be allocated in memory space. The longer the string, the more memory consumed, the less efficient the code becomes. Using join() can improve speed >30% vs ‘+’ operator.
There are other methods to improve speed and save memory, check details here.
Or flatten a list with itertools.chain()
### Concatenate string using '+' operationdef add_string_with_plus(iters):    s = ""    for i in range(iters):        s += "abc"    assert len(s) == 3*iters    ### Concatenate strings using join() functiondef add_string_with_join(iters):    l = []    for i in range(iters):        l.append("abc")    s = "".join(l)    assert len(s) == 3*iters    ### Compare speed>>> timeit(add_string_with_plus(10000))100 loops, best of 5: 3.74 ms per loop>>> timeit(add_string_with_join(10000))100 loops, best of 5: 2.3 ms per loop
Check out itertools documentation for more methods. I recommend exploring:
itertools.accumulate(iterable , func): accumulate through iterable. func can be an operator.func or default Python functions such as max, min...
itertools.compress(iterable, selectors): filters the iterable with another (the other object can be treated as a condition)
itertools.filterfalse(predicate, iterable): filter and drop values that satisfy the predicate. This is useful and fast for filtering a list object.
itertools.repeat(object[, times]): repeat a object value N times. However, I prefer using list multiplication ['hi']*1000 to repeat ‘hi’ 1000 times than using itertools.repeat('hi', 1000) (12.2 μs per loop vs 162 μs per loop respectively)
itertools.zip_longest(*iterables, fillvalue=None): zip multiple iterables into tuples and fill None value with the value specified in fillvalue.
import statement can be executed from anywhere. However, executing outside of a function will run much faster than the inside one, even though the package is declared as a global variable (doit2), but in return, takes up more memory space than the other one.
I’m confident to say that most of the time you don’t use all of your data all at once and loading them in 1 big batch may crash the memory. Therefore, chunking data or load in small chunks, process the chunk of data, and save the result is one of the most useful techniques to prevent memory error.
pandas lets you do that with the help of chunksize or iterator parameters in pandas.read_csv() and pandas.read_sql(). sklearn also supports training in small chunks with partial_fit()method for most models.
Handling Memory Error in Python is a tricky task and the root cause might be never been detected if you go another way around.
First, investigate which process/variable is the core reason leading to the memory overflown problem.
Second, apply the applicable memory optimization methods for that object, estimate the new memory footprint and examine if the new one solves the issue.
If not, try to optimize related processes (such as reducing whole process memory consumption to save more space for the core object).
Try the above tips and if the trouble remains, consider building the process with chunk/batch operation with the support of an outside database service.