Python, Memory, and Objects. The basics of memory management for... | by Naser Tamimi | Towards Data Science

As data scientists, normally, we don’t pay attention to how Python and the underlying operating system handle memory for our code. After all, Python is the most popular language among data scientists, partly because it automatically handles those details. As long as we are working on small datasets, ignoring how Python manages memory (i.e., memory allocation and deallocation) does not impact our code performance. But, as soon as we switch to large datasets (big data) or heavy processing projects, basic knowledge about memory management becomes crucial.
As an example, I was working on a data science project regarding indexing human DNA. I used a python dictionary object to keep track of sequences (i.e., sequences of nucleotides) and store their location in a reference human DNA. About 10% into the process, the dictionary object took all my RAM and started swapping between disk and RAM. It made the process super slow (as the disk is much slower in data transmission). As a data scientist, if I knew the basics of Python and memory management, I could prevent it and make much more memory-efficient codes.
In this article and an upcoming article, I explain some basic concepts around memory management in Python. At the end of this article, you have good basic knowledge of how Python handles memory allocation and deallocation. Let’s get started ...
A python program is a collection of
methodsreferencesobjects
methods
references
objects
Methods or operations are easy. When you add two numbers, you are basically applying the add (or sum) method to two values. References are a little bit tricky to explain. A reference is a name that we use to access a data value (i.e., an object). The most famous references in programming are variables. When you define x = 1 , x is the variable or reference and 1 is its value (more accurate an integer object). In addition to variables, attributes and items are two other popular references in programming.
Now, let's get deeper and introduce objects. As a Python programmer, you must have heard that “Everything in Python is an object.” An integer number is an object. A string is an object. Lists, dictionaries, tuples, pandas data frames, NumPy arrays are objects. Even a function is an object. When we create an object, it will be stored in memory. When we defined references in the previous paragraph, I should have told you that a reference does not point to a value in Python but points to the memory address of an object. For example, in our simple example x = 1 the reference x is pointing to a memory address that the integer object 1 is stored.
At the run time, computer memory gets divided into different parts. Three important memory sections are:
CodeStackHeap
Code
Stack
Heap
Code (also called Text or Instructions) section of the memory stores code instructions in a form that the machine understands. The machine follows instructions in the code section. According to the instruction, the Python interpreter load functions and local variables in the Stack Memory (also called the Stack). Stack memory is static and temporary. Static means the size of values stored in the Stack cannot be changed. Temporary means, as soon as the called function returned its value, the function and the related variable will be removed from the Stack. As a data scientist and programmer, you don’t have access to Stack memory. Python interpreter and OS memory management together take care of this section of memory.
As you learned, variables (or references in general) only stores memory addresses of objects. So, where are the objects? Are they in Stack memory? No, they are in a different memory called “Heap Memory” (also called the Heap). To store objects, we need memory with dynamic memory allocation (i.e., size of memory and objects can change). Python interpreter actively allocates and deallocates the memory on the Heap (what C/C++ programmers should do manually!!! Thanks, Python!!!). Python uses a garbage collection algorithm (called Garbage Collector) that keeps the Heap memory clean and removes objects that are not needed anymore.
You don’t need to mess with the Heap, but it is better to understand how Python manages the Heap since most of your data is stored in this section of the memory.
Let’s find the memory address on the Heap that the variable x points to. To find it out, we can use a function called id().
>>> x = 1>>> id(x)140710407579424>>> hex(id(x))'0x7ff9b1dc2720'
When we run the first line (x = 1), Python stores integer object 1 in a memory address 140710407579424 on my computer (different from yours). In computer science, we normally show memory addresses in hexadecimal numbers; therefore, I used the hex() function (note: the prefix 0x is used in computer science to indicate that the number is in hex). After storing the int object 1 in Heap memory, Python tells the reference (or variable) x to memorize this address (140710407579424 or 0x7ff9b1dc2720) as its value.
Take a look at this example.
>>> x = 1>>> y = 1>>> hex(id(x))'0x7ffdf176a190'>>> hex(id(y))'0x7ffdf176a190'
Here, I defined two variables (x and y). I assigned them an integer object (i.e. 1) to both of them. Surprisingly, the memory addresses that both variables point to are the same. Look at another example.
>>> str1 = "Python">>> str2 = "Python">>> hex(id(str1))'0x1e3adfe2830'>>> hex(id(str2))'0x1e3adfe2830'
I defined two variables (str1 and str2) and assigned a string object (Python) to both of them. The memory addresses that both variables point to are the same. If you test the same thing for boolean objects, you will see a similar observation. Why?
To optimize memory allocation. Python does a process called “interning.” For some objects (will be discussed later), Python only stores one object on Heap memory and ask different variables to point to this memory address if they use those objects. The objects that Python does interning on them are integer numbers [-5, 256], boolean, and some strings. Interning does not apply to other types of objects such as large integers, most strings, floats, lists, dictionaries, tuples.
So far, we have shown examples of simple data structures such as integers, strings, or booleans. What about more complex data structures such as lists or dictionaries.
>>> lst = [1, 2, 3, 257]>>> hex(id(lst))'0x236330edf88'>>> hex(id(lst[0]))'0x7ffdf176a190'>>> hex(id(lst[3]))'0x7ffdf176a1b0'
The example clearly shows that the memory address of the list object is different from its items. It makes sense since a list is a collection of objects, each of its items has its own identity and is a separate object with a different memory address.
If each item in a list is a single object, does interning (from the previous section) apply to each item in a list? It is easy to check.
>>> a = 1>>> b = 257>>> hex(id(a))‘0x7ffdf176a190’>>> hex(id(b))‘0x236330dc450’
As you see, both a and lst[0] are pointing to the same memory address due to integer interning. Also, you can see, when the integer number goes beyond 256, both b and lst[1] are pointing to different memory addresses.
What happens when we add a new item to a list. Does the memory address change? Let’s test it.
>>> lst = [1, 2, 3]>>> hex(id(lst))'0x23633104888'>>> lst.append(4)>>> lst[1, 2, 3, 4]>>> hex(id(lst))'0x23633104888'
Interesting, the memory address for the list remains the same. The reason is that a list is a mutable object, and if you add items to it, the object is still the same object with one more item.
Another important fact about mutable objects is that if you instantiate different variables from an object, all of them will change if you make any change to the object. Let me show it with a simple code.
>>> lst1 = [1, 2, 3]>>> lst2 = lst1>>> lst1.append(4)>>> lst2[1, 2, 3, 4]>>> lst2.append(5)>>> lst1[1, 2, 3, 4, 5]
In this example, both variables lst1 and lst2 are pointing to the same mutable object (i.e. [1, 2, 3]). If any of those variables change the object (e.g., append a new item), the value of another variable (which is pointing to the same object) will also change. The only way to get a separate copy of a mutable object is to use .copy() method.
>>> lst1 = [1, 2, 3]>>> lst2 = lst1.copy()>>> lst1.append(4)>>> lst2[1, 2, 3]>>> hex(id(lst1))'0x236330dfe08'>>> hex(id(lst2))'0x236330e0c88'
As you see, using the .copy() method, we are creating two separate list objects with different memory addresses that changing one of them does not change the other one.
Almost anything we said about the list objects also applies to the dictionary objects. There are some subtle differences that are beyond this article. For example, the way their memory size grows after adding a new item will be different.
To check if two or more variables are pointing to the same object, you don't need to check their memory address. You can check if two variables are pointing to the same object using is. Here is an example.
>>> lst1 = [1, 2, 3]>>> lst2 = lst1>>> lst3 = lst1.copy()>>> lst2 is lst1True>>> lst3 is lst1False
Remember, is is different from == . is tells you if two objects are the same but == tells you if their content or value is the same. For example, in the previous code, the contents of both lst3 and lst1 are the same (i.e. [1, 2, 3]), but they are two separate and different objects. The following code shows it clearly.
>>> lst3 == lst1True>>> lst3 is lst1False
This article gave you the basic knowledge about how Python (more accurate the CPython implementation) allocates memory to objects. When working on big data in Python, you need to know these fundamental concepts to write more memory-efficient codes.
Follow me on Twitter for the latest stories: https://twitter.com/TamimiNas