Speeding up Postgres writing with Python multiprocessing
By: Gabriela Matos Guedes
Tags: Multiprocessing, Python, Performance, Mongodb, Postgres

I had the following problem: I was supposed to move 10 millions documents from a collection in MongoDB to 3 tables in Postgres in a relational way using Python. In MongoDB, I had documents like:

restaurant: "Some restaurant"

restaurant_address: "Some address"

dish_name: "Roasted chicken"

base_ingredients: ["chicken", "oil", "garlic", "pepper"]

is_vegan: false

price: 10.0

code: 235

size: "big"

They represent each dish “product” that there was in each restaurant menu. They needed to be broken into 3 tables from Postgres:

This way, I would have 10 millions records in the table dish_products, but it would also mean I would have less records in the other two tables, because there were documents in Mongo from the same restaurant and also the same dish, but with different sizes. For example, if there were 4 millions of differents restaurants and 8 millions of dishes, at the end, Postgres would receive ~22 millions of write operations.

Due to the relations and the need to save foreign keys in the tables dishes and dish_products, I couldn’t get the documents from Mongo in batches, create a dataframe and insert many records at once. Also, I would have to do an upsert, instead of a simple insert, because there were some documents with null value for the restaurant_address, for example.

To compare the performance between each solution I have tested, I am going to consider here only the number of records in dish_products , because it is the last table in which I would write for each document and a document would only be written on this table if the other two had also been written successfully. The database I was writting on wasn’t at my local computer (and not even at the same region) where my script was running.

All the queries from Mongo were done with pagination (in batches of 1000 documents), to not request the 10 millions documents at once.

Writing it the “normal” way

At first, I wrote it in a simple way. I fetched the data from Mongo and wrote on Postgres opening, committing and closing the connection for each record saved. The main script was like this:

And the storer class used by it:

Then, when I runned it for the first time, I realized it would really take too long to move all this data. In 5 minutes, it had moved 66 documents. Thus, for 10 millions documents, this method would take ~526 days, something completly impracticable.

With the autocommit option set

My second try was to stop closing and reopening the connection for each operation and use the autocommit attribute from psycopg as True. So, there were some little changes in the code, I had to connect to Postgres with the Storer initialization:

class Storer:

def __init__(self):

con = pg_connect()

con.autocommit = True

self.cursor = con.cursor

# ...

And the write_in_postgres function now wouldn’t need to connect/commit/close:

def write_in_postgres(self, query, values):

self.cursor.execute(query, values)

return self.cursor.fetchone()['id']

The other functions from Storer(the ones with the queries) and the main script remained the same.

Running this was ~89% faster than the previous method! But it was still very slow, moving 595 documents in 5 mintues, therefore it was going to take ~58 days for all the documents.

Committing in batches

When I searched faster ways to write many records in Postgres, I saw somewhere in Stack Overflow that the autocommit is the same as doing “manually” the commit after each query execution. So, I decided to turn off the autocommit and commit manually only after some records were inserted. I had chosen, then, to commit after 1000 records being inserted into dish_products so I could use the same while I was already using in the main section (just to remember, this while is looped for each batch requested from Mongo, which size is 1000). The main script now would look like this (the changes are in bold):

# ... while products != []:

for product in products:

restaurant_id = s.store_in_restaurant_table(product)

dish_id = s.store_in_dish_table(product, restaurant_id)

s.store_in_product_table(product, dish_id)

s.con.commit()

products = fetch()

The Storer initializer without autocommit:

class Storer:

def __init__(self):

self.con = pg_connect()

self.cursor = con.cursor #...

There were no changes in the others sections.

The difference between this method and the last one was very tiny. This method moved 618 documents in 5 minutes and, for all the documents, it would take ~56 days.

Using threads

Thread is the most recommended alternative when your program performance depends on I/O things, just like this case. Therefore, I have created a thread for each batch of documents that would be saved, without limits on the number of threads. I had also set the autocommit as True again. Changes in the main script:

import threading products = fetch()

s = Storer() while products != []:

threading.Thread(target=save_products, args=(products)).start()

products = fetch()

The function called for each thread (which was basically what already existed):

def save_products(products):

for product in products:

restaurant_id = s.store_in_restaurant_table(product)

dish_id = s.store_in_dish_table(product, restaurant_id)

s.store_in_product_table(product, dish_id)

And the Storer initializer:

class Storer:

def __init__(self):

con = pg_connect()

con.autocommit = True

self.cursor = con.cursor

For my surprise, it was even slower than the last 2 solutions. In 5 minutes, it has moved 422 documents. To move everything, the thread solution would take ~82 days.

Using multiprocessing

Although the script problem wasn’t due to CPU limitations (the script was being limited by the database operations), I decided to try using multiprocessing. The computer I was using had 4 processors, so I set a limit of 4 simultaneous processes. I also had to create a new connection in Postgres for each process, because Postgres doesn’t allow different processes using the same connection.

The main script changes:

from multiprocessing import Process

from multiprocessing.sharedctypes import Value

import time processes_running = Value('i', 0)

products = fetch() while products != []:

p = Process(target=save_products, args=(products)).start()

while processes_running.value >= 4:

time.sleep(10)

p.start()

products = fetch()

The save_products function:

def save_products(products):

processes_running.value += 1

s = Storer()

for product in products:

restaurant_id = s.store_in_restaurant_table(product)

dish_id = s.store_in_dish_table(product, restaurant_id)

s.store_in_product_table(product, dish_id)

processes_running.value -= 1

In 5 minutes, it had stored 2338 documents! For all the documents, it would take ~14.9 days, which is excellent (compared to the previous numbers), but it could still be improved. Then, I decided to test increasing the limit from 4 processes to 50:

# ... p = Process(target=save_products, args=(products)).start()

while processes_running.value >= 50:

time.sleep(10)

p.start() #...

This last time, it had saved 9947 documents!! For all the base in Mongo, it would take only ~3.5 days, which is enough for my project requirements, also ~99% faster than the first method and muuuch better than the previous solutions. :)