Title: multiprocessing on pybind11 c++ function hangs on pool.join()
Tags: python,multiprocessing,pybind11

Question: I'm trying to run a c++ function in parallel on python. I've wrapped the c++ function with pybind11, and then again define a wrapper function in python, that I call using multiprocessing.pool.map_async().
Here is the basic code.
displayNameLst is just a list of strings in python
displayNameLensLst is a list of ints, just gives displayNameLensLst[j] = len(displayNameLst[j])
cutoffLst is a list of ints that gives the cutoff to Levenshtein distance where the inner c++ Levenstein distance function can just return False.
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <vector>
#include <string>
#include <iostream>
#include <string.h>
#include <sstream>
using namespace std;

bool LevenshteinDistanceCutoff(const char* str1, const char* str2, const uint_fast16_t l1, const uint_fast16_t l2, const uint_fast16_t cutoff, uint_fast16_t** d)
{
   # skipped for brevity
}

vector<vector<uint_fast32_t>> CheckPairwiseLevensteinDistOfList(vector<string> &str_list, vector<uint_fast16_t> &strLens, const uint_fast16_t cutoff)
{    
   uint_fast32_t size = str_list.size();
   cout << "length of str_list = " << size << endl;
   
   uint_fast16_t maxLen = strLens[size-1];
   uint_fast16_t penultimateLen = strLens[size-2];

   cout << "maxLen " << maxLen << endl;
   cout << "penultimateLen " << penultimateLen << endl;

   vector<vector<uint_fast32_t>> indPotMatches(size);

   cout << "current size indPotMatches " << indPotMatches.size() << endl;
   cout << "max size indPotMatches " << indPotMatches.max_size() << endl;

   uint_fast16_t** d = new uint_fast16_t*[penultimateLen+1];
   for (uint_fast16_t i=0; i<penultimateLen+1; ++i)
   {
      d[i] = new uint_fast16_t[maxLen+1];
   }
   
   for (uint_fast32_t j=0; j<size-1; ++j)
   {
      for (uint_fast32_t k=j+1; k<size; ++k)
      {
      
     if (LevenshteinDistanceCutoff(str_list[j].c_str(), str_list[k].c_str(), strLens[j], strLens[k], cutoff, d))
     {
        indPotMatches[j].push_back(k);
     }
      }
   }
   for (uint_fast16_t i=0; i<penultimateLen+1; ++i)
   {
      delete[] d[i];
   }
   
   delete[] d;

   return indPotMatches;
      
}


PYBIND11_MODULE(CHelpers, m) {   
      
   m.def("CheckPairwiseLevensteinDistOfList", &CheckPairwiseLevensteinDistOfList, pybind11::return_value_policy::take_ownership);
   
   
}

and the python code:
import multiprocessing as mp
sys.path.append(DIR of C++.so)
import CHelpers
sys.path.remove(DIR of C++.so)

NBR_PROC = 12

# python wrapper around C++ function
def CheckPairwiseLEvensteinDistOFListWrapper(arg):
    locStartValRange = arg[0]
    locEndValRange = arg[1]
    locDisplayNameLst = arg[2]
    locDisplayNameLensLst = arg[3]
    locCutoff = arg[4]
    tmpInd = CHelpers.CheckPairwiseLevensteinDistOfList(locDisplayNameLst, locDisplayNameLensLst, locCutoff)
    return locStartValRange, locEndValRange, locCutoff, tmpInd

# some other minor stuff like global variables

if __name__ == '__main__':
    # A bunch of other stuff to prepare the args list to be sent to the multiprocessing pool
    lstArgsForMap = [(startValRangeLst[j], endValRangeLst[j], displayNameLst[startValRangeLst[j]:endValRangeLst[j]].copy(), displayNameLensLst[startValRangeLst[j]:endValRangeLst[j]].copy(), cutoffLst[j])  for j in range(len(startValRangeLst))]

    mp.set_start_method('spawn')
    pool = mp.Pool(processes = NBR_PROC, maxtasksperchild=1)
    res = pool.map_async( CheckPairwiseLEvensteinDistOFListWrapper, lstArgsForMap)
    pool.close()
    pool.join()

    res = res.get()

    # other post-processing of res    

Now the weird thing is that this executes without any issues on my laptop, but when I try this on my remote server, it always hangs on the pool.join(). Eventually I see that there is 1 process left, taking 0% of CPU and ~20% of memory. After a very long time (12+ hours) the process eventually gets killed, and I get the error:
“/usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d 'w”
Both the server and laptop run ubuntu 20.04. python3.8 and python3 both point to python 3.8.10.
The only difference between the server and my laptop is the size of the list of strings that I'm trying to compare. On my laptop it's roughly 1M, while on the server it's about 250M. (This does take up a lot of memory, but the server has ~500GB of RAM, and it can easily hold a pandas dataframe with all the strings + a bunch of other columns of lists.)

I changed the start method from "fork" to "spawn" after reading about issues that fork causes with these kinds of memory leaks.
I also added maxtaksperchild to create a new process for each task.
Added pybind11::return_value_policy::take_ownership to make sure python knows it's suppose to handle the deletion of vector of vectors that each task returns (maybe this is not working and that's the issue)

I've done a lot of googling about this, but the issue seems very murky. It's the first time I use pybind11 and multiprocessing together and I am basically a total noob with both. I'm not sure where to even start to continue troubleshooting this. Any help would be much appreciated, since I am hoping to use this combination of parallel python + inner c++ loops more in the future. Worst case I could rewrite the entire thing in c++ using threads.

Answer: It turns out the problem was mostly that my return values of vectors of vectors of ints were too large in some cases. I am doing a pairwise comparison of strings, and for some inputs, there were just way too many matches, which meant I ended up essentially with a non-sparse 1M x 1M matrix. I don't know if I should delete the question since it's not an issue with multiprocessing at all.
The memory management of multiprocessing with forkserver duplicating memory for every process and then tripling that memory usage when pickling (is my understanding of how it works) didn't help of course, and is very relevant when you're operating close to memory limits.