Question: Apologies in advance for the extremely long question--it has been a journey... so thanks in advance for your patience. TL;DR: python throws a segfault when using this set up: [numpy code] <-- [python/C api] <-- [C-implemented .so library] <-- [py driver using ctypes' cdll] Note the py driver is just for testing the .so library, the lib is ultimately intended to be used in another 3rd party app. Background Our team has some python code that we are trying to integrate into another 3rd party application, and the application in question expects symbol implementations in the form of a shared object file (.so library). To accomplish this, I am trying to make a shared object library in C that leverages the Python/C API. I have been running into some issues getting things to run as expected, so I've tried to distill the problem down to an "as-simple-as-possible" example: I am making a basic shared object library written in C, using the Python/C API, and in particular this library's methods have a numpy dependency (this will be important later). The lib is ultimately intended to be used by a "runtime application" (via dynamically loading symbols, for example with dlfcn.h) that is "pre-existing" (I don't have control over the compilation/linking etc. of said program). I am running in to what seems like a linking issue, but may be an issue with either how the so is compiled and/or how I am using/configuring python via the C API. I know this is possible to do because of some existing use cases I've found about on the internets, but I cannot seem to zero in on where my setup is failing. Setup Let's say I am on ubuntu 20.04 with python3.8-dev installed system-wide (in this example I am actually working in a docker container, so if it ends up possibly a system setup issue I am happy to provide the dockerfile as well). Further, I have a virtual environment set up in e.g. ~/venv, it is activated, and numpy is installed--I can confirm this with, for example, (venv) user@4189d31a5bbe:~$ which python && python -c "import numpy; print(numpy)" /home/user/venv/bin/python <module 'numpy' from '/home/user/venv/lib/python3.8/site-packages/numpy/__init__.py'>  Library Files I have the following header/source files: mylibwithpy.h: #ifndef __MYLIBWITHPY__ #define __MYLIBWITHPY__ #include <stdio.h> #include <Python.h> void someFunctionWithPython(); #endif  mylibwithpy.c: All the function someFunctionWithPython does is check if python is initialized, initializes if it isn't, and then tries to import numpy. #include "mylibwithpy.h" void someFunctionWithPython() { if (!Py_IsInitialized()) { printf("Initializing python...\n"); Py_Initialize(); } else { printf("python alread initialized.\n"); } printf("importing numpy...\n"); PyObject* numpy = PyImport_ImportModule("numpy"); if (numpy == NULL) { printf("Warning: error during import:\n"); PyErr_Print(); Py_Finalize(); exit(1); } return; }  The library *.so file is compiled via these Makefile targets: mylibwithpy.o: gcc -L/usr/lib/x86_64-linux-gnu -I/usr/include/python3.8 -Wall -c mylibwithpy.c -o $@ -lpython3.8 mylibwithpy.so: mylibwithpy.o gcc -L/usr/lib/x86_64-linux-gnu -Wall -fPIC -shared -Wl,-soname,$@ -o $@ mylibwithpy.o -lpython3.8  At this point, ldd seems to check out as so far "ok": (venv) user@4189d31a5bbe:~$ ldd mylibwithpy.so linux-vdso.so.1 (0x00007ffc36ba6000) libpython3.8.so.1.0 => /lib/x86_64-linux-gnu/libpython3.8.so.1.0 (0x00007f295c5ea000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f295c3f8000) libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f295c3ca000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f295c3ae000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f295c38b000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f295c385000) libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f295c37e000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f295c22f000) /lib64/ld-linux-x86-64.so.2 (0x00007f295cb4f000)  Example runtime This is where things start going wrong... First, let's try just a basic python driver, using python's ctypes library. driver.py from ctypes import cdll if __name__ == "__main__": print("opening mylibwithpy.so..."); my_so = cdll.LoadLibrary("mylibwithpy.so") print(".so object: ", my_so) print(".so object's 'someFunctionWithPython': ", my_so.someFunctionWithPython) print("calling someFunctionWithPython..."); my_so.someFunctionWithPython()  This basic script will result in a (Segmentation fault) error at the point that numpy is trying to be imported in the lib function: (venv) user@4189d31a5bbe:~$ LD_LIBRARY_PATH=. python driver.py opening mylibwithpy.so... .so object: <CDLL 'mylibwithpy.so', handle 19f43b0 at 0x7fd873b72610> .so object's 'someFunctionWithPython': <_FuncPtr object at 0x7fd873ac91c0> calling someFunctionWithPython... python alread initialized. importing numpy... Segmentation fault (core dumped)  Ok, so I'm not really sure how to even start to debug this guy, so let's try again with an equivalent driver in C: driver.c: #include <dlfcn.h> #include <stdio.h> #include <stdlib.h> int main() { printf("opening mylibwithpy.so...\n"); void* mylibwithpy_so = dlopen("mylibwithpy.so", RTLD_LAZY); if (mylibwithpy_so == NULL){ printf("an error occurred during loading mylibwithpy.so: \n%s\n", dlerror()); exit(1); } void (*soFunc)(); soFunc = dlsym(mylibwithpy_so, "someFunctionWithPython"); if (soFunc == NULL){ printf("an error occurred during loading symbol someFunctionWithPython: \n%s\n", dlerror()); exit(1); } soFunc(); return 0; }  Compiling this program via: gcc -L/usr/lib/x86_64-linux-gnu -Wall driver.c -o cdriver -ldl  And running this driver results in an interestingly-much-more verbose error being reported: (venv) user@4189d31a5bbe:~$ LD_LIBRARY_PATH=. ./cdriver opening mylibwithpy.so... Initializing python... importing numpy... Warning: error during import: Traceback (most recent call last): File "/home/user/venv/lib/python3.8/site-packages/numpy/core/__init__.py", line 23, in <module> from . import multiarray File "/home/user/venv/lib/python3.8/site-packages/numpy/core/multiarray.py", line 10, in <module> from . import overrides File "/home/user/venv/lib/python3.8/site-packages/numpy/core/overrides.py", line 6, in <module> from numpy.core._multiarray_umath import ( ImportError: /home/user/venv/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyObject_SelfIter During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/user/venv/lib/python3.8/site-packages/numpy/__init__.py", line 141, in <module> from . import core File "/home/user/venv/lib/python3.8/site-packages/numpy/core/__init__.py", line 49, in <module> raise ImportError(msg) ImportError: IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE! Importing the numpy C-extensions failed. This error can happen for many reasons, often due to issues with your setup or how NumPy was installed. We have compiled some common reasons and troubleshooting tips at: https://numpy.org/devdocs/user/troubleshooting-importerror.html Please note and check the following: * The Python version is: Python3.8 from "/home/user/venv/bin/python3" * The NumPy version is: "1.24.2" and make sure that they are the versions you expect. Please carefully study the documentation linked above for further help. Original error was: /home/user/venv/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyObject_SelfIter  AHA!(...?) According to this, it seems numpy has some shared objects of its own, but is somehow missing some symbols (PyObject_SelfIter, to be precise--side note, this symbol is listed in the Python/C API "stable ABI contents"): /home/user/venv/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyObject_SelfIter  (Another side note: the listed numpy docs reference https://numpy.org/devdocs/user/troubleshooting-importerror.html does not seem very applicable to the situation or the error I'm getting, but a more discerning eye may find something helpful that I overlooked...) Also, a quick ldd check shows that indeed libpython is not among the dynamically linked libraries: (venv) user@4189d31a5bbe:~$ ldd /home/user/venv/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so linux-vdso.so.1 (0x00007ffed7df2000) libopenblas64_p-r0-15028c96.3.21.so => /home/user/venv/lib/python3.8/site-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-15028c96.3.21.so (0x00007f98a1ae6000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f98a198f000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f98a196c000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f98a177a000) /lib64/ld-linux-x86-64.so.2 (0x00007f98a3f29000) libgfortran-040039e1.so.5.0.0 => /home/user/venv/lib/python3.8/site-packages/numpy/core/../../numpy.libs/libgfortran-040039e1.so.5.0.0 (0x00007f98a12ed000) libquadmath-96973f99.so.0.0.0 => /home/user/venv/lib/python3.8/site-packages/numpy/core/../../numpy.libs/libquadmath-96973f99.so.0.0.0 (0x00007f98a10ae000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f98a1092000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f98a1077000)  Testing the linker issue hypothesis Since we're getting an undefined symbol error printed out in the cdriver program, I can try to force-link libpython to the cdriver via: gcc -L/usr/lib/x86_64-linux-gnu -Wall driver.c -o cdriver -ldl -Wl,--no-as-needed -lpython3.8  and lo and behold this time the program completes without error: (venv) user@4189d31a5bbe:~$ LD_LIBRARY_PATH=. ./cdriver opening mylibwithpy.so... Initializing python... importing numpy... (venv) user@4189d31a5bbe:~$  Note that in the real build I will not have access to compiling/linking the runtime program, so this check is NOT a solution but seems to help diagnose the problem? SO... down to the actual questions  What is needed to, for example, get the python driver to work as expected? Why do numpy's internal shared objects files not link to libpython3.8.so? What am I missing?! :sob:  I'm hoping I'm just missing some small but crucial step in compiling the .so or configuring python.  EDIT: I am able to consistently recreate the issue using a very simple docker container: Dockerfile: FROM ubuntu:20.04 RUN apt-get update; \ DEBIAN_FRONTEND=noninteractive apt-get install -y \ build-essential \ vim \ python3.8-dev \ python3.8-venv RUN useradd --create-home --shell /bin/bash user USER user WORKDIR /home/user RUN python3 -m venv venv RUN /bin/bash -c "source venv/bin/activate && pip install numpy"  Makefile: all: mylibwithpy.so cdriver clean: rm mylibwithpy.o mylibwithpy.so cdriver cdriver: gcc -L/usr/lib/x86_64-linux-gnu -Wall driver.c -o $@ -ldl mylibwithpy.o: gcc -L/usr/lib/x86_64-linux-gnu -I/usr/include/python3.8 -Wall -c mylibwithpy.c -o $@ -lpython3.8 mylibwithpy.so: mylibwithpy.o gcc -L/usr/lib/x86_64-linux-gnu -Wall -fPIC -shared -Wl,-soname,$@ -o $@ mylibwithpy.o -lpython3.8  Then executing with the same command as above LD_LIBRARY_PATH=. ./cdriver 

Answer: When using functions exported by cdll.LoadLibrary, you're releasing the Global Interpreter Lock (GIL) as you enter the method. If you want to call python code, you need to re-acquire the lock. e.g. void someFunctionWithPython() { ... PyGILState_STATE state = PyGILState_Ensure(); printf("importing numpy...\n"); PyObject* numpy = PyImport_ImportModule("numpy"); if (numpy == NULL) { printf("Warning: error during import:\n"); PyErr_Print(); Py_Finalize(); PyGILState_Release(state); exit(1); } PyObject* repr = PyObject_Repr(numpy); PyObject* str = PyUnicode_AsEncodedString(repr, "utf-8", "~E~"); const char *bytes = PyBytes_AS_STRING(str); printf("REPR: %s\n", bytes); Py_XDECREF(repr); Py_XDECREF(str); PyGILState_Release(state); return; }  $ gcc $(python3.9-config --includes --ldflags --embed) -shared -o mylibwithpy.so mylibwithpy.c $ LD_LIBRARY_PATH=. python driver.py opening mylibwithpy.so... .so object: <CDLL 'mylibwithpy.so', handle 1749f50 at 0x7fb603702fa0> .so object's 'someFunctionWithPython': <_FuncPtr object at 0x7fb603679040> calling someFunctionWithPython... python alread initialized. importing numpy... REPR: <module 'numpy' from '/home/me/test/.venv/lib/python3.9/site-packages/numpy/__init__.py'>  Also if you look at PyDLL it says:  Instances of this class behave like CDLL instances, except that the Python GIL is not released during the function call, and after the function execution the Python error flag is checked. If the error flag is set, a Python exception is raised.  So if you use PyDLL for your driver then you wouldn't need to re-acquire the lock in the C code: from ctypes import PyDLL if __name__ == "__main__": print("opening mylibwithpy.so..."); my_so = PyDLL("mylibwithpy.so") print(".so object: ", my_so) print(".so object's 'someFunctionWithPython': ", my_so.someFunctionWithPython) print("calling someFunctionWithPython..."); my_so.someFunctionWithPython()   UPDATE  Why do numpy's internal shared objects files not link to libpython3.8.so?  I believe numpy is setup this way because it expects to be called by the python interpreter where libpython will already be loaded and have the symbols made available. That said, we can make the python libraries available for when mylibwithpy calls the import of numpy by using RTLD_GLOBAL.  The symbols defined by this shared object will be made available for symbol resolution of subsequently loaded shared objects.  The update to your code is simple: void* mylibwithpy_so = dlopen("mylibwithpy.so", RTLD_LAZY | RTLD_GLOBAL);  Now all of the python libraries will be included because they are a dependency of mylibwithpy, meaning they will be available by the time that numpy loads its own shared libraries. Alternatively, you could choose to load just libpythonX.Y.so with RTLD_GLOBAL to prior to loading mylibwithpy.so to minimize the amount symbols made globally available. printf("opening libpython3.9.so...\n"); void* libpython3_so = dlopen("libpython3.9.so", RTLD_LAZY | RTLD_GLOBAL); if (libpython3_so == NULL){ printf("an error occurred during loading libpython3.9.so: \n%s\n", dlerror()); exit(1); } printf("opening mylibwithpy.so...\n"); void* mylibwithpy_so = dlopen("mylibwithpy.so", RTLD_LAZY); if (mylibwithpy_so == NULL){ printf("an error occurred during loading mylibwithpy.so: \n%s\n", dlerror()); exit(1); }  Docker setup I used to recreate and test: FROM ubuntu:20.04 ARG DEBIAN_FRONTEND=noninteractive RUN apt-get update && apt-get install -y \ build-essential \ python3.9-dev \ python3.9-venv RUN mkdir /workspace WORKDIR /workspace RUN python3.9 -m venv .venv RUN .venv/bin/python -m pip install numpy COPY . /workspace RUN gcc -o mylibwithpy.so mylibwithpy.c -fPIC -shared \ $(python3.9-config --includes --ldflags --embed --cflags) RUN gcc -o cdriver driver.c -L/usr/lib/x86_64-linux-gnu -Wall -ldl ENV LD_LIBRARY_PATH=/workspace # Then run: . .venv/bin/activate && ./cdriver