TRIQS/TRIQS — Issue #26: Segfault due to RTLD_GLOBAL dlopen'ed numpy (multiarray)
Author: nstange
Created: 2012-02-14 21:19:41+00:00
State: open

Hi everybody,

this issue is very likely related to
https://github.com/TRIQS/TRIQS/issues/17 (I encountered this problem
while trying to do sth. very similar). Maybe you can confirm this and
merge these two issues somehow?

Minimal testcase:
<--snip-->
from pytriqs.Base.Lattice.pytriqs_LatticeTools import bravais_lattice
from scipy.optimize import leastsq
<--snap-->

Results (running in gdb):
<--snip-->
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff2beba40 in PyArray_API ()
from /usr/lib/pymodules/python2.7/numpy/core/multiarray.so
<--snap-->

Everything on Ubuntu 11.10 (x86_64) with python/numpy/scipy/gcc from
my distribution and boost 1.48.0.

The Reason is the following:
>>> from pytriqs.Base.Lattice.pytriqs_LatticeTools import bravais_lattice
already brings numpy's 'multiarray.so' into my python process and this
happens with RTLD_GLOBAL:

Excerpt from 'pytriqs/Base/Lattice/pytriqs_LatticeTools.py':
<--snip-->
sys.setdlopenflags(flag|ctypes.RTLD_GLOBAL)
from pytriqs.Base.Lattice._pytriqs_LatticeTools import *
sys.setdlopenflags(flag)
<--snap-->
'TRIQS/pytriqs/Base/Lattice/C++/wrap.cpp' now calls numpy's
'_import_array()' and this in turn brings 'multiarray.so' in (while
'RTLD_GLOBAL' is still in effect).

Now,
>>> from scipy.optimize import leastsq
crashes because it tries to write-access the "wrong" 'PyArray_API'
symbol instead of its own one (actually the one from 'multiarray.so'
mapped into a read-only memory section while 'RTLD_GLOBAL' had been in
effect). (See
http://docs.python.org/extending/extending.html#providing-a-c-api-for-an-extension-module
for further details on this PyFoo_API technique).

Please note, that the leastsq is also imported in
'TRIQS/pytriqs/Base/GF_Local/GFBloc_ImFreq.py:117' (method
'fitTail'). This is what brings me to the conclusion that this issue
might be related to the one mentioned above.

As a fix I tried to just remove the calls to 'setdlopenflags'
enclosing the import statements:
<--snip-->
diff --git a/cmake/FindPythonModuleEmbeddingTool.cmake b/cmake/FindPythonModuleE
index fbe5a93..dd9d5da 100644
--- a/cmake/FindPythonModuleEmbeddingTool.cmake
+++ b/cmake/FindPythonModuleEmbeddingTool.cmake
@@ -27,7 +27,7 @@ else (TRIQS_BUILD_STATIC)
#nSET (FIX "from pytriqs.alps import pyhdf5_c \n")
#file (WRITE ${CMAKE_BINARY_DIR}/${ModuleIncludeFile} "import sys,dl\nflag =
# not ok : dl is not present on 64 bits machines and it is deprecated anyway,
- file (WRITE ${CMAKE_BINARY_DIR}/${ModuleIncludeFile} "${FIX}\nimport sys,cty
+ file (WRITE ${CMAKE_BINARY_DIR}/${ModuleIncludeFile} "${FIX}\nfrom pytriqs.$
endif (TRIQS_BUILD_STATIC)
install( FILES ${CMAKE_BINARY_DIR}/${ModuleIncludeFile} DESTINATION ${TRIQS_PYT
endfunction (python_build_module ModuleName)
<--snap-->

For a quick check, I ran a 'make test' and at least I got no
"undefined reference" errors (I got some errors, but these are related
to HDF5 comparisons).

I also tried to import all pytriqs-Modules implemented in C++ (each in its own, fresh python process):
pytriqs.Base.GF_Local._pytriqs_GF
pytriqs.Base.Lattice._pytriqs_LatticeTools
pytriqs.Solvers.HubbardI.hubbard_I
pytriqs.Solvers.HybridizationExpansion._pytriqs_Solver_HybridizationExpansion
pytriqs.Wien2k.vertex
pytriqs.boost.mpi
An again, no "unresolved references".

Best,

Nicolai

P.S:
The people here seem to have a very similar problem:
http://projects.scipy.org/numpy/ticket/1148