open-mpi/ompi — pull request #240 by artpol84 (2014-10-17 08:13:06+00:00)
Title: Fix building system: correct libltdl check for advise capability.

Hello,

Short story: The build sustem libltdl check for advise option is broken. This PR fix this.

Long story is under the cut.
-------------------------------------
I faced interesting problem on my laptop running Mint 17 with SLURM inside Linux Containers. I use near-latest SLURM and OMPI versions from the corresponding github trunks.

The problem I see was the following. If I use PMI1 (srun ./hellompi) from the batch script I receive the following error:
/home/artpol/WORK/Development/SLURM/mpi_plugin/jobs/hello_job/./hellompi: symbol lookup error: /home/artpol/sandboxes/slurm/lib/slurm/auth_none.so: undefined symbol: slurm_debug
srun: error: slurm_receive_msg[10.0.3.6]: Zero Bytes were transmitted or received
srun: error: cn4: task 1: Exited with exit code 127
srun: error: slurm_receive_msg: Zero Bytes were transmitted or received

The problem analysis:

1. slurm_debug is located in libslurm.

2. pmi1 component is linked with it:
$ ldd mca_pmix_s1.so
linux-vdso.so.1 => (0x00007fff2b9fe000)
libpmi.so.0 => /home/artpol/sandboxes/slurm/lib/libpmi.so.0 (0x00007f3259cbd000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3259a7e000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f32596b8000)
libslurm.so.28 => /home/artpol/sandboxes/slurm/lib/libslurm.so.28 (0x00007f32592f1000)
/lib64/ld-linux-x86-64.so.2 (0x00007f325a0cc000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f32590ec000)

3. If I attach to hellompi process right before the call to the PMI_KVS_Commit I can see libslurm in /proc//maps.

4. PMI_KVS_Commit uses dlopen to load auth plugin.

Conclusion: that mca_pmix_s1.so (and subsequently libslurm) is dlopen'd by libltdl without GLOBAL flag. So lately dlopen'd auth_none.so can't see slurm_debug.so. Tracking it down to opal/libltdl/loaders/dlopen.c:vm_open() confirms this guess. And if I use LD_PRELOAD in the batch script I fix this problem.

I was confused that this error was discovered only by me and not rised before. So I compared the results with the real cluster that I use. It turns out that on my laptop OPAL_HAVE_LTDL_ADVISE is set to zero preventing OMPI from setting is_symglobal flag. So the issue was with configuration on my system.

./configure creates the conftest.c file with the following content:
... < lots of defines > ....
#define OMPI_MPI_CONTRIBS "libompitrace, vt"
#define OPAL_C_HAVE_VISIBILITY 1
#define HAVE_DLFCN_H 1
#define LT_OBJDIR ".libs/"
/* end confdefs.h. */
#include <./opal/libltdl/ltdl.h>

and compiles it with:
gcc -E -I. conftest.c

On my system subsequent include in ltdl.h:
....
#include
#include
...

caused an error:
$ gcc -E -I. ltdl_error_bkp.c
# 1 "ltdl_error_bkp.c"
# 1 ""
# 1 "/usr/include/stdc-predef.h" 1 3 4
# 1 "" 2
# 1 "ltdl_error_bkp.c"
# 715 "ltdl_error_bkp.c"
# 1 "././opal/libltdl/ltdl.h" 1
In file included from ltdl_error_bkp.c:715:0:
././opal/libltdl/ltdl.h:36:31: fatal error: libltdl/lt_system.h: No such file or dirctory
#include
^
compilation terminated.

So the suggested PR adds -I./opal/libltdl to the building system to let gcc know where to find lt_system.h.

On the real system gcc also detect this errors, however this doesn't force compilation to terminate:
$ gcc -E -I. ltdl_error_bkp.c
# 1 "ltdl_error_bkp.c"
# 1 ""
# 1 ""
# 1 "ltdl_error_bkp.c"
# 715 "ltdl_error_bkp.c"
# 1 "././opal/libltdl/ltdl.h" 1
In file included from ltdl_error_bkp.c:715:
././opal/libltdl/ltdl.h:36:31: error: libltdl/lt_system.h: No such file or dirctory
././opal/libltdl/ltdl.h:37:30: error: libltdl/lt_error.h: No such file or dirctory
././opal/libltdl/ltdl.h:38:33: error: libltdl/lt_dlloader.h: No such file or dirctory
# 40 "././opal/libltdl/ltdl.h"
LT_BEGIN_C_DECLS
# 50 "././opal/libltdl/ltdl.h"
typedef struct lt__handle *lt_dlhandle;
...