Repository: numba/numba

Issue:
Reenable OpenMP threading backend in wheel builds
- [x] I am using the latest released version of Numba (most recent is visible in
 the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG).

On Linux:
* The 0.40.0 wheel build of Numba had an OpenMP backend that segfaulted/sigabrt'd `ld.so`.
* The 0.40.1 wheel build of Numba disabled OpenMP to avoid the above problem.

This issue is to act as a reminder to debug and fix the root cause of the problem, reproducer etc. is here https://github.com/numba/numba/issues/3341


Discussion:
After considerable debugging to get to the bottom of this, I'm of the view that the issue is essentially the same as that described here https://github.com/NixOS/patchelf/issues/84.

Things that helped lead here:
* https://sourceware.org/bugzilla/show_bug.cgi?id=16457
* Searching on a search engine for `edit '.gnu.version'` lead to the `patchelf` issue above.

```bash
$readelf -WV <path>/lib/python3.7/site-packages/numba/npyufunc/omppool.cpython-37m-x86_64-linux-gnu.so

Version symbols section '.gnu.version' contains 69 entries:
 Addr: 00000000000011f8  Offset: 0x0011f8  Link: 3 (.dynsym)
  000:   0 (*local*)       2 (GLIBC_2.2.5)   3 (GOMP_1.0)      4 (GLIBC_2.2.5)
  004:   5 (GLIBCXX_3.4)   6 (CXXABI_1.3)    2 (GLIBC_2.2.5)   2 (GLIBC_2.2.5)
  008:   3 (GOMP_1.0)      0 (*local*)       2 (GLIBC_2.2.5)   5 (GLIBCXX_3.4)
  00c:   5 (GLIBCXX_3.4)   7 (GLIBC_2.2.5)   8 (OMP_1.0)       0 (*local*)    
  010:   0 (*local*)       0 (*local*)       2 (GLIBC_2.2.5)   2 (GLIBC_2.2.5)
  014:   6 (CXXABI_1.3)    2 (GLIBC_2.2.5)   5 (GLIBCXX_3.4)   2 (GLIBC_2.2.5)
  018:   0 (*local*)       8 (OMP_1.0)       2 (GLIBC_2.2.5)   6 (CXXABI_1.3) 
  01c:   6 (CXXABI_1.3)    4 (GLIBC_2.2.5)   2 (GLIBC_2.2.5)   0 (*local*)    
  020:   9 (GCC_3.0)       8 (OMP_1.0)       0 (*local*)       0 (*local*)    
  024:   2 (GLIBC_2.2.5)   5 (GLIBCXX_3.4)   1 (*global*)      1 (*global*)   
  028:   1 (*global*)      1 (*global*)      1 (*global*)      1 (*global*)   
  02c:   1 (*global*)      1 (*global*)      1 (*global*)      1 (*global*)   
  030:   1 (*global*)      1 (*global*)      1 (*global*)      1 (*global*)   
  034:   1 (*global*)      1 (*global*)      1 (*global*)      1 (*global*)   
  038:   1 (*global*)      1 (*global*)      1 (*global*)      1 (*global*)   
  03c:   1 (*global*)      1 (*global*)      1 (*global*)      1 (*global*)   
  040:   1 (*global*)      1 (*global*)      1 (*global*)      1 (*global*)   
  044:   1 (*global*)   

Version needs section '.gnu.version_r' contains 6 entries:
 Addr: 0x0000000000001288  Offset: 0x001288  Link: 37 (.dynstr)
  000000: Version: 1  File: libgcc_s.so.1  Cnt: 1
  0x0010:   Name: GCC_3.0  Flags: none  Version: 9
  0x0020: Version: 1  File: libpthread.so.0  Cnt: 1
  0x0030:   Name: GLIBC_2.2.5  Flags: none  Version: 7
  0x0040: Version: 1  File: libstdc++.so.6  Cnt: 2
  0x0050:   Name: CXXABI_1.3  Flags: none  Version: 6
  0x0060:   Name: GLIBCXX_3.4  Flags: none  Version: 5
  0x0070: Version: 1  File: libm.so.6  Cnt: 1
  0x0080:   Name: GLIBC_2.2.5  Flags: none  Version: 4
  0x0090: Version: 1  File: libgomp.so.1  Cnt: 2   # <--- NOTE libgomp.so.1 for versions of {G,}OMP_1.0
  0x00a0:   Name: OMP_1.0  Flags: none  Version: 8
  0x00b0:   Name: GOMP_1.0  Flags: none  Version: 3
  0x00c0: Version: 1  File: libc.so.6  Cnt: 1
  0x00d0:   Name: GLIBC_2.2.5  Flags: none  Version: 2

$ readelf -d <path>/lib/python3.7/site-packages/numba/npyufunc/omppool.cpython-37m-x86_64-linux-gnu.so
Dynamic section at offset 0x6d98 contains 30 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgomp-24cd7153.so.1.0.0] # <--- NOTE: DT_NEEDED shared lib name
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/../.libs]
```


I think `auditwheel repair` is most likely the part of the build that's executing `patchelf --replace-needed`. Anaconda distribution `patchelf` (and the latest tagged release) is version 0.9 (29 Feb 2016), the patch for `patchelf` that fixed the issue is https://github.com/NixOS/patchelf/pull/85 and went in after the last release.

Doing:
```bash
$ patchelf --replace-needed libgomp-24cd7153.so.1.0.0 libgomp.so.1 <omplib>
```
fixes the ELF to refer to Anaconda shipped/system `libgomp.so.1` which matches the `gnu.version_r`. This leads to a valid ELF.


Alternatively doing (circuitous as `--replace-needed X X` is a NOP):
```bash
$ patchelf_built_from_master --replace-needed libgomp-24cd7153.so.1.0.0 libgomp.so.1 <omplib>
$ patchelf_built_from_master --replace-needed libgomp.so.1 libgomp-24cd7153.so.1.0.0 <omplib>
```
fixes the ELF so that `DT_NEEDED` has ` 0x0000000000000001 (NEEDED)             Shared library: [libgomp-24cd7153.so.1.0.0]`  and `.gnu.version_r` has:
```
  0x0090: Version: 1  File: libgomp-24cd7153.so.1.0.0  Cnt: 2
  0x00a0:   Name: OMP_1.0  Flags: none  Version: 8
  0x00b0:   Name: GOMP_1.0  Flags: none  Version: 3
```
This also leads to a valid ELF.

Further, the ELF linter actually spots the problem with the DSO that caused the segfault/sigabrt in the first place:
```
$ eu-elflint <path>/lib/python3.7/site-packages/numba/npyufunc/omppool.cpython-37m-x86_64-linux-gnu.so|grep .gnu.version_r
section [ 5] '.gnu.version_r': entry 1 references unknown dependency
```
we {c,sh}ould add this to the wheel build scripts?
Ping @sklam 

Also, for toolchain purposes, ping @mingwandroid @nehaljwani @msarahan @jjhelmus
The required patch has been backported to patchelf (build 3). https://github.com/AnacondaRecipes/patchelf-feedstock/commit/822959cfe17086d7d9f92dc4e5b7b5b17878f132
@nehaljwani @jjhelmus, thanks for the rebuild. I tried the Anaconda patchelf build 3 as suggested:

```bash
#!/bin/bash

function dso_info(){
printf "Info for bad DSO:\n"
printf "===>OMP symbols in DT_NEEDED:"
readelf -d $OFFENDING_DSO|grep omp
printf "===>OMP symbols in Versioned:"
readelf -WV $OFFENDING_DSO|grep omp
printf "===>ELF linter output:"
eu-elflint $OFFENDING_DSO
printf "\n"
}

OFFENDING_DSO="offending_dso.so"
ENVNAME='_patchpatchedelf'
conda remove -n $ENVNAME --all -y
conda create -n $ENVNAME patchelf -y
source activate $ENVNAME

echo "----------------------------------------------------------------------"
echo "Using conda patchelf build 3:"

which patchelf


# get file, demo issue
cp <path>/envs/numba_wheel_40/lib/python3.7/site-packages/numba/npyufunc/omppool.cpython-37m-x86_64-linux-gnu.so.ORIG $OFFENDING_DSO
dso_info

# patch
printf "Patching...\n"
patchelf --replace-needed  libgomp-24cd7153.so.1.0.0 libgomp.so.1 $OFFENDING_DSO
patchelf --replace-needed  libgomp.so.1 libgomp-24cd7153.so.1.0.0 $OFFENDING_DSO

# check if fixed
dso_info

# clean up
source deactivate $ENVNAME
rm $OFFENDING_DSO

echo "----------------------------------------------------------------------"
echo "Using source build:"

PATCHELF_FROM_SOURCE="<path>/src/patchelf"
# get file, demo issue
cp <path>/envs/numba_wheel_40/lib/python3.7/site-packages/numba/npyufunc/omppool.cpython-37m-x86_64-linux-gnu.so.ORIG $OFFENDING_DSO
dso_info

# patch
printf "Patching...\n"
$PATCHELF_FROM_SOURCE --replace-needed  libgomp-24cd7153.so.1.0.0 libgomp.so.1 $OFFENDING_DSO
$PATCHELF_FROM_SOURCE --replace-needed  libgomp.so.1 libgomp-24cd7153.so.1.0.0 $OFFENDING_DSO

# now fixed.
dso_info
```
and it gave this:
```
$ bash omp_issue.sh 

Remove all packages in environment <path>/envs/_patchpatchedelf:


## Package Plan ##

  environment location: <path>/envs/_patchpatchedelf


The following packages will be REMOVED:

    libgcc-ng:    8.2.0-hdf63c60_1
    libstdcxx-ng: 8.2.0-hdf63c60_1
    patchelf:     0.9-he6710b0_3  

Solving environment: done

## Package Plan ##

  environment location: <path>/envs/_patchpatchedelf

  added / updated specs: 
    - patchelf


The following NEW packages will be INSTALLED:

    libgcc-ng:    8.2.0-hdf63c60_1
    libstdcxx-ng: 8.2.0-hdf63c60_1
    patchelf:     0.9-he6710b0_3  

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate _patchpatchedelf
#
# To deactivate an active environment, use
#
#     $ conda deactivate

----------------------------------------------------------------------
Using conda patchelf build 3:
<path>/envs/_patchpatchedelf/bin/patchelf
Info for bad DSO:
===>OMP symbols in DT_NEEDED: 0x0000000000000001 (NEEDED)             Shared library: [libgomp-24cd7153.so.1.0.0]
===>OMP symbols in Versioned:  0x0090: Version: 1  File: libgomp.so.1  Cnt: 2
===>ELF linter output:section [ 5] '.gnu.version_r': entry 1 references unknown dependency
section [34] '.symtab': _DYNAMIC symbol size 0 does not match dynamic segment size 544
section [34] '.symtab': _GLOBAL_OFFSET_TABLE_ symbol size 0 does not match .got.plt section size 392
loadable segment [7] is writable but contains no writable sections

Patching...
Info for bad DSO:
===>OMP symbols in DT_NEEDED: 0x0000000000000001 (NEEDED)             Shared library: [libgomp-24cd7153.so.1.0.0]
===>OMP symbols in Versioned:  0x0090: Version: 1  File: libgomp.so.1  Cnt: 2
===>ELF linter output:section [ 5] '.gnu.version_r': entry 1 references unknown dependency
section [34] '.symtab': _DYNAMIC symbol size 0 does not match dynamic segment size 544
section [34] '.symtab': _GLOBAL_OFFSET_TABLE_ symbol size 0 does not match .got.plt section size 392
loadable segment [7] is writable but contains no writable sections
loadable segment [8] is writable but contains no writable sections
loadable segment [9] is writable but contains no writable sections

----------------------------------------------------------------------
Using source build:
Info for bad DSO:
===>OMP symbols in DT_NEEDED: 0x0000000000000001 (NEEDED)             Shared library: [libgomp-24cd7153.so.1.0.0]
===>OMP symbols in Versioned:  0x0090: Version: 1  File: libgomp.so.1  Cnt: 2
===>ELF linter output:section [ 5] '.gnu.version_r': entry 1 references unknown dependency
section [34] '.symtab': _DYNAMIC symbol size 0 does not match dynamic segment size 544
section [34] '.symtab': _GLOBAL_OFFSET_TABLE_ symbol size 0 does not match .got.plt section size 392
loadable segment [7] is writable but contains no writable sections

Patching...
Info for bad DSO:
===>OMP symbols in DT_NEEDED: 0x0000000000000001 (NEEDED)             Shared library: [libgomp-24cd7153.so.1.0.0]
===>OMP symbols in Versioned:  0x0090: Version: 1  File: libgomp-24cd7153.so.1.0.0  Cnt: 2
===>ELF linter output:phdr[2]: cannot get content of note section: invalid section flags
section [32] '.symtab': _DYNAMIC symbol size 0 does not match dynamic segment size 544
section [32] '.symtab': _GLOBAL_OFFSET_TABLE_ symbol size 0 does not match .got.plt section size 392
loadable segment [7] is writable but contains no writable sections
loadable segment [8] is writable but contains no writable sections
loadable segment [9] is writable but contains no writable sections
```
so it looks like that build is still not patching the versioned symbols section of the ELF correctly whereas the source build is. The offending DSO can be obtained from `pip install numba=0.40.0` and fishing out the file `site-packages/numba/npyufunc/omppool.cpython-37m-x86_64-linux-gnu.so`.
Update:

* patchelf build from HEAD works
* to re-enable omp in wheel, we need to update setup.py to remove the check on `is_building_wheel()`
ping? :)
@slayoo thanks for the ping. I think we are just waiting on someone to submit a PR for the `setup.py`.
Using the `patchelf` from `main|defaults` I get:

```
----------------------------------------------------------------------
Using conda patchelf build 3:
/home/vhaenel/miniconda3/envs/_patchpatchedelf/bin/patchelf
Info for DSO:
===>OMP symbols in DT_NEEDED: 0x0000000000000001 (NEEDED)             Shared library: [libgomp-24cd7153.so.1.0.0]
===>OMP symbols in Versioned:  0x0090: Version: 1  File: libgomp.so.1  Cnt: 2
===>ELF linter output:section [ 5] '.gnu.version_r': entry 1 references unknown dependency
section [ 6] '.rela.dyn': relocation 0: invalid type
section [ 6] '.rela.dyn': relocation 1: invalid type
section [ 6] '.rela.dyn': relocation 2: invalid type
section [ 6] '.rela.dyn': relocation 3: invalid type
section [ 6] '.rela.dyn': relocation 4: invalid type
section [ 6] '.rela.dyn': relocation 5: invalid type
section [ 6] '.rela.dyn': relocation 6: invalid type
section [ 6] '.rela.dyn': relocation 7: invalid type
section [ 6] '.rela.dyn': relocation 8: invalid type
section [ 6] '.rela.dyn': relocation 9: invalid type
section [ 6] '.rela.dyn': relocation 10: invalid type
section [ 6] '.rela.dyn': relocation 11: invalid type
section [ 6] '.rela.dyn': relocation 12: invalid type
section [ 6] '.rela.dyn': relocation 13: invalid type
section [ 6] '.rela.dyn': relocation 14: invalid type
section [ 6] '.rela.dyn': relocation 15: invalid type
section [ 7] '.rela.plt': relocation 0: invalid type
section [ 7] '.rela.plt': relocation 1: invalid type
section [ 7] '.rela.plt': relocation 2: invalid type
section [ 7] '.rela.plt': relocation 3: invalid type
section [ 7] '.rela.plt': relocation 4: invalid type
section [ 7] '.rela.plt': relocation 5: invalid type
section [ 7] '.rela.plt': relocation 6: invalid type
section [ 7] '.rela.plt': relocation 7: invalid type
section [ 7] '.rela.plt': relocation 8: invalid type
section [ 7] '.rela.plt': relocation 9: invalid type
section [ 7] '.rela.plt': relocation 10: invalid type
section [ 7] '.rela.plt': relocation 11: invalid type
section [ 7] '.rela.plt': relocation 12: invalid type
section [ 7] '.rela.plt': relocation 13: invalid type
section [ 7] '.rela.plt': relocation 14: invalid type
section [ 7] '.rela.plt': relocation 15: invalid type
section [ 7] '.rela.plt': relocation 16: invalid type
section [ 7] '.rela.plt': relocation 17: invalid type
section [ 7] '.rela.plt': relocation 18: invalid type
section [ 7] '.rela.plt': relocation 19: invalid type
section [ 7] '.rela.plt': relocation 20: invalid type
section [ 7] '.rela.plt': relocation 21: invalid type
section [ 7] '.rela.plt': relocation 22: invalid type
section [ 7] '.rela.plt': relocation 23: invalid type
section [ 7] '.rela.plt': relocation 24: invalid type
section [ 7] '.rela.plt': relocation 25: invalid type
section [ 7] '.rela.plt': relocation 26: invalid type
section [ 7] '.rela.plt': relocation 27: invalid type
section [ 7] '.rela.plt': relocation 28: invalid type
section [ 7] '.rela.plt': relocation 29: invalid type
section [ 7] '.rela.plt': relocation 30: invalid type
section [ 7] '.rela.plt': relocation 31: invalid type
section [ 7] '.rela.plt': relocation 32: invalid type
section [ 7] '.rela.plt': relocation 33: invalid type
section [ 7] '.rela.plt': relocation 34: invalid type
section [ 7] '.rela.plt': relocation 35: invalid type
section [ 7] '.rela.plt': relocation 36: invalid type
section [ 7] '.rela.plt': relocation 37: invalid type
section [ 7] '.rela.plt': relocation 38: invalid type
section [ 7] '.rela.plt': relocation 39: invalid type
section [ 7] '.rela.plt': relocation 40: invalid type
section [ 7] '.rela.plt': relocation 41: invalid type
section [ 7] '.rela.plt': relocation 42: invalid type
section [ 7] '.rela.plt': relocation 43: invalid type
section [ 7] '.rela.plt': relocation 44: invalid type
section [ 7] '.rela.plt': relocation 45: invalid type
section [34] '.symtab': _DYNAMIC symbol size 0 does not match dynamic segment size 544
section [34] '.symtab': _GLOBAL_OFFSET_TABLE_ symbol size 0 does not match .got.plt section size 392
loadable segment [7] is writable but contains no writable sections

Patching...
Info for DSO:
===>OMP symbols in DT_NEEDED: 0x0000000000000001 (NEEDED)             Shared library: [libgomp-24cd7153.so.1.0.0]
===>OMP symbols in Versioned:  0x0090: Version: 1  File: libgomp-24cd7153.so.1.0.0  Cnt: 2
===>ELF linter output:phdr[2]: unknown object file note type 364544 with owner name '' at offset 24
phdr[2]: extra 12 bytes after last note
section [ 4] '.rela.dyn': relocation 0: invalid type
section [ 4] '.rela.dyn': relocation 1: invalid type
section [ 4] '.rela.dyn': relocation 2: invalid type
section [ 4] '.rela.dyn': relocation 3: invalid type
section [ 4] '.rela.dyn': relocation 4: invalid type
section [ 4] '.rela.dyn': relocation 5: invalid type
section [ 4] '.rela.dyn': relocation 6: invalid type
section [ 4] '.rela.dyn': relocation 7: invalid type
section [ 4] '.rela.dyn': relocation 8: invalid type
section [ 4] '.rela.dyn': relocation 9: invalid type
section [ 4] '.rela.dyn': relocation 10: invalid type
section [ 4] '.rela.dyn': relocation 11: invalid type
section [ 4] '.rela.dyn': relocation 12: invalid type
section [ 4] '.rela.dyn': relocation 13: invalid type
section [ 4] '.rela.dyn': relocation 14: invalid type
section [ 4] '.rela.dyn': relocation 15: invalid type
section [ 5] '.rela.plt': relocation 0: invalid type
section [ 5] '.rela.plt': relocation 1: invalid type
section [ 5] '.rela.plt': relocation 2: invalid type
section [ 5] '.rela.plt': relocation 3: invalid type
section [ 5] '.rela.plt': relocation 4: invalid type
section [ 5] '.rela.plt': relocation 5: invalid type
section [ 5] '.rela.plt': relocation 6: invalid type
section [ 5] '.rela.plt': relocation 7: invalid type
section [ 5] '.rela.plt': relocation 8: invalid type
section [ 5] '.rela.plt': relocation 9: invalid type
section [ 5] '.rela.plt': relocation 10: invalid type
section [ 5] '.rela.plt': relocation 11: invalid type
section [ 5] '.rela.plt': relocation 12: invalid type
section [ 5] '.rela.plt': relocation 13: invalid type
section [ 5] '.rela.plt': relocation 14: invalid type
section [ 5] '.rela.plt': relocation 15: invalid type
section [ 5] '.rela.plt': relocation 16: invalid type
section [ 5] '.rela.plt': relocation 17: invalid type
section [ 5] '.rela.plt': relocation 18: invalid type
section [ 5] '.rela.plt': relocation 19: invalid type
section [ 5] '.rela.plt': relocation 20: invalid type
section [ 5] '.rela.plt': relocation 21: invalid type
section [ 5] '.rela.plt': relocation 22: invalid type
section [ 5] '.rela.plt': relocation 23: invalid type
section [ 5] '.rela.plt': relocation 24: invalid type
section [ 5] '.rela.plt': relocation 25: invalid type
section [ 5] '.rela.plt': relocation 26: invalid type
section [ 5] '.rela.plt': relocation 27: invalid type
section [ 5] '.rela.plt': relocation 28: invalid type
section [ 5] '.rela.plt': relocation 29: invalid type
section [ 5] '.rela.plt': relocation 30: invalid type
section [ 5] '.rela.plt': relocation 31: invalid type
section [ 5] '.rela.plt': relocation 32: invalid type
section [ 5] '.rela.plt': relocation 33: invalid type
section [ 5] '.rela.plt': relocation 34: invalid type
section [ 5] '.rela.plt': relocation 35: invalid type
section [ 5] '.rela.plt': relocation 36: invalid type
section [ 5] '.rela.plt': relocation 37: invalid type
section [ 5] '.rela.plt': relocation 38: invalid type
section [ 5] '.rela.plt': relocation 39: invalid type
section [ 5] '.rela.plt': relocation 40: invalid type
section [ 5] '.rela.plt': relocation 41: invalid type
section [ 5] '.rela.plt': relocation 42: invalid type
section [ 5] '.rela.plt': relocation 43: invalid type
section [ 5] '.rela.plt': relocation 44: invalid type
section [ 5] '.rela.plt': relocation 45: invalid type
section [32] '.symtab': _DYNAMIC symbol size 0 does not match dynamic segment size 544
section [32] '.symtab': _GLOBAL_OFFSET_TABLE_ symbol size 0 does not match .got.plt section size 392
section [36] '.note.gnu.build-id': cannot get content of note section
loadable segment [7] is writable but contains no writable sections
loadable segment [8] is writable but contains no writable sections
loadable segment [9] is writable but contains no writable sections
```
The `eu-elflint` warning `section [ 5] '.gnu.version_r': entry 1 references unknown dependency` is no longer showing up after patching.
Looks fixed, thanks for checking.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -76,14 +76,6 @@ def is_building():
     return True
 
 
-def is_building_wheel():
-    if len(sys.argv) < 2:
-        # No command is given.
-        return False
-
-    return 'bdist_wheel' in sys.argv[1:]
-
-
 def get_ext_modules():
     """
     Return a list of Extension instances for the setup() call.
@@ -238,9 +230,8 @@ def check_file_at_path(path2file):
     else:
         print("TBB not found")
 
-    # Disable OpenMP if we are building a wheel or
-    # forced by user with NUMBA_NO_OPENMP=1
-    if is_building_wheel() or os.getenv('NUMBA_NO_OPENMP'):
+    # Disable OpenMP if forced by user with NUMBA_NO_OPENMP=1
+    if os.getenv('NUMBA_NO_OPENMP'):
         print("OpenMP disabled")
     elif have_openmp:
         print("Using OpenMP from:", have_openmp)


Test patch: