Repository: TileDB-Inc/TileDB-Py

Issue:
sporadic failure on Python 3.8
We are periodically seeing the following error on Python 3.8, after an apparently successful pip install of tiledb in a clean virtual environment.  In particular, it occurs regularly in our CI tests (Linux/Ubuntu), and I've seen it once on Windows/WSL/Ubuntu.  

~~I haven't found a reliable reproducible test case yet.~~

Update: found a test.  Will include it in a second comment.


```
Python 3.8.2 (default, Feb 26 2020, 04:23:39) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tiledb
Traceback (most recent call last):
  File "/mnt/c/Users/bruce/projects/cellxgene/venv38/lib/python3.8/site-packages/tiledb/__init__.py", line 20, in <module>
    from .libtiledb import Ctx
ImportError: libtiledb.so.1.7: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/c/Users/bruce/projects/cellxgene/venv38/lib/python3.8/site-packages/tiledb/__init__.py", line 24, in <module>
    ctypes.CDLL(os.path.join(lib_dir, lib_name))
  File "/usr/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /mnt/c/Users/bruce/projects/cellxgene/venv38/lib/python3.8/site-packages/tiledb/native/libtiledb.so: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/c/Users/bruce/projects/cellxgene/venv38/lib/python3.8/site-packages/tiledb/__init__.py", line 27, in <module>
    ctypes.CDLL(lib_name)
  File "/usr/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libtiledb.so: cannot open shared object file: No such file or directory

```

After this error, I confirmed that the shared library is missing as stated in the error.  When I deleted the virtual environment, and recreated it, everything worked fine.

There are example failures in our CI tests [example](https://github.com/chanzuckerberg/cellxgene/runs/499525064), where it shows stuff like:

```
2020-03-11T02:23:51.9743921Z Traceback (most recent call last):
2020-03-11T02:23:51.9748172Z   File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/tiledb/__init__.py", line 20, in <module>
2020-03-11T02:23:51.9749227Z     from .libtiledb import Ctx
2020-03-11T02:23:51.9751419Z ImportError: libtiledb.so.1.7: cannot open shared object file: No such file or directory
2020-03-11T02:23:51.9751796Z 
2020-03-11T02:23:51.9752201Z During handling of the above exception, another exception occurred:
2020-03-11T02:23:51.9752484Z 
2020-03-11T02:23:51.9752839Z Traceback (most recent call last):
2020-03-11T02:23:51.9754072Z   File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/tiledb/__init__.py", line 24, in <module>
2020-03-11T02:23:51.9754571Z     ctypes.CDLL(os.path.join(lib_dir, lib_name))
2020-03-11T02:23:51.9755577Z   File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/ctypes/__init__.py", line 373, in __init__
2020-03-11T02:23:51.9756018Z     self._handle = _dlopen(self._name, mode)
2020-03-11T02:23:51.9757000Z OSError: /opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/tiledb/native/libtiledb.so: cannot open shared object file: No such file or directory
2020-03-11T02:23:51.9757350Z 
2020-03-11T02:23:51.9757724Z During handling of the above exception, another exception occurred:
2020-03-11T02:23:51.9757998Z 
2020-03-11T02:23:51.9758348Z Traceback (most recent call last):
2020-03-11T02:23:51.9758820Z   File "/opt/hostedtoolcache/Python/3.8.2/x64/bin/cellxgene", line 11, in <module>
2020-03-11T02:23:51.9759618Z     load_entry_point('cellxgene==0.14.0', 'console_scripts', 'cellxgene')()
2020-03-11T02:23:51.9760578Z   File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 829, in __call__
2020-03-11T02:23:51.9761661Z     return self.main(*args, **kwargs)
2020-03-11T02:23:51.9762605Z   File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 782, in main
2020-03-11T02:23:51.9763058Z     rv = self.invoke(ctx)
2020-03-11T02:23:51.9763952Z   File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
2020-03-11T02:23:51.9764428Z     return _process_result(sub_ctx.command.invoke(sub_ctx))
2020-03-11T02:23:51.9765332Z   File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
2020-03-11T02:23:51.9765798Z     return ctx.invoke(self.callback, **ctx.params)
2020-03-11T02:23:51.9766691Z   File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/click/core.py", line 610, in invoke
2020-03-11T02:23:51.9767165Z     return callback(*args, **kwargs)
2020-03-11T02:23:51.9768069Z   File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/server/cli/launch.py", line 69, in wrapper
2020-03-11T02:23:51.9768532Z     return func(*args, **kwargs)
2020-03-11T02:23:51.9769436Z   File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/server/cli/launch.py", line 116, in wrapper
2020-03-11T02:23:51.9769893Z     return func(*args, **kwargs)
2020-03-11T02:23:51.9770775Z   File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/server/cli/launch.py", line 152, in wrapper
2020-03-11T02:23:51.9771237Z     return func(*args, **kwargs)
2020-03-11T02:23:51.9771746Z   File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/server/cli/launch.py", line 202, in wrapper
2020-03-11T02:23:51.9772002Z     return func(*args, **kwargs)
2020-03-11T02:23:51.9772527Z   File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/server/cli/launch.py", line 233, in wrapper
2020-03-11T02:23:51.9772969Z     return func(*args, **kwargs)
2020-03-11T02:23:51.9773489Z   File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/server/cli/launch.py", line 322, in launch
2020-03-11T02:23:51.9773752Z     matrix_data_loader = MatrixDataLoader(datapath)
2020-03-11T02:23:51.9774288Z   File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/server/data_common/matrix_loader.py", line 152, in __init__
2020-03-11T02:23:51.9774561Z     from server.data_cxg.cxg_adaptor import CxgAdaptor
2020-03-11T02:23:51.9775168Z   File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/server/data_cxg/cxg_adaptor.py", line 9, in <module>
2020-03-11T02:23:51.9775446Z     import tiledb
2020-03-11T02:23:51.9775964Z   File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/site-packages/tiledb/__init__.py", line 27, in <module>
2020-03-11T02:23:51.9776219Z     ctypes.CDLL(lib_name)
2020-03-11T02:23:51.9776491Z   File "/opt/hostedtoolcache/Python/3.8.2/x64/lib/python3.8/ctypes/__init__.py", line 373, in __init__
2020-03-11T02:23:51.9776741Z     self._handle = _dlopen(self._name, mode)
2020-03-11T02:23:51.9776950Z OSError: libtiledb.so: cannot open shared object file: No such file or directory
```

The install looked normal (highlights here):
```
2020-03-11T02:15:07.0693249Z Collecting tiledb>=0.5.3
2020-03-11T02:15:07.0883409Z   Downloading tiledb-0.5.6.tar.gz (100 kB)
<snip>
2020-03-11T02:15:25.6759796Z     Running setup.py install for tiledb: started
2020-03-11T02:16:34.2381601Z     Running setup.py install for tiledb: still running...
2020-03-11T02:18:01.8356121Z     Running setup.py install for tiledb: still running...
2020-03-11T02:19:02.3270758Z     Running setup.py install for tiledb: still running...
2020-03-11T02:20:04.6740979Z     Running setup.py install for tiledb: still running...
2020-03-11T02:21:06.6901161Z     Running setup.py install for tiledb: still running...
2020-03-11T02:22:20.4495865Z     Running setup.py install for tiledb: still running...
2020-03-11T02:22:21.0442019Z     Running setup.py install for tiledb: finished with status 'done'
```

I'll update this if I find a test case.




Discussion:
Here is the test case I found - it requires disabling the pip cache.

This was done on Windows/WSL/Ubuntu, which should be more-or-less same as ubuntu (and I see the failure on actual ubuntu in my CI pipeline).

```
# python3.8 -m venv venv38
# source venv38/bin/activate
(venv38) # pip install tiledb --no-cache-dir
Collecting tiledb
  Downloading https://files.pythonhosted.org/packages/b4/1a/d54f1a426149114a651753c89646fb4e0562d2c399084a32632ea33ca34c/tiledb-0.5.6.tar.gz (100kB)
     |████████████████████████████████| 102kB 1.9MB/s 
Collecting numpy>=1.7 (from tiledb)
  Downloading https://files.pythonhosted.org/packages/41/38/b278d96baebc6a4818cfd9c0fb6f0e62013d5b87374bcf0f14a0e9b83ed5/numpy-1.18.1-cp38-cp38-manylinux1_x86_64.whl (20.6MB)
     |████████████████████████████████| 20.6MB 5.2MB/s 
Collecting wheel>=0.30 (from tiledb)
  Downloading https://files.pythonhosted.org/packages/8c/23/848298cccf8e40f5bbb59009b32848a4c38f4e7f3364297ab3c3e2e2cd14/wheel-0.34.2-py2.py3-none-any.whl
Installing collected packages: numpy, wheel, tiledb
  Running setup.py install for tiledb ... done
Successfully installed numpy-1.18.1 tiledb-0.5.6 wheel-0.34.2
WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(venv38) # python
Python 3.8.2 (default, Feb 26 2020, 04:23:39) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tiledb
Traceback (most recent call last):
  File "/mnt/c/Users/bruce/projects/cellxgene/venv38/lib/python3.8/site-packages/tiledb/__init__.py", line 20, in <module>
    from .libtiledb import Ctx
ImportError: libtiledb.so.1.7: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/c/Users/bruce/projects/cellxgene/venv38/lib/python3.8/site-packages/tiledb/__init__.py", line 24, in <module>
    ctypes.CDLL(os.path.join(lib_dir, lib_name))
  File "/usr/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /mnt/c/Users/bruce/projects/cellxgene/venv38/lib/python3.8/site-packages/tiledb/native/libtiledb.so: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/c/Users/bruce/projects/cellxgene/venv38/lib/python3.8/site-packages/tiledb/__init__.py", line 27, in <module>
    ctypes.CDLL(lib_name)
  File "/usr/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libtiledb.so: cannot open shared object file: No such file or directory
>>> 

(venv38) # python --version
Python 3.8.2
(venv38) # pip --version
pip 19.2.3 from /mnt/c/Users/bruce/projects/cellxgene/venv38/lib/python3.8/site-packages/pip (python 3.8)
(venv38) # pip list
Package    Version
---------- -------
numpy      1.18.1 
pip        19.2.3 
setuptools 41.2.0 
tiledb     0.5.6  
wheel      0.34.2 

```

Hi @bkmartinjr, I believe TileDB (core) is being built from source in this example, because we don't have a Python 3.8 linux wheel on PyPI yet. There were downstream issues when I last looked at Python 3.8, but I believe those are now resolved on all platforms. Barring any issues, I will put a wheel up first thing in the morning which should provide install-only usage (and hopefully speed up your CI as well).

btw, how often are the builds failing? The main reason I can think of for a libtiledb build failure, in an otherwise consistent environment, is network issues. However, all of our deps should come from github so I would expect that to be relatively rare.
I believe you are correct that a full build is occurring (it takes some time).  There are no errors produced by the build on stdout.

Builds are very consistently failing since we started them a week ago (we just added this test).  I don't think any have succeeded on github actions.

One interesting point - I can resolve the issue in my local environment by doing a `pip uninstall tiledb` followed by a `pip install tiledb`.

```
(venv38) # python
Python 3.8.2 (default, Feb 26 2020, 04:23:39) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tiledb
Traceback (most recent call last):
  File "/mnt/c/Users/bruce/projects/cellxgene/venv38/lib/python3.8/site-packages/tiledb/__init__.py", line 20, in <module>
    from .libtiledb import Ctx
ImportError: libtiledb.so.1.7: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/c/Users/bruce/projects/cellxgene/venv38/lib/python3.8/site-packages/tiledb/__init__.py", line 24, in <module>
    ctypes.CDLL(os.path.join(lib_dir, lib_name))
  File "/usr/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /mnt/c/Users/bruce/projects/cellxgene/venv38/lib/python3.8/site-packages/tiledb/native/libtiledb.so: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/c/Users/bruce/projects/cellxgene/venv38/lib/python3.8/site-packages/tiledb/__init__.py", line 27, in <module>
    ctypes.CDLL(lib_name)
  File "/usr/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libtiledb.so: cannot open shared object file: No such file or directory
>>> 
(venv38) # pip uninstall tiledb
Uninstalling tiledb-0.5.6:
  Would remove:
    /mnt/c/Users/bruce/projects/cellxgene/venv38/lib/python3.8/site-packages/tiledb-0.5.6-py3.8.egg-info
    /mnt/c/Users/bruce/projects/cellxgene/venv38/lib/python3.8/site-packages/tiledb/*
Proceed (y/n)? y
  Successfully uninstalled tiledb-0.5.6
(venv38) # pip install tiledb
Collecting tiledb
Requirement already satisfied: numpy>=1.7 in ./venv38/lib/python3.8/site-packages (from tiledb) (1.18.1)
Requirement already satisfied: wheel>=0.30 in ./venv38/lib/python3.8/site-packages (from tiledb) (0.34.2)
Installing collected packages: tiledb
Successfully installed tiledb-0.5.6
WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(venv38) # python
Python 3.8.2 (default, Feb 26 2020, 04:23:39) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tiledb
>>> 
(venv38) # 
```
Hi @bkmartinjr,
- python 3.8 pypi [wheel is up](https://files.pythonhosted.org/packages/d9/5c/fe0eb94a89be77d631a5870d7d48736f765b92173f4609e518e6ce39da79/tiledb-0.5.6-cp38-cp38-manylinux1_x86_64.whl), so `pip install tiledb` should be fast and consistent now in your CI environment. I will have a PR shortly to add that output to our build process. 
- for the build error, with ubuntu 18.04 docker and `apt install cmake build-essential`, it is failing in the AWS SDK build step -- most likely due to a missing dependency. To make this less cryptic in the future I will add some config options which will at least surface the underlying build error when `pip install -v` is used, and I will try add more checks and failure reporting in a non-verbose build (somewhat tricky to control because pip hides all stdout by default).
sounds great - thank you! 

I did go back and re-test with verbose pip output, but you had already landed the wheel :-)


@ihnorton - the presence of the wheel has resolved our build failures.  Feel free to close this when you see fit.  Thanks for the rapid resolution!

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -1,5 +1,6 @@
 from __future__ import absolute_import, print_function
 
+import ctypes
 import multiprocessing
 import os
 import shutil
@@ -289,12 +290,32 @@ def do_copy(src, dest):
         tiledb_ext.library_dirs += [os.path.join(install_dir, lib_subdir)]
         tiledb_ext.include_dirs += [os.path.join(install_dir, "include")]
         # Update package_data so the shared object gets installed with the Python module.
-        libtiledb_objects = [os.path.join(native_subdir, libname) for libname in libtiledb_library_names()]
+        libtiledb_objects = [os.path.join(native_subdir, libname)
+                             for libname in libtiledb_library_names()]
+
+        # Make sure the built library is usable
+        for shared_obj in libtiledb_objects:
+            if is_windows(): continue
+            test_path = os.path.join(TILEDB_PKG_DIR, shared_obj)
+            # should only ever be 1, not sure why libtiledb_library_names -> List
+            try:
+                ctypes.CDLL(test_path)
+            except:
+                print("\n-------------------")
+                print("Failed to load shared library: {}".format(test_path))
+                print("-------------------\n")
+                raise
+
+        # This needs to be a glob in order to pick up the versioned SO
+        libtiledb_objects = [so + '*' for so in libtiledb_objects]
+
         if is_windows():
             libtiledb_objects.extend(
                 [os.path.join(native_subdir, libname) for libname in
                               ["tiledb.lib", "tbb.dll", "tbb.lib"]])
+        print("\n-------------------")
         print("libtiledb_objects: ", libtiledb_objects)
+        print("-------------------\n")
         setuptools_cmd.distribution.package_data.update({"tiledb": libtiledb_objects})
 
 


Test patch: