Repository: TileDB-Inc/TileDB-Py

Issue:
array.query no longer works for string attributes
I have a case that looks basically like this:

```
array = tiledb.DenseArray(uri, mode="r")
q = array.query(attrs=["attrname"])
data = q[:]
```
This works when the type of the attribute is a float or an int.
But if the type of the attribute is a string, this fails with an exception:

```
  File "tiledb/libtiledb.pyx", line 3874, in tiledb.libtiledb.Query.__getitem__
  File "tiledb/libtiledb.pyx", line 4107, in tiledb.libtiledb.DenseArrayImpl.subarray
  File "tiledb/libtiledb.pyx", line 4156, in tiledb.libtiledb.DenseArrayImpl._read_dense_subarray
  File "tiledb/libtiledb.pyx", line 3760, in tiledb.libtiledb.Array._unpack_varlen_query
  File "tiledb/libtiledb.pyx", line 3760, in tiledb.libtiledb.Array._unpack_varlen_query
  File "tiledb/libtiledb.pyx", line 3805, in tiledb.libtiledb.Array._unpack_varlen_query
SystemError: Negative size passed to PyUnicode_FromStringAndSize
```

This worked in v1.7.6, and I ran into this problem when porting to 2.0.0.
If there is a better way to pull out just one attribute from the array, then
I'm open to suggestions.

[example.zip](https://github.com/TileDB-Inc/TileDB-Py/files/4589870/example.zip)

Attached is a script, output from 1.7.6, and output from 2.0.0






Discussion:
Followup:  I misdiagnosed the problem.  

It is not just with query, but also with indexing into the array.
My array was created with 1.7.6, if that matters.  It has 5 attributes, 2 of which are strings.  I found that slicing into the array works up to a point.  I did a binary search to
see exactly where it fails.  My array has 1305994 rows, but I can only slice up to 1305008.

Let me know I you'd like me to send the array.

```
array = tiledb.DenseArray(uri, mode="r", ctx=ctx)
array.schema.dump()
print(array.shape)
print(array[:1305008])
print(array[:1305009])
```

output:

```
- Array type: dense
- Cell order: row-major
- Tile order: row-major
- Capacity: 10000
- Allows duplicates: false
- Coordinates filters: 1
  > ZSTD: COMPRESSION_LEVEL=-1
- Offsets filters: 1
  > ZSTD: COMPRESSION_LEVEL=-1

### Dimension ###
- Name: __dim_0
- Type: UINT32
- Cell val num: 1
- Domain: [0,1305993]
- Tile extent: 1000
- Filters: 0

### Attribute ###
- Name: name_0
- Type: STRING_UTF8
- Cell val num: var
- Filters: 1
  > ZSTD: COMPRESSION_LEVEL=22

### Attribute ###
- Name: n_counts_all
- Type: FLOAT32
- Cell val num: 1
- Filters: 1
  > ZSTD: COMPRESSION_LEVEL=22

### Attribute ###
- Name: n_counts
- Type: FLOAT32
- Cell val num: 1
- Filters: 1
  > ZSTD: COMPRESSION_LEVEL=22

### Attribute ###
- Name: louvain
- Type: STRING_UTF8
- Cell val num: var
- Filters: 1
  > ZSTD: COMPRESSION_LEVEL=22

### Attribute ###
- Name: n_genes
- Type: INT32
- Cell val num: 1
- Filters: 1
  > ZSTD: COMPRESSION_LEVEL=22



(1305994,)

OrderedDict([('name_0', array(['AAACCTGAGATAGGAG-1', 'AAACCTGAGCGGCTTC-1', 'AAACCTGAGGAATCGC-1',
       ..., 'TGCGTGGAGAAACCGC-133', 'TGCGTGGAGAATAGGG-133',
       'TGCGTGGAGATGTCGG-133'], dtype=object)), ('n_counts_all', array([4046., 2087., 4654., ..., 2575., 4662., 3479.], dtype=float32)), ('n_counts', array([ 793.81116,  935.5113 ,  658.3262 , ..., 1326.0614 ,  997.1257 ,
       1113.8953 ], dtype=float32)), ('louvain', array(['5', '2', '6', ..., '12', '26', '19'], dtype=object)), ('n_genes', array([ 85,  76, 114, ...,  67,  79, 103], dtype=int32))])

Traceback (most recent call last):
  File "server/test/tiledb1.py", line 20, in <module>
    print(array[:1305009])
  File "tiledb/libtiledb.pyx", line 4002, in tiledb.libtiledb.DenseArrayImpl.__getitem__
  File "tiledb/libtiledb.pyx", line 4107, in tiledb.libtiledb.DenseArrayImpl.subarray
  File "tiledb/libtiledb.pyx", line 4156, in tiledb.libtiledb.DenseArrayImpl._read_dense_subarray
  File "tiledb/libtiledb.pyx", line 3760, in tiledb.libtiledb.Array._unpack_varlen_query
  File "tiledb/libtiledb.pyx", line 3760, in tiledb.libtiledb.Array._unpack_varlen_query
  File "tiledb/libtiledb.pyx", line 3805, in tiledb.libtiledb.Array._unpack_varlen_query
SystemError: Negative size passed to PyUnicode_FromStringAndSize
```
Hi @bmccandless,
If array is shareable that would be very helpful (email if you prefer not
to post). Otherwise, if you can show the repr of ‘array.schema’ (which now
returns a valid python expression) then I will try to repro with synthetic
data.


Resolving patch:
diff --git a/tiledb/multirange_indexing.py b/tiledb/multirange_indexing.py
--- a/tiledb/multirange_indexing.py
+++ b/tiledb/multirange_indexing.py
@@ -128,7 +128,7 @@ def __getitem__(self, idx):
 
         for name, item in result_dict.items():
             if len(item[1]) > 0:
-                arr = self.array._unpack_varlen_query(item, name)
+                arr = q.unpack_buffer(name, item[0], item[1])
             else:
                 arr = item[0]
                 arr.dtype = schema.attr_or_dim_dtype(name)


Test patch:
diff --git a/tiledb/tests/test_libtiledb.py b/tiledb/tests/test_libtiledb.py
--- a/tiledb/tests/test_libtiledb.py
+++ b/tiledb/tests/test_libtiledb.py
@@ -1227,6 +1227,45 @@ def test_incomplete_dense(self):
             res_idx = A[:]
             assert_array_equal(res_idx, data)
 
+    def test_incomplete_dense_varlen(self):
+        ncells = 100
+
+        path = self.path("incomplete_dense_varlen")
+        str_data = [rand_utf8(random.randint(0, n)) for n in range(ncells)]
+        data = np.array(str_data, dtype=np.unicode_)
+
+        # basic write
+        ctx = tiledb.Ctx()
+        dom = tiledb.Domain(tiledb.Dim(domain=(1, len(data)), tile=len(data), ctx=ctx), ctx=ctx)
+        att = tiledb.Attr(dtype=np.unicode_, var=True, ctx=ctx)
+
+        schema = tiledb.ArraySchema(dom, (att,), ctx=ctx)
+
+        tiledb.DenseArray.create(path, schema)
+        with tiledb.DenseArray(path, mode='w', ctx=ctx) as T:
+            T[:] = data
+
+        with tiledb.DenseArray(path, mode='r', ctx=ctx) as T:
+            assert_array_equal(data, T[:])
+
+        # set the memory to the max length of a cell
+        # these settings force ~100 retries
+        # TODO would be good to check repeat count here; not yet exposed
+        #      Also would be useful to have max cell config in libtiledb.
+        init_buffer_bytes = 1024**2
+        config = tiledb.Config({'sm.memory_budget': ncells,
+                                'sm.memory_budget_var': ncells,
+                                'py.init_buffer_bytes': init_buffer_bytes })
+
+        ctx2 = tiledb.Ctx(config=config)
+        self.assertEqual(
+            config['py.init_buffer_bytes'],
+            str(init_buffer_bytes)
+        )
+
+        with tiledb.DenseArray(path, mode='r', ctx=ctx2) as T2:
+            assert_array_equal(data, T2[:])
+
     def test_written_fragment_info(self):
         uri = self.path("test_written_fragment_info")