Repository: scikit-hep/awkward

Issue:
Sliced arrays yields inconsistent values in numba jitted function
When passing sliced arrays to a numba jitted function, I'm receiving inconsistent values in the array. It's probably easiest to see in the reproducer code:

```python
import awkward1 as ak
import numba as nb

parquet_arrays = ak.from_parquet("AnalysisResults.18q.parquet")
# zip because I usually have other arrays
arrays = ak.zip({"parent_index": parquet_arrays["matched.fJetSplittings.fParentIndex"]})

@nb.njit
def reproduce(arrays):
    i = 0
    for values in arrays:
        print("======== i =", i)
        parent_indices = values.parent_index
        print("parent_indices", parent_indices)
        j = 0
        for p in parent_indices:
            print(j, ":", p)
            assert p == parent_indices[j]  # How??
            j += 1
        i += 1
```

This gives a truncated output of:

```
>>> reproduce(arrays[:, :2])
======== i = 0
parent_indices []
======== i = 1
parent_indices [-1]
0 : -1
======== i = 2
parent_indices [-1]
0 : -1
======== i = 3
parent_indices [-1, 0]
0 : 1
1 : 2
======== i = 4
parent_indices []
======== i = 5
parent_indices [-1, 0]
0 : 0
1 : -1
...
```

The trouble starts at i = 3. Based on the printed values from `parent_indices`, it should be 0 : -1, 1: 0 (checked outside of the jitted function). Instead, the values are incremented by 2. Then for i = 5, the values are switched. Some observations:

- `njit` -> `jit` has the same issue.
- Asserting the `p` values vs parent_indices at the index claims that they agree.
- If I drop the numba decorator, everything works fine.
- If I remove the `:2` slice, everything works fine.

For reference, the truncated expected output:

```
======== i = 0
parent_indices []
======== i = 1
parent_indices [-1]
0 : -1
======== i = 2
parent_indices [-1]
0 : -1
======== i = 3
parent_indices [-1, 0]
0 : -1
1 : 0
======== i = 4
parent_indices []
======== i = 5
parent_indices [-1, 0]
0 : -1
1 : 0
...
```

and the layout:
```python
>>> arrays.parent_index.layout
<ListOffsetArray64>
    <offsets><Index64 i="[0 0 1 2 8 ... 47597 47598 47598 47598 47599]" offset="0" length="25110" at="0x7fcb16199000"/></offsets>
    <content><NumpyArray format="h" shape="47599" data="-1 -1 -1 0 1 ... 0 -1 -1 -1 -1" at="0x7fcb162b2000"/></content>
</ListOffsetArray64>
```

Unfortunately, I'm only able to reproduce this via a stored file. Maybe it's something about parquet, or maybe I didn't come up with a complex enough example. I need to keep it private, so I'm emailing it to you (it's similar but not identical to the file that I sent for the uproot4 issue last week). It was basically produced with:

```python
import uproot4 as uproo4
tree = uproot.open("AnalysisResults.root")
arrays = tree.arrays(["branch1", ...])
ak.to_parquet(arrays, "AnalysisResults.18q.parquet")
```

I can also provide the original root file if it helps. Thanks!


Discussion:
It's a bug.

Here's a simpler reproducer:

```python
import awkward1 as ak
import numpy as np
import numba as nb

numpyarray = ak.layout.NumpyArray(np.arange(100, 200, 10))
indexedarray = ak.layout.IndexedArray64(ak.layout.Index64(np.array([0, 1, 2, 3, 8, 9])), numpyarray)
listoffsetarray = ak.layout.ListOffsetArray64(ak.layout.Index64(np.array([0, 0, 1, 2, 4, 4, 6])), indexedarray)
array = ak.Array(listoffsetarray)

def reproduce(arrays):
    i = 0
    for values in arrays:
        print("======== i =", i)
        parent_indices = values
        print("parent_indices", parent_indices)
        j = 0
        for p in parent_indices:
            print(j, ":", p)
            assert p == parent_indices[j]  # How??
            j += 1
        i += 1

print(array.tolist())
reproduce(array)
nb.njit(reproduce)(array)
```

Output:

```
[[], [100], [110], [120, 130], [], [180, 190]]
======== i = 0
parent_indices []
======== i = 1
parent_indices [100]
0 : 100
======== i = 2
parent_indices [110]
0 : 110
======== i = 3
parent_indices [120, 130]
0 : 120
1 : 130
======== i = 4
parent_indices []
======== i = 5
parent_indices [180, 190]
0 : 180
1 : 190
======== i = 0
parent_indices []
======== i = 1
parent_indices [100]
0 : 100
======== i = 2
parent_indices [110]
0 : 120
======== i = 3
parent_indices [120, 130]
0 : 140
1 : 150
======== i = 4
parent_indices []
======== i = 5
parent_indices [180, 190]
0 : 93967057837984
1 : 0
```

The IndexedArray is essential—without that, there's no bug. There's also no error if the IndexedArray is used alone, not nested within a ListOffsetArray. Apparently, the Numba implementation of the IndexedArray iterator (`for p in parent_indices`) is not correctly managing the indexing between ListOffsetArray and IndexedArray. I'll start there.

(The `print` statement in a Numba-compiled function acquires the GIL and creates a Python context, so that's why it looked okay in `print`: the bug is in the Numba implementation, which isn't used in `print`.)
A ListOffsetArray is being added twice. These even-simpler reproducers show it:

```python
def reproduce(arrays):
    for values in arrays:
        print("values", values)
        j = 0
        for p in values:
            print(j, ":", p)
            assert p == values[j]  # How??
            j += 1
```

with

```python
numpyarray = ak.layout.NumpyArray(np.arange(100, 200, 10))
indexedarray = ak.layout.IndexedArray64(ak.layout.Index64(np.array([0, 1, 2, 3, 4, 5])), numpyarray)
listoffsetarray = ak.layout.ListOffsetArray64(ak.layout.Index64(np.array([0, 1, 5])), indexedarray)
array = ak.Array(listoffsetarray)

print(array.tolist())
print("\nNo Numba")
reproduce(array)
print("\nYes Numba")
nb.njit(reproduce)(array)
```

prints

```python
[[100], [110, 120, 130, 140]]

No Numba
values [100]
0 : 100
values [110, 120, 130, 140]
0 : 110
1 : 120
2 : 130
3 : 140

Yes Numba
values [100]
0 : 100
values [110, 120, 130, 140]
0 : 120
1 : 130
2 : 140
3 : 150
```

but

```python
numpyarray = ak.layout.NumpyArray(np.arange(100, 200, 10))
indexedarray = ak.layout.IndexedArray64(ak.layout.Index64(np.array([0, 1, 2, 3, 4, 5])), numpyarray)
listoffsetarray = ak.layout.ListOffsetArray64(ak.layout.Index64(np.array([0, 2, 5])), indexedarray)
array = ak.Array(listoffsetarray)

print(array.tolist())
print("\nNo Numba")
reproduce(array)
print("\nYes Numba")
nb.njit(reproduce)(array)
```

prints

```
[[100, 110], [120, 130, 140]]

No Numba
values [100, 110]
0 : 100
1 : 110
values [120, 130, 140]
0 : 120
1 : 130
2 : 140

Yes Numba
values [100, 110]
0 : 100
1 : 110
values [120, 130, 140]
0 : 140
1 : 150
2 : 160
```

The only difference between these two is that the first has ListOffsetArray offsets of `[0, 1, 5]` and the second has `[0, 2, 5]` (second list starts one later). In the first case, the second list starts `1` too late; in the second case, the second list starts `2` too late.

Resolving patch:
diff --git a/src/awkward1/_connect/_numba/layout.py b/src/awkward1/_connect/_numba/layout.py
--- a/src/awkward1/_connect/_numba/layout.py
+++ b/src/awkward1/_connect/_numba/layout.py
@@ -935,12 +935,12 @@ def lower_getitem_at(
         )
         proxynext = context.make_helper(builder, nextviewtype)
         proxynext.pos = nextpos
-        proxynext.start = viewproxy.start
+        proxynext.start = context.get_constant(numba.intp, 0)
         proxynext.stop = builder.add(
             awkward1._connect._numba.castint(
                 context, builder, self.indextype.dtype, numba.intp, nextat
             ),
-            builder.add(viewproxy.start, context.get_constant(numba.intp, 1)),
+            context.get_constant(numba.intp, 1),
         )
         proxynext.arrayptrs = viewproxy.arrayptrs
         proxynext.sharedptrs = viewproxy.sharedptrs


Test patch:
diff --git a/tests/test_0395-fix-numba-indexedarray.py b/tests/test_0395-fix-numba-indexedarray.py
new file mode 100644
--- /dev/null
+++ b/tests/test_0395-fix-numba-indexedarray.py
@@ -0,0 +1,88 @@
+# BSD 3-Clause License; see https://github.com/scikit-hep/awkward-1.0/blob/master/LICENSE
+
+from __future__ import absolute_import
+
+import sys
+
+import pytest
+
+import numpy
+import awkward1
+
+numba = pytest.importorskip("numba")
+
+
+def test():
+    def reproduce(arrays):
+        out = numpy.zeros(len(arrays), numpy.int64)
+        i = 0
+        for values in arrays:
+            for p in values:
+                out[i] = p
+                i += 1
+                break
+        return out
+
+    numpyarray = awkward1.layout.NumpyArray(numpy.arange(100, 200, 10))
+    indexedarray = awkward1.layout.IndexedArray64(awkward1.layout.Index64(numpy.array([5, 4, 3, 2, 1, 0])), numpyarray)
+    listoffsetarray = awkward1.layout.ListOffsetArray64(awkward1.layout.Index64(numpy.array([0, 1, 4])), indexedarray)
+    array = awkward1.Array(listoffsetarray)
+
+    assert reproduce(array).tolist() == [150, 140]
+    assert numba.njit(reproduce)(array).tolist() == [150, 140]
+
+    numpyarray = awkward1.layout.NumpyArray(numpy.arange(100, 200, 10))
+    indexedarray = awkward1.layout.IndexedArray64(awkward1.layout.Index64(numpy.array([5, 4, 3, 2, 1, 0])), numpyarray)
+    listoffsetarray = awkward1.layout.ListOffsetArray64(awkward1.layout.Index64(numpy.array([0, 2, 4])), indexedarray)
+    array = awkward1.Array(listoffsetarray)
+
+    assert reproduce(array).tolist() == [150, 130]
+    assert numba.njit(reproduce)(array).tolist() == [150, 130]
+
+    numpyarray = awkward1.layout.NumpyArray(numpy.arange(100, 200, 10))
+    indexedarray = awkward1.layout.IndexedArray64(awkward1.layout.Index64(numpy.array([5, 4, 3, 2, 1, 0])), numpyarray)[2:]
+    listoffsetarray = awkward1.layout.ListOffsetArray64(awkward1.layout.Index64(numpy.array([0, 1, 4])), indexedarray)
+    array = awkward1.Array(listoffsetarray)
+
+    assert reproduce(array).tolist() == [130, 120]
+    assert numba.njit(reproduce)(array).tolist() == [130, 120]
+
+    numpyarray = awkward1.layout.NumpyArray(numpy.arange(100, 200, 10))
+    indexedarray = awkward1.layout.IndexedArray64(awkward1.layout.Index64(numpy.array([5, 4, 3, 2, 1, 0])), numpyarray)[2:]
+    listoffsetarray = awkward1.layout.ListOffsetArray64(awkward1.layout.Index64(numpy.array([0, 2, 4])), indexedarray)
+    array = awkward1.Array(listoffsetarray)
+
+    assert reproduce(array).tolist() == [130, 110]
+    assert numba.njit(reproduce)(array).tolist() == [130, 110]
+
+    numpyarray = awkward1.layout.NumpyArray(numpy.arange(100, 200, 10))[3:]
+    indexedarray = awkward1.layout.IndexedArray64(awkward1.layout.Index64(numpy.array([5, 4, 3, 2, 1, 0])), numpyarray)
+    listoffsetarray = awkward1.layout.ListOffsetArray64(awkward1.layout.Index64(numpy.array([0, 1, 4])), indexedarray)
+    array = awkward1.Array(listoffsetarray)
+
+    assert reproduce(array).tolist() == [180, 170]
+    assert numba.njit(reproduce)(array).tolist() == [180, 170]
+
+    numpyarray = awkward1.layout.NumpyArray(numpy.arange(100, 200, 10))[3:]
+    indexedarray = awkward1.layout.IndexedArray64(awkward1.layout.Index64(numpy.array([5, 4, 3, 2, 1, 0])), numpyarray)
+    listoffsetarray = awkward1.layout.ListOffsetArray64(awkward1.layout.Index64(numpy.array([0, 2, 4])), indexedarray)
+    array = awkward1.Array(listoffsetarray)
+
+    assert reproduce(array).tolist() == [180, 160]
+    assert numba.njit(reproduce)(array).tolist() == [180, 160]
+
+    numpyarray = awkward1.layout.NumpyArray(numpy.arange(100, 200, 10))[3:]
+    indexedarray = awkward1.layout.IndexedArray64(awkward1.layout.Index64(numpy.array([5, 4, 3, 2, 1, 0])), numpyarray)[2:]
+    listoffsetarray = awkward1.layout.ListOffsetArray64(awkward1.layout.Index64(numpy.array([0, 1, 4])), indexedarray)
+    array = awkward1.Array(listoffsetarray)
+
+    assert reproduce(array).tolist() == [160, 150]
+    assert numba.njit(reproduce)(array).tolist() == [160, 150]
+
+    numpyarray = awkward1.layout.NumpyArray(numpy.arange(100, 200, 10))[3:]
+    indexedarray = awkward1.layout.IndexedArray64(awkward1.layout.Index64(numpy.array([5, 4, 3, 2, 1, 0])), numpyarray)[2:]
+    listoffsetarray = awkward1.layout.ListOffsetArray64(awkward1.layout.Index64(numpy.array([0, 2, 4])), indexedarray)
+    array = awkward1.Array(listoffsetarray)
+
+    assert reproduce(array).tolist() == [160, 140]
+    assert numba.njit(reproduce)(array).tolist() == [160, 140]