Repository: scikit-hep/awkward

Issue:
Regression: reading parquet file leads to ByteMaskedArray length error
### Version of Awkward Array

1.10.0

### Description and code to reproduce

Using data from http://s3.amazonaws.com/datashader-data/nyc_buildings.parq.zip and the following code
```python
import awkward._v2 as ak
ak.from_parquet("/home/data/hv/nyc_buildings.parq")
```
works fine with awkward 1.9.0 but not 1.10.0 onwards, giving this error:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/iant/.venv/awk/lib/python3.10/site-packages/awkward/_v2/operations/ak_from_parquet.py", line 74, in from_parquet
    return _load(
  File "/home/iant/.venv/awk/lib/python3.10/site-packages/awkward/_v2/operations/ak_from_parquet.py", line 250, in _load
    return ak._v2.operations.ak_concatenate._impl(
  File "/home/iant/.venv/awk/lib/python3.10/site-packages/awkward/_v2/operations/ak_concatenate.py", line 121, in _impl
    out = batch[0].mergemany(batch[1:])
  File "/home/iant/.venv/awk/lib/python3.10/site-packages/awkward/_v2/contents/recordarray.py", line 668, in mergemany
    merged = forfield[0].mergemany(forfield[1:])
  File "/home/iant/.venv/awk/lib/python3.10/site-packages/awkward/_v2/contents/bytemaskedarray.py", line 669, in mergemany
    return ByteMaskedArray(
  File "/home/iant/.venv/awk/lib/python3.10/site-packages/awkward/_v2/contents/bytemaskedarray.py", line 80, in __init__
    raise ak._v2._util.error(
ValueError: while calling (from <stdin>, line 1)

    ak._v2.from_parquet(
        path = '/home/data/hv/nyc_buildings.parq'
        columns = None
        row_groups = None
        storage_options = None
        max_gap = 64000
        max_block = 256000000
        footer_sample_size = 1000000
        generate_bitmasks = False
        highlevel = True
        behavior = None
    )

Error details: ByteMaskedArray len(mask) (1157859) must be <= len(content) (1007927)
```
Data file was created using `spatialpandas` and `pyarrow` quite a long time ago.

Installed packages:
```
$ pip list
Package    Version
---------- --------
awkward    1.10.0
fsspec     2022.8.2
numpy      1.23.3
packaging  21.3
pip        22.0.2
pyarrow    9.0.0
pyparsing  3.0.9
setuptools 59.6.0
```
If I then
```
$ pip uninstall awkward
$ pip install awkward==1.9.0
```
it works fine.


Discussion:
So far, I've narrowed it down: it's something in the concatenation, the last step of making an array out of a dataset directory of Parquet files. Moreover, you only need a RecordArray of one field, and no symptom if the field is completely projected out.

That is,

```python
>>> import awkward as ak   # this is _v2
>>> one = ak.from_parquet("nyc_buildings.parq/part.0.parquet")
>>> two = ak.from_parquet("nyc_buildings.parq/part.1.parquet")
>>> ak.concatenate([one, two])
```

reproduces the error, but in `ak.concatenate` instead of the final stages of `ak.from_parquet`.

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jpivarski/irishep/awkward/awkward/operations/ak_concatenate.py", line 48, in concatenate
    return _impl(arrays, axis, merge, mergebool, highlevel, behavior)
  File "/Users/jpivarski/irishep/awkward/awkward/operations/ak_concatenate.py", line 116, in _impl
    out = batch[0].mergemany(batch[1:])
  File "/Users/jpivarski/irishep/awkward/awkward/contents/recordarray.py", line 665, in mergemany
    merged = forfield[0].mergemany(forfield[1:])
  File "/Users/jpivarski/irishep/awkward/awkward/contents/bytemaskedarray.py", line 661, in mergemany
    return ByteMaskedArray(
  File "/Users/jpivarski/irishep/awkward/awkward/contents/bytemaskedarray.py", line 80, in __init__
    raise ak._util.error(
ValueError: while calling (from <stdin>, line 1)

    ak.concatenate(
        arrays = [<Array [{geometry: [[[...]]], ...}, ..., {...}] type='31579...
        axis = 0
        merge = True
        mergebool = True
        highlevel = True
        behavior = None
    )

Error details: ByteMaskedArray len(mask) (66665) must be <= len(content) (63158)
```

This (making RecordArrays of only one field, `"name"`) also raises the error:

```python
>>> ak.concatenate([one[["name"]], two[["name"]]])
```

but this does not:

```python
>>> ak.concatenate([one["name"], two["name"]])
<Array [None, None, None, None, ..., None, None, None] type='66665 * ?string'>
```

So the existence of a RecordArray is important. The types of `one` and `two` are the same:

```python
>>> one.type.show()
31579 * {
    geometry: option[var * option[var * option[var * ?float64]]],
    type: ?string,
    name: ?string,
    hilbert_distance: ?int64
}
>>> two.type.show()
35086 * {
    geometry: option[var * option[var * option[var * ?float64]]],
    type: ?string,
    name: ?string,
    hilbert_distance: ?int64
}
```

and even the Forms of `one[["name"]]` and `two[["name"]]` (which has the error-symptom) are the same:

```python
>>> one[["name"]].layout
<RecordArray is_tuple='false' len='31579'>
    <content index='0' field='name'>
        <BitMaskedArray valid_when='true' lsb_order='true' len='31579'>
            <mask><Index dtype='uint8' len='3948'>[0 0 0 ... 0 0 0]</Index></mask>
            <content><ListOffsetArray len='31579'>
                <parameter name='__array__'>'string'</parameter>
                <offsets><Index dtype='int32' len='31580'>
                    [   0    0    0 ... 3491 3491 3491]
                </Index></offsets>
                <content><NumpyArray dtype='uint8' len='3491'>
                    <parameter name='__array__'>'char'</parameter>
                    [ 85 110 105 ...  97 114 121]
                </NumpyArray></content>
            </ListOffsetArray></content>
        </BitMaskedArray>
    </content>
</RecordArray>
>>> two[["name"]].layout
<RecordArray is_tuple='false' len='35086'>
    <content index='0' field='name'>
        <BitMaskedArray valid_when='true' lsb_order='true' len='35086'>
            <mask><Index dtype='uint8' len='4386'>[0 0 0 ... 0 0 0]</Index></mask>
            <content><ListOffsetArray len='35086'>
                <parameter name='__array__'>'string'</parameter>
                <offsets><Index dtype='int32' len='35087'>
                    [  0   0   0 ... 581 581 581]
                </Index></offsets>
                <content><NumpyArray dtype='uint8' len='581'>
                    <parameter name='__array__'>'char'</parameter>
                    [ 86  97 110 100 101 114  98 105 108 116  32  97 116  32
                      83 111 117 116 104  32  66 101  97  99 104  67 101 110
                     ...
                     111 115 116  32  79 102 102 105  99 101  66 117 114 103
                     101 114  32  75 105 110 103]
                </NumpyArray></content>
            </ListOffsetArray></content>
        </BitMaskedArray>
    </content>
</RecordArray>
```

But the projected out layouts, `one["name"]` and `two["name"]`, do not have the error-symptom:

```python
>>> one["name"].layout
<BitMaskedArray valid_when='true' lsb_order='true' len='31579'>
    <mask><Index dtype='uint8' len='3948'>[0 0 0 ... 0 0 0]</Index></mask>
    <content><ListOffsetArray len='31579'>
        <parameter name='__array__'>'string'</parameter>
        <offsets><Index dtype='int32' len='31580'>
            [   0    0    0 ... 3491 3491 3491]
        </Index></offsets>
        <content><NumpyArray dtype='uint8' len='3491'>
            <parameter name='__array__'>'char'</parameter>
            [ 85 110 105 ...  97 114 121]
        </NumpyArray></content>
    </ListOffsetArray></content>
</BitMaskedArray>
>>> two["name"].layout
<BitMaskedArray valid_when='true' lsb_order='true' len='35086'>
    <mask><Index dtype='uint8' len='4386'>[0 0 0 ... 0 0 0]</Index></mask>
    <content><ListOffsetArray len='35086'>
        <parameter name='__array__'>'string'</parameter>
        <offsets><Index dtype='int32' len='35087'>
            [  0   0   0 ... 581 581 581]
        </Index></offsets>
        <content><NumpyArray dtype='uint8' len='581'>
            <parameter name='__array__'>'char'</parameter>
            [ 86  97 110 100 101 114  98 105 108 116  32  97 116  32  83 111
             117 116 104  32  66 101  97  99 104  67 101 110 116 101 114  32
             ...
             111 115 116  32  79 102 102 105  99 101  66 117 114 103 101 114
              32  75 105 110 103]
        </NumpyArray></content>
    </ListOffsetArray></content>
</BitMaskedArray>
```

and their BitMaskedArray lengths, as well as the lengths of their bit masks, are the same between `x[["name"]]` and `x["name"]`. One of these is triggering the error-symptom while the other doesn't, even though they have the same lengths.

Also, in the error message,

```
Error details: ByteMaskedArray len(mask) (66665) must be <= len(content) (63158)
```

I'd predict a ByteMaskedArray length of `66665`, by adding the `len` of each BitMaskedArray: `31579 + 35086`. (It's not the length of the bit mask because the mask gets expanded by a factor of 8 when converting from bit mask to byte mask, and that byte mask might be a little too big because it's the first multiple of 8 larger than the true length. That's why the BitMaskedArray _separately_ stores its length, and those add up.)

I'd predict the `len(content)` to also be `66665`, by adding the `len` of each ListOffsetArray: `31579 + 35086`. (With ListOffsetArray, beware of looking at the `offsets` length, since that's always one longer than the ListOffsetArray.)

So the `len(content)` somehow got wrong. Where does `63158` come from? It's a lot smaller than the size you'd get by adding the two contents, and I don't see why it would be _any_ smaller.

And why does it depend on https://github.com/scikit-hep/awkward/compare/v1.9.0...v1.10.0? The way parameters affect merging was changed, but we see the same error-symptom if we use the `"hilbert_distance"` field, rather than `"name"`, and this field has no parameters at all. (`"name"` has an `__array__ = "string"` parameter.)

```python
>>> one[["hilbert_distance"]]
<Array [{hilbert_distance: 1078494}, ...] type='31579 * {hilbert_distance: ...'>
>>> one[["hilbert_distance"]].layout
<RecordArray is_tuple='false' len='31579'>
    <content index='0' field='hilbert_distance'>
        <BitMaskedArray valid_when='true' lsb_order='true' len='31579'>
            <mask><Index dtype='uint8' len='3948'>[255 255 255 ... 255 255   7]</Index></mask>
            <content><NumpyArray dtype='int64' len='31579'>
                [  1078494   1087198   1189938 ... 113899921 113899975
                 113899983]
            </NumpyArray></content>
        </BitMaskedArray>
    </content>
</RecordArray>
>>> two[["hilbert_distance"]].layout
<RecordArray is_tuple='false' len='35086'>
    <content index='0' field='hilbert_distance'>
        <BitMaskedArray valid_when='true' lsb_order='true' len='35086'>
            <mask><Index dtype='uint8' len='4386'>[255 255 255 ... 255 255  63]</Index></mask>
            <content><NumpyArray dtype='int64' len='35086'>
                [113900034 113900330 113900368 ... 127703116 127703404
                 127703509]
            </NumpyArray></content>
        </BitMaskedArray>
    </content>
</RecordArray>
```

Could it be in the way that broadcasting changed? Concatenation at `axis=0` shouldn't be doing any broadcasting.

In `RecordArray._mergemany` (lines 582‒583 in src/awkward/contents/recordarray.py), this changed:

```diff
        for_each_field = []
        for field in self.contents:
            trimmed = field[0 : self.length]
-           for_each_field.append([field])
+           for_each_field.append([trimmed])
```

But it ought to be the right thing to do.

Testing it, **that did make a difference.** With this one change reverted, it does not show the error-symptom. But this is weird because we really should be trimming to the length of the RecordArray. In fact, it's exactly the same: at that point in the code, `len(field)`, `len(trimmed)`, and `self.length` are all `31579`. (`self` is only the first in the concatenation, `one[["name"]]`. The rest are trimmed a few lines later.)

Very weird—it's trimmed to its own length, so it should be no change at all, and yet, that's what makes the content 3507 items shorter two levels of recursion further down. It's probably not this fix specifically—I think this fix is correct—but it might be revealing something else wrong with slicing. It looks like `field[0 : self.length]` is not doing what it's supposed to do.

I'm going to have to leave it at that for right now.
In v2, we added a new implementation for `ByteMaskedArray.mergemany`, which previously was using `toIndexedOptionArray`. I *think* we have some bugs here with respect to length handling, as the following fixes the test case:

```diff
diff --git a/src/awkward/contents/bytemaskedarray.py b/src/awkward/contents/bytemaskedarray.py
index e8a2f173..ae8b6bfb 100644
--- a/src/awkward/contents/bytemaskedarray.py
+++ b/src/awkward/contents/bytemaskedarray.py
@@ -649,13 +649,13 @@ class ByteMaskedArray(Content):
             for x in others
         ):
             parameters = self._parameters
-            masks = [self._mask.data]
+            masks = [self._mask.data[:self.length]]
             tail_contents = []
             length = 0
             for x in others:
                 parameters = ak._util.merge_parameters(parameters, x._parameters, True)
-                masks.append(x._mask.data)
-                tail_contents.append(x._content[: self.length])
+                masks.append(x._mask.data[: x.length])
+                tail_contents.append(x._content[: x.length])
                 length += x.length
 
             return ByteMaskedArray(

```

I've not properly thought about this patch, it just looks like we're not properly trimming everything / using the wrong lengths. 
We can keep that in mind, but at the end of my long debugging session, I found that the RecordArray

```diff
        for_each_field = []
        for field in self.contents:
            trimmed = field[0 : self.length]
-           for_each_field.append([field])
+           for_each_field.append([trimmed])
```

toggled the error-symptom, and this is doing a slice that shouldn't be changing anything. I guess the next step would be to see what _does_ happen to the layout of `field` when it becomes `trimmed`. Given that `self.length` is equal to `len(field)` in this case, `field` and `trimmed` ought to be exactly identical. But somehow it changes the outcome.

The changes you point out in ByteMaskedArray also look like corrections/improvements.
@jpivarski to start with, yes we currently have a bug:

In `main`, we are not properly handling the case where the `len(mask) != len(contents)`, as we were only trimming the contents, not the masks. You can see that we trigger (a) this bug by manually crafting such a layout:

```python
import awkward as ak
import numpy as np

x = ak.contents.RecordArray(
    [ak.contents.ByteMaskedArray(
        ak.index.Index8(np.r_[0, 1, 1]),
        ak.contents.NumpyArray(np.arange(12)),
        valid_when=True,
    )],
    ["x"],
)
y = ak.contents.RecordArray(
    [ak.contents.ByteMaskedArray(
        ak.index.Index8(np.r_[1, 1, 1, 0, 0]),
        ak.contents.NumpyArray(np.arange(12)),
        valid_when=True,
    )],
    ["x"],
)
z = ak.concatenate((x, y))

```
I don't think that RecordArray is at fault here. I think it's simply introducing the _valid_ state that `ByteMaskedArray` cannot properly handle (see #1749). 

What's happening in the above reproducer is that 1ebf1dbe732b6b634fab95c8406498a59b71695f adds a slice, which leads to the bit masked array becoming a byte masked array, where we now have this buggy code path. You can see this by modifying `recordarray.py`:
```diff
diff --git a/src/awkward/contents/recordarray.py b/src/awkward/contents/recordarray.py
index 2923b645..2f9a0194 100644
--- a/src/awkward/contents/recordarray.py
+++ b/src/awkward/contents/recordarray.py
@@ -580,7 +580,7 @@ class RecordArray(Content):
         for_each_field = []
         for field in self.contents:
             trimmed = field[0 : self.length]
-            for_each_field.append([trimmed])
+            for_each_field.append([field.toBitMaskedArray(True, True)])
 
         if self.is_tuple:
             for array in headless:
```
which no longer fails.

Resolving patch:
diff --git a/src/awkward/_v2/contents/bytemaskedarray.py b/src/awkward/_v2/contents/bytemaskedarray.py
--- a/src/awkward/_v2/contents/bytemaskedarray.py
+++ b/src/awkward/_v2/contents/bytemaskedarray.py
@@ -655,15 +655,15 @@ def mergemany(self, others):
             for x in others
         ):
             parameters = self._parameters
-            masks = [self._mask.data]
+            masks = [self._mask.data[: self.length]]
             tail_contents = []
             length = 0
             for x in others:
                 parameters = ak._v2._util.merge_parameters(
                     parameters, x._parameters, True
                 )
-                masks.append(x._mask.data)
-                tail_contents.append(x._content[: self.length])
+                masks.append(x._mask.data[: x.length])
+                tail_contents.append(x._content[: x.length])
                 length += x.length
 
             return ByteMaskedArray(


Test patch:
diff --git a/tests/v2/test_1747-bytemaskedarray-mergemany.py b/tests/v2/test_1747-bytemaskedarray-mergemany.py
new file mode 100644
--- /dev/null
+++ b/tests/v2/test_1747-bytemaskedarray-mergemany.py
@@ -0,0 +1,22 @@
+# BSD 3-Clause License; see https://github.com/scikit-hep/awkward-1.0/blob/main/LICENSE
+
+import numpy as np  # noqa: F401
+import pytest  # noqa: F401
+
+import awkward as ak  # noqa: F401
+
+
+def test():
+    x = ak._v2.contents.ByteMaskedArray(
+        ak._v2.index.Index8(np.r_[0, 1, 1]),
+        ak._v2.contents.NumpyArray(np.arange(12)),
+        valid_when=True,
+    )
+    y = ak._v2.contents.ByteMaskedArray(
+        ak._v2.index.Index8(np.r_[1, 1, 1, 0, 0]),
+        ak._v2.contents.NumpyArray(np.arange(12)),
+        valid_when=True,
+    )
+    z = x.merge(y)
+
+    assert z.to_list() == [None, 1, 2, 0, 1, 2, None, None]