Repository: scikit-hep/awkward

Issue:
Simplify output of {Bit,Byte}MaskedArray
At the moment, v1 calls `simplify_optiontype` before returning from particular `getitem` methods in these content types, whilst v2 does not. Is this a planned change, or do we need to restore these?

https://github.com/scikit-hep/awkward-1.0/blob/fca80a69b23b2f5b2d952cc03fee722d9a867665/src/awkward/_v2/contents/bytemaskedarray.py#L161-L210

vs

https://github.com/scikit-hep/awkward-1.0/blob/fca80a69b23b2f5b2d952cc03fee722d9a867665/src/libawkward/array/ByteMaskedArray.cpp#L664-L752


Discussion:
It's not a planned change. Failing to include `simplify_optiontype` is a bug, and the symptom would be that the operation that's missing the `simplify_optiontype` has any of the following node types as a direct `content` of any of the following node types:

   * UnmaskedArray
   * BitMaskedArray
   * ByteMaskedArray
   * IndexedOptionArray
   * IndexedArray

That is, any of the 25 combinations like `UnmaskedArray(UnmaskedArray(XYZ))`, `UnmaskedArray(BitMaskedArray(XYZ))`, ... `IndexedArray(IndexedArray(XYZ))` are all not allowed in arrays that return to the user: they must be eliminated by a call to `simplify_optiontype` before that happens. (They can't be excluded in the constructors of these types because they're allowed as temporary states in the midst of a calculation, before being returned to the user.)

I can't think of any situation where having an unnecessary `simplify_optiontype` does harm, but missing one is bad. If we need to implement a generic

```python
class Content:
    ...
    def simplify_optiontype(self):
        return self
```

so that node types other than the above 5 don't complain about the method not existing, then that's fine. It would be better from a code maintenance point of view to not call `simplify_optiontype` where we know it's useless, because then the code would be suggesting something false.

Also, there are no situations where we'd want `simplify_optiontype` in v1 but not v2 or in v2 but not v1. If one of the two versions has this as a bug, the other should get it fixed. We may not be adding major new features to v1, but we _are_ fixing bugs.

(Also also, it might seem strange that IndexedArray is included in the set of node types that `simplify_optiontype` should be applied to, even though IndexedArray does not have option type. It's because of the way that it mixes with IndexedOptionArray. We don't want arrays with `ByteMaskedArray(IndexedArray(XYZ))` because it would use less memory and suffer less indirection to be `IndexedOptionArray(XYZ)`. IndexedArray has the cost of extra indirection, but the benefit of delaying expensive calculations or minimizing memory use or both, but `ByteMaskedArray(IndexedArray(XYZ))` doesn't have either of these two advantages.)

Is there a minimal reproducer that demonstrates it, showing a different layout in v1 and v2? (Maybe in #1249 that I haven't looked at yet...)
Yes, uncommenting a ported v1 test in #1249 led me to report this, and fix it in #1259
I was just looking at that, trying to type it up. I was disturbed by this output:

```
arr1       = <Array [{a: 1, b: 1}, {a: 2, b: None}] type='2 * {a: int64, b: ?int64}'>
arr2       = <Array [{a: 1, b: 1}, {a: 2, b: None}] type='2 * ?{a: int64, b: ?int64}'>
```

because those are supposed to be different types; a failure of `simplify_optiontype` would show up as double question marks (`??`), not this. But now I realize that's not what the test is complaining about, it's just useful information about two variables that are defined at the time of the test—the test is not asserting that they are the same or that they should be.
Here's the reproducer:

```python
>>> v1 = ak.Array(ak.layout.ByteMaskedArray(
...     ak.layout.Index8(np.array([True, True])),
...     ak.Array({"a": [1, 2], "b": [1, None]}).layout,
...     valid_when=True)
... )
>>> v1
<Array [{a: 1, b: 1}, {a: 2, b: None}] type='2 * ?{"a": int64, "b": ?int64}'>
>>> v1.b
<Array [1, None] type='2 * ?int64'>

>>> v2 = ak._v2.Array(ak._v2.contents.ByteMaskedArray(
...     ak._v2.index.Index8(np.array([True, True])),
...     ak._v2.Array({"a": [1, 2], "b": [1, None]}).layout,
...     valid_when=True)
... )
>>> v2
<Array [{a: 1, b: 1}, {a: 2, b: None}] type='2 * ?{a: int64, b: ?int64}'>
>>> v2.b
<Array [1, None] type='2 * ??int64'>
```

v1 is right and v2 is wrong (double `??`).

Resolving patch:
diff --git a/src/awkward/_v2/contents/bitmaskedarray.py b/src/awkward/_v2/contents/bitmaskedarray.py
--- a/src/awkward/_v2/contents/bitmaskedarray.py
+++ b/src/awkward/_v2/contents/bitmaskedarray.py
@@ -271,7 +271,7 @@ def _getitem_field(self, where, only_fields=()):
             self._field_identifier(where),
             None,
             self._nplike,
-        )
+        ).simplify_optiontype()
 
     def _getitem_fields(self, where, only_fields=()):
         return BitMaskedArray(
@@ -283,7 +283,7 @@ def _getitem_fields(self, where, only_fields=()):
             self._fields_identifier(where),
             None,
             self._nplike,
-        )
+        ).simplify_optiontype()
 
     def _carry(self, carry, allow_lazy, exception):
         assert isinstance(carry, ak._v2.index.Index)
diff --git a/src/awkward/_v2/contents/bytemaskedarray.py b/src/awkward/_v2/contents/bytemaskedarray.py
--- a/src/awkward/_v2/contents/bytemaskedarray.py
+++ b/src/awkward/_v2/contents/bytemaskedarray.py
@@ -197,7 +197,7 @@ def _getitem_field(self, where, only_fields=()):
             self._field_identifier(where),
             None,
             self._nplike,
-        )
+        ).simplify_optiontype()
 
     def _getitem_fields(self, where, only_fields=()):
         return ByteMaskedArray(
@@ -207,7 +207,7 @@ def _getitem_fields(self, where, only_fields=()):
             self._fields_identifier(where),
             None,
             self._nplike,
-        )
+        ).simplify_optiontype()
 
     def _carry(self, carry, allow_lazy, exception):
         assert isinstance(carry, ak._v2.index.Index)


Test patch:
diff --git a/tests/v2/test_1260-simplify-masked-option-types.py b/tests/v2/test_1260-simplify-masked-option-types.py
new file mode 100644
--- /dev/null
+++ b/tests/v2/test_1260-simplify-masked-option-types.py
@@ -0,0 +1,51 @@
+# BSD 3-Clause License; see https://github.com/scikit-hep/awkward-1.0/blob/main/LICENSE
+
+
+import pytest  # noqa: F401
+import numpy as np  # noqa: F401
+import awkward as ak  # noqa: F401
+
+
+def test_byte_masked_array():
+    array = ak._v2.highlevel.Array(
+        ak._v2.contents.ByteMaskedArray(
+            ak._v2.index.Index8(np.array([True, True])),
+            ak._v2.contents.RecordArray(
+                [
+                    ak._v2.contents.NumpyArray(np.arange(2)),
+                    ak._v2.contents.IndexedOptionArray(
+                        ak._v2.index.Index64(np.array([0, -1])),
+                        ak._v2.contents.NumpyArray(np.array([1], dtype=np.int64)),
+                    ),
+                ],
+                ["a", "b"],
+            ),
+            # ak._v2.Array({"a": [1, 2], "b": [1, None]}).layout,
+            valid_when=True,
+        )
+    )
+
+    assert str(array.b.type) == "2 * ?int64"
+
+
+def test_bit_masked_array():
+    array = ak._v2.highlevel.Array(
+        ak._v2.contents.BitMaskedArray(
+            ak._v2.index.IndexU8(np.array([1 << 0 | 1 << 1], dtype=np.uint8)),
+            ak._v2.contents.RecordArray(
+                [
+                    ak._v2.contents.NumpyArray(np.arange(2)),
+                    ak._v2.contents.IndexedOptionArray(
+                        ak._v2.index.Index64(np.array([0, -1])),
+                        ak._v2.contents.NumpyArray(np.array([1], dtype=np.int64)),
+                    ),
+                ],
+                ["a", "b"],
+            ),
+            valid_when=True,
+            length=2,
+            lsb_order=True,
+        )
+    )
+
+    assert str(array.b.type) == "2 * ?int64"