Repository: scikit-hep/awkward

Issue:
`ak.is_none` does not descend into option contents
### Version of Awkward Array

1.7.0

### Description and code to reproduce

Currently, `ak.is_none` returns during the result building stage once it encounters an option type. This produces some interesting results when `ak.is_none` over nested option types:
  ```python3
  >>> index_of_index = ak.Array(
  ...     ak.layout.IndexedOptionArray64(
  ...         ak.layout.Index64(np.r_[0, 1, 2, 3]),
  ...         ak.layout.IndexedOptionArray64(
  ...             ak.layout.Index64(np.r_[0, -1, 2, 3]),
  ...             ak.layout.NumpyArray(np.r_[1, 2, 3, 4]),
  ...         ),
  ...     )
  ... )
  >>> index_of_index
  <Array [1, None, 5, 6] type='4 * ?union[?int64, ?int64]'>
  >>> ak.is_none(index_of_index)
  <Array [False, False, False, False] type='4 * bool'>
  ```

In these cases, it's visible in the type string that there is something funky going on with the layout, but the correct result would be a `NumpyArray` layout of union of bytemask arrays

It might be that the simplest solution here is to use the `transform_child_layouts` utility, and return the union of the option-layout bytemask and the result from `transform_child_layouts`, e.g.
```python3
def is_none(array, axis=0, highlevel=True, behavior=None):
    """
    Args:
        array: Data to check for missing values (None).
        axis (int): The dimension at which this operation is applied. The
            outermost dimension is `0`, followed by `1`, etc., and negative
            values count backward from the innermost: `-1` is the innermost
            dimension, `-2` is the next level up, etc.
        highlevel (bool): If True, return an #ak.Array; otherwise, return
            a low-level #ak.layout.Content subclass.
        behavior (None or dict): Custom #ak.behavior for the output array, if
            high-level.

    Returns an array whose value is True where an element of `array` is None;
    False otherwise (at a given `axis` depth).
    """

    def apply(layout, depth, posaxis):
        posaxis = layout.axis_wrap_if_negative(posaxis)
        if posaxis == depth - 1:
            nplike = ak.nplike.of(layout)
            if isinstance(layout, ak._util.optiontypes):
                mask = nplike.asarray(layout.bytemask()).view(np.bool_)

                inner_mask = ak.to_numpy(
                    ak._util.transform_child_layouts(apply, layout, depth, posaxis)
                )
                return ak.layout.NumpyArray(mask | inner_mask)

            if isinstance(
                layout,
                (
                    ak._util.unknowntypes,
                    ak._util.listtypes,
                    ak._util.recordtypes,
                    ak.layout.NumpyArray,
                ),
            ):
                return ak.layout.NumpyArray(nplike.zeros(len(layout), dtype=np.bool_))

        return ak._util.transform_child_layouts(apply, layout, depth, posaxis)

    layout = ak.operations.convert.to_layout(array)

    out = apply(layout, 1, axis)

    return ak._util.maybe_wrap_like(out, array, behavior, highlevel)
```

This feels a bit clunky, but it makes the point :)

Alternatively, we could just call `ak.packed` here, which will ensure that merge-able layouts are merged. This solution won't fix other cases, though, e.g.
```python3
ak.layout.IndexedOptionArray64(
    ak.layout.Index64(np.r_[0, 1, 2, 3]),
    ak.layout.UnionArray8_64(
        ak.layout.Index8(np.r_[0, 0, 1, 1]),
        ak.layout.Index64(np.r_[0, 1, 0, 1]),
        [
            ak.layout.IndexedOptionArray64(
                ak.layout.Index64(np.r_[0, -1, 2, 3]),
                ak.layout.NumpyArray(np.r_[1, 2, 3, 4]),
            ),
            ak.layout.NumpyArray(np.r_[4, 5, 6]),
        ],
    ),
)
```

Related to this: at present, multple option layouts grow the type string `??` corresponds to two nested options. Is this what we want? I would expect to see a single logical option (`any(l.optional for l in layouts_at_depth)`).
`ak.is_none` does not descend into option contents
### Version of Awkward Array

1.7.0

### Description and code to reproduce

Currently, `ak.is_none` returns during the result building stage once it encounters an option type. This produces some interesting results when `ak.is_none` over nested option types:
  ```python3
  >>> index_of_index = ak.Array(
  ...     ak.layout.IndexedOptionArray64(
  ...         ak.layout.Index64(np.r_[0, 1, 2, 3]),
  ...         ak.layout.IndexedOptionArray64(
  ...             ak.layout.Index64(np.r_[0, -1, 2, 3]),
  ...             ak.layout.NumpyArray(np.r_[1, 2, 3, 4]),
  ...         ),
  ...     )
  ... )
  >>> index_of_index
  <Array [1, None, 5, 6] type='4 * ?union[?int64, ?int64]'>
  >>> ak.is_none(index_of_index)
  <Array [False, False, False, False] type='4 * bool'>
  ```

In these cases, it's visible in the type string that there is something funky going on with the layout, but the correct result would be a `NumpyArray` layout of union of bytemask arrays

It might be that the simplest solution here is to use the `transform_child_layouts` utility, and return the union of the option-layout bytemask and the result from `transform_child_layouts`, e.g.
```python3
def is_none(array, axis=0, highlevel=True, behavior=None):
    """
    Args:
        array: Data to check for missing values (None).
        axis (int): The dimension at which this operation is applied. The
            outermost dimension is `0`, followed by `1`, etc., and negative
            values count backward from the innermost: `-1` is the innermost
            dimension, `-2` is the next level up, etc.
        highlevel (bool): If True, return an #ak.Array; otherwise, return
            a low-level #ak.layout.Content subclass.
        behavior (None or dict): Custom #ak.behavior for the output array, if
            high-level.

    Returns an array whose value is True where an element of `array` is None;
    False otherwise (at a given `axis` depth).
    """

    def apply(layout, depth, posaxis):
        posaxis = layout.axis_wrap_if_negative(posaxis)
        if posaxis == depth - 1:
            nplike = ak.nplike.of(layout)
            if isinstance(layout, ak._util.optiontypes):
                mask = nplike.asarray(layout.bytemask()).view(np.bool_)

                inner_mask = ak.to_numpy(
                    ak._util.transform_child_layouts(apply, layout, depth, posaxis)
                )
                return ak.layout.NumpyArray(mask | inner_mask)

            if isinstance(
                layout,
                (
                    ak._util.unknowntypes,
                    ak._util.listtypes,
                    ak._util.recordtypes,
                    ak.layout.NumpyArray,
                ),
            ):
                return ak.layout.NumpyArray(nplike.zeros(len(layout), dtype=np.bool_))

        return ak._util.transform_child_layouts(apply, layout, depth, posaxis)

    layout = ak.operations.convert.to_layout(array)

    out = apply(layout, 1, axis)

    return ak._util.maybe_wrap_like(out, array, behavior, highlevel)
```

This feels a bit clunky, but it makes the point :)

Alternatively, we could just call `ak.packed` here, which will ensure that merge-able layouts are merged. This solution won't fix other cases, though, e.g.
```python3
ak.layout.IndexedOptionArray64(
    ak.layout.Index64(np.r_[0, 1, 2, 3]),
    ak.layout.UnionArray8_64(
        ak.layout.Index8(np.r_[0, 0, 1, 1]),
        ak.layout.Index64(np.r_[0, 1, 0, 1]),
        [
            ak.layout.IndexedOptionArray64(
                ak.layout.Index64(np.r_[0, -1, 2, 3]),
                ak.layout.NumpyArray(np.r_[1, 2, 3, 4]),
            ),
            ak.layout.NumpyArray(np.r_[4, 5, 6]),
        ],
    ),
)
```

Related to this: at present, multple option layouts grow the type string `??` corresponds to two nested options. Is this what we want? I would expect to see a single logical option (`any(l.optional for l in layouts_at_depth)`).


Discussion:
@agoose77, could you handle this one? From my perspective, it is not necessary to fix the v1 `is_none`, but if you can take the above into account for v2, that would be great. From your perspective, you might also need to fix v1 because you depend on it for your thesis (unless you have a work-around).

There's a project (https://github.com/scikit-hep/awkward-1.0/projects/4) where we can take ownership of high-level function porting. I was catching up on bug-fixes today and plan to take some of those porting tasks tomorrow, but I can avoid this one because you've thought more deeply about it and see something that needs to get fixed.
Thanks! (I linked the card to this issue.) If you're sure the above is a bug, you can switch the label, too.
@agoose77, could you handle this one? From my perspective, it is not necessary to fix the v1 `is_none`, but if you can take the above into account for v2, that would be great. From your perspective, you might also need to fix v1 because you depend on it for your thesis (unless you have a work-around).

There's a project (https://github.com/scikit-hep/awkward-1.0/projects/4) where we can take ownership of high-level function porting. I was catching up on bug-fixes today and plan to take some of those porting tasks tomorrow, but I can avoid this one because you've thought more deeply about it and see something that needs to get fixed.
Thanks! (I linked the card to this issue.) If you're sure the above is a bug, you can switch the label, too.

Resolving patch:
diff --git a/src/awkward/_v2/operations/structure/ak_is_none.py b/src/awkward/_v2/operations/structure/ak_is_none.py
--- a/src/awkward/_v2/operations/structure/ak_is_none.py
+++ b/src/awkward/_v2/operations/structure/ak_is_none.py
@@ -7,54 +7,64 @@
 
 
 def is_none(array, axis=0, highlevel=True, behavior=None):
-    raise NotImplementedError
-
-
-#     """
-#     Args:
-#         array: Data to check for missing values (None).
-#         axis (int): The dimension at which this operation is applied. The
-#             outermost dimension is `0`, followed by `1`, etc., and negative
-#             values count backward from the innermost: `-1` is the innermost
-#             dimension, `-2` is the next level up, etc.
-#         highlevel (bool): If True, return an #ak.Array; otherwise, return
-#             a low-level #ak.layout.Content subclass.
-#         behavior (None or dict): Custom #ak.behavior for the output array, if
-#             high-level.
-
-#     Returns an array whose value is True where an element of `array` is None;
-#     False otherwise (at a given `axis` depth).
-#     """
-
-#     def getfunction(layout, depth, posaxis):
-#         posaxis = layout.axis_wrap_if_negative(posaxis)
-#         if posaxis == depth - 1:
-#             nplike = ak.nplike.of(layout)
-#             if isinstance(layout, ak._v2._util.optiontypes):
-#                 return lambda: ak._v2.contents.NumpyArray(
-#                     nplike.asarray(layout.bytemask()).view(np.bool_)
-#                 )
-#             elif isinstance(
-#                 layout,
-#                 (
-#                     ak._v2._util.unknowntypes,
-#                     ak._v2._util.listtypes,
-#                     ak._v2._util.recordtypes,
-#                     ak._v2.contents.NumpyArray,
-#                 ),
-#             ):
-#                 return lambda: ak._v2.contents.NumpyArray(
-#                     nplike.zeros(len(layout), dtype=np.bool_)
-#                 )
-#             else:
-#                 return posaxis
-#         else:
-#             return posaxis
-
-#     layout = ak._v2.operations.convert.to_layout(array)
-
-#     out = ak._v2._util.recursively_apply(
-#         layout, getfunction, pass_depth=True, pass_user=True, user=axis
-#     )
-
-#     return ak._v2._util.maybe_wrap_like(out, array, behavior, highlevel)
+    """
+    Args:
+        array: Data to check for missing values (None).
+        axis (int): The dimension at which this operation is applied. The
+            outermost dimension is `0`, followed by `1`, etc., and negative
+            values count backward from the innermost: `-1` is the innermost
+            dimension, `-2` is the next level up, etc.
+        highlevel (bool): If True, return an #ak.Array; otherwise, return
+            a low-level #ak.layout.Content subclass.
+        behavior (None or dict): Custom #ak.behavior for the output array, if
+            high-level.
+
+    Returns an array whose value is True where an element of `array` is None;
+    False otherwise (at a given `axis` depth).
+    """
+
+    # Determine the (potentially nested) bytemask
+    def getfunction_inner(layout, depth, **kwargs):
+        if not isinstance(layout, ak._v2.contents.Content):
+            return
+
+        nplike = ak.nplike.of(layout)
+
+        if layout.is_OptionType:
+            layout = layout.toIndexedOptionArray64()
+
+            # Convert the option type into a union, using the mask
+            # as a tag.
+            tag = nplike.asarray(layout.bytemask())
+            index = nplike.where(tag, 0, nplike.asarray(layout.index))
+
+            return ak._v2.contents.UnionArray(
+                ak._v2.index.Index8(tag),
+                ak._v2.index.Index64(index),
+                [
+                    layout.content.recursively_apply(getfunction_inner),
+                    ak._v2.contents.NumpyArray(nplike.array([True], dtype=np.bool_)),
+                ],
+            ).simplify_uniontype()
+
+        elif (
+            layout.is_UnknownType
+            or layout.is_ListType
+            or layout.is_RecordType
+            or layout.is_NumpyType
+        ):
+            return ak._v2.contents.NumpyArray(nplike.zeros(len(layout), dtype=np.bool_))
+
+    # Locate the axis
+    def getfunction_outer(layout, depth, depth_context, **kwargs):
+        depth_context["posaxis"] = layout.axis_wrap_if_negative(
+            depth_context["posaxis"]
+        )
+        if depth_context["posaxis"] == depth - 1:
+            return layout.recursively_apply(getfunction_inner)
+
+    layout = ak._v2.operations.convert.to_layout(array)
+    behavior = ak._v2._util.behavior_of(array, behavior=behavior)
+    depth_context = {"posaxis": axis}
+    out = layout.recursively_apply(getfunction_outer, depth_context=depth_context)
+    return ak._v2._util.wrap(out, behavior, highlevel)
diff --git a/src/awkward/operations/structure.py b/src/awkward/operations/structure.py
--- a/src/awkward/operations/structure.py
+++ b/src/awkward/operations/structure.py
@@ -2851,36 +2851,50 @@ def is_none(array, axis=0, highlevel=True, behavior=None):
     False otherwise (at a given `axis` depth).
     """
 
-    def getfunction(layout, depth, posaxis):
+    def transform_inner(layout, depth, user=None):
+        nplike = ak.nplike.of(layout)
+        if isinstance(layout, ak._util.optiontypes):
+            if not isinstance(layout, ak.layout.IndexedOptionArray64):
+                layout = layout.toIndexedOptionArray64()
+
+            # Convert the option type into a union, using the mask
+            # as a tag.
+            tag = nplike.asarray(layout.bytemask())
+            index = nplike.where(tag, 0, nplike.asarray(layout.index))
+
+            return ak.layout.UnionArray8_64(
+                ak.layout.Index8(tag),
+                ak.layout.Index64(index),
+                [
+                    transform_inner(layout.content, depth),
+                    ak.layout.NumpyArray(nplike.array([True], dtype=np.bool_)),
+                ],
+            ).simplify()
+        elif isinstance(
+            layout,
+            (
+                ak._util.unknowntypes,
+                ak._util.listtypes,
+                ak._util.recordtypes,
+                ak.layout.NumpyArray,
+            ),
+        ):
+            return ak.layout.NumpyArray(nplike.zeros(len(layout), dtype=np.bool_))
+        else:
+            return ak._util.transform_child_layouts(transform_inner, layout, depth)
+
+    def transform_outer(layout, depth, posaxis):
         posaxis = layout.axis_wrap_if_negative(posaxis)
-        if posaxis == depth - 1:
-            nplike = ak.nplike.of(layout)
-            if isinstance(layout, ak._util.optiontypes):
-                return lambda: ak.layout.NumpyArray(
-                    nplike.asarray(layout.bytemask()).view(np.bool_)
-                )
-            elif isinstance(
-                layout,
-                (
-                    ak._util.unknowntypes,
-                    ak._util.listtypes,
-                    ak._util.recordtypes,
-                    ak.layout.NumpyArray,
-                ),
-            ):
-                return lambda: ak.layout.NumpyArray(
-                    nplike.zeros(len(layout), dtype=np.bool_)
-                )
-            else:
-                return posaxis
+        if posaxis != depth - 1:
+            return ak._util.transform_child_layouts(
+                transform_outer, layout, depth, posaxis
+            )
         else:
-            return posaxis
+            return transform_inner(layout, depth)
 
     layout = ak.operations.convert.to_layout(array)
 
-    out = ak._util.recursively_apply(
-        layout, getfunction, pass_depth=True, pass_user=True, user=axis
-    )
+    out = transform_outer(layout, 1, axis)
 
     return ak._util.maybe_wrap_like(out, array, behavior, highlevel)
 


Test patch:
diff --git a/tests/test_1193-is-none-nested-option.py b/tests/test_1193-is-none-nested-option.py
new file mode 100644
--- /dev/null
+++ b/tests/test_1193-is-none-nested-option.py
@@ -0,0 +1,20 @@
+# BSD 3-Clause License; see https://github.com/scikit-hep/awkward-1.0/blob/main/LICENSE
+
+import pytest  # noqa: F401
+import numpy as np  # noqa: F401
+import awkward as ak  # noqa: F401
+
+
+def test():
+    index_of_index = ak.Array(
+        ak.layout.IndexedOptionArray64(
+            ak.layout.Index64(np.r_[0, 1, 2, -1]),
+            ak.layout.IndexedOptionArray64(
+                ak.layout.Index64(np.r_[0, -1, 2, 3]),
+                ak.layout.NumpyArray(np.r_[1, 2, 3, 4]),
+            ),
+        )
+    )
+
+    mask = ak.is_none(index_of_index)
+    assert ak.to_list(mask) == [False, True, False, True]
diff --git a/tests/v2/test_0713-getitem_field-should-simplify_optiontype.py b/tests/v2/test_0713-getitem_field-should-simplify_optiontype.py
--- a/tests/v2/test_0713-getitem_field-should-simplify_optiontype.py
+++ b/tests/v2/test_0713-getitem_field-should-simplify_optiontype.py
@@ -8,7 +8,6 @@
 to_list = ak._v2.operations.convert.to_list
 
 
-@pytest.mark.skip(reason="FIXME: ak._v2.operations.structure.is_none not implemented")
 def test():
     arr1 = ak._v2.highlevel.Array({"a": [1, 2], "b": [1, None]})
     arr2 = ak._v2.operations.structure.mask(arr1, [True, True])
diff --git a/tests/v2/test_1193-is-none-nested-option.py b/tests/v2/test_1193-is-none-nested-option.py
new file mode 100644
--- /dev/null
+++ b/tests/v2/test_1193-is-none-nested-option.py
@@ -0,0 +1,20 @@
+# BSD 3-Clause License; see https://github.com/scikit-hep/awkward-1.0/blob/main/LICENSE
+
+import pytest  # noqa: F401
+import numpy as np  # noqa: F401
+import awkward as ak  # noqa: F401
+
+
+def test():
+    index_of_index = ak._v2.highlevel.Array(
+        ak._v2.contents.IndexedOptionArray(
+            ak._v2.index.Index64(np.r_[0, 1, 2, -1]),
+            ak._v2.contents.IndexedOptionArray(
+                ak._v2.index.Index64(np.r_[0, -1, 2, 3]),
+                ak._v2.contents.NumpyArray(np.r_[1, 2, 3, 4]),
+            ),
+        )
+    )
+
+    mask = ak._v2.operations.structure.is_none(index_of_index)
+    assert mask.tolist() == [False, True, False, True]