Repository: scikit-hep/awkward

Issue:
When ak.to_numpy makes a np.ma.MaskedArray, the values behind the mask must be initialized
### Version of Awkward Array

HEAD

### Description and code to reproduce

When an Awkward Array with missing values in it is converted to NumPy with [ak.to_numpy](https://awkward-array.readthedocs.io/en/latest/_auto/ak.to_numpy.html) and `allow_missing=True`, the return type is `np.ma.MaskedArray` with `mask[i] = True` for entries `i` corresponding to missing values. The `data[i]` for those `i` are uninitialized or otherwise undefined. That would be fine if MaskedArray were an iron-clad thing that never let you see the `data[i]` for which `mask[i]` is `True`, but it's quite easy to look at these things, and even

```python
>>> np.asarray(np.ma.MaskedArray([0, 1, 2, 3], [False, False, True, True]))
array([0, 1, 2, 3])
```

So MaskedArrays are [very leaky](https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/).

Instead of leaving `data[i]` for these `i` uninitialized/undefined, we should ensure that they correspond to a common value. My preferred solution is:

   * These `data[i]` are `np.nan` for floating-point types, `np.iinfo(dtype).max` for integer types, datetimes, and timedeltas, empty strings for strings, and I guess `False` for boolean.

All of these values except `np.nan` could be considered "good data"—they're in-band sentinels—but they were _supposed to be_ behind a mask anyway. Even with `np.nan`, there's a distinction in Awkward Array between `np.nan` and `None` that gets lost when the mask is taken away by somebody else. Hey, we did everything we could.

Another solution is:

   * Set `data.dtype=object`, so that we can put `None` values in `data[i]` at each `i` for which `mask[i] = True`.

This solution boxes all of the numbers, a big performance hit, just in case someone disregards the `mask` that we put in place to say that these `i` aren't valid. I don't think that's the right thing to do.


Discussion:
I agree that this is the pragmatic approach and the best you can do in this case where someone has left Awkward, gone to a masked array, and then discarded the mask!  Always going to the same "in-band" sentinel is definitely better than leaving them uninitialized (which can have very interesting undefined behavior  https://github.com/h5py/h5py/issues/2069).

Resolving patch:
diff --git a/src/awkward/_v2/contents/indexedoptionarray.py b/src/awkward/_v2/contents/indexedoptionarray.py
--- a/src/awkward/_v2/contents/indexedoptionarray.py
+++ b/src/awkward/_v2/contents/indexedoptionarray.py
@@ -1544,6 +1544,24 @@ def _to_numpy(self, allow_missing):
                     mask[~mask0] |= mask1
 
                 data[~mask0] = content
+
+                if issubclass(content.dtype.type, (bool, np.bool_)):
+                    data[mask0] = False
+                elif issubclass(content.dtype.type, np.floating):
+                    data[mask0] = np.nan
+                elif issubclass(content.dtype.type, np.complexfloating):
+                    data[mask0] = np.nan + np.nan * 1j
+                elif issubclass(content.dtype.type, np.integer):
+                    data[mask0] = np.iinfo(content.dtype).max
+                elif issubclass(content.dtype.type, (np.datetime64, np.timedelta64)):
+                    data[mask0] = numpy.array([np.iinfo(np.int64).max], content.dtype)[
+                        0
+                    ]
+                else:
+                    raise ak._v2._util.error(
+                        AssertionError(f"unrecognized dtype: {content.dtype}")
+                    )
+
                 return numpy.ma.MaskedArray(data, mask)
             else:
                 raise ak._v2._util.error(


Test patch: