Repository: scikit-hep/awkward

Issue:
`NumpyArray` with `ndim>1` does not broadcast correctly
### Version of Awkward Array

1.4.0

### Description and code to reproduce
The result of `broadcast_arrays` differs according to whether the regular dimension in a *jagged* array is a `RegularArray` or a `NumpyArray[ndim == 2]]`

#### Reproducer:
```python3
form, length, container = (
    {
        "class": "ListOffsetArray64",
        "offsets": "i64",
        "content": {
            "class": "ListOffsetArray64",
            "offsets": "i64",
            "content": {
                "class": "NumpyArray",
                "inner_shape": [2],
                "itemsize": 8,
                "format": "d",
                "primitive": "float64",
                "form_key": "node2",
            },
            "form_key": "node1",
        },
        "form_key": "node0",
    },
    4,
    {
        "part0-node0-offsets": np.array([0, 1, 2, 3, 4], dtype=np.int64),
        "part0-node1-offsets": np.array([0, 1, 6, 7, 8], dtype=np.int64),
        "part0-node2-data": np.array(
            [
                [40.25, 41.25],
                [-8.0, -11.75],
                [-8.0, -13.75],
                [35.0, 41.25],
                [70.0, 75.25],
                [24.0, 34.25],
                [-3.75, -3.75],
                [-6.25, -9.75],
            ]
        ),
    },
)

x = ak.from_buffers(form, length, container)
y = ak.Array(
    [
        [[42.5]],
        [[-13.25, -14.25, 40.416666666666664, 76.75, 35.75]],
        [[-4.916666666666667]],
        [[-8.833333333333334]],
    ]
)
ak.broadcast_arrays(x, y)
```
produces 
```python3
[<Array [[[[40.2, 41.2]], ... [[-6.25, -9.75]]]] type='4 * var * var * 2 * float64'>,
 <Array [[[42.5]], [[-13.2, ... [[-8.83]]] type='4 * var * var * float64'>]
```
whereas
```python3
z = ak.to_regular(ak.Array(x.tolist()), axis=-1)
ak.broadcast_arrays(z, y)
```
produces
```python3
[<Array [[[[40.2, 41.2]], ... [[-6.25, -9.75]]]] type='4 * var * var * 2 * float64'>,
 <Array [[[[42.5, 42.5]], ... [[-8.83, -8.83]]]] type='4 * var * var * 2 * float64'>]
```


Discussion:
I *think* the bug is that the getfunction doesn't expect an `ndim>1` array, and doesn't set `numpy_to_regular=True` which would otherwise guarantee that it would never receive ndim arrays. 

https://github.com/scikit-hep/awkward-1.0/blob/2c679f385cb9cbd7af013958c54c0e0be8d8837c/src/awkward/operations/structure.py#L1369-L1373

The fix would then be either:
1. To check that the dimensions are correct
   ```python3
   def getfunction(inputs):
       if all(isinstance(x, ak.layout.NumpyArray) and x.ndim == 1 for x in inputs):
           return lambda: tuple(inputs)
       else:
           return None
   ```
   and if not, the `broadcast_and_apply` helper will convert the `NumpyArray` to a `RegularArray` on the next loop
2. Alternatively set `numpy_to_regular=True` in the call to `broadcast_and_apply`.

I'm not sure whether the purpose of `numpy_to_regular` is clear to me, could you elaborate @jpivarski ? It seems like it's used to ensure that there any `NumpyArray`s that are passed to the `getfunction` have `ndim == 1` (which is why setting it to `True` would solve this bug. It doesn't affect the recursion routine itself, as later in the recursion body we have

https://github.com/scikit-hep/awkward-1.0/blob/2c679f385cb9cbd7af013958c54c0e0be8d8837c/src/awkward/_util.py#L750-L757

The result itself is always converted to a `RegularArray` in either case.
Handling layout details like NumpyArray with ndim > 1 is considered the particular getfunction's responsibility. That means that no one has experienced this bug before in `ak.broadcast_arrays` specifically. (It would be different in other high-level functions.) Maybe that should be collected into a "regulararray=True" option on `broadcast_and_apply`, though?

Resolving patch:
diff --git a/src/awkward/operations/structure.py b/src/awkward/operations/structure.py
--- a/src/awkward/operations/structure.py
+++ b/src/awkward/operations/structure.py
@@ -1380,6 +1380,7 @@ def getfunction(inputs):
         left_broadcast=left_broadcast,
         right_broadcast=right_broadcast,
         pass_depth=False,
+        numpy_to_regular=True,
     )
     assert isinstance(out, tuple)
     if highlevel:


Test patch:
diff --git a/tests/test_1017-numpyarray-broadcast.py b/tests/test_1017-numpyarray-broadcast.py
new file mode 100644
--- /dev/null
+++ b/tests/test_1017-numpyarray-broadcast.py
@@ -0,0 +1,19 @@
+# BSD 3-Clause License; see https://github.com/scikit-hep/awkward-1.0/blob/main/LICENSE
+
+from __future__ import absolute_import
+
+import pytest  # noqa: F401
+import numpy as np  # noqa: F401
+import awkward as ak  # noqa: F401
+
+
+def test():
+    x = ak.layout.ListOffsetArray64(
+        ak.layout.Index64(np.array([0, 2, 2, 4, 6])),
+        ak.layout.NumpyArray(np.arange(8 * 8).reshape(8, -1)),
+    )
+    y = ak.layout.ListOffsetArray64(
+        ak.layout.Index64(np.array([0, 2, 2, 4, 6])), ak.layout.NumpyArray(np.arange(8))
+    )
+    u, v = ak.broadcast_arrays(x, y)
+    assert u.ndim == v.ndim