Repository: scikit-hep/awkward

Issue:
Intersecting parameters introduced "unhashable type" bug.
### Version of Awkward Array

1.10.0

### Description and code to reproduce

PR #1679 added parameter-carrying to broadcasting, but if the parameter values are dicts, rather something hashable, like strings, the implementation fails.

```python
>>> import awkward._v2 as ak
>>> array = ak.with_parameter(ak.Array([[1, 2], [], [3]]), "attrs", "hashable")
>>> array
<Array [[1, 2], [], [3]] type='3 * [var * int64, parameters={"attrs": "hash...'>
>>> array + array
<Array [[2, 4], [], [6]] type='3 * [var * int64, parameters={"attrs": "hash...'>
```

versus

```python
>>> array = ak.with_parameter(ak.Array([[1, 2], [], [3]]), "attrs", {"not": "hashable"})
>>> array
<Array [[1, 2], [], [3]] type='3 * [var * int64, parameters={"attrs": {"not...'>
>>> array + array
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jpivarski/mambaforge/lib/python3.10/site-packages/numpy/lib/mixins.py", line 21, in func
    return ufunc(self, other)
  File "/Users/jpivarski/irishep/awkward/awkward/_v2/highlevel.py", line 1339, in __array_ufunc__
    return ak._v2._connect.numpy.array_ufunc(ufunc, method, inputs, kwargs)
  File "/Users/jpivarski/irishep/awkward/awkward/_v2/_connect/numpy.py", line 278, in array_ufunc
    out = ak._v2._broadcasting.broadcast_and_apply(
  File "/Users/jpivarski/irishep/awkward/awkward/_v2/_broadcasting.py", line 1019, in broadcast_and_apply
    out = apply_step(
  File "/Users/jpivarski/irishep/awkward/awkward/_v2/_broadcasting.py", line 998, in apply_step
    return continuation()
  File "/Users/jpivarski/irishep/awkward/awkward/_v2/_broadcasting.py", line 720, in continuation
    outcontent = apply_step(
  File "/Users/jpivarski/irishep/awkward/awkward/_v2/_broadcasting.py", line 409, in apply_step
    parameters_factory = parameters_factory_impl(inputs)
  File "/Users/jpivarski/irishep/awkward/awkward/_v2/_broadcasting.py", line 277, in _intersection_parameters_factory
    functools.reduce(operator.and_, list(parameters_to_intersect))
TypeError: unhashable type: 'dict'
```

It fails in the `functools.reduce` of

https://github.com/scikit-hep/awkward/blob/ca1b9d7835570af8a11fe7af61a7c53d095a80e2/src/awkward/_v2/_broadcasting.py#L262-L276

in the hashable case, the `parameters_to_intersect` is

```python
[dict_items([('attrs', 'hashable')]), dict_items([('attrs', 'hashable')])]
```

and in the non-hashable case, it's

```python
[dict_items([('attrs', {'not': 'hashable'})]), dict_items([('attrs', {'not': 'hashable'})])]
```

The `operator.and_` is a bitwise `&` operator—I doubt that's what you want there. I think this can be done more simply in a for loop.

Also, possibly related, we're losing parameters in simple arrays:

```python
>>> array = ak.with_parameter(ak.Array([1, 2, 3]), "attrs", {"not": "hashable"})
>>> array
<Array [1, 2, 3] type='3 * int64[parameters={"attrs": {"not": "hashable"}}]'>
>>> array + array
<Array [2, 4, 6] type='3 * int64'>
```

In this case, it reaches the `break` and therefore skips the `reduce`. (Why would `_parameters_is_empty` be true for this array that clearly has parameters? I think the first step in broadcasting creates a RegularArray around this NumpyArray—it doesn't collapse the RegularArray + NumpyArray into a multidimensional NumpyArray, and lose the parameter, does it?)

Discovered by @philippemiron.
Intersecting parameters introduced "unhashable type" bug.
### Version of Awkward Array

1.10.0

### Description and code to reproduce

PR #1679 added parameter-carrying to broadcasting, but if the parameter values are dicts, rather something hashable, like strings, the implementation fails.

```python
>>> import awkward._v2 as ak
>>> array = ak.with_parameter(ak.Array([[1, 2], [], [3]]), "attrs", "hashable")
>>> array
<Array [[1, 2], [], [3]] type='3 * [var * int64, parameters={"attrs": "hash...'>
>>> array + array
<Array [[2, 4], [], [6]] type='3 * [var * int64, parameters={"attrs": "hash...'>
```

versus

```python
>>> array = ak.with_parameter(ak.Array([[1, 2], [], [3]]), "attrs", {"not": "hashable"})
>>> array
<Array [[1, 2], [], [3]] type='3 * [var * int64, parameters={"attrs": {"not...'>
>>> array + array
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jpivarski/mambaforge/lib/python3.10/site-packages/numpy/lib/mixins.py", line 21, in func
    return ufunc(self, other)
  File "/Users/jpivarski/irishep/awkward/awkward/_v2/highlevel.py", line 1339, in __array_ufunc__
    return ak._v2._connect.numpy.array_ufunc(ufunc, method, inputs, kwargs)
  File "/Users/jpivarski/irishep/awkward/awkward/_v2/_connect/numpy.py", line 278, in array_ufunc
    out = ak._v2._broadcasting.broadcast_and_apply(
  File "/Users/jpivarski/irishep/awkward/awkward/_v2/_broadcasting.py", line 1019, in broadcast_and_apply
    out = apply_step(
  File "/Users/jpivarski/irishep/awkward/awkward/_v2/_broadcasting.py", line 998, in apply_step
    return continuation()
  File "/Users/jpivarski/irishep/awkward/awkward/_v2/_broadcasting.py", line 720, in continuation
    outcontent = apply_step(
  File "/Users/jpivarski/irishep/awkward/awkward/_v2/_broadcasting.py", line 409, in apply_step
    parameters_factory = parameters_factory_impl(inputs)
  File "/Users/jpivarski/irishep/awkward/awkward/_v2/_broadcasting.py", line 277, in _intersection_parameters_factory
    functools.reduce(operator.and_, list(parameters_to_intersect))
TypeError: unhashable type: 'dict'
```

It fails in the `functools.reduce` of

https://github.com/scikit-hep/awkward/blob/ca1b9d7835570af8a11fe7af61a7c53d095a80e2/src/awkward/_v2/_broadcasting.py#L262-L276

in the hashable case, the `parameters_to_intersect` is

```python
[dict_items([('attrs', 'hashable')]), dict_items([('attrs', 'hashable')])]
```

and in the non-hashable case, it's

```python
[dict_items([('attrs', {'not': 'hashable'})]), dict_items([('attrs', {'not': 'hashable'})])]
```

The `operator.and_` is a bitwise `&` operator—I doubt that's what you want there. I think this can be done more simply in a for loop.

Also, possibly related, we're losing parameters in simple arrays:

```python
>>> array = ak.with_parameter(ak.Array([1, 2, 3]), "attrs", {"not": "hashable"})
>>> array
<Array [1, 2, 3] type='3 * int64[parameters={"attrs": {"not": "hashable"}}]'>
>>> array + array
<Array [2, 4, 6] type='3 * int64'>
```

In this case, it reaches the `break` and therefore skips the `reduce`. (Why would `_parameters_is_empty` be true for this array that clearly has parameters? I think the first step in broadcasting creates a RegularArray around this NumpyArray—it doesn't collapse the RegularArray + NumpyArray into a multidimensional NumpyArray, and lose the parameter, does it?)

Discovered by @philippemiron.


Discussion:
I've assigned it to you, @agoose77, so that you can take a first look (since you wrote #1679), but I can look at it too, with you, if you want.
> but if the parameter values are dicts, rather something hashable, like strings, the implementation fails.

An oversight on my part. 

> The operator.and_ is a bitwise & operator—I doubt that's what you want there. I think this can be done more simply in a for loop.

Yes, in the case of `set()`,  [`x & y` computes the intersection of `x` and `y`](https://docs.python.org/3/library/stdtypes.html#frozenset.intersection)
I've assigned it to you, @agoose77, so that you can take a first look (since you wrote #1679), but I can look at it too, with you, if you want.
> but if the parameter values are dicts, rather something hashable, like strings, the implementation fails.

An oversight on my part. 

> The operator.and_ is a bitwise & operator—I doubt that's what you want there. I think this can be done more simply in a for loop.

Yes, in the case of `set()`,  [`x & y` computes the intersection of `x` and `y`](https://docs.python.org/3/library/stdtypes.html#frozenset.intersection)

Resolving patch:
diff --git a/src/awkward/_v2/_broadcasting.py b/src/awkward/_v2/_broadcasting.py
--- a/src/awkward/_v2/_broadcasting.py
+++ b/src/awkward/_v2/_broadcasting.py
@@ -6,7 +6,6 @@
 import enum
 import functools
 import itertools
-import operator
 from typing import Any, Callable, Dict, List, Union
 from collections.abc import Sequence
 
@@ -201,7 +200,7 @@ def _parameters_of(obj: Any, default: Any = NO_PARAMETERS) -> Any:
         return default
 
 
-def _all_or_nothing_parameters_factory(
+def all_or_nothing_parameters_factory(
     inputs: Sequence,
 ) -> BroadcastParameterFactory:
     """
@@ -240,7 +239,7 @@ def apply(n_outputs: int) -> list[dict[str, Any] | None]:
     return apply
 
 
-def _intersection_parameters_factory(
+def intersection_parameters_factory(
     inputs: Sequence,
 ) -> BroadcastParameterFactory:
     """
@@ -268,11 +267,11 @@ def _intersection_parameters_factory(
         if ak._v2.forms.form._parameters_is_empty(parameters):
             break
         else:
-            parameters_to_intersect.append(parameters.items())
+            parameters_to_intersect.append(parameters)
     # Otherwise, build the intersected parameter dict
     else:
-        intersected_parameters = dict(
-            functools.reduce(operator.and_, parameters_to_intersect)
+        intersected_parameters = functools.reduce(
+            ak._v2.forms.form._parameters_intersect, parameters_to_intersect
         )
 
     def apply(n_outputs: int) -> list[dict[str, Any] | None]:
@@ -283,7 +282,7 @@ def apply(n_outputs: int) -> list[dict[str, Any] | None]:
     return apply
 
 
-def _one_to_one_parameters_factory(
+def one_to_one_parameters_factory(
     inputs: Sequence,
 ) -> BroadcastParameterFactory:
     """
@@ -315,7 +314,7 @@ def apply(n_outputs) -> list[dict[str, Any] | None]:
     return apply
 
 
-def _none_parameters_factory(
+def none_parameters_factory(
     inputs: Sequence,
 ) -> BroadcastParameterFactory:
     """
@@ -337,10 +336,10 @@ def apply(n_outputs: int) -> list[dict[str, Any] | None]:
 
 # Mapping from rule enum values to factory implementations
 BROADCAST_RULE_TO_FACTORY_IMPL = {
-    BroadcastParameterRule.INTERSECT: _intersection_parameters_factory,
-    BroadcastParameterRule.ALL_OR_NOTHING: _all_or_nothing_parameters_factory,
-    BroadcastParameterRule.ONE_TO_ONE: _one_to_one_parameters_factory,
-    BroadcastParameterRule.NONE: _none_parameters_factory,
+    BroadcastParameterRule.INTERSECT: intersection_parameters_factory,
+    BroadcastParameterRule.ALL_OR_NOTHING: all_or_nothing_parameters_factory,
+    BroadcastParameterRule.ONE_TO_ONE: one_to_one_parameters_factory,
+    BroadcastParameterRule.NONE: none_parameters_factory,
 }
 
 
diff --git a/src/awkward/_v2/_connect/numpy.py b/src/awkward/_v2/_connect/numpy.py
--- a/src/awkward/_v2/_connect/numpy.py
+++ b/src/awkward/_v2/_connect/numpy.py
@@ -140,21 +140,6 @@ def _array_ufunc_signature(ufunc, inputs):
     return signature
 
 
-def _array_ufunc_deregulate(inputs):
-    nextinputs = []
-    for x in inputs:
-        if isinstance(x, ak._v2.contents.RegularArray):
-            y = x.maybe_toNumpyArray()
-            if y is not None:
-                nextinputs.append(y)
-            else:
-                nextinputs.append(x)
-        else:
-            nextinputs.append(x)
-
-    return nextinputs
-
-
 def array_ufunc(ufunc, method, inputs, kwargs):
     if method != "__call__" or len(inputs) == 0 or "out" in kwargs:
         return NotImplemented
@@ -175,14 +160,17 @@ def action(inputs, **ignore):
             if custom_matmul is not None:
                 return custom_matmul()
 
-        inputs = _array_ufunc_deregulate(inputs)
-
         if all(
             isinstance(x, NumpyArray) or not isinstance(x, ak._v2.contents.Content)
             for x in inputs
         ):
             nplike = ak.nplike.of(*inputs)
 
+            # Broadcast parameters against one another
+            parameters_factory = ak._v2._broadcasting.intersection_parameters_factory(
+                inputs
+            )
+            (parameters,) = parameters_factory(1)
             if nplike.known_data:
                 args = []
                 for x in inputs:
@@ -211,8 +199,7 @@ def action(inputs, **ignore):
                 assert shape is not None
                 tmp = getattr(ufunc, method)(*args, **kwargs)
                 result = nplike.empty((shape[0],) + tmp.shape[1:], tmp.dtype)
-
-            return (NumpyArray(result, nplike=nplike),)
+            return (NumpyArray(result, nplike=nplike, parameters=parameters),)
 
         for x in inputs:
             if isinstance(x, ak._v2.contents.Content):
diff --git a/src/awkward/_v2/forms/form.py b/src/awkward/_v2/forms/form.py
--- a/src/awkward/_v2/forms/form.py
+++ b/src/awkward/_v2/forms/form.py
@@ -2,7 +2,7 @@
 from __future__ import annotations
 
 import json
-from typing import Any
+from typing import Any, Mapping
 
 import awkward as ak
 
@@ -210,6 +210,24 @@ def _parameters_equal(one, two, only_array_record=False):
         return True
 
 
+def _parameters_intersect(
+    left: Mapping[str, Any], right: Mapping[str, Any]
+) -> dict[str, Any]:
+    """
+    Args:
+        left: first parameters mapping
+        right: second parameters mapping
+
+    Returns the intersected key-value pairs of `left` and `right` as a dictionary.
+
+    """
+    result = {}
+    for key in left.keys() & right.keys():
+        if left[key] == right[key]:
+            result[key] = left[key]
+    return result
+
+
 def _parameters_update(one, two):
     for k, v in two.items():
         if v is not None:


Test patch:
diff --git a/tests/v2/test_1672-broadcast-parameters.py b/tests/v2/test_1672-broadcast-parameters.py
--- a/tests/v2/test_1672-broadcast-parameters.py
+++ b/tests/v2/test_1672-broadcast-parameters.py
@@ -255,12 +255,22 @@ def test_transform_float_int_2d_different_intersect():
     this = ak._v2.contents.ListOffsetArray(
         ak._v2.index.Index64(numpy.array([0, 3, 4], dtype="int64")),
         ak._v2.contents.NumpyArray(numpy.array([1.0, 2.0, 3.0, 4.0], dtype="float64")),
-        parameters={"name": "this"},
+        parameters={
+            "name": "this",
+            "key": "value",
+            "parent": {"child": 1},
+            "pets": [{"name": "fido"}],
+        },
     )
     that = ak._v2.contents.ListOffsetArray(
         ak._v2.index.Index64(numpy.array([0, 3, 4], dtype="int64")),
         ak._v2.contents.NumpyArray(numpy.array([1, 2, 1, 9], dtype="int64")),
-        parameters={"name": "that"},
+        parameters={
+            "name": "that",
+            "key": "value",
+            "parent": {"child": 2},
+            "pets": [{"name": "fido"}],
+        },
     )
     this_next, that_next = ak._v2.operations.ak_transform.transform(
         lambda *a, **k: None,
@@ -270,8 +280,8 @@ def test_transform_float_int_2d_different_intersect():
         broadcast_parameters_rule="intersect",
     )
 
-    assert this_next.parameters == {}
-    assert that_next.parameters == {}
+    assert this_next.parameters == that_next.parameters
+    assert that_next.parameters == {"key": "value", "pets": [{"name": "fido"}]}
 
 
 def test_transform_float_int_2d_one_to_one_error():