Repository: scikit-hep/awkward

Issue:
ak.fields (v2) passes a RecordArray's internal fields by reference
Okay, so I hadn't noticed that Awkward v2's fields are passed by reference, which exposes them to the danger that someone might modify them downstream:

v1:

```python
>>> array = awkward.Array([{"x": 1, "y": 1.1}])
>>> fields = awkward.fields(array)
>>> array
<Array [{x: 1, y: 1.1}] type='1 * {"x": int64, "y": float64}'>
>>> fields
['x', 'y']
>>> fields[0] = "XXX"
>>> fields
['XXX', 'y']
>>> array
<Array [{x: 1, y: 1.1}] type='1 * {"x": int64, "y": float64}'>
```

v2:

```python
>>> array = awkward._v2.Array([{"x": 1, "y": 1.1}])
>>> fields = awkward._v2.fields(array)
>>> array
<Array [{x: 1, y: 1.1}] type='1 * {x: int64, y: float64}'>
>>> fields
['x', 'y']
>>> fields[0] = "XXX"
>>> fields
['XXX', 'y']
>>> array
<Array [{XXX: 1, y: 1.1}] type='1 * {XXX: int64, y: float64}'>
```

It could be fixed [here, in Awkward](https://github.com/scikit-hep/awkward/blob/352b0dead74846ad2a56d385be4694ec87072a08/src/awkward/_v2/contents/recordarray.py#L162), or maybe [here](https://github.com/scikit-hep/awkward/blob/352b0dead74846ad2a56d385be4694ec87072a08/src/awkward/_v2/operations/ak_fields.py#L30) (to only suffer the list-copy when handing it off to a user, so that internal uses can still be by reference).

I'll use this comment to open an issue in Awkward. Once `awkward.fields` is guarded, your `.copy()` can be removed, but it can also not be removed with no consequences but a little performance.

_Originally posted by @jpivarski in https://github.com/scikit-hep/vector/pull/226#discussion_r958660705_


Discussion:
I'm leaning toward doing the copy at the public interface:

https://github.com/scikit-hep/awkward/blob/352b0dead74846ad2a56d385be4694ec87072a08/src/awkward/_v2/operations/ak_fields.py#L28-L30

just wrapping that `layout.fields` as `copy.copy(layout.fields)`. (It could be `None`.)
We also have little protection for `layout.parameters` either (e.g. via `ak.parameters`). I suppose we need to decide whether the layout interface is immutable by design or by convention. This is perhaps more troubling, because parameters are shared all over the place.

So the above also goes for `ak.parameters`. Nothing is _supposed_ to be assigned in place, though we might do it in an internal function if we can prove to ourselves that the influence will not leak beyond that function.

Early in v1, there was a shift to complete immutability, even in the internals: #176 and #177. The pybind11 layer had to copy all the data (such as `recordlookup,` which is `fields`, and `parameters`) in the conversion from C++ to Python.

In v2, we pass around a lot more references, such as this `_fields` and, as you point out, `_parameters`. Parameters are either `None` or a dict, and if the latter, it can get expensive to be copying dicts all the time. (`None` will turn into `{}` when needed, too. The `None` representation of a layout's `_parameters` is an internal optimization that we hide in utility functions like `merge_parameters`.)

A lot of things changed from v1 to v2, making it hard to narrow in on causes, but there was a noticeable factor-of-several speed-up in the overhead (_O(1)_ time) of most of the functions that I actually tested. I strongly suspect it has to do with fewer copies of metadata.

You're right that we should clarify, to ourselves, a policy about this. I think we should keep using references for the low-level interfaces (e.g. `layout.parameters`) but defensively copy the high-level interfaces (e.g. `ak.parameters`). Low-level users/developers of downstream dependencies already run the risk of mutating data because they can cast a `NumpyArray` as a `np.ndarray` and mutate that. This policy of leaving the low-level interfaces as references has the same risk of mutation, but in metadata, rather than data.
As for action items on this issue:

  - [x] `copy.copy(layout.fields)` [here](https://github.com/scikit-hep/awkward/blob/352b0dead74846ad2a56d385be4694ec87072a08/src/awkward/_v2/operations/ak_fields.py#L30)
  - [x] `copy.copy` the `parameters` in [these two cases](https://github.com/scikit-hep/awkward/blob/352b0dead74846ad2a56d385be4694ec87072a08/src/awkward/_v2/operations/ak_parameters.py#L29-L36)

If I've forgotten any, let me know and we'll add them to this list. (@Saransh-cpp has expressed an interest in fixing this, and we can coordinate the timing of a release with Vector.)
> I think we should keep using references for the low-level interfaces (e.g. layout.parameters) but defensively copy the high-level interfaces 

Yes, the purist in me itches to enforce this at the API level for LL and HL APIs, but if we do this by forcing copying on all users ... that might not be great. I don't have a feel for the performance impact of such a decision, which would strongly depend on how often we're calling `copy` and what we're actually copying. I would remark, however, that we're definitely not crossing the Python-C++ boundary as often, which I suspect was also a significant contributor to the reduced O(1) performance of v1 w.r.t v2.

At the same time as addressing this, I think we might want to formalise our contents API too - right now there are methods / properties that all contents implement, and are expected to implement, but are not defined on the base class. Given that layouts are low-level w.r.t the high-level API, we certainly could get away with only copying at the high level, which would also be guaranteed to mostly happen once per call. We could alternatively choose to implement a zero-copy private interface for contents, which the `Content` classes can call, but external layout users should avoid, e.g. `_parameters` vs `def parameters(self)`.

I'm not too worried about the low-level side of things all considered, because it already requires a degree of domain knowledge.

As an aside, I'd like to make `fields` be a tuple, but I think we're past being able to change that, at least with the v1-v2 transition preserving API.
> As an aside, I'd like to make `fields` be a tuple, but I think we're past being able to change that, at least with the v1-v2 transition preserving API.

I thought about that, too, and it's not too late to change the API to make `fields` a tuple. (Changes on that scale are still open.) It's just that it doesn't strike me as a tuple-kind of thing: I think of tuples as having known or relatively stable lengths, such as pairs or triples, and frequently different types. This `fields` is always strings, but varies in length arbitrarily from one RecordArray to another.

On the other hand, Python doesn't care about that—tuple is a synonym for "frozen list."

One thing that might be hard, in changing it, is that it might be set via list comprehensions in some parts of the code. It would take some care to be certain that it's not sometimes converted from a tuple back into a list. (That would have to be code that modifies `_fields` in place, since building a new RecordArray with new `fields` can always wrap input lists and save them as tuples. I don't _think_ there's a lot of in-place RecordArray manipulation happening, and I guess it could be easy to search for, as "`self._fields`" on the left-hand side of an assignment.)

> I think we might want to formalise our contents API too

Yes! If the documentation just has a list of the ones that are expected on all Contents, that would already help us a lot. We can find the common set algorithmically:

```python
>>> # Content subclasses (same for Forms and Types)
>>> clss = [getattr(ak._v2.contents, x) for x in dir(ak._v2.contents) if x.endswith("Array")]

>>> # methods defined on all of them
>>> tmp = set.intersection(*[set(dir(x)) for x in clss])

>>> # maybe just the public ones
>>> tmp = set.intersection(*[set([y for y in dir(x) if not y.startswith("_")]) for x in clss])
>>> print("\n".join(sorted(tmp)))
Form
all
any
argmax
argmin
argsort
axis_wrap_if_negative
branch_depth
combinations
completely_flatten
copy
count
count_nonzero
dimension_optiontype
dummy
fields
fill_none
flatten
forget_length
form
form_with_key
identifier
is_IndexedType
is_ListType
is_NumpyType
is_OptionType
is_RecordType
is_RegularType
is_UnionType
is_UnknownType
is_tuple
is_unique
jax_flatten
jax_unflatten
layout_equal
length
local_index
max
maybe_to_array
merge
merge_as_union
merge_parameters
mergeable
mergemany
min
minmax_depth
nbytes
nplike
num
numbers_to_type
packed
pad_none
pad_none_axis0
parameter
parameters
prod
purelist_depth
purelist_isregular
purelist_parameter
recursively_apply
sort
sum
to_arrow
to_backend
to_buffers
to_json
to_list
to_numpy
tolist
typetracer
unique
validity_error
validity_error_parameters
with_parameter
```

There might be others that are accidentally missing from one or two classes, and still others that are supposed to be only defined on one class or all `is_ListType` or all `is_OptionType`. Some variants of Awkward had a non-trivial class hierarchy to group things as list-like or option-like, but it's not a strict tree because `is_IndexType` is an important qualifier, and it straddles option-type and non-option-type.

Resolving patch:
diff --git a/src/awkward/_v2/operations/ak_fields.py b/src/awkward/_v2/operations/ak_fields.py
--- a/src/awkward/_v2/operations/ak_fields.py
+++ b/src/awkward/_v2/operations/ak_fields.py
@@ -27,4 +27,4 @@ def fields(array):
 
 def _impl(array):
     layout = ak._v2.operations.to_layout(array, allow_record=True, allow_other=False)
-    return layout.fields
+    return layout.fields.copy()
diff --git a/src/awkward/_v2/operations/ak_parameters.py b/src/awkward/_v2/operations/ak_parameters.py
--- a/src/awkward/_v2/operations/ak_parameters.py
+++ b/src/awkward/_v2/operations/ak_parameters.py
@@ -1,5 +1,8 @@
 # BSD 3-Clause License; see https://github.com/scikit-hep/awkward-1.0/blob/main/LICENSE
 
+import copy
+import numbers
+
 import awkward as ak
 
 np = ak.nplike.NumpyMetadata.instance()
@@ -27,13 +30,13 @@ def parameters(array):
 
 def _impl(array):
     if isinstance(array, (ak._v2.highlevel.Array, ak._v2.highlevel.Record)):
-        return array.layout.parameters
+        return _copy(array.layout.parameters)
 
     elif isinstance(
         array,
         (ak._v2.contents.Content, ak._v2.record.Record),
     ):
-        return array.parameters
+        return _copy(array.parameters)
 
     elif isinstance(array, ak._v2.highlevel.ArrayBuilder):
         return array.snapshot().layout.parameters
@@ -43,3 +46,10 @@ def _impl(array):
 
     else:
         return {}
+
+
+def _copy(what):
+    if all(isinstance(x, (str, numbers.Real)) for x in what.values()):
+        return what.copy()
+    else:
+        return copy.deepcopy(what)


Test patch: