internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: scikit-hep/awkward

Issue:
Add a "field does not exist" `IndexError` subclass
### Description of new feature

This will enable framework authors to identify problems with typetracer usage leading to loss of important fields.


Discussion:

Resolving patch:
diff --git a/src/awkward/_errors.py b/src/awkward/_errors.py
--- a/src/awkward/_errors.py
+++ b/src/awkward/_errors.py
@@ -352,3 +352,7 @@ def deprecate(
         version, date, will_be, message
     )
     warnings.warn(warning, category, stacklevel=stacklevel + 1)
+
+
+class FieldNotFoundError(IndexError):
+    ...
diff --git a/src/awkward/forms/recordform.py b/src/awkward/forms/recordform.py
--- a/src/awkward/forms/recordform.py
+++ b/src/awkward/forms/recordform.py
@@ -128,7 +128,7 @@ def field_to_index(self, field):
             else:
                 return i
         raise ak._errors.wrap_error(
-            IndexError(
+            ak._errors.FieldNotFoundError(
                 "no field {} in record with {} fields".format(
                     repr(field), len(self._contents)
                 )


Test patch:
diff --git a/tests/test_2120_missing_field_error.py b/tests/test_2120_missing_field_error.py
new file mode 100644
--- /dev/null
+++ b/tests/test_2120_missing_field_error.py
@@ -0,0 +1,20 @@
+# BSD 3-Clause License; see https://github.com/scikit-hep/awkward-1.0/blob/main/LICENSE
+
+import pytest  # noqa: F401
+
+import awkward as ak
+
+
+def test():
+    array = ak.Array(
+        [
+            {"x": 100, "y": 200},
+            {"x": 300, "y": 400},
+        ]
+    )
+    with pytest.raises(ak._errors.FieldNotFoundError):
+        array["z"]
+    with pytest.raises(ak._errors.FieldNotFoundError):
+        array[["z", "k"]]
+    with pytest.raises(ak._errors.FieldNotFoundError):
+        array[0, "z"]
Repository: scikit-hep/awkward

Issue:
ak.any Error message gives suggestion which will also error
### Version of Awkward Array

2.0.6

### Description and code to reproduce

```python
import awkward as ak

ak.any(ak.Array([{"a": False}]))
```

Throws an error that says:

```pytb
TypeError: while calling

    ak.any(
        array = <Array [{a: False}] type='1 * {a: bool}'>
        axis = None
        keepdims = False
        mask_identity = False
        highlevel = True
        behavior = None
    )

Error details: cannot combine record fields unless flatten_records=True
```

But the suggested command also errors, telling me to do something else:

```python
ak.any(ak.Array([{"a": False}]), flatten_records=True)
```

```pytb
ValueError: while calling

    ak.any(
        array = <Array [{a: False}] type='1 * {a: bool}'>
        axis = None
        keepdims = False
        mask_identity = False
        highlevel = True
        behavior = None
    )

Error details: `flatten_records` is no longer a supported argument for reducers. Instead, use `ak.ravel(array)` first to remove the record structure and flatten the array.
```

`ak.ravel` should probably be suggested at the first point.


Discussion:

Resolving patch:
diff --git a/src/awkward/contents/recordarray.py b/src/awkward/contents/recordarray.py
--- a/src/awkward/contents/recordarray.py
+++ b/src/awkward/contents/recordarray.py
@@ -934,9 +934,11 @@ def _remove_structure(self, backend, options):
                 in_function = " in " + options["function_name"]
             raise ak._errors.wrap_error(
                 TypeError(
-                    "cannot combine record fields{} unless flatten_records=True".format(
-                        in_function
-                    )
+                    (
+                        "encountered a record whilst removing array structure{}, "
+                        "but this operation does not support erasing records. "
+                        "Try first calling `ak.ravel` or `ak.flatten(..., axis=None)."
+                    ).format(in_function)
                 )
             )
 


Test patch:
Repository: scikit-hep/awkward

Issue:
Example in merge_union_of_records is a no-op
### Which documentation?

Python docstrings

### What needs to be documented?

In the example given, the result type is the same as the input type:

<details>
<summary> Quote from the docstring </summary>

```python
Simplifies unions of records, e.g.

    >>> array = ak.Array([{"a": 1}, {"b": 2}])

into records of options, i.e.

    >>> ak.merge_union_of_records(array)
    <Array [{a: 1, b: None}, {a: None, ...}] type='2 * {a: ?int64, b: ?int64}'>
```

</details>

```python
>>> import awkward as ak
>>> ak.Array([{"a": 1}, {"b": 2}]).type.show()
2 * {
    a: ?int64,
    b: ?int64
}
>>> ak.merge_union_of_records(ak.Array([{"a": 1}, {"b": 2}])).type.show()
2 * {
    a: ?int64,
    b: ?int64
}
```


Discussion:
Gah, this is the reason for doctests. I made a last minute change here. Thanks for catching it!

Resolving patch:
diff --git a/src/awkward/operations/ak_merge_union_of_records.py b/src/awkward/operations/ak_merge_union_of_records.py
--- a/src/awkward/operations/ak_merge_union_of_records.py
+++ b/src/awkward/operations/ak_merge_union_of_records.py
@@ -23,7 +23,7 @@ def merge_union_of_records(array, axis=-1, *, highlevel=True, behavior=None):
 
     Simplifies unions of records, e.g.
 
-        >>> array = ak.Array([{"a": 1}, {"b": 2}])
+        >>> array = ak.concatenate(([{"a": 1}], [{"b": 2}]))
 
     into records of options, i.e.
 


Test patch:
Repository: scikit-hep/awkward

Issue:
Tests are failing in "Build Docs", possibly due to new Sphinx theme
              Is pydata_sphinx_theme broken, perhaps? It had a 0.13 release 12 hours ago.

_Originally posted by @henryiii in https://github.com/scikit-hep/awkward/issues/2268#issuecomment-1448934363_

For example, https://github.com/scikit-hep/awkward/actions/runs/4297198800/jobs/7489883737

This open issue, https://github.com/pydata/pydata-sphinx-theme/issues/1149, has the same error message and they say

> I think it ought to be possible to run our basic dev commands (test, docs, docs-live) in the dev environment we recommend contributors to use

which sounds like something we do.


Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -84,13 +84,13 @@
 html_show_sourcelink = True
 html_theme_options = {
     "logo": {
-        "image_light": "image/logo-300px.png",
-        "image_dark": "image/logo-300px-white.png",
+        "image_light": "_static/image/logo-300px.png",
+        "image_dark": "_static/image/logo-300px-white.png",
     },
     "github_url": "https://github.com/scikit-hep/awkward",
     # Add light/dark mode and documentation version switcher:
     "navbar_end": ["theme-switcher", "navbar-icon-links"],
-    "footer_items": ["copyright", "sphinx-version", "funding"],
+    "footer_start": ["copyright", "sphinx-version", "funding"],
     "icon_links": [
         {
             "name": "PyPI",


Test patch:
Repository: scikit-hep/awkward

Issue:
Deprecate `highlevel=True` for `Form.length_zero_array`
### Description of new feature

It seems to me that `Form.length_zero_array` is an L2 function, and therefore doesn't need to be concerned with behaviours or `ak.Array`s. I propose that we deprecate the `highlevel=True` variant of this function, to keep the separation of concerns.


Discussion:
I agree.

Resolving patch:
diff --git a/src/awkward/forms/form.py b/src/awkward/forms/form.py
--- a/src/awkward/forms/form.py
+++ b/src/awkward/forms/form.py
@@ -9,6 +9,7 @@
 import awkward as ak
 from awkward._backends.numpy import NumpyBackend
 from awkward._behavior import find_typestrs
+from awkward._errors import deprecate
 from awkward._nplikes.numpylike import NumpyMetadata
 from awkward._nplikes.shape import unknown_length
 from awkward._parameters import parameters_union
@@ -427,6 +428,12 @@ def _type(self, typestrs):
     def length_zero_array(
         self, *, backend=numpy_backend, highlevel=True, behavior=None
     ):
+        if highlevel:
+            deprecate(
+                "The `highlevel=True` variant of `Form.length_zero_array` is now deprecated. "
+                "Please use `ak.Array(form.length_zero_array(...), behavior=...)` if an `ak.Array` is required.",
+                version="2.3.0",
+            )
         return ak.operations.ak_from_buffers._impl(
             form=self,
             length=0,
diff --git a/src/awkward/operations/ak_from_parquet.py b/src/awkward/operations/ak_from_parquet.py
--- a/src/awkward/operations/ak_from_parquet.py
+++ b/src/awkward/operations/ak_from_parquet.py
@@ -1,6 +1,7 @@
 # BSD 3-Clause License; see https://github.com/scikit-hep/awkward-1.0/blob/main/LICENSE
 __all__ = ("from_parquet",)
 import awkward as ak
+from awkward._layout import wrap_layout
 from awkward._regularize import is_integer
 
 
@@ -230,7 +231,9 @@ def _load(
         )
 
     if len(arrays) == 0:
-        return subform.length_zero_array(highlevel=highlevel, behavior=behavior)
+        return wrap_layout(
+            subform.length_zero_array(highlevel=False), behavior=behavior
+        )
     elif len(arrays) == 1:
         # make high-level
         if isinstance(arrays[0], ak.record.Record):


Test patch:
Repository: scikit-hep/awkward

Issue:
Add uniqueness of `fields` to `_validity_error`
### Version of Awkward Array

main

### Description and code to reproduce

It's currently possible to create "invalid" `RecordArrays` that contain duplicate fields. We should check this.


Discussion:

Resolving patch:
diff --git a/src/awkward/contents/recordarray.py b/src/awkward/contents/recordarray.py
--- a/src/awkward/contents/recordarray.py
+++ b/src/awkward/contents/recordarray.py
@@ -905,6 +905,14 @@ def _validity_error(self, path):
             sub = cont._validity_error(f"{path}.field({i})")
             if sub != "":
                 return sub
+
+        # Check for duplicate fields
+        if not self.is_tuple:
+            seen_fields = set()
+            for field in self._fields:
+                if field in seen_fields:
+                    return f"at {path} ({type(self)!r}): duplicate field {field!r}"
+                seen_fields.add(field)
         return ""
 
     def _nbytes_part(self):


Test patch:
diff --git a/tests/test_2296_duplicate_field.py b/tests/test_2296_duplicate_field.py
new file mode 100644
--- /dev/null
+++ b/tests/test_2296_duplicate_field.py
@@ -0,0 +1,29 @@
+# BSD 3-Clause License; see https://github.com/scikit-hep/awkward-1.0/blob/main/LICENSE
+
+import re
+
+import pytest  # noqa: F401
+
+import awkward as ak
+
+
+def test_invalid():
+    layout = ak.contents.RecordArray(
+        [
+            ak.contents.NumpyArray([1, 2, 3]),
+            ak.contents.NumpyArray([1, 2, 3]),
+        ],
+        ["x", "x"],
+    )
+    assert re.match(r".*duplicate field 'x'.*", ak.validity_error(layout)) is not None
+
+
+def test_valid():
+    layout = ak.contents.RecordArray(
+        [
+            ak.contents.NumpyArray([1, 2, 3]),
+            ak.contents.NumpyArray([1, 2, 3]),
+        ],
+        ["x", "y"],
+    )
+    assert re.match(r".*duplicate field 'x'.*", ak.validity_error(layout)) is None
Repository: scikit-hep/awkward

Issue:
`ak.flatten` flattens strings with `axis != None`
### Version of Awkward Array

main

### Description and code to reproduce

We are leaning towards strings being a robust abstraction — if you want to erase a string, remove the parameters (or use `ak.enforce_type`). 

However, there are some holes in this, notably with `ak.flatten`:
```python
>>> ak.flatten(["hello", "moto"], axis=1)
'hellomoto'
```

We should ensure that this raises an error.


Discussion:

Resolving patch:
diff --git a/src/awkward/contents/listoffsetarray.py b/src/awkward/contents/listoffsetarray.py
--- a/src/awkward/contents/listoffsetarray.py
+++ b/src/awkward/contents/listoffsetarray.py
@@ -718,6 +718,15 @@ def _offsets_and_flattened(self, axis, depth):
             raise AxisError("axis=0 not allowed for flatten")
 
         elif posaxis is not None and posaxis + 1 == depth + 1:
+            if (
+                self.parameter("__array__") == "string"
+                or self.parameter("__array__") == "bytestring"
+            ):
+                raise ValueError(
+                    "array of strings cannot be directly flattened. "
+                    'To flatten this array, drop the `"__array__"="string"` parameter using '
+                    "`ak.enforce_type`, `ak.with_parameter`, or `ak.without_parameters`/"
+                )
             listoffsetarray = self.to_ListOffsetArray64(True)
             stop = listoffsetarray.offsets[-1]
             content = listoffsetarray.content._getitem_range(0, stop)


Test patch:
diff --git a/tests/test_2471_flatten_string.py b/tests/test_2471_flatten_string.py
new file mode 100644
--- /dev/null
+++ b/tests/test_2471_flatten_string.py
@@ -0,0 +1,26 @@
+# BSD 3-Clause License; see https://github.com/scikit-hep/awkward-1.0/blob/main/LICENSE
+
+import pytest
+
+import awkward as ak
+
+
+def test():
+    array = ak.Array(
+        ["is", "this", "real", "life", "or", "is", "this", "just", "fantasy"]
+    )
+    with pytest.raises(ValueError, match=r"strings cannot be directly flattened"):
+        ak.flatten(array, axis=1)
+
+    array = ak.Array(
+        ["is", "this", "real", "life", "or", "is", "this", "just", "fantasy"]
+    )
+    assert ak.to_layout(array) == ak.flatten(array, axis=0, highlevel=False)
+
+    array_ragged = ak.Array(
+        [["is", "this", "real", "life"], ["or", "is", "this", "just", "fantasy"]]
+    )
+    assert ak.almost_equal(array, ak.flatten(array_ragged, axis=1))
+
+    with pytest.raises(ValueError, match=r"strings cannot be directly flattened"):
+        ak.flatten(array_ragged, axis=2)
Repository: scikit-hep/awkward

Issue:
`NumpyArray` should not be string-like
### Version of Awkward Array

main

### Description and code to reproduce

At present, we have [code that implies `NumpyArray` can be a string](https://github.com/scikit-hep/awkward/blob/f3ce29e0a6014ebbce8a722826abc6fc7ae14791/src/awkward/contents/numpyarray.py#L665) (list of char/byte). Whilst this is only a single line of code, it makes me realise that we don't have a clear policy here. Without the ability to store parameters on a child content, we would need to implement this _at present_ with special logic on `NumpyArray` to add these parameters when e.g. pulling out a string. 

I don't think this is necessary; 2D `NumpyArray`s are only really useful if you have a non-contiguous layout, as it defers the need to render it contiguous. Given that strings are already converted into `ListArray` layouts with `from_numpy`, I think we could impose the restriction that `NumpyArray` cannot have `__array__ = "string"`, and simply require that a `RegularArray` be used for such a purpose.

Thoughts? @jpivarski 


Discussion:

Resolving patch:
diff --git a/src/awkward/contents/numpyarray.py b/src/awkward/contents/numpyarray.py
--- a/src/awkward/contents/numpyarray.py
+++ b/src/awkward/contents/numpyarray.py
@@ -674,9 +674,7 @@ def _as_unique_strings(self, offsets):
 
     def _numbers_to_type(self, name, including_unknown):
         if (
-            self.parameter("__array__") == "string"
-            or self.parameter("__array__") == "bytestring"
-            or self.parameter("__array__") == "char"
+            self.parameter("__array__") == "char"
             or self.parameter("__array__") == "byte"
         ):
             return self


Test patch:
Repository: scikit-hep/awkward

Issue:
We shouldn't decorate the `KeyboardInterrupt` exception
### Version of Awkward Array

HEAD

### Description and code to reproduce

If I press control-C (or ■ in Jupyter), I don't need to be told what high-level function failed because it didn't really "fail," it was cancelled.

```
KeyboardInterrupt: 

This error occurred while attempting to slice

    <MomentumArray4D [[{pt: 63, eta: ..., ...}, ...], ...] type='299973 * v...'>

with

    'phi'
```

Similarly, we have to think about what will happen if someone cancels an asynchronous GPU process—does the background thread get informed? That's beyond this issue, though.


Discussion:
Yes, we should catch exception and not base exception. I'm surprised this happens (without looking at the source), as it's a common pattern  that I'd expect us to have anticipated. I'll make a patch soon. 

Resolving patch:
diff --git a/src/awkward/_errors.py b/src/awkward/_errors.py
--- a/src/awkward/_errors.py
+++ b/src/awkward/_errors.py
@@ -59,7 +59,11 @@ def __enter__(self):
     def __exit__(self, exception_type, exception_value, traceback):
         try:
             # Handle caught exception
-            if exception_type is not None and self.primary() is self:
+            if (
+                exception_type is not None
+                and issubclass(exception_type, Exception)
+                and self.primary() is self
+            ):
                 self.handle_exception(exception_type, exception_value)
         finally:
             # `_kwargs` may hold cyclic references, that we really want to avoid


Test patch:
Repository: scikit-hep/awkward

Issue:
Support placeholders in `to_buffers`
### Version of Awkward Array

main

### Description and code to reproduce

The title says it all!


Discussion:

Resolving patch:
diff --git a/src/awkward/_nplikes/array_module.py b/src/awkward/_nplikes/array_module.py
--- a/src/awkward/_nplikes/array_module.py
+++ b/src/awkward/_nplikes/array_module.py
@@ -55,9 +55,13 @@ def ascontiguousarray(self, x: ArrayLike) -> ArrayLike:
     def frombuffer(
         self, buffer, *, dtype: np.dtype | None = None, count: int = -1
     ) -> ArrayLike:
-        assert not isinstance(buffer, PlaceholderArray)
-        assert not isinstance(count, PlaceholderArray)
-        return self._module.frombuffer(buffer, dtype=dtype, count=count)
+        if isinstance(buffer, PlaceholderArray):
+            if count == -1:
+                return self.asarray(buffer)
+            else:
+                return self.asarray(buffer[:count])
+        else:
+            return self._module.frombuffer(buffer, dtype=dtype, count=count)
 
     def from_dlpack(self, x: Any) -> ArrayLike:
         return self._module.from_dlpack(x)
diff --git a/src/awkward/_singleton.py b/src/awkward/_singleton.py
--- a/src/awkward/_singleton.py
+++ b/src/awkward/_singleton.py
@@ -9,6 +9,9 @@ class Singleton(Protocol):
     def __new__(cls, *args, **kwargs):
         return super().__new__(cls, *args, **kwargs)
 
+    def __reduce__(self):
+        return type(self).instance, ()
+
     @classmethod
     def instance(cls) -> Self:
         try:


Test patch:
diff --git a/tests/test_2693_to_buffers_placeholders.py b/tests/test_2693_to_buffers_placeholders.py
new file mode 100644
--- /dev/null
+++ b/tests/test_2693_to_buffers_placeholders.py
@@ -0,0 +1,34 @@
+# BSD 3-Clause License; see https://github.com/scikit-hep/awkward-1.0/blob/main/LICENSE
+import pickle
+
+import numpy as np
+
+import awkward as ak
+from awkward._nplikes.numpy import Numpy
+from awkward.typetracer import PlaceholderArray
+
+nplike = Numpy.instance()
+
+
+def test_round_trip_placeholder():
+    layout = ak.contents.RecordArray(
+        [
+            ak.contents.NumpyArray(
+                PlaceholderArray(nplike, (4,), np.dtype(np.float64))
+            ),
+            ak.contents.NumpyArray([1, 2, 3, 4]),
+        ],
+        ["x", "y"],
+    )
+
+    form, length, container = ak.to_buffers(layout)
+    result = ak.from_buffers(form, length, container, highlevel=False)
+
+    assert isinstance(result.content("x").data, PlaceholderArray)
+
+    assert result.content("x").data.shape == layout.content("x").data.shape
+    assert result.content("x").data.dtype == layout.content("x").data.dtype
+
+
+def test_pickle_nplike():
+    assert pickle.loads(pickle.dumps(nplike)) is nplike
Repository: scikit-hep/awkward

Issue:
`ak.argmax` and `ak.argmin`: support for Boolean arrays
### Description of new feature

`ak.argmax` and `ak.argmin` do not support Boolean arrays unless they are first cast to integer. Meanwhile, `np.argmax` and `np.argmin` can operate on Boolean arrays directly:

```python
import awkward as ak
import numpy as np

assert ak.__version__ == '2.5.1'
assert np.__version__ == '1.24.4'

np_bool_array = np.array([[1, 2, 3], [3, 2, 1]]) > 2
assert np.argmax(np_bool_array, axis=1).tolist() == [2, 0]
assert np.argmin(np_bool_array, axis=1).tolist() == [0, 1]

ak_bool_array = ak.Array([[1, 2, 3], [4, 3, 2, 1]]) > 2

try:
    ak.argmax(ak_bool_array, axis=1)
except KeyError:
    print("this doesn't work :(")

try:
    ak.argmax(np_bool_array, axis=1)
except KeyError:
    print("this doesn't work either :(")

# casting to `int` works:
ak_int_array = ak.values_astype(ak_bool_array, int)
assert ak.argmax(ak_int_array, axis=1).tolist() == [2, 0]
assert ak.argmin(ak_int_array, axis=1).tolist() == [0, 2]
```


Discussion:
@delannoy - thanks for brining it up! I think, we do want to be as close as possible to `Numpy`. Python, from version 2.3 forward, has a `bool` type which is a subclass of `int`, the standard integer type. It has two possible values: `True` and `False`, which are special versions of `1` and `0` respectively and behave as such in arithmetic contexts.
Agreed with @ianna, we should support this. I'll make a PR! Thanks for the report :)

Resolving patch:
diff --git a/src/awkward/_reducers.py b/src/awkward/_reducers.py
--- a/src/awkward/_reducers.py
+++ b/src/awkward/_reducers.py
@@ -129,6 +129,14 @@ class ArgMin(KernelReducer):
     preferred_dtype: Final = np.int64
     needs_position: Final = True
 
+    @classmethod
+    def _dtype_for_kernel(cls, dtype: DTypeLike) -> DTypeLike:
+        dtype = np.dtype(dtype)
+        if dtype == np.bool_:
+            return np.dtype(np.int8)
+        else:
+            return super()._dtype_for_kernel(dtype)
+
     def apply(
         self,
         array: ak.contents.NumpyArray,
@@ -183,6 +191,14 @@ class ArgMax(KernelReducer):
     preferred_dtype: Final = np.int64
     needs_position: Final = True
 
+    @classmethod
+    def _dtype_for_kernel(cls, dtype: DTypeLike) -> DTypeLike:
+        dtype = np.dtype(dtype)
+        if dtype == np.bool_:
+            return np.dtype(np.int8)
+        else:
+            return super()._dtype_for_kernel(dtype)
+
     def apply(
         self,
         array: ak.contents.NumpyArray,


Test patch:
diff --git a/tests/test_2934_argmin_argmax_bool.py b/tests/test_2934_argmin_argmax_bool.py
new file mode 100644
--- /dev/null
+++ b/tests/test_2934_argmin_argmax_bool.py
@@ -0,0 +1,12 @@
+# BSD 3-Clause License; see https://github.com/scikit-hep/awkward/blob/main/LICENSE
+
+from __future__ import annotations
+
+import awkward as ak
+
+
+def test():
+    array = ak.Array([[True, False, True], [False, False, True, False], [True, False]])
+
+    assert ak.argmin(array, axis=1).tolist() == [1, 0, 1]
+    assert ak.argmax(array, axis=1).tolist() == [0, 2, 0]