Repository: scikit-hep/awkward

Issue:
Performance regressions compared to ak1
### Version of Awkward Array

2.1.4

### Description and code to reproduce

Long story short in the `trijet["p4"] = trijet.j1 + trijet.j2 + trijet.j3` I am seeing up to a 2.5x longer execution time in ak2 in the following code. 

I see also about 2x more peak RSS usage in top, but using python's internal max RSS monitor from resource they show nearly the same in terms of max RSS (but that may not count something?). I'll see if I can get a better measure.

Here's the code (you will need to `pip install pyinstrument`):
```python3
import numpy as np
import awkward as ak

from pyinstrument import Profiler

np.random.seed(1)

behavior = {}

@ak.mixin_class(behavior)
class LorentzVector:
    @property
    def mass2(self):
        """Squared `mass`"""
        return self.t**2 - self.x**2 - self.y**2 - self.z**2

    @property
    def mass(self):
        r"""Invariant mass (+, -, -, -)
        :math:`\sqrt{t^2-x^2-y^2-z^2}`
        """
        return np.sqrt(self.mass2)
    
    @ak.mixin_class_method(np.add, {"LorentzVector"})
    def add(self, other):
        """Add two vectors together elementwise using `x`, `y`, `z`, and `t` components"""
        return ak.zip(
            {
                "x": self.x + other.x,
                "y": self.y + other.y,
                "z": self.z + other.z,
                "t": self.t + other.t,
            },
            with_name="LorentzVector",
            behavior=self.behavior,
        )

rows = 2_000_000

counts = np.rint(np.random.lognormal(mean=1.0, sigma=0.8, size=rows)).astype(np.int64)
total = np.sum(counts)
print(np.mean(counts), np.min(counts), np.max(counts))
x = np.random.uniform(low=-250.0/1.74, high=250.0/1.74, size=total)
y = np.random.uniform(low=-250.0/1.74, high=250.0/1.74, size=total)
z = np.random.uniform(low=-250.0/1.74, high=250.0/1.74, size=total)
t = np.random.uniform(low=250.0, high=500.0, size=total)
btag = np.random.uniform(low=0.0, high=1.0, size=total)

jets = ak.zip(
    {
        "x": ak.unflatten(x, counts),
        "y": ak.unflatten(y, counts),
        "z": ak.unflatten(z, counts),
        "t": ak.unflatten(t, counts),
        "btag": ak.unflatten(btag, counts),
    },
    with_name="LorentzVector",
    behavior=behavior,
)

prof = Profiler()

prof.start()
trijet = ak.combinations(jets, 3, fields=["j1", "j2", "j3"])
prof.stop()

print(prof.output_text(unicode=True, color=True, show_all=True))

prof = Profiler()

prof.start()
trijet["p4"] = trijet.j1 + trijet.j2 + trijet.j3
prof.stop()

print(prof.output_text(unicode=True, color=True, show_all=True))

prof = Profiler()

prof.start()
trijet = ak.flatten(
    trijet[ak.singletons(ak.argmin(abs(trijet.p4.mass - 172.5), axis=1))]
)
prof.stop()

print(prof.output_text(unicode=True, color=True, show_all=True))
```
produces the following profiles in ak2:
```
(coffea-dev) lgray@Lindseys-MBP dask-demo-day-20042023 % python regression_repro.py
3.7437375 0 112

  _     ._   __/__   _ _  _  _ _/_   Recorded: 17:05:59  Samples:  24
 /_//_/// /_\ / //_// / //_'/ //     Duration: 1.174     CPU time: 1.173
/   _/                      v4.3.0

Program: regression_repro.py

1.174 <module>  regression_repro.py:1
└─ 1.174 combinations  awkward/operations/ak_combinations.py:11
   └─ 1.174 _impl  awkward/operations/ak_combinations.py:205
      └─ 1.174 combinations  awkward/_do.py:125
         ├─ 1.138 _combinations  awkward/contents/listoffsetarray.py:1353
         │  ├─ 0.574 __call__  awkward/_kernels.py:89
         │  └─ 0.560 _carry  awkward/contents/recordarray.py:496
         │     ├─ 0.300 [self]  
         │     ├─ 0.239 ndarray.copy  <built-in>:0
         │     └─ 0.021 any  awkward/_nplikes/array_module.py:349
         │        └─ 0.021 any  <__array_function__ internals>:177
         │           └─ 0.021 any  numpy/core/fromnumeric.py:2307
         │              └─ 0.021 _wrapreduction  numpy/core/fromnumeric.py:69
         │                 └─ 0.021 ufunc.reduce  <built-in>:0
         └─ 0.036 [self]  



  _     ._   __/__   _ _  _  _ _/_   Recorded: 17:06:01  Samples:  37
 /_//_/// /_\ / //_// / //_'/ //     Duration: 9.148     CPU time: 6.215
/   _/                      v4.3.0

Program: regression_repro.py

9.148 <module>  regression_repro.py:1
└─ 9.143 func  numpy/lib/mixins.py:18
   └─ 9.143 __array_ufunc__  awkward/highlevel.py:1289
      └─ 9.143 array_ufunc  awkward/_connect/numpy.py:219
         └─ 9.143 broadcast_and_apply  awkward/_broadcasting.py:1046
            └─ 9.143 apply_step  awkward/_broadcasting.py:369
               └─ 9.143 continuation  awkward/_broadcasting.py:991
                  └─ 9.143 broadcast_any_list  awkward/_broadcasting.py:499
                     └─ 9.143 apply_step  awkward/_broadcasting.py:369
                        └─ 9.143 continuation  awkward/_broadcasting.py:991
                           └─ 9.143 broadcast_any_list  awkward/_broadcasting.py:499
                              └─ 9.137 apply_step  awkward/_broadcasting.py:369
                                 └─ 9.137 continuation  awkward/_broadcasting.py:991
                                    └─ 9.074 broadcast_any_indexed  awkward/_broadcasting.py:947
                                       ├─ 5.246 apply_step  awkward/_broadcasting.py:369
                                       │  └─ 5.246 action  awkward/_connect/numpy.py:228
                                       │     └─ 5.246 _array_ufunc_adjust  awkward/_connect/numpy.py:168
                                       │        └─ 5.246 add  regression_repro.py:24
                                       │           ├─ 4.914 func  numpy/lib/mixins.py:18
                                       │           │  └─ 4.914 __array_ufunc__  awkward/highlevel.py:1289
                                       │           │     └─ 4.914 array_ufunc  awkward/_connect/numpy.py:219
                                       │           │        └─ 4.914 broadcast_and_apply  awkward/_broadcasting.py:1046
                                       │           │           └─ 4.914 apply_step  awkward/_broadcasting.py:369
                                       │           │              └─ 4.914 continuation  awkward/_broadcasting.py:991
                                       │           │                 └─ 4.914 broadcast_any_list  awkward/_broadcasting.py:499
                                       │           │                    └─ 4.914 apply_step  awkward/_broadcasting.py:369
                                       │           │                       └─ 4.914 action  awkward/_connect/numpy.py:228
                                       │           │                          └─ 4.914 apply_ufunc  awkward/_backends/backend.py:46
                                       │           └─ 0.332 zip  awkward/operations/ak_zip.py:11
                                       │              └─ 0.332 _impl  awkward/operations/ak_zip.py:159
                                       │                 └─ 0.332 broadcast_and_apply  awkward/_broadcasting.py:1046
                                       │                    └─ 0.332 apply_step  awkward/_broadcasting.py:369
                                       │                       └─ 0.332 continuation  awkward/_broadcasting.py:991
                                       │                          └─ 0.332 broadcast_any_list  awkward/_broadcasting.py:499
                                       │                             └─ 0.332 apply_step  awkward/_broadcasting.py:369
                                       └─ 3.827 <listcomp>  awkward/_broadcasting.py:948
                                          └─ 3.797 project  awkward/contents/indexedarray.py:408
                                             ├─ 3.352 _carry  awkward/contents/recordarray.py:496
                                             │  └─ 3.352 <listcomp>  awkward/contents/recordarray.py:517
                                             │     └─ 3.348 _carry  awkward/contents/numpyarray.py:315
                                             └─ 0.445 __call__  awkward/_kernels.py:89



  _     ._   __/__   _ _  _  _ _/_   Recorded: 17:06:10  Samples:  37
 /_//_/// /_\ / //_// / //_'/ //     Duration: 7.386     CPU time: 5.410
/   _/                      v4.3.0

Program: regression_repro.py

7.386 <module>  regression_repro.py:1
├─ 6.550 mass  regression_repro.py:17
│  ├─ 4.398 mass2  regression_repro.py:12
│  │  └─ 4.398 func  numpy/lib/mixins.py:18
│  │     └─ 4.398 __array_ufunc__  awkward/highlevel.py:1289
│  │        └─ 4.398 array_ufunc  awkward/_connect/numpy.py:219
│  │           ├─ 4.013 recursively_apply  awkward/_do.py:20
│  │           │  └─ 4.013 _recursively_apply  awkward/contents/listoffsetarray.py:2069
│  │           │     └─ 4.006 continuation  awkward/contents/listoffsetarray.py:2084
│  │           │        └─ 4.006 _recursively_apply  awkward/contents/numpyarray.py:1261
│  │           │           └─ 4.006 unary_action  awkward/_connect/numpy.py:309
│  │           │              └─ 4.006 action  awkward/_connect/numpy.py:228
│  │           │                 └─ 4.006 apply_ufunc  awkward/_backends/backend.py:46
│  │           └─ 0.385 broadcast_and_apply  awkward/_broadcasting.py:1046
│  │              └─ 0.385 apply_step  awkward/_broadcasting.py:369
│  │                 └─ 0.385 continuation  awkward/_broadcasting.py:991
│  │                    └─ 0.385 broadcast_any_list  awkward/_broadcasting.py:499
│  │                       └─ 0.385 apply_step  awkward/_broadcasting.py:369
│  │                          └─ 0.385 continuation  awkward/_broadcasting.py:991
│  │                             └─ 0.385 broadcast_any_list  awkward/_broadcasting.py:499
│  │                                └─ 0.379 apply_step  awkward/_broadcasting.py:369
│  │                                   └─ 0.379 action  awkward/_connect/numpy.py:228
│  │                                      └─ 0.379 apply_ufunc  awkward/_backends/backend.py:46
│  └─ 2.152 __array_ufunc__  awkward/highlevel.py:1289
│     └─ 2.152 array_ufunc  awkward/_connect/numpy.py:219
│        └─ 2.152 recursively_apply  awkward/_do.py:20
│           └─ 2.152 _recursively_apply  awkward/contents/listoffsetarray.py:2069
│              └─ 2.150 continuation  awkward/contents/listoffsetarray.py:2084
│                 └─ 2.150 _recursively_apply  awkward/contents/numpyarray.py:1261
│                    └─ 2.150 unary_action  awkward/_connect/numpy.py:309
│                       └─ 2.150 action  awkward/_connect/numpy.py:228
│                          └─ 2.150 apply_ufunc  awkward/_backends/backend.py:46
├─ 0.406 func  numpy/lib/mixins.py:18
│  └─ 0.406 __array_ufunc__  awkward/highlevel.py:1289
│     └─ 0.406 array_ufunc  awkward/_connect/numpy.py:219
│        └─ 0.406 recursively_apply  awkward/_do.py:20
│           └─ 0.406 _recursively_apply  awkward/contents/listoffsetarray.py:2069
│              ├─ 0.222 __getitem__  awkward/contents/content.py:536
│              │  └─ 0.222 _getitem  awkward/contents/content.py:539
│              │     └─ 0.222 _getitem_range  awkward/contents/numpyarray.py:295
│              │        └─ 0.222 __init__  awkward/contents/numpyarray.py:97
│              │           └─ 0.222 __instancecheck__  typing.py:1009
│              └─ 0.179 continuation  awkward/contents/listoffsetarray.py:2084
│                 └─ 0.179 _recursively_apply  awkward/contents/numpyarray.py:1261
│                    └─ 0.179 unary_action  awkward/_connect/numpy.py:309
│                       └─ 0.179 action  awkward/_connect/numpy.py:228
│                          └─ 0.179 apply_ufunc  awkward/_backends/backend.py:46
├─ 0.279 argmin  awkward/operations/ak_argmin.py:14
│  └─ 0.279 _impl  awkward/operations/ak_argmin.py:152
│     └─ 0.279 reduce  awkward/_do.py:259
│        └─ 0.267 _reduce_next  awkward/contents/listoffsetarray.py:1464
│           ├─ 0.167 _reduce_next  awkward/contents/numpyarray.py:1074
│           │  └─ 0.119 apply_reducer  awkward/_backends/backend.py:37
│           │     └─ 0.119 apply  awkward/_reducers.py:81
│           │        └─ 0.119 __call__  awkward/_kernels.py:89
│           └─ 0.099 __call__  awkward/_kernels.py:89
└─ 0.107 func  numpy/lib/mixins.py:53
   └─ 0.107 __array_ufunc__  awkward/highlevel.py:1289
      └─ 0.107 array_ufunc  awkward/_connect/numpy.py:219
         └─ 0.107 recursively_apply  awkward/_do.py:20
            └─ 0.107 _recursively_apply  awkward/contents/listoffsetarray.py:2069
               └─ 0.104 continuation  awkward/contents/listoffsetarray.py:2084
                  └─ 0.104 _recursively_apply  awkward/contents/numpyarray.py:1261
                     └─ 0.104 unary_action  awkward/_connect/numpy.py:309
                        └─ 0.104 action  awkward/_connect/numpy.py:228
                           └─ 0.104 apply_ufunc  awkward/_backends/backend.py:46
```
and in ak1:
```
(coffea-testing) lgray@Lindseys-MBP dask-demo-day-20042023 % python regression_repro.py
3.7437375 0 112

  _     ._   __/__   _ _  _  _ _/_   Recorded: 17:06:25  Samples:  1
 /_//_/// /_\ / //_// / //_'/ //     Duration: 0.570     CPU time: 0.568
/   _/                      v4.4.0

Program: regression_repro.py

0.570 <module>  regression_repro.py:1
└─ 0.570 combinations  awkward/operations/structure.py:3679



  _     ._   __/__   _ _  _  _ _/_   Recorded: 17:06:26  Samples:  19
 /_//_/// /_\ / //_// / //_'/ //     Duration: 3.171     CPU time: 2.911
/   _/                      v4.4.0

Program: regression_repro.py

3.171 <module>  regression_repro.py:1
├─ 3.133 LorentzVectorArray.func  numpy/lib/mixins.py:18
│  └─ 3.133 LorentzVectorArray.__array_ufunc__  awkward/highlevel.py:1353
│     └─ 3.133 array_ufunc  awkward/_connect/_numpy.py:54
│        └─ 3.133 broadcast_and_apply  awkward/_util.py:619
│           └─ 3.132 apply  awkward/_util.py:695
│              └─ 3.132 apply  awkward/_util.py:695
│                 ├─ 2.976 apply  awkward/_util.py:695
│                 │  ├─ 1.627 [self]  None
│                 │  └─ 1.350 apply  awkward/_util.py:695
│                 │     └─ 1.350 <lambda>  awkward/_connect/_numpy.py:177
│                 │        └─ 1.350 adjust  awkward/_connect/_numpy.py:70
│                 │           └─ 1.350 LorentzVectorArray.add  regression_repro.py:24
│                 │              └─ 1.350 Array.func  numpy/lib/mixins.py:18
│                 │                 └─ 1.350 Array.__array_ufunc__  awkward/highlevel.py:1353
│                 │                    └─ 1.350 array_ufunc  awkward/_connect/_numpy.py:54
│                 │                       └─ 1.350 broadcast_and_apply  awkward/_util.py:619
│                 │                          └─ 1.350 apply  awkward/_util.py:695
│                 │                             └─ 1.350 getfunction  awkward/_connect/_numpy.py:150
│                 └─ 0.153 [self]  None
└─ 0.036 [self]  None



  _     ._   __/__   _ _  _  _ _/_   Recorded: 17:06:29  Samples:  27
 /_//_/// /_\ / //_// / //_'/ //     Duration: 4.520     CPU time: 4.517
/   _/                      v4.4.0

Program: regression_repro.py

4.519 <module>  regression_repro.py:1
├─ 4.079 LorentzVectorArray.mass  regression_repro.py:17
│  ├─ 3.987 LorentzVectorArray.mass2  regression_repro.py:12
│  │  ├─ 3.932 Array.func  numpy/lib/mixins.py:18
│  │  │  └─ 3.932 Array.__array_ufunc__  awkward/highlevel.py:1353
│  │  │     └─ 3.932 array_ufunc  awkward/_connect/_numpy.py:54
│  │  │        └─ 3.932 broadcast_and_apply  awkward/_util.py:619
│  │  │           └─ 3.932 apply  awkward/_util.py:695
│  │  │              └─ 3.930 apply  awkward/_util.py:695
│  │  │                 └─ 3.926 apply  awkward/_util.py:695
│  │  │                    └─ 3.926 getfunction  awkward/_connect/_numpy.py:150
│  │  └─ 0.055 [self]  None
│  └─ 0.083 Array.__array_ufunc__  awkward/highlevel.py:1353
│     └─ 0.083 array_ufunc  awkward/_connect/_numpy.py:54
│        └─ 0.083 broadcast_and_apply  awkward/_util.py:619
│           └─ 0.083 apply  awkward/_util.py:695
│              └─ 0.083 apply  awkward/_util.py:695
│                 └─ 0.083 apply  awkward/_util.py:695
│                    └─ 0.083 getfunction  awkward/_connect/_numpy.py:150
├─ 0.234 argmin  awkward/operations/reducers.py:691
├─ 0.070 Array.func  numpy/lib/mixins.py:18
│  └─ 0.070 Array.__array_ufunc__  awkward/highlevel.py:1353
│     └─ 0.070 array_ufunc  awkward/_connect/_numpy.py:54
│        └─ 0.070 broadcast_and_apply  awkward/_util.py:619
│           └─ 0.070 apply  awkward/_util.py:695
│              └─ 0.070 apply  awkward/_util.py:695
│                 └─ 0.070 apply  awkward/_util.py:695
│                    └─ 0.070 getfunction  awkward/_connect/_numpy.py:150
└─ 0.068 Array.func  numpy/lib/mixins.py:53
   └─ 0.068 Array.__array_ufunc__  awkward/highlevel.py:1353
      └─ 0.068 array_ufunc  awkward/_connect/_numpy.py:54
         └─ 0.068 broadcast_and_apply  awkward/_util.py:619
            └─ 0.068 apply  awkward/_util.py:695
               └─ 0.068 apply  awkward/_util.py:695
                  └─ 0.068 apply  awkward/_util.py:695
                     └─ 0.068 getfunction  awkward/_connect/_numpy.py:150
```
On an m1 max macbook pro with 32GB of memory.


Discussion:
Indeed I can show very clear differences in memory use using memray (`pip install memray`):
Here's ak2:
<img width="632" alt="image" src="https://user-images.githubusercontent.com/1068089/235263921-7009b798-d2d2-4a6a-92ca-acede18c56c8.png">

Here's ak1:
<img width="668" alt="image" src="https://user-images.githubusercontent.com/1068089/235263948-60e0d70e-e65f-43f2-b2bc-dcbd78a86029.png">

ak1 uses approximately 3x less memory than ak2 for this set of operations with the same data!

The particular point of nastiness for memory and time appears to be: 
`trijet["p4"] = trijet.j1 + trijet.j2 + trijet.j3`
followed by
```
trijet = ak.flatten(
    trijet[ak.singletons(ak.argmin(abs(trijet.p4.mass - 172.5), axis=1))]
)
```

`ak.combinations` is also slower/more memory hungry but is in the noise compared to the above.
I will also add that the above test data is pretty similar in multiplicity to real data from the CMS detector - so regular users will hit this performance difference.
It seems likely to me that the performance regression is in the sum of 4-vectors,

```python
p4 = trijet.j1 + trijet.j2   # just a single + to narrow it down
```

not the broadcast-and-assign,

```python
trijet["p4"] = p4
```

Is that correct?
That is correct in both ak1 and ak2.
Profiling the script _up to_ the addition using memray, I see 
- Total usage of 9.7 GB usage on v1, and 13.9 GB on v2.
- The `IndexedArray.project` call uses the most memory; 6.9 GB on v1, and 10.4 GB on v2.
- The ufunc `apply` call uses 2.8 GB on v1, 3.5 GB on v2.

These aren't _significantly_ worse, but let's dig further.

I logged when a carry was performed; the memory usage here is dominated by buffer carry operations. We perform the same number of `NumpyArray` carrys in both v1 and v2, over the same shape buffers. The obvious thing to point the finger at here is the garbage collector; in v1, these buffers are freed the moment they lose enough references, whereas in v2 we may have to wait for the Python GC to do a sweep (cyclic references). I added `gc.collect(2)` into the `NumpyArray._carry` method, and saw the total memory usage drop to 8.3 GB, though the actual carry is 7.6 GB.

This implies that we have some reference cycles that are making it harder to free these buffers, because only cycles are freed by the GC; simple 0-refcounts are immediately freed.

However, this is probably not the full picture; I suspect the most likely cause could be the existence of concurrent partial results in a single stack frame. If function `X` has `N` intermediate steps, the peak memory usage of `X` is `N * M`, even if the final memory usage is `M`. Perhaps we have more of these cases in v2 than v1. I will keep looking at this over the next few days to get a bigger picture.

Performance wise, up to (and including) the vector addition, v1 and v2 produce very similar times (9.31 s for v1, 10.83 s for v2 (wall)). Again, not a thorough test.
Odd that it seems OS dependent? @nsmith- was seeing things being rather more memory hungry in linux in the [adl-benchmarks](https://github.com/CoffeaTeam/coffea-benchmarks/blob/coffea2023/coffea-adl-benchmarks.ipynb) so far as well. 

I'd argue as well that in absolute term 3GB more memory usage is pretty significant given our 2GB/core standard slot size in CMS (yes here we're talking about a 6-7GB anyway but...). Implies significantly reduced partition sizes in regular slots to be successful -> more data inefficiencies.
I'm starting to suspect this might be platform dependent:
![image](https://user-images.githubusercontent.com/1068089/235357531-95fd6d86-5b69-43ed-8a36-04e1cb74e899.png)

(this was on a centos7 image, coffea 0.7 = awkward1, coffea2023 = awkward2) but also smaller chunk sizes compared to where I see issues... so take this with a grain of salt.
I've looked at this further today. Here, I disabled the GC, and added debug information to the destructor to provide an insight into what's being freed, and when. These are the results:
```
Freeing container with 0 referrers to our array@ 139669413579440 with  709 MB
Freeing container with 0 referrers to our array@ 139669413579344 with  709 MB
Freeing container with 0 referrers to our array@ 139669413579248 with  709 MB
Freeing container with 0 referrers to our array@ 139669413581168 with  709 MB
Freeing container with 0 referrers to our array@ 139669413580976 with  709 MB
Freeing container with 0 referrers to our array@ 139669413580880 with  709 MB
Freeing container with 0 referrers to our array@ 139669413581360 with  709 MB
Freeing container with 0 referrers to our array@ 139669413580592 with  709 MB
Freeing container with 0 referrers to our array@ 139669413581552 with  709 MB
Freeing container with 0 referrers to our array@ 139669413580688 with  709 MB
Freeing container with 0 referrers to our array@ 139669413297968 with  709 MB
Freeing container with 0 referrers to our array@ 139669413581936 with  709 MB
Freeing container with 0 referrers to our array@ 139669413582128 with  709 MB
Freeing container with 0 referrers to our array@ 139669413582416 with  709 MB
Freeing container with 0 referrers to our array@ 139669413581456 with  709 MB
Freeing container with 0 referrers to our array@ 139669413582992 with  709 MB
Freeing container with 0 referrers to our array@ 139669413580400 with  709 MB
Freeing container with 0 referrers to our array@ 139669413583184 with  709 MB
Freeing container with 0 referrers to our array@ 139669413298064 with  709 MB
Freeing container with 0 referrers to our array@ 139669413582608 with  709 MB
Freeing container with 0 referrers to our array@ 139669413581744 with  709 MB
Freeing container with 0 referrers to our array@ 139669413582032 with  709 MB
Freeing container with 0 referrers to our array@ 139669413583472 with  709 MB
Freeing container with 0 referrers to our array@ 139669413582896 with  709 MB
Freeing container with 0 referrers to our array@ 139669413583664 with  709 MB
Freeing container with 0 referrers to our array@ 139669413583088 with  709 MB
Freeing container with 0 referrers to our array@ 139669413583856 with  709 MB
Freeing container with 0 referrers to our array@ 139669413582512 with  709 MB
Freeing container with 0 referrers to our array@ 139669413580784 with  709 MB
Freeing container with 0 referrers to our array@ 139669413584240 with  709 MB
Freeing container with 0 referrers to our array@ 139669413579632 with  709 MB
```

To break this down, this output is printed after a manual `gc.collect(2)` at the end of a simple program that evaluates `trijet.j1 + trijet.j2 + trijet.j3`, which means that by the time that `__del__` is called, it sees no referrers. The ID is printed alongside the number of bytes used by the array. 

Clearly this is a scenario in which we're seeing a lot of refcount cycles. Whilst it is perfectly feasible to use the GC cycle detection to correct this, in a real analysis we'd all prefer to avoid these cycles such that we naturally free these objects much earlier.

Digging into this, it looks like our error handling machinery is introducing a ref-cycle via delayed formatting that's keeping these buffers around. Disabling lazy formatting leads to reduced memory consumption under normal GC configuration (12.1 GB vs 17.6 GB in `main`).

It should be fairly trivial to break this cycle.

Resolving patch:
diff --git a/src/awkward/_errors.py b/src/awkward/_errors.py
--- a/src/awkward/_errors.py
+++ b/src/awkward/_errors.py
@@ -55,6 +55,11 @@ def __exit__(self, exception_type, exception_value, traceback):
             if exception_type is not None and self.primary() is self:
                 self.handle_exception(exception_type, exception_value)
         finally:
+            # `_kwargs` may hold cyclic references, that we really want to avoid
+            # as this can lead to large buffers remaining in memory for longer than absolutely necessary
+            # Let's just clear this, now.
+            self._kwargs.clear()
+
             # Step out of the way so that another ErrorContext can become primary.
             if self.primary() is self:
                 self._slate.__dict__.clear()


Test patch: