Repository: scikit-hep/awkward
Pull request #3425: fix: `ak.almost_equal` and `ak.array_equal` should check that jagged arrays have the same offsets
Author: login: ikrommyd
Created: 2025-03-18T18:58:55Z | Merged: 2025-03-19T09:25:55Z

Fixes https://github.com/scikit-hep/awkward/issues/3426
Addresses https://github.com/scikit-hep/awkward/discussions/3418

Jagged arrays can have the same content but different offsets. `ak.almost_equal` and `ak.array_equal` do not check for that and only check whether the contents are equal. Example:
```python
In [15]: a = ak.Array([[1, 2], [3]])

In [16]: b = ak.Array([[1], [2, 3]])

In [17]: ak.almost_equal(a, b)
Out[17]: True

In [18]: ak.array_equal(a,b)
Out[18]: True
```
This isn't proper behavior in my opinion.


Diff:
diff --git a/src/awkward/operations/ak_almost_equal.py b/src/awkward/operations/ak_almost_equal.py
index 949f955a45..f462ccfcf4 100644
--- a/src/awkward/operations/ak_almost_equal.py
+++ b/src/awkward/operations/ak_almost_equal.py
@@ -161,6 +161,11 @@ def visitor(left, right) -> bool:
             return (left.size == right.size) and visitor(left.content, right.content)
         # List-list
         elif left.is_list and right.is_list:
+            # Check that indexes are equal
+            left_index = left.to_ListOffsetArray64(True).offsets
+            right_index = right.to_ListOffsetArray64(True).offsets
+            if not backend.index_nplike.array_equal(left_index, right_index):
+                return False
             # Mixed regular-var
             if left.is_regular and not right.is_regular:
                 return (not check_regular) and visitor(
diff --git a/tests/test_1105_ak_aray_equal.py b/tests/test_1105_ak_aray_equal.py
index 7512f50316..922863ae25 100644
--- a/tests/test_1105_ak_aray_equal.py
+++ b/tests/test_1105_ak_aray_equal.py
@@ -38,6 +38,30 @@ def test_array_equal_on_listoffsets():
     )
     assert ak.array_equal(a1, a2)
 
+    # double jagged array
+    a = ak.Array([[[1], [2, 3]], [[4, 5], [6]]])
+    assert ak.array_equal(a, a)
+
+    # different index same content
+    a = ak.Array([[1], [2, 3]])
+    b = ak.Array([[1, 2], [3]])
+    assert not ak.array_equal(a, b)
+
+    # different outer index, same inner index
+    a = ak.Array([[[], [1]], [[], [0]]])
+    b = ak.Array([[[], [1], []], [[0]]])
+    assert not ak.array_equal(a, b)
+
+    # same outer index, different inner index
+    a = ak.Array([[[0, 1], [1]], [[0], []]])
+    b = ak.Array([[[0], [1, 1]], [[], [0]]])
+    assert not ak.array_equal(a, b)
+
+    # nested
+    a = ak.Array([[[[]], [[0, 1], [1]]], [[[0]], [[]]]])
+    b = ak.Array([[[[0]], [[1], [1]]], [[[]], [[0]]]])
+    assert not ak.array_equal(a, b)
+
 
 def test_array_equal_mixed_content_type():
     a1 = ak.Array([[1, 2, 3], [4, 5, 6], [7, 8]])
diff --git a/tests/test_2198_almost_equal.py b/tests/test_2198_almost_equal.py
index 64e6544e90..d791e93c22 100644
--- a/tests/test_2198_almost_equal.py
+++ b/tests/test_2198_almost_equal.py
@@ -133,6 +133,38 @@ class CustomList(ak.Array): ...
     assert not ak.almost_equal(other_array, another_array)
 
 
+def test_ragged():
+    # single jagged array
+    a = ak.Array([[1], [2, 3]])
+    b = ak.Array([[1], [2, 3]])
+    assert ak.almost_equal(a, b)
+
+    # double jagged array
+    a = ak.Array([[[1], [2, 3]], [[4, 5], [6]]])
+    b = ak.Array([[[1], [2, 3]], [[4, 5], [6]]])
+    assert ak.almost_equal(a, b)
+
+    # different index same content
+    a = ak.Array([[1], [2, 3]])
+    b = ak.Array([[1, 2], [3]])
+    assert not ak.almost_equal(a, b)
+
+    # different outer index, same inner index
+    a = ak.Array([[[], [1]], [[], [0]]])
+    b = ak.Array([[[], [1], []], [[0]]])
+    assert not ak.almost_equal(a, b)
+
+    # same outer index, different inner index
+    a = ak.Array([[[0, 1], [1]], [[0], []]])
+    b = ak.Array([[[0], [1, 1]], [[], [0]]])
+    assert not ak.almost_equal(a, b)
+
+    # nested
+    a = ak.Array([[[[]], [[0, 1], [1]]], [[[0]], [[]]]])
+    b = ak.Array([[[[0]], [[1], [1]]], [[[]], [[0]]]])
+    assert not ak.almost_equal(a, b)
+
+
 def test_empty_outer_ragged():
     array = ak.Array([[1]])[0:0]
     assert not ak.almost_equal(array, [])


Reviews:
user: login: ianna
body: @ikrommyd - Thanks for looking into it! Could you, please, add more tests for the `ListOffsetArrays` of `ListOffsetArrays` of `ListOffsetArrays` etc. Where offsets are the same at the top level, but different in the nested ones? Thanks. 
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T22:49:02Z
user: login: ikrommyd
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-18T23:07:02Z
user: login: ianna
body: @ikrommyd - great, thanks! The tests pass, I'll merge.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-19T09:02:05Z
user: login: ikrommyd
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-19T09:08:27Z
user: login: ianna
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-19T10:03:02Z
user: login: ikrommyd
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-19T10:08:37Z

Review comments:
diff_hunk: @@ -161,6 +161,11 @@ def visitor(left, right) -> bool:
             return (left.size == right.size) and visitor(left.content, right.content)
         # List-list
         elif left.is_list and right.is_list:
+            # Check that indexes are equal
+            left_index = left.to_ListOffsetArray64(True).offsets
+            right_index = right.to_ListOffsetArray64(True).offsets
+            if not backend.index_nplike.array_equal(left_index, right_index):
+                return False
path: src/awkward/operations/ak_almost_equal.py
user: login: ianna
body: @ikrommyd - what about the nested `ListOffsetArrays` offsets?
created_at: 2025-03-18T22:46:00Z
author_association: MEMBER
diff_hunk: @@ -161,6 +161,11 @@ def visitor(left, right) -> bool:
             return (left.size == right.size) and visitor(left.content, right.content)
         # List-list
         elif left.is_list and right.is_list:
+            # Check that indexes are equal
+            left_index = left.to_ListOffsetArray64(True).offsets
+            right_index = right.to_ListOffsetArray64(True).offsets
+            if not backend.index_nplike.array_equal(left_index, right_index):
+                return False
path: src/awkward/operations/ak_almost_equal.py
user: login: ikrommyd
body: The function is recursive. It will be called on the contents again. If the contents are list-like, they have their own offsets and this will be checked.
created_at: 2025-03-18T23:07:02Z
author_association: COLLABORATOR
diff_hunk: @@ -161,6 +161,11 @@ def visitor(left, right) -> bool:
             return (left.size == right.size) and visitor(left.content, right.content)
         # List-list
         elif left.is_list and right.is_list:
+            # Check that indexes are equal
+            left_index = left.to_ListOffsetArray64(True).offsets
+            right_index = right.to_ListOffsetArray64(True).offsets
+            if not backend.index_nplike.array_equal(left_index, right_index):
+                return False
path: src/awkward/operations/ak_almost_equal.py
user: login: ianna
body: Thanks! I've added a couple of more tests.
created_at: 2025-03-19T09:01:13Z
author_association: MEMBER
diff_hunk: @@ -161,6 +161,11 @@ def visitor(left, right) -> bool:
             return (left.size == right.size) and visitor(left.content, right.content)
         # List-list
         elif left.is_list and right.is_list:
+            # Check that indexes are equal
+            left_index = left.to_ListOffsetArray64(True).offsets
+            right_index = right.to_ListOffsetArray64(True).offsets
+            if not backend.index_nplike.array_equal(left_index, right_index):
+                return False
path: src/awkward/operations/ak_almost_equal.py
user: login: ikrommyd
body: Oh I'm sorry. I was gonna do them once I woke up. I just added them to the other test as well (mainly for obsessive reasons 🤣). Merge whenever ready!
created_at: 2025-03-19T09:08:27Z
author_association: COLLABORATOR
diff_hunk: @@ -161,6 +161,11 @@ def visitor(left, right) -> bool:
             return (left.size == right.size) and visitor(left.content, right.content)
         # List-list
         elif left.is_list and right.is_list:
+            # Check that indexes are equal
+            left_index = left.to_ListOffsetArray64(True).offsets
+            right_index = right.to_ListOffsetArray64(True).offsets
+            if not backend.index_nplike.array_equal(left_index, right_index):
+                return False
path: src/awkward/operations/ak_almost_equal.py
user: login: ianna
body: It's me, sorry. I assumed you are done because the PR was marked as "ready for review". Please, use "draft" mode to avoid misunderstanding in the future. Thanks!
created_at: 2025-03-19T10:03:01Z
author_association: MEMBER
diff_hunk: @@ -161,6 +161,11 @@ def visitor(left, right) -> bool:
             return (left.size == right.size) and visitor(left.content, right.content)
         # List-list
         elif left.is_list and right.is_list:
+            # Check that indexes are equal
+            left_index = left.to_ListOffsetArray64(True).offsets
+            right_index = right.to_ListOffsetArray64(True).offsets
+            if not backend.index_nplike.array_equal(left_index, right_index):
+                return False
path: src/awkward/operations/ak_almost_equal.py
user: login: ikrommyd
body: It's alright. I just didn't expect you to do it when you said "please add more tests" 😃. Noted for the future.
created_at: 2025-03-19T10:08:37Z
author_association: COLLABORATOR

PR comments:
user: login: agoose77
created_at: 2025-03-18T19:45:24Z
body: I think this is a bug, yes — we should likely convert to a list with `start=0`
author_association: COLLABORATOR
user: login: ikrommyd
created_at: 2025-03-19T09:37:36Z
body: @all-contributors please add @ikrommyd for code
author_association: COLLABORATOR
user: login: allcontributors[bot]
created_at: 2025-03-19T09:37:47Z
body: @ikrommyd 

I've put up [a pull request](https://github.com/scikit-hep/awkward/pull/3427) to add @ikrommyd! :tada:
author_association: CONTRIBUTOR