Repository: dask/dask

Issue:
dataframe assert_eq sorts the inputs when asserting equality
<!-- Please include a self-contained copy-pastable example that generates the issue if possible.

Please be concise with code posted. See guidelines below on how to provide a good bug report:

- Craft Minimal Bug Reports http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
- Minimal Complete Verifiable Examples https://stackoverflow.com/help/mcve

Bug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.
-->

**What happened**:
`dask.dataframe.utils.assert_eq` sorts the two inputs when asserting equality between two dataframes.

This can be problematic in tests such as this where the tests would pass irrespective of whether the sort succeeded or not. https://github.com/dask/dask/blob/bc43d692e8f5fc55e6d81c91053598bd89f79267/dask/dataframe/tests/test_shuffle.py#L1353 


**What you expected to happen**:
One possible solution could be to add a param to the `assert_eq` method that allows to control whether we ignore ordering or not and sort based on that. The default for now could be set to `ignore_ordering=True` so as to not change existing behavior.

Relevant modification here: https://github.com/dask/dask/blob/4d6a5f08c45be56302f696ca4ef6038a1cd1e734/dask/dataframe/utils.py#L549

**Minimal Complete Verifiable Example**:

```python
from dask import dataframe as dd

import numpy as np
import pandas as pd


df = pd.DataFrame()
df["a"] = [5,4,3,2,1] * 20
df["b"] = np.arange(100,  200)
ddf = dd.from_pandas(df, npartitions=10)
sorted_ddf = ddf.sort_values(["a"])
dd.assert_eq(df, sorted_ddf)
```

**Anything else we need to know?**:
N/A

**Environment**:

- Dask version: 2022.4.1
- Python version: 3.8
- Operating System: Ubuntu 20.04
- Install method (conda, pip, source): conda

<!-- If you are reporting an issue such as scale stability, cluster deadlock.
Please provide a cluster dump state with this issue, by running client.dump_cluster_state()

https://distributed.dask.org/en/stable/api.html?highlight=dump_cluster_state#distributed.Client.dump_cluster_state

-->



Discussion:

Resolving patch:
diff --git a/dask/dataframe/utils.py b/dask/dataframe/utils.py
--- a/dask/dataframe/utils.py
+++ b/dask/dataframe/utils.py
@@ -523,6 +523,7 @@ def assert_eq(
     check_dtype=True,
     check_divisions=True,
     check_index=True,
+    sort_results=True,
     scheduler="sync",
     **kwargs,
 ):
@@ -545,7 +546,7 @@ def assert_eq(
         a = a.to_pandas()
     if hasattr(b, "to_pandas"):
         b = b.to_pandas()
-    if isinstance(a, (pd.DataFrame, pd.Series)):
+    if isinstance(a, (pd.DataFrame, pd.Series)) and sort_results:
         a = _maybe_sort(a, check_index)
         b = _maybe_sort(b, check_index)
     if not check_index:


Test patch:
diff --git a/dask/dataframe/tests/test_shuffle.py b/dask/dataframe/tests/test_shuffle.py
--- a/dask/dataframe/tests/test_shuffle.py
+++ b/dask/dataframe/tests/test_shuffle.py
@@ -1361,7 +1361,7 @@ def test_sort_values(nelem, by, ascending):
     with dask.config.set(scheduler="single-threaded"):
         got = ddf.sort_values(by=by, ascending=ascending)
     expect = df.sort_values(by=by, ascending=ascending)
-    dd.assert_eq(got, expect, check_index=False)
+    dd.assert_eq(got, expect, check_index=False, sort_results=False)
 
 
 @pytest.mark.parametrize("ascending", [True, False, [False, True], [True, False]])
diff --git a/dask/dataframe/tests/test_utils_dataframe.py b/dask/dataframe/tests/test_utils_dataframe.py
--- a/dask/dataframe/tests/test_utils_dataframe.py
+++ b/dask/dataframe/tests/test_utils_dataframe.py
@@ -542,13 +542,29 @@ def test_nonempty_series_nullable_float():
 
 
 def test_assert_eq_sorts():
-    df1 = pd.DataFrame({"A": np.linspace(0, 1, 10), "B": np.random.random(10)})
-    df2 = df1.sort_values("B")
-    df2_r = df2.reset_index(drop=True)
-    assert_eq(df1, df2)
-    assert_eq(df1, df2_r, check_index=False)
+    df = pd.DataFrame({"A": np.linspace(0, 1, 10), "B": np.random.random(10)})
+    df_s = df.sort_values("B")
+    assert_eq(df, df_s)
     with pytest.raises(AssertionError):
-        assert_eq(df1, df2_r)
+        assert_eq(df, df_s, sort_results=False)
+
+    df_sr = df_s.reset_index(drop=True)
+    assert_eq(df, df_sr, check_index=False)
+    with pytest.raises(AssertionError):
+        assert_eq(df, df_sr)
+    with pytest.raises(AssertionError):
+        assert_eq(df, df_sr, check_index=False, sort_results=False)
+
+    ddf = dd.from_pandas(df, npartitions=2)
+    ddf_s = ddf.sort_values(["B"])
+    assert_eq(df, ddf_s)
+    with pytest.raises(AssertionError):
+        assert_eq(df, ddf_s, sort_results=False)
+
+    ddf_sr = ddf_s.reset_index(drop=True)
+    assert_eq(df, ddf_sr, check_index=False)
+    with pytest.raises(AssertionError):
+        assert_eq(df, ddf_sr, check_index=False, sort_results=False)
 
 
 def test_assert_eq_scheduler():