Repository: modin-project/modin

Issue:
Fuzzy Testing to find Correctness/Performance bugs
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. What kind of performance improvements would you like to see with this new API?

Hi Modin Team, I've designed a system called [fuzzydata](https://github.com/suhailrehman/fuzzydata) which allows for the encoding, random generation, and replay of dataframe workflows using systems such as Pandas, Modin, and SQLite. In developing fuzzydata I uncovered a bug wherein multiple groupby operations performed on an in-memory generated dataframe fail (#4287). 

I think some of the techniques in fuzzydata would be useful to further the development of modin, please consider using/implementing this system for better performance/correctness testing. A preprint of our [paper](https://github.com/suhailrehman/fuzzydata/blob/main/docs/fuzzydata-dbtest22.pdf) is also in the repo and goes into detail about the design and methodology of our system. 

I would caution against using completely randomized dataframe workflow testing in your CI, however, since bugs may be related to a specific workflow or dataframe that was randomly generated - I think its best to randomly generate a workflow and then replay /scale workflows once generated, and that might be a good way to force a specific edge case that needs to be tested. 




Discussion:
Hi @suhailrehman thanks for posting! Your work is really interesting and I think we should try to leverage some fuzziness in our testing to find corner cases we don't yet cover.

>I would caution against using completely randomized dataframe workflow testing in your CI, however, since bugs may be related to a specific workflow or dataframe that was randomly generated

I agree, we would ideally separate the testing out into some recurring sanity testing job rather than directly in our CI. It can be difficult to reproduce such issues without some very clear ways to reproduce. We could start with adding a new GitHub Actions job on push (rather than on each pull request) and start expanding from there. That way we can also monitor the tests separately. Would starting there make sense to you?
> We could start with adding a new GitHub Actions job on push (rather than on each pull request) and start expanding from there. That way we can also monitor the tests separately. Would starting there make sense to you?

Thanks, @devin-petersohn Let me see if I can whip up an action that runs against a small workflow using the version of code that is pushed to the repo. Might be a while, but I'll respond with a PR if I find a good integration and a quick way to display/inspect the results of the test and save the resulting workflow JSON spec to reproduce. 

Resolving patch:
diff --git a/modin/experimental/fuzzydata/__init__.py b/modin/experimental/fuzzydata/__init__.py
new file mode 100644
--- /dev/null
+++ b/modin/experimental/fuzzydata/__init__.py
@@ -0,0 +1,14 @@
+# Licensed to Modin Development Team under one or more contributor license agreements.
+# See the NOTICE file distributed with this work for additional information regarding
+# copyright ownership.  The Modin Development Team licenses this file to you under the
+# Apache License, Version 2.0 (the "License"); you may not use this file except in
+# compliance with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software distributed under
+# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
+# ANY KIND, either express or implied. See the License for the specific language
+# governing permissions and limitations under the License.
+
+"""Module holds experimental fuzzydata specific functionality for Modin."""


Test patch:
diff --git a/modin/experimental/fuzzydata/test/__init__.py b/modin/experimental/fuzzydata/test/__init__.py
new file mode 100644
--- /dev/null
+++ b/modin/experimental/fuzzydata/test/__init__.py
@@ -0,0 +1,12 @@
+# Licensed to Modin Development Team under one or more contributor license agreements.
+# See the NOTICE file distributed with this work for additional information regarding
+# copyright ownership.  The Modin Development Team licenses this file to you under the
+# Apache License, Version 2.0 (the "License"); you may not use this file except in
+# compliance with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software distributed under
+# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
+# ANY KIND, either express or implied. See the License for the specific language
+# governing permissions and limitations under the License.
diff --git a/modin/experimental/fuzzydata/test/test_fuzzydata.py b/modin/experimental/fuzzydata/test/test_fuzzydata.py
new file mode 100644
--- /dev/null
+++ b/modin/experimental/fuzzydata/test/test_fuzzydata.py
@@ -0,0 +1,65 @@
+# Licensed to Modin Development Team under one or more contributor license agreements.
+# See the NOTICE file distributed with this work for additional information regarding
+# copyright ownership.  The Modin Development Team licenses this file to you under the
+# Apache License, Version 2.0 (the "License"); you may not use this file except in
+# compliance with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software distributed under
+# the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
+# ANY KIND, either express or implied. See the License for the specific language
+# governing permissions and limitations under the License.
+
+import os
+import glob
+import uuid
+import shutil
+from fuzzydata.core.generator import generate_workflow
+from fuzzydata.clients.modin import ModinWorkflow
+
+from modin.config import Engine
+
+
+def test_fuzzydata_sample_workflow():
+    # Workflow Generation Options
+    wf_name = str(uuid.uuid4())[:8]  # Unique name for the generated workflow
+    num_versions = 10  # Number of unique CSV files to generate
+    cols = 33  # Columns in Base Artifact
+    rows = 1000  # Rows in Base Artifact
+    bfactor = 1.0  # Branching Factor - 0.1 is linear, 10.0 is star-like
+    exclude_ops = ["groupby"]  # In-Memory groupby operations cause issue #4287
+    matfreq = 2  # How many operations to chain before materialization
+
+    engine = Engine.get().lower()
+
+    # Create Output Directory for Workflow Data
+    base_out_directory = (
+        f"/tmp/fuzzydata-test-wf-{engine}/"  # Must match corresponding github-action
+    )
+    if os.path.exists(base_out_directory):
+        shutil.rmtree(base_out_directory)
+    output_directory = f"{base_out_directory}/{wf_name}/"
+    os.makedirs(output_directory, exist_ok=True)
+
+    # Start Workflow Generation
+    workflow = generate_workflow(
+        workflow_class=ModinWorkflow,
+        name=wf_name,
+        num_versions=num_versions,
+        base_shape=(cols, rows),
+        out_directory=output_directory,
+        bfactor=bfactor,
+        exclude_ops=exclude_ops,
+        matfreq=matfreq,
+        wf_options={"modin_engine": engine},
+    )
+
+    # Assertions that the workflow generation worked correctly
+    assert len(workflow) == num_versions
+    assert len(list(glob.glob(f"{output_directory}/artifacts/*.csv"))) == len(
+        workflow.artifact_dict
+    )
+    assert os.path.exists(f"{output_directory}/{workflow.name}_operations.json")
+    assert os.path.getsize(f"{output_directory}/{workflow.name}_operations.json") > 0
+    assert os.path.exists(f"{output_directory}/{workflow.name}_gt_graph.csv")