Repository: modin-project/modin (branch: merge_without_to_pandas)
Commit date: 2024-01-10T15:49:40Z

Workflow file: .github/workflows/fuzzydata-test.yml (fuzzy)
name: fuzzy
on:
  pull_request:
    paths:
      # NOTE: keep these paths in sync with the paths that trigger the CI Github
      # Actions in .github/workflows/ci.yml
      - .github/workflows/**
      - '!.github/workflows/push-to-master.yml'
      - asv_bench/**
      - modin/**
      - requirements/**
      - scripts/**
      - environment-dev.yml
      - requirements-dev.txt
      - setup.cfg
      - setup.py
      - versioneer.py
concurrency:
  # Cancel other jobs in the same branch. We don't care whether CI passes
  # on old commits.
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}
env:
  MODIN_GITHUB_CI: 'true'
  PIP_INDEX_URL: http://localhost:8629/2024-01-10
  UV_INDEX_URL: http://localhost:8629/2024-01-10

jobs:
  test-fuzzydata:
    runs-on: ubuntu-latest
    services:
      pypi_wayback:
        image: ghcr.io/waleko/pypi-wayback
        ports:
          - 8629:8080
    defaults:
      run:
        shell: bash -l {0}
    strategy:
      matrix:
        python-version: ["3.9"]
        engine: ["ray", "dask"]
    steps:
      - uses: actions/checkout@v3
      - uses: ./.github/actions/mamba-env
        with:
          environment-file: environment-dev.yml
          python-version: ${{matrix.python-version}}
      - name: test-fuzzydata (engine ${{matrix.engine}}, python ${{matrix.python-version}})
        run: python -m pytest modin/experimental/fuzzydata/test/test_fuzzydata.py -Wignore::UserWarning --log-file=/tmp/fuzzydata-test-wf-${{matrix.engine}}/run.log --log-file-level=INFO
        env:
          MODIN_ENGINE: ${{matrix.engine}}
      - uses: actions/upload-artifact@v3
        if: success() || failure()
        with:
           name: fuzzydata-test-workflow-${{matrix.engine}}
           path: /tmp/fuzzydata-test-wf-${{matrix.engine}}/* # Must match output dir in test_fuzzydata.py
           if-no-files-found: error


Failed build logs:
step_name: test-fuzzydata (3.9, ray)/4_test-fuzzydata (engine ray, python 3.9).txt
log: ##[group]Run python -m pytest modin/experimental/fuzzydata/test/test_fuzzydata.py -Wignore::UserWarning --log-file=/tmp/fuzzydata-test-wf-ray/run.log --log-file-level=INFO
[36;1mpython -m pytest modin/experimental/fuzzydata/test/test_fuzzydata.py -Wignore::UserWarning --log-file=/tmp/fuzzydata-test-wf-ray/run.log --log-file-level=INFO[0m
shell: /usr/bin/bash -l {0}
env:
  MODIN_GITHUB_CI: true
  INPUT_RUN_POST: true
  CONDA: /usr/share/miniconda3
  CONDA_PKGS_DIR: /home/runner/conda_pkgs_dir
  MODIN_ENGINE: ray
##[endgroup]
============================= test session starts ==============================
platform linux -- Python 3.9.18, pytest-7.4.4, pluggy-1.3.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/runner/work/modin/modin
configfile: setup.cfg
plugins: benchmark-4.0.0, xdist-3.5.0, anyio-4.2.0, Faker-22.1.0, cov-4.1.0
collected 1 item

modin/experimental/fuzzydata/test/test_fuzzydata.py F                    [100%]

=================================== FAILURES ===================================
________________________ test_fuzzydata_sample_workflow ________________________

    def test_fuzzydata_sample_workflow():
        # Workflow Generation Options
        wf_name = str(uuid.uuid4())[:8]  # Unique name for the generated workflow
        num_versions = 10  # Number of unique CSV files to generate
        cols = 33  # Columns in Base Artifact
        rows = 1000  # Rows in Base Artifact
        bfactor = 1.0  # Branching Factor - 0.1 is linear, 10.0 is star-like
        exclude_ops = ["groupby"]  # In-Memory groupby operations cause issue #4287
        matfreq = 2  # How many operations to chain before materialization
    
        engine = Engine.get().lower()
    
        # Create Output Directory for Workflow Data
        base_out_directory = (
            f"/tmp/fuzzydata-test-wf-{engine}/"  # Must match corresponding github-action
        )
        if os.path.exists(base_out_directory):
            shutil.rmtree(base_out_directory)
        output_directory = f"{base_out_directory}/{wf_name}/"
        os.makedirs(output_directory, exist_ok=True)
    
        # Start Workflow Generation
>       workflow = generate_workflow(
            workflow_class=ModinWorkflow,
            name=wf_name,
            num_versions=num_versions,
            base_shape=(cols, rows),
            out_directory=output_directory,
            bfactor=bfactor,
            exclude_ops=exclude_ops,
            matfreq=matfreq,
            wf_options={"modin_engine": engine},
        )

modin/experimental/fuzzydata/test/test_fuzzydata.py:47: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/share/miniconda3/envs/modin/lib/python3.9/site-packages/fuzzydata/core/generator.py:381: in generate_workflow
    raise e
/usr/share/miniconda3/envs/modin/lib/python3.9/site-packages/fuzzydata/core/generator.py:367: in generate_workflow
    wf.execute_current_operation(next_label)
/usr/share/miniconda3/envs/modin/lib/python3.9/site-packages/fuzzydata/core/workflow.py:197: in execute_current_operation
    raise e
/usr/share/miniconda3/envs/modin/lib/python3.9/site-packages/fuzzydata/core/workflow.py:168: in execute_current_operation
    new_artifact = self.current_operation.execute(new_label)
/usr/share/miniconda3/envs/modin/lib/python3.9/site-packages/fuzzydata/core/operation.py:172: in execute
    result = self.materialize(new_label)
/usr/share/miniconda3/envs/modin/lib/python3.9/site-packages/fuzzydata/clients/pandas.py:114: in materialize
    new_df = eval(self.code)
<string>:1: in <module>
    ???
modin/logging/logger_decorator.py:129: in run_and_log
    return obj(*args, **kwargs)
modin/pandas/dataframe.py:1301: in merge
    query_compiler=self._query_compiler.merge(
modin/logging/logger_decorator.py:129: in run_and_log
    return obj(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler object at 0x7f9fccfaedf0>
right = <modin.core.storage_formats.pandas.query_compiler.PandasQueryCompiler object at 0x7f9fccfae940>
kwargs = {'copy': True, 'how': 'inner', 'indicator': False, 'left_index': False, ...}
how = 'inner', left_index = False, right_index = False, sort = False
map_func = <function PandasQueryCompiler.merge.<locals>.map_func at 0x7f9fd4036dc0>
new_columns = Index(['urhjs__ipv4_public', '19caY__sha1', 'uqdyN__bothify',
       'Fpmv9__credit_card_number', 'cTFks__ipv4_private...     'NN7sX__day_of_month', 'Z2GvM__safe_color_name', 'RZGm2__pyfloat',
       'JW1cO__address'],
      dtype='object')
new_dtypes = ModinDtypes:
	value type: <class 'modin.core.dataframe.pandas.metadata.dtypes.DtypesDescriptor'>;
	value:
	DtypesDescr...graph', '7tHsJ__sentence'];
	schema is known: False;
	has parent df: False;
	columns order: None;
	know all names: True

    def merge(self, right, **kwargs):
        how = kwargs.get("how", "inner")
        on = kwargs.get("on", None)
        left_on = kwargs.get("left_on", None)
        right_on = kwargs.get("right_on", None)
        left_index = kwargs.get("left_index", False)
        right_index = kwargs.get("right_index", False)
        sort = kwargs.get("sort", False)
    
        if how in ["left", "inner"] and left_index is False and right_index is False:
            kwargs["sort"] = False
    
            def should_keep_index(left, right):
                keep_index = False
                if left_on is not None and right_on is not None:
                    keep_index = any(
                        o in left.index.names
                        and o in right_on
                        and o in right.index.names
                        for o in left_on
                    )
                elif on is not None:
                    keep_index = any(
                        o in left.index.names and o in right.index.names for o in on
                    )
                return keep_index
    
            def map_func(
                left, right, *axis_lengths, kwargs=kwargs, **service_kwargs
            ):  # pragma: no cover
                df = pandas.merge(left, right, **kwargs)
    
                if kwargs["how"] == "left":
                    partition_idx = service_kwargs["partition_idx"]
                    if len(axis_lengths):
                        if not should_keep_index(left, right):
                            # Doesn't work for "inner" case, since the partition sizes of the
                            # left dataframe may change
                            start = sum(axis_lengths[:partition_idx])
                            stop = sum(axis_lengths[: partition_idx + 1])
    
                            df.index = pandas.RangeIndex(start, stop)
    
                return df
    
            # Want to ensure that these are python lists
            if left_on is not None and right_on is not None:
                left_on = list(left_on) if is_list_like(left_on) else [left_on]
                right_on = list(right_on) if is_list_like(right_on) else [right_on]
            elif on is not None:
                on = list(on) if is_list_like(on) else [on]
    
            new_columns = None
            new_dtypes = None
            if self._modin_frame.has_materialized_columns:
                if left_on is None and right_on is None:
                    if on is None:
                        on = [c for c in self.columns if c in right.columns]
                    _left_on, _right_on = on, on
                else:
                    if left_on is None or right_on is None:
                        raise MergeError(
                            "Must either pass only 'on' or 'left_on' and 'right_on', not combination of them."
                        )
                    _left_on, _right_on = left_on, right_on
    
                try:
                    new_columns, left_renamer, right_renamer = join_columns(
                        self.columns,
                        right.columns,
                        _left_on,
                        _right_on,
                        kwargs.get("suffixes", ("_x", "_y")),
                    )
                except NotImplementedError:
                    # This happens when one of the keys to join is an index level. Pandas behaviour
                    # is really complicated in this case, so we're not computing resulted columns for now.
                    pass
                else:
                    # renamers may contain columns from 'index', so trying to merge index and column dtypes here
                    right_index_dtypes = (
                        right.index.dtypes
                        if isinstance(right.index, pandas.MultiIndex)
                        else pandas.Series(
                            [right.index.dtype], index=[right.index.name]
                        )
                    )
                    right_dtypes = pandas.concat([right.dtypes, right_index_dtypes])[
                        right_renamer.keys()
                    ].rename(right_renamer)
    
                    left_index_dtypes = (
                        self._modin_frame._index_cache.maybe_get_dtypes()
                    )
                    left_dtypes = (
                        ModinDtypes.concat(
                            [self._modin_frame._dtypes, left_index_dtypes]
                        )
                        .lazy_get(left_renamer.keys())
                        .set_index(list(left_renamer.values()))
                    )
                    new_dtypes = ModinDtypes.concat([left_dtypes, right_dtypes])
    
            new_self = self.__constructor__(
                self._modin_frame.broadcast_apply_full_axis(
                    axis=1,
                    func=map_func,
                    enumerate_partitions=how == "left",
                    other=right._modin_frame,
                    # We're going to explicitly change the shape across the 1-axis,
                    # so we want for partitioning to adapt as well
                    keep_partitioning=False,
                    num_splits=merge_partitioning(
                        self._modin_frame, right._modin_frame, axis=1
                    ),
                    new_columns=new_columns,
                    sync_labels=False,
                    dtypes=new_dtypes,
                    pass_axis_lengths_to_partitions=how == "left",
                )
            )
    
            # Here we want to understand whether we're joining on a column or on an index level.
            # It's cool if indexes are already materialized so we can easily check that, if not
            # it's fine too, we can also decide that by columns, which tend to be already
            # materialized quite often compared to the indexes.
            keep_index = False
            if self._modin_frame.has_materialized_index:
>               keep_index = should_keep_index(self, right_pandas)
E               NameError: name 'right_pandas' is not defined

modin/core/storage_formats/pandas/query_compiler.py:643: NameError
----------------------------- Captured stderr call -----------------------------
2024-01-10 15:52:55,130	INFO worker.py:1664 -- Started a local Ray instance. View the dashboard at [1m[32m127.0.0.1:8265 [39m[22m
------------------------------ Captured log call -------------------------------
INFO     fuzzydata.core.workflow:workflow.py:53 Creating new Workflow cc54e466
INFO     fuzzydata.core.generator:generator.py:72 Generating base df with 1000 rows and 33 columns
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=True; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=True; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=True; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=True; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=True; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=True; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=True; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     fuzzydata.core.generator:generator.py:301 Selected Artifact: Artifact(label=artifact_0), initializing operation chain
INFO     fuzzydata.core.generator:generator.py:339 Chaining Operation: sample
INFO     fuzzydata.core.generator:generator.py:70 Generating right-merge df df with 998 rows and 5 columns
INFO     modin.logger.default:partition_manager.py:89 Using sequential splitting in '.from_pandas()' because of some of the conditions are False: enough_elements=False; all_numeric_types=False; async_mode_on=False
INFO     fuzzydata.core.generator:generator.py:339 Chaining Operation: merge
INFO     fuzzydata.core.generator:generator.py:365 Executing current operation list: {'sources': ['artifact_0', 'artifact_1'], 'new_label': None, 'op_list': [{'op': 'sample', 'args': {'frac': 0.73}}, {'op': 'merge', 'args': {'key_col': 'uqdyN__bothify'}}]}
ERROR    fuzzydata.core.workflow:workflow.py:192 Could not execute Operation: [{'op': 'sample', 'args': {'frac': 0.73}}, {'op': 'merge', 'args': {'key_col': 'uqdyN__bothify'}}]
ERROR    fuzzydata.core.generator:generator.py:376 Error during generation, stopping...
ERROR    fuzzydata.core.generator:generator.py:377 Was trying to execute operation: {'sources': ['artifact_0', 'artifact_1'], 'new_label': None, 'op_list': [{'op': 'sample', 'args': {'frac': 0.73}}, {'op': 'merge', 'args': {'key_col': 'uqdyN__bothify'}}]} on soruce artifact(s): [Artifact(label=artifact_0), Artifact(label=artifact_1)]
ERROR    fuzzydata.core.generator:generator.py:378 Was trying to execute code: self.sources[0].table.sample(frac=0.73).merge(self.sources[1].table, on="uqdyN__bothify")
ERROR    fuzzydata.core.generator:generator.py:379 Writing out all files to /tmp/fuzzydata-test-wf-ray//cc54e466/
=============================== warnings summary ===============================
modin/experimental/fuzzydata/test/test_fuzzydata.py::test_fuzzydata_sample_workflow
  DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html

modin/experimental/fuzzydata/test/test_fuzzydata.py::test_fuzzydata_sample_workflow
modin/experimental/fuzzydata/test/test_fuzzydata.py::test_fuzzydata_sample_workflow
modin/experimental/fuzzydata/test/test_fuzzydata.py::test_fuzzydata_sample_workflow
  DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages

modin/experimental/fuzzydata/test/test_fuzzydata.py::test_fuzzydata_sample_workflow
  DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.cloud')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages

modin/experimental/fuzzydata/test/test_fuzzydata.py::test_fuzzydata_sample_workflow
  DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('sphinxcontrib')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED modin/experimental/fuzzydata/test/test_fuzzydata.py::test_fuzzydata_sample_workflow - NameError: name 'right_pandas' is not defined
======================== 1 failed, 6 warnings in 11.54s ========================
##[error]Process completed with exit code 1.


Changed files:
modin/core/storage_formats/pandas/query_compiler.py

Fix diff:
diff --git a/modin/core/storage_formats/pandas/query_compiler.py b/modin/core/storage_formats/pandas/query_compiler.py
index 32b379994..7ca5bb8f1 100644
--- a/modin/core/storage_formats/pandas/query_compiler.py
+++ b/modin/core/storage_formats/pandas/query_compiler.py
@@ -640,19 +640,19 @@ class PandasQueryCompiler(BaseQueryCompiler):
             # materialized quite often compared to the indexes.
             keep_index = False
             if self._modin_frame.has_materialized_index:
-                keep_index = should_keep_index(self, right_pandas)
+                keep_index = should_keep_index(self, right)
             else:
                 # Have to trigger columns materialization. Hope they're already available at this point.
                 if left_on is not None and right_on is not None:
                     keep_index = any(
-                        o not in right_pandas.columns
+                        o not in right.columns
                         and o in left_on
                         and o not in self.columns
                         for o in right_on
                     )
                 elif on is not None:
                     keep_index = any(
-                        o not in right_pandas.columns and o not in self.columns
+                        o not in right.columns and o not in self.columns
                         for o in on
                     )