internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: modin-project/modin

Issue:
REFACTOR: remove unused '_read_parquet_columns' function
https://github.com/modin-project/modin/blob/72ca672f3e79cb4cdd589692db2d0e19d696da5c/modin/experimental/core/execution/ray/implementations/pandas_on_ray/io/io.py#L47


Discussion:

Resolving patch:
diff --git a/modin/experimental/core/execution/ray/implementations/pandas_on_ray/io/io.py b/modin/experimental/core/execution/ray/implementations/pandas_on_ray/io/io.py
--- a/modin/experimental/core/execution/ray/implementations/pandas_on_ray/io/io.py
+++ b/modin/experimental/core/execution/ray/implementations/pandas_on_ray/io/io.py
@@ -42,45 +42,6 @@
 import ray
 
 
-# Ray functions are not detected by codecov (thus pragma: no cover)
-@ray.remote
-def _read_parquet_columns(path, columns, num_splits, kwargs):  # pragma: no cover
-    """
-    Read columns from Parquet file into a ``pandas.DataFrame`` using Ray task.
-
-    Parameters
-    ----------
-    path : str or List[str]
-        The path of the Parquet file.
-    columns : List[str]
-        The list of column names to read.
-    num_splits : int
-        The number of partitions to split the column into.
-    kwargs : dict
-        Keyward arguments to pass into ``pyarrow.parquet.read`` function.
-
-    Returns
-    -------
-    list
-        A list containing the splitted ``pandas.DataFrame``-s and the Index as the last
-        element.
-
-    Notes
-    -----
-    ``pyarrow.parquet.read`` is used internally as the parse function.
-    """
-    import pyarrow.parquet as pq
-
-    df = (
-        pq.ParquetDataset(path, **kwargs)
-        .read(columns=columns, use_pandas_metadata=True)
-        .to_pandas()
-    )
-    df = df[columns]
-    # Append the length of the index here to build it externally
-    return _split_result_for_readers(0, num_splits, df) + [len(df.index)]
-
-
 class ExperimentalPandasOnRayIO(PandasOnRayIO):
     """
     Class for handling experimental IO functionality with pandas storage format and Ray engine.
@@ -102,7 +63,6 @@ class ExperimentalPandasOnRayIO(PandasOnRayIO):
         (RayTask, PandasPickleExperimentalParser, PickleExperimentalDispatcher),
         build_args,
     )._read
-    read_parquet_remote_task = _read_parquet_columns
 
     @classmethod
     def read_sql(


Test patch:
Repository: modin-project/modin

Issue:
Update copyright year



Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -48,7 +48,7 @@ def noop_decorator(*args, **kwargs):
 export_config_help(configs_file_path)
 
 project = "Modin"
-copyright = "2018-2021, Modin"
+copyright = "2018-2022, Modin"
 author = "Modin contributors"
 
 # The short X.Y version


Test patch:
Repository: modin-project/modin

Issue:
test docs CI job starts failed after update to pandas 1.3.5 (now isn't working)
https://github.com/modin-project/modin/runs/4524559469?check_suite_focus=true


Discussion:
cc @YarShev 

Resolving patch:
diff --git a/modin/core/io/io.py b/modin/core/io/io.py
--- a/modin/core/io/io.py
+++ b/modin/core/io/io.py
@@ -271,7 +271,7 @@ def _read(cls, **kwargs):
         return pd_obj
 
     @classmethod
-    @_inherit_docstrings(pandas.read_json, apilink="pandas.io.json.read_json")
+    @_inherit_docstrings(pandas.read_json, apilink="pandas.read_json")
     @doc(
         _doc_default_io_method,
         summary="Convert a JSON string to query compiler",


Test patch:
Repository: modin-project/modin

Issue:
DOCS: Synchronize the readthedocs copyright footer and NOTICE copyright
In our [NOTICE file](https://github.com/modin-project/modin/blob/master/NOTICE) we have the copyright listed as Modin Developers, but in the docs footer, it is just Modin, see:

<img width="270" alt="Screen Shot 2022-01-13 at 1 24 58 PM" src="https://user-images.githubusercontent.com/15958254/149387752-bd56dbcd-c93a-4961-8c4f-1c3602216f99.png">

https://modin.readthedocs.io/en/latest/developer/architecture.html#:~:text=%C2%A9%20Copyright%202018%2D2022%2C%20Modin.

See also: https://github.com/modin-project/modin/pull/3937/files#r784185877




Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -48,7 +48,7 @@ def noop_decorator(*args, **kwargs):
 export_config_help(configs_file_path)
 
 project = "Modin"
-copyright = "2018-2022, Modin"
+copyright = "2018-2022, Modin Developers."
 author = "Modin contributors"
 
 # The short X.Y version


Test patch:
Repository: modin-project/modin

Issue:
Update pyarrow in environment files
When creating an env from enviroment.yml pyarrow 3.0.0 is installed. At the same time, when installing modin-ray and modin-omnisci in the same env pyarrow 5.0.0 is installed. So we should update pyarrow in requirement-dev.txt and enviroment.yml files to check correctness tests pass with a newer pyarrow version.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -5,7 +5,7 @@
     long_description = fh.read()
 
 dask_deps = ["dask>=2.22.0", "distributed>=2.22.0"]
-ray_deps = ["ray[default]>=1.4.0", "pyarrow>=1.0"]
+ray_deps = ["ray[default]>=1.4.0", "pyarrow>=4.0.1"]
 remote_deps = ["rpyc==4.1.5", "cloudpickle", "boto3"]
 spreadsheet_deps = ["modin-spreadsheet>=0.1.0"]
 sql_deps = ["dfsql>=0.4.2", "pyparsing<=2.4.7"]


Test patch:
Repository: modin-project/modin

Issue:
scripts/ is getting installed in site-packages
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 20.04
- **Modin version** (`modin.__version__`): 0.12.0
- **Python version**:  3.8
- **Code we can use to reproduce**:

I believe the scripts/ folder is getting installed in site-packages with modin. I used ```pip install modin[dask]```. We ran into an import issue since we have a scripts module (conflict between using ours vs the site-packages installed.) but when we install modin we found that your scripts module is getting installed in the site-packages. I am not sure if this is a mistake or intentional.



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -16,7 +16,7 @@
     version=versioneer.get_version(),
     cmdclass=versioneer.get_cmdclass(),
     description="Modin: Make your pandas code run faster by changing one line of code.",
-    packages=find_packages(),
+    packages=find_packages(exclude=["scripts", "scripts.*"]),
     include_package_data=True,
     license="Apache 2",
     url="https://github.com/modin-project/modin",


Test patch:
Repository: modin-project/modin

Issue:
Propagate index in Modin frame raises error when setting column
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:
- **Modin version** (`modin.__version__`):
- **Python version**:
- **Code we can use to reproduce**:

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->
This line here
https://github.com/modin-project/modin/blob/406af7c8d6c973d680ab9e07d6edc13eb69481f0/modin/core/dataframe/pandas/dataframe/dataframe.py#L534
Needs to be moved to the end of the function and be unindented to be outside of the if statement right? Cause otherwise we will error out whenever we try and propagate columns.

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->


Discussion:

Resolving patch:
diff --git a/modin/core/dataframe/pandas/dataframe/dataframe.py b/modin/core/dataframe/pandas/dataframe/dataframe.py
--- a/modin/core/dataframe/pandas/dataframe/dataframe.py
+++ b/modin/core/dataframe/pandas/dataframe/dataframe.py
@@ -531,10 +531,11 @@ def apply_idx_objs(df, cols):
                     for i in range(len(self._partitions))
                 ]
             )
+            self._deferred_column = False
+        else:
             ErrorMessage.catch_bugs_and_request_email(
                 axis is not None and axis not in [0, 1]
             )
-            self._deferred_column = False
 
     @lazy_metadata_decorator(apply_axis=None)
     def mask(


Test patch:
Repository: modin-project/modin

Issue:
`ray.cloudpickle` can't serialize empty Modin DataFrame
### System information
Modin Version: 0.12.0
Ray 1.7.1
Python 3.9.7


### Describe the problem

```
df1 = modin_pd.DataFrame([])
ray.cloudpickle.dumps(df1)

---------------------------------------------------------------------------
PicklingError                             Traceback (most recent call last)
/var/folders/9l/q_y0w5yx2_1_h65cprqgczyc0000gq/T/ipykernel_64814/415989427.py in <module>
      1 df1 = pd.DataFrame([])
----> 2 ray.cloudpickle.dumps(df1)

/python3.9/site-packages/ray/cloudpickle/cloudpickle_fast.py in dumps(obj, protocol, buffer_callback)
     71                 file, protocol=protocol, buffer_callback=buffer_callback
     72             )
---> 73             cp.dump(obj)
     74             return file.getvalue()
     75 

/python3.9/site-packages/ray/cloudpickle/cloudpickle_fast.py in dump(self, obj)
    578     def dump(self, obj):
    579         try:
--> 580             return Pickler.dump(self, obj)
    581         except RuntimeError as e:
    582             if "recursion" in e.args[0]:

PicklingError: args[0] from __newobj__ args has the wrong class
```


Discussion:
@c3-cjazra Thanks for reporting the issue. I can reproduce it. The root cause is that while an empty pandas dataframe is pickleable

```
import pandas
import pickle
pickle.dumps(pandas.DataFrame())
```

, an empty Modin dataframe is not:

```
import modin.pandas as pandas
import pickle
pickle.dumps(pandas.DataFrame())
```

@mvashishtha, it is worth looking into this https://discuss.modin.org/t/reusing-the-same-modin-dataframe-across-different-processes/240.

Resolving patch:
diff --git a/modin/pandas/base.py b/modin/pandas/base.py
--- a/modin/pandas/base.py
+++ b/modin/pandas/base.py
@@ -91,6 +91,7 @@
     "_inflate_light",
     "_inflate_full",
     "__reduce__",
+    "__reduce_ex__",
 } | _ATTRS_NO_LOOKUP
 
 


Test patch:
diff --git a/modin/pandas/test/dataframe/test_pickle.py b/modin/pandas/test/dataframe/test_pickle.py
--- a/modin/pandas/test/dataframe/test_pickle.py
+++ b/modin/pandas/test/dataframe/test_pickle.py
@@ -39,6 +39,9 @@ def persistent(request):
     PersistentPickle.put(old)
 
 
+@pytest.mark.parametrize(
+    "modin_df", [pytest.param(modin_df), pytest.param(pd.DataFrame(), id="empty_df")]
+)
 def test_dataframe_pickle(modin_df, persistent):
     other = pickle.loads(pickle.dumps(modin_df))
     df_equals(modin_df, other)
Repository: modin-project/modin

Issue:
Omit thrift logs when using OmniSci engine
Currently, any code that uses OmniSci engine in Modin prints OmniSci's internal thrift logs:
```python
>>> pd.DataFrame({"a": [1, 2, 3]}).sum()
UserWarning: Distributing <class 'dict'> object. This may take some time.
Thrift: Fri Feb  4 17:55:48 2022 TSocket::open() error on socket (after THRIFT_POLL) <Host: localhost Port: 6279>: Connection refused
Thrift: Fri Feb  4 17:55:48 2022 TSocket::open() error on socket (after THRIFT_POLL) <Host: localhost Port: 6279>: Connection refused
Thrift: Fri Feb  4 17:55:48 2022 TSocket::open() error on socket (after THRIFT_POLL) <Host: localhost Port: 6279>: Connection refused
Thrift: Fri Feb  4 17:55:48 2022 TSocket::open() error on socket (after THRIFT_POLL) <Host: localhost Port: 6279>: Connection refused
Thrift: Fri Feb  4 17:55:48 2022 TSocket::open() error on socket (after THRIFT_POLL) <Host: localhost Port: 6279>: Connection refused
Thrift: Fri Feb  4 17:55:48 2022 TSocket::open() error on socket (after THRIFT_POLL) <Host: localhost Port: 6279>: Connection refused
Thrift: Fri Feb  4 17:55:48 2022 TSocket::open() error on socket (after THRIFT_POLL) <Host: localhost Port: 6279>: Connection refused
Thrift: Fri Feb  4 17:55:48 2022 TSocket::open() error on socket (after THRIFT_POLL) <Host: localhost Port: 6279>: Connection refused
Thrift: Fri Feb  4 17:55:49 2022 TSocket::open() error on socket (after THRIFT_POLL) <Host: localhost Port: 6279>: Connection refused
UserWarning: <function DataFrame.transpose> defaulting to pandas implementation.
To request implementation, send an email to feature_requests@modin.org.
a    6
dtype: int64
```
We can omit the logs by passing `--enable_thrift_logs=0` argument to the OmniSci server:
```python
>>> cfg.OmnisciLaunchParameters.put({"enable_thrift_logs": 0})
>>> import modin.pandas as pd
>>> pd.DataFrame({"a": [1, 2, 3]}).sum()
UserWarning: Distributing <class 'dict'> object. This may take some time.
UserWarning: <function DataFrame.transpose> defaulting to pandas implementation.
To request implementation, send an email to feature_requests@modin.org.
a    6
dtype: int64
```
We should consider disabling thrift logs by default.


Discussion:

Resolving patch:
diff --git a/modin/config/envvars.py b/modin/config/envvars.py
--- a/modin/config/envvars.py
+++ b/modin/config/envvars.py
@@ -384,6 +384,7 @@ class OmnisciLaunchParameters(EnvironmentVariable, type=dict):
         "enable_lazy_fetch": 0,
         "null_div_by_zero": 1,
         "enable_watchdog": 0,
+        "enable_thrift_logs": 0,
     }
 
     @classmethod


Test patch:
Repository: modin-project/modin

Issue:
REFACTOR: add assert check into `__init__` method of `PandasOnDaskDataframePartition` class
This should help avoid undefined behavior.


Discussion:

Resolving patch:
diff --git a/modin/core/execution/dask/implementations/pandas_on_dask/partitioning/partition.py b/modin/core/execution/dask/implementations/pandas_on_dask/partitioning/partition.py
--- a/modin/core/execution/dask/implementations/pandas_on_dask/partitioning/partition.py
+++ b/modin/core/execution/dask/implementations/pandas_on_dask/partitioning/partition.py
@@ -44,6 +44,7 @@ class PandasOnDaskDataframePartition(PandasDataframePartition):
     """
 
     def __init__(self, future, length=None, width=None, ip=None, call_queue=None):
+        assert isinstance(future, Future)
         self.future = future
         if call_queue is None:
             call_queue = []


Test patch:
Repository: modin-project/modin

Issue:
Pin `Dask<2022.2.0` as a temporary fix of CI
Investigation of the problem should be continued in #4202.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -4,7 +4,7 @@
 with open("README.md", "r", encoding="utf-8") as fh:
     long_description = fh.read()
 
-dask_deps = ["dask>=2.22.0", "distributed>=2.22.0"]
+dask_deps = ["dask>=2.22.0,<2022.2.0", "distributed>=2.22.0,<2022.2.0"]
 ray_deps = ["ray[default]>=1.4.0", "pyarrow>=4.0.1"]
 remote_deps = ["rpyc==4.1.5", "cloudpickle", "boto3"]
 spreadsheet_deps = ["modin-spreadsheet>=0.1.0"]


Test patch:
Repository: modin-project/modin

Issue:
REFACTOR: remove unused `dask_client` global variable in `modin\pandas\__init__`



Discussion:

Resolving patch:
diff --git a/modin/pandas/__init__.py b/modin/pandas/__init__.py
--- a/modin/pandas/__init__.py
+++ b/modin/pandas/__init__.py
@@ -96,14 +96,12 @@
 os.environ["OMP_NUM_THREADS"] = "1"
 
 _is_first_update = {}
-dask_client = None
 _NOINIT_ENGINES = {
     "Python",
 }  # engines that don't require initialization, useful for unit tests
 
 
 def _update_engine(publisher: Parameter):
-    global dask_client
     from modin.config import StorageFormat, CpuCount
     from modin.config.envvars import IsExperimental
     from modin.config.pubsub import ValueSource


Test patch:
diff --git a/modin/pandas/test/test_api.py b/modin/pandas/test/test_api.py
--- a/modin/pandas/test/test_api.py
+++ b/modin/pandas/test/test_api.py
@@ -67,7 +67,6 @@ def test_top_level_api_equality():
         "os",
         "multiprocessing",
         "Client",
-        "dask_client",
         "get_client",
     ]
Repository: modin-project/modin

Issue:
ASV: some of bench failed with `AttributeError: 'PandasOnRayDataframeColumnPartition' object has no attribute 'oid'`
This problem arose after #3886.

The best solution here is not to rely on the internals of the partition.
`wait` function implemented at the partition level can help with this.


Discussion:

Resolving patch:
diff --git a/asv_bench/benchmarks/utils/common.py b/asv_bench/benchmarks/utils/common.py
--- a/asv_bench/benchmarks/utils/common.py
+++ b/asv_bench/benchmarks/utils/common.py
@@ -498,21 +498,26 @@ def execute(
         if ASV_USE_STORAGE_FORMAT == "omnisci":
             df._query_compiler._modin_frame._execute()
             return
-        partitions = df._query_compiler._modin_frame._partitions
+        partitions = df._query_compiler._modin_frame._partitions.flatten()
+        if hasattr(partitions[0], "wait"):
+            all(map(lambda partition: partition.wait(), partitions))
+            return
+
+        # compatibility with old Modin versions
         all(
             map(
                 lambda partition: partition.drain_call_queue() or True,
-                partitions.flatten(),
+                partitions,
             )
         )
         if ASV_USE_ENGINE == "ray":
             from ray import wait
 
-            all(map(lambda partition: wait([partition.oid]), partitions.flatten()))
+            all(map(lambda partition: wait([partition.oid]), partitions))
         elif ASV_USE_ENGINE == "dask":
             from dask.distributed import wait
 
-            all(map(lambda partition: wait(partition.future), partitions.flatten()))
+            all(map(lambda partition: wait(partition.future), partitions))
         elif ASV_USE_ENGINE == "python":
             pass
 
diff --git a/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py b/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py
--- a/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py
+++ b/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py
@@ -393,6 +393,12 @@ def drain(df):
         self.list_of_partitions_to_combine = drained
         self.call_queue = []
 
+    def wait(self):
+        """Wait completing computations on the object wrapped by the partition."""
+        self.drain_call_queue()
+        futures = self.list_of_blocks
+        ray.wait(futures, num_returns=len(futures))
+
     def add_to_apply_calls(self, func, *args, **kwargs):
         """
         Add a function to the call queue.


Test patch:
Repository: modin-project/modin

Issue:
Upgrade pandas to 1.4.1



Discussion:

Resolving patch:
diff --git a/modin/pandas/__init__.py b/modin/pandas/__init__.py
--- a/modin/pandas/__init__.py
+++ b/modin/pandas/__init__.py
@@ -14,7 +14,7 @@
 import pandas
 import warnings
 
-__pandas_version__ = "1.4.0"
+__pandas_version__ = "1.4.1"
 
 if pandas.__version__ != __pandas_version__:
     warnings.warn(
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -22,7 +22,7 @@
     url="https://github.com/modin-project/modin",
     long_description=long_description,
     long_description_content_type="text/markdown",
-    install_requires=["pandas==1.4.0", "packaging", "numpy>=1.18.5", "fsspec"],
+    install_requires=["pandas==1.4.1", "packaging", "numpy>=1.18.5", "fsspec"],
     extras_require={
         # can be installed by pip install modin[dask]
         "dask": dask_deps,


Test patch:
Repository: modin-project/modin

Issue:
Remove  User-defined function verification is still under development in Modin. The function provided is not verified.
This warning is no longer needed and seems kind of scary. Let's remove it from `apply`, `applymap`, and other places where it occurs.


Discussion:

Resolving patch:
diff --git a/modin/pandas/base.py b/modin/pandas/base.py
--- a/modin/pandas/base.py
+++ b/modin/pandas/base.py
@@ -847,7 +847,6 @@ def error_raiser(msg, exception):
 
         self._validate_function(func, on_invalid=error_raiser)
         axis = self._get_axis_number(axis)
-        ErrorMessage.non_verified_udf()
         if isinstance(func, str):
             # if axis != 1 function can be bounded to the Series, which doesn't
             # support axis parameter
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -350,7 +350,6 @@ def applymap(
         """
         if not callable(func):
             raise ValueError("'{0}' object is not callable".format(type(func)))
-        ErrorMessage.non_verified_udf()
         return DataFrame(query_compiler=self._query_compiler.applymap(func))
 
     def apply(
@@ -1681,7 +1680,6 @@ def query(self, expr, inplace=False, **kwargs):  # noqa: PR01, RT01, D200
         """
         Query the columns of a ``DataFrame`` with a boolean expression.
         """
-        ErrorMessage.non_verified_udf()
         self._update_var_dicts_in_kwargs(expr, kwargs)
         self._validate_eval_query(expr, **kwargs)
         inplace = validate_bool_kwarg(inplace, "inplace")


Test patch:
Repository: modin-project/modin

Issue:
Refresh default to pandas info
**Is your feature request related to a problem? Please describe.**
Update warnings and docs regarding defaulting to pandas


Discussion:

Resolving patch:
diff --git a/modin/error_message.py b/modin/error_message.py
--- a/modin/error_message.py
+++ b/modin/error_message.py
@@ -15,8 +15,8 @@
 
 
 class ErrorMessage(object):
-    # Only print the request implementation one time. This only applies to Warnings.
-    printed_request_implementation = False
+    # Only print full ``default to pandas`` warning one time.
+    printed_default_to_pandas = False
     printed_warnings = set()
 
     @classmethod
@@ -45,13 +45,13 @@ def default_to_pandas(cls, message=""):
         else:
             message = "Defaulting to pandas implementation."
 
-        if not cls.printed_request_implementation:
+        if not cls.printed_default_to_pandas:
             message = (
                 f"{message}\n"
-                + "To request implementation, send an email to "
-                + "feature_requests@modin.org."
+                + "Please refer to "
+                + "https://modin.readthedocs.io/en/stable/supported_apis/defaulting_to_pandas.html for explanation."
             )
-            cls.printed_request_implementation = True
+            cls.printed_default_to_pandas = True
         warnings.warn(message)
 
     @classmethod


Test patch: