Repository: modin-project/modin
Pull request: FIX-#3823: Fix TypeError when creating Series from SparseArray
## What do these changes do?

<!-- Please give a short brief about these changes. -->

- [x] first commit message and PR title follow format outlined [here](https://modin.readthedocs.io/en/latest/development/contributing.html#commit-message-formatting)
  > **_NOTE:_**  If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.
- [x] passes `flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py`
- [x] passes `black --check modin/ asv_bench/benchmarks scripts/doc_checker.py`
- [x] signed commit with `git commit -s` <!-- you can amend your commit with a signature via `git commit -amend -s` -->
- [x] Resolves #3823? <!-- issue must be created for each patch -->
- [x] tests added and passing
- [ ] module layout described at `docs/development/architecture.rst` is up-to-date <!-- if you have added, renamed or removed files or directories please update the documentation accordingly -->


Problem statement:
``Series`` doesn`t work correctly with ``sparse`` at MODIN_STORAGE_FORMAT=omnisci
When I run at MODIN_STORAGE_FORMAT=omnisci:
```
import numpy as np
import pandas
import modin.pandas as pd
pd.Series(pandas.arrays.SparseArray(np.array([3, 1, 2, 3, 4, np.nan])))
```
I have error:
```
TypeError: Sparse pandas data (column F___reduced__) not supported.
```

Hints from issue discussion:
@modin-project/modin-omnisci  could someone please check if this still happens?

@devin-petersohn do we still want to support sparse data?..

Commit message: FIX-#3823: Fixed TypeError when creating Series from SparseArray

Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>

Patch to review:
diff --git a/modin/experimental/core/execution/native/implementations/hdk_on_native/partitioning/partition_manager.py b/modin/experimental/core/execution/native/implementations/hdk_on_native/partitioning/partition_manager.py
index b0a73a7dfe6..7e69693ede4 100644
--- a/modin/experimental/core/execution/native/implementations/hdk_on_native/partitioning/partition_manager.py
+++ b/modin/experimental/core/execution/native/implementations/hdk_on_native/partitioning/partition_manager.py
@@ -12,7 +12,7 @@
 # governing permissions and limitations under the License.
 
 """Module provides a partition manager class for ``HdkOnNativeDataframe`` frame."""
-
+from modin.error_message import ErrorMessage
 from modin.pandas.utils import is_scalar
 import numpy as np
 
@@ -24,6 +24,7 @@
 from ..calcite_builder import CalciteBuilder
 from ..calcite_serializer import CalciteSerializer
 from modin.config import DoUseCalcite
+from pandas import SparseDtype
 
 import pyarrow
 import pandas
@@ -184,13 +185,26 @@ def _get_unsupported_cols(cls, obj):
                 pyarrow.lib.ArrowTypeError,
                 pyarrow.lib.ArrowInvalid,
                 ValueError,
+                TypeError,
             ) as err:
+                # The TypeError could be raised when converting a sparse data to
+                # arrow table - https://github.com/apache/arrow/pull/4497. If this
+                # is the case - fall back to pandas, otherwise - rethrow the error.
+                if type(err) == TypeError:
+                    if any([isinstance(t, SparseDtype) for t in obj.dtypes]):
+                        ErrorMessage.single_warning(
+                            "Sparse data is not currently supported!"
+                        )
+                    else:
+                        raise err
+
                 # The ValueError is raised by pyarrow in case of duplicate columns.
                 # We catch and handle this error here. If there are no duplicates
                 # (is_unique is True), then the error is caused by something different
                 # and we just rethrow it.
                 if (type(err) == ValueError) and obj.columns.is_unique:
                     raise err
+
                 regex = r"Conversion failed for column ([^\W]*)"
                 unsupported_cols = []
                 for msg in err.args:
diff --git a/modin/experimental/core/execution/native/implementations/hdk_on_native/test/test_dataframe.py b/modin/experimental/core/execution/native/implementations/hdk_on_native/test/test_dataframe.py
index 21999c86d28..b6c68ec10e7 100644
--- a/modin/experimental/core/execution/native/implementations/hdk_on_native/test/test_dataframe.py
+++ b/modin/experimental/core/execution/native/implementations/hdk_on_native/test/test_dataframe.py
@@ -2399,5 +2399,13 @@ def test_dict(self):
         df_equals(mdf, pdf)
 
 
+class TestSparseArray:
+    def test_sparse_series(self):
+        data = pandas.arrays.SparseArray(np.array([3, 1, 2, 3, 4, np.nan]))
+        mds = pd.Series(data)
+        pds = pandas.Series(data)
+        df_equals(mds, pds)
+
+
 if __name__ == "__main__":
     pytest.main(["-v", __file__])


Code review comments:
diff_hunk: @@ -12,7 +12,7 @@
 # governing permissions and limitations under the License.
 
 """Module provides a partition manager class for ``HdkOnNativeDataframe`` frame."""
-
+from modin.error_message import ErrorMessage
path: modin/experimental/core/execution/native/implementations/hdk_on_native/partitioning/partition_manager.py
text: @user1:
```suggestion

from modin.error_message import ErrorMessage
```

diff_hunk: @@ -24,6 +24,7 @@
 from ..calcite_builder import CalciteBuilder
 from ..calcite_serializer import CalciteSerializer
 from modin.config import DoUseCalcite
+from pandas import SparseDtype
path: modin/experimental/core/execution/native/implementations/hdk_on_native/partitioning/partition_manager.py
text: @user1:
```suggestion
```
You can use pandas.SparseDtype to fix CodeQL scanning.