Repository: modin-project/modin

Issue:
in OSX, parquet formatted dataframe errors on tail() - "length should not be negative"
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04): OSX
- **Modin version** (`modin.__version__`):0.11.1
- **Python version**: 3.8.3
- **Code we can use to reproduce**:
- 

df = pd.read_parquet('/var/tmp/prod.parquet')
df.head() # works!
df.tail() #  fails! "length should not be negative"


<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

This happens only on OSX

When a parquet dataset is read by modin, the tail operation errors.

exporting to a csv and reread with pd.read_csv, both head() and tail() work.

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:
Hi @farhadabar thanks for the report!

Can you share the full traceback you received? I can't reproduce with any of the files we use for testing yet, but the traceback would give more information.
here you go,


>>> df
      MPG  Cylinders  Displacement  ...  Acceleration  Model_Year  Origin
0    18.0          8         307.0  ...          12.0          70     usa
1    15.0          8         350.0  ...          11.5          70     usa
2    18.0          8         318.0  ...          11.0          70     usa
3    16.0          8         304.0  ...          12.0          70     usa
4    17.0          8         302.0  ...          10.5          70     usa
..    ...        ...           ...  ...           ...         ...     ...
393  27.0          4         140.0  ...          15.6          82     usa
394  44.0          4          97.0  ...          24.6          82  europe
395  32.0          4         135.0  ...          11.6          82     usa
396  28.0          4         120.0  ...          18.6          82     usa
397  31.0          4         119.0  ...          19.4          82     usa

[398 rows x 8 columns]

>>> df.head()
    MPG  Cylinders  Displacement  Horsepower  Weight  Acceleration  Model_Year Origin
0  18.0          8         307.0       130.0    3504          12.0          70    usa
1  15.0          8         350.0       165.0    3693          11.5          70    usa
2  18.0          8         318.0       150.0    3436          11.0          70    usa
3  16.0          8         304.0       150.0    3433          12.0          70    usa
4  17.0          8         302.0       140.0    3449          10.5          70    usa

>>> df.tail()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Applications/Genetica/Cortex10/miniconda3/lib/python3.8/site-packages/modin/pandas/base.py", line 2469, in tail
    return self.iloc[-n:]
  File "/Applications/Genetica/Cortex10/miniconda3/lib/python3.8/site-packages/modin/pandas/indexing.py", line 754, in __getitem__
    result = super(_iLocIndexer, self).__getitem__(row_lookup, col_lookup, ndim)
  File "/Applications/Genetica/Cortex10/miniconda3/lib/python3.8/site-packages/modin/pandas/indexing.py", line 300, in __getitem__
    qc_view = self.qc.view(row_lookup, col_lookup)
  File "/Applications/Genetica/Cortex10/miniconda3/lib/python3.8/site-packages/modin/backends/pandas/query_compiler.py", line 3056, in view
    self._modin_frame.mask(row_numeric_idx=index, col_numeric_idx=columns)
  File "/Applications/Genetica/Cortex10/miniconda3/lib/python3.8/site-packages/modin/engines/base/frame/data.py", line 474, in mask
    new_row_lengths = [
  File "/Applications/Genetica/Cortex10/miniconda3/lib/python3.8/site-packages/modin/engines/base/frame/data.py", line 475, in <listcomp>
    len(range(*idx.indices(self._row_lengths[p])))
ValueError: length should not be negative





@farhadabar I have tried to reproduce this multiple times and have not been sucessful. Is it possible to share the file that produces the issue?
I was able to reproduce this issue, but it wasn't an issue with `tail()` rather an issue with indexing into a column by name. I suspect they are related to how we are partitioning parquet files.

I did this on MacOS: 12.0.1, python==3.9.10, modin==0.13.0

Here is a script:
```
import modin.pandas as pd 

df = pd.read_parquet("s3://modin-test/test_parquet_file.parquet", storage_options={"anon":True})
df["numerical_1"]
```

I can also confirm this breaks when reading the same parquet file locally and not from s3. I can also confirm this is broken on older versions of Modin as far back as 0.12.0.

Here is the stack trace: https://gist.github.com/jeffreykennethli/f0ed5408755b96b8b5133140f7def18a

Resolving patch:
diff --git a/modin/core/dataframe/pandas/dataframe/dataframe.py b/modin/core/dataframe/pandas/dataframe/dataframe.py
--- a/modin/core/dataframe/pandas/dataframe/dataframe.py
+++ b/modin/core/dataframe/pandas/dataframe/dataframe.py
@@ -192,6 +192,10 @@ def __init__(
                     sum(row_lengths), len(self._index_cache)
                 ),
             )
+            ErrorMessage.catch_bugs_and_request_email(
+                any(val < 0 for val in row_lengths),
+                "Row lengths cannot be negative: {}".format(row_lengths),
+            )
         self._row_lengths_cache = row_lengths
         if column_widths is not None and len(self.columns) > 0:
             ErrorMessage.catch_bugs_and_request_email(
@@ -200,6 +204,10 @@ def __init__(
                     sum(column_widths), len(self._columns_cache)
                 ),
             )
+            ErrorMessage.catch_bugs_and_request_email(
+                any(val < 0 for val in column_widths),
+                "Column widths cannot be negative: {}".format(column_widths),
+            )
         self._column_widths_cache = column_widths
         self._dtypes = dtypes
         self._filter_empties()
diff --git a/modin/core/io/column_stores/column_store_dispatcher.py b/modin/core/io/column_stores/column_store_dispatcher.py
--- a/modin/core/io/column_stores/column_store_dispatcher.py
+++ b/modin/core/io/column_stores/column_store_dispatcher.py
@@ -134,8 +134,8 @@ def build_index(cls, partition_ids):
         else:
             row_lengths = [
                 index_chunksize
-                if i != num_partitions - 1
-                else index_len - (index_chunksize * (num_partitions - 1))
+                if (i + 1) * index_chunksize < index_len
+                else max(0, index_len - (index_chunksize * i))
                 for i in range(num_partitions)
             ]
         return index, row_lengths


Test patch:
diff --git a/modin/pandas/test/test_io.py b/modin/pandas/test/test_io.py
--- a/modin/pandas/test/test_io.py
+++ b/modin/pandas/test/test_io.py
@@ -19,6 +19,7 @@
 from pandas.core.dtypes.common import is_list_like
 from pathlib import Path
 from collections import OrderedDict
+from modin.config.envvars import MinPartitionSize
 from modin.db_conn import (
     ModinDatabaseConnection,
     UnsupportedDatabaseException,
@@ -1282,6 +1283,25 @@ def test_read_parquet(self, make_parquet_file, columns):
             columns=columns,
         )
 
+    @pytest.mark.xfail(
+        condition="config.getoption('--simulate-cloud').lower() != 'off'",
+        reason="The reason of tests fail in `cloud` mode is unknown for now - issue #3264",
+    )
+    def test_read_parquet_indexing_by_column(self, make_parquet_file):
+        # Test indexing into a column of Modin with various parquet file row lengths.
+        # Specifically, tests for https://github.com/modin-project/modin/issues/3527
+        # which fails when min_partition_size < nrows < min_partition_size * (num_partitions - 1)
+
+        nrows = (
+            MinPartitionSize.get() + 1
+        )  # Use the minimal guaranteed failing value for nrows.
+        unique_filename = get_unique_filename(extension="parquet")
+        make_parquet_file(filename=unique_filename, nrows=nrows)
+
+        parquet_df = pd.read_parquet(unique_filename)
+        for col in parquet_df.columns:
+            parquet_df[col]
+
     @pytest.mark.parametrize("columns", [None, ["col1"]])
     @pytest.mark.xfail(
         condition="config.getoption('--simulate-cloud').lower() != 'off'",