Repository: modin-project/modin

Issue:
ValueError: bins must be monotonically increasing or decreasing
### System information
- **OS Platform and Distribution** : Window 10 Pro 
- **Modin installed from (source or binary)** : via pip 
- **Modin version** : 0.6.2 
- **Python version** : 3.6.5 
- **Exact command to reproduce** : 
import modin.pandas as pd 
import numpy as np 
trans = pd.read_csv('./Users/data.csv',parse_dates=['par_day']) 
trans.drop_duplicates(inplace=True) 
trans[['col1','col2',...]]



### Describe the problem
I have tried to process my csv file. when I used ' drop_duplicates ' or double backet to select the colunm (ex. trans[['col1','col2',..]]. this error occurs

### Source code / logs
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-7-981102ef416d> in <module>
      1 # drop dubplicate
----> 2 trans.drop_duplicates(inplace=True)
      3 dfs = trans[['par_day','invoice_number','doc_type','cv_group','cv_group_province','saleman_code', 'cv_code',
      4                       'product_code', 'product_name','quantity_weight', 'unit_selling_price', 'extended_amount','warehouse_id',
      5                        'warehouse_code', 'warehouse_name']]

~\Anaconda3\lib\site-packages\modin\pandas\dataframe.py in drop_duplicates(self, subset, keep, inplace)
    187     def drop_duplicates(self, subset=None, keep="first", inplace=False):
    188         return super(DataFrame, self).drop_duplicates(
--> 189             subset=subset, keep=keep, inplace=inplace
    190         )
    191 

~\Anaconda3\lib\site-packages\modin\pandas\base.py in drop_duplicates(self, keep, inplace, **kwargs)
   1055             duplicates = self.duplicated(keep=keep, **kwargs)
   1056         indices, = duplicates.values.nonzero()
-> 1057         return self.drop(index=self.index[indices], inplace=inplace)
   1058 
   1059     def duplicated(self, keep="first", **kwargs):

~\Anaconda3\lib\site-packages\modin\pandas\base.py in drop(self, labels, axis, index, columns, level, inplace, errors)
    968 
    969         new_query_compiler = self._query_compiler.drop(
--> 970             index=axes["index"], columns=axes["columns"]
    971         )
    972         return self._create_or_update_from_compiler(new_query_compiler, inplace)

~\Anaconda3\lib\site-packages\modin\backends\pandas\query_compiler.py in drop(self, index, columns)
    973             )
    974         new_modin_frame = self._modin_frame.mask(
--> 975             row_numeric_idx=index, col_numeric_idx=columns
    976         )
    977         return self.__constructor__(new_modin_frame)

~\Anaconda3\lib\site-packages\modin\engines\base\frame\data.py in mask(self, row_indices, row_numeric_idx, col_indices, col_numeric_idx)
    338         if row_numeric_idx is not None:
    339             row_partitions_list = self._get_dict_of_block_index(
--> 340                 1, row_numeric_idx, ordered=True
    341             )
    342             new_row_lengths = [len(indices) for _, indices in row_partitions_list]

~\Anaconda3\lib\site-packages\modin\engines\base\frame\data.py in _get_dict_of_block_index(self, axis, indices, ordered)
    564             )
    565 
--> 566         partition_ids = np.digitize(indices, cumulative)
    567         # If the output order doesn't matter or if the indices are monotonically
    568         # increasing, the computation is significantly simpler and faster than doing

ValueError: bins must be monotonically increasing or decreasing



Discussion:
Hi @ggpython, thanks for the report!

Is the CSV file something you can share? It seems like something is going wrong with the metadata calculation after the data is read, but I can't reproduce the issue. If you could share the data file, that would help a lot!
Hi @ggpython, I took another stab at reproducing this today and I found it is a subtle issue with metadata related to the Dask backend on Windows specifically. 

Thanks again for reporting!

Resolving patch:
diff --git a/modin/engines/base/frame/data.py b/modin/engines/base/frame/data.py
--- a/modin/engines/base/frame/data.py
+++ b/modin/engines/base/frame/data.py
@@ -544,15 +544,11 @@ def _get_dict_of_block_index(self, axis, indices, ordered=False):
         else:
             indices = np.array(indices)
         if not axis:
-            # INT_MAX to make sure we don't try to compute on partitions that don't
-            # exist.
-            cumulative = np.array(
-                np.append(self._column_widths[:-1], np.iinfo(np.int32).max)
-            ).cumsum()
+            bins = np.array(self._column_widths)
         else:
-            cumulative = np.array(
-                np.append(self._row_lengths[:-1], np.iinfo(np.int32).max)
-            ).cumsum()
+            bins = np.array(self._row_lengths)
+        # INT_MAX to make sure we don't try to compute on partitions that don't exist.
+        cumulative = np.append(bins[:-1].cumsum(), np.iinfo(bins.dtype).max)
 
         def internal(block_idx, global_index):
             return (


Test patch: