Repository: ray-project/ray

Issue:
[Pandas on Ray] Cannot .loc by toDateTime Index
Hi @devin-petersohn , I retried my example on the new machine @pcmoritz gave me, and I ran into an issue where I could not run .loc by index after I converted the index to DateTime. The same code works in Pandas.

This time, I did figure how to print out the version of Ray that is installed on the new machine, so I hope you can let me know if this is indeed an old version of Ray or if this is a legitimate bug.

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 16.04.4 LTS (Xenial Xerus)
- **Ray installed from (source or binary)**: Unknown. Please ask Philipp Moritz, he was the one who installed Ray for me.
- **Ray version**: 0.4.0
- **Python version**: Anaconda Python 3.6.4
- **Exact command to reproduce**: 
```
import ray.dataframe as pd
# import pandas as pd

df = pd.read_csv("yellow_tripdata_2015-01-01.csv")
df = df.set_index('tpep_pickup_datetime')
df.index = pd.to_datetime(df.index)  # Ray does fine without this line

df.loc['2015-01-10 19:01:44']  #Error here
```


### Describe the problem
I get the traceback copied below in the **Source code/logs** section, when running the line `df.loc['2015-01-10 19:01:44']`. It complains that the Index is not unique-- indeed, there are multiple rows in the data that have the same timestamp for that column, and it is not monotonic either. 

However, in Pandas, this line normally returns a DataFrame of multiple rows that match the`2015-01-10 19:01:44` DateTime index. Ray correctly returns multiple rows if the index was not converted into DateTime first.

In case it helps, I'm under the impression this problem may have to do with the `pd.to_datetime` more than `DataFrame.loc` in this case. I once saw `pd.to_datetime` return all rows as `1970-01-01 00:00:00` when converting this column to DateTime (though I haven't reproduced that error since then), so I have seen funny behavior from `to_datetime` before from Pandas on Ray.

A similar error may have used to exist in an older version of Pandas, but was fixed in this Pandas Github issue: [https://github.com/pandas-dev/pandas/issues/3659](url)

Also, this Pandas Github issue mentions in the reply chain that the correct behavior for .loc with non-unique indices is a DataFrame of matching rows: [https://github.com/pandas-dev/pandas/issues/9466](url)

Additionally, I noticed when running Ray on this new machine that the memory errors seem **much** worse than before. In my original email example, I was using the January dataset (1/12 of all of 2015 data), and was able to run several parts in succession. 

However, this time when running Ray, it would hang and spam `ObjectID already exists in the object store` error messages immediately, on any next line, right after `pd.read_csv` (with my 1/12 data, running **isolated** from pandas or anything else). I wouldn't even be able to run `df.head()` right after reading in the csv this time around.

I had to split my January dataset in half, and use only 1/24 of the 2015 data, in order to just run this test. It may have just been that the new machine has less memory, but I do hope that I am indeed running from the right version of Ray.

You can get the source data csv I used at this page [http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtml](url) under 2015>January>Yellow. Note that this csv file is 2 GB.

### Source code / logs
Traceback returned at `df.loc['2015-01-10 19:01:44']` line:
```
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2319, in intersection
    indexer = Index(other._values).get_indexer(self._values)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2684, in get_indexer
    tolerance=tolerance)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2687, in get_indexer
    raise InvalidIndexError('Reindexing only valid with uniquely'
pandas.core.indexes.base.InvalidIndexError: Reindexing only valid with uniquely valued Index objects

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/ray/python/ray/dataframe/indexing.py", line 286, in __getitem__
    self._handle_enlargement(row_loc, col_loc)
  File "/home/ubuntu/ray/python/ray/dataframe/indexing.py", line 313, in _handle_enlargement
    new_meta = self._enlarge_axis(loc, axis=axis)
  File "/home/ubuntu/ray/python/ray/dataframe/indexing.py", line 335, in _enlarge_axis
    nan_labels = self._compute_enlarge_labels(locator, major_meta.index)
  File "/home/ubuntu/ray/python/ray/dataframe/indexing.py", line 376, in _compute_enlarge_labels
    common_labels = locator_as_index.intersection(base_index)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2309, in intersection
    return this.intersection(other)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2324, in intersection
    Index(other._values).get_indexer_non_unique(self._values)[0])
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2816, in get_indexer_non_unique
    indexer, missing = self._engine.get_indexer_non_unique(tgt_values)
  File "pandas/_libs/index.pyx", line 310, in pandas._libs.index.IndexEngine.get_indexer_non_unique
TypeError: 'NoneType' object is not iterable
``` 



Discussion:
Thanks @crystalzyan!

@simon-mo Do you know why we are getting a `NoneType` error? 

@kunalgosar Can you check the `to_datetime` problems?

Resolving patch:
diff --git a/python/ray/dataframe/indexing.py b/python/ray/dataframe/indexing.py
--- a/python/ray/dataframe/indexing.py
+++ b/python/ray/dataframe/indexing.py
@@ -370,7 +370,11 @@ def _compute_enlarge_labels(self, locator, base_index):
         Returns:
              nan_labels: The labels needs to be added
         """
-        locator_as_index = pandas.Index(locator)
+        # base_index_type can be pd.Index or pd.DatetimeIndex
+        # depending on user input and pandas behavior
+        # See issue #2264
+        base_index_type = type(base_index)
+        locator_as_index = base_index_type(locator)
 
         nan_labels = locator_as_index.difference(base_index)
         common_labels = locator_as_index.intersection(base_index)


Test patch: