Repository: pandas-dev/pandas
Pull request: BUG: Series.rank modifies inplace with NaT
- [x] closes #18521
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Looks like `_ensure_data` converts NaT to -9223372036854775808 then that gets converted back to datetime64.

Problem statement:
rank() makes unexpected inplace changes
#### Code Sample

```python
import pandas as pd

dt = pd.Series([pd.Timestamp('2017-01-05 10:20:27.569000'), pd.NaT])
print("\nBefore")
print(dt)

# Should not modify the object
ranks = dt.rank()

print("\nAfter")
print(dt)
```

```
Before
0   2017-01-05 10:20:27.569
1                       NaT
dtype: datetime64[ns]

After
0   2017-01-05 10:20:27.569000000
1   2262-04-11 23:47:16.854775807
dtype: datetime64[ns]
```

#### Problem description

The rank() method replaces NaT with a concrete date value in the object.

#### Expected Output

No change in the object.

#### Output of ``pd.show_versions()``

<details>

/Users/naure/Library/Python/3.6/lib/python/site-packages/xarray/core/formatting.py:16: FutureWarning: The pandas.tslib module is deprecated and will be removed in a future version.
  from pandas.tslib import OutOfBoundsDatetime

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.1.final.0
python-bits: 64
OS: Darwin
OS-release: 15.4.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.20.1
pytest: None
pip: 9.0.1
setuptools: 36.0.1
Cython: None
numpy: 1.13.1
scipy: 0.19.0
xarray: 0.9.5
IPython: 6.1.0
sphinx: None
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.0
tables: 3.4.2
numexpr: 2.6.2
feather: None
matplotlib: 2.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.999999999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None

</details>

Hints from issue discussion:
Yikes!  I dislike surprises like these as much as the next developer.  Patch for this behavior is most welcome!

Commit message: Fixed whatsnew; copy value in rank_1d

Patch to review:
diff --git a/doc/source/whatsnew/v0.22.0.txt b/doc/source/whatsnew/v0.22.0.txt
index c2da0c420f643..1e8aa6eaf2ef4 100644
--- a/doc/source/whatsnew/v0.22.0.txt
+++ b/doc/source/whatsnew/v0.22.0.txt
@@ -336,4 +336,4 @@ Other
 ^^^^^
 
 - Improved error message when attempting to use a Python keyword as an identifier in a ``numexpr`` backed query (:issue:`18221`)
--
+- Bug in :func:`Series.rank` where ``Series`` containing ``NaT`` modifies the ``Series`` inplace (:issue:`18521`)
diff --git a/pandas/_libs/algos_rank_helper.pxi.in b/pandas/_libs/algos_rank_helper.pxi.in
index 0e46530e20d1c..cc31ab1cd4a2d 100644
--- a/pandas/_libs/algos_rank_helper.pxi.in
+++ b/pandas/_libs/algos_rank_helper.pxi.in
@@ -84,6 +84,9 @@ def rank_1d_{{dtype}}(object in_arr, ties_method='average', ascending=True,
     mask = np.isnan(values)
     {{elif dtype == 'int64'}}
     mask = values == iNaT
+    # create copy in case of iNaT
+    if mask.any():
+        values = values.copy()
     {{endif}}
 
     # double sort first by mask and then by values to ensure nan values are
diff --git a/pandas/tests/frame/test_analytics.py b/pandas/tests/frame/test_analytics.py
index 4bba6d7601ae8..10573558d13cd 100644
--- a/pandas/tests/frame/test_analytics.py
+++ b/pandas/tests/frame/test_analytics.py
@@ -2214,3 +2214,11 @@ def test_series_broadcasting(self):
             df_nan.clip_lower(s, axis=0)
             for op in ['lt', 'le', 'gt', 'ge', 'eq', 'ne']:
                 getattr(df, op)(s_nan, axis=0)
+
+    def test_series_nat_conversion(self):
+        # GH 18521
+        df = DataFrame(np.random.randn(10, 3), dtype='float64')
+        expected = df.copy()
+        df.rank()
+        result = df
+        tm.assert_frame_equal(result, expected)
diff --git a/pandas/tests/series/test_rank.py b/pandas/tests/series/test_rank.py
index 311d14e928caa..bdc98f7c2e9a8 100644
--- a/pandas/tests/series/test_rank.py
+++ b/pandas/tests/series/test_rank.py
@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-from pandas import compat
+from pandas import compat, Timestamp
 
 import pytest
 
@@ -368,3 +368,12 @@ def test_rank_object_bug(self):
         # smoke tests
         Series([np.nan] * 32).astype(object).rank(ascending=True)
         Series([np.nan] * 32).astype(object).rank(ascending=False)
+
+    def test_rank_modify_inplace(self):
+        # GH 18521
+        s = Series([Timestamp('2017-01-05 10:20:27.569000'), NaT])
+        expected = s.copy()
+
+        s.rank()
+        result = s
+        assert_series_equal(result, expected)


Code review comments:
diff_hunk: @@ -336,4 +336,4 @@ Other
 ^^^^^
 
 - Improved error message when attempting to use a Python keyword as an identifier in a ``numexpr`` backed query (:issue:`18221`)
--
path: doc/source/whatsnew/v0.22.0.txt
text: @user1:
can you move to the reshaping section (n bug fix)