Repository: pandas-dev/pandas

Issue:
DateTimeIndex values are assigned across entire df when using .loc
I posted a 2-part answer to this question on SO: http://stackoverflow.com/questions/28482553/pandas-set-value-of-column-to-value-of-index-based-on-condtion

What I noticed is that if your index is a datetimeindex then assigning the values is not respecting the column selection and is blatting all rows.

I am using pandas 0.15.2 using numpy 1.9.1 and python 3.4 64-bit

example:

```
In [46]:

rows = 3
df = pd.DataFrame(np.random.randn(rows,2), columns=list('AB'), index=pd.date_range('1/1/2000', periods=rows, freq='1H'))
print(df)
df.loc[df.A > 0.5, 'LAST_TIME_A_ABOVE_X'] = df.loc[df.A > 0.5].index
df
                            A         B
2000-01-01 00:00:00 -0.761643  0.969167
2000-01-01 01:00:00  0.050335 -1.346953
2000-01-01 02:00:00  0.663857 -0.272247
Out[46]:
                             A                             B  \
2000-01-01 00:00:00 1970-01-01           1970-01-01 00:00:00   
2000-01-01 01:00:00 1970-01-01 1969-12-31 23:59:59.999999999   
2000-01-01 02:00:00 1970-01-01           1970-01-01 00:00:00   

                    LAST_TIME_A_ABOVE_X  
2000-01-01 00:00:00                 NaT  
2000-01-01 01:00:00                 NaT  
2000-01-01 02:00:00 2000-01-01 02:00:00 
```

If the index is an Int64 type then this doesn't happen.

If you reset the index and then assign the values using .loc then it works correctly



Discussion:
This is kind of tricky. You are assigning possibly a single value (or maybe a list, depends on what A is). with a list-like, IOW, and index (regardless of the value of A, it may have 0 or more elements).
If its a list on the rhs, then it will try to broadcast it, that's why you get values in every column.

So I would say this is not really safe to do

```
In [57]: df = pd.DataFrame(np.random.randn(rows,2), columns=list('AB'), index=pd.date_range('1/1/2000', periods=rows, freq='1H'))

In [58]: df
Out[58]: 
                            A         B
2000-01-01 00:00:00  0.117055  0.554529
2000-01-01 01:00:00 -1.587738 -0.913139
2000-01-01 02:00:00  1.439404 -0.521966

In [59]: df.loc[df.A>0.5].index[0]
Out[59]: Timestamp('2000-01-01 02:00:00', offset='H')

In [60]: df.loc[df.A>0.5,'new'] = (df.loc[df.A>0.5].index)[0]

In [61]: df
Out[61]: 
                            A         B                 new
2000-01-01 00:00:00  0.117055  0.554529                 NaT
2000-01-01 01:00:00 -1.587738 -0.913139                 NaT
2000-01-01 02:00:00  1.439404 -0.521966 2000-01-01 02:00:00
```

Here is a better/safe way to do this.

```
In [91]: df['new'] = df.index.to_series().where(df.A>0.5)

In [92]: df
Out[92]: 
                            A         B        new
2000-01-01 00:00:00  0.559047 -0.366489 2000-01-01
2000-01-01 01:00:00  0.249399 -0.780957        NaT
2000-01-01 02:00:00 -1.244441  1.364961        NaT
```

OK but is there a reason why the datetimeindex behaves differently to when it's just an ordinary column of datetime64 dtype? For instance after df creation you then did `df.reset_index(inplace=True)
temp = df.loc[df.A > 0.5,'index']
df.loc[df.A > 0.5, 'LAST_TIME_A_ABOVE_X'] = temp` then this works as expected

@alan-wong what you are posting is not the same thing. you are assigning with an aligned series. this makes all the difference. The purpose of the index is to _align_ things. When you use `df.loc[df.A>0.5].index` you get a `DatetimeIndex`, which is not aligned to anything, its just like a list of values. 

```
In [131]: mask=df.A>0.5

In [132]: mask
Out[132]: 
2000-01-01 00:00:00     True
2000-01-01 01:00:00    False
2000-01-01 02:00:00    False
Freq: H, Name: A, dtype: bool

In [133]: df.loc[mask,'new'] = df.loc[mask].index.tolist()

In [134]: df
Out[134]: 
                            A         B        new
2000-01-01 00:00:00  1.321158 -1.546906 2000-01-01
2000-01-01 01:00:00 -0.202646 -0.655969        NaT
2000-01-01 02:00:00  0.193421  0.553439        NaT
```

I'll mark this as a bug, it seems the index (on the rhs) is not being treated as list-like, and instead maybe its trying to align.

OK I see the semantic difference but just to point out that you don't observe this behaviour if the index is int64 : `rows=3 df = pd.DataFrame(np.random.randn(rows,2), columns=list('AB'), index=np.arange(rows))
df.loc[df.A > 0.5, 'LAST_TIME_A_ABOVE_X'] = df.loc[df.A > 0.5].index` works as expected which is why I think this is something specific with datetimeindex


Resolving patch:
diff --git a/pandas/core/common.py b/pandas/core/common.py
--- a/pandas/core/common.py
+++ b/pandas/core/common.py
@@ -1011,6 +1011,27 @@ def conv(r, dtype):
     return [conv(r, dtype) for r, dtype in zip(result, dtypes)]
 
 
+def _infer_fill_value(val):
+    """
+    infer the fill value for the nan/NaT from the provided scalar/ndarray/list-like
+    if we are a NaT, return the correct dtyped element to provide proper block construction
+
+    """
+
+    if not is_list_like(val):
+        val = [val]
+    val = np.array(val,copy=False)
+    if is_datetimelike(val):
+        return np.array('NaT',dtype=val.dtype)
+    elif is_object_dtype(val.dtype):
+        dtype = lib.infer_dtype(_ensure_object(val))
+        if dtype in ['datetime','datetime64']:
+            return np.array('NaT',dtype=_NS_DTYPE)
+        elif dtype in ['timedelta','timedelta64']:
+            return np.array('NaT',dtype=_TD_DTYPE)
+    return np.nan
+
+
 def _infer_dtype_from_scalar(val):
     """ interpret the dtype from a scalar, upcast floats and ints
         return the new value and the dtype """
diff --git a/pandas/core/index.py b/pandas/core/index.py
--- a/pandas/core/index.py
+++ b/pandas/core/index.py
@@ -345,7 +345,10 @@ def _get_attributes_dict(self):
         return dict([ (k,getattr(self,k,None)) for k in self._attributes])
 
     def view(self, cls=None):
-        if cls is not None and not issubclass(cls, Index):
+
+        # we need to see if we are subclassing an
+        # index type here
+        if cls is not None and not hasattr(cls,'_typ'):
             result = self._data.view(cls)
         else:
             result = self._shallow_copy()
diff --git a/pandas/core/indexing.py b/pandas/core/indexing.py
--- a/pandas/core/indexing.py
+++ b/pandas/core/indexing.py
@@ -8,7 +8,7 @@
 from pandas.core.common import (_is_bool_indexer, is_integer_dtype,
                                 _asarray_tuplesafe, is_list_like, isnull,
                                 ABCSeries, ABCDataFrame, ABCPanel, is_float,
-                                _values_from_object)
+                                _values_from_object, _infer_fill_value)
 import pandas.lib as lib
 
 import numpy as np
@@ -238,7 +238,9 @@ def _setitem_with_indexer(self, indexer, value):
                             self.obj[key] = value
                             return self.obj
 
-                        self.obj[key] = np.nan
+
+                        # add a new item with the dtype setup
+                        self.obj[key] = _infer_fill_value(value)
 
                         new_indexer = _convert_from_missing_indexer_tuple(
                             indexer, self.obj.axes)


Test patch:
diff --git a/pandas/tests/test_index.py b/pandas/tests/test_index.py
--- a/pandas/tests/test_index.py
+++ b/pandas/tests/test_index.py
@@ -297,6 +297,9 @@ def test_view(self):
         i_view = i.view()
         self.assertEqual(i_view.name, 'Foo')
 
+        # with arguments
+        self.assertRaises(TypeError, lambda : i.view('i8'))
+
     def test_legacy_pickle_identity(self):
 
         # GH 8431
@@ -1469,6 +1472,12 @@ def test_view(self):
         i_view = i.view()
         self.assertEqual(i_view.name, 'Foo')
 
+        i_view = i.view('i8')
+        tm.assert_index_equal(i, Int64Index(i_view))
+
+        i_view = i.view(Int64Index)
+        tm.assert_index_equal(i, Int64Index(i_view))
+
     def test_coerce_list(self):
         # coerce things
         arr = Index([1, 2, 3, 4])
@@ -1856,7 +1865,21 @@ def test_slice_keep_name(self):
         idx = Int64Index([1, 2], name='asdf')
         self.assertEqual(idx.name, idx[1:].name)
 
-class TestDatetimeIndex(Base, tm.TestCase):
+class DatetimeLike(Base):
+
+    def test_view(self):
+
+        i = self.create_index()
+
+        i_view = i.view('i8')
+        result = self._holder(i)
+        tm.assert_index_equal(result, i)
+
+        i_view = i.view(self._holder)
+        result = self._holder(i)
+        tm.assert_index_equal(result, i)
+
+class TestDatetimeIndex(DatetimeLike, tm.TestCase):
     _holder = DatetimeIndex
     _multiprocess_can_split_ = True
 
@@ -1926,7 +1949,7 @@ def test_time_overflow_for_32bit_machines(self):
         self.assertEqual(len(idx2), periods)
 
 
-class TestPeriodIndex(Base, tm.TestCase):
+class TestPeriodIndex(DatetimeLike, tm.TestCase):
     _holder = PeriodIndex
     _multiprocess_can_split_ = True
 
@@ -1936,7 +1959,7 @@ def create_index(self):
     def test_pickle_compat_construction(self):
         pass
 
-class TestTimedeltaIndex(Base, tm.TestCase):
+class TestTimedeltaIndex(DatetimeLike, tm.TestCase):
     _holder = TimedeltaIndex
     _multiprocess_can_split_ = True
 
diff --git a/pandas/tests/test_indexing.py b/pandas/tests/test_indexing.py
--- a/pandas/tests/test_indexing.py
+++ b/pandas/tests/test_indexing.py
@@ -3158,6 +3158,19 @@ def f():
         df.loc[3] = [6,7]
         assert_frame_equal(df,DataFrame([[6,7]],index=[3],columns=['A','B'],dtype='float64'))
 
+    def test_partial_setting_with_datetimelike_dtype(self):
+
+        # GH9478
+        # a datetimeindex alignment issue with partial setting
+        df = pd.DataFrame(np.arange(6.).reshape(3,2), columns=list('AB'),
+                          index=pd.date_range('1/1/2000', periods=3, freq='1H'))
+        expected = df.copy()
+        expected['C'] = [expected.index[0]] + [pd.NaT,pd.NaT]
+
+        mask = df.A < 1
+        df.loc[mask, 'C'] = df.loc[mask].index
+        assert_frame_equal(df, expected)
+
     def test_series_partial_set(self):
         # partial set with new index
         # Regression from GH4825