BUG/ENH: Index constructor with dtype now delegates
State: closed

...to Index subclass (if possible). Simplifies the Index constructor
slightly too, which is a nice side bonus. Will even better after Index
refactor.

Fixes #5196.


Labels:

Comments:
whoops, broke something!

This goes back to at least 0.11. I need to throw out what I have, but I'll reopen this once I fix it.

It turns out this is non-trivial. There are a bunch of places that just assume it's okay to pass dtypes with lists and work assuming this casting to object and then need to handle edge cases &c.

``` python
    def insert(self, loc, item):
        """
        Make new Index inserting new item at location

        Parameters
        ----------
        loc : int
        item : object

        Returns
        -------
        new_index : Index
        """
        index = np.asarray(self)
        # because numpy is fussy with tuples
        item_idx = Index([item], dtype=index.dtype)
        new_index = np.concatenate((index[:loc], item_idx, index[loc:]))
        return Index(new_index, name=self.name)
```

one example has item as `date1` and then the `item_idx` line just works no matter what because it gets casted to object index. e.g. this example:

``` python
In [3]: ind = pd.Index(range(10))
In [4]: ind
Out[4]: Int64Index([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype='int64')
In [5]: ind.insert(5, 5)
Out[5]: Int64Index([0, 1, 2, 3, 4, 5, 5, 6, 7, 8, 9], dtype='int64')
In [6]: ind.insert(5, 'apples')
Out[6]: Index([0, 1, 2, 3, 4, u'apples', 5, 6, 7, 8, 9], dtype='object')
```

If I can make everything pass just by comparing dtype, then I'll do it, but otherwise I don't want to screw with something that is assumed in many places.

More generally, the question is whether the Index constructor should raise when passing a non-matching dtype, just like numpy constructor raises with:  `np.array(['apples'], dtype='int64')`.  Currently, it sometimes returns object Index and sometimes just fails utterly.

Right now it only raises if the conversion causes a `ValueError` rather than `TypeError` (because the try/except only catches TypeError).

``` python
In [3]: import numpy as np

In [4]: ind = pd.Index(np.array([1, 2, 3]), dtype='datetime64[ns]')

In [5]: ind
Out[5]: Index([1, 2, 3], dtype='object')

In [6]: ind = pd.Index(np.array(['a', 'b', 'c'], dtype='int64')
   ...: )
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-dab2e51f9808> in <module>()
----> 1 ind = pd.Index(np.array(['a', 'b', 'c'], dtype='int64')
      2 )

ValueError: invalid literal for long() with base 10: 'a'

In [7]: ind = pd.Index(np.array(['a', 'b', 'c'], dtype='datetime64[ns]')
)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-7-f598c33a6c8a> in <module>()
----> 1 ind = pd.Index(np.array(['a', 'b', 'c'], dtype='datetime64[ns]')
      2 )

ValueError: Error parsing datetime string "a" at position 0
```

@cpcloud btw -this could be why those alignment tests fail weirdly only on certain machines. I'm guessing they're being passed an explicit dtype and then returning object Index instead of raising.

Diff for me to remember with test cases.

``` diff
diff --git a/pandas/tests/test_index.py b/pandas/tests/test_index.py
index cd26016..0f30845 100644
--- a/pandas/tests/test_index.py
+++ b/pandas/tests/test_index.py
@@ -48,6 +48,56 @@ class TestIndex(unittest.TestCase):
         for name, ind in self.indices.items():
             setattr(self, name, ind)

+    def test_explicit_dtype_in_constructor(self):
+        # first, smoke tests to make sure result is the same
+        # (with NON-ndarray)
+        pairs = [(np.dtype(unicode), self.unicodeIndex),
+                 (np.dtype(str), self.strIndex),
+                 ('datetime64[ns]', self.dateIndex),
+                 (np.dtype(int), self.intIndex),
+                 (np.dtype(float), self.floatIndex),
+                 (np.dtype(object), self.empty),
+                 (np.dtype(object), self.tuples)]
+        for dtype, original_index in pairs:
+            new_index = Index(list(original_index), dtype=str(dtype))
+            # key is that type is the same
+            tm.assert_isinstance(new_index, type(original_index))
+            tm.assert_index_equal(new_index, original_index)
+
+        # float coerces to Float64Index (even if it could be Int64)
+        ind = Index([1, 3, 5], dtype=float)
+        expected = Float64Index([1., 3., 5.])
+        tm.assert_isinstance(ind, Float64Index)
+        tm.assert_index_equal(ind, expected)
+
+        # These don't need to stay the same if new Index types are added
+        # object-like without explicit dtype
+        ind = Index(['s', 5, 3, None])
+        assert ind.dtype == np.object_
+        assert type(ind) is Index, "Expected Index type, found %s" % type(ind)
+        tm.assert_almost_equal(ind.values, ['s', 5, 3, None])
+
+        # int with nan goes to Float64Index
+        vals = [1, 3, 5, np.nan, 3, -3, -4]
+        ind1 = Index(vals, dtype=float)
+        # as should list-like without dtype
+        ind2 = Index(vals)
+
+        for ind in (ind1, ind2):
+            tm.assert_almost_equal(ind.values, vals)
+            tm.assert_isinstance(ind1, Float64Index)
+
+        # doesn't rescue with explicit (but wrong) dtype
+        with tm.assertRaises(ValueError):
+            Index(vals, dtype=int)
+
+        # bad dtypes with list-likes
+        with tm.assertRaisesRegexp(TypeError, "data type .banana"):
+            Index(['apple', 1, 2], dtype='banana')
+
+        with tm.assertRaisesRegexp(TypeError, "data type .ribbit"):
+            Index(range(3), dtype='ribbit')
+
     def test_wrong_number_names(self):
         def testit(ind):
             ind.names = ["apple", "banana", "carrot"]
```