Repository: pandas-dev/pandas

Issue:
Failing interpolation test
```
$ nosetests pandas/tests/test_generic.py:TestSeries.test_interp_quad
F
======================================================================
FAIL: test_interp_quad (pandas.tests.test_generic.TestSeries)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/phillip/Documents/code/py/pandas/pandas/tests/test_generic.py", line 339, in test_interp_quad
    assert_series_equal(result, expected)
  File "/home/phillip/Documents/code/py/pandas/pandas/util/testing.py", line 452, in assert_series_equal
    assert_attr_equal('dtype', left, right)
  File "/home/phillip/Documents/code/py/pandas/pandas/util/testing.py", line 369, in assert_attr_equal
    assert_equal(left_attr,right_attr,"attr is not equal [{0}]" .format(attr))
  File "/home/phillip/Documents/code/py/pandas/pandas/util/testing.py", line 354, in assert_equal
    assert a == b, "%s: %r != %r" % (msg.format(a,b), a, b)
AssertionError: attr is not equal [dtype]: dtype('int64') != dtype('float64')

----------------------------------------------------------------------
Ran 1 test in 0.041s

FAILED (failures=1)
```



Discussion:
cc @TomAugspurger

Does this make any sense? A float block with array of items with int dtype?

```
ipdb> self
SingleBlockManager
Items: Int64Index([1, 2, 3, 4], dtype=int64)
FloatBlock: 4 dtype: float64
```

I'm in `core.internals.apply`

items are the 'index'...so that is right

@cpcloud where do you see this failing? I can't repro on 64 or 32-bit

spoke too soon!

@TomAugspurger  that test just needs to have the expected be `int64` otherwise looks fine. as an FYI, maybe need 
some tests that don't infer dtypes (e.g. set `downcast=False` to have no inferring of the results)

So are you saying to change expected to `expected = Series([1, 4, 9, 16], index=[1, 2, 3, 4])` (int type), because that fails for me. I'm trying to figure out why the result [1., 4., 9., 16.] doesn't get downcast for me right now.

@TomAugspurger you may also want to put some more creative logic in there for inference. Since we know that we are going to only float/int coming in, you could always infer the ints so that you will get ints if possible and the floats will stay floats.

@TomAugspurger the result IS downcast (to int64), its the expected that is float64

> @TomAugspurger the result IS downcast (to int64), its the expected that is float64

Not for me: 

``` python
In [5]: result = sq.interpolate(method='quadratic')
In [6]: result
Out[6]: 
1     1
2     4
3     9
4    16
dtype: float64
```

---

Can you clear this up for me?  I think this is where things aren't going the same way.  `b` is the float block with the nan interpolated.

``` python
ipdb> !b
FloatBlock: 4 dtype: float64
ipdb> !b.values
array([  1.,   4.,   9.,  16.])
ipdb> !b.downcast(downcast)[0].values  # should be ints?
array([  1.,   4.,   9.,  16.])
ipdb> downcast
'infer'
```

That's in `pandas/core/internals.py(337)_maybe_downcast()`

I'll dig a bit deeper.

umm... in `/pandas/core/common.py(1064)_possibly_downcast_to_dtype()`:

``` python
ipdb> result
array([  1.,   4.,   9.,  16.])
ipdb> result.astype(dtype)
array([ 1,  4,  8, 16])
ipdb> dtype
dtype('int64')
ipdb> 
```

but back at in the interpreter:

``` python
In [10]: a.astype(np.int64)
Out[10]: array([ 1,  4,  9, 16])

In [11]: a = np.array([1., 4., 9., 16.])

In [12]: a.astype(np.int64)
Out[12]: array([ 1,  4,  9, 16])
```

This is a precision issue

```
array([  1.,   4.,   9.,  16.])
(Pdb) p result[0]
1.0
(Pdb) p result[1]
4.0
(Pdb) p result[2]
9.0000000000000036
(Pdb) p result[3]
16.0
```

thus this array is NOT equal to array([1,4,9,16])

thus should not be downcasted (though you can make a case that it close 'enough') to be....

```
(Pdb) result == new_result
array([ True,  True, False,  True], dtype=bool)
(Pdb) result.round(8) == new_result
array([ True,  True,  True,  True], dtype=bool)
```

should we round when trying to downcast to int?

I think I should just do `allclose` with the default tolerances (1e-5,1e-8).....

Fair enough.  And users can override that with `s.interpolate(…, infer=False)` right?   Where would the necessary changes need to be made?

Or were you saying `allclose` for the test?

yes...they can specify `infer=False` to turn off downcasting; I am going to put up a PR to basically use `allclose` to figure out if the values are downcastable, so going to change your test.


Resolving patch:
diff --git a/pandas/core/common.py b/pandas/core/common.py
--- a/pandas/core/common.py
+++ b/pandas/core/common.py
@@ -1053,7 +1053,7 @@ def _possibly_downcast_to_dtype(result, dtype):
             # do a test on the first element, if it fails then we are done
             r = result.ravel()
             arr = np.array([ r[0] ])
-            if (arr != arr.astype(dtype)).item():
+            if not np.allclose(arr,arr.astype(dtype)):
                 return result
 
             # a comparable, e.g. a Decimal may slip in here
@@ -1062,8 +1062,14 @@ def _possibly_downcast_to_dtype(result, dtype):
 
             if issubclass(result.dtype.type, (np.object_,np.number)) and notnull(result).all():
                 new_result = result.astype(dtype)
-                if (new_result == result).all():
-                    return new_result
+                try:
+                    if np.allclose(new_result,result):
+                        return new_result
+                except:
+
+                    # comparison of an object dtype with a number type could hit here
+                    if (new_result == result).all():
+                        return new_result
     except:
         pass
 
diff --git a/pandas/core/internals.py b/pandas/core/internals.py
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -376,10 +376,10 @@ def downcast(self, dtypes=None):
                 dtype = dtypes.get(item, self._downcast_dtype)
 
             if dtype is None:
-                nv = _block_shape(values[i])
+                nv = _block_shape(values[i],ndim=self.ndim)
             else:
                 nv = _possibly_downcast_to_dtype(values[i], dtype)
-                nv = _block_shape(nv)
+                nv = _block_shape(nv,ndim=self.ndim)
 
             blocks.append(make_block(nv, Index([item]), self.ref_items, ndim=self.ndim, fastpath=True))
 


Test patch:
diff --git a/pandas/tests/test_generic.py b/pandas/tests/test_generic.py
--- a/pandas/tests/test_generic.py
+++ b/pandas/tests/test_generic.py
@@ -59,7 +59,7 @@ def _axes(self):
         """ return the axes for my object typ """
         return self._typ._AXIS_ORDERS
 
-    def _construct(self, shape, value=None, **kwargs):
+    def _construct(self, shape, value=None, dtype=None, **kwargs):
         """ construct an object for the given shape
             if value is specified use that if its a scalar
             if value is an array, repeat it as needed """
@@ -74,7 +74,7 @@ def _construct(self, shape, value=None, **kwargs):
                     # remove the info axis
                     kwargs.pop(self._typ._info_axis_name,None)
                 else:
-                    arr = np.empty(shape)
+                    arr = np.empty(shape,dtype=dtype)
                     arr.fill(value)
             else:
                 fshape = np.prod(shape)
@@ -184,6 +184,32 @@ def test_numpy_1_7_compat_numeric_methods(self):
             if f is not None:
                 f(o)
 
+    def test_downcast(self):
+        # test close downcasting
+
+        o = self._construct(shape=4, value=9, dtype=np.int64)
+        result = o.copy()
+        result._data = o._data.downcast(dtypes='infer')
+        self._compare(result, o)
+
+        o = self._construct(shape=4, value=9.)
+        expected = o.astype(np.int64)
+        result = o.copy()
+        result._data = o._data.downcast(dtypes='infer')
+        self._compare(result, expected)
+
+        o = self._construct(shape=4, value=9.5)
+        result = o.copy()
+        result._data = o._data.downcast(dtypes='infer')
+        self._compare(result, o)
+
+        # are close
+        o = self._construct(shape=4, value=9.000000000005)
+        result = o.copy()
+        result._data = o._data.downcast(dtypes='infer')
+        expected = o.astype(np.int64)
+        self._compare(result, expected)
+
 class TestSeries(unittest.TestCase, Generic):
     _typ = Series
     _comparator = lambda self, x, y: assert_series_equal(x,y)
@@ -335,7 +361,7 @@ def test_interp_quad(self):
         _skip_if_no_scipy()
         sq = Series([1, 4, np.nan, 16], index=[1, 2, 3, 4])
         result = sq.interpolate(method='quadratic')
-        expected = Series([1., 4., 9., 16.], index=[1, 2, 3, 4])
+        expected = Series([1, 4, 9, 16], index=[1, 2, 3, 4])
         assert_series_equal(result, expected)
 
     def test_interp_scipy_basic(self):
@@ -589,7 +615,7 @@ def test_spline(self):
         _skip_if_no_scipy()
         s = Series([1, 2, np.nan, 4, 5, np.nan, 7])
         result = s.interpolate(method='spline', order=1)
-        expected = Series([1., 2, 3, 4, 5, 6, 7])  # dtype?
+        expected = Series([1, 2, 3, 4, 5, 6, 7])
         assert_series_equal(result, expected)