Safer is dtype
State: closed

Closes https://github.com/pandas-dev/pandas/issues/22960

Labels:
name: CI
description: Continuous Integration

Comments:
Hello @TomAugspurger! Thanks for updating the PR.

 - There are no PEP8 issues in the file [`pandas/core/dtypes/base.py`](https://github.com/TomAugspurger/pandas/blob/073df470ef2e645dd886d568490da50e4574acb4/pandas/core/dtypes/base.py) !

 - There are no PEP8 issues in the file [`pandas/core/frame.py`](https://github.com/TomAugspurger/pandas/blob/073df470ef2e645dd886d568490da50e4574acb4/pandas/core/frame.py) !

 - There are no PEP8 issues in the file [`pandas/core/series.py`](https://github.com/TomAugspurger/pandas/blob/073df470ef2e645dd886d568490da50e4574acb4/pandas/core/series.py) !

 - There are no PEP8 issues in the file [`pandas/tests/dtypes/test_dtypes.py`](https://github.com/TomAugspurger/pandas/blob/073df470ef2e645dd886d568490da50e4574acb4/pandas/tests/dtypes/test_dtypes.py) !

 - There are no PEP8 issues in the file [`pandas/tests/frame/test_operators.py`](https://github.com/TomAugspurger/pandas/blob/073df470ef2e645dd886d568490da50e4574acb4/pandas/tests/frame/test_operators.py) !



##### Comment last updated on October 03, 2018 at 20:20 Hours UTC
@jreback @pambot thoughts on broadening the caught exceptions in https://github.com/pandas-dev/pandas/pull/22975/files#diff-8c2f3d0cbd1523cafd79c45e87124103R100?. According to the docs, only TypeError is "allowed" to be raised, but python has no way to enforce that. I'm not sure what's best here.
Failure is from #22019 (more evidence that moving the numpydev warnings build to the main section is a good idea).

I'm going to include a fix for that here.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22975?src=pr&el=h1) Report
> Merging [#22975](https://codecov.io/gh/pandas-dev/pandas/pull/22975?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/fea27f0736a4b8f6626da60a6abc2f6e26b8a365?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/22975/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/22975?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #22975      +/-   ##
==========================================
+ Coverage   92.19%   92.19%   +<.01%     
==========================================
  Files         169      169              
  Lines       50821    50825       +4     
==========================================
+ Hits        46852    46857       +5     
+ Misses       3969     3968       -1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.61% <100%> (ø)` | :arrow_up: |
| #single | `42.36% <60%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/22975?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/dtypes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/22975/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2Jhc2UucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/22975/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `93.75% <100%> (ø)` | :arrow_up: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/22975/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.2% <100%> (ø)` | :arrow_up: |
| [pandas/util/\_depr\_module.py](https://codecov.io/gh/pandas-dev/pandas/pull/22975/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvX2RlcHJfbW9kdWxlLnB5) | `65.11% <0%> (-2.33%)` | :arrow_down: |
| [pandas/core/resample.py](https://codecov.io/gh/pandas-dev/pandas/pull/22975/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzYW1wbGUucHk=) | `96.98% <0%> (+0.01%)` | :arrow_up: |
| [pandas/core/groupby/grouper.py](https://codecov.io/gh/pandas-dev/pandas/pull/22975/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9ncm91cGVyLnB5) | `98.16% <0%> (+0.73%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22975?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22975?src=pr&el=footer). Last update [fea27f0...073df47](https://codecov.io/gh/pandas-dev/pandas/pull/22975?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

cc @jbrockmendel for the ops changes to catch that warning from numpy. Does  https://github.com/pandas-dev/pandas/pull/22975/files#diff-1e79abbbdd150d4771b91ea60a4e1cc7R4911 look like the right place to do it? Looking at the call stack, we can do it in either `_combine_match_index` or  `_combine_series_frame` Given that `combine_match_index` dispatches to series (which I think *may* go to EA ops), I don't think we should be catching NumPy warnings there. I'd rather leave that up to the EA authors.
Darn, we're back to the failure in `test_ndarray_compat`, which I cannot reproduce locally. https://travis-ci.org/pandas-dev/pandas/jobs/436672887#L2765


```
______________________ TestSeriesMisc.test_ndarray_compat ______________________
[gw1] linux -- Python 3.7.0 /home/travis/miniconda3/envs/pandas/bin/python
self = <pandas.tests.series.test_api.TestSeriesMisc object at 0x7f94a64fdbe0>
    def test_ndarray_compat(self):
    
        # test numpy compat with Series as sub-class of NDFrame
        tsdf = DataFrame(np.random.randn(1000, 3), columns=['A', 'B', 'C'],
                         index=date_range('1/1/2000', periods=1000))
    
        def f(x):
            return x[x.idxmax()]
    
>       result = tsdf.apply(f)
pandas/tests/series/test_api.py:405: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/core/frame.py:6240: in apply
    return op.get_result()
pandas/core/apply.py:151: in get_result
    return self.apply_standard()
pandas/core/apply.py:245: in apply_standard
    dummy = Series(empty_arr, index=index, dtype=values.dtype)
pandas/core/series.py:277: in __init__
    raise_cast_failure=True)
pandas/core/series.py:4270: in _sanitize_array
    subarr = _try_cast(data, True)
pandas/core/series.py:4228: in _try_cast
    subarr = maybe_cast_to_integer_array(arr, dtype)
pandas/core/dtypes/cast.py:1350: in maybe_cast_to_integer_array
    if np.array_equal(arr, casted):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
a1 = array([ 6.93066801e-310,  6.93066801e-310,  4.63759004e-310,
        4.63759004e-310,  6.93058871e-310, -3.80607598e+0...93066821e-310, -9.21064254e+018,
        6.93058666e-310, -3.04661570e+290,  0.00000000e+000,
        0.00000000e+000])
a2 = array([ 6.93066801e-310,  6.93066801e-310,  4.63759004e-310,
        4.63759004e-310,  6.93058871e-310, -3.80607598e+0...93066821e-310, -9.21064254e+018,
        6.93058666e-310, -3.04661570e+290,  0.00000000e+000,
        0.00000000e+000])
    def array_equal(a1, a2):
        """
        True if two arrays have the same shape and elements, False otherwise.
    
        Parameters
        ----------
        a1, a2 : array_like
            Input arrays.
    
        Returns
        -------
        b : bool
            Returns True if the arrays are equal.
    
        See Also
        --------
        allclose: Returns True if two arrays are element-wise equal within a
                  tolerance.
        array_equiv: Returns True if input arrays are shape consistent and all
                     elements equal.
    
        Examples
        --------
        >>> np.array_equal([1, 2], [1, 2])
        True
        >>> np.array_equal(np.array([1, 2]), np.array([1, 2]))
        True
        >>> np.array_equal([1, 2], [1, 2, 3])
        False
        >>> np.array_equal([1, 2], [1, 4])
        False
    
        """
        try:
            a1, a2 = asarray(a1), asarray(a2)
        except Exception:
            return False
        if a1.shape != a2.shape:
            return False
>       return bool(asarray(a1 == a2).all())
E       DeprecationWarning: elementwise comparison failed; this will raise an error 
```
> [...] look like the right place to do it?

I think it should be higher up the stack where `func` is defined.
You're thinking something like this?

```python
def func2(...):
    with np.errstate(all='ignore'):
        func(...)
```

and `func2` is whats actually passed down through the chain.
> You're thinking something like this?

Close.  `_combine_match_index` only gets called from one place in `_combine_series_frame`.  `_combine_series_frame` gets called from 3 places.  I'd want to track the problem back and see which of these paths is actually relevant.  i.e. do this particular wrapping as early as possible so as to wrap as little as possible.

Does the problem show up exclusively in `comparison(DataFrame, Series)` ops?  That would be the simplest explanation with the cleanest fix.
Will take a look at who hits this.

jumping back to the other failure, this seems strange. The error is coming in `FrameApply.apply_standard` when we do `dummy = Series(empty_arr, index=index, dtype=values.dtype)`.

Eventually, we get to `_sanitize_array`, which calls `maybe_cast_to_integer_dtype`. But why the `if is_float_dtype(dtype)`?

```python
In [1]: import pandas as pd

In [2]: pd.Series([1., 2], dtype=float)
> /Users/taugspurger/sandbox/pandas-travis-37-numpydev/pandas/core/series.py(4229)_try_cast()
-> subarr = maybe_cast_to_integer_array(arr, dtype)
(Pdb) l
4224            try:
4225                # gh-15832: Check if we are requesting a numeric dype and
4226                # that we can convert the data to the requested dtype.
4227                if is_float_dtype(dtype) or is_integer_dtype(dtype):
4228                    breakpoint()
4229 ->                 subarr = maybe_cast_to_integer_array(arr, dtype)
4230
4231                subarr = maybe_cast_to_datetime(arr, dtype)
4232                # Take care in creating object arrays (but iterators are not
4233                # supported):
4234                if is_object_dtype(dtype) and (is_list_like(subarr) and
(Pdb) dtype
dtype('float64')
```

cc @gfyoung that's from https://github.com/pandas-dev/pandas/pull/21456, if you recall.
@jbrockmendel updated with what should be a more focused filter, though I'm not sure. https://github.com/pandas-dev/pandas/pull/22975/files#diff-f38e1e99cb32831f13ced48de4220eacR141

AFAICT, the only warning leaking through was truediv when the RHS had 0s. So I replaced `operator.truediv` with a wrapper that sets the errstate.

Compared to my earlier commit, I think the difference is that

1. The errstate is applied just to truediv and rtruediv, not every function that reached that else block in `_combine_match_index`.
2. The errstate is applied to `truediv`, regardless of which path it takes. In particular, `ops.dispatch_to_series` is also called with the `silent_truediv` function. That ends up not mattering here, since another errstate is applied down in `expressions._evaluate_standard`.
I'm not sure what was wrong with 
https://github.com/pandas-dev/pandas/pull/22975/commits/d88929f38c061d0e4506ba5bc00ab8002a7acb5e, but there were warnings coming through everywhere. Gone with the original approach of all was calling `func` with the errstate set. That matches what we eventually do for series anyway.
> Gone with the original approach of all was calling func with the errstate set.

OK.  It's not my favorite thing, but I can live with it.
Any other thoughts @jreback, or are we good to merge?
thanks!