BUG: support casting from bool array to EA Integer dtype
State: closed

Cast boolean array to int before casting to EA Integer dtype.

- [x] closes #25211
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25265?src=pr&el=h1) Report
> Merging [#25265](https://codecov.io/gh/pandas-dev/pandas/pull/25265?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/5f73594d49ea06ada4c6779006c00aaf41935595?src=pr&el=desc) will **decrease** coverage by `49.79%`.
> The diff coverage is `50%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25265/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25265?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25265      +/-   ##
==========================================
- Coverage   91.93%   42.13%   -49.8%     
==========================================
  Files         166      166              
  Lines       52187    52189       +2     
==========================================
- Hits        47978    21991   -25987     
- Misses       4209    30198   +25989
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `?` | |
| #single | `42.13% <50%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25265?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/arrays/integer.py](https://codecov.io/gh/pandas-dev/pandas/pull/25265/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2ludGVnZXIucHk=) | `41.19% <50%> (-55.13%)` | :arrow_down: |
| [pandas/io/formats/latex.py](https://codecov.io/gh/pandas-dev/pandas/pull/25265/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/core/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/25265/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY2F0ZWdvcmljYWwucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/25265/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/25265/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/25265/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvY29udmVydGVyLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/formats/html.py](https://codecov.io/gh/pandas-dev/pandas/pull/25265/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvaHRtbC5weQ==) | `0% <0%> (-99.35%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/25265/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `0% <0%> (-95.46%)` | :arrow_down: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/25265/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `0% <0%> (-91.17%)` | :arrow_down: |
| [pandas/io/sas/sas\_xport.py](https://codecov.io/gh/pandas-dev/pandas/pull/25265/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfeHBvcnQucHk=) | `0% <0%> (-90.15%)` | :arrow_down: |
| ... and [124 more](https://codecov.io/gh/pandas-dev/pandas/pull/25265/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25265?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25265?src=pr&el=footer). Last update [5f73594...b3e1d76](https://codecov.io/gh/pandas-dev/pandas/pull/25265?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25265?src=pr&el=h1) Report
> Merging [#25265](https://codecov.io/gh/pandas-dev/pandas/pull/25265?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/304d8d4bc3b3cde7e2b463426bf484701df142cd?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25265/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25265?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25265      +/-   ##
==========================================
- Coverage   91.68%   91.67%   -0.01%     
==========================================
  Files         174      174              
  Lines       50704    50706       +2     
==========================================
- Hits        46488    46486       -2     
- Misses       4216     4220       +4
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.18% <100%> (ø)` | :arrow_up: |
| #single | `41.18% <50%> (-0.15%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25265?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/25265/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `93.67% <ø> (ø)` | :arrow_up: |
| [pandas/core/arrays/integer.py](https://codecov.io/gh/pandas-dev/pandas/pull/25265/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2ludGVnZXIucHk=) | `96.35% <100%> (+0.02%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/25265/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `78.94% <0%> (-10.53%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/25265/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.01% <0%> (-0.12%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25265?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25265?src=pr&el=footer). Last update [304d8d4...3bce8a1](https://codecov.io/gh/pandas-dev/pandas/pull/25265?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

https://github.com/pandas-dev/pandas/issues/25211#issuecomment-462694916

@vladserkoff : Don't worry about the Codecov.  Windows failure looks weird though.
this needs a test, pls merge master
@jreback, I've added a test. Casting boolean to integer as `pandas.core.arrays.integer.integer_array(array, dtype=None)`(note that dtype is set to None) will still fail, but it seems it's usually called with a dtype specified.
``pandas.core.arrays.integer.integer_array(array, dtype=None)`` should work as well, can you show what array is when it *doesn't* work? the dtype is optional (though maybe an explicit passed ``None`` is not well handled)
Regarding `pandas.core.arrays.integer.integer_array(array, dtype=None)` fails for boolean array without nans because this function just calls `coerce_to_array`. I don't handle cases when dtype is None.
https://github.com/pandas-dev/pandas/blob/d0acd206a59279f9cea9615bfc8b9279273d495c/pandas/core/arrays/integer.py#L190-L191
If I change the condition to `elif is_bool_dtype(values) and (is_integer_dtype(dtype) or dtype is None)` then all the test in `TestComparisonOps.test_compare_[array|scalar]` fail with following errors:
```python-traceback
self = <pandas.tests.extension.test_integer.TestComparisonOps object at 0x1c1c3687b8>
s = 0       1
1       2
2       3
3       4
4       5
5       6
6       7
7       8
8     NaN
9      10
10     11
11     1...91
91     92
92     93
93     94
94     95
95     96
96     97
97    NaN
98     99
99    100
Length: 100, dtype: UInt64
op = <built-in function gt>
other = 0     1
1     1
2     1
3     1
4     1
5     1
6     1
7     1
8     1
9     1
10    1
11    1
12    1
13    1
14    ...    1
89    1
90    1
91    1
92    1
93    1
94    1
95    1
96    1
97    1
98    1
99    1
Length: 100, dtype: int64
op_name = '__gt__', exc = None

    def _check_op(self, s, op, other, op_name, exc=NotImplementedError):
        if exc is None:
            result = op(s, other)
            expected = s.combine(other, op)
>           self.assert_series_equal(result, expected)
E           AssertionError: Attributes are different
E
E           Attribute "dtype" are different
E           [left]:  bool
E           [right]: Int64
```
yeah i think passing an explict ``dtype=None`` for just do inference, which I think we have a path for. If you want to take a stab here great, otherwise can do a followup (but create an issue in that case).
@jreback, not sure what you mean by taking a stab here 🙂 
@vladserkoff in fixing passing ``dtype=None``
@jreback, I've tried to fix this but with no success. The issue lies here and I'm not sure how to handle this:
The tests that fail when trying to handle dtype=None is comparisonOps: https://github.com/pandas-dev/pandas/blob/7c24ea824d6089adf7bf662cc7b2eab79bea7a21/pandas/tests/arrays/test_integer.py#L362-L365
`_compare_other` in turn calls https://github.com/pandas-dev/pandas/blob/7c24ea824d6089adf7bf662cc7b2eab79bea7a21/pandas/tests/extension/base/ops.py#L29-L33
while `s.combine` is a method in which https://github.com/pandas-dev/pandas/blob/7c24ea824d6089adf7bf662cc7b2eab79bea7a21/pandas/core/series.py#L2644-L2656
method `_from_sequence` internally calls `integer_array` without specifying dtype. When passed boolean array it just fails to convert it to integer, and everything is okay as expected.
Cant understand why call `integer_array` on an array with no intention to actually cast it to integer.
@vladserkoff ok, can you create an issue for this and we can deal with it separately.
see my comment
Hello @vladserkoff! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-05-14 10:40:34 UTC
ping when updated
@jreback, pls check out my latest commit, the main idea here in `pandas.core.series` is to pass the resulting dtype to the EA constrictor. If you think that the logic here is ok, then I'll try to find a better way to get the expected dtype.
can you merge master
can you merge master and i'll have a look
Hey, @jreback. Before I proceed, let me clear one thing a bit. The main difficulty here is handling the case when `integer_array(...)` is called with `dtype=None`. It is difficult because for some cases it should fail when `dtype=None` like in this place when `op` is some comparison operation:
https://github.com/pandas-dev/pandas/blob/f2e2eac1e8d3c2adfb72bcc792a2e34d174e5a7e/pandas/tests/extension/base/ops.py#L31-L33
If `s` is an Integer EA, this call to `s.combine` will subsequently call `integer_array` and it should not convert boolean array to ints.
What I propose is that we don't convert to int if it is not explicitly stated by specifying dtype, as I first suggested. This way we address this specific issue I was trying to resolve, meaning handling the cases like this one `pd.Series([True, False, ...], dtype=pd.Int64())`, AND not breaking any other cases.
Does it sound okay to you?
Otherwise I don't seem to be qualified enough to fix this in a general way :)
@vladserkoff ok let's try what you suggest above and see
also pls merge master
@jreback, done. Checks are ok.
thanks @vladserkoff 

happy to have an issue about ``integer_array(..., dtype=None)`` (and PR too)!