EA: BoolArray
State: closed

this builds on #22226 and #22238 by @WillAyd  and the ``ArrowBoolArray`` by @TomAugspurger to create a first class bool array that is backed by a numpy EA array or a pyarrow array if available. Uses this MaskArray as the mask inside IntegerArray. The is a POC to see how feasible this is generally. Note that I did this with ``bitarray`` as well, but this is more general purpose / forward looking.

Some decent memory savings as the bits are packed by arrow.

PR:
```
In [2]: s = pd.Series(np.arange(100000), dtype='Int64')                                                                                                                                                                                 

In [3]: s.nbytes                                                                                                                                                                                                                        
Out[3]: 812500

In [4]: s._values._mask.nbytes                                                                                                                                                                                                          
Out[4]: 12500
```

0.24.1 / master
```
In [2]: s = pd.Series(np.arange(100000), dtype='Int64')                                                                                                                                                                                 

In [3]: s.nbytes                                                                                                                                                                                                                        
Out[3]: 900000

In [4]: s._values._mask.nbytes                                                                                                                                                                                                          
Out[4]: 100000
```

TODO:
- [x] ``pyarrow`` uses ``pandas`` as a dependency, so we have a hoop to jump to actually have this import in a sane way
- [ ] we need a ``pd.bool_`` dtype to avoid exposing the pyarrow dtypes directly (mainly because we have this as an optional dep and now BoolArray can either be ``np.bool_`` or ``arrow_bool`` so need a single type to expose to end users; this is not hard just not done).
- [x] The ``BoolArray`` allows NaN's when its an ArrowBoolArray but not for a NumpyBoolArray; I avoided this entirely but just using this as a masking array (so its guaranteed as only bool types), but eventually want to allow this
- [ ] A number of missing methods on ArrowBoolArray that we should push upstream (implemented now by simply coercing and using numpy)
- [x] ~Should just rename ``MaskArray`` to ``BoolArray`` I think.~ This is now internal only, ``BoolArray`` will use this.

cc @pandas-dev/pandas-core 

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Hello @jreback! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:













































There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-04-21 17:30:58 UTC
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25415?src=pr&el=h1) Report
> Merging [#25415](https://codecov.io/gh/pandas-dev/pandas/pull/25415?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/b6731886db957b3e667449f67b6c95a638f2ac81?src=pr&el=desc) will **decrease** coverage by `50.08%`.
> The diff coverage is `45.73%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25415/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25415?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #25415       +/-   ##
===========================================
- Coverage   91.73%   41.64%   -50.09%     
===========================================
  Files         173      175        +2     
  Lines       52845    53041      +196     
===========================================
- Hits        48479    22091    -26388     
- Misses       4366    30950    +26584
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `?` | |
| #single | `41.64% <45.73%> (-0.09%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25415?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/25415/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tbW9uLnB5) | `67.2% <100%> (-31.19%)` | :arrow_down: |
| [pandas/core/arrays/integer.py](https://codecov.io/gh/pandas-dev/pandas/pull/25415/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2ludGVnZXIucHk=) | `41.33% <100%> (-54.99%)` | :arrow_down: |
| [pandas/core/dtypes/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/25415/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2dlbmVyaWMucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [pandas/core/arrays/bool.py](https://codecov.io/gh/pandas-dev/pandas/pull/25415/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2Jvb2wucHk=) | `10.25% <15.9%> (ø)` | |
| [pandas/core/arrays/\_mask.py](https://codecov.io/gh/pandas-dev/pandas/pull/25415/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL19tYXNrLnB5) | `57.33% <57.33%> (ø)` | |
| [pandas/core/indexing.py](https://codecov.io/gh/pandas-dev/pandas/pull/25415/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhpbmcucHk=) | `52.34% <66.66%> (-39.95%)` | :arrow_down: |
| [pandas/io/formats/latex.py](https://codecov.io/gh/pandas-dev/pandas/pull/25415/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvbGF0ZXgucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/core/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/25415/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY2F0ZWdvcmljYWwucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/25415/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `0% <0%> (-100%)` | :arrow_down: |
| [pandas/tseries/plotting.py](https://codecov.io/gh/pandas-dev/pandas/pull/25415/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvcGxvdHRpbmcucHk=) | `0% <0%> (-100%)` | :arrow_down: |
| ... and [138 more](https://codecov.io/gh/pandas-dev/pandas/pull/25415/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25415?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25415?src=pr&el=footer). Last update [b673188...d539290](https://codecov.io/gh/pandas-dev/pandas/pull/25415?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25415?src=pr&el=h1) Report
> Merging [#25415](https://codecov.io/gh/pandas-dev/pandas/pull/25415?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/61edc765ee3d12a2341ee667d6aee3d96be3db72?src=pr&el=desc) will **decrease** coverage by `51.26%`.
> The diff coverage is `44.39%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25415/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25415?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #25415       +/-   ##
===========================================
- Coverage   91.98%   40.72%   -51.27%     
===========================================
  Files         175      179        +4     
  Lines       52371    52764      +393     
===========================================
- Hits        48175    21488    -26687     
- Misses       4196    31276    +27080
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `?` | |
| #single | `40.72% <44.39%> (-0.14%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25415?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/groupby/groupby.py](https://codecov.io/gh/pandas-dev/pandas/pull/25415/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9ncm91cGJ5LnB5) | `23.9% <0%> (-73.33%)` | :arrow_down: |
| [pandas/core/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/25415/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tbW9uLnB5) | `67.74% <100%> (-30.65%)` | :arrow_down: |
| [pandas/core/nanops.py](https://codecov.io/gh/pandas-dev/pandas/pull/25415/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvbmFub3BzLnB5) | `31.52% <100%> (-62.31%)` | :arrow_down: |
| [pandas/core/dtypes/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/25415/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2dlbmVyaWMucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [pandas/core/internals/blocks.py](https://codecov.io/gh/pandas-dev/pandas/pull/25415/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2Jsb2Nrcy5weQ==) | `51.06% <16.66%> (-43.03%)` | :arrow_down: |
| [pandas/core/arrays/mask/\_pyarrow.py](https://codecov.io/gh/pandas-dev/pandas/pull/25415/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL21hc2svX3B5YXJyb3cucHk=) | `21.15% <21.15%> (ø)` | |
| [pandas/core/arrays/mask/\_base.py](https://codecov.io/gh/pandas-dev/pandas/pull/25415/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL21hc2svX2Jhc2UucHk=) | `44.18% <44.18%> (ø)` | |
| [pandas/core/arrays/integer.py](https://codecov.io/gh/pandas-dev/pandas/pull/25415/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2ludGVnZXIucHk=) | `43.93% <50%> (-52.42%)` | :arrow_down: |
| [pandas/core/arrays/mask/\_numpy.py](https://codecov.io/gh/pandas-dev/pandas/pull/25415/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL21hc2svX251bXB5LnB5) | `55.81% <55.81%> (ø)` | |
| [pandas/core/indexing.py](https://codecov.io/gh/pandas-dev/pandas/pull/25415/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhpbmcucHk=) | `52.25% <66.66%> (-38.64%)` | :arrow_down: |
| ... and [150 more](https://codecov.io/gh/pandas-dev/pandas/pull/25415/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25415?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25415?src=pr&el=footer). Last update [61edc76...d030c57](https://codecov.io/gh/pandas-dev/pandas/pull/25415?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

so this could be simplfiied if we make pyarrow a hard dependency :-D, then we don't need the numpy compat layer.
fixes a few buggies.
Do we want to do this for 0.25?
this really needs pyarrow as a hard dep 
but that needs some discussion
Ah yeah I forgot about that complication. OK with pushing for now then.

On Tue, May 21, 2019 at 12:03 PM Jeff Reback <notifications@github.com>
wrote:

> this really needs pyarrow as a hard dep
> but that needs some discussion
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/25415?email_source=notifications&email_token=AAKAOIUFEY7DGZCDEPEGYG3PWQTMPA5CNFSM4GZIJUCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV4RRBQ#issuecomment-494475398>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIRDO263BNLMTFYTVBDPWQTMPANCNFSM4GZIJUCA>
> .
>

It's also not fully clear to me what the goal / scope is of this PR. It seems to introduce a MaskArray / MaskDtype, but is that the way we want to expose this to the user? (instead of BooleanArray?) Or is it not meant for general usage, but only for the mask of the IntegerArray? But then why make this into a full ExtensionArray? 
so this PR serves a couple of cases:

- formalize using a MaskType backed by pyarrow / numpy boolean (non-nullable) types (based off of @TomAugspurger original example); basically seeing what indexing things pop up when we can return a mask type that is not a numpy boolean array (it works pretty reasonably actually)
- use the MaskType as a backing for the mask in Integer implementation to save memory (1 bit per vs 1 byte now); this also works reasonably well
- see if I could do the above optionally using ``pyarrow``. The implementation does work, but I had to jump thru quite a few hoops. I think we should simply adopt ``pyarrow`` as a formal dependency and this (and other things) become quite a bit easier to do and we can then start using pyarrow kernels and other machinery (hash tables perhaps) in the pandas internals.

cc @pandas-dev/pandas-core 
Thoughts on when we would want to take on pyarrow as a required dependency?
After 1.0?

On Thu, May 23, 2019 at 7:29 AM Jeff Reback <notifications@github.com>
wrote:

> so this PR serves a couple of cases:
>
>    - formalize using a MaskType backed by pyarrow / numpy boolean
>    (non-nullable) types (based off of @TomAugspurger
>    <https://github.com/TomAugspurger> original example); basically seeing
>    what indexing things pop up when we can return a mask type that is not a
>    numpy boolean array (it works pretty reasonably actually)
>    - use the MaskType as a backing for the mask in Integer implementation
>    to save memory (1 bit per vs 1 byte now); this also works reasonably well
>    - see if I could do the above optionally using pyarrow. The
>    implementation does work, but I had to jump thru quite a few hoops. I think
>    we should simply adopt pyarrow as a formal dependency and this (and
>    other things) become quite a bit easier to do and we can then start using
>    pyarrow kernels and other machinery (hash tables perhaps) in the pandas
>    internals.
>
> cc @pandas-dev/pandas-core
> <https://github.com/orgs/pandas-dev/teams/pandas-core>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/25415?email_source=notifications&email_token=AAKAOIX2Q64HEFKWHGAYDXDPW2E2XA5CNFSM4GZIJUCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWCB6OY#issuecomment-495198011>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOITLIHXYYEFQVBBX45TPW2E2XANCNFSM4GZIJUCA>
> .
>

closing for now.