BUG: Fix make_sparse mask generation
State: closed

- [x] closes #xxxx
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This is the part of https://github.com/pandas-dev/pandas/pull/17386.
The mask generation sequence in `make_sparse` treated two data, which have same bits, as same if array is `dtype=object`.
This bug made `SparseArray([False, 0, 100.0, 0.0], fill_value=False, dtype=np.object)` is evaluated as `[False, False, 100.0, False]`

Labels:
name: Bug
name: Sparse
description: Sparse Data Type

Comments:
Hello @Licht-T! Thanks for updating the PR.

 - In the file [`asv_bench/benchmarks/sparse.py`](https://github.com/pandas-dev/pandas/blob/2c6bca40cd17b46ee95d2c26ca3efe671506fd2d/asv_bench/benchmarks/sparse.py), following are the PEP8 issues :

> [Line 31:80](https://github.com/pandas-dev/pandas/blob/2c6bca40cd17b46ee95d2c26ca3efe671506fd2d/asv_bench/benchmarks/sparse.py#L31): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (119 > 79 characters)
> [Line 32:80](https://github.com/pandas-dev/pandas/blob/2c6bca40cd17b46ee95d2c26ca3efe671506fd2d/asv_bench/benchmarks/sparse.py#L32): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (117 > 79 characters)
> [Line 34:80](https://github.com/pandas-dev/pandas/blob/2c6bca40cd17b46ee95d2c26ca3efe671506fd2d/asv_bench/benchmarks/sparse.py#L34): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (128 > 79 characters)
> [Line 35:80](https://github.com/pandas-dev/pandas/blob/2c6bca40cd17b46ee95d2c26ca3efe671506fd2d/asv_bench/benchmarks/sparse.py#L35): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (126 > 79 characters)
> [Line 37:80](https://github.com/pandas-dev/pandas/blob/2c6bca40cd17b46ee95d2c26ca3efe671506fd2d/asv_bench/benchmarks/sparse.py#L37): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (123 > 79 characters)
> [Line 38:80](https://github.com/pandas-dev/pandas/blob/2c6bca40cd17b46ee95d2c26ca3efe671506fd2d/asv_bench/benchmarks/sparse.py#L38): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (121 > 79 characters)
> [Line 40:80](https://github.com/pandas-dev/pandas/blob/2c6bca40cd17b46ee95d2c26ca3efe671506fd2d/asv_bench/benchmarks/sparse.py#L40): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (122 > 79 characters)
> [Line 41:80](https://github.com/pandas-dev/pandas/blob/2c6bca40cd17b46ee95d2c26ca3efe671506fd2d/asv_bench/benchmarks/sparse.py#L41): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (120 > 79 characters)
> [Line 80:80](https://github.com/pandas-dev/pandas/blob/2c6bca40cd17b46ee95d2c26ca3efe671506fd2d/asv_bench/benchmarks/sparse.py#L80): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (80 > 79 characters)







##### Comment last updated on September 27, 2017 at 11:39 Hours UTC
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17574?src=pr&el=h1) Report
> Merging [#17574](https://codecov.io/gh/pandas-dev/pandas/pull/17574?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/37e23d03e1ba2298d9df05ded69028dfac0e823e?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17574/graphs/tree.svg?height=150&token=eZ4WkYLtcO&src=pr&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/17574?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17574      +/-   ##
==========================================
- Coverage   91.22%    91.2%   -0.02%     
==========================================
  Files         163      163              
  Lines       49625    49632       +7     
==========================================
- Hits        45270    45268       -2     
- Misses       4355     4364       +9
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.99% <100%> (ø)` | :arrow_up: |
| #single | `40.19% <25%> (-0.08%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17574?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/sparse/array.py](https://codecov.io/gh/pandas-dev/pandas/pull/17574?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3BhcnNlL2FycmF5LnB5) | `91.44% <100%> (+0.14%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17574?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17574?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.77% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17574?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17574?src=pr&el=footer). Last update [37e23d0...05bbd54](https://codecov.io/gh/pandas-dev/pandas/pull/17574?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17574?src=pr&el=h1) Report
> Merging [#17574](https://codecov.io/gh/pandas-dev/pandas/pull/17574?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/44747c8e91cee3599f96a4f154cb0323269b4ef1?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17574/graphs/tree.svg?src=pr&token=eZ4WkYLtcO&width=650&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/17574?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17574      +/-   ##
==========================================
+ Coverage   91.24%   91.25%   +<.01%     
==========================================
  Files         163      163              
  Lines       49762    49764       +2     
==========================================
+ Hits        45406    45412       +6     
+ Misses       4356     4352       -4
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.05% <100%> (+0.02%)` | :arrow_up: |
| #single | `40.34% <66.66%> (-0.07%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17574?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/sparse/array.py](https://codecov.io/gh/pandas-dev/pandas/pull/17574?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3BhcnNlL2FycmF5LnB5) | `91.58% <100%> (+0.28%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17574?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17574?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.73% <0%> (-0.1%)` | :arrow_down: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/17574?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `65.2% <0%> (+1.81%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17574?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17574?src=pr&el=footer). Last update [44747c8...2c6bca4](https://codecov.io/gh/pandas-dev/pandas/pull/17574?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@jreback Thanks for your review.
- Fixed the mask generation method
- Fixed the insufficient array comparison method in test case 
@jreback The method is now simplified.
@jreback Added asvs for `SparseArray` constructor, but it does not run due to the bug in `asv_bench/benchmarks/sparse.py`.
I created another PR: #17659.
what do benchmarks show?
@jreback Here is the results of tests.
This is pretty much slow. Should I reimplement by Cython?

```bash
[asv_bench] asv continuous -f 1.1 pandas-dev/master HEAD -b sparse.sparse_array_constructor                        21:39:21  ☁  fix-make_spase-mask-generation ☂ 𝝙 ⚡ ✭ 𝝙
· Creating environments
· Discovering benchmarks
·· Uninstalling from conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt.
·· Installing into conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt...
· Running 12 total benchmarks (2 commits * 1 environments * 6 benchmarks)
[  0.00%] · For pandas commit hash a48f9572:
[  0.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt....
[  0.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[  8.33%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_float64_10percent                                                            39.3±2ms
[ 16.67%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_float64_1percent                                                             7.67±1ms
[ 25.00%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_int64_10percent                                                            39.6±0.4ms
[ 33.33%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_int64_1percent                                                             6.69±0.3ms
[ 41.67%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_object_10percent                                                             556±20ms
[ 50.00%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_object_1percent                                                               486±6ms
[ 50.00%] · For pandas commit hash 5279a172:
[ 50.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt...................................................................
[ 50.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 58.33%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_float64_10percent                                                            37.3±3ms
[ 66.67%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_float64_1percent                                                           6.70±0.3ms
[ 75.00%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_int64_10percent                                                              38.0±1ms
[ 83.33%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_int64_1percent                                                             6.16±0.1ms
[ 91.67%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_object_10percent                                                             53.8±2ms
[100.00%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_object_1percent                                                            32.4±0.5ms
       before           after         ratio
     [5279a172]       [a48f9572]
+      32.4±0.5ms          486±6ms    15.01  sparse.sparse_array_constructor.time_sparse_array_constructor_object_1percent
+        53.8±2ms         556±20ms    10.34  sparse.sparse_array_constructor.time_sparse_array_constructor_object_10percent

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
```
@Licht-T yes that is slow. Sure if you can do that in cython would be great. its only a single case so shouldn't be much code. not sure how much speedup to get though as you can't really type anything.
also, would actually be ok with NOT allowing a not-nan fill_value for ``object`` dtypes as well. Slightly more inflexible but generally using sparse with object dtypes is not very performant anyhow.
@jreback Here is the Cython result.
```bash
[asv_bench] asv continuous -f 1.1 pandas-dev/master HEAD -b sparse.sparse_array_constructor                        23:04:10  ☁  fix-make_spase-mask-generation-new ☂ ⚡ ✭
· Creating environments
· Discovering benchmarks
·· Uninstalling from conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt.
·· Installing into conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt...
· Running 12 total benchmarks (2 commits * 1 environments * 6 benchmarks)
[  0.00%] · For pandas commit hash ae079f69:
[  0.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt..................................................................
[  0.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[  8.33%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_float64_10percent                                                          36.9±0.6ms
[ 16.67%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_float64_1percent                                                           6.91±0.4ms
[ 25.00%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_int64_10percent                                                              38.5±2ms
[ 33.33%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_int64_1percent                                                            5.88±0.05ms
[ 41.67%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_object_10percent                                                             85.9±2ms
[ 50.00%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_object_1percent                                                            52.7±0.6ms
```
This is rough reimplement and only checking whether this passes tests. It might be a little more faster.
@jreback Here is the final result. `NaN` and non-`NaN` are almost same.

```bash
[asv_bench] asv continuous -f 1.1 pandas-dev/master HEAD -b sparse.sparse_array_constructor                         0:35:38  ☁  fix-make_spase-mask-generation ☂ 𝝙 ⚡ ✭ 𝝙
· Creating environments
· Discovering benchmarks
·· Uninstalling from conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt.
·· Installing into conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt...
· Running 16 total benchmarks (2 commits * 1 environments * 8 benchmarks)
[  0.00%] · For pandas commit hash 35890726:
[  0.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt....
[  0.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[  6.25%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_float64_10percent                                                            40.7±1ms
[ 12.50%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_float64_1percent                                                           7.34±0.1ms
[ 18.75%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_int64_10percent                                                            37.3±0.6ms
[ 25.00%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_int64_1percent                                                            6.34±0.09ms
[ 31.25%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_object_nan_fill_value_10percent                                              78.2±1ms
[ 37.50%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_object_nan_fill_value_1percent                                             44.0±0.6ms
[ 43.75%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_object_non_nan_fill_value_10percent                                          87.0±2ms
[ 50.00%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_object_non_nan_fill_value_1percent                                         51.6±0.5ms
[ 50.00%] · For pandas commit hash 5279a172:
[ 50.00%] ·· Building for conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt....
[ 50.00%] ·· Benchmarking conda-py3.6-Cython-matplotlib-numexpr-numpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 56.25%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_float64_10percent                                                            37.2±2ms
[ 62.50%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_float64_1percent                                                           7.07±0.3ms
[ 68.75%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_int64_10percent                                                              46.8±3ms
[ 75.00%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_int64_1percent                                                            6.66±0.05ms
[ 81.25%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_object_nan_fill_value_10percent                                              82.8±1ms
[ 87.50%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_object_nan_fill_value_1percent                                               46.9±1ms
[ 93.75%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_object_non_nan_fill_value_10percent                                          55.0±2ms
[100.00%] ··· Running sparse.sparse_array_constructor.time_sparse_array_constructor_object_non_nan_fill_value_1percent                                         34.4±0.9ms
       before           after         ratio
     [5279a172]       [35890726]
+        55.0±2ms         87.0±2ms     1.58  sparse.sparse_array_constructor.time_sparse_array_constructor_object_non_nan_fill_value_10percent
+      34.4±0.9ms       51.6±0.5ms     1.50  sparse.sparse_array_constructor.time_sparse_array_constructor_object_non_nan_fill_value_1percent
-        46.8±3ms       37.3±0.6ms     0.80  sparse.sparse_array_constructor.time_sparse_array_constructor_int64_10percent

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
```
ok this looks good
going to give a final look in a bit
lgtm ping on green.
thanks @Licht-T keep em coming!