Repository: pandas-dev/pandas
Pull request: Support hard-masked numpy arrays
- [x] closes #24574
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

For the whatsnew entry, is this reasonable?
```rst
- :func:`DataFrame` and :func:`Series` now properly handle 
numpy masked arrays with hardened masks. Previously, constructing 
a DataFrame or Series from a masked array with a hard mask would 
create a pandas object containing the underlying value, rather than the
expected NaN. (:issue:`24574`)
```

Problem statement:
Masked arrays with hard masks don't read as missing
#### Code Sample, a copy-pastable example if possible

```python
arr = np.ma.array([1.0, 2.0], mask=[True, False], dtype='float64')
arr_hard = np.ma.array([1.0, 2.0], mask=[True, False], dtype='float64', hard_mask=True)
print(pd.DataFrame(arr))
print(pd.DataFrame(arr_hard))
```

Normal masked array:

| | 0 |
|:---|---:|
|0 | **NaN**|
|1 |2.0|

Hard masked array:

| | 0 |
|:---|---:|
|0 | **1.0**|
|1 |2.0|

#### Problem description

I expected the output for a "hard" masked array to be the same as a regular masked array – I thought both would give NaN for the masked value. If the two cases behaved the same, it would be less surprising.

The issue is the same for `Series` as well as `DataFrame`, as well as other types of float.

#### Expected Output


Normal masked array: no change

Hard masked array:

| | 0 |
|:---|---:|
|0 | **NaN**|
|1 |2.0|

#### Output of ``pd.show_versions()``

<details>


INSTALLED VERSIONS
------------------
commit: None
python: 3.7.1.final.0
python-bits: 64
OS: Linux
OS-release: 4.18.0-13-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.4
pytest: None
pip: 18.1
setuptools: 40.6.3
Cython: None
numpy: 1.15.4
scipy: None
pyarrow: None
xarray: None
IPython: 7.2.0
sphinx: None
patsy: None
dateutil: 2.7.5
pytz: 2018.7
blosc: None
bottleneck: None
tables: 3.4.4
numexpr: 2.6.8
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>

Hints from issue discussion:
we have minimal support for masked array conversions

welcome to have a PR to fix / test though

Commit message: Support hard-masked numpy arrays

Patch to review:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
index d6aa3117570af..dda9fffacd91c 100644
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -400,6 +400,7 @@ def __init__(self, data=None, index=None, columns=None, dtype=None,
                 mask = ma.getmaskarray(data)
                 if mask.any():
                     data, fill_value = maybe_upcast(data, copy=True)
+                    data.soften_mask() # set hardmask False if it was True
                     data[mask] = fill_value
                 else:
                     data = data.copy()
diff --git a/pandas/core/internals/construction.py b/pandas/core/internals/construction.py
index b3c893c7d84be..906d2861924bd 100644
--- a/pandas/core/internals/construction.py
+++ b/pandas/core/internals/construction.py
@@ -547,6 +547,7 @@ def sanitize_array(data, index, dtype=None, copy=False,
         mask = ma.getmaskarray(data)
         if mask.any():
             data, fill_value = maybe_upcast(data, copy=True)
+            data.soften_mask() # set hardmask False if it was True
             data[mask] = fill_value
         else:
             data = data.copy()
diff --git a/pandas/tests/frame/test_constructors.py b/pandas/tests/frame/test_constructors.py
index 8a5ec1a16d1df..0e795e99629e7 100644
--- a/pandas/tests/frame/test_constructors.py
+++ b/pandas/tests/frame/test_constructors.py
@@ -757,6 +757,20 @@ def test_constructor_maskedarray_nonfloat(self):
         assert frame['A'][1] is True
         assert frame['C'][2] is False
 
+        # Check hardened masks
+        mat_hard = ma.masked_all((2, 3), dtype=float).harden_mask()
+        frame = DataFrame(mat_hard, columns=['A', 'B', 'C'], index=[1, 2])
+        assert len(frame.index) == 2
+        assert len(frame.columns) == 3
+        assert np.all(~np.asarray(frame == frame))
+        # Check case where mask is hard but no data are masked
+        mat_hard = ma.ones((2,3), dtype=float).harden_mask()
+        frame = DataFrame(mat_hard, columns=['A', 'B', 'C'], index=[1, 2])
+        assert len(frame.index) == 2
+        assert len(frame.columns) == 3
+        assert np.all(np.asarray(frame == 1.0))
+
+
     def test_constructor_mrecarray(self):
         # Ensure mrecarray produces frame identical to dict of masked arrays
         # from GH3479
diff --git a/pandas/tests/series/test_constructors.py b/pandas/tests/series/test_constructors.py
index f5a445e2cca9a..5bd10fa7989ed 100644
--- a/pandas/tests/series/test_constructors.py
+++ b/pandas/tests/series/test_constructors.py
@@ -386,6 +386,11 @@ def test_constructor_maskedarray(self):
         expected = Series([nan, nan, nan])
         assert_series_equal(result, expected)
 
+        data_hard = ma.copy(data).harden_mask()
+        result = Series(data_hard)
+        expected = Series([nan, nan, nan])
+        assert_series_equal(result, expected)
+
         data[0] = 0.0
         data[2] = 2.0
         index = ['a', 'b', 'c']


Code review comments:
diff_hunk: @@ -757,6 +757,20 @@ def test_constructor_maskedarray_nonfloat(self):
         assert frame['A'][1] is True
         assert frame['C'][2] is False
 
+        # Check hardened masks
+        mat_hard = ma.masked_all((2, 3), dtype=float).harden_mask()
+        frame = DataFrame(mat_hard, columns=['A', 'B', 'C'], index=[1, 2])
path: pandas/tests/frame/test_constructors.py
text: @user1:
use the format
```
result = 
expected = pd.DataFrame(....)
tm.assert_frame_equal(result, expected)
```
construct the expected exactly