Repository: pandas-dev/pandas

Issue:
Masked arrays with hard masks don't read as missing
#### Code Sample, a copy-pastable example if possible

```python
arr = np.ma.array([1.0, 2.0], mask=[True, False], dtype='float64')
arr_hard = np.ma.array([1.0, 2.0], mask=[True, False], dtype='float64', hard_mask=True)
print(pd.DataFrame(arr))
print(pd.DataFrame(arr_hard))
```

Normal masked array:

| | 0 |
|:---|---:|
|0 | **NaN**|
|1 |2.0|

Hard masked array:

| | 0 |
|:---|---:|
|0 | **1.0**|
|1 |2.0|

#### Problem description

I expected the output for a "hard" masked array to be the same as a regular masked array – I thought both would give NaN for the masked value. If the two cases behaved the same, it would be less surprising.

The issue is the same for `Series` as well as `DataFrame`, as well as other types of float.

#### Expected Output


Normal masked array: no change

Hard masked array:

| | 0 |
|:---|---:|
|0 | **NaN**|
|1 |2.0|

#### Output of ``pd.show_versions()``

<details>


INSTALLED VERSIONS
------------------
commit: None
python: 3.7.1.final.0
python-bits: 64
OS: Linux
OS-release: 4.18.0-13-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.4
pytest: None
pip: 18.1
setuptools: 40.6.3
Cython: None
numpy: 1.15.4
scipy: None
pyarrow: None
xarray: None
IPython: 7.2.0
sphinx: None
patsy: None
dateutil: 2.7.5
pytz: 2018.7
blosc: None
bottleneck: None
tables: 3.4.4
numexpr: 2.6.8
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>



Discussion:
we have minimal support for masked array conversions

welcome to have a PR to fix / test though

Resolving patch:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -400,6 +400,7 @@ def __init__(self, data=None, index=None, columns=None, dtype=None,
                 mask = ma.getmaskarray(data)
                 if mask.any():
                     data, fill_value = maybe_upcast(data, copy=True)
+                    data.soften_mask()  # set hardmask False if it was True
                     data[mask] = fill_value
                 else:
                     data = data.copy()
diff --git a/pandas/core/internals/construction.py b/pandas/core/internals/construction.py
--- a/pandas/core/internals/construction.py
+++ b/pandas/core/internals/construction.py
@@ -547,6 +547,7 @@ def sanitize_array(data, index, dtype=None, copy=False,
         mask = ma.getmaskarray(data)
         if mask.any():
             data, fill_value = maybe_upcast(data, copy=True)
+            data.soften_mask()  # set hardmask False if it was True
             data[mask] = fill_value
         else:
             data = data.copy()


Test patch:
diff --git a/pandas/tests/frame/test_constructors.py b/pandas/tests/frame/test_constructors.py
--- a/pandas/tests/frame/test_constructors.py
+++ b/pandas/tests/frame/test_constructors.py
@@ -757,6 +757,28 @@ def test_constructor_maskedarray_nonfloat(self):
         assert frame['A'][1] is True
         assert frame['C'][2] is False
 
+    def test_constructor_maskedarray_hardened(self):
+        # Check numpy masked arrays with hard masks -- from GH24574
+        mat_hard = ma.masked_all((2, 2), dtype=float).harden_mask()
+        result = pd.DataFrame(mat_hard, columns=['A', 'B'], index=[1, 2])
+        expected = pd.DataFrame({
+            'A': [np.nan, np.nan],
+            'B': [np.nan, np.nan]},
+            columns=['A', 'B'],
+            index=[1, 2],
+            dtype=float)
+        tm.assert_frame_equal(result, expected)
+        # Check case where mask is hard but no data are masked
+        mat_hard = ma.ones((2, 2), dtype=float).harden_mask()
+        result = pd.DataFrame(mat_hard, columns=['A', 'B'], index=[1, 2])
+        expected = pd.DataFrame({
+            'A': [1.0, 1.0],
+            'B': [1.0, 1.0]},
+            columns=['A', 'B'],
+            index=[1, 2],
+            dtype=float)
+        tm.assert_frame_equal(result, expected)
+
     def test_constructor_mrecarray(self):
         # Ensure mrecarray produces frame identical to dict of masked arrays
         # from GH3479
diff --git a/pandas/tests/series/test_constructors.py b/pandas/tests/series/test_constructors.py
--- a/pandas/tests/series/test_constructors.py
+++ b/pandas/tests/series/test_constructors.py
@@ -451,6 +451,13 @@ def test_constructor_maskedarray(self):
                            datetime(2001, 1, 3)], index=index, dtype='M8[ns]')
         assert_series_equal(result, expected)
 
+    def test_constructor_maskedarray_hardened(self):
+        # Check numpy masked arrays with hard masks -- from GH24574
+        data = ma.masked_all((3, ), dtype=float).harden_mask()
+        result = pd.Series(data)
+        expected = pd.Series([nan, nan, nan])
+        tm.assert_series_equal(result, expected)
+
     def test_series_ctor_plus_datetimeindex(self):
         rng = date_range('20090415', '20090519', freq='B')
         data = {k: 1 for k in rng}