Repository: pandas-dev/pandas

Issue:
str.split on np.nan gives np.nan in one column but None in another column


```python
import pandas as pd
import numpy as np

s = pd.Series(['19HT|C2', np.nan, '20ZT|C1'])
print(s)
```

    0    19HT|C2
    1        NaN
    2    20ZT|C1
    dtype: object



```python
s_split = s.str.split('|', expand=True)
print(s_split)
```

          0     1
    0  19HT    C2
    1   NaN  None
    2  20ZT    C1



```python
print(s_split.dtypes)
```

    0    object
    1    object
    dtype: object



```python
print(type(s_split.loc[1,0]))
```




    float




```python
print(type(s_split.loc[1,1]))
```




    NoneType




#### Problem description

When `np.nan` gets split, it becomes `np.nan` (of type `float`) in the first column but `None` (of type `NoneType`) in the second column. I'd consider this unexpected behavior. How come splitting a value of one type results in two values of different types?

#### Expected Output

          0     1
    0  19HT    C2
    1   NaN   NaN
    2  20ZT    C1

Either `np.nan` or `None` in both columns, but not a mix of both. I'd say `np.nan` makes most sense, since that's the original value of the row.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.0.final.0
python-bits: 64
OS: Linux
OS-release: 4.10.0-40-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.21.0
pytest: 3.0.5
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.13.1
scipy: 0.19.1
pyarrow: 0.7.1
xarray: None
IPython: 5.1.0
sphinx: 1.5.1
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2016.10
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.2
feather: 0.4.0
matplotlib: 2.0.2
openpyxl: 2.4.1
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.6
lxml: 4.1.1
bs4: 4.5.3
html5lib: 0.9999999
sqlalchemy: 1.1.5
pymysql: None
psycopg2: None
jinja2: 2.9.4
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
</details>



Discussion:
Just to be clear the issue here is how ``expand=True`` handles ``NaN`` values. You'd still mix ``str`` and ``None`` instances using expansion if the split character isn't in one of the records. For example, 

    s = pd.Series(['19HT|C2', np.nan, '20ZT|C1', 'foo'])
    s.str.split("|", expand=True)

yields

          0     1
    0  19HT    C2
    1   NaN  None
    2  20ZT    C1
    3   foo  None

The str_split docstring is a little ambiguous because it says it should propagate NaN values, but I think that references the value returned irrespective of the expansion mechanism. Will leave it to others here to comment as to whether or not we think this is a bug with how expansion works, or if the docstring should be modified.

https://github.com/pandas-dev/pandas/blob/97fea48945163ec27349918c14ec9bdca2335460/pandas/core/strings.py#L1005
this is a bug; we want to use np.nan as the missing value indicator 

Resolving patch:
diff --git a/pandas/core/strings.py b/pandas/core/strings.py
--- a/pandas/core/strings.py
+++ b/pandas/core/strings.py
@@ -1423,6 +1423,10 @@ def cons_row(x):
                     return [x]
 
             result = [cons_row(x) for x in result]
+            if result:
+                # propogate nan values to match longest sequence (GH 18450)
+                max_len = max(len(x) for x in result)
+                result = [x * max_len if x[0] is np.nan else x for x in result]
 
         if not isinstance(expand, bool):
             raise ValueError("expand must be True or False")


Test patch:
diff --git a/pandas/tests/test_strings.py b/pandas/tests/test_strings.py
--- a/pandas/tests/test_strings.py
+++ b/pandas/tests/test_strings.py
@@ -2086,6 +2086,18 @@ def test_rsplit_to_multiindex_expand(self):
         tm.assert_index_equal(result, exp)
         assert result.nlevels == 2
 
+    def test_split_nan_expand(self):
+        # gh-18450
+        s = Series(["foo,bar,baz", NA])
+        result = s.str.split(",", expand=True)
+        exp = DataFrame([["foo", "bar", "baz"], [NA, NA, NA]])
+        tm.assert_frame_equal(result, exp)
+
+        # check that these are actually np.nan and not None
+        # TODO see GH 18463
+        # tm.assert_frame_equal does not differentiate
+        assert all(np.isnan(x) for x in result.iloc[1])
+
     def test_split_with_name(self):
         # GH 12617