Repository: pandas-dev/pandas
Pull request: BUG: Fix (22477) dtype=str converts NaN to 'n'
- [x] closes #22477
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Now:
```
>>> import pandas as pd
>>> result = pd.Series(index=range(5), dtype=str)
>>> result
0    NaN
1    NaN
2    NaN
3    NaN
4    NaN
dtype: object
```

This is implemented by adding a check so if the `dtype` is `str` is will create an empty array type object and then pass the values. Two tests have been implemented:

- test for an empty series. To check that it fills the series with NaN and not with 'n'. For example now:
- test for cases that no string values are given.

Problem statement:
BUG: dtype=str in 0.23.0 converts NaN to 'n'
```python
>>> import pandas as pd
>>> series = pd.Series(index=range(5), dtype=str)
>>> series
0    n
1    n
2    n
3    n
4    n
dtype: object
>>> pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.0.final.0
python-bits: 64
OS: Darwin
OS-release: 14.3.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.0
pytest: 3.0.5
pip: 9.0.1
setuptools: 40.0.0
Cython: 0.25.2
numpy: 1.11.3
scipy: 0.18.1
pyarrow: None
xarray: None
IPython: 5.1.0
sphinx: 1.5.1
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2016.10
blosc: None
bottleneck: 1.2.0
tables: 3.3.0
numexpr: 2.6.1
feather: None
matplotlib: 2.0.0
openpyxl: 2.4.1
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.6
lxml: 3.7.2
bs4: 4.5.3
html5lib: None
sqlalchemy: 1.1.5
pymysql: None
psycopg2: None
jinja2: 2.9.4
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
```

Before I upgraded pandas to 0.23.0, pd.Series(dtype=str, index=range(5)) gave me a series filled with NaN values. However, with this recent upgrade, dtype=str converts null values to lower case letter ’n’. I’m appending the normal null value series output below as a comparison. 

```python
>>> import pandas as pd
>>> series = pd.Series(index=range(5), dtype=str)
>>> series
0    NaN
1    NaN
2    NaN
3    NaN
4    NaN
dtype: object
>>> pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.0.final.0
python-bits: 64
OS: Darwin
OS-release: 14.3.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.22.0
pytest: 3.0.5
pip: 9.0.1
setuptools: 40.0.0
Cython: 0.25.2
numpy: 1.11.3
scipy: 0.18.1
pyarrow: None
xarray: None
IPython: 5.1.0
sphinx: 1.5.1
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2016.10
blosc: None
bottleneck: 1.2.0
tables: 3.3.0
numexpr: 2.6.1
feather: None
matplotlib: 2.0.0
openpyxl: 2.4.1
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.6
lxml: 3.7.2
bs4: 4.5.3
html5lib: None
sqlalchemy: 1.1.5
pymysql: None
psycopg2: None
jinja2: 2.9.4
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

```

Hints from issue discussion:
Thanks for the report! That certainly seems like a regression.

I seem to remember we had some issues about it last year, but can't directly find it.

cc @toobaz you worked a bit on the series constructor and related things, not sure if this is related
Hello, is this related or a different issue?

```
x = np.array(['x','y'])
pd.Series(x).str.decode(encoding='UTF-8',errors='strict')
0   NaN
1   NaN
dtype: float64
```

I think this prevents me from loading HDF-Files that were written with earlier pandas versions. Accessing certain HDF-nodes I  get an ValueError leading back to this line in pytables.py:

[data = Series(data).str.decode(encoding, errors=errors).values](https://github.com/pandas-dev/pandas/blob/55d176d1cd624f8921379c0ceed06fb8e00625b9/pandas/io/pytables.py#L4698)

Its raising because the original "data" (numpy-ndarray-object) was something like the above ``x`` but Series.str.decode makes it all NaNs.
git bisect reveals this as the first bad commit: c8fcfcb6035d2bd76e4e83e9eabf4cfd86f74377

The bisect script did just this

```
import pandas
assert pandas.Series(index=range(5), dtype=str).isnull().iloc[0] == True
```
@abrakababra that looks different.

@kokes thanks (and sorry). I think the issue is that `dtype` should be sanitized to be a dtype pandas expects by the time we get [here](https://github.com/pandas-dev/pandas/pull/20865/files#diff-150fd3c5a732ae915ec47bc54a933c41R316). Right now it's `<U'` and it should just be object.
> cc @toobaz you worked a bit on the series constructor and related things, not sure if this is related

Maybe... I'm quite busy until mid September, but for the moment I think this is the issue you were looking at: #9428 . It is extremely similar, so the fix might be too.
Hey everyone!
I am currently at a sprint at EuroScipy 2017. I worked with @jorisvandenbossche on this and will open a `PR` to close the issue :+1:

Commit message: style tweak

Patch to review:
diff --git a/pandas/core/dtypes/cast.py b/pandas/core/dtypes/cast.py
index c7c6f89eb13a4..4b14529914294 100644
--- a/pandas/core/dtypes/cast.py
+++ b/pandas/core/dtypes/cast.py
@@ -6,7 +6,7 @@
 
 from pandas._libs import lib, tslib, tslibs
 from pandas._libs.tslibs import OutOfBoundsDatetime, Period, iNaT
-from pandas.compat import PY3, string_types, text_type
+from pandas.compat import PY3, string_types, text_type, to_str
 
 from .common import (
     _INT64_DTYPE, _NS_DTYPE, _POSSIBLY_CAST_DTYPES, _TD_DTYPE, _string_dtypes,
@@ -1217,11 +1217,14 @@ def construct_1d_arraylike_from_scalar(value, length, dtype):
             dtype = dtype.dtype
 
         # coerce if we have nan for an integer dtype
-        # GH 22858: only cast to float if an index
-        # (passed here as length) is specified
-        if length and is_integer_dtype(dtype) and isna(value):
-            dtype = np.float64
-        subarr = np.empty(length, dtype=dtype)
+        if is_integer_dtype(dtype) and isna(value):
+            dtype = np.dtype('float64')
+        if isinstance(dtype, np.dtype) and dtype.kind in ("U", "S"):
+            subarr = np.empty(length, dtype=object)
+            if not isna(value):
+                value = to_str(value)
+        else:
+            subarr = np.empty(length, dtype=dtype)
         subarr.fill(value)
 
     return subarr
diff --git a/pandas/core/dtypes/common.py b/pandas/core/dtypes/common.py
index 94e9b72b001b1..2756753855071 100644
--- a/pandas/core/dtypes/common.py
+++ b/pandas/core/dtypes/common.py
@@ -392,7 +392,7 @@ def is_datetime64_dtype(arr_or_dtype):
         return False
     try:
         tipo = _get_dtype_type(arr_or_dtype)
-    except TypeError:
+    except (TypeError, UnicodeEncodeError) as e:
         return False
     return issubclass(tipo, np.datetime64)
 
diff --git a/pandas/tests/series/test_constructors.py b/pandas/tests/series/test_constructors.py
index ce0cf0d5c089e..6eb6f63de40da 100644
--- a/pandas/tests/series/test_constructors.py
+++ b/pandas/tests/series/test_constructors.py
@@ -134,6 +134,23 @@ def test_constructor_no_data_index_order(self):
         result = pd.Series(index=['b', 'a', 'c'])
         assert result.index.tolist() == ['b', 'a', 'c']
 
+    def test_constructor_no_data_string_type(self):
+        # GH 22477
+        result = pd.Series(index=[1], dtype=str)
+        assert np.isnan(result.iloc[0])
+
+    @pytest.mark.parametrize('item', ['13'])
+    def test_constructor_single_element_string_type(self, item):
+        # GH 22477
+        result = pd.Series(int(item), index=[1], dtype=str)
+        assert result.iloc[0] == item
+
+    @pytest.mark.parametrize('item', ['entry', 'ѐ'])
+    def test_constructor_string_element_string_type(self, item):
+        # GH 22477
+        result = pd.Series(item, index=[1], dtype=str)
+        assert result.iloc[0] == item
+
     def test_constructor_dtype_str_na_values(self, string_dtype):
         # https://github.com/pandas-dev/pandas/issues/21083
         ser = Series(['x', None], dtype=string_dtype)


Code review comments:
diff_hunk: @@ -134,6 +134,23 @@ def test_constructor_no_data_index_order(self):
         result = pd.Series(index=['b', 'a', 'c'])
         assert result.index.tolist() == ['b', 'a', 'c']
 
+    def test_constructor_no_data_string_type(self):
+        # GH 22477
+        result = pd.Series(index=[1], dtype=str)
+        assert np.isnan(result.iloc[0])
+
+    @pytest.mark.parametrize('item', ['13'])
path: pandas/tests/series/test_constructors.py
text: @user1:
you don't need to parameterize this test (only 1 case), and you need to change the name as the next test overwrites it.

diff_hunk: @@ -1221,8 +1221,13 @@ def construct_1d_arraylike_from_scalar(value, length, dtype):
 
         # coerce if we have nan for an integer dtype
         if is_integer_dtype(dtype) and isna(value):
-            dtype = np.float64
-        subarr = np.empty(length, dtype=dtype)
+            dtype = np.dtype('float64')
+        if isinstance(dtype, np.dtype) and dtype.kind in ("U", "S"):
path: pandas/core/dtypes/cast.py
text: @user1:
this can also be an elif