Repository: pandas-dev/pandas

Issue:
read_excel with dtype=str converts empty cells to the string 'nan'
#### Code Sample, a copy-pastable example if possible

```python
In [10]: df = pd.DataFrame({'a': ['x', 'y', '', 'z']})

In [11]: df
Out[11]: 
   a
0  x
1  y
2   
3  z

In [12]: df.to_excel('temp.xlsx')

In [13]: df = pd.read_excel('temp.xlsx', dtype=str)

In [14]: df
Out[14]: 
     a
0    x
1    y
2  nan
3    z

In [15]: df.loc[2, 'a']
Out[15]: 'nan'

In [16]: type(df.loc[2, 'a'])
Out[16]: str

```

#### Problem description
The empty string of the original dataframe becomes the string 'nan', instead of numpy.nan.


#### Expected Output

```python
In [14]: df
Out[14]: 
     a
0    x
1    y
2  NaN
3    z

In [15]: df.loc[2, 'a']
Out[15]: nan

In [16]: type(df.loc[2, 'a'])
Out[16]: float
```


#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.13.0-36-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: ca_ES.UTF-8
LOCALE: ca_ES.UTF-8

pandas: 0.22.0
pytest: None
pip: 9.0.1
setuptools: 38.5.2
Cython: 0.27.3
numpy: 1.14.1
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: None
patsy: 0.5.0
dateutil: 2.7.0
pytz: 2018.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.2.0
openpyxl: 2.5.1
xlrd: 1.1.0
xlwt: None
xlsxwriter: 0.7.3
lxml: None
bs4: None
html5lib: 1.0.1
sqlalchemy: 1.2.5
pymysql: 0.8.0
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>



Discussion:
Thanks for the report. Mind taking a look to see what's going on?
I've found the function that converts the `np.nan` to `'nan'`: `pandas.core.dtypes.astype_nansafe`.

`read_excel` uses `PythonParser`. This parser has a function called `self._cast_types`, which uses `astype_nansafe`.

https://github.com/pandas-dev/pandas/blob/master/pandas/io/parsers.py#L1570

An example of  `astype_nansafe`:
```python
In [4]: from pandas.core.dtypes.cast import astype_nansafe

In [5]: arr = np.array([np.nan, 'a', 'b'], object)

In [6]: arr
Out[6]: array([nan, 'a', 'b'], dtype=object)

In [7]: astype_nansafe(arr, str)
Out[7]: array(['nan', 'a', 'b'], dtype=object)
```

I don't know if it's the expected behavior or not.
Should we modify `astype_nansafe`? or `_cast_types`?.

I'm not sure how to fix the issue.
Is that what one should expect when using `dtype=str`?  Not specifying `dtype` produces the expected result.  The default value is `None` and dtype conversion is done by row.

```
In [16]: df = pd.read_excel('temp.xlsx')

In [17]: df
Out[17]:
     a
0    x
1    y
2  NaN
3    z

In [18]: df.loc[2, 'a']
Out[18]: nan

In [19]: type(df.loc[2, 'a'])
Out[19]: float

In [20]: type(df.loc[1, 'a'])
Out[20]: str
```

I think that the output for `dtype=str` is expected behavior.
`read_csv` returns what you would expect
```python
from io import StringIO
import pandas as pd

arr = pd.read_csv(StringIO('col1, col2, col3\n,a,b'), dtype=str)
```
it will give you
```
  col1  col2  col3
0  NaN     a     b
```
so, I guess, what you expect from `read_excel` seems to be consistent with this.

I guess, if this needs to be changed, the best would be to change `astype_unicode` in `pandas/_libs/lib.pyx`.
What do you think?
@cbertinato just saw your comment! Good point. 
@cbertinato, if I have a column with numbers, but I want to read them as strings I need to specify `dtype=str`, otherwise they will be read as float.
```python
In [1]: df = pd.DataFrame({'a': ['1', '2', '', '3']})

In [2]: df.to_excel('temp.xlsx')

In [3]: pd.read_excel('temp.xlsx', dtype=str)
Out[3]: 
     a
0   1
1   2
2  nan
3   3

In [5]: pd.read_excel('temp.xlsx')
Out[5]: 
     a
0  1.0
1  2.0
2  NaN
3  3.0

```

Moreover, as @nikoskaragiannakis says, it should be consistent with `read_csv`.
I missed the use case for numbers, and @nikoskaragiannakis's comment. It should probably be consistent with `read_csv`.
@cbertinato if the case is as @arnau126 says, then I can make a fix for this.
Question: When we use `dtype=str` do we want `np.nan` to turn to a string `"nan"` or to stay as it is?
I made a fix for this issue, but the above functionality is affected.
The purpose of this issue is just change this functionality: currently `np.nan` turns to a string `"nan"`, and I think that it should stay as it.
As @arnau126 points out, the result from `pd.read_excel` with `dtype=str` is inconsistent with that from `pd.read_csv`.  The value of placing a `np.nan` instead of the string representation is that you can use `pd.isna`, which does not work for 'nan'. So the thought is to make `read_excel` consistent with `read_csv`.
@cbertinato @arnau126 maybe my question wasn't very clear.
I suggested changing `astype_unicode` and `astype_str` (https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/lib.pyx#L460-L483), which would mean that `'nan'` would **always** become `np.nan`.

If we only want to change the `read_excel` function, then we could just substitute all the `'nan'` to `np.nan` before we return the result to the user. Not very elegant, but it would only affect `read_excel`.

Which one do we want here?

Sorry if I misunderstood what you guys are saying.
I think that the latter would be safer and less likely to break something else. This is clearly only an issue for `read_excel` and not `read_csv`.
It is clear now, thanks. I'll do the changes.

Resolving patch:
diff --git a/pandas/core/dtypes/cast.py b/pandas/core/dtypes/cast.py
--- a/pandas/core/dtypes/cast.py
+++ b/pandas/core/dtypes/cast.py
@@ -645,9 +645,9 @@ def conv(r, dtype):
     return [conv(r, dtype) for r, dtype in zip(result, dtypes)]
 
 
-def astype_nansafe(arr, dtype, copy=True):
-    """ return a view if copy is False, but
-        need to be very careful as the result shape could change!
+def astype_nansafe(arr, dtype, copy=True, skipna=False):
+    """
+    Cast the elements of an array to a given dtype a nan-safe manner.
 
     Parameters
     ----------
@@ -655,7 +655,9 @@ def astype_nansafe(arr, dtype, copy=True):
     dtype : np.dtype
     copy : bool, default True
         If False, a view will be attempted but may fail, if
-        e.g. the itemsizes don't align.
+        e.g. the item sizes don't align.
+    skipna: bool, default False
+        Whether or not we should skip NaN when casting as a string-type.
     """
 
     # dispatch on extension dtype if needed
@@ -668,10 +670,12 @@ def astype_nansafe(arr, dtype, copy=True):
 
     if issubclass(dtype.type, text_type):
         # in Py3 that's str, in Py2 that's unicode
-        return lib.astype_unicode(arr.ravel()).reshape(arr.shape)
+        return lib.astype_unicode(arr.ravel(),
+                                  skipna=skipna).reshape(arr.shape)
 
     elif issubclass(dtype.type, string_types):
-        return lib.astype_str(arr.ravel()).reshape(arr.shape)
+        return lib.astype_str(arr.ravel(),
+                              skipna=skipna).reshape(arr.shape)
 
     elif is_datetime64_dtype(arr):
         if is_object_dtype(dtype):
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -1685,7 +1685,8 @@ def _cast_types(self, values, cast_type, column):
 
         else:
             try:
-                values = astype_nansafe(values, cast_type, copy=True)
+                values = astype_nansafe(values, cast_type,
+                                        copy=True, skipna=True)
             except ValueError:
                 raise ValueError("Unable to convert column %s to "
                                  "type %s" % (column, cast_type))


Test patch:
diff --git a/pandas/tests/io/parser/na_values.py b/pandas/tests/io/parser/na_values.py
--- a/pandas/tests/io/parser/na_values.py
+++ b/pandas/tests/io/parser/na_values.py
@@ -5,6 +5,7 @@
 parsing for all of the parsers defined in parsers.py
 """
 
+import pytest
 import numpy as np
 from numpy import nan
 
@@ -380,3 +381,18 @@ def test_inf_na_values_with_int_index(self):
         expected = DataFrame({"col1": [3, np.nan], "col2": [4, np.nan]},
                              index=Index([1, 2], name="idx"))
         tm.assert_frame_equal(out, expected)
+
+    @pytest.mark.parametrize("na_filter", [True, False])
+    def test_na_values_with_dtype_str_and_na_filter(self, na_filter):
+        # see gh-20377
+        data = "a,b,c\n1,,3\n4,5,6"
+
+        # na_filter=True --> missing value becomes NaN.
+        # na_filter=False --> missing value remains empty string.
+        empty = np.nan if na_filter else ""
+        expected = DataFrame({"a": ["1", "4"],
+                              "b": [empty, "5"],
+                              "c": ["3", "6"]})
+
+        result = self.read_csv(StringIO(data), na_filter=na_filter, dtype=str)
+        tm.assert_frame_equal(result, expected)
diff --git a/pandas/tests/io/test_excel.py b/pandas/tests/io/test_excel.py
--- a/pandas/tests/io/test_excel.py
+++ b/pandas/tests/io/test_excel.py
@@ -14,7 +14,7 @@
 import pandas as pd
 import pandas.util.testing as tm
 import pandas.util._test_decorators as td
-from pandas import DataFrame, Index, MultiIndex
+from pandas import DataFrame, Index, MultiIndex, Series
 from pandas.compat import u, range, map, BytesIO, iteritems, PY36
 from pandas.core.config import set_option, get_option
 from pandas.io.common import URLError
@@ -371,7 +371,34 @@ def test_reader_dtype(self, ext):
         tm.assert_frame_equal(actual, expected)
 
         with pytest.raises(ValueError):
-            actual = self.get_exceldf(basename, ext, dtype={'d': 'int64'})
+            self.get_exceldf(basename, ext, dtype={'d': 'int64'})
+
+    @pytest.mark.parametrize("dtype,expected", [
+        (None,
+         DataFrame({
+             "a": [1, 2, 3, 4],
+             "b": [2.5, 3.5, 4.5, 5.5],
+             "c": [1, 2, 3, 4],
+             "d": [1.0, 2.0, np.nan, 4.0]
+         })),
+        ({"a": "float64",
+          "b": "float32",
+          "c": str,
+          "d": str
+          },
+         DataFrame({
+             "a": Series([1, 2, 3, 4], dtype="float64"),
+             "b": Series([2.5, 3.5, 4.5, 5.5], dtype="float32"),
+             "c": ["001", "002", "003", "004"],
+             "d": ["1", "2", np.nan, "4"]
+         })),
+    ])
+    def test_reader_dtype_str(self, ext, dtype, expected):
+        # see gh-20377
+        basename = "testdtype"
+
+        actual = self.get_exceldf(basename, ext, dtype=dtype)
+        tm.assert_frame_equal(actual, expected)
 
     def test_reading_all_sheets(self, ext):
         # Test reading all sheetnames by setting sheetname to None,