Repository: pandas-dev/pandas
Pull request: BUG: Fix segfault on dir of a DataFrame with a unicode surrogate character in the column name
Return a `repr()` version if the column name string is not printable. This also means the the column name is not present in the output of `dir()`

- [x] closes #25509 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
Dir fails on dataframes with pathological column names
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd
df = pd.DataFrame({'\ud83d': []})
_ = dir(df)
```

```python-traceback
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
<ipython-input-3-4e949fe17c82> in <module>
----> 1 _ = dir(df)

~/miniconda/envs/dev/lib/python3.7/site-packages/pandas/core/accessor.py in __dir__(self)
     37         """
     38         rv = set(dir(type(self)))
---> 39         rv = (rv - self._dir_deletions()) | self._dir_additions()
     40         return sorted(rv)
     41

~/miniconda/envs/dev/lib/python3.7/site-packages/pandas/core/generic.py in _dir_additions(self)
   5110         If info_axis is a MultiIndex, it's first level values are used.
   5111         """
-> 5112         additions = {c for c in self._info_axis.unique(level=0)[:100]
   5113                      if isinstance(c, string_types) and isidentifier(c)}
   5114         return super(NDFrame, self)._dir_additions().union(additions)

~/miniconda/envs/dev/lib/python3.7/site-packages/pandas/core/indexes/base.py in unique(self, level)
   1999         if level is not None:
   2000             self._validate_index_level(level)
-> 2001         result = super(Index, self).unique()
   2002         return self._shallow_copy(result)
   2003

~/miniconda/envs/dev/lib/python3.7/site-packages/pandas/core/base.py in unique(self)
   1312         else:
   1313             from pandas.core.algorithms import unique1d
-> 1314             result = unique1d(values)
   1315
   1316         return result

~/miniconda/envs/dev/lib/python3.7/site-packages/pandas/core/algorithms.py in unique(values)
    360
    361     table = htable(len(values))
--> 362     uniques = table.unique(values)
    363     uniques = _reconstruct_data(uniques, dtype, original)
    364     return uniques

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.StringHashTable.unique()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.StringHashTable._unique()

UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d' in position 0: surrogates not allowed
```

#### Problem description

Dir fails on dataframes with pathalogical column names

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.7.1.final.0
python-bits: 64
OS: Darwin
OS-release: 18.2.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.1
pytest: 3.10.1
pip: 18.1
setuptools: 40.6.2
Cython: None
numpy: 1.15.4
scipy: 1.1.0
pyarrow: 0.11.1
xarray: 0.11.3
IPython: 7.2.0
sphinx: 1.8.4
patsy: None
dateutil: 2.7.5
pytz: 2018.7
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: 0.2.0
fastparquet: 0.1.6
pandas_gbq: None
pandas_datareader: None
gcsfs: None

</details>

Hints from issue discussion:
On OSX this segfaults
What is the expectation here? Is this the first half of a surrogate pair?
Tracking this down, it looks like we get to tslibs.util.get_c_string_buf_and_size and within that we call `PyUnicode_AsUTF8AndSize` and fail there
So with regards to the OP I don't *think* this is a bug with pandas - an exception gets thrown when passing it as an argument to `print` in python though strangely enough the object seems OK held in a string

```python
>>> print('\ud83d')
>>> type('\ud83d')
<class 'str'>
>>> alist = ['\ud83d']
>>> alist[0] # surprised this works
'\ud83d'
>>> print(alist[0]) # this failure matches pandas
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d' in position 0: surrogates not allowed
```

w.r.t. Cython I see the following warnings before segfault, so maybe something of interest there:

```python
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d' in position 0: surrogates not allowed
Exception ignored in: 'pandas._libs.tslibs.util.get_c_string_buf_and_size'
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud83d' in position 0: surrogates not allowed
[1]    32913 segmentation fault  python
```
Doing some googling, is there a specific range of unicode characters that are surrogates that we might be able to screen for?  Are there non-surrogate "pathological" cases we need to worry about?
In `pd._libs.tslibs.util.get_c_string_buf_and_size` if we change:

```
    if PyUnicode_Check(py_string):
        buf = PyUnicode_AsUTF8AndSize(py_string, length)
```

to

```
    if PyUnicode_Check(py_string):
        if not py_string.isprintable():
            py_string = repr(py_string)
        buf = PyUnicode_AsUTF8AndSize(py_string, length)
```

then a) the `dir(df)` call doesn't raise/segfault, b) `dir(df)` does _not_ contain the column name like we would expect, c) `df.columns.unique()` doesn't raise/segfault, d) `df.columns.unique()` _does_ contain the 1 entry we expect.

Not sure what to do with this information, but its out there.
I have traced this down to `get_c_string` (from `util.pxd`) in `hashtable_class_helper.pxi.in`.
The unicode string with the surrogate character in it is passed to `get_c_string` which passes it to `get_c_string_buf_and_size`, which passes it to `PyUnicode_AsUTF8AndSize`.
This last one will happily return a NULL pointer when it receives a surrogate character. This is passed down the line to a point where it is dereferenced in `khash.h` (probably to calculate a hash value).
Hence the segfault.

This is where the NULL pointer gets assigned to an array value in `hashtable_class_helper.pxi.in`:791:
```python
                # if ignore_na is False, we also stringify NaN/None/etc.
                v = get_c_string(<str>val)
                vecs[i] = v
```

Commit message: Performance fix

Patch to review:
diff --git a/doc/source/whatsnew/v1.1.0.rst b/doc/source/whatsnew/v1.1.0.rst
index 0d3a9a8f969a4..b97dd580e2d8a 100644
--- a/doc/source/whatsnew/v1.1.0.rst
+++ b/doc/source/whatsnew/v1.1.0.rst
@@ -402,6 +402,7 @@ Other
 - Fixed :func:`pandas.testing.assert_series_equal` to correctly raise if left object is a different subclass with ``check_series_type=True`` (:issue:`32670`).
 - :meth:`IntegerArray.astype` now supports ``datetime64`` dtype (:issue:32538`)
 - Fixed bug in :func:`pandas.testing.assert_series_equal` where dtypes were checked for ``Interval`` and ``ExtensionArray`` operands when ``check_dtype`` was ``False`` (:issue:`32747`)
+- Bug in :meth:`DataFrame.__dir__` caused a segfault when using unicode surrogates in a column name (:issue:`25509`)
 
 .. ---------------------------------------------------------------------------
 
diff --git a/pandas/_libs/hashtable_class_helper.pxi.in b/pandas/_libs/hashtable_class_helper.pxi.in
index 811025a4b5764..1fa672be6f1f9 100644
--- a/pandas/_libs/hashtable_class_helper.pxi.in
+++ b/pandas/_libs/hashtable_class_helper.pxi.in
@@ -12,6 +12,11 @@ WARNING: DO NOT edit .pxi FILE directly, .pxi is generated from .pxi.in
 from pandas._libs.tslibs.util cimport get_c_string
 from pandas._libs.missing cimport C_NA
 
+cdef extern from "Python.h":
+    # Note: importing extern-style allows us to declare these as nogil
+    # functions, whereas `from cpython cimport` does not.
+    void PyErr_Clear() nogil
+
 {{py:
 
 # name, dtype, c_type
@@ -790,6 +795,9 @@ cdef class StringHashTable(HashTable):
             else:
                 # if ignore_na is False, we also stringify NaN/None/etc.
                 v = get_c_string(<str>val)
+                if v == NULL:
+                    PyErr_Clear()
+                    v = get_c_string(<str>repr(val))
                 vecs[i] = v
 
         # compute
diff --git a/pandas/tests/frame/test_api.py b/pandas/tests/frame/test_api.py
index a021dd91a7d26..940a76601b75e 100644
--- a/pandas/tests/frame/test_api.py
+++ b/pandas/tests/frame/test_api.py
@@ -127,6 +127,14 @@ def test_not_hashable(self):
         with pytest.raises(TypeError, match=msg):
             hash(empty_frame)
 
+    def test_column_name_contains_unicode_surrogate(self):
+        # GH 25509
+        colname = "\ud83d"
+        df = DataFrame({colname: []})
+        # this should not crash
+        assert colname not in dir(df)
+        assert df.columns[0] == colname
+
     def test_new_empty_index(self):
         df1 = DataFrame(np.random.randn(0, 3))
         df2 = DataFrame(np.random.randn(0, 3))
diff --git a/pandas/tests/io/parser/test_dtypes.py b/pandas/tests/io/parser/test_dtypes.py
index 11dcf7f04f76b..e68dcb3aa577e 100644
--- a/pandas/tests/io/parser/test_dtypes.py
+++ b/pandas/tests/io/parser/test_dtypes.py
@@ -192,7 +192,7 @@ def test_categorical_dtype_utf16(all_parsers, csv_dir_path):
     pth = os.path.join(csv_dir_path, "utf16_ex.txt")
     parser = all_parsers
     encoding = "utf-16"
-    sep = ","
+    sep = "\t"
 
     expected = parser.read_csv(pth, sep=sep, encoding=encoding)
     expected = expected.apply(Categorical)


Code review comments:
diff_hunk: @@ -12,6 +12,11 @@ WARNING: DO NOT edit .pxi FILE directly, .pxi is generated from .pxi.in
 from pandas._libs.tslibs.util cimport get_c_string
 from pandas._libs.missing cimport C_NA
 
+cdef extern from "Python.h":
+    # Note: importing extern-style allows us to declare these as nogil
path: pandas/_libs/hashtable_class_helper.pxi.in
text: @user1:
This is interacting with Python objects so it needs to be called with the GIL

@author:
Will remove it, I am quite new to cython, so I have to read up a bit on these things.

@user1:
If you are getting warnings / errors when trying to cythonize you might need a `with gil:` block in your error handler

@author:
I am not getting any cython errors, so I am probably good.