Repository: pandas-dev/pandas

Issue:
The feather example from docs segfault with pyarrow 0.8.0
The example from the docs succeeds to write and read the feather file, but the read-in dataframe is corrupted. 
This seems to be due to the Categorical column (reduced example):

```
In [1]: pd.__version__                                                                                                                                                                                              
Out[1]: '0.24.0rc1+12.g453fa85a8'

In [2]: import pyarrow                                                                                                                                                                                              

In [3]: pyarrow.__version__                                                                                                                                                                                         
Out[3]: '0.8.0'

In [6]: df = pd.DataFrame({'a': [1, 2, 3], 'b': pd.Categorical(['a', 'b', 'a'])})                                                                                                                                   

In [7]: df.to_feather('example.feather')                                                                                                                                                                            

In [8]: result = pd.read_feather('example.feather')                                                                                                                                                                 
/home/joris/miniconda3/envs/dev/lib/python3.5/site-packages/pyarrow/pandas_compat.py:628: FutureWarning: .labels was deprecated in version 0.24.0. Use .codes instead.
  labels, = index.labels

In [9]: result._data                                                                                                                                                                                                
Out[9]: 
BlockManager
Items: Index(['a', 'b'], dtype='object')
Axis 1: RangeIndex(start=0, stop=3, step=1)
IntBlock: slice(0, 1, 1), 1 x 3, dtype: int64
CategoricalBlock: slice(1, 2, 1), 1 x 3, dtype: category

In [10]: result._data.blocks[1].values                                                                                                                                                                              
Out[10]: Segmentation fault (core dumped)
```

On general displaying of the `result`, you also get a segfault.

The above is with latest master of pandas, and pyarrow 0.8.0 installed from conda-forge (quite old, but still in the supported range). Still need to try with a newer version of pyarrow.





Discussion:
The above was in my development environment (where I apparently still has an old pyarrow), but I now created a clean new env just with installing pyarrow and pandas, and can confirm the issue. With pyarrow 0.11 it works fine, but reading the file with pyarrow 0.8.0 gives a segfault. 

Of course, it can also be a bug in the old version of pyarrow. But the question is then if we need to guard users against it (if there is some way to detect the invalid data coming from pyarrow).
Bumping to pyarrow 0.9 was favourably discussed in #24617. Also xref #23482.
OK, let's maybe do that then for 0.24 ? That would at least prevent the segmentation fault from happening (I checked that 0.9 doesn't give the error)
I think that seems reasonable to squeeze in at this time.

Should we also bump up the min version of `fastparquet` (similar to #23482), or is that not necessary?
We bumped fastparquet to its latest.


________________________________
From: gfyoung <notifications@github.com>
Sent: Sunday, January 20, 2019 18:05
To: pandas-dev/pandas
Cc: Subscribed
Subject: Re: [pandas-dev/pandas] The feather example from docs segfault with pyarrow 0.8.0 (#24767)


I think that seems reasonable to squeeze in at this time.

Should we also bump up the min version of fastparquet (similar to #23482<https://github.com/pandas-dev/pandas/pull/23482>), or is that not necessary?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<https://github.com/pandas-dev/pandas/issues/24767#issuecomment-455915287>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABQHItbhPjJBj1eyWsi0fOclSBQ59etHks5vFQQ5gaJpZM4Z-5BB>.


Resolving patch:
diff --git a/pandas/io/feather_format.py b/pandas/io/feather_format.py
--- a/pandas/io/feather_format.py
+++ b/pandas/io/feather_format.py
@@ -24,8 +24,8 @@ def _try_import():
                           "or via pip\n"
                           "pip install -U pyarrow\n")
 
-    if LooseVersion(pyarrow.__version__) < LooseVersion('0.4.1'):
-        raise ImportError("pyarrow >= 0.4.1 required for feather support\n\n"
+    if LooseVersion(pyarrow.__version__) < LooseVersion('0.9.0'):
+        raise ImportError("pyarrow >= 0.9.0 required for feather support\n\n"
                           "you can install via conda\n"
                           "conda install pyarrow -c conda-forge"
                           "or via pip\n"
diff --git a/pandas/io/parquet.py b/pandas/io/parquet.py
--- a/pandas/io/parquet.py
+++ b/pandas/io/parquet.py
@@ -89,9 +89,9 @@ def __init__(self):
                 "\nor via pip\n"
                 "pip install -U pyarrow\n"
             )
-        if LooseVersion(pyarrow.__version__) < '0.7.0':
+        if LooseVersion(pyarrow.__version__) < '0.9.0':
             raise ImportError(
-                "pyarrow >= 0.7.0 is required for parquet support\n\n"
+                "pyarrow >= 0.9.0 is required for parquet support\n\n"
                 "you can install via conda\n"
                 "conda install pyarrow -c conda-forge\n"
                 "\nor via pip\n"


Test patch:
diff --git a/pandas/tests/io/test_parquet.py b/pandas/tests/io/test_parquet.py
--- a/pandas/tests/io/test_parquet.py
+++ b/pandas/tests/io/test_parquet.py
@@ -47,8 +47,6 @@ def engine(request):
 def pa():
     if not _HAVE_PYARROW:
         pytest.skip("pyarrow is not installed")
-    if LooseVersion(pyarrow.__version__) < LooseVersion('0.7.0'):
-        pytest.skip("pyarrow is < 0.7.0")
     return 'pyarrow'
 
 
@@ -289,11 +287,6 @@ def test_read_columns(self, engine):
     def test_write_index(self, engine):
         check_names = engine != 'fastparquet'
 
-        if engine == 'pyarrow':
-            import pyarrow
-            if LooseVersion(pyarrow.__version__) < LooseVersion('0.7.0'):
-                pytest.skip("pyarrow is < 0.7.0")
-
         df = pd.DataFrame({'A': [1, 2, 3]})
         check_round_trip(df, engine)
 
@@ -386,10 +379,8 @@ def test_basic(self, pa, df_full):
         df = df_full
 
         # additional supported types for pyarrow
-        import pyarrow
-        if LooseVersion(pyarrow.__version__) >= LooseVersion('0.7.0'):
-            df['datetime_tz'] = pd.date_range('20130101', periods=3,
-                                              tz='Europe/Brussels')
+        df['datetime_tz'] = pd.date_range('20130101', periods=3,
+                                          tz='Europe/Brussels')
         df['bool_with_none'] = [True, None, True]
 
         check_round_trip(df, pa)