Repository: pandas-dev/pandas

Issue:
pandas/io/feather_format.py should call use_threads instead of nthreads to prevent breakage in pyarrow 0.11.0
#### Code Sample

```python
d = {'one' : [1., 2., 3., 4.],
        'two' : [4., 3., 2., 1.]}
df = pandas.DataFrame(d)
df.to_feather('example.feather')

# with pyarrow 0.10.0 this succeeds with a deprecation warning
# with pyarrow 0.11.0 this errors with a TypeError: unexpected argument 'nthreads'
df = pandas.read_feather('example.feather')

# attempt to manually set nthreads results in TypeError: unexpectect argument 'nthreads'
df = pandas.read_feather('example.feather', nthreads=4)

# attempt to pass 'use_threads' results in TypeError: unexpected argument 'nthreads'
df = pandas.read_feather('example.feather', use_threads=True)

```
#### Problem description
Pandas introduced nthreads for reading feather files in issue [16359](https://github.com/pandas-dev/pandas/issues/16359)

With PyArrow 0.10.0 a deprecation warning is shown from [this source](https://github.com/apache/arrow/blob/master/python/pyarrow/util.py): "`nthreads` argument is deprecated, pass `use_threads` instead"

When PyArrow version 0.11.0, Python errors with: TypeError: read_feather() got an unexpected keyword argument 'nthreads'.

I've searched with 'pyarrow' and 'nthreads' keywords and didn't see this issue posted.

Specifically [feather-format.py line 112](https://github.com/pandas-dev/pandas/blob/master/pandas/io/feather_format.py#L112) should be changed to 
`return feather.read_dataframe(path, use_threads=True)` or changing the method signature to all overriding use_threads:
`return feather.read_dataframe(path, use_threads=use_threads)`
I will submit a PR if the only barrier to fix is code effort.
#### Expected Output
I expect no error output upon running pandas.read_feather() with PyArrow 0.11.0

#### Output of ``pd.show_versions()``

<details>

[paste the output of ``pd.show_versions()`` here below this line]
INSTALLED VERSIONS                                           
------------------                                           
commit: None                                                 
python: 3.6.6.final.0                                        
python-bits: 64                                              
OS: Windows                                                  
OS-release: 7                                                
machine: AMD64                                               
processor: Intel64 Family 6 Model 79 Stepping 1, GenuineIntel
byteorder: little                                            
LC_ALL: None                                                 
LANG: None                                                   
LOCALE: None.None                                            
                                                             
pandas: 0.23.4                                               
pytest: None                                                 
pip: 18.1                                                    
setuptools: 40.3.0                                           
Cython: None                                                 
numpy: 1.15.1                                                
scipy: 1.1.0                                                 
pyarrow: 0.10.0                                              
xarray: None                                                 
IPython: 6.5.0                                               
sphinx: None                                                 
patsy: 0.5.0                                                 
dateutil: 2.7.3                                              
pytz: 2018.5                                                 
blosc: None                                                  
bottleneck: None                                             
tables: None                                                 
numexpr: None                                                
feather: 0.4.0                                               
matplotlib: 2.2.2                                            
openpyxl: None                                               
xlrd: 1.1.0                                                  
xlwt: None                                                   
xlsxwriter: None                                             
lxml: None                                                   
bs4: None                                                    
html5lib: None                                               
sqlalchemy: 1.2.11                                           
pymysql: None                                                
psycopg2: None                                               
jinja2: None                                                 
s3fs: None                                                   
fastparquet: None                                            
pandas_gbq: None                                             
pandas_datareader: None                                      

</details>

DEPS/DEPR: Allow import of feather through pyarrow
Currently, `DataFrame.to_feather()` tries to `import feather`, which only works if `feather-format` is installed. The same capabilities could also be imported as follows: `import pyarrow.feather as feather`.

The issue is that `pyarrow` is more easily available (e.g. `conda install feather-format` doesn't work without specifying a specific channel; but `pyarrow` works), and is under more active development.

I asked if there is a preference between the two imports in wesm/feather#341, and Wes answered [my bold]:
> You can use either method; there is no explicit recommendation right now. **`import feather` is provided for backwards compatibility**

I think the import should at least try `import pyarrow.feather as feather` as well, if `import feather` fails.


Discussion:
AFAICT, pyarrow doesn't have nightly builds we can test against, so I'm not sure what the best way to test this is. Will probably just have to be manual until 0.11 is released.
Hmmm, well it's in [PyPi](https://pypi.org/project/pyarrow/) but the mailing list indicates [its an RC](http://mail-archives.apache.org/mod_mbox/arrow-dev/201810.mbox/%3CCAJPUwMDC-hxg5FPLs_Cnq6KaxiATsFci%3D90bEjeN%2B0nhaSmg-A%40mail.gmail.com%3E).  

I've been waiting on a fix for another issue to be fixed with PyArrow on Python 3.7, so I may be one of the first to the punch on trying it out.
`pyarrow` has nightly conda packages on the twosigma channel. 

`pyarrow==0.11` is released so you can use it to test against it. We're missing Python 3.7 wheels at the moment but at least for Linux, these will appear in the next days.
From https://anaconda.org/twosigma/pyarrow/files?sort=time&sort_order=desc,
it looks like nightlies stopped being uploaded a few months ago.

Is @cpcloud the right person to ping here?

On Tue, Oct 9, 2018 at 9:11 AM Uwe L. Korn <notifications@github.com> wrote:

> pyarrow has nightly conda packages on the twosigma channel.
>
> pyarrow==0.11 is released so you can use it to test against it. We're
> missing Python 3.7 wheels at the moment but at least for Linux, these will
> appear in the next days.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/23053#issuecomment-428207341>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHImGDhp4TL6NdN0YN8GneVGT2dH-vks5ujK5-gaJpZM4XPmBW>
> .
>

We are seeing the same issue as tests were broken since pyarrow 0.11.0. Using `use_threads=True` seem to fix this issue for us.
@johnolos thanks. AFAIK, no one has submitted a PR updating pandas. If you could do so we'll include it in the next pandas release.
would take a PR to use pyarrow.feather
i suppose should continue to accept feather-format but deprecate it as a dep

Resolving patch:
diff --git a/pandas/io/feather_format.py b/pandas/io/feather_format.py
--- a/pandas/io/feather_format.py
+++ b/pandas/io/feather_format.py
@@ -3,6 +3,7 @@
 from distutils.version import LooseVersion
 
 from pandas.compat import range
+from pandas.util._decorators import deprecate_kwarg
 
 from pandas import DataFrame, Int64Index, RangeIndex
 
@@ -10,31 +11,27 @@
 
 
 def _try_import():
-    # since pandas is a dependency of feather
+    # since pandas is a dependency of pyarrow
     # we need to import on first use
-
     try:
-        import feather
+        import pyarrow
+        from pyarrow import feather
     except ImportError:
-
         # give a nice error message
-        raise ImportError("the feather-format library is not installed\n"
+        raise ImportError("pyarrow is not installed\n\n"
                           "you can install via conda\n"
-                          "conda install feather-format -c conda-forge\n"
+                          "conda install pyarrow -c conda-forge\n"
                           "or via pip\n"
-                          "pip install -U feather-format\n")
+                          "pip install -U pyarrow\n")
 
-    try:
-        LooseVersion(feather.__version__) >= LooseVersion('0.3.1')
-    except AttributeError:
-        raise ImportError("the feather-format library must be >= "
-                          "version 0.3.1\n"
+    if LooseVersion(pyarrow.__version__) < LooseVersion('0.4.1'):
+        raise ImportError("pyarrow >= 0.4.1 required for feather support\n\n"
                           "you can install via conda\n"
-                          "conda install feather-format -c conda-forge"
+                          "conda install pyarrow -c conda-forge"
                           "or via pip\n"
-                          "pip install -U feather-format\n")
+                          "pip install -U pyarrow\n")
 
-    return feather
+    return feather, pyarrow
 
 
 def to_feather(df, path):
@@ -51,7 +48,7 @@ def to_feather(df, path):
     if not isinstance(df, DataFrame):
         raise ValueError("feather only support IO with DataFrames")
 
-    feather = _try_import()
+    feather = _try_import()[0]
     valid_types = {'string', 'unicode'}
 
     # validate index
@@ -83,10 +80,11 @@ def to_feather(df, path):
     if df.columns.inferred_type not in valid_types:
         raise ValueError("feather must have string column names")
 
-    feather.write_dataframe(df, path)
+    feather.write_feather(df, path)
 
 
-def read_feather(path, nthreads=1):
+@deprecate_kwarg(old_arg_name='nthreads', new_arg_name='use_threads')
+def read_feather(path, use_threads=True):
     """
     Load a feather-format object from the file path
 
@@ -99,6 +97,11 @@ def read_feather(path, nthreads=1):
         Number of CPU threads to use when reading to pandas.DataFrame
 
        .. versionadded 0.21.0
+       .. deprecated 0.24.0
+    use_threads: bool, default True
+        Whether to parallelize reading using multiple threads
+
+       .. versionadded 0.24.0
 
     Returns
     -------
@@ -106,10 +109,13 @@ def read_feather(path, nthreads=1):
 
     """
 
-    feather = _try_import()
+    feather, pyarrow = _try_import()
     path = _stringify_path(path)
 
-    if LooseVersion(feather.__version__) < LooseVersion('0.4.0'):
-        return feather.read_dataframe(path)
+    if LooseVersion(pyarrow.__version__) < LooseVersion('0.11.0'):
+        int_use_threads = int(use_threads)
+        if int_use_threads < 1:
+            int_use_threads = 1
+        return feather.read_feather(path, nthreads=int_use_threads)
 
-    return feather.read_dataframe(path, nthreads=nthreads)
+    return feather.read_feather(path, use_threads=bool(use_threads))


Test patch:
diff --git a/pandas/tests/io/test_common.py b/pandas/tests/io/test_common.py
--- a/pandas/tests/io/test_common.py
+++ b/pandas/tests/io/test_common.py
@@ -135,9 +135,7 @@ def test_iterator(self):
         (pd.read_csv, 'os', FileNotFoundError, 'csv'),
         (pd.read_fwf, 'os', FileNotFoundError, 'txt'),
         (pd.read_excel, 'xlrd', FileNotFoundError, 'xlsx'),
-        pytest.param(
-            pd.read_feather, 'feather', Exception, 'feather',
-            marks=pytest.mark.xfail(reason="failing for pyarrow < 0.11.0")),
+        (pd.read_feather, 'feather', Exception, 'feather'),
         (pd.read_hdf, 'tables', FileNotFoundError, 'h5'),
         (pd.read_stata, 'os', FileNotFoundError, 'dta'),
         (pd.read_sas, 'os', FileNotFoundError, 'sas7bdat'),
@@ -162,10 +160,7 @@ def test_read_non_existant_read_table(self):
         (pd.read_csv, 'os', ('io', 'data', 'iris.csv')),
         (pd.read_fwf, 'os', ('io', 'data', 'fixed_width_format.txt')),
         (pd.read_excel, 'xlrd', ('io', 'data', 'test1.xlsx')),
-        pytest.param(
-            pd.read_feather, 'feather',
-            ('io', 'data', 'feather-0_3_1.feather'),
-            marks=pytest.mark.xfail(reason="failing for pyarrow < 0.11.0")),
+        (pd.read_feather, 'feather', ('io', 'data', 'feather-0_3_1.feather')),
         (pd.read_hdf, 'tables', ('io', 'data', 'legacy_hdf',
                                  'datetimetz_object.h5')),
         (pd.read_stata, 'os', ('io', 'data', 'stata10_115.dta')),
diff --git a/pandas/tests/io/test_feather.py b/pandas/tests/io/test_feather.py
--- a/pandas/tests/io/test_feather.py
+++ b/pandas/tests/io/test_feather.py
@@ -1,6 +1,5 @@
 """ test feather-format compat """
 from distutils.version import LooseVersion
-from warnings import catch_warnings
 
 import numpy as np
 
@@ -9,15 +8,13 @@
 from pandas.util.testing import assert_frame_equal, ensure_clean
 
 import pytest
-feather = pytest.importorskip('feather')
-from feather import FeatherError  # noqa:E402
+pyarrow = pytest.importorskip('pyarrow')
 
 from pandas.io.feather_format import to_feather, read_feather  # noqa:E402
 
-fv = LooseVersion(feather.__version__)
+pyarrow_version = LooseVersion(pyarrow.__version__)
 
 
-@pytest.mark.xfail(reason="failing for pyarrow < 0.11.0")
 @pytest.mark.single
 class TestFeather(object):
 
@@ -34,8 +31,7 @@ def check_round_trip(self, df, **kwargs):
         with ensure_clean() as path:
             to_feather(df, path)
 
-            with catch_warnings(record=True):
-                result = read_feather(path, **kwargs)
+            result = read_feather(path, **kwargs)
             assert_frame_equal(result, df)
 
     def test_error(self):
@@ -65,13 +61,6 @@ def test_basic(self):
         assert df.dttz.dtype.tz.zone == 'US/Eastern'
         self.check_round_trip(df)
 
-    @pytest.mark.skipif(fv >= LooseVersion('0.4.0'), reason='fixed in 0.4.0')
-    def test_strided_data_issues(self):
-
-        # strided data issuehttps://github.com/wesm/feather/issues/97
-        df = pd.DataFrame(np.arange(12).reshape(4, 3), columns=list('abc'))
-        self.check_error_on_write(df, FeatherError)
-
     def test_duplicate_columns(self):
 
         # https://github.com/wesm/feather/issues/53
@@ -85,17 +74,6 @@ def test_stringify_columns(self):
         df = pd.DataFrame(np.arange(12).reshape(4, 3)).copy()
         self.check_error_on_write(df, ValueError)
 
-    @pytest.mark.skipif(fv >= LooseVersion('0.4.0'), reason='fixed in 0.4.0')
-    def test_unsupported(self):
-
-        # timedelta
-        df = pd.DataFrame({'a': pd.timedelta_range('1 day', periods=3)})
-        self.check_error_on_write(df, FeatherError)
-
-        # non-strings
-        df = pd.DataFrame({'a': ['a', 1, 2.0]})
-        self.check_error_on_write(df, ValueError)
-
     def test_unsupported_other(self):
 
         # period
@@ -103,11 +81,26 @@ def test_unsupported_other(self):
         # Some versions raise ValueError, others raise ArrowInvalid.
         self.check_error_on_write(df, Exception)
 
-    @pytest.mark.skipif(fv < LooseVersion('0.4.0'), reason='new in 0.4.0')
     def test_rw_nthreads(self):
-
         df = pd.DataFrame({'A': np.arange(100000)})
-        self.check_round_trip(df, nthreads=2)
+        expected_warning = (
+            "the 'nthreads' keyword is deprecated, "
+            "use 'use_threads' instead"
+        )
+        with tm.assert_produces_warning(FutureWarning) as w:
+            self.check_round_trip(df, nthreads=2)
+        assert len(w) == 1
+        assert expected_warning in str(w[0])
+
+        with tm.assert_produces_warning(FutureWarning) as w:
+            self.check_round_trip(df, nthreads=1)
+        assert len(w) == 1
+        assert expected_warning in str(w[0])
+
+    def test_rw_use_threads(self):
+        df = pd.DataFrame({'A': np.arange(100000)})
+        self.check_round_trip(df, use_threads=True)
+        self.check_round_trip(df, use_threads=False)
 
     def test_write_with_index(self):