Repository: pandas-dev/pandas

Issue:
pd.read_pickle, pd.DataFrame.to_pickle do not accept pathlib.Path objects
#### Code Sample

```python
In [1]: import pandas as pd
   ...: import pathlib as pl
   ...: path = pl.Path.home() / 'tmp.pickle' # I have prepared a pickle in this path
   ...: 

In [2]: df = pd.read_pickle(str(path)) # This works, no problem
   ...: 

In [3]: df2 = pd.read_pickle(path) # This breaks
   ...: 
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/home/ntaw/anaconda3/lib/python3.5/site-packages/pandas/io/pickle.py in try_read(path, encoding)
     50         try:
---> 51             with open(path, 'rb') as fh:
     52                 return pkl.load(fh)

TypeError: invalid file: PosixPath('/home/ntaw/tmp.pickle')

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
/home/ntaw/anaconda3/lib/python3.5/site-packages/pandas/io/pickle.py in try_read(path, encoding)
     55             try:
---> 56                 with open(path, 'rb') as fh:
     57                     return pc.load(fh, encoding=encoding, compat=False)

TypeError: invalid file: PosixPath('/home/ntaw/tmp.pickle')

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
/home/ntaw/anaconda3/lib/python3.5/site-packages/pandas/io/pickle.py in read_pickle(path)
     64     try:
---> 65         return try_read(path)
     66     except:

/home/ntaw/anaconda3/lib/python3.5/site-packages/pandas/io/pickle.py in try_read(path, encoding)
     60             except:
---> 61                 with open(path, 'rb') as fh:
     62                     return pc.load(fh, encoding=encoding, compat=True)

TypeError: invalid file: PosixPath('/home/ntaw/tmp.pickle')

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
/home/ntaw/anaconda3/lib/python3.5/site-packages/pandas/io/pickle.py in try_read(path, encoding)
     50         try:
---> 51             with open(path, 'rb') as fh:
     52                 return pkl.load(fh)

TypeError: invalid file: PosixPath('/home/ntaw/tmp.pickle')

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
/home/ntaw/anaconda3/lib/python3.5/site-packages/pandas/io/pickle.py in try_read(path, encoding)
     55             try:
---> 56                 with open(path, 'rb') as fh:
     57                     return pc.load(fh, encoding=encoding, compat=False)

TypeError: invalid file: PosixPath('/home/ntaw/tmp.pickle')

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-3-d33de9a00518> in <module>()
----> 1 df2 = pd.read_pickle(path)

/home/ntaw/anaconda3/lib/python3.5/site-packages/pandas/io/pickle.py in read_pickle(path)
     66     except:
     67         if PY3:
---> 68             return try_read(path, encoding='latin1')
     69         raise
     70 

/home/ntaw/anaconda3/lib/python3.5/site-packages/pandas/io/pickle.py in try_read(path, encoding)
     59             # compat pickle
     60             except:
---> 61                 with open(path, 'rb') as fh:
     62                     return pc.load(fh, encoding=encoding, compat=True)
     63 

TypeError: invalid file: PosixPath('/home/ntaw/tmp.pickle')
```

#### Problem description

`pandas` breaks at `df = pd.read_pickle(path)`. It seems to me that this should have been solved along with issue [#11033](https://github.com/pandas-dev/pandas/issues/11033), but it evidently doesn't work for my setup.

I encountered the same issue with `pd.DataFrame.to_pickle`.

The usecase appears to be related to open issue [#14705](https://github.com/pandas-dev/pandas/issues/14705) and closed pull request [#12930](https://github.com/pandas-dev/pandas/pull/12930).

#### Expected Output

 I would expect the `pandas.read_*` functions and `pandas.DataFrame.to_*`  methods to recognise `pathlib` objects, and handle the conversion `str(path)` under the hood.

#### Output of ``pd.show_versions()``

<details>
INSTALLED VERSIONS
------------------
commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-53-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.19.1
nose: 1.3.7
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.1
numpy: 1.11.2
scipy: 0.18.1
statsmodels: 0.6.1
xarray: None
IPython: 5.1.0
sphinx: 1.4.8
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2016.7
blosc: None
bottleneck: 1.1.0
tables: 3.3.0
numexpr: 2.6.1
matplotlib: 1.5.3
openpyxl: 2.4.0
xlrd: 1.0.0
xlwt: 1.1.2
xlsxwriter: 0.9.3
lxml: 3.6.4
bs4: 4.5.1
html5lib: None
httplib2: 0.9.2
apiclient: 1.5.1
sqlalchemy: 1.1.3
pymysql: None
psycopg2: None
jinja2: 2.8
boto: 2.43.0
pandas_datareader: None
</details>



Discussion:
original issue to implement for excel, hdf, csv was here: https://github.com/pandas-dev/pandas/pull/11051/files

pickle should be quite easy if you'd like to do a PR.
xref https://github.com/pandas-dev/pandas/issues/11666

this should simply use ``pandas.io.common._get_handle`` which will then facilitate both of these requests (pathlib and compression) as that now is our generic handler.
xref #13823
Sounds like a fun way to start contributing :) I'll give it a shot!
`pandas.io.common._get_handle` does not support `pathlib.Path` with python 3.5.2, but it seems that the [`open` call supports it in 3.6](https://docs.python.org/3.6/whatsnew/3.6.html#pep-519-adding-a-file-system-path-protocol). [`get_filepath_or_buffer`](https://github.com/pandas-dev/pandas/blob/dc4b0708f36b971f71890bfdf830d9a5dc019c7b/pandas/io/common.py#L193), however, seems to handle it well. So, what did you mean with 

> this should simply use `pandas.io.common._get_handle` 

?

Resolving patch:
diff --git a/pandas/io/common.py b/pandas/io/common.py
--- a/pandas/io/common.py
+++ b/pandas/io/common.py
@@ -314,6 +314,9 @@ def _get_handle(path_or_buf, mode, encoding=None, compression=None,
 
     handles = list()
     f = path_or_buf
+
+    # Convert pathlib.Path/py.path.local or string
+    path_or_buf = _stringify_path(path_or_buf)
     is_path = isinstance(path_or_buf, compat.string_types)
 
     if compression:
diff --git a/pandas/io/formats/format.py b/pandas/io/formats/format.py
--- a/pandas/io/formats/format.py
+++ b/pandas/io/formats/format.py
@@ -32,7 +32,8 @@
                            OrderedDict, unichr)
 from pandas.io.formats.terminal import get_terminal_size
 from pandas.core.config import get_option, set_option
-from pandas.io.common import _get_handle, UnicodeWriter, _expand_user
+from pandas.io.common import (_get_handle, UnicodeWriter, _expand_user,
+                              _stringify_path)
 from pandas.io.formats.printing import adjoin, justify, pprint_thing
 from pandas.io.formats.common import get_level_lengths
 import pandas.core.common as com
@@ -1475,7 +1476,7 @@ def __init__(self, obj, path_or_buf=None, sep=",", na_rep='',
         if path_or_buf is None:
             path_or_buf = StringIO()
 
-        self.path_or_buf = _expand_user(path_or_buf)
+        self.path_or_buf = _expand_user(_stringify_path(path_or_buf))
         self.sep = sep
         self.na_rep = na_rep
         self.float_format = float_format


Test patch:
diff --git a/pandas/tests/io/parser/common.py b/pandas/tests/io/parser/common.py
--- a/pandas/tests/io/parser/common.py
+++ b/pandas/tests/io/parser/common.py
@@ -679,6 +679,19 @@ def test_file(self):
 
         tm.assert_frame_equal(url_table, local_table)
 
+    def test_path_pathlib(self):
+        df = tm.makeDataFrame()
+        result = tm.round_trip_pathlib(df.to_csv,
+                                       lambda p: self.read_csv(p, index_col=0))
+        tm.assert_frame_equal(df, result)
+
+    def test_path_localpath(self):
+        df = tm.makeDataFrame()
+        result = tm.round_trip_localpath(
+            df.to_csv,
+            lambda p: self.read_csv(p, index_col=0))
+        tm.assert_frame_equal(df, result)
+
     def test_nonexistent_path(self):
         # gh-2428: pls no segfault
         # gh-14086: raise more helpful FileNotFoundError
diff --git a/pandas/tests/io/sas/test_sas7bdat.py b/pandas/tests/io/sas/test_sas7bdat.py
--- a/pandas/tests/io/sas/test_sas7bdat.py
+++ b/pandas/tests/io/sas/test_sas7bdat.py
@@ -3,6 +3,7 @@
 import pandas.util.testing as tm
 import os
 import io
+import pytest
 import numpy as np
 
 
@@ -65,6 +66,29 @@ def test_from_iterator(self):
                 tm.assert_frame_equal(df, df0.iloc[2:5, :])
                 rdr.close()
 
+    @pytest.mark.xfail(reason="read_sas currently doesn't work with pathlib")
+    def test_path_pathlib(self):
+        tm._skip_if_no_pathlib()
+        from pathlib import Path
+        for j in 0, 1:
+            df0 = self.data[j]
+            for k in self.test_ix[j]:
+                fname = Path(os.path.join(self.dirpath, "test%d.sas7bdat" % k))
+                df = pd.read_sas(fname, encoding='utf-8')
+                tm.assert_frame_equal(df, df0)
+
+    @pytest.mark.xfail(reason="read_sas currently doesn't work with localpath")
+    def test_path_localpath(self):
+        tm._skip_if_no_localpath()
+        from py.path import local as LocalPath
+        for j in 0, 1:
+            df0 = self.data[j]
+            for k in self.test_ix[j]:
+                fname = LocalPath(os.path.join(self.dirpath,
+                                               "test%d.sas7bdat" % k))
+                df = pd.read_sas(fname, encoding='utf-8')
+                tm.assert_frame_equal(df, df0)
+
     def test_iterator_loop(self):
         # github #13654
         for j in 0, 1:
diff --git a/pandas/tests/io/test_excel.py b/pandas/tests/io/test_excel.py
--- a/pandas/tests/io/test_excel.py
+++ b/pandas/tests/io/test_excel.py
@@ -1858,6 +1858,16 @@ def test_freeze_panes(self):
             result = read_excel(path)
             tm.assert_frame_equal(expected, result)
 
+    def test_path_pathlib(self):
+        df = tm.makeDataFrame()
+        result = tm.round_trip_pathlib(df.to_excel, pd.read_excel)
+        tm.assert_frame_equal(df, result)
+
+    def test_path_localpath(self):
+        df = tm.makeDataFrame()
+        result = tm.round_trip_localpath(df.to_excel, pd.read_excel)
+        tm.assert_frame_equal(df, result)
+
 
 def raise_wrapper(major_ver):
     def versioned_raise_wrapper(orig_method):
diff --git a/pandas/tests/io/test_feather.py b/pandas/tests/io/test_feather.py
--- a/pandas/tests/io/test_feather.py
+++ b/pandas/tests/io/test_feather.py
@@ -9,6 +9,7 @@
 
 from feather import FeatherError
 from pandas.util.testing import assert_frame_equal, ensure_clean
+import pandas.util.testing as tm
 
 
 @pytest.mark.single
@@ -114,3 +115,15 @@ def test_write_with_index(self):
         df.index = [0, 1, 2]
         df.columns = pd.MultiIndex.from_tuples([('a', 1), ('a', 2), ('b', 1)]),
         self.check_error_on_write(df, ValueError)
+
+    @pytest.mark.xfail(reason="feather currently doesn't work with pathlib")
+    def test_path_pathlib(self):
+        df = tm.makeDataFrame().reset_index()
+        result = tm.round_trip_pathlib(df.to_feather, pd.read_feather)
+        tm.assert_frame_equal(df, result)
+
+    @pytest.mark.xfail(reason="feather currently doesn't work with localpath")
+    def test_path_localpath(self):
+        df = tm.makeDataFrame().reset_index()
+        result = tm.round_trip_localpath(df.to_feather, pd.read_feather)
+        tm.assert_frame_equal(df, result)
diff --git a/pandas/tests/io/test_packers.py b/pandas/tests/io/test_packers.py
--- a/pandas/tests/io/test_packers.py
+++ b/pandas/tests/io/test_packers.py
@@ -134,6 +134,18 @@ def test_string_io(self):
             result = read_msgpack(p)
             tm.assert_frame_equal(result, df)
 
+    @pytest.mark.xfail(reason="msgpack currently doesn't work with pathlib")
+    def test_path_pathlib(self):
+        df = tm.makeDataFrame()
+        result = tm.round_trip_pathlib(df.to_msgpack, read_msgpack)
+        tm.assert_frame_equal(df, result)
+
+    @pytest.mark.xfail(reason="msgpack currently doesn't work with localpath")
+    def test_path_localpath(self):
+        df = tm.makeDataFrame()
+        result = tm.round_trip_localpath(df.to_msgpack, read_msgpack)
+        tm.assert_frame_equal(df, result)
+
     def test_iterator_with_string_io(self):
 
         dfs = [DataFrame(np.random.randn(10, 2)) for i in range(5)]
diff --git a/pandas/tests/io/test_pickle.py b/pandas/tests/io/test_pickle.py
--- a/pandas/tests/io/test_pickle.py
+++ b/pandas/tests/io/test_pickle.py
@@ -299,6 +299,18 @@ def test_pickle_v0_15_2():
     tm.assert_categorical_equal(cat, pd.read_pickle(pickle_path))
 
 
+def test_pickle_path_pathlib():
+    df = tm.makeDataFrame()
+    result = tm.round_trip_pathlib(df.to_pickle, pd.read_pickle)
+    tm.assert_frame_equal(df, result)
+
+
+def test_pickle_path_localpath():
+    df = tm.makeDataFrame()
+    result = tm.round_trip_localpath(df.to_pickle, pd.read_pickle)
+    tm.assert_frame_equal(df, result)
+
+
 # ---------------------
 # test pickle compression
 # ---------------------
diff --git a/pandas/tests/io/test_pytables.py b/pandas/tests/io/test_pytables.py
--- a/pandas/tests/io/test_pytables.py
+++ b/pandas/tests/io/test_pytables.py
@@ -4249,6 +4249,49 @@ def test_select_filter_corner(self):
             result = store.select('frame', [crit])
             tm.assert_frame_equal(result, df.loc[:, df.columns[:75:2]])
 
+    def test_path_pathlib(self):
+        df = tm.makeDataFrame()
+
+        result = tm.round_trip_pathlib(
+            lambda p: df.to_hdf(p, 'df'),
+            lambda p: pd.read_hdf(p, 'df'))
+        tm.assert_frame_equal(df, result)
+
+    @pytest.mark.xfail(reason='pathlib currently doesnt work with HDFStore')
+    def test_path_pathlib_hdfstore(self):
+        df = tm.makeDataFrame()
+
+        def writer(path):
+            with pd.HDFStore(path) as store:
+                df.to_hdf(store, 'df')
+
+        def reader(path):
+            with pd.HDFStore(path) as store:
+                pd.read_hdf(store, 'df')
+        result = tm.round_trip_pathlib(writer, reader)
+        tm.assert_frame_equal(df, result)
+
+    def test_pickle_path_localpath(self):
+        df = tm.makeDataFrame()
+        result = tm.round_trip_pathlib(
+            lambda p: df.to_hdf(p, 'df'),
+            lambda p: pd.read_hdf(p, 'df'))
+        tm.assert_frame_equal(df, result)
+
+    @pytest.mark.xfail(reason='localpath currently doesnt work with HDFStore')
+    def test_path_localpath_hdfstore(self):
+        df = tm.makeDataFrame()
+
+        def writer(path):
+            with pd.HDFStore(path) as store:
+                df.to_hdf(store, 'df')
+
+        def reader(path):
+            with pd.HDFStore(path) as store:
+                pd.read_hdf(store, 'df')
+        result = tm.round_trip_localpath(writer, reader)
+        tm.assert_frame_equal(df, result)
+
     def _check_roundtrip(self, obj, comparator, compression=False, **kwargs):
 
         options = {}
diff --git a/pandas/tests/io/test_stata.py b/pandas/tests/io/test_stata.py
--- a/pandas/tests/io/test_stata.py
+++ b/pandas/tests/io/test_stata.py
@@ -1283,3 +1283,15 @@ def test_invalid_encoding(self):
         with pytest.raises(ValueError):
             with tm.ensure_clean() as path:
                 original.to_stata(path, encoding='utf-8')
+
+    @pytest.mark.xfail(reason="stata currently doesn't work with pathlib")
+    def test_path_pathlib(self):
+        df = tm.makeDataFrame()
+        result = tm.round_trip_pathlib(df.to_stata, read_stata)
+        tm.assert_frame_equal(df, result)
+
+    @pytest.mark.xfail(reason="stata currently doesn't work with localpath")
+    def test_pickle_path_localpath(self):
+        df = tm.makeDataFrame()
+        result = tm.round_trip_localpath(df.to_stata, read_stata)
+        tm.assert_frame_equal(df, result)
diff --git a/pandas/util/testing.py b/pandas/util/testing.py
--- a/pandas/util/testing.py
+++ b/pandas/util/testing.py
@@ -117,6 +117,63 @@ def round_trip_pickle(obj, path=None):
         return pd.read_pickle(path)
 
 
+def round_trip_pathlib(writer, reader, path=None):
+    """
+    Write an object to file specifed by a pathlib.Path and read it back
+
+    Parameters
+    ----------
+    writer : callable bound to pandas object
+        IO writing function (e.g. DataFrame.to_csv )
+    reader : callable
+        IO reading function (e.g. pd.read_csv )
+    path : str, default None
+        The path where the object is written and then read.
+
+    Returns
+    -------
+    round_trip_object : pandas object
+        The original object that was serialized and then re-read.
+    """
+
+    import pytest
+    Path = pytest.importorskip('pathlib').Path
+    if path is None:
+        path = '___pathlib___'
+    with ensure_clean(path) as path:
+        writer(Path(path))
+        obj = reader(Path(path))
+    return obj
+
+
+def round_trip_localpath(writer, reader, path=None):
+    """
+    Write an object to file specifed by a py.path LocalPath and read it back
+
+    Parameters
+    ----------
+    writer : callable bound to pandas object
+        IO writing function (e.g. DataFrame.to_csv )
+    reader : callable
+        IO reading function (e.g. pd.read_csv )
+    path : str, default None
+        The path where the object is written and then read.
+
+    Returns
+    -------
+    round_trip_object : pandas object
+        The original object that was serialized and then re-read.
+    """
+    import pytest
+    LocalPath = pytest.importorskip('py.path').local
+    if path is None:
+        path = '___localpath___'
+    with ensure_clean(path) as path:
+        writer(LocalPath(path))
+        obj = reader(LocalPath(path))
+    return obj
+
+
 def assert_almost_equal(left, right, check_exact=False,
                         check_dtype='equiv', check_less_precise=False,
                         **kwargs):