Infer compression from non-string paths
State: closed

Currently, `compression='infer'` only works on paths that are strings. However, pandas supports non-string paths, such as `pathlib.Path` objects. This PR adds support for inferring compression for all path types (for io interfaces that use `io.common._infer_compression`).

## Checklist

- [x] passes ``git diff upstream/master -u -- "*.py" | flake8 --diff``
- [x] No issue opened
- [x] tests added / passed
- [x] whatsnew entry


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: IO CSV
description: read_csv, to_csv

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=h1) Report
> Merging [#17206](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/556effcba52f4712fed21b269e9782f1a309ea93?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17206/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&height=150&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17206      +/-   ##
==========================================
- Coverage   91.03%   91.01%   -0.02%     
==========================================
  Files         162      162              
  Lines       49511    49511              
==========================================
- Hits        45070    45061       -9     
- Misses       4441     4450       +9
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.79% <100%> (ø)` | :arrow_up: |
| #single | `40.26% <100%> (-0.06%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `67.79% <100%> (ø)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.71% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=footer). Last update [556effc...b4e280f](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=h1) Report
> Merging [#17206](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/556effcba52f4712fed21b269e9782f1a309ea93?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17206/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&height=150&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17206      +/-   ##
==========================================
- Coverage   91.03%   91.01%   -0.02%     
==========================================
  Files         162      162              
  Lines       49511    49511              
==========================================
- Hits        45070    45061       -9     
- Misses       4441     4450       +9
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.79% <100%> (ø)` | :arrow_up: |
| #single | `40.26% <100%> (-0.06%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `67.79% <100%> (ø)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.71% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=footer). Last update [556effc...b4e280f](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=h1) Report
> Merging [#17206](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/924b43359b2450cd1e6e364c468c30ee7694f0c1?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17206/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&height=150&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17206      +/-   ##
==========================================
- Coverage      91%   90.99%   -0.02%     
==========================================
  Files         162      162              
  Lines       49531    49535       +4     
==========================================
- Hits        45078    45073       -5     
- Misses       4453     4462       +9
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.77% <100%> (ø)` | :arrow_up: |
| #single | `40.25% <100%> (-0.07%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/parsers.py](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnNlcnMucHk=) | `95.45% <ø> (ø)` | :arrow_up: |
| [pandas/io/pickle.py](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=tree#diff-cGFuZGFzL2lvL3BpY2tsZS5weQ==) | `81.25% <ø> (ø)` | :arrow_up: |
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `67.79% <100%> (ø)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.72% <0%> (-0.1%)` | :arrow_down: |
| [pandas/core/reshape/concat.py](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9jb25jYXQucHk=) | `97.6% <0%> (ø)` | :arrow_up: |
| [pandas/core/reshape/reshape.py](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9yZXNoYXBlLnB5) | `99.28% <0%> (ø)` | :arrow_up: |
| [pandas/core/reshape/merge.py](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9tZXJnZS5weQ==) | `94.26% <0%> (+0.01%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=footer). Last update [924b433...8a15074](https://codecov.io/gh/pandas-dev/pandas/pull/17206?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@TomAugspurger @gfyoung, we're passing. Did you want me to make a What's New entry?
Yes, absolutely!
Currently, this PR is ready, except for @jreback's outstanding review: https://github.com/pandas-dev/pandas/pull/17206#pullrequestreview-55496287. I'm happy to "add a generalized compression test", but I need some additional guidance. Refer to the line-by-line discussion under the review.
Moving that out of the inline comment that was squashed:

> @jreback, Just to make sure we're on the same page. All path types would be:
> 
> - local paths via strings
> - local paths via fspath defining objects
> - URLs
> All compression types would be None, xz, bz2, zip, and gzip. But in all cases, we'd be inferring > compression?

@dhimmel it'll be similar to https://github.com/pandas-dev/pandas/blob/330b8c1c195174f729a1d2ee6f916ebd1579217e/pandas/tests/io/test_common.py#L130

You can parametrize over readers, extensions, and path types (string, pathlib, py.path, and `CustomFSPath`).

What all readers should this support?
quite a lot of these tests already exists

just the case for the oathlib type objects i think are a bit lite

however the fspath tests are centralized 
i would move these to the io readers themselves (where the reg path compression tests are)
Okay I moved the tests to `tests/io/test_common.py`. The new test only tests `_infer_compression` rather than testing entire file reading, so it's lighter weight. 

https://github.com/pandas-dev/pandas/blob/0f925c15d6d8a3c2248139f07fef3bb61385fecd/pandas/tests/io/test_common.py#L91-L102

> You can parametrize over ... path types (string, pathlib, py.path, and CustomFSPath).

@TomAugspurger see what I did in https://github.com/pandas-dev/pandas/pull/17206/commits/0f925c15d6d8a3c2248139f07fef3bb61385fecd to parametrize over path types. It requires creating a module level variable `path_types`... I couldn't identify a more elegant way.
> You could potentially add that to conftest.py, but I think it's fine as is.

Probably best for me to avoid touching `pandas/conftest.py`. Let's save that for someone who knows the the testing architecture a bit better (:
> You could potentially add that to conftest.py, but I think it's fine as is.

I agree.  There is no need to do this.  Ping on green!
Thanks @dhimmel !
@gfyoung pls don't merge things that I have an outstanding review on.
> - `read_*` methods can now infer compression from non-string paths, such as ``pathlib.Path`` objects (:issue:`17206`).

If this is claimed, then this needs tests for each type (e.g. html, pytables, prob some others).

@dhimmel pls post a new PR with this.

@gfyoung pls create a new issue to track this fix.
> pls post a new PR with this

Okay the actual scope of this PR is `read` methods that infer compression using `pandas.io.common._infer_compression`. Would that be appropriate to state in the What's New? 

Ideally, all read methods would use the common functions in `io.common`, but that's not the case (see https://github.com/pandas-dev/pandas/issues/15008). I don't really have time right now to go through all these read methods and see which are affected. I am interesting in consolidating the `read_*` code, but would prefer to wait till `pandas2`... since ensuring Python 2 support makes this work quite tedious.

Given these considerations, this PR's scope is to add path-like object support to `_infer_compression`. @jreback let me know how to update the What's New.