Pass kwargs from read_parquet() to the underlying engines.
State: closed

This allows e.g. to specify filters for predicate pushdown to fastparquet.
This is a followup to #18155/#18154

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: IO Parquet
description: parquet, feather

Comments:
Hello @Corni! Thanks for updating the PR.





Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on November 14, 2017 at 13:47 Hours UTC
@hoffmann Is this what you intended?
need tests
@wesm @cpcloud @martindurant

is there a reasonable API that we could expose here to users to facilitate some sort of row group filttering (w/o resorting to a full query language or multiple functions)?
@jreback https://github.com/dask/fastparquet/blob/master/fastparquet/api.py#L296-L300 sounds reasonable. We should be able to implement that in `pyarrow` quite easily.
Created https://issues.apache.org/jira/browse/ARROW-1796 and https://issues.apache.org/jira/browse/PARQUET-1158 to do this in `pyarrow`.
@jreback How do you imagine tests for this feature, which would not rely on testing backend-specific parameters and their behauviour?
@Corni actually I would have a couple of tests that pass thru kwargs specifically for the backend (IOW separate tests). just to make sure things are passed thru. e.g. write a file with row_groups and exercise reading with row_groups. (the kwargs for this type of predicate pushdown hopefully will be synchronized at some point, but for now you can have tests for each engine separately).
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=h1) Report
> Merging [#18216](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/9050e38bf8afea458a531b7af5b6b00b892f2b57?src=pr&el=desc) will **decrease** coverage by `0.06%`.
> The diff coverage is `80%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/18216/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #18216      +/-   ##
==========================================
- Coverage   91.42%   91.36%   -0.07%     
==========================================
  Files         163      164       +1     
  Lines       50068    49884     -184     
==========================================
- Hits        45777    45575     -202     
- Misses       4291     4309      +18
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.16% <80%> (-0.06%)` | :arrow_down: |
| #single | `39.42% <40%> (-1%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/parquet.py](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnF1ZXQucHk=) | `65.38% <80%> (ø)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/io/clipboard/clipboards.py](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree#diff-cGFuZGFzL2lvL2NsaXBib2FyZC9jbGlwYm9hcmRzLnB5) | `24.05% <0%> (-2.54%)` | :arrow_down: |
| [pandas/tseries/frequencies.py](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvZnJlcXVlbmNpZXMucHk=) | `94.09% <0%> (-1.92%)` | :arrow_down: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `63.44% <0%> (-1.77%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.8% <0%> (-0.1%)` | :arrow_down: |
| [pandas/core/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY2F0ZWdvcmljYWwucHk=) | `95.75% <0%> (-0.05%)` | :arrow_down: |
| [pandas/core/indexes/timedeltas.py](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy90aW1lZGVsdGFzLnB5) | `91.14% <0%> (-0.04%)` | :arrow_down: |
| [pandas/core/dtypes/concat.py](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2NvbmNhdC5weQ==) | `99.13% <0%> (-0.02%)` | :arrow_down: |
| [pandas/core/api.py](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXBpLnB5) | `100% <0%> (ø)` | :arrow_up: |
| ... and [19 more](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=footer). Last update [9050e38...1c045a5](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=h1) Report
> Merging [#18216](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/9050e38bf8afea458a531b7af5b6b00b892f2b57?src=pr&el=desc) will **decrease** coverage by `0.06%`.
> The diff coverage is `83.33%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/18216/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #18216      +/-   ##
==========================================
- Coverage   91.42%   91.36%   -0.07%     
==========================================
  Files         163      164       +1     
  Lines       50068    49880     -188     
==========================================
- Hits        45777    45571     -206     
- Misses       4291     4309      +18
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.16% <83.33%> (-0.06%)` | :arrow_down: |
| #single | `39.42% <33.33%> (-1%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/parquet.py](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnF1ZXQucHk=) | `65.38% <83.33%> (ø)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/io/clipboard/clipboards.py](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree#diff-cGFuZGFzL2lvL2NsaXBib2FyZC9jbGlwYm9hcmRzLnB5) | `24.05% <0%> (-2.54%)` | :arrow_down: |
| [pandas/tseries/frequencies.py](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvZnJlcXVlbmNpZXMucHk=) | `94.09% <0%> (-1.92%)` | :arrow_down: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `63.44% <0%> (-1.77%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.8% <0%> (-0.1%)` | :arrow_down: |
| [pandas/core/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY2F0ZWdvcmljYWwucHk=) | `95.75% <0%> (-0.05%)` | :arrow_down: |
| [pandas/core/indexes/timedeltas.py](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy90aW1lZGVsdGFzLnB5) | `91.14% <0%> (-0.04%)` | :arrow_down: |
| [pandas/core/indexes/multi.py](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9tdWx0aS5weQ==) | `96.38% <0%> (-0.02%)` | :arrow_down: |
| [pandas/core/dtypes/concat.py](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2NvbmNhdC5weQ==) | `99.13% <0%> (-0.02%)` | :arrow_down: |
| ... and [21 more](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=footer). Last update [9050e38...edbd937](https://codecov.io/gh/pandas-dev/pandas/pull/18216?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

I included now I included now a test for filtering rowgroups with fastparquet, though there is no kwargs in pyarrow yet which I could include in a test (see https://arrow.apache.org/docs/python/generated/pyarrow.parquet.read_table.html), so I'm not testing the pyarrow yet.
@Corni thanks, just the minor test issue otherwise like I intended it.
lgtm. ping on green.

@wesm @xhochy @martindurant
+1
ping :)
Can you update the docstring for this new functionality?
> Can you update the docstring for this new functionality?

No matter, it already seems to (incorrectly) have been there ..
@Corni Thanks!