BrandonFors/pandas-github-issues huggingface dataset:
ENH: Add query functionality to skiprows param in read_csv
State: closed

This allows users to pass in strings to query through the ``skiprows`` parameter in ``pandas.read_csv``. This is done through querying the Dataframe as soon as it is created, and returning a Dataframe with the desired columns to the user. 

Included in this PR are the relevant whatsnew entry, documentation changes, and new tests.

- [X] closes #32072
- [x] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry

Labels:
name: API Design
name: IO CSV
description: read_csv, to_csv

Comments:
Agreed that we shouldn't proceed in the current form. Passing a string expression implies (at least to me) that the filtering happens before the data is loaded into a DataFrame.
Skiprows being separate makes sense to me. In the latest change I added a new parameter ``conditionalrows`` that performs the query functionality instead. I feel this is a nice thing for users to have when reading in data, so hopefully this is a better integration of the idea.

If it's still a better idea to separate query from read_csv entirely, I can take a look at improving the documentation instead.
I don't think we should add any kind of "query" functionality to `read_csv` unless it's reducing the peak memory usage. For anything that's equivalent to `read_csv(...).query(...)` (i.e. read all the data, construct the DataFrame, then query) the user should just write that.
I think an example in the docs or cookbook along the lines of 

```result = pd.concat([df.query(...) for df in pd.read_csv(...., chunksize=...)], ignore_index=True)```

is ok
> Skiprows being separate makes sense to me. In the latest change I added a new parameter `conditionalrows` that performs the query functionality instead. I feel this is a nice thing for users to have when reading in data, so hopefully this is a better integration of the idea.
> 
> If it's still a better idea to separate query from read_csv entirely, I can take a look at improving the documentation instead.

I totally agree on this. Remaining `skiprows` separate and adding a new query param is a lot nicer and reasonable. I think it'd be great to support query-like param while reading the data. I know pretty much `df.read_csv(..., chunksize=...).query(...)` does the trick, but this query-like feature would be nice for users.
closing this as wont fix
but would take a doc example if you want
CI: Not deleting docs symlink in prod
State: closed

In #33341 I forgot to exclude the `docs` symlink for deletion, meaning that it'll be deleted when a PR is merged. And the docs will be unavailable. Fixing it here. 

@pandas-dev/pandas-core can we prioritize this please.

Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
Thanks. Feel free to self-merge hotfix things like this I think.
PERF: statically define classes for is_dtype checks
State: closed

This moves the definition of the functions passed to `_is_dtype_type` from dynamically generated functions is `is_integer_dtype` to top-level functions. I can't run asv right now (https://github.com/pandas-dev/pandas/issues/33315#issuecomment-610329344) but here are some timeits

master

```python
In [2]: t = np.dtype('int64')

In [3]: %timeit pd.api.types.is_integer_dtype(t)
2.22 µs ± 51.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
```
This PR

```python
In [3]: %timeit pd.api.types.is_integer_dtype(t)
1.54 µs ± 39.6 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
in fact could simply instantiate classes 
Nice, I didnt expect `classes` to account for that much overhead.

> I can't run asv right now ([#33315 (comment)](https://github.com/pandas-dev/pandas/issues/33315#issuecomment-610329344)) but here are some timeits

Hmm don't think that issue is the root of any ASV issues - are you getting any other error messages from the run? Maybe worth opening a dedicated issue
Which logic would you move? It seems like `_is_dtype_or_type` requires a callable, and I want to keep the definition of that callable out of e.g. `is_bool_dtype`.
@TomAugspurger is this still something you want to merge, or is it superceded by #33400?
Probably a non-issue (or less pressing) after #33400 is done. I'll re-run
benchmarks after that's merged.

On Thu, Apr 16, 2020 at 12:23 PM jbrockmendel <notifications@github.com>
wrote:

> @TomAugspurger <https://github.com/TomAugspurger> is this still something
> you want to merge, or is it superceded by #33400
> <https://github.com/pandas-dev/pandas/pull/33400>?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/33364#issuecomment-614786585>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKAOISFT56BCX3OG7EYTJ3RM45J5ANCNFSM4MDBB6BQ>
> .
>

needs rebase
ok with this if we can update to comments.
conceptually ok with this if you can update to comments
closing as stale
DEP: Bump min version of dateutil to 2.7.3
State: closed

- [x] xref https://github.com/pandas-dev/pandas/pull/32465#discussion_r404449469
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

---

Not sure where to put the note in the "whatsnew"

Labels:
name: Dependencies
description: Required and optional dependencies

Comments:
In 1.0, we had dependency bumps under the "Backwards incompatible" section. https://pandas.pydata.org/docs/whatsnew/v1.0.0.html#increased-minimum-versions-for-dependencies. We can do that again, or make a new top-level section.
> In 1.0, we had dependency bumps under the "Backwards incompatible" section.

so, by definition, this is an breaking change? is this necessary or can it keep till 2.0?
> > In 1.0, we had dependency bumps under the "Backwards incompatible" section.
> 
> so, by definition, this is an breaking change? is this necessary or can it keep till 2.0?

Haven't thought of that.

---

I'll push more commits when someone from the pandas-core team will tell me if this PR is relevant :)
I wouldn't consider this an API breaking change. That's just the header it went under in older versions.
> Can you have a look at the CI and see what's wrong please?

@datapythonista I have found out that there is a bug in 2.7.0 that was fixed in 2.7.3 when parsing a ```NaN``` (ref https://github.com/dateutil/dateutil/issues/662)
thanks @MomIsBestFriend 

as a followon can you see if we can change code for anything that needed < 2.7.3 (i don't know if there is anything, but its possible).
TST: iat with duplicate column names
State: closed

- [x] closes #11754
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
ping on green
ping @simonjayhawkins 
Thanks @MomIsBestFriend
BUG/REF: unstack with EA dtypes
State: closed

1) `ser.unstack(args)` should behave like `ser.to_frame().unstack(args).droplevel(level=0, axis=1)`.  This fails on master, is fixed by this PR.
2) `reshape._unstack_extension_series` currently re-implements logic that is in `ExtensionBlock._unstack`.  This de-duplicates by dispatching.
3) We currently transpose in cases where we shouldn't.  This implements `DataFrame._can_fast_transpose` to avoid that.
4) `test_astype_object_series` has what looks like a typo that is making us test not-the-intended-thing; @TomAugspurger can you confirm im reading this right?
5) The dtype comparisons in `test_astype_object_frame` currently raise, are commented out for exposition, should be fixed in a separate PR.

Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
> How do the unstack asvs look here?

with "-b reshape" no change
thanks nice de-duplicating
REF: remove BlockManager.set
State: closed

xref #33052, #33332.

This touches several DataFrame/NDFrame methods.  Those methods need a refactor in part because the ones defined on NDFrame are only ever called for DataFrame.  Will refactor in follow-up.

Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
updated to remove the need for `clear` kwarg, so we end up not needing to touch DataFrame methods at all
thanks!
BUG: Timestamp comparison with ndarray[dt64]
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug

Comments:
@jreback gentle ping; this is a blocker for simplifiying BlockManager._replace_list
TST: add read_json test #GH32383
State: closed

- [x] closes #32383
- [x] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
thanks @BenjaminLiuPenrose
BUG: scalar indexing on 2D DTA/TDA/PA
State: closed

cc @jorisvandenbossche this fixes the issue with `DatetimeLikeBlockMixin.iget` discussed in #33252

Note: this sits on top of #33290.

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
@jbrockmendel now #33290 is merged, this can be updated / merged?
> this can be updated / merged?

This is good to go
CI: Sync web and dev docs automatically with prod server
State: closed

- [X] closes #28528

This forgets about the OVH server, and after every merge to master will synchronize the repo web and docs in the production server.

**Note** that this will delete any file in the server that is not known, except the cheat sheets, and all the docs versions in `pandas-doc`. I don't think anything else should be kept, but if there is anything else that we upload manually, please let me know. I made a backup in the server, so if we delete anything by accident with this, it can be easily restored.

CC: @TomAugspurger @jorisvandenbossche 

Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
Looks good to me. 

But not very familiar with the server, to know whether we uploaded things manually in the past. 
> delete any file in the server that is not known

Starting in what directory? `/usr/share/nginx/` or `/usr/share/nginx/pandas`?
I suppose the diff answers my question: `nginx/pandas`. So this looks good.

Merge when you're ready.
Yes, `nginx/pandas`. Thanks for the reviews.
BUG: Don't raise on value_counts for empty Int64
State: closed

- [ ] closes #33317
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
thanks @dsaxton 

we don't have a full fledged ExtensionIndex yet so anytime an EA is set-index its object
DOC/PLT: Add `stacked` in doc and doc example for barh and bar plot
State: closed

- [x] closes #32759
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs
name: Visualization
description: plotting

Comments:
Hello @charlesdong1991! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-05-05 14:36:22 UTC
would appreciate a lot for some reviews, thanks! ^^ @jreback @WillAyd 
Can you post a screenshot of the built docs? Doesn't this place the argument out of order (i.e. after the Returns section)?

@datapythonista 
Hi, @WillAyd thanks for your review!

You are right, it is not the right place for parameters. And I removed it because i noticed another PR #33596 has included `stacked` also. So might be better to let that PR deal with parameter doc.

So the remaining question is the code example, I think it would be nice for users to see the example. But if you think it is not needed because its quite straightforward, then let's close this PR. I am open to hearing from your preference!
ping @WillAyd 

pls let me know if it is still worth it to have this one in doc? ^^
Thanks @charlesdong1991
PLT: Order of plots does not preserve the column orders in df.hist
State: closed

- [x] closes #29235
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Visualization
description: plotting

Comments:
could you please take a look? ^^  @WillAyd @jreback 

thanks! 
thanks for your review! @WillAyd I rephrase the whatsnew note

the build failure is unrelated

I will take a look at the other use of `try_sort` and see if it is needed, if not needed, I will deprecate it @WillAyd 
ping

seems it is the azure machine issue @WillAyd @jreback 
ping @WillAyd @jreback 
Great thanks @charlesdong1991
BLD: Increase minimum version of Cython to 0.29.16
State: closed

- In particular, this fixes a bug in code returning ctuples
    - cython/cython#2745 
    - cython/cython#1427
- This is a prereq for #33220

- [x] whatsnew entry

Labels:
name: Dependencies
description: Required and optional dependencies

Comments:
I think you need to bump `min_cython_ver` in `setup.py` and the `pyproject.toml`
also if you can PR to pandas-wheels repo:  https://github.com/MacPython/pandas-wheels/blob/master/azure/posix.yml and https://github.com/MacPython/pandas-wheels/blob/master/azure/windows.yml (these 2 files).
@jreback Tests green, PR in wheels repo opened
thanks @mgmarino (and for the pandas-wheels one)
CLN: Static types in `pandas/_lib/lib.pyx`
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Clean

Comments:
Thanks @MomIsBestFriend
DOC: Fix examples in `pandas/core/strings.py`
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
LGTM
Thanks @MomIsBestFriend
CI: Add argument doc/source/development to formatting docstrings in code_checks.sh
State: closed

Regarding issue #32550

Labels:
name: Docs
name: Clean

Comments:
Thanks for the PR @cleconte987 

I think the changes to both files are independent. Can you create separate PRs for them, and for the changes to the Python script add a description on what you're proposing please?
No sorry.. I didn't mean to change again 'validate_rst_title_capitalization.py'. I will do another PR.
TST: Don't use 'is' on strings to avoid SyntaxWarning
State: closed

This avoids the below warning (in Python 3.8, [user-visible in Debian](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=956021) because they byte-compile everything on install), and possibly unspecified behaviour (though I haven't seen that in practice).

```
  /usr/lib/python3/dist-packages/pandas/tests/frame/test_alter_axes.py:241: SyntaxWarning: "is" with a literal. Did you mean "=="?
    False if (keys[0] is "A" and keys[1] is "A") else drop  # noqa: F632
```



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
thanks @rebecca-palmer
Changed files permissions to be the same
State: closed

Not 100% sure about this change, feel free to close at anytime.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
this correct, all files *should* be 644, actually we could use a code check for this to avoid accidently setting it.
> this correct, all files _should_ be 644, actually we could use a code check for this to avoid accidently setting it.

@jreback All files? including ```setup.py```, ```test.sh```, ```release_stats.sh``` etc..?

---

Writing a code check for this is simple, have exclusions might be tricker
DOC: do not include type hints in signature in html docs
State: closed

See https://github.com/pandas-dev/pandas/issues/33025. Removing the type hints in the online docs until we have a better solution to improve readability.

Labels:
name: Docs

Comments:
Can you post a screenshot of the output? The types are just completely hidden from the signature?
It simply removed type hints from the signature, so you get

![image](https://user-images.githubusercontent.com/1020496/78566835-1a139900-7820-11ea-9bc9-380bdfae4e49.png)

instead of

![image](https://user-images.githubusercontent.com/1020496/78566880-27308800-7820-11ea-9211-21b360ce0c7b.png)

using the example from https://github.com/pandas-dev/pandas/issues/33025 (but there are other pages that have much more elaborate type hints, where the improvement in readability is bigger)
lgtm
LGTM
I think @WillAyd had thoughts in the issue. Are you OK with disabling them until we can provide a better UI for controlling whether you want to see them or not?
lgtm
Thanks for confirming.
DOC: include Offset.__call__ to autosummary to fix sphinx warning
State: closed


Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
@datapythonista is this the right way to fix the doc build?
I'm not sure, sphinx still confuses me with those. @jorisvandenbossche knows better.
It clearly fixes it, because CI is green ;)

Now, the actual answer: it depends. Are we fine with including the `__call__` docstring? Is this considered public / documented in the user guide?
> It clearly fixes it, because CI is green ;)
> 
> Now, the actual answer: it depends. Are we fine with including the `__call__` docstring? Is this considered public / documented in the user guide?

these *are* public methods.
Then the fix is fine
Thanks @jbrockmendel !
@Meeseeksdev backport to 1.0.x
ENH: Named aggregations with multiple columns
State: closed

- [ ] closes #29268
- [x] tests added
  - [ ] tests passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Enhancement
name: Groupby
name: API Design

Comments:
Hello @fpunny! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-04-08 02:24:07 UTC
Thanks @fpunny for the PR. can you please add tests and a whatsnew and update docstrings/examples etc so users would know how to use this functionality.
@simonjayhawkins Hi simon, we created the whatsnew entry, added some test cases, and passed all of the checks. This being our first enhancement for Pandas, we're hoping to get some feedback on our implementation. We're fairly new to the codebase, hence we would love to learn a bit more about how we can improve - in hopes of merging this
This MR seems inactive, how complex is this issue to pick up as a first issue? @jreback 
this particular issue is a bit complex

however my comments above about simplifying the existing code could be a good first issue
Closing as I think this has gone stale, but @fpunny ping if you'd like to pick back up
@WillAyd If OP does not have time, I can give this a try, not sure if I can make it, but I would like to see how far I come. 
Sounds good! Feel free to clone this and push a new PR
REGR: Fix construction of PeriodIndex from strings
State: closed

- [ ] closes #26109
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Period
description: Period data type
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
Thanks for the feedback @simonjayhawkins . Do you know by chance what's happening in the CI? The error is cryptic and seems unrelated: https://dev.azure.com/pandas-dev/e01c9671-576d-40f3-a1a1-7bef78604087/_apis/build/builds/32518/logs/72
The failurs actually seem related to me. Eg on one of the builds (https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=32640&view=logs&jobId=a67b4c4c-cd2e-5e3c-a361-de73ac9c05f9&j=a67b4c4c-cd2e-5e3c-a361-de73ac9c05f9&t=33d2fdd0-c376-5f94-e6d3-957bdd23a3b8) I see 

```
==================================== ERRORS ==================================== ERRORS ====================================
______________ ERROR collecting pandas/tests/indexes/test_base.py ______________
pandas/tests/indexes/test_base.py:46: in <module>
    class TestIndex(Base):
        tm.assert_index_equal(PeriodIndex(idx.values), idx)
        tm.assert_index_equal(PeriodIndex(list(idx.values)), idx)
    
        msg = "freq not specified and cannot be inferred"
        with pytest.raises(ValueError, match=msg):
            PeriodIndex(idx.asi8)
        with pytest.raises(ValueError, match=msg):
            PeriodIndex(list(idx.asi8))
    
        msg = "'Period' object is not iterable"
        with pytest.raises(TypeError, match=msg):
            PeriodIndex(data=Period("2007", freq="A"))
    
>       result = PeriodIndex(iter(idx))

pandas/tests/indexes/period/test_constructors.py:158: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/core/indexes/period.py:215: in __new__
    data = period_array(data=data, freq=freq)
pandas/core/arrays/period.py:855: in period_array
    return PeriodArray._from_sequence(data, dtype=dtype)
pandas/core/arrays/period.py:197: in _from_sequence
    freq = freq or libperiod.extract_freq(periods)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   def extract_freq(ndarray[object] values):
E   ValueError: Buffer has wrong number of dimensions (expected 1, got 0)

pandas/_libs/tslibs/period.pyx:1469: ValueError

```

which is at least period related.
thanks @dsaxton