BrandonFors/pandas-github-issues huggingface dataset:
TST: add test for rolling max with DatetimeIndex
State: closed

- [x] closes #21096
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Window
description: rolling, ewma, expanding

Comments:
thanks @ganevgv 

I would love if you could parametrize some of these tests (e.g. also test min) in a followup if you can.
Thanks for the feedback @jreback, @WillAyd - created a new PR using parametrize over different operations and frequencies - #29932
TST: add test for unused level indexing raises KeyError
State: closed

- [x] closes #20410
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: MultiIndex

Comments:
thanks @ganevgv
ANN: types for _create_storer
State: closed

1) Annotate return type for _create_storer
2) Annotate other functions that we can reason about given 1
3) Fixups to address mypy warnings caused by 1 and 2

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@jbrockmendel looking good. `Union["GenericFixed", "Table"]` used 4 times. not sure we have a policy for when we should use an alias, i.e. number of occurrences vs complexity of type expression.
> used 4 times

In this case I think they're better kept as the Union.  In a perfect world id like to get separate functions one of which returns Table and the other returns GenericFixed, but i couldn't come up with a nice way of pre-checking
i think i merged in an odd order, pls rebase
merged lots of stuff, can you rebase just to be sure here.
rebased+green
k thanks
Remove Ambiguous Behavior of Tuple as Grouping
State: closed

follow up to #18314 

Labels:
name: Groupby
name: Deprecate
description: Functionality to remove in pandas

Comments:
thanks @WillAyd small followon request.
@WillAyd pls be sure to update #6581/#13777 if called for
@WillAyd looks like you marked the checkbox in #6581; the process is to move the entry to #13777 and add in parens the PR reference.

My mistake - should be updated now
PERF: speed-up when scalar not found in Categorical's categories
State: closed

I took a look at the code in core/category.py and found a usage of np.repeat for creating boolean arrays. np.repeat is much slower than np.zeros/np.ones for that purpose.

```python
>>> n = 1_000_000
>>> c = pd.Categorical(['a'] * n + ['b'] * n + ['c'] * n)
>>> %timeit c == 'x'
17 ms ± 270 µs per loop  # master
82.6 µs ± 488 ns per loop  # this PR
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Categorical
description: Categorical Data Type

Comments:
thanks @topper-123 any additional asv's for this / categorical stuff surely welcome.
io/parsers: ensure decimal is str on PythonParser
State: closed

- [x] closes #29650
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
Also can you add a whatsnew note for v1.0.0?
@WillAyd thanks for the feedback! 🎉 

Just pushed the changes you suggested and rebased master, please take another look.
ptal @WillAyd @jreback 

Thanks for the feedback!
thanks @fsouza 
🎉
TYP: disallow comment-based annotation syntax
State: closed


Labels:
name: Code Style
description: Code style, linting, code_checks
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@WillAyd happy to have specific questions in a followup for @simonjayhawkins 

can you rebase.
@simonjayhawkins one more rebase and merge on green.
sure
DOC: Add link to dev calendar and meeting notes
State: closed


Labels:
name: Docs

Comments:
great idea! merge on green.
Updated the Google calendar to be fixed to UTC, and updated the text in the docs.
Fixed the links and removed the tz from the URL
OK, merging this in. I'd recommend subscribing to this calendar for meeting changes. Right now just Joris and I can edit it. Happy to add anyone else.
DEPR: enforce deprecations for kwargs in factorize, FrozenNDArray.ser…
State: closed

…achsorted, DataFrame.update, DatetimeIndex.shift, TimedeltaIndex.shift, PeriodIndex.shift



Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Clean

Comments:
thanks @jbrockmendel 

with all of these, pls check them off / move when merged.
DEPR: remove Index fastpath kwarg
State: closed


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
@WillAyd are the mypy complaints actually caused by something here or just a coincidence?
Doesn't make sense on first glance; just restarted let's see
@simonjayhawkins any ideas whats going on with mypy here?
might be relevant to this issue @jbrockmendel 

https://github.com/python/mypy/issues/1174
> @simonjayhawkins any ideas whats going on with mypy here?

we had this issue before, see https://github.com/pandas-dev/pandas/pull/29205#issuecomment-546030059

did reach an agreement whether it is a bug or expected behaviour.

the change to the function signatures has triggered this.
> did reach an agreement whether it is a bug or expected behaviour.

suggestions on what to do here?  It looks like @TomAugspurger was working on making a fixture for just [Index, Series] at one point
> fixture for just [Index, Series] at one point

Do you mean a TypeVar?

I don't recall the details, but I think my preference was to add `type: ignore` to the lines in the test files that were causing the (maybe buggy) inference by mypy. But I didn't really understand the issue enough to say what the expected / right behavior is.
> lgtm. I see your typing TODOs, any objections to simply merging and fixing later? @simonjayhawkins

sure
Thanks @jbrockmendel
DEPR: enforce deprecations in core.internals
State: closed

Not sure if the whatsnew is needed for this; IIRC it was mainly pyarrow/fastparquet that were using these keywords

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
lgtm. rebase and merge.
the travis-36-locale build uses fastparquet=0.2.1 which apparently still passes dtype to make_block_same_class.  any objection to bumping the required version?
Looks like we last bumped the fastparquet version in #24590
Yes, I think bumping the minimum fastparquet version is fine. 
cc @TomAugspurger 
> Yes, I think bumping the minimum fastparquet version is fine.
> cc @TomAugspurger

agreed here as well.
Happy to bump.

On Wed, Nov 20, 2019 at 5:32 PM jbrockmendel <notifications@github.com>
wrote:

> Looks like we last bumped the fastparquet version in #24590
> <https://github.com/pandas-dev/pandas/pull/24590>
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/29723?email_source=notifications&email_token=AAKAOIT3P4CQOUA7FP5TQIDQUXCIJA5CNFSM4JPKHGJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEWEFYA#issuecomment-556548832>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKAOIXB2FXCSVDBUSTVBBDQUXCIJANCNFSM4JPKHGJQ>
> .
>

bumped fastparquet to 0.3.0.  Wasnt sure what to put in the whatsnew, so I copied the whole minimum-version section from 0.25.0, updated the fastparquet entry, and cut the sentence about py27 having just been dropped.
Looks like 0.3.2 is the version when fastparquet stopped using the deprecated keyword, but that is also the most recent version.  are we OK with bumping that far?
looks good, IIRC you have a nother PR where you add the Change column, can update then.
DEPR: remove encoding kwarg from read_stata, DataFrame.to_stata
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
cc @bashtage 
Lgtm. 
lgtm. rebase and merge.
Thanks @jbrockmendel
DEPR: remove deprecated keywords in read_excel, to_records
State: closed

There are a handful of these PRs on the way and they're inevitably going to have whatsnew conflicts.  I'm trying to group together related deprecations so as to have a single-digit number of PRs for these.

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
lgtm. needs a rebase
rebased+green
Thanks @jbrockmendel
DEPR: remove Series.from_array, DataFrame.from_items, as_matrix, asobject, as_blocks, blocks
State: closed


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Hello @jbrockmendel! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-11-20 15:14:37 UTC
lgtm. once rebased, pls merge. (for the deprecated removals PRs this is ok)
thanks
Improve return description in `droplevel` docstring
State: closed

Use https://pandas-docs.github.io/pandas-docs-travis/reference/api/pandas.DataFrame.dropna.html#pandas.DataFrame.dropna
docstring as a\ template.

MINOR DOCUMENTATION CHANGE ONLY (ignored below)

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Great thanks @deepyaman !
WIP: All the CI in GitHub actions (DO NOT MERGE)
State: closed

Many things are incomplete or broken here, but opening this so people can see the progress.

This needs to be discussed, and if we move forward splitted in many PRs, before it can be merged.

This adds all our current builds to GitHub actions, with two new features:
- Creating an artifact for the built web and docs
- A `Finalize` job is triggered when all the others are finished, that should summarize the errors in a GitHub comment (WIP)

Some things that are broken:
- The builds with locale different than English
- The grep that checks if there are ipython errors in the sphinx logs
- The builds other than Linux (because the matrix tries to create the containers for MySQL and PostgreSQL for all builds now)
- The `Finalize` job is WIP, and not sure if even triggered

I'll probably work on #29685 before adding any test build to actions. I think we can do much better in terms of simplicity and coverage with the creation of the matrix (which builds we run, and what's tested in each).

Labels:
name: CI
description: Continuous Integration
name: Needs Discussion
description: Requires discussion from core team before further action

Comments:
@datapythonista anything i/others can do to help with this?
> @datapythonista anything i/others can do to help with this?

So far I'm mainly experimenting, to see challenges and what can be done. There are several builds that are working now, but I'd probably wait for now, before moving anything. I'd probably start by the web+docs build, and the ones in travis, and those are the ones that are failing (besides the windows ones).

What's your idea? If you'd like to see things in GitHub actions soon, we can discuss and see how can split this already. Is there anything that you think we should prioritize?
I don't have any ideas, no, just was going through PRs and seeing where I might be useful
Closing this, was just for testing. Will propose new builds to GitHub actions incrementally.
TST: Silence lzma output
State: closed

This was leaking to stdout when the pytest `-s` flag was used.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
https://travis-ci.org/pandas-dev/pandas/jobs/614108628?utm_medium=notification&utm_source=github_status and https://travis-ci.org/pandas-dev/pandas/jobs/614108629?utm_medium=notification&utm_source=github_status are both the flaky clipboard test
thanks
format replaced with f-strings
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

ref #29547 

Labels:
name: Clean

Comments:
@datapythonista @WillAyd Great! Thanks for the approval. One more thing, I'm not sure what went wrong with the CI tests, but it doesn't seem related to these changes. What should I do?

Cheers
> One more thing, I'm not sure what went wrong with the CI tests, but it doesn't seem related to these changes. What should I do?

The affected builds are being flaky for unrelated reasons.  We're trying to fix them, and will probably ask you to rebase when that is done
> > One more thing, I'm not sure what went wrong with the CI tests, but it doesn't seem related to these changes. What should I do?
> 
> The affected builds are being flaky for unrelated reasons. We're trying to fix them, and will probably ask you to rebase when that is done

No problem!

alright can you rebase and we'll see if the CI goes through
> alright can you rebase and we'll see if the CI goes through

Rebased
thanks @lucassa3
REF: dont _try_cast for user-defined functions
State: closed

Also add a whatsnew note for #29641 and a comment in core.apply that ive been meaning to get in

Labels:
name: Groupby
name: Timezones
description: Timezone data dtype

Comments:
Can you add test(s) for this?
> Can you add test(s) for this?

This is turning out to be surprisingly difficult.  Which suggests its all-the-more worthwhile
> Can you add test(s) for this?

So far the answer is "no".  i haven't been able to find a case that fails in master but works in this PR.
How did you come across the behavior in the whatsnew?
> How did you come across the behavior in the whatsnew?

Thats the whatsnew note I forgot to include in #29641.
Ah OK. So not necessarily tied to the rest of the PR?
> So not necessarily tied to the rest of the PR?

Right.  ditto for the comment in core.apply
thanks
restore clipboard setup?
State: closed

@datapythonista do you recall why these lines were removed in https://github.com/pandas-dev/pandas/pull/28531/files?

Labels:
name: CI
description: Continuous Integration
name: Unreliable Test
description: Unit tests that occasionally fail

Comments:
We call xvfb with the command directly in https://github.com/pandas-dev/pandas/blob/master/ci/run_tests.sh#L33, we shouldn't need that. Is that causing any trouble?
OK, thanks. No idea if it's causing issues. I'll rerun the travis job that has been failing a few times to see if it's present on this branch.
Dunno if this is related, but I do see

> QXcbClipboard::setMimeData: Cannot set X11 selection owner

In the output log. I assume that the tests are run under the same `user` as everything else though, so not sure what that actually means.
Things passed with https://github.com/pandas-dev/pandas/pull/29695/commits/a93b9d8074a768f79b6da45eb4859a4efac421ed. Going to restart the problematic job a few times.

1 / 1
I've been going through logs. The "Cannot set X11 selection owner" warning seems relevant. It's been present in all the failed builds, and absent in all the successful ones.
I found this post very useful when dealing with clipboard test errors in the past, and trying to understand how things work: https://www.uninformativ.de/blog/postings/2017-04-02/0/POSTING-en.html

Still very confused, and I don't know how to fix that error yet, but I think the article is worth reading.
I'm wondering if we're copying and pasting too fast, and if a short sleep in between would get rid of the error...
I have one theory to try out before then. I'm worried about the module-level `to_clipboard` we attempt. That would be executed on import. Putting that behind a fixture.
With some luck #29712 may fix the problem. Some info here: http://elementalselenium.com/tips/38-headless

If that doesn't work, I'll probably focus on moving the db builds to GitHub actions, where I guess the X server is provided, we don't need `xvfb-run`, and I think the test won't fail.
Thanks. Hopefully https://github.com/pandas-dev/pandas/pull/29712 fixes it.
TST: Split pandas/tests/frame/test_indexing into a directory (#29544)
State: closed

- [x] closes #29544 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Split test cases in pandas/tests/frame/test_indexing into a sub directory (like pandas/tests/indexing). Still main file is pretty huge. Will update this PR with after suggestions and will fix style in a separate commit.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Hello @prakhar987! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-11-19 14:01:53 UTC
lgtm. can you merge master. ping on green.
@jreback done
thanks @prakhar987
BUG: Series groupby does not include nan counts for all categorical labels (#17605)
State: closed

- [x] closes #17605
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This is a simple, low-impact fix for #17605.

Labels:
name: Groupby
name: Categorical
description: Categorical Data Type

Comments:
FYI 

<details>
 <summary>Output of `python scripts/validate_docstrings.py pandas.core.groupby.GroupBy._reindex_output`</summary>
################################################################################
########### Docstring (pandas.core.groupby.GroupBy._reindex_output)  ###########
################################################################################

If we have categorical groupers, then we might want to make sure that
we have a fully re-indexed output to the levels. This means expanding
the output space to accommodate all values in the cartesian product of
our groups, regardless of whether they were observed in the data or
not. This will expand the output space if there are missing groups.

The method returns early without modifying the input if the number of
groupings is less than 2, self.observed == True or none of the groupers
are categorical.

Parameters
----------
output : Series or DataFrame
    Object resulting from grouping and applying an operation.
fill_value : scalar, default np.NaN
    Value to use for unobserved categories if self.observed is False.

Returns
-------
Series or DataFrame
    Object (potentially) re-indexed to include all possible groups.

################################################################################
################################## Validation ##################################
################################################################################

1 Errors found:
	Summary should fit in a single line
2 Warnings found:
	See Also section not found
	No examples section found

</details>


Thanks for the great feedback so far @TomAugspurger, @WillAyd, @jreback! I've made the requested changed, feel free to re-review.
thanks @OliverHofkens nice patch!
CI: Fixing error in code checks in GitHub actions
State: closed

xref https://github.com/pandas-dev/pandas/pull/29546#issuecomment-554807243

Looks like the code in one of the checks in `ci/code_checks.sh` uses another encoding for spaces and other characters. I guess it was generated from Windows, Azure-pipelines is not sensitive to the different encoding, but GitHub actions is.

Probably worth doing some more research and adding a check that makes sure this new encoding is not introduced anymore.

CC: @gfyoung @jreback 

Labels:
name: CI
description: Continuous Integration

Comments:
@datapythonista : If it doesn't produce the error in the output, LGTM.

Do you know why the [pipeline](https://github.com/pandas-dev/pandas/runs/306852441#step:3:36) didn't fail despite the encoding issue?
It doesn't produce the error, that step at least works as expected.

We usually run scripts with `set -e` so at any step it fails. In this case we can't do that because we don't want to fail at the first error, but report all them. We control the exit code of the script with `$RET`. If one step fails (fails the way we expect it to fail) we increment `$RET`, and the script will fail. But in this case:
- The first error was the `echo $MSG`, which we don't control if it fails or not
- The second failed, and is the one we cared
- But the third failed again, which was the one trying to increment `$RET`, so `$RET` stayed at zero, the script continued, and at the end if was successful

Not perfect, but I don't think we can do any better.
> Not perfect, but I don't think we can do any better.

Perhaps, but good enough for now to put the explanation out there so someone can prove otherwise if there is in fact a way to improve upon this.
Bug fix GH 29624: calling str.isalpha on empty series returns object …
State: closed

…dtype, not bool

Added dtype=bool argument to make _noarg_wrapper() return a bool Series

- [x] closes #29624  
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Strings
description: String extension data type and string data

Comments:
thanks @ellequelle very nice!
CI: Use bash for windows script on azure
State: closed

- Towards #26344
- This makes `windows.yml` and `posix.yml` steps pretty similar and they could potentially be merged/reduce duplication? (Will leave this for a separate PR)
- We also now use `ci/run_tests.sh` for windows test stage.

Reference https://github.com/pandas-dev/pandas/pull/27195 where I began this.

Labels:
name: CI
description: Continuous Integration

Comments:
Thanks @alimcmaster1
CI: Pin black to version 19.10b0
State: closed

This will pin black version in ci and for local dev.
(This will avoid code checks failing when a new black version is released)

As per comment from @jreback here https://github.com/pandas-dev/pandas/pull/29508#discussion_r346316578

Labels:
name: CI
description: Continuous Integration
name: Code Style
description: Code style, linting, code_checks
name: Dependencies
description: Required and optional dependencies

Comments:
sure 
ping on green
thanks @alimcmaster1