BrandonFors/pandas-github-issues huggingface dataset:
CLN: Avoid importing Series in core.aggregation
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Breaks import cycles allowing aggregation to be imported at the top.

Labels:
name: Clean
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
thanks @rhshadrach
REGR: DataFrame.apply() with raw option and func returning string
State: closed

- [ ] closes #35940
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
thanks @simonjayhawkins nicely documented work-around
@meeseeksdev backport 1.1.x
CI: rerun flaky test in html #36467
State: closed

Part of #36467


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
Hmm so is this test failing or just sometimes leaking a resource? If the latter it seems like there is an issue elsewhere we need to find
> Hmm so is this test failing or just sometimes leaking a resource? If the latter it seems like there is an issue elsewhere we need to find

The coverage build generates many many ResourceWarnings (I wonder if it has something to do with the coverage plugin). This test just happens to have an assert_produces_warning in it. Therefore it's the only one that failed. 
> This test just happens to have an assert_produces_warning in it.

We have quite a few tests that would do this though, no? 
> > Hmm so is this test failing or just sometimes leaking a resource? If the latter it seems like there is an issue elsewhere we need to find
> 
> The coverage build generates many many ResourceWarnings (I wonder if it has something to do with the coverage plugin). This test just happens to have an assert_produces_warning in it. Therefore it's the only one that failed.

happy to try to fix that, but is this really the fix?
I'll take a deeper look at this. Close for now.
CLN: assorted
State: closed

- [x] closes #26982
- [x] closes #22373
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs
name: Clean

Comments:
lgtm. @ivanovmg has a comment 
reverted non-unanimous bit
lgtm.

whats up with the py38_np_dev build, any idea?
> whats up with the py38_np_dev build, any idea?

Looks like a new exception message
BUG: propagate dropna in pd.Grouper
State: closed

- [x] closes #36620
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

A precursor to #35751

Labels:
name: Bug
name: Groupby

Comments:
I noticed that the `Grouper` docstring didn't have a `dropna` arg so I added that here.

I said `added in 1.2.0` - a bit of a slight of hand since it the attribute was already there in the code in `Grouper` (but it didn't exist/propagate in `BaseGrouper` so it was unusable)
The test is fine it was a general question in the PR. Was addressed by Jeff as well so otherwise lgtm

Sent from my iPhone

> On Sep 25, 2020, at 11:49 AM, Richard Shadrach <notifications@github.com> wrote:
> 
> ﻿
> @rhshadrach commented on this pull request.
> 
> In pandas/tests/groupby/test_groupby_dropna.py:
> 
> > @@ -162,6 +162,13 @@ def test_groupby_dropna_series_by(dropna, expected):
>      tm.assert_series_equal(result, expected)
>  
>  
> +@pytest.mark.parametrize("dropna", (False, True))
> +def test_grouper_dropna_propagation(dropna):
> +    df = pd.DataFrame({"A": [0, 0, 1, None], "B": [1, 2, 3, None]})
> +    gb = df.groupby("A", dropna=dropna)
> +    assert gb.grouper.dropna == dropna
> @WillAyd maybe you're thinking we should only be testing the public API and not any internals?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

ok this is fine
numpy version in py36_locale_slow_old_np on 1.1.x
State: closed

PR against 1.1.x

not sure if important but numpy version in https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=43144&view=logs&jobId=a69e7846-138e-5465-0656-921e8964615b&j=a69e7846-138e-5465-0656-921e8964615b&t=56da51de-fd5a-5466-5244-b5f65d252624 is 1.19.2

xref #33729

Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
marking as ready for review.

test failure unrelated and numpy 1.15.4 used in py36_locale_slow_old_np and passing
LGTM
failure unrelated. see https://github.com/pandas-dev/pandas/pull/36396#issuecomment-695195148
Partial Revert "ENH: infer freq in timedelta_range (#32377)"
State: closed

closes #35897

maybe alternative to #36582 if no changes since #32377 now fail (not tested locally)

cc @phofl @jbrockmendel 

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Timedelta
description: Timedelta data type

Comments:
> maybe alternative to #36582 if no changes since #32377 now fail (not tested locally)

looks like maybe just a doctest added since.

the release note will need to explain that the enhancement has been reverted, so a proper fix maybe more preferable.

will leave this a draft for now.
LGTM
@simonjayhawkins yep looks good here too; if you can add a release note and merge master; ping on green (and you are in draft status)
Hello @simonjayhawkins! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-09-25 18:36:20 UTC
ci restarted

```
=========================== short test summary info ===========================
FAILED pandas/tests/plotting/test_groupby.py::TestDataFrameGroupByPlots::test_series_groupby_plotting_nominally_works
= 1 failed, 75523 passed, 1809 skipped, 1165 xfailed, 67 warnings in 1223.71s (0:20:23) =
```
> @simonjayhawkins yep looks good here too; if you can add a release note and merge master; ping on green (and you are in draft status)

@jreback green
3rd time the charm, thanks @simonjayhawkins 
@meeseeksdev backport 1.1.x
CI: Add rst backtick checker
State: closed

Adding a pre-commit hook for detecting single backticks around code in RST files. Running for instance on the v1.2.0 whatsnew shows errors:
```
doc/source/whatsnew/v1.2.0.rst:120:- `Styler` now allows direct CSS class name addition to individual data cells (:issue:`36159`)
doc/source/whatsnew/v1.2.0.rst:225:- Performance improvements when creating DataFrame or Series with dtype `str` or :class:`StringDtype` from array with many string elements (:issue:`36304`, :issue:`36317`, :issue:`36325`, :issue:`36432`)
doc/source/whatsnew/v1.2.0.rst:230:- Performance improvement in :meth:`pd.to_datetime` with non-`ns` time unit for `float` `dtype` columns (:issue:`20445`)
doc/source/whatsnew/v1.2.0.rst:265:- Bug in :func:`date_range` was raising AmbiguousTimeError for valid input with `ambiguous=False` (:issue:`35297`)
doc/source/whatsnew/v1.2.0.rst:306:- Bug in :meth:`SeriesGroupBy.transform` now correctly handles missing values for `dropna=False` (:issue:`35014`)
doc/source/whatsnew/v1.2.0.rst:312:- Bug in :meth:`DataFrame.xs` when used with :class:`IndexSlice` raises ``TypeError`` with message `Expected label or tuple of labels` (:issue:`35301`)
doc/source/whatsnew/v1.2.0.rst:320:- In :meth:`read_csv` `float_precision='round_trip'` now handles `decimal` and `thousands` parameters (:issue:`35365`)
doc/source/whatsnew/v1.2.0.rst:322:- :meth:`to_csv` passes compression arguments for `'gzip'` always to `gzip.GzipFile` (:issue:`28103`)
doc/source/whatsnew/v1.2.0.rst:324:- :meth:`to_csv` and :meth:`read_csv` did not honor `compression` and `encoding` for path-like objects that are internally converted to file-like objects (:issue:`35677`, :issue:`26124`, and :issue:`32392`)
doc/source/whatsnew/v1.2.0.rst:327:- Bug in :meth:`read_csv` with `engine='python'` truncating data if multiple items present in first row and first element started with BOM (:issue:`36343`)
doc/source/whatsnew/v1.2.0.rst:328:- Removed ``private_key`` and ``verbose`` from :func:`read_gbq` as they are no longer supported in `pandas-gbq` (:issue:`34654` :issue:`30200`)
```

~This is going to fail for now so need to figure out how to make it pass.~

Labels:
name: CI
description: Continuous Integration
name: Code Style
description: Code style, linting, code_checks

Comments:
nitpicks, generally looks good
CLN: clean up new detected trailing whitespace
State: closed

- [N/A] closes #xxxx (xref [#36386 (comment)](https://github.com/pandas-dev/pandas/pull/36386#discussion_r493865987))
- [N/A] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [N/A] whatsnew entry

Fixed the remaining trailing whitespace problems that weren't being found by `ci/code_checks.sh` because of the file type filters.


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Would it work to add the `trailing-whitespace` hook from https://github.com/pre-commit/pre-commit-hooks ?
Might need some of the same exclusions as for the end-of-file-fixer (maybe just svg and html?)
> Might need some of the same exclusions as for the end-of-file-fixer (maybe just svg and html?)

Yeah just noticed that after pushing, was going to fix but something else came up. Should be all done now 🙂
Thanks @plammens !
TST: 32bit dtype compat #36579
State: closed

Part of #36579



Labels:
name: 32bit
description: 32-bit systems

Comments:
Hello @fangchenli! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-09-24 16:57:24 UTC
thanks!

do we have a 32-bit build on CI? e.g. where were these failing
> thanks!
> 
> do we have a 32-bit build on CI? e.g. where were these failing

The 32-bit build is still work in progress https://github.com/pandas-dev/pandas/pull/35898. It's working but not integrated with our existing setup yet. I also saw those failures on Raspberry Pi.
BUG: use cmath to test complex number equality in pandas._testing
State: closed

- [x] closes #28235
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Adds `cmath`-based equality testing for complex numeric types (`complex`, `np.complex64` and `np.complex128`) 

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Complex
description: Complex Numbers

Comments:
Travis failure looks unrelated
Green barring unrelated doctest failure:
```
=================================== FAILURES ===================================
_________________ [doctest] pandas.core.reshape.pivot.crosstab _________________
549     Examples
550     --------
551     >>> a = np.array(["foo", "foo", "foo", "foo", "bar", "bar",
552     ...               "bar", "bar", "foo", "foo", "foo"], dtype=object)
553     >>> b = np.array(["one", "one", "one", "two", "one", "one",
554     ...               "one", "two", "two", "two", "one"], dtype=object)
555     >>> c = np.array(["dull", "dull", "shiny", "dull", "dull", "shiny",
556     ...               "shiny", "dull", "shiny", "shiny", "shiny"],
557     ...              dtype=object)
558     >>> pd.crosstab(a, [b, c], rownames=['a'], colnames=['b', 'c'])
Differences (unified diff with -expected +actual):
    @@ -1,5 +1,6 @@
    -b   one        two
    +Index(['__dummy__'], dtype='object')
    +b    one        two      
     c   dull shiny dull shiny
    -a
    +a                        
     bar    1     2    1     0
     foo    2     2    1     2

/home/runner/work/pandas/pandas/pandas/core/reshape/pivot.py:558: DocTestFailure

```
Thanks @arw2019 
Thanks @WillAyd @jreback @jbrockmendel for reviewing!
Backport PR #36535 on branch 1.1.x (Regr/period range large value/issue 36430)
State: closed

Backport PR #36535: Regr/period range large value/issue 36430

Labels:
name: Period
description: Period data type

Comments:
failure unrelated. see https://github.com/pandas-dev/pandas/pull/36396#issuecomment-695195148
ENH: return RangeIndex from difference, symmetric_difference
State: closed

- [x] closes #12034
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Index
description: Related to the Index class or subclasses

Comments:
@iamlemec can you take a look and tell me what needs to be done here to be consistent with what you're doing in #36413?
> @iamlemec can you take a look and tell me what needs to be done here to be consistent with what you're doing in #36413?

I think the most consistent way here would be to use `_wrap_setop_result` on the final return statement and `_get_reconciled_name_object` on the quick return statements up top. I actually didn't really touch `difference` and `symmetric_difference` in #36413, but it would make sense that they get a similar treatment.

My only concern regarding `difference` is that maybe since it's not a symmetric operation, it's naming rule should be left-biased. Like, if you're just dropping a few nameless elements, should that clobber the original name?
added name checks to tests
i guess this is technically a change, can you add a note; ping on green.
ping
Closes #36541 (BUG: ValueError: cannot convert float NaN to integer when resetting MultiIndex with NaT values)
State: closed

- [x] closes #36541
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: MultiIndex

Comments:
I might need some help regarding the build error (Arm64, Xenial, Py3.7). Seems like there are some files missing, but I’m not sure what I could do to fix that...?
pls merge master as well
ready to merge @jreback
thanks @ssche very nice!
[BUG]: Fix bug with pre epoch normalization
State: closed

- [x] closes #36294
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

The fix propsed by @jbrockmendel worked.

Hope the test is in the right place

Labels:
name: Datetime
description: Datetime data dtype
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
@simonjayhawkins  Added the test
thanks @phofl
[TST]: Add test for duplicate keys in concat
State: closed

- [x] closes #20816
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Added a test

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Test failure seems unrelated
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
can you merge master and ping on green.
@jreback green
thanks!
PERF: TimedeltaArray.__iter__
State: closed

- [x] closes #26713
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Also 2d compat for TDA/DTA `__iter__`

Labels:
name: Performance
description: Memory or execution speed performance
name: Timedelta
description: Timedelta data type

Comments:
gentle ping
CI: fix failing  pre-commit
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration
name: Code Style
description: Code style, linting, code_checks

Comments:
The npdev CI failure is unrelated, so I'm OK with merging @simonjayhawkins. But have you seen it before?

```
=================================== FAILURES ===================================
________________ test_raw_roundtrip[\u03a9\u0153\u2211\xb4...] _________________
[gw0] linux -- Python 3.8.5 /home/vsts/miniconda3/envs/pandas-dev/bin/python

data = 'Ωœ∑´...'

    @pytest.mark.single
    @pytest.mark.clipboard
    @pytest.mark.parametrize("data", ["\U0001f44d...", "Ωœ∑´...", "abcd..."])
    def test_raw_roundtrip(data):
        # PR #25040 wide unicode wasn't copied correctly on PY3 on windows
        clipboard_set(data)
>       assert data == clipboard_get()
E       AssertionError: assert 'Ωœ∑´...' == '👍...'
E         - 👍...
E         + Ωœ∑´...

pandas/tests/io/test_clipboard.py:259: AssertionError
------------ generated xml file: /home/vsts/work/1/s/test-data.xml -------------
```
> But have you seen it before?

not that I'm aware of.
OK, let's see if it goes away.
CLN: Break up wrap applied output
State: closed

Splitting off the Series case seemed to be a natural way to break up this method. Always open to other thoughts.

Labels:
name: Groupby
name: Clean
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
can you merge master and ill take a look
@jbrockmendel Merged and passing thus far. Much of the diff is due to indentation being one level less, easier to view if you ignore whitespace.
Regr/period range large value/issue 36430
State: closed

#  Checklist

- [x] closes #36430 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

# Solution

Culprit was the multiplication `unix_date * 24 * 3600 * 10**9 / factor`, for 
```
unix_date = 106752
np.log2(unix_date * 24 * 3600 * 10**9)
# 63.00000011936912
```
That probably lead to an integer overflow somewhere and the observed behaviours.

Splitting the multiplication did the trick.


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Period
description: Period data type

Comments:
looks good ping when addressed @WillAyd comments and green.
@jreback @WillAyd green and comment addressed, even if I am not sure about the const comment.

I also explicited in response to comment how the fix work, and why we should not be using unsigned long long here.
@WillAyd anything further?
Great thanks @nrebena - nice PR!
@meeseeksdev backport 1.1.x
BUG: Fix issue in preserving index name on empty DataFrame
State: closed

- [x] closes #36527
- [x] tests added / passed
   - `tests/indexing/test_partial.py:tesst_index_name_empty`
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry
   - v1.1.3


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Hello @Dr-Irv! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-09-21 20:17:46 UTC
thanks @Dr-Irv 
@MeeseeksDev backport to 1.1.x
REF: Categorical.fillna match patterns in other methods
State: closed

Also:

- always return a copy even if there is nothing to fill
- validate the fill_value even if there is nothing to fill

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Categorical
description: Categorical Data Type
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
what is the visible user change here?
> what is the visible user change here?

A couple of exception messages changed (see changed tests), always returning a copy even if a no-op, always validate fill value even if no-op (see OP, xref #36466)
Possibly closes #19705 cc @TomAugspurger 
whatsnew added + green
nice code deletion!
REF: test_to_latex
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Refactor/clean-up ``test_to_latex.py``.

- Split big test functions with multiple assertions into multiple functions
- Make readable expected strings by first indenting for the good visual appearance and then dedenting by the leading whitespace for the assertion.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean
name: IO LaTeX
description: to_latex

Comments:
yeah agree with @WillAyd comment as well. Note that would be also fine to organize these into different classes (if that's useful). same / followon ok with this

ping on green.
> 
> 
> yeah agree with @WillAyd comment as well. Note that would be also fine to organize these into different classes (if that's useful). same / followon ok with this
> 
> ping on green.

Yes, I though of re-organizing the tests into separate classes. I would prefer to do that in a separate PR, after this one.

What is the policy on the test functions naming? I mean if we organize tests in the appropriate classes, then would that be necessary for us to keep ``to_latex`` in all of them? As I understand, this is useful for ``pytest .. -k to_latex`` filtering, but similarly one can run ``pytest -k ToLatex`` if all classes contain ``ToLatex`` in them.

Similarly, if we have a separate class on, let's say multiindex, then I would rather remove ``multiindex`` from the test functions as well, as the information will be available in the class name.
What do you think about it?

thanks @ivanovmg very nice.

so a couple of things for followons.

- prefer shorter PRs, though I do get that sometimes it makes sense to change everything in one go
- we use classes for organization, i would *still* leave the `to_latex` in the test name as with pytest you could search on this as well
- using composable fixtures is nice if  you can, but again use what makes sense
DOC: a few sphinx fixes in release notes
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
I've labelled 1.1.3 but only the changes in doc/source/whatsnew/v1.1.3.rst can be backported.
TST: check inequality by comparing categorical with NaN ( #28384 )
State: closed

- [x] closes #28384
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Categorical
description: Categorical Data Type

Comments:
Please review.

And I need some help.  
I don't know why CI is failing.From the looks of it, it looks like it's timed out.
> I think possibly add a non-categorical example since the OP was about the difference in behavior between categorical vs. non-categorical. See [#28384 (comment)](https://github.com/pandas-dev/pandas/issues/28384#issuecomment-530514218)
> 
> Re: CI failures those are unrelated I think. Merge with master and they'll be rerun

Thank you for your input. I added test case.
And I rebased my branch.
thanks @junjunjunk