BrandonFors/pandas-github-issues huggingface dataset:
BUG: Series.groupby fails with InvalidIndexError on time series with …
State: closed

…a tuple-named grouper. (#42731)

- [x] closes #42731
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Skipped any "What's New" for now, since it's not decided which version will have it. Will update as soon as this is decided.

Labels:
name: Bug
name: Groupby
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Hello @suoniq! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-09-05 02:10:58 UTC
Please note that I'll be on vacation until mid-August and plan to not come close to any computers. So I won't respond Do any necessary changes to the MR if you don't want to wait. See you then.
cc @rhshadrach if you can look
@suoniq - friendly ping. It'd be great to get this in for 1.3.2 and I think it's almost there. If you don't have time to finish it up, would you mind if I pushed it across the finish line?
@simonjayhawkins - I've tagged this as 1.3.2 but certainly don't hold up the release if it doesn't get in by tomorrow.

@suoniq - Thanks for the PR and apologies for the ping above, I missed your note about being out.

cc @jreback @jbrockmendel
moving to 1.3.3
Hi, back from vacation.  
@rhshadrach Of course I am fine with you taking over. Still consider the original test as good b/c it pin-pointed exactly the problem: time series behave differently than series indexed differently. Was red before the fix as well. `(-;`  
Anyway, please let me know, if there is anything left I can do.
@suoniq I didn't think the intention was to test on empty objects, but let me know if this is incorrect. I think we're in good shape here, just waiting on review. More than happy if you want to take the reins back again!
@jreback and @jbrockmendel for review
Thanks @jreback - comment addressed and green.
thanks @suoniq and @rhshadrach 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
Fix typing issues for CI
State: closed

- [X] closes #42769
- [N/A] tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [N/A] whatsnew entry

Should allow CI to proceed.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks @Dr-Irv
REG: DataFrame.agg where func returns lists and axis=1
State: closed

- [x] closes #42727
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

This reverts #40428. Once agg/apply paths are sorted out so that agg doesn't fall back to apply, we can redo #40428. See https://github.com/pandas-dev/pandas/issues/42727#issuecomment-887191683 for more details.

Labels:
name: Bug
name: Regression
description: Functionality that used to work in a prior pandas version
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
thanks @rhshadrach 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
PERF: nancorr pearson
State: closed

- [x] xref #40956
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Benchmarks:
```
       before           after         ratio
     [e64fcfa7]       [c9a99585]
     <master>         <nancorr_perf>
            52.3M            52.1M     1.00  stat_ops.Correlation.peakmem_corr_wide('pearson')
-         307±5μs         217±10μs     0.71  stat_ops.Correlation.time_corr('pearson')
         132±10μs         138±10μs     1.05  stat_ops.Correlation.time_corr_series('pearson')
-      7.74±0.6ms       2.80±0.2ms     0.36  stat_ops.Correlation.time_corr_wide('pearson')
       9.04±0.7ms       8.22±0.8ms     0.91  stat_ops.Correlation.time_corr_wide_nans('pearson')
       3.92±0.2ms       4.00±0.3ms     1.02  stat_ops.Correlation.time_corrwith_cols('pearson')
       6.99±0.2ms       7.62±0.5ms     1.09  stat_ops.Correlation.time_corrwith_rows('pearson')
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
sure, thanks @mzeitlin11
TST: doctest fix for pandas.io.formats.style_render._parse_latex_tabl…
State: closed

The following commit fixes the doctest in _parse_latex_table_styles.
- [X] closes #42674



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Docs

Comments:
@datapythonista do we need to enable running this particular doc-test?
i c, this is enabled in #42705 
thanks @aneesh98
TST GH27994 Add test for dropping empty list / index for a DataFrame …
State: closed

…with a non-unique datetime index.

- [ ] closes GH#27994 pandas-dev/pandas/issues/27994
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
I spent some time to look at the original ticket and testing out some of the scenarios mentioned in pandas==0.25.0 and master.  

It looks like only in the exact reported example would an error appear in 0.25.0:
1. Drop Empty List on Index - Returns Same DataFrame [Passed]
2. Drop Empty List on Non-Unique Index - Returns Same DataFrame [Passed]
3. Drop Empty List on DateTimeIndex - Returns Same DataFrame [Passed]
4. Drop Empty List on Non-Unique DateTimeIndex - "TypeError: 'NoneType' object is not iterable" [Failed]

There was no difference between dropping using an empty list vs. empty datetimeindex.  However, I decided to test both because list-like values are supported and although a list is the most simple the example used a datetimeindex.

Like mentioned, this is now fixed on master.

I only wrote a test for [3], the very specific scenario in which the Bug was reported.  But let me know if anyone thinks we should add more of these scenarios.
Looks like there are some failures on 'Checks' - 'Running doctests'
> FAILED pandas/io/formats/style_render.py::pandas.io.formats.style_render.StylerRenderer.format
FAILED pandas/io/formats/style_render.py::pandas.io.formats.style_render._parse_latex_table_styles

I don't think it's related to my Pull Request but going to take a look to see why it's failing or if there's an Open Issue.
Yeah they are unrelated 
thanks @mikephung122
[BUG] don't mangle null-objects in value_counts
State: closed

- [x] closes #42688
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Null-like values are no longer mangled in `value_counts`.

This was overlooked in #22296, as decision was made not to mangle null-like values (np.nan, None, pd.NaT and so on).

It also has impact on `mode`, as it uses `value_counts` under the hood.

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
asv shows, what cost mangling of null-like values would have:

```
       before           after         ratio
     [9731fd07]       [d5454938]
-         174±2ms          138±2ms     0.79  series_methods.ValueCountsObjectDropNAFalse.time_value_counts(100000)
-     1.16±0.01ms         916±20μs     0.79  series_methods.ValueCountsObjectDropNAFalse.time_value_counts(1000)
-         144±2ms        111±0.6ms     0.77  series_methods.ModeObjectDropNAFalse.time_mode(100000)
-      7.03±0.1ms      4.40±0.05ms     0.63  series_methods.ModeObjectDropNAFalse.time_mode(10000)

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@mroeschke some tests are failing, but it looks as if it is not due to this PR.
thanks @realead very nice!
REF: GroupBy._get_cythonized_result
State: closed

This is preparatory to making get_cythonized_result operate blockwise, which should improve perf a decent amount

Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby

Comments:
I think this PR introduced two of the mypy failures on master.
ill make a pr to fix the mypy issues shortly
I fixed it in #42770
DEPR: special-casing dt64/td64 in Series.unique
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

```
dti = pd.date_range('2016-01-01', periods=3).repeat(2)
ser = pd.Series(dti)

>>> ser.unique()
<stdin>:1: FutureWarning: Series.unique behavior with datetime64[ns] dtype is deprecated. In a future version this will return a <class 'pandas.core.arrays.datetimes.DatetimeArray'> instead of a np.ndarray
array(['2016-01-01T00:00:00.000000000', '2016-01-02T00:00:00.000000000',
       '2016-01-03T00:00:00.000000000'], dtype='datetime64[ns]')

```

Labels:
name: Datetime
description: Datetime data dtype
name: Deprecate
description: Functionality to remove in pandas
name: Stale

Comments:
can you add a release note. IIUC this will *always* warn? how does one avoid this?
> IIUC this will always warn? how does one avoid this?

i guess could do ._values.unique
i am confused over when the user would see this, can you update the top of the PR
added example to OP
so the problem is this will *always* warn right? 
yah,  i guess could do it without deprecation in 2.0

On Sat, Aug 7, 2021 at 4:45 PM Jeff Reback ***@***.***> wrote:

> so the problem is this will *always* warn right?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/42741#issuecomment-894719150>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB5UM6HI7OJSRP5YVDP2SUTT3XAS3ANCNFSM5BBBDEUA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>
> .
>

This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@jbrockmendel worth fixing this up?
> worth fixing this up?

i think this is worth doing.  IIRC the trouble was that if we issue a FutureWarning, we don't have a good way to say "do X instead to get the future behavior".  Best idea ATM is to just make the change in 2.0 without a deprecation cycle.
yeah sounds good to me
@jbrockmendel this for 2.0? if so let's just close it (and add to the list itself which i think you did)
> this for 2.0? if so let's just close it (and add to the list itself which i think you did)

sure.
REGR: DataFrame.from_records with empty records #42456
State: closed

- [x] closes #42456
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
CI: Add numba back to 3.9 env
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Oversight from my other PR syncing 3.9 builds

Labels:
name: CI
description: Continuous Integration

Comments:
thanks @lithomas1
CI: Fix asv
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
@jreback gentle ping gets us back to green
great
DOC: move release note PR #42644
State: closed

move release note PR #42644

no backport required.

Labels:
name: Docs

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.3.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 353ac5cab1310d3755339ef5fcd2cec2fed4e246
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #42730: DOC: move release note PR #42644'
```

4. Push to a named branch :

```
git push YOURFORK 1.3.x:auto-backport-of-pr-42730-on-1.3.x
```

5. Create a PR against branch 1.3.x, I would have named this PR:

> "Backport PR #42730 on branch 1.3.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
TST: test_column_types_consistent
State: closed

- [x] closes #26779
- [x] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

This PR adds a test for the following issue: #26779 

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
thanks @shiv-io
CLN: clean the conda-to-pip script
State: closed

precursor of #42304


Labels:
name: Clean

Comments:
No objections here, good to remove the now-unnecessary `azure` flag
thanks @fangchenli 
this is a precursor to #42304, see https://github.com/pandas-dev/pandas/pull/42720#issue-696688995

needs backport for #42901
@meeseeksdev backport 1.3.x
TST: Fixed doctest in Styler.to_latex #42672
State: closed

- [x] closes #42672 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Docs
name: Styler
description: conditional formatting using DataFrame.style

Comments:
I'm new to contributing to open source and I was wondering how I would go about fixing the failed checks
New failures are not related to your changes, they were caused by a xarray upgrade see #42716
hmm do we need to backport this?
looks fine, gets us back to (or closer to) a green CI i think
thanks @willie3838
Pin xarray to get ci passing
State: closed

- [x] xref #42716
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Pinning until fixed

Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
Thanks @phofl
fixed doctest in styler.to_latex
State: closed

- [] closes #42672
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Docs

Comments:
Hello @willie3838! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-07-25 16:19:49 UTC
the reason I suggested this as skip doc test is because this is the docstring for `Styler.to_latex` but the code example here which is `s.render()` is generating HTML. It is  unnecessary to show the output in this docstring - the code is provided as a comparative example to demonstrate the difference in input between HTML and LaTeX, where users will typically be used to the former.


Thanks @attack68, I missed this was the issue you made the comment to. Happy with that too. Using ellipsis also seems as a reasonable approach to me, also fine, whatever you prefer @willie3838 I think.
DOC use pepy-tech badge in README instead of conda-forge one
State: closed

cf Gitter conversation

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
@MarcoGorelli status of this?
> @MarcoGorelli status of this?

I've edited this badge to read "PyPI downloads per month"

I don't see how to edit the conda-forge one (https://anaconda.org/conda-forge/pandas/badges)

So, my suggestion is to just remove the conda downloads count, unless someone can suggest a way to customize its badge's text
sure
DOC: NumericIndex
State: closed

- [x] closes #41272

Docs etc. for the newly added NumericIndex (#41153) + adding NumericIndex to the public namespace.

Labels:
name: Docs
name: Index
description: Related to the Index class or subclasses
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
I’ve updated this.

@jreback 
Ping.
thanks @topper-123 

if you think we need a followup issue for any items pls create one. Also ok to create a deprecation remove issue for 2.0
PERF: unstack
State: closed

```
from asv_bench.benchmarks.reshape import *
self = ReshapeExtensionDtype()
self.setup("Period[s]")

%timeit self.time_unstack_fast("Period[s]")
23.2 ms ± 1.15 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)   # <- master
9.37 ms ± 128 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <- PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
this may have broken one of the asvs, will check
PERF: DataFrame constructor
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Re-implement #42631 without the bug-causing caching of Block.values

Labels:
name: Performance
description: Memory or execution speed performance
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
do we need any kind of test to replicate the doc-test failure to avoid regressions?
updated with test that ports doctest to regular test
TST: fix read_stata doctest #42670
State: closed

- [x] closes #42670
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] example docstring code fixed for read_strata


Labels:
name: Docs
name: IO Stata
description: read_stata, to_stata

Comments:
Yes but the only solution I can see is creating a dummy .dta file for it to work since it needs a file to be read and I don't know where you would store that. For example, should I create an examples folder to store it so all future example documentation code can run as well?
Hello @KrishnaSai2020! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-07-31 07:38:51 UTC
Made the requested changes @fangchenli. 
This is ruining the docstring's purpose just to pass a test. It is not a good idea.  The entire point of reading with chunks to to parse a large data file.  IT should have > 10,000 roww to be illustrative. 

@bashtage I see what you mean. I made the changes to the df's size please review the new changes.
@bashtage thanks for you help. I have now made the changes.
@bashtage I think I've fixed the formatting now. Please do let me know if there is anything else 
Think I've fixed them please review.
Need to add `pandas/io/stata.py \ ` after this line so that the state doctests are run.

https://github.com/pandas-dev/pandas/blob/b8d750fb90c68d4af25af89442aba001483adffa/ci/code_checks.sh#L123


@bashtage I've added this doctest into the scripts and resolved the conflicts
This should now pass all checks.
From what I can tell everything works apart from the doctests in Style_render.py which is unrelated to this pr.
@fangchenli Could I get an update as to what is going to happen with this pr, please? Is it going to get merged? or have I done something wrong which means it can't get merged?
@fangchenli please re-review
Could I get a second opinion on this PR, please? it's passed all checks and already has one approval
@KrishnaSai2020 This LGMT.  I'll leave it to @jreback to make the final decision. You don't need to merge master in anymore unless there is a conflict. 
@KrishnaSai2020 Thanks.