BrandonFors/pandas-github-issues huggingface dataset:
CLN: remove _maybe_update_attributes
State: closed

It is only non-trivial for DTI and TDI, and only actually needed in `__array_wrap__`.  It is cleaner to implement the relevant patch in `DatetimeIndexOpsMixin.__array_wrap__` and avoid the need for maybe_update_attributes altogether.



Labels:
name: Clean

Comments:
can you rebase
Took a glance at the ASV portal this might have caused a lot of performance regressions:

https://pandas.pydata.org/speed/pandas/#/regressions

@jbrockmendel do you have a quick way of validating that?
> do you have a quick way of validating that?

Not a quick way, no.  I'll take a look, but likely won't get to it today.
Oh yea no rush - thanks!
@WillAyd looking at this it isn't obvious which regressions you think may have been caused here.  I see a bunch from August 15 linking to 15b11c83; are those the ones you have in mind?
Might have misread but thought that was the commit that preceded this one on master

Sent from my iPhone

> On Oct 18, 2019, at 7:06 PM, jbrockmendel <notifications@github.com> wrote:
> 
> ﻿
> @WillAyd looking at this it isn't obvious which regressions you think may have been caused here. I see a bunch from August 15 linking to 15b11c8; are those the ones you have in mind?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

It's indeed this commit that asv reports for the biggest regressions. But there was also a gap in asv runs between August 2 and this commit (August 15), so probably it can be anything in that window
@TomAugspurger it also seems that the last runs were in September? 
Will check today. We had some issues with a bad ASV env failing builds. 

> On Oct 19, 2019, at 07:05, Joris Van den Bossche <notifications@github.com> wrote:
> 
> ﻿
> @TomAugspurger it also seems that the last runs were in September?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.
Infer names in MultiIndex.from_product
State: closed

- [x] closes #27292
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: MultiIndex

Comments:
@TomAugspurger in regards to your first comment, the reason I opened an issue about this in the first place was due to the addition of `from_frame`, which *does* infer names from the Series inputs, so it seemed like an inconsistency.   If that behavior usually isn't wanted/needed in `from_product`, then maybe this PR isn't necessary.
you would *also* have to change ``from_arrays`` for the same reason. I think if you did that would be ok with this (ex using @TomAugspurger suggestion, that an explicit ``names=None`` works)
@christopherzimmerman do you have time to make `names=None` mean "no names for any levels"?

The default value will be something like

```
_no_default = object()

def from_product(self, ..., names=_no_default, ...):
    if names is None or names is _no_default:
        names = [None] * nlevels

```
@TomAugspurger  sorry for the inactivity, I'll make those changes.  I believe I messed up the commit history for this PR when I rebased from master (which is resulting in a check failing), and have a lot of junk commits included, I may close this PR and open another one.
BUG: fix Sparse reduction
State: closed

- [x] closes #27080
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Sparse
description: Sparse Data Type

Comments:
needs a whatsnew note i think for this.
updated+green
Thanks.
DOC: Updated Series.items 'See also' section for clarity
State: closed

I changed the See also to:

DataFrame.items : Iterate over (column name, Series) pairs.
DataFrame.iterrows : Iterate over DataFrame rows as (index, Series) pairs.

No assertions about the equivalence of DataFrame.items and Series.items

closes #27859


Labels:
name: Docs

Comments:
Thanks! I agree that is clearer now!
CLN: Index.__new__
State: closed

underlying goal is to share code between Index.__new__ and pd.array

Unrelated cleanup in DataFrame.select_dtypes because I found it cumbersome.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
thanks @jbrockmendel yeah this is some pretty hard core logic, thanks for simplifying
TST: Raise ValueError and suggestion to use header=None if header=-1 is pa…
State: closed

…ssed

- [x] closes #27779
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
Thanks @MarcoGorelli!
Happy to help, and massive thanks for your assistance/guidance!
DOC:Use of "Yields" for documentation of DataFrame.iteritems()
State: closed

changed `Returns` to `Yields`

closes #27860

Labels:
name: Docs

Comments:
@datapythonista @WillAyd : The fix looks good, but the documentation checks seem to be failing.
See the also the discussion on the issue: https://github.com/pandas-dev/pandas/issues/27860. Someone will need to understand why the docstring validation sees iteritems and items differently
Ah, it is because `iteritems` does `return self.items()` to do the same, therefore the docstring validation sees 'return' and thinks it needs to return something. See logic in 

https://github.com/pandas-dev/pandas/blob/master/scripts/validate_docstrings.py#L529-L562

That's of course a bit brittle. @datapythonista I suppose the easiest here is to ignore that rule? 
But currently we can only fully ignore a rule or not in code_checks ?
Since we're Python 3 only now, my preference is to use `yield from self.items()` instead of `return self.items()`, and that should fix the doc validation I think.
thanks @datapythonista that indeed seems a good idea
Thanks @sameshl !
Glad to help! @jorisvandenbossche
DOC: clarify see also for DataFrame.iterrows()
State: closed

changed ` .x()` to `Dataframe.x()` in See Also for DataFrame.iterrows.

Labels:
name: Docs

Comments:
@sameshl sorry, I just merged https://github.com/pandas-dev/pandas/pull/27893, another PR that fixed the same issue. Because it was their first contribution, I merged the other one, but still, thanks a lot for your contribution, and sorry I didn't merge it.

BTW, one thing you can do to prevent this to some extent is to include a `closes #27862` (with the issue number) in your PR. Then at least that shows up as a referenced PR in the issue, and other people that want to work on it might see that. You are also always welcome to explicitly mention on the issue that you are working on it.
@jorisvandenbossche No problem! I should have included the issue number in my PR. :sweat_smile:
Giving beginners a chance should be the priority anyway. Thanks!
requested edit to comparison method
State: closed

cc @jorisvandenbossche discussed in #27803

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
thanks, ideally add to this to a todo list
@jreback if it is a PR specifically for my comments, can you please give me a bit of time to review it?

There were actually more things we were discussing than this.
sure
@jbrockmendel what I wanted to ask in addition:

- can you add a test (for a non-frozenset case) for a case that would have broken by the previous PR?
- can update the wrong "# rename is needed in case res_name is None and result.name" comment?
- ideally, can you also do the rename only if needed? (or use a cheaper method that does not copy) 
> can you add a test (for a non-frozenset case) for a case that would have broken by the previous PR?

I'd rather not test not-necessarily-intentional behavior until we pin down the desired behavior in #27911 

> can update the wrong "# rename is needed in case res_name is None and result.name" comment?

Sure.

> ideally, can you also do the rename only if needed? (or use a cheaper method that does not copy)

Doesn't it do a shallow copy?  I guess creating a new Series object is more costly than the check in the case where it isnt needed?  Wouldn't it be cheaper yet to unconditionally just set `result.name = res_name`?
> I'd rather not test not-necessarily-intentional behavior

Well, it's working behaviour that people will rely upon, so we shouldn't just break it IMO. So in that case a test would be useful. It would eg have catched the changes you were doing in the previous PR, and will do if you or someone further work on that code in the future (we are now well aware, but if we only pick it up again in some months, we might well have forgotten again)

> Doesn't it do a shallow copy?

Nope, it returns a new object, which currently means copying the data (we don't yet have a copy-on-write)

> Well, it's working behaviour that people will rely upon, so we shouldn't just break it IMO. 

But will take a look at the issue you linked
> I'd rather not test not-necessarily-intentional behavior

And it would also have helped to catch https://github.com/pandas-dev/pandas/pull/27769/files#r314060256 :-)
DOC: Clarify that unique() promotes dtype to 64-bit
State: closed

I found this behavior surprising:

```python
In [1]: pd.Series([1,2,3], dtype=np.uint8).unique()
Out[1]: array([1, 2, 3], dtype=uint64)
```
... because I did not expect a different dtype in the result.  This PR adds a sentence to the docs to clarify this.

- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- (n/a) closes #xxxx
- (n/a) tests added / passed
- (n/a) whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
FWIW, These changes do pass `black` and `flake8`.  The CI failures appear unrelated.
CI failure is fixed on master.

Is this deliberate, or an implementation detail? We should document that.
>Is this deliberate, or an implementation detail?

My hunch is that it's an implementation detail.  For example, `pd.unique()` differs from `np.unique()` in this respect, and I don't know why pandas would go out of its way to be different.  It's probably just incidental.

Heck, maybe it's even accidental (i.e. a bug).  Should [this line][1] be changed as follows?

[1]: https://github.com/pandas-dev/pandas/blob/8993facf0897a192c3a933dea99d5c759959c16b/pandas/core/algorithms.py#L407

```diff
-    uniques = _reconstruct_data(uniques, dtype, original)
+    uniques = _reconstruct_data(uniques, original.dtype, original)
```

In case that looks right to you, I've opened a PR: #27874
superseded by #27874
CI: Pin Python to 3.7.3
State: closed


Labels:

Comments:
I'm currently looking to see what other projects are affected.   Maybe try master from 2 weeks ago?
Oh, we're on python 3.7.4 now. That's most likely it.
Let's get CI passing, and investigate later.
The windows 3.7.4 build passed just fine, so I didn't pin that to 3.7.3.
This may break the pip dep handling in https://github.com/pandas-dev/pandas/blob/master/scripts/generate_pip_deps_from_conda.py#L22. We skip exactly on `python=3` there.

But I'd like to have a single commit that we can revert when the conda 3.7.4 packages are sorted out.
all green, LGTM
BUG: DataFrameGroupBy.transform and ngroup do not work with cumcount
State: closed

- [ ] closes #27472 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Groupby

Comments:
Hello @charlesdong1991! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-08-23 19:38:32 UTC
yeah, you are right, pushing to my pr again will somehow pass flaky tests! @TomAugspurger 
@jbrockmendel can you take a glance at this?
will do
is flaky test failure back 😢? also saw the same failure in other PR @TomAugspurger 
Restarted the azure builds.
Thanks Tom, looks okay except one thing: what is `codecov/project` about? how to fix such issue? @TomAugspurger 😅 
You can ignore that.

On Thu, Aug 15, 2019 at 9:19 AM Kaiqi Dong <notifications@github.com> wrote:

> Thanks Tom, looks okay except one thing: what is codecov/project about?
> how to fix such issue? @TomAugspurger <https://github.com/TomAugspurger>
> 😅
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/27858?email_source=notifications&email_token=AAKAOIX2X44TKQ7AFY5S243QEVQYHA5CNFSM4IK373YKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4L56JY#issuecomment-521658151>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIQ3U353VUT77DYHMRLQEVQYHANCNFSM4IK373YA>
> .
>

any follow-up review? ^^ @TomAugspurger 
probably this is not gonna be merged today, then shall I move whatsnew note to 1.0.0? @TomAugspurger 
Yes, that's probably best. I haven't had a chance to go through it again.

On Wed, Aug 21, 2019 at 3:30 PM Kaiqi Dong <notifications@github.com> wrote:

> if this is not gonna be merged today, then shall I move whatsnew note to
> 1.0.0? @TomAugspurger <https://github.com/TomAugspurger>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/27858?email_source=notifications&email_token=AAKAOIX5VHOLTYPDD7TYZ3DQFWQV7A5CNFSM4IK373YKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD43A4JA#issuecomment-523636260>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOISBW4IMYL7QBNGGL2DQFWQV7ANCNFSM4IK373YA>
> .
>

ok, will move after the 0.25.1 releases
Can you also move to v1.0.0?
> Can you also move to v1.0.0?

yeah, of course.. was planning to do so
also needs a whatsnew note (1.0, groupby bug fix)
@charlesdong1991 can you merge master and update to any open comments
@charlesdong1991 can you rebase
BUG: add back check for MultiIndex case and take_split_path
State: closed

Closes #27841


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
This is reverting part of https://github.com/pandas-dev/pandas/pull/27351: there was a seemingly unused check removed (as we thought it was for panels), but apparently it was still used (given the regression report in https://github.com/pandas-dev/pandas/issues/27841)

There was actually discussion about this here: https://github.com/pandas-dev/pandas/pull/10838/files#r302728324
Updated.

I could simplify the case, and understand now better when this was triggered: it's when adding a single column, but where the single column selection is a slice.

In a non-MultiIndex case we actually don't allow this (and you get the same error as now in 0.25 with MultiIndex):

```In [60]: df = pd.DataFrame(np.random.randn(5,3), columns=['a', 'b', 'c'])                                                                                     

In [61]: df
Out[61]: 
          a         b         c
0 -2.791318 -0.238227  0.791717
1  0.635844 -1.494229  0.445973
2  0.575157  0.295088 -0.978239
3  0.894169  0.300125 -0.949338
4  0.716657 -0.634217 -2.311306

In [62]: df.loc[:, 'a':'a']
Out[62]: 
          a
0 -2.791318
1  0.635844
2  0.575157
3  0.894169
4  0.716657

In [63]: df.loc[:, 'a':'a'] = np.arange(5) 
...
ValueError: could not broadcast input array from shape (5) into shape (5,1)
```

But for the MultiIndex, you can get a "slice" without that the user explicitly slices the columns (as I did for the non-MultiIndex case above). So there we are more flexible. 
We could maybe discuss if we want to be more strict here as well (in the future, on purpose with a deprecation), but those MultiIndex corner cases are all a bit tricky.
> I could simplify the case, and understand now better when this was triggered: it's when adding a single column, but where the single column selection is a slice.

Simplification sounds good.  If there's a reasonable scope to refactor into its own method, that would be worthwhile; the setitem_with_indexer method is a beast. 
Merged master.

I think this is OK to fix the regression. I doubt we would want to backport any simplification code anyway.
Thanks @jorisvandenbossche
DOC:add append example in to_excel documentation
State: closed

closes #27051



Labels:
name: Docs

Comments:
Merged master to hopefully fix CI. Ping on green.
@TomAugspurger Looks good now!
Btw, could you explain why the build was failing and merging master to my branch fixed it? :sweat_smile: 
Thanks!
Thanks @sameshl 

If there is something broken in master (e.g. a pandas dependency has been updated and it causes a test to fail), then all PRs will fail even if they are correct. So, master needs to be fixed first, and then the fixes in master need to be merged into the PRs that failed.
@datapythonista Thanks for the explanation!
DOC: Add CoC to the README
State: closed

@datapythonista : Add CoC to README file

Labels:
name: Docs

Comments:
Sorry about the CI failures again @sparalic. If you merge master and push, things should be fixed

```
$ git fetch upstream
$ git merge upstream/matser
$ # git add and commit stuff
$ git push origin pandas-coc
```

You can also put in @gfyoung's request change at the same time if you want
@TomAugspurger  and @gfyoung thank you for all of your help. @TomAugspurger, I tried merging with the master and pushing as you suggested, but some of the CI test failed again.
@sparalic : I restarted the failing build for you to see if the failure goes away.  If not, you should merge with `master` and push.
@datapythonista and @gfyoung thank you!
You are just not having luck with our CI @sparalic 😄  The windows failure is unrelated. Tracking it in https://github.com/pandas-dev/pandas/issues/27902.

This is good to merge once the doc build passes.
Thanks @sparalic!
BUG: boxplot does not work when data has datetime column
State: closed

- [ ] closes #22799
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Visualization
description: plotting
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Thanks @charlesdong1991!
CI: pytest, xdist versions
State: closed


Labels:
name: Dependencies
description: Required and optional dependencies

Comments:
Merging, to get CI passing again.
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 0.25.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 e26fa2b9b7c44ba7a1865c9206ae213e1e908b99
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #27845: CI: pytest, xdist versions'
```

4. Push to a named branch :

```
git push YOURFORK 0.25.x:auto-backport-of-pr-27845-on-0.25.x
```

5. Create a PR against branch 0.25.x, I would have named this PR:

> "Backport PR #27845 on branch 0.25.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
DOC: GroupBy.head()/tail() documentation
State: closed

- [x] closes #6721
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Update DOC to make it consistent with actual behavior

Labels:
name: Docs
name: Groupby

Comments:
added double backticks to `as_index` as suggested
Hello @mtrbean! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-08-20 17:29:07 UTC
@jorisvandenbossche I've updated the PR per your comment
@jorisvandenbossche @datapythonista ping again
PERF: Break reference cycle for all Index types
State: closed

- [x] xref #27585 & #27607
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Generalizes the test in #27607 to be used for all index types. This finds ref cycle issues for ``PeriodIndex`` and ``CategoricalIndex`` which are solved in this PR.


Labels:
name: Performance
description: Memory or execution speed performance
name: Index
description: Related to the Index class or subclasses

Comments:
Thanks for providing this extended fix :)
I think this should be ok to pull in after the rebase?

Agreed, thanks!