BrandonFors/pandas-github-issues huggingface dataset:
BUG: Fix return of missing values when applying loc to single level of MultiIndex
State: closed

- [x] closes #27104
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This confused ``-1`` representing missing values in input with the ``-1`` signaling which components to exclude from output.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
cc @jreback green


Thanks for merging master
very nice @phofl
REF: simplify NDFrame.replace, ObjectBlock.replace
State: closed

This makes it so that we only call BlockManager.replace from one spot, so we can rule out more cases when reasoning about the arguments in ObjectBlock.replace.

This will have a merge conflict with #37696; doesnt matter which one goes in first.  One both are in, there are further simplifications available.

I'm hopeful that we'll be able to get rid of Categorical.replace altogether (and with it, CategoricalBlock.replace, CategoricalBlock._replace_list), but that's a few steps away at best.


Labels:
name: Clean

Comments:
sure pls rebase the other
remove unnecessary use of Appender
State: closed

- [x] ref #31942
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] removes the last use of Appender() in NDFrame


Labels:
name: Docs

Comments:
nice! @smartvinnetou
CLN: Simplify groupby head/tail tests
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Going to add axis=1 tests as part of resolution of #9772, simplifying test first.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Clean

Comments:
thanks!
DOC: Highlight pre-commit section
State: closed

- [x] closes #37358 
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Docs

Comments:
~~I noticed that https://dev.pandas.io/ is not up to date, is this intended?~~

it seems like this is intentional and that latest docs from `master` are at https://pandas.pydata.org/docs/dev/, see #37248.
looks fine to me. @MarcoGorelli if you can have a look see
Thanks @MarcoGorelli , applied your suggestions.
lgtm. cc @MarcoGorelli merge when ready
Thanks @hongshaoyang
DOC: Add caveat to to_stata with version=119
State: closed

- [x] closes #37490
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Docs
name: IO Stata
description: read_stata, to_stata

Comments:
LGTM.
thanks @hongshaoyang
DOC: capitalize Gregorian and fix word order
State: closed

Gregorian is a proper noun
(https://en.wikipedia.org/wiki/Proleptic_Gregorian_calendar). Also,
"proleptic" is modifying the compound noun "Gregorian epoch" rather than
"Gregorian" modifying a "proleptic epoch".

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Thanks @erictleung
REF: implement replace_regex, remove unreachable branch in ObjectBlock.replace
State: closed


Labels:
name: Categorical
description: Categorical Data Type
name: Clean

Comments:
yeah this ones need a rebase
rebased+green
I'm pretty sure we can get rid of CategoricalBlock.replace (and with it, deprecate Categorical.replace) by implementing EABlock._can_hold_element/putmask correctly.  will give it a go this week
Dataframe.sum() returns an error with MultiIndex columns and skipna=False
State: closed

- [ ] closes #37622
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
> Thanks @GYHHAHA for the PR!
> 
> Can you add tests? As reviewers that's the first thing we look for

I find it needs more investigations. Close now and open this again after fixed.
DOC: add examples to insert and update generally
State: closed

This commit mainly adds examples on usage. This commit also fills in
information suggested by `validate_docstrings.py` script.

Ran

```
python scripts/validate_docstrings.py pandas.DataFrame.insert
```

to check docs and added in suggested information (e.g., See Also section).

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
@arw2019 changes made. Thanks for the feedback!
Sorry this PR has been unaddressed for a while. LGTM but could you merge master one more time?
@mroeschke thanks for taking a look at this! I rebased my branch onto the most recent commits on `master`. That should resolve any issues merging. Let me know if there are any issues going forward in merging.
Thanks @erictleung
CLN: Clean indexing tests
State: closed

- [x] tests passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Found a few unuesed statements

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Clean

Comments:
small comment, otherwise LGTM
thanks @phofl
pd.Series.loc.__getitem__ promotes to float64 instead of raising KeyError
State: closed

- [x] closes #25927
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

cc @jreback Added a test. The test above was the only one explicitly added for the ``KeyError``


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Needs Tests
description: Unit test(s) needed to prevent regressions

Comments:
thanks @phofl
CLN: _replace_single
State: closed

Confirmed that the `regex` kwarg is always bool, which lets us remove some gymnastics (L2570-L2586).  Then removed always-True `convert` kwarg.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Clean

Comments:
ok nice clean, can you run some bencharmarks; iirc this was a performance case. prob not a big deal (and maybe i am wrong).
> ok nice clean, can you run some bencharmarks; iirc this was a performance case. prob not a big deal (and maybe i am wrong).

I'll give it a go.  I'm pretty sure the perf issue would have been in replace_list; thats where we have some semi-kludgy optimizations in place.
> > ok nice clean, can you run some bencharmarks; iirc this was a performance case. prob not a big deal (and maybe i am wrong).
> 
> I'll give it a go. I'm pretty sure the perf issue would have been in replace_list; thats where we have some semi-kludgy optimizations in place.

yeah certainly possible.
asv with `-b replace` show no change
more replace-related PRs coming up
BUG: setitem with boolean mask and series as value is broken for Series with EA type
State: closed

- [x] closes #26468
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Was fixed already, added a test

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Needs Tests
description: Unit test(s) needed to prevent regressions

Comments:
thanks @phofl
Bug in loc raised for numeric label even when label is in Index
State: closed

- [x] closes #26491
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Additionally checking for the label inside values, this fixes the problem.

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
thanks @phofl very nice
CLN: no need to call _wrap_results in nanvar
State: closed


Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Clean

Comments:
this is likley true for only float returning methods, e.g. sem, kurt, skew as well (though i don't think its actually a big deal if you leave this here).
> this is likley true for only float returning methods, e.g. sem, kurt, skew as well (though i don't think its actually a big deal if you leave this here).

_wrap_results is only relevant for td64/dt64, so my plan is to combine it with _mask_datetimelike_result (will end up being a decorator).  You're right that its harmless here.
Closing in favor of #37723
CLN: only call _wrap_results one place in nanmedian
State: closed


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
~see my comments on #37674 I don't think this improves clarity generally by removing this call as then a future reader will have to parse individual paths and determine what is float returning and what is not. (now general cleaning of course is good).~

nvm didn't look clearly first. this is a good change.
BUG: Fix bug in loc setitem changing the dtype when condition is False
State: closed

- [x] closes #37550
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


This fixes the bug, but I am not sure, if the solution is good enough....

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@phofl can you merge master to resolve conflicts
@phofl can you rebase and will look again
merged master
@jbrockmendel ok here?
@phofl can you update
Looks like the `ser[[]] = [1, 2, 3]` thing hasnt been addressed
ok let's move off the milestone for rn, can always do after the rc
@phofl status of this (merge master)
@phofl worth fixing up?
Yeah definitely.

i hope that I can get back to this friday or friday next week 
I moved the fix for the all False case. This fixes an inconsistency in _align_series and covers the all False indexer case. 

I could not figure the empty list thing out. The values look exactly the same as in the all False case. We probably will have to fix this someplace else, but not sure right now.

I woild propose open an issue about that and go from there.
lgtm @jbrockmendel if any comments
will take a look this afternoon.  (got booster yesterday, moving slow today)
Does this relate to #43632?
I don't think so. The all False case is an empty not a boolean indexer when jumping into the setitem logic
thanks @phofl
TST: fix warning for pie chart
State: closed

- [ ] closes #37668
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Fix MatplotlibDeprecationWarning for pie chart by specifically passing ``normalize=True``.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Visualization
description: plotting

Comments:
thanks @ivanovmg
BUG: CategoricalIndex.equals casting non-categories to np.nan
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Categorical
description: Categorical Data Type

Comments:
im hopeful we'll be able to just get rid of _is_dtype_compat entirely, but taking it one step at a time
imagine this is the root cause of at least a bug or 2 on our tracker
TST: match matplotlib warning message
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Follow up on #36982.
Since, currently ``assert_produces_warning`` now allows matching warning message,
instead of the comment we will match the message.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Nice!

Not a blocker, but why not match the entire message?
```python
        msg = (
            "To output multiple subplots, the figure containing "
            "the passed axes is being cleared"
        )
```
> 
> 
> Nice!
> 
> Not a blocker, but why not match the entire message?
> 
> ```python
>         msg = (
>             "To output multiple subplots, the figure containing "
>             "the passed axes is being cleared"
>         )
> ```

I did that to have a slight immunity to minor modifications of the warning message by matplotlib team, just in case.
This concerns only the second part of the message, so it may not be fully justifiable.
Anyway, I will update to the full message.
It is greenish since there are some internal CI errors. All relevant tests seem to pass successfully.
Thanks @ivanovmg !
Start testing pypy3
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration

Comments:
Thanks for taking a look, our preference is to run more of our CI on Azure compared to Travis. Is it easy enough to move over? (I would wait for some other maintainers to validate this before making any changes)

Ref: Email from @bashtage this week.
I'll confess I'm unsure how to provide a PR for the azure bits....
@jcpunk might help to look at the yaml docs for pipelines:

https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema%2Cparameter-schema
now that I've chased down the azure bits and gotten the manylinux wheels somewhat sorted, looks like the manylinux containers aren't ready for pypy yet.

I'll close this out for now and revisit when the manylinux container has pypy support.  No sense in hacking it in now.
Manylinux support pypy3.7 and pypy3.8. They are installed in manylinux2014:latest (from 5 days ago). For instance, pypy3.8 is in /opt/python/pp38-pypy38_pp73
REF: simplify cycling through colors
State: closed

- [ ] closes #37604
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

I tried to address #37604, but it turns out that on some occasions it is absolutely mandatory to return more colors than ``num_colors`` says.
In particular, this is the test that cannot be updated and the underlying code cannot be easily modified as well.

```
pandas/tests/plotting/test_frame.py
...
class TestDataFramePlots(TestPlotBase):
...
    def test_bar_user_colors(self):
        df = DataFrame(
            {"A": range(4), "B": range(1, 5), "color": ["red", "blue", "blue", "red"]}
        )
        # This should *only* work when `y` is specified, else
        # we use one color per column
        ax = df.plot.bar(y="A", color=df["color"])
        result = [p.get_facecolor() for p in ax.patches]
        expected = [
            (1.0, 0.0, 0.0, 1.0),
            (0.0, 0.0, 1.0, 1.0),
            (0.0, 0.0, 1.0, 1.0),
            (1.0, 0.0, 0.0, 1.0),
        ]
        assert result == expected
```

When "y" is specified, then the number of colors must be the length of the series df["A"],
but inside the BarPlot class there is no opportunity to check for that condition.

So, let ``get_standard_colors`` return maximum of ``num_colors`` and ``len(colors)``.
This way all tests work without modifications,
just minor improvement in the cycling function.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Visualization
description: plotting
name: Clean

Comments:
thanks @ivanovmg very nice!
thanks @ivanovmg
Move inconsistent namespace check to pre-commit, fixup more files
State: closed

xref https://github.com/pandas-dev/pandas/pull/37401#discussion_r515312381

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Good catch, thiat one wasn't picked up because the first instance of `Categorical` in that file doesn't have an open parenthesis after it. Have updated (with a little test too)
ok this looks good, can you merge master one more time and ping on green.
restarted the docbuild, which looked like an HTTPError
Thanks - @jreback green now
BUG: unpickling modifies Block.ndim
State: closed

- [x] closes #37631
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
@meeseeksdev backport 1.1.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.1.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 dbee8faecf9a3ea65a2861e32c12695eae81f3e6
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #37657: BUG: unpickling modifies Block.ndim'
```

4. Push to a named branch :

```
git push YOURFORK 1.1.x:auto-backport-of-pr-37657-on-1.1.x
```

5. Create a PR against branch 1.1.x, I would have named this PR:

> "Backport PR #37657 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
TST: split up tests/plotting/test_frame.py into subdir & modules #34769
State: closed

Created instead of #37538

- [x] closes #34769
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

I moved the file to a new directory.
What is the best structure for this folder?

[before_test_ploting_frame](https://gist.github.com/Mikhaylov-yv/999de1451e8ecc836215b30db34ef841)
[after_test_ploting_frame](https://gist.github.com/Mikhaylov-yv/98eca32c21f59cc9e7653cd96d5fabd2)

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Visualization
description: plotting

Comments:
in pre-commit:

```pandas/tests/plotting/frame/test_frame_groupby.py:7:1: F401 'pandas as pd' imported but unused```

also merge master and ping on green.
I am very sorry. I don’t understand what to do next?
> I am very sorry. I don’t understand what to do next?

run `git pull upstream master` to merge master, then in the new files use `from pandas import foo` instead of `pd.foo`
Thanks @jbrockmendel. After that, I get a message like this:
![image](https://user-images.githubusercontent.com/44315225/98464889-e8f4b000-21d6-11eb-9f1b-0722a8f7219e.png)
Am I doing the right thing?

Hello @Mikhaylov-yv! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-11-12 05:34:48 UTC
> Am I doing the right thing?

I dont see anything wrong in the screenshot you posted.

Near the bottom of this page you should see a box with red text saying "Some checks were not successful".  In that box there are two red "x"s for the failed checks.  Click through to the details on those to see what still needs to be addressed.
@jbrockmendel Thanks. I think I figured out what to do.
I hope I took into account all my mistakes
The errors don't seem to be relevant to my pull request. I'm wrong?
> The errors don't seem to be relevant to my pull request. I'm wrong?

You are not wrong.  This is affecting all the builds, you can ignore this failure for now.
Thanks @jbrockmendel Does this mean the job is ready?  or do you need to do something else?
@Mikhaylov-yv is this pretty much cut/paste?  if so, can you merge master and ping on green
@jbrockmendel Yes it is.  the number of tests is the same as the master.  
Now I'm going to merge
LGTM cc @jreback
thanks @Mikhaylov-yv 
@jreback @jbrockmendel  Thank you very much for your help
BUG: preserve fold in Timestamp.replace
State: closed

- [X] closes #37610
- [X] 1 tests added / 1 passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry

## Problem
We currently lose `fold` information (whether the Timestamp corresponds to the first or second instance of wall clock time in a DST transition) when calling `Timestamp.replace`.

## Solution
A simple addition to the code of `Timestamp.replace` should fix this.

## Test
I added the test for the use case I came up with in the issue discussion. The OP example is losing `fold` when deleting timezone information with `Timetsamp.replace`, and that's not really a bug, but replacing a valid dateutil timezone with itself and losing `fold` definitely is.

The proposed solution fixes the original example as well. I just don't think we should be tracking it in tests, as it's not clear to me why `fold` must be preserved in a `Timestamp` with no timezone information (but it is the convention recommended in [PEP 495, fold section](https://www.python.org/dev/peps/pep-0495/#the-fold-attribute) to let users keep invalid fold and to just ignore it).

## Some details
IIRC, we ignore `fold`, when it doesn't do anything, so we should be safe preserving `fold` while replacing tzinfo with None, as in the OP example. I remember this coming up when we introduced fold support, and we made sure that the fold-aware functions don't care what fold is outside of a DST transition with a dateutil timezone (this was done to satisfy the requirements of [PEP 495, fold section](https://www.python.org/dev/peps/pep-0495/#the-fold-attribute)).

Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Timezones
description: Timezone data dtype

Comments:
All green, should be ready for a review.
cc @pganssle @jbrockmendel 
@jreback @jbrockmendel @pganssle 
Accepted all suggestions, all green, detailed comments above.
Looks fine to me, but I'm going to agree with whatever @pganssle says here
thanks @AlexKirko
DOC: test organization
State: closed

Documenting the MO behind the test organization we've been doing.

1) I expect this is not valid rst ATM
2) The style does not match the rest of the docs, so probably needs a re-write.



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Docs

Comments:
thanks @jbrockmendel very nice
i think this might have been merged prematurely, the docbuild is complaining about rst stuff i need to figure out
@jbrockmendel if you push your branch again, I *think* you should be able to reopen the PR (unless you also already deleted the branch locally), and then we can work out the failures here