BrandonFors/pandas-github-issues huggingface dataset:
DEPR: `Styler.set_na_rep` and `.set_precision` in favour of `.format(na_rep='x', precision=3)`
State: closed

- [x] closes issue  #40032  and PR #40060 
- [x] deprecates the methods `Styler.set_na_rep()` and `Styler.set_precision()` in favour of, and incorporating `precision` into `Styler.format()`. Maintains some backwards compatibility with a deprecation warning, the copy mechanics remain broken.
    - [x] `Styler.set_na_rep('MISS')` is replaced and equivalent to `Styler.format(na_rep='MISS')`
    - [x]  `Styler.set_precision(5)` is replaced and equivalent to `Styler.format(precision=5)`
    - [x]  `Styler.set_na_rep('MISS').set_precision(5)` is equivalent to `Styler.format(na_rep='MISS', precision=5)`
    - [x]  `Styler.set_precision(5).set_na_rep('MISS')` is equivalent to `Styler.format(na_rep='MISS', precision=5)` 
- [x] moves the `default_formatter` and `maybe_wrap_formatter` methods to `module` level. 
- [x] documentation is added to show new functionality
- [x] most existing tests still pass with, some needed minor modification and some were no longer necessary
- [x] tests added for new combinations
- [x] whatsnew entry


This PR was suggested after review of #40060 as an alternative. It seems to be a simpler and more consistent solution for API.

Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Styler
description: conditional formatting using DataFrame.style

Comments:
The idea seems good to me. Looks like we'll need a release note.
@jreback ping this is greenish, if you approve the changes to the docstring ``formatter``above
thanks @attack68 very nice!
@attack68 want to make a PR enforcing this deprecation?
yes will do. been waiting for the 2.0 tag to push to
REF: share recarray constructor code
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
looks good can you merge master
BUG: Set dtypes of new columns when stacking (#36991)
State: closed

- [x] closes #36991
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Hello @maroth96! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-03-10 01:29:17 UTC
I extracted a function for stacking the column index. Let me know if you prefer it inlined. Otherwise, the same is possible for the index (lines 714--732).
thanks @maroth96 very nice! keep em coming!
STYLE update pre-commit-config
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Am doing this manually (instead of the automated job tomorrow) in order to fixup some minor things

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
LGTM; npdev failures look unrelated
thanks @MarcoGorelli
Use tempfile for creating a file in home directory for a IO test
State: closed

- [X] closes #40091
- [X] tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [X] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
lgtm ping on greenish (there is a failing numpy_dev build ATM)
thanks @chrispe very nice!
TYP: to_arrays, BUG: from_records empty dtypes
State: closed

1) the argument annotation for columns in to_arrays isnt quite right.  this is fixed by adding an ensure_index inside `DataFrame.__init__`
2) call ensure_index in the empty case in to_arrays so we can tighten from `Tuple[Any, Any]` to Tuple[Any, Index]`
3) in the structured array case in to_arrays, return a list of empty ndarrays instead of list of empty lists, so we can further tighten to `Tuple[List[ArrayLike], Index]`
4) This breaks test_from_records_empty_with_nonempty_fields_gh3682, at which point we decide that the new behavior is better, so calling it a bugfix.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
+1 on calling this a bug fix, if you can add a whatsnew note and ping on green.
whatsnew added + greenish
CI: skip array-manager test
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

should get the CI back to green

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
lgtm merge in green
Thanks. Removing the skip again in https://github.com/pandas-dev/pandas/pull/39991, since that fixes it
PERF: DTA/TDA _simple_new disallow i8 values
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype
name: Performance
description: Memory or execution speed performance
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
thanks, assume the perf benefit is for microbenchmarks
FTR I think this fixes https://github.com/HypothesisWorks/hypothesis/issues/2894
API: silent overflow in Series(bigints, dtype="int8")
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

This is made complicated by a) #40110 and b) MaskedArray behavior, which i need to check but may be changeable.

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
So the current behaviour is to wrap around, which follows numpy:

```
In [5]: pd.Series([1, 1000], dtype="int8")
Out[5]: 
0     1
1   -24
dtype: int8

In [6]: np.array([1, 1000], dtype="int8")
Out[6]: array([  1, -24], dtype=int8)
```

Personally, I think it would be good to be more strict (and only allow safe casting by default) in the constructors, as you are doing in this PR. 
But I think this could (relatively easily) be done with a deprecation warning that it will raise in the future?
> But I think this could (relatively easily) be done with a deprecation warning that it will raise in the future?

It could, but leaving it until 2.0 will block most remaining simplification of the construction/casting code, which is bound to uncover more bugs.

We do explicitly catch OverflowError when passing negative integers and specifying a uint dtype, so there's a precedent for not matching numpy's behavior.
mothballing to clear the queue
CLN/TST: normalize test_frame_apply
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Some minor normalizations before breaking up or parametrizing these tests

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
@jorisvandenbossche there's an ArrayManager test currently failing on master; any obvious candidates for something recently-merged that might account for that?
@jbrockmendel @jorisvandenbossche - Just looking at recent commits, the earliest one I see failing is #40050. This failed in the PR as well. Both as well as this PR have a single test failure, `test_groupby_dtype_inference_empty`.
Yes, that PR was simply merged with failing tests .. Merging master will resolve it now thanks to Brock
BLD: updated mypy version to 0.812
State: closed

- [x] closes #40106 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


After making necessary changes
```bash
mypy pandas
```
O/P
```
Success: no issues found in 1237 source files
```
![Screenshot from 2021-02-27 19-13-10](https://user-images.githubusercontent.com/31338369/109389161-5157d080-7931-11eb-9ead-82a0c8ca0d91.png)


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
lgtm. @simonjayhawkins if you have any concerns.
Thanks @Muktan
STYLE exclude io from inconsistent namespace check
State: closed

xref https://github.com/pandas-dev/pandas/pull/40069#issuecomment-786931356

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
is this still needed? we merged the patch on master.
My thinking was that this'd prevent unnecessary future error messages from the hook, but maybe the correct thing would be to change such objects so they're imported directly from `pandas.io` (as was done in the patch) as that's more explicit anyway - closing then
DEPS: replace pyperclip with pyclip #39834
State: closed

- [x] closes #39834



Labels:
name: Dependencies
description: Required and optional dependencies

Comments:
wouldn't we need to include this as a dependency?
> thoughts @pandas-dev/pandas-core

Here's my 2 cents, and I will leave it to the more experienced members of the core team to weigh in.

The proposal is that we depend on `pyclip` instead of `pyperclip`.  Prior to today, I had never heard of either project, so my analysis is based on looking at their GitHub repos. 

`pyperclip` is a project that has been around for almost 10 years.  There has not been much activity lately in terms of commits , and it has 71 open issues. But the latest PR was merged on 2/20/21.  There have been multiple contributors over the 10 years.  As an outsider, I would say that it is probably a pretty stable piece of code. 

`pyclip` is a much younger project, started in January of 2021.  There is one contributor. So it is most likely not as "battle tested" as `pyperclip`.  I'd be a little concerned about having pandas depend on such a young project.

So unless `pyclip` is fixing an issue that affects pandas users that is a problem with using `pyperclip`, I'd vote for not making this change.

But others may feel differently, because `pyclip` is the more active project.
I'll only add that it appears pyperclip requires less on our side, which I love. But I agree with @Dr-Irv assessment. I feel  wary adding a 2 month old project, even as an optional dependency. I also wonder how used this feature is (SO gives 161 results for `pandas clipboard`)
I think this is a nearly interactive only function, so I'm not sure it will show up much in github. I have used it a few times in the past month in quick experiments. 
~@Dr-Irv you have an unfortunate type in the fourth paragraph.~ (fixed)
thanks @fangchenli let's reconsider this after the project is a bit more mature.
REF: unify casting logic in Categorical.__init__
State: closed

Big picture, trying to call maybe_infer_to_datetimelike in fewer places so we can simplify scattered datetimelike casting.


Labels:
name: Refactor
description: Internal refactoring of code
name: Categorical
description: Categorical Data Type

Comments:
pls have a check on perf, IIRC we were pretty sensitive here (prob unchanged but who knows)
BUG: df.loc setitem-with-expansion with duplicate index
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

also CategoricalIndex.insert

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
cc @phofl
I think we should deprecate indexing expansion, pretty sure we have an issue for this.
updted per comments + greenish
TST/REF: Consolidate tests in apply.test_invalid_arg
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Moves relevant tests into test_invalid_arg. Had to rename two tests because they were of the same name, but no other changes. I organized the file by the reason for the exception, and indicated this at the top of the file. Certainly open to any other ways of organizing.

Labels:
name: Refactor
description: Internal refactoring of code
name: Testing
description: pandas testing functions or related to the test suite
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
great. can you merge master to assure no actual failtures, ping on green.
lgtm. can you merge master.
thanks @rhshadrach
REG: DataFrame/Series.transform with list and non-list dict values
State: closed

- [x] closes #40018
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Currently on master, `agg` does some preprocessing on dict-like arguments that `transform` does not, leading to incorrect behavior in `transform`. This PR moves `agg`'s preprocessing into `validate_dictlike_arg` which is used by both `agg` and `transform`, fixing the issue.

As identified [in #40018](https://github.com/pandas-dev/pandas/issues/40018#issuecomment-785678614), this is a regression from 1.1 -> 1.2 (almost certainly from one of my PRs, but I haven't checked which). Backporting this to 1.2 will probably require a bit of a different fix.

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
seems fine. can you merge master. and then can backport (agreed may require some work on that branch).
thanks @rhshadrach pls track the backport itself as it likely won't merge cleanly
@meeseeksdev backport 1.2.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 47c6d16972e52db35b046e862b6f2aad79d7bf5b
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #40090: REG: DataFrame/Series.transform with list and non-list dict values'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-40090-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #40090 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
CI: typo fixup for ResourceWarning
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
can you merge master (all looks fine, just making sure)
ping gets us back to green.  note the scope of that expanded since you last looked
k thanks
REF: tighten types in maybe_downcast_numeric, maybe_downcast_to_dtype
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
looks like this broke a couple of checks in the code_checks and possibly docbuild, am updating #40088 to fix these
REF: tighter types in maybe_infer_to_datetimelike
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
great. +1 on things like this.
> great. +1 on things like this.

bunch of em on deck
[ArrayManager] TST: resample tests
State: closed

xref https://github.com/pandas-dev/pandas/issues/39146/

This will only pass once https://github.com/pandas-dev/pandas/pull/40050 is merged

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
can you merge master on this (merged the precursor)
API: dont-special-case datetimelike in setting new column
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
What's the change in behaviour exactly? It will no longer infer datetime, or will no longer *only* infer datetimes? 

(I mentioned that before, but I would find it helpful if you can generally provide a bit more context when opening PRs)
> (I mentioned that before, but I would find it helpful if you can generally provide a bit more context when opening PRs)

I generally prefer brevity in cases where the title is descriptive and the code change is small: one line of code and 1 affected test
> I generally prefer brevity in cases where the title is descriptive and the code change is small: one line of code and 1 affected test

"descriptive" for you (who has been doing the change and thus of course understands what it is about) is something else as descriptive enough for someone else who is not you ;) 
To be clear: the reason I gave this comment was exactly because the title was *not* descriptive for me. Without diving into the code myself, I don't know what special case you are talking about, or what it means in practice to no longer special case something.

It's not because it is only a 1 line change that it is necessarily easy to understand what it actually changes. To know that, I would need to know by heart all the possible inferences that `Series` does, vs what the `maybe_infer_to_datetimelike` function does. And no, I don't know this.


>  possible inferences that Series does, vs what the maybe_infer_to_datetimelike function does. And no, I don't know this.

So big picture, the goal of this and other recent/upcoming PRs is to make this clearer, or at least less.  In this case by removing usages of maybe_infer_to_datetimelike so we have fewer (ideally just one) flavor of casting.

maybe_infer_to_datetimelike(object_array_of_datetimes) will cast to dt64, but will not cast for object_array_of_intervals.  As a result, `df[newcol] = object_array_of_intervals` stays as object dtype.  This change makes `df[newcol] = object_array_of_intervals` infer to IntervalDtype.
REF: avoid object-dtype casting in Block.replace
State: closed

ATM we do `self.astype(object)`, where the idiomatic thing to do is `self.coerce_to_target_dtype(value)`, which in some cases means we dont need to cast on the back-end

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
is this user visible? if so can you add a whatsnew note, otherwise lgtm.
> is this user visible?

no, just more idiomatic
REF, TYP: factor out _mark_right from _add_legend_handle
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

This PR factors out adding ` (right)` to labels which are meant to go on the right y-axis. This does not change anything user-facing, and is simply a precursor to fixing #40044  and #39522 .

The change is that on `master`, only `leg.get_texts()` shows `b (right)` while `ax.right_ax.get_legend_handles_labels()` shows `b`. The former is used for constructing the user-facing legend. The reason I think it needs doing is that I believe that to address #40044  and #39522, we need to use `.get_legend_handles_labels()` for both handles and labels, while currently `.get_legend_handles_labels` is used for handles and `leg.get_texts()` is used for labels.

Putting a breakpoint after

https://github.com/pandas-dev/pandas/blob/ab687aec38668cdb0139ca02d6e37e8c8386ae7d/pandas/plotting/_matplotlib/core.py#L591

results in , on this branch:
```python
(Pdb) ax.right_ax.get_legend_handles_labels()
([<matplotlib.lines.Line2D object at 0x7fc94adf1dc0>], ['b (right)'])
```

and on master:
```python
(Pdb) ax.right_ax.get_legend_handles_labels()
([<matplotlib.lines.Line2D object at 0x7ff2892abbe0>], ['b'])
```

Labels:
name: Visualization
description: plotting

Comments:
codecov/patch shows the lines from `_mark_right_label` as not being covered - however, running `pytest pandas/tests/plotting/frame/test_frame_subplots.py -k test_subplots_dup_columns` hits them
@fangchenli I think the coverage check here is failing because the slow plotting tests' coverage isn't currently uploaded to codecov, any suggestions for how to deal with that? Should `posix.yml` be edited so there's a step at the end which uploads to codecov, like there is in `database.yml`?
> @fangchenli I think the coverage check here is failing because the slow plotting tests' coverage isn't currently uploaded to codecov, any suggestions for how to deal with that? Should `posix.yml` be edited so there's a step at the end which uploads to codecov, like there is in `database.yml`?

Yes, enable coverage for the build that actually run those tests and upload the report. Codecov'll automatically merge multiple reports. xref #39758
codecov check passes now @jreback 
thanks @MarcoGorelli I think it's fine to merge!
REF: move logic of 'block manager axis' into the BlockManager
State: closed

@jreback @jbrockmendel what are your thoughts about doing this? 
Currently the 0/1 -> 1/0 axis switch for the BlockManager happens on the DataFrame side, while I think it would be a bit cleaner to let this be the responsibility of the BlockManager method being called.

For illustration, the diff of this PR is such a change for the `diff()` method: instead of using `_get_block_manager_axis` in frame.py, it's the `BlockManager.diff()` method that takes care of this (so just moving it down a level). 

Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
+1
yep +1 on moving this to the BM.
I also added the change for `where` and `shift`, in addition to `diff`. For keeping the PR small, that's probably enough for a first PR.

Other uses cases are eg in `set_axis`, `reindex_indexer`, `take`.
can you merge master
once more merge master pls