BrandonFors/pandas-github-issues huggingface dataset:
CI: Migrate Python 3.10 testing to Posix GHA/Azure Pipelines
State: closed

- [ ] closes #43890
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration

Comments:
Can we backport this? It would be nice to keep CI as shared as possible between the release and main branches before the actual release is cut. 
This is also pretty low risk given that it only changes CI and not code.
yep ok to merge and backport 
thanks @lithomas1 
@meeseeksdev backport 1.4.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.4.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -m1 d437902f46acbff4a03d748b30620bc75fa5ea1f
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #45120: CI: Migrate Python 3.10 testing to Posix GHA/Azure Pipelines'
```

4. Push to a named branch:

```
git push YOURFORK 1.4.x:auto-backport-of-pr-45120-on-1.4.x
```

5. Create a PR against branch 1.4.x, I would have named this PR:

> "Backport PR #45120 on branch 1.4.x (CI: Migrate Python 3.10 testing to Posix GHA/Azure Pipelines)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.4.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -m1 d437902f46acbff4a03d748b30620bc75fa5ea1f
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #45120: CI: Migrate Python 3.10 testing to Posix GHA/Azure Pipelines'
```

4. Push to a named branch:

```
git push YOURFORK 1.4.x:auto-backport-of-pr-45120-on-1.4.x
```

5. Create a PR against branch 1.4.x, I would have named this PR:

> "Backport PR #45120 on branch 1.4.x (CI: Migrate Python 3.10 testing to Posix GHA/Azure Pipelines)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
REF: avoid object-casting in _get_codes_for_values
State: closed

Index.get_indexer_for ends up going through the same hashtable-based lookup anyway.  We add a little constructor overhead, but avoid object-dtype casting in cases where the common-dtype is e.g. numeric.

Labels:
name: Refactor
description: Internal refactoring of code
name: Categorical
description: Categorical Data Type

Comments:
wow, how's performance?
> how's performance?

Looks like a small improvement; i expect we could come up with cases that would go the other way:

```
import numpy as np
import pandas as pd

cats = np.arange(100)
values = np.arange(200).repeat(1000)

%timeit pd.Categorical(values, categories=cats)
911 µs ± 99.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <- master
880 µs ± 45.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <- PR

%timeit pd.Categorical(values, categories=["a", "b", "c"])
9.95 ms ± 66.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <- master
8.8 ms ± 239 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)    # <- PR

objvals = np.repeat(np.array(["a", "b", "c", "d"]), 1000)

%timeit pd.Categorical(objvals, categories=["a", "b", "c"])
478 µs ± 9.43 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <- master
443 µs ± 4.42 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <- PR
```
CI fail unrelated
BUG: to_xml raising for pd.NA
State: closed

- [x] closes #43903
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: IO XML
description: read_xml, to_xml

Comments:
hmm some failures that could be relevant
It looks like this was some kind of hiccup. But merging master to be sure
thanks @phofl
TST: Mock unstack and pivot_table test for PerformanceWarning
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

xref https://github.com/pandas-dev/pandas/pull/45084#issuecomment-1002708717

Makes these tests run quicker locally.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
thanks @mroeschke 
Thanks @mroeschke
Typ excel writer _base
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

If this is ok, I will type the other classes too


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@phofl The issue is already fixed in the just released pyright 1.1.202 :)

Feel free to remove the "Remove typevar" commit and bump the version of pyright [here](https://github.com/pandas-dev/pandas/blob/ae7916c56fd3323448fbdc136681162737229650/.github/workflows/code-checks.yml#L77) and [here](https://github.com/pandas-dev/pandas/blob/ae7916c56fd3323448fbdc136681162737229650/.pre-commit-config.yaml#L81).
@phofl ping when ready as @twoertwein ok here
Reverted the commit, let's see if it works
> Reverted the commit, let's see if it works

You also need to bump the pyright version (pyright is pinned).
Ah, thx. Should have expected that

Edit: And missed your comment above
Looks good now, cc @jreback
thanks @phofl
BUG: Fix nanosecond timedeltas
State: closed

- [x] closes #43764
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

Fixes the construction of `Timedelta` objects with any nanosecond contribution.

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Timedelta
description: Timedelta data type

Comments:
@simonjayhawkins no more PRs on 1.4 actively trying to reduce the number. we can certainly decide later to backport or merge on the release.
sure. we had the discussion on the last release about the milestone and adopted the blocker for rc label. Admittedly, I don't recall if that allowed adding new issues/prs to the milestone.
> sure. we had the discussion on the last release about the milestone and adopted the blocker for rc label. Admittedly, I don't recall if that allowed adding new issues/prs to the milestone.

right but the point is i don't want to label these with a version until we are ready to merge (otherwise these tend to stick for no good reason)
thanks @deponovo
DEPR: downcast kwd in fillna
State: closed

- [x] closes #40988
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

I'm on the fence about this.  `downcast="infer"` seems pretty useful, as evidenced by how often it shows up in the tests.  We don't have anything analogous to `infer_objects` for "see if i can losslessly downcast numeric values".

If we go forward with this, probably need to deprecate within NDFrame.interpolate too, as apparently some but not all interpolate paths see this deprecation.

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Deprecate
description: Functionality to remove in pandas

Comments:
> i'm on the fence about this. downcast="infer" seems pretty useful, as evidenced by how often it shows up in the tests

this was intended original to be a performance hit avoider, e.g. if the fillna ended up in object dtype we would automatically try to reconvert. but this seems silly as we do this elsewhere. so +1 on deprecating.

it was never intended to be a numeric downcaster (nor should it), we have to_numeric for that and it need to be explict
Mothballing pending re-thinking.
DEPR: make to_datetime('now') match Timestamp('now')
State: closed

- [x] closes #18705
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype
name: Deprecate
description: Functionality to remove in pandas

Comments:
@jbrockmendel small comment and a rebase and can get this in
CI: Ensure all minimum optional dependencies are tested
State: closed

- [x] tests added / passed

Progress towards https://github.com/pandas-dev/pandas/issues/29685

- `actions-38-minimum_versions.yaml` now contains all optional dependencies with their min versions
    - Other builds that have min version pinned are now removed
- `actions-38-db.yaml` -> `actions-38-downstream_compat.yaml`
    - Build for compat with libraries like statsmodels, pytorch, yaml, etc
- `actions-38-db-min.yaml` removed as redundant with `actions-38-minimum_versions.yaml`

Discoveries:
Min version of lxml and openpyxl with `.xlsm` or `.xlsx` doesn't seem to work. Need to bump to openpyxl to 3.0.9 https://openpyxl.readthedocs.io/en/stable/changes.html#id47

Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
@mroeschke 

>  Need to bump to openpyxl to 3.0.9

you updated the dev deps for this?

cc @lithomas1 can you review
> you updated the dev deps for this?

Don't think so. lxml and openpyxl don't have pins in `environment.yml` and `requirements-dev.txt` so the more recent versions should be installed.
thanks @mroeschke @lithomas1
BUG: head and tail not dropping groups with nan
State: closed

- [x] closes #45089
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

We could also create a specific function for this.

Also that nth respects ``as_index`` while it is ignored by ``head`` and ``tail`` looks inconsistent

Labels:
name: Bug
name: Groupby

Comments:
> We could also create a specific function for this.

what does this mean
Adjusting the mask for the dropna flag in a dedicated function, because the place I put this is a bit awkward
BUG: merge raising KeyError with differently named indexes and on keywords
State: closed

- [x] closes #45094
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
thanks @phofl keep em coming!
CI: Split CI workflow into code checks, docbuild & web, and arraymanager
State: closed

- [x] closes #37242
- [x] tests added / passed


`ci.yml` split into:

* `arraymanager.yml`
* `code-checks.yml` (includes `pre-commit.yml` now)
* `docbuild-and-upload.yml`

Labels:
name: CI
description: Continuous Integration

Comments:
Does this mean that if one fails we can re-run just that one?
> Does this mean that if one fails we can re-run just that one?

Yes where "one" is the entire workflow. 

e.g. If the arraymanager tests fail, the docbuld or code checks won't be re-run
e.g. If one code check fails, all the code checks still need to be rereun https://github.com/actions/runner/issues/432
I think it might be worth waiting for github/roadmap#271. Of course we could also merge this now and revert when that is implemented.
> I think it might be worth waiting for [github/roadmap#271](https://github.com/github/roadmap/issues/271). Of course we could also merge this now and revert when that is implemented.

github/roadmap#271 would make this a lot nicer (as well as our other workflows). 

I have slight preference just splitting now to make re-running things more focused in the short term. Additionally I think the splitting here has a better separation of concerns IMO.
thanks @mroeschke seems ok to do now
TST: Add regression tests for old issues
State: closed

- [x] closes #34455
- [x] closes #25412
- [x] closes #19136
- [x] closes #32619
- [x] closes #28613
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them



Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
ty @mroeschke
REF: de-duplicate DataFrame indexing code
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
awesome
wip REF: redirect `df.to_html` to Styler if new kwargs are used.
State: closed

This is an alternatative way of transitioning `DataFrame.to_html` to `Styler.to_html`.

It is a very minimalist introduction:

  - It will **fallback to the original HTMLFormatter implementation** if:
    - any of the deprecated kwargs are specifically input, and will show a **deprecation warning**.
    - **none** of the new Styler implementation kwargs are input.
  - It will **only use the new Styler implementation** if no deprecated kwargs are input and at least one **new** kwarg is used.


here is the rendered doc-strings:
<img width="602" alt="Screenshot 2022-01-02 at 15 46 11" src="https://user-images.githubusercontent.com/24256554/147883230-ff6e5e55-b30c-4429-bb9c-f6b4c6c0024c.png">
<img width="544" alt="Screenshot 2022-01-02 at 15 46 23" src="https://user-images.githubusercontent.com/24256554/147883233-d0b179a2-01ae-4a4a-b832-5666f13744dc.png">
<img width="571" alt="Screenshot 2022-01-02 at 15 46 34" src="https://user-images.githubusercontent.com/24256554/147883235-f37f1492-1526-47fe-aa73-db50f2a2e338.png">
<img width="561" alt="Screenshot 2022-01-02 at 15 46 45" src="https://user-images.githubusercontent.com/24256554/147883236-03132b3b-3626-4bb9-8948-07dc544b07a1.png">
<img width="574" alt="Screenshot 2022-01-02 at 15 46 56" src="https://user-images.githubusercontent.com/24256554/147883237-c589680e-4c8a-41e4-9c36-e8788701b364.png">
<img width="599" alt="Screenshot 2022-01-02 at 15 47 07" src="https://user-images.githubusercontent.com/24256554/147883238-c297586d-c5e4-4b19-87ea-4d1e64ccadb7.png">
<img width="606" alt="Screenshot 2022-01-02 at 15 47 17" src="https://user-images.githubusercontent.com/24256554/147883239-5e6607b9-dfc9-480a-97c2-3f35f2d855b4.png">
<img width="587" alt="Screenshot 2022-01-02 at 15 47 25" src="https://user-images.githubusercontent.com/24256554/147883240-658f752f-516d-4a80-963c-77a32289001d.png">
<img width="610" alt="Screenshot 2022-01-02 at 15 47 32" src="https://user-images.githubusercontent.com/24256554/147883242-6de1853c-04dc-4487-8173-41fd47c3f374.png">



Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@attack68 do you still want to work on this?
@datapythonista I do but it requires some pre requisites and a community consensus. There is a bit more progress on `DataFrame.to_latex`. Hopefully in future will try to use that as a template, so will close this for now.
BUG: Unstack/pivot raising ValueError on large result
State: closed

- [x] closes #26314
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
thanks @mroeschke 
@mroeschke this seems to significantly increase the run time running the tests locally.

also the tests fail with pytest-xdist... node down.
> @mroeschke this seems to significantly increase the run time running the tests locally.
> 
> also the tests fail with pytest-xdist... node down.

Addressed in https://github.com/pandas-dev/pandas/pull/45112. Now takes ~4 seconds locally.
BUG: Series.replace with value=None explicitly
State: closed

- [x] closes #36984
- [x] closes #19998
- [x] closes #44498
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

Not sure if there is a way to keep the default values rendered in the docs unchanged.

Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: replace
description: replace method

Comments:
i think doc-string needs updating

```
> /home/runner/work/pandas/pandas/pandas/core/generic.py:6801: DocTestFailure
_________________ [doctest] pandas.core.series.Series.replace __________________
5211     dtype: object
5212 
5213     When ``value=None`` and `to_replace` is a scalar, list or
5214     tuple, `replace` uses the method parameter (default 'pad') to do the
5215     replacement. So this is why the 'a' values are being replaced by 10
5216     in rows 1 and 2 and 'b' in row 4 in this case.
5217     The command ``s.replace('a', None)`` is actually equivalent to
5218     ``s.replace(to_replace='a', value=None, method='pad')``:
5219 
5220     >>> s.replace('a', None)
Differences (unified diff with -expected +actual):
    @@ -1,6 +1,6 @@
    -0    10
    -1    10
    -2    10
    -3     b
    -4     b
    +0      10
    +1    None
    +2    None
    +3       b
    +4    None
     dtype: object
```
very nice!
DEPR: resample/groupby.pad/backfill in favor of ffill/bfill
State: closed

- [x] closes #33396
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Groupby
name: Resample
description: resample method
name: Deprecate
description: Functionality to remove in pandas

Comments:
do we use 'pad' *anywhere* else? if so should deprecate those as well
> do we use 'pad' _anywhere_ else? if so should deprecate those as well

`method='pad'` and `method='backfill'` appear in several function like `reindex`, `interpolate` etc. Probably best tackled in subsequent PRs.
> > do we use 'pad' _anywhere_ else? if so should deprecate those as well
> 
> `method='pad'` and `method='backfill'` appear in several function like `reindex`, `interpolate` etc. Probably best tackled in subsequent PRs.

kk great
Groupby Agg not working if different partials with same name on the same column
State: closed

- [x] closes #28570
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry
a unit test was added in accordance with issue [28570](https://github.com/pandas-dev/pandas/issues/28570). It asserts the equality of the expected output with the tested df. 



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby

Comments:
Hello @aliraeisdanaei! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-01-10 15:17:01 UTC
Hey, this is my first pull request to pandas, so please do tell me if I have made any mistakes. 

Thank you & God bless,
Ali Raeisdanaei
@jbrockmendel Thanks for the feedback. 
Thanks for the PR, but it appears to have gone stale. If you'd like to continue and investigate the bug fix, please ping and we can reopen.
DEPR/ENH: support axis=None in min/max
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Deprecate the current axis=None behavior so we can change it in 2.0 xref #21597

Labels:
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: Deprecate
description: Functionality to remove in pandas

Comments:
what is the status of the other functions here? (e.g. sum, mean etc) do these need to do the same?
> what is the status of the other functions here? (e.g. sum, mean etc) do these need to do the same?

the idea is to do the same for all of them, but it was less obvious how to implement it for the others.  i guess if we're just doing the deprecation now, can deprecate them all at once.
kk yeah would want this for 1.4 but for all if possible (separate PRs are ok or all at once)
> separate PRs are ok or all at once

i'll try locally adding the deprecation for all of them and see if it Breaks The World.  if not, will just add to this PR.
CI: Fix pytorch installation error
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

The conda installation expects mkl locally, which does not work . So installing via pip for now to get ci back to green

Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
thanks @phofl
ENH: Index[bool]
State: closed

- [x] closes #15890
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Enhancement
name: Index
description: Related to the Index class or subclasses

Comments:
> Thoughts on deprecation path?

i think it might be ok to actually just change this, are you finding that you need to change test behavior?
>  are you finding that you need to change test behavior?

Yes.  Largely around factorize and value_counts where this branch is currently very ad-hoc.
looks ok for 1.4?
> looks ok for 1.4?

I'd rather not rush it.
> minor comment. ready to go?

just pushed with a fix.  i think ready, yes.
can you rebase again (as the scipy fix merged)
thanks @jbrockmendel very nice!