BrandonFors/pandas-github-issues huggingface dataset:
MAINT: Fix some typos.
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
thanks @Yulv-git
CI: Move 32 bit Linux build to GHA
State: closed

- [x] closes #46351 (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature

Also removes references to Azure since no builds run there anymore.

Labels:
name: CI
description: Continuous Integration

Comments:
thanks @mroeschke 
@meeseeksdev backport 1.4.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.4.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 97b99797d7e3ba605ff9d7e039739a529936a1d8
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #47020: CI: Move 32 bit Linux build to GHA'
```

4. Push to a named branch:

```
git push YOURFORK 1.4.x:auto-backport-of-pr-47020-on-1.4.x
```

5. Create a PR against branch 1.4.x, I would have named this PR:

> "Backport PR #47020 on branch 1.4.x (CI: Move 32 bit Linux build to GHA)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
REF: merge datetime_to_datetime64 into array_to_datetime
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code
name: Datetime
description: Datetime data dtype

Comments:
looks fine, can you merge master.

cc @mroeschke 
Looks like doctests are failing, otherwise LGTM
The failed doctest looks like the new behavior is just better than the old behavior.

```
1001     - Finally, mixing timezone-aware strings and :class:`datetime.datetime` always
1002       raises an error, even if the elements all have the same time offset.
1003 
1004     >>> from datetime import datetime, timezone, timedelta
1005     >>> d = datetime(2020, 1, 1, 18, tzinfo=timezone(-timedelta(hours=1)))
1006     >>> pd.to_datetime(["2020-01-01 17:00 -0100", d])
Expected:
    Traceback (most recent call last):
        ...
    ValueError: Tz-aware datetime.datetime cannot be converted to datetime64
                unless utc=True
Got:
    DatetimeIndex(['2020-01-01 17:00:00-01:00', '2020-01-01 18:00:00-01:00'], dtype='datetime64[ns, pytz.FixedOffset(-60)]', freq=None)
```
Yup agreed. Looks like this example could be removed from the docstring
Or rather, this example this doctest be corrected and added as a unit test
@jbrockmendel  looks like this is still failing the doc tests

```
=================================== FAILURES ===================================
______________ [doctest] pandas.core.tools.datetimes.to_datetime _______________
1013 
1014     >>> pd.to_datetime(['2018-10-26 12:00 -0530', '2018-10-26 12:00 -0500'],
1015     ...                utc=True)
1016     DatetimeIndex(['2018-10-26 17:30:00+00:00', '2018-10-26 17:00:00+00:00'],
1017                   dtype='datetime64[ns, UTC]', freq=None)
1018 
1019     - Inputs can contain both naive and aware, string or datetime, the above
1020       rules still apply
1021 
1022     >>> pd.to_datetime(['2018-10-26 12:00', '2018-10-26 12:00 -0530',
UNEXPECTED EXCEPTION: NameError("name 'timezone' is not defined")
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/pandas-dev/lib/python3.8/doctest.py", line 1336, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest pandas.core.tools.datetimes.to_datetime[18]>", line 4, in <module>
NameError: name 'timezone' is not defined
/home/runner/work/pandas/pandas/pandas/core/tools/datetimes.py:1022: UnexpectedException
```
Unpin xarray
State: closed

- [X] closes [#42716](https://github.com/pandas-dev/pandas/issues/42716) 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Dependencies
description: Required and optional dependencies

Comments:
> looks like this is breaking a doctest example

xref https://github.com/pandas-dev/pandas/issues/42716#issuecomment-886271371 for the doctest issue (the mypy part seems fine now)
It seems that the CircleCI: test-arm is not passing. Upon inspecting the error, it seems there might be a [numpy dependency missing](https://app.circleci.com/pipelines/github/pandas-dev/pandas/11048/workflows/10dc60d8-ff99-407f-8f7d-ee5d065df965/jobs/41887?invite=true#step-102-1166). I think that the previous doctest issue is fixed now, but will check back in the morning.
CI: Ensure no-use-pep517 with no-build-isolation with new pip version
State: closed

xref https://github.com/scipy/oldest-supported-numpy/issues/53

We may even be able to replace no-build-isolation with no-use-pep517 https://github.com/scikit-learn/scikit-learn/pull/23339 but that could be a followup.

Labels:
name: Build
description: Library building on various platforms
name: CI
description: Continuous Integration

Comments:
can you merge master (i merged the 32-bit build one first)
Thanks @mroeschke 
@meeseeksdev backport 1.4.x
TYP: resolve mypy ingores in core/indexing.py
State: closed

xref https://github.com/pandas-dev/pandas/issues/37715

Notes: `axis` has to be passed as an argument to `_tupleize_axis_indexer` in order to assert that it is not `None`. Since this kind of decouples it from the `_LocationIndexer` class, I made it a standalone utility function.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @iasoon
ENH: do not sort resulting columns when sort=False
State: closed

- [x] closes #17041
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Currently, `pivot_table` sorts the resulting columns even when `sort=False`. For example, the following code snippet produces the column ["age", "height"] in alphabetical order instead of the order in the original `df` or passed by `values=`. 

```py
import pandas as pd

df = pd.DataFrame(
    {
        "firstname": ["John", "Michael"],
        "lastname": ["Foo", "Bar"],
        "height": [173, 182],
        "age": [47, 33],
    }
)
result = df.pivot_table(index=["lastname", "firstname"], values=["height", "age"], sort=False)
print(result)
```

```
                    age  height
lastname firstname             
Foo      John        47     173
Bar      Michael     33     182
```

This PR fixes this issue by not sorting the resulting columns when `sort=False`

FYI, #17041 has some discussion about the order of the columns. PR #40954 added the keyword `sort` to `pivot_table` and was merged in v1.3. 

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Hello @ShawnZhong! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-05-15 18:21:51 UTC
I think ok to close https://github.com/pandas-dev/pandas/issues/17041 with this PR (if you can add a test for this).

also pls add a whatsnew note. put in reshaping bug fixes section for 1.5
I have added the whatsnew note and changed the PR description to close the issue. 

Is there a specific case that you want me to add a test for? 
thanks @ShawnZhong
CLN: DatetimeTZBlock don't override values_for_json
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO JSON
description: read_json, to_json, json_normalize
name: Clean

Comments:
looks good, if you can merge master & mark as ready
Thanks @lithomas1
CI/TST: Fix test for pyarrow 8.0 release
State: closed

- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Arrow
description: pyarrow functionality

Comments:
@meeseeksdev backport 1.4.x
ENH: Implement nlargest and nsmallest for DataFrameGroupBy
State: closed

- [ ] closes #46924 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v1.5.0.rst` file if fixing a bug or adding a new feature.

Try to implement `nlargest` and `nsmallest` methods for `DataFrameGroupBy`. Appreciate any comments.

```python
>>> df
  group name  data
0   bar   b1     1
1   bar   b2     3
2   bar   b3     3
3   foo   f1     1
4   foo   f2     1
5   foo   f3     3


>>> df.groupby("group").nlargest(n=2)
        name  data
group
bar   1   b2     3
      2   b3     3
foo   5   f3     3
      3   f1     1


>>> df.groupby("group").nlargest(n=2, keep="all")
        name  data
group
bar   1   b2     3
      2   b3     3
foo   5   f3     3
      3   f1     1
      4   f2     1

>>> df.groupby("group").nsmallest(n=2)
        name  data
group
bar   0   b1     1
      1   b2     3
foo   3   f1     1
      4   f2     1


>>> df.groupby("group").nsmallest(n=2, keep="all")
        name  data
group
bar   0   b1     1
      1   b2     3
      2   b3     3
foo   3   f1     1
      4   f2     1
```

Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
TST: Assign back multiple column to datetime
State: closed

This tests make sure when converting multiple columns to datetimes
and when assiging back it remains as datetime not as unix date
as mentioned in GH #20511.

- [x] closes #20511
- [x] [Tests added and passed]
- [x] All [code checks passed]


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
thanks @parthi-siva
CI: Move MacOS build from Azure to GHA
State: closed

- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Similar to https://github.com/pandas-dev/pandas/pull/46960


Labels:
name: CI
description: Continuous Integration
name: OS X
description: Related to Mac OS & hardware issues (M1)

Comments:
> Thanks for doing this. Should we add slow builds too? (I know we don't run any currently, but would be nice to have that coverage on Windows and MacOS too)

no objections, though not sure this adds a whole lot (but let's do in another PR).
@meeseeksdev backport 1.4.x
> Thanks for doing this. Should we add slow builds too? (I know we don't run any currently, but would be nice to have that coverage on Windows and MacOS too)

I can try removing the `not slow` mark from the current Windows/MacOS builds (I'm not fond of create whole separate build for those test)
TST: avoid chained assignment in tests outside of specific tests on chaining
State: closed

A small part of the test changes from #46958 that can be done separately.

We have specific tests about chained indexing (eg `test_chaining_and_caching.py`), so outside those specific tests, I can think we can avoid using chained indexing (regardless of #46958, this would follow our own recommendation on best indexing practices, although it also shows that some cases of mixed positional/label based setting is somewhat convoluted)

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @jorisvandenbossche
follow-up to 44787, use pandas compat for platform specifics in added test
State: closed

follow-up to #44787, as requested by @mroeschke in https://github.com/pandas-dev/pandas/pull/44787#issuecomment-1120292009.

Updates one added test to use `is_platform_windows()` for dealing with carriage returns. 

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
The Docs test fails, but I don't think that's related.
Awesome, thanks for the follow up!
WIP: ENH Add int[pyarrow] dtype
State: closed

- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
didn't like the parametrized-dtype idea?
> didn't like the parametrized-dtype idea?

Hmm I was mainly mimicking your structure for the nullable numeric dtypes. What's the parameterized-dtype route?

EDIT: Oh do you mean like

```
pd.array([1, 2], dtype="Int64")
pd.array([1, 2], dtype="Int64[pandas]")  # same as above
pd.array([1, 2], dtype="Int64[pyarrow]")


...

class IntegerDtype(...)

   def construct_from_string(string):
       if string == "Int64":
           return cls(storage="pandas")
       if string == "Int64[pyarrow]":
           return cls(storage="pyarrow")
```
The suggestion is to have a single `PyarrowDtype` class and instances of `PyarrowDtype[int64]` instead of a bunch of `FooPyarrowDtype` classes.

If it were up to move we'd move to having just `NullableDtype[foo]` instead of `FooNullableDtype` as well. (and just MaskedArray instead of BooleanArray/IntegerArray/FloatingArray, kinda xref #43002)
Hmm okay I think I am starting to understand. So to confirm with relevant examples:


```
class ArrowDtype(StorageExtensionDtype):

    def __init__(self, storage, pa_dtype):
        # validate
        self.storage = storage
        self.pa_dtype = pa_dtype

    def _is_numeric(self):
        return pa.types.is_integer(self.pa_dtype) or pa.types.is_float(self.pa_dtype)


class ArrowExtensionArray:

    def __init__(self, values, pa_dtype):
        # validate
        self.storage = storage
        self._dtype = ArrowDtype("pyarrow", pa_dtype)

    @cache_readonly
    def dtype(self) -> NumericArrowDtype:
        return self._dtype

    @classmethod
    def _from_sequence_of_strings(cls, strings, *, dtype=None, copy: bool = False):
        if self.dtype._is_numeric:
            from pandas.core.tools.numeric import to_numeric

            scalars = to_numeric(strings, errors="raise")
        elif other_dtype:
            # something else
        return cls._from_sequence(scalars, dtype=dtype, copy=copy)

    def mean(self):
        if self.dtype._is_numeric:
            # call pyarrow compute mean
        else:
            # raise TypeError?
```


thats pretty much what i had in mind, yah
@mroeschke I tried to do a similar approach for the floating[arrow] dtype, based on your discussion:
- https://github.com/pandas-dev/pandas/pull/47027
Closing in favor of https://github.com/pandas-dev/pandas/pull/47034
TYP: enable reportOverlappingOverload
State: closed

reportOverlappingOverload needs a newer pyright version (otherwise it has a few false positives). I intended to update pyright together with mypy in #46905 first, but mypy still hasn't been updated on conda-forge.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
can you merge master, assume the docs failure is unrelated.
rebased and green
Fix link to be *true* raw data
State: closed

Which would be handy for `wget`

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
thanks @theidexisted
CI: Move Windows build from Azure to GHA
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature

Took parts from https://github.com/pandas-dev/pandas/pull/46611


Labels:
name: CI
description: Continuous Integration
name: Windows
description: Windows OS

Comments:
@meeseeksdev backport 1.4.x
thanks @mroeschke
DOC: fix cookbook groupby & transform example
State: closed

In the example, all negative values of a group should be replaced by the mean of the rest of the group. The linked stackoverflow output has it right: https://stackoverflow.com/questions/14760757/replacing-values-with-groupby-means

We have to negate the first argument of `DataFrame.where` here, since this should be the condition when the value stays the same (and this should be the case for `g>=0`.)

pandas `df.where` behavior is pretty weird when coming from Spark world :)


- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @julianste
ENH: plot.scatter passes errorbar specific keyword arguments to ax.er…
State: closed

…rorbar() (#46952)

- [x] closes #46952 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: Visualization
description: plotting
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, please explore the feasibility of https://github.com/pandas-dev/pandas/pull/46954#discussion_r882161765, and we can reopen.
CLN: Remove special case for rank in groupby.ops
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby
name: Clean

Comments:
@mroeschke - I realized I could removed the TODO here. While I don't know why we couldn't do `out[:, k] = result` before, we certainly can't do it now.
Thanks @rhshadrach
Fix for issue #46870 DataFrame.select_dtypes(include='number') includes BooleanDtype 
State: closed

- [x] closes #46870 (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Stale

Comments:
Output for the following test has been confirmed after installing this branch:
```python
df = pd.DataFrame({
    'a': [1, 2, 3],
    'b': pd.Series([True, False, True], dtype=pd.BooleanDtype()),
})
print(df.select_dtypes(include='number').columns)
```
Returns:
Index(['a'], dtype='object')
> always add tests

and a release note in doc/source/whatsnew/v1.4.3.rst
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks @MichaelRocchio for the PR. closing as stale. (needs green ci, tests for issue code sample and release note.)

ping if you want to continue and will re-open.