BrandonFors/pandas-github-issues huggingface dataset:
DOC: add note for ExtensionArray authors on relation between argmax/min and _values_for_argsort
State: closed

See discussion / conclusion in https://github.com/pandas-dev/pandas/issues/33276#issuecomment-1027853573

Labels:
name: Docs
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
thanks @jorisvandenbossche
Bug fix - Integer overflow in Series.astype('datetime64[D]')
State: closed

- [x] closes #28045 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Datetime
description: Datetime data dtype
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Hello @weikhor! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-03-19 09:02:12 UTC
I close this since I stuck this for long time. I get simpler task. Thank for review.
BUG: pd.concat produces frames with inconsistent order when concating the ones with categorical indices
State: closed

- [ ] closes #44099 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v1.5.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Categorical
description: Categorical Data Type
name: Stale

Comments:
Hello @tyuyoshi! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-03-25 07:30:01 UTC
> can you add the example from the original OP as well and put a test in pandas/tests/reshape/test_concat.py

Thanks! I added the test case in this commit(https://github.com/pandas-dev/pandas/pull/46019/commits/a66a60fe6284b7380303d514cacfac14f29693d8)
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to be stale. If interested in continuing, please merge in the main branch, address the review, and we can reopen.
CI: Unpin Cython on PyPy testing
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
thanks @lithomas1
PERF: join empty frame
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Similar to #45838, but for `DataFrame.join`. 

`DataFrame.join` already had a fast path for joining with an empty frame but it only covered a limited set of cases. This PR makes the fast path a bit faster and covers additional cases. 

```
import pandas as pd 
import numpy as np

N = 10_000_000

df = pd.DataFrame({'A': np.arange(N)})
df_empty = pd.DataFrame(columns=['B', 'C'], dtype='int64')

%timeit df.join(df_empty, how='inner')
932 ms ± 83.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)       <- main
285 µs ± 4.29 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)  <- PR



Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
> do we have sufficient asv's to cover this?

Added an asv to cover this.
@jreback - greenish, errors look unrelated
Could you merge in main one more time? (failures look unrelated but good to be sure)
@mroeschke - merged main. greenish again, let me know if you think the error is related
Thanks @lukemanley. The failure was unrelated
CI: Don't run xdist if pytest.mark.single_cpu
State: closed

- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Appears some of the single builds are still timing out. Checking if not activating xdist for this build does the trick

e.g. https://github.com/pandas-dev/pandas/runs/5206822069?check_suite_focus=true

EDIT: The timeouts are due to pyarrow engine in read_csv since mamba is installing pyarrow=2.0.0 (conda seems to install pyarrow=6.0.0 when unpinned)


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
close/open a couple of times to make sure this doesn't affect
@mroeschke can you comment on the status of the CI and/or your gameplan?
My ideal CI Goals:

1) Ensure we're testing PY3.8-10 on Linux/MacOS/Windows with all required + optional dependencies (min versions + unpinned)
   1a) Ensure all tests that are supposed to run are run
2) Eliminate flaky behaviors (Have the red x represent an actual addressable failure)

Current CI Status:

1) Testing PY3.8-10 on Linux/MacOS/Windows with required + optional dependencies (partial dependencies on MacOS)
   1a) Some tests are still not running that could be run
2) Inching closer to eliminating flaky behaviors (it's usually always the optional dependencies s3/pyarrow)

TODO:

1) Finish testing alignment: https://github.com/pandas-dev/pandas/issues/29685
2) Implement dependency `conda-lock` files to increase stability

Closing in favor of https://github.com/pandas-dev/pandas/pull/46059
Remove hack and change a benchmark to fail
State: closed

We are improving the use of asv for Pandas benchmarks. On this PR, I'm removing a hack on the .github/workflows/code-checks.yml file and creating an error on the `attrs_caching.py` file to make the benchmark fail and check the CI output.

cc @datapythonista 

Labels:

Comments:
Here the CI output for the Code Checks / ASV Benchmarks (pull_request)
<img width="1413" alt="image" src="https://user-images.githubusercontent.com/13194519/154178996-16e8267b-6c74-4030-a05c-d696ee82ee84.png">

> Here the CI output for the Code Checks / ASV Benchmarks (pull_request) <img alt="image" width="1413" src="https://user-images.githubusercontent.com/13194519/154178996-16e8267b-6c74-4030-a05c-d696ee82ee84.png">

Sorry, just saw this now. Seems like asv is not returning the right exit code then, and this hack is still needed. We should start by making sure the problem persists in the latest asv. But we may need to fix its problem first.
Closing for now, we can reopen when asv is fixed.
DEPR: groupby nuisance warnings
State: closed

Follow up to #41475

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

The warning currently generated for pivot_table says groupby - but it points to the correct line of user code, and I think it's easy to understand what's going on from a user perspective. E.g.

```
df = pd.DataFrame(
    {
        "A": ["foo", "bar"],
        "B": ["small", "large"],
        "C": [1, 2],
        "D": ["3", "4"],
    }
)
result = pd.pivot_table(df, values=["C", "D"], index=["A"], columns=["B"], aggfunc="sum")
```

gives


> scratch.py:61: FutureWarning: Dropping invalid columns in DataFrameGroupBy.sum is deprecated. In a future version, a TypeError will be raised. Before calling .sum, select only columns which should be valid for the function.
  result = pd.pivot_table(df, values=["C", "D"], index=["A"], columns=["B"], aggfunc="sum")

cc @jbrockmendel 

Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
The deprecations are essentially correct, but for the wrong reason. They should be deprecated due to #46072 and not #41475. Will be reworking this.
BUG: to_csv not respecting float_format for Float64
State: closed

- [x] closes #45991 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO CSV
description: read_csv, to_csv

Comments:
thank you @phofl 
@phofl I think this is failing on main on Windows builds https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=74146&view=logs&jobId=2b18e8df-2ee7-5e01-73e9-0533d995635b&j=2b18e8df-2ee7-5e01-73e9-0533d995635b&t=13dd5051-9e2f-5a32-09fe-1d338e0833eb

```
_________________ TestToCSV.test_to_csv_float_ea_float_format _________________
[gw1] win32 -- Python 3.10.2 C:\Miniconda\envs\pandas-dev\python.exe

self = <pandas.tests.io.formats.test_to_csv.TestToCSV object at 0x0000025FCC572B00>

        def test_to_csv_float_ea_float_format(self):
            # GH#45991
            df = DataFrame({"a": [1.1, 2.02, pd.NA, 6.000006], "b": "c"})
            df["a"] = df["a"].astype("Float64")
            result = df.to_csv(index=False, float_format="%.5f")
            expected = """a,b
    1.10000,c
    2.02000,c
    ,c
    6.00001,c
    """
>           assert result == expected
E           AssertionError: assert 'a,b\r\n1.100...6.00001,c\r\n' == 'a,b\n1.10000...\n6.00001,c\n'
E             - a,b
E             + a,b

E             ?    +
E             - 1.10000,c
E             + 1.10000,c

E             ?          +
E             - 2.02000,c...
E             
E             ?      +
E             - 2.02,c...
E             
E             ...Full output truncated (9 lines hidden), use '-vv' to show
```
Thx @mroeschke, opened #46024
BUG: PeriodArray subtraction returning wrong results
State: closed

- [x] closes #45999 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Period
description: Period data type

Comments:
@phofl Thanks a lot for fixing this!!!
TST: Don't mark all plotting tests as slow
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Ideally we should be more targeted regarding a slow test

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
hmm windows builds taking a long time (but succeeding)
> hmm windows builds taking a long time (but succeeding)

Added some `@pytest.mark.slow` to some of the plotting tests (ones in the top 30 slowest)
hmm this are you sure we should not mark these slow? seems to be increasing the ci times
> hmm this are you sure we should not mark these slow? seems to be increasing the ci times

That would mean _no_ plotting tests would run on MacOS/Windows builds since we're running `not slow`. I was able to discover a PY3.10 entrypoints bug since we weren't running these tests before

IMO testing platform coverage should be more important than CI times since a lot of pandas development is pretty async
can we run a slow build on those platforms?
> can we run a slow build on those platforms?

I can follow up by removing `not slow` from these builds
can you rebase
can we reduce the matrix a bit to lower the number of total jobs?
Hmm the current jobs were deemed "important" at some point. If we need to prune jobs, my vote would be to remove the 2 locale jobs first.

* Runs a lot of redundant tests
* Relevant tests cannot be targeted
* Encourage users to locally tests, patches accepted, but not practical to test (every locale) in our CI. Similar to our 32 bit testing eventually: https://github.com/pandas-dev/pandas/issues/44453
We could also remove testing pyarrow 6. I think that would reduce 3 builds.
for pyarrow oldest and newest are fine (are we testing more)?
We're testing 1.0.1 (min), 5, 6, 7 right now. (we're not testing 2 or 3 since we've observed read_csv timeouts with those versions)

IIRC @jorisvandenbossche aimed to test more pyarrow versions since some pandas-pyarrow functionality is only available in certain versions.


ok then dropping 6 is fine as 5,7 are testing
REF: Share NumericArray/NumericDtype methods
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
> Any recommendations on where to start? Would I need to essentially create MaskedArrowArray and go from there?

I've been thinking we should refactor as much of ArrowStringArray as possible out into a general arrow-backed EA.  Not sure if that could be a fully-formed EA or if it would need to be a mixin.  (also in the tests there's an ArrowExtensionArray but I think its half-baked)

FWIW my current thought with MaskedArray is to try to get as much of NumericArray and BooleanArray pushed down into MaskedArray as possible so it can just become a wrapper around an arbitrary non-masked array (including e.g. other EAs)
luv it
CI: Add single_cpu build
State: closed

- [x] Closes #45705
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


As well as marking all pyarrow csv engine tests as `pytest.mark.single_cpu` since they use threads


Labels:
name: CI
description: Continuous Integration

Comments:
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.
TST: Clean tests/plotting
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

* Remove old, unsupported compat checks/code
* Minimize setup_method in favor of fixtures
* parameterize tests


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Visualization
description: plotting

Comments:
WOW!
A good cleanup!  🧹
Removed broken link from NDFrame#interpolate docs.
State: closed

The link at https://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html pointing to the SciPy Interpolation tutorial is broken. An alternative tutorial was not found in the new documentation.

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Hello @bicarlsen! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-02-15 07:47:49 UTC
/precommit
https://github.com/pandas-dev/pandas/runs/5191233428?check_suite_focus=true

is failing i think you need to remove a line
Thanks @bicarlsen
DOCS: Add comment about index_col and forward filling missing values
State: closed

- [x] closes #44837 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


cc @rhshadrach 

Labels:
name: Docs
name: IO Excel
description: read_excel, to_excel

Comments:
Greenish @rhshadrach 
Thanks @phofl
BUG: groupby.size and groupby.transform('size') incorrect for axis=1
State: closed

- [x] closes #45715 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Groupby

Comments:
thanks @rhshadrach
TST: Simplify test_register_writer
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Excel
description: read_excel, to_excel
name: Clean

Comments:
Awesome, thanks @rhshadrach
REGR: drop raising with ea index and duplicates
State: closed

- [x] closes #45860 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

We handle this case similarly for loc

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Index
description: Related to the Index class or subclasses

Comments:
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.
Test case dataframe.groupby incorrect with multiindex and None value
State: closed

- [✔] closes #32492 
- [✔] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [✔] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: MultiIndex

Comments:
Hello @weikhor! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-02-15 12:03:06 UTC
lgtm @phofl merge when ready
Thanks @weikhor
BUG: Convert <br> to space in `pd.read_html`
State: closed

- [x] closes #29528
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Labels:
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap
name: Stale

Comments:
since `<br>` is a line break is it better to convert it to `\n`?
> since `<br>` is a line break is it better to convert it to `\n`?

I do convert it to `\n` in the code - however pandas either replaces it or renders it as a space. I'm not sure if it is worth forcing the matter. Thoughts?
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
> This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.

Ping @attack68 
Can you merge master.

This looks good to me, the solution opens the door to a list of general replacements if we also want to go that way in the future.
> needs merge

OK now?
@attack68 ok here?
thanks @abmyii 
> > since `<br>` is a line break is it better to convert it to `\n`?
> 
> I do convert it to `\n` in the code - however pandas either replaces it or renders it as a space. I'm not sure if it is worth forcing the matter. Thoughts?

function `_remove_whitespace` caused it

![image](https://github.com/user-attachments/assets/b0ac82b9-7574-4bd3-ab59-4db7cfeee820)

@attack68 @abmyii