BrandonFors/pandas-github-issues huggingface dataset:
BUG: assert_index_equal ignoring names when check_order is false
State: closed

- [x] closes #47328 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This should get merged after #47325 (will probably have a conflict then)



Labels:
name: Bug
name: Testing
description: pandas testing functions or related to the test suite
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Index
description: Related to the Index class or subclasses

Comments:
Resolved conflicts
Thanks @phofl
REGR: Fix nan comparison for same Index object
State: closed

- [x] closes #47105 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Regression
description: Functionality that used to work in a prior pandas version
name: Index
description: Related to the Index class or subclasses

Comments:
thanks @phofl 
@meeseeksdev backport 1.4.x
Could not push to auto-backport-of-pr-47326-on-1.4.x due to error, aborting.
REGR: Avoid regression warning with ea dtype and assert_index_equal order False
State: closed

- [x] closes #47207 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
thanks @phofl 
@meeseeksdev backport 1.4.x
Could not push to auto-backport-of-pr-47325-on-1.4.x due to error, aborting.
BUG: pd.DataFrame.from_records() raises a KeyError if passed a string index and an empty iterable
State: closed

- [x] closes #47285(Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors
name: Stale

Comments:
Hello @Kelvingandhi! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-06-24 20:23:11 UTC
cc @phofl (from issue)

Hello @simonjayhawkins , Just wanted to follow up on this PR. I'm not sure, is it waiting for another approval or is it being checked or need more discussion. If you could provide your feedback, that would be appreciated. 
Did you address our comments?
> Did you address our comments?

Please check my response above.

https://github.com/pandas-dev/pandas/pull/47319#discussion_r901163462
@github-actions pre-commit
All checks are successful except these 2 which are keep getting canceled after 90 min timeout. Not sure what action is needed now.

- Windows-MacOS / windows-latest actions-38.yaml (pull_request)
- Windows-MacOS / windows-latest actions-310.yaml (pull_request)
@phofl @simonjayhawkins Have you gotten a chance to look at this 2 checks getting cancelled issue. https://github.com/pandas-dev/pandas/pull/47319#issuecomment-1165973025
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
CI: Pin PYTEST_WORKERS=1 for Windows builds due to memory errors
State: closed

1. Running tests without xdist is okay
2. It's not due to pyarrow v8 (I think)
3. Same error with `windows-2019` as `windows-latest`

Labels:
name: CI
description: Continuous Integration
name: Windows
description: Windows OS

Comments:
im not seeing a way to download the log file (which is too big to display in the browser)
> im not seeing a way to download the log file (which is too big to display in the browser)

https://github.com/pandas-dev/pandas/suites/6902339888/logs?attempt=2 is the direct download link for the 3.9 build
Looks like you're doing trial/error to figure out what test is failing?  LMK if i can be helpful
> Looks like you're doing trial/error to figure out what test is failing? LMK if i can be helpful

Yeah, I think there's a memory leak somewhere but for now I'm just skipping tests to hopefully skip the one that leaks. If you have time/ideas how to identify the leaking test(s) that would be helpful 
@jbrockmendel the first PR merged into main that exhibited these failures was https://github.com/pandas-dev/pandas/pull/47299. If you have a chance could you comb over that PR again to see if any change there could cause a memory leak?
The changes in tzconversion.pyx are the most plausible candidate, though
not a slam dunk

On Sun, Jun 19, 2022 at 1:27 PM Matthew Roeschke ***@***.***>
wrote:

> @jbrockmendel <https://github.com/jbrockmendel> the first PR merged into
> main that exhibited these failures was #47299
> <https://github.com/pandas-dev/pandas/pull/47299>. If you have a change
> could you comb over that PR again to see if any change there could cause a
> memory leak?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/47318#issuecomment-1159805646>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB5UM6AXF6OJIHNI4KNSIPDVP57BVANCNFSM5YSCUS4A>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>

lgtm to get ci green(ish) for now.
@simonjayhawkins should we backport to 1.4.x or just merge to main?
probably should backport as could miss things when ci is not green?
Okay merging to get the CI back to green for Windows builds
sure. have not seen windows failures on 1.4.x but if we don't know the root cause then no harm.
DOC: added example of valid input dict in dfgroupby.aggregate
State: closed

- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

This edit adds an example in [dfgroupby.aggregate](https://pandas.pydata.org/docs/reference/api/pandas.core.groupby.DataFrameGroupBy.aggregate.html), for the more complicated case where the input is a "dict of axis labels -> functions, function names or list of such". It also explicitly states that it is possible to use `list` as a "function" (where the output will be a list of values for each group in the target column, as seen below).

![image](https://user-images.githubusercontent.com/90637415/173214630-6bdc6072-3d00-496c-8d52-49390f7d7f4f.png)

Labels:
name: Docs

Comments:
Hello @jasonjiajs! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-06-13 22:48:39 UTC
Hi, this is my first attempt at contributing to the documentation. I've made what I think is a very small change, but it failed 2 tests and I'm not sure why. Would appreciate any help on this!
Great, thanks @jasonjiajs
ENH: Timestamp.normalize support non-nano
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Datetime
description: Datetime data dtype
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
> Also minor, I noticed some of the typings in the cython code for timezones are e.g. tzinfo tz when it looks like it should be tzinfo | None tz like in _from_value_and_reso.

good catch, will edit in an upcoming PR
BUG: Fix date_format and datetime_format arguments to ExcelWriter being ignored
State: closed

- [X] closes #44284
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Notes:
* The fix is as suggested in the [latest comment](https://github.com/pandas-dev/pandas/issues/44284#issuecomment-992227644) on the issue.
* I applied the fix to all the writers, not just Openpyxl. This was following the pattern of changes made in the PR where the bug was introduced: https://github.com/pandas-dev/pandas/pull/40430
* The fix is very straightforward (2-line change in each of 3 files); the challenge was adding tests for it, since to properly test it we need to inspect the formatting info on the Excel file. Crucially, there is not a common API for this within pandas across the different engines, so initially adding the test into the `TestExcelWriter` class, which is parametrized over the different writers, gave some quite awkward code branching between the different APIs for the different engines. So I decided to move the tests to the test files for each individual engine, though this leaves some duplication between them.

Potential outstanding issues:
* Moving the test from `TextExcelWriter` to the individual test files meant losing the `openpyxl-.xlsm` parameter, since there didn't seem to be a file where this extension was being tested.
* Even with the arguments being passed through, `ODSWriter` does not seem to respect them, and this may be a limitation of the ODF specification. This felt like it was out-of-scope for this bug. Consequently, there was not a test added for the ODF engine.

Labels:
name: Bug
name: IO Excel
description: read_excel, to_excel
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: Disallow non-increasing multi-index header arguments
State: closed

- [x] closes #47011 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- NA - Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I didn't add separate tests for read_excel, read_html, or read_fwf because they all pass their data to TextParser, where the header validation occurs. Let me know if you'd like separate tests for those.

Labels:
name: Bug
name: MultiIndex

Comments:
> I didn't add separate tests for read_excel, read_html, or read_fwf because they all pass their data to TextParser, where the header validation occurs. Let me know if you'd like separate tests for those.

If you could find if these are already tested, please reference them in this issue. Otherwise would be good to add test for these
ENH: Timestamp.replace support non-nano
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Datetime
description: Datetime data dtype
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
@mroeschke looks like the windows failure is affecting all PRs ATM?
> @mroeschke looks like the windows failure is affecting all PRs ATM?

Yeah :/ something must have updated recently
TYP: Expanding.__init__
State: closed


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
LGTM. Could you rebase? The code checks were not working when you originally opened this PR.
> LGTM. Could you rebase? The code checks were not working when you originally opened this PR.

This time it ran :) Might be worth skipping the typing tests only if building pandas fails.
Thanks @twoertwein 

> Might be worth skipping the typing tests only if building pandas fails.

I think this should be the case since we have `if: ${{ steps.build.outcome == 'success' }}` in the GHA job.
I think the typing tests are also skipped when the doc-build/tests fail?
Hmm that shouldn't be the case.

The docbuild is in a separate workflow, `docbuild-and-upload.yml`, from the typing checks, `code-checks.yml` which each build a separate version of pandas. So if the docbuild fails the typing checks in code-checks still run. 
Sorry, I meant the `Docstring and typing validation` workflow:

![image](https://user-images.githubusercontent.com/6618166/173249453-2210606c-0dab-4fea-a505-5c5778cc7c16.png)

Ah gotcha, yes correct. If any of the prior checks fail, then the whole workflow will stop. Feel free to add a `continue-on-error: true` in a subsequent PR for all the actual code checking steps
ENH: DTA to_pydatetime, time, timetz, date, iter support non-nano
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
This turned up a few other places where non-nano needs to get implemented first.  see #47316, #47312, #47313, and a few more coming up.
rebased on top of #47346
looks like some unrelated things snuck in the last rebase
TST: GH39126 Test barh plot with string and integer at the same column
State: closed

- [x] closes #39126
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests)
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Visualization
description: plotting

Comments:
Great! Thanks @arch1baald
BUG: StataWriterUTF8 removing some valid characters in column names (#47276)
State: closed

- [x] closes #47276
- [x] Tests added and passed
- [x] All code checks passed
- [x] Added an entry in  `doc/source/whatsnew/v1.5.0.rst` 


Labels:
name: IO Stata
description: read_stata, to_stata

Comments:
thanks @eirki
Use setup-conda action
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration

Comments:
@mroeschke
Will send more PRs with caching changes soon :) The current caching is broken/badly implemented 
Thanks @jonashaag, looking forward to the followups :)
Add run-tests action
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration

Comments:
@mroeschke 
Awesome, thanks @jonashaag
DOC: Additions/updates to documentation GH47282 
State: closed

- [x] closes #47282 


Add brief definitions of Series and DataFrame into the 10 Minute Guide [https://pandas.pydata.org/docs/dev/user_guide/10min.html#min](url). 

Labels:
name: Docs
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Closing as stale. @Capobiaj if you want to address the comments and continue working on this, please let us know to reopen.
PERF: reindex default fill_value dtype
State: closed

- [x] closes #45857
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.

Default fill_value (np.nan) to match interleaved_dtype when possible to avoid upcasting and allow for block consolidation. 

Using the example from the OP:

```
import pandas as pd
import numpy as np

df = pd.concat([
    pd.DataFrame(np.zeros((1000, 1000), dtype='f4')),
], axis=1).reindex(columns=np.arange(5, 1005))
print(df._data.nblocks) 

df.values
print(df._data.nblocks)  # <- now consolidating

%timeit df.values
```

main:
```
2
2
789 µs ± 135 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops 
```

PR:
```
2
1
2.01 µs ± 48.8 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
@jbrockmendel - any suggestions here? The test added in this PR is failing for certain builds. Unfortunately, I'm unable to reproduce the failures locally so I'm not sure where to start. 
@lukemanley do you happen to be using a windows machine or 32bit build locally?
There's a comment in NDFrame._reindex_with_indexers `# TODO: speed up on homogeneous DataFrame objects (see _reindex_multi)` that i think might be useful in these cases.  Could that be helpful?
> do you happen to be using a windows machine or 32bit build locally?

Nope. I'm unable to reproduce those errors on mac or 64bit windows. 

> There's a comment in NDFrame._reindex_with_indexers # TODO: speed up on homogeneous DataFrame objects (see _reindex_multi) that i think might be useful in these cases. Could that be helpful?

I'll take a look, thanks. 
I’m going to close this PR. I cannot reproduce the failures locally. I tried a few different things but was unsuccessful in identifying where the float64 dtype comes from. I do not see that behavior locally.