BrandonFors/pandas-github-issues huggingface dataset:
TST: Test CoW with DataFrame.items()
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Copy / view semantics

Comments:
@lithomas1 can you update this for Matt's small comment?
DOC: Add missing attributes and methods to MultiIndex API documentation
State: closed

- [x] xref #50496  (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: MultiIndex

Comments:
I see now that this only partially addresses the issue. I changes `closes` in the description to `xref` for now, so we don't close the issue yet if this is merged. But would be great if you can also fix the problem in the left side menu, where some methos/attributes of MultiIndex are under the MultiIndex object, and some are listed independently after `IndexSlice`.

The problem seems to be in https://github.com/pandas-dev/pandas/blob/main/doc/source/reference/indexing.rst#multiindex

I'm not fully sure how Sphinx manages those autosummaries. Seems like some of the attributes/methods are added automatically, while many others are added manually in different sections. This doesn't only affect `MultiIndex`.

Maybe I can suggest to move `IndexSlice`  to not be in the middle of `MultiIndex` in this PR, and then address the problem in the sidebar separately.

What do you think @Elvislim1991 ? Thanks for working on this!
Ya, I realize the issues on the sidebar as well. As you said, It happens on all the methods so I thought it's good to keep it. I think it will be cleaner to just show the higher level as well. I can try doing that as well in this PR. What do you think @datapythonista?

For the IndexSlice, I agree for aesthetics better to place it out of MultiIndex but close to it as IndexSlice is closely related to it. 
I expect this PR, including moving IndexSlice to be non-controversial and merged quickly. While grouping things in the sidebar can be trickier, and also more opinionated. If we use separate PRs, we'll get this part merged faster, and we won't need to keep reviewing the changes here while discussing about the sidebar grouping.

No big deal, since changes here are small, but that's why I suggested splitting the PRs this way. But up yo you.
Okay, I will just touch the IndexSlice for this PR. 
Ok, I will move back IndexSlice under MultiIndex. 
Thanks @Elvislim1991
ENH: Add fast array equal function for indexers
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

The all equal case is as fast as numpy on 2 million entries, the early exist is significantly faster obviously


Labels:
name: Copy / view semantics

Comments:
Scrolling through issues, it looks like this is basically #32339.
(Takeaway from there was that perf for non short-circuit case needed to be checked. It's pretty hard to beat numpy's SIMD extensions.)
Commented on this in the initial post. Both are running around 700micro seconds if the whole array is checked (2 million entries). Ours is actually a bit faster, but not by much. 

Numpy is doing some things that are more expensive, e.g. allocating a Boolean array and then calling all on it. +

Edit: Looks like NumPy is faster with less elements, but not by that much:

```
10**6
%timeit np.array_equal(left, right)
269 µs ± 5.64 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
%timeit array_equal_fast(left, right)
303 µs ± 2.44 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
```

and with 2 Million:
```
%timeit array_equal_fast(left, right)
701 µs ± 79.4 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
%timeit np.array_equal(left, right)
729 µs ± 7.5 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
```

Edit2: I'd try using this only for indexers in CoW in the beginning. The advantage is, that if an indexer is equal, we avoid a copy which brings a big speedup. And if it's not equal, we should find different values early on.
CI: Temporary pin sqlalchemy
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

looks like it's quite tricky to fix the sqlalchemy issues, so perhaps it's OK to just pin temporarily to get CI green again?


Labels:
name: CI
description: Continuous Integration

Comments:
I am quite optimistic that mine is ready now, but lets see what the ci says
@meeseeksdev please backport to 1.5.x
TST: Copy on Write for filter
State: closed

- [ ] xref #49473 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Copy / view semantics

Comments:
thx @lithomas1
BUG: parsing mixed-offset Timestamps with errors='ignore' and no format raises 
State: closed

This is a refactor I'd been wanting to do for a while. Fortunately, it closes #50585, and also addresses the performance issue reported in https://github.com/pandas-dev/pandas/issues/35296#issuecomment-1371347855

- [ ] closes #50585  (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I'd highly recommend setting "ignore whitespace" when reviewing this

Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
sure - it's actually changed already, because I've removed the try-except (within the try-except, within the try-except...who'd have thought all these nested try-excepts would cause issues 😄 ) which re-reraised ValueError as TypeError - now the original error is raised (see one of the fixed tests)

---

The performance gain on the snippet from #35296 is huge here BTW:

1.5.2:
```python
In [4]: %%time
   ...: df = pd.read_csv(csv_file, usecols=['ts_col'],
   ...:     date_parser=lambda x: pd.to_datetime(x, format=timestamp_format),
   ...:     parse_dates=['ts_col']
   ...: )
   ...:
   ...:
<timed exec>:1: FutureWarning:
        Use pd.to_datetime instead.

CPU times: user 21.7 s, sys: 9.87 ms, total: 21.7 s
Wall time: 21.7 s
```

Here:
```python
In [2]: %%time
   ...: 
   ...: df = pd.read_csv(csv_file, usecols=['ts_col'],
   ...:     date_parser=lambda x: pd.to_datetime(x, format=timestamp_format),
   ...:     parse_dates=['ts_col']
   ...: )
   ...: 
   ...: 
CPU times: user 306 ms, sys: 22.9 ms, total: 329 ms
Wall time: 331 ms
```

Thanks @sm-Fifteen for your report, this was really useful!
Thanks @MarcoGorelli
CI: Try with sqlalchemy warnings
State: closed

- [ ] xref #50558 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query
name: CI
description: Continuous Integration
name: Stale

Comments:
Thanks. Would the goal be to fully address https://github.com/pandas-dev/pandas/issues/50558?
Not sure if the con.begin() will fix everything. If yes, then this should cover the issue. 
cc @mroeschke I am optimistic that this will do it (hopefully :))

Do we want to keep the warnings active on main as well till we can test against alchemy 2.0 or should I remove the env variable again?
I'm okay keeping the sqlalchemy warnings active on main (should hopefully aspire for sqlalchemy 2.0 compatibility so keeping these active will hopefully help) 
CircleCI is green, this is a good sign :)

cc @MarcoGorelli 
The lite tests yes, I don’t have MySQL or Postgres set up locally…
i think ive got postgres running on my server.  would it help if i test this branch there?
I am leaning towards not fixing the auto commit problems now, since @cdcadman mentioned that he has a branch tackling these. The text problems are all fixed. 

But generally speaking, help is welcome here :)
@phofl , my branch needed some modifications due to 1.4.46 warnings, and I fixed it.  I don't mind if you use anything from that branch here: #48576 .
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
CLN: MultiIndex._union
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.

No need to special-case missing/NA following the fix in #49442

Labels:
name: MultiIndex

Comments:
> We have tests covering this?

yes, a handful in pandas/tests/indexes/multi/test_setops.py:
test_union_with_missing_values_on_both_sides
test_union_nan_got_duplicated
test_union_keep_ea_dtype_with_na
test_union_with_na_when_constructing_dataframe


cc @lukemanley The Minimum versions failure could be legit, could you have a look? Might be possible that the PerformanceWarning is avoided now
> cc @lukemanley The Minimum versions failure could be legit, could you have a look? Might be possible that the PerformanceWarning is avoided now

Yep, looks like the PerformanceWarning is now avoided. I've removed the warning check. 
Thanks @lukemanley
ENH: Add lazy copy for swapaxes no op
State: closed

- [ ] xref #49473 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Copy / view semantics

Comments:
Not totally sure what to do in the other copy case. This can return views in some cases, but copy is performed on the underlying array
> Not totally sure what to do in the other copy case. This can return views in some cases, but copy is performed on the underlying array

What I think we _ideally_ should do is in case that ``self.values`` is a view on the underlying data (i.e. only if we have a single block I assume) and so the swapped values is also a view of the original data, in that case we should construct a new BlockManager with a ref to the original manager's block. 

This can be done here using some internal APIs, but to do this properly, it might be easier to have this as a method on the Manager. 


I've used ndarray_to_mgr for now, that's basically what we need here.
> we will need to same logic in DataFrame.transpose() as well

Another thought here: now that we are limited to max 2 dimensions in NDFrame, isn't swapaxes just the same as transpose? (for the case i and j are not the same) We could then maybe also consolidate those implementations (eg swapaxes calling transpose when appropriate)


> Another thought here: now that we are limited to max 2 dimensions in NDFrame, isn't swapaxes just the same as transpose? (for the case i and j are not the same) We could then maybe also consolidate those implementations (eg swapaxes calling transpose when appropriate)

Sounds good! But would take a look as follow up, the implementations are quite different and want to understand how this would impact stuff. I think transpose is better, as it keeps dtypes more often if possible
Merging for now, happy to address comments in a follow up if there are any
BUG: is_numeric_dtype(ArrowDtype[numeric]) not returning True
State: closed

- [x] closes #50563 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Arrow
description: pyarrow functionality

Comments:
LGTM. failures look unrelated (same sqlalchemy issues hitting other PRs)
TYP: Better numeric test types
State: closed

Currently we have that:
* UNSIGNED_INT_NUMPY_DTYPES
* SIGNED_INT_NUMPY_DTYPES
* ALL_INT_NUMPY_DTYPES

have dtype `list[Dtype]`. This is a too broad dtype, and the better dtype for these objects is `list[NpDtype]`.

Extracted from newest version of #49560.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @topper-123
TYP: mypy fix in pandas/io/formats
State: closed

Fixes a mypy error I'm seeing:

```bash
mypy.....................................................................Failed
- hook id: mypy
- exit code: 1

pandas/io/formats/printing.py:269: error: Unused "type: ignore" comment
pandas/tests/test_register_accessor.py:16: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
pandas/tests/io/xml/test_xml.py:1127: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
pandas/tests/io/xml/test_xml.py:1306: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
pandas/tests/io/xml/test_to_xml.py:995: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
Found 1 error in 1 file (checked 1397 source files)
```


Labels:

Comments:
This probably re-enables the error? This was ignored because the ci was failing
Locally I currently get a mypy error on main, and this fixes the issue, so IDK if it's different on the CI? Let's wait and see if this PR passes the CI?
Did you upgrade IPython to 8.8.0? That is what caused the error on the ci
Ok, no, I'm on a lower version. I'll close this PR then.

Should the ipython entry in `requirements-dev.txt` be changed to `ipython >= 8.8.0`?
No, we don't want to pin to anything higher than the minimum dependency
Backport PR #50562 on branch 1.5.x (CI Ipython ci fixup again)
State: closed

Backport PR #50562: CI Ipython ci fixup again

Labels:
name: CI
description: Continuous Integration

Comments:
CI failure was related to the github quota when building the CI. The cache we're using now is different for each branch. Not sure how many website builds we can do before hitting the quota (assuming all CI workers use the same quota, that I don't think it's the case), but if it's just one, maybe it'd be good to increase the cache life to few hours instead of one. Please ping me if you see more web failures, and I'll have a look.
ok, thanks for taking look!
BUG: Fix more interchange Column.size method
State: closed

- [x] closes #50554 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Like https://github.com/pandas-dev/pandas/pull/48426 part of the updated interchange protocol introduced in 1.5


Labels:
name: Interchange
description: Dataframe Interchange Protocol

Comments:
thx @mroeschke 
FWIW, I think it would be useful to add a test that actually covers this
CI Ipython ci fixup again
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration

Comments:
Looks like with ipython 8.8.0, it's necessary to `# type: ignore` this line, as opposed to 8.7.0 (that's the difference in the environments between today and yesterday):
```
                                                  Name Version_x Version_y
0    ──────────────────────────────────────────────...      None      None
61                                            cfn-lint    0.72.5    0.72.6
68                                            coverage     7.0.2     7.0.3
144                                           identify    2.5.11    2.5.12
151                                            ipython     8.7.0     8.8.0
169                                       jupyter_core     5.1.1     5.1.2
377                                              scipy     1.9.3    1.10.0
399                                         sqlalchemy    1.4.45    1.4.46
```
thx @MarcoGorelli 
Probably due to https://github.com/ipython/ipython/pull/13858/
gh-49656:Removing unwanted line in `.pre-commit-config.yaml`
State: closed

- [x] closes #49656
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
thaanks @ramvikrams 
> thaanks @ramvikrams

thanks
(wait til 2.0) STYLE rename fixtures and enable redefined-outer-name on tests
State: closed

- [ ] closes #50553 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This changes a lot of files, but it's practically all automated

I've split this into 2 separate commits:
1. add the script / pre-commit hook, and run hook on all files
2. make a handful of manual adjustments (for when a fixture is set equal to another one to generate a cartesian product)

The vast majority of changes here are automated and driven by the script (which is 100% tested)

This is what the pytest docs recommend doing https://docs.pytest.org/en/stable/reference/reference.html#pytest-fixture :

> If a fixture is used in the same module in which it is defined, the function name of the fixture will be shadowed by the function arg that requests the fixture; one way to resolve this is to name the decorated function fixture_<fixturename> and then use @pytest.fixture(name='<fixturename>').

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
let's not bother