BrandonFors/pandas-github-issues huggingface dataset:
DEPR: deprecate exposing blocks in core.internals
State: closed

Motivation: tried to clean up the blocks code bc DatetimeTZBlock and DatetimeLikeBlock are no longer really needed.  Found that we can't do that bc DatetimeTZBlock is exposed in the pseudo-public API.  That got me looking at `internals.__init__`, where comments suggest that we are importing the block subclasses for io.pytables (which no longer needs them) and io.packers (which no longer exists).  So we could remove them directly, but apparently a recent PR deprecated NumericBlock and ObjectBlock so this piggy-backs on that deprecation (and adds a missing test).

In internals.api I need to deprecate DatetimeTZBlock in order to eventually get rid of it entirely.  Also deprecating Block and ExtensionBlock is orthogonal just to wean downstream packages off of them.

cc @jorisvandenbossche 

xref https://github.com/pandas-dev/pandas/issues/40226

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Deprecate
description: Functionality to remove in pandas

Comments:
As mentioned in https://github.com/pandas-dev/pandas/issues/40226, pyarrow accesses DateimeTZBlock, but never instantiates it. So in theory that use case could easily be kept without a full working class. Now, the only reason we were using it was to pass it to `make_block`, and that's something we can easily remove in pyarrow, which I am doing in https://github.com/apache/arrow/pull/38321.  
So from pyarrow's point of view, this deprecation is fine.

If there are no more comments, I plan to merge this in a week.
> Can't comment on the exact line, but can you fix the stacklevel to 2 for this DeprecationWarning?

Updated. Can you confirm this is what you have in mind
Yep, that's what I meant
PERF: groupby aggregations on pyarrow timestamp and duration types
State: closed

- [x] closes #55031
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.0.rst` file if fixing a bug or adding a new feature.

Seeing ~identical timing between pyarrow and numpy in the OP example with this PR. 

```
import pandas as pd
import numpy as np

N = 1_000_000

df = pd.DataFrame(
    {
        "group": np.arange(N//1000).repeat(1000),
        "timestamp": pd.array(np.arange(N), dtype="timestamp[ns][pyarrow]"),
        "duration": pd.array(np.arange(N), dtype="duration[s][pyarrow]"),
    }
)

gb = df.groupby("group")

%timeit gb["timestamp"].max()
# 215 ms ± 9.72 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)   <- main
# 13 ms ± 839 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  <- PR

%timeit gb["duration"].max()
# 272 ms ± 2.72 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)     <- main
# 12.8 ms ± 341 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  <- PR
```

Labels:
name: Datetime
description: Datetime data dtype
name: Groupby
name: Performance
description: Memory or execution speed performance
name: Arrow
description: pyarrow functionality

Comments:
Thanks @lukemanley
TYP: Correct type annotation for to_dict.
State: closed

The `into` argument of DataFrame.to_dict and Series.to_dict can be either a class or instance of a class of dict; this is covariant - subclasses of dict can also be used. The argument was annotated as `type[dict]` though, so type checkers marked passing initialized objects (required for collections.defaultdict) as an incorrect argument type.

Fix by annotating `into` to take either a subclass of dict or an initialized instance of a subclass of dict.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @jsspencer
Use pandasSQL transactions in sql test suite to avoid engine deadlocks
State: closed

When trying to integrate the engine fixtures into these tests I have been running into a lot of deadlocking issues. From some [research](https://github.com/sqlalchemy/sqlalchemy/issues/5405#issuecomment-646607306) it looks like SQLAlchemy strongly discourages running doing [Connectionless Execution](https://docs.sqlalchemy.org/en/13/core/connections.html#connectionless-execution-implicit-execution) 

Note the comment:

> Modern SQLAlchemy usage, especially the ORM, places a heavy stress on working within the context of a transaction at all times; the “implicit execution” concept makes the job of associating statement execution with a particular transaction much more difficult.

We probably need an overhaul of the sql module to more natively account for this, but I think it makes sense to unlock our test suite first so that getting better coverage via fixtures in https://github.com/pandas-dev/pandas/pull/55074 would help us more confidently refactor

Here's an MRE to reproduce this deadlock locally:

```python
import sqlalchemy as sa

from pandas.io.sql import pandasSQL_builder

engine = sa.create_engine("mysql+pymysql://root@localhost:3306/pandas")

pandasSQL = pandasSQL_builder(engine)
with pandasSQL.run_transaction() as trans:
    stmt = sa.text("CREATE TABLE test_trans (A INT, B TEXT)")
    trans.execute(stmt)

with pandasSQL.run_transaction() as trans:
    stmt = sa.text("INSERT INTO test_trans (A,B) VALUES (1, 'blah')")
    trans.execute(stmt)

result = pandasSQL.read_query("SELECT * FROM test_trans")

# Assume fixture tries to clean up all tables after function exit
with engine.begin() as conn:
    stmt = sa.text("DROP TABLE test_trans")
    conn.execute(stmt)
```

If you wrap the read_query call in a transaction there is no deadlock

(tip: if you ran the above locally you will probably want to kill the thread so it doesn't lock up your database. run `show engine innodb status;` inside the db, look for the hanging thread id and run `KILL <thread_id>`)


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
Thanks @WillAyd
BUG: add pyarrow autogenerated prefix
State: closed

- [x] closes #54459
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

~Any suggestions where to add a test for this bug (and fix)?~

edit: took a shot and put it in `tests/io/parser/test_header.py`

Labels:
name: IO CSV
description: read_csv, to_csv
name: Arrow
description: pyarrow functionality

Comments:
Please review and address the failed pre-commit failure
> Please review and address the failed pre-commit failure

wasn't seeing any issue in my local pre-commit for some reason
Thanks @hedeershowk
BUG: Index.difference not always returning a unique set of values
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.0.rst` file if fixing a bug or adding a new feature.

`Index.difference` has two short-circuit paths which currently differ in behavior from the main path in that they do not return a unique set of values when duplicates exist:

```
In [1]: import pandas as pd

In [2]: idx = pd.Index([1, 1])

In [3]: idx.difference([2])
Out[3]: Index([1], dtype='int64')      <- main path: unique

In [4]: idx.difference([])
Out[4]: Index([1, 1], dtype='int64')   <- short circuit path when right is empty: non-unique

In [5]: idx.difference([True])
Out[5]: Index([1, 1], dtype='int64')   <- short circuit path when right is considered "non-comparable": non-unique
```

Labels:
name: Bug
name: setops
description: union, intersection, difference, symmetric_difference

Comments:
Thanks @lukemanley
Assorted UBSAN cleanups
State: closed

Found the first few from running the io test suite. Will be stuck at https://github.com/pandas-dev/pandas/issues/55111 which requires closer attention than I wanted to tackle here

Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
tests?
UBSAN is a runtime check, so all of these were hit from values in the existing test suite. Nothing needs to be added
Thanks @WillAyd
Better support for UserDict on constructor
State: closed

I noticed that if I create my DataFrame passing a dictionary which is a custom one of a subclass of `collections.UserDict` then we have incompatible behaviour with the one if I pass a builtin `dict`.

The idea is that the `UserDict` should have the same type interface as the `dict` and used for custom behaviour, instead of subclassing direct from `dict`. 

This PR attempts to fix that incompatibility issue.

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors
name: Stale

Comments:
Well observed, @marcelotrevisani - Mapping is better here
@mroeschke is this good to go? :-)
@datapythonista 
Marc, if you don't mind, can you take a look please? :)
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
PERF: Index.difference
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.0.rst` file if fixing a bug or adding a new feature.


asv continuous -f 1.1 upstream/main perf-index-difference -b index_object.SetDisjoint

```
       before           after         ratio
-      7.97±0.4ms       3.16±0.4ms     0.40  index_object.SetDisjoint.time_datetime_difference_disjoint
```

asv continuous -f 1.1 upstream/main perf-index-difference -b multiindex_object.Difference

```
       before           after         ratio
-      3.55±0.7ms       1.29±0.3ms     0.36  multiindex_object.Difference.time_difference('int')
-      3.79±0.5ms       1.20±0.2ms     0.32  multiindex_object.Difference.time_difference('datetime')
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Index
description: Related to the Index class or subclasses
name: setops
description: union, intersection, difference, symmetric_difference

Comments:
Thanks @lukemanley 
This patch may have induced a performance regression. If it was a necessary behavior change, this may have been expected and everything is okay.

Please check the links below. If any ASVs are parameterized, the combinations of parameters that a regression has been detected for appear as subbullets.

 - [index_object.SetDisjoint.time_datetime_difference_disjoint](https://asv-runner.github.io/asv-collection/pandas/#index_object.SetDisjoint.time_datetime_difference_disjoint)
 - [multiindex_object.Difference.time_difference](https://asv-runner.github.io/asv-collection/pandas/#multiindex_object.Difference.time_difference)
   - [dtype='int'](https://asv-runner.github.io/asv-collection/pandas/#multiindex_object.Difference.time_difference?p-dtype=%27int%27)
   - [dtype='datetime'](https://asv-runner.github.io/asv-collection/pandas/#multiindex_object.Difference.time_difference?p-dtype=%27datetime%27)
   - [dtype='ea_int'](https://asv-runner.github.io/asv-collection/pandas/#multiindex_object.Difference.time_difference?p-dtype=%27ea_int%27)
   - [dtype='string'](https://asv-runner.github.io/asv-collection/pandas/#multiindex_object.Difference.time_difference?p-dtype=%27string%27)

Subsequent benchmarks may have skipped some commits. The link below lists the commits that are between the two benchmark runs where the regression was identified.

https://github.com/pandas-dev/pandas/compare/79067a76adc448d17210f2cf4a858b0eb853be4c...0bdbc44babac09225bdde02b642252ce054723e3
DOC: Update contributing.rst to give suggestions on non-code contributions
State: closed

Based on a conversation at the NumFOCUS project summit.

Labels:
name: Docs
name: Stale

Comments:
I am not an RST expert so apologies if there are any weird mistakes here
Also, this CI problems seem to be caused by your changes, if you can have a look: https://github.com/pandas-dev/pandas/actions/runs/6157790957/job/16709259450?pr=55107#step:6:65
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
DEPS: remove duplicated dependency in requirement-dev.txt
State: closed


Labels:
name: Dependencies
description: Required and optional dependencies

Comments:
I think this is already tested here essentially? https://github.com/pandas-dev/pandas/blob/705d4312cf1d94ef2497bcd8091e0eabd1085f4a/.github/workflows/code-checks.yml#L156
> I think this is already tested here essentially?
> 
> https://github.com/pandas-dev/pandas/blob/705d4312cf1d94ef2497bcd8091e0eabd1085f4a/.github/workflows/code-checks.yml#L156

Seems so. Then I wondered why it didn't catch the double requirements given error. I'll remove the CI check and make this PR a simple fix.
Thanks @fangchenli
wip
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for your pull request. For early development I suggest iterating locally before opening up a PR in this forum or, if absolutely needed, opening up a PR against your open pandas fork. Going to close for now.
Dev groupby
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for your pull request. For early development I suggest iterating locally before opening up a PR in this forum or, if absolutely needed, opening up a PR against your open pandas fork. Going to close for now.
TST: Make test_hash_equality_invariance xfail more generic
State: closed

The `raises` isn't too novel compared to raising a more specific behavior exception like a `ValueError`

Additionally, seems like hypothesis raises an ExceptionGroup here now which is PY311 specific (or using a backport package).

Labels:
name: Unreliable Test
description: Unit tests that occasionally fail

Comments:
Going to merge to avoid flaky failures
BUG: This fixes #55009 (`raw=True` caused `apply` method of `DataFrame` to ignore passed arguments)
State: closed

This should fix #55009. I documented the fix and also added some code to the tests that should pick it up next time.

Unfortunately, I was not able to run the tests manually yet with this error:

```
ImportError while loading conftest.
__init__.py:46: in <module>
    from pandas.core.api import (
core/api.py:1: in <module>
    from pandas._libs import (
_libs/__init__.py:16: in <module>
    import pandas._libs.pandas_parser  # noqa: E501 # isort: skip # type: ignore[reportUnusedImport]
E   ModuleNotFoundError: No module named 'pandas._libs.pandas_parser'
```

I probably do not have something properly installed. So let me know if I should run them in addition to what is done on the CI pipelines.

## Checklist

- [x] closes #55009
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests)
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file


Labels:
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
pre-commit.ci autofix
Thanks @martin-sicho 
@​mroeschke Hi, could you please consider making this fix available sooner, perhaps in the next patch release? I believe that using `.apply` with `raw=True` is a commonly used pattern. If this fix is to be included in version 2.2 which is scheduled for December, it could potentially affect the usability of pandas 2.1.x for some users.
BUG: concat(axis=1) ignoring sort parameter for DatetimeIndex
State: closed

- [x] closes #54769
- [x] closes #51210
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Thanks @lukemanley
Updated future warning msg in transform() for Series.groupby
State: closed

- [ ] closes #55066 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Simple change to make the warning message a lot more clear, thank you!
@rhshadrach can you please review this?
Thanks @rsm-23
ENH: make guess_datetime_format public
State: closed

- [X] closes #54727 (Replace xxxx with the GitHub issue number)
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [NA] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/v2.1.1.rst` file if fixing a bug or adding a new feature.


Labels:
name: Datetime
description: Datetime data dtype

Comments:
> Could you add this to the API reference documentation?

is the API reference documentation in` doc/source/reference/api`? If so, I'm seeing that path in the .gitignore file - is there a procedure to update the API reference under this context?

I think you can just add an entry to `doc/source/reference/general_functions.rst`
Thanks for the review and comments @mroeschke and @MarcoGorelli 
- added an entry to doc/source/reference/general_functions.rst ✅ 
- updated warning in docstring as sphinx directive ✅ 
- added examples to docstring ✅ 
Can you fix the failing ci checks please
👋  I'm getting a bit stuck with these failing checks - which seem to be mostly due to the following attribute error:
`AttributeError: module 'pandas' has no attribute 'guess_datetime_format'`

I updated the docstring in parsing.pyx with a sphinx `..warning::` directive and added two examples (learned that formatting was important which was a good lesson); however not sure how to address the failing jobs - any feedback or advice on how to troubleshoot?


This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
hi @nnlnr - is this still active? if you could fixup the conflicts I can finish it up
👋 @MarcoGorelli  - thanks! That would be great - let me fix up the conflicts and circle back here
website build looks good:

![image](https://github.com/pandas-dev/pandas/assets/33491632/ce87e0b4-f2ac-417f-a5ba-e67c020370a1)

merging then, thanks @nnlnr !
Could you also add a whatsnew note in 2.2 under the `Other Enhancements` section?
Thanks @nnlnr  and @MarcoGorelli 
Thanks @MarcoGorelli  @mroeschke!!  🎉 🙏