BrandonFors/pandas-github-issues huggingface dataset:
Revert "BUG: Timestamp origin takes no effect in resample for 'MS' frequency (#53938)"
State: closed

closes #55064

This reverts commit dfc4c3900a181e8a9dff724217d5a14e59ca2e4f.

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

---

I'm just:
- reverting the commit
- fixing some merge conflicts due to:
  - random number generator usage
  - 'T' -> 'min' deprecation
- adding a docs note noting that `origin` only takes effect for tick-frequencies

and adding a 2.1.1 whatsnew note

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Frequency
description: DateOffsets

Comments:
Thanks @MarcoGorelli
Replace classes with pytest in test_sql
State: closed

A lot of this is very intertwined so hard to break up into small diffs.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
The diff here is unfortunate, but I am not sure there is a lot that can be broken off piece-wise. 

With the new design, we have a lot more coverage of the different connection types, particularly when using engines directly (and have also uncovered some broken behavior). Total number of tests increase from 824 to 1449

There is still more that can be done to clean up the fixtures (in particular clean up standard vs iris vs type table requiring fixtures) but as is this should help us much more effectively onboard the ADBC drivers
Might it be more digestible if one test class was broken down at a time? Even if that might mean there is duplication in code that setups & tearsdown db connections & data loading
Hmm I'm not sure. I'll have to do another pass, but the hard part is that you end up with a lot of duplication in the intermediate state. Taking the first method of `TestSQLiteFallback` as an example:

```python
class TestSQLiteFallback(SQLiteMixIn, PandasSQLTest):
    """
    Test the fallback mode against an in-memory sqlite database.

    """

    flavor = "sqlite"

    @pytest.fixture(autouse=True)
    def setup_method(self, iris_path, types_data):
        self.conn = self.connect()
        self.load_iris_data(iris_path)
        self.load_types_data(types_data)
        self.pandasSQL = sql.SQLiteDatabase(self.conn)

    def test_read_sql_parameter(self, sql_strings):
        self._read_sql_iris_parameter(sql_strings)
```

We would have to duplicate load_iris_data, load_types_data and read_sql_iris_parameter just to get that one test to run. We would also have to reimplemnt connect, which in this case fortunately isn't that different from the `sqlite_buildin` fixture, but then puts us in a state where we have a fixture that does part of the setup/teardown and custom functions to do the rest.

In the current PR there is no intermediate state that mixes fixtures, functions and class-based setup - everything is managed via the fixture. Its a big diff but a cleaner end state

The issue with hanging now is from updating the fixture to test engines directly, not just connections. I did this in hopes of increasing test coverage but may just need to scale it back some more

OK finally got rid of all the hanging behavior when using engine arguments directly. This should be good to go
Very cool! It's a been a clean up that's long been needed
BUG: dt.tz with ArrowDtype returned string
State: closed

- [x] closes #55003 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Timezones
description: Timezone data dtype
name: Arrow
description: pyarrow functionality

Comments:
Looks like tests are passing here so merging. Can follow up if needed
BUG: read_csv(on_bad_lines='warn') did not raise a Python warning
State: closed

- [x] closes #54296 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO CSV
description: read_csv, to_csv
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
Yeah maybe it was added to the python engine for compat. Not sure why the c engine wrote to stderr
Let isocalendar() return int32 (no NaT) and float64 (in presence of NaT)
State: closed

- [x] closes #54657
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale

Comments:
This should align with the other dt accessors, like dt.year, dt.month etc. which return "int32" without NaN and "float64" with NaN.

If this PR is considered valid, I'll fix the failing doctests. For the typing errors I would need a little help.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
I'm still waiting for a comment from @pandas-dev/pandas. 
Thanks for the PR here so far, but let's reopen this PR once another core developer has chimed in on the original issue.
Convert to compatible NumPy dtype for MaskedArray to_numpy
State: closed

- [ ] closes #48891 (Replace xxxx with the GitHub issue number)
- [ ] closes #40630
- [ ] closes #37460
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This still needs a whatsnew

Labels:
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
I think this is pretty reasonable. Is this part of a larger set of changes or is this self-contained?

Probably worth a whatsnew
This is something we wanted to clean up for a while now, but yes I have to add a whatsnew (didn't get to it yet because this needs more than a single line)
Added the whatsnew
Does this close #48891?
yep
cc @mroeschke 

I think this is ready for a review
Can we get this in if ci is green now? Just merged main to resolve whatsnew conflicts
Thanks @phofl
DOC: fix an example which raises an Error in whatsnew/v0.10.0.rst
State: closed

Fix an example in doc/source/whatsnew/v0.10.0.rst, which raises an IndentationError (see 
https://pandas.pydata.org/docs/dev/whatsnew/v0.10.0.html)

Labels:
name: Docs

Comments:
@mroeschke, could you please take a look at this PR?
Thanks @natmokval
Improve error message for StringDtype with invalid storage
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Strings
description: String extension data type and string data
name: Arrow
description: pyarrow functionality

Comments:
Thanks @phofl
Fix pickle roundtrip for new arrow string dtype
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Strings
description: String extension data type and string data
name: Arrow
description: pyarrow functionality

Comments:
Thanks @phofl
Fix docstring of Index.join in base.py
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
That's a very strange thing and it's better if pandas devs can have a closer look here.

I can see the `Examples` have been added with https://github.com/pandas-dev/pandas/pull/53403 but there is no sign of any failure. Everything looks good over there (cc @DeaMariaLeon @MarcoGorelli).


> Everything looks good over there (

I did not look far down enough. The "Examples" heading is indeed one space to far to the right. So this PR should fix it.
Nice thanks @kmuehlbauer 
@mroeschke @lithomas1 Could this be included into 2.1.1? This breaks doc-building downstream.
@meeseeksdev backport 2.1.x
COMPAT: bump pyarrow min version for div on duration
State: closed

- [x] closes #55020
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Arrow
description: pyarrow functionality

Comments:
Looks like there are some code check and precommit failures otherwise looks good
I fixed the code check.

Note that this does not address the fact that there is an extensive branch structure depending on the pyarrow version but there is no test matrix covering it. Hence the formerly unnoticed incompatibility with pyarrow 13.0.0
Thanks @bnavigator
DEPR: ArrayManager
State: closed

- [x] closes #55043 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
I'm seeing warnings related to the dateutil package upon import of pandas; is this expected?

https://github.com/pandas-dev/pandas/actions/runs/6177889227/job/16769987130?pr=55044#step:5:92

> /opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/dateutil/tz/tz.py:37: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC).
>  EPOCH = datetime.datetime.utcfromtimestamp(0)

cc @jbrockmendel @mroeschke @MarcoGorelli 
> I'm seeing warnings related to the dateutil package upon import of pandas; is this expected?

I'm not surprised by the warning itself, but I'd expect it to be showing up on all builds, not specific to this PR
> I'd expect it to be showing up on all builds, not specific to this PR

Thanks - maybe it's only incurred by using subprocess like the test added here. And even then, it only seems to happen in certain jobs.
handful of comments, generally looks good.  the complexity of this deprecation (as opposed to just warning inside `ArrayManager.__init__`) makes me want to avoid the environment variable pattern in the future
Thanks @rhshadrach
REGR: DataFrameGroupBy.agg with duplicate column names and a dict
State: closed

- [x] closes #55006 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Unlike what I reported in #55006, the core issue is not how we treat kwargs. Rather, I believe the issue occurred in #52115 where logic specific for duplicate columns was added for Series/DataFrame but also impacted groupby.

This PR fixes the regression, but groupby still does not work with duplicate columns. I've confirmed groupby.agg does not work with duplicate columns in this case on 2.0.x, and believe it hasn't for a long time (possibly forever). For this, I've opened #55041.

Labels:
name: Bug
name: Groupby
name: Regression
description: Functionality that used to work in a prior pandas version
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Thanks @rhshadrach
Cythonize is_null_slice
State: closed

This makes the diff in #54506 smaller

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Is there a reason you want to Cythonize this? (impact on benchmarks?)

This seems to be used only 6 times.
https://github.com/search?q=repo%3Apandas-dev%2Fpandas%20is_null_slice&type=code
Yea it also would get used in https://github.com/pandas-dev/pandas/pull/54506 in the getitem / setitem methods of the BitmaskArray
I won't claim that this has a significant impact on benchmarks. The main thing is that it shows less Python interaction in Cython annotate, so it causes less visual noise when searching through the annotate to find bottlenecks. If there's unease about this then no problem closing
BUG: Interval with Timestamp with tz shows tz
State: closed

- [x] closes #55015 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Timezones
description: Timezone data dtype
name: Interval
description: Interval data type

Comments:
Going to merge this. Happy to follow up
Doc: lines added
State: closed


DOC: 
- [ ] [Tests added and passed](doc/source/getting_started/comparison/comparison_with_r.rst) 
- This addition helps readers understand the purpose of the page and the context of the comparison between pandas and the R language.
- Provide additional clarity to the sentence, making it explicit that the comparison is related to these specific aspects.


Labels:

Comments:
Please as noted before https://github.com/pandas-dev/pandas/pull/54840#issuecomment-1697128112, pull requests should reference existing issues
BUG: merge with left and/or right empty returning mis-ordered columns
State: closed

- [x] closes #51929
- [x] closes #43087
- [x] closes #9937
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Thanks again @lukemanley
BLD: Build wheels for Python 3.12
State: closed

- [ ] closes #54447 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Build
description: Library building on various platforms
name: Python 3.12

Comments:
Let's merge and backport this for 2.1.1.

This means we'll have nightlies up for Python 3.12 and also the option to upload Python 3.12 wheels for 2.1.1 if we want.

(I am still planning on waiting for numpy to release 1.26 before wheels are uploaded to PyPI. This is more so downstream can test with pandas).
Thanks @lithomas1
TYP: Add typing.overload signatures to DataFrame/Series.clip
State: closed

This adds overloads so that a type checker can determine whether clip returns a  Series/DataFrame or None based on the value of the inplace argument.


Labels:

Comments:
pre-commit.ci autofix
Thank you @caneff !
Thanks for the quick review and merge!

On Mon, Sep 4, 2023, 1:23 PM Torsten Wörtwein ***@***.***>
wrote:

> Thank you @caneff <https://github.com/caneff> !
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/55002#issuecomment-1705560449>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABG4SH2C4P2WMN76HMQQ4BTXYYFCZANCNFSM6AAAAAA4KRCADU>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
BUG: ArrowDtype raising for fixed size list
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Arrow
description: pyarrow functionality

Comments:
Thanks @phofl
TYP: Add typing.overload signatures to DataFrame/Series.interpolate
State: closed

This adds overloads so that a type checker can determine whether interpolate returns DataFrame/Series or None based on the value of the inplace argument.

This is more of the same thing that was done in [this pull](https://github.com/pandas-dev/pandas/pull/54281).

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
pre-commit.ci autofix
Thank you @caneff! 

Pyright/mypy pass, CI failures seem unrelated.