BrandonFors/pandas-github-issues huggingface dataset:
DEPS: Make pytz an optional dependency
State: closed

- [ ] closes #34916 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


This makes pandas return `zoneinfo.ZoneInfo` objects by default from string inputs

Labels:
name: Timezones
description: Timezone data dtype
name: Dependencies
description: Required and optional dependencies

Comments:
might be easier to break into steps e.g. one PR just changing which UTC, one just changing FixedOffset, then the hard one for the general case.
> one PR just changing which UTC, one just changing FixedOffset

These are already changed (I think you changed it for 2.0), so now this is tackling the general case.
Closing in favor of https://github.com/pandas-dev/pandas/pull/59089
fix: Expose `to_pandas_kwargs` in `read_parquet` for pyarrow engine
State: closed

- [x] closes #49236
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO Parquet
description: parquet, feather
name: Stale

Comments:
Hmm this design feels weird to me. What arguments would realistically be passed through this? 
In the underlying issue, the OP needs to send `timestamp_as_object=True` to `.to_pandas`, to control how the data is formatted as a dataframe.
If it is not a real concern, I can close this PR.
I am +1 on this, the most important is probably types_mapper, which is a real PITA that this is not possible at the moment
Something like `read_pandas(to_pandas_kwargs=...)`  feels strange to me from an API perspective, and has the downside of really intertwining the APIs of the libraries. If we later decide to change the allowable `to_pandas_kwargs` or pyarrow decides to change their API it would be quite messy

Why not just encourage users to call `pa.parquet.read_table` in such a case directly?
Would updating the documentation be sufficient in your opinion?
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@WillAyd ?
Sure I think documenting this is a reasonable approach
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
API: MultiIndex.names|codes|levels returns tuples
State: closed

- [ ] closes #53531 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: MultiIndex

Comments:
Just playing devil's advocate but why would we prefer tuple here over `Index`? The latter is what we return from DataFrame.columns which maybe should harmonize with this
> Just playing devil's advocate but why would we prefer tuple here over Index?

I picked tuple to align with the prior `FrozenList` which seemed to act like an immutable `list`. If we were to go with `Index` I think each would have to be `Index[object]

* `MultiIndex.codes` would be an Index of ndarrays
* `MultiIndex.levels` would be an Index of Indexes
* `MultiIndex.names` would be an Index of object/inferred dtype which would be reasonable.

But yeah the biggest thing is that the returned object should not be able to be resized.
Will merge this week unless there's any objections
@mroeschke - just the request to fix the docs: https://github.com/pandas-dev/pandas/pull/57042#discussion_r1472031470
> @mroeschke - just the request to fix the docs: [#57042 (comment)](https://github.com/pandas-dev/pandas/pull/57042#discussion_r1472031470)

Ah thanks for the reminder. Removed
This is quite a breaking change .. I know you knew this and therefore kept it for 3.0, but I still do wonder if this is worth the breakage, and if we really want to change it, whether we couldn't first deprecate some aspects of it.

The breaking change I ran into with geopandas, is the fact that a tuple cannot be concatenated like a list. We actually used the same pattern in pandas itself, as the diff in this PR has this change in the groupby code:

```diff
-        mi = MultiIndex(levels=levels, codes=codes, names=idx.names + [None])
+        mi = MultiIndex(levels=levels, codes=codes, names=list(idx.names) + [None])
```

I can imagine quite some external libraries use this when manipulating index / multi-index objects. 

We could first deprecate those methods on FrozenList that won't work anymore for tuple (I see @rhshadrach mentioned the same on the issue for the `difference`/`union` methods)
DOC: Docstrings for fixtures in pandas/indexes/numeric/test_numeric.py
State: closed

Partially addresses #19159

Added docstring descriptions for the `pandas/indexes/numeric/test_numeric.py` file

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
REGR: perf regression in Series.combine_first
State: closed

- [x] closes #55845
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.1.rst` file if fixing a bug or adding a new feature.

```
import pandas as pd
import numpy as np

N = 1_000_000


# different indexes
s1 = pd.Series(np.random.randn(N))
s1 = s1[::2].sample(frac=1.0)
s2 = pd.Series(np.random.randn(N))
%timeit s1.combine_first(s2)

# 523 ms ± 7.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)   <- main
# 227 ms ± 4.33 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  <- PR


# same index
s1 = pd.Series(np.random.randn(N))
s1[::2] = np.nan
s2 = pd.Series(np.random.randn(N))
%timeit s1.combine_first(s2)

# 282 ms ± 5.64 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)   <- main
# 7 ms ± 64.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  <- PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Thanks @lukemanley
BUG: Inconsistent behavior while constructing a Series with large integers in a int64 masked array
State: closed

- [x] closes #56566 
- [x] [Tests added and passed]
- [x] All [code checks passed]
- [ ] Added [type annotations]
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
DOC: Add `DataFrame.to_numpy` method
State: closed

- [x] closes #57016  (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @luke396 
This method seems to be not the only one missing from the current (and dev) docs, e.g. https://pandas.pydata.org/pandas-docs/version/2.1/reference/api/pandas.Index.array.html. Not sure how to check for all occurrences, because this is likely not the only additional method not working...
@cbrnr Thank you for the reminder. I will make an effort to locate and include the missing documents, as you mentioned earlier
@meeseeksdev backport 2.2.x
TST: Interchange implementation for timestamp[ns][pyarrow]
State: closed

- [x] closes #56712
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Labels:
name: Interchange
description: Dataframe Interchange Protocol

Comments:
Hi, thanks for pointing out, I added the necessary `pytest.importorskip` to prevent the test from failing.
DOC: fix EX03 errors in docstrings - pandas.Series.plot.line, pandas.Series.to_sql, pandas.read_json, pandas.DataFrame.to_sql
State: closed

All EX03 Errors resolved in the following cases:

1. scripts/validate_docstrings.py --format=actions --errors=EX03 pandas.Series.plot.line
2. scripts/validate_docstrings.py --format=actions --errors=EX03 pandas.Series.to_sql
3. scripts/validate_docstrings.py --format=actions --errors=EX03 pandas.read_json
4. scripts/validate_docstrings.py --format=actions --errors=EX03 pandas.DataFrame.to_sql

OUTPUT:

1. scripts/validate_docstrings.py --format=actions --errors=EX03 pandas.Series.plot.line
```
################################################################################
################################## Validation ##################################
################################################################################

1 Errors found for `pandas.Series.plot.line`:
	PR02	Unknown parameters {'color'}
```

2. scripts/validate_docstrings.py --format=actions --errors=EX03 pandas.Series.to_sql
```
################################################################################
################################## Validation ##################################
################################################################################

Docstring for "pandas.Series.to_sql" correct. :)
```

3. scripts/validate_docstrings.py --format=actions --errors=EX03 pandas.read_json
```
################################################################################
################################## Validation ##################################
################################################################################

1 Errors found for `pandas.read_json`:
	ES01	No extended summary found
```

4. scripts/validate_docstrings.py --format=actions --errors=EX03 pandas.DataFrame.to_sql
```
################################################################################
################################## Validation ##################################
################################################################################

Docstring for "pandas.DataFrame.to_sql" correct. :)
```


- [x] closes https://github.com/pandas-dev/pandas/issues/56804
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
@datapythonista you are correct, this will close stale PRs https://github.com/pandas-dev/pandas/pull/56866 and https://github.com/pandas-dev/pandas/pull/56865 

I'll ping you when this PR is merged into main
Thanks for all your effort here @jordan-d-murphy
PERF: Index.join to maintain cached attributes in more cases
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.


```
import pandas as pd

data = [f"i-{i:05}" for i in range(100_000)]
dtype = "string[pyarrow_numpy]"

idx1 = pd.Index(data, dtype=dtype)
idx2 = pd.Index(data[1:], dtype=dtype)

# the is_unique call at the end is cached in this PR
%timeit idx1.join(idx2, how="outer").is_unique

# 59.1 ms ± 1.29 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  -> main
# 41.9 ms ± 894 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)   -> PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Index
description: Related to the Index class or subclasses

Comments:
Thanks @lukemanley
REGR: merge_ordered with fill_method="ffill" and how="left"
State: closed

- [x] closes #57010
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.1.rst` file if fixing a bug or adding a new feature.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Thanks @lukemanley
DOC : ambiguous takes bool as an argument, not bool-ndarray
State: closed

`ambiguous` takes bool as an argument, not bool-ndarray. `[True, False]` wouldn't make sense in this context. 

Docstring update so checklist N/A. 



Labels:
name: Docs
name: Timezones
description: Timezone data dtype

Comments:
Thanks @Will-So @datapythonista
Fix docstring
State: closed

- [x] xref #27976 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @thiphan94
Remove SKIP summary from CI logs
State: closed

I think this was @rhshadrach idea originally. This currently generates about 3k lines of SKIPs in the CI output, which can be mildly annoying to have to scroll through every log

Labels:
name: CI
description: Continuous Integration

Comments:
+1 on this - sometimes the large amount of output does make my logs lag.
+1 as well
Thanks @WillAyd
DOC: mention both mirror methods
State: closed

closes #21825  (Replace xxxx with the GitHub issue number)
 [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
 All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
 Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
 Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @ronarbo
Use Python Timedelta C-API where applicable
State: closed

- [ ] closes #54682

Kind of a strange intermediate point but maybe no worse than where we are today

Labels:
name: Timedelta
description: Timedelta data type

Comments:
@jorisvandenbossche @jbrockmendel for thoughts
Ah OK I see the problem now - I thought the internal timedelta encoded days / months separately but things are not that simple
> I thought the internal timedelta encoded days / months separately but things are not that simple

I don't really understand how that is related to this being feasible or not?
> I don't really understand how that is related to this being feasible or not?

I think a significant effort would have to go into overhauling this to make it so 64 bit ns resolution integers can be subdivided into the appropriates components. Originally I thought simple math would suffice but you have to convert the integer into a time struct to make this work
Doesn't do cpython that for us? 
At least when using it from python, I can do `datetime.timedelta(milliseconds=value)` with a large value, and then under the hood python will convert that in a number for days and a number for seconds. 

Or `timedelta.__new__` is more restricted?

Looking at the cpython source (https://github.com/python/cpython/blob/c63c6142f9146e1e977f4c824c56e8979e6aca87/Modules/_datetimemodule.c#L2503), it seems that the timedelta new method converts the input first to a single number of microseconds, and then converts that number of microseconds into the three numbers of days/seconds/microseconds. 

In any case, it seems that passing whathever value we have should just work (unless there is an overflow, but that we have to either catch or first check for)
BUG: df.iloc not working with NamedTuple
State: closed

- [x] closes #48188(Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

@rhshadrach I kept the `type(key) is tuple` since index can contain namedtuple, used as key in `df.loc[key]`, as shown in the test case below:
https://github.com/pandas-dev/pandas/blob/488f6bd5c08111603549d42effc6dbb7fec935f0/pandas/tests/indexing/test_loc.py#L1615-L1624

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Stale

Comments:
> I kept the type(key) is tuple since index can contain namedtuple, used as key in df.loc[key], as shown in the test case below:

Sorry, I don't understand. In particular, the test you highlighted passes with this patch applied and from your comment above I expected it not to.
I mean if we only change `type(key) is tuple` to `isinstance(key, tuple)`, this particular test will fail.

Because in this test, namedtuples are elements of the index, and are used as labels in `df.loc`. It was handled by the else branch in the original code below.
https://github.com/pandas-dev/pandas/blob/72814750548b5ead5b08cd9b90d56a23c9a520ea/pandas/core/indexing.py#L1180-L1192
Thanks - makes sense. I think this might need further discussion in that case. I do not see a way to support tuple (and subclasses of tuple) in a flat Index (i.e. not a MultiIndex) and using tuples for accessing (row, column) pairs. Indeed, the following variant of that test using tuples fails on main:

```
idx1 = ("foo", "bar")
idx2 = ("baz", "bof")
index = Index([idx1, idx2], name="composite_index", tupleize_cols=False)
df = DataFrame([(1, 2), (3, 4)], index=index, columns=["A", "B"])
result = df.loc[("foo", "bar")]["A"]
assert result == 1
```

The only way I can see making it all work is to try one and if it fails try the other - which I think we want to avoid.

While we don't run into this problem with iloc, I wonder if we want to avoid supporting subclasses with iloc but not loc.

cc @jbrockmendel @phofl 
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.