BrandonFors/pandas-github-issues huggingface dataset:
BUG: Index with duplicate labels raises ValueError in Dataframe.query
State: closed

- [x] closes #51815 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale

Comments:
Hey @mroeschke , just wanted to give you a quick update on my progress. I've integrated the latest version of my code into the 'query()' function, which should help streamline the overall process. Let me know if you have any questions or concerns.
Hello @mroeschke, I have made modifications to the query() function to check for duplicate index labels, but some tests have failed. As a new contributor to this open-source project, I would appreciate your expert opinion on whether I should make further changes to the code or focus on fixing the failing tests.
> As a new contributor to this open-source project

Honestly, I would suggest tackling an issue label `good first issue` first rather than this bug. A fix should ideally not fail existing tests and the solution should not involve falling back to a try/except 
Hello @mroeschke , I'm not sure if you saw, but I have made changes to the implementation. It no longer requires falling back to a try/except. Instead, I added an if statement to check for duplicate indexes.It covers all the constraints that you have informed me of, and it is currently working.

   
 if self.index.duplicated().any():
            engine='numexpr'
            # Create a copy of the dataframe with a unique index to avoid reindexing errors
            unique_index = RangeIndex(len(self.index))
            df_copy = self.copy()
            df_copy.index = unique_index

            # Filter the copied dataframe
            filtered_df = df_copy.query(expr, engine=engine)

            # Map the filtered index back to the original index labels
            index_mapping = dict(zip(unique_index, self.index))
            filtered_df.index = filtered_df.index.map(index_mapping)
            
            return filtered_df

 I apologize for the frequent comments, I don't mean to be a bother.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: __from_arrow__ doesn't accept pyarrow null arrays for numeric ma…
State: closed

…sked types

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Labels:
name: Bug
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays
name: Arrow
description: pyarrow functionality

Comments:
Looks like a merge conflict otherwise LGTM 
Thanks for the reviews all, going to merge this to unblock myself.
Ugh, I'm going to need to backport this as well to backport #52087.

Ok to do this, @phofl?
I'd rather not create chains that we have to backport. If fixes aren't totally trivial we should probably hold off?
Seems reasonable to me.
DOC: Clarifies the description of if_sheet_exists in pd.ExcelWriter (#52189)
State: closed

- [ ] closes #52189
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: IO Excel
description: read_excel, to_excel

Comments:
Thanks @abokey1
BUG: zero-pad shorter years in `Timestamp.isoformat`
State: closed

As discussed in #50867, this changes `Timestamp.isoformat` to zero-pad years to 4 digits.

After applying the change, I also had to change a existing test because of the now-changed standard format, but also remove a condition which assumed that `"001-01-01"` and `"0001-01-01"` would result in different objects (not sure, maybe there was a time where the resulting `Timestamp` object would not be equal?). However, on current `main`:
```python
a = pd.Timestamp("001-01-01")
b = pd.Timestamp("0001-01-01")
a == b and a.unit == b.unit
```

cc @spencerkclark

No `whatsnew` entry yet because I was not sure where to put it.

- [x] closes #50867
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] ~Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.~
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Timestamp
description: pd.Timestamp and associated methods

Comments:
I wonder if it would be possible to include this in `pandas` 2.0? The actual change is very small and it fixes a bug in one of the new features of 2.0, but of course we're pretty late in the release process already.
Thanks @keewis
ENH: make DataFrame.applymap uses the .map method of ExtensionArrays
State: closed

Currently `DataFrame.applymap` ignores the `.map`method of `ExtensionArrays`. This fixes that.

Example:
```python
>>> import pandas as pd
>>>
>>> arr = pd.array(["a", np.nan, "b"], dtype=object)
>>> sparse_arr = pd.array(arr.tolist(), dtype=pd.SparseDtype(object))
>>> df = pd.DataFrame(data={'a': arr, "b": sparse_arr})
>>> df.applymap(str.upper, na_action="ignore").dtypes  # main
a    object
b    object
dtype: object
>>> df.applymap(str.upper, na_action="ignore").dtypes  # this PR
a                 object
b    Sparse[object, nan]
dtype: object
```
 

Labels:
name: Apply
description: Apply, Aggregate, Transform, Map
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @topper-123
datetime64 series and dataframe test GH#28273
State: closed


- [ ] closes #28273 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

I used main codes to verify the GH28273. Unlike the original codes, the series and data frame types are same. So my codes just test its current behavior.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
I have tested the ujson.{decode,encode} with np.datetime64. I can't find a method to decode the result of encode to datetime64. So I compare the decode and encode result directly.
Actually, turns out this code doesn't exist in the current code base so it no longer needs testing. Closing but thanks for attempting to add a test for this
BUG: assert_frame_equal still checks category dtypes even when asked not to check index type
State: closed

- [x] closes #52126
- [ ] [Tests added and passed]
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Categorical
description: Categorical Data Type

Comments:
Looks pretty good. Could use a whatsnew note in `v2.0.1.rst`
@mroeschke done
Thanks @jayendra-patil33
ENH: Adding engine_kwargs to Excel engines for issue #40274
State: closed

- [X] closes #40274 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO Excel
description: read_excel, to_excel

Comments:
@rhshadrach Changes have been made! Please let me know if I have missed anything else!

Quick question. Once this PR is approved, could this also close out issue [43053](https://github.com/pandas-dev/pandas/issues/43053)
@rhshadrach Wondering if I should submit a GH issue to implement the same feature for the `to_excel` method?
> @rhshadrach Wondering if I should submit a GH issue to implement the same feature for the `to_excel` method?

Yea - I think that makes sense to do!
@rmhowe425 Please be aware that each time you merge main into this branch, it kicks off all the tests (30+hours of compute). This should be done only when needed - for example:
 - if there are conflicts
 - if you're aware of a PR that has been merged to main that touches on similar parts of the code
 - right before we merge this branch into main (to make sure all the tests still pass)

I don't think it needs to be done daily, or even once every couple of days.
@rhshadrach 
![image](https://user-images.githubusercontent.com/45905457/229359156-0d935b7c-5665-43d2-be15-740fd378a993.png)
For some reason it seems like the Code Checks test isn't liking my attempts to revert this change...looking at the failing test, it appears that the specific test that failed was: `sort whatsnew entries by issue number.`

The reason that I initially moved this line in the documentation was due to a merge conflict...I'm wondering if the line in question does in fact belong where I placed it?
Fixed merge conflict
@MarcoGorelli wow okay never mind. Not sure what I did wrong before....Thanks for pointing that mistake out to me!
looks right now, nice :) 
@rhshadrach Pinging on green!


@MarcoGorelli Looks like we're good to go! I think we just need your sign off?
Awesome! Thanks @rmhowe425 
Thanks for all the help guys! Really appreciate it!
hi team, pls what pandas version is this targeted for? I cant find it on pandas 2.0.3

@samukweku 2.1.0, which I believe will be released later today. If you check the "Files Changed", you'll see an entry in whatsnew/v2.1.0
API/BUG: infer_dtype_from_scalar with non-nano
State: closed

- [x] closes #51196 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Needs whatsnew and targeted tests, opening to see if there are opinions about doing this as a bugfix or waiting to do it as an API change in 3.0

Labels:
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@MarcoGorelli thoughts on if/how this is worth pursuing?
sorry, missed this. I'd have thought this was fine as a bug fix - I'm at jupytercon next week though so may be a bit slow to review, but will take a look when I get a chance
🤔 not sure if this might be related
```
FAILED pandas/tests/frame/methods/test_reindex.py::TestDataFrameSelectReindex::test_reindex_tzaware_fill_value - ValueError: Unexpected value for 'dtype': 'int32'. Must be 'datetime64[s]', 'datetime64[ms]', 'datetime64[us]', 'datetime64[ns]' or DatetimeTZDtype'.
```
Thanks @jbrockmendel
CI: Use auto pytest workers for Windows
State: closed

- [ ] closes #47443 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration
name: Windows
description: Windows OS

Comments:
Looks like it hasn't been fixed in the interim.
POC: rollback_array
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

xref #7449, #52105

This implements rollback_array for Month, Quarter, and Year offsets.  I've only checked the FooBegin cases and only in non-tzaware cases:

```
dti = pd.date_range("2016-01-01", periods=60, freq="15D")

mth = pd.offsets.MonthBegin(1)
qtr = pd.offsets.QuarterBegin(1)
yr = pd.offsets.YearBegin(1)


res_mth = pd.DatetimeIndex(mth.rollback_array(dti))
res_qtr = pd.DatetimeIndex(qtr.rollback_array(dti))
res_yr = pd.DatetimeIndex(yr.rollback_array(dti))

exp_mth = pd.DatetimeIndex([mth.rollback(x) for x in dti])
exp_qtr = pd.DatetimeIndex([qtr.rollback(x) for x in dti])
exp_yr = pd.DatetimeIndex([yr.rollback(x) for x in dti])

assert res_mth.equals(exp_mth)
assert res_qtr.equals(exp_qtr)
assert res_yr.equals(exp_yr)
```

Posting this as a POC so someone else can take it the rest of the way.  If no one does I'll circle back to it after focusing on my "do less work on the weekends" goal.

Labels:
name: Frequency
description: DateOffsets
name: Mothballed
description: Temporarily-closed PR the author plans to return to
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Mothballing.
ASV: Add benchmark when comparing datetimes with different reso
State: closed

- [ ] xref #52080 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
Thanks @phofl
ENH: Add dtype of categories to repr of CategoricalDtype
State: closed

- [x] closes #52179 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Categorical
description: Categorical Data Type

Comments:
```
=================================== FAILURES ===================================
__ [doctest] pandas.core.dtypes.dtypes.CategoricalDtype._from_values_or_dtype __
246             If ``CategoricalDtype``, cannot be used together with
247             `categories` or `ordered`.
248 
249         Returns
250         -------
251         CategoricalDtype
252 
253         Examples
254         --------
255         >>> pd.CategoricalDtype._from_values_or_dtype()
Expected:
    CategoricalDtype(categories=None, ordered=None)
Got:
    CategoricalDtype(categories=None, ordered=None, categories_dtype=None)

/home/runner/work/pandas/pandas/pandas/core/dtypes/dtypes.py:255: DocTestFailure
___ [doctest] pandas.io.json._table_schema.convert_json_field_to_pandas_type ___
167     ------
168     ValueError
169         If the type of the provided field is unknown or currently unsupported
170 
171     Examples
172     --------
173     >>> convert_json_field_to_pandas_type({"name": "an_int", "type": "integer"})
174     'int64'
175 
176     >>> convert_json_field_to_pandas_type(
Expected:
    CategoricalDtype(categories=['a', 'b', 'c'], ordered=True)
Got:
    CategoricalDtype(categories=['a', 'b', 'c'], ordered=True, categories_dtype=object)
```
Thanks @phofl
ENH: add `__from_pyarrow__` support to `DatetimeTZDtype`
State: closed

- [x] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Closes #52200

Labels:
name: Datetime
description: Datetime data dtype
name: Arrow
description: pyarrow functionality

Comments:
Any additional feedback on this? All checks have been green, and it's aligned with what we do for the other Extension Dtypes.
I'd like to see more buy-in on ignoring iNaT
I would be okay with ignoring iNaT for now. Would be good to write a comment in `__from_arrow__`
Thanks @mroeschke @jbrockmendel I've added a docstring to `__from_arrow__` clarifying the `iNaT` behavior.
Looks like a merge conflict otherwise LGTM
Looks like `test_astype_from_non_pyarrow` is failing the pyarrow nightly build. Possibly this changes something with the upcoming pyarrow release?
> Looks like test_astype_from_non_pyarrow is failing the pyarrow nightly build. Possibly this changes something with the upcoming pyarrow release?

Thanks for the FYI. Looks like maybe some different assumptions about timezones in latest pyarrow? I'll investigate.
Possible bug with: https://github.com/apache/arrow/pull/34270 ? If the timezones aren't the same (or UTC and naive) then it shouldn't be no-copy, right?
@mroeschke I think I found the issue. I don't think I should be calling `tz_localize`, but there was a bug where the units weren't being copied over from the DatetimeArray constructor that I vaguely recall trying to work around. Should be fixed now.
Thanks for sticking with this @tswast
add extra test case in the test_constructor_str_infer_reso
State: closed

- [x] closes #51025
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

An extra test case is added to `test_constructor_str_infer_reso`  to check `Timestamp('300-01-01')`.

I suggest doing a parametrization for this test.

Labels:
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution
name: Timestamp
description: pd.Timestamp and associated methods

Comments:
Looks good thanks, if you mark as ready for review we can get it in