BrandonFors/pandas-github-issues huggingface dataset:
DEPR: maybe_infer_ndim
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel
BUG: fix to_json() with JSON Table Schema work correctly with string dtype
State: closed

Fixes:  #61889 

To ensure consistent behavior for to_json(), when dtype="str" is used, it will now output _"type": "string"_ instead of _"type": "any"_.

Before Fix:
```
>>> pd.Series(["a", "b", None], dtype="str").to_json(orient="table", index=False)
'{"schema":{"fields":[{"name":"values","type":"any","extDtype":"str"}],"pandas_version":"1.4.0"},"data":[{"values":"a"},{"values":"b"},{"values":null}]}'
```

After Fix:
```
>>> pd.Series(["a", "b", None], dtype="str").to_json(orient="table", index=False)
'{"schema":{"fields":[{"name":"values","type":"string","extDtype":"str"}],"pandas_version":"1.4.0"},"data":[{"values":"a"},{"values":"b"},{"values":null}]}'

```

- [x] closes #61889
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: IO JSON
description: read_json, to_json, json_normalize
name: Strings
description: String extension data type and string data

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 a067fff92540118cd784349b6e79021087347768
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #61900: BUG: fix to_json() with JSON Table Schema work correctly with string dtype'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-61900-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #61900 on branch 2.3.x (BUG: fix to_json() with JSON Table Schema work correctly with string dtype)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Thanks @khemkaran10!
Manual backport -> https://github.com/pandas-dev/pandas/pull/61965
BUG: Fix concat dtype preservation through concat
State: closed

- [x] closes #51362
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


---
Expected behavior:

- An empty categorical, and a categorical should concat as a categorical.


---
Linked issues:
#13524 see https://github.com/pandas-dev/pandas/issues/13524#issuecomment-233792648
https://github.com/pandas-dev/pandas/issues/39443

#14177 (seems a dev call agreed on putting this as an option in `concat`)

https://github.com/pandas-dev/pandas/pull/13767



Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Categorical
description: Categorical Data Type

Comments:
As mentioned in #51362, this is not a bug, but we are willing to support this via a keyword in pd.concat.

I think there are also some older related issues this would close.  Can you make sure those are listed in the OP.
Closing this for now — I ran into more complexity than expected. Might revisit later with a cleaner approach.
WEB: Remove Roadmap points pending a PDEP section from Roadmap
State: closed

closes https://github.com/pandas-dev/pandas/issues/60913

In https://github.com/pandas-dev/pandas/issues/51471#issuecomment-3075831365 @jbrockmendel wrote

> No evidence the ecosystem wants this, closing.

in https://pandas.pydata.org/about/roadmap.html#roadmap-points-pending-a-pdep

> pandas is in the process of moving roadmap points to PDEPs (implemented in August 2022). During the transition, some roadmap points will exist as PDEPs, while others will exist as sections below.

If we don't have actionable plans than can be written up as a PDEP, let's remove this from the roadmap.

---

In https://github.com/pandas-dev/pandas/pull/27478 @TomAugspurger wrote

> Do we want this? Roadmaps tend to go stale. How can we keep this up to date?

So maybe there are other sections that we also want to remove now?

@pandas-dev/pandas-core 

Labels:
name: Web
description: pandas website
name: Roadmap
description: A proposal for the roadmap.

Comments:
following suggestion by @mroeschke and thumbs up from @jbrockmendel i've removed all Roadmap points pending a PDEP.

Thanks @simonjayhawkins
Output formatting: preserve quoting for string categories
State: closed

POC for https://github.com/pandas-dev/pandas/issues/61890 if we want to keep the quoting behaviour

Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Strings
description: String extension data type and string data
name: Categorical
description: Categorical Data Type

Comments:
It's an easy change, so let's add to 2.3.x (I might just not backport all test fixes)
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 c849d39c4c956495c0c86b85ab561bf74bf5df8d
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #61891: Output formatting: preserve quoting for string categories'
```

4. Push to a named branch:

```
git push YOURFORK 2.3.x:auto-backport-of-pr-61891-on-2.3.x
```

5. Create a PR against branch 2.3.x, I would have named this PR:

> "Backport PR #61891 on branch 2.3.x (Output formatting: preserve quoting for string categories)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport -> https://github.com/pandas-dev/pandas/pull/61966
DOC: fix doctests for string dtype changes (top-level)
State: closed

Part of https://github.com/pandas-dev/pandas/issues/61886). Some first doctest fixes for the new string dtype, for things that pytest sees as part of `pandas/__init__.py` and some files that only had one failure.

Labels:
name: Docs
name: Strings
description: String extension data type and string data

Comments:
Thanks @jorisvandenbossche
Fix warning for extra fields in read_csv with on_bad_lines callable
State: closed

- [ ] closes #61837 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
trailing white space is annoying
pre-commit.ci autofix
API: Index.__cmp__(Series) return NotImplemented
State: closed

- [x] closes #36759 (Replace xxxx with the GitHub issue number)
- [x] closes #54475
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
Thanks @jbrockmendel
BUG: Timedelta with invalid keyword
State: closed

- [x] closes #53801 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Timedelta
description: Timedelta data type

Comments:
Thanks @jbrockmendel
BUG: disallow exotic np.datetime64 unit
State: closed

- [x] closes #25611 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This doesn't work locally for reasons described [here](https://github.com/pandas-dev/pandas/issues/25611#issuecomment-3079363946). @seberg says it should work, so let's see if the CI can prove him right.

Labels:
name: Datetime
description: Datetime data dtype

Comments:
Thanks @jbrockmendel
Set up ty in CI
State: closed

Hello,

This PR sets up [`ty`](https://github.com/astral-sh/ty), a type checker developed by the creator of `ruff`, in CI.

I'm aware that `ty` is still in preview but there can be a couple of benefits in adding it now. 
- It's already very fast and can help us debug faster. `mypy` is quite slow and disrupts the coding flow.
- The cost of setup is pretty cheap, as it's mostly similar to `mypy` and `pyright`. Currently I ignore all errors so that we can fix and enable them gradually, like the other linting and typing errors.
- This also allows us to evaluate if/when `ty` can be a replacement of `mypy` and `pyright`.

Let me know what you think.

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks for the PR but closing until ty is more matrue
DOC: show Parquet examples with default engine (without explicit pyarrow/fastparquet engine keyword)
State: closed

Encountered this in https://github.com/pandas-dev/pandas/pull/61864, but in general for the readability of our doc page, I feel that it is not needed to show every single code example in this section with both pyarrow and fastparquet (certainly because in practice the fastparquet result is then ignored, and we only show the resulting dtypes for the pyarrow one). 
We already mention in the text itself the engine keyword and the different options.



Labels:
name: Docs
name: IO Parquet
description: parquet, feather

Comments:
Thanks @jorisvandenbossche
ERR: improve exception message from timedelta64-datetime64
State: closed

- [x] closes #59571 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Thanks @jbrockmendel
API: IncompatibleFrequency subclass TypeError
State: closed

- [x] closes #55782 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Thanks @jbrockmendel
API: np.isinf on Index return Index[bool]
State: closed

- [x] closes #52676 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Index
description: Related to the Index class or subclasses

Comments:
Thanks @jbrockmendel
TST: add test for `dtype` argument in `str.decode`
State: closed

This PR adds a test case for `str.decode` and ensures it correctly infers the string datatype, when `dtype=None` and the option `future.infer_string` is used. This argument was introduced in PR https://github.com/pandas-dev/pandas/pull/60940 but has no test for None. This test adds coverage to the following line:


```py
    def decode(
        self, encoding, errors: str = "strict", dtype: str | DtypeObj | None = None
    ):
        ...
        if dtype is not None and not is_string_dtype(dtype):
            raise ValueError(f"dtype must be string or object, got {dtype=}")
        if dtype is None and get_option("future.infer_string"):
            dtype = "str" #✅ NOW COVERED
        # TODO: Add a similar _bytes interface.
        if encoding in _cpython_optimized_decoders:
            # CPython optimized implementation
            f = lambda x: x.decode(encoding, errors)
        ...
```

Note: Parts of this test have been automatically generated by a novel technique that we're currently developing as part of an academic research project aiming at improving test coverage. *To not waste developer time, two researchers manually checked the test before submitting it.* We appreciate the developers' time and any feedback is welcomed.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @hippowm
BUG FIX: None of the included dtypes present in df will raise ValueError with clear error message.
State: open

Before Fix:
```python
>>> df = pd.DataFrame({"a": [1, 2, 3]})
>>> df.describe(include=["datetime"])
...
ValueError: No objects to concatenate
```
After Fix:
```python
>>> df = pd.DataFrame({"a": [1, 2, 3]})
>>> df.describe(include=["datetime"])
...
ValueError: No columns match the specified include or exclude data types
```

- [x] closes #61863
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
pre-commit.ci autofix
```
df = pd.DataFrame({"a": [1, 2, 3]})
df.describe(exclude=[np.int64])
...
ValueError: None of the included dtypes are present in the DataFrame
```

After the fix, this example will also show the same error. Should we change the error message to something like, 
_No columns match the specified include or exclude data types_
@mroeschke I have made the requested changes. can you please review it.
BUG: Fix inconsistency with DateOffset near DST
State: open

- [x] closes #61862
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] ~Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.~
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This fix ensures that `pd.offsets.DateOffset(1)` and `pd.offsets.DateOffset(days=1)` return the same value near a DST transition. 

Labels:
name: Frequency
description: DateOffsets

Comments:
can you add a whatsnew note
pre-commit.ci autofix
BUG: Fix logical method Non 1D Extension Arrays
State: closed

- [x] closes #61866 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Now, ExtensionArrays handle these cases like NumPy arrays do by returning NotImplemented when they can’t perform the operation.
So if the other operand  can handle it, it takes over and the operation succeeds instead of just showing error and stopping execution for the entire code.
DOC: Add Raises section to to_numeric docstring
State: closed

- [x] closes #61811 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
I don't think any problem should occur with the checks. 
Thanks @tisjayy
Fix logical operations broadcasting for 2D ExtensionArrays
State: closed

- [ ] closes #61866 
- [ ] Tests added and passed (local tests not run due to environment setup, but CI will run them)
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] No entry added to whatsnew since this is an internal bug fix


Labels:

Comments:
GitHub says the diff is almost 9 million lines. What’s happening here?
DOC: make doc build run with string dtype enabled
State: closed

~First commit is from https://github.com/pandas-dev/pandas/pull/61722 (will be removed here after that PR is merged and this one is rebased)~, then subsequent commits enable errors on the doc build again and fix issues.

Labels:
name: Docs

Comments:
Going to merge this already to make sure the dev docs are back up. But happy to follow-up with any comment.
Doc simplify footer
State: closed

[x] closes #51536 - Simplified pandas theme footer by removing custom template dependency
[x] Tests added and passed - Documentation changes don't require additional tests, but the build process validates the changes
[x] All code checks passed - Changes follow pandas documentation standards and use proper reStructuredText formatting
[x] Added type annotations - Not applicable for documentation-only changes
[x] Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file - Entry already exists in doc/source/whatsnew/v3.0.0.rst under "Documentation changes" section

Labels:
name: Docs

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.