BrandonFors/pandas-github-issues huggingface dataset:
PERF: Performance Improvement on `DataFrame.to_csv()` when `index=False`
State: closed

- [x] closes #59312 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


I added an alternative `ndarray` with the same length on `_get_values_for_csv`'s output, used on `write_csv_rows` crude testing
https://github.com/pandas-dev/pandas/blob/642d2446060afb11f9860c79a7339eb6ec96fea7/pandas/_libs/writers.pyx#L42

# Tests
Using the same code as the referenced issue:
```python
import pandas as pd
import pyarrow as pa
import pyarrow.csv as csv
import time

NUM_ROWS = 10000000
NUM_COLS = 20

# Example Multi-Index DataFrame
df = pd.DataFrame(
    {
        f"col_{col_idx}": range(col_idx * NUM_ROWS, (col_idx + 1) * NUM_ROWS)
        for col_idx in range(NUM_COLS)
    }
)
df = df.set_index(["col_0", "col_1"], drop=False)

# Timing Operation A
start_time = time.time()
df.to_csv("file_A.csv", index=False)
end_time = time.time()
print(f"Operation A time: {end_time - start_time} seconds")

# Timing Operation B
start_time = time.time()
df_reset = df.reset_index(drop=True)
df_reset.to_csv("file_B.csv", index=False)
end_time = time.time()
print(f"Operation B time: {end_time - start_time} seconds")
```

Output before performance improvement
```
Operation A time: 869.2354643344879 seconds
Operation B time: 42.1906418800354 seconds
```

Output after performance improvement
```
Operation A time: 51.408071756362915 seconds
Operation B time: 45.78637385368347 seconds
```

**Operation B** is used for time comparison when resetting index, the change improves the performance on **Operation A**

Labels:
name: Performance
description: Memory or execution speed performance
name: IO CSV
description: read_csv, to_csv

Comments:
For you benchmark could you show before and after timings
Output before performance improvement
```
Operation A time: 869.2354643344879 seconds
Operation B time: 42.1906418800354 seconds
```

Output (after performance improvement)
```
Operation A time: 51.408071756362915 seconds
Operation B time: 45.78637385368347 seconds
```

**Operation B** is used for time comparison when resetting index, the change improves the performance on **Operation A**

### Added output times to the description as well
Thanks @KevsterAmp
DOC: fix docstring validation errors for pandas.io.formats.style.Styler
State: closed

Addresses #59606.

Fixes:
https://github.com/pandas-dev/pandas/blob/360597c349f4309364af0d5ac3bab158fd83d9fa/ci/code_checks.sh#L308-L335

Note: order of args for `pandas.io.formats.style.Styler.to_excel` shifted around in order to maintain compatibility with shared doc template handler.

Labels:
name: Docs

Comments:
Thanks @hlakams
DOC: Fix Numpy Docstring errors in pandas.api.extensions.ExtensionArray
State: closed

Related to: https://github.com/pandas-dev/pandas/issues/58539

Fixes the following docstring checks:
```
        -i "pandas.api.extensions.ExtensionArray.duplicated RT03,SA01" \
        -i "pandas.api.extensions.ExtensionArray.fillna SA01" \
        -i "pandas.api.extensions.ExtensionArray.insert PR07,RT03,SA01" \
        -i "pandas.api.extensions.ExtensionArray.isin PR07,RT03,SA01" \
        -i "pandas.api.extensions.ExtensionArray.tolist RT03,SA01" \
        -i "pandas.api.extensions.ExtensionArray.unique RT03,SA01" \
        -i "pandas.api.extensions.ExtensionArray.view SA01" \
```

Labels:
name: Docs

Comments:
Thanks @abhinav-thimma
DOCS: fix docstring validation errors for pandas.Series
State: closed

Partially addresses #59592 

Fixes:
```
 -i "pandas.Series.pop SA01" \
 -i "pandas.Series.list.__getitem__ SA01" \
 -i "pandas.Series.list.flatten SA01" \
 -i "pandas.Series.list.len SA01" \
 -i "pandas.Series.reorder_levels RT03,SA01" \
 -i "pandas.Series.sparse.density SA01" \
 -i "pandas.Series.gt SA01" \
 -i "pandas.Series.lt SA01" \
 -i "pandas.Series.ne SA01" \
 -i "pandas.Series.prod RT03" \
 -i "pandas.Series.product RT03" \
```

Labels:
name: Docs

Comments:
Thanks @hlakams
BUG: groupby.agg with UDF changing pyarrow dtypes
State: closed

Continuation of #58129

- [ ] closes #53030 (Replace xxxx with the GitHub issue number)
- [ ] closes #61636
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Root cause:
-  `agg_series` always forces output dtype to be the same as input dtype, but depending on the lambda, the output dtype can be different

Fix:
- replace all NA with nan
- convert the `results' to respective pyarrow extension array, using pyarrow library methods
- pyarrow library methods is used instead of `maybe_convert_object`, as `maybe_convert_object` does not check for NA, and forces dtype to float if NA is present (NA is not float in pyarrow),


Labels:
name: Bug
name: Groupby
name: pyarrow dtype retention
description: op with pyarrow dtype -> expect pyarrow result

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
> as maybe_convert_object does not check for NA, and forces dtype to float if NA is present (NA is not float in pyarrow),

maybe_convert_object has a convert_to_nullable keyword.  if you pass that as True you'll get back a numpy-nullable array, which you can then convert to pyarrow.  Not sure if that is actually better than what you're doing here.
The correct solution here and to several related issues is an EA._construct_with_inference method that behaves similarly to maybe_convert_objects but preserves dtype backend.  xref #56430.

(Felt the need to plug that, but doesn't need to be a blocker for this in the interim)
thanks @rhshadrach 
There are 2 issues listed in the OP but the check-boxes aren't checked.  I suspect that means they won't be auto-closed.  Should they be manually closed?
Looks like they got closed.

Github only checks that a issue link follows a [set of keywords](https://docs.github.com/en/issues/tracking-your-work-with-issues/using-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword) to close that issue after a PR is merged, so it's not dependent on the check-boxes
They were indeed meant to be closed.
DOCS: fix docstring validation errors for pandas.Series
State: closed

Partially addresses https://github.com/pandas-dev/pandas/issues/59592

Fixes:
```
-i "pandas.Series.struct.dtypes SA01" \ 
-i "pandas.Series.to_markdown SA01" \ 
```

Labels:
name: Docs
name: Series
description: Series data structure

Comments:
Thanks @githubalexliu
DOC: add See Also section to groupby.DataFrameGroupBy.prod
State: closed

This PR fixes a numpy docstring validation error described in https://github.com/pandas-dev/pandas/issues/58063 for pandas.core.groupby.DataFrameGroupBy.prod
```
################################################################################
################################## Validation ##################################
################################################################################

1 Errors found for `pandas.core.groupby.DataFrameGroupBy.prod`:
	ES01	No extended summary found
```

Removed _groupby_agg_method_template since having a template for a single use case seems un-necessary.

Labels:
name: Docs
name: Groupby

Comments:
Thanks @ktseng4096
PERF: Improve efficiency of `BlockValuesRefs`
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Improve efficiency of `BlockValuesRefs` by allowing for more efficient C code-gen

Labels:
name: Enhancement
name: Performance
description: Memory or execution speed performance
name: Copy / view semantics

Comments:
@Tolker-KU - can you run and post [ASV results](https://pandas.pydata.org/docs/dev/development/contributing_codebase.html#running-the-performance-test-suite)
> @Tolker-KU - can you run and post [ASV results](https://pandas.pydata.org/docs/dev/development/contributing_codebase.html#running-the-performance-test-suite)

Thanks for taking a look a this.

I've tried hard to get meaningful runs of the asv benchmarks but I think my computer is so old and slow that noise is dominating the results. I'm getting weird results where some benchmarks are much faster and some are much slower

To show the gains from this I've run below micro-benchmark

```python
>>> N = 10_000
>>> arr = np.random.normal(size=(50, N))
>>> df = pd.DataFrame(arr)
>>> %timeit -n10000 list(df.iterrows())
2.43 ms ± 268 μs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)  <- before
2.1 ms  ± 117 μs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)  <- after
```
Thanks @Tolker-KU
DOCS: fix docstring validation errors for pandas.Series.str
State: closed

Partially addresses https://github.com/pandas-dev/pandas/issues/59592

Fixes:
````
 -i "pandas.Series.str.wrap RT03,SA01" \ 
 -i "pandas.Series.str.zfill RT03" \ 
````

Labels:
name: Docs
name: Strings
description: String extension data type and string data

Comments:
Thanks @wenchen-cai 
hii @wenchen-cai  i am also trying to contribute but facing some issue can you help me please
DOCS: fix docstring validation errors for pandas.Series
State: closed

Partially addresses https://github.com/pandas-dev/pandas/issues/59592

Fixes:
-i "pandas.Series.str.match RT03" \
-i "pandas.Series.str.normalize RT03,SA01" \
-i "pandas.Series.str.repeat SA01" \
-i "pandas.Series.str.replace SA01" \


Labels:
name: Docs
name: Strings
description: String extension data type and string data

Comments:
Thanks @ivonastojanovic
DOCS: fix docstring validation errors for pandas.Series
State: closed

Partially addresses https://github.com/pandas-dev/pandas/issues/59592

Fixes:
-i "pandas.Series.sparse.fill_value SA01" \
-i "pandas.Series.sparse.from_coo PR07,SA01" \
-i "pandas.Series.sparse.npoints SA01" \
-i "pandas.Series.sparse.sp_values SA01" \


Labels:
name: Docs
name: Sparse
description: Sparse Data Type

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: Fix for pandas.to_datetime reports incorrect index when failing
State: closed

- [x] closes #59298 
- [x] Tests added and passed
- [x] All code checks passed
- [x] Added type annotations
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
As per the discussion on the linked issue, removed the index from the error message. As the changes are in the .pyx files, the package will have to be compiled again to reflect the changes if to be tested locally.
Thanks @maushumee
DOC: Enforce Numpy Docstring Validation (Issue #59458)
State: closed

- [x] Addresses one task of #59458 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.



Labels:
name: Docs
name: Timestamp
description: pd.Timestamp and associated methods

Comments:
adding a method with `@property` decorator to add docstring for pandas.Timestamp.day. Please provide feedback if this is an acceptable approach. Can do the same for other properties if approved. 
Thanks @ankit-dhokariya
DOC: Fix pandas.Series.dt seconds, nanoseconds GL08, SA01
State: closed

- [x] xref https://github.com/pandas-dev/pandas/issues/58504

Fixes:
`pandas.Series.dt.nanoseconds SA01`
`pandas.Series.dt.seconds SA01`


Labels:
name: Docs

Comments:
thanks @KatsiarynaDzibrova , looks like there's a test failure
```
_______ [doctest] pandas.core.indexes.accessors.DatetimeProperties.freq ________
394         ...         "2022-02-22 05:22:22-06:00",
395         ...         "2022-02-22 06:22:22-06:00",
396         ...         "2022-02-22 07:22:22-06:00",
397         ...         "2022-02-22 08:22:22-06:00",
398         ...         "2022-02-22 09:22:22-06:00",
399         ...         "2022-02-22 10:22:22-06:00",
400         ...         "2022-02-22 11:22:22-06:00",
401         ...     ]
402         ... )
403         >>> datetimeindex.freq
UNEXPECTED EXCEPTION: AttributeError("'Series' object has no attribute 'freq'")
Traceback (most recent call last):
  File "/home/runner/micromamba/envs/test/lib/python3.10/doctest.py", line 1350, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest pandas.core.indexes.accessors.DatetimeProperties.freq[1]>", line 1, in <module>
  File "/home/runner/micromamba/envs/test/lib/python3.10/site-packages/pandas/core/generic.py", line 6066, in __getattr__
    return object.__getattribute__(self, name)
AttributeError: 'Series' object has no attribute 'freq'
```
Thanks @KatsiarynaDzibrova
PERF: Only clear cached .levels when setting MultiIndex.names
State: closed

When setting `MutilIndex.names`, all cached operations were invalidated. These names are only really tied to `self.levels` (which is also cached), but all usages of `self.levels` only requires the level values and not names, so we shouldn't need to invalidate operations which accessed the old `self.levels` values only.

Also removes a `validate` keyword in `_set_names` which was never set to `False`

Labels:
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
Going to merge since the tests are passing, but happy to follow up if needed
BUG: Fix Ability to set both color and style in pandas plotting
State: closed

- [x] closes #59461  (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Visualization
description: plotting

Comments:
Thanks @AntiKnot
PERF: avoid calling `DataFrame.dtypes` in loop
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

`DataFrame.dtypes` property isn't cheap to compute, this regressed `DataFrame.eval` performance massively (when upgrading from 2.1.2 to 2.2.2).


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
Thanks @messense
DOC: Fix docstrings for Timestamp: strptime, timetz, to_datetime64, to_julian_date
State: closed

Part of #59458

Addresses:

pandas.Timestamp.strptime PR01,SA01
pandas.Timestamp.timetz SA01
pandas.Timestamp.to_datetime64 SA01
pandas.Timestamp.to_julian_date ES01

Changes:
Edited Short Summary of strptime
Added See Also, Extended Summary, Parameters to strptime
Added See Also, Extended Summary to timetz
Edited Short Summary of to_datetime64
Added See Also, Extended Summary to to_datetime64
Edited Extended Summary of to_julian_date
Removed 3 of 4 methods from code_checks.sh (to_julian_date not in the file)

Labels:
name: Docs

Comments:
Thanks @jsngn
REF (string): de-duplicate str_endswith, startswith
State: closed


This adds a _object_compat attribute to govern whether the subclass uses object-dtype semantics in cases where they differ.
<b>update</b>_object_compat turned out to be unnecessary.



Labels:
name: Refactor
description: Internal refactoring of code
name: Strings
description: String extension data type and string data
name: backported

Comments:
> This adds a _object_compat attribute to govern whether the subclass uses object-dtype semantics in cases where they differ.

The question also is to what extent we necessarily _want_ to keep this difference. I don't have a strong opinion on what `ArrowDtype(pa.string())` should do (I am also fine with it being a more "strictly using pyarrow" version, for now), but aligning the behaviour would simplify the implementation .. (cc @mroeschke)
>  I don't have a strong opinion on what ArrowDtype(pa.string()) should do (I am also fine with it being a more "strictly using pyarrow" version, for now)

This is my general hope for `ArrowExtensionArray` and it's behaviors generally to match pyarrow as closely as possible.
> The question also is to what extent we necessarily want to keep this difference. I don't have a strong opinion on what ArrowDtype(pa.string()) should do (I am also fine with it being a more "strictly using pyarrow" version, for now), but aligning the behaviour would simplify the implementation

I don't have an opinion on the behavior, but would prefer to keep behavior-changing PRs separate from REF/de-dup PRs.
```
  /home/runner/work/pandas/pandas/pandas/core/arrays/_arrow_string_mixins.py:129:16 - error: Invalid conditional operand of type "bool | NDArray[bool_] | NDFrame"
    Method __bool__ for type "NDFrame" returns type "NoReturn" rather than "bool" (reportGeneralTypeIssues)
  /home/runner/work/pandas/pandas/pandas/core/arrays/_arrow_string_mixins.py:154:16 - error: Invalid conditional operand of type "bool | NDArray[bool_] | NDFrame"
    Method __bool__ for type "NDFrame" returns type "NoReturn" rather than "bool" (reportGeneralTypeIssues)
```

i dont understand these complaints
lint complaints fixed!
I think comments have been addressed here
CI: Uninstall nomkl & 32 bit Interval tests
State: closed


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Interval
description: Interval data type
name: 32bit
description: 32-bit systems

Comments:
looks like the un-xfailed tests are still complaining on the pyiodide build.  that has numpy 1.26; are other builds now using newer numpys?
Yeah it looks like the closest equivalent 32 bit build is using numpy 2.1.0. I'll just xfail these tests for `WASM` for now until numpy 2 is installed
Merging to get the CI back to green
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.2.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 1044cf442109953987c1a47f476dc90d286b9f0f
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #59553: CI: Uninstall nomkl & 32 bit Interval tests'
```

4. Push to a named branch:

```
git push YOURFORK 2.2.x:auto-backport-of-pr-59553-on-2.2.x
```

5. Create a PR against branch 2.2.x, I would have named this PR:

> "Backport PR #59553 on branch 2.2.x (CI: Uninstall nomkl & 32 bit Interval tests)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
Series replace error
State: closed

- [ ] closes #59452 
- [ ] [Tests added and passed](https://github.com/pandas-dev/pandas/blob/main/pandas/tests/series/methods/test_replace.py) 
- [ ] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst`


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Series
description: Series data structure

Comments:
```bash
ruff....................................................................................................Failed
- hook id: ruff
- exit code: 1

pandas/core/generic.py:7493:89: E501 Line too long (89 > 88)
     |
7491 |                     if isinstance(self, ABCSeries):
7492 |                         raise ValueError(
7493 |                             "to_replace and value cannot be dict-like for Series.replace"
     |                                                                                         ^ E501
7494 |                         )
```
Thanks @fbourgey