BrandonFors/pandas-github-issues huggingface dataset:
Changed read_csv type hint for na_values and added tests Issue#53813
State: closed

- [ ] closes #53813
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.

Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
Hi @twoertwein and @lithomas1 thank you for the feedback. Do you know where the ellipses issue could be coming from? I only changed the na_values type hint for the read_csv() function.
> Hi @twoertwein and @lithomas1 thank you for the feedback. Do you know where the ellipses issue could be coming from? I only changed the na_values type hint for the read_csv() function.

You seem to have changed it :) You can always use `git add --patch` to confirm individual changes you want to add.

You can simply replace `= ...` with `= None` in the non-overload definition.

The easiest way to remove the test is probably `git checkout <commit id from before your changes> pandas/tests/io/parser/test_na_values.py` and then add and commit that.
Hi @twoertwein do the changes I made look good?
> Hi @twoertwein do the changes I made look good?

Now you use `= None` everywhere. You want `= None` for the function definition that doesn't have the `@overload` but otherwise use `= ...`.

It might be better to use `Sequence` instead of `Iterable` (not sure whether any iterable is accepted).

You can remove the test changes you made.
> > Hi @twoertwein do the changes I made look good?
> 
> Now you use `= None` everywhere. You want `= None` for the function definition that doesn't have the `@overload` but otherwise use `= ...`.
> 
> It might be better to use `Sequence` instead of `Iterable` (not sure whether any iterable is accepted).
> 
> You can remove the test changes you made.

Thank you for the feedback. I made the changes
Thanks @kvn4 !
Update whatsnew
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @phofl
Remove GIL from ``take_2d_axis_0``
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This came up, we should get rid of the Gil here

Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
nice!
CLN: Cython 3 cleanups
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This doesn't get all of #34213, but it gets us 90% of the way there.


Labels:
name: Clean

Comments:
Nice thanks @lithomas1
BUG: interval_range with float step
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Interval
description: Interval data type

Comments:
thx @mroeschke
Add whatsnew for arrow
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @mroeschke

I could add my blogpost https://medium.com/p/2891d3d96d2b as evidence for the performance improvements

Labels:
name: Docs

Comments:
Sorry, wrote too many blogs...
Thanks @phofl
DOC: Simplify reshaping.rst
State: closed

* Added intro section and reorganized sections corresponding to the intro
* Added more sphinx references
* Simplified language
* Remove sections covered in the docstrings or that were outdated

Labels:
name: Docs

Comments:
/preview
No preview found for PR #54465. Did the docs build complete?
Looks like the preview mechanism is broken, but the doc build is passing here so merging. Happy to follow up if needed
Fixes grouped rank bug with nullable types.
State: closed

- [x] closes #54206
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Groupby
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays
name: Transformations
description: e.g. cumsum, diff, rank

Comments:
@rhshadrach sir, I have made all the suggested changes, can you once quickly review.
@rhshadrach sir, I have made all the suggested changes, and all the tests have passed. Please check now, Thank you.
@rhshadrach sir, I made the required changes and all the tests are passing, Can you please check now, Thank you.
Thanks @Sai-Suraj-27
TYP: ExtensionDtype.name/type as ClassVar
State: closed

xref #51736

ExtensionDtype.name/type is sometimes a property and sometimes a class variable. Going either way would require non-typing changes. I believe the ClassVar approach is used more frequently and also mandated by some of the register methods.

If ClassVar is the main way of defining type/name, let's annotate as much as possible with that and ignore the few cases that use properties:
```
pandas/core/dtypes/dtypes.py:854: error: Cannot override writeable attribute with read-only property  [override]
pandas/core/dtypes/dtypes.py:1048: error: Cannot override writeable attribute with read-only property  [override]
pandas/core/dtypes/dtypes.py:1422: error: Cannot override writeable attribute with read-only property  [override]
pandas/core/dtypes/dtypes.py:1429: error: Cannot override writeable attribute with read-only property  [override]
pandas/core/dtypes/dtypes.py:1766: error: Cannot override writeable attribute with read-only property  [override]
```

I will fix this PR after a decision has been made.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Since the base EDtype defined it as property previously, going to ClassVar would be an API change that would make it writable (which I think we should make raise) cc @jbrockmendel 
> Since the base EDtype defined it as property previously, going to ClassVar would be an API change that would make it writable (which I think we should make raise) cc @jbrockmendel

I would agree with that (and the previous PR was aimed to making it a property; it also makes it more explicit that subclasses have to implement it) but in practice, it seems that it has to be a ClassVar (at least for `.name`)
https://github.com/pandas-dev/pandas/blob/2451b420d7bd783763971230726616e94ff24904/pandas/core/dtypes/base.py#L287
above is called when trying to register a user-defined ExtensionDtype

Might be good to discuss ClassVar/property separately for name and type (both are themselves not consistent).


edit: When changing these two lines to a property in the pandas-stubs tests, and then run the tests, it fails
https://github.com/pandas-dev/pandas-stubs/blob/main/tests/extension/decimal/array.py#L41
seems like we have a check for .name but not for .type?  .type being a property matters for ArrowDtype.

In general I follow @simonjayhawkins approach of not wanting to futz with code just to making typing checks happy
I'm happy to focus only on `name` and try to create a minimally invasive PR :) Personally, I think it would be better to also change `name` in ExtensionDtype to a `ClassVar` (as it also confuses users - clearly, they cannot use a property (asserts) but they are expected to use one from an inheritance perspective) but then we cannot have a doc-string anymore.
I updated the PR (removed all `.type` changes and all runtime code changes). I think the current state is better than on main but far from ideal: it should be consistent but that requires code changes and maybe even API changes if the property approach is preferred.
Thanks @twoertwein
Clean up autosummary usage
State: closed

This PR stops numpydoc from producing the summary tables for a class's methods and attributes, preferring to instead rely on autosummary. This approach allows using a class.rst template that isn't relying on hacks to get autosummary to still generate stub pages. In a follow-up PR we can look into removing more of those hacks throughout the code base, this PR is just scoped to resolve the issues with the main class.rst template.

- [x] closes #54454 
- [x] closes https://github.com/pandas-dev/pandas/issues/52937
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
In my local builds this PR does not appear to increase the number of warnings at all, suggesting that the changes are safe.
I pushed an update to add accessor attributes to the API reference
Awesome, thanks @vyasr
REF: update decimal tests to TestExtension
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @jbrockmendel
BLD: Add Cython Coverage option
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Build
description: Library building on various platforms

Comments:
You need to specify it as a option.

For debug builds and Cython Coverage enabled you can do
```
pip install -ve . --config-settings=editable-verbose=true --config-settings=setup-args=-DCYTHON_COVERAGE=true --config-settings=setup-args=-Dbuildtype=debug --no-build-isolation
```
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Closing to clear the queue, but feel free to reopen if you have time to circle back
BUG: Fixes #54617 Dataframe to html for empty array with complex dtypes
State: closed

- [x] closes #54167  
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap
name: Complex
description: Complex Numbers

Comments:
Thanks could you add an entry in `2.1.0.rst` in the IO section?
@mroeschke yeah done!
@mroeschke  scripts/tests/test_sort_whatsnew_note.py test_sort_whatsnew_note hook seems to be failing in pre-commit.ci
Thanks @soumya1729 
@mroeschke Was a great learning curve,thanks!
test_map: use data_missing to exercise na_action
State: closed

Change `data` to `data_missing` to exercise the `na_action` parameter.

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @MichaelTiemannOSC
ENH: Implement more efficient merge for arrow strings
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


lets see if this passes ci

```
def random_string():
    return ''.join(random.choices(string.ascii_letters, k=7))


df = pd.DataFrame({"a": pd.Series([random_string() for _ in range(3_000_000)], dtype=pd.ArrowDtype(pa.string())), "b": 1})
df2 = pd.DataFrame({"a": pd.Series([random_string() for _ in range(1_000_000)], dtype=pd.ArrowDtype(pa.string())), "c": 1})

df.merge(df2, how="outer")
```


Main: 1.4701979160308838
PR: 0.6503970623016357

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Arrow
description: pyarrow functionality

Comments:
A whatsnew note would be good otherwise LGTM
Thanks @phofl
Enable many tests for complex numbers
State: closed

Add `complex` to `test_numpy.py` and add `complex128` to `test_numeric.py`.  Then fix the many small problems revealed by the test suite.

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Complex
description: Complex Numbers

Comments:
One of the reasons I made these changes was to get a better understanding of how, in `tests/base/ops.py`, the function `_get_expected_exception` should be written for handling the case of complex numbers.  In this first draft of changes, I didn't really use it, warded off by the comment that "the `self.obj_bar` pattern isn't great in part because it can depend on `op_name` or `dtype`s, but we use it here for backward-compatibility".    I could rewrite the changes so that `BaseArithmenticOpsTests` uses `_get_expected_exception` instead of skipping the tests, but that would also entail lots more work in terms of creating test cases that notice an exception being raised.  I didn't want to do that work speculatively.
I don't understand the mypy error messages (which I've never seen before).
This is not what I meant to do.  GitHub fooled me again!!
REF: move misplaced tests
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @jbrockmendel
REF: use updated patterns in pyarrow TestReduce
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Arrow
description: pyarrow functionality

Comments:
Thanks @jbrockmendel
CI: Try to fix the tar issue on macOS
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Take 2 at fixing the macOS builds. I misdiagnosed the issue the first time (doesn't seem to be an issue with artifacts anymore).

My latest guess is there's something gone wrong with the tarring

Labels:
name: Build
description: Library building on various platforms
name: OS X
description: Related to Mac OS & hardware issues (M1)

Comments:
Hopefully this fixes it.
REF: update to use extension test patterns
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
thx @jbrockmendel
BUG: Fix for pandas-dev#54167
State: closed

- [x] closes #54167 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
@phofl sorry about that, quite new to this. Have added the test now. Pls recheck