BrandonFors/pandas-github-issues huggingface dataset:
CI: start testing Python 3.11
State: closed

- [ ] xref #46680 (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Python 3.11

Comments:
Is this going to supersede https://github.com/pandas-dev/pandas/pull/47032? 
> 

Didn't realize there is an existing PR...
You can just push to my PR directly if you want, or pull the changes here. It's a little farther down the pipeline, but I will not have time to work on it in the near future.
```
FAILED pandas/tests/arrays/categorical/test_api.py::TestCategoricalAPI::test_set_ordered
FAILED pandas/tests/arrays/categorical/test_api.py::TestPrivateCategoricalAPI::test_codes_immutable
FAILED pandas/tests/indexes/multi/test_get_set.py::test_set_levels_codes_directly
FAILED pandas/tests/indexes/period/test_freq_attr.py::TestFreq::test_freq_setter_deprecated
FAILED pandas/tests/io/parser/test_quoting.py::test_null_quote_char[python--0]
FAILED pandas/tests/io/parser/test_quoting.py::test_null_quote_char[python--3]
FAILED pandas/tests/io/parser/common/test_read_errors.py::test_null_byte_char[python]
ERROR pandas/tests/scalar/test_nat.py - TypeError: '<' not supported between ...
ERROR pandas/tests/scalar/test_nat.py - TypeError: '<' not supported between ...
```
FYI Windows will also fail until #47467 is merged in.
Hello @fangchenli! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-08-17 19:19:05 UTC
@fangchenli 
Thanks for picking this up. There haven't been commits on this PR in a while, so I was wondering whether you had the time to finish this PR up.

I've merged main into this PR for you just to see where we're at with the failures.

I'm pretty close to finishing off #47467, and am going to try to get it merged this week. I think I have enough time where I'll be able to finish it up, hopefully before 3.11rc1.
I accidentally pushed some commits here(I misconfigured github cli-meant to push to my own branch and pull request to this one), so I decided to go all the way to getting CI green. Sorry about the noise.

At any rate, this is ready now pending #47467 merge. 
@lithomas1 Thank you very much for finishing this up.
> @lithomas1 Thank you very much for finishing this up.

Thanks for picking this up again.
REF: stop passing self as arg from ExtensionArray methods
State: closed

- [x] closes #46843
- [ ] All [code checks passed].
- [x] Added [type annotations] to new arguments/methods/functions.

🐼



Labels:
name: Refactor
description: Internal refactoring of code
name: Clean
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Stale

Comments:
Hello @brunocoutinhoo! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-07-10 01:01:23 UTC
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
To be fair, it's also from our side that the feedback is limited / unclear. We should maybe update https://github.com/pandas-dev/pandas/issues/46843 to be clearer about what the issue actually proposes the change (or close it if it is no longer actionable)
REF:  if statements merge + typehint correction
State: closed

- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

- if statements merge on `pandas/core/frame.py` and `pandas/io/xml.py`

- Function `_generate_index_names_row()`  on `pandas/io/formats/style_render.py` expected a tuple as a parameter but its only reference uses a list, so i altered the typehint


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
Hello @gugazan! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-06-21 01:16:12 UTC
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
CLN: some simple code cleanup
State: closed

i've changed some inconsistencies on some magic methods's parameters and removed a return from a object initialization

Labels:
name: Clean

Comments:
Thanks for the feedback, @datapythonista! Gonna do some exploring looking for what you mentioned then.
Hey @datapythonista! Sorry for the delay. 

So, i didn't really know how to check if the tests were being executed in the CI, but i tried analyzing the classes on each test (how they are being used and for what purpose) and got to the conclusion that they're not wrong, just the parameters of the magic methods that are not entirely correct. I also checked if these cases appears in other places but they dont.

I really want to help, so if you can give me some pointers i really would appreciate it!
Sorry for the delay @hydratedguy. In general, there are few ways to check if a test is executed. One would be to check the [list of skipped tests](https://github.com/pandas-dev/pandas/runs/6976030372?check_suite_focus=true#step:8:150) in one of the builds. The faster and most reliable could be to add an `assert False` into the test, push, and see if the CI fails.

I had another look, and I think in this case it's not the tests that are not being used, but the functionality itself. What I'd try is to delete the `__len__` and `__missing__` methods completely. They don't seem to be called. Maybe there is a reason why they are there, and then your fix will be fine. But if they are not needed, probably worth removing them instead of fixing them.
Also, if you can please merge upstream into your branch, the CI should be working now.
Thanks for the clean up @hydratedguy 

If in a follow up PR you want to try to delete these methods, which may not be needed, that would be great.
DOC: clarify to_csv float format docstring
State: closed

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

I stumbled across this myself a few days ago. I think this would help


Labels:
name: Docs
name: IO CSV
description: read_csv, to_csv

Comments:
Thanks @phofl
Fix `vertical-align: middle;` for `Styler.to_excel` with xlsxwriter engine
State: closed

- [x] closes #30107
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v1.5.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO Excel
description: read_excel, to_excel

Comments:
Looks to be a merge conflict and some code checks failing otherwise LGTM
@github-actions pre-commit
> @github-actions pre-commit

This functionality was removed recently unfortunately.
Thanks @tehunter
DOC clarify inplace operation section in 1.5 whats_new
State: closed

xref https://github.com/pandas-dev/pandas/issues/47381

Coming from scikit-learn tests failing on scikit-learn with the pandas development version, I found the whats_new entry not very helpful at all. This does a few things:
- use an example with different dtypes so that the behaviour is as advertised for old behaviour (i.e. `ser` is not updated in place). This was not the case before, since when dtype matches, `ser` is updated in place so the code snippet was not showing the right behaviour.
- make it clearer that there is no behaviour change in pandas 1.5

Some of the wording may be not completely accurate, as I don't have a very good grasp of the pandas internals, feel free to suggest improvements!


Labels:
name: Docs

Comments:
I have tried to make the diff smaller as advised
> If you want to spend the time, I think it'd help read this section is the example had some more "interesting" data. 

I did this in my last commit, I also showed `df.iloc[:, 0]` which I think helps in understanding what is happening (whether the column keeps its dtype or not).
Thanks @lesteve
Fix segmentation fault when JSON serializing a PeriodIndex
State: closed

Fixes #46683

- [x] closes #46683
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO JSON
description: read_json, to_json, json_normalize
name: Segfault
description: Non-Recoverable Error

Comments:
lgtm ex @WillAyd comment.
Thanks @roberthdevries
TST: GH26650, added new test to validate numpy matmul function with dataframes
State: closed

- [X] closes #26650 
- [X] [Tests added and passed]
- [X] All [code checks passed]


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Please let the reviewer resolve the conversations
Thanks @carbonleakage 
Thanks @mroeschke and @phofl!
BUG: DataFrame.setitem raising when rhs is ea dtype Series
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

This is a regression on main, so no whatsnew needed.

Is it on purpose, that np_can_hold_element raises when feeding a ea dtype Series as element?


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Thanks @phofl 
> Is it on purpose, that np_can_hold_element raises when feeding a ea dtype Series as element?

I don't think so, no.
ENH: TDA.total_seconds support non-nano
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Timedelta
description: Timedelta data type
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Looks like there's a doctest that's failing.
> Looks like there's a doctest that's failing.

neat, looks like this improves floating point error in the relevant case


```
pandas/io/excel/_base.py:793: error: List item 0 has incompatible type "Union[int, Sequence[int]]"; expected "int"  [list-item]
pandas/io/excel/_base.py:797: error: Argument 1 to "len" has incompatible type "Union[int, Sequence[int], None]"; expected "Sized"  [arg-type]
pandas/io/excel/_base.py:798: error: Argument 1 to "len" has incompatible type "Union[int, Sequence[int], None]"; expected "Sized"  [arg-type]
Found 3 errors in 1 file (checked 1391 source files)
```
Add tests
State: closed

- [x] closes #24091
- [x] closes #25169
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Hello @jackgoldsmith4! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-06-21 13:48:48 UTC
@github-actions pre-commit
thx @jackgoldsmith4
BUG FIX: incorrect type when casting to nullable type in multiindex dataframe
State: closed

- [ ] closes #46896
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: MultiIndex
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
> Did not see this yesterday when opening my pr. #48094
> 
> We have to set column wise here, otherwise we will cast dtypes as soon as we have multiple dtypes in our value to set, for example #20476.
> 
> We are doing the same if the indexer is a list of keys

Ok I will close my PR. Your PR is much better. Will look at your solution.
REF: GH38174 - Refactoring DataFrame.combine_first
State: closed

- [ ] closes #38174 
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Stale

Comments:
Am i using the right where function? Do i have to use self._mgr.where(...) or self.where(...) ?
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@hydratedguy closing as stale. Please ping of you want to continue
Hey @phofl ! From what i've seen i think that to solve [#38174](https://github.com/pandas-dev/pandas/issues/38174) it's needed to do changes on a Internals file, so there's not much i can do. What do you think?
You can fix this through making changes in the internals or use a different function 
But only a pandas dev can do changes in the internals right? And if i used a different function i woulnd't be following what the PR asked for. 
This is the pandas dev repo. Every contributor can make changes
Hmm got it. Even so, I believe that as a beginner I am not able to solve this problem rn. Sorry for bothering you and thanks for answering me!
BUG: .fillna(-1) does not work in pd.eval 
State: closed

- [x] closes #46471
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: iterparse of read_xml not parsing duplicate element and attribute names
State: closed

- [X] closes #47343 (Replace xxxx with the Github issue number)
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/v1.5.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO XML
description: read_xml, to_xml

Comments:
Thanks @ParfaitG
TYP: read_sas
State: closed


Labels:
name: IO SAS
description: SAS: read_sas
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Does this need documentation because we’re breaking the API or do we consider it internal with no guarantees?
> Does this need documentation because we’re breaking the API or do we consider it internal with no guarantees?

If we want this change @jreback  (I certainly would like it from a typing perspective), I think it should have a whatsnew. Otherwise, I'm happy to keep `None` but then adjust the overloads of `read_sas` to include `None`
i think it's fine

just add a note on the very slight api change
Thanks @twoertwein
PERF: Performance improvement on dataframe.update. 
State: closed

the setitem under the hood is doing df copy per https://github.com/pandas-dev/pandas/issues/46267
therefore, the performance is compromised, per the suggestion, using loc for assignment instead 
see the detail example in the issue link for metrics.

- [X] closes #47392 (Replace xxxx with the Github issue number)

~Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature~
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

~Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.~
~Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.~


Labels:

Comments:
@phofl  would you mind taking look at this PR
@warm200 Thanks for the PR. https://github.com/pandas-dev/pandas/pull/47327 has been merged which includes the change in this PR so closing.
Revert "REF: remove JoinUnit.shape (#43651)"
State: closed

This reverts commit bb9a9852265915a4688f772dd062d3fcf4159a32.

This is also included in https://github.com/pandas-dev/pandas/pull/47372/, but reverting this single commit separately will make it easier to backport the other PR, since this is the only commit that was not already included in 1.4, but was only done in 1.5 (cc @simonjayhawkins)


Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
sure
SAS7BDAT parser: Faster string parsing
State: closed

Speed up SAS7BDAT string reading.

Today this brings a modest 10% performance improvement. But together with the other changes I will be proposing it will be a major bottleneck.

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Performance
description: Memory or execution speed performance
name: IO SAS
description: SAS: read_sas

Comments:
I've ran against all SAS7BDAT test files in the repo, here's the ASV results.

```
asv continuous main sas/rstrip -b sas
```

Run 1

```
       before           after         ratio
     [e915b0a4]       [e20cf659]
     <main>           <sas/rstrip>
-     8.13±0.03ms      5.02±0.02ms     0.62  io.sas.SAS.time_productsalessas7bdat
-      14.9±0.3ms      7.32±0.05ms     0.49  io.sas.SAS.time_load_logsas7bdat

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```

Run 2

```
-      8.35±0.2ms      4.96±0.02ms     0.59  io.sas.SAS.time_productsalessas7bdat
-      14.8±0.2ms      7.29±0.02ms     0.49  io.sas.SAS.time_load_logsas7bdat
```

Shall I add those to the ASV suite (or simply all of the files)?
thanks @jonashaag
BUG: read_excel raising uncontrolled IndexError when header references non-existing rows
State: closed

- [x] closes #43143 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO Excel
description: read_excel, to_excel

Comments:
Thanks @phofl
BUG: read_csv may interpret second row as index names even if index_col is False
State: closed

- [x] closes #46569 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I think index_col should take precedence over guessing

Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
Thanks @phofl