BrandonFors/pandas-github-issues huggingface dataset:
REF: share some constructor code
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
```
mypy.....................................................................Failed
- hook id: mypy
- exit code: 1
pandas/core/arrays/datetimelike.py:2054: error: Item "ndarray[Any, Any]" of "Union[ndarray[Any, Any], ExtensionArray]" has no attribute "categories"  [union-attr]
pandas/core/arrays/datetimelike.py:2054: error: Item "ExtensionArray" of "Union[ndarray[Any, Any], ExtensionArray]" has no attribute "categories"  [union-attr]
pandas/core/arrays/datetimelike.py:2054: error: Item "ndarray[Any, Any]" of "Union[ndarray[Any, Any], ExtensionArray]" has no attribute "codes"  [union-attr]
pandas/core/arrays/datetimelike.py:2054: error: Item "ExtensionArray" of "Union[ndarray[Any, Any], ExtensionArray]" has no attribute "codes"  [union-attr]
Found 4 errors in 1 file (checked 1392 source files)
```
fixed
REF: Reduce duplicative methods between XML parser classes
State: closed

- [X] Relates to #40131
- [X] [Tests passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests).
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) _(unnecessary)_
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file _(unnecessary)_


Labels:
name: Refactor
description: Internal refactoring of code
name: IO XML
description: read_xml, to_xml

Comments:
Thanks @ParfaitG
ENH/TST: Add BaseMethodsTests tests for ArrowExtensionArray
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Arrow
description: pyarrow functionality

Comments:
Will take a look today (and start whittling down the ping backlog)
ASV: Non-unique DataFrame index
State: closed

- [x] closes #26064 
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

I added an asv benchmark for non-unique indices in DataFrames - based on the benchmark of indices in Series.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
Thanks @kjanker
Update `to_gbq` wrapper method to contain `api_method` parameter
State: closed

ENH: add api_method parameter and some documentation
DOC: update dependencies to require at least version 0.16.0 of pandas-gbq

- [ ] closes #46117
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Dependencies
description: Required and optional dependencies
name: Stale

Comments:
looks fine. @tswast ok here?
Yeah appears that pandas-gbq requires pyarrow>=3.0 while pandas still supports pyarrow>=1.0.1. To support this for 1.5 we would need to do the kwargs trick mentioned https://github.com/pandas-dev/pandas/pull/47539#pullrequestreview-103004336
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
ENH: DatetimeIndex.indexer_between_time support non-nano
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Datetime
description: Datetime data dtype
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
(Failure unrelated)
WEB: Update to `jupyterlite==0.1.0b10`
State: closed

Update to the latest `jupyterlite` release for the interactive terminal on the website.

The latest release adds support for caching assets and offline mode, which should help speed things up a little bit, at least when the page is reloaded.

See the release notes for more information: https://github.com/jupyterlite/jupyterlite/releases/tag/v0.1.0b10

- [x] This should help with #47530
- [x] ~[Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature~
- [x] ~All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).~
- [x] ~Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.~
- [x] ~Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.~


Labels:
name: Docs
name: Web
description: pandas website

Comments:
Any high level before/after load times after this change?
Guessing from the changelog that this will be a net positive speedup. Thanks @jtpio
WARN: Clarify datetime warning when can't parse in accordance with dayfirst 
State: closed

- [x] closes #46210 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

(also, typing swapped_day_and_month whilst I'm here as I hadn't done that when introducing the original warning message)

Labels:
name: Datetime
description: Datetime data dtype
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
Thanks @MarcoGorelli
ENH: TimedeltaArray add/sub with NaT preserve reso
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Timedelta
description: Timedelta data type
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
updated + green
added link to advanced indexing guide in .loc docstring
State: closed

closes #46620
Not sure if the online documentation also needs to change with this docstring.
if it does, how can I change that?

Labels:
name: Docs

Comments:
Hello @INDIG0N! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-06-27 22:50:54 UTC
Thanks @INDIG0N
PERF: use c-division in nancorr
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

There's three divisions in this function that can safely be done at the C-level:

        meanx += 1 / nobs * dx
        meany += 1 / nobs * dy

because `nobs` is at least `1` when we get here (it starts at `0`, `nobs+=1` is called a few lines
above, and `nobs` can't decrease), and

        result[xi, yi] = result[yi, xi] = covxy / divisor

because `divisor` is checked to not be equal to `0` in the preceding line.

This makes a noticeable difference, example:
```
arr1 = np.random.randn(100, 100)
```
On main:
```
%%timeit
nancorr(arr1)

5.39 ms ± 21.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

On this branch:
```
%%timeit
nancorr(arr1)

3.86 ms ± 58.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

I didn't find much difference in nancorr_spearman, I think because that one calls
the Python function rank_1d within the loop, so that might be a bigger bottleneck there

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
nvm, looks like this changes results (failing tests) - not sure I see why, will look into it, but closing for now

EDIT ok might have fixed this now, just a matter of marking the numerator in the division as float

running the same timing test as above now gives:

main:
```
5.68 ms ± 123 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

here:
```
4.97 ms ± 60.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```
thanks @MarcoGorelli
TYP: Return annotations for io/{formats,json}
State: closed


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Looks good. It'd be good to have an issue about this for more context, and to somehow group the different issues in this effort. Also to add to the CI the validation you're using to find those. Does this make sense to you?
> It'd be good to have an issue about this for more context

I will create an issue later today about it (many of these annotations could be easy first contributions). Inspired by @MarcoGorelli's #47497 I'm using pyright to find missing return annotations that it can infer. While this doesn't help pyright much (it infers them already), it helps mypy, and in some cases it also helps pyright (when pyright infers a Union (might be an overload) or infers Any/Unknown (might be able to find a narrower type).


> Also to add to the CI the validation you're using to find those.

At the moment I'm just doing:

```sh
pyright --createstub pandas
grep -R "# -> " typings/pandas/
```
and then see which cases I feel comfortable annotating.

In terms of integrating something like that into the CI:

- [soon'ish?] @MarcoGorelli's #47497 hook could be a way to catch some of the missing return values
- [in a few blue moons] If all functions have tight returns values, we have one fewer discrepancy between mypy and pyright: might be able to remove [`--skipunannotated`](https://github.com/pandas-dev/pandas/blob/7d2f9b8d59908fbf57c6453bc41891efbfe981a6/.pre-commit-config.yaml#L101) run reportGeneralTypeIssues on all files (or the subset of files that currently pass). This would only catch missing return annotations that cause pyright to report errors.
- [in many blue moons] Using #39813
> Inspired by @MarcoGorelli's https://github.com/pandas-dev/pandas/pull/47497 I'm using pyright to find missing return annotations that it can infer.

Nice! Does this obliviate the need for #47497 and associated tool, or does pyright not find some obvious `-> None` statements?
> Does this obliviate the need for #47497 and associated tool, or does pyright not find some obvious `-> None` statements?

Pyright is really good at inferring types :) but `--createstubs` does not care about private functions. Your hook is not only helpful to cover private functions but it is also helpful to enforce obvious `-> None` annotations. I believe that even #39813 (`--verifytypes`) does not enforce unannotated private functions.
Thanks @twoertwein As @datapythonista mentioned, would be good to have a general issue about this pyright specific check.
Opened #47521
REGR: concat materializing index even if sort is not necessary
State: closed

- [x] closes #47501 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
this version prob ok for 1.4

can do enhanced for 1.5
Thanks @phofl 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.4.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 6fd33769b1831d39decbe7932756d33b1bd9eff9
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #47508: REGR: concat materializing index even if sort is not necessary'
```

4. Push to a named branch:

```
git push YOURFORK 1.4.x:auto-backport-of-pr-47508-on-1.4.x
```

5. Create a PR against branch 1.4.x, I would have named this PR:

> "Backport PR #47508 on branch 1.4.x (REGR: concat materializing index even if sort is not necessary)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Could you manual backport when you get the chance @phofl? 
Yep, will do tomorrow
#47565

The conflict was caused by tests :)
TYP: _item_cache and _ixs
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Noticed these in #47497

I have a question about the `_ixs` docstring in `frame.py` - it says that if a slice is passed, then
the result is a view. However, the arguments are annotated to be `int` only. Is the docstring
correct, or is the type annotation too tight?

I presume it's the former, since I'm not able to pass a slice:

```python
>>> df = pd.DataFrame({'a': [1,2, 3], 'b': [4,5,6]})
>>> df
   a  b
0  1  4
1  2  5
2  3  6
>>> df._ixs(0, axis=1)
0    1
1    2
2    3
Name: a, dtype: int64
>>> df._ixs(slice(0, 2, None), axis=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/megorelli/pandas-dev/pandas/core/frame.py", line 3588, in _ixs
    col_mgr = self._mgr.iget(i)
  File "/home/megorelli/pandas-dev/pandas/core/internals/managers.py", line 988, in iget
    block = self.blocks[self.blknos[i]]
TypeError: only integer scalar arrays can be converted to a scalar index
```

Also, typing `_ixs` as that's what's called within `get_item_cache`.
For `Series._ixs`, I've annotated the return type as `Any`, as that's what the return
type of `Series.pop` is.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
I think slice support was removed in c6335792f187b2904c2fdd77662d1048dde631cc.

Looks like this was forgotten back then
Thanks @MarcoGorelli
DEPR/WIP: Deprecate pandas.core
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

WIP, just curious what the CI says.

Labels:

Comments:
Drat, git didn't pickup on the renaming. Going to have to try this again.
Going to try just doing a single submodule instead.
BUG: read_xml iterparse doesn't handle multiple toplevel elements with lxml parser
State: closed

- [X] closes #47422
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature. *(not necessary for unreleased new 1.5 feature)*


Labels:
name: IO XML
description: read_xml, to_xml

Comments:
Thanks @ParfaitG
DOC: Resolve links in whatsnew
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Only added `.` in places where it would get the link to resolve.

Labels:
name: Docs

Comments:
Thanks @rhshadrach
DOC: Minor fixes in whatsnew 1.5.0
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.



Labels:
name: Docs

Comments:
Thanks @rhshadrach
(WIP) TYP: add some obvious -> None to pandas/core
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

via https://github.com/MarcoGorelli/add-obvious-return-none - adding `-> None` where possible so at least those functions will be type-checked

there's gonna be a tonne of CI failures so opening as draft for now

Labels:
name: Typing
description: type annotations, mypy/pyright type checking
name: Stale

Comments:
> via https://github.com/MarcoGorelli/add-obvious-return-none - adding `-> None` where possible so at least those functions will be type-checked

Does this also add `None` to `__init__`? Just surprised that no `__init__` had missing return annotations :)

Not sure whether it is worth the effort: you could extend your hook to also check pyi files: they have obviously no implementation but at least `__init__`'s that have no return annotation should probably return `None`.

If you want to cover non-obvious `None`s, you could use pyright: it infers missing return types.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Closing for now, this'd need some other adjustments first
> Closing for now, this'd need some other adjustments first

Would be great to have it to close #46337
Thanks - I hope to be able to address it soonish
Might be great to set up https://github.com/JelleZijlstra/autotyping (I failed to get it running locally)