BrandonFors/pandas-github-issues huggingface dataset:
PERF: Remove unnecessary asof join functions
State: closed

Several functions in the join cython are module are basically just calling others, and can probably be removed. This wouldn't
be a big deal inside python, but in this case it cuts down considerably on the amount of generated cython code.
    
The tests pass for me locally when I do this and there seems to be significant space savings.

Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Hello @wlach! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-05-05 11:37:26 UTC
> significant space savings

cool, how much?
> > significant space savings
> 
> cool, how much?

I think I was seeing about a megabyte on join.pyx, although I think there might be the potential for further gains.

Unfortunately I wasn't running the tests properly so I didn't notice that my PR didn't actually work. :frowning: Need to dig in a bit further.
Ok, think I got this working now. It's not as big an improvement as I thought (about 100k or ~4% off of the generated .so for join, not sure why I got it wrong) but I don't think it impacts readability (arguably it helps) so it seems worth doing.

Before

- 2617696 bytes
- nm: 2949 symbols

After

- 2508880 bytes
- nm: 2873 symbols

Ok, I think this is ready for a look!

docstring and typing information is giving an error but I don't see how it could be related to the code changes here (a very similar pull request from last night was passing):

```
Error: /home/runner/work/pandas/pandas/pandas/core/generic.py:2529:PR03:pandas.DataFrame.to_hdf:Wrong parameters order. Actual: ('path_or_buf', 'key', 'mode', 'complevel', 'complib', 'append', 'format', 'index', 'min_itemsize', 'nan_rep', 'dropna', 'data_columns', 'errors', 'encoding'). Documented: ('path_or_buf', 'key', 'mode', 'complevel', 'complib', 'append', 'format', 'index', 'errors', 'encoding', 'min_itemsize', 'nan_rep', 'dropna', 'data_columns')
```
> Ok, I think this is ready for a look!
> 
> docstring and typing information is giving an error but I don't see how it could be related to the code changes here (a very similar pull request from last night was passing):
> 
> ```
> Error: /home/runner/work/pandas/pandas/pandas/core/generic.py:2529:PR03:pandas.DataFrame.to_hdf:Wrong parameters order. Actual: ('path_or_buf', 'key', 'mode', 'complevel', 'complib', 'append', 'format', 'index', 'min_itemsize', 'nan_rep', 'dropna', 'data_columns', 'errors', 'encoding'). Documented: ('path_or_buf', 'key', 'mode', 'complevel', 'complib', 'append', 'format', 'index', 'errors', 'encoding', 'min_itemsize', 'nan_rep', 'dropna', 'data_columns')
> ```

this is unrelated
thanks @wlach
TYP: narrow type bounds on extract_array
State: closed

xref https://github.com/pandas-dev/pandas/issues/37715

Narrowing the type bound allows resolving some ignored mypy errors.
The other modified code is needed because `extract_array` no longer returns `Any`, causing more strict type checking in the calling methods.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@twoertwein @simonjayhawkins good here?
Thanks @iasoon. https://github.com/pandas-dev/pandas/pull/46942#discussion_r866212272 can be followed up with in a subsequent PR if needed 
> Thanks @iasoon. [#46942 (comment)](https://github.com/pandas-dev/pandas/pull/46942#discussion_r866212272) can be followed up with in a subsequent PR if needed

not needed. Thanks @iasoon
TYP: enable reportUnusedImport
State: closed

pyright's reportUnusedImport checks py and pyi files for unused imports. If an import is explicitly marked as public (re-exported using `as` or in `__all__`) it is "used" (flake8 seems to use the same definition).

Adding unused imports to `__all__` becomes messy when the file does not yet have `__all__`: need to list also all public symbols (constants, classes, functions).

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
This generally looked okay if you want to merge main.
`__all__` as two functions: 1) it affects `from xyz import *` (imports only functions/classes inside `__all__`) and 2) in a py.typed library `__all__` defines the public interface of a module (unless the module is private, starts with `_`).

>     1. Nothing changed from the "public" API?

Technically, users will import fewer modules if they use `from xyz import *` which could be a public API change.

>     2. In terms of code flow, we will need to ensure all "unused" imports are defined in `__all__`?

flake8/pyright will let you know about unused imports :) They can then either be removed, be ignored (if they have some side effects), or added to `__all__` if they are meant to be public.
> Technically, users will import fewer modules if they use from xyz import * which could be a public API change.

Could you do a quick before/after check when `from pandas import *` is run? I imagine it should hopefully not change in a negative way, and maybe can be noted as a benefit that less stuff is imported :)
Seems there is actually no differences between main and this PR when imported everything from a file:

```py
import importlib
from pathlib import Path
import pickle

import ipdb

imported = {}
for py in Path("pandas").glob("**/*.py"):
    print(py)
    py = str(py).replace("/", ".").removesuffix(".py")
    mod = importlib.import_module("py")
    imported[py] = [x for x in dir(mod) if not x.startswith("_")]
with open("before.pickle", mode="wb") as file:
    pickle.dump(imported, file)

# ran the above first on this PR and save as after.pickle
# and then the entire script on main

after = {}
with open("after.pickle", mode="rb") as file:
    after = pickle.load(file)

for py, imports_b in imported.items():
    imports_a = set(after[py])
    imports_b = set(imports_b)
    remove = imports_b.difference(imports_a)
    added = imports_a.difference(imports_b)

    if len(remove) > 0 or len(added) > 0:
        ipdb.set_trace()  # never ended up here
```
Thanks for double checking @twoertwein!
PERF: Remove docstrings from inline cython code
State: closed

These functions aren't externally available, and removing them saves
a small amount of space (approximately 250k in the size of the generated
.so's on a linux64 machine).


Labels:
name: Docs
name: Stale

Comments:
More details:

base sizes: https://gist.github.com/wlach/d1a70c3246e51137e71d6f992ff34149
"no docstrings" sizes: https://gist.github.com/wlach/ac2151d94ba1c895f530ee1585c545c9

Calculating savings:

```
>>> import pandas
>>> df1 = pandas.read_csv('cython-sizes-base.csv')
>>> df1['bytes'].sum()
90027048
>>> df2 = pandas.read_csv('cython-sizes-no-docstrings.csv')
>>> df2['bytes'].sum()
89744976
```

I am especially interested in this for the case of pyodide, where the load times of pandas is a very large bottleneck: https://github.com/iodide-project/pyodide/issues/347

I don't expect this to help that much, but any savings (which doesn't affect end user functionality) is welcome.
Thanks for the PR! I'm not sure they are really private, because this does break some of the tests checking docstrings.
> I don't expect this to help that much, but any savings (which doesn't affect end user functionality) is welcome.

If you can get this passing the CI, this sounds good to me.
> Thanks for the PR! I'm not sure they are really private, because this does break some of the tests checking docstrings.

Yes, I think you're right on closer inspection (I had a hard time understanding the test runs) -- and it's quite likely it's the larger docstrings which are the ones that are really needed. This is probably not worth pursuing, but I'll leave this open for a bit in case I'm missing something obvious.
Perhaps you could try making this a compile time option that is disabled by default?
> Perhaps you could try making this a compile time option that is disabled by default?

Not my call, but I'd personally be a bit reluctant to do this -- the savings aren't IMO great enough to justify the increased complexity.
As an aside, if you're looking to implement space savings, there has been consensus to remove tests from the distribution: https://github.com/pandas-dev/pandas/issues/30741
We are already unvendoring pandas tests as a separate package in Pyodide, so that is not an issue.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Looks like we're not moving forward with this, and the discussion is stale, closing.
DOC: added index, dropna description for HDF methods #45030
State: closed

- [ ] closes #45030
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: IO HDF5
description: read_hdf, HDFStore

Comments:
thanks @alexondor 
Hello @jreback ! Could you please explain about failed test "pandas.Series.to_hdf:Wrong parameters order"? Is it correct?  I inserted comments in order to function input parameters. But there was previously violated order.
DataFrame.replace with dict doesn't work when value=None
State: closed

- [x] closes #46606 (Replace xxxx with the Github issue number)

Labels:
name: Docs

Comments:
Hello @stanleycai95! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-05-02 22:49:26 UTC
Thanks @stanleycai95
TYP: overload maybe_downcast_numeric and maybe_downcast_to_dtype
State: closed

xref https://github.com/pandas-dev/pandas/issues/37715

Overload type definitions for these methods so that passing in an ndarray returns an ndarray.
This allows removing two more ignored mypy errors.

Labels:

Comments:
> Instead of using overloads, does it make sense to have an ArrayLikeT, i.e. a TypeVar bound to `np.ndarray | ExtensionArray`?
> 
> cc @simonjayhawkins @Dr-Irv

I think (but I'm not sure) that the times to use a `TypeVar` are when subclassing gets involved with respect to `self` .  I think the overload approach is correct here, but also agree with @twoertwein that with `ArrayLike` including `np.ndarray`, it's better to have one overload for `np.ndarray` and a different one for `ExtensionArray`
Thanks for the comments!
I played around a bit with the typevar route, but I do not think it is sound in this case: both methods will return `np.ndarray` when passed `np.ndarray`, but could return either `ExtensionArray` or `np.ndarray` when passed ExtensionArray.
I agree non-overlapping overloads are cleaner here, I resolved that.
Thank you @iasoon!
TYP: fix a few annotations in offsets.pyi
State: closed

- [ ] closes #46908
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Some doc-strings refer to `pd.HolidayCalendar` but that doesn't seem to exist?!

There are a few more un-annotated variables and a few containers (list and dict) without types but I'm not familair enough with them to annotate them.

Labels:

Comments:
> Have we standardized on whether we put `None` last or first?

Not that I'm aware of. The only example I found is here: https://pandas.pydata.org/docs/development/contributing_codebase.html#style-guidelines (None at the end).


> > Have we standardized on whether we put `None` last or first?
> 
> Not that I'm aware of. The only example I found is here: https://pandas.pydata.org/docs/development/contributing_codebase.html#style-guidelines (None at the end).

I think we use the style of `str | None` more than `None | str`, so can you use the former throughout in this file for this PR?
Thanks @twoertwein
TYP: fix MultiIndex._names type
State: closed

xref https://github.com/pandas-dev/pandas/issues/37715


_names was never really a FrozenList.
Also adapted some related code in _count_level
The change in _count_level is required because `list` cannot be indexed by `Hashable` - this issue had not been detected by mypy up until now because the argument of `__getitem__` is untyped for `FrozenList`.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks @iasoon
CI: More targeted pyarrow version testing
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature

Hoping to supersede https://github.com/pandas-dev/pandas/pull/46386. cc @lithomas1 

While the job queue bandwidth is larger now, probably don't need to test different pyarrow versions with different Python versions. Instead:

* PY38 will test pyarrow 5, 6, 7
* PY39 and PY310 will just test pyarrow 5

Labels:
name: CI
description: Continuous Integration
name: Arrow
description: pyarrow functionality

Comments:
> Thanks for picking this up! Do you know if we should remove pyarrow from the env files, given that we're installing it manually here?

I don't think we can since the Azure Windows/MacOS builds use these files and don't install pyarrow manually
@meeseeksdev backport 1.4.x
BUG: added finalize to merge, GH28283
State: closed

Progress towards #28283

This PR gives the `merge` method the ability to propagate metadata using `__finalize__`.

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v1.5.0.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Hello @jQuinRivero! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-05-01 02:49:59 UTC
DOC: update shortened link to full (#46899)
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
could you rebase onto `main` please?
Thanks for the follow up @charlogazzo (failure unrelated)
TYP/CI: bump mypy&pyright
State: closed

Mypy aims to have a monthly release.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Marking as draft until mypy 0.950 is available through conda
mypy 0.950 is now available on conda-forge :)
doc-test failure seems to be unrelated but mypy&pyright passed :)

@simonjayhawkins
@twoertwein planning to restart the ci here once #47015 is green and merged, so nothing more to do here as far as i'm concerned.
BUG: DatetimeIndex.resolution with nanosecond reso
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Timezones
description: Timezone data dtype
name: Deprecate
description: Functionality to remove in pandas

Comments:
Rebased + green
ENH: fields.get_start_end_field support non-nano
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Thanks @jbrockmendel
Update Git.io deprecated link (#46888)
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
thanks @charlogazzo
BUG: rolling aggregate() with a list of functions along axis 1 raises ValueError #46132
State: closed

- [x] closes #46132 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Window
description: rolling, ewma, expanding

Comments:
can you merge master and resolve conflicts
@github-actions pre-commit
Hello @xr-chen! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-06-05 20:54:19 UTC
REF: handle 2D in tslibs.vectorized
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code
name: Datetime
description: Datetime data dtype

Comments:
Added a comment pointing back to the discussion here; i think that's the best we'll be able to do for the forseeable future.  cc @mroeschke
DOC: GH27557 Updates Python support documentation to match NumPy NEP 29
State: closed

- [x] closes [#27557](https://github.com/pandas-dev/pandas/issues/27557)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This updates the documentation for Python support to match NEP 29, and includes a link to NEP 29. The change is at line 410 of docs/development/policies.html.

Labels:
name: Docs

Comments:
Thanks, I included the change to the policies.rst document as well. I resubmitted the PR using the double underscore. Is there a way for me to test that a link in an .rst file works in my local environment first? With the .html file, I checked it in my browser, but am I correct in understanding something similar is not possible with .rst files?
You can test this locally by building the docs with these instructions: https://pandas.pydata.org/pandas-docs/stable/development/contributing_documentation.html#how-to-build-the-pandas-documentation
Backport PR #45247 on branch 1.4.x (PERF: find_stack_level)
State: closed

Backport PR #45247: PERF: find_stack_level

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
Looks like we have perf issues from find_stack_level(#46874), so I'm backporting this PR from main. 
thanks @lithomas1
PERF: use C version of np.empty
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
> Is this something that can be wrapped upstream in Cython too?

I've considered making a helper function `empty_1d` and putting in util.pxd, haven't thought about upstreaming it.