BrandonFors/pandas-github-issues huggingface dataset:
BUG: Add fix for hashing timestamps with folds
State: closed

- [x] closes #33931, #40817
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Edit: 

https://github.com/pandas-dev/pandas/issues/42305#issuecomment-895218386
This comment identifies the problem as being upstream in datetime's construction of timestamps which fails during some DST changes.

The datetime hash function then being used tries to unset the fold representing the DST change, (https://github.com/python/cpython/blob/main/Lib/datetime.py#L2117). But it does this using the datetime replace function - this change fixes that problem by removing the fold prior to calling the datetime hash function using timestamp's replace function instead.

Labels:
name: Timezones
description: Timezone data dtype

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@jreback I don't see an asv set datetime specifically, so I tried to run the all of the ones for `tslibs`, let me know if you want me to run a different set.

```
-> asv continuous -f 1.1 -E virtualenv upstream/master fold_hash_fix -b ^tslibs
...

     [95123939]       [c6400d66]
     <master>         <fold_hash_fix>
-     1.47±0.02μs      1.34±0.03μs     0.91  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(0, 1000, tzlocal())
-     4.40±0.03μs      3.99±0.09μs     0.91  tslibs.tslib.TimeIntsToPydatetime.time_ints_to_pydatetime('time', 0, <DstTzInfo 'US/Pacific' LMT-1 day, 16:07:00 STD>)
-     1.49±0.03μs      1.35±0.04μs     0.91  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(0, 2011, tzlocal())
-     1.47±0.08μs      1.32±0.02μs     0.90  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(0, 3000, None)
-     1.53±0.07μs      1.37±0.02μs     0.89  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1, 2000, tzlocal())
-     1.50±0.08μs      1.33±0.02μs     0.88  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1, 1000, None)

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```


There aren't any asvs that specifically hit this.  If you wanted to add one, it would go in benchmarks.tslibs.timestamp.  Ideally in TimestampOps, but i dont know if it the `ts` defined in `setup` there makes a difference.

(an asv for this is likely overkill)
can you add the test here: https://github.com/pandas-dev/pandas/issues/40817 as well. ping on green for merge
@jreback Sounds good, I've added the reindex unit test and mentioned it in whatsnew
@jreback Any more thoughts?
thanks @JonWiggins very nice!
CI: Revert splitting 3.10 tests
State: closed

Maths says this should work now.

- [ ] closes #44173
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Don't backport.

Labels:
name: CI
description: Continuous Integration
name: Python 3.10

Comments:
hmm something taking a long time
> hmm something taking a long time

Not the Python 3.10 tests, though. Just the usual suspects :)
gentle ping @jreback. This is ready.
can u rebase once more
thanks @lithomas1
CLN: missing f for f-string
State: closed


Labels:
name: Docs

Comments:
wonder if there is a way to actually check for this (out of scope here).

cc @MarcoGorelli would be a nice plugin for style checking.
I think pylint detects this... might be worth bringing some of its (less pedantic) rules 
https://github.com/MichaelKim0407/flake8-use-fstring That's how I found it but we have many false positives because of the shared documentation using "{}"
BUG: broadcasting listlike values in Series.__setitem__ GH#44265
State: closed

- [x] closes #44265
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

This should in turn allow us to avoid some special-casing that we do in internals/putmask_smart.

Sits on top of #44261

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Nested Data
description: Data where the values are collections (lists, sets, dicts, objects, etc.).

Comments:
@jreback following this we can get rid of both putmask_preserve and putmask_without_repeat
can you rebase
rebased + greenish
updated + greenish
lgtm @phofl if you can have a look
thx @jbrockmendel
DOC: added examples to DataFrame.var #44162
State: closed

- [x] closes #44162 by adding two examples to docstring of DataFrame.var
- [x] tests added / passed
- [x] All linting tests passed (pre-commit)

Screenshot of locally built documentation:

![var](https://user-images.githubusercontent.com/58558211/139727701-f6c92ad1-948d-42bd-8989-2d874e3cfa58.PNG)




Labels:
name: Docs

Comments:
Thanks @MarcoGorelli for the motivation via cat images.
Apologies for the old commits. Next time I'll create a new branch from scratch.
It seems like there is a check that keeps failing `CI > Checks (pull_request) > Typing validation`. I do not understand the error report so I'm not sure what is going wrong. Please see the details of the error report below and advise:

```console
Run ci/code_checks.sh typing
mypy --version
mypy 0.910
Performing static analysis using mypy
pandas/core/frame.py:997: error: Incompatible return value type (got "Optional[str]", expected "str")  [return-value]
Found 1 error in 1 file (checked 1318 source files)
Performing static analysis using mypy DONE
Performing static analysis using pyright
No configuration file found.
pyproject.toml file found at /home/runner/work/pandas/pandas.
Loading pyproject.toml file at /home/runner/work/pandas/pandas/pyproject.toml
Assuming Python platform Linux
Searching for source files
Found 311 source files
0 errors, 0 warnings, 0 infos 
Completed in 48.921sec
Performing static analysis using pyright DONE
Error: Process completed with exit code 1.
```
> It seems like there is a check that keeps failing `CI > Checks (pull_request) > Typing validation`. I do not understand the error report so I'm not sure what is going wrong. Please see the details of the error report below and advise:
> 
> ```
> Run ci/code_checks.sh typing
> mypy --version
> mypy 0.910
> Performing static analysis using mypy
> pandas/core/frame.py:997: error: Incompatible return value type (got "Optional[str]", expected "str")  [return-value]
> Found 1 error in 1 file (checked 1318 source files)
> Performing static analysis using mypy DONE
> Performing static analysis using pyright
> No configuration file found.
> pyproject.toml file found at /home/runner/work/pandas/pandas.
> Loading pyproject.toml file at /home/runner/work/pandas/pandas/pyproject.toml
> Assuming Python platform Linux
> Searching for source files
> Found 311 source files
> 0 errors, 0 warnings, 0 infos 
> Completed in 48.921sec
> Performing static analysis using pyright DONE
> Error: Process completed with exit code 1.
> ```

It's unrelated, and fixed on master, don't worry about it
> It's unrelated, and fixed on master, don't worry about it

Ok great. Is there anything else I can do regarding this PR before it can be merged?
Looks good to me, I'll just leave it open a bit in case others have comments
Good idea, thanks @MarcoGorelli 
Thanks, @mroeschke 
Just made the changes you recommended.
Thanks @ruankie
TYP: use TYPE_CHECKING for import_optional_dependency("numba")
State: closed

xref https://github.com/pandas-dev/pandas/pull/44233#issuecomment-955594933 or could go with https://github.com/pandas-dev/pandas/pull/44254#issuecomment-956450632 instead.

cc @twoertwein 

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
There is one more: pandas/core/window/numba_.py:261 After that one you can also remove the pyright comment from the file.

If `numba.generated_jit` can be typed in `typings.numba`, we can also remove the pyright comment in pandas/core/util/numba_.py.
@simonjayhawkins I'd be happy to merge this PR (after rebasing), but I'm not sure whether I (as the only reviewer so far) am allowed/supposed to also merge it? 
@twoertwein can you merge master and @mroeschke if you can have a look
thanks @simonjayhawkins
BUG: Styler `hide` compatible with `max_columns`
State: closed

closes #43704

Description of render trimming and maximums bug:

```
pd.options.styler.render.max_columns = 5
pd.options.styler.render.max_rows = 5
df = DataFrame(np.random.rand(10,10))
df.columns = pd.MultiIndex.from_product([[0,1,2,3,4], [0,1]])
df.index = pd.MultiIndex.from_product([[0,1,2,3,4], [0,1]])
df.index.names, df.columns.names = ["a", "b"], ["c", "d"]
df.style.hide([(0,0), (0,1), (1,0)], axis=1).hide([(0,0), (0,1), (1,0)], axis=0)
```

## Pre #44248
![Screen Shot 2021-11-01 at 19 02 48](https://user-images.githubusercontent.com/24256554/139718765-88bce097-e5f0-445f-8614-9b351d15e336.png)

## After #44248 (merged)
![Screen Shot 2021-11-01 at 19 03 47](https://user-images.githubusercontent.com/24256554/139718908-c3ce3e29-4360-4f6b-b917-192b516d8ec6.png)


## After this PR
![Screen Shot 2021-11-01 at 19 05 04](https://user-images.githubusercontent.com/24256554/139719076-16d303ed-9688-4305-9f74-04988193f4bb.png)



Labels:
name: Bug
name: Styler
description: conditional formatting using DataFrame.style

Comments:
thanks @attack68
BUG: Series.replace(method='pad') with EA dtypes
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
I guess technically a user facing change? if so can you add a note
TST: Add test for where inplace
State: closed

This adds a test for ensuring that None-to-NaN type-casting is consistent with `where` when `inplace=True` and otherwise.

- [x] closes #22051 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
thanks @rlvoyer
TYP: check typings with pyright
State: closed

small followup to #44233

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @twoertwein

Any changes needed for the pre-commit check?
Do you think it makes sense to replace 

```py
numba = import_optional_dependency("numa")
```

with

```py
import_optional_dependency("numa")
import numba
```

Unfortunately, I didn't find a way to type `import_optional_dependency` (even with overloads, a module by itself is no type).

> Any changes needed for the pre-commit check?

No, pre-commit just calls `pyright` same as for the CI.

I have a PR whch i'm just running the tests etc for before opening in which I use

```python
    if TYPE_CHECKING:
        import numba
    else:
        numba = import_optional_dependency("numba")
```
Thanks @twoertwein
TYP: Subset of "Improved the type stubs in the _libs directory to help with type checking"
State: closed

The two big parts missing from #43744 are interval.pyi and offsets.pyi as they result in many mypy errors.

I had already made changes to somewhat accommodate for interval.pyi and offsets.pyi (if these files are included, we have now "only" ~100 mypy errors). I kept these changes in this PR.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Hello @twoertwein! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-12-12 19:24:28 UTC
cc @Dr-Irv 
> I see that you put some typing in the `pandas/tests/tseries/...` . Have we started typing the test code, and running `mypy` on parts of the test code?

Some tests are typed checked, including those files.

> If not, I don't think that should be included in the PR.

The test changes I made are technically not needed for this PR. They will be needed for `offsets.pyi` from #43744. Happy to remove them and then add them with `offsets.pyi` later.


> The test changes I made are technically not needed for this PR. They will be needed for `offsets.pyi` from #43744. Happy to remove them and then add them with `offsets.pyi` later.

IMHO, I think the changes to the test files should be left out for now.  That will make the PR smaller.  But I would defer to @simonjayhawkins on this
looks good, can you rebase

@simonjayhawkins if any comments.
> IMHO, I think the changes to the test files should be left out for now. That will make the PR smaller. But I would defer to @simonjayhawkins on this

The test modules are type checked but the test functions are not (check_untyped_defs is False). Saying that we have several test modules ignored in the mypy config
@twoertwein can you merge master
thanks @twoertwein very nice
DataFrame.convert_dtypes doesn't preserve subclasses
State: closed

- [x] closes #43668 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

I've added a fixture for the specific construction of a subclassed dataframe from the issue - which mirrors the setup in geopandas, I expect this probably needs to be relocated (I could have constructed this inline with the test and maybe that's the right solution, but I can see analogous changes for `astype` perhaps being useful in the geopandas context as well). 

I've also added the call to `__finalize__` although it's not strictly needed, since that's also needed by #28283 (Note that the 1 dimensional case already has `__finalize__` called indirectly via `astype`.


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors
name: Subclassing
description: Subclassing pandas objects

Comments:
CI failures now don't seem related, but I could be wrong

Edit: now it definitely does, trying to sort out the type hints
> looks ok, need to remove the print. ping on green.

@jreback looks like everything is passing now.
thanks @m-richards
BUG: styler render when using `hide`, `MultiIndex` and `max_rows` in combination
State: closed

There was a previous fix for #43703 but that dealt only with the SingleIndex case

- [x] closes #44247
- [x] tests added / passed
- [x] whatsnew entry

Visuals of the fix compared to the issue:
![Screen Shot 2021-10-31 at 08 51 43](https://user-images.githubusercontent.com/24256554/139573572-1663d3fa-ceac-4e9a-b677-2108b6f2cef7.png)



Labels:
name: MultiIndex
name: Styler
description: conditional formatting using DataFrame.style

Comments:
thanks @attack68
TST: Old issues
State: closed

- [x] closes #16380
- [x] closes #14135
- [x] closes #16577
- [x] closes #16925
- [x] closes #17093
- [x] closes #17382
- [x] closes #18050
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them



Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
CI failure looks unrelated
thanks @mroeschke
TST: fixturize, collect
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
That’s the dunder strings

On Sat, Oct 30, 2021 at 4:57 PM Jeff Reback ***@***.***>
wrote:

> ***@***.**** commented on this pull request.
> ------------------------------
>
> In pandas/conftest.py
> <https://github.com/pandas-dev/pandas/pull/44242#discussion_r739734701>:
>
> > @@ -921,6 +921,23 @@ def all_binary_operators(request):
>      return request.param
>
>
> ***@***.***(
>
> isn't this all_compare_operators( ?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/44242#pullrequestreview-793648543>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB5UM6HZOBOP7JW4FAELPJLUJSA6BANCNFSM5HBTEHYQ>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>
CLN/TST: Remove redundant/unnecessary windows/moments tests
State: closed

- [x] closes #37535
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them


* Avoids running similar `def test_*` checks more than once
* Thins out the `consistency_data` pytest fixture to test data with certain properties only once (constant values, all nans, monotonically increasing, etc)

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Window
description: rolling, ewma, expanding

Comments:
great
CLN: de-duplicate Ellipsis-handling
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Clean

Comments:
> pandas/core/arrays/sparse/array.py:886: error: Non-overlapping identity check (left operand type: "Union[Union[Union[int, integer[Any]], Union[slice, List[int], ndarray[Any, Any]]], Tuple[Union[int, ellipsis], ...]]", right operand type: "ellipsis")  [comparison-overlap]


otherwise lgtm.
BUG: setitem into td64/dt64 series/frame with Categorical[strings]
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
this a user facing bug?
> this a user facing bug?

yes, updated with whatsnew
TYP: numba stub
State: closed

partial stub to preserve type signatures for decorated functions.

draft since will rebase once #43828 is merged.

cc @twoertwein @mroeschke 

Labels:
name: Typing
description: type annotations, mypy/pyright type checking
name: numba
description: numba-accelerated operations

Comments:
There are a few more cases of `numba.jit` that have ignored comments. 

pandas/core/window/online.py:    @numba.jit(nopython=nopython, nogil=nogil, parallel=parallel)  # type: ignore[misc]
pandas/core/window/numba_.py:    @numba.jit(nopython=nopython, nogil=nogil, parallel=parallel)  # type: ignore[misc]
pandas/core/window/numba_.py:    @numba.jit(nopython=nopython, nogil=nogil, parallel=parallel)  # type: ignore[misc]
pandas/core/window/numba_.py:    @numba.jit(nopython=nopython, nogil=nogil, parallel=parallel)  # type: ignore[misc]
pandas/core/window/numba_.py:    @numba.jit(nopython=nopython, nogil=nogil, parallel=parallel)  # type: ignore[misc]
pandas/core/_numba/executor.py:    @numba.jit(nopython=nopython, nogil=nogil, parallel=parallel)  # type: ignore[misc]
pandas/core/groupby/numba_.py:    @numba.jit(nopython=nopython, nogil=nogil, parallel=parallel)  # type: ignore[misc]
pandas/core/groupby/numba_.py:    @numba.jit(nopython=nopython, nogil=nogil, parallel=parallel)  # type: ignore[misc]

edit: because of `import_optional_dependency`
@twoertwein merge when ready here. can certainly be followups to fix more stuff.
DOC: local py.typed for users
State: closed

xref https://github.com/pandas-dev/pandas/pull/44223#issuecomment-954672983

Labels:
name: Docs
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Starting with 1.4, it might be useful to slowly start advertising pandas's built-in type annotations (before pandas becomes a py.typed library). Specifically, I can imagine informing the maintainers of https://github.com/microsoft/pylance-release They have often pandas type issues as pylance ships the Microsoft type stubs.

Could maybe even have a note in the whatsnew?
happy to have a section in the whats new and the docs as well
can you merge master
@jreback  green'ish, doc seems to fail on master as well
I don't think it makes sense to advertise experimenting with a local `py.typed` at the moment. People who will test it will be swamped in error messages. I tested it with my own code: many functions return a Union instead of using overloads (`read_csv`, basically all functions taking `inplace`, and indexing with list-like vs. scalars).

I think the earliest it makes sense to advertise experiments with `py.typed` is with 2.0 (when we can make most arguments keyword-only arguments: makes most overloads possible).
thanks @twoertwein this is good.