BrandonFors/pandas-github-issues huggingface dataset:
BUG: pandas.to_datetime raises exception when more than 50 values needs coercion to NaT (#43732)
State: closed

- [x] closes #43732 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
can you write a test
Constructing an Index from a value that will resolve to NaT causes an error, but I'm not sure if this is by design.
```py
import datetime
import pytz
import pandas as pd

# t is a date after 2262 that should be coerced into NaT
t = datetime.datetime(3000, 1, 1, 0, 0, 0, 0, pytz.UTC)
print(pd.Index([t], tz="utc")) # pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 3000-01-01 00:00:00
```
> Constructing an Index from a value that will resolve to NaT causes an error, but I'm not sure if this is by design.
> 
> ```python
> import datetime
> import pytz
> import pandas as pd
> 
> # t is a date after 2262 that should be coerced into NaT
> t = datetime.datetime(3000, 1, 1, 0, 0, 0, 0, pytz.UTC)
> print(pd.Index([t], tz="utc")) # pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 3000-01-01 00:00:00
> ```

this is expected, it is an out of bounds date.
@jbrockmendel we wrote some tests, is there anything else we need to do?
pls merge master, but tests seem to be failing
pls merge master, but tests seem to be failing
is worthwhile but needs to be passing (and pls merge master)
would welcome this change but its stale. pls ping if you want to continue.
[PERF] Improve performace of read_csv with memory_map=True when file encoding is UTF-8
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

This PR improves performance of read_csv with memory_map=True and a UTF-8-encoded file by eliminating unnecessary `decode()` call. The code below demonstrates the speed improvement:
```
import timeit
import pandas as pd

def perftest_readcsv_memmap_utf8():
    lines = []
    for lnum in range(0x20, 0x10080, 0x80):
        line = "".join([chr(c) for c in range(lnum, lnum + 0x80)]) + "\n"
        try:
            line.encode("utf-8")
        except UnicodeEncodeError:
            continue
        lines.append(line)
    df = pd.DataFrame(lines)
    df = pd.concat([df for n in range(1000)], ignore_index=True)
    fname = "test_readcsv_utf8.csv"
    df.to_csv(fname, index=False, header=False, encoding="utf-8")
    ti_rep = 5
    ti_num = 10
    time_dfmem = timeit.repeat(f'dfnomem = pd.read_csv("{fname}", header=None, memory_map=True, engine="c")',
                  setup='import pandas as pd',
                  repeat=ti_rep, number=ti_num)
    print(f"Read CSV (memory_map=True), repeat={ti_rep}: {time_dfmem}")
    print(f"Median: {pd.Series(time_dfmem).median()}")

perftest_readcsv_memmap_utf8()
```
On my machine the results are:
Without patch:
```
Read CSV (memory_map=True), repeat=5: [8.107480439008214, 8.100845866953023, 8.135483622085303, 8.090781628969125, 8.068992758984677]
Median: 8.100845866953023
```
With patch:
```
Read CSV (memory_map=True), repeat=5: [5.280414769076742, 5.290980814024806, 5.127453051973134, 5.150275847059675, 5.276113986037672]
Median: 5.276113986037672
```
The improved code runs in ~65% of time of the current code.
The spedup depends on the contents of the file; the test code above creates a 182 MB file containing almost all of Unicode Plane 0 and a several of Plane 1 characters UTF-8-encoded in 1, 2, 3 and 4 bytes; in this respect, it is a worst case. I also tested this patch on my 8.8 GB CSV file consisting of 1 and 2-byte encoded characters and the code ran in ~75% of time of the current code.


Labels:
name: Performance
description: Memory or execution speed performance
name: Unicode
description: Unicode strings
name: IO CSV
description: read_csv, to_csv

Comments:
@michal-gh yeah if you can run the existing asv's and report here (and you might need to add one to cover this case).

cc @twoertwein 
The last commit above includes asv perf test. On my machine, running asv's io.csv returns:
```
       before           after         ratio
     [bd94bb12]       [52ac1bed]
     <perf-readcsv~2^2^2>       <perf-readcsv>
-      81.6±0.5ms       55.3±0.4ms     0.68  io.csv.ReadCSVMemMapUTF8.time_read_memmapped_utf8

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```
I also corrected the whatsnew file.
thanks @michal-gh very nice!
BUG: Merge with str/StringDtype keys and multiindex
State: closed

- [x] closes #43734
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
thanks @benoit9126 very nice
TYP: enable reportUntypedNamedTuple
State: closed

Prefer `typing.NamedTuple` to `collections.namedtuple`.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Does the change to `NamedAgg` need a whatsnew? I think the other cases are all private.
WDYM? NameddAgg should still be public 
> WDYM? NameddAgg should still be public

It still is public, but its type changed. This change might need a whatsnew
Except the public NamedAgg, the other `collections.namedtuple`s are I think all used only internally. So, they shouldn't need a whatsnew 
cc @simonjayhawkins if ok here
thanks @twoertwein
TST: fixed cython doctests
State: closed

- [x] closes #43757

Skipped some of the tests which were failing

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Docs
name: CI
description: Continuous Integration

Comments:
thanks @MarcoGorelli for reviewing this!

- yes I think that's valid for cython docstrings as well. I have removed the `# doctest: +SKIP` for the line `pandas supports PEP 3141 numbers:` and it's running fine.

- I have tried my best and was able to do the first two examples, but not able to replicate the third one (I don't have much experience working with `tz`).

So currently, only two tests are being skipped, i.e. last example in `tz` and `Int64Factorizer.factorize` in hashtable.pyx

Edit - Another point worth highlighting, is the below is failing on my local (as the output is different than from the CI).
```
??? >>> pd.Timestamp.fromtimestamp(1584199972)
Expected:
    Timestamp('2020-03-14 15:32:52')
Got:
    Timestamp('2020-03-14 21:02:52')
> Edit - Another point worth highlighting, is the below is failing on my local (as the output is different than from the CI). [...] `??? >>> pd.Timestamp.fromtimestamp(1584199972)`

This is going to depend on what timezone you are in locally.
> > Edit - Another point worth highlighting, is the below is failing on my local (as the output is different than from the CI). [...] `??? >>> pd.Timestamp.fromtimestamp(1584199972)`
> 
> This is going to depend on what timezone you are in locally.

would be a good idea to skip this no? (or maybe make it timezone independent if possible, to ensure it doesn't fail in anyones local)
IMO the 
```
In [33]: from dateutil.tz.tz import tzutc as tz

In [34]: tz
Out[34]: dateutil.tz.tz.tzutc

In [35]: from pandas._libs.tslibs.timezones import tz_standardize

In [36]: tz_standardize(tz)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-36-94d2ff882b92> in <module>
----> 1 tz_standardize(tz)

TypeError: Argument 'tz' has incorrect type (expected datetime.tzinfo, got _TzSingleton)
```
test can be removed, if it doesn't run.

Other than that, the rest looks good to me
thanks @debnathshoham 

i suppose that if we had more doc-tests in cython this would pick them up. if someone wants to create an issue about adding more doc-strings wouldn't object (for example there is an older PR that adds a ton to the timestamp functions).
BUG: to_coo silently converting non-zero fill values
State: closed

- [X] closes #24817 
- [X] tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [X] whatsnew entry


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Sparse
description: Sparse Data Type

Comments:
Thanks @saehuihwang! As a potential followup if interested, this issue might be present for the `Series` case too
REF: share _cython_agg_general
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby

Comments:
great
BUG: groupby mean fails for complex
State: closed

- [x] closes #43701
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Groupby
name: Complex
description: Complex Numbers

Comments:
thanks @debnathshoham 

@mzeitlin11 happy to have more consolidation of fused types
CLN: Remove unused functions/variables
State: closed

Found by pyright's `reportUnused*`.

Unused functions/variables are not directly related to typing, but pyright warns about these cases because it is asked to do unnecessary work.

Labels:
name: Clean

Comments:
looks good, some failures
A few of the private functions are only called by tests which caused them to fail. The only failing test now is codecov/patch for some reason.
can you rebase
@jreback rebased & green
thanks @twoertwein did you need to turn on a pyright option to have this reported? e.g. can we have it always run? (or is it expensive)

I didn't turn them one because we have false-positives. We "exclude" pandas/tests - if we were to "ignore" pandas/tests, we probaly wouldn't have these false-positives anymore but pyright might be 30-50s slower if it has to work through the mostly un-typed tests.

The four options from pyright are:

- reportUnusedClass: is only "offended" in one file (pickle compat) where a private class is refernced using only a string. We could disable reportUnusedClass for this file using with a comment at the top of the file.
- reportUnusedFunction: needs a discussion whether these private functions should either be moved (often they are used only one time, but defiend in a different file) or made "public" (at least remove the leading "_")
- reportUnusedImport: could probably be enabled if `__all__` or redundant import were used (main "offenders" are the api.py files)
- reportUnusedVariable: needs consensus to start unused variables with _ (flake8-bugbear has the same rules but it is not enabled because it is "controversial").
TYP: add mypy defaults to pyproject.toml
State: closed

may make it easier for contributors to experiment with enabling mypy strictness flags

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
When we did check_untyped_defs we created an exclude list.

IIRC it takes some maintenance to have an exclude list with more than one flag as IIRC later configs override earlier ones.

Since none of the remaining flags are as important (except maybe the imports resolving to Any now that we have stubs for the cython code) as check_untyped_defs, we have not yet enabled any other flags.

We probably also still have a few "fix later" ignores from enabling check_untyped_defs globally to deal with.

for the untyped decorators, if #43744 solves the @cache_readonly issue, then probably only numba stubs for the numba.jit decorator is needed on that one.

In the past we also discussed using the module level comment syntax to make mypy checking more strict, but IIRC there was preference to keep all mypy config in the global config file.

So if that is still the case, I find implementing a level system like https://github.com/numba/numba/pull/6222/files an interesting alternative. How would something like that fit with pyright type checking.
> In the past we also discussed using the module level comment syntax to make mypy checking more strict, but IIRC there was preference to keep all mypy config in the global config file.

Pyright supports comments in files https://github.com/microsoft/pyright/blob/main/docs/comments.md

> So if that is still the case, I find implementing a level system like https://github.com/numba/numba/pull/6222/files an interesting alternative. How would something like that fit with pyright type checking.

I don't think pyright supports this https://github.com/microsoft/pyright/issues/953

If we enable decorator checking in both mypy and pyright, we could use ignore statements for the numba functions.
> > In the past we also discussed using the module level comment syntax to make mypy checking more strict, but IIRC there was preference to keep all mypy config in the global config file.
> 
> Pyright supports comments in files https://github.com/microsoft/pyright/blob/main/docs/comments.md

The ability in pyright to add `# pyright: strict` and then exclude some checks with say `# pyright: strict, reportPrivateUsage=false` at first glance seems to be easier to maintain and grok than the equivalent mypy syntax of enabling individual checks

In light of this, and it would be good to start to lock down some strictness, I would be happy to see comments added to modules for pyright now that we have pyright checking on CI.
Improved the type stubs in the _libs directory to help with type checking
State: closed

This doesn't change any code, so it should have no impact on runtime behavior.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@erictraut If you don't have time for this PR, I wouldn't mind forking your branch, rebasing it, and making adjustments to get it merged (after #43828 is merged).

Out of curiosity: Did pyright generate the type annotations in this PR or are they from Microsoft's pandas type stubs?
@twoertwein, thanks for offering. Yeah, I'd appreciate it if you'd take over this PR. Addressing its issues involved more complexity than I anticipated.

These type annotations are based on Microsoft's pandas type stubs plus some additional hand-written signatures. They weren't auto-generated by pyright.
pls rebase
> pls rebase

I think this PR could be closed (but please don't delete the fork). I cherry-pick parts of this PR into more manageable portions.
closing as per @twoertwein comment https://github.com/pandas-dev/pandas/pull/43744#issuecomment-996185887
DEPR: Silent dropping of nuisance columns in transform_dict_like
State: closed

Part of #43740

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Apply
description: Apply, Aggregate, Transform, Map
name: Nuisance Columns
description: Identifying/Dropping nuisance columns in reductions, groupby.add, DataFrame.apply

Comments:
@rhshadrach want to make a PR enforcing this deprecation?
Yep, will do.
DEPR: Silent dropping of nuisance columns in agg_list_like
State: closed

Part of #43740 

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Apply
description: Apply, Aggregate, Transform, Map
name: Nuisance Columns
description: Identifying/Dropping nuisance columns in reductions, groupby.add, DataFrame.apply

Comments:
can you rebase
thanks @rhshadrach very nice.
DOC: Added sentence to show that unmatched values return -1.
State: closed

- [x] closes #38482
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew: Added sentence to show that unmatched values return -1 in get_indexer method

In [1]: python make.py --single pandas.Index.get_indexer
Out [1]:![image](https://user-images.githubusercontent.com/50010452/134762263-3be88054-ddfa-4fee-889c-112c00cf780e.png)

In [2]: pre-commit run --files pandas/core/indexes/base.py
Out [2] : ![image](https://user-images.githubusercontent.com/50010452/134768894-a75b8ade-175e-477d-94ff-fefa224f98f9.png)



Labels:
name: Docs
name: Index
description: Related to the Index class or subclasses
name: Stale

Comments:
Hello @DhruvBShetty! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-11-01 12:05:03 UTC
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Hi @DhruvBShetty can you address the comment please
> This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

Interested in finishing this.
> Can you run the [validating docstrings](https://pandas.pydata.org/docs/development/contributing_documentation.html#updating-a-pandas-docstring) script for this function?

Done, It's the first error
![image](https://user-images.githubusercontent.com/50010452/139662439-9a7d1f8c-5d3e-4e0d-8657-1ad0e45eaae5.png)

After which it's this
![image](https://user-images.githubusercontent.com/50010452/139662605-d267c753-630b-426d-b46d-5e7ea5a9ff39.png)

Left a line between %(raises_section)s and notes accidentally.
![image](https://user-images.githubusercontent.com/50010452/139662878-b94ea7d9-aefc-4e61-8362-3045f8c18d81.png)



Thanks @DhruvBShetty
PERF: `Styler.tooltips` 
State: closed

When tooltips were developed there were no hidden_rows or hidden_columns and no max_rows or max_columns.

This adds some benchmarks and a minimal change to improve performance. Scope to do a bit better here in future also.

```
       before           after         ratio
     [cd13e3a4]       [7766f055]
     <master>         <styler_tooltips_perf>
-        52.5±2ms       47.4±0.7ms     0.90  io.style.Render.time_apply_format_hide_render(24, 120)
-       196±0.6ms        173±0.9ms     0.88  io.style.Render.time_tooltips_render(36, 120)
-        78.0±2ms       67.0±0.4ms     0.86  io.style.Render.time_tooltips_render(12, 120)
-         140±2ms        120±0.3ms     0.85  io.style.Render.time_tooltips_render(24, 120)

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```



Labels:
name: Performance
description: Memory or execution speed performance
name: Styler
description: conditional formatting using DataFrame.style

Comments:
thanks @attack68
ENH: new .agg for list-likes
State: closed

Part of #43678

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

Replaces the list-like implementation for `.agg` used by Series, DataFrame, SeriesGroupBy, DataFrameGroupBy, Window, and Resample. This will transpose agg results, and swap multiindex levels as well as the order from within the levels. Adds the new implementation behind the option "new_udf_methods".

Docs: [future_udf_behavior.zip](https://github.com/pandas-dev/pandas/files/7368201/future_udf_behavior.zip)


Labels:
name: Enhancement
name: Stale

Comments:
just a comment - I have always tried to avoid names like "new_routine" because at some point, it won't be, and you wished you never called it that in the first place! :)
@attack68 - The plan is to put all usage of these methods behind the option "new_udf_methods"; when the old are deprecated and then removed, they will be renamed by dropping all the "new_" prefixes. If you have any suggestions for different names, please do make them!
no, no suggestions. But, and I kid you not, in my line of work I have seen `swap_pricer()`, `new_swap_pricer()`, `new_swap_pricer2()` and `new_new_swap_pricer()` - imagine explaining that to new recruits!
Agreed, with @attack68. Helpful to have a slightly clearer naming even during the transition period. Maybe `shared_list_like` if the plan is for `method` to be agg, transform, etc? 
@mroeschke and @attack68 - is there opposition to the "new_udf_methods" option as well, or is it just on the method names themselves? I am not a particular fan of "new" myself, but I could not think of anything concise and better. Regardless of the name that is picked, I would like the relationship between the option name and the methods that are behind that option name to be clear.

Edit: could go with "experimental" and prefix the methods with that?

Edit: "experimental" is a bad choice because there will be a period where the default for this option is True and thus it is no longer experimental.
> I would like the relationship between the option name and the methods that are behind that option name to be clear

+1 I would hope these are consistent.

Maybe `future_udf_behavior` if the idea is that this will be the default some day?
Thanks @mroeschke - I like "future".
I've added docs (built attachment in OP) summarizing the changes here.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
TST: multi indexing when one column exclusively contains NaT(#38025)
State: closed

- [X] closes #38025
- [X] tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: MultiIndex

Comments:
Thanks for the PR @AliSayyah, but seems your test is broken
> Thanks for the PR @AliSayyah, but seems your test is broken

I'll fix it. Thank you.
thanks @AliSayyah
ENH: Add engine="numba" to groupby mean
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Groupby

Comments:
rebase to make sure typing is ok (merged the typing numba PR).
thanks @mroeschke