BrandonFors/pandas-github-issues huggingface dataset:
DEPR: **kwargs in ExcelWriter
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Currently ExcelWriter has **engine_kwargs to pass through to the engine, [although this is undocumented](https://pandas.pydata.org/pandas-docs/dev/reference/api/pandas.ExcelWriter.html). All other places in pandas where engine_kwargs is used, they are a regular argument.

This PR 

 - Renames **engine_kwargs to **kwargs (not an API change)
 - Adds **kwargs to the docstring of ExcelWriter
 - Deprecates **kwargs
 - Introduces engine_kwargs as a regular argument for their replacement
 

Labels:
name: IO Excel
description: read_excel, to_excel
name: Deprecate
description: Functionality to remove in pandas
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
Thanks @jreback - changes made and green. I might not have understood your comment about setting `engine_kwargs = kwargs`, response is above.
thanks @rhshadrach
CLN: Don't modify state in FrameApply.agg
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


Labels:
name: Clean
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
thanks @rhshadrach
CLN/TST: Parametrize/deduplicate test_invalid_arg
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

 - Combined test_series_agg_nested_renamer and test_series_nested_renamer via parametrization
 - test_transform_nested_renamer is already a part of test_nested_renamer
 - test_transform_missing_columns is essentially a duplicate of test_missing_column

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
thanks!
PERF: make Styler default formatter arguments statics instead of repeated dynamic
State: closed

This PR is a minor fix/improvement to a recent PR which deprecated `set_na_rep` and `set_precision`. #40134

For a dataframe with 100,000 floats the current Styler default formatter will make 100,000 `get_option` lookups for the pandas precision.

Doing this calculation once prevents all these lookups and saves just over 10% render time for larger tables.

```
rows                       THIS PR   
------ -------------------------
 cols       12          120     
====== ============ ============
  12    11.8±0.2ms   40.8±0.8ms 
  24    22.5±0.3ms    69.2±2ms  
  36    32.3±0.6ms    96.9±1ms  
====== ============ ============
```

```
rows                        MASTER         
------ -------------------------
 cols       12          120     
====== ============ ============
  12    12.9±0.4ms    43.4±1ms  
  24    22.3±0.1ms   75.2±0.7ms 
  36    32.3±0.3ms   105±0.5ms  
====== ============ ============
```



Labels:
name: Performance
description: Memory or execution speed performance
name: Styler
description: conditional formatting using DataFrame.style

Comments:
thanks @attack68
Fix typing for extension arrays and extension dtypes without isin and astype
State: closed

Same as #39501 by removing changes related to `isin` and the overloads for `astype` and return type for `astype`

Per discussion with @simonjayhawkins  https://github.com/pandas-dev/pandas/pull/39501#issuecomment-798617105



Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@Dr-Irv I think we still need to reduce the size of this PR, so need to start breaking non controversial bits off so we can start merging some of this great work (i.e. changing typvars, the \_\_getitem__ overloads and maybe privatising registry need further input from others) 
> @Dr-Irv I think we still need to reduce the size of this PR, so need to start breaking non controversial bits off so we can start merging some of this great work (i.e. changing typvars, the __getitem__ overloads and maybe privatising registry need further input from others)

OK, next commit will remove those three things, which were changing the `TypeVar` on `DatetimeLikeScalar`, removing the `__getitem__` overloads and the changing of `registry` to `_registry`
@simonjayhawkins I believe I've addressed all your requests on this.  Anything else to do?
Thanks @Dr-Irv for addressing the comments. I want to make another thorough pass before approving.
@simonjayhawkins pinging you again on review of this. Thanks!
@Dr-Irv can you resolve conflicts.
> @Dr-Irv can you resolve conflicts.

@simonjayhawkins I've done this.  Note that with respect to what was in #41097, I changed the name of the `TypeVar` to be `ExtensionDtypeT` rather than `E` to follow convention that we use elsewhere.

Note that the way that I use `pyright` didn't pick that up because the example was from `CategoricalDtype` and I was only verifying the base extension types.

Closing this as I've started making even smaller PRs and tracking status here: https://github.com/pandas-dev/pandas/pull/39501#issuecomment-798922245 in the comment from the original big PR
DOC: Update backticks missing in pandas.DataFrame.query GH40375
State: closed

- [ ] closes #40375 
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

A new suggestion was made to include code styling. 

Beginner's question: What is the procedure when my pull request (in this case #40400) is merged, which closes an issue after which a new suggestion for further improvement is made in the closed issue. Can I just implement the new suggestion and make a new pull request like this? Or is the procedure different?


Labels:
name: Docs
name: expressions
description: pd.eval, query

Comments:
thanks @sstiijn 

thanks for checking the built docs. pls do so again & if any issues, pls let us know.
BUG: Do not attempt to cache unhashable values in to_datetime (#39756)
State: closed

- [x] closes #39756
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
Hello @maroth96! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-03-14 00:53:28 UTC
thanks @maroth96
DEPR: error_bad_lines and warn_bad_lines for read_csv
State: closed

- [x] closes #15122
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Summary of contents
    - Adds new on_bad_lines parameter (I found on_bad_lines a more explicit name than bad_lines)
        - This defaults to None at first in order to preserve compatibility, however it should be changed to error in 2.0 after
        error_bad_lines and warn_bad_lines are removed.
    - Cleanup of some C/Python Parser code ( add enum instead of using 2 variables for C and only use on_bad_lines in Python)

Labels:
name: IO CSV
description: read_csv, to_csv
name: Deprecate
description: Functionality to remove in pandas

Comments:
Hello @lithomas1! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-05-28 00:23:35 UTC
cc @gfyoung if you'd have a look
thanks @lithomas1 this was a bear! thanks for sticking with it
Thanks for the reviews @jreback. Would it be possible to mention this in the deprecations tracker #30228?(I can't edit it myself.)
updated
TYP: fix ignores
State: closed

cc @simonjayhawkins 

Locally when I run mypy on master I get 427 errors mostly "unused 'type: ignore' comment".  Same result when i switch from py39 to py38, or from OSX to Ubuntu.  mypy==0.812 in all cases.  Any guesses what config the CI has different?

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
> Locally when I run mypy on master I get 427 errors mostly "unused 'type: ignore' comment". Same result when i switch from py39 to py38, or from OSX to Ubuntu. mypy==0.812 in all cases. Any guesses what config the CI has different?

can you confirm that you have numpy 1.20.1 in those envs.
> can you confirm that you have numpy 1.20.1 in those envs.

that fixed it, thanks
BUG: Index constructor silently ignoring dtype
State: closed

- [x] closes #21311
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

We can go a step further and re-use sanitize_array for the `dtype is None` case, but that ever-so-slightly changes our datetime-inference behavior, so we'd need to either deprecate or call it a bugfix.

Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
very nice!
CLN: unreachable code in algos.diff
State: closed

it looks like some code is unreachable following #37140 (assuming 1d or 2d only) cc @jbrockmendel 

Labels:
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: Clean

Comments:
nice cleanup, LGTM pending green(ish)
nice, thanks @simonjayhawkins
Use take_1d in take_nd
State: closed

xref https://github.com/pandas-dev/pandas/pull/40300#issuecomment-797637165

Labels:
name: Refactor
description: Internal refactoring of code
name: Stale

Comments:
can remove comment in take_1d about being only for ArrayManager

Update: ill just do this in one of my cleanup branches
test failure is unrelated (and the flaky test is fixed on master).  LGTM
cool, does this allow refactor of other code?
can this be moved out of Draft mode?
@jorisvandenbossche gentle ping
@jorisvandenbossche gentle ping to see if we can move this out of draft mode
@jorisvandenbossche gentle ping
Updated the PR. 
But personally not sure if it is worth it, though:

```python
In [1]: from pandas.core.array_algos.take import take_1d, take_nd

In [2]: dtype = 'int'
   ...: m = 100
   ...: n = 1000

In [3]: values = np.arange(m * n)

In [4]: index = np.arange(m)

# this is not affected, just including as reference
In [5]: %timeit take_1d(values, index)
5.67 µs ± 27 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  <-- master
5.59 µs ± 50.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  <-- PR

In [6]: %timeit take_nd(values, index)
10.5 µs ± 104 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  <-- master
8.86 µs ± 68.3 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  <-- PR
```

So it does give a small improvement, but when increasing the size of the array, the improvement almost completely disappears. 
3 lines for a  15% improvement for a fairly common case seems like a good deal to me
agree this can't hurt (and you know me about not liking special cases)
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Thanks @jorisvandenbossche for the PR. closing as stale. re-open when ready.
BUG: fix convert_dtypes to handle empty df
State: closed

Added a conditional to check a DataFrame had at least one column and if not return the original (empty) DataFrame if so.

- [x] closes #40393
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Thanks @jorisvandenbossche

All good call outs (particularly the test in the wrong place - I should've noticed that). I hope it looks better now. 
Thanks for the feedback @rhshadrach I'm still not sure I've got the assert written the way you'd like - didn't know whether to compare the DataFrame to itself or to a new empty DataFrame. Please let me know if you'd like anything else updated.
Thanks a lot @owenlamont for your first contribution!
Thank you! I'm honoured to contribute to such a foundational part of the PyData ecosystem. First of many hopefully.
DOC: Backticks missing in pandas.DataFrame.query
State: closed

- [ ] closes #40375 
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

I also noticed that in the same sentence double quotes were missing, so I added those as well. 

Labels:
name: Docs

Comments:
thanks @sstiijn 
See https://github.com/pandas-dev/pandas/issues/40375#issuecomment-797882281 for an improvement.
REF/PERF: move np.errstate out of core array_ops up to higher level
State: closed

Currently, we suppress numpy warnings with `np.errstate(all="ignore")` inside the `evaluate` function in `core.computation.expressions`, which is is only used in `core.ops.array_ops._na_arithmetic_op`, which in itself is only used in `core.ops.array_ops` `arithmetic_op()` and `comparison_op()` (where we actually called `np.errstate` again, duplicatively). 

So, in summary, we suppress the warnings at the level of the "array op". For the ArrayManager, we call this array op many times for each column, and repeatedly calling `np.errstate(all="ignore")` gives a big overhead. Luckily, it is easy to suppress the warnings once at a higher level, at the DataFrame/Series level, where those array ops are called.

That's what this PR is doing: removing `np.errstate(all="ignore")` in the actual array ops, and adding it in all places where we currently call the array ops. 

With the benchmark case of an arithmetic op with two dataframes, this gives a considerable improvement:

```python
import numpy as np
import pandas as pd

df = pd.DataFrame(np.random.randn(1000, 1000))
df_am = df._as_manager("array")
```

```
In [2]: %timeit df_am + df_am
18.1 ms ± 1.04 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)   <-- master
8.57 ms ± 167 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  <-- PR
```

For BM it doesn't matter for this case, since that's a single block and `np.errstate` would be called only once anyway. But for certain cases of `df+s` ops where we potentially also work column-wise for BM, it should benefit there as well.

Labels:
name: Performance
description: Memory or execution speed performance
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
This is unfortunate, since it is both a) uglier to do this at a higher level and b) a really nice perf improvement
Is it that much uglier? I removed it in 5 places, and I added it in 4 places in `_dispatch_frame_op` (which in theory could be combined into a single one) and 2 places in Series ops. So it's not that big of a difference.
LGTM, merging this afternoon unless someone chimes in to stop me
thanks @jorisvandenbossche
CI run coverage on multiple builds
State: closed

- [x] closes #39758

Moving the posix builds to GitHubActions + uploading coverage on every build, as they're merged by default by Codecov

----

Trying to get this in so that the codecov check in #40078 will pass

Labels:
name: CI
description: Continuous Integration

Comments:
@jreback Would there be a capacity/credit issue if we run all Linux builds on GitHub Actions?
no we can run on github actions

w/o restrictions 
this looks pretty good. cc @jbrockmendel as i know you look at coverage lot. cc @datapythonista if any comments here.
No objections.  The CI config is pretty mysterious to me.
> Looks nice. Do we have enough builds in actions to not slow the CI results with this?

do you know where this info is?
> > Looks nice. Do we have enough builds in actions to not slow the CI results with this?
> 
> do you know where this info is?

I couldn't find information online, not sure why. But checking the messages I sent to Azure and GitHub, Azure gives 10 parallel workers by default, but we had 30, and GitHub actions gives 20, and I asked them if they could increase it, but afaik that never happened.

I think we balanced things so both azure and actions had the same load, with the 30+20. If this is still the more or less the case, and we move lots of builds from azure to actions, feels like the 20 on actions will be a bottleneck, and we won't use much the 30 in azure.

But not sure if things changed since I had all these discussions.
thanks @MarcoGorelli 

very nice!

@pandas-dev/pandas-core so this moves the linux builds to github actions, if anything looks awry pls open an issue
CI/TYP: Window typing followup
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

I believe https://github.com/pandas-dev/pandas/pull/40293 may have caused the current typing failures in the CI

Labels:
name: Window
description: rolling, ewma, expanding
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks for the follow-up!
PERF: avoid double-verify of take indices
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Performance
description: Memory or execution speed performance
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
Thanks!
TYP: Index.join, get_indexer, reindex
State: closed

@simonjayhawkins this is technically incorrect bc im specifying returned dtypes as np.intp.  Is there a valid way to do this?  In working in core.internals trying to figure out whether we want int64 or intp I had to track all of these down to be sure.  Would prefer to avoid re-doing this in a few weeks when I forget.

Labels:

Comments:
> @simonjayhawkins this is technically incorrect bc im specifying returned dtypes as np.intp. Is there a valid way to do this? In working in core.internals trying to figure out whether we want int64 or intp I had to track all of these down to be sure. Would prefer to avoid re-doing this in a few weeks when I forget.

will need to use comments to retain this info for now because `"ndarray" expects no type arguments, but 1 given` in numpy 1.20.x

I'll start another branch shortly, in the same vien as #36092 checking against numpy master to get visibility of upcoming changes

OK, I guess I'll mothball this to clear the queue, will re-open when these are supported
ENH: Make to_csv('filename.csv.zip') compress the output
State: closed

- [x] closes #39465 #26023
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] dataframe.to_csv  with zip compression now works like gzip/xz, when use 'infer' and destination path 'xxxx.csv.zip' which endwith '.zip', to_csv will create a zip file with archive_name=xxxx.csv

close old PR,do some work after read pre commit guidelines

Labels:
name: IO CSV
description: read_csv, to_csv
name: Stale

Comments:
Hello @HQDragon! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-03-16 00:40:58 UTC
CI check failed is not caused by changed files, it exist in master branch.
It seems checks failed randomly. commit "582f454“ passed all checks ,commit "a555acd" failed, i only changed imports position between these commits.
How to restart partial checks without new commit
@HQDragon can you add a whatsnew note also the precommit checks are failing
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
> Thanks @HQDragon, can you merge master and address the comments please? I think this is close to ready.

@HQDragon, can you merge master and address the comments please?
Appears that this PR has gone stale thought it's close to ready. Going to close for now but let us know if you'd like to continue working on this and we'd be happy to reopen.
Fix frame_or_series.asfreq() dropping rows on unordered indices
State: closed

- [x] closes #39805
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

This prevents `frame_or_series.asfreq()` from dropping rows/items if the index is not sorted.

This seemed like a relatively easy fix, and it should probably be done sooner than later. However, there wasn't much discussion in the issue so I still have some questions:

1. How should we respond to unsorted indices? For now, I went ahead with option a.
  a. Silently "sort" the index before reindexing. Or we could
  b. Raise a warning before "sorting." Or
  c. Raise an error.
2. If it should be in the whatsnew, which version would it be added to?
3. I was thinking of converting the `asfreq` tests from class-based to functions. Can I do that in this PR or should I do that in another one?

Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Frequency
description: DateOffsets

Comments:
also pls add a whatsnew note, 1.3. bug fixes, resample section
@jreback Look good?
@jreback I just changed the test to resemble the previous test in format. Hopefully, that makes its intention more clear?
thanks @nmay231
REF: share Categorical.fillna with NDArrayBackedExtensionArray
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: Categorical
description: Categorical Data Type

Comments:
very nice