BrandonFors/pandas-github-issues huggingface dataset:
REF: Consolidate validation of dictionary argument in agg/transform
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Currently there are no tests for dict-like argument with agg when a DataFrame is missing a column (edit: in the apply tests, there are tests in groupby, resample, and window). This PR changes the error slightly from

> f"Column '{k}' does not exist!"

to

> f"Column(s) {cols} do not exist"

in order to make it consistent (and I think more helpful, in the case of multiple columns) with the error message from transform.

Labels:
name: Refactor
description: Internal refactoring of code
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
@jreback - added test_invalid_arg to start reorganizing the apply tests to something with a better layout. If this seems like a good idea, will move other relevant tests here in a followup.
great, this is a very small api change (e.g. now get SpecificationError rather than KeyError in some cases), can you add a whatsnew note. ping on green.
@jreback Ah - thanks for catching this. Before adding the note - is it clear that we want a SpecificationError here and not KeyError? Looking throughout pandas.core, I only see SpecificationError used in three places:

 - groupby.generic._aggregate_multiple_funcs
 - aggregation.reconstruct_func  - only used by DataFrame.agg and DataFrameGroupBy.agg
 - apply for transform (and with this PR, agg too)

In each case, I believe either KeyError or ValueError is also appropriate. It seems to me maybe changing to KeyError for transform here would be better, and removing SpecificationError altogether in the future. This would also having the benefit of a smaller API change (transform is only used by Series/DataFrame here, whereas agg is used by GroupBy/Resampler/Rolling).


very nice!
STYLE: Inconsistent namespace - Indexes (#39992)
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [X] whatsnew entry

Don't think tests apply here. By itself, does not close 39992.

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Hi @alexprincel - no need for a whatsnewnote - other than that and the merge conflicts, this looks good, thanks
Thanks @alexprincel for updating - there's still a conflict, could you resolve it please?
> Thanks @alexprincel for updating - there's still a conflict, could you resolve it please?

Hello, yes will do. Could not finish yesterday but it's fixed on my local environment now. Still learning how to use git properly :)
No worries, I'd recommend the first 3 chapters of the [Pro Git](https://git-scm.com/book/en/v2) book to learn the common commands
CLN: follow-ups
State: closed


Labels:
name: Clean

Comments:
whoo hoo issue 40000!
looks like the test thats failing here is failing everywhere
> pandas/_libs/lib.html

got commited :_>
> looks like the test thats failing here is failing everywhere

can you xfail / fix ?
> got commited :_>

woops, will revert

> can you xfail / fix ?

was planning to try to fix it in a bit
> was planning to try to fix it in a bit

sure
DOC: fix obsolete URL for Vega
State: closed

https://github.com/trifacta/vega -> https://github.com/vega/vega

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Thanks
Remove numpy pin from environment.yml on 1.2.x
State: closed

xref #39513, expected to fail until #39994 backported

This PR is directly against 1.2.x and ignores site packages for mypy checks.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
can remove blocker label from #39513 on merge, https://github.com/pandas-dev/pandas/issues/39513#issuecomment-771006563
> expected to fail until #39994 backported

the problematic benchmark is only on master
lgtm. @simonjayhawkins merge when ready
REF: move Block construction in groupby aggregation to internals
State: closed

Possible precursor for https://github.com/pandas-dev/pandas/pull/39885

Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
For the current BlockManager code, this doesn't actually simplify that much in the groupby code (it *does* remove accessing the blocks and an explicit BlockManager(..) call, so removing BlockManager internals). 

In principle I could re-use `BlockManager.apply` inside `BlockManager.grouped_reduce`, but I don't find it very clean to use `apply` for something for something that doesn't generate a same-shape BlockManager (it basically now creates an invalid BlockManager object because the length of the index is not correct).

But it will probably make it a bit cleaner to add ArrayManager support. 
lgtm. some failing tests :->
fix benchmark failure with numpy 1.20+
State: closed

cherry-pick from #36092 in case we don't backport it as this change needs to be backported, follow-up to #39795

Labels:
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
> cherry-pick from #36092 in case we don't backport it as this change needs to be backported, follow-up to #39795

only on master. see https://github.com/pandas-dev/pandas/pull/39842#issuecomment-779915656
thanks @simonjayhawkins
[ArrayManager] DataFrame constructors
State: closed

xref https://github.com/pandas-dev/pandas/issues/39146

In this PR, I am trying to ensure what when we construct a DataFrame, we directly construct the proper manager (depending on the option settings) instead of converting at the end of the constructor (as is done now). To that end, I am passing through a keyword to indicate the type of the manager to be created to the different internal construction methods like `arrays_to_mgr` etc.

Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
@jbrockmendel other comments?
lgtm cc @jreback
@jreback more answers / comments here?
All good here? (this is blocking a few other PRs that are mostly ready: https://github.com/pandas-dev/pandas/pull/39700, https://github.com/pandas-dev/pandas/pull/40050)
Moving the datetime wrapping into the ArrayManager constructor revealed some other issues, so first doing a precursor for this here -> https://github.com/pandas-dev/pandas/pull/40147
https://github.com/pandas-dev/pandas/pull/40147 is merged, so updated this (I think the only remaining discussion item was about where the datetime wrapping happened, which is resolved by https://github.com/pandas-dev/pandas/pull/40147)
sort_index not sorting when multi-index made by different categorical types
State: closed

- [ ] closes #24271
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex

Comments:
Hello @ftrihardjo! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-03-13 05:02:37 UTC
Hi @ftrihardjo - could you put "closes #24271" in the pull request description and put a descriptive title (e.g. the name of the issue) please?
thanks @ftrihardjo
Backport PR #39944 on branch 1.2.x (Fix regression for setitem not aligning rhs with boolean indexer)
State: closed

Backport PR #39944: Fix regression for setitem not aligning rhs with boolean indexer

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
> Run source activate pandas-dev
mypy --version
mypy 0.782
Performing static analysis using mypy
pandas/tests/frame/indexing/test_setitem.py:408: error: Module has no attribute "setitem"  [attr-defined]
pandas/tests/frame/indexing/test_setitem.py:408: error: Module has no attribute "loc"  [attr-defined]
Found 2 errors in 1 file (checked 1129 source files)
Performing static analysis using mypy DONE
Error: Process completed with exit code 1.


@simonjayhawkins what do you want do here :-<
DOC: fix template substitution in pad/bfill docstrings
State: closed

- [ ] closes #39640
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

#39640 Added doc tag above ffill() to substitute klass

Labels:
name: Docs
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
cc @simonjayhawkins @jorisvandenbossche 

lgtm.
yeah I can do that. I'm not sure what "Igtm" or "L6516" mean/referring to, if I need to know then could someone let me know please. 


@dylanp-git lgtm means 'looks good to me'; L6516 is line number 6516

> @dylanp-git lgtm means 'looks good to me'; L6516 is line number 6516

aha thankyou, I'm slowly learning the lingo!!
Thanks a lot @dylanp-git !
> Thanks a lot @dylanp-git !

No worries! I was going to look into it again later as only 20/22 checks passed?

On another note, how many "good first issue" issues do you think I should do before trying something a bit more in-depth?
> only 20/22 checks passed?

the failures were unrelated (sometimes that happens)

> how many "good first issue" issues do you think I should do before trying something a bit more in-depth?

IMO if you've got familiar with the contribution workflow then you could start immediately
BUG: Incomplete Styler copy methods fix (#39708)
State: closed

- [x] closes #39708
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

I haven't made so many pull requests before, so I hope this is going into the right direction. 

Labels:
name: Styler
description: conditional formatting using DataFrame.style
name: Copy / view semantics

Comments:
When committing my changes based on your comments, do I always have to add 'BUG: ... (#39708)' to my commit messages? Or is it okay if I just write something like 'Mention issue number in enhancements section'?

I prefer atomic commits, but do all my commits end up upstream after the PR is merged? Shall I do only one single commit for all my changes instead?
> When committing my changes based on your comments, do I always have to add 'BUG: ... (#39708)' to my commit messages? Or is it okay if I just write something like 'Mention issue number in enhancements section'?
> 
> I prefer atomic commits, but do all my commits end up upstream after the PR is merged? Shall I do only one single commit for all my changes instead?

when merged to master will be squashed to one commit. do as many commits as you want, when you need a new review generally ping for one, wait patiently, or request here. :)

> when merged to master will be squashed to one commit. do as many commits as you want, when you need a new review generally ping for one, wait patiently, or request here. :)

Thanks, and good to know!

Hi, here's my next try! 
thanks @rlukevie
update pyperclip link
State: closed

looks like the link is outdated. updated it.

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
thanks @rohitgr7 
@meeseeksdev backport 1.2.x
Something went wrong ... Please have  a look at my logs.
DOC: fix a mis-formatted external link
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
thanks @partev 
@meeseeksdev backport 1.2.x
Something went wrong ... Please have  a look at my logs.
PERF: halve render time in Styler with itertuples
State: closed

```
 before           after         ratio
     [dfb92d0f]       [e5c2790b]
     <master>         <perf_itertups>
-         149±1ms         74.4±2ms     0.50  io.style.RenderApply.time_render(24, 120)
-        31.9±1ms       15.9±0.8ms     0.50  io.style.RenderApply.time_render(24, 12)
-       86.3±20ms         42.3±1ms     0.49  io.style.RenderApply.time_render(12, 120)
-      12.2±0.2ms       5.93±0.2ms     0.49  io.style.RenderApply.time_render(12, 12)
-      44.8±0.5ms       21.7±0.3ms     0.48  io.style.RenderApply.time_render(36, 12)
-         227±2ms         99.9±1ms     0.44  io.style.RenderApply.time_render(36, 120)

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```

FYI `Styler` now faster than `to_html`, even without any function stripping...

```
df = pd.DataFrame(np.random.rand(10000,10))
%timeit df.to_html()
1.9 s ± 63.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit df.style.render()
1.71 s ± 14.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Styler
description: conditional formatting using DataFrame.style

Comments:
great. Can you add a whatsnew note (IIRC you may already have one about perf rendering, if so just add this on). ping on greenish.
@jreback ping greenish. `test_from_coo` failure is unrelated.
thanks @attack68
DOC: fix a warning using obsolete package
State: closed

currently there is a warning:
"/home/runner/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/IPython/html.py:12: ShimWarning: The `IPython.html` package has been deprecated since IPython 4.0. You should import from `notebook` instead. `IPython.html.widgets` has moved to `ipywidgets`.
  warn("The `IPython.html` package has been deprecated since IPython 4.0. ""

this change fixes the warning.

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
thanks @partev
CI: conditional xfail flaky test
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
ok can you open an issue for this (and reference it in the comments)
we have #38630 for this
ENH: NDArrayBackedExtensionArray swapaxes, delete-with-axis
State: closed

Sits on top of #39880


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
can you rebase and will look
rebased
hmm looks like master is failing on this:
I guess need to pin for 3.7 the dateutil version? 

```
pytest -m "not slow and not network" -n auto --dist=loadfile -s --strict-markers --durations=30 --junitxml=test-data.xml pandas
ImportError while loading conftest '/Users/runner/work/1/s/pandas/conftest.py'.
pandas/__init__.py:51: in <module>
    from pandas.core.api import (
pandas/core/api.py:24: in <module>
    from pandas.core.algorithms import (
pandas/core/algorithms.py:39: in <module>
    from pandas.core.dtypes.cast import (
pandas/core/dtypes/cast.py:29: in <module>
    from dateutil.parser import ParserError
E   ImportError: cannot import name 'ParserError' from 'dateutil.parser' (/Users/runner/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/dateutil/parser/__init__.py)
```
CLN: Remove some duplicated tests and parametrize
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


There were a few duplicates from the ix removal

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Clean

Comments:
couple comments, ping on green
ping @jbrockmendel green
thanks @phofl
DOC: remove unnecessary index.html from the URL
State: closed

remove unnecessary index.html from the URL

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
thanks @partev
REF: Move transform into apply
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Cleanups and consolidations to follow.

Labels:
name: Refactor
description: Internal refactoring of code
name: Apply
description: Apply, Aggregate, Transform, Map
name: DataFrame
description: DataFrame data structure
name: Series
description: Series data structure

Comments:
thanks @rhshadrach 

let the deduplication begin!
REF: Move the rest of DataFrame.agg into apply
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run 

Labels:
name: Refactor
description: Internal refactoring of code
name: Apply
description: Apply, Aggregate, Transform, Map
name: DataFrame
description: DataFrame data structure

Comments:
very nice
REF: Move the rest of Series.agg into apply
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Labels:
name: Refactor
description: Internal refactoring of code
name: Apply
description: Apply, Aggregate, Transform, Map
name: Series
description: Series data structure

Comments:
thanks @rhshadrach
PERF: Styler render improvement by limiting .join()
State: closed

Currently 
`<th>` only have attributes when there is a row or column span.
`<td>` never have attributes currently.
So there is no need for repititive `join()` operations, especially inside the jinja2 template. Could be preconfigured outside.

We can get around 10-15% better render times.

```
[ 75.00%] ··· io.style.RenderApply.time_render                                                                                                           ok
[ 75.00%] ··· ====== ============ ============
              --                rows          
              ------ -------------------------
               cols       12          120     
              ====== ============ ============
                12    11.6±0.5ms   77.0±0.5ms 
                24    29.2±0.5ms    144±1ms   
                36    43.7±0.9ms    209±2ms   
              ====== ============ ============

```
versus master
```
[100.00%] ··· io.style.RenderApply.time_render                                                                                                           ok
[100.00%] ··· ====== ============ ==========
              --               rows         
              ------ -----------------------
               cols       12         120    
              ====== ============ ==========
                12    12.9±0.4ms   89.8±4ms 
                24    31.4±0.9ms   166±2ms  
                36     46.5±2ms    250±40ms 
              ====== ============ ==========
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Styler
description: conditional formatting using DataFrame.style

Comments:
thanks @attack68
TYP: hashing
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
lgtm. @simonjayhawkins if you have a chance.
if theres no interest here we can close it to clear the queue
> if theres no interest here we can close it to clear the queue

its fine, this is worth merging.
REF: implement array_algos.take
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
if you can rebase
rebased + green
maybe just more all of algorithms.py to array_algos? though i think these should probably be in somewhat separated modules rather than one giant one. so ok with moveing take but let's put it separate as its pretty meaty by itself.
> maybe just more all of algorithms.py to array_algos?

there are some that depend on Index/Series, so they dont all fit
k thanks
TYP: stubfile for _libs.lib
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

cc @simonjayhawkins @Dr-Irv there are a couple of kludges (ndarray_obj_2d, const memoryviews), interested in your thoughts on how to handle these

Labels:

Comments:
Some general comments:

1. Use `List[Any]` rather than `list`  (or `List[float]` if it's a list of floats, etc.)
2. Consider using `Any` rather than `object`
3. With respect to `numpy`, does this work with both numpy 1.19 and numpy 1.20 (which has typing) ?  I'm not sure how you test that...
4. You might consider creating in `pandas._typing` aliases for the different numpy types (e.g., `np.ndarray`, `np.ndarray[object]` ).  And maybe be consistent in the use of those, since there are places where you use the former, and other places where you use the latter.  Creating our own types for that might make it easier to support different versions of numpy as their typing stuff evolves.

I haven't done enough with `numpy` typing to know how to handle `ndarray_obj_2d` .