BrandonFors/pandas-github-issues huggingface dataset:
CLN: rename init_dict/ndarray to dict/ndarray_to_mgr for consistency
State: closed

See https://github.com/pandas-dev/pandas/pull/39991#discussion_r582632186, in case this is preferred for consistency with `arrays_to_mgr` (I don't have a strong preference myself, since they are only used in `__init__` the current name is also nice)

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
thxs
PERF: EWMA with times
State: closed

- [x] closes #39784
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

```
In [1]: idx=pd.date_range('20000101','20201231',periods=50000)
   ...:
   ...: df=pd.DataFrame(data=range(50000),index=idx)

In [2]: %timeit df.ewm(halflife=pd.Timedelta('100d'),times=df.index).mean()

1.2 ms ± 13.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) <-- PR

10.2 s ± 32.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) <-- master
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Window
description: rolling, ewma, expanding

Comments:
very nice!
This seems to have resulted in a performance regression for the existing ASV benchmark: https://pandas.pydata.org/speed/pandas/#rolling.EWMMethods.time_ewm?python=3.8&Cython=0.29.21&p-constructor='DataFrame'&p-constructor='Series'&p-window=1000&p-dtype='float'&p-method='mean'&commits=ab687aec-f066cda6
opened #42333 to keep track of this regression.
CLN: remove unused arg from _ensure_data
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: Clean

Comments:
great this was always slightly janky
STYLE: inconsistent namespace (io, reshape)
State: closed

- [x] xref #39992
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
@MarcoGorelli this is `io` and `reshape`. First time contributing to pandas so let me know if there are any style things I should change - I read over the dev docs and think I've got everything.

Thanks!
can you merge master
Rebased & pushed
Hi @AA-Turner - `reshape` is already covered by #40058, could you revert those changes and just keep this to `io` please?
Thanks Marco, mutst've missed that when I was looking for what had been done already. Will do as you ask
Done
Thanks @AA-Turner !
Looks like this PR had some failure - seeing them on master as well now. Are these related?
Hi Richard - will look into this. ~~Given this just changes namespace usage, may be that `pandas/__init__.py` exposes names or something that aren't in submodules.~~

Edit: found the cause, see below

Given a lot of tests/CI checks failed - is there a known subset that are ingored? I'll look for `io` related tests in the logs first, though.
Found three failures I think are related - all in `test_stata.py`. The changes moved to using `io.stata.blah` (stripping the leading `pd.`), which shadows the `io` builtin, and I didn't catch when looking over.

PR to fix is at #40094

Adam
BUG: construct_1d_ndarray_preserving_na with dt64/td64 and object dtype
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
does this surface to user facing?
> does this surface to user facing?

not AFAICT.  same as the maybe_convert_objects PR, this surfaces when trying to simplify our casting code
BUG: maybe_convert_objects with timedelta64 nat
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Timedelta
description: Timedelta data type

Comments:
user facing at all?
> user facing at all?

not AFAICT.  discovered while trying to simplify our casting code by passing convert_timedelta/convert_datetime=True and skipping some of the casting calls.
TYP: incorrect annotations in internals.construction
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
cc @simonjayhawkins if you have a chance to look
> any perf difference? (e.g. wrapping with Index / ensure_index) a few times

i dont think so; the Index wrapping would have to happen eventually anyway.  if anything, early consistent wrapping should make it easier to avoid re-checking later
BUG/ENH: `Styler.format()` always inherits from `.set_na_rep()`
State: closed

- [X] closes #40032 
- [X] tests added: `test_display_format_subset_interaction`: why: see below.
- [X] tests removed: `test_format_bad_na_rep`: why: `na_rep` works if not string in some cases - changed to duck-typing.
- [X] tests lefts unchanged that pass: `test_display_format`, `test_display_set_precision`, `test_format_subset`, `test_display_dict`
- [X] tests expanded: `test_display_format_raises`
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

The issue is that there are currently two competing methods in `Styler` for missing value represenatation: 

  - `.set_na_rep('x')` and
  -  `.format(na_rep='y')`. 

The two do not interact well together dynamically. This PR brings the `format wrapping` into the class so that `format` can be updated and inherit values in `set_na_rep` after it has been called. The image below highlights the impact this PR has and the added test replicates this pattern:

![Screen Shot 2021-02-25 at 20 38 45](https://user-images.githubusercontent.com/24256554/109210160-bfbf5600-77ac-11eb-8be7-8938d80f4ce6.png)

Also at the bottom here, we add a `KeyError` to remove the ambiguity between what happens when a dict is provided with a column Key that is not specifically included within the provided `subset` slice.


Labels:
name: Code Style
description: Code style, linting, code_checks
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
> .set_na_rep('x') and
.format(na_rep='y').

if there are 2 let's deprecate one. we use `.format()` everywhere else, I don't really see having a specific method na rep as all that useful (we *also* have set_precision as well).

we need one canoncial way to do things.
try not to mix bug fixes with api changes in a single pr. keep them as focused as possible.
@jreback I agree somewhat that maybe multiple different `set_na_rep` and `set_precision` and `format` methods are not ideal and maybe incorporating everything to a format function is preferable. I've never suggested a deprecation previously (and all of these are legacy decisions from my perspective) but am happy to work on a solution otherwise, if you think thats better.
I think the PR above is better both in terms of API and code maintenance. Do close this PR if you agree.
STYLE: Inconsistent namespace - reshape (#39992)
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
can you merge master
merging as test failures are unrelated (they're in `pandas/tests/extensions`), thanks @alexprincel
STYLE: Inconsistent namespace - groupby (#39992)
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Nice, thanks @alexprincel ! I'll merge this and the other one you opened - shall we leave the other folder to other new contributors?
> Nice, thanks @alexprincel ! I'll merge this and the other one you opened - shall we leave the other folder to other new contributors?

Yes, of course. I'll try to pick another issue or enhancement that I could tackle.
BUG: revert parts of #40037
State: closed

- [x] closes #40048
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@jreback this gets us back to green
[ArrayManager] Remaining GroupBy tests (fix count, pass on libreduction for now)
State: closed

Follow-up on #39885 and #40047

This passes / skips all groupby tests (in `/tests/groupby`). Actual code changes:

- fix `groupby().count()` to not create a ArrayManager with 2D arrays
- for now skip the usage of `fast_apply` / `libreduction.apply_frame_axis0` for DataFrames with ArrayManager. This can be tackled separately, while for now already getting more test coverage by using the python fallback (as we eg also do for EAs).
- implement `ignore_failures` for `ArrayManager.grouped_reduction` (this keyword is actually always True at the moment, so we could also leave away the keyword itself)

Most of the skips are related `quantile` (or `describe`) not yet being implemented. And it also uncovered a few other failures, that can be worked on as follow-ups.


Labels:
name: Groupby
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
> quantile

good reminder.  ive got a terminal window open i need to revisit...
thanks @jorisvandenbossche
VIS Wrong legend when combining multiple plots on the same axes , plotting with `legend=False` messes up legend 
State: closed

- [x] closes #39522
- [x] closes #40044
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

So, in #27808 , `leg.legendHandles` was replaced with `ax.get_legend_handles_labels()[0]` in order to preserve marker info.

However, `ax.get_legend_handles_labels()[0]` may contain extra elements which aren't (yet) visible, which means we may end up at 

https://github.com/pandas-dev/pandas/blob/272435073c70b4421f72184663837f4864601f11/pandas/plotting/_matplotlib/core.py#L625

with
```python
(Pdb) handles
[<matplotlib.lines.Line2D object at 0x7f0938a37430>, <ErrorbarContainer object of 3 artists>, <matplotlib.lines.Line2D object at 0x7f0938a37430>]
(Pdb) labels
['orange', 'blue']
```

This causes the permuting noted in #39522

Ideally, there would be a way to get `leg.legendHandles` in a way that preserves the marker info. However, I can't find a way to do that - have asked on the matplotlib discourse. Also, the current test doesn't quite work (as it passes on `master` too). I'm working on finding a satisfactory solution

---

On master:

![image](https://user-images.githubusercontent.com/33491632/109154213-e4440f80-7765-11eb-8c76-7d70ccddf447.png)

On this branch:

![image](https://user-images.githubusercontent.com/33491632/109154268-f8880c80-7765-11eb-839f-e239a909b8fe.png)

---

cc @charlesdong1991 in case you have comments / suggestions

Labels:

Comments:
closing as I'm not happy with this solution - will get back to it
BUG: Sampling over selected groupbys does not reflect the selection
State: closed

- [x] closes #39928
- [x] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Groupby
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
thanks @wjsi
CI: troubleshoot OOB timedelta test
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Timedelta
description: Timedelta data type

Comments:
@jreback looks like this gets us back to green
great, weird that the f-string wouldn't string-ify the same, but I guess it does repr?
must be something either in cython or numpy?
FYI, you can use `f'{value!s}'` to call str() instead of repr()
PERF: make Categorical _ndarray, attribute, _codes property
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Categorical
description: Categorical Data Type

Comments:
How does this rename improve performance?
Ah, I see your comment in https://github.com/pandas-dev/pandas/pull/40007
STYLE: Inconsistent namespace - base (pandas-dev#39992)
State: closed

- [x] xref #39992
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @punitvara - I'll merge this one, but perhaps shall we leave some other folders for other first-time contributors?
Sure @MarcoGorelli. No Problem. I will not submit  new one. Is there any other issue I can look into ? Any recommendations ?
> Any recommendations ?

My general advice is to look for an issue which you understand and which makes you think "wow, I can't believe this is wrong, someone needs to fix it"

If you know matplotlib well, then there's lots of visualisation issues which aren't getting much attention
@MarcoGorelli Thanks. Appreciate your pointers.
REF: share common code in ArrayManager.get_numeric_data/get_bool_data
State: closed

See https://github.com/pandas-dev/pandas/pull/39719/files#r578859353 @jreback

Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Merging, so I can update https://github.com/pandas-dev/pandas/pull/39719/ to use it (failures are unrelated)
STYLE: Inconsistent namespace - apply (pandas-dev#39992)
State: closed

Fix inconsistent namespace in apply directory.

- [x] xref #39992
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
@MarcoGorelli  Sure. I have reverted it back. Thanks for guiding me. Hope this looks fine now.
> Hope this looks fine now.

You still need to fixup the error from the `black` formatter

```
pre-commit run --files pandas/tests/apply/test_frame_apply.py pandas/tests/apply/test_series_apply.py
```

---

Or just comment `@github-actions pre-commit` and that'll trigger a workflow which'll fix it up for you
Done.
@MarcoGorelli go ahead and merge PRs like this that are all green-ish
sure, thanks - flake8 and the rest pass, so this should be good to go
REF: catch less in maybe_cast_to_datetime
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
mothballing while i double-check something
REF: extract_array earlier in block construction
State: closed

The idea is to eventually get the extract_array out of maybe_coerce_values which will improve `Block.__init__` perf.

The main remaining blocker to that is downstream usage.  @jorisvandenbossche i know pyarrow accesses some of core.internals, do you know if it is just `make_block` or is anything else accessed?

Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
> The main remaining blocker to that is downstream usage. @jorisvandenbossche i know pyarrow accesses some of core.internals, do you know if it is just make_block or is anything else accessed?

I think it is mainly `make_block` and the `BlockManager(..)` constructor.
CI/TST: Supply dtype to coo_matrix until scipy is fixed
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Fixes CI for Windows 3.7 np1.6; regression in scipy 1.6.1 (https://github.com/scipy/scipy/issues/13585)

Alternative would be to pin build, wasn't sure which is prefered over the other. Will open tracking issue once the method is decided.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
going to backport as well, ping on green.
Windows 3.7 np1.6 was successful, failures look unrelated. Running again.
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
@jreback - greenish

Failing test is `TestTimedeltas.test_to_timedelta_oob_non_nano`, I'm seeing this fail on the most recent commit to master as well. 
> @jreback - greenish
> 
> Failing test is `TestTimedeltas.test_to_timedelta_oob_non_nano`, I'm seeing this fail on the most recent commit to master as well.

oh that's prob something we just merged cc @jbrockmendel 
thanks @rhshadrach 
@meeseeksdev backport 1.2.x
> test_to_timedelta_oob_non_nano

will take a look
BUG: Cannot sample on DataFrameGroupBy with weights when index is specified
State: closed

- [x] closes #39927
- [x] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Groupby

Comments:
Hello @wjsi! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-02-24 16:53:56 UTC
Also need a note in the 1.3.0 whatsnew under GroupBy Bug Fixes 
Thanks @wjsi!
REGR: compressed to_json with URL-like paths and binary objects
State: closed

- [ ] closes #39985
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: IO Network
description: Local or Cloud (AWS, GCS, etc.) IO Issues
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
thanks @twoertwein very nice as always, keep em coming!
@meeseeksdev backport 1.2.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 bd9a70264b2b990311911930e3fa27873a51bfc6
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #40010: REGR: compressed to_json with URL-like paths and binary objects'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-40010-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #40010 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Yes, thank you for the quick turnaround on this.
BUG: overflow in astype(td64ns)
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Bug
name: Timedelta
description: Timedelta data type

Comments:
seem to remember a bug report about this, but could be imagintion
This results in a 5.22x slowdown in one of our benchmarks

https://simonjayhawkins.github.io/fantastic-dollop/#inference.ToTimedelta.time_convert_int?branch=1.2.x&branch=1.3.x&commits=b7618487
> This results in a 5.22x slowdown in one of our benchmarks

Yah AFAICT this is unavoidable if we want to fix the bug.  https://github.com/numpy/numpy/issues/16352 is about fixing the astype upstream.

BTW looking at the regressions listed, I think we can hit "ignore" for the TimestampProperties.time_freqstr ones, since those are driven by a deprecation of freqstr (and are still measured in ns)
> > lts in a 5.22x slowdown in one of our benchmarks
> 
> Yah AFAICT this is unavoidable if we want to fix the bug. [numpy/numpy#16352](https://github.com/numpy/numpy/issues/16352) is about fixing the astype upstream.

ok. have ignored https://simonjayhawkins.github.io/fantastic-dollop/#regressions?sort=3&dir=desc&branch=1.3.x
BUG: rolling not calculating window bounds correctly with offset and desc dates
State: closed

- [x] closes #40002
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Window
description: rolling, ewma, expanding

Comments:
Thanks @phofl. CI failure was unrelated
POC: dynamically choose which asvs to run
State: closed

The main idea is in trim_asv_paths.  Ideally it would be expanded to allow finer-tuning (e.g. #39917 would allow some more)


Labels:
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
> Any chance you've run this against currently open PRs?

no i havent.  i guess could experiment with it by making a GH action to display included or excluded files