BrandonFors/pandas-github-issues huggingface dataset:
Change in astype leads to no longer casting to numpy string dtypes
State: closed

- [x] closes #39566
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

@jreback as discussed we no longer cast to numpy string dtypes after this

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Strings
description: String extension data type and string data
name: Stale

Comments:
I am not familiar with the experimental data manager. What could be the reason that the tests fail there but not on azure? Additionally only the DataFrame tests seem to fail, not the series tests
> What could be the reason that the tests fail there but not on azure?

they only are run if you specify add `--array-manger` to the pytest command, so only get run in that one build
Thx, this explains the behavior and the not raising in series methods.

@jorisvandenbossche You added a TODO about the numpy arrays in there. Did you have any concrete plans handling this case?
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@phofl can you merge master to resolve conflicts and address comments
@phofl closing as stale. reopen when ready.
Yes will reopen when fixed, not much time right now unfortunately
Fix regression for setitem not aligning rhs with boolean indexer
State: closed

- [x] closes #39931
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

reindexing solves the issue here before dispatching to iloc

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
> reindexing solves the issue here before dispatching to iloc

maybe add words to that effect as a comment in the code

test failure unreleated. restarted job
Added something
thanks @phofl 
@meeseeksdev backport 1.2.x
BUG: Groupby ops on empty objects loses index, columns, dtypes
State: closed

- [x] closes #26411
- [x] closes #13483
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Groupby
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
@jreback - thanks, whatsnew and test have been updated.
REF: share _maybe_coerce_values
State: closed

Gets us within spittting distance of combining DTBlock/TDBlock (and not far from DT64TZBlock too)

Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
can you rebase
rebased + green
sorry one more time
Rebased + green
ENH: Add function to create DateOffset objects from offset aliases
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:

Comments:
Hello @Pawel-Kranzberg! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-02-20 17:34:15 UTC
Is this PR in reference to an existing issue?

There is already https://pandas.pydata.org/docs/reference/api/pandas.tseries.frequencies.to_offset.html
> Is this PR in reference to an existing issue?
> 
> There is already https://pandas.pydata.org/docs/reference/api/pandas.tseries.frequencies.to_offset.html

My bad.
BUG: DataFrame.agg and apply with 'size' returns a scalar
State: closed

- [x] closes #39934
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Labels:
name: Bug
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
@jreback - I merged master.

> It might be better to detect this argument 'size' and do a translation in 
```pandas/core/base.py:    _builtin_table```

This isn't a builtin; `DataFrame.size` reports the number of cells in the frame (rows * columns). Also not sure what we'd map it to, count is similar and operates on rows/columns but drops nans whereas size does not.
my point is to map this to a lambda function to avoid special casing like this
@jreback - makes sense, and I agree your suggestion is the prefered way. The one issue is that when replacing `func` with a lambda, the resulting Series has name None whereas tests currently expect "size". I see two ways to handle (hopefully not overlooking something simple):

 1. Mark as TODO and tackle when consolidating code for wrapping results
 2. Add a "result_name" attribute the the Apply class, set to None in all cases except this one where it is "size", and apply it in various methods where appropriate.

It seems easier and less error-prone to go with 1, but I can do 2 here if you prefer.
> @jreback - makes sense, and I agree your suggestion is the prefered way. The one issue is that when replacing `func` with a lambda, the resulting Series has name None whereas tests currently expect "size". I see two ways to handle (hopefully not overlooking something simple):
> 
> 1. Mark as TODO and tackle when consolidating code for wrapping results
> 2. Add a "result_name" attribute the the Apply class, set to None in all cases except this one where it is "size", and apply it in various methods where appropriate.
> 
> It seems easier and less error-prone to go with 1, but I can do 2 here if you prefer.

yeah happy to do this in the future. 2 sounds interesting actually. why don't you add a TODO for now (and issue) then can merge this one.
@jreback TODO added, issue opened, and green(ish - failure is unrelated).
can you merge master & ping on greenish
thanks @rhshadrach
Grammatically update the readme file
State: closed

Small grammatical update the readme file 

Labels:

Comments:
Thanks @mohamed-khoualed - some of these aren't correct (e.g. "Here are just a few of the things that pandas do well", the current phrasing with "does" is fine as it is), I'm not sure this is an improvement, and some of the linting checks are broken

Instead of trying to change the README, why not try tackling on of the many [good first issue](https://github.com/pandas-dev/pandas/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)s? See [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html) for how to contribute, and feel free to reach out on Gitter if you have trouble getting set up
DOC: updated wrong type hint for pandas.Series.droplevel
State: closed

- [x] closes #39907 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
thanks @Muktan
Updated read_html to add option
State: closed

Adds optional boolean parameter "remove_whitespace" to skip the remove_whitespace functionality. Defaults to true to support backwards compatibility. See https://github.com/pandas-dev/pandas/issues/24766

- [ x ] closes #24766 
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Enhancement
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap
name: Stale

Comments:
Hello @Derekt2! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/io/html.py`](https://github.com/pandas-dev/pandas/blob/3d5f48df7aba938afaa4409ad8402cb829a11775/pandas/io/html.py):

> [Line 175:1](https://github.com/pandas-dev/pandas/blob/3d5f48df7aba938afaa4409ad8402cb829a11775/pandas/io/html.py#L175): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace
> [Line 197:1](https://github.com/pandas-dev/pandas/blob/3d5f48df7aba938afaa4409ad8402cb829a11775/pandas/io/html.py#L197): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace
> [Line 911:89](https://github.com/pandas-dev/pandas/blob/3d5f48df7aba938afaa4409ad8402cb829a11775/pandas/io/html.py#L911): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (92 > 88 characters)
> [Line 918:89](https://github.com/pandas-dev/pandas/blob/3d5f48df7aba938afaa4409ad8402cb829a11775/pandas/io/html.py#L918): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (90 > 88 characters)
> [Line 1061:1](https://github.com/pandas-dev/pandas/blob/3d5f48df7aba938afaa4409ad8402cb829a11775/pandas/io/html.py#L1061): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace



##### Comment last updated at 2021-02-26 18:43:17 UTC
Added the ```.. versionadded:: 1.3.0``` to the doc strings
@Derekt2 this needs some tests.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
> @Derekt2 this needs some tests.

Thanks @Derekt2 for the PR. closing as stale. ping if you want to continue.
BENCH: collect isin asvs
State: closed

Pushes these as far towards parametrization as I can get them.  We end up adding a few cases, dropping none.

Decides on a policy to benchmark Series.isin and leave the others alone (see `algos/__init__.py`)

Labels:
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
great. nice to collect these. only concern is that these are kept to a reasonable amount of time (more of a general concern about benchmarks).
TYP: create a Frequency alias in _typing
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Spinning off a piece of #38416

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks @arw2019
BENCH: collect low-dependency asvs
State: closed

The idea here is to collect benchmarks in such a way that we can say "This PR doesn't change code in files [...] -> We can skip running asvs in files [...]".  (putting together a POC script to automate a bit of this)

There is a cost to refactoring benchmarks, since we have to re-run the new benchmarks on older commits (and im not sure if that happens automatically or if some human intervention is needed cc @TomAugspurger ?).  On the flip side, this is the kind of problem that we can throw hardware at, so I don't worry about it that much.  Still, the bar for merging is higher than it would be for non-benchmark refactors.

With that in mind, the other refactors I'm looking at:

1) We have benchmarks for e.g. `isin` scattered around, benchmarking `[algos.isin(x, y), pd.array(x).isin(y), Index(s).isin(y), Series(x).isin(y)]`.  I'd like to collect these in one place and parametrize them.

2) Similar for Index methods, we have a lot of similar benchmarks for different index subclasses that id like to collect+parametrize.


Labels:
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
seems fine can you merge master
rebased + green
ENH: Implement unary operators for FloatingArray class
State: closed

- [ ] closes #38749
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Enhancement
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
@zitorelova will want to rebase this after https://github.com/pandas-dev/pandas/pull/39971 (merging soon). just to ensure copied masks.
Hello @zitorelova! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:













There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-02-27 18:31:34 UTC
throwing errors with stata files
> throwing errors with stata files

fixed in #40094. restarted ci. if not passing with restart, will need to merge master.
lgtm ping on green.
checks complete @jreback 
Thanks @zitorelova
STYLE keep local imports relative
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

xref [this comment](https://github.com/pandas-dev/pandas/pull/39561#discussion_r575644482):

> > more generally, i like relative imports for same-directory as it clarified dependency structure
> 
> +1, I also find this helpful. I think it should certainly be doable to have a check that relative imports are never outside the local directory

@jorisvandenbossche @jbrockmendel is this what you had in mind?

Like this, imports in the same folder are relative, and those outside are absolute.

So, for example, in `pandas/_testing/_io.py`:
- `from pandas.io.common import urlopen` stays absolute
- `from pandas._testing.contexts import ensure_clean` becomes `from .contexts import ensure_clean`

Labels:

Comments:
Sure :+1:
REF: consolidate paths for astype
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
>  is the added code covered?

yes.  this is broken off from a branch that makes DTBlock/TDBlock directly backed by DTA/TDA; these are edits needed to make that pass
BUG: Breaking change to offsets.pyx - quarter offset classes
State: closed

Set default months for `BQuarterEnd`, `BQuarterBegin`, `QuarterEnd`, `QuarterBegin` classes in order; rename `startingMonth` parameter to `month` for those classes.
Addresses #8435 and #5307

- [ ] closes #8435 #5307
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Frequency
description: DateOffsets
name: Stale

Comments:
pls just one PR - just push to the same one if changing 
> pls just one PR - just push to the same one if changing

But this one has different scope from #39887 . The change regarding months alone in #39887 should hopefully be easier to process, and less breaking.
ok that's fine

you always need a test
does this handle #5307?
> does this handle #5307?

Not at present. I'd like to cover it as well - please let me know if that's OK.
> does this handle #5307?

It does now.
Hello @Pawel-Kranzberg! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:























There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-05-06 07:50:12 UTC
> can you add a sub-section in the whatsnew 1.3 api breaking changes to indicate what a user would except (aside from the deprection)

Description drafted and added.
> so you are adding a new parameter 'month', so now we have the confusing 'startingMonth' and 'month'. I would rather deprecate 'startingMonth', then add new; not even sure 'month' is a good name here. can you write a sample whatsnew note indicate what the change is here from a user POV.

BTW, maybe `anchor` would be a better replacement name?
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
> I would _really_ like to do the deprecation first and separate

@Pawel-Kranzberg what's the status. close this and open separate PRs?
> > I would _really_ like to do the deprecation first and separate
> 
> @Pawel-Kranzberg what's the status. close this and open separate PRs?

Yes, I plan to do so next week.
Thanks for the PR, but it has appeared to have gone stale. Closing due to inactivity, but we would be happy to take a new PR working on the deprecation first.
correction to get time interpolation correct
State: closed

- [ ] closes #21351
- [ ] One test does not pass and I don't know why


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Hello @oricou! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-02-19 12:03:39 UTC
> you need to either add a test or change current ones to show what you are changing.

Done but the "Missing-data" is still here...
@oricou open source software is contributed by the community

the maintainers are donating time to review and comment and once the code meets all criteria it be merged 



Thanks @oricou for taking the time to address this issue with the PR, but since this PR has gotten stale I'm going to close for now. If anyone else is interested in taking this up, happy to reopen.
[ArrayManager] GroupBy cython aggregations (no fallback)
State: closed

xref https://github.com/pandas-dev/pandas/issues/39146/

This implements one aspect of groupby: basic cython-based aggregations (so not yet general apply, python fallback, or other methods, etc, only the basic aggregations that take the `_cython_agg_general` path).

Similarly to how we currently have a `_cython_agg_blocks`, this PR adds an equivalent `_cython_agg_arrays` which calls the cython operation on each column instead of each block.

Labels:
name: Groupby
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
> rather than adding new things & constructing the AM / BM here. can you do a pre-cursor PR that pushes this to an internals routine, maybe internals/blockwise. We do not want to keep adding code in here (and in fact want to remove the block manager / internals code).

It might be less easy to fully get internals out of groupby, as we still use it in several places. But focusing on this specific usage: in the same line as we have `BlockManager.reduce` for DataFrame reductions, I could look into adding a `BlockManager.grouped_reduce` (or similar name) for the grouped reductions. 

One possible experiment to move some internals code out of groupby: https://github.com/pandas-dev/pandas/pull/39997
Updated this now #39997 is merged. Could now easily reuse the same `blk_func` and `cast_agg_result`, so resulting diff looks better now.
> so resulting diff looks better now.

much nicer.  perf impact?
You mean performance impact of the latest change? Or in general BlockManager vs ArrayManager?
> You mean performance impact of the latest change? Or in general BlockManager vs ArrayManager?

i mean perf impact of the PR on the non-AM paths
Ah. Do you see a change that you think is potentially performance sensitive? For BlockManager paths, it's mostly some renaming of functions/arguments, an additional check for `isinstance(obj, ArrayManager)` and a replacement of `mgr.axes[1] = ..` with `mgr.set_axis(1, ..)`
> Do you see a change that you think is potentially performance sensitive? 

i dont see anything obvious, but ive often been surprised by results in this part of the code
```
$ asv continuous -f 1.1 upstream/master HEAD
...
BENCHMARKS NOT SIGNIFICANTLY CHANGED.
```
also pls merge master
Merging this so I can do follow-up PRs. I think I answered to the remaining open comments, but otherwise I can further address those in the follow-up PRs.
> Merging this so I can do follow-up PRs

i know timezones are a hassle, but please try to avoid the temptation to do this
CLN: remove redundant code related to Styler
State: closed

`non_reducing_slice` is performed within the `apply` so no need to execute twice.

`maybe_numeric_slice` is a convoluted wrapper for selecting numeric columns so removed.


Labels:
name: Refactor
description: Internal refactoring of code
name: Styler
description: conditional formatting using DataFrame.style

Comments:
Nice.  IIRC non_reducing_slice is only used in this file, would it make sense to move it there?
> Nice. IIRC non_reducing_slice is only used in this file, would it make sense to move it there?

Yes I observed the same but was bigger change. updated now.
thanks @attack68 

would you mind doing a PR splitting up test_style.py (you can create a sub-dir and put in several tests modules) as its getting kind of long.
TST: port Dim2CompatTests
State: closed

broken off from #38992

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
> didn't realize 2d reshape works

yep.  very nearly all of it at this point.  #39963 gets some stragglers
I don't have time to review this, but so also not going to block it ;)
TST: collect indexing tests by method
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: MultiIndex

Comments:
@jreback gentle ping for this and #39873, blockers for more test sharing
ping on green
DOC: update link to user guide
State: closed

- [x] closes #39863 



Labels:
name: Docs

Comments:
Hello @morrme! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-02-17 19:18:12 UTC
Built this locally and the link now works fine - thanks @morrme !
@jorisvandenbossche I put the 1.3 milestone, but now I'm seeing other doc changes (e.g. #39858) being backported - should I change to 1.2.3 and backport?
This fixes a doc change concerning a 1.2 deprecation, so am pretty convinced it needs backporting

@MeeseeksDev backport to 1.2.x
normally we don't backport too much except for CI issues that are likely to affect 1.2.x 
ok here if fixing something broken
CI: filter json DeprecationWarning
State: closed

- [x] closes #39854
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

We're already using the same filter in two other places in this file.

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
@jreback gentle ping gets the CI back to green
thanks @jbrockmendel 
@meeseeksdev backport 1.2.x