BrandonFors/pandas-github-issues huggingface dataset:
REF: raise scalar_data_error to make mypy happy
State: closed

@simonjayhawkins do I need to update a config file or enable something?

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Have you tried using `NoReturn` for the return value of `_scalar_data_error`? It wasn't added until Python 3.5.4 so will fail the CI pinned to 3.5.3, but could be reasonable to use going into 1.0.0
I haven't tried that, will look into it for future reference.  In this particular case I agree with mypy that this is clearer than the status quo
> In this particular case I agree with mypy that this is clearer than the status quo

tend to agree, although the `NoReturn` is intended for patterns like this. 
lgtm. @WillAyd
Thanks!
CLN: Clean up of CI scripts
State: closed

Making the CI scripts simpler, and fixing typos (this was part of #26949, which couldn't be merged because `--dist=loadfile` was making the tests much slower).


Labels:
name: CI
description: Continuous Integration
name: Clean

Comments:
@jreback I think we discussed this in the past, do you know why `import pandas; pandas.get_option("display.encoding")` is not getting the system locale?

I see that #23923 was closed, but I'm unsure on whether we're actually testing the locales or not.
This should only affect the CI (and other than the small changes to the visualization of the skipped tests, there is no impact more than in the code clarity. So I don't think anything needs to be backported.
BUG: Raise when casting infinity to int
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] whatsnew entry

Raises a `ValueError` when attempting to cast a `Float64Index` with infinite values to a (non-`ExtensionArray`) integer dtype.  Currently this returns a garbage value:

```
idx                                                               
# Float64Index([1.0, 2.0, inf], dtype='float64')
idx.astype(int)                                                   
# Int64Index([1, 2, -9223372036854775808], dtype='int64')
```

Related PR: https://github.com/pandas-dev/pandas/pull/28438

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Haven't followed entire discussion but high level lgtm - thanks for the PR! I think @TomAugspurger can have ultimately say here though
thanks @dsaxton
POC: deprecation with strict removal version
State: closed

xref #28415 

Decorator that just registers the deprecation in a dict that we can check and test-time.

Labels:

Comments:
Thinking out loud but is it possible to move some of this to `scripts`? Putting a code-maintenance task in the code base itself seems a little strange to me so `scripts` might be a better home
> Putting a code-maintenance task in the code base itself seems a little strange to me so scripts might be a better home

Agreed on the principle, no idea about the implementation.  Maybe code_checks?
ENH: Explode multiple columns of DataFrame
State: closed

Now .explode() can take a list of column names and will explode multiple at the same time (given that each element across all the columns have the same length in every single row
- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Hello @stahl085! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-09-16 16:48:53 UTC
Just a heads up - this was part of the original implementation in #27267 so you can check there for inspiration on tests and implementation. The big blocker there though was how to handle duplicate values, i.e. whether we should generate a cartesian product or not. Do you know how other similar tools would handle that?
Thanks for the info, I like that #27267 implementation much better!
By duplicate do you mean the user passes in the same column name twice? ex: ['A', 'A']

I haven't seen this implemented elsewhere, but to me it seems un-natural for this to return a cartesian product. Would it make sense to include that as an optional argument `cartesian=False`?
Pandas get_dummies validate `columns` input
State: closed

- [x] closes #28383
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

The Test code is inspired from a similar tests in
`RaiseError`: https://github.com/pandas-dev/pandas/blob/master/pandas/tests/arrays/categorical/test_operators.py#L97
Valid `list_like`: https://github.com/pandas-dev/pandas/blob/master/pandas/tests/reshape/test_pivot.py#L706

This is my first pull request here and so I have taken all the necessary precautions, also I read the contributing guidelines. Not much sure of  `black pandas` . 

**Problem:**
If `columns` is not passed a list-like value a substantial decrease in performance was observed.

**Proposed Solution:**
As mentioned in the issue by @WillAyd that function should raise and hence based on [similar situation](https://github.com/pandas-dev/pandas/blob/2c9c4223442cd555a1fbc894eb5e89792c09ea63/pandas/core/arrays/categorical.py#L2484) in the codebase, I have edited the file to raise `TypeError`.



Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
What used to happen without this change?  Would a less-useful error message be raised or would you get an incorrect result?  For the whatsnew, add a bullet point in the appropriate subsection in doc/source/whatsnew/v1.0.0.rst
Doc at doc/source/whatsnew/v1.0.0.rst was updated also, I have updated the PR with problem and proposed solution.
As per the instructions, I have removed the `test_get_dummies_with_list_like_values` 
Any update on the PR? 
needs one more rebase, otherwise looks good to go cc @jreback
Just fixed up the merge conflict. This should be good to go when CI is green
looks like trailing whitespace is choking the linter
I removed trailing white spaces. Now it should pass
Thanks @R1j1t !
Thank everyone for the reviews. It was something new for me (adding test and updating docs) but I am sure this is my first of the many PR. I will start small but I love pandas and hence contributing in my free time.
BUG: fix size()/count() when resamping empty series (#28427)
State: closed

- [X] closes #28427
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype
name: Resample
description: resample method

Comments:
I'm not sure this is the best way to fix this.
this looked ok, can you merge master and ping on green.
@jreback, For some reason, there's still one requested change (which was done). I guess github is expecting yout to validate it?
> @jreback, For some reason, there's still one requested change (which was done). I guess github is expecting yout to validate it?

yes I need to still approve / re-request.
@randomstuff can you merge master and ping on green.
@jreback: all green.
thanks @randomstuff 

sometimes these things take a while, thanks for sticking with it!

happy to have a followup to consolidate size/count across all grouping types :->
Build table schema version fix
State: closed

- [x] Issue #28455 
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:

Comments:
Thanks for the PR! As discussed in the issue though I don't think this is a change we actually want to make. Certainly welcome your continued feedback and of course we have a ton of other issues we would love PRs for
REF: Refactor sparse HDF5 read / write
State: closed

In preparation for the removal of SparseSeries and SparseDataFrame, we
read into a Series[sparse] / DataFrame[sparse]. https://github.com/pandas-dev/pandas/pull/28425

cc @jreback @jorisvandenbossche.

I've just done the minimal amount to get this working. It may shed some light on how we can design an API for dispatching writing for 3rd party EAs (I think we should ask the EA for the values (arrays) to write, and the bits of extra metadata. Pandas should take care of the actual writing though. But that's for another day).

Labels:
name: IO HDF5
description: read_hdf, HDFStore
name: Sparse
description: Sparse Data Type

Comments:
Hmm, can someone confirm something for me on master / pandas 0.25.1?

```python
df = pd.DataFrame({"A": [1, None, 2, 3]}).to_sparse()
df.to_hdf("foo.h5", "sdf")
pd.read_hdf("foo.h5", "sdf")
```

I see `NotImplementedError: start and/or stop are not supported in fixed Sparse reading`, so reading / writing SparseDataFrame is already broken?
Hmm, yes, I see the same (with another example). If it is actually not working (and also not even tested, as you noticed), then that simplifies things I think ;)
Oh, SparseSeries reading is broken too, not just SparseFrame? So things are simplified quite a bit indeed :)

I have a local fix for SparseSeries, but it's quite ugly. I think for now I'll ignore reading old sparse HDF5 files, and only support reading new files. I'll add additional tests for newly created files though.
Given that it didn't work before, I would say this is necessarily a blocker for #28425. But if this is already working mostly for the new way, fine with adding this capability.
Good point.

This solution doesn't feel 100% correct. In particularly, I don't like dynamically modifying `self.attributes` when we write the array.
i would likely just remove sparse entirely from HDF
it was barely supported to begin with
Yes, my preference right now is probably not to merge this PR and to followup with a proper API for dispatching array / attribute writing for all ExtensionArrays.
Sounds good to me
Going to close this for now. This should be covered as part of https://github.com/pandas-dev/pandas/issues/20612 when that's picked up.
Annotate DataFrame Part 2 (Rendering Methods)
State: closed

- [x] part of  #26792 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] annotate all rendering methods of `pandas.core.frame.DataFrame`


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @vaibhavhrt - will take as many more as you've got!
Yeah, will keep them coming, trying not to do too many in same PR as it's gets harder to review then.
DOC: fix docstring of core.comon.pipe (PR02)
State: closed

Solves:
- Unknown parameters {kwargs, args}
- Parameter "func" type should use "str" instead of "string"
- Parameter "*args" description should start with a capital letter
- Parameter "**kwargs" description should start with a capital letter
- pandas.core.common.pipe : Unknown parameters {kwargs, args} in #27976 


Labels:
name: Docs
name: Code Style
description: Code style, linting, code_checks

Comments:
@AminooZ thanks a lot!
CLN: remove merge-pr script, clean Exception in tm, doc
State: closed

I count 131 of these ATM, some of which we'll end up marking as intentionally non-specific.

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Clean

Comments:
I never use it anymore, and I think we fully switched to using github's button to merge PRs. But maybe some people still use it to fetch PR's branchs and push back there?
yeah i think we can blow away the merge script
Updated to remove merge-pr.py
Thanks!
BUG: Don't cast categorical nan to int
State: closed

- [x] closes #28406
- [x] passes `black pandas`
- [x] tests added / passed
- [x] whatsnew entry

This raises an error when attempting to cast a `Categorical` or `CategoricalIndex` containing `nans` to an integer dtype.  Also had to remove the casting within `get_indexer_non_unique` since this won't always be possible.

Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Categorical
description: Categorical Data Type

Comments:
FWIW the issue with `test_get_indexer_non_unique ` is here https://github.com/pandas-dev/pandas/blob/master/pandas/core/indexes/base.py#L4717.  If `target` is `Categorical` then the values can't always be cast to the dtype of the categories.
should the use_inf_as_na option matter here?
> should the use_inf_as_na option matter here?

Wasn't familiar with that option, but it shouldn't affect `numpy` behavior I think?
> I think this PR should focus on just NaN handling.
> 
> I'd rather see `Float64Index` properly handle astyping inf values to int (raise), and then do `self.categories.take(self.codes)` with a check if any of the codes is negative, so that we raise.

Ok, so for the purposes of this PR remove the references to `np.inf` in the code and tests?
That would be my preference. Did my earlier comment make sense?

The `NaN` issue can be solved by checking `Categorical.codes`

```python
In [7]: c = pd.Categorical([1, None, 2])

In [8]: c.codes < 0
Out[8]: array([False,  True, False])
```

The inf issue is deeper, and is present in Float64Index, so I think it should be split to its own PR.

```python
In [9]: c = pd.Categorical([1, np.inf])

In [10]: c.categories.astype(int)
Out[10]: Int64Index([1, -9223372036854775808], dtype='int64')
```
> That would be my preference. Did my earlier comment make sense?
> 
> The `NaN` issue can be solved by checking `Categorical.codes`
> 
> ```python
> In [7]: c = pd.Categorical([1, None, 2])
> 
> In [8]: c.codes < 0
> Out[8]: array([False,  True, False])
> ```
> 
> The inf issue is deeper, and is present in Float64Index, so I think it should be split to its own PR.
> 
> ```python
> In [9]: c = pd.Categorical([1, np.inf])
> 
> In [10]: c.categories.astype(int)
> Out[10]: Int64Index([1, -9223372036854775808], dtype='int64')
> ```

I believe so, you want to fix the `inf` casting behavior in one place (`Float64Index`) and then use the fixed `astype` method here?  Regarding the `nan` issue, is there a reason to prefer `self.codes < 0` over `self.isna()`?
thanks @dsaxton
BUG: Fix merging non-indexes causes Index dtype promotion in when keys are missing
State: closed

... from left or right side. (GH28220)
Also closes GH24897, GH24212, and GH17257

- [x] closes #28220, #24897, #24212, and #17257
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Probably requires a bit more cleanup and reduction of spaghetti... Not in love with the the solution (as it's quite similar to using both indexes) and requires extension test case changes, but hoping to get some comments and feedback on making it better.


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Looks like a MultiIndex is getting mixed up with a RangeIndex. Did this pass locally?
I ran the merging unit tests in test_merge. I'll spend some time to fix the others
Hello @dworvos! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-11-09 03:05:47 UTC
can you merge master and will have a look
I've rebased with upstream/master. Thanks!
Just checking in to see if I did what you asked of correctly.

Cheers
@dworvos needs another rebase
@dworvos i think we'd take this, but needs some work if you want to continue, pls ping
Sure, I can take a look, but I'd like some feedback on what needs to be done rather than just constantly rebasing and getting nowhere?
BLD: clean-up sdist creation
State: closed

I messed up in https://github.com/pandas-dev/pandas/pull/28374/files#diff-2eeaed663bd0d25b7e608891384b7298R531. I meant to revert the lines that excluded cythonized files in the sdist to save it for a followup.

So I'm using this to clean up the sdist logic a bit, and exclude a few test data files that have snuck into our distribution.

Labels:
name: Build
description: Library building on various platforms

Comments:
LGTM
@jorisvandenbossche done and passing.
Whoops, this should have a release note. 1 sec.
REF: implement arithmetic, comparison, logical ops on arrays
State: closed

Following #28413, #28396, and #28395, these are all pretty straightforward refactors.

`arithmetic_op`, `comparison_op`, and `logical_op` are going to become the ops that we call `Block`-wise.  Hopefully also `PandasArray` will use them directly.

Other follow-ups I have in mind:
- Move `should_extension_dispatch` and `dispatch_to_extension_op` so we dont need runtime imports
- get rid of `eval_kwargs` entirely, they're not really necessary at this point
- see if we can use `numexpr` for comparison/logical ops
- docstrings
- address inconsistencies in when we call `extract_array`
- address inconsistencies in how `Series` methods handle `__finalize__` and alignment
- simplify `logical_op`'s `na_op`

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
Added requested comments and types.  Holding off on docstrings since doing those right will be non-trivial.
reverted types because mypy was complaining about indexes.base
@jreback gentle ping.  This is the last blocker before I can implement block-wise ops for the scalar case (and one of the two Series cases)
updated with types and bare-bones docstrings
> this is mostly just a cut/paste yes?

Correct.  types and basic docstrings have been added.
> can you move both extract_array's in here? would make groking this simpler

I agree that having extract_array in exactly one place is weird.  We'll be able to get rid of it after the integer-addition deprecation is enforced (and keep_null_freq becomes unnecessary).

The tradeoff is that doing all 6 extract_array calls inside the functions would break the typing "guarantees", which I'm starting to like.
> > [@jreback] can you move both extract_array's in here? would make groking this simpler
>
> I agree that having extract_array in exactly one place is weird. We'll be able to get rid of it after the integer-addition deprecation is enforced (and keep_null_freq becomes unnecessary).
>
> The tradeoff is that doing all 6 extract_array calls inside the functions would break the typing "guarantees", which I'm starting to like.

I agree with @jbrockmendel that it is nice to already know what types are passed to those functions. So I think what is done in the PR (do extract_array outside, except for the one case where it is needed for the deprecated functionality) is fine. It's only temporary (we should remove that deprecation for 1.0), and gives the least code flux afterwards (only that one case can be moved out afterwards.
rebased+green
thanks @jbrockmendel couple of followup requests noted above
BUG: Remove error raise to allow same input function on the same column in named aggregation
State: closed

- [x] closes #28426 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


https://github.com/pandas-dev/pandas/pull/27921#issuecomment-521794197

this was kind of mentioned in the PR27921, and the case used in the issue is basically the same as having `min` or `max`. Previously it was not raising error is because I deliberately skipped this case since i thought this removal should be treated in different PR. Feel free to review and comment your ideas.


Labels:
name: Bug
name: Groupby

Comments:
this solution looks tricky though, but i tested on other cases like lambda or `min`, 'max` stuff, they are all working except the partials. So I made this change (although looks like two different issues) @TomAugspurger 
can you merge master
@jreback @TomAugspurger  merged to master and fixed tests

since this PR was quite long ago, so i hereby put little summary to the outcome of this PR, with the PR merged:
```python
df = pd.DataFrame({"A": [1, 2]})
df.groupby([0, 0]).agg(['sum', 'sum'])
```
will still raise an error, but
```python
df = pd.DataFrame({"A": [1, 2], "B": [3, 4]})
df.groupby("A").agg(a=("B", "sum"), b=("B", "sum"))
```
will not raise because of users specifying different new column names for it

emm, not sure what caused it. 🤔 
@charlesdong1991 we had some CI issues a few weeks back; if you merge master and repush should be resolved
thanks, @WillAyd merged! feel free to review and see if you think it's worth it to leave this PR open!
can you merge master
Hello @charlesdong1991! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-12-03 14:45:06 UTC
merged! 
could you pls take a look? @jreback 
this looks good @charlesdong1991 can you merge master and ping on green.
ping

@jreback 
ping  @jreback 
thanks @charlesdong1991
BUG: inconsistency when passing dict to Series with index
State: closed

* Adds ValueError if passing a dict into pd.Series with a different size than the passed index parameter.  Closes #28418 



- [x] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

(Will add whatsnew / test if this solution is acceptable.)


Labels:
name: Needs Discussion
description: Requires discussion from core team before further action

Comments:
Hello @jsal13! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-09-12 17:54:41 UTC
Discussing this in the issue: https://github.com/pandas-dev/pandas/issues/28418
Closing due to it being not necessary.
ENH: MultiIndex.from_product infers names from inputs if not explicitly provided
State: closed

- [x] closes #27292
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Infers names in `MultiIndex.from_product`  and normalizes handing of `names=None` between `from_arrays` and `from_product`

Labels:
name: Enhancement
name: MultiIndex

Comments:
do we have enough testing on ``from_arrays``  for this?
Thanks!