BrandonFors/pandas-github-issues huggingface dataset:
DOC: Fix EX01 in pandas.DataFrame.idxmin
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Related to #27977. 

```
################################################################################
################################## Validation ##################################
################################################################################

Labels:
name: Docs

Comments:
Thanks @farhanreynaldo 
so this just triggered in azure, so maybe ok now
Accept dateutil frequency constants in date_range
State: closed

- [x] closes #9314
- [x] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype

Comments:
Hello @mason-98! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-03-13 19:31:15 UTC
edit: ignore my comment above; overlooked the first module!
Thanks for your interest in this PR @mason-98. I think we need more discussion on whether we want to accept this feature in the original issue as the dateutil frequencies are just integers and not enums.

Feel free to tackle other `good first issue` tags if interested! Going to close for now
TST: Parametrize in pandas/tests/internals/test_internals.py
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean

Comments:
can you rebase
##[error]./pandas/tests/internals/test_internals.py:558:5:F811:redefinition of unused 'test_interleave' from line 549
@MomIsBestFriend can you rebase and ping on green
Restarting azure as I am not sure why it failed.
ping @jbrockmendel, the CI is green.
thanks @MomIsBestFriend
BUG: Series.__getitem__ with downstream scalars
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

cc @spencerclark I think this fixes a subset of the issues reported in https://github.com/pydata/xarray/issues/3751, can you confirm?

@jorisvandenbossche IIRC geopandas scalars not being recognized by lib.is_scalar has caused some issues there; does this address any of those?

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Thanks @jbrockmendel -- sorry I just saw this now.  Yes, as written, this change indeed fixes the remaining issue for us, though I think @jreback's suggestion would work too.
Updated with a better implementation, and a test that fails in master: `df.dtypes[some_dtype]` currently raises, is fixed here.
much better, thanks.
CLN: Remove PY2 compat code
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

---

Benchmarks:

#### Master:

```
In [1]: import pandas as pd                                                                                   

In [2]: s = pd.Series(list("ABCA"))                                                                           

In [3]: %timeit pd.get_dummies(s)                                                                             
482 µs ± 752 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```

---

#### PR:

```
In [1]: import pandas as pd                                                                                   

In [2]: s = pd.Series(list("ABCA"))                                                                           

In [3]: %timeit pd.get_dummies(s)                                                                             
479 µs ± 2.43 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```

Labels:
name: Clean

Comments:
LGTM.
Thanks, @MomIsBestFriend.
Avoid bare pytest.raises in dtypes/test_dtypes.py
State: closed

* [x]  ref #30999
 
* [x]  tests added / passed
 
* [x]  passes `black pandas`

* [x]  passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Please be aware that I changed the dtypes.py file. I believe that there's a spelling mistake in one of the error messages. Could someone please confirm whether this is the case? Please note that one of the other testing files uses the other verbiage.

It's the difference between Categorical and Categorial.
Thanks @Vlek
REF: put all post-processing at end of DataFrame._reduce
State: closed

there's some nasty try/except logic in DataFrame._reduce.  This is simplifying the code _around_ that.

Labels:
name: Refactor
description: Internal refactoring of code
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
thanks. _reduce defiinitely needs teasing apart.
TST: reintroduce check_series_type in assert_series_equal
State: closed

- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Re-introduced `check_series_type` in `assert_series_equal` which was recently removed in #32513. It was a part of public API which is not used internally, but it is used elsewhere (like geopandas). Discussion in #32513 suggests that it should be put back where it was.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Dependencies
description: Required and optional dependencies

Comments:
I think worth completely reverting rather than part of the original PR

@jbrockmendel @jorisvandenbossche 
> I think worth completely reverting rather than part of the original PR

Why is that?  #32513 included some non-downstreaming-breaking improvements
Yes, just adding back only this is perfect IMO. The rest of the PR was actually completely unrelated to this change (regardless of the title of the PR that seemed to indicate otherwise).
We might want to add a test though
Added a test
@simonjayhawkins this is passing now. Are you OK with including the fix here?

cc @jbrockmendel 
> lgtm. maybe #32670 (comment)

Ah, yes, forgot that. That actually simplified it a lot
lgtm
BUG: DatetimeArray._from_sequence accepting bool dtype
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Identified when making the change this makes in Series.

Labels:
name: Datetime
description: Datetime data dtype
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
looks fine. need a whatsnew as a bug fix for pd.to_datetime erroneously accepting bools
can you rebase your PRs again, now that azure if running. (likely not a big deal, but just to make sure)
thanks
implemented additionally functionality of formatters_col in to_latex
State: closed

Not sure what I should do for the whatsnew entry, bit confused about versioning. 

- [ x ] closes #26278
- [ x] tests added / passed
- [ x] passes `black pandas`
- [ x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Enhancement
name: IO LaTeX
description: to_latex
name: Stale

Comments:
Hello @wlgranados! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-10-06 01:25:04 UTC
i am not sure the proposed api is in line how we format for example
i’m to_csv is there a reason you are diverging?
> i am not sure the proposed api is in line how we format for example
> i’m to_csv is there a reason you are diverging?

No particular reason, just wasn't paying attention, I believe I have addressed this issue with the latest commit. 
@wlgranados do you have time to merge in master and to address the review comments? 
@mroeschke yes, I have some time.
I believe I addressed some comments on my local branch, just forgot to mention it here. 
I'll update this shortly. 
@wlgranados is this still active? Can you address latest comments?
@WillAyd @mroeschke  Yeah still active, I found that the test case I included was incorrect. There is an issue with the lambda function generation inside the list comprehension, so attempting to fix that currently. Also I read in the original thread that it might be possible to use the formatters variable instead, I think this should address the api usage comments then what I had before. 
@WillAyd @mroeschke this should be code complete, so any comments would be appreciated 
Looks pretty good. Just needs a whatsnew entry for 1.2.0
@wlgranados looks like some merge conflicts have piled up and there are some outstanding CI failures - can you take a look and try to get things green?
Closing in favor of #37552
REF: values-> _values
State: closed

For the affected Index subclasses .values and ._values are the same

Labels:
name: Refactor
description: Internal refactoring of code

Comments:
@WillAyd tangential to this PR: in `io.json._json `there are a couple of uses of `obj.values`; is it clear whether those can be changed to `obj._values`?
I don't know all of the differences between the two enough to say for sure but I would think so
> I don't know all of the differences between the two enough to say for sure but I would think so

Yah it can be hard to keep track of.  Rule of thumb: `._values` is non-lossy and non-copying, `.values` may doesn't assure either for EA Series/Index.

I'll try to changing those two usages and see if it breaks anything.
DOC: Fix EX02 in pandas.Series.memory_usage
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Related to #27977. 

```
################################################################################
################################## Validation ##################################
################################################################################

Labels:

Comments:
I think for now should just stick with what CI is yielding. I don't think the 8 byte difference is worth worrying too much about, though if you see something more problematic feel free to open an issue for it.

Thanks anyway for the PR!
BUG: process Int64 as ints for preservable ops, not as float64
State: closed

Calling `.min()` or `.max()` on `pd.Int64` data returns integers, but uses `float64` as an intermediate representation, leading to data corruption.

This PR attempts to whitelist them to be processed entirely as `Int64`s. Also add a test that verifies this behavior, as existing tests failed to detect the bug.

- [x] closes #32651
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
@qwhelan thanks for looking into this!

I am personally not fully convinced this is the best approach long term (I rather see us looking into https://github.com/pandas-dev/pandas/pull/30982, but there is still discussion on the way forward). So short term this can certainly fix the issue at hand.
@jorisvandenbossche Yeah, I agree this isn't the long-term approach. The issue is that `df[df[col] == df[col].max()]` is a fairly popular idiom and fails when the left side is an `Int64` and the right side is a `float64` (if not for type reasons, for the latter losing precision).
so would be ok with this for *only* min/max right now.
@jreback Updated to the bare minimum to fix `IntegerArray.min/max`
we are likey to merge: https://github.com/pandas-dev/pandas/pull/30982 shortly. Happy to take this as a followon using that approach.
@qwhelen does #30982 affect the implementation here?
@jbrockmendel Looks like this is unnecessary now, closing
BUG: DTI/TDI/PI get_indexer_non_unique with incompatible dtype
State: closed

I'm pretty sure we can share more code here, but PeriodEngine needs some work first.

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
lgtm. needs a rebase. is this user visible at a high level? e.g. in .loc or someting?
>  is this user visible at a high level? e.g. in .loc or someting?

Yes, will update the test introduced in #30446
also needs a rebase
hmm this should probably raise a KeyError instead of a TypeError?

```
df = pd.DataFrame(
    np.random.randn(5, 3),
    columns=["a", "b", "c"],
    index=pd.date_range("2012", freq="H", periods=5),
)

df = df.iloc[[0, 2, 2, 3]].copy() 

dti = df.index
tdi = pd.TimedeltaIndex(dti.asi8)  # matching i8 values

with pytest.raises(TypeError, match="is not convertible to datetime"):
    df.loc[tdi]

with pytest.raises(TypeError, match="is not convertible to datetime"):
    df["a"].loc[tdi]
```
> hmm this should probably raise a KeyError instead of a TypeError?
> 
> ```
> df = pd.DataFrame(
>     np.random.randn(5, 3),
>     columns=["a", "b", "c"],
>     index=pd.date_range("2012", freq="H", periods=5),
> )
> 
> df = df.iloc[[0, 2, 2, 3]].copy() 
> 
> dti = df.index
> tdi = pd.TimedeltaIndex(dti.asi8)  # matching i8 values
> 
> with pytest.raises(TypeError, match="is not convertible to datetime"):
>     df.loc[tdi]
> 
> with pytest.raises(TypeError, match="is not convertible to datetime"):
>     df["a"].loc[tdi]
> ```

hmm, yep sounds right
thanks
MultiIndex union/intersection with non-object other
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

ATM if we call MultiIndex.union with e.g. a Float64Index, we get `ValueError: Buffer dtype mismatch, expected 'Python object' but got 'double'` from `lib.fast_unique_multiple`.  This PR changes that to raise NotImplementedError with a reasonable message, but we could alternatively return `self.to_flat_index().union(other)`

I prefer making the user do to_flat_index on their own because this seems like something that would often be reached by accident.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex

Comments:
gentle ping; we're on the verge of being rid of _ndarray_values
thanks
Simplified get_blkno_indexers
State: closed

This function is the last that appears to be throwing warnings in #32163

This PR doesn't solve that, but simplifies the logic to make it easier to grok

Labels:
name: Clean

Comments:
nice cleanup assuming perf isnt hurt
Here are the results for reshaping and concat. A mixed bag but I think net benefit

```sh
+     1.50±0.08ms       2.80±0.2ms     1.87  reshape.SparseIndex.time_unstack
-        30.6±3ms         18.9±2ms     0.62  reshape.Cut.time_qcut_datetime(10)
-        89.8±5ms         50.9±6ms     0.57  reshape.Cut.time_qcut_timedelta(1000)
-         101±7ms         48.7±5ms     0.48  reshape.Cut.time_qcut_datetime(1000)
-        31.9±3ms         14.8±1ms     0.46  reshape.Cut.time_qcut_datetime(4)
-       94.1±10ms         36.6±8ms     0.39  reshape.Cut.time_cut_datetime(1000)
-        19.7±5ms       7.51±0.2ms     0.38  reshape.Cut.time_cut_datetime(4)

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE DECREASED.
```

```sh
       before           after         ratio
     [1f2385ea]       [dec831fa]
+        18.4±1ms         26.5±2ms     1.44  join_merge.Concat.time_concat_small_frames(1)
-        843±80μs          649±9μs     0.77  join_merge.Concat.time_concat_mixed_ndims(1)
-         327±7μs          245±3μs     0.75  join_merge.Concat.time_concat_empty_right(1)
-         373±5μs         254±10μs     0.68  join_merge.Concat.time_concat_mixed_ndims(0)
-        377±30μs          251±3μs     0.67  join_merge.Concat.time_concat_empty_left(1)
-        286±70μs          182±4μs     0.64  join_merge.Concat.time_concat_empty_left(0)

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE DECREASED.
```
thanks
fix infer_freq raises section
State: closed

Fixes the raises section of the `infer_freq` doc string. Seems small enough to skip a whats new entry.

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry



Labels:
name: Docs

Comments:
Thanks @wholmgren
DEPR: Index.ravel behavior
State: closed

The future behavior should match Series.ravel.

- [x] closes #19956
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Index
description: Related to the Index class or subclasses

Comments:
Mothballing to clear the queue, will revisit.
DEPR: Categorical.to_dense
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Ideally I'd want to keep the method and change the behavior to (an improved variant of) _internal_get_values, but so it goes.

Labels:
name: Categorical
description: Categorical Data Type
name: Deprecate
description: Functionality to remove in pandas

Comments:
can you add a release note & link to the deprecation removal issue.
~~rebase again pls~~
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
thanks
TST: tighten check_categorical=False tests
State: closed

This removes all check_categorical=False usages except for a) those in tests/util and b) a skipped json test test_latin_encoding (cc @WillAyd is that likely to be enabled in the foreseeable future?)

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
> skipped json test test_latin_encoding (cc @WillAyd is that likely to be enabled in the foreseeable future?)

I'm not familiar with that test / background but would think not. Looks like its part of #13774 will comment there
@jorisvandenbossche any objection to removing check_categorical kwarg from assert_series_equal now that we no longer use it?
> @jorisvandenbossche any objection to removing check_categorical kwarg from assert_series_equal now that we no longer use it?

Yes, it's a public keyword, so needs to be deprecated first if we want to remove it.
TYP: Remove  _ensure_type
State: closed

- [ ] closes #31925
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@TomAugspurger do we also want to add a test and a whatsnew, or merge these changes into original PR?
Thanks! This seems good for now. I don't think we would need any additional tests, but a whatsnew would be nice (fixed regression).
ok can revert test. will add whatsnew.
No harm in keeping the test.
lgtm. w.r.t. test no strong preference, though do we see ourselves getting confused with that in the future? 

Maybe just my lack of familiarity with our subclassing, but not sure why we would want a reindex to return a different type than the caller
I think we'll be able to track it back to this discussion.

But yes, I think in general the return type on `_constructor` should be the same as the input, but how we go about encouraging / requiring that is a bigger discussion.
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.0.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 a3642ecb34b8a65a9d899f17475612b3bff411d4
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #32633: TYP: Remove  _ensure_type'
```

4. Push to a named branch :

```
git push YOURFORK 1.0.x:auto-backport-of-pr-32633-on-1.0.x
```

5. Create a PR against branch 1.0.x, I would have named this PR:

> "Backport PR #32633 on branch 1.0.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
[WIP] Numpy dev 32bit (do not accept)
State: closed

This will hopefull catch 32bit issues with newer numpy versions

Do NOT accept this at this point.  Right now I am trying to figure out where a bug is originating and I need the tests to run.

Labels:
name: CI
description: Continuous Integration
name: 32bit
description: 32-bit systems

Comments:
@toddrme2178 I don't think you're working on this anymore. Ping me to reopen if you want to keep testing.
DOC: fix formatting / links of API refs in 1.0.2 whatsnew (#32620)
State: closed

https://github.com/pandas-dev/pandas/pull/32620

Labels:

Comments:
Unrelated CI failure

```
2020-03-11T14:07:00.9385237Z WARNING: failed to reach any of the inventories with the following issues:
2020-03-11T14:07:00.9389172Z intersphinx inventory 'https://dateutil.readthedocs.io/en/latest/objects.inv' not fetchable due to <class 'requests.exceptions.HTTPError'>: 502 Server Error: Bad Gateway for url: https://dateutil.readthedocs.io/en/latest/objects.inv
```
DOC: fix formatting / links of API refs in 1.0.2 whatsnew
State: closed

cc @TomAugspurger (did a few other clean-ups directly as well)

Labels:
name: Docs

Comments:
Thanks.
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.0.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 c1fd95bece36815d7b3fa09f1e103a5218c5f5e9
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #32620: DOC: fix formatting / links of API refs in 1.0.2 whatsnew'
```

4. Push to a named branch :

```
git push YOURFORK 1.0.x:auto-backport-of-pr-32620-on-1.0.x
```

5. Create a PR against branch 1.0.x, I would have named this PR:

> "Backport PR #32620 on branch 1.0.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
I'll backport this manually.
DOC: Clarify pivot_table fill_value description
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

---

Clarify that `fill_value` gets used after performing the aggregation (i.e. it fills values in the resulting pivot table).
(Depending on the aggfunc and df, the result will be different if performing the filling before/after aggregation).

---

### Actually, why not remove it, requiring the user to more explicitly do `df.pivot_table(...).fillna(...)`?

Labels:
name: Docs

Comments:
Thanks @taljaards