BrandonFors/pandas-github-issues huggingface dataset:
BUG: Use args and kwargs in Rolling.apply
State: closed

- [x] closes #33433
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Window
description: rolling, ewma, expanding

Comments:
Hello @mproszewska! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-05-15 11:38:03 UTC
thanks @mproszewska 

cc @simonjayhawkins marked for 1.04 as well
BUG: Handling columns from index_col in _is_potential_multi_index
State: closed

- [x] closes #33476
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [] whatsnew entry


Labels:
name: IO Excel
description: read_excel, to_excel
name: MultiIndex

Comments:
Hello @mproszewska! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-06-01 00:42:28 UTC
I added requested changes.

Everything should be fine now
thanks @mproszewska 
@mproszewska , thanks very much!

But @mproszewska , @jreback , should we add an argument to [this line](https://github.com/pandas-dev/pandas/blob/c71bfc36211b5e2d860a06d8fbef902b757bd6e4/pandas/io/parsers.py#L1601), too, shouldn't we?
https://github.com/pandas-dev/pandas/blob/c71bfc36211b5e2d860a06d8fbef902b757bd6e4/pandas/io/parsers.py#L1601
PERF: make _Tick into a cdef class
State: closed

The actual "PERF" part is that in the follow-up we can cimport `is_tick_object` which will be appreciably faster than the `getattr(other, "_typ", None) == "dateoffset" and hasattr(other, "delta")` checks we now do

Labels:
name: Performance
description: Memory or execution speed performance
name: Frequency
description: DateOffsets

Comments:
also assume we have sufficient benchmarks on this (e.g. how you found this at some point)
DOC: start 1.0.4
State: closed

NOTE: opened against 1.0.x branch as to not commit to release at this stage.

I think could backport a few PRs moving the whatsnews and only after a release remove from 1.1 whatsnew on master. This gives us the chance to easily abandon the release at any time.

Labels:
name: Docs

Comments:
lgtm. feel free to merge things as needed to this branch.
This needed to be merged in master first / as well (or did that already happen in another PR?)
> This needed to be merged in master first / as well (or did that already happen in another PR?)

purposefully done it this way, see OP

> NOTE: opened against 1.0.x branch as to not commit to release at this stage.

was planning to copy release notes on backports verbatim, then when ready to do a release, tidy up the whats new to be more applicable and then following a release, remove the applicable release notes from 1.1 whatsnew.

but could also work against master. wdyt?
> was planning to copy release notes on backports verbatim, then when ready to do a release, tidy up the whats new to be more applicable and then following a release, remove the applicable release notes from 1.1 whatsnew.

Yes, I think that is fine, but for backporting the PRs verbatim, you don't need this file (yet) in the 1.0.x branch, right?

But also in that workflow, the PR that fixes up the whatsnew files (moving the appropriate whatsnew entries from 1.1.0 to 1.0.4) can be done against master + backported?
> Yes, I think that is fine, but for backporting the PRs verbatim, you don't need this file (yet) in the 1.0.x branch, right?

i meant copy the release note from 1.1 whatsnew to 1.0.4 whatsnew without changes in the backport PRs

so the 1.0.x branch doesn't have a 1.1. whatsnew and master doesn't have a 1.0.4 whatsnew (yet)
Ah, yes, I forgot that the 1.0.x branch has no 1.1 file, so you can't just backport without changing the whatsnew. Yeah, then that workflow seems fine to me to move the whatsnew *inside* the backport PR

> Yeah, then that workflow seems fine to me to move the whatsnew _inside_ the backport PR

it does mean that each backport needs to be done manually, since auto backports fail. so there maybe a better workflow involving having a 1.1 whatsnew in 1.0.x temporarily.

the other downside is that any PRs in the pipeline that are suitable for backport can't add the release note to 1.0.4.
Was thinking the same and wondering if adding a "blank" 1.1.0 file (or only with the headers) would help, but it's quite possible that this would still give conflicts though in almost all cases in the whatsnew file. 
But it could be worth to try out.
(in general, for future releases, I think it will be easier to assume the bugfix release will happen, and when it's clear that it doesn't happen, it's easy to move the whatsnew entries from eg 1.0.4 to 1.1.0 and delete 1.0.4 file in one PR, instead of the other way around)
CI: Bump numpydev URL
State: closed

NumPy and scipy are uploading their wheels to a new URL.


Labels:
name: CI
description: Continuous Integration

Comments:
great this is the same URL that we recommended in #33505  

cc @alimcmaster1 
merge on green.
We got

```
numpy                     1.19.0.dev0+8dfaa4a          pypi_0    pypi
```

That's from 12 days ago, but there are newer wheels available (one from yesterday). Is pip just grabbing the "highest" version number, which happens to be `...8dfaa4a`?

@mattip has NumPy's development builds always used `.dev0+<git hash>` instead of something like `.dev0+<increasing-count>.<git hash>`? Perhaps I'm just noticing that now.
> @mattip has NumPy's development builds always used `.dev0+<git hash>` instead of something like `.dev0+<increasing-count>.<git hash>`? Perhaps I'm just noticing that now.

hmm, that is problematic.
Looking at the [old location](http://66ce22ef0bf26a5e2cca-4ffdece11fd3f72855e4665bc61c7445.r49.cf2.rackcdn.com/) I see it used the dates as well, for instance numpy-1.19.0.dev0+20200504031348_58dbafa-cp37-cp37m-linux_x86_64.whl. I must have missed that in the move to azure scripts
@mattip I didn't immediately see where version was set with the old MacPython/numpy-wheels setup. Do you want me to open an issue somewhere as a reminder?
ping me here again if it is not fixed soonish.
It is [something the uploader did](https://github.com/matthew-brett/wheelhouse-uploader/blob/master/wheelhouse_uploader/upload.py#L180) to rename dev wheels. 
Is this related to the issue breaking the npdev builds?
@TomAugspurger newer wheels are available. Is there a reason not to delete the old ones?
Thanks! No reason to keep the old ones around I think.

@jbrockmendel are you referring to the

```
Cython.Compiler.Errors.CompileError: pandas/_libs/reduction.pyx
```

in https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=34915&view=logs&j=3a03f79d-0b41-5610-1aa4-b4a014d0bc70&t=fe74a338-551b-5fbb-553d-25f48b1836e8&l=443? That looks unrelated from the changes here.
triggering to see if this is picked up
hmm not sure this is picking up the right version.
#34050 should fix the non-npdev CI failure
I think we are still getting the older versions.
Actually I think we're OK. https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=35269&view=logs&j=3a03f79d-0b41-5610-1aa4-b4a014d0bc70&t=fe74a338-551b-5fbb-553d-25f48b1836e8&l=664 has

```
numpy                     1.19.0.dev0+e664535
```

Looking at https://anaconda.org/scipy-wheels-nightly/numpy/files, that's from yesterday.

On anaconda.org, it included the full tag `numpy-1.19.0.dev0+20200510035734_e664535-cp37-cp37m-manylinux1_i686.whl`. Hopefully it's not just by accident that we got the latest wheel :)
ok great, thanks @TomAugspurger 

I guess just let's monitor to make sure we keep getting updates.
@Meeseeksdev backport to 1.0.x
fix bfill, ffill and pad when calling with df.interpolate with column…
State: closed

… dtype string (#33956)

- [x] closes #33956 
- [x] closes #12918 
- [x] closes #29146 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
xref #12918 maybe the tests could test for equality with either DataFrame.ffill or DataFrame.fillna(method='ffill) and update the DataFrame.interpolate docstring, see https://github.com/pandas-dev/pandas/issues/12918#issuecomment-211424101 and https://github.com/pandas-dev/pandas/issues/12918#issuecomment-419425660 
can you merge master
> xref #12918 maybe the tests could test for equality with either DataFrame.ffill or DataFrame.fillna(method='ffill) and update the DataFrame.interpolate docstring, see [#12918 (comment)](https://github.com/pandas-dev/pandas/issues/12918#issuecomment-211424101) and [#12918 (comment)](https://github.com/pandas-dev/pandas/issues/12918#issuecomment-419425660)

I went to the interpolate function back and forth and I come to the conclusion that we totally should fix this first. When we want to ensure that `fillna(method='ffill')` and `interpolate(method='ffill')` behave the same way, why shouldn't we just call `fillna` inside of `interpolate` in this case?

The whole `interpolate` function looks pretty much like things have grown over time and the quick fix is prefered to the generic, clean solution (with transposing back and forth, and so on). I am working on a cleaner solution. Am curious for your feedback.

EDIT:
I found a quick solution which fixes the referenced tickets (#33956 and #12918 and #29146). I am not totally satisfied with this solution since it is adding complexity (though only a little) to an already complex code. I would still suggest that a refactoring of the total `interpolate` logic would make things much clearer.
Hello @CloseChoice! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-05-29 16:26:02 UTC
> Thanks @CloseChoice for the updates, is it feasible for the fixes for #12918 and #29146 be split off into a pre-cursor PR. smaller PRs with more atomic changes are easier to review/quicker to merge. also needs a whatsnew.

Thanks for your review. Actually fixing #12918 and #29146 has the side effect of fixing #33956, since the `fillna`-methods don't have the issue described in #33956. So I don't see possibility to split this into two PRs though I totally agree with you to have small, atomic PRs.
thanks @CloseChoice very nice
DOC: update conf.py to use new numpy doc url
State: closed

- [X] closes #33953
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Thanks @codeloop, did you build the docs locally and checked that the link is working all right? I guess the intersphinx file is available at the new URL, but we need to make sure before merging this. Thanks!
@datapythonista , I'm building the docs, following the steps from here [Contributing to the documentation](https://pandas-docs.github.io/pandas-docs-travis/development/contributing.html#contributing-to-the-documentation), will update after build is complete & tested locally.
successfully built documentation locally using this commit, now the links point to numpy URL. 
For eg: In the pandas docs at https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.argsort.html
earlier it was referencing to : https://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.argsort.html#numpy.ndarray.argsort
After this commit & build it references to : https://numpy.org/doc/stable/reference/generated/numpy.ndarray.argsort.html#numpy.ndarray.argsort

Looks good to me.
@jorisvandenbossche may be you want to have a look
@Meeseeksdev backport to 1.0.x
WIP: BUG: Setting DataFrame values via iloc aligns when arguments are lists
State: closed

- [x] closes #22046
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This PR currently fails two tests on Windows due to int32 vs int64, but I cant' duplicate on my machine.

````
pandas\tests\indexing\test_indexing.py:588
test_astype_assignment

df = df_orig.copy()
df.iloc[:, 0:2] = df.iloc[:, 0:2].astype(np.int64)
expected = DataFrame(
    [[1, 2, "3", ".4", 5, 6.0, "foo"]], columns=list("ABCDEFG")
)
>       tm.assert_frame_equal(df, expected)
E       AssertionError: Attributes of DataFrame.iloc[:, 0] (column name="A") are different
E       
E       Attribute "dtype" are different
E       [left]:  int32
E       [right]: int64
````

Within these test, the change made causes the setting in iloc to be done here:

https://github.com/pandas-dev/pandas/blob/c6ad13d57c8157cd175a294f06c427f8a239dcea/pandas/core/indexing.py#L1722-L1724

as opposed to the usual path here:

https://github.com/pandas-dev/pandas/blob/c6ad13d57c8157cd175a294f06c427f8a239dcea/pandas/core/indexing.py#L1695-L1705

My best guess is that this is a bug in the former path that is being uncovered by this change, but I am not sure how to fix. Any suggestions or help is much appreciated.

Labels:

Comments:
Needs #34217 to be addressed first. Closing to take off the queue.
BUG: CategoricalIndex.__contains__ incorrect NaTs
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Categorical
description: Categorical Data Type

Comments:
thanks. I suppose could add a whatsnew, this is pretty esoteric though.
TST/REF: Fixturize constant functions in ConsistencyBase
State: closed

This PR fixturizes the last constants (`base_functions` and `no_nan_functions`) in ConsistencyBase, and take corresponding tests out of class. 

After this one, only constants in `Base` class needs to be fixed (if `Base` is also wanted to be cleaned)

cc @jreback 

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Window
description: rolling, ewma, expanding

Comments:
are tests here there any that need marking as slow?
if 5s per test function is acceptable, then we are fine for tests in `expanding` i think

i marked two functions as slow in `rolling` one, one is above which replied to @jbrockmendel comment, that test takes 40s, and the other is `test_rolling_apply_consistency` which is involved in the change in this pr, and this takes ~105s. 

generally tests in rolling takes longer and for the rest of tests in rolling, basically all <15s, and in total ~250s, so maybe it's okay to leave them not as slow? @jreback 

I will double check a bit, but would be nice to get the ballpark for the threshold for slow? 

Update:

other relatively big ones:
`test_rolling_consistency_std`: ~20s
`test_rolling_consistency_var`: ~25s

i also mark them as slow

emm, the error does not seem related.
gentle ping @jreback 
thanks @charlesdong1991 very nice!
TST: add pattern for nearest
State: closed

- [ ] closes #33895
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Timezones
description: Timezone data dtype
name: Resample
description: resample method

Comments:
thanks @kotamatsuoka
TST: raise InvalidIndexError with IntervalIndex.get_value and get_loc
State: closed

ensure InvalidIndexError is raised when non-scalar is passed as key to
get_loc and get_value methods for IntervalIndex

- [X] closes #31117 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Interval
description: Interval data type

Comments:
thanks @prashant616
PERF: Datetimelike lookups
State: closed

cc @TomAugspurger this should address at least some of the regressions you posted about this morning.

Labels:
name: Datetime
description: Datetime data dtype
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance

Comments:
does this link to a specific issue?
> does this link to a specific issue?

Tom opened a handful of perf-regression issues yesterday.  I havent checked which of those this affects, but I think the fastpath in `__getitem__` is a big one.
Thanks!
BUG: Fix small memory access errors in ujson objToJSON.c
State: closed

These were found (and validated fixed) using valgrind. It is likely
that none of these changes should create any issues (except maybe
on a debug build of python?). But silencing valgrind warnings is
good on its own, to ease possible future debugging using valgrind.

----

NOTE: I removed skipping of `test_astype_generic_timestamp_no_frequency`, this is just out of curiosity, I would be very surprised if it made a difference. So this should probably be only merged after undoing the test change.

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: IO JSON
description: read_json, to_json, json_normalize
name: Clean

Comments:
JSON changes look good

Out of curiosity is there a guide you can share for running this with valgrind? I’ve tried unsuccessfully in the past but would be interested in knowing how
There is no secret, in this case I did not even to bother debug builds. The only thing to remember nowadays is to use `PYTHONMALLOC=malloc` as an environment variable, otherwise you will get thousands of false positives (and generally less useful output).
OK, this (unsurprisingly) did not change anything with the test, so removed it from the PR, I guess it should be all set, considering that it is a tiny bug fix really?
Travis fail is unrelated, has been affecting a bunch of tests for the last few days
thanks @seberg
TST/REF: Move static methods out of consistency classes
State: closed

In order to keep such tests refactor PR more readable, in this PR, I only move those methods which are actually staticmethods out of huge consistency classes to be independent tests, so no code changes! And after this PR, the big consistency classes are broken down to much smaller.

This one will pave the way for replacing some other final fixtures & staticmethods out of classes 

cc @jreback 

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Window
description: rolling, ewma, expanding

Comments:
emm, seems that unrelated error is back again. 
thanks, keep em coming!
ENH: print DataFrame columns names at the end
State: closed

- [ ] closes #32296
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

I've added the possibility of printing columns names again at the end of DataFrame. I'm not sure whether the number of rows should be checked again (max number of rows could be exceeded with that extra row).


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
@MarcoGorelli Thanks for your comments. I wasn't sure if `ending_header` is the best name for such a parameter. Is it fine?
I've added whatsnew entry, I hope that it is enough descriptive. Please let me know if it should be improved.
This looks good - can you merge master? I think should resolve the CI issue
> This looks good - can you merge master? I think should resolve the CI issue

@WillAyd Done
@jreback thoughts here?
Do we consider changing the repr to be API breaking, and so would have to
wait for 2.0? I think it probably is.

But in the mean time it's fine to do this as opt-in for `to_string`,
possibly adding a config option for adding it to `__repr__`.

On Tue, Jun 9, 2020 at 5:43 PM Jeff Reback <notifications@github.com> wrote:

> *@jreback* requested changes on this pull request.
>
> this PR is only adding to to_string() which doesn't appear to be very
> useful, the default formatters won't have this and now you have confusing.
> I am ok with this in principle but would need to comprehensively audit the
> display outputs.
>
> cc @jorisvandenbossche <https://github.com/jorisvandenbossche>
> @TomAugspurger <https://github.com/TomAugspurger>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/33914#pullrequestreview-427595174>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKAOIVNLKT6VG4GHHBEBMDRV23IHANCNFSM4MXBXSGA>
> .
>

Hi @JDkuba - is this still active? If so, can you address Jeff's comment about the versionadded tag?
@JDkuba can you merge master and update for comments
Closing as stale (no response in a month and a half), but please ping if you'd like to bring this back up
API: make min/max on empty datetime df consistent with datetime serie…
State: closed

…s (#33704)

- [x] closes #33704
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Fixes the following issue:
```python
import pandas as pd
df = pd.DataFrame(dict(x=pd.to_datetime([])))
df.max()
```
throws a `ValueError` but 
```python
pd.Series(pd.to_datetime([])).max()
```
results in `NaT`.
With this change, calling an DataFrame on en empty `pd.to_datetime([])`, results in
```
In[1]: df = pd.DataFrame(dict(x=pd.to_datetime([])))
In[2]: df.max()
Out[5]: 
x   NaT
dtype: datetime64[ns]
```


Labels:
name: Datetime
description: Datetime data dtype
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Hello @CloseChoice! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-05-07 22:16:22 UTC
> pls add a whatsnew note as well, 1.1 bug fixes datetimelike section

done
thanks @CloseChoice
REF: de-duplicate listlike validation in DTA._validate_foo
State: closed

Implements a helper `_validate_listlike` that is re-used across 4 of the _validate_foo methods.  We can then smooth out the remaining differences by passing the same flags in each of the four usages.

Labels:
name: Refactor
description: Internal refactoring of code

Comments:
looks fine, needs a rebase
rebased+green
gentle ping; follow-ups on deck