BrandonFors/pandas-github-issues huggingface dataset:
Reallow usecols to reference OOB indices - reverts 25623
State: closed

- [x] closes #27252
reverts #25623

@heckeop 

@gfyoung I know you asked for a FutureWarning to be raised but I didn't want to mess around with the validation_function in place here so just did a simple revert for sake of time / effort


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
@WillAyd : I think it's fine to revert in this PR, but I would investigate adding a warning in a follow-up.
thanks @WillAyd 
may need to reopen the original issue?
BUG: fix+test fillna with non-nano datetime64; closes #27419
State: closed

- [x] closes #27419
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
thanks @jbrockmendel
CLN: fix build warning in c_timestamp.pyx
State: closed

```
pandas/_libs/tslibs/c_timestamp.c: In function ‘__pyx_f_6pandas_5_libs_6tslibs_11c_timestamp_10_Timestamp__get_start_end_field’:
pandas/_libs/tslibs/c_timestamp.c:7257:25: warning: ‘__pyx_t_10’ may be used uninitialized in this function [-Wmaybe-uninitialized]
 7257 |   __pyx_t_5numpy_int8_t __pyx_t_10[1];
      |  
```

Retrying fix dropped from https://github.com/pandas-dev/pandas/pull/27371

a review [comment](https://github.com/pandas-dev/pandas/pull/27371#discussion_r303179721) there disliked reintroducing `ndarray` here because the file had been converted to using memoryviews. `get_start_end_field` returns a view of type `npy_bool` so`out` should therefore be either a memoryview or an ndarray, and bool memoryviews will only be supported in cython 0.30
(see cython/cython/pull/2676).


Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
Haven't closely followed that conversation or checked links so apologies if stupid question but does a `unit8` memory view not work in the meantime? IIRC we've used that in the past instead
No, that's a reasonable question. But no, that doesn't work either because on cython<0.30 bool arrays require an explicit cast in the cdef even for assigning to a `ndarray[npy_bool]`, while memory views just don't work. If you try a memory view (bool or int8_t) what you get  is

```
ValueError: Does not understand character buffer dtype format string ('?')
```

The linked cython PR fixes this in the next release.
It looks like the linked cython issue makes it unnecessary to specify `cast=True` in some ndarray declarations.  We use that in a few places, so once we update to require cython>=0.30.0, feel free to take point on removing that no-longer-necessary usage
ok. There's also the conditional gil support in 0.30 which is needed. Should probably update soon after release.
add some type annotations io/formats/format.py
State: closed

typing not added to all function signatures in io/formats/format.py in this pass to reduce the diff

~~this PR also contains a refactor to simplify the addition of type annotations; `truncate_h` and `tr_col_num` are dependant variables. `truncate_h` can be `True` or `False`. if `True`, `tr_col_num` is an `int` else if `truncate_h` is `False`, `tr_col_num` is `None`~~

~~same applies for `truncate_v` and `tr_row_num`~~


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@WillAyd to reduce the diff and confusion, i've added `tr_row_num` (and `tr_col_num`) back in, but defined `truncate_v` ( and `truncate_h`) using `tr_row_num` (and `tr_col_num`) to appease the static type checking
lgtm. @WillAyd 
Thanks @simonjayhawkins !
thanks @WillAyd for the review.
CLN/REF: Unify Arithmetic Methods
State: closed

2 goals here.

First is to end up with a single implementation for our arithmetic ops.  By implementing the missing methods directly on DTA/PA, we make it so we don't need to special-case inside the `Index.__foo__` methods, so can dispatch to Series more directly.

Second is to get the wrapper defined in_arith_method_SERIES to the point where we can use it block-wise instead of column-wise, to address the recent perf issues.  This makes progress towards that goal by tightening up the allowed types in masked_arith_op

Oh, and we also get rid of a usage of values_from_object, which is adjacent to #27165 and #27167

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Looks quite nice, thanks.

I think @jreback is fixing the numpy dev CI failure.
Didn't look at the actual PR, but just to note: I think we should stop merging such cleaning / refactoring PRs before 0.25 (or already branch). Which goes for most of Brock's PRs (nothing personal, Brock, it is just that you are doing useful stuff that touches existing code ;))
> Didn't look at the actual PR, but just to note: I think we should stop merging such cleaning / refactoring PRs before 0.25 (or already branch). Which goes for most of Brock's PRs (nothing personal, Brock, it is just that you are doing useful stuff that touches existing code ;))

well then we should branch / release. These are + PRs and to be honest if all of the tests pass I don't see a reason to not merge.
> we should stop merging such cleaning / refactoring PRs before 0.25 (or already branch).

+1
thanks @jbrockmendel nice code removal.
CLN: fix compiler warnings in tzconversion.pyx
State: closed

Fixes 
```
pandas/_libs/tslibs/tzconversion.c:3157:21: warning: ‘__pyx_pybuffernd_dst_hours.diminfo[0].strides’ may be used uninitialized in this function [-Wmaybe-uninitialized]
 3157 |   __Pyx_LocalBuf_ND __pyx_pybuffernd_dst_hours;
      |                     ^~~~~~~~~~~~~~~~~~~~~~~~~~
pandas/_libs/tslibs/tzconversion.c:3149:21: warning: ‘__pyx_pybuffernd_ambiguous_array.diminfo[0].strides’ may be used uninitialized in this function [-Wmaybe-uninitialized]
 3149 |   __Pyx_LocalBuf_ND __pyx_pybuffernd_ambiguous_array;

```

Labels:
name: Build
description: Library building on various platforms
name: Clean

Comments:
LGTM 
thanks
CLN: Unify Window._apply_window and Rolling._apply functions
State: closed

- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

`roll_window` function is split into two to enable implementing other weighted functions e.g. var, std (ref #26597)


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Clean

Comments:
@jreback ok kept original function unchanged while calling it in separate functions. need to pass str functions in `_apply_window` also. `_apply` doesn't implement weights
in the spirit of #27409, any objections to renaming to `roll_weighted_window_foo` or some such? it's confusing having `Window`, `Rolling` and `roll_window`, where the latter two can both do ops like sum/mean.
In favour of renaming it to `window_foo` or `weighted_roll_foo`
It seems some network tests have failed
Would this be a good opportunity to include a fix for issue #26462 ? Currently it looks like the `win_type` parameter is ignored in a `df.goupby(...).rolling(..., win_type=xyz)` aggregation
@ihsansecer pls merge master and update when you can, ping on green.
> Would this be a good opportunity to include a fix for issue #26462 ? Currently it looks like the `win_type` parameter is ignored in a `df.goupby(...).rolling(..., win_type=xyz)` aggregation

@Connossor we try to keep 1 PR to 1 thing; this is a refactor only.
disallow np.timedelta64 in is_integer
State: closed

xref #27347

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Ci fail false positive @jreback @TomAugspurger 
thanks @jbrockmendel 

what's up with the codecov ? I thought we turned that off?
We changed it from leaving comments to a code check (I think at the
sprint). Haven't had time to diagnose why it's acting strangely.

On Mon, Jul 15, 2019 at 1:16 PM Jeff Reback <notifications@github.com>
wrote:

> thanks @jbrockmendel <https://github.com/jbrockmendel>
>
> what's up with the codecov ? I thought we turned that off?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/27401?email_source=notifications&email_token=AAKAOIQC34VU2QY4TKBKURDP7S5GTA5CNFSM4IDYTB4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ6Q6RY#issuecomment-511512391>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIT3KFUO2DKN4Y47CTDP7S5GTANCNFSM4IDYTB4A>
> .
>
Initial implementation of rolling iterators
State: closed

- [x] closes #11704
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Hi, I have yet to implements tests, use flake8 and create a whatsnew entry as I am uncertain whether this implementation would be correct.

I would really appreciate it if someone could give me some feedback on the code.

This PR will implement iterators for window functions.

Some questions I have:

* Does DataFrame.rolling(x) yield DataFrames?
* Does Series.rolling(x) yield Series?

Labels:
name: Enhancement
name: Window
description: rolling, ewma, expanding

Comments:
Hello @ThomasKluiters! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-07-21 11:48:05 UTC
Can you add tests for this?
Also I think yes to both of your questions
> Can you add tests for this?

I wouldn't mind adding tests! I had to know what the expected output would be, but now that I know I'll be writing them! Thanks :)
@mroeschke I have changed the files according to your feedback, could you have a look?
> Can you add `expanding` to the tests here as well?

They're already there, view the changes under test_expanding :)
@ThomasKluiters thanks - sorry misread that on my end
> Looking good. Could you also add tests where `min_periods` is utilized?

@mroeschke 
Expanding kind of already does this, but I have added three new unit tests that use window and min_periods, please check!
For the `min_periods` tests, could you also test where there's `nan`s in the Series/DataFrames?
> For the `min_periods` tests, could you also test where there's `nan`s in the Series/DataFrames?

Well, it turns out that this required a bit of work - I've added some tests for it and had to change some code in window.py
@jreback 

I have moved the tests to `test_iterator` and added the reference to the issue at the top.
I have moved the whatsnew to 1.0.0.
I have added the documentation to computation.rst

Though, I can't seem to figure out how to reference computation.rst in the Whatsnew, could you help me with this please :)? Thanks!

can you merge master and see if can get passing
IIRC this looked reasonable, can you merge master and update to any comments
closing this as stale, though its a nice change, needs a rebase and comments addressed.
ENH: Infer integer-na in infer_dtype
State: closed

- [x] closes #27283
- [x] tests added/passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

As mentioned in #27335, this PR serves as an infrastructure for future changes in the default int type


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
@gfyoung  Tests added
@jreback Green now
@gfyoung ok here?
thanks @jiangyue12392 keep em coming! you are tackling non-trivial issues and making very nice PRs!
TYPING: pandas/core/window.py
State: closed

- [x] tests passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Includes simplifying some code paths

Labels:
name: Window
description: rolling, ewma, expanding
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks for reviewing @WillAyd!
in groupby sorting index when length of groupings is one
State: closed

#### Reference Issues/PRs
Example: Fixes #27369  "Bug: groupby(..., observed=True) doesn't respect sort keys"


#### What does this implement/fix? Explain your changes.
Sorts the index when the length of groupings is one (if `sefl.sorted` equals `True`)




Labels:
name: Groupby
name: Categorical
description: Categorical Data Type

Comments:
Can you add a test for the behavior you are fixing?
@camilaagw can you add a test and merge master?
I think this is stale but @camilaagw feel free to ping if you'd like to pick back up
PERF: significantly improve performance of MultiIndex.shape
State: closed

`MultiIndex.shape` is currently extremely slow as it triggers the creation of `._values`, which can be quite expensive for datetime levels. The one mitigating factor is that this result is cached and thus making `._values.shape` near-instant on subsequent calls, but also hard to catch in `asv` benchmarks; this commit adds a suite dedicated to measuring such cached properties on `Index` objects.

`asv` results show a `~400,000x` speedup for a relatively straightforward case:
```
       before           after         ratio
     [269d3681]       [d205acf6]
     <master>       <shape>   
-      3.52±0.02s       8.33±0.2μs     0.00  index_cached_properties.MultiIndexCached.time_shape
```

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
Updated `asv` results show moving into `Index` benefits a few other classes significantly as well:
```
       before           after         ratio
     [a4c19e7a]       [3c946017]
     <unsorted_cats~1>       <shape>   
-     2.59±0.07μs      2.16±0.06μs     0.83  index_cached_properties.IndexCache.time_shape('PeriodIndex')
-     2.74±0.09μs       2.26±0.1μs     0.83  index_cached_properties.IndexCache.time_shape('DatetimeIndex')
-      5.06±0.2μs       3.57±0.2μs     0.70  index_cached_properties.IndexCache.time_shape('UInt64Index')
-      5.80±0.4μs       3.70±0.3μs     0.64  index_cached_properties.IndexCache.time_shape('Float64Index')
-      6.40±0.4μs       4.08±0.3μs     0.64  index_cached_properties.IndexCache.time_shape('TimedeltaIndex')
-      6.80±0.3μs       3.88±0.2μs     0.57  index_cached_properties.IndexCache.time_shape('IntervalIndex')
-        65.2±1μs         903±20ns     0.01  index_cached_properties.IndexCache.time_shape('Int64Index')
-      65.1±0.9μs         892±10ns     0.01  index_cached_properties.IndexCache.time_shape('RangeIndex')
-         214±2ms       4.45±0.2μs     0.00  index_cached_properties.IndexCache.time_shape('MultiIndex')
```
ENH: Validation to only allow positive integers for options
State: closed

- [x] closes #23348
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

I looked at several different options for providing this functionality.  In the end it seemed best to create a a new function that returns a validation function to only allow positive integers.  So now we have is_pos_int available along side the existing is_int and is_text function.  display.max_rows has been modified to use this new validation.  I went through the other integer based display options and tested each with a negative int to see what else might be broken.  display.precision and display.max_columns both had issues when set to a negative number prior to displaying a dataframe.  They have been changed as well.

Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
py37_np_dev test is failing  with the error below.  Not sure why.  I notice this is happening on other pull requests as well:

```
File "/home/vsts/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pytest_nunit/plugin.py", line 276, in <genexpr> list(case for case in self.cases.values() if case["outcome"] == "passed") KeyError:  outcome'
```
Is there anything else I need to do on this one? It still shows as Changes requested, but those changes have been integrated into later commits.  I just want to make sure nobody is waiting on me for this one.
lgtm. over to you @WillAyd 
thanks @Adam-Klaum !
DOC: document optional arguments on plot submethods
State: closed

- [x] closes #11042 
- [ ] passes `black pandas`
- [ ] Signatures and doc strings for plot sub-methods opt-args added.


Labels:
name: Docs

Comments:
Is there an issue for this?
> Is there an issue for this

#11042
@hancar can you fix up merge conflicts and address comment above?
> @hancar can you fix up merge conflicts and address comment above?

Oh, thank you for reminding, today I will find some time.
Hello @hancar! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/plotting/_core.py`](https://github.com/pandas-dev/pandas/blob/002fc51536147df6d2b2361b6fec95eba9af4ca2/pandas/plotting/_core.py):

> [Line 1336:89](https://github.com/pandas-dev/pandas/blob/002fc51536147df6d2b2361b6fec95eba9af4ca2/pandas/plotting/_core.py#L1336): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (89 > 88 characters)
> [Line 1558:89](https://github.com/pandas-dev/pandas/blob/002fc51536147df6d2b2361b6fec95eba9af4ca2/pandas/plotting/_core.py#L1558): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (89 > 88 characters)



##### Comment last updated at 2019-11-19 08:22:52 UTC
Seems like some test and code checks are failing. Let us know if you need help looking into them.
> Seems like some test and code checks are failing. Let us know if you need help looking into them.

Yes please, now I would really appreciate some help. It seems like the tests fail on some linting or docstring validation, but I didn't manage to rerun the tests locally to find the problem. This is from `Checks/12_Testing docstring validation script.txt` in the logs:

```
2019-10-10T22:31:39.8687326Z scripts/tests/test_validate_docstrings.py ..............................................x..x..............................................docstring1: err desc
2019-10-10T22:31:39.8688602Z docstring1: err desc
2019-10-10T22:31:39.8689131Z docstring1: err desc
2019-10-10T22:31:39.8689537Z docstring2: err desc
2019-10-10T22:31:39.8689947Z docstring2: err desc
2019-10-10T22:31:39.8717964Z ..EXECUTED
2019-10-10T22:31:39.8732189Z {"docstring1": {"errors": [["ER01", "err desc"], ["ER02", "err desc"], ["ER03", "err desc"]]}, "docstring2": {"errors": [["ER04", "err desc"], ["ER05", "err desc"]]}}.Series.foo: err desc
2019-10-10T22:31:39.8732941Z DataFrame.bar: err desc
2019-10-10T22:31:39.8733428Z Series.foobar: err desc
2019-10-10T22:31:39.8733849Z Series.foo: err desc
2019-10-10T22:31:39.8746350Z .
```
Can you give me some hint, please?
Thanks for the PR @hancar - mind merging master and addressing @WillAyd comments whenever you have time- thanks!
@hancar looks like some CI issues can you take a look?
Closing as stale, @hancar if you want to finish this, please let us know to reopen.
arrays/period: allow parsing of PeriodDtype columns from read_csv
State: closed

Fixes: https://github.com/pandas-dev/pandas/issues/26934

Signed-off-by: Antonio Gutierrez <chibby0ne@gmail.com>

- [x] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: IO CSV
description: read_csv, to_csv
name: Period
description: Period data type

Comments:
Hello @chibby0ne! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-07-17 01:10:07 UTC
> tests

Done! Thank you for the swift reply ;)!
Can someone explain how the codecov/project failed? It looks as if the coverage in all the files dropped, even though I only changed 2 files. 
> Can someone explain how the codecov/project failed? 

Happens sometimes, seems random.  Rebase and push again and it usually fixes itself
> > Can someone explain how the codecov/project failed?
> 
> Happens sometimes, seems random. Rebase and push again and it usually fixes itself

Weird. Thanks for the suggestion though ;)
thanks @chibby0ne 
> thanks @chibby0ne

Thank you for your support @jreback @TomAugspurger @jbrockmendel @WillAyd !
TST: add NaN tests for all data types (#12535)
State: closed

- [x] closes #12535
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Window
description: rolling, ewma, expanding

Comments:
Any interest in converting this to a fixture(s)? 
@WillAyd yes, I might try to do that, but not right now, though.
OK thanks. This is in real need of it so any contribution there would be *greatly* appreciated.

I think this looks OK otherwise though @jreback 
i’ll look soon 
@tq0 can you use paramterize here instead; this is getting long and is non-idiomatic.
I'd prefer parametreziation be done separately. The diff here is relatively
easy to understand.

On Mon, Jul 15, 2019 at 7:23 AM Jeff Reback <notifications@github.com>
wrote:

> *@jreback* requested changes on this pull request.
>
> comments
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/27378?email_source=notifications&email_token=AAKAOISL4JI7QXKOKLKNDWDP7RT55A5CNFSM4IC4DRBKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOB6NID5Y#pullrequestreview-261784055>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIQ5EH2Y6VGFQ2AXQ7LP7RT55ANCNFSM4IC4DRBA>
> .
>

thanks @tq0 if you can follow up with a PR to parameterize would be great.
Fix compilation warnings
State: closed

- [x] tests passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff` 

Labels:
name: Clean

Comments:
One of the fixes introduces a failing test on CI which I can't reproduce locally on either py37 or py35. Removed it for now.
One more fix added.
@willayd, ready.