BrandonFors/pandas-github-issues huggingface dataset:
Added get_train_test_stratified_by_time_and_col
State: closed

A lot of time we want to split dataframe into train and test by time and make sure it's stratified by a key. I suggest **get_train_test_stratified_by_time_and_col** which does this. 

**stratify_split_col_name**: The column name which has to be split in train and test in a stratified way.
**time_sort_col_name**: The column name used for sorting rows by time.
**test_size**: Test size
**ascending:** For sorting time_sort_col_name by ascending or descending.

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Hello @bhavsarpratik! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/core/groupby/groupby.py`](https://github.com/pandas-dev/pandas/blob/56ebc73ad6fe39095eb590fc6c6eeae090c0abf7/pandas/core/groupby/groupby.py):

> [Line 2480:1](https://github.com/pandas-dev/pandas/blob/56ebc73ad6fe39095eb590fc6c6eeae090c0abf7/pandas/core/groupby/groupby.py#L2480): [E302](https://duckduckgo.com/?q=pep8%20E302) expected 2 blank lines, found 1
> [Line 2493:42](https://github.com/pandas-dev/pandas/blob/56ebc73ad6fe39095eb590fc6c6eeae090c0abf7/pandas/core/groupby/groupby.py#L2493): [E226](https://duckduckgo.com/?q=pep8%20E226) missing whitespace around arithmetic operator


you should open an issue to discuss adding API before a PR
we have rejected splitting apis like this is past as they are out of scope for pandas
For context, see https://github.com/pandas-dev/pandas/pull/6687 for some previous discussion about this
TST: parametrize sparse array arithmetic tests
State: closed

Motivated by a tedious troubleshooting process in a different branch.

I'm pretty sure we can take this even further, saving it for another pass since the diff here is already pretty big.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Sparse
description: Sparse Data Type

Comments:
Codecov fail looks like false positive
any deal-breakers here?  I'd like to get this in before trying to troubleshoot the (hopefully) last division-by-zero branch
thanks @jbrockmendel nice!
REF: Avoid dispatching Series ops to pd.Index
State: closed

We have some semi-circular dispatch logic between non-EA Series and pd.Index ops.  This avoids that circularity by implementing the relevant logic directly in the Series implementation.  After this I'm going to start migrating the Index implementations to use the Series implementation, after which we will have One True Implementation (also have to migrate RangeIndex and IntegerArray implementations)

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Timedelta
description: Timedelta data type
name: Clean

Comments:
thanks
DOC: Clarify column type for 'on' parameter of rolling
State: closed

- [x] closes #21687


Labels:
name: Docs
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Might be reading it wrong but not sure about the referenced issue. Commented there
@WillAyd I believe he is confused if an integer column can be used while calculating rolling window
thanks @ihsansecer
COMPAT: catch InvalidIndexError in base Indexer getitem
State: closed

Closes https://github.com/pandas-dev/pandas/issues/27258

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Dependencies
description: Required and optional dependencies

Comments:
thanks @jorisvandenbossche though would avoid using the internals in geopandas :->
I'd still like a comment here about why we are now catching InvalidIndexerError.  Until recently this was catching Exception and I tracked down what actually gets raised, would prefer not to have to do that again in a few months.
@jbrockmendel yes, fully agree, was planning to do that so will do a follow up PR
REF: ops.missing
State: closed

- Move arithmetic-centric functions from core.missing to core.ops.missing.
- module-level docstring giving the motivation for these functions
- tiny optimization:

```
if op in [operator.floordiv, operator.truediv, getattr(operator, "div", None)]:
    result = mask_zero_div_zero(left, right, result)
```

becomes

```
if op is operator.floordiv:
    result = mask_zero_div_zero(left, right, result)
```

because in py3 numpy behaves the way we want for truediv.

Labels:
name: Refactor
description: Internal refactoring of code
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
thanks @jbrockmendel
BUG: Enable changing dtype of column where column names are not unique
State: closed

- [ ] closes #22035, partially #24798
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
@rok is this still active?
@WillAyd yeah, I still plan to do this but have to make time. If I'm blocking anything please let me know. :)
closing as stale, though we'd still like this when you can comeback; ping to re-open.
Fix _table_mod is 'NoneType' issue
State: closed

- [x] closes #27253 
- [ ] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Currently in get_node() it is not guaranteed that _table_mod is not none, and this breaks some packages that depend on Pandas, forcing use of older versions of Pandas. I added `_tables()` to fix this issue
see #27253 for more details

Labels:
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
closing; suclasses are not following the api of HDFStore; would take a patch with proper tests. (though I don't see any good reason to actually subclass HDFStore).
REF: rename _data-->_mgr
State: closed

At the sprint there was discussion of trimming down the exposed surface of internals.  Among the things that makes this difficult is identifying exactly what that surface is.  This makes it feasiable to grep for all accesses by using the unambiguous `._mgr` instead of the overloaded `._data`.

Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
> what is the upside of changing this?

Discussed in the OP: it is hard to tell right now where BlockManager is accessed across the code-base (most of which we want to remove).  With this, you can find it immediately by grepping.
> > what is the upside of changing this?
> 
> Discussed in the OP: it is hard to tell right now where BlockManager is accessed across the code-base (most of which we want to remove). With this, you can find it immediately by grepping.

I am not sure this is a positive return given the churn on other packages.
TST: add tests to validate margin results for pivot (#25815)
State: closed

- [x ] closes #25815
- [x ] tests added / passed
- [x ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Categorical
description: Categorical Data Type

Comments:
see #27233 for details on black code style to resolve the CI failure.
BUG: merge_asof with multiple by columns with tz
State: closed

- [x] closes #26649
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Timezones
description: Timezone data dtype

Comments:
thanks @mroeschke
BUG: Fix divmod fill value, closes #26987
State: closed

- [x] closes #26987
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Takes over from #27130 by implementing dispatch_fill_zeros.  This is unfortunately similar to dispatch_missing, and eventually the two functions will be merged.  ATM I'm trying to do exactly that in the branch for 27130, having lots of trouble pop up with sparse and IntegerArray.

Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
this supersedes #27130 right?
yes, but I'm keeping that open because that branch has evolved into addressing other bugs
thanks @jbrockmendel
CLN: remove never-used branch from Series ops
State: closed

_Almost_ gets to a point where Series and DataFrame ops can share the na_op definition.

AFAICT this is the only non-test usage of libalgos.arrmap_object.

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
So what's the conclusion re arrmap_*?
pandas/core/algorithms.py
```
        q = np.asarray(q, np.float64)
        return algos.arrmap_float64(q, _get_score)
```

I think you can just do ``np.vectorize`` on this and it will work, if so, then let's completely blow away arrmap
I'll open an issue to look into this more carefully.
cool, ping on green.
ping
thanks @jbrockmendel
Adding fastpath and name parameters to docstring for Series class and others. 
State: closed

- [x] closes #27178
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

I modified the docstring for the Series class to include the "fastpath" and "name" parameters.  I searched the code base, and also modified the docstrings for the following classes in a similar way:

Categorial
CategoricalIndex
Grouping
Index
MultiIndex
PeriodIndex
RangeIndex

It appears that fastpath is definitely used internally, and is not deprecated.  There were no other examples of fastpath being documented in the parameters section of a docstring, so I went with this convention:

fastpath : bool, default False
        Internal use only

Labels:
name: Docs

Comments:
Thanks for the PR! I'm not sure about documenting this more. The note is pretty clear that it's *internal only* I think this generates more questions than it solves.

Would the alternate just to be to remove it from all documentation?
Will, happy to help!

I thought about removing it entirely, as that would probably be the clearest message to the user.  However, it looks like Sphinx is using autosummary to build the first part of the documentation and is querying the class' __init__ method to do so.  So as long as fastpath is a valid parameter to the constructor, it will show in the documentation.  

I'm new to Sphinx so I'm not sure if there is an easy way to override this behavior to hide certain parameters.  So I think our options are:

1. Do as I have done in the pull request to at least let users know to leave the fastpath parameter alone.

2. Do some Sphinx wizardry so that it ignores the fastpath parameter when building the doc. 

3. Eliminate the fastpath methodology from the code base.

I think I understand why the fastpath functionality was originally used internally.  There are way more sanity checks and exception handlers used when fastpath=False.  fastpath=True assumes that the caller has done their homework and isn't passing in garbage.  Makes sense for an internally used function.  

I'm open to any of the three options,  what do you think?
I _think_ I've identified the only place where Series fastpath is used and plan to get rid of it.
I only found one use of it for Series in this block of code in pandas/core/frame.py

```python
    def _box_col_values(self, values, items):
        """
        Provide boxed values for a column.
        """
        klass = self._constructor_sliced
        return klass(values, index=self.index, name=items, fastpath=True)
```
It sounds like the direction is to purge fastpath from the code base.  I'm going to go ahead and close this request in lieu of future pull requests that propose code changes to remove fastpath from classes other than Series.  
As long as `fastpath` is not removed (and I don't see any immediate motion on that), it's better to document it as "Internal use only" than having people wonder what this is for and maybe even use it.

So I think this is still a valuable addition.
DOC: update contributing guidelines for black
State: closed

Closes https://github.com/pandas-dev/pandas/issues/27226 
Starting this by copying some content from dask (https://github.com/dask/dask/pull/4983)

Also related to https://github.com/pandas-dev/pandas/issues/23616 (recommending pre-commit hooks)



Labels:
name: Docs

Comments:
ok lgtm.
thanks @jorisvandenbossche
REF: separate indexer utilities from indexing.py
State: closed

A couple things going on here.

- Take simple, low-dependency, not-used-in-indexing.py functions out of indexing.py and put them in indexers.py.
  - A couple of runtime-defined functions in internals get moved here too.
  - Avoids some runtime imports (i.e. simplify dependency structure)
  - Fixes some incorrect private names
  - Adds typing
- Simplify some indexing.py methods
  - Avoid duplicated validation calls
  - Add typing
  - De-duplicate _get_slice_axis method
  - Move code outside of try/except blocks to clarify what the failure modes are.

Medium-term goals in indexing.py
  - Isolate internals access
  - refactor _setitem_with_indexer (over 300 lines)

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
note as a post or pre-cursor to this I would form pandas/core/indexing/ and move indexing.py (and indexers.py) there as well
> note as a post or pre-cursor to this I would form pandas/core/indexing/ and move indexing.py (and indexers.py) there as well

Sure.  Obvious preference for post-cursor.
Hello @jbrockmendel! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-07-08 13:51:16 UTC
Added one more docstring, left un-docstringed is_scalar_indexer and is_empty_indexer because I'm not totally clear on what those functions are doing (possibly an extra indentation in is_scalar_indexer?  it is _almost_ identical to the last 5 lines of is_empty_indexer)

Added a note about InvalidIndexError being for geopandas.
thanks. in followups we need to move / consolidate these things; also a typing pass would be helpful.
CLN: Move code outside of try/except blocks
State: closed

Trying to make it clearer exactly what the failure modes being caught are.

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
Travis fail unrelated
lgtm. ping on green.
ping
thanks
Didn't yet look in detail, but it seems this broke geopandas: https://travis-ci.org/jorisvandenbossche/geopandas/jobs/554786189 (at least it started failing the last two hours, and is related to a get_value)
Opened an issue: https://github.com/pandas-dev/pandas/issues/27258
BUG: Fixup bench for deprecation
State: closed

Closes #27205 

Haven't reproduced the tz one yet. Will see if it shows up here.


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
fixes?
> fixes?

What do you mean?

> Haven't reproduced the tz one yet. Will see if it shows up here.

The one in the issue may have been a random failure. It's not showing up here.
ok great.
Misleading error for pd.read_msgpack
State: closed

- [x] closes #27160 
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry



Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
I'm not really sure why it fails on Linting
We just merged in a PR to use black as a code formatting tool. If you install that and run locally against pandas/io/packers.py and pandas/tests/io/test_packers.py I think should resolve
Hello @U09Kane! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-07-05 22:08:47 UTC
Hopefully, everything is good now.
thanks sticking it out @U09Kane 

nice patch!
Thanks for the feedback
DEPR: make Categorical.ravel() return Categorical
State: closed

Discussed in #27142.

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Labels:
name: Categorical
description: Categorical Data Type
name: Deprecate
description: Functionality to remove in pandas

Comments:
conflict and failing

```
pandas/tests/extension/test_categorical.py(100,): error :  Found unwanted pattern:         with pytest.warns(FutureWarning, match="will return a Categorical")
```
otherwise lgtm.
I am a little concerned by the change in testing pattern this introduces. Could we remove this for now.
testing pattern reverted
lgtm. merge on green.
thanks @jbrockmendel
DEPR: Deprecate outer ufunc in Series.__array_ufunc__
State: closed

Closes https://github.com/pandas-dev/pandas/issues/27186

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
[Travis finished](https://travis-ci.org/pandas-dev/pandas/builds/553709108?utm_source=github_status&utm_medium=notification), GitHub just isn't showing it in the UI yet.
TST: handle inconsistent ordering in resample_api compat test
State: closed


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
should fix the 3.5 occasionally failing tests
> LGTM. Are we having pytest 5 issues? I can take a look.

yep, I pinned < 5.0 on the 3.6 builds (already pinned on 3.5); so I think this is ok.

will merge and rebase other that looks ready.
CLN: fix cython build warning
State: closed

```
warning: pandas/_libs/tslib.pyx:724:49: Buffer unpacking not optimized away.
warning: pandas/_libs/tslib.pyx:724:49: Buffer unpacking not optimized away.
```

the error is due to a cdef inline function which uses ndarray[object]. The function has one caller, and it isn't in any perf-critical inner-loop, so the inline isn't important. Remove it and avoid warning during cythonization.


Labels:
name: Build
description: Library building on various platforms

Comments:
you don't need to rebase things; they are squashed on merge; also pls don't force push generally as it makes reviewing harder (doesn't matter here).
improve test for timestamp addition/subtraction
State: closed

This PR adds regression test cases for the addition/subtraction of time deltas to timestamps. In fact, this is causing issues in 0.24.2 when `freq="M"` and `freq="W"`, and the problem is solved in `master`.

- [ ] closes #27182 
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Frequency
description: DateOffsets

Comments:
thanks @lostella
CLN: simplify MultiIndex._shallow_copy
State: closed

``MultiIndex._shallow_copy``  doesnt pass on kwargs, unless ``values`` is passed also, but just silently drops the kwargs. This makes ``_shallow_copy`` currently a bit unreliable.

This PR fixes that problem by calling ``.copy`` directly and passing kwargs to ``copy``and not passing the call through ``.view``. Also, ``_shallow_copy`` doesn't call ``.view`` on other index types, so this PR makes ``MultiIndex._shallow_copy`` conform with other index types.

Labels:
name: MultiIndex
name: Clean

Comments:
this looks ok. does this close any current issues?

does this have any user facing effects?
No, there's no issue for this, as this is small & internal. For an upcoming PR I need to call ``mi._shallow_copy(names=...)``, so made a separate PR for this, instead of mixing it in the other PR.

This has no user-facing effects.
> No, there's no issue for this, as this is small & internal. For an upcoming PR I need to call `mi._shallow_copy(names=...)`, so made a separate PR for this, instead of mixing it in the other PR.
> 
> This has no user-facing effects.

k thanks
thanks @topper-123
ERR: Raise on duplicates names in read_csv
State: closed

- [x] xref #6581
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO CSV
description: read_csv, to_csv
name: Clean

Comments:
thanks @mroeschke 
@mroeschke should the entry for this in #6581 be moved to #13777?  (also the docstring for `_validate_names` is now inaccurate) 
@jbrockmendel Good catch. Looks like I forgot to move it to #13777 after this PR was merged.
which version did this make it into?
Looks like 0.25.0
BUG: Fix Index constructor with mixed closed Intervals
State: closed

- [X] closes #27172
- [X] tests added / passed
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry

Note that this solution uses the same pattern that's used to support a list of mixed frequency `Period` objects:
https://github.com/pandas-dev/pandas/blob/527e714647a16ad31f80ea535af5a63156515861/pandas/core/indexes/base.py#L444-L448

Labels:
name: Bug
name: Interval
description: Interval data type
name: Index
description: Related to the Index class or subclasses

Comments:
lgtm merge one green
thanks @jschendel
CLN: convert argument in .take method
State: closed

- [x] xref #6581
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Clean

Comments:
so i think we can collapse _take back into take; it was split out to be the internal take to accomdate this deprecation. can be a separate PR if u want (make an issue though)
Created https://github.com/pandas-dev/pandas/issues/27174 for the follow up consolidation.