BrandonFors/pandas-github-issues huggingface dataset:
BUG: Fix windowing over read-only arrays
State: closed

- [x] closes #27766
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Window
description: rolling, ewma, expanding

Comments:
https://github.com/pandas-dev/pandas/pull/27736, which touches these files, is scheduled for 1.0. I'd prefer to see this in 0.25.1. :grin: 
thanks @kernc 

very nice putting up a bug report AND a PR!
TYPING: more type hints for io.formats.printing
State: closed

xref #27568

```
$ mypy pandas/io/formats/printing.py --check-untyped-defs --follow-imports skip
pandas\io\formats\printing.py:194: error: "str" has no attribute "decode"; maybe "encode"?
```


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@simonjayhawkins can you merge master?
> @simonjayhawkins can you merge master?

done.
Thanks!
CLN/REF: Remove _try_cast_result, _try_coerce_and_cast_result
State: closed

This finishes getting the post-call casting/coersion out of the Blocks.

Pre-call coercion is still in there, exclusively for TimedeltaBlock and DatetimeBlock, i.e. will not be necessary if/when those are backed by EA.

Labels:
name: Clean

Comments:
thanks @jbrockmendel
CLN: Move base.StringMixin to computations.common
State: closed

``StringMixin`` should be removed, but I can't figure out how to remove it without breaking tests, so this moves ``StringMixin`` to core.computation, which is the only module, where it is currently used.

This makes ``core.base.py`` a bit cleaner.

Labels:
name: Clean

Comments:
Generally +1.  If you were inclined to try to separate out the just-for-`io.pytables` parts I'd offer moral support for that too.
I’ve updated the PR.
thanks @topper-123
CLN: deprivatize names in pd.core.common
State: closed

Some clean-up, as these functions don't need to start with a "_".

Labels:
name: Clean

Comments:
+1 on all of this.

Do are the `any_not_none` and similar still needed?  do they perhaps predate the built-in equivalents?  I haven't timed it, but would expect them to be less performant than `any(comphrension)`
``any`` and ``all`` are quite old, from python2.5, if I remember correctly, so these funcs were not added for compat reasons.

I like having these functions (a bit less repetitive than spelling them out), thought it's not a strong preference either way, as they're not used all that much.
I've made a small update where ``any``/``all`` is used instead of a for loop; I think this is more readable. 
lgtm. merge on green.
BUG: Fix NaT +/- DTA/TDA
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

After this bugfix, we can change all existing uses of `dispatch_to_index_op` in `core.ops.__init__` to use `dispatch_to_extension_op`.  Once we do that, we can collapse ~50 lines of typechecking-dispatching code to all use `dispatch_to_extension_op`

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Timedelta
description: Timedelta data type

Comments:
does this affect anything user facing?
> does this affect anything user facing?

No.
ping on green
Ping
BUG: fix+test PA+all-NaT TDA
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

In conjunction with one more branch coming up, we're going to simplify the tar out of Series arithmetic ops.

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Timedelta
description: Timedelta data type

Comments:
thanks
REF: pandas/core/window.py into multiple files
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Splits `pandas/core/window.py` into more logical division of:

`pandas/core/window/rolling.py`
`pandas/core/window/expanding.py`
`pandas/core/window/ewm.py`
`pandas/core/window/common.py`


Labels:
name: Refactor
description: Internal refactoring of code
name: Window
description: rolling, ewma, expanding

Comments:
Doesn't need to be part of this PR, but could use your input: I've been wondering if it makes sense to put window and resample into groupby/, or otherwise organize in a way to denote that those two modules depend on groupby/
@jbrockmendel definitely. I imagine a lot of the aggregation functions could be collapsed as well if rolling, resample, and groupby lived under the same umbrella.
> Doesn't need to be part of this PR, but could use your input: I've been wondering if it makes sense to put window and resample into groupby/, or otherwise organize in a way to denote that those two modules depend on groupby/

this would be too much currently: each of these is already a tremendous amount of tests and code
nesting more doesn’t help
Note that #27767 was just merged so need to account for that in rebase
yep this needs a rebase, ping on green.
Ping all green.
thanks @mroeschke 

did we have an issue for this? if so pls close it
REF: separate out invalid ops
State: closed

We moved ops.py to `ops.__init__` a while back, still need to get the bulk of it out of `__init__`.  This separates out templated invalid operations, which are the main things that outside modules import  (this helps us move towards getting rid of the `import pandas as pd` in the main file)

Labels:
name: Clean

Comments:
can you rebase
Rebased and green
BUG: fix to_datetime(dti, utc=True)
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


ATM `to_datetime(naive_dti, utc=True)` returns naive incorrect, same for naive `DatetimeArray`

Labels:
name: Datetime
description: Datetime data dtype
name: Timezones
description: Timezone data dtype

Comments:
updated with whatsnew and suggested edits
Thanks @jbrockmendel
REF: define concat classmethods in the appropriate places
State: closed

cc @jorisvandenbossche we briefly discussed at the sprint the idea that `dtype.concat` is a weird place to define these functions.

This PR takes the subset of `dtypes.concat` methods that are a) private and b) equivalent to `klass._concat_same_dtype` for some `klass` and moves the implementation to the appropriate class.

The categorical one is left in place for now because a) it is public and b) it'd be a pretty big move in and of itself.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
> dtypes.concat was meant to hold all the concat things so they are in 1 place.

Totally on board with the principle here.  But `dtypes` is a weird place for what is a collection of _array_ functions.  And in the cases that are being moved here, they don't really share any code, they're just class methods that are being implemented away from the relevant classes.

> Is this pr meant as a precusor to moar consolidation? 

Yes.  union_categorical and concat_categorical are left in place because just-do-the-incorrectly-private-stuff seemed like a good stopping point.  But these will belong in a location that reflects the fact that they are array functions, possibly with some of the other helper functions in arrays.categorical that are used in not-just-Categorical places.
thanks for the expl @jbrockmendel 

I think a good goal then would be to remove completely pandas.core.dtypes.concat and move to appropriate places the remaining.
BUG: fix+test DTA/TDA/PA add/sub Index
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Looking forward to doing a couple of test-cleanup passes.

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Index
description: Related to the Index class or subclasses

Comments:
_data -> array edits made
lgtm ping on green.
ping
thanks!
BUG: fix replace_list
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Timezones
description: Timezone data dtype

Comments:
added whatsnew
lgtm. question.
thanks @jbrockmendel
remove confusing instructions and link.
State: closed

- [ ] closes #27707 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
I'd still prefer to link to the platform-specific instructions.

* https://devguide.python.org/setup/#windows-compiling
* https://devguide.python.org/setup/#macos
* https://devguide.python.org/setup/#unix-compiling

On Fri, Aug 2, 2019 at 11:17 AM Nico Cernek <notifications@github.com>
wrote:

>
>    - closes #27707 <https://github.com/pandas-dev/pandas/issues/27707>
>    - tests added / passed
>    - passes black pandas
>    - passes git diff upstream/master -u -- "*.py" | flake8 --diff
>    - whatsnew entry
>
> ------------------------------
> You can view, comment on, or merge this pull request online at:
>
>   https://github.com/pandas-dev/pandas/pull/27717
> Commit Summary
>
>    - remove confusing instructions and link.
>
> File Changes
>
>    - *M* doc/source/development/contributing.rst
>    <https://github.com/pandas-dev/pandas/pull/27717/files#diff-0> (5)
>
> Patch Links:
>
>    - https://github.com/pandas-dev/pandas/pull/27717.patch
>    - https://github.com/pandas-dev/pandas/pull/27717.diff
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/27717?email_source=notifications&email_token=AAKAOIWM6Z5DD2E6JKRLRWTQCRMX7A5CNFSM4II7F5DKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HDDDSBA>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOISPN7NL3YEFNRX7RA3QCRMX7ANCNFSM4II7F5DA>
> .
>

lgtm. @TomAugspurger this is what you were looking for?
Thanks @ncernek!
pandas.PeriodIndex and pandas.DateTimeIndex docstring quotes fix
State: closed

- [x] closes #27713
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
I think if you run black against the files modified should fix CI
Thanks!
TST: troubleshoot inconsistent xfails
State: closed


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks for taking a look at these - great to get these working again. Took a quick look at a couple builds and noticed they weren't always the same failures. Could split this up into a few PRs if easier to troubleshoot
Hello @jbrockmendel! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:































There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-08-02 23:33:25 UTC
black and flake8 are OK with this diff, pep8speaks must have its own config or a version mismatch?
@WillAyd can I get a "booyah"?
booyah it is
any reason we shouldn't do this for 0.25.1?
> any reason we shouldn't do this for 0.25.1?

Not that I'm aware of
thanks @jbrockmendel
REF/CLN: maybe_downcast_to_dtype
State: closed

Separate out a numeric-only maybe_downcast_to_dtype_numeric from maybe_downcast_to_dtype.  Avoid a giant try/except by checking the correct things up front.

The non-numeric portion of maybe_downcast_to_dtype casts to datetime64, datetime64tz, or PeriodArray.  Following #27683 the next step in cleaning up internals is going to be getting rid of _try_cast_result, which will involve replacing a usage of maybe_cast_to_dtype with the less-broadly-scoped maybe_cast_to_dtype_numeric.

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
thanks, this is pretty dense code, good to simplify a bit.
BUG: partial string indexing with scalar
State: closed

Closes #27516

cc @jreback if you have thoughts on this approach.

Labels:
name: Datetime
description: Datetime data dtype
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
thanks @TomAugspurger 
@jbrockmendel can you create an issue to see if we can remove .is_all_dates entirely?
Linebreak is bleeding into the documentation page
State: closed

https://pandas.pydata.org/pandas-docs/stable/reference/series.html

```html
<table><td>
...
<tr class="row-even"><td><a class="reference internal" href="api/pandas.Series.T.html#pandas.Series.T" title="pandas.Series.T"><code class="xref py py-obj docutils literal notranslate"><span class="pre">Series.T</span></code></a></td>
<td>Return the transpose, which is by</td>
</tr>
...
</td>
</table>
```
And looks weird on https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.T.html#pandas.Series.T

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
I think need to add an explicit continuation and maybe decent to fix while passing code checks 
> I think need to add an explicit continuation and maybe decent to fix while passing code checks

@WillAyd I don't get the explicit continuation

The linter failures are not informative (exit 1)

My line is 74 characters long and the file has longer lines.
Sorry meant escape the line break. 

On phone so didn’t see exact check failure - do you need help with message?
Looks like you need to run black against the module
Anyone know why we need the newlines to begin with?
If I had to guess just matches our standard of 

```python
def func():
    """
    Docstring goes here.
    """
```

Instead of having that all on one line.
Good guess.

This diff from master seems to pass the docstring checks

```diff
diff --git a/pandas/core/base.py b/pandas/core/base.py
index cfa8d2521..38a8bf717 100644
--- a/pandas/core/base.py
+++ b/pandas/core/base.py
@@ -687,8 +687,9 @@ class IndexOpsMixin:
 
     T = property(
         transpose,
-        doc="""\nReturn the transpose, which is by
-                                definition self.\n""",
+        doc="""
+        Return the transpose, which is by definition self.
+        """,
     )
 
     @property
```

@clemens-tolboom do you want to try that?
@TomAugspurger  your suggestion is way better :-)

Done
Looks good! Just as one last check can you see how this renders? We *might* need to wrap the string in dedent with this approach
It looked fine to me locally. 
Thanks!
CLN: rename reduce-->do_reduce
State: closed

The current name makes it difficult to grep for.

Assorted cleanups; added type annotations in a few places.

Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
> I would also change _reduce in ExtensionArray to match

need this too
The `_reduce` is part of the EA interface, so we cannot just rename that.

Besides, that seems like to defeat Brock's purpose to be able to search for a specific function (and thus not having multiple functions or methods with the same name that do a different thing)
We have _reduce defined in a bunch of places, renaming it should be a separate discussion
can you rebase
rebased+green
Thanks!
BUG: Avoid try/except in blocks, fix setitem bug in datetimelike EA
State: closed

The initial impetus for this was avoiding two try/excepts in Block methods (i.e. the diff in internals.blocks).  This uncovered the bug in DTA/TDA, which accounts for the rest of the diff.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Clean

Comments:
does this need rebase? as it included the just merged set indexing patch
> does this need rebase?

should be OK as is
thanks
BUG: Concatenation warning still appears with sort=False
State: closed

- [x] closes #21952 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
thanks @ncernek
BUG: DTA/TDA incorrectly accepting iNaT for setitem
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

First of several related PRs.

Labels:
name: Datetime
description: Datetime data dtype
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
@TomAugspurger the Travis failure here seems to be affecting everything.  The traceback looks like its somewhere in xdist or pytest.  Didn't we recently deal with something really similar?
Yeah, I think pytest 5.x is incompatible with pytest-xdist<=1.28

pytest                    5.0.1                    py36_0
pytest-forked             1.0.2                    py36_0
pytest-mock               1.10.4                   py36_0
pytest-xdist              1.28.0                     py_0


So probably bump pytest-xdist>=1.29 in this env if possible.

On Thu, Aug 1, 2019 at 2:08 PM jbrockmendel <notifications@github.com>
wrote:

> @TomAugspurger <https://github.com/TomAugspurger> the Travis failure here
> seems to be affecting everything. The traceback looks like its somewhere in
> xdist or pytest. Didn't we recently deal with something really similar?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/27699?email_source=notifications&email_token=AAKAOIS2SMCUAC7VNOT42NLQCMYCJA5CNFSM4IITHBW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3LS4RY#issuecomment-517418567>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIUVGH6FUTXTORPDINDQCMYCJANCNFSM4IITHBWQ>
> .
>

green
thanks
DOC: 0.25 fixups
State: closed

* fix tag for contributors
* fix 0.25.1 contributors
* remove orphan

Closes https://github.com/pandas-dev/pandas/issues/27687

Labels:
name: Docs

Comments:
thanks @TomAugspurger 
Contributors section in 0.25.0 notes is still empty, see https://github.com/pandas-dev/pandas/issues/27687#issuecomment-544931258
ENH: Implement weighted rolling var and std
State: closed

- [x] closes #26597
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Used West's online [algorithm](https://dl.acm.org/citation.cfm?id=359153). [Here](http://www.nowozin.net/sebastian/blog/streaming-mean-and-variance-computation.html) is an explanation of the algorithm and link to pdf of the paper. Tested implementation with `win_type="boxcar"` comparing it with the result of `Rolling.std`.

Additionaly, `_get_window` function is split into two. `_get_kwargs` function is used to split kwargs for window function and rolling function (since std and var takes an additional `ddof` argument).

Shared docs for `std` and `var` are moved to be used with `Window` functions


Labels:
name: Window
description: rolling, ewma, expanding

Comments:
@jreback could you review this?
can you merge master and will review
@jreback merged
can you merge master and will have another look
@jreback merged
@ihsansecer I think this is good; can you merge master and fix up conflict?
thanks @ihsansecer nice patch.

can you do a followup to make these nogil (@jbrockmendel recently added most window functions like this, e.g. see roll_sum)
@jreback sure but I have been a bit busy these days. I will do it when I have time.
BUG: fixes formatted value error for missing sheet (#27676)
State: closed

Fixes a formatted message at [io.excel._odfreader line 62](https://github.com/pandas-dev/pandas/blob/master/pandas/io/excel/_odfreader.py#L63)

- [x] closes #27676
- [x] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO Excel
description: read_excel, to_excel

Comments:
Thanks. Could you release note under "Bug fixes" in `doc/source/v0.25.1.rst` and a test ensuring that the sheet name is including the the error message?
You can ignore the codecov failure.
Looks good. Can you check test coverage for this though? Should have one for all excel / odf engines 
Interesting that all of the engines raise differently... @ilogue do you have an idea of effort to align all of these messages? Would ideally like to do here if not too much effort but could also leave to a follow up if non-trivial
@WillAyd by align do you mean catch the exceptions thrown by the other packages and rethrow a common pandas exception? Or ask the maintainers of those packages to change their error message?
Yea I mean raise one common error message from `get_sheet_by_name` in the various excel readers. So for example since xlrd raises an XLRDError would catch and re-raise a ValueError that matches what would happen with openpyxl and odfpy
@ilogue looks like the CI is failing. Do these tests pass for you locally?
No they don't work, because they currently target all engines. So we could 
1. Moved the test back to the engine specific module
2. Implement https://github.com/pandas-dev/pandas/pull/27677#issuecomment-517372330
Sorry didn't mean to close, on my phone
Do you have an opinion on which option is better?
Hello @ilogue! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-10-25 16:02:56 UTC
> Do you have an opinion on which option is better?

Yes I'd say 1. and went ahead and did that. I absolutely agree with @WillAyd taht going forward it would be better to have a uniform way of dealing with these, but think it may be outside of the scope of this PR
Can you merge master and move this to the 1.0.0 whatsnew? Also OK with approach but can you open a follow up to unify messaging across all engines?
@ilogue can you fix up merge conflicts and revert any changes to 0.25.2 whatsnew?
> @ilogue can you fix up merge conflicts and revert any changes to 0.25.2 whatsnew?

@WillAyd done
Great thanks @ilogue