BrandonFors/pandas-github-issues huggingface dataset:
Run clang-format on objToJSON
State: closed

POC for #28143

@TomAugspurger may have interest

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Failed due to the reorder of imports. Quite a few options outlined here:

https://stackoverflow.com/a/37999072/621736

The easiest of which is to turn off the sort-includes flag (though there are more ideal alternatives as well)

I guess for now looking for feedback on style and can mess with getting build to work later
Option here was `clang-format -sort-includes=0 -i -style="{IndentWidth: 4}" pandas/_libs/src/ujson/python/objToJSON.c`
No opinion on this.
Cool - I'll probably merge later today just to make the file more readable as I've been looking at it
Sounds good to me.

On Mon, Aug 26, 2019 at 9:29 AM William Ayd <notifications@github.com>
wrote:

> Cool - I'll probably merge later today just to make the file more readable
> as I've been looking at it
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/28144?email_source=notifications&email_token=AAKAOIQ77INXCVS6IU4YCJTQGPSC5A5CNFSM4IPKOOZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5ERHXI#issuecomment-524882909>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIVYUQ6X5DY5SMNACWDQGPSC5ANCNFSM4IPKOOZA>
> .
>
TYPING: --check-untyped-defs for Index.__new__
State: closed

xref #27568

```
pandas\core\indexes\base.py:335: error: Incompatible types in assignment (expression has type "TimedeltaIndex", variable has type "DatetimeIndex")
pandas\core\indexes\base.py:338: error: Incompatible types in assignment (expression has type "TimedeltaIndex", variable has type "DatetimeIndex")
pandas\core\indexes\base.py:346: error: Incompatible types in assignment (expression has type "PeriodIndex", variable has type "DatetimeIndex")
pandas\core\indexes\base.py:408: error: Name 'PeriodIndex' already defined on line 74
pandas\core\indexes\base.py:408: error: Module 'pandas.core.indexes.period' has no attribute 'IncompatibleFrequency'
pandas\core\indexes\base.py:419: error: Name 'Float64Index' already defined on line 349
pandas\core\indexes\base.py:443: error: Name 'Float64Index' already defined on line 349
pandas\core\indexes\base.py:447: error: Name 'IntervalIndex' already defined on line 170
pandas\core\indexes\base.py:459: error: Name 'DatetimeIndex' already defined on line 88
pandas\core\indexes\base.py:470: error: Name 'TimedeltaIndex' already defined on line 57
```


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
LGTM
Thanks @simonjayhawkins
TYPING: --disallow-any-expr for HTMLFormatter.__init__
State: closed

```
pandas\io\formats\html.py:44: error: Expression type contains "Any" (has type "Union[str, List[Any], Tuple[Any, ...], None]")
pandas\io\formats\html.py:49: error: Expression type contains "Any" (has type "Dict[str, Any]")
pandas\io\formats\html.py:49: error: Expression has type "Any"
pandas\io\formats\html.py:50: error: Expression type contains "Any" (has type "Dict[str, Any]")
pandas\io\formats\html.py:50: error: Expression has type "Any"
pandas\io\formats\html.py:54: error: Expression type contains "Any" (has type "Union[int, Any]")
```

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Maybe wait a day or two to see if any other devs would like to comment. Otherwise merge away
can wait. it is `gradual` typing! very gradual.
Thanks @simonjayhawkins
Remove outdated docstring that no longer applies
State: closed

- [X] closes #28136
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


For a change this minor that affects only a comment, not sure if a `whatsnew entry` update is needed?

Labels:
name: Docs
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
Thanks @bmoscon !
@WillAyd thank you!
TYPING: add stubs for _packer and _unpacker
State: closed

xref #28133

```
pandas\io\packers.py:800: error: Base type _Packer becomes "Any" due to an unfollowed import
pandas\io\packers.py:820: error: Base type _Unpacker becomes "Any" due to an unfollowed import
```

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Are there packaging considerations to be made here? Given these modules are deprecated I think we should just ignore them if causing problems
`pd.to_msgpack` and `pd.read_msgpack` will not be deprecated until pandas 1.2. #27103

we will still be accepting bugfixes? if so, type hints should be a benefit. no reason to ignore?
we haven’t really quite decided on the length of this deprecation
but adding types would be fine 
Sounds good. Would we need to do anything with setup.py to get these distributed?
atm we're primarily concerned with code quality? (and reducing the pull request review overhead if mypy can find problems)

making pandas PEP 561 compatible is outside the scope of the PR. https://mypy.readthedocs.io/en/latest/installed_packages.html#making-pep-561-compatible-packages

Cool that's what I was asking about. Yea doesn't need to be solved here but seems like already some people have started using types hints with IDEs, so would be good to ship stubs alongside releases. If you want to open an issue for that discussion would be helpful
personally, my time spent in an IDE is with pandas itself and use notebook for using pandas. so i'll raise the issue but add contributions welcome so that someone using pandas as a library in an IDE can perhaps take it on.
Thanks @simonjayhawkins
TYPING: _pytest.mark.structures.MarkDecorator -> Callable
State: closed

xref #28133

```
pandas\util\_test_decorators.py:106: error: Return type becomes "Any" due to an unfollowed import
pandas\util\_test_decorators.py:120: error: Return type becomes "Any" due to an unfollowed import
```

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @simonjayhawkins
Fix slicer assignment bug
State: closed

- [x] closes #26939 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
LGTM
Nice PR - thanks @RoeiRaz !
DOC: Fix RangeIndex and other docstrings for missing period in summary
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

@datapythonista 

I added the period to the following cases:
pandas.Categorical.dtype: Summary does not end with a period
pandas.merge_ordered: Summary does not end with a period
pandas.bdate_range: Summary does not end with a period
pandas.period_range: Summary does not end with a period
pandas.timedelta_range: Summary does not end with a period
pandas.interval_range: Summary does not end with a period
pandas.util.hash_pandas_object: Summary does not end with a period
pandas.Grouper: Summary does not end with a period
pandas.Index.memory_usage: Summary does not end with a period
pandas.Index.fillna: Summary does not end with a period
pandas.Index.dropna: Summary does not end with a period
pandas.RangeIndex.start: Summary does not end with a period
pandas.RangeIndex.stop: Summary does not end with a period
pandas.RangeIndex.step: Summary does not end with a period

Labels:
name: Docs

Comments:
@datapythonista - have made the requested changes. Thanks
Thanks @bhuvanakundumani 
Thanks @datapythonista @WillAyd.
CLN: change regex try/except to checks
State: closed


Labels:
name: Clean

Comments:
Just a general question - is this a performance gain or just an approach you find more appropriate?
> Just a general question - is this a performance gain or just an approach you find more appropriate?

The latter.  In a lot of places we have try/excepts that obscure what the expected failure mode is, sometimes with real consequences (eg #28122).  This case is pretty benign.
Sounds good thanks for clarifying
BUG: Fixed PandasArray.__setitem__ with str
State: closed

Closes https://github.com/pandas-dev/pandas/issues/28118
Closes https://github.com/pandas-dev/pandas/issues/28150

Labels:
name: Bug
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
LGTM

That said, I'm just noticing that below the changed code PandasArray can change its dtype and pin a new underlying ndarray.  Does that seem sketchy to anyone else?
> PandasArray can change its dtype and pin a new underlying ndarray.

I'm not too bothered by the new underlying ndarray part, since it's private. What part bothers you?

I do notice that `.astype` should probably be with `copy=False`.
> I'm not too bothered by the new underlying ndarray part, since it's private. What part bothers you?

It's liable to cause surprises with view-like semantics.  e.g. I'd expect `parr[:]` or `np.asarray(parr)` to stay in sync with `parr`.  (this discussion probably belongs in a separate issue)
Agreed that this can go in a followup :)
Merging in a few hours.
Fixed the whatsnew. Merging.
BUG: DataFrame.drop raising TypeError when passing empty DatetimeIndex/list
State: closed

- [x] closes #27994
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: DataFrame
description: DataFrame data structure

Comments:
small comments, otherwise looks good
Pushed a new commit fixing the above.
Is this fixing a regression in 0.25? If not, can you move the release note to 1.0.0.rst?
The release note has been moved.
@TomAugspurger Can you check if PR can be merged?
@katsuya-horiuchi can you merge master and address latest comments?
pls merge master
Sorry guys I have been extremely busy recently. I will take another look as soon as possible.
can you merge master and update
Closing as I think this is stale but @katsuya-horiuchi ping if you'd like to continue
BUG: Fix groupby quantile array
State: closed

xref https://github.com/pandas-dev/pandas/pull/28085#issuecomment-524066959

This is a regression in 0.25.2. It was partially fixed in #28085, but there was an issue with the first fix.

Labels:
name: Groupby
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
LGTM
@TomAugspurger thanks for the quick fix and adding the test! 👍 
@WillAyd care to glance at this?

Note that this whole section is a bit hacky, but can be removed entirely once #10710 is fixed.
lgtm - can you add a whatsnew?
Whoops, added.
BUG: SparseDataFrame op incorrectly casting to float
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Sparse
description: Sparse Data Type

Comments:
rebased and whatsnew added
thanks @jbrockmendel
COMPAT: implement visit_Constant for 3.8 compat
State: closed

Closes https://github.com/pandas-dev/pandas/issues/27261

xref https://bugs.python.org/issue32892


Labels:

Comments:
Do we plan to do a 0.25.2, or keep open the option? (basically meaning: do we keep backporting for now?) If so, this would be a candidate I think.
would be +1 on. potential 0.25.2 esp if we can have 3.8 support in it
According to https://www.python.org/dev/peps/pep-0569/, the release date is targeted for October 21.

I'm not sure we'll get 1.0 out by then so a (conservative) 0.25.2 makes sense. Just plan to backport 3.8 fixes and 0.25 regressions?
FYI, I need to read through https://bugs.python.org/issue36917 a bit more to make sure this is a valid fix. So consider this WIP for now.
Once we get to 3.8 as a minimum we could get rid of the other types represented here right? If so maybe worth a comment of some sort
Updated with all the tests that were failing for me locally with 3.8.

Note that we can't test against 3.8 beta3 because a segfault would crash the test workers (I think). That's fixed in https://github.com/python/cpython/pull/15390 which will be in beta 4.
> Once we get to 3.8 as a minimum we could get rid of the other types represented here right? If so maybe worth a comment of some sort

I'm not sure exactly which ones can be removed once 3.8 is our min version. I don't think I understand this code / the changes well enough to say.
Sounds good. Can cross that bridge when we get there
Unless someone else has objections I so go ahead and merge
Bug: merge_asof() cannot use datetime.timedelta as tolerance kwarg 
State: closed

- [x] Fixes issue #28098
- [x] tests added / passed [1/0]
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Timedelta
description: Timedelta data type

Comments:
datetime.timedelta objects do not have nanoseconds like pd.Timedelta

Another check and conversion is necessary to convert datetime.timedelta correctly to nanoseconds similarly to pd.Timedelta.value
@TomAugspurger
The offending test

https://github.com/pandas-dev/pandas/blob/def01cf7bbb5ef8c9bf2e19737ea918e6a76a143/pandas/tests/reshape/merge/test_merge_asof.py#L593-L611
I only see a whatsnew - did you have an actual fix and tests for this?
@WillAyd

See this conversation. I found this bug while writing a fix for a separate issue.

https://github.com/pandas-dev/pandas/pull/27650/files/04c18e9f15358ac2b91631abfaf0f4151cf289cf#r308457291

I was asked to add this test in a separate pull request. 
You are right I have not written a fix, simply investigated the bug.
Oh OK thanks. Let's close this for now though until a fix is in place. Usually prefer to keep our PR queue down to actionable items so can come back to this when there is something to review
Thank you, I'm new at this. I'll reopen when a fix is in place.

Ian

On Tue, Aug 27, 2019, 5:08 PM William Ayd <notifications@github.com> wrote:

> Closed #28100 <https://github.com/pandas-dev/pandas/pull/28100>.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/28100?email_source=notifications&email_token=AICYMWMVCY746UAHQWSXYI3QGWQWZA5CNFSM4IOYKNYKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOTJFXXWQ#event-2588638170>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AICYMWOYWBNCIFAOCTPOLB3QGWQWZANCNFSM4IOYKNYA>
> .
>
PERF: replace with list, closes #28084
State: closed

- [x] closes #28084
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

based on the OP's example:

```
In [2]: df = pd.DataFrame({"A": 0, "B": 0}, index=range(4*10**7))

In [3]: %timeit df.replace([np.inf, -np.inf], np.nan)
5.18 s ± 423 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)    # <-- master
414 ms ± 7.19 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)   # <-- PR

In [4]: %timeit df.replace([np.inf, -np.inf], np.nan, inplace=True)
2.89 s ± 111 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)      # <-- master
69.6 µs ± 4 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <-- PR

In [5]: %timeit df.replace([np.inf, -np.inf, 1], np.nan)
4.88 s ± 228 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)   # <-- master
466 ms ± 11.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  # <-- PR
```


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
good idea, will do
@WillAyd suggestions for how to make asv happy?
Thanks!
Whoops, just noticed this needs a release note: https://github.com/pandas-dev/pandas/pull/28154
ENH: groupby missing data in index
State: closed

- [x] closes #20519
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

`.groupby` intialize "grouper" by _get_grouper_for_level. Base on "code", "level_index" fills with non-NA value if there is NA value. Problem is if there are only NA values, "level_index" can't fill with values. 
So, check "level_index" whether all "level_index" values is NA. then "level_index" are all NA values, fill with NaN


Labels:
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: MultiIndex

Comments:
@jreback could you merge this pr?
Hello @proost! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-10-25 04:59:47 UTC
@jreback or @jbrockmendel any thoughts on this one?
@proost can you fix up merge conflict?
Thanks @proost 
@jreback @WillAyd @jbrockmendel @TomAugspurger 
Thank you for you review and feedback
DOC: Fixes to docstrings formatting
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Hello @realjohnward! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:











There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-08-24 15:09:22 UTC
@datapythonista Here is the new PR I said I'd make in the PR you requested changes on.
How do I resolve the failed checks? @datapythonista 
Thanks @realjohnward
Repeat headings every n-th row in Jupyter-lab notebook
State: closed

- [ ] closes #27563
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Adds a new option in `pd.set_option()` method.

Usage:
```python
# n is an int that represents the interval of rows which the headers are repeated (default = 0, no repetition)
pd.set_option("display.repeating_row_index", n)   
```
Before | After (n = 3)
----------- | -------------
![Imgur](https://i.imgur.com/jPjDAu0.png) | ![Imgur](https://i.imgur.com/uodLczI.png)


@leostayner 

Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: API Design
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
Hello @elisamalzoni! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-08-22 14:55:02 UTC
Hmm, I'm not entirely sure about this. It doesn't mesh well with my idea of an HTML table.

If we proceed, we'd need tests, and we would need to handle it similar to the other options (a parameter that gets plumbed through from DataFrame.to_html down to the formatter, and looked up from the config if not specified)
Yea I think this is a little arbitrary. From the linked issue I get the problem we are trying to solve is that the headers aren't visible while scrolling through. Wouldn't it be possible to just pin the headers then with appropriate CSS?

cc @simonjayhawkins for insights
I do think that this could be out-of-scope for pandas. presumably, it's an issue for any application/library display HTML in the notebook and the issue should be raised there?
@elisamalzoni will look in detail, but on initially glance suspect that this may have issues with a multiIndex because of rowspan. Changes to _write_hierarchical_rows would also be needed. Can you add a test for this?
Thanks all for the input. Given there doesn't appear to be a lot of appetite for this from the maintainers I think going to close, but @elisamalzoni appreciate for sure the effort on this and would welcome that on any other issues out there
DOC: Remove alias for numpy.random.randn from the docs
State: closed

Based on this comment: https://github.com/pandas-dev/pandas/issues/28038#issuecomment-523439376

Supersedes #28068 

Labels:
name: Docs

Comments:
@datapythonista Hi Marc, I've done this
do you mind having both changes in the same PR? thanks @WuraolaOyewusi 
ok @datapythonista 
@WuraolaOyewusi Thanks a lot!
REF: separate bloated test
State: closed

Working on logical ops I'm finding a bunch of internal inconsistencies that will need to be addressed.  Since those are liable to have large diffs, this starts off with refactor that leaves behavior unchanged: simplify the method itself, and split a giant test into well-scoped ones.


Labels:
name: Refactor
description: Internal refactoring of code
name: Testing
description: pandas testing functions or related to the test suite

Comments:
> Perhaps we could clear out some of the TODO's you added in this PR in the PR itself though.

Separate PR(s).  Lots of small steps.
thanks @jbrockmendel
DOC: redirect from_csv search
State: closed

A web search for "DataFrame from CSV" leads to
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.from_csv.html

which is currently a 404, since DataFrame.from_csv was removed. Redirect
it to read_csv.

Normally we shouldn't worry about this, but I suspect this is a common search term.

Labels:
name: Docs

Comments:
Not super familiar with this file but aren't there already redirects for `from_csv` therein? Maybe just need to be updated?
I think those from_csv redirects were from the old API locations to the new
ones.

This is for a removed page.

On Fri, Aug 23, 2019 at 8:05 AM William Ayd <notifications@github.com>
wrote:

> Not super familiar with this file but aren't there already redirects for
> from_csv therein? Maybe just need to be updated?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/28075?email_source=notifications&email_token=AAKAOIVT5GCRF4JK5LILEF3QF7OAXA5CNFSM4ION54L2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5AE2FQ#issuecomment-524307734>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIRS3MHPB5MEUISX6VDQF7OAXANCNFSM4ION54LQ>
> .
>

Cool thanks for confirming
I assume this is for 1.0.0 but merge for whichever milestone you feel best
@TomAugspurger did you want to merge this?
I still need to check if I have the path correct first.

On Thu, Sep 5, 2019 at 10:43 AM William Ayd <notifications@github.com>
wrote:

> @TomAugspurger <https://github.com/TomAugspurger> did you want to merge
> this?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/28075?email_source=notifications&email_token=AAKAOIXI7G3GI2P2HXKILCDQIESJZA5CNFSM4ION54L2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD57TTEQ#issuecomment-528431506>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIRVR6NQH5TVY3EFPUDQIESJZANCNFSM4ION54LQ>
> .
>

I fixed the redirect, and tested locally that this works.