BrandonFors/pandas-github-issues huggingface dataset:
REF: use inherit_names for TimedeltaIndex
State: closed

ATM we have two separate ways of doing inheritance/pass-through for our EA-backed Indexes.  This moves TDI over to all-inherit_names,  Will do the same for other EA-Indexes in their own PRs.

Labels:
name: Refactor
description: Internal refactoring of code
name: Timedelta
description: Timedelta data type

Comments:
Can you explain a bit what's the difference between the two mechanisms? (the delegate_names vs inherit_names)
> Can you explain a bit what's the difference between the two mechanisms? (the delegate_names vs inherit_names)

1) inherit_names has a kwarg that allows us to inherit a property as a cache_readonly
2) inherit_names works for class attributes, i.e. not properties or methods
3) inherit_names lets us to avoid repeating boilerplate of PandasDelegate
thanks @jbrockmendel 
> inherit_names works for class attributes, i.e. not properties or methods

What do you mean exactly? In this PR it's being used for methods? (eg sum, median, etc)

> What do you mean exactly? In this PR it's being used for methods? (eg sum, median, etc)

e.g. We use inherit names on DatetimeIndex to inherit _bool_ops, which is just a list
REF: do all convert_tolerance casting inside Index.get_loc
State: closed

As a bonus we get better exception messages in the affected tests.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Timedelta
description: Timedelta data type

Comments:
is this actually a bug fix? e.g.the error message before was ints and not its (correctly) timestamps.?
> e.g.the error message before was ints and not its (correctly) timestamps.?

I guess so, sure
lgtm ex merge conflict
rebased+green
added whatsnew for improved exception messages, rebased+green
ok!
PERF: postpone imports in Index constructor
State: closed

xref https://github.com/pandas-dev/pandas/issues/30790, it's for improving this one mentioned in that issue:

```
arr_str = np.array(["foo", "bar", "baz"], dtype=object)
%timeit pd.Index(arr_str)
```

Only importing the index classes when we actually need them gives a 15-20% speedup.

Last week when profiling the Index constructor, I noticed that those imports actually take quite a bit of time (and it's a change that was introduced after 0.25, https://github.com/pandas-dev/pandas/pull/28141). I didn't get any further with profiling (this alone doesn't explain the full slowdown), but thought it's an easy one to already do anyway.

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
cc @simonjayhawkins 
`%timeit` already runs it several times, so that should be OK (but also repeating that multiple times confirms it) 
I have to say I was also surprised, I would have thought this to be cheaper, but apparently getting those from the import cache also takes time ..


LGTM

> but apparently getting those from the import cache also takes time ..

This is surprising and a good catch.  I'd be curious whether there's any perf difference between

```
from pandas.core.indexes.numeric import Float64Index
from .numeric import Float64Index
from pandas import Float64Index
```
Sanitize column names
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Is there an open issue you can link this to?

EDIT: just noticed you opened #31417. Could you put that in the "closes #xxxx" part?
https://github.com/pandas-dev/pandas/issues/31417
so, should I make it df.columns.str.sanitize_column_names() ???
not averse to a very welll defined convenience function that is composed of other accessors; .sanitize() might be ok
where are the source files for ``df.columns.str.*`` ??
so that maybe I can change the method from df.sanitize_column_names() to df.columns.str.sanitize() as "jreback" suggested.

If anyone has any suggestions or changes to the above said plan, feel free to comment

EDIT:
but if it is a .str method, then you can use it on not just columns but on everything (like Series) which is not the idea.
Fix bug in calculation of holidays
State: closed

Closes #31415

- [x] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype
name: Frequency
description: DateOffsets

Comments:
Hello @dhirschfeld! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-02-07 04:22:19 UTC
Can you add tests for the behavior this fixes
CI is passing! :tada:
code change looks fine. I think you need to isort the files you change. btw we have a pre-commit hook for this (pip install precommit turns it on).

merge master and ping on green.
I rebased on master but I think I'm now getting an unrelated failure bout `libindex`:
```
black --version
black, version 19.10b0
Checking black formatting
All done! ✨ 🍰 ✨
941 files would be left unchanged.
Checking black formatting DONE
flake8 --version
3.7.9 (flake8-comprehensions: 3.2.2, mccabe: 0.6.1, pycodestyle: 2.5.0, pyflakes: 2.1.1) CPython 3.7.6 on Linux
Linting .py code
##[error]./pandas/core/series.py:1025:9:F821:undefined name 'libindex'
##[error]./pandas/core/series.py:1108:17:F821:undefined name 'libindex'
Linting .py code DONE
```
Possibly related to #31506?
ping @jreback - rebased on (the latest) master and the tests are passing! :tada:
ping @jreback! 
thanks @dhirschfeld very nice
CLN: Replace '.format' with f-strings in some test files #29547
State: closed

Updates the below 2 test files to use f-strings (fast and preferred) instead of '.format' (old and slow)

1) pandas/tests/reshape/test_reshape.py
2) pandas/tests/scalar/period/test_period.py
 
For more details on the issue, refer: https://github.com/pandas-dev/pandas/issues/29547

- [x] tests passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
LGTM
It is my first contribution to the pandas library and I am all excited :)
Thanks for the review @jbrockmendel and thanks for adding the label @MomIsBestFriend.
@vandana-iyer Thank you for the PR :)
thanks @vandana-iyer
PERF/CLN: float-to-int casting
State: closed

using float.is_integer is apparently much faster than casting and checking equality, especially for values between int32max and int64max (anecdotal based on small sample size)

```
cast1 = com.cast_scalar_indexer_master
cast2 = com.cast_scalar_indexer_PR

def cast3(key):
    # Effectively what we have in _maybe_cast_indexer in master
    if lib.is_float(key):
        try:
            ckey = int(key)
            if ckey == key:
                key = ckey
        except (OverflowError, ValueError, TypeError):
            pass
    return key

key1 = np.float64(17179869184.0)
key2 = float(key1) * 2**30             
key3 = np.float64(np.iinfo(np.int64).max)

In [87]: for func in [cast1, cast2, cast3]: 
    ...:     for key in [key1, key2, key3]: 
    ...:                 %timeit func(key) 
    ...:                                                                                                                                                                                                                                          

647 ns ± 3.71 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
559 ns ± 27.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
1.98 µs ± 19.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

360 ns ± 4.1 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
353 ns ± 3.11 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
377 ns ± 6.54 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

568 ns ± 3.92 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
445 ns ± 15 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
1.91 µs ± 36.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
```

The middle three timeit results are the ones we get from this PR.

Labels:
name: Performance
description: Memory or execution speed performance
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
There should be a similar case in `arrays/integer.py` that you can update, I think
more cases in pandas/core/dtypes/cast.py no?
> more cases in pandas/core/dtypes/cast.py no?

Just did a pass through the file, didnt find any.

> also cast_scalar_indexer really belongs in pandas/core/dtypes/cast.py as well (common.py is a hodge podge)

Probably.  I've got a lot of refactor-esque PRs/branches going, will move this after the current batch.
lgtm. needs a rebase.
rebased+green
thanks
REF: consolidate DTI/TDI/PI get_value in ExtensionIndex, associated cleanups
State: closed

@jschendel does the holds_integer change for IntervalIndex make sense?

@jreback is this a correct usage of self.holds_integer() as meaning "we only fall back to positional indexing if not self.holds_integer()"

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
can you rebase
rebased+green
thanks
Add pandas blog to feed
State: closed

I don't think this needs to be backported.

Labels:
name: Web
description: pandas website

Comments:
FYI @datapythonista 
i get a 404 when i visit the new URL.  is that what this PR fixes?
Looks good, I guess you're setting up blog.pandas.io as the domain for the blog? I think the web build will fail if we merge this before the new domain is working. 

There was some discussion where some people wanted to keep using the old domain and discard pandas.io,not sure if a final decision was made. 
Oh, that's the URL from the footer on http://dev.pandas.io/pandas-blog/.

So the right one at the current URL would be http://dev.pandas.io/pandas-blog/feeds/all.atom.xml.
The npdev failure is fixed on master / 1.0.x.
PERF: avoid is_bool_indexer check where possible
State: closed

xref #30349, not a lot of ground to pick up here, but we can avoid a few calls

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance

Comments:
> does this make any difference or just moving code around?

This should not change any behavior.

> prefer you just do a return rather than assign and return

I like doing it in two lines inside try/excepts so i can see in coverage measurements whether the call is ever successful
can you rebase; i am not sure if @jorisvandenbossche had more comments.
Just my question from above: Can you show some timings?
> Can you show some timings?

It's pretty small (especially since is_scalar and is_iterator were recently optimized)

```
In [1]: import pandas as pd 
   ...: import numpy as np 
   ...:  
   ...: ser = pd.Series(range(10**5)) 
   ...:  
   ...: indexer = np.random.random(10**5).astype(bool) 
   ...:                                                                                                                                                                                                                                           

In [2]: %timeit ser[indexer]                                                                                                                                                                                                                      
220 µs ± 3.93 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <-- master
213 µs ± 9.02 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <-- PR
```
And with a scalar indexer?
for the scalar case I expect the difference to be zero, or even slower by the amount of a `isinstance(self.index, MultiIndex)` check

```
In [4]: %timeit ser[999]                                                        
1.85 µs ± 81.1 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- master
1.82 µs ± 57.6 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- PR
```
> or even slower 

That's what I expected as well.

(in general, if you do changes specifically for performance, it's helpful for reviewing that you show some relevant timings)
@jorisvandenbossche pls make sure labels and versions on the issue and PR are good before merging
GH31391 - Fixed typos and fixed inexistant path in a SAS class comment
State: closed

Simple change: just fixed some typos and incorrect path as reported in #31391 

Labels:

Comments:
Thanks, @jeandersonbc !

I don't think we would typically update an old whatsnew note, could you please revert that one to how it is on the master branch?
> I don't think we would typically update an old whatsnew note,

@TomAugspurger whats the policy on this?
I don’t think we have one. What would you recommend?

> On Jan 28, 2020, at 18:33, jbrockmendel <notifications@github.com> wrote:
> 
> ﻿
> I don't think we would typically update an old whatsnew note,
> 
> @TomAugspurger whats the policy on this?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

I think for legitimate typos we're OK retroactively fixing.
thanks @jeandersonbc
ENH: truncate output of Groupby.groups
State: closed

Resurrection of #24853. Have resolved conflicts and updated according to the last review it received. The original PR fixed some unrelated minor typos, so I've kept them in (and added a fix for another absolutely minor typo I'd noticed)

- [x] closes #1135
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Enhancement
name: Groupby
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
thanks @MarcoGorelli
Raise error in read_csv when arguments header and prefix both are not None
State: closed

- [x] closes #27394
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO CSV
description: read_csv, to_csv

Comments:
All checks successful.
No, in my view it doesn't need depreciation cycle.

And I have made all the requested changes @simonjayhawkins 
> No, it doesn't need depreciation cycle.

IIUC then currently `prefix` is ignored if `header` passed. 

```python
>>> from io import StringIO
>>> s = StringIO("0,1\n2,3")
>>>
>>> import pandas as pd
>>> pd.__version__
'1.0.0rc0+228.g4edcc5541'
>>>
>>> pd.read_csv(s, header=0, prefix="_X")
   0  1
0  2  3
>>>
```

This would now raise a ValueError.

IMO this is a breaking change. see what others think.
>IMO this is a breaking change

oh, I think you are right.
Hello @rushabh-v! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-02-03 09:25:48 UTC
@gfyoung 2 tests are failing, I don't see any obvious connection of them with this PR. 
can you take a look, please? https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=27452&view=logs&j=a3a13ea8-7cf0-5bdb-71bb-6ac8830ae35c&t=add65f64-6c25-5783-8fd6-d9aa1b63d9d4&l=119 
Indeed, those test failures look unrelated.  I just restarted the build.
>Indeed, those test failures look unrelated. I just restarted the build.

same test failure again.
You may need to rebase or merge.  Unclear why these warnings are popping up.

cc @pandas-dev/pandas-core : is there something with `matplotlib` that changed recently?
Fails after merge too
All greens now @gfyoung 
Thanks @rushabh-v !
DEPR: Deprecate core
State: closed

- [x] closes #27522 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

I haven't actually performed the refactor(pandas.core -> pandas._core) yet so tests are failing because of the warnings. 


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
I think I'd prefer doing this as a DeprecationWarning before elevating to a FutureWarning.

That will give downstream libraries a chance to report the pieces of functionality they need from `pandas.core` a chance to report and fix the warnings before their users see it.
TYP: check_untyped_defs core.reshape.reshape
State: closed

pandas\core\reshape\reshape.py:378: error: "Index" has no attribute "levels"; maybe "nlevels"?
pandas\core\reshape\reshape.py:381: error: "Index" has no attribute "codes"
pandas\core\reshape\reshape.py:383: error: "Index" has no attribute "codes"
pandas\core\reshape\reshape.py:436: error: Incompatible types in assignment (expression has type "_Unstacker", variable has type "partial[_Unstacker]")      
pandas\core\reshape\reshape.py:444: error: "partial[_Unstacker]" has no attribute "get_result"
pandas\core\reshape\reshape.py:1000: error: Incompatible types in assignment (expression has type "None", variable has type "Index")
pandas\core\reshape\reshape.py:1009: error: Incompatible types in assignment (expression has type "float", variable has type "int")
pandas\core\reshape\reshape.py:1013: error: Need type annotation for 'sp_indices'

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks
BUG/REG: multiindex nested tuples with duplicates
State: closed

- [x] closes #30892
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

there is an underlying problem with MultiIndex.get_loc that needs to get worked out, but that's going to take a while.  This fixes the reported regression.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
thanks @jbrockmendel 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.0.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 54a6490ed050a693f04c740af2dddbd2fd19c4e1
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #31358: BUG/REG: multiindex nested tuples with duplicates'
```

4. Push to a named branch :

```
git push YOURFORK 1.0.x:auto-backport-of-pr-31358-on-1.0.x
```

5. Create a PR against branch 1.0.x, I would have named this PR:

> "Backport PR #31358 on branch 1.0.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@jbrockmendel can you push a backport for this, didn't work automatically.
ENH: `Styler.highlight_null` can accepts `subset` argument
State: closed

As proposed in #31345 and as a follow-up PR for [#29118](https://github.com/pandas-dev/pandas/pull/29118#discussion_r340733887)

- [x] closes #31345
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Styler
description: conditional formatting using DataFrame.style

Comments:
Won't have time near-term to look.

On Tue, Mar 3, 2020 at 7:24 PM William Ayd <notifications@github.com> wrote:

> *@WillAyd* requested changes on this pull request.
>
> minor nit on annotation otherwise lgtm
>
> @TomAugspurger <https://github.com/TomAugspurger> care to look?
> ------------------------------
>
> In pandas/io/formats/style.py
> <https://github.com/pandas-dev/pandas/pull/31350#discussion_r387398995>:
>
> > @@ -1003,19 +1003,23 @@ def hide_columns(self, subset) -> "Styler":
>
>      def _highlight_null(v, null_color: str) -> str:
>
>          return f"background-color: {null_color}" if pd.isna(v) else ""
>
>
>
> -    def highlight_null(self, null_color: str = "red") -> "Styler":
>
> +    def highlight_null(self, null_color: str = "red", subset=None) -> "Styler":
>
>
> ⬇️ Suggested change
>
> -    def highlight_null(self, null_color: str = "red", subset=None) -> "Styler":
>
> +    def highlight_null(self, null_color: str = "red", subset: Optional[Union[Label, Sequence[Label]]]=None) -> "Styler":
>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/31350?email_source=notifications&email_token=AAKAOIVIA555AZZ3BF32TA3RFWUUFA5CNFSM4KMDQDK2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCX3AUCA#pullrequestreview-368445960>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKAOIXD3STROVYY3QN3RKLRFWUUFANCNFSM4KMDQDKQ>
> .
>

Thanks, it's ok now.
Thanks @immaxchen nice PR!
REF: share code for set-like ops in DTI/TDI/PI
State: closed

The DatetimeIndexOpsMixin._concat_same_type this has can probably become the ExtensionIndex._concat_same_type default.  I expect IntervalIndex can be updated to look the same pretty easily, not so sure about CategoricalIndex based on a quick look.

Labels:
name: Refactor
description: Internal refactoring of code
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
can you merge master here and i'll look again
k some followup requests
DEPR: row-based indexing in DataFrame.__getitem__
State: closed

Discussed recently in #9595.  The two tests this edits are the only two that go down this path AFAICT.



Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Deprecate
description: Functionality to remove in pandas
name: Mothballed
description: Temporarily-closed PR the author plans to return to

Comments:
@jorisvandenbossche discussion notwithstanding, can i get your help in troubleshooting the doc build?  i put in "okwarnings" to fix the previous set of failures, not clear on why dft_minute is now undefined
Repeating what I said above again: please open an issue first to discuss this.
iirc there is an issue about this (though woudl have to search)
The issue Brock mentioned is #9595, but that is much broader (about ``__getitem`` in general), so if that is the one you are thinking about, I would propose to create a new, specific issue for this topic
> The issue Brock mentioned is #9595, but that is much broader (about `__getitem` in general), so if that is the one you are thinking about, I would propose to create a new, specific issue for this topic

i believe there is a specific issue this was discussed long ago
> can you give examples of what user-facing behavior is being deprecated? I'd be in favor of deprecating fallback indexing for partial string slicing dataframes with datetime indexes

Exactly the example you gave.  The relevant cases are where we have all of the following:
- a DataFrame `df`
- `df.index` is either `DatetimeIndex` or `PeriodIndex`
- `key` is a date-like string
- `key not in df.columns`

(these conditions are basically psuedo-code for `convert_to_index_sliceable`)
closing to clear the queue, will reivisit