BrandonFors/pandas-github-issues huggingface dataset:
REF: implement indexes.extension to share delegation
State: closed

In following steps, I plan to

- remove the DatetimeDelegateMixin entirely and just use inherit_names
- de-duplicate the slightly-different comparison method code from CategoricalIndex vs DatetimelikeIndex
- move the remaining wrapping utilities from datetimelike to extension.py, after double-checking if we can remove any layers
- delegate more methods, some of which depends on smoothing out small idiosyncrasies, e.g. #30627.

Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Index
description: Related to the Index class or subclasses

Comments:
k merge on green.
green
thanks
DOC: Add strings for dtypes in basic.rst
State: closed

- [x] closes #30590
- [ ] tests added / passed
    - N/A
- [ ] passes `black pandas`
    - N/A
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
    - N/A
- [ ] whatsnew entry
    - N/A

Decided to add all valid strings for all the types in the table about dtypes.  I didn't want to decide which ones to leave out, and if we want to leave some of them out, we should decide whether they should be removed from the code as well (e.g., `'Sparse[int, 0]'`)

Had to reformat the table so it would look nice, by splitting the list of strings into multiple lines (e.g., a merged cell)



Labels:
name: Docs
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Thanks @Dr-Irv, nice addition
REF: delegate more IntervalIndex methods
State: closed

cc @jschendel is the usage of _simple_new instead of _shallow_copy `take` important?  Should it always be the case that `idx.close == idx._data.closed`?

Labels:
name: Refactor
description: Internal refactoring of code
name: Interval
description: Interval data type

Comments:
> is the usage of `_simple_new` instead of `_shallow_copy` in `take` important?

I don't think so.

> should it always be the case that `idx.closed == idx._data.closed`?

Yes.
thanks @jbrockmendel
TST: Adding test to concat where copy=False for ExtensionArrays
State: closed

The test ensures that ExtensionArrays are correctly constructed when
concat(copy=False) is used.

- [x] closes #20756
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
thanks @dwhu
TYP: enable strict_equality to prohibit comparisons of non-overlappin…
State: closed

…g types

By default, mypy allows always-false comparisons like 42 == 'no'. Use this flag to prohibit such comparisons of non-overlapping types, and similar identity and container checks
```
pandas\core\dtypes\base.py:239: error: Non-overlapping equality check (left operand type: "str", right operand type: "Callable[[ExtensionDtype], str]")      
pandas\core\arrays\period.py:657: error: Non-overlapping container check (element type: "Callable[[Any], Any]", container item type: "Callable[[Any, Any], Any]")
pandas\core\arrays\period.py:658: error: Non-overlapping identity check (left operand type: "Callable[[Any], Any]", right operand type: "Callable[[Any, Any], Any]")
pandas\io\formats\format.py:307: error: Non-overlapping equality check (left operand type: "bool", right operand type: "Literal['truncate']")
pandas\io\formats\format.py:461: error: Non-overlapping equality check (left operand type: "bool", right operand type: "Literal['truncate']")
pandas\io\excel\_odfreader.py:159: error: Non-overlapping equality check (left operand type: "str", right operand type: "float")
```

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks @simonjayhawkins 
Master seems to be failing now with `pandas/core/generic.py:464: error: Non-overlapping identity check (left operand type: "str", right operand type: "Type[int]")` which I think might be a result of this. Giving it a look
TYP: Add TypeVars to NDFrame
State: closed

Adding TypeVars to NDFrame methods that don't return optional values. This ensures that e.g. ``(DataFrame|Series).astype`` and ``(DataFrame|Series).copy`` have a known return type, which is nice.

Also adds ``PandasObject._ensure_type``, which is a method used to solve the problem with optional return values, but where we actually know the return type. This is a helper method to help with the tediousness of optional return values experienced in #30565.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks @topper-123
CLN: replacing str.format with f-strings in several files. #29547
State: closed

- [ ] xref #29547 
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks for the clean up @jlamborn324
Improve ASV Environment Creation Performance
State: closed

...by leveraging a lot of the recent work to enable parallel CI

Overriding default values documented here:

https://asv.readthedocs.io/en/stable/asv.conf.json.html#build-command-install-command-uninstall-command

I've picked 4 as the value assuming most people running benchmarks will have 2-4 cores. YMMV 

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
LGTM.  any ballpark on the perf imrpovement?
It varies depending on hardware, with physical cores I think being the big thing. My quad core desktop can build clean in about 20 seconds. Duo core laptop Takes 2-3 minutes. Non parallel I think around 8-10 minutes

Whatever the savings are multiply by two for the ASV environment creation

Sent from my iPhone

> On Jan 1, 2020, at 9:27 PM, jbrockmendel <notifications@github.com> wrote:
> 
> ﻿
> LGTM. any ballpark on the perf imrpovement?
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.

Thanks! Will report how it does on the benchmark machine.
REF: delegate attrs for CategoricalIndex, IntervalIndex
State: closed

Working towards creating an ExtensionIndexMixin to be shared by all our backed-be-EA indexes, which would ideally grow into ExtensionIndex.

Removes CategoricalIndex._codes_for_groupby, which would raise AttributeError if it were ever called.

Labels:
name: Refactor
description: Internal refactoring of code
name: Categorical
description: Categorical Data Type
name: Interval
description: Interval data type

Comments:
thanks @jbrockmendel comments above, but i think you have already noted them
CLN: Replace old format strings to f-strings in pandas/tests/base
State: closed

- [x] Contributes to #29547 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
I am unable to for the life in my run pytest on the repository and by this I mean that it doesn't work for the unchanged `master` branch. That means that I have been unable to run it on my changes.
If anyone can help me as to where I'm going wrong I'd appreciate it. 
thanks @rbharadwaj9 

pls make sure your environment is setup and you have compiled the extensions
this might help @rbharadwaj9 if following through:

https://pandas.pydata.org/pandas-docs/stable/development/contributing.html
I got it. Thanks for the help!
Added 'pearson' to methods list in pandas/core/nanops.py
State: closed

- [x] ref https://github.com/pandas-dev/pandas/pull/30461#discussion_r362323260
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Closed&Reopened because the build failed during compile time.

Checking if that was a false positive.
thanks @MomIsBestFriend
TST: Add test for TypeError when using datetime.time in scatter plot
State: closed

- [ ] closes #8113 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Visualization
description: plotting
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
@jreback ping
thanks !
CLN: Use fstring instead of .format in io/excel and test/generic
State: closed

- [x] Contributes to #29547 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @Bharat123rox
BUG: Ensure df.itertuples() uses plain tuples correctly
State: closed

Currently DataFrame.itertuples() has an off by one error
when it inspects whether or not it should return namedtuples
or plain tuples in it's response.

This PR addresses that bug by correcting the condition
that is used when making the check.

Closes: #28282

- [x] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
thanks @simongibbons very nice!
CLN: Remove int32 and float32 dtypes from IntervalTree
State: closed

There isn't a practical way to actually get an `IntervalTree` with `int32`/`float32` dtypes since `IntervalIndex` is backed by two pandas indexes and we don't have a `Int32Index` or `Float32Index`. These indexes are used to create the underlying `IntervalTree` that backs an `IntervalIndex`, so we're guaranteed to have `int64`/`float64` dtype data when initializing.

The only way that comes to mind that a user could create an `IntervalTree` with `int32`/`float32` dtype would be by explicitly initializing a standalone `IntervalTree` with `int32`/`float32` arrays, which doesn't seem particularly likely.

This should also help with build times as it results in 8 less node classes being generated.

Didn't add a whatsnew note since I don't think `IntervalTree` is user facing (could be wrong?) but can add one if desired since this is technically a breaking change.

Labels:
name: Build
description: Library building on various platforms
name: Interval
description: Interval data type
name: Clean

Comments:
thanks @jschendel
REF: share join methods for DTI/TDI
State: closed

The only actual changed behavior is in the new `_is_convertible_to_index_for_join` method where it first checks:

```
if isinstance(other, cls):
    return False
```

In master, TimedeltaIndex uses _is_convertible_to_index which returns True here.  The DTI method behavior is unchanged.

Labels:
name: Refactor
description: Internal refactoring of code
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
In a follow-up, there will be more set methods we can share.
ping
BUG: DTA/TDA/PA add/sub object-dtype
State: closed

Bonus: we get to get rid of is_offsetlike

Labels:
name: Datetime
description: Datetime data dtype
name: Timedelta
description: Timedelta data type
name: Period
description: Period data type

Comments:
> is this a bug fix? e.g. is there an issue

There is no associated issue.  I stumbled on this when implementing block-wise arithmetic for the non-scalar cases.
ping
k thanks.
REF: separate casting out of Index.__new__
State: closed

first of two PRs to separate array casting/inference out of `Index.__new__`.  Once both are in place, we'll be able to do all inference/casting up-front and simplify the constructor quite a bit.  We'll also be able to look into sharing code between Index/Series/array, and address a handful of outstanding issues with the Index constructor.

Labels:
name: Refactor
description: Internal refactoring of code
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
thanks
BUG: Disable parallel cythonize on Windows (GH 30214)
State: closed

- [x ] closes #30356
- [ ] tests added / passed
  - N/A
- [ ] passes `black pandas`
  - Didn't run - changes too many files
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Added test in `setup.py` to check if nthreads is positive and on Windows.


Labels:
name: Build
description: Library building on various platforms
name: Windows
description: Windows OS

Comments:
> Can you also enable this for Windows CI? Would be curious to see how much time it saves there

@WillAyd If I "enable" it for Windows CI, then all compiles would run sequentially.  So that wouldn't save time (assuming the parallel stuff is working now). What am I missing here?
I meant should add the “-j N” flag to the build command for Windows in the CI folder. We currently don’t pass anything so Cythonization and compilation are sequential

Sent from my iPhone

> On Dec 31, 2019, at 4:04 PM, Irv Lustig <notifications@github.com> wrote:
> 
> ﻿
> Can you also enable this for Windows CI? Would be curious to see how much time it saves there
> 
> @WillAyd If I "enable" it for Windows CI, then all compiles would run sequentially. So that wouldn't save time (assuming the parallel stuff is working now). What am I missing here?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

> I meant should add the “-j N” flag to the build command for Windows in the CI folder. We currently don’t pass anything so Cythonization and compilation are sequential

OK, I put that in the last commit
thanks @Dr-Irv
BUG: Series __setitem__ gives wrong result with bool indexer
State: closed

Series.__setitem__ gives wrong result with bool indexer and when length of new data matches the number of Trues and new data is neither a Series nor a numpy array

- [x] closes #30567
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
thanks @fujiaxiang very nice
@fujiaxiang, thank you so much for this!
ENH: Add ignore_index to sort_index
State: closed

- [x] closes #30114
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


@jreback 
I just looked back at #30114 and found out that i overlooked to add `ignore_index` to `sort_index`, and this might be the reason this issue is still left open. I added this to `sort_index` to close this issue.

Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design

Comments:
Lgtm
thanks! will do @jreback 
Hi,`ignore_index = True` works  just fine and meets my expectations when `axis =0`,but when `axis = 1`, the number of index after reset using `ignore = Ture` is equal to the number of columns instead of the original index number,like `new_index = range(len(columns))`.
Is this a bug, or where am I going wrong?
THKS!
CLN: Clean _test_moments_consistency in common.py
State: closed

- [ ] xref #30486 #30542 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean

Comments:
thanks @charlesdong1991 

ok now that you have cleaned up some of this. what I *really* want is to break up these giant consistency tests into much smaller tests. Love for you to have a go at this.

Also using fixtures for creating (rather than the setup methods) is a plus
moving helper functions out of the classes would be good; just make free functions.

just do small pieces incrementally. thanks!
API: Raise when setting name via level
State: closed

Closes https://github.com/pandas-dev/pandas/issues/29032

This is extremely ugly, but gets the job done. It's probably worth doing to avoid silently failing, but not sure.

cc @topper-123.

Labels:
name: MultiIndex

Comments:
I think it has to be on all index subclasses since they make up the levels of a MultiIndex. 
@TomAugspurger , but if it's set on ``Index`` all index subclasses inherit the attribute?
Right, since it’s the level of the MI that can’t have its name set, not the MI itself. 
My apologies, I've simplified things a bit. I misunderstood what happened to class attributes when the attribute is set on an instance.
> This will be removed later, when we feel confident that people have switched idiom, right?

I'm not sure. We can weigh the maintenance cost down the road and remove it if it's burdensome.
I'm fine with this as-is.

Spitballing another solution: we could have Index._names be an ndarray for all Index subclasses, with

```
@property
def name(self):
    return self._names[0]     

@name.setter
def name(self, value):
    if np.is_view_or_whatever(self._names):
        raise RuntimeError(...)
    self._names[0] = value
```
this looks fine, butI would consider something slightly different. We could have an attribute defined on Index._parent = None

then set a weakref when it is contained in a MultiIndex. This gives us provenance on where things are coming from. might be useful.
Mmm, that's also reasonable. My initial concern is that indexes are potentially shared, so they could have multiple parents? For this purpose it's not a problem since we're creating the Index from scratch, just something to keep in mind going forward.
> Mmm, that's also reasonable. My initial concern is that indexes are potentially shared, so they could have multiple parents? For this purpose it's not a problem since we're creating the Index from scratch, just something to keep in mind going forward.

I don’t think we share for a contained Index inside a MultiIndex but not really sure
You're right, these levels of a MI are created on the fly.

In theory, a user-action could end up with an Index that's created from the level of a MI

```python
s = pd.Series(index=idx.levels[0])
```

but that probably doesn't affect your proposed "parent" reference.

That implementation is going to be more complex though, since subclasses don't call `Index._simple_new`. Thoughts on waiting till we need it?

yeah was just an idea i think what u did here is ok
let me have another glance 
thanks @TomAugspurger
TYP: check_untyped_defs io.json._normalize
State: closed

pandas\io\json\_normalize.py:282: error: Argument 1 to "zip" has incompatible type "Union[str, List[Any], None]"; expected "Iterable[Any]"
pandas\io\json\_normalize.py:299: error: Argument 1 to "zip" has incompatible type "Union[str, List[Any], None]"; expected "Iterable[Any]"

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
lgtm
TYP: check_untyped_defs various
State: closed

pandas\core\dtypes\generic.py:7: error: 'classmethod' used with a non-method
pandas\core\config_init.py:310: error: Name 'get_ipython' is not defined
pandas\core\indexes\category.py:588: error: Need type annotation for 'missing' (hint: "missing: List[<type>] = ...")

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
LGTM
Thanks!
PERF: Fixed performance regression in Series init
State: closed

Closes https://github.com/pandas-dev/pandas/issues/30564

PR

```
[ 50.00%] ··· series_methods.SeriesConstructor.time_constructor                                                                                                                   ok
[ 50.00%] ··· ====== ==========
               data
              ------ ----------
               None   670±0μs
               dict   84.4±0ms
              ====== ==========


```

master

```
[ 50.00%] ··· series_methods.SeriesConstructor.time_constructor                                                                                                                   ok
[ 50.00%] ··· ====== =========
               data
              ------ ---------
               None   756±0μs
               dict   725±0ms
              ====== =========
```

No whatsnew, since the regression was only on master.

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
Also pushed a change to avoid calling `construct_from_string` from `ExtensionDtype.is_dtype` for non-string inputs in the first place.
Lgtm
Thanks @TomAugspurger
CLN: refactor tests in test_moments_ewm.py
State: closed

- [ ] xref: #30486 #30542 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Window
description: rolling, ewma, expanding

Comments:
thanks @charlesdong1991 
this scope is good. moving common functions out and simplifiying logic & parmeterizing. keep em coming here.
thanks, will do @jreback
BUG: Change IntervalDtype.kind from None to "O"
State: closed

- [X] closes #30568
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry

This is partially blocking #28399 as tests are failing there due to `kind` being `None`.

cc @TomAugspurger 


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Interval
description: Interval data type

Comments:
could add a test to the interface tests requiring that dtype.kind be a 1-character str?
There's an existing test for `dtype.kind` that allows for it to be `None` but otherwise checks that it's one of the valid single character string values:

https://github.com/pandas-dev/pandas/blob/fd7db9819b8c7dba86b2887bee33f670b2715afc/pandas/tests/extension/base/dtype.py#L17-L20

I removed the `if` block and all the tests are passing for me locally, so I've pushed the change here too.  Not sure if there's a specific reason it was written that way and we'd want to allow `None` though, so can revert if we want to still allow it.  Given that `None` can cause things to break I think we'd want to disallow it though.
Thanks!
CLN: Clean test moments for expanding
State: closed

- xref #30486 #30542 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Window
description: rolling, ewma, expanding

Comments:
Hello, @jreback and @mroeschke 

This is the first follow-up PR for #30542 , and I plan to clean it in three steps in order to keep PR small and readable:

1. this PR will focus on `expanding`
2. the second PR will focus on `ewm`
3. the last PR will focus on `_test_moments_consistency` which is in `common.py` and this will probably influence all three files since they all inherit from this (`expanding`, `ewm` and `rolling`)

If you have any other suggestions on how to proceed, feel free to bring up!
thanks @charlesdong1991 this is good.
DEPR: tz-aware array
State: closed

Enforces the deprecation on tz-aware series / index
converting to UTC and returning a datetime64[ns] ndarray.

Closes https://github.com/pandas-dev/pandas/issues/6581 (this is the last one).

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Whoops, just saw https://github.com/pandas-dev/pandas/pull/30516. Closing in favor of that