BrandonFors/pandas-github-issues huggingface dataset:
API: Store name outside attrs
State: closed

This aligns with xarray and h5py:
https://github.com/pandas-dev/pandas/pull/29062#issuecomment-545703586

Labels:
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
sure x- @jbrockmendel comment.
@TomAugspurger maybe also add a note somewhere that `attrs` is experimental?
When will this feature be not experimental?  Is allows for persistence of data when pickling on accessors.  I would encourage to remove the experimental tag.
You can remove the experimental label if you want. The basics aren't go to change. However, there are two future changes that will affect things in attrs

1. `__finalize__` isn't called everywhere: https://github.com/pandas-dev/pandas/issues/31452
2. We might provide an API for merging `.attrs` of multiple dataframes in operations like `concat`. This might be doable in a backwards-compat way. I'm not sure yet.
IntegerArray.to_numpy
State: closed

This implements IntegerArray.to_numpy with similar semantics to BooleanArray.to_numpy. The implementation is now identical between BooleanArray & IntegerArray. #30789 will merge them.

1. `.to_numpy(dtype=float/bool/int)` will raise if there are missing values
2. `.astype(float)` will convert NA to NaN.

I've made a slight change from the BooleanArray implementation on master, which I'll annotate inline.

Closes https://github.com/pandas-dev/pandas/issues/30038

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
Thanks @TomAugspurger
BUG: DTI/TDI/PI `where` accepting non-matching dtypes
State: closed

This bug was hidden by _ensure_datetimelike_to_i8, and the only other place where that is used is in _round.  _round is clearer without using it, so ensure_datetimelike_to_i8 gets ripped out, and with it we can get rid of _ensure_localized.

Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Timedelta
description: Timedelta data type

Comments:
thanks @jbrockmendel 

love the code cleanups!
REF: Implement BaseMaskedArray class for integer/boolean ExtensionArrays
State: closed

Todo item of https://github.com/pandas-dev/pandas/issues/29556, consolidating common code for IntegerArray and BooleanArray.

This is only a start, there is more to share.

Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks for starting this. Just out of curiosity did you ever consider implementing this as a Mixin? I suppose the advantage of that versus something like this would be it theoretically allows EA authors to compose masked extension types without having to worry about class hierarchies
For now, this is not supposed to be public (actually, I just didn't consider that yet, that can certainly be a discussion). For internal use, it doesn't really matter much? What would be the advantage of a mixin in that case?
For me (being less exposed to the EA stuff) a mixin just seemed a more natural way to add a mask. Let's see what others think
For now, I think just straight inheritance from EA is best. With a mixin, we'd need to include a bunch of the EA interface anyway to get this to work (dtype, isna, etc.)
> can you comment on what is not just a straight cut/paste

Everything is still a straight cut and paste, except for:

- The `_internal_fill_value` class member, which is used in `take`
- I edited the `to_numy` docstring a little bit to be generic for both datatypes
> This is only a start, there is more to share.

@jorisvandenbossche were you planning to move any more methods in this PR, or should we merge this and do more as followups? Something like `_create_comparison_method` *may* be sharable but will be a be tricky. Might warrant its own PR.
this ready?
thanks @jorisvandenbossche
CLN: Removed outdated comment
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

I think this is a bit outdated, since we are using black and it's doing the work for us, right?

according to git blame, this is 2 years old.

Labels:
name: Clean

Comments:
Thanks @MomIsBestFriend 
@MomIsBestFriend - I don’t think this is referring to black.

Think it’s related to https://github.com/pandas-dev/pandas/issues/22524
@alimcmaster1 should it be reverted then?
ASV: use pandas.util.testing for back compat
State: closed

cc @TomAugspurger I propose to keep the old deprecated imports (as long as they are not removed), so the benchmarks can still be run when eg doing a comparison of 0.25 with current master. 



Labels:
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
Actually, maybe doing a try/except is a better option (this can be put in `pandas_vb_common.py`, we have something similar for `lib`)
A try / except sounds better to me, so we don't have the warnings in the logs.
> A try / except sounds better to me, so we don't have the warnings in the logs.

maybe we just filter out FutureWarnings from asvs? (generally)
Though we'll need to make the changes at some point, for when we enforce the deprecation. Don't have a preference.
> Though we'll need to make the changes at some point, for when we enforce the deprecation.

When using the try/except, that is in principle not needed (until the point we want to clean up compatibility with older pandas versions in the benchmarks of course, but that doesn't necessarily need be at the same time as when enforcing the deprecation)
thanks @jorisvandenbossche 
@jreback given the discussion, this was not really ready 
See https://github.com/pandas-dev/pandas/pull/30810 for a PR based on the discussion above
[WIP] style NA in reprs
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Hello @TomAugspurger! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

































* In the file [`pandas/tests/arrays/test_integer.py`](https://github.com/pandas-dev/pandas/blob/c6d593f97e98b648a5d4416da7ef4207fd711860/pandas/tests/arrays/test_integer.py):

> [Line 109:89](https://github.com/pandas-dev/pandas/blob/c6d593f97e98b648a5d4416da7ef4207fd711860/pandas/tests/arrays/test_integer.py#L109): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (113 > 88 characters)
> [Line 115:89](https://github.com/pandas-dev/pandas/blob/c6d593f97e98b648a5d4416da7ef4207fd711860/pandas/tests/arrays/test_integer.py#L115): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (98 > 88 characters)
> [Line 119:89](https://github.com/pandas-dev/pandas/blob/c6d593f97e98b648a5d4416da7ef4207fd711860/pandas/tests/arrays/test_integer.py#L119): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (164 > 88 characters)








FYI I just made a "Mothballed" label for temporarily closed PRs the author plans to return to
I'll just close this. No plans to return any time soon.
TST: Add tests for fixed issues
State: closed

- [x] closes #13230
- [x] closes #13820
- [x] closes #13758
- [x] closes #13228
- [x] closes #13208
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
lgtm. if you'd update to @simonjayhawkins comments, then merge on green.
CLN: Simplify logic in _format_labels function for cut/qcut
State: closed

Small simplification: modify the `breaks` metadata before creating an `IntervalIndex` then create and an `IntervalIndex` from the modified `breaks`.  The existing approach creates an `IntervalIndex`, modifies the first `Interval`, then creates a new `IntervalIndex` with the updated first `Interval`.

This yields a slight performance improvement but doesn't seem dramatic enough to warrant a whatsnew entry, though I can add one if desired.

On this branch:
```python
In [1]: import numpy as np; import pandas as pd; pd.__version__
Out[1]: '0.26.0.dev0+1668.ga6c08fc02'

In [2]: a = np.arange(10**5)

In [3]: %timeit pd.qcut(a, 10**4)
273 ms ± 914 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

On `master`:
```python
In [1]: import numpy as np; import pandas as pd; pd.__version__
Out[1]: '0.26.0.dev0+1667.g40bff2fed'

In [2]: a = np.arange(10**5)

In [3]: %timeit pd.qcut(a, 10**4)
317 ms ± 1.14 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Clean

Comments:
Thanks!
in tests, change pd.arrays.SparseArray to SparseArray
State: closed

- [x] closes https://github.com/pandas-dev/pandas/pull/30656#discussion_r363060184
- [x] tests added / passed
    - modified most tests that use `pd.arrays.SparseArray` to just import `SparseArray`
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry
    - N/A

For @jreback to review based on his comment previous PR #30656 

In a few files (see below), left it as is because usage was pretty local (and allows `pd.arrays.SparseArray` reference to be tested in code)
```bash
$ grep -c -r arrays.SparseArray . | grep -v ":0"
./dtypes/test_generic.py:1
./frame/methods/test_quantile.py:2
./series/test_missing.py:2
```




Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
My style preference is to just use `pd.arrays` namespace.
> My style preference is to just use `pd.arrays` namespace.

So in the various test modules, 
So @TomAugspurger prefers to use `pd.arrays.SparseArray`, i.e., writing `pd.arrays.SparseArray([1,2,3])` in a test.
And @jreback prefers to `from pd.arrays import SparseArray`, i.e., writing `SparseArray([1,2,3])` in a test

I did the latter because of feedback from @jreback , so you guys need to work out your differences.

Yep, that's what we're doing.

On Tue, Jan 7, 2020 at 3:00 PM Irv Lustig <notifications@github.com> wrote:

> My style preference is to just use pd.arrays namespace.
>
> So in the various test modules,
> So @TomAugspurger <https://github.com/TomAugspurger> prefers to use
> pd.arrays.SparseArray, i.e., writing pd.arrays.SparseArray([1,2,3]) in a
> test.
> And @jreback <https://github.com/jreback> prefers to from pd.arrays
> import SparseArray, i.e., writing SparseArray([1,2,3]) in a test
>
> I did the latter because of feedback from @jreback
> <https://github.com/jreback> , so you guys need to work out your
> differences.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/30765?email_source=notifications&email_token=AAKAOIQHVMHOKCWZ2KU5BRLQ4TUQBA5CNFSM4KDND7J2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIKIQTA#issuecomment-571770956>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKAOITI5E6XVDNSADVFVWDQ4TUQBANCNFSM4KDND7JQ>
> .
>

this looks fine to me, i slightly prefer the shorter syntax. unless @TomAugspurger objects, ok to merge this.
No objections. 

> On Jan 7, 2020, at 17:13, Jeff Reback <notifications@github.com> wrote:
> 
> ﻿
> this looks fine to me, i slightly prefer the shorter syntax. unless @TomAugspurger objects, ok to merge this.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.
BUG: TDI/DTI _shallow_copy creating invalid arrays
State: closed

Following this we should be able to use shallow_copy in indexes.extension more, which will help with perf (xref #30717)

Labels:
name: Datetime
description: Datetime data dtype
name: Timedelta
description: Timedelta data type
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Index
description: Related to the Index class or subclasses

Comments:
thanks, some comments
TYP: type up parts of series.py
State: closed

More typing.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Hello @topper-123! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-12 13:32:45 UTC
I think this is the best I can do here: I agree a TypeVar would be best, but it doesn't play well with ``._constructor`` AFAICS.

I wonder if we should deprecate ``._constructor`` and use ``type(self)`` for instantiation, but that's probably a different discussion wit its own traps.
lgtm.
ping
Thanks @topper-123
BUG: TDI.insert with empty TDI raising IndexError
State: closed

This started out as a cosmetic-only branch and ended up finding a broken corner case.  The relevant change is in timedeltas L416 where `if self.freq is not None:` is now `if self.size and self.freq is not None:`

Using _check_compatible_with causes us to raise TypeError instead of ValueError in a couple of the DatetimeIndex.insert cases.

Labels:
name: Timedelta
description: Timedelta data type

Comments:
@simonjayhawkins suggestions for fixing mypy?
```
mypy 0.730
Performing static analysis using mypy
pandas/core/indexes/timedeltas.py:56: error: Definition of "_data" in base class "ExtensionIndex" is incompatible with definition in base class "DatetimelikeDelegateMixin"
pandas/core/indexes/timedeltas.py:56: error: Definition of "_data" in base class "Index" is incompatible with definition in base class "DatetimelikeDelegateMixin"
pandas/core/indexes/datetimes.py:87: error: Definition of "_data" in base class "ExtensionIndex" is incompatible with definition in base class "DatetimelikeDelegateMixin"
pandas/core/indexes/datetimes.py:87: error: Definition of "_data" in base class "Index" is incompatible with definition in base class "DatetimelikeDelegateMixin"
Found 4 errors in 2 files (checked 844 source files)
```
> @simonjayhawkins suggestions for fixing mypy?

am AFK for the rest of the day (Pydata London). will look tomorrow.
ping on green.
ping
can you rebase
thanks @jbrockmendel
BUG: DTI/TDI .insert accepting incorrectly-dtyped NaT
State: closed

Also TDI.insert trying to parse strings to Timedelta, which neither DTI nor PI do.

Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Timedelta
description: Timedelta data type

Comments:
thanks (technically should have a whatsnew, but this is a very very tiny edge case), but if you can in a followon ok.
Fix: Warning(Attempt to set value to copy of a slice)
State: closed

Getting a warning emitted from this line of code.  Here is the fix I used to get rid of the warning, and here is the code that generated it on my setup (Anaconda3, Windows 10):

```
import pandas as pd

ts_columns = []
for col in df.columns:
    if isinstance(df[col].dtype, Timestamp):
        ts_columns.append(col)
```

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Usage Question

Comments:
Hello @chrisdonlan! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-06 20:29:15 UTC
@jreback That doesn't seem to be the case.  The warning is intermittent, the dataframe is in memory, I am in a debugger, and while slowly stepping through the code produces no issues, **occasionally** this warning is emitted when the code is allowed to run (over and over on the same data, with same configs).   

Can't really devote much more time to this.  It seems to be a Pandas issue, because I can perform the same task in numpy/native python without flaws (i.e. by accessing the underlying values and performing type checking there). 

![image](https://user-images.githubusercontent.com/14854647/71848703-dd57ca80-3095-11ea-8630-8dfeb521c1a1.png)

![image](https://user-images.githubusercontent.com/14854647/71848670-bf8a6580-3095-11ea-90ab-4f293df39feb.png)

Agreed that we won't be able to accept this without a test case demonstrating the issue. Let us know if you can provide one @chrisdonlan, otherwise, I think we'll pass on changing this line.
@TomAugspurger @jreback  No problem, I won't be able to lock this issue down until later. 
@TomAugspurger @jreback  

Looks like I get rid of the warning by deep copying the dataframe before passing into the function which inspects the column types.  I am not sure why checking the column types should incur a deep-copy, or trigger any modification of dataframe values.  
closing as not reproducible. if you think you have a found a bug, then we need a self-contained reproducible example.
Fix PR08 errors
State: closed

Fixes PR08 errors. When I ran the script, a lot of them seem to be false positives, these are the ones I'm pretty sure should be fixed:
```
pandas.infer_freq: Parameter "index" description should start with a capital letter
pandas.MultiIndex.get_loc_level: Parameter "drop_level" description should start with a capital letter
```
Related to #27977 cc @datapythonista

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Thanks @galuhsahid
PERF: Categorical getitem perf
State: closed

Convert to an array earlier on.
Closes https://github.com/pandas-dev/pandas/issues/30744

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance
name: Categorical
description: Categorical Data Type

Comments:
Master

```python
   ...: %timeit data[list_]
1.44 ms ± 5.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```

This PR

```
   ...: %timeit data[list_]
740 µs ± 5.92 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```

Will post the pre-indexing with NA timing in a bit.

Edit:

7b350997467f868aa563c42cfec1d0a3bd2a9887 (1 commit prior to NA indexing)

```
728 µs ± 7.57 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```

so we're back to where we were.
cool, can you add an asv
> can you add an asv

I think Joris caught this because the existing asv `CategoricalSlicing.time_getitem_list` https://pandas.pydata.org/speed/pandas/#categoricals.CategoricalSlicing.time_getitem_list?commits=6efc2379-b9de33e3
LGTM
So I reported this for Categorical (indeed because of the existing benchmark), but a similar change was made for other dtypes in https://github.com/pandas-dev/pandas/pull/30308. Also the boolean and integer (and numpy) arrays have the same issue.
CI: Using docstring validator from numpydoc
State: closed

- [X] xref #28822
- [x] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

We moved our script to numpydoc, and it already had some improvements there. The script does like 80% of our validation, so what I'm doing here is to call numpydoc validation, and then call our custom validation (things that for different reasons weren't moved to numpydoc).

Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
> For the codes that are left over do you see another home for them, like code checks or any of the other validators?

The assumption here is that we want to be able to have a script that reports all the errors for a specific docstring. The one we used in the sprints, which we'll be using again when fixing one docstring at a time.

Otherwise, we could do what you say. And the validation for pep8 errors in the examples, which is the most tricky one, could just be deleted, since we can use pytest for that.

One thing to consider is that the numpydoc script is expected to be wrapped if used in the CI. It can be used directly for a single object, but we need to implement the functionality to get which objects we want to validate, which I think it's more than half of the new script.
@WillAyd can you remove the requested changes flag if you don't have further comments. Thanks!
Merging this today if there are no further comments.
DEPR/REGR: Fix pandas.util.testing deprecation
State: closed

Closes https://github.com/pandas-dev/pandas/issues/30735

This avoids using _DeprecatedModule, which doesn't work for
direct imports from a module. Sorry for the importlib magic, but
I think this is the correct way to do things.

cc @jorisvandenbossche.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Deprecate
description: Functionality to remove in pandas

Comments:
I don't have a strong opinion on this approach vs just leaving the pandas.util.testing module and importing what is needed there (and add a deprecation warning in the module, which will only trigger a warning once?). But if we keep the approach in this PR, should this be factored out in a utility and just replace the current DeprecatedModule utility?
Sorry, I think @jreback is right. We don't need the import hook stuff. Just

```python
# pandas/util/testing.py
warnings.warn("pandas.util.testing is deprecated. Use pandas.testing instead")
from pandas._testing import *
```

should accomplish the same thing with less hassle. Will update shortly (I do think I need to restructure a few things to avoid a circular import from core)
Sorry about that. Just using standard stuff now.
LGTM
thanks @TomAugspurger