BrandonFors/pandas-github-issues huggingface dataset:
Tests for where() with categorical
State: closed

- [X] closes #18888
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Hey everyone, 

I wrote some tests for issue #18888. I don't know exactly if the format is correct but if any change is needed I'm happy to make it right. 

Which 'whatsnew entry' should I report for both new tests?

Also, if you think this is a decent contribution and if there's no problem with doing that, it would be nice to tag this PR with hacktoberfest-accepted label :)

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Categorical
description: Categorical Data Type

Comments:
@arw2019 Joined both tests the way you suggested!

@jreback Don't know how is the project policy with Hacktoberfest this year but if it's ok to label this PR with 'hacktoberfest-accepted' I would appreciate it very much!
thanks @gabriellm1
BUG: Fix isin with read-only target
State: closed

- [x] closes #37174
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
@jbrockmendel Seems this has been talked about before in https://github.com/pandas-dev/pandas/issues/31710. This fix worked for me locally, but is this less trivial than it seems?
> This fix worked for me locally, but is this less trivial than it seems?

i dont think so, no
@simonjayhawkins @dsaxton 
Sorry for the stupid question but how could I get the version with this fix (with conda if possible)? 
1.1.4 is currently scheduled for release on October 30, 2020, see https://github.com/pandas-dev/pandas/milestone/78 

wheels are uploaded to PyPI moreless immediately and then shortly after for conda-forge and defaults.
BUG: use cls instead of MultiIndex in MultiIndex classmethods
State: closed

- [x] closes #11267
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This doesn't go through all the places where MultiIndex is hard-coded, just the classmethods where in `cls` should be used as a matter of principle.

Labels:
name: MultiIndex
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
yep merge on green
API: tighten DTA/TDA _from_sequence signature
State: closed

cc @jorisvandenbossche 

need to decide how to handle test_from_2d_ndarray_with_dtype which this currently xfails

Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: API - Consistency
description: Internal Consistency of API/Behavior
name: Mothballed
description: Temporarily-closed PR the author plans to return to

Comments:
> this looks like a substantial (but good) api change, or is this _just_ on master as _from_sequence is actually strict? (in 1.1.x)

not just on master, this makes _from_sequence substantially stricter than 1.1.x
Thanks for looking at this!

Now, for the actual behavioural change, `_from_sequence` is still being used for eg `pd.array(.., dtype="datetime64[ns"])` as well, so we first need to decide what behaviour we want for that. 
Otherwise could also keep this in a `_from_sequence_strict` (and rename `_from_sequence_not_strict` to `_from_sequence`)
just pushed with a couple new tests that get us to full coverage on the affected methods
if u can rebase; i think a couple of questions
cc @WillAyd most recent commit made small edits to json code, can you take a look
lgtm - I think the explicitness here is nice
@jorisvandenbossche any objection to this as a temporary workaround?  im pretty sure It will allow us to clean up a bunch of code in .isin and .equals xref #37528
updated per discussion so that _from_sequence remains non-strict while a new _from_sequence_strict has the hopefully-future behavior of from_sequence.

im hoping to use from_sequence strict to simplify+de-duplicate .equals and .isin
> We don't have one that is more strict on the input that could be used for this purpose?

Good idea, we can use `tslibs.conversion.datetime_to_datetime64`.  AFAICT we dont have an equivalent for timedelta, but it wouldn't be hard to implement one.  `libperiod.extract_ordinals` can also be adapted to do the same thing for PeriodArray (separate PR).

Will update.
mothballing to clear the queue while i address comments locally
BUG: infer_dtype with decimal/complex
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Does this need a whatsnew?

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
thanks
BUG: fix tab completion
State: closed

Currently tab completion doesn't work on attributes defined on the instance, only ones defined on the class. This fixes that.

Example:
```python
>>> class MyCls(pd.core.accessor.DirNamesMixin):
...     x = 1
... 
...     def __init__(self):
...         self.y = 2
...
>>> x = MyCls()
>>> "x" in dir(x)
True  # master and this PR
>>> "y" in dir(x)
False  # master
True  # this PR
```

Also some type fixups.

Labels:
name: Groupby
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
LGTM pending green
I suppose worth a whatsnew note
Updated. I´ve renamed _deprecations to _hidden_attribs, because being deprecated and hidden may be two different things.
thanks @topper-123
CLN: ensure we pass correct type to DTI/TDI shallow_copy
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Clean
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
This seems to have caused a performance regression, see https://pandas.pydata.org/speed/pandas/#index_object.SetDisjoint.time_datetime_difference_disjoint?Cython=0.29.21&python=3.8&x-axis-scale=date

Example from the benchmark:
```
In [5]: N = 10 ** 5
   ...: B = N + 20000
   ...: datetime_left = pd.DatetimeIndex(range(N))
   ...: datetime_right = pd.DatetimeIndex(range(N, B))

In [6]: %timeit datetime_left.difference(datetime_right)
3.27 ms ± 230 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

(note I didn't verify it's this commit, but it seems the most related one in the last days)
BUG: MultiIndex comparison with tuple #21517
State: closed

- [x] closes #21517
- [x] closes #34180
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
also closes #34180, just pushed with a test for that, will update OP
thanks
BUG: Bug in quantile() and median() returned wrong result for non monotonic window borders
State: closed

- [x] closes #37153
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

cc @mroeschke 

Values were not added to the window again if starting point was moved back and values were not removed from the window, when ending point was moved back

Should I create functions for the deletion and addition?

Labels:
name: Window
description: rolling, ewma, expanding

Comments:
Yeah would be great to create a function that median and quantile could share for the skiplist inserts and removal
Should we move the for the ``for j in range(s, e):`` blocks into the function too? Would be differing from the other parts, but otherwise the assignment of ``err`` is a bit long winded
Hmm if it makes the code more DRY then sure.
this is orthogonal to #36933 ?
> this is orthogonal to #36933 ?

Yes, median and quantile are slightly different from sum/mean/var
We can use ``is_monotonic_start_end_bounds`` here too nevertheless. Did miss that previously. I would rename the function after #36933 is merged.
Mind merging in master? https://github.com/pandas-dev/pandas/pull/36933 was recently pulled in (but shouldn't directly interfere) 
Thanks for the heads up. Had to rename the method, but this was expected. I think this could be merged now.
Appears that there's another merge conflict. 
Thanks @phofl
ENH: DataFrame __divmod__, __rdivmod__
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Enhancement
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
> is this new? or just a better definition for the op? (e.g. do you need a whatsnew)

new, will add whatsnew
DEPR: Series/Index .values special cases
State: closed

- [x] closes #36957
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:

Comments:
Is this motivated by code-sharing or trying to reduce special casing of our extension blocks?

IMO, if there's code in pandas that relies on `.values` returning an ndarray or not then that's a sign that we shouldn't be using `.values`. We should be using `.array` or `.to_numpy()` instead.
> Is this motivated by code-sharing or trying to reduce special casing of our extension blocks?

That, plus the general "this is a code smell" thing
> Let's discuss further on the issue first?

Sounds good.  I'll keep this open to debug the npdev build.
Not worth the effort to get the npdev build passing.  Closing.
API: Index.__cmp__(Series) return NotImplemented
State: closed

- [x] closes #36759
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Gets rid of an unnecessary special case.

Could do this as a deprecation first.


Labels:
name: Index
description: Related to the Index class or subclasses
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
if this isnt something we want to do, can close
Discussed on Nov 2020 dev call, decided to put this off until we have BoolIndex
BUG: Fix parsing of ISO8601 durations
State: closed

This PR fixes the following behavior:

- Adds "W" as a valid weeks designator
- Fixes parsing of durations with empty periods (e.g. "PT10M")
- Fixes partial parsing of durations (e.g. "P1DT1H")

Fixes #29773.
Fixes #36204.

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Labels:
name: Bug
name: Timedelta
description: Timedelta data type

Comments:
thanks @mgmarino very nice!
CLN/REF: Refactor min_periods calculation in rolling
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: Window
description: rolling, ewma, expanding

Comments:
cool. was there a referenced issue?
No, just discovered independently
REF: avoid access to _mgr.blocks -> _mgr._is_single_block
State: closed

@jbrockmendel a small follow-up on https://github.com/pandas-dev/pandas/pull/36873. It keeps the same spirit of that PR (I think), but just replacing the access of `len(self._mgr.blocks)` with an existing attribute on the manager (could also remove the leading underscore on the attribute). 
(this makes it easier to override this attribute in the ArrayManager, and limits the access to the blocks outside of the internals)

Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Good idea.  Probably should be public
(failure is unrelated, "##[error]Insufficient system resources exist to complete the requested service.")
BUG: indexing bugs #26490, #13691
State: closed

- [x] closes #26490
- [x] closes #13691
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Closing in favor of #37161, #37152
BUG: DataFrame[td64].sum(skipna=False)
State: closed

- [x] closes #17235
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

The same underlying problem affects mean, so this fixes that too.


Labels:
name: Bug
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
i think roughly the same fix will end up being used for #36907
updated per requests + green.  several followups in the pipeline
gentle ping; id like to re-use the helpers implemented here in PR(s) fixing other reductions
i guess this needs a whats new note but can be a follow on
BUG: Fix bug in quantile() for resample and groupby with Timedelta
State: closed

- [x] closes #29485
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Groupby
name: Timedelta
description: Timedelta data type
name: Resample
description: resample method

Comments:
Sorry, was waiting for some kind of feedback :) Changed the whatsnew
Windows py38_np18 failed to start, Windows py37_np16 failed on `test_binary_arith_ops` (expected).
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
Same result. @phofl Can you merge master? `test_binary_arith_ops` should now pass and hopefully py38_np18 will start.
Done
Failure seems unrelated
thanks @phofl
BLD: update build requirement for py39 #37135
State: closed

- [x] closes #37135 



Labels:
name: Build
description: Library building on various platforms

Comments:
> Many thanks for considering my request. But I'm really new with programming and cmd. How can I apply this code when I pip install in cmd?
> 
> ```
> "wheel",
> "Cython>=0.29.21,<3",  # Note: sync with setup.py
> "numpy==1.16.5; python_version=='3.7' and platform_system!='AIX'",
> "numpy==1.17.3; python_version>='3.8' and platform_system!='AIX'",
> "numpy==1.17.3; python_version=='3.8' and platform_system!='AIX'",
> "numpy==1.16.5; python_version=='3.7' and platform_system=='AIX'",
> "numpy==1.17.3; python_version>='3.8' and platform_system=='AIX'",
> "numpy==1.17.3; python_version=='3.8' and platform_system=='AIX'",
> "numpy; python_version>='3.9'",
> ```
> 
> ]
> 
> [tool.black]
> target-version = ['py37', 'py38']
> target-version = ['py37', 'py38', 'py39']
> exclude = '''
> (

I'll reply in the issue.
thanks @fangchenli 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.1.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 58dcafadfa50bb160afb2657a4fbda326d4802a6
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #37144: BLD: update build requirement for py39 #37135'
```

4. Push to a named branch :

```
git push YOURFORK 1.1.x:auto-backport-of-pr-37144-on-1.1.x
```

5. Create a PR against branch 1.1.x, I would have named this PR:

> "Backport PR #37144 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
DOC: Remove redundant table in Series docs
State: closed

The attribute `Series.index` was separated in another table, which makes
the rendering of the documentation slightly inconsistent. This commit
consolidates the tables together to make the documentation consistent
and aligned.

**Live site** (https://pandas.pydata.org/docs/reference/series.html):

![image](https://user-images.githubusercontent.com/2754821/96168089-20708500-0ed5-11eb-8bab-cc34ebb7f7bc.png)

**Locally rendered change**:

![image](https://user-images.githubusercontent.com/2754821/96168171-4007ad80-0ed5-11eb-99a2-48c200602213.png)

I'm not sure why the color changed. My guess is because I just rendered that page with `python make.py --single reference/series.rst`, so the links to those pages are nonexistent.

If this separate table was intentional, feel free to close this PR. Thanks for considering this!

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
good catch thanks @erictleung
BUG: algos.diff with datetimelike and NaT
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Timedelta
description: Timedelta data type

Comments:
where is the bug?
> where is the bug?

`algos.diff`.  the tests this adds fails in master
Remove no-longer-necessary CI checks
State: closed

xrange is no longer valid in Python3, so this check is unnecessary.

`# -\*- coding: utf-8 -\*-"`, `class\s\S*\((object)?\):`, and `"super\(\w*, (self|cls)\)` are fixed by pyupgrade


closes #37133 


Labels:
name: CI
description: Continuous Integration

Comments:
thanks!
TYP: use overload to refine return type of reset_index
State: closed

maybe alternative to #33519

with `from __future__ import annotations` on py3.7 the types in the function signature are not evaluated at run-time, so this shouldn't fail 🤞 

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
LGTM
Cool thanks @simonjayhawkins
CI move check incorrect sphinx directives to pre-commit
State: closed

Currently, the check (which uses `grep` with the return codes inverted) doesn't catch

```python
    .. warning:
```
, while using `pygrep` from `pre-commit` (which is a Python implementation of `grep` with inverted return codes) does. I think this is because grep doesn't count the newline as a character

Advantages of moving this (and, later, other) `invgrep` checks to pre-commit are:

- they'll be cross-platform
- they'll be simpler to write/maintain because `pre-commit` comes with `pygrep` built-in, so there's no need to manually define something like `invgrep`
- they'll give devs faster feedback on when they introduce unwanted patterns

Labels:
name: Docs
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @MarcoGorelli these pre-commit hooks are nifty
BUG: raise when doing arithmetic on DataFrame and list of iterables
State: closed

- [X] closes #36702
- [X] tests 1 added / 1 passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry

## Problem

We currently allow doing arithmetic on a DataFrame, and, for example, a list of Series. The latter gets broadcast with `align_method_FRAME` and this leads to weirdness.

## Solution

We should raise. There is no definite answer to how such arithmetic should be handled, and the user should just make another DataFrame and add that to the frame instead of doing stuff like `[Series, Series]`. 


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
cc @jbrockmendel 
Ran indexing benchmarks three times, no stable regressions on my machine.
I'm running the benchmark suite on Ubuntu inside Windows with plenty of free CPU and memory, but I still saw different 2-5 regressions / improvements during the three runs that I did. Sorry, don't have a dedicated machine for benchmarking, and my setup isn't always stable (probably because of all the junk in Windows grabbing and releasing resources).
We should be fine, since no regression or improvement held across runs.
@jreback 

Made the change suggested by @jbrockmendel , all green.
thanks @AlexKirko nice!
Unfortunately this fix breaks `test_arith_frame_with_scalar` for my custom `ExtensionArray`. I have to say that my type is a bit "weird" in the sense that indexing it gives a object of the same type (like the string type in the standard library). This fix does not seem to like that very much.
@vnmabus can you give a link or example?
I have two extension arrays in https://github.com/GAA-UAM/scikit-fda, `FDataBasis` and `FDataGrid`. Both of them are used to represent a sample of functions (functional data). However, for simplicity, as I did not want to create a "just one function" object to represent a scalar value which won't be used much, I made `__getitem__` with an integer key return an object of the same kind and length 1.

These extension arrays (mostly) work with Pandas, but this fix raises an exception, as they are "list like" and the elements (that is, the same class) are "array like". Yo can see the error here: https://github.com/GAA-UAM/scikit-fda/runs/3037343824.
Haven't looked at this too closely, but I think _maybe_ if you make `FDataAFoo.__iter__` yield the scalars (functions) that might avoid this particular problem.  That would mean an inconsistency between `list(fdata)` vs `[fdata[i] for i in range(len(fdata))]` though.
The thing is that currently there isn't a separate class for the scalars. A scalar in my case is just an object of the same kind but with length one.
Yah I don't see any obvious way to avoid inconsistencies.  If this is a serious problem, I'd suggest opening a new issue for dedicated discussion
TST: test for expanding window with min_periods GH25857
State: closed

- [x] closes #25857
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Hi @mroeschke, here's the new test - please let me know if it needs any changes.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Window
description: rolling, ewma, expanding

Comments:
@mroeschke should close the OP ?
thanks @theMogget 
Thanks! Could anyone help at the `hacktoberfest-accepted` label to this please? :) 
Thanks @theMogget. added.
PERF: compare RangeIndex to equal RangeIndex
State: closed

Fastpath for RangeIndexes with equal `_range` attribute.

Notice this works even if the RangeIndexes are not identical.

Example:

```python
>>>  n = 100_000
>>> rng1 = pd.RangeIndex(n)
>>> rng2 = pd.RangeIndex(n)
>>> %timeit rng1 == rng2
374 µs ± 15.1 µs per loop  # master
10.4 µs ± 157 ns per loop  # this PR
```

Somewhat related to #37109

Labels:
name: Performance
description: Memory or execution speed performance
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Index
description: Related to the Index class or subclasses

Comments:
thanks @topper-123
ENH: DataFrame.to_parquet() returns bytes if path_or_buf not provided
State: closed

- [x] closes #37105 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Not sure if this is an API breaking change. Prior to this patch `path` was a required positional argument but it becomes optional here. It is now consistent with, for example, the csv writer.

Labels:
name: IO Parquet
description: parquet, feather

Comments:
lgtm. can you merge master and ping on green just to make sure.

@jorisvandenbossche if any comments.
Thanks @arw2019 !