BrandonFors/pandas-github-issues huggingface dataset:
CLN: Simplify gathering of results in aggregate
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Reduces the number of paths when collecting results in `aggregation.aggregate`; one where we are gathering NDFrames using `concat` and the other where we are gathering scalars using `Series`.

The reason for the one test change is as follows. In one case, we previously gathered results using `DataFrame`. When this occurs and the indexes are not all equal, `DataFrame` will sort the index whereas `concat` will have the index in order of appearance. For example with

```
df = DataFrame(
    {
        'A': pd.Series([1, 2], index=['b', 'a']),
        'B': pd.Series([3, 4], index=['c', 'a'])
    }
)
```

gives 

```
     A    B
a  2.0  4.0
b  1.0  NaN
c  NaN  3.0
```

whereas using `concat` instead of `DataFrame` on the first line with `axis=1` gives:

```
     A    B
b  1.0  NaN
a  2.0  4.0
c  NaN  3.0
```

If in this example you replace the 2nd index with ['b', 'a'] (so that they are equal), then both `Dataframe` and `concat` will produce the same result with index `['b', 'a']`. If on the other hand you replace the 2nd index with `['a', 'b']`, then `DataFrame` will result in index `['a', 'b']` whereas concat will result in index `['b', 'a']`.

Labels:
name: Refactor
description: Internal refactoring of code
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
lgtm merge on green
thanks @rhshadrach
DOC: Fix typos and broken formatting
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This PR fixes minor typos in the getting started tutorial [How to manipulate textual data?](https://pandas.pydata.org/pandas-docs/stable/getting_started/intro_tutorials/10_text_data.html). I built the documentation to confirm that I fixed the formatting of the Wikipedia link. I also fixed the inconsistent use of "element-wise" versus "element wise".

Labels:
name: Docs

Comments:
Thanks
TYP: indexes
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@jbrockmendel can you respond to https://github.com/pandas-dev/pandas/pull/37224#discussion_r510798365
Thanks!
Fix regression for is_monotonic_increasing with nan in MultiIndex
State: closed

- [x] closes #37220
- [x] closes #27498 (closes that one too probably, because we need this case)
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Found this while debugging a different issue. 

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: MultiIndex

Comments:
@phofl can you resolve conflict.
@simonjayhawkins Done
this looks fine. can you run some asv's to see if any perf diffs here? (prob don't need to add, want to see the existing ones).
@jreback 
```
 before           after         ratio
     [57c06d9b]       [31ef0b98]
     <36915^2>        <37220>   
-        63.5±5ms       56.5±0.4ms     0.89  multiindex_object.SetOperations.time_operation('non_monotonic', 'string', 'symmetric_difference')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```
Non monotonic indices increased (not surprising). No changes otherwise
thanks @phofl 
@meeseeksdev backport 1.1.x
DOC: Updated resample.py and groupby.py to fix SA04 Errors
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry: Fixed Partial SA04 Errors 



Labels:
name: Docs
name: Groupby
name: Resample
description: resample method

Comments:
Hello @nagesh-chowdaiah! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



























There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-10-23 07:52:28 UTC
@simonjayhawkins can you explain me on this test errors.  I just updated the docs and logic is not changed. but still the pipelines are failing
@jreback Fixed all the issues or SA04 errors.
@jreback and @jorisvandenbossche i have fixed the suggested chagnes. 

can you merge master and ping on green.
Windows py37_np16 unrelated see #37328

```
=========================== short test summary info ===========================
FAILED pandas/tests/computation/test_eval.py::TestEvalNumexprPython::test_binary_arith_ops[4-3-/]
= 1 failed, 96826 passed, 4046 skipped, 1425 xfailed, 1 xpassed, 105 warnings in 1454.58s (0:24:14) =
```

Windows py38_np18 failure 
```
FATAL ERROR: Committing semi space failed. Allocation failed - process out of memory
```

restarting ci

@jreback and @simonjayhawkins does this needs a wait on #37328 to get merged
@jreback can you please merge this PR and can you please add the tag hacktoberfest-accepted. I am participating in that.. would be great if you can help in achieving that target
thanks @nagesh-chowdaiah
CLN: de-duplicate in arithmetic/test_numeric
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

- Move one test to another one parametrized.
- Address one part of TODO item

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @ivanovmg
BUG: Join behaved like an inner join when only one side had MultiIndex
State: closed

- [x] closes #34292
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

We can use the regular path now, since ``droplevel`` can handle an ``Index``. Only difference is a resorted result, because ``Index.join()`` does not preserve the order of the leading input

```
left = pd.Index([0, 1, 2, 1], name="b")
right = pd.Index([1, 2, 5], name="b")
print(left.join(right, how="left"))
```

returns 

```
Int64Index([0, 1, 1, 2], dtype='int64', name='b')
```

Edit: We may be able to delete the ``_join_level`` function, if we add a ``level`` keyword to the ``_join_multi`` function to determine the overlap.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex
name: Stale

Comments:
The change caused a few other tests to fail, which I have not run locally. All because the new result has a differently sorted index. Is this ok or do we have to figure a way out to get the old behavior?
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@phofl status here?
> @phofl status here?


@phofl closing as stale. reopen when ready.
DOC: add value counts as related method to count
State: closed

Counting values using `.value_counts()`, more generally, is a counting task, so it felt natural to link the `.count()` page to `.value_counts()`.

I copied the description for `.value_counts()` from the "See also" box in [`.drop_duplicates()`](https://github.com/pandas-dev/pandas/blob/9fed16cd4c302e47383480361260d63dc23cbefc/pandas/core/frame.py#L5102).

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
thanks @erictleung
BUG: Join did not work correctly when one MultiIndex had only one level
State: closed

- [x] closes #36909
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

``droplevel`` for ``MultiIndex`` returns an ``Index`` when the resulting ``MultiIndex`` had only one level. But if the input had only one level and no level should be dropped, the return was a ``MultiIndex``. This did not seem consistent, so I changed it, that in this case an Index would be returned too. If this is not the desired behavior, we could fix the ``join`` problems after calling this functions.

If this is not desired, we should add a note to the docstring, that the return for ``droplevel`` is a MultiIndex, if no level is dropped.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Done
@jreback green
thanks
ENH: Support closed for fixed windows in rolling
State: closed

- [x] closes #34315
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Enhancement
name: Window
description: rolling, ewma, expanding

Comments:
Hello @mroeschke! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-10-21 22:44:16 UTC
thanks @mroeschke
DOC: remove debug messages by adding semicolons
State: closed

remove last few remaining debug messages that slipped through the cracks by adding semicolons after plot functions

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs
name: Visualization
description: plotting

Comments:
thanks @partev
BUG: Don't copy DataFrame columns as metadata
State: closed

See https://github.com/pandas-dev/pandas/pull/37186/files#r506978889.
Basically, we don't want to accidentally do `series.name =
dataframe.name` and try to set a column.

cc @jorisvandenbossche. This could conceivably affect geopandas, since geodataframe has different metadata than geoseries: https://github.com/geopandas/geopandas/blob/924cdf65c7c15b01749d1cdd036c5c291e87b0f4/geopandas/geodataframe.py#L83

But I don't think it'll actually break anything for you.

Labels:
name: metadata
description: _metadata, .attrs

Comments:
Also cc @arw2019
thanks @TomAugspurger 
> This could conceivably affect geopandas, since geodataframe has different metadata than geoseries: https://github.com/geopandas/geopandas/blob/924cdf65c7c15b01749d1cdd036c5c291e87b0f4/geopandas/geodataframe.py#L83
>
> But I don't think it'll actually break anything for you.

Indeed, I also can't think of a situation where it would break anything (and our tests using pandas master didn't show up any failure). Whenever a metadata attribute existed on `self` but not on `other`, it would not be set to any value anyway
ENH: Support all Scipy window types in rolling(..., win_type)
State: closed

- [x] closes #34556
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Additionally cleans up some helper methods

Labels:
name: Enhancement
name: Window
description: rolling, ewma, expanding

Comments:
Hello @mroeschke! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-10-30 19:32:34 UTC
All green besides the (expected) failing 32-bit build
thanks @mroeschke
CLN: clean color selection in _matplotlib/style
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

There was a comment in the top of the module that the code was a bit too dynamic.
The logic in the function was too complex.

I refactored the code
 - Extract functions
 - Simplify logic
 - Add type annotations

Note: there is one issue with mypy, which I do not know how to handle. Currently I suggested to ignore the error.

Labels:
name: Refactor
description: Internal refactoring of code
name: Visualization
description: plotting

Comments:
Here is a problem. I want to install Cycler (https://matplotlib.org/cycler/) for testing purposes, to mimic matplotlib behavior on default color selection. When I add it into ``environment.yml``, it is rejected by pre-commit check. Moreover, it seems that this package is never installed anyway, which leads to the failure of the tests requiring Cycler.
I have some weird problem with the new test module ``test_style.py``. It works fine in my local virtual environment, but fails in CI (some importing problem).

```
_____________ ERROR collecting pandas/tests/plotting/test_style.py _____________
ImportError while importing test module '/home/vsts/work/1/s/pandas/tests/plotting/test_style.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../miniconda3/envs/pandas-dev/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
pandas/tests/plotting/test_style.py:5: in <module>
    from pandas.plotting._matplotlib.style import get_standard_colors
pandas/plotting/_matplotlib/__init__.py:3: in <module>
    from pandas.plotting._matplotlib.boxplot import (
pandas/plotting/_matplotlib/boxplot.py:5: in <module>
    from matplotlib.artist import setp
E   ModuleNotFoundError: No module named 'matplotlib'
```
All checks are good except Windows py38_np18, where tests related to numba failed. I guess it is irrelevant of my changes.
cc @charlesdong1991 

I checked that we actually can use cycler.
But the problem is that tests must be changed.
Currently it is acceptable to return more colors that ``num_colors``.

When I use cycler I would prefer to keep only required number of colors:

```
def _cycle_colors(colors: List[Color], num_colors: int) -> Iterator[Color]:
    import matplotlib.pyplot as plt

    cycle = plt.cycler(color=colors)
    for _, item in zip(range(num_colors), cycle()):
        yield item["color"]

```

This looks simple to understand, IMHO.
And it seems more reasonable that we return exactly ``num_colors`` colors, without any extra (although extra colors are ignored anyway).
But are we good that I go ahead and change some tests?
Or maybe leave it to another PR?

@jreback 
thanks @ivanovmg very nice
BUG: Non deterministic level order in MultiIndex with join
State: closed

- [x] closes #36910 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This fixes the issue, if the number should be deterministic. If not, we should add a note in the docs

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex

Comments:
thanks @phofl nice!
BUG: Regression in Resample.apply raised error when apply affected only a Series
State: closed

- [x] closes #36951
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Error was no longer caught as before.

Sorry made a mistake when pushing...

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Resample
description: resample method
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
thanks @phofl 
@meeseeksdev backport 1.1.x
BUG: JoinUnit.is_na wrong for CategoricalDtype
State: closed

- [x] closes #20833
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

JoinUnit.is_na is basically checking `isna(self.block.values).all()`.  The check for is_categorical is an attempted optimization bc values.categories is often much smaller than values.  But Categorical represents its NAs in its codes, not in its categories.  So this will incorrectly always return False in the status quo. 

Having trouble coming up with a useful test.  I can adapt a test from test_concat that returns an incorrect answer from is_na, but that does not appear to affect the result of the higher-level pd.concat call.

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Internals
description: Related to non-user accessible pandas implementation
name: Categorical
description: Categorical Data Type

Comments:
surpised this doesn't break anything.
CLN: parametrize test_nat_comparisons
State: closed

Parametrize ``TestDatetime64SeriesComparison.test_nat_comparisons``
using ``operator`` module.

If this kind of cleanup is considered reasonable,
then I will go through the remaining tests in this test module.

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
yes this type of cleanup is good. targeted PRs are the best.

cc @jbrockmendel if comments.
i do not find this clearer than the status quo.

the lowest-hanging fruit in the tests/arithmetic directory are ones that do not use the box_with_array fixture

there are also very likely redundant tests that can be de-duplicated
@jbrockmendel ok so you are -1 on this?
-0.5, i.e. i dont think this is an improvement, but not a hill to die on if others have strong opinions
ok closing this
> -0.5, i.e. i dont think this is an improvement, but not a hill to die on if others have strong opinions

Parameterisation is beneficial when there a multiple test failures. It is sometimes easier to debug when all the passing and failing test cases are visible together.

This PR looked good to me.
ok @ivanovmg can u merge master
Merged master.
Also I renamed local variable ``operator`` into ``op`` for consistency with other test and not to confuse with module ``operator``.
However, probably more readable option would be ``operator_`` with a trailing underscore, so it is more clear that we are talking about the operator.
Alternatively, ``binary_operator`` - longer name, but more clear IMHO.
> Parameterisation is beneficial when there a multiple test failures. It is sometimes easier to debug when all the passing and failing test cases are visible together.

This is true in a general sense, but I'd point out:

1) we have many tests where we don't parametrize these particular 6 methods

2) The 6 Series objects get created at test collection time, which increases the memory footprint of the test suite.  In isolation these 6 small Series is not a big deal, but we are already at a point where memory is causing the py38 windows build to fail regularly.

Again, not a hill worth dying on, just saying.
thanks @ivanovmg
CI: Check for inconsistent pandas namespace usage
State: closed

Adding a CI check that we aren't (for instance) using Series(...) and pd.Series(...) in the same file. ~This is kept intentionally small in scope (checking only DataFrame and Series for one file name right now) since this is very common in the code base and I'm not sure if this is something we'd actually want to enforce.~

Labels:
name: CI
description: Continuous Integration
name: Code Style
description: Code style, linting, code_checks

Comments:
I agree with @jreback: small preference towards not `pd.`, preference for within-file consistency, not worth bikeshedding on this topic
yeah it think this is ok to add. will have to figure out the best way to expand this coverage. maybe a file for includes with the test files to check? (or a config like file).
> yeah it think this is ok to add. will have to figure out the best way to expand this coverage. maybe a file for includes with the test files to check? (or a config like file).

The diffs will be very large but it may be easier just to fix this globally for the different classes. I did this for Series just now (can revert if the PR is too large) and could do for DataFrame, etc. if worthwhile.
> > yeah it think this is ok to add. will have to figure out the best way to expand this coverage. maybe a file for includes with the test files to check? (or a config like file).
> 
> The diffs will be very large but it may be easier just to fix this globally for the different classes. I did this for Series just now (can revert if the PR is too large) and could do for DataFrame, etc. if worthwhile.

oh i think this is fine (yeah do DataFrame separately); only issue is have to make sure all passing (in case we are missing imports). ci is a bit cranky now.
and actually merge master once again (as just merged big PR). ping when you are ready.
> and actually merge master once again (as just merged big PR). ping when you are ready.

Looks to be mostly green (only annoying thing that can happen are false positives due to Series(...) being used in comments but not code, so I changed those as well)
thanks @dsaxton
call __finalize__ in more methods
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

The goal here is to make progress on some of the more uncontroversial/easier content of #28283. I'll split off into another PR if this gets too big

Labels:
name: metadata
description: _metadata, .attrs

Comments:
cc @TomAugspurger 

lgtm.
https://github.com/pandas-dev/pandas/pull/37205 handles the dataframe.name issue.

I don't know if we want to add a test case for all of those, but one or two might not hurt?
> #37205 handles the dataframe.name issue.
> 
> I don't know if we want to add a test case for all of those, but one or two might not hurt?

Great! 

I can add the tests here and then probably we would want to merge #37205 before this goes in
can you merge master and ping on green as merged @TomAugspurger patch
Green
thanks @arw2019
DOC: Add example to Series.divmod
State: closed


- [x] closes #24589
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

-----
[Current doc](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.divmod.html); Rendered doc:
![Screenshot 2020-10-17 at 10 44 26 AM](https://user-images.githubusercontent.com/19281800/96326790-3d888d80-1066-11eb-94e7-0a4e0db5d4ba.png)


Labels:
name: Docs
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
thanks @hongshaoyang
CLN: core/dtypes/cast.py::maybe_casted_values
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

xref https://github.com/pandas-dev/pandas/pull/36985#issuecomment-705804862

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Clean

Comments:
@jbrockmendel @jreback this is the follow-up to #36985. It seems like it's ok to just drop the special casing for EAs in `maybe_casted_values`
great thanks @arw2019