BrandonFors/pandas-github-issues huggingface dataset:
TST: GroupBy on 2 rows of MultiIndex returns correct group indices
State: closed

- [x] closes #18451 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

There was previously a bug where grouping on precisely two rows of a MultiIndex would return a wrongly formatted index. This PR is adding a test to enforce this correct behaviour. The new test fails on 0.20.3 but passes on >=1.0.5.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: MultiIndex

Comments:
thanks @smithto1 very nice, keep em coming!
DOC: Add note to docstring DataFrame.compare about identical labels
State: closed

- [x] closes #35491
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Docs

Comments:
Hello @erfannariman! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-09-01 16:35:46 UTC
@simonjayhawkins do you have time to check this PR? Thanks
will look soon
@MarcoGorelli I added `DataFrame.equals` to "see also"  section as well since these methods are quite alike. If that is not oke, let me know and I'll remove it.

@MarcoGorelli Do you have any idea what's the cause of this test failing? I can't figure out.
> @MarcoGorelli Do you have any idea what's the cause of this test failing? I can't figure out.

fixed in #36011, restarted job so should pass.
ping @MarcoGorelli @simonjayhawkins 
ping @jreback 
LGTM
Thanks @erfannariman
MAINT: Fix broadcasting of arrays
State: closed

Prevent empty object arrays from being broadcast

- [X] closes #35481
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Probably not the right fix for the future empty array assignment issue, but seems to work.
@bashtage if this closes #35481, I think tests should be failing (with XPASS), see #35502
I thought it worked prior to rebase...I'll need to see what happens.
Definitely not a fix.
MAINT: Use float arange when required or intended
State: closed

Ensure arange is float when intened or required by NumPy

- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Clean

Comments:
Use float initialization to allow NaN to be set of when setting floating points later. 
Used to be part of #35427, but split it out to have one PR one issue.  Required by Future NumPy.
Fixes about 12 of the np-dev fails.
Thanks @bashtage 
@MeeseeksDev backport to 1.1.x
REGR: Fix conversion of mixed dtype DataFrame to numpy str
State: closed

- [x] closes https://github.com/pandas-dev/pandas/issues/35455
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
lgtm. ping on green.
> lgtm. ping on green.

@jreback Green, thanks for reviewing
thanks!
CI: unpin isort 5 (#35134)
State: closed

- [x] closes #35134
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: CI
description: Continuous Integration
name: Code Style
description: Code style, linting, code_checks

Comments:
@TomAugspurger i'm thinking that if/when we merge this to master maybe we should backport this otherwise we could end up with some auto backports failing and needing to do them manually. wdyt? (or we could defer this until post 1.1.1)
Merging and backporting sounds fine.

On Thu, Jul 30, 2020 at 9:38 AM Simon Hawkins <notifications@github.com>
wrote:

> @TomAugspurger <https://github.com/TomAugspurger> i'm thinking that
> if/when we merge this to master maybe we should backport this otherwise we
> could end up with some auto backports failing and needing to do them
> manually. wdyt?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/35470#issuecomment-666407014>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKAOIWWSOSR4IW6HGNGCBTR6GAXXANCNFSM4PL6UEUQ>
> .
>

will merge this tomorrow if no objections. 
Thanks @VirosaLi 
@MeeseeksDev backport to 1.1.x
CI: activate github actions on 1.1.x (PR only)
State: closed

same as #35459 against master. so should be OK when backported.

Labels:
name: CI
description: Continuous Integration

Comments:
some failures not seen in other PRs. will re-run and see if they repeat.
```
=========================== short test summary info ============================
FAILED pandas/tests/io/formats/test_format.py::TestDataFrameFormatting::test_repr_deprecation_negative_int
FAILED pandas/tests/io/formats/test_format.py::TestDataFrameFormatting::test_repr_min_rows
FAILED pandas/tests/io/formats/test_format.py::TestDataFrameFormatting::test_str_max_colwidth
FAILED pandas/tests/io/formats/test_format.py::TestDataFrameFormatting::test_east_asian_unicode_false
FAILED pandas/tests/io/formats/test_format.py::TestDataFrameFormatting::test_east_asian_unicode_true
FAILED pandas/tests/io/formats/test_format.py::TestDataFrameFormatting::test_datetimelike_frame
= 6 failed, 73402 passed, 1017 skipped, 1051 xfailed, 240 warnings in 1450.34s (0:24:10) =
```
The option to re-run has disappeared from Travis again. currently bisecting an issue in dev env. will create an empty commit later/tomorrow.
test failures unrelated
Do we even need the branches option or can we just remove and have it run against any target?
github actions updates the dev docs on the website. so only want that on master.
@simonjayhawkins feel free to merge things like this when you need.
@MeeseeksDev backport to 1.1.x
CI: activate github actions on 1.1.x (PR only) - DO NOT MERGE
State: closed

IIUC without changes to other scripts, activating GitHub actions for merge would push the docs. so this is only for PRs

If works. will open PR against master instead. and backport.

xref https://github.com/pandas-dev/pandas/pull/34800#issuecomment-644669496

Labels:
name: CI
description: Continuous Integration

Comments:
creating a PR using same branch re-triggered CI here. Github actions (and Travis) were green
Add note about limited propagation of attrs
State: closed

Explicitly mention that `attrs` are not retained once a series is placed into a dataframe, related to issue #35425

Labels:
name: Docs
name: Stale

Comments:
@TomAugspurger 
@RobertRosca is this still active? Could you address Tom's comment if so?
Thanks for updating - could you also address the comment

>  for now I think adding this note to basics.rst is best.

please?
Hey, sorry for the delay, forgot about the PR :sweat_smile: 

Had  a look at the `user_guide/basics.rst` page and I'm not sure where this would fit in there. There's a section ["Attributes and underlying data"](https://pandas.pydata.org/pandas-docs/stable/user_guide/basics.html#attributes-and-underlying-data) which has a reference `_basics.attrs` in the file, but that talks about metadata intrinsic to the table itself (shape, axis labels) not metadata added by users to `.attrs`. The `.attrs` dict itself doesn't seem to be mentioned on the basics page.

And since attrs is [marked as experimental](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.attrs.html), and adding your own metadata isn't really 'basic', I don't think it should be mentioned much on the page about essential basic functionality.

Personally I'd expect to find this kind of warning about the use of attrs on the doc page for  `pandas.Series.attrs`. I'm happy to put it somewhere in user_guide/basics.rst as well, but still think that a gotcha for the series attrs should be mentioned on the series attrs doc page.
Hey, I added the note but to a different location to what @TomAugspurger suggested, if Tom's okay with where it is then hopefully it can be merged :smiley: 
I don't think that a paragraph in the reference docs is the right place. It
should either go in the docstring or the user guide.

That said, the wording will need to be clarified. Right now it reads like
DataFrame just doesn't propagate attrs in any operation.
It shouldn't mention propagation, instead it should focus on the DataFrame
constructor not extracting attrs from individual Series.
Things like Series.to_frame() do correctly propagate attrs, so the focus
really should be on the constructor.

On Tue, Nov 3, 2020 at 2:54 AM Robert Rosca <notifications@github.com>
wrote:

> Hey, I added the note but to a different location to what @TomAugspurger
> <https://github.com/TomAugspurger> suggested, if Tom's okay with where it
> is then hopefully it can be merged 😃
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/35456#issuecomment-720985332>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKAOIQXSMGMZ7XGLJCXTKTSN7ALRANCNFSM4PLSR7PQ>
> .
>

Closing for now. @RobertRosca ping us know whenever you'd like to pick up and address @TomAugspurger feedback - we'll reopen
DOC: Add 1.1.1 whatsnew
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Whoop,s sorry I missed that you had opened this.
you got there first!
WIP: ENH: Add numba engine to groupby apply
State: closed

- [x] closes #31845
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Some notes:

- The passed function must be a reduction op
- The numba engine will drop the grouping column by default
- Can only operate on numeric data and will return float64

Preliminary default timing:

```
(pandas-dev) matthewroeschke:pandas-mroeschke matthewroeschke$ ipython
Python 3.8.3 | packaged by conda-forge | (default, Jun  1 2020, 17:21:09)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: df_g = pd.DataFrame({'a': range(10**4), 'b': range(10**4), 'c': range(10**4)})

In [2]: def f(x):
   ...:     return np.sum(x) + 1
   ...:

In [3]: df_g.groupby('a').apply(f)
Out[3]:
          a      b      c
a
0         1      1      1
1         2      2      2
2         3      3      3
3         4      4      4
4         5      5      5
...     ...    ...    ...
9995   9996   9996   9996
9996   9997   9997   9997
9997   9998   9998   9998
9998   9999   9999   9999
9999  10000  10000  10000

[10000 rows x 3 columns]

In [4]: %timeit df_g.groupby('a').apply(f)
3.07 s ± 57.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [5]: df_g.groupby('a').apply(f, engine='numba', engine_kwargs={'parallel': True})
Out[5]:
            0        1
0         1.0      1.0
1         2.0      2.0
2         3.0      3.0
3         4.0      4.0
4         5.0      5.0
...       ...      ...
9995   9996.0   9996.0
9996   9997.0   9997.0
9997   9998.0   9998.0
9998   9999.0   9999.0
9999  10000.0  10000.0

[10000 rows x 2 columns]

In [6]: %timeit df_g.groupby('a').apply(f, engine='numba', engine_kwargs={'parallel': True})

510 ms ± 3.46 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

Labels:

Comments:
Hello @mroeschke! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-08-05 18:13:49 UTC
BUG: GroupBy.apply() throws erroneous ValueError with duplicate axes
State: closed

- [x] closes #16646 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

*Behavioural Change*
`GroupBy.apply` would sometimes throw an erroneous `ValueError: cannot reindex from duplicate axis` because when it checked if the output was mutated from its original shape it would only check the index, when sometimes it should check the columns. 

*Tests*
One new test added on `test_apply.py` which fails on master because of the bug; passes with this PR. One existing test was marked *xfail* but the *xfail* is now removed. One existing test previously had to manipulate the expected index, but that is no longer necessary. 

All of the copy-pasteable examples from #16646 are fixed with this PR. 

Labels:
name: Bug
name: Groupby
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
lgtm. pls merge master and ping on green.
@jreback I think this is ready to merge. 
thanks @smithto1 very nice
BUG: CategoricalIndex.format
State: closed

- [x] closes #35439
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

I've temporarily put the whatsnewentry in the v.1.1.0 release note, because there isn't a v.1.1.1 version yet. I'll move it, before this is merged.

Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Categorical
description: Categorical Data Type

Comments:
Thanks @topper-123 for the PR.

The regression was caused by #35118. Categorical types other than object were also affected. maybe need to parameterise test with other values for `cols`

```
>>> pd.__version__
'1.2.0.dev0+10.g3b1d4f1ee'
>>>
>>> data = [[4, 2], [3, 2], [4, 3]]
>>> cols = [1, None]
>>> res = pd.DataFrame(data, columns=cols)
>>> print(res)
   1  NaN
0  4    2
1  3    2
2  4    3
>>>
>>> res = pd.DataFrame(data, columns=pd.CategoricalIndex(cols))
>>> print(res)
   1    NaN
0    4    2
1    3    2
2    4    3
>>>
```

```
>>> pd.__version__
'1.0.5'
>>>
>>> data = [[4, 2], [3, 2], [4, 3]]
>>> cols = [1, None]
>>> res = pd.DataFrame(data, columns=cols)
>>> print(res)
   1  NaN
0  4    2
1  3    2
2  4    3
>>>
>>> res = pd.DataFrame(data, columns=pd.CategoricalIndex(cols))
>>> print(res)
   1.0  NaN
0    4    2
1    3    2
2    4    3
>>>
```

> I've temporarily put the whatsnewentry in the v.1.1.0 release note, because there isn't a v.1.1.1 version yet.

doc\source\whatsnew\v1.1.1.rst now merged to master
Updated.
thanks @topper-123 very nice!
@MeeseeksDev backport to 1.1.x
BUG: tuple-of-tuples indexing results in NumPy VisibleDeprecationWarning
State: closed

- [ ] closes #35434
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Nested Data
description: Data where the values are collections (lists, sets, dicts, objects, etc.).

Comments:
> this needs either an actual fix in index construction

to prevent nested tuples in the index?
I'll close this for now to clear the queue, I'll revisit once on top of 1.1.0 regressions. There could be lots to do here to avoid 7 repeated calls to np.asarray and maintain state for the indexer checks.
CLN/PERF: move RangeIndex._cached_data to RangeIndex._cache
State: closed

The ``._cached_data`` attribute is not necessary. It was originally added to allow a check to see if the ``._data`` ndarray had been created, but that's also possible to do by a ``"_data" in _cache`` check in the new implemention, which IMO would be more idiomatic.

The new implementation has the benefit that the ``_data`` will be available to new copies of a RangeIndex, saving the need to create a new ndarray for each new copy of the RangeIndex.

```python
>>> idx = pd.RangeIndex(1_000_000)
>>> idx[[1, 4]]  # this accesses ._data and saves it in cached_data (master) or _cache["_data"](this PR)
>> %timeit idx._shallow_copy()[[1, 4]]
2.55 ms ± 69.3 µs per loop  # master
17.7 µs ± 405 ns per loop  # this PR
```

xref #26565.


Labels:
name: Performance
description: Memory or execution speed performance
name: Index
description: Related to the Index class or subclasses
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
@topper-123 can you rebase
LGTM pending rebase+green
MAINT: Fix issue in StataReader due to upstream changes
State: closed

Avoid creating an array of dtypes to workaround NumPy future change

closes #35426

- [X] closes #35426
- [ ] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: IO Stata
description: read_stata, to_stata

Comments:
Probably no reason to do this for 1.1.0. Are you OK with waiting for 1.1.1 @simonjayhawkins?
> Probably no reason to do this for 1.1.0. Are you OK with waiting for 1.1.1 @simonjayhawkins?

no problem.
moving milestone. The Stata tests are no longer failing with Linux py37_np_dev on master so this won't need backport.
Should we just close this as unneeded?  The second fix does appear to be
needed or at least a good idea since assigning Nan to an int array has a
funny smell.

On Wed, Jul 29, 2020, 11:33 Simon Hawkins <notifications@github.com> wrote:

> moving milestone. The Stata tests are no longer failing with Linux
> py37_np_dev on master so this won't need backport.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/35427#issuecomment-665585142>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABKTSRJF4NFNURKEPDR4JMDR573I7ANCNFSM4PIUO5VA>
> .
>

> The second fix does appear to be needed or at least a good idea since assigning Nan to an int array has a funny smell.

Thanks. I'm not sure where we are with numpy-dev. We still have 20 fails on master. not sure whether further changes in numpy will resolve these or we still have work to do.

> Should we just close this as unneeded? 

up to you. 
@simonjayhawkins I reverted the second commit so this just a small, sensible always safe refactor.
Restarted the Travis build, LGTM pending green
@jbrockmendel green.
Thanks @bashtage
TST: adding test for .describe() with duplicate columns
State: closed

- [x] closes #35423 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

While working on #35314 I made a change that broke `GroupBy.apply` if you called it on a DataFrame with duplicate columns; however, this break was not caught by any test. Was just caught on inpsection: https://github.com/pandas-dev/pandas/pull/35314#discussion_r460000612

This PR is just to add a test for this case to ensure it works and doesn't break again in the future. 

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
@smithto1 tests are failing

```
=========================== short test summary info ============================
FAILED pandas/tests/groupby/test_function.py::test_describe_with_duplicate_output_column_names[True]
FAILED pandas/tests/groupby/test_function.py::test_describe_with_duplicate_output_column_names[False]
=== 2 failed, 70016 passed, 4046 skipped, 1022 xfailed in 547.03s (0:09:07) ====
```
Hello @smithto1! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-08-03 20:58:22 UTC
thanks @smithto1 very nice
CLN: _wrap_applied_output
State: closed

The majority of this change is the result of three operations:

1. Move code that returns up top, to avoid the heavily nested structure.
2. Move variable creation as close as possible to where they are used. Previously, in certain cases computations were being done and then going unused.
3. Combine duplicated code.

There are two sections that I was able to entirely remove:

````
ping = self.grouper.groupings[0]
if len(keys) == ping.ngroups:
    key_index = ping.group_index
    key_index.name = key_names[0]

    key_lookup = Index(keys)
    indexer = key_lookup.get_indexer(key_index)

    # reorder the values
    values = [values[i] for i in indexer]

    # update due to the potential reorder
    first_not_none = next(com.not_none(*values), None)
````

and

````
# GH5788 instead of stacking; concat gets the
# dtypes correct
from pandas.core.reshape.concat import concat

result = concat(
    values,
    keys=key_index,
    names=key_index.names,
    axis=self.axis,
).unstack()
result.columns = index
````

For the single test that I touched, it was checking that the categorical dtype of an index was being dropped after a groupby. I don't believe that is the correct behavior - that the categorical dtype should remain. I checked groupby/categorical bugs and didn't find any issues this closes.

Labels:
name: Groupby
name: Clean
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Codecov is finding a lot in this method that isn't hit. I'm going to see what tests can be added or branches that can be removed.
> The majority of this change is the result of three operations:

To the extent that you can split this into independent pieces, it will be easier to review.
CLN: clarify TypeError for IndexSlice argument to pd.xs
State: closed

- [x] closes #35301 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

As per discussion in #35301 we do not support `IndexSlice` arguments to `xs`. This PR aims to clarify the `TypeError` thrown when that happens.

I created a separate issue to house the discussion re: functionality of `loc` vs `xs` (#35418).

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Error Reporting
description: Incorrect or improved errors from pandas
name: MultiIndex

Comments:
thanks @arw2019 very nice (failure is unrelated)
great! thanks @jreback for reviewing
DOC: update DataFrame.to_feather docstring
State: closed

- [x] closes #29532
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

The issue was that `DataFrame.to_feather` had an `fname` argument which in fact could be either a filename or a buffer. Most of the work to close #29532 was done in #30338 where `fname` argument was deprecated in favor of `path`. Here I update the docstring to let users know that they can use a buffer. 

Labels:
name: Docs

Comments:
@simonjayhawkins 
LGTM
thanks @arw2019