BrandonFors/pandas-github-issues huggingface dataset:
REGR: undocumented astype("category").astype(str) type inconsistency between pandas 1.1 & 1.2
State: closed

- [ ] closes #41797
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Categorical
description: Categorical Data Type

Comments:
thanks @simonjayhawkins 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 817b25a3e8f216a528450f5a70393577e0b1404b
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #42087: REGR: undocumented astype("category").astype(str) type inconsistency between pandas 1.1 & 1.2'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-42087-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #42087 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@meeseeksdev backport 1.2.x
@meeseeksdev backport 1.3.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 817b25a3e8f216a528450f5a70393577e0b1404b
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #42087: REGR: undocumented astype("category").astype(str) type inconsistency between pandas 1.1 & 1.2'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-42087-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #42087 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
cc @simonjayhawkins attempted backport to 1.2.x & 1.3.x
> cc @simonjayhawkins attempted backport to 1.2.x & 1.3.x

1.3.x backport ok, 1.2.x needs manual. will do shortly.
BUG: groupby any/all raising with pd.NA object data
State: closed

- [x] closes #37501
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
thanks @mzeitlin11 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
PERF/CLN: to_csv
State: closed

- [ ] xref #41468
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

More of a cleanup than a performance improvement. Perf only seems to improve for multiindexes.
Will try to squeeze out more improvement.

Here are the asv's (run 3 times)
```
asv continuous -f 1.1 upstream/master HEAD -b ^io.csv.ToCSV
```
```
       before           after         ratio
     [5153334c]       [a7a839bd]
     <perf-to-csv~2^2>       <perf-to-csv>
-        28.3±5ms       13.6±0.8ms     0.48  io.csv.ToCSVDatetime.time_frame_date_formatting
-       2.77±0.1s       1.10±0.01s     0.40  io.csv.ToCSVIndexes.time_head_of_multiindex

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.

       before           after         ratio
     [5153334c]       [a7a839bd]
     <perf-to-csv~2^2>       <perf-to-csv>
-        29.8±6ms       15.9±0.3ms     0.53  io.csv.ToCSVDatetime.time_frame_date_formatting
-       2.51±0.2s       1.25±0.06s     0.50  io.csv.ToCSVIndexes.time_head_of_multiindex

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.

       before           after         ratio
     [5153334c]       [a7a839bd]
     <perf-to-csv~2^2>       <perf-to-csv>
-        934±30ms         767±30ms     0.82  io.csv.ToCSVIndexes.time_multiindex
-      3.49±0.08s       1.08±0.03s     0.31  io.csv.ToCSVIndexes.time_head_of_multiindex

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.

```


Labels:
name: Performance
description: Memory or execution speed performance
name: IO CSV
description: read_csv, to_csv
name: Stale

Comments:
@lithomas1 can you merge master and run the memory asvs
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
This PR has appear to have gone stale. Closing for now, but feel free to reopen if you're still interested in working on this.
ENH: `Styler.set_sticky` for maintaining index and column headers in HTML frame
State: closed

- [x] closes #29072

This allows you to quickly make the index and/or column headers stick in place when scrolling in HTML.
Additional args allow the control of subselecting specific multiindex levels and controlling the width /or height of the sticking elements.

https://user-images.githubusercontent.com/24256554/122395757-99017700-cf77-11eb-93f0-f3c401afca4e.mov




Labels:
name: Styler
description: conditional formatting using DataFrame.style

Comments:
thanks @attack68 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
BUG: avoid unnecessary casting in CategoricalIndex.reindex
State: closed

Doesn't appear to be anything user-facing.  The only unequivocally wrong bit is on L420 where the current condition is always truthy.

Labels:
name: Categorical
description: Categorical Data Type

Comments:
can you rebase. what's the bug?
> what's the bug?

It's not user-facing, just clearly-unintended code.  On L420 `if not isinstance(cats, CategoricalIndex) or (cats == -1).any():` always holds bc `cats` is an ndarray.
rebased + greenish
looks fine can you rebase
rebased + green
booyah.  bunch of MultiIndex PRs that are blockers for... more MultiIndex PRs.  Nothing urgent.
Allow passing custom sentinel into MultiIndex.format
State: closed

Commit b0468aa45f3 seems to have unintentionally removed functionality
to pass custom sentinel values for sparsifying in MultiIndex.format().
Previously, any "Truthy" value other than 1 and True was used as the
sentinel, which is useful particularly with adjoin=False. The current
logic does not make sense: it checks sparsify for "Falsey" values under
"if sparsify:" condition. Under this if sparsify cannot be Falsey.

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: MultiIndex
name: Deprecate
description: Functionality to remove in pandas

Comments:
Hello @eltoder! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-06-17 18:59:55 UTC
@jreback I agree. Ok to keep this change though? The current code doesn't make sense and changed the API
@jreback How about this?
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
closing as stale, if you want to continue working, please ping.
PERF: contiguity, less gil in join algos
State: closed

Broken off a branch working towards #13745

This doesn't have noticeable user-facing impact on its own since this is a smaller part of the merge operation. Some timings:

```
import numpy as np
import pandas._libs.join as libjoin
np.random.seed(0)
arr1 = np.random.randint(0, 100, 100000)
arr2 = np.random.randint(0, 100, 100000)
```

Master:
```
In [2]: %timeit libjoin.inner_join(arr1, arr2, 100)
1.19 s ± 17.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [3]: %timeit libjoin.left_outer_join(arr1, arr2, 100)
1.22 s ± 16.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [4]: %timeit libjoin.full_outer_join(arr1, arr2, 100)
1.26 s ± 33.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

This pr:
```
In [2]: %timeit libjoin.inner_join(arr1, arr2, 100)
729 ms ± 17.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [3]: %timeit libjoin.left_outer_join(arr1, arr2, 100)
714 ms ± 11 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [4]: %timeit libjoin.full_outer_join(arr1, arr2, 100)
715 ms ± 6.99 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
wow!
migth as well backport this
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
DOC: update styler user guide for ref to hiding cols and index and link to to_LaTeX
State: closed

should be backported to 1.3.0 as it references the features add there.


Labels:
name: Docs
name: Styler
description: conditional formatting using DataFrame.style

Comments:
Thanks @attack68 web and docs is failing on rate limit exceeded. As this PR changes docs will need to re-run later to make sure this is green.
@simonjayhawkins this is green now
thanks @attack68 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
BUG: hash_pandas_object ignores optional arguments when the input is a DataFrame.
State: closed

- [x] closes #41404
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

In this PR, I will try to solve issue #41404.
I would appreciate it if you could review this.

## About the issue

In the current `pandas.util.hash_pandas_object` function implementation, the hash_key argument is ignored when the input object type is a DataFrame.
This is a reproducible example.

```py
df = pd.DataFrame({"x": list("abc")})
new_hash_key = "a"*16
print("Series")
print(pd.util.hash_pandas_object(df["x"]).values)
print(pd.util.hash_pandas_object(df["x"], hash_key=new_hash_key).values)
print("DataFrame")
print(pd.util.hash_pandas_object(df).values)
print(pd.util.hash_pandas_object(df, hash_key=new_hash_key).values)
```

The output is here.

```
Series
[ 4578374827886788867 17338122309987883691  5473791562133574857]
[14372590553486244284 12017144478307119910    34790805896362026]
DataFrame
[ 4578374827886788867 17338122309987883691  5473791562133574857]
[ 4578374827886788867 17338122309987883691  5473791562133574857]
```

The last output should be
```
[14372590553486244284 12017144478307119910    34790805896362026]
```

## About the fix

When the input object is a DataFrame, the hash values are calculated by applying `hash_array` function for each Series.
However, optional arguments, including `hash_key`, are not passed to the hash_array function in the current implementation.
To fix this, I passed the optional arguments to the hash_array function.

Labels:
name: Bug
name: hashing
description: hash_pandas_object

Comments:
lgtm @i-aki-y can you add a release note bug fix section (Other is fine) for 1.3 ping on green.
@jreback Thanks, I have added a whatsnew entry.
thanks @i-aki-y 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
CI/TST: fix typo in #41982
State: closed

fix a typo in #41982, which is causing CI failure.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration
name: ARM
description: aarch64 architecture

Comments:
The PR causing the ci failures was backported. so backporting this too.
Thanks @fangchenli
BUG: Don't supply name to Series when using 'size' with agg/apply
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Ref: https://github.com/pandas-dev/pandas/pull/39935#discussion_r651989373

Doing e.g.

    print(DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]}).apply('sum'))
    print(DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]}).agg('sum'))

produce a Series with `name=None`. This was introduced in 1.3, so does not need a whatsnew.

Labels:
name: Bug
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
@jreback greenish; failure is unrelated

> FAILED pandas/tests/indexes/numeric/test_numeric.py::TestUInt64Index::test_constructor_does_not_cast_to_float
= 1 failed, 170123 passed, 6374 skipped, 874 xfailed, 6 xpassed, 352 warnings in 1414.92s (0:23:34) =

> @jreback greenish; failure is unrelated
> 
> > FAILED pandas/tests/indexes/numeric/test_numeric.py::TestUInt64Index::test_constructor_does_not_cast_to_float
> > = 1 failed, 170123 passed, 6374 skipped, 874 xfailed, 6 xpassed, 352 warnings in 1414.92s (0:23:34) =

now fixed.
Thanks @rhshadrach
BUG: MultiIndex.reindex with level and EA dtype
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Preliminary to a pretty nice de-duplication.

Labels:
name: Bug
name: MultiIndex

Comments:
release note?
PERF: groupsort_indexer contiguity
State: closed

Some low-hanging fruit
```
In [1]: import numpy as np
   ...: import pandas._libs.algos as libalgos
   ...:
   ...: np.random.seed(0)
   ...: inds = np.random.randint(0, 100, 500000)
   ...: ngroups = inds.max() + 1

In [2]: %timeit libalgos.groupsort_indexer(inds, ngroups)
```
`1.06 ms ± 11.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # this pr`
` 1.2 ms ± 39.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # master`


Labels:
name: Performance
description: Memory or execution speed performance
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
thanks @mzeitlin11
Regression raising Error when having dup cols with single dtype for read csv
State: closed

- [x] closes #42022
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


regression on master, so no whatsnew

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: IO CSV
description: read_csv, to_csv

Comments:
> is this for 1.3?

looks like it
Yes this is a regression on the rc
thanks @phofl 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
TST: Reduce number of numba tests run
State: closed

- xref https://github.com/pandas-dev/pandas/pull/41971#issuecomment-860607472
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: numba
description: numba-accelerated operations

Comments:
great merge on green.
CI: pin fsspec
State: closed

Hopefully fixes new Windows py37_np17 failures

Labels:
name: CI
description: Continuous Integration

Comments:
ok exciting, ping on green.
@simonjayhawkins we *may* want to do this for 1.2.5
sure. maybe best to just backport ci changes to 1.3 for now, and i'll go through the backports and backport any needed to 1.2.5
The azure failure timed out on this latest run, but succeeded on previous commit, so greenish
thanks @mzeitlin11 

@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
@meeseeksdev backport 1.2.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 89ef88a3a0b8c4e2670d8b020df7156878e27306
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #42023: CI: pin fsspec'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-42023-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #42023 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
BUG: passing str to GroupBy.apply
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Groupby

Comments:
lgtm, can you merge master one more time.
Thanks @jbrockmendel
Fix module name for _LoadSparseFrame
State: closed


Labels:
name: Stale

Comments:
what exactly reproduces this
@jreback that's just not where _LoadSparseFrame is defined. Check the other entry below. pandas.core.sparse.frame does not exist anymore.
> @jreback that's just not where _LoadSparseFrame is defined. Check the other entry below. pandas.core.sparse.frame does not exist anymore.

sure i get that. but what broke, we need a test that exercises this. we have older pickles which are deserialized just fine.
You can construct a case by pickling SparseDataFrame with an old version and trying to unpickle it. I couldn't find any unit tests for pickle_compat though.
> You can construct a case by pickling SparseDataFrame with an old version and trying to unpickle it. I couldn't find any unit tests for pickle_compat though.

we would for sure need a test case. we *already* have old pickled files that have sparse in them. So my question is how did you get this to fail?
> we would for sure need a test case. we _already_ have old pickled files that have sparse in them.

Mind pointing me to these?

> So my question is how did you get this to fail?

I didn't. I just read the code and it's obviously wrong.
https://github.com/pandas-dev/pandas/blob/master/pandas/tests/io/generate_legacy_storage_files.py and https://github.com/pandas-dev/pandas/tree/master/pandas/tests/io/data/pickle

we are prob not properly testing these legacy things. if you can generate an old pickle then the tests should fail w/o the patch and work with.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@eltoder i know writing tests for old pickles is a PITA, but the process is to use pandas/tests/io/generate_legacy_storage_files.py to write a pickle file containing objects created using old versions.  looks like we don't have any SparseDataFrame objects in there, so you may need to modify the script
Thanks for the PR, but it appears to have gone stale and just needs some tests to exercise this case. If you're interested in continuing we would be happy to reopen this PR, but closing in the meantime.
DOC: Fix wrongly associated docstring of Series.str.isspace
State: closed

cc @simonjayhawkins

- [ ] closes #42015
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
thanks @felixdivo 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
Updated the docstring for reset_index in frame.py
State: closed

If not all levels are removed, the index is not reset to default integer values.

import pandas as pd

x = pd.Series(range(3), index=pd.MultiIndex.from_arrays([['a', 'b', 'c'], [1, 2, 3]], names=['letters', 'numbers']))

print(x.reset_index('numbers', drop=True))

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs
name: MultiIndex

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
doc string update looks ok, but can we add the example you have above in the Examples section to make this clear.
can you merge master and update tto comments
Thanks for the PR but appears to have gone stale. If interested in continuing, please merge master and address the comments and we can reopen
Update frame.py
State: closed

If not all levels are removed, the index is not reset to default integer values.

import pandas as pd

x = pd.Series(range(3), index=pd.MultiIndex.from_arrays([['a', 'b', 'c'], [1, 2, 3]], names=['letters', 'numbers']))

print(x.reset_index('numbers', drop=True))

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:

Comments:
Hello @cf-vrgl! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/core/frame.py`](https://github.com/pandas-dev/pandas/blob/30fc3eeff5bab70b37658759988b957f7a39f366/pandas/core/frame.py):

> [Line 5618:74](https://github.com/pandas-dev/pandas/blob/30fc3eeff5bab70b37658759988b957f7a39f366/pandas/core/frame.py#L5618): [W291](https://duckduckgo.com/?q=pep8%20W291) trailing whitespace
CLN: Verify non-ndarray is an ExtensionArray in hash_array GH42003
State: closed

Closes #42003 

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Stale

Comments:
Hello @TheNeuralBit! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-06-14 20:31:18 UTC
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Thanks for the PR, but it appears it has gone stale and needs to address the review. Closing but happy to reopen if you're still interested in continuing.