BrandonFors/pandas-github-issues huggingface dataset:
CLN: remove internal usage of integer_array()
State: closed

The `integer_array()` function stems from a time that `pd.array(..)` didn't exist yet. But since now `pd.array` can do this, IMO we shouldn't be using that outside of the integer module (eg I didn't add a `floating_array(..)`).

Only in `tests/arrays/integer/tst_construction.py` I still left the usage of `integer_array()` as is (but probably could also change it there). Could in principle remove the function altogether, since it is just a small wrapper around the private `arrays.integer.coerce_to_array`

Labels:
name: Clean
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
>  need a whatsnew to mention that we are removing

This is not public, so no need for a whatnew I think? (I am only removing *internal* usage, there was no public exposure of this)
> > need a whatsnew to mention that we are removing
> 
> This is not public, so no need for a whatnew I think? (I am only removing _internal_ usage, there was no public exposure of this)

i diagree this has been exposed for a while (in pandas.core.arrays), which is public-ish. i am ok with just removing it, but need to have a note.
`pandas.core.arrays` is not public. The array classes *are* exposed, but in `pandas.arrays`
(anyway, I won't further argue against adding a note if you insist (it's less work to write that sentence than writing this comment ;)), but just want to be clear this is something completely internal)
> (anyway, I won't further argue against adding a note if you insist (it's less work to write that sentence than writing this comment ;)), but just want to be clear this is something completely internal)

thanks, i would also completely remove the usage of this as noted above.
Yes, will update `test_construction.py` to stop using it as well
@jorisvandenbossche i think this is worth putting on 1.2. ping when green.
can you merge master
>  i think this is worth putting on 1.2. 

Is there a reason to include it for 1.2? (since it's only an internal clean-up)
> > i think this is worth putting on 1.2.
> 
> Is there a reason to include it for 1.2? (since it's only an internal clean-up)

no this is fine for 1.3. thanks
REF: avoid catching all exceptions in libreduction
State: closed

Most of the libreduction edits are from #35417

Labels:
name: Refactor
description: Internal refactoring of code

Comments:
> Do we know now that the function in libreduction will never raise an error?

The relevant question isn't "will it ever raise" but "will it raise in the cython path but not in the python path".  AFAICT IndexError is the only non-contrived way that happens
do you have objections @jorisvandenbossche 
> "will it raise in the cython path but not in the python path". AFAICT IndexError is the only non-contrived way that happens

OK, I see, it's only removing the error catching specifically from calling the function on the chunk (which was re-raised as InvalidApply), not general errors from libreduction code
this looks fine. @jorisvandenbossche any comments?
BUG: IntervalIndex.union with mismatched dtypes both empty
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Ever so slowly ironing out the wrinkles between the setops.

Labels:
name: Bug
name: Index
description: Related to the Index class or subclasses

Comments:
technically needs a whatsnew
pandas/core/strings/accesory.py Unnecessary dict call - rewrite as a literal #38138
State: closed

xref #38138

File modified: pandas/core/strings/accesory.py

Changed unnecessary dict calls for flake8 (C408) standard:

- Rewrite dict() as {}

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @UrielMaD
BUG: isin numeric vs string
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Using the new hashtables!

```
In [3]: arr = np.arange(10**7).astype(np.int32)

In [4]: %timeit isin(arr, arr[:10])
88.8 ms ± 497 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)   # <-- master
42.7 ms ± 385 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)   # <-- PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
nice!
I don't think a whatsnew note is needed here
@jbrockmendel can you check if this regresssion is related? https://pandas.pydata.org/speed/pandas/#series_methods.IsIn.time_isin?python=3.8&Cython=0.29.21&p-dtype='int64'&commits=11d0176c-5b91febe

(this commit seems the most likely in the range https://github.com/pandas-dev/pandas/compare/11d0176c...5b91febe)
Good catch, looks like we're casting [1, 2] to ndarray[object] instead of ndarray[int].  ill see if theres an easy fix.
BUG: Validate drop_duplicates ignore_index argument for bool
State: closed

- [x] closes #38274
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Saw this on the tracker. I think we should validate this, some inputs could lead to weird results.
The last drop duplicates entry in whatsnew was also under reshaping, so put it there

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
thanks @phofl
Removing dict calls
State: closed

- [ ] xref #38138 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Removed the dict calls for the files:

`./pandas/core/arrays/floating.py`
`./pandas/core/arrays/numpy_.py`
`./pandas/core/arrays/interval.py`

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
@arw2019 thanks for your attention! I have used flake8 for the three files that I modified and it returned nothing for me, can you help me on how I can solve this issue?
> @arw2019 thanks for your attention! I have used flake8 for the three files that I modified and it returned nothing for me, can you help me on how I can solve this issue?

if you run the precommit tool locally it will take care of this
Hey @arw2019 looks like my modifications passed the checks by removing the conflict with another branch. Thanks for your help.
thanks @zanuttin
test groupby.indices for multiple groupby and mix of types
State: closed

Creates tests for GH26859

- [X] closes #26859
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Does this warrant an entry in whatsnew?

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Stale

Comments:
> Thanks @alexifm for the PR!
> 
> Some comments. More generally the second function is a lot to take in, wondering if it can be simplified somehow

yea, probably can just run the groupby on all the columns and remove the parametrization.  I'd have to check again if there was anything special about 1 vs multiple fields in a groupby.  Or perhaps split into two simpler tests?
> yea, probably can just run the groupby on all the columns and remove the parametrization. I'd have to check again if there was anything special about 1 vs multiple fields in a groupby. 

Unless they're direct replicas of each other we likely want to test both

>Or perhaps split into two simpler tests?

i'd say split into two tests
@arw2019 
> Unless they're direct replicas of each other we likely want to test both

The difference is whether the keys are tuples or not:
```
>>> df.groupby(['a', 'b']).indices
{('a', Timestamp('2018-01-01 00:00:00')): array([0]),
 ('b', Timestamp('2018-02-01 00:00:00')): array([1]),
 ('c', Timestamp('2018-03-01 00:00:00')): array([2])}

>>> df.groupby(['a']).indices
{'a': array([0]), 'b': array([1]), 'c': array([2])}
```
So nearly identical behavior but subtly different.  I split the tests but can bring them back together.
@alexifm can you merge master and update to comments
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
closing as stale. if you want to continue, pls ping and can re-open.
BUG: DataFrame.apply with axis=1 and EA dtype
State: closed

- [x] closes #36785
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Apply
description: Apply, Aggregate, Transform, Map
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @jbrockmendel for the PR.

The code sample in #36785 still fails, but now with `ValueError: Length of passed values is 2, index implies 4.`

but happy to close #36785 if we feel that the underlying issue is solved.
can you add a release note to 1.1.5 in the regression fixes section
> but happy to close #36785 if we feel that the underlying issue is solved.

looks like there maybe still an issue. 
ill take another look this afternoon
> looks like there maybe still an issue.

thanks @jbrockmendel. code sample in OP is now passing.

before the edit, the existing `test_invert_array` was failing on ci which had more than 1 row. so that case is covered.

The sample in the OP was still failing and the dataframe had more than 1 column. Since only the one test was failing, it appears that this case is not covered by existing tests.

maybe we should an additional test case to `test_apply_axis1_with_ea`, maybe...
```
        df = DataFrame(date_range("2020", periods=3, tz="UTC")).T
        result = df.apply(lambda x: x, axis=1)
        tm.assert_frame_equal(result, df)
```
@meeseeksdev backport 1.1.x
@simonjayhawkins im falling behind on GH notifications.  do i need to make a PR adding more tests or something here?
The PR was merged so I assume not.
BUG: name retention in Index.difference corner cases
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug

Comments:
lgtm. i guess needs a whatsnew note; ping on green.
ping
BUG: Inserting array of same size with Series.loc raises ValueError
State: closed

closes #37748
closes #37486
closes #38271
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
cc @phofl
Also does this solve #38271
@arw2019 
> Also does this solve #38271 

It seems so:

```
In [1]: import numpy as np 
   ...:     ...: import pandas as pd 
   ...:     ...:  
   ...:     ...: ser = pd.Series(1, index=list("abcde"), dtype="object") 
   ...:     ...:  
   ...:     ...: expected = pd.array([0,0,3,0,0]) 
   ...:     ...: ser.loc["a"] = expected 
   ...:     ...: result = ser[0]                                                                                                                                                                                                               

In [2]: ser                                                                                                                                                                                                                                    
Out[2]: 
a    [0, 0, 3, 0, 0]
b                  1
c                  1
d                  1
e                  1
dtype: object
```
cc @phofl and @jbrockmendel if any comments.
lgtm otherwise
thanks @ma3da
C408 rewrite dict call to dict literals
State: closed

- [ ] xref #38138
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry
Identify unnecessary dict call - rewrite as a literal in  `./pandas/tests/io/pytables/test_store.py`, `./pandas/core/arrays/interval.py`



Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
I am not sure how to assign someone for review.
@jreback  Would you be able to review this?
thanks @vmdhhh 

you don't assign reviews, they will just happen.
BUG: to_hdf and HDFStore for subclasses
State: closed

- [ ] closes #33748
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Fix the bug that the subclasses of a dataframe and a series cannot be saved ``to_hdf``.
The tests were added.
When testing I found that ``assert_frame_equal`` does not work on subclasses as well - probably need to fix it in a separate PR.
For now in the tests I compare values only.

Labels:
name: IO HDF5
description: read_hdf, HDFStore

Comments:
thanks @ivanovmg
C408 fix
State: closed

- [x] xref #38138 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Rewrote dict calls to dict literals in the following files
./pandas/tests/io/json/test_ujson.py
./pandas/tests/io/json/test_normalize.py

Also changed list calls to literals in line 730 of the following file
pandas/tests/io/json/test_ujson.py

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @gerardjorgensen
BUG: groupby.apply on the NaN group drops values with original axes return
State: closed

- [x] closes #38227
- [x] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Groupby
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Hello @GYHHAHA! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-12-03 08:24:42 UTC
thanks @GYHHAHA very nice, keep em coming!
CI: delete unused import in groupby/groupby.py
State: closed

Fixes a `flake8` bug on current master:
```
(pandas-dev) andrewwieteska@Andrews-MacBook-Pro pandas % flake8 pandas/core/groupby/groupby.py
pandas/core/groupby/groupby.py:53:1: F401 'pandas.core.dtypes.cast.maybe_cast_result' imported but unused
```

Labels:
name: CI
description: Continuous Integration

Comments:
thanks @arw2019
ENH: implement _should_compare/_is_comparable_dtype for all Index subclasses
State: closed

Fixes some problems in get_indexer_non_unique that AFAICT are not actually causing any problems.

Categorical kludge is needed because of #38240.

Labels:
name: Refactor
description: Internal refactoring of code
name: Index
description: Related to the Index class or subclasses

Comments:
de-duplicate away!
Fix dict call for pandas/tests/util/test_assert_series_equal.py
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Hello @UrielMaD! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-12-03 00:59:49 UTC
Fix C408 Errors in core/strings, core/arrays, core/dtypes...
State: closed

...core/computation, core/indexes, core/ops, io/pytables.py, tests/dtypes, tests/io/generate_legacy_storage_files.py, tests/frame/methods/test_to_records.py, tests/groupby/transform/test_transform.py

- [X] xref #[38138](https://github.com/pandas-dev/pandas/issues/38138)
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:

Comments:
Hello @austiezr! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





































There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-12-02 19:55:09 UTC
need to merge master.
closing as these have already been done, sorry about that
PERF: timeseries comparison ops
State: closed

- [x] closes #37462
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Locally this gets me to within 3-4% of 1.1.4 on the asv mentioned in #37462

Labels:
name: Datetime
description: Datetime data dtype
name: Performance
description: Memory or execution speed performance

Comments:
@jbrockmendel just confirming, but I believe this is for 1.1.5 right?
i think so, best to ask @simonjayhawkins 
> @jbrockmendel just confirming, but I believe this is for 1.1.5 right?

comment from https://github.com/pandas-dev/pandas/issues/37462#issue-731193131

> This seems to be caused by #37313

so IIUC no need to backport.
thanks @jbrockmendel
BUG: Limited available colors
State: closed

(GH37967)

- [x] closes #37967
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Visualization
description: plotting

Comments:
Not sure if bug is the good label...
Two small scripts used for that.
```
from pandas.io.formats.excel import CSSToExcelConverter
from matplotlib.colors import CSS4_COLORS

# check if existing colors match the ones of matplotlib
for name, color in CSSToExcelConverter.NAMED_COLORS.items():
    assert "#" + color == CSS4_COLORS[name]

# generate code for CSSToExcelConverter
for name, color in CSS4_COLORS.items():
    print(f'"{name}": "{color[1:]}",')
```
cc @charlesdong1991 
(removing 1.2 milestone, since this is not critical to include)
thanks @ma3da
BUG: Allow numeric ExtensionDtypes in DataFrame.select_dtypes
State: closed

- [x] closes #35340
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Picking up #35341 (original PR addressed all comments - here I merged master + added whatsnew note)

cc @simonjayhawkins @jreback @jbrockmendel 

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Green + addressed comments
Green + addressed comments
thanks @arw2019
BUG: CategoricalDtype.__repr__ with RangeIndex categories
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

The problem was introduced a couple weeks ago, so shouldn't need a whatsnew.

Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Regression
description: Functionality that used to work in a prior pandas version
name: Categorical
description: Categorical Data Type

Comments:
what is the issue where this is are regression? can you add a whatsnew note
There is no issue.  The bug isnt in any released version, so doesn't need a whatsnew.
kk
REGR: unstack on 'int' dtype prevent fillna to work
State: closed

- [ ] closes #37115
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
thanks @simonjayhawkins hopefuly will backport cleanly
@meeseeksdev backport 1.1.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.1.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 cb04ad78b91dbb176cf11454285f9794f5a96ab6
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am "Backport PR #38244: REGR: unstack on 'int' dtype prevent fillna to work"
```

4. Push to a named branch :

```
git push YOURFORK 1.1.x:auto-backport-of-pr-38244-on-1.1.x
```

5. Create a PR against branch 1.1.x, I would have named this PR:

> "Backport PR #38244 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
TST: string representation of IntervalIndex
State: closed

- [x] closes #32553
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Picking up #36628

Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Interval
description: Interval data type

Comments:
thanks @arw2019
TST: setitem in presence of duplicate cols
State: closed

- [ ] closes #23239
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Add tests required to close the issue.
The tests are based on the example in the issue.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
thanks @ivanovmg
REF: consolidate result-casting in _cython_operation
State: closed

Orthogonal to #38235.  After this and 38235, we'll be down to only two usages of maybe_cast_result, only one of which is sketchy.

Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby

Comments:
great
yep, couple more nice PRs in this sequence and then one that involves a tough choice + ugly code
TYP: handle mypy ignore in pandas/_testing
State: closed

- [ ] xref #37715
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Handle mypy ignore comments in ``pandas/_testing.py``, function ``write_to_compressed``.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
It's green, except failing ``TestHDFStore.test_append_with_data_columns`` on Linux py39.
REF: casting in _python_agg_general
State: closed

Consolidate two for-loops into one.

Instead of using maybe_cast_result in two places, use it only once, immediately after the function call being wrapped.  Replace those two usages with the less-invasive `maybe_downcast_to_dtype`

Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby

Comments:
looks like between this and the other groupby-casting PR we caused an unused import in groupby.py.  I'll push a fix to the next PR I rebase