BrandonFors/pandas-github-issues huggingface dataset:
BUG: RangeIndex.astype('category')
State: closed

There's currently inconsistent behaviour when converting `RangeIndex` to `CategoricalIndex` using different methods. This fixes that.

```python
>>> ridx = pd.RangeIndex(5)
>>> pd.CategoricalIndex(ridx).categories
RangeIndex(start=0, stop=5, step=1)  # both master and this PR
>>> ridx.astype("category").categories
Int64Index([0, 1, 2, 3, 4], dtype='int64')  # master
RangeIndex(start=0, stop=5, step=1)  # this PR
```

In general, when supplying a Index subclass to `Categorical`/`CategoricalIndex`, the new categories should be of that type (unless the supplied index is a `CategoricalIndex` itself).

Discovered working on tests for #41153.

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Index
description: Related to the Index class or subclasses

Comments:
looks fine ex @jbrockmendel comment.
I`ve updated.
thanks @topper-123  (and also for fixing exact!)
REF: simplify libreduction
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
sure
many more tiny PRs in libreduction and groupby coming up.  this is a PITA so im taking it in tiny steps.
COMPAT: fix _bootlocale import error on py310
State: closed

The _bootlocale module has been removed from Python 3.10.

https://github.com/python/cpython/commit/b62bdf71ea0cd52041d49691d8ae3dc645bd48e1

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Python 3.10

Comments:
thanks @fangchenli
TYP: ExtensionArray unique and repeat
State: closed

Typing fixes for `ExtensionArray.unique()` and `ExtensionArray.repeat()`


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Right, my thought is making those Literal would make it more obvious to
readers that it is for DataFrame

On Sun, May 2, 2021 at 6:57 PM Irv Lustig ***@***.***> wrote:

> *@Dr-Irv* commented on this pull request.
> ------------------------------
>
> In pandas/core/arrays/interval.py
> <https://github.com/pandas-dev/pandas/pull/41260#discussion_r624845240>:
>
> > @@ -1518,7 +1518,11 @@ def delete(self: IntervalArrayT, loc) -> IntervalArrayT:
>          return self._shallow_copy(left=new_left, right=new_right)
>
>      @Appender(_extension_array_shared_docs["repeat"] % _shared_docs_kwargs)
> -    def repeat(self: IntervalArrayT, repeats: int, axis=None) -> IntervalArrayT:
> +    def repeat(
> +        self: IntervalArrayT,
> +        repeats: int | Sequence[int],
> +        axis: int | None = None,
>
> @jbrockmendel <https://github.com/jbrockmendel> No, we can't accept those
> arguments in ExtensionArray.repeat(), because we call numpy.repeat
>
> —
> You are receiving this because you were mentioned.
>
>
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/41260#discussion_r624845240>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB5UM6DILM4G7I7URYA2SDTTLX7I7ANCNFSM4364DQ3A>
> .
>

> Right, my thought is making those Literal would make it more obvious to readers that it is for DataFrame

@jbrockmendel Except that there is no `DataFrame.repeat()`, so I'm not sure what you mean here. 
pd._typing.Axis is intended for use in Series/DataFrame methods, not in EA methods.  But that is not obvious to all readers (as evidenced by the initial suggestion to use it here).  So I was suggesting an edit that might make it clearer to the next reader.

Out of scope for this PR.
> as

right my main point here is that we should have exactly 1 canonical way to type things and it should be obvious. If its not it needs lots of comments / expl and is not ideal at all, otherwise we are going to get inconsistencies across the board. if there is a way to enforce this would be great.

in any event this PR is fine.
fix(documentation/user_guides/indexing): Highlight `where`, add Todo for "wrong" example
State: closed

- [X] add Todo marker for "wrong" example (description said `where` should be used but actually it isn't)
- [ ] add actual example using `where` (beyond my skillset)



Labels:

Comments:
I am not sure this is worth merging with the TODO. you are welcome to actually change to what you believe is the correct docs.
I understand, but I actually was looking at this part of the docs, because I don't know how to do it... 🤷
Thanks @JoeSchr for the PR. rather than the TODO, if not fixing, maybe open an issue describing the problem instead. That would have more visibility.
TYP: Typing for ExtensionArray.__getitem__ 
State: closed

Changes for typing of `ExtensionArray.__getitem__()`.

Required changes to `DateTimeLikeArrayMixin` for correct compatibility.


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@simonjayhawkins can you review, please?
> Thanks @Dr-Irv generally lgtm. issue with api regarding using Sequence that may require some discussion

Thinking some more, probably best to remove Sequence altogether here.

A tuple of ints is a Sequence[int] and returns a Scalar on our 1d arrays in some cases and errors in others, so we need to add an extra overload for tuple to make the the annotations correct. This will then likely create LSP issues with the subclasses that can be backed by a 2d array as the return type of and tuple of ints will be inconsistent will the base implementation.

> @Dr-Irv can you also add annotations to the other array subclasses, Categorical, Interval, Masked, ArrowStringArray and Sparse before catering for the 2d array subclasses

Done in next commit.  Created issues in the block and array managers where now the `type ignore` comments have to change, as we are strictly defining EA's to be 1D.  So I changed those comments.
@simonjayhawkins  pushed changes a week ago, so looking forward to a new review.
@simonjayhawkins @jbrockmendel Based on our discussion today, I think this is where we want to be.  The base EA class `getitem()` is accepting single dimensional arguments, but the mixin classes accept 2D versions, including tuples.

Can you do a pass through the current version and see if there is anything else to change?
@simonjayhawkins ping...
pinging @simonjayhawkins 
weekly ping @simonjayhawkins 
another ping @simonjayhawkins 
same @Dr-Irv 

cc @twoertwein if you can have a look as you are looking at similar things
@jreback all green now (except for the one test on Windows that always seems to time out)
thanks @Dr-Irv
Backport PR #40994: REGR: memory_map with non-UTF8 encoding
State: closed

Backport PR #40994

Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
I think this PR depends on some not-backported PRs?
this might be non-trival to make work. ping when / if ready.
I will not have time to look into this for the next ~two weeks. The whatsnew entry should probably be moved to 1.3.
This makes this compatible with the logic on master
> The whatsnew entry should probably be moved to 1.3.

this fixed a regression in 1.2 https://github.com/pandas-dev/pandas/issues/40986#issuecomment-847005267, so would be good to backport.

The changes do no look trivial which would normally be desirable for a backport.

@twoertwein @phofl @jreback happy to merge this?
I think the only change is 
```
            errors = self.errors if self.errors is not None else "strict"
```

which is necessary because we have a new keyword on master setting this flag.
Thanks @twoertwein @phofl
TYP: Simple type fixes for ExtensionArray
State: closed

Typing changes for the following methods in `ExtensionArray`

- `__iter__`
- `__contains__`
- `fillna`
- `transpose`
- `ravel`
- `__hash__`

Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks @Dr-Irv very nice
CLN: Move stuff in tests.indexes.test_numeric.py to more logical locations
State: closed

This does four things to better organize the numeric index's tests:
1. Moves test method `test_index_groupby` from `tests.indexes.test_numeric.py::TestNumericIndex` to `tests.indexes.common.py::Base` and makes it use a fixture. This means this tests will be used for all index types. Also deletes `TestNumericIndex`, as its empty now.
2. Moves the `Numeric` class from `test_numeric.py` to `common.py` and renames it `NumericBase`.
3. Moves the `tests.indexes.numeric.py::TestArithmetic::test_arithmetic_explicit_conversions` tests to the new `common.py::NumericBase` and delete `test_numeric.py::TestArithmetic` class, as it's now empty.
4. Moves the file  `tests.indexes.test_numeric.py` to `tests.indexes.numeric.test_numeric.py`.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean
name: Index
description: Related to the Index class or subclasses

Comments:
looks fine, can you rebase.

@jbrockmendel ok here?
LGTM; needs rebase
Ok, I’ve rebased.
thanks @topper-123
TYP: Typing changes for ExtensionArray.astype
State: closed

Minimal (hopefully) typing changes for `ExtensionArray.astype` .  


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@simonjayhawkins can you review, please?
This is now failing mypy with master merged. can you merge master and fixup.
@simonjayhawkins  pushed changes a week ago, so looking forward to a new review.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@jbrockmendel @simonjayhawkins got marked stale due to lack of review/approval....
> Thanks @Dr-Irv
> 
> is it worth posting the results of a test file with reveal_types for this?

Can you point me to an example?
@simonjayhawkins ping...
pinging @simonjayhawkins 
@Dr-Irv thanks for the patience. can you rebase

@simonjayhawkins if you can have a look
weekly ping @simonjayhawkins 
another ping @simonjayhawkins 
biweekly ping @simonjayhawkins 
@Dr-Irv ok if you can merge master and make sure still passing (i think it was last) time, will merge these.
@jreback all green except for a Windows timeout on one of the tests
can you rebase again just to be sure (as merged the other)

cc @twoertwein 
@jreback all green after merge with the delete/searchsorted changes
thanks @Dr-Irv
CLN: remove outdated arrow compat
State: closed

precursor of #41207 


Labels:
name: Clean

Comments:
Thanks @fangchenli 
> I suppose that in the actual code the only change is the change in indentation? (git/github doesn't seem to understand that)

You can ask github to ignore whitespace changes:
![image](https://user-images.githubusercontent.com/6618166/116788136-057ef000-aa76-11eb-9a12-23e71ab38888.png)
[ArrowStringArray] PERF: use pyarrow.compute.utf8_length if available
State: closed

```
[ 25.00%] ··· strings.Methods.time_len                                                                                                                    ok
[ 25.00%] ··· ============== ==========
                  dtype                
              -------------- ----------
                   str        19.1±0ms 
                  string      6.82±0ms 
               arrow_string   1.93±0ms 
              ============== ==========
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Strings
description: String extension data type and string data

Comments:
thanks comment above
CLN: standardize naming in core.groupby
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Groupby
name: Clean

Comments:
looks fine. can you rebase and ping on green.
ping
REGR: Avoid error when defining non existing usecols with python engine
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Regression on master so no whatsnew. Restored behaviro of 1.2.4
 cc @gfyoung

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: IO CSV
description: read_csv, to_csv

Comments:
thanks @phofl
REF: make libreduction behavior match _aggregate_series_pure_python
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Groupby

Comments:
nice!
There's more coming up in this sequence of PRs.  At some point further simplification will need decisions on #41090 and #28900.
Optimising Series.nunique for Nan values #40865
State: closed

- [ ] closes #40865


Labels:
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
Hello @KenilMehta! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-05-03 05:05:01 UTC
thanks @KenilMehta 

can you add a whatsnew entry (improved perf of Series.nunique)

and do we have asv's for this? (if not we would like to add them), and run to show the improvements. (could also just show via a timeout on master vs your PR)
@jreback 
I am new to this. Can you please tell me what do you mean by a asv? 
https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#running-the-performance-test-suite
1. What is new in the PR?
As @rhshadrach pointed out while raising this issue, 
Currently we first remove nans, then use len() on the result of Series.unique. Except for Series that are mostly null values, it is more performant to switch the order of these operations.
I have done exactly the same thing in the PR by first finding the unique values of series and then removing nans from the uniques values.

2. Is there an asv for this?
I could not find any asv for this. I have added an asv for this in the PR.

3. Show the improvements.
I have written a small script which I ran against my PR and the master.

```
import time
import pandas as pd
import numpy as np

def calculateTime(part_nan):
    n = 100_000
    ser = pd.Series(n * (part_nan * [np.nan] + list(range(100)))).astype(float)

    startTime = time.time()
    n = ser.nunique()
    endTime = time.time()
    print(f"time taken {(endTime - startTime)*1000} {part_nan}")


calculateTime(0)
calculateTime(10)
calculateTime(100)
calculateTime(250)
calculateTime(300)

```

Following are the results from master:
```
time taken 134.84644889831543 0
time taken 128.6320686340332 10
time taken 159.5449447631836 100
time taken 196.3047981262207 250
time taken 206.98261260986328 300

```
Following are the results from PR:
```
time taken 72.15404510498047 0
time taken 75.88934898376465 10
time taken 108.44254493713379 100
time taken 166.73660278320312 250
time taken 187.24393844604492 300

```
We can clearly see that the time taken is less in the PR as compared to master.

@jreback 
Tried to ans all the things that you asked for.

can you add a whatsnew note in doc/source/whatsnew/v1.3.0.txt in the perf section. pls ping when greeen.
@jreback 
Added whatsnew in doc/source/whatsnew/v1.3.0.txt file. 
All builds passed, it is green.
thanks @KenilMehta
REF: simplify kwargs unpacking in resample
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Simplified kwargs unpacking in resample, by utilizing keyword-only arguments.

Labels:
name: Refactor
description: Internal refactoring of code
name: Resample
description: resample method

Comments:
thanks @ivanovmg
COMPAT: float() exception type for py310
State: closed

`float()` raises TypeError in py310

```
________________ TestEnsureNumeric.test_non_convertable_values _________________

self = <pandas.tests.test_nanops.TestEnsureNumeric object at 0x7f7da1e029e0>

    def test_non_convertable_values(self):
        msg = "Could not convert foo to numeric"
        with pytest.raises(TypeError, match=msg):
            nanops._ensure_numeric("foo")
    
        # with the wrong type, python raises TypeError for us
        msg = "argument must be a string or a number"
        with pytest.raises(TypeError, match=msg):
>           nanops._ensure_numeric({})

pandas/tests/test_nanops.py:771: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

x = {}

    def _ensure_numeric(x):
        if isinstance(x, np.ndarray):
            if is_integer_dtype(x) or is_bool_dtype(x):
                x = x.astype(np.float64)
            elif is_object_dtype(x):
                try:
                    x = x.astype(np.complex128)
                except (TypeError, ValueError):
                    try:
                        x = x.astype(np.float64)
                    except (TypeError, ValueError) as err:
                        # GH#29941 we get here with object arrays containing strs
                        raise TypeError(f"Could not convert {x} to numeric") from err
                else:
                    if not np.any(np.imag(x)):
                        x = x.real
        elif not (is_float(x) or is_integer(x) or is_complex(x)):
            try:
>               x = float(x)
E               TypeError: float() argument must be a string or a real number, not 'dict'

pandas/core/nanops.py:1600: TypeError
```

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Python 3.10

Comments:
thanks @fangchenli
BUG: read_excel trailing blank rows and columns
State: closed

- [x] closes #41167
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: IO Excel
description: read_excel, to_excel

Comments:
Not an expert here, but could you run asvs for read_excel?
@phofl here are the asvs on my machine (asv run -E existing --bench ReadExcel)

engine | this branch | master
-|-|-
xlrd | 35.8±0.1ms | 36.1±0.1ms 
openpyxl | 166±3ms | 165±0.4ms
odf | 678±5ms | 692±4ms

The asv test data has no trailing cells, so we don't see a measurable impact. I also tested both branches on a sample .xlsx file with 1000 rows × 2 columns and a single formatted cell on row 0, column 2**14.
[blank_cell_XFD1.xlsx](https://github.com/pandas-dev/pandas/files/6418200/blank_cell_XFD1.xlsx)

master 2.84 s
this branch 48.9 ms
(both times best of 5)
thanks @ahawryluk 
lgtm thanks @ahawryluk
DOC: Add hint for "Product space too large"
State: closed

Adds a note to the documentation for `DataFrame.groupby` suggesting that the cause of a `Product space too large to allocate arrays!`  error may be from unexpected behavior when grouping on a combination of sparse categorical columns.

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs
name: Groupby

Comments:
Feels like it should be better to add this information to the error message itself instead.
Hello @mobiusklein! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-10-04 01:08:36 UTC
+1 on @datapythonista's comment, I would recommend clarifying the error message itself.
I have modified the error message, and adjusted the test to expect the new error message. Note, this error isn't thrown in `groupby` but inside an implementation detail method, which looked generic but apparently is only called by `groupby` right now. If it starts being called by other things, the error message may be misleading.
@rhshadrach Thank you for finding the exact call site. I'll do that.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
I will work on this when time permits.
@mobiusklein - can you merge master. Some failures on the CI, but fairly certain they are unrelated.
@rhshadrach I think that there is some sanity checking at a lower level, but the tests may fail on a 32 bit system where even the reduced cross-product space is too large to allocate. I've merged master in and will check where the failures are again.
@mobiusklein - my apologies! I failed to notice that the context manager I asked to be added does not work on Windows. With this, I think it'd be best to just remove it (but otherwise leave the tests as-is).
@mobiusklein - do you still want to work on this? Looks almost there 
Looks like  `test_groupby_categorical_crossproduct` is still failing on py38 32 bit
if you can merge master will look
I've merged in `master` and added a branch case to the test. I was not able to conveniently find the right shape for 32 bit architectures, so settled for at least testing the expectation the call will fail on 32-bit architectures  
this PR is way too complicated. I am ok with changing the error message but there is a lot of extraneous stuff here.
Agreed. I think it should be suitable to append to the existing error message to use `observed=True` if data is categorical. Don't think it's completely necessary to test for the extra message 
closing. this would need a much simpler PR to merge.
REF: maybe_apply_* in core.apply
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Followup to #41067. cc @jbrockmendel 

Labels:
name: Refactor
description: Internal refactoring of code
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
sorry for the slow feedback; LGTM
BUG: to_string truncation row with index=False
State: closed

- [x] closes #40904 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry
  - Already have one for the same bug, not sure if I need one for this.


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
lgtm. if @pandas-dev/pandas-core or @pandas-dev/pandas-triage can have a look to double check.
I checked how ``to_string`` looks on this branch.
I confirm that the new behavior is favorable.
@jreback @ivanov Is the PR ready for merge? I don't see the option to complete it from my side.
> ...@ivanov Is the PR ready for merge? I don't see the option to complete it from my side.

I think you meant to tag @ivanovmg 
thanks @mikeronayne