BrandonFors/pandas-github-issues huggingface dataset:
PERF: DataFrame.insert
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

```
from asv_bench.benchmarks.indexing import *

self = InsertColumns()
self.setup()

%timeit self.setup(); self.time_insert()
32.5 ms ± 382 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)   # <- master
20.7 ms ± 744 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)   # <- PR

%timeit self.setup(); self.time_insert_middle()
37.2 ms ± 407 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  # <- master
25.1 ms ± 513 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  # <- PR

```

Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
looks fine. can you rebase
Add return_inverse to .unique in EA interface closes
State: closed

- [x] closes #24245 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
i am not sure how this PR helps here. this *already* raises an error if you try to use the parameter.
In the original issue, it was stated that it 
> should be adapted to allow for the possibility of the return_inverse-kwarg.

I will try to work on the actual implementation if I can. 
Looking at the implementation of `ExtensionArray.unique()`

> def unique(self: ExtensionArrayT) -> ExtensionArrayT:
        """
        Compute the ExtensionArray of unique values.
        Returns
        -------
        uniques : ExtensionArray
        """
        uniques = unique(self.astype(object))
        return self._from_sequence(uniques, dtype=self.dtype)

My suggestion would be to reverse `uniques` before it's passed to `self._from_sequence()`. Since `uniques` is a sequence of scalars, simply doing `uniques = uniques[::-1]` would reverse the values before being parsed into an ExtensionArray. Please let me know if that would work, and I will make changes accordingly.
> In the original issue, it was stated that it
> 
> > should be adapted to allow for the possibility of the return_inverse-kwarg.
> 
> I will try to work on the actual implementation if I can.

I am not sure we want to support this api, we don't anywhere else.
Backport fastparquet 0.7 compat (PR #42954 and #42919)
State: closed

- [ ] xref #42954 and #42919
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

**This is a non-trivial backport. Please first review changed files before merging. Thanks**


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: IO Parquet
description: parquet, feather

Comments:
What? How is CI picking up fastparquet 0.7.0. This needs more investigation...
Anyone on the core team want to restart just the database builds? I just want to confirm that this is not flaky.
Restarted, but it was indeed installing fastparquet 0.7.0 ...
Ok figured it out. fastparquet decided to actually enforce numpy>1.18 in 0.7.1.
Thanks! (and indeed, a follow-up PR for my comment is the easiest ;))
BUG: AbstractMethodError msg should begin w/ 'types'
State: closed

`types` and `methodtype` are swapped inside the `ValueError` raised in `AbstractMethodError`.

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. Closing for now, but if interested in continuing please address the review and we can reopen.
DOC/CLN: 1.3.2 release notes
State: closed


Labels:
name: Docs
name: Clean

Comments:
lgtm
i've converted to draft till #42984 is merged otherwise the auto-backport will fail
Something went wrong ... Please have  a look at my logs.
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
BUG: pd.to_datetime with format doesn't work with pd.NA
State: closed

- [x] closes #42957
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Hi @jreback - reverted to cython solution. Let me know if this looks good.
Hi @jreback - do you mind taking a look, if this is fine?
thanks @debnathshoham
DOC: updates documentations Closes #21749
State: closed

- [x] xref #21749 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Hello @salomondush! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:











There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-08-16 18:33:32 UTC
great does this close all of the items in the original issue?
I only changed those in the directory ./pandas/util/testing.py. Others have remained the same. 
thanks @salomondush happy to have a PR on other items in the original issue as well
TST: use fixtures for some sql database connections
State: closed

xref #40686

This PR:

1. Create fixtures for all supported SQL connections, including sqlalchemy engine/connection and sqlite3 connection. Setup and cleanup are handled within fixtures.
2. Migrate several common tests to use connection fixtures. They serve as examples for future migrations.

We probably should put all the fixtures in a separate conftest.py file. But since there are unmerged PRs related to this test file, I kept everything in one place for now to avoid conflict.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Thanks for the PR but appears that it has gone stale. Feel free to reopen if you have time to continue working on it!
@fangchenli happyt o take this if you can fixup
thanks @fangchenli
Backport PR #42838 on branch 1.3.x (REGR: Series.nlargest with masked arrays)
State: closed

Backport PR #42838: REGR: Series.nlargest with masked arrays

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
hmm I think this needs some mods to backport @jbrockmendel e.g. `final` is not allowed in 3.7
ENH: `max_rows` `max_cols` kw for `Styler.to_html` + `styler.options`
State: closed

one more item towards deprecating `DataFrame.to_html` for #41693 

- [x] adds options to config
- [x] update `options.rst`
- [x] add tests
- [x] update to_html docstring
- [x] whats new
- [ ] update user guide (will be follow on after all options in master tracker added)   


and an item towards consistent `pandas.options` for #41395

![Screen Shot 2021-08-29 at 09 26 45](https://user-images.githubusercontent.com/24256554/131242297-0ba1a7c6-c058-43f0-af94-c3acf14eca4a.png)


Labels:
name: Enhancement
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap
name: Styler
description: conditional formatting using DataFrame.style

Comments:
can you merge master
PERF: avoid repeating checks in interpolation
State: closed

Not a particularly big difference.

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
> pandas/core/missing.py:349: error: Incompatible return value type (got "Union[ExtensionArray, ndarray[Any, Any]]", expected "ndarray[Any, Any]")  [return-value]
pandas/core/missing.py:436: error: unused "type: ignore" comment
CI: Pin dateutil
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration

Comments:
can u open an issue about dateutil - making it work 
#42961
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
DOC: erroneous initialization of a DataFrame with Series objects #42818
State: closed

- [ ] closes #42818 
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

issue url : https://github.com/pandas-dev/pandas/issues/42818


Labels:
name: Docs
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
thanks @tyuyoshi
CLN: remove unused Series._index attribute
State: closed

Noticed this in https://github.com/pandas-dev/pandas/pull/42950, the attribute seems completely unused. 

(cc @jbrockmendel do you have any idea what the reason is/was for this?)

Labels:
name: Clean

Comments:
> do you have any idea what the reason is/was for this?)

no idea
yep, no idea what this was used for, thanks @jorisvandenbossche
TST: move index tests to correct files (#27045)
State: closed

- [ ] xref #27085 (chips away at, yes)
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

* https://github.com/pandas-dev/pandas/pull/42284 
* https://github.com/pandas-dev/pandas/pull/42896 
* https://github.com/pandas-dev/pandas/pull/42909
* https://github.com/pandas-dev/pandas/pull/42930
* https://github.com/pandas-dev/pandas/pull/42932

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Frequency
description: DateOffsets

Comments:
thanks @felixDulys
ERR: clarify PerformanceWarning for fragmented frame
State: closed

xref some confusion in #42477. If this is thought an improvement, could target 1.3.2 since more users now see this error than in 1.2.x

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: DataFrame
description: DataFrame data structure
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
lgtm ping on green
thanks @mzeitlin11 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
TST: GH34037 add test for type error when using parameter 'min_period'
State: closed

- [x] closes #34037 
- [ x] tests added / passed
- [x ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Window
description: rolling, ewma, expanding

Comments:
Thanks @horaceklai
BUG: df.drop not separating missing labels with commas
State: closed

- [x] closes #42881 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
thanks @zeitlinv
CLN: clean warning assertions in sql tests
State: closed


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
sorry should have merged this first, can you rebase
e.g. https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=64435&view=logs&j=bf526160-cc38-5046-7b49-5141c64a483a&t=6a92c4fe-2b0b-5acd-cfba-e7901042d9bd&l=57 i think there are some actual errors
thanks @fangchenli
BUG:Can't calculate quantiles from Int64Dtype Series when results are floats
State: closed


- [x] closes #42626
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
needs a release note. fixing a regression so target 1.3.2
thanks @debnathshoham 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
In summary and for the record this changed behavior from 1.2.5.

1.2.5 always returned an object array of floats when `q` is list-like. If `q` is a scalar the return type was always a Python float (as documented https://pandas.pydata.org/pandas-docs/dev/reference/api/pandas.Series.quantile.html)


<details>

```
>>> pd.__version__
'1.2.5'
>>> 
>>> result = pd.Series([1, 2, 3, pd.NA], dtype="Int64").quantile([0, 0.75])
>>> result
0.00    1.0
0.75    2.5
dtype: object
>>> 
>>> type(result[0])
<class 'float'>
>>> 
>>> result = pd.Series([1, 2, 3, pd.NA], dtype="Int64").quantile([0])
>>> result
0.0    1.0
dtype: object
>>> 
>>> type(result[0])
<class 'float'>
>>> 
>>> result = pd.Series([1, 2, 3, pd.NA], dtype="Int64").quantile([0.75])
>>> result
0.75    2.5
dtype: object
>>> 
>>> type(result[0.75])
<class 'float'>
>>> 
>>> result = pd.Series([1, 2, 3, pd.NA], dtype="Int64").quantile(0.75)
>>> result
2.5
>>> 
>>> type(result)
<class 'float'>
>>> 
>>> result = pd.Series([1, 2, 3, pd.NA], dtype="Int64").quantile(0)
>>> result
1.0
>>> 
>>> type(result)
<class 'float'>
```

</details>

1.3.2 will now return nullable integer (Int64) or numpy float (float64) depending on the values in the result when `q` is list-like. If `q` is a scalar the return type is now a numpy float or a numpy int. (inconsistent with documentation)

<details>

```
>>> pd.__version__
'1.4.0.dev0+415.g99cf794ae2'
>>> 
>>> result = pd.Series([1, 2, 3, pd.NA], dtype="Int64").quantile([0, 0.75])
>>> result
0.00    1.0
0.75    2.5
dtype: float64
>>> 
>>> type(result[0])
<class 'numpy.float64'>
>>> 
>>> result = pd.Series([1, 2, 3, pd.NA], dtype="Int64").quantile([0])
>>> result
pd.Series([1, 2, 3, pd.NA], dtype="Int64").quantil0.0    1
dtype: Int64
>>> 
>>> type(result[0])
<class 'numpy.int64'>
>>> 
>>> result = pd.Series([1, 2, 3, pd.NA], dtype="Int64").quantile([0.75])
>>> result
0.75    2.5
dtype: float64
>>> 
>>> type(result[0.75])
Series([1, 2, 3, pd.NA], dtype="Int64").quantile(0<class 'numpy.float64'>
>>> 
>>> result = pd.Series([1, 2, 3, pd.NA], dtype="Int64").quantile(0.75)
>>> result
2.5
>>> 
>>> type(result)
<class 'numpy.float64'>
>>> 
>>> result = pd.Series([1, 2, 3, pd.NA], dtype="Int64").quantile(0)
>>> result
1
>>> 
>>> type(result)
<class 'numpy.int64'>
```

</details>
DOC add example to reorder levels
State: closed

Response to changes suggested on my previous pull request

- [x] closes #42124
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Docs

Comments:
Hello @trevorkask! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-08-23 15:16:42 UTC
Are you going to close #42667 or this? You should really continue that PR and close this one.
use this one
Thanks @trevorkask
TST: reorganize misplaced quarter / business quarter tests (#27085)
State: closed

- [ ] xref #27085 (chips away at, yes)
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

* https://github.com/pandas-dev/pandas/pull/42284 
* https://github.com/pandas-dev/pandas/pull/42896 
* https://github.com/pandas-dev/pandas/pull/42909
* https://github.com/pandas-dev/pandas/pull/42930

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Frequency
description: DateOffsets

Comments:
thanks @felixDulys
BUG: Attributes skipped when serialising plain Python objects to JSON (#42768)
State: closed

This patch closes #42768 and fixes a previously-reported issue #33043.

When `ujson` falls back to encoding a Python object, it iterates over all of the non-callable attributes of the object which do not start with an underscore, using them as JSON keys. While iterating the `dirs(obj)` list, an index was incremented twice causing every second attribute to be skipped.

I've fixed this increment, and moved some of the declarations inside the function so that control flow is more clear. I've also added a regression test for this behaviour.

- [x] closes #42768
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

This patch also xref #41174, but I'm not sure that turning complex numbers into `{"real": ..., "imag": ...}` JSON objects was intended in the first place, so I haven't added a test for this.

Labels:
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
can you replicate the test in the OP
@jreback The test more or less duplicates the OP, just using regular classes rather than dataclasses. However, the expected output in the OP included properties beginning with underscores, wheras I have left these out of the resulting JSON because this is what the existing (intended, I assume) behaviour did. 

It does have user-visible effects, for example if classes are stored as elements in series or dataframes. The program:

```
import pandas as pd

class A:
    def __init__(self, a, b):
        self.a = a
        self.b = b

series = pd.Series([A(a=1, b=2)])
print(series.to_json())
```

used to return `{"0":{"a":1}}`, and now returns `{"0":{"a":1,"b":2}}`.
Thanks @mzeitlin11 - I would usually only change the part of the function which affected the logic, but the logic of that function was so unclear it seemed to warrant a cleanup anyway. For example on 928 it looks like something interesting is being assigned but actually NULL is being assigned. (Incidentally, how should I go about putting in the cleanup PR - just title it with "CLN" and reference this discussion?). 

@jreback ping
thanks @joelgibson
TST: reorganize misplaced year / business year tests (#27085)
State: closed

- [ ] xref #27085 (chips away at, yes)
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

* https://github.com/pandas-dev/pandas/pull/42284 
* https://github.com/pandas-dev/pandas/pull/42896 
* https://github.com/pandas-dev/pandas/pull/42909

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Frequency
description: DateOffsets

Comments:
thanks @felixDulys keep em coming!
PERF: Sparse Series to scipy COO sparse matrix
State: closed

- [x] closes #42880 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

I made the test more thorough, as it only checked the output's type, but not that the output matrix was actually the expected one.

Here is the output from the asv benchmark:
```
       before           after         ratio
     [e045034e]       [042dbbc3]
     <master>         <sseries-to-coo-perf>
-      12.6±0.6ms       2.35±0.1ms     0.19  sparse.ToCoo.time_sparse_series_to_coo_single_level(False)
-      19.9±0.6ms      3.68±0.05ms     0.19  sparse.ToCoo.time_sparse_series_to_coo(True)
-      19.6±0.4ms      3.51±0.09ms     0.18  sparse.ToCoo.time_sparse_series_to_coo(False)
-        12.5±1ms         220±10μs     0.02  sparse.ToCoo.time_sparse_series_to_coo_single_level(True)
```
As you can see the performance improvement is the most dramatic when there is only one row and column level, and for an output with sorted labels. This is simply due to the fact that the labels are sorted in the `levels` attribute of an `Index`, so I covered this case, which should be the most common, with a simpler (and faster) implementation. For this reason I was thinking maybe `sort_labels` could default to `True`, but that would probably require prior deprecation warnings and all. In any case I can add this information as a performance tip to the docs, let me know what you think.


Labels:
name: Performance
description: Memory or execution speed performance
name: Sparse
description: Sparse Data Type

Comments:
@jreback I think everything's ok now, the failing checks seem unrelated.
The one for sort_labels=False but row_level/column_levels a single level is the first one reported below:
```
       before           after         ratio
     [e045034e]       [621b1028]
     <master>         <sseries-to-coo-perf>
-      11.8±0.9ms      2.17±0.05ms     0.18  sparse.ToCoo.time_sparse_series_to_coo_single_level(False)
-        21.2±1ms       3.29±0.5ms     0.16  sparse.ToCoo.time_sparse_series_to_coo(True)
-      19.0±0.5ms       2.78±0.1ms     0.15  sparse.ToCoo.time_sparse_series_to_coo(False)
-      12.5±0.9ms         200±10μs     0.02  sparse.ToCoo.time_sparse_series_to_coo_single_level(True)
```
the one I mention in the docstring, which is the last one above and has better performance, is for sort_labels=True.
@TLouf lgtm.

@pandas-dev/pandas-core if any comments.
thanks @TLouf very nice!
STYLE: pre-commit check on get/set_option usage
State: closed

- [x] closes #38813
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

This is failing across multiple locations. Is there a specific location where get/set_option can be used? or across everything?


Labels:
name: Code Style
description: Code style, linting, code_checks
name: Stale

Comments:
> This is failing across multiple locations. Is there a specific location where get/set_option can be used? or across everything?

I think the idea is to replace with the context manager everywhere that `get/set_option` is not the system under test. Maybe best to start with a precursor PR changing all the places you find where it makes sense to do this replacement. Then in the PR adding the precommit check, can discuss which files it makes sense to skip for this check.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.