BrandonFors/pandas-github-issues huggingface dataset:
PERF: Improve performance for df.duplicated with one column subset
State: closed

- [x] closes #45236
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

```
%timeit df["a"].duplicated()
3.16 ms ± 96 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
%timeit df.duplicated(subset=['a'])
3.1 ms ± 75.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

The Index creation and the ``get_group_index`` were both at fault for the slowdown. Removing the Index call should improve the performance for longer subsets too.

Labels:
name: Performance
description: Memory or execution speed performance
name: duplicated
description: duplicated, drop_duplicates

Comments:
thanks @phofl 
@phofl several cases where the changes maybe viewed as breaking changes by users. maybe need to expand the release note or revert this PR.
I'd rather expand the release note than reverting, because the new behavior seems to be correct?
It is now only "correct" where we have a single column DataFrame and previously the DataFrame methods had bugs. I think that an inconsistency between a single column DataFrame (or single subset) is less desirable than the existing inconsistency between the Series and DataFrame methods.

Of course this is only applicable to the few cases reported and that all the bugs reported apply to object dtype columns only.

So probably not necessary to revert.
Update development docs recommended pyenv version to a supported version
State: closed

**Only doc change**

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


As far as i understand - python development should be done on python 3.8 or 3.9


Labels:
name: Docs

Comments:
thanks @alonme
Add test for concat with categorical + datetime dataframes
State: closed

- [x] closes #39443
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry (N/A)


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Stale

Comments:
can you merge master and address comments
Okay - looks like the behaviour of the code in the CI is working to concat: 

```python
# test catergorical dataframes both containing datetimes
df1 = DataFrame({"x": Series(datetime(2021, 1, 1), index=[0], dtype="category")})
df2 = DataFrame({"x": Series(datetime(2021, 1, 2), index=[1], dtype="category")})
result = pd.concat([df1, df2])
```

With a result of: 

```python
DataFrame({"x": Series([datetime(2021, 1, 1), datetime(2021, 1, 2)]), dtype="category"})
```

This is different to what I have locally, but probably just because I have an old version built? 

Can I just assume this is the expected behaviour and write tests to satisfy that? It seems logical that if both DataFrames are categorical, the results should be the union of the two categories. 
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@ryangilmour if you can merge master and update
Thanks for the pull request, but it appears to have gone stale. Closing, but if interested in continuing please merge in main, address the comments and the team can have another look.
BUG: df.getitem returning copy instead of view for unique column in dup index
State: closed

- [x] closes #45316
- [x] closes #41062
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

I think this is more intuitive (and "more" correct) than the previous behavior.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Copy / view semantics

Comments:
perf impact?
> perf impact?

Good point. Let's check for uniqueness or that the key is unique, so we only have the drop duplicates call if we have non unique columns
@phofl can you merge master and some comments
Done
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Merged master again
@phofl could you merge in one more time. Looks pretty good.
Done
thanks @phofl very nice as always
test replace on out of bound datetime
State: closed

Just adds a test for the functionality that is working on `main`. Placed the test in amongst the other `DataFrame.replace` tests. 

- [x] closes #36782
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: replace
description: replace method

Comments:
thanks @ryangilmour
REGR: fix period dtype <-> arrow roundtrip for pyarrow < 4
State: closed

Closes #45470

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: IO Parquet
description: parquet, feather
name: Arrow
description: pyarrow functionality

Comments:
Is this ready? can we merge and get this backported or move to 1.4.1 before tomorrow morning since if still open and on the 1.4 milestone tomorrow I will not start the release.
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.
REGR: read_fwf interpreting infer as list of colspecs when checking names length
State: closed

- [x] closes #45337
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

No whatsnew needed if this goes into 1.4, otherwise I will add one for 1.4.1 when available

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: IO Fixed Width
description: read_fwf

Comments:
thanks @phofl 
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.
REF: use more explicit to_numpy(object) instead of astype(object) in EA implementation
State: closed

Somewhat related to https://github.com/pandas-dev/pandas/issues/24877 / https://github.com/pandas-dev/pandas/pull/43699, but regardless of whether we want to stop `astype` returning numpy arrays or not, I think when we know we want to get an object-dtype numpy array as result, using `to_numpy(object)` is more explicit about this when reading the code.

Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Stale

Comments:
Can't speak to the general case, but DTA is more performant in astype

```
In [3]: dti = pd.date_range('2016-01-01', periods=10**5, freq='s', tz="US/Pacific")

In [4]: dta = dti._data

In [5]: %timeit dta.astype(object)
50.8 ms ± 2.13 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [6]: %timeit dta.to_numpy(object)
178 ms ± 6.26 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
```
Good catch. That seems to be because in datetimelike `__array__` we explicitly go through a list, which shouldn't be needed:

https://github.com/pandas-dev/pandas/blob/fefccfe10a34d244ec1df0bba5792fbc894c4402/pandas/core/arrays/datetimelike.py#L313-L316

We can probably call astype there (or do the same than astype does). Astype special cases object dtype for datetime64 to directly call `ints_to_pydatetime`:

https://github.com/pandas-dev/pandas/blob/fefccfe10a34d244ec1df0bba5792fbc894c4402/pandas/core/arrays/datetimelike.py#L410-L422

(which is what `list(self)` will also call, but then iteratively)
xref #44484 for dispatch logic
Yes, although the behaviour differences I note in that issue are for other dtypes, not object dtype (for object, it's only a matter of performance, as far as I see)
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
get_dummies to select string dtype in addition to object and categorical
State: closed

- [x] closes #44965
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Sure - would this go under `Bug Fixes > Strings` or `Other enhancements`?
Would put it under bug fixes
Should a test that uses `df = df.convert_dtypes() ` be added? #44965
thanks @lukemanley
ENH: ExtensionEngine
State: closed

Follow-up on https://github.com/pandas-dev/pandas/pull/43930/

Avoid object-dtype cast.

Next/last PR in this sequence introduces NullableEngine.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
(given that I asked for the status of this work, a small ping is always handy)

Repeating my question from on the original PR (https://github.com/pandas-dev/pandas/pull/43930/#pullrequestreview-778767750):

Can you give a bit more high-level context on how you approached it? 
For example, for the `NullableEngine` (here only `ExtensionEngine` for now), you are currently not using any hash table. Did you look at that / decide that is not possible or desirable? Or is that a potential future improvement, and you are focusing first on getting it working with a base implementation? 
The general `ExtensionEngine` seems to work with an actual ExtensionArray. An alternative could be to have it work an an ndarray that the EA could provide? Although a potential disadvantage of that approach is then that such an ndarray needs to be materialized always (in case this EA -> ndarray conversion is costly), while the the current way doesn't need that (but also cannot make use of existing optimized engines). So for the general ExtensionEngine, this is maybe a good approach. But are you also planning to do that for the NullableEngine?

This might also require some performance checking to investigate which approach is preferable. For example, I did a quick test comparing the performance of the ExtensionEngine (doesn't use hash-table) vs ObjectEngine (uses hash-table, which was used as fall-back before this PR) on one use case (`get_indexer`, which makes use of the hashtable if possible):

```
In [1]: idx_ea = pd.Index(np.arange(1_000_000), dtype="Int64")

In [2]: idx_object = idx_ea.astype(object)

In [4]: indexer = np.arange(500, 1000)

In [5]: %timeit idx_ea.get_indexer(indexer)
19 ms ± 418 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [6]: %timeit idx_object.get_indexer(indexer)
122 µs ± 4.43 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
```

So this is a case where the custom ExtensionEngine of this PR actually caused a big slowdown compared to the object engine fallback that we used before for Index[EA]. 
(now, I didn't profile this specifically, there might be some bottleneck in some EA method that gets called here, so there might also be other ways to improve the performance here instead of using a hashtable

(and specifically for the masked arrays as I used in this example, this might also be solvable in a specialized `NullableEngine`
https://github.com/pandas-dev/pandas/issues/45652 reported a performance regression which is I think essentially the same as what I showed above (or in any case, also caused by ExtensionIndex being slower than ObjectIndex). 

We should probably also add benchmark cases for those new index dtypes / engines (if we don't have that yet, didn't check)
Test to datetime null to NaT
State: closed

- [x] closes #35888
- [x] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Looks like there's typing errors:

```
pandas/tests/tools/test_to_datetime.py:1074: error: List item 0 has incompatible type "None"; expected "NaTType"  [list-item]
```
@mroeschke - thanks for the review. I'll reduce the size of the test series. I've now got the dev environment set up to run mypy locally now. But I'm unsure how to change the code to avoid typing errors.

Mypy complains with either of the following `Series` objects because they have an incompatible type. 

```
Series([NaT] * 20 + [None] * 20, dtype="object")
Series([None] * 20 + [NaT] * 20, dtype="object")
```

Possible fixes: 

- use `# type: ignore` 
   - not seen this in any other tests, so doesn't seem appropriate 
- ignore the test case 
   - is it not a valid test case? 
   - it is a reasonable/possible input to the `to_datetime()` function, so I feel it ought to be tested 
 - something else?
I would just use `# type: ignore[list-item]` for now, and if you're interested you can look into adjusting the typing of `to_datetime` in a follow up PR.
I think we DO need to test a larger input (as well) here e.g. 'It only occurs if the input is large enough to force the caching mechanism.'
@ryangilmour can you merge master and address comments
Yep - i'll get round to this in the next few days. 
thanks @ryangilmour
TST: Add test inconsistency in group by (#44803)
State: closed

- [X] closes #44803
- [X] tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby

Comments:
The code fails one test, but the test it's named _pytest (actions-310-numpydev.yaml, not slow and not network, xclip, deprecate, -W error)_.

Can someone help me with the error?
The failure is unrelated. Could you merge main again?
Done !
thanks @NumberPiOso
CI: Troubleshoot
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration

Comments:
My theory proved incorrect in https://github.com/pandas-dev/pandas/pull/45500. Looks like you're having more luck here?
looks like azure is green.  time to whittle these down...
great
any luck here?
can we force the windows builds to a single cpu? (and maybe disable some expensive tests / keep 2 builds)
> any luck here?

slowly narrowing it down
OK! looks like its somewhere in usecols...
im bashing my head against the wall here, need to tag someone else in on this
can we look at the past CI runs when we were green and see what changed in the config?
The build that just failed is almost identical to one that passed a few hours ago (see the "remove usecols" commit above 71bf824).  That commit also removed the entire usecols directory, while this one left `usecols/__init__.py`
> can we force the windows builds to a single cpu? (and maybe disable some expensive tests / keep 2 builds)

Trying this approach in https://github.com/pandas-dev/pandas/pull/45547. The CI is really backed up right now so it might take a while to get to
> The CI is really backed up right now so it might take a while to get to

Could/should we just cancel some of the non-fixing-this azure jobs to speed this up?
> > The CI is really backed up right now so it might take a while to get to
> 
> Could/should we just cancel some of the non-fixing-this azure jobs to speed this up?

Yeah if there are lot of queued up jobs it would allow to debug this quicker. But looks like the queue cleared up overnight
azure green!
> azure green!

Confident it was this or do you think retriggering the CI will confirm this?
ok let's merge on green (i retriggered)
let's also an issue to troubleshoot this as well (this case you are skipping)
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.
TST: Add tests string series min max
State: closed

Add tests for max and min aggregation at string series

In line with #31746

- [X] closes #31746
- [X] tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Strings
description: String extension data type and string data

Comments:
thanks @NumberPiOso
BLD: Fix pydata-sphix-theme version GH45374
State: closed

- [x] closes #45374 
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Build
description: Library building on various platforms
name: Docs
name: Dependencies
description: Required and optional dependencies

Comments:
@jorisvandenbossche should we backport this?
changing milestone to 1.4.1
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.
(sidenote: if we don't use the dev version from git anymore, we can probably move it to the conda packages section of the file)
REF: mask values in loc.__setitem__ with bool indexer
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

The motivation here is the 1-line change in Block.setitem, which will enable us to share Block.setitem with EABackedBlock.setitem.

cc @phofl thoughts on how to make this prettier or more robust?

Labels:
name: Refactor
description: Internal refactoring of code
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
This is tricky, in case of Series/DataFrame I think reindexing the value should work. In case of objects without indexes (e.g. np.array) this should raise I think
> This is tricky, in case of Series/DataFrame I think reindexing the value should work. In case of objects without indexes (e.g. np.array) this should raise I think

I like this idea, will update to let non-Series/DataFrame fall through.
Updated to address comments and add tests.  special-cases all-False cases with FIXME comments to punt on https://github.com/pandas-dev/pandas/pull/45501#discussion_r789214644.

https://github.com/pandas-dev/pandas/runs/4909709364?check_suite_focus=true
status here?
> status here?

ready to go.  looking forward to some nice de-duplication this will allow.
CI/TST: Disable tests/window/moments for Azure timeouts
State: closed

- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

My hypothesis (so far) why Azure is timing out is because `pandas/tests/window/moments` tests are hanging on the teardown step. Skipping these tests to see if it fixes things.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
I'm having not much luck here. My idea was incorrect that it was strictly the moments tests
>I'm having not much luck here. My idea was incorrect that it was strictly the moments tests

OK if i cancel the azure builds for this so #45507 gets run sooner?
> > I'm having not much luck here. My idea was incorrect that it was strictly the moments tests
> 
> OK if i cancel the azure builds for this so #45507 gets run sooner?

Yup go for it. (I don't have access to canceling the Azure builds from the Azure site)
Going to close. This wasn't the right approach
BUG: Adds newline after info() call on empty dataframe
State: closed

- [ ] closes #45494 
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

This fix improves the formatting of the output of pd.DataFrame.info() when called on an empty DataFrame by adding a newline, per the above issue.

I don't know if this is the proper style for adding newlines, but it appears to pass the linting tests. I'm pretty new to open-source development so let me know if anything should be changed.

Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
Thanks - please add a test too
Thanks for working with me!
thanks @EzraBrauner
BUG: allow DBAPI2 object #45416
State: closed

- [x] closes #45416



Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
thanks @fangchenli 
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.
Thanks @fangchenli
DOC: Update import statement
State: closed

Corrected Conventional import statements

- [x] xref https://github.com/pandas-dev/pandas/issues/21749
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Hello @parthi-siva! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-01-20 12:01:38 UTC
thanks @parthi-siva
Rename to main
State: closed

- [ ] see #39577
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
@mroeschke maybe backport? 1.4 or 1.4.1
@meeseeksdev backport 1.4.x
thanks @jarrodmillman 
Something went wrong ... Please have a look at my logs.