BrandonFors/pandas-github-issues huggingface dataset:
CI/TST: Use CI moto service instead of subprocess
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them



Labels:
name: CI
description: Continuous Integration

Comments:
@lithomas1 how feasible do you think it would be to move our Windows and MacOS builds to Github Actions given the current jobs there?

For Azure Windows, appears we can't use Linux containers
https://stackoverflow.com/a/55351920
https://stackoverflow.com/a/57751095

For Azure MacOS, appears it doesn't support containers in general
https://docs.microsoft.com/en-us/azure/devops/pipelines/process/service-containers?view=azure-devops&tabs=yaml#requirements


The goal here was to hopefully avoid flaky behavior of running a moto service in a subprocess by running it in a container instead.

Darn. For Github Actions, it looks like service containers only work for Ubuntu runners :(

https://docs.github.com/en/actions/using-containerized-services/about-service-containers#about-service-containers
@jreback thoughts about only running these s3 tests only in Linux Python 38 & 39 (and the Linux db build)? Before it was running in the Windows Python 38 & 39 builds and the Linux db build 

One can still run this locally on Windows or Mac with the subprocesses
> @jreback thoughts about only running these s3 tests only in Linux Python 38 & 39 (and the Linux db build)? Before it was running in the Windows Python 38 & 39 builds and the Linux db build
> 
> One can still run this locally on Windows or Mac with the subprocesses

sounds ok to me
ping when ready here
Ready and all green here. 
@lithomas1 if any comments
can you merge master
thanks @mroeschke 
18 failed tests when running tests locally on ubuntu with `aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host localhost:5000 ssl:default [Connect call failed ('127.0.0.1', 5000)]`

Is there something that needs to now be changed in the local dev setup also or should these tests be skipped locally?

on WSL 44 failed.
TST: Use subprocess.Popen as context managers
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

Might help address the elusive `ResourceWarnings` in the CI
https://github.com/pandas-dev/pandas/runs/4317219787?check_suite_focus=true

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
looks fine. you might want to propagate this upstream to the clipboard library itself as we vendor this.
PERF: Improve performance in read_csv with numeric index col
State: closed

- [x] closes #44158
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

The isin was calling the np.in1d and was checking against a bunch of strings causing the performance issues

Labels:
name: Performance
description: Memory or execution speed performance
name: IO CSV
description: read_csv, to_csv

Comments:
the benchmark runner failed, not really sure why
I think I have used the wrong base class, changed it
greenish now
thanks @phofl very nice
keep em coming!
ENH: improved dtype inference for Index.map
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Broken off from #43930

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
looks good. is this a user visible thing? if so can you add a whatsnew
An example to illustrate, assume we have an uint32 index:

```
In [1]: idx = pd.NumericIndex([1, 2, 3], dtype="uint32")

In [2]: idx
Out[2]: NumericIndex([1, 2, 3], dtype='uint32')

In [3]: idx.map(lambda x: x)
Out[3]: NumericIndex([1, 2, 3], dtype='int64')
```

On master we always get an int64 index back, while with this PR it tries to preserve the original dtype:

```
In [3]: idx.map(lambda x: x)
Out[3]: NumericIndex([1, 2, 3], dtype='uint32')
```

Now, I don't know if this is actually a user visible change compared to the released version, since the above example is with a non-int64 dtype, which is only in master (so in that sense you could see it as a bug fix in the new not-yet-released feature)
> Now, I don't know if this is actually a user visible change compared to the released version, since the above example is with a non-int64 dtype, which is only in master (so in that sense you could see it as a bug fix in the new not-yet-released feature)

Ah, we of course already had UInt64Index, for example, and that will now also preserve the dtype instead of returning Int64Index in the example above.
> is this a user visible thing? if so can you add a whatsnew

yes, will add whatsnew.
> The current tests you edited are (all?) simple lambda x: x checks that ensure the dtype is now preseved. Can you ensure we also test the case where a maybe_cast_pointwise_result needs to fallback? Eg idx.map(lambda x: x*1000) with an uint8 index, where the resulting values thus don't fit in the original dtype.

I do intend to do this (https://github.com/pandas-dev/pandas/pull/44609/files#diff-9090cca4ac914e526dfb58dd84ce7c75e2935e41cdeb3ee1618bb5068f05e2dbR549), the question is when.  This PR in its current state is just splitting the Index.map changes off from #43930.  I'm happy to take the time to do this The Right Way, but that does mean marginally slowing down the ExtensionIndex PR.
PERF: check first for recognized scalars in DatetimeArray._validate_scalar
State: closed

Something small, but most of the time we will be setting with actual scalars, so putting that first improves this case a bit:

```python
arr = pd.date_range("2012-01-01", periods=3).array
scalar = arr[0]

In [3]: %timeit arr._validate_setitem_value(scalar)
3.04 µs ± 58.5 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- master
2.08 µs ± 27.2 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- PR
```

(it gives a noticeable difference in eg fillna benchmarks)

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
The second commits gives another small improvement (the `_assert_tzawareness_compat` just before already ensures that self and other have either both a tz or both not):

```python
In [3]: %timeit arr._validate_setitem_value(scalar)
3.04 µs ± 58.5 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- master
1.85 µs ± 82 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- pR
```
thanks. at some point if you can add the collected PR numbers to the whatsnew in the perf improvements section.
@jreback they're mostly related to (or most relevant for) ArrayManager, so I should probably start with some actual basic docs about it before starting to mention it in the whatsnew
well u can certainly make a section in the whatsnew to indicate that perf for these have changed
DEP: df.insert not showing warning for allow_dups and timezoneaware and naive indexing
State: closed

cc @jbrockmendel 

This ensures that the warning is raised also for allow_duplicates.

Additional question:

Should this:

```
index = pd.date_range("1/1/2013", "1/2/2013")
index.insert(0, Timestamp("2012-12-30", tz="UTC"))
```

raise the warning?

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Timezones
description: Timezone data dtype
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Stale

Comments:
> Should this: [inserting tzaware Timestamp into tznaive DTI] raise the warning?

No.  It's behavior is not going to change.
In this case we could try avoiding the warning for insert completly, because its only raised because of column in self.columns and not because of the actual insert operation
failures i think are legit on https://github.com/pandas-dev/pandas/runs/4315745835?check_suite_focus=true
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
if you can rebase @phofl 
Thanks for the PR, but it appears to have gone stale. Feel free to reopen if you have time to revisit.
BUG: parse_dates=False while passing date_parser tries to use date parser
State: closed

- [x] closes #44366 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

I have updated the code in the `pandas/io/parsers/readers.py` since if we pased `parse_dates=False`, it would still change it to True.

Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
thx @loicdiridollou
BUG: concat_compat with 2D PeriodArray
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Period
description: Period data type

Comments:
looks fine. wasn't sure if this is user facing in any way, but if it is, pls add a note in followon
CI: Add single CPU core build for pytest.mark.single
State: closed

- [x] closes #44584
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

1. Adds MacOS, Windows and Linux builds where `PYTEST_WORKERS = 1` for `@pytest.mark.single` tests
2. Convert `@pytest.mark.high_memory` to `@pytest.mark.single` (seem to serve the same purpose)
2. Tries to mark all `s3_resource` fixture tests as `@pytest.mark.single` to hopefully reduce timeouts
3. Moves sqlite tests as  `@pytest.mark.db` tests

Labels:
name: CI
description: Continuous Integration

Comments:
Going to try another approach first.
BUG: Series[object].unstack preserve object dtype
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

Sits on top of #44593

Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
looks good if u can add a whatsnew
@jbrockmendel can you confirm whether the slowdown in https://pandas.pydata.org/speed/pandas/#reshape.ReshapeExtensionDtype.time_transpose?python=3.8&Cython=0.29.24&p-dtype='datetime64%5Bns%2C%20US%2FPacific%5D'&commits=8374c656 pointing to this is expected? I think it is, but wanted to confirm.
> can you confirm whether the slowdown in https://pandas.pydata.org/speed/pandas/#reshape.ReshapeExtensionDtype.time_transpose?python=3.8&Cython=0.29.24&p-dtype='datetime64%5Bns%2C%20US%2FPacific%5D'&commits=8374c656 pointing to this is expected? I think it is, but wanted to confirm.

No. I don't see how this would affect DataFrame.transpose.  More plausibly #44593 which this sits on top of?  Even then I'd need to take a look bc I did not expect that to hurt perf.
OK I think #44593 caused us to split a 2D DatetimeArray into column arrays, which would account for the slowdown, but is not intended.  Will look into a fix.
Perf regression fixed in #44724
PERF: improve find_common_type perf for special case of all-equal dtypes
State: closed

In several benchmarks, the `find_common_type` spends quite some time on checking if all dtypes are equal, which can be improved a bit by 1) assuming all dtypes are actual dtypes (`is_dtype_equal` is generic and will first convert the input to a dtype, giving overhead) and 2) writing the for loop generator in small cython helper:

```python
types = [np.dtype("float64") for _ in range(10000)]

In [2]: %timeit pd.core.dtypes.cast.find_common_type(types)
7.18 ms ± 627 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <-- master
368 µs ± 10.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <-- PR
```

(based on some testing, both aspects above contribute significantly to the improvement)


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
nice speedup!
DOC: timedelta return type
State: closed

- [x] closes #42316
- [NA] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [NA] whatsnew entry


Labels:
name: Docs
name: Timedelta
description: Timedelta data type

Comments:
Thanks @loicdiridollou! Awesome job. Happy to have more of these doc issues addressed!
BUG: join MultiIndex with overlapping IntervalIndex level (#44096)
State: closed

`BaseMultiIndexCodesEngine._extract_level_codes` relies on the assumption that
`get_indexer()` is always usable as any index level `is_unique`.  However, that
is not the case for overlapping `IntervalIndex` levels.  `get_indexer()` can
not be used on an `IntervalIndex` that `is_overlapping` even if it `is_unique`.

This patch uses `get_indexer_for()` instead.

~This patch checks if the index level is an overlapping `IntervalIndex` and then
calls `get_indexer_non_unique()` instead.~

~We should not switch to `get_indexer_non_unique()` for other index levels than
overlapping `IntervalIndex`, even though that actually would work as well.
That is because `get_indexer_non_unique()` comes with a performance penalty of
roughly 30% compared to `get_indexer()`, measured by simple `%timeit`
measurements.~

- [x] closes #44096 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex
name: Interval
description: Interval data type

Comments:
@johannes-mueller can you merge master and ping on green
@jreback rebased


thanks @johannes-mueller
REF: remove always-True DataFrame._iset_item kwarg
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
@jorisvandenbossche any idea why this would be failing on the AM build?  my only guess is that yesterday's change from skipping to xfailing is interacting with the weird statefullness of the tests in test_expressions
I have seen that before the last days (maybe in that PR, not fully sure, the tests were passing in the last commit there), but always only appeared from time to time (restarting might "solve" it), and I am not able to reproduce it locally. 

> change from skipping to xfailing is interacting with the weird statefullness of the tests in test_expressions

The `test_expressions.py` itself is not using any skip/xfails, so I have a hard time understanding how that would be related. 

Looking at the failures, I think all have to do with for some reason not actually using numexpr, when the test is thought to be using it. Will have a check if any of the tests we currently skip->xfail might meddle with those settings.
> I have seen that before the last days

that's pretty strong evidence that the skip->xfail change is unrelated.  thanks for taking a look.
>> I have seen that before the last days
>
> that's pretty strong evidence that the skip->xfail change is unrelated. thanks for taking a look.

In the end it _was_ related, but not to any ArrayManager stuff in that PR, but by enabling a dask test (which we don't run anywhere else), which changes the use_numexpr setting ... and our tests are not robust against that changed setting ... -> https://github.com/pandas-dev/pandas/pull/44601
CLN: TODOs/noqas/xfails
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Clean

Comments:
> can merge or wait for questions

let's go for it.  harder to get started on the next round of FIXMEs until these are in
BUG: Series.mode with dt64tz or PeriodDtype
State: closed

- [x] closes #41927
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

Sits on top of #44581

Labels:
name: Bug
name: Timezones
description: Timezone data dtype

Comments:
i ithink this needs a rebase as merged the underlying PR
Rebased +greenish
BUG: Series.view from int64 to datetimelike
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Broken off of a branch fixing #41927

Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
this or #44582 first?
this
ok then same comment on using super
> ok then same comment on using super

added + greenish
Deprecate skipna=None for mad
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
hmm are any of the doc-string failures related here? (don't think so but hard to tell at a glance).

also in future, feel free to link to the deprecation removal issue when appropriate
Could not find anything which is related. Will link to the issue in the future
oh sorry what meant was add an entry in the deprecation removal issue (we have 1 master one)

i did for this one already - but FYI
Sorry my answer was very misleading. I meant:

I could not find any failing doctests which are related to this pr.
Will link to the deprecation issue in the future.


Sorry for the confusion
can you rebase and ping on green
ping @jreback
thanks @phofl
Raise if None is passed for skipna
State: closed

- [x] closes #44178
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


will deprecate mad in follow up


Labels:
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
thanks @phofl
BUG: Series.reset_index not ignoring name with inplace and drop
State: closed

- [x] closes #44575
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
thanks @phofl
Holiday rename; Juneteenth `start_date`
State: closed

- [x] closes #44574

Rebased to origin/master.  Fixes #42328 and #44574.  Not sure if this is the right way to add to someone else's (closed) PR though...


Labels:
name: Enhancement
name: Frequency
description: DateOffsets

Comments:
> Thanks for picking this up - as requested in the original PR, could you add a test please?

Yep, on it, plus fixing tests that broke.
I don't think I changed anything that caused the "Test experimental data manager (not slow..."  failure.  Is there a way to re-run this one step?
It looks unrelated, don't worry about it
Thanks @Bklyn! CI failure was unrelated.
[ArrayManager] TST: Convert skip into xfail and clean-up tests that now work
State: closed

xref https://github.com/pandas-dev/pandas/issues/39146

As discussed before, turning the generic "skip" into an "xfail", so it's easier to keep track of which tests start to pass (and also clean-up a whole bunch of tests that were indeed already passing). 
For parametrized tests, sometimes it's only a subset of the tests that are failing/passing. So for those cases, added a helper function that adds the xfail mark inside the test based on a condition (instead of as decorator for the full test).


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @jorisvandenbossche
DOC: update shared_docs to clarify groupby with lists
State: closed

- [x] closes #44333

Following @rhshadrach 's and @asishm 's suggestion, this PR modifies groupby docstring.

Labels:
name: Docs
name: Groupby

Comments:
Hello @arushi-08! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-11-24 05:22:52 UTC
thanks @arushi-08
PERF: only apply nanops rowwise optimization for narrow arrows
State: closed

See the timings in https://github.com/pandas-dev/pandas/pull/43311#issuecomment-974891737

(this is a regression compared to previous release mostly when using ArrayManager (and thus doesn't need a whatsnew) and is covered by the FrameOps benchmarks)

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
Verified that the remaining failure is only an unrelated ResourceWarning
thanks @jorisvandenbossche 
LGTM.  merits an asv?
> LGTM. merits an asv?

It gave a huge slowdown in the current ASVs when running with ArrayManager, so in that way it's already captured. 

We could add one for wide dataframes as well, ~but you already need an extreme shape (like 10, 1000), so I personally don't find that a very interesting one to add.~ Misremembered this, so you actually already see a speed-up with something like (1000, 10).
> so in that way it's already captured

id be more comfortable with one that always runs
BUG: DataFrame.shift with axis=1 and mismatched fill_value
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
user facing?
> user facing?

yes, there's a whatsnew (forgot to check the checkbox until just now)