BrandonFors/pandas-github-issues huggingface dataset:
BUG: Change of behavior in casting of datetime-like types in MultiIndex
State: closed

- [x] closes #43091
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

Added try-except block in multi.py to reset the behaviour back to 1.2.5

Labels:

Comments:
Hello @shubham11941140! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-10-28 01:57:38 UTC
Please see contributing guide for help: https://pandas.pydata.org/pandas-docs/stable/development/contributing.html
> pls look at the 1.3 release. I am pretty sure we change this on purpose. datetime.date is retained which is the point here.

@jreback the change described in #43091 was caused by #38552, which was about `Index([date]).astype("category").astype(object)` round-tripping.
Thanks @shubham11941140 for the PR. These changes do not appear to correspond with the changes in the PR that was identified in the issue as causing the reported change in behavior.

We also need to decide if we are reverting to 1.2.5 behavior.

I am closing for now but feel free to contribute to the discussion in #43091
BUG: to_datetime with xarray DataArray and specifie unit errors
State: closed

- [x] closes #44053
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] coerce to numpy array before calling ```tslib.array_with_unit_to_datetime```function

Let me know if any change need to be done. Let me know if you need to make any changes.
I love the package hope I can start helping more :)

Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Hello @rocabrera! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-10-18 23:38:45 UTC
I will add a note tomorrow ! 
https://github.com/pandas-dev/pandas/pull/44074#pullrequestreview-781512885


thanks @rocabrera
BUG: to_datetime with xarray DataArray and specifie unit errors
State: closed

- [x] closes #44053
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

Converted values of xarray.DataArray to numpy.ndarray to solve the problem.

Labels:

Comments:
Hello @shubham11941140! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



* In the file [`pandas/tests/tools/test_to_datetime.py`](https://github.com/pandas-dev/pandas/blob/155563fcc3efe4e8fe9db4649d3f127caf9206fd/pandas/tests/tools/test_to_datetime.py):

> [Line 2656:5](https://github.com/pandas-dev/pandas/blob/155563fcc3efe4e8fe9db4649d3f127caf9206fd/pandas/tests/tools/test_to_datetime.py#L2656): [E265](https://duckduckgo.com/?q=pep8%20E265) block comment should start with '# '
> [Line 2661:16](https://github.com/pandas-dev/pandas/blob/155563fcc3efe4e8fe9db4649d3f127caf9206fd/pandas/tests/tools/test_to_datetime.py#L2661): [E128](https://duckduckgo.com/?q=pep8%20E128) continuation line under-indented for visual indent
> [Line 2662:16](https://github.com/pandas-dev/pandas/blob/155563fcc3efe4e8fe9db4649d3f127caf9206fd/pandas/tests/tools/test_to_datetime.py#L2662): [E128](https://duckduckgo.com/?q=pep8%20E128) continuation line under-indented for visual indent
> [Line 2663:15](https://github.com/pandas-dev/pandas/blob/155563fcc3efe4e8fe9db4649d3f127caf9206fd/pandas/tests/tools/test_to_datetime.py#L2663): [E128](https://duckduckgo.com/?q=pep8%20E128) continuation line under-indented for visual indent



##### Comment last updated at 2021-10-17 19:46:06 UTC
closing in favor of https://github.com/pandas-dev/pandas/pull/44074
BUG: 43909 - check monoticity of rolling groupby
State: closed

- [x] closes #43909 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry
Whats new:

- Exception now raised when non-monotonic data passed to rolling groupby
- Ammendment of tests to reflect this changed behaviour
- Addition of test for the new behaviour 


Labels:
name: Window
description: rolling, ewma, expanding

Comments:
Hello @jamesholcombe! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-12-22 13:40:13 UTC
@github-actions pre-commit
This PR should close issue 43909. Previous behaviour of a rolling window on a groupby was inconsistent with behaviour of rolling window on a dataframe. The former raises an error for non-monotonic "on". I have amended the latter to match this behaviour.
@github-actions pre-commit
@github-actions pre-commit
Whats New: Value Error now raised when a non-monotonic time series passed to "_on" attribute of a rolling groupby. This is to match the error raised for a rolling window on a dataframe. Tests have been added to confirm expected behaviour.
@github-actions pre-commit
@github-actions pre-commit
Hi Jeff, 
master is merged no? 
which sections require commenting?
Cheers
@github-actions pre-commit

NOTE: master branch contains breaking changes so have not merged most recent.
> NOTE: master branch contains breaking changes so have not merged most recent.

what does this mean?


still needs a whatsnew note and merge master
still needs a whatsnew note and merge master
@github-actions pre-commit
@github-actions pre-commit
Find whats new note at top of PR. master merged.
@github-actions pre-commit
whatsnew note goes in the doc/ folder under 1.4.0 for bug fixes
pre-commit is failing, pls merge master and ping on green.
pre commit now passing
thanks @jamesholcombe
BUG: sort_index did not respect ignore_index when not sorting
State: closed

- [X] closes #43591
- [X] tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [X] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Index
description: Related to the Index class or subclasses

Comments:
thanks @saehuihwang very nice!
DEPS: bump pyarrow min to 1.0 #41329
State: closed

- [x] closes #41329
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Dependencies
description: Required and optional dependencies

Comments:
> Can we remove pa_under_1p0 too? Thanks. Looks good otherwise.
> 
> https://github.com/pandas-dev/pandas/blob/776329fdd7fef357cc7ef3a37ffcdda3df8e6fc1/pandas/compat/pyarrow.py#L10

I'll remove all the compat in a follow-up PR.
prob worth bumping to the latest patch version of 1.x (e.g. 1.0.1)
thanks @fangchenli
BUG: 43909 - check monoticity of rolling groupby
State: closed

- [y ] closes #43909 
- [y ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ test and bugfix ] whatsnew entry


Labels:

Comments:
Hello @jamesholcombe! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-10-16 22:38:12 UTC
@github-actions pre-commit
Removed a manual file handler pitfall
State: closed

It is a simple PR. 

It only removes a manual file handler pitfall from versioneer.py

As stated by [Pylint documentation](https://pylint.pycqa.org/en/latest/technical_reference/features.html)

"
consider-using-with (R1732)
Consider using 'with' for resource-allocating operations Emitted if a resource-allocating assignment or call may be replaced by a 'with' block. By using 'with' the release of the allocated resources is ensured even in the case of an exception.
"

Labels:

Comments:
> This file is generated by versioneer - we generally don't edit it manually.
> 
> Can you make the fix upstream and we can then regenerate? https://github.com/python-versioneer/python-versioneer

Sounds fair.

In this case, I'm closing this PR.

Thanks for the attention!
> > This file is generated by versioneer - we generally don't edit it manually.
> > Can you make the fix upstream and we can then regenerate? https://github.com/python-versioneer/python-versioneer
> 
> Sounds fair.
> 
> In this case, I'm closing this PR.
> 
> Thanks for the attention!

Agree thanks!
added test to indexing on groupby, #32464
State: closed

- [x] closes #32464
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

This makes sure an index of a groupby with more then one column is always a MultiIndex.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: MultiIndex

Comments:
I tried using `tm.assert_frame_equal` to test if a.groupby(["a", "b", "c"]).sum() has a MultiIndex like this:

```python
    a = DataFrame({"a": [], "b": [], "c": []})
    result = a.groupby(["a", "b", "c"]).sum()

    expected_index = MultiIndex.from_arrays([[], [], []], names=("a", "b", "c"))
    expected = DataFrame(
        np.ndarray((0, 0)), dtype="float64", index=expected_index, columns=Index([])
    )

    # Tests if groupby with all columns has a multiindex
    tm.assert_frame_equal(result, expected)
```

But it didn't work, so I'm keeping just the other assertion, that tests if both groupby's are equal.
can you merge master and update to comments
Yes, I was working on other stuff, will work on it now!
Hm I don't really know why the check fail, can someone help me?
can you merge master again and ping on green
@jreback can you help me with the error?
is worthwhile but needs to be passing (and pls merge master)
@gabrieldi95 - The CI failure appears unrelated to me; can you try merging master again
@gabrieldi95 can you merge master and address comments
Closing to take this off the queue. @gabrieldi95 please let me know if you'd like to continue this!
@rhshadrach sorry for taking so long! Yes, I would like to continue
can you merge master
can you merge master and update to comments
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in main and address the review and we can reopen.
Fix several typos under pandas/_libs
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Would it make sense to enable codespell for more than just these file types?
https://github.com/pandas-dev/pandas/blob/c30afb23dfa948e3b34b4d71abb16ce455012428/.pre-commit-config.yaml#L19
> 
> 
> Would it make sense to enable codespell for more than just these file types?
> 
> https://github.com/pandas-dev/pandas/blob/c30afb23dfa948e3b34b4d71abb16ce455012428/.pre-commit-config.yaml#L19

Perhaps, but it could easily lead to false-positives.
Thanks @PApostol. Might be worthwhile opening up a new issue to discuss expanding the codespell check.
Fixed metadata propagation in Dataframe.apply (issue #28283)
State: closed

Co-authored-by: Mohamad Rkein <mohamad_rkein@usp.br>
Co-authored-by: Rafael Rodrigues <rrvsrafael@gmail.com>

In reference to #28283

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
this appears to have xpassed some previously xfailed tests. @moha-rk you may need to adjust some other tests (as they maybe indirectly calling .apply)
> this appears to have xpassed some previously xfailed tests. @moha-rk you may need to adjust some other tests (as they maybe indirectly calling .apply)

Thank you for your feedback, @jreback. I didn't notice the 'F's that appeared after my changes.

The method `DataFrame.nunique()` calls the method `DataFrame.apply()` on its implementation (pandas/core/frame.py), applying `Series.nunique()` (pandas/core/base.py) on each Series (column), so it already handles the metadata propagation. I don't know if that's a problem, but the "method" attribute used on the `__finalize__` call will stay as apply.

As for "mode()", the same thing happens when it's called by a DataFrame (pandas/core/frame.py). However, there's a catch. When "mode()" is called directly from a Series (pandas/core/base.py), this apply call doesn't exist. This is a problem because "Series.mode()" returns a Series containing the mode(s). I created a test with the "not_implemented_mark" to observe this behaviour. I don't know if calling __finalize__ on the Series.mode() function is a good idea, since this would cause it to be called one time for each column on the DataFrame call. Do you have any suggestion? For now, I'll leave it with the said mark.

Finally, the last method that changed was "DataFrame.transform()". This function creates a "FrameApply" object, and calls its function ".transform()". When following the series of function calls, we end on "transform_str_or_callable()" (pandas/core/apply.py), which uses the "DataFrame.apply()" method at its end, propagating the attributes. I removed the "not_implemented_mark".
DOC: Fix Docstring for DataFrame nlargest `keep` option (#44040)
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] Add double backticks for DataFrame nlargest `keep` option to keep the style consistent

From
https://pandas.pydata.org/docs/dev/reference/api/pandas.DataFrame.nlargest.html
![image](https://user-images.githubusercontent.com/25895405/137416950-97b0db11-382f-410f-ac90-50ef2b25c32b.png)
To
![image](https://user-images.githubusercontent.com/25895405/137420535-275670da-094c-4533-9eac-82a36f52d5f4.png)



Labels:
name: Docs

Comments:
Thanks for the PR @Zeroto521
REF: dispatch DTI/TDI setops to RangeIndex
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Includes #44019 (but not that whatsnew)

I'm optimistic we can eventually dispatch all the relevant cases and then can get rid of the special-cased _fast_union/_fast_intersect.

Labels:
name: Refactor
description: Internal refactoring of code
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
can you rebase
TST: added groupby apply test for nan coerce
State: closed

- [x] closes #24903
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] Bug was reported in #24903 that showed empty strings being coerced into Nan when certain apply was used with groupby.  Although bug has been fixed, this test identifies the issue if it were to happen again.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @tyvich
DOC: Document and annotate Index.reindex (#40328).
State: closed

- [x] closes #40328 
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them




Labels:
name: Docs
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
thanks @ShivnarenSrinivasan
Backport PR #43729: DEPS: Upgrade Deps for Python 3.10
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

**THIS IS A MANUAL BACKPORT OF LARGE BUILD CHANGES. PLEASE REVIEW CAREFULLY BEFORE MERGING**

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Dependencies
description: Required and optional dependencies
name: Python 3.10

Comments:
thanks @lithomas1 very nice
TST: adds test for .loc on multiindex for series GH43908
State: closed

- [x] closes #43908
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: MultiIndex

Comments:
Excellent thanks @KoenSR!
CLN: no need for suffices anymore in test_hashtable.py
State: closed

- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Since we have version with fused types, there is no need to call precise version (i.e. with suffices like int64, int32 and so on) any longer.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean

Comments:
Thanks @realead
REGR: Barplot broken on Index(dtype='object')
State: closed

- [ ] closes #38947
- [x] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Visualization
description: plotting
name: Stale

Comments:
Hello @SabrinaMB! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-10-14 20:20:54 UTC
Thanks for the PR @SabrinaMB , please could you fix the flake8 issues below as per CI.

MIght be worth reading our contributing guide here: https://pandas.pydata.org/pandas-docs/stable/development/contributing.html

Please could you also update the PR title to something more descriptive.

Thanks!

```
flake8..........................................................................................................Failed
- hook id: flake8
- exit code: 1

pandas/tests/plotting/test_misc.py:470:20: C407 Unnecessary list comprehension - 'all' can take a generator.
pandas/tests/plotting/test_misc.py:476:9: F841 local variable 'e' is assigned to but never used
```
can you merge master and update to comments
can you merge master and update to comments
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Thanks for the PR, but it appears to have gone stale. If interested in continuing please merge the main branch and address the review and we can reopen.
[PERF] fixing memory leak in aggregation.pyx
State: closed

- [x] closes #43339
- [x] tests passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

The old memory needs to be released, before the pointer switches to the new list.


Labels:
name: Performance
description: Memory or execution speed performance
name: Window
description: rolling, ewma, expanding

Comments:
would be ok including this in 1.3.x if you can move the note
cc @simonjayhawkins ok to include this for 1.3.4


> cc @simonjayhawkins ok to include this for 1.3.4

sure.
thanks @realead 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
ENH/CI: Add Cache for Azure pipelines  - posix
State: closed

- [x] closes ##42580
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Trying to leverage the use of cache on Azure pipelines - currently WIP will make some investigations 

Labels:
name: CI
description: Continuous Integration

Comments:
Hey folks it seems I have most of this figured, some notes

- [x] MacOs is working now as expected (see [Azure pipeline](https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=68127&view=logs&j=054c7612-f473-5cf0-0a65-77b54574553a))
- [x] Windows - some nagging issues with paths are still present but can work on that
- [ ] Up to discussion: it would be best to explore the use of https://github.com/conda-incubator/conda-lock since the dependencies are not explicitly pinned in `environment.yaml` 

Update: The MacOS pipelines are still working as expected, however since yesterday I noticed that the Windows pipelines are getting stuck in `conda-update` and eventually timeout ([see here](https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=68142&view=logs&j=e2350215-d973-503e-a41e-d287ff823797&t=f0a61e35-cb95-523a-e4e8-f80405bb57f0))

I might for now try and do some debugging and see if I can pinpoint to the root cause, but in the meantime a review to the rest of the PR would be grand
Thanks for the PR. Do you know how much time this saves?


> Update: The MacOS pipelines are still working as expected, however since yesterday I noticed that the Windows pipelines are getting stuck in `conda-update` and eventually timeout ([see here](https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=68142&view=logs&j=e2350215-d973-503e-a41e-d287ff823797&t=f0a61e35-cb95-523a-e4e8-f80405bb57f0))
> 
> I might for now try and do some debugging and see if I can pinpoint to the root cause, but in the meantime a review to the rest of the PR would be grand

If it's easier for you, you can update the posix pipelines first and do the Windows pipelines in a follow up.

Also, I noticed that as of the latest commit ``conda update`` is not being performed. We probably want to make sure that the any old packages in the cache are updated.
Hey @lithomas1  thanks for having a look - I have gone ahead and split the posix and Windows cache parts as suggested.

As is the `posix` cache  seems to be saving ~5 mins per job 

can you merge master
Sorted!
does this help in build time? can you show before / after
@trallard can you merge master and address comments
Sorry folks I took some time off at the end of year - will merge master and add the details requested today 

@trallard what's the status here?
Thanks for the PR, but it appears to have gone stale. Additionally, there is some discussion about moving aware from Azure pipelines as well https://github.com/pandas-dev/pandas/issues/46351. Closing, but would be happy to take similar changes to our GHA builds!