BrandonFors/pandas-github-issues huggingface dataset:
[BUG]: Groupy and Resample miscalculated aggregation
State: closed

- [x] closes #27343
- [x] closes #33548
- [x] closes #35275
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

I think I found a solution for this issue. We used `obj.index` to get the Index values. In case of resample `obj.index` had the original index. This index was not necessarily a RangeIndex of course -> resulted in a IndexError. In case of a RangeIndex the object `self._grouper` may have been resorted -> Wrong elements were selected. The object `self.obj` stored the original DataFrame, but already sorted in the first go around (is overwritten later). I stored the mapping from original index to a new synthetic RangeIndex (sorted after the grouping column). This allows us to replace the Index of obj in a way, that the take function works.


Labels:
name: Groupby
name: Resample
description: resample method
name: Stale

Comments:
Hello @phofl! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-09-09 07:42:50 UTC
@jreback  would you mind looking over my changes?

I passed the indices of the single groups through now. This information is lost otherwise. If there is a better way piping this through to this function please let me know.

We could sort self._grouper immediately after storing it. Did not know, which would be preferred.

The issue is matching a Series without the date column with the corresponding DatetimeIndex, which was resorted (probably).

the .grouper property holds a subclass of a Grouper (BinGrouper in this case) which has all of the info

TimeGrouper is just a simple class that is used to create the others 

pls step thru and look at object at each stage 
self.grouper or self._grouper respectively is a DateTimeIndex not a grouper in this case. This does not contain any indices. 

I look through tomorrow anyway, but I already looked into self.grouper, I could not find the necessary information.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
closing in favor of https://github.com/pandas-dev/pandas/pull/37905
CLN: remove unnecessary trailing commas on issues #35925
State: closed

#35925

- pandas/tests/arrays/categorical/test_replace.py
- pandas/tests/arrays/sparse/test_array.py
- pandas/tests/arrays/test_array.py
- pandas/tests/arrays/test_timedeltas.py
- pandas/tests/base/test_conversion.py
- pandas/tests/dtypes/test_missing.py
- pandas/tests/extension/base/methods.py
- pandas/tests/extension/test_sparse.py
- pandas/tests/frame/indexing/test_setitem.py

please give me a feedback if there is something that I have to change.. thank you before

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Hi @satrio-hw 

You will need to run `black` again on the files you've modified to make sure they won't be changed
@MarcoGorelli I've tried to rerun black on modified files, some of them revert back with commas on it.. And still got some error when chacking after I push back.. Something I could do to fix this?
@satrio-hw I think you can not include pandas/tests/arrays/sparse/test_array.py as part of this pull request, as the two versions of `black` will not agree on how to reformat it and we need to the trailing comma there. So, we'll update it when we update the version of black. The rest of your PR looks good!
@MarcoGorelli I've revert back pandas/tests/arrays/sparse/test_array.py to origin/master. To make sure I've run a command `./ci/code_checks.sh lint` and check there is only one changes to commit.. linting check done successful after pushing changes, but now I got failed tests on pandas-dev.pandas

is there something wrong in the procedures that I've done? thanks before
@MarcoGorelli I've fetch and merge master, and push it again.. now all green.. thank you very much for your guidance
thanks @satrio-hw
TST: test_datetime64_factorize on 32bit
State: closed

- [ ] closes #35921
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: 32bit
description: 32-bit systems

Comments:
thanks @simonjayhawkins
TST: update test_series_factorize_na_sentinel_none for 32bit
State: closed

xref https://github.com/pandas-dev/pandas/issues/35831#issuecomment-688163069

(I've not tested this locally, after backport can retrigger https://github.com/MacPython/pandas-wheels/pull/97 to check)

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: 32bit
description: 32-bit systems

Comments:
thanks @simonjayhawkins 
@meeseeksdev backport 1.1.x
DOC: move release note for #36155
State: closed

xref https://github.com/pandas-dev/pandas/pull/36155#issuecomment-687846461

does not need backport. release note already in 1.1.2 on backport branch

Labels:
name: Docs

Comments:
shouldn't this just be in 1.2?
> shouldn't this just be in 1.2?

see https://github.com/pandas-dev/pandas/pull/36187#issue-481307046

> does not need backport. release note already in 1.1.2 on backport branch

changes should not really have been applied to the backport branch directly

feel free to change milestone if you think more appropriate.
ahh i c, yeah in retrospect we shouldn't have merged the docs in 1.2 at all.
REF: use _validate_foo pattern in Categorical
State: closed

Similar to how we did it in datetimelike, eventually these validator methods are going to end up sharing most of their code

Labels:
name: Refactor
description: Internal refactoring of code
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Categorical
description: Categorical Data Type

Comments:
The related others are  #36173 (just rebased), #36180, #36171, #36176
DEPR: string indexing along index for datetimes
State: closed

- [x] closes #31476
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Let's get ourselves one less special case to worry about

Labels:
name: Datetime
description: Datetime data dtype
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Deprecate
description: Functionality to remove in pandas

Comments:
> also the ???

placeholder before I had a PR number.  just updated
could use some help with the ipython directive errors
ping
thanks, can you add the entry to the deprecation removal issue as well when you have a sec
Move sort index to generic
State: closed

- [x] closes #8283
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Refactor
description: Internal refactoring of code
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
thanks @VirosaLi very nice!
COMPAT: match numpy behavior for searchsorted on dt64/td64
State: closed

- [x] closes #29884
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Should whatsnew be considered a bug or breaking change?

Labels:
name: Datetime
description: Datetime data dtype
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
Note the _rebox_native this implements is identical to that in #36171.
I guess this needs a whatsnew note for the change in searchsorted?
whatsnew added + green
BUG: read_excel for ods files raising UnboundLocalError in certain cases
State: closed

- [x] closes #36122 
- [x] closes #35802
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Need some guidance on tests here. Do we need tests for other file formats here as well? Also unsure about naming conventions.

The primary bug in this case (apart from the indentation problem in the original code) is that the code was ignoring cases where the XML of the cell had multiple child nodes that were not spaces

reverted implementation from #33233 incorporating part of its modifications.

Labels:
name: IO Excel
description: read_excel, to_excel

Comments:
Looks good - can you add a what’s new note for 1.2 bug fix section?
@WillAyd added what's new note
thanks @asishm 
from https://github.com/pandas-dev/pandas/issues/36122#issue-693379165

> This used to work fine (one month ago on my machine). I tried upgrading to Pandas 1.1.1 and get the same bug. It looks like what's below:

could this fix go in 1.1.3
yep could backport
ok will raise PR on master first to move release note and then do a manual backport.

can't auto backport this (followed by release note move) as no doc/source/whatsnew/v1.2.0.rst on 1.1.x
> ok will raise PR on master first to move release note and then do a manual backport.

second thoughts will do the backport first and then if all ok move release note on master to be in sync
BUG: Ensure read_spss accepts pathlib Paths (GH33666)
State: closed

- [x] closes #33666
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Wrapped the `path` passed into `pandas.io.spss.read_spss` in a `pandas.io.common.stringify_path` call prior to passing it to `pyreadstat.read_sav`. This fixes #33666. I've also included tests validating `pd.read_spss` accepts strings and `pathlib.Path`s.

Labels:
name: IO Stata
description: read_stata, to_stata

Comments:
cc @simonjayhawkins possible if can be done before the tag (otherwise nbd)
@jreback I addressed the changes you requested earlier this week, and I believe the PR is ready for another round of review.
thanks @drmrd very nice
@meeseeksdev backport 1.1.x
Clarify docs for df.to_sql `chunksize`
State: closed

- [x] closes #35891
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Note: this is a docs pr, so not adding tests or a whatsnew entry. 

Labels:
name: Docs
name: Needs Discussion
description: Requires discussion from core team before further action

Comments:
Thanks for the PR but looks like more discussion is needed in original issue before doing anything here
DTA/TDA/PA use self._data instead of self.asi8 for self._ndarray
State: closed

This in turn will let us share more methods with the parent class, at the cost of re-shuffling where we do some of the view-casting.

Labels:
name: Datetime
description: Datetime data dtype
name: Performance
description: Memory or execution speed performance
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
sure
Comma cleanup
State: closed

- [x] pandas/tests/indexing/test_iloc.py
- [x] pandas/tests/indexing/test_indexing.py
- [x] pandas/tests/indexing/test_loc.py
- [x] pandas/tests/internals/test_internals.py
- [x] pandas/tests/io/formats/test_css.py
- [x] pandas/tests/io/formats/test_info.py
- [x] pandas/tests/io/json/test_compression.py
- [x] pandas/tests/io/json/test_pandas.py
- [x] pandas/tests/io/parser/test_c_parser_only.py
- [x] pandas/tests/io/parser/test_parse_dates.py
- [x] pandas/tests/io/parser/test_usecols.py


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @JonathanShrek
TST verify groupby doesn't alter unit64s to floats #30859
State: closed

- [x] closes #30859 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Logic of the test is to check that the value hasn't changes and that the dtype remains uint64.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
thanks @TAJD
CLN: _wrap_applied_output
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

All if statements above the changed block return, so we can remove the else's and flatten out the structure a bit.

Labels:
name: Groupby
name: Clean
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Failure is unrelated.
thanks @rhshadrach 

wouldn't be averse to simplifying this function *even* more with have some helper functions (and would location them in *another* module if possible as free functions and just passing in what you need), rather than locating them on the groupby objects themselves (e.g. to make this even easier to follow the cases)
ENH: add set_td_classes method for CSS class addition to data cells
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Simple extension to Styler to allow cells to be assigned CSS classes from a DataFrame of strings.
Seems to be a reasonably common request.
The existing framework, i.e. the `cell_context` dictionary (which previously did nothing) is leveraged.

@TomAugspurger 


Labels:
name: Enhancement
name: Styler
description: conditional formatting using DataFrame.style

Comments:
reindexing tests like these added?
thanks @attack68 
can you do a followon which updates the notebook to use this feature (or is this pretty esoteric)?
@jreback I have a few Styler PRs being worked, plan is to update the docs notebook after they all get finalised, preferably well before 1.2 release? I can link them with each other, since there will be multiple ways now of achieving same results.
DOC: Fix capitalization among headings in doc/source/user_guide/ part…
State: closed

… 1 (#32550)

- [ ] xref #32550 
- [ ] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs
name: Clean
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@ShyamDesai can you merge master to fix conflicts? This generally looks good.
Thanks but going to close this PR as stale. Please ping us if you would like to keep working on this.
Fix compressed multiindex for output of groupby.rolling
State: closed

- [x] closes #36018 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

The number of leves for MultiIndex in input was compressed to one, because the tuples representing the Index rows were not unpacked. Also name was used instead of names -> resulted in missing name for every MultiIndex.

Labels:
name: Bug
name: Groupby
name: Window
description: rolling, ewma, expanding

Comments:
thanks @phofl very nice, keep em coming!
@meeseeksdev backport 1.1.x
REF: use BlockManager.apply in csv code
State: closed

This doesn't get rid of internals usage entirely, but makes that access 1 layer less deep

Labels:
name: IO CSV
description: read_csv, to_csv
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
@jbrockmendel needs a rebase, but I agree @jorisvandenbossche suggsetion might be good in this PR
just rebased.  i encourage @jorisvandenbossche to push the edit he has in mind
thanks @jbrockmendel 

yeah agree that followups to push down to_native_types makes sense
REGR: Series access with Index of tuples/frozenset
State: closed

- [x] closes #35534 
- [x] closes #35747
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version
name: Series
description: Series data structure

Comments:
Thanks @jreback, I reworked the logic. This patch now retains much more of the original behavior. 

When one tries to find a tuple key in a Series with an Index, originally you got the message `Can only tuple-index with a MultiIndex`. As this is no longer true, I've changed this to `key of type tuple not found and not a MultiIndex`.
@simonjayhawkins this is ok to merge on green if you haven't tagged (otherwise can move to 1.1.3)
> @simonjayhawkins this is ok to merge on green if you haven't tagged (otherwise can move to 1.1.3)

yep
restarting ci. known failure

```
=========================== short test summary info ===========================
FAILED pandas/tests/plotting/test_groupby.py::TestDataFrameGroupByPlots::test_series_groupby_plotting_nominally_works
= 1 failed, 74726 passed, 1786 skipped, 1130 xfailed, 6 xpassed, 67 warnings in 1183.92s (0:19:43) =
```
ok let's defer this one
LGTM whatsnew-placement notwithstanding
Failure is different this time.

```
=========================== short test summary info ===========================
FAILED pandas/tests/io/test_parquet.py::TestParquetPyArrow::test_s3_roundtrip_for_dir[partition_col0]
= 1 failed, 74674 passed, 1905 skipped, 1108 xfailed, 6 xpassed, 284 warnings in 1178.42s (0:19:38) =
```

> Failure is different this time.

I think this one is affecting all PRs
@jreback Same unrelated parquet failure; the groupby plots failure no longer occurs.
Thanks for the review @jbrockmendel, changes made and checks pass.
thanks @rhshadrach for the followon's can open an issue if you want so we can track.
@meeseeksdev backport 1.1.x
CLN: Separate transform tests
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Setup for #35964. There were two tests that are accidental duplicates; I've just copied them over to the transform modules. Will cleanup in #35964.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
thanks @rhshadrach
DOC: add userwarning doc about mpl #35684
State: closed

- [x] closes #35684
- [x] whatsnew entry


Labels:
name: Docs
name: Visualization
description: plotting

Comments:
thanks @VirosaLi 

if you'd like to backport the original change to 1.1.2 would be ok. you can do a PR vs the 1.1.2 branch directly. (if we do that we'd also need an additonal to master to move the whatsnew note).
> thanks @VirosaLi
> 
> if you'd like to backport the original change to 1.1.2 would be ok. you can do a PR vs the 1.1.2 branch directly. (if we do that we'd also need an additonal to master to move the whatsnew note).

Do I need to put a note in whatsnew 1.1.2 for that?
STY+CI: check for private function access across modules
State: closed

The added checks here are broken off from #36055.

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
nice!
Hi @jbrockmendel - I have a question about this check - why only check for function calls?

Specifically, should this (`generic._shared_docs`)

https://github.com/pandas-dev/pandas/blob/e97c7667a648dc2ce5dbe550b117f3c9cac17b43/pandas/io/pickle.py#L14-L19

not count as a private import across modules?
> Specifically, should this (generic._shared_docs) not count as a private import across modules?

It should, we just whitelisted it in the code check.  I think we did that out of convenience more than any principled choice, so I'd be fine with changing it.
Arrow string array dtype
State: closed

This is a continuation of https://github.com/pandas-dev/pandas/pull/35259, updated to use a single `StringDtype` parameterized by `storage` (python or pyarrow). We do use different classes for the array though.

I wouldn't recommend reviewing this yet. In particular, I need to refactor the string ops implementation. That will likely be done in a separate PR to keep things smaller here.

```python
In [1]: import pandas as pd

In [2]: a = pd.Series(pd.array([pd.NA, pd.NA], dtype=pd.StringDtype("python")))

In [3]: b = pd.Series(pd.array([pd.NA, pd.NA], dtype=pd.StringDtype("pyarrow")))

In [4]: a.array
Out[4]:
<StringArray>
[<NA>, <NA>]
Length: 2, dtype: StringDtype[python]

In [5]: b.array
Out[5]:
<ArrowStringArray>
[<NA>, <NA>]
Length: 2, dtype: StringDtype[pyarrow]
```

Closes https://github.com/pandas-dev/pandas/issues/35169

Labels:
name: Strings
description: String extension data type and string data

Comments:
Just FYI @xhochy.
@TomAugspurger I would continue to work on this starting next week again. Do we want to coordinate somehow?
Closing in favor of https://github.com/pandas-dev/pandas/pull/35259 (hopefully)
@TomAugspurger @xhochy OK to merge these changes into #35259?
I'm not sure if we have complete agreement over how this will be exposed to the user yet in https://github.com/pandas-dev/pandas/issues/35169. This PR was taking a strong opinion on

1. One user-facing `pd.StringDtype[storage]`, parametrized by storage = `python/pyarrow`.
2. One or two (or three) user-facing  StringArray classes, an `ArrowStringArray`, `PythonStringArray`, and maybe a base `StringArray`.

Given the uncertainty around the API here, I'd prefer to just move forward with an internals-only `ArrowStringArray` and decide on the exact API later (unless we can agree on something quickly).
> I'd prefer to just move forward with an internals-only `ArrowStringArray`

OK, i've just changed #35259 so that the doctest passed. I should probably revert that change and change the doctest so that ArrowStringDtype is not a top level import.





Personally, I think there is enough agreement to at least start making a concrete PR, which will more easily trigger further discussion on it. 
(but let's bring this up again in the issue).
That said, I think it's certainly fine to focus on the actual implementation in #35259, and leave the user exposing API for a separate PR. There is already more than enough to review and discuss for the internal implementation details ;)
I don't seem to able to push to this branch. will open a new PR
> I don't seem to able to push to this branch. will open a new PR

#39908
BUG: copying series into empty dataframe does not preserve dataframe index name
State: closed

- [x] closes #31368
- [x] tests added / passed
   `indexing/test_partial.py:TestPartialSetting:test_index_name_empty`
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Index
description: Related to the Index class or subclasses

Comments:
also pls merge master
@simonjayhawkins ok for 1.1.2 (if you haven't tagged), otherwise let's push to 1.1.3
> @simonjayhawkins ok for 1.1.2 (if you haven't tagged), otherwise let's push to 1.1.3

ok for 1.1.2
Thanks @Dr-Irv 
@meeseeksdev backport 1.1.x
@meeseeksdev backport 1.1.x
@meeseeksdev backport 1.1.x
Something went wrong ... Please have  a look at my logs.
Something went wrong ... Please have  a look at my logs.