BrandonFors/pandas-github-issues huggingface dataset:
DOC: GH48089 Update doc for df to latex
State: closed

- [x] closes #48089  (Replace xxxx with the Github issue number)
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Docs
name: IO LaTeX
description: to_latex
name: Styler
description: conditional formatting using DataFrame.style

Comments:
Thanks @samrao1997
DOC: Altered capitalization validation script to handle edge cases
State: closed

references  #32550

I fixed the capitalization validation script to stop considering urls.

I also altered the script to consider hyphenated words as a single word. Previously this was asking you to change the capitalization of packages that had "Pandas" in the name, and adding that to the exceptions list would have let the script pass through instances where Pandas DID need to be changed to lowercase.


Labels:
name: Docs
name: Code Style
description: Code style, linting, code_checks
name: Stale

Comments:
Hello @INDIG0N! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-08-23 04:15:11 UTC
@phofl I'm a bit new to this, how can I run pre-commit locally, and will doing that automatically resolve the failed check here or do I need to do something else?
@INDIG0N you can simply run the validation script and make sure it's happy locally: `./scripts/validate_rst_title_capitalization.py` (or any other script that pre-commit calls and fails in the CI, such as `black`, `isort`...
@datapythonista oh, the script worked fine locally, even used it to find some capitalization errors and fix them. I see there's a conflict though, I'll take care of that and then the branch should be good to merge
@INDIG0N do you have time to address the last comments?
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
CI/TST: Fix test_resample_empty_dataframe for 32 bit build
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

xref https://github.com/pandas-dev/pandas/pull/48072


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Resample
description: resample method
name: 32bit
description: 32-bit systems

Comments:
Merging to get the CI to green
Thx for this
BUG: df.__setitem__ casting ea to object when setting with scalar key and  frame value
State: closed

- [x] closes #46896 (Replace xxxx with the Github issue number)
- [x] closes #20476 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Setting with duplicate columns is still buggy, but could not figure this out efficiently yet, will open a follow up when this is merged


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Thanks @phofl
ENH: move an exception and add a prehook to check for exception place…
State: closed

- [x] closes #27656.
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

This is a follow up from my last MR: https://github.com/pandas-dev/pandas/pull/47901. I'm new to prehooks, so I may have went in the wrong direction for the prehook. But, I was thinking of checking the files for class that inherit Exceptions and flagging them. Let me know if another approach is preferred.

I also noticed some other exceptions while working on this.

- InvalidVersion. I tried moving this one, but got a circular import so I'm importing it in the errors.__init__.py instead.
- LossySetitemError, NoBufferPresent, and InvalidComparison are moved to errors.__init__.py.
- NotThisMethod. It seem like a code generated exception rather than a custom pandas one, so I'm importing it in the errors.__init__.py instead.


Labels:
name: Enhancement
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Code Style
description: Code style, linting, code_checks
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
Looked pretty good. Could you fix the merge conflict and see if the Ci failures are related?
> 



> Looked pretty good. Could you fix the merge conflict and see if the Ci failures are related?

Sounds good. I've merged main
Awesome work on this @dataxerik! This will really help make sure users are able to publicly access Warnings and Exceptions in the future! 
(just getting here from the notification that the issue was closed, so sorry for the late comment)

I _think_ that `InvalidComparison` is only used internally to raise certain cases which is then catched internally again to do something else or raise a public error. Assuming that's the case, we don't need to expose this exception class publicly? 

(the same might be true for `LossySetitemError`, didn't check in detail, cc @jbrockmendel)
That's a fair point. 

Maybe we can clearly document the exceptions or warnings that are supposed to be internal? My concern is that overtime the "internal" exceptions or warnings may unintentionally become publicly thrown, and at least users could still catch them through `pandas.errors`
> I think that InvalidComparison is only used internally to raise certain cases which is then catched internally again to do something else or raise a public error. Assuming that's the case, we don't need to expose this exception class publicly?

correct.  It should always be caught and re-raised.  Could clarify that in the docstring.

Ditto for LossySetitemError.
ENH: Add axis parameter to add_prefix and add_suffix
State: closed

- [X] closes #47819
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement

Comments:
Hi @mroeschke, are there any other changes required (or) should we hold back changes in this API for now ?
Accordingly (if required) can update the whatsnew file along with other review suggestions
Thanks in advance. :)
Hello @dannyi96! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-08-30 22:27:53 UTC
IMO I think this is more appropriate for 1.6, so I would be in favor of moving the release notes to `1.6.0.rst`. I could be convinced otherwise though
No preference on my side. @dannyi96 could you move whatsnew and change Versionadded?
Sure thing @mroeschke @phofl. Will move the notes to 1.6
updated the whatsnew. 
( have` added it as part of "Other enhancements" in whatsnew v1.6 file )
> One small comment otherwise LGTM. If you could merge in main once more, that would be great.

done 🙂 
Thanks @dannyi96, great work!
REF: avoid internals in merge code
State: closed

After this, concatenate_block_managers is only used in one place.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Thanks @jbrockmendel 
I haven't yet investigated why, but this has broken a lot of tests in geopandas' CI against pandas main (so if we want to do an RC tomorrow, the easiest might be to revert this for now, since it's only a refactor?)

Will try to make a minimal reproducer
I can't reproduce this without geopandas, but the issue boils down to that `merge` now calls `pd.concat` (and thus `_constructor` / `__finalize__`) before adding suffices for duplicated column names. 
So maybe this might be sufficient:

```diff
--- a/pandas/core/reshape/merge.py
+++ b/pandas/core/reshape/merge.py
@@ -763,9 +763,12 @@ class _MergeOperation:
         right.index = join_index
 
         from pandas import concat

+        left.columns = llabels
+        right.columns = rlabel
         result = concat([left, right], axis=1, copy=copy)
-        result.columns = llabels.append(rlabels)
         return result

```




One reproducer with geopandas is:

```
import geopandas
df1 = geopandas.GeoDataFrame({"key": [1, 2, 3]}, geometry=geopandas.points_from_xy([1, 2, 3], [1, 2, 3]))
df2 = geopandas.GeoDataFrame({"key": [2, 3, 4]}, geometry=geopandas.points_from_xy([1, 2, 3], [1, 2, 3]))

df1.merge(df2, on="key")
```
I have no objections to the suggested patch.  Can you open an issue in geopandas to figure out how they are relying on our internals and ween off of it (xref #40226)
> Can you open an issue in geopandas to figure out how they are relying on our internals and ween off of it (xref #40226)

To be clear, geopandas is not _directly_ relying on the internals (we are not calling an internal method), but we are indirectly relying on the logic here, because we disallow dataframes with duplicate geometry columns (and thus the fact whether the suffixes are applied before or after the concat step matters). 

Opened https://github.com/pandas-dev/pandas/pull/48416 with the patch proposed above.
It looks like this caused a performance regression in our merge benchmarks, https://asv-runner.github.io/asv-collection/pandas/#join_merge.JoinEmpty.time_inner_join_right_empty

This is the most relevant commit in the timespan when the regression occurred


Bummer.  Not ideal, but won't be the end of the world if we have to revert.
ENH: Add support for groupby.ohlc for ea dtypes
State: closed

- [x] xref #37493 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This did not work at all before

cc @jorisvandenbossche 

Labels:
name: Groupby
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Thanks @phofl
ENH: Support mask for groupby var and mean
State: closed

- [x] xref #37493 (Replace xxxx with the Github issue number)
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This saves around 30-35% in the aggregation operation for nullable dtypes with 1/3 missing values compared to the previous implementation

cc @jorisvandenbossche 

Labels:
name: Groupby
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
Thx for review @rhshadrach
Bug: Unexpected behavior when assigning multi-dimensional array to DataFrame column 
State: closed

- [ ] closes #40827
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: DataFrame
description: DataFrame data structure

Comments:
I will close this first. Need some time to understand.  Thank for review
DEPR: `sort_columns` in `plot` (#47563)
State: closed

- [x] closes #47563
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v1.5.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Visualization
description: plotting
name: Deprecate
description: Functionality to remove in pandas

Comments:
@phofl thanks for reviewing! I just updated the PR with the requested changes & rebased with the latest `main`.

Let me know if further changes are needed. Thank you!
Fixed - thanks!
CI seems to be breaking. Marking as draft while it is fixed.
Thanks @YagoGG 
linked to deprecation log
TST: Check dtype in test_resample_empty_dataframe
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Noticed from https://github.com/pandas-dev/pandas/pull/47672#discussion_r945175211

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Resample
description: resample method

Comments:
Thanks @rhshadrach (failure unrelated)
STYLE: upgrade flake8
State: closed

- [x] closes #48061
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Looks good - could you make this target the `main` branch please?
TST: Check index when grouping all columns of empty DF (#32464)
State: closed

- [x] closes #32464
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature. _(does not apply)_


Labels:
name: Bug
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: MultiIndex

Comments:
Giving @rhshadrach attribution for the code as the test is basically his suggestion over #44046.

Let me know if any changes are needed. Thanks!
Rebased with the latest `main`, as it looks that the build has been fixed now.
Thanks @YagoGG
CI/DEPS: Fix timezone test due to pytz upgrade
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature



Labels:
name: CI
description: Continuous Integration
name: Timezones
description: Timezone data dtype

Comments:
Looks like a change in pytz

```
In [1]: import pytz

In [2]: pytz.__version__
Out[2]: '2020.4'

In [3]: pytz.timezone('Africa/Kinshasa')._utc_transition_times
Out[3]: [datetime.datetime(1, 1, 1, 0, 0), datetime.datetime(1919, 8, 31, 23, 46, 24)]

In [1]: import pytz

In [2]: pytz.timezone('Africa/Kinshasa')._utc_transition_times
Out[2]:
[datetime.datetime(1, 1, 1, 0, 0),
 datetime.datetime(1905, 6, 30, 23, 46, 25),
 datetime.datetime(1908, 7, 1, 0, 0),
 datetime.datetime(1913, 12, 31, 23, 46, 25),
 datetime.datetime(1919, 8, 31, 23, 30)]

In [3]: pytz.__version__
Out[3]: '2020.5'

In [1]: import pytz

In [2]: pytz.__version__
Out[2]: '2022.1'

In [3]: pytz.timezone('Africa/Kinshasa')._utc_transition_times
Out[3]:
[datetime.datetime(1, 1, 1, 0, 0),
 datetime.datetime(1905, 6, 30, 23, 46, 25),
 datetime.datetime(1908, 7, 1, 0, 0),
 datetime.datetime(1913, 12, 31, 23, 46, 25),
 datetime.datetime(1919, 8, 31, 23, 30)]

In [1]: import pytz

In [2]: pytz.__version__
Out[2]: '2022.2'

In [3]: pytz.timezone('Africa/Kinshasa')._utc_transition_times
Out[3]: [datetime.datetime(1, 1, 1, 0, 0), datetime.datetime(1901, 12, 13, 20, 45, 52)]
```
BUG: Groupby cumprod nan influences other columns with skipna False
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Saw this when working on mask support

previously this returned

```
      b    c
0  22.0  1.0
1   NaN  NaN
2   NaN  3.0
```

because the calculation in the second row was skipped for the second column after the NaN was encountered

Labels:
name: Bug
name: Groupby

Comments:
Thanks @phofl
fixed docstring error with pandas.Series.plot.area:Unknown parameter: {'stacked'}
State: closed

When you run python `./scripts/validate_docstrings.py --errors=PR02`, one of the errors is `/home/pandas/pandas/plotting/_core.py:1470:PR02:pandas.Series.plot.area:Unknown parameters {'stacked'}`

The error I believe is because "stacked" is declared in the docstring, but is not a parameter within the function. Even though stacked is only passed in when setting stacked to False, it should still be defined in the function parameter.

- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Visualization
description: plotting

Comments:
Hi, I don't believe the changes I have made, contributed to the errors in the checks. Could you please clarify?
> Hi, I don't believe the changes I have made, contributed to the errors in the checks. Could you please clarify?

There are some related test failures: https://github.com/pandas-dev/pandas/runs/7811767621?check_suite_focus=true
Hi, I have tried to reproduce the test cases that are failing locally but could not. How can I go about debugging these errors?
The issues look unrelated to this PR. Could you merge in main once more?
thx @joshuabello2550
REGR: Avoid overflow with groupby sum
State: closed

This is the alternative to #48044.

I would very much prefer this one, even though it is a bit slower.

cc @mroeschke cc @jorisvandenbossche 

Labels:
name: Groupby
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
I would be okay with this version of the change
> even though it is a bit slower.

Do you have a rough estimate of how much slower? 

I certainly think this version is good enough (in the end, before we were casting as well, to float64), but if it is a significant difference, we can still table it as a future possible improvement if we find a better way to avoid the many compilation combinations
Around 5% of the ``.sum()`` runtime is taken up by the astype call. So no big deal (1_000_000 values)

Memory footprint is something that would make it worth, if we can avoid the duplications.

But for now, this should be sufficient since we were castint to float anyway
Will open an issue, after this is merged.
Thanks joris

If ok, I would merge when ci is greenish
Go ahead!
Failures unrelated