BrandonFors/pandas-github-issues huggingface dataset:
Streamline docker usage
State: closed

Took a look at this setup today and found a few things that could be improved to make it easier for new contributors. Some quick comparisons:

- Image size current 6.1 GB vs new 2.75 GB
- Image build time of current somewhere in the 30-45 minute range versus new at 2.5 min
- Current image blurs the lines of responsibility between a Docker image and a container
- Current image puts another layer of virtualization in with mamba that is arguably unnecessary with Docker

The remaining bottleneck with image creation is https://github.com/pandas-dev/pandas/issues/48828 

Labels:
name: Build
description: Library building on various platforms
name: Docs

Comments:
Is this compatible with VSCode's `.devcontainer.json` setup? I don't use VSCode personally but not sure if that assumes the Dockerfile needs to be configured a certain way
Good question...at the very least it should match whatever the previous image had in terms of compatibility, but also not a VSCode user. Maybe @MarcoGorelli knows?
@natmokval I think you said you were using Docker? Was it via vscode? If so, just tagging in case you felt like having a look

> Good question...at the very least it should match whatever the previous image had in terms of compatibility, but also not a VSCode user. Maybe @MarcoGorelli knows?

I don't remember any specific requirements in terms of compatibility when I tried it years' ago, but I'll give this a go
I think this line will need changing:

https://github.com/pandas-dev/pandas/blob/39e0964ad7896ed758fb1f8fb7c2326685277b5e/.devcontainer.json#L13

Anyway, I tried out the rest in vscode, and it worked perfectly, well done!
> @natmokval I think you said you were using Docker? Was it via vscode? If so, just tagging in case you felt like having a look
> 
Hi @MarcoGorelli , I use docker with VSCode, but without `.devcontainer.json`. I run docker in terminal and mount my repository into container by `-v /Users/natalia/pandas-natmokval:/home/pandas`. Haven't tried dev containers so far.
Awesome thanks @MarcoGorelli for the VSCode help
Is it OK to backport this so that the correct instructions show up in the `stable` version of the docs?
I am fine with that
@meeseeksdev please backport to 1.5.x
CLN: assorted
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Clean

Comments:
LGTM merge when ready @WillAyd 
Thanks @jbrockmendel
REF: Refactor merge array conversions for factorizer
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
thanks @phofl
DEP: Enforce deprecation of mangle_dupe_cols
State: closed

#48137

Labels:
name: IO CSV
description: read_csv, to_csv
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @phofl 
The pull request says

> Deprecating `mangle_dupe_cols`, which was never implemented, and will be replaced by a new standard argument to allow change the format of the renamed columns.

Which is the new standard argument for this? In https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html I cannot find any option for dealing with duplicate column names in `pd.read_csv`
add multi index with categories test
State: closed

This PR adds test to address the issue in #49337, which have already been fixed on main

- [x] closses #49337
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Do you want a entry in the changelog since this is not a fix, just a test to ensure we dont run into this again?

- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Stale

Comments:
Failing test seems unrelated to this
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
rebased master and addresses the comment, I dont think the PR sync while it is closed so you cant see the newest commit
For some reason I cannot re-open this pull request. Mind opening up another one?
Fix `Styler.format()` Excel example
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: Styler
description: conditional formatting using DataFrame.style

Comments:
Thanks @janosh 
That example didnt work.
```
TypeError: <lambda>() takes 0 positional arguments but 1 was given
```
Needs to be
```
df.style.applymap(lambda v: pseudo_css).to_excel("formatted_file.xlsx")
```
not
```
df.style.applymap(lambda: pseudo_css).to_excel("formatted_file.xlsx")
```
Can you re-submit.
@attack68 Sorry about that. Submitted a fix.
TST : Adding Oracle Database support to run SQL Tests.
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.



Labels:

Comments:
Thanks for the pull request, but since this feature currently has a blocker, closing
BUG: Avoid intercepting TypeError in DataFrame.agg
State: closed

- [x] closes #49399
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
thanks @yuvanist
REF: reverse dispatch in factorize
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Needed for upcoming removal of DTA/TDA.freq

Labels:
name: Refactor
description: Internal refactoring of code

Comments:
Thanks @jbrockmendel
ENH: Add io.nullable_backend=pyarrow support to read_excel
State: closed

- [x] xref #48957 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO Excel
description: read_excel, to_excel
name: Arrow
description: pyarrow functionality

Comments:
thx @mroeschke for addressing the doc comments!
DOC: See also box should refer to same type (Series.dt.tz_convert --> Series.dt.tz_localize)
State: closed

- [x] closes #49297
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
TST/CoW: copy-on-write tests for df.head and df.tail
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Progress towards #49473
Because `.head()` and `.tail()` use `.iloc` under the hood, copy-on-write is already implemented for them, but this PR adds explicit tests for both methods.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Copy / view semantics

Comments:
Thanks @seljaks
STYLE: #49656 - generic.py
State: closed

- [ ] closes #49656 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
@MarcoGorelli I am very glad to help
48779-doc-MonthEnd
State: closed

- [x] closes #48779
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
As proposed by @MarcoGorelli I updated docs for `MonthEnd` and added one more example to highlight “the last day of the month” behavior.

I checked that build of documentation shows the new description.

Labels:
name: Docs

Comments:
Thank you @MarcoGorelli, for your comments and hints. I corrected my docstring and added an example, which uses `n=0`

Unfortunately, my docker environment is broken by absence of visioner.py and I can not validate my changes locally.
If I am not mistaken, visioner.py is missing after recent commit: BLD: use nonvendor versioneer (#49924)
thanks for updating!

could you try updating your environment? `versioneer` was added to the `environment.yml` file in that PR. I don't know about how it works in the Docker container, but for me `mamba env update -n pandas-dev --file environment.yml` worked
Assuming you are using the Dockerfile in this repo, a force rebuild of the image should pick up the versioneer dependency
> Assuming you are using the Dockerfile in this repo, a force rebuild of the image should pick up the versioneer dependency

Thank you @WillAyd, I fixed my problem with docker image.
> could you try updating your environment? `versioneer` was added to the `environment.yml` file in that PR. I don't know about how it works in the Docker container, but for me `mamba env update -n pandas-dev --file environment.yml` worked

Thank you, @MarcoGorelli, my environment works well now. I fixed errors in docs for `MonthEnd` and added an example for `n=0`. Docstring validation also requested me to add `See Also` section. I added the reference to `DateOffset`.

I noticed one check during CI failed, but I assume it was caused by timeout and not by my changes.

> If there's other DateOffsets where similar confusion can arise, PRs to add examples / clarification there too would be welcome
> 
Certainly. Checking other DateOffsets looks like a good idea. I will look into it.
> Regarding the "see also" note - this isn't strictly necessary to pass CI, as the codes in
> 
> https://github.com/pandas-dev/pandas/blob/4d0a4365870893e232f639af13fea44c7d3ff9d4/ci/code_checks.sh#L83
> 
> are currently checked - but it's nice to have it anyway, thanks!
> 
I will use the same list of errors locally the next time.

Thank you @MarcoGorelli, for your guidance and support. It makes contributing to pandas pleasant and not too complicated.
STYLE #49656: format.py - 9.99/10
State: closed

- [ ] xref #49656 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
@MarcoGorelli I updated the format.py. There is one test that failed, but I am not sure what is the reason. Can you help me check it out, please?
hey @bang128 - there's already a PR open to address `format.py` I'm afraid. Closing then, but thanks for your PR!
@MarcoGorelli Oh, okay. Thank you for letting me know.
DEPR: Change numeric_only default to False in remaining groupby methods
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby
name: Deprecate
description: Functionality to remove in pandas
name: Nuisance Columns
description: Identifying/Dropping nuisance columns in reductions, groupby.add, DataFrame.apply

Comments:
thanks @rhshadrach, exciting to see this go!  dont forget to update #30228
> dont forget to update #30228

Done.
There are checkboxes for 40238 and 47561 that i'd expect to be checked off.  am i being too hopeful here?
Nope - those are done as well. Thanks.
ASV: Fix asv after versioneer change
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


#49924 broke the local ci workflow. We have to install versioner before building. Adding it to the other dependencies did not work

Labels:
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
Thanks @phofl
DEPR: enforce not-automatically aligning in DataFrame comparisons
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel
STYLE pre-commit autoupdate
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Awesome, thanks for breaking this up @MarcoGorelli
PERF: read_html: Reading one HTML file with multiple tables is much slower than loading each table separatly
State: closed

- [x] closes #49929
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

```py
# setup from referenced issue
In [2]: %timeit combined = pd.read_html(StringIO(combined_html))
324 ms ± 1.79 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) # <- PR

In [3]: %timeit combined = pd.read_html(StringIO(combined_html)) # <- main
44.3 s ± 4.28 s per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

Labels:
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
Added whatsnew!
Thanks @asishm
REF: simplify DTI/PI.get_loc
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
Thanks @jbrockmendel
STYLE #49656: fixed redefined-outer-name linting issue to format.py 
State: closed

Fixed redefined-outer-name issue in:
- pandas/io/formats/format.py

redefined-outer-name t issues raised for:
- justify function (import from pandas.io.formats.printing)
- decimal (changed to from decimal import Decimal) 
- is_dates_only parameter conflict with function name 
- updated get_format_datetime64 function with renamed parameter

- passed related tests in pytest pandas
- running:  pylint --disable=all --enable=redefined-outer-name pandas/io/
-- > Results in 10.00/10 rating



Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @MarieKMiko