BrandonFors/pandas-github-issues huggingface dataset:
BUG: ArrowExtensionArray.factorize with chunked dictionary array
State: closed

- [x] closes #54844 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Arrow
description: pyarrow functionality

Comments:
thx @mroeschke
CI: add empty line in `no-bool-in-generic` to avoid `black` complaining
State: closed

*No reference issues (I ran into this during another contribution).*


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Why is this needed?
For instance if I used `bool` in generic, this hook will delete the last empty line. Then if I add the changes and commit again, `black` will complain about it. Well then if you add the changes and commit again everything will be fine, but I just think it would be better to avoid such conflict.

Or did I miss something? @mroeschke
Can you show a commit where this happened?
Does this suffice? @mroeschke

https://github.com/pandas-dev/pandas/assets/108576690/858ec0dc-b550-4e65-93fe-55d2ec53bd53

Ah great thanks for that demonstration 👍
Fix GH #54853: BUG: DeprecationWarning for frontend.OptionParser when building docs
State: closed

- [x] closes #54853 (Replace xxxx with the GitHub issue number)
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

See https://github.com/docutils/docutils/commit/6548b56d9ea9a3e101cd62cfcd727b6e9e8b7ab6#diff-a033583f6ace19fed2adc108b1c130e17ea00e1afee38c22c993cb477ad27a5fR453

Labels:
name: Docs

Comments:
Thanks @DavidToneian
DOC: Add date for v2.1.0
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
thx @lithomas1
DEPR: remove use of `nan_as_null` from callers of `__dataframe__`
State: closed

No one was using it yet and it seemed easier to clean it up from libraries than to ask everyone to add support for it - see https://github.com/data-apis/dataframe-api/issues/125.

Labels:
name: Interchange
description: Dataframe Interchange Protocol

Comments:
pre-commit.ci autofix
@phofl gentle ping (your review is blocking the pr)

let's get the text in, and if/when we start a formal deprecation cycle we can add the `.. deprecated::` directive and issue a warning
Fine by me if we deprecated properly in the future (I don't care when we start the process)
Thanks - let's get this in then

if/when we start the deprecation process we can add `.. deprecated::` and raise a warning, but for now noting that it's deprecated in the spec and has no effect seems like a net positive

adding either now might result in something accidentally enforcing the deprecation in 3.0
ENH: Bring `devcontainer` closer to GitHub CI environment, improve `vscode` setup
State: closed

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This PR serves two purposes:

1. **Harmonizing devcontainer with the CI environment in GitHub Actions**
GitHub Actions use `micromamba` for Python environment management, while the old devcontainer setup used to use `pip` (with a list of dependencies fetched from `requirements-dev.txt` once at container image creation).
As I was trying to reproduce an unrelated issue locally, I was unsure whether deviations from the CI reults may have been caused by different Python environments. This change should reduce uncertainties in this regard.
2. **Modernizing the Visual Studio Code setup**
This PR attempts to produce a devcontainer that has high utility out of the box (e.g., installation and configuration of extensions for `mypy`, `flake8`, `black`, etc.)
Furthermore, it replaces deprecated configuration options with the preferred ones.

If you like this PR, please let me know which `whatsnew` file this should be mentioned in (`v2.1.0.rst` I presume?)

Also, do you need me to open an issue for this?

Labels:

Comments:
I see the check `docker run --rm pandas-dev-env python -c "import pandas as pd; print(pd.show_versions())"` is failing.

It did not previously because the old `Dockerfile` installed whatever `pandas` it pulled in via `pip`. I'm not sure what value this has, as I suppose the users of the docker image would be building and using their development versions anyway.

So I see two approaches here:

1. Remove the `docker run` check above, if you agree it serves little value; or
2. Have a separate CI process build and publish a complete docker image with the respective HEAD installed; this image could then be checked.

The second approach would be quite a bit more involved I think, as I don't think there is an established Docker publication infrastructure in `pandas`, is there?
> Also, do you need me to open an issue for this?

I would recommend you open an issue about this first before moving further with this PR, the Dockerfile for example used to be closer to what you had but it was deemed to large/slow to build and it was slimmed down. It would be good to discuss in an issue since it might also break people's workflows
Thanks for opening up https://github.com/pandas-dev/pandas/issues/54862. Closing to ensure the discussion happens in https://github.com/pandas-dev/pandas/issues/54862 first before proceeding here
BUG: ArrowExtensionArray.to_numpy avoid object dtype when na_value provided
State: closed

- [x] closes #54808
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.


Labels:
name: Bug
name: Arrow
description: pyarrow functionality

Comments:
Thanks @lukemanley
remove unnecessary conditional from equals method
State: closed

- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Currently, the `equals` method has two conditional expressions. The second conditional expression is not only a duplicate of the first one, but because it does not have the "not" operator in front of it like the first conditional expression does, it causes incorrect behavior of the equals function. 

Labels:

Comments:
thanks for your PR, but this is causing tests to fail so perhaps it's not so unnecessary
Shinyp
State: closed

- DOC: Rewrite a sentence for a better understanding of users.
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
thnx but please check good first issues
TST: added test for missing categories for `value_counts`
State: closed

- [x] closes #54836 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Labels:
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: Categorical
description: Categorical Data Type
name: Needs Tests
description: Unit test(s) needed to prevent regressions

Comments:
@rhshadrach can you please review this PR?
Thanks @rsm-23
ENH: add `ExtensionArray._explode` method; adjust pyarrow extension for use of new interface
State: closed

- [x] closes #54833 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
```bash
Error: /home/runner/work/pandas/pandas/pandas/core/arrays/base.py:1928:GL01:pandas.api.extensions.ExtensionArray._explode:Docstring text (summary) should start in the line immediately after the opening quotes (not in the same line, or leaving a blank line in between)
Error: /home/runner/work/pandas/pandas/pandas/core/arrays/base.py:1928:GL02:pandas.api.extensions.ExtensionArray._explode:Closing quotes should be placed in the line after the last text in the docstring (do not close the quotes in the same line as the text, or leave a blank line between the last text and the quotes)
Error: /home/runner/work/pandas/pandas/pandas/core/arrays/base.py:1928:GL03:pandas.api.extensions.ExtensionArray._explode:Double line break found; please use only one blank line to separate sections or paragraphs, and do not leave blank lines at the end of docstrings
Error: /home/runner/work/pandas/pandas/pandas/core/arrays/base.py:1928:SA03:pandas.api.extensions.ExtensionArray._explode:Description should be capitalized for See Also "Series.explode" reference
Error: /home/runner/work/pandas/pandas/pandas/core/arrays/base.py:1928:EX01:pandas.api.extensions.ExtensionArray._explode:No examples section found
```
Could you fix the merge conflict? I think this should be good to merge after that
yep, fixed! (CI failures look unrelated; spelling in a file untouched but this PR and an installation issue associated with BLAS)
Thanks @douglasdavis
ENH: allow EADtype to specify _supports_2d
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

The can_fast_transpose bit isn't used but is future-proofing for a time when we have e.g. pyarrow tables backing a 2D EA where transposing isn't cheap.  If asked, I'll revert that bit until it is necessary.

Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @jbrockmendel
DOC: update information of parameter value_name of pandas.melt
State: closed

- [x] closes #54715 
- [x] *Pre-commit* and *python3 scripts/validate_docstrings.py pandas.melt* were executed with no mistakes

The update developed was to add the information of no repeating an already existing column name.





Labels:
name: Docs

Comments:
@myles could you take a look in this PR, please? I know it's a small thing, but I am new in the open source community then I started with something simple.
Thanks @thiagogquinto
Infer large_string type as pyarrow_numpy strings
State: closed

- [ ] ref #54798 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

this covers the case where we don't overflow

Labels:
name: Strings
description: String extension data type and string data
name: Arrow
description: pyarrow functionality

Comments:
Looks good to me. We should handle the case of a too large array that would need to be split for being able to cast to "string", but that's fine for a follow up. Can you open an issue for that?
I was planning on keeping the other issue open
CLN: Remove unused parameter
State: closed

- [ ] closes #54806 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

not urgent, that's not public anyway

Labels:
name: Clean
name: Stale

Comments:
Looks like the groupby shift needs aligning too
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Closing to clear the queue, but feel free to reopen when you have time
#50990 Increase to_dict('list') performance
State: closed

- [x] closes #50990 
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.0.rst` file if fixing a bug or adding a new feature.

`25.8±2ms       1.61±0.4ms     0.06  frame_methods.ToDict.time_to_dict_datetimelike('list')`

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
Thanks @taytzehao
DOC: Removing repeated NA
State: closed

- [x] closes #53404 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
/preview
No preview found for PR #54821. Did the docs build complete?
@MarcoGorelli 
I'm trying to do what you suggested about this issue but I'm missing something. Would you mind taking a look please? thanks
thanks - looks like there's a CI failure, but I'm hoping it's unrelated - have tried merging upstream/main, let's see
/preview
No preview found for PR #54821. Did the docs build complete?
I downloaded the docs build, and it looks fine - what were you referring to when you wrote "but I'm missing something"?

Only comment I have is that `NaT` is also repeated, and that if we're going to have a missing values section, then I think that should also be moved there together with NA?
I wrote "but I'm missing something" because the preview didn't work. I thought that there was an error on what I did. I'll add the `NaT` to the section then. Thanks
/preview
No preview found for PR #54821. Did the docs build complete?
> I downloaded the docs build

Hey @MarcoGorelli, how did you do that? (I tried and couldn't find it).
hey - if you click on https://github.com/pandas-dev/pandas/actions/runs/6341541518/job/17225406933?pr=54821 and then "summary", at the bottom of the page it says "artifacts"

you can download then, extract it, and then from `python3 -m http.server` from inside (I think from the docs folder), and then open `localhost:8000` in your browser
Thanks @MarcoGorelli. 
Now, is this OK? : 

![Screenshot 2023-09-29 at 12 50 39](https://github.com/pandas-dev/pandas/assets/11835246/253715d6-1689-4cd0-96ad-104e618293b8)
Fix minor cython warning
State: closed

Fixes 

```
2023-08-26T19:21:26.6203637Z   warning: /home/runner/work/pandas/pandas/pandas/_libs/writers.pyx:123:50: Unknown type declaration 'Py_ssize_t' in annotation, ignoring
2023-08-26T19:21:26.6253780Z   warning: /home/runner/work/pandas/pandas/pandas/_libs/writers.pyx:123:50: Unknown type declaration 'Py_ssize_t' in annotation, ignoring
2023-08-26T19:21:26.6254699Z   warning: /home/runner/work/pandas/pandas/pandas/_libs/writers.pyx:123:50: Unknown type declaration 'Py_ssize_t' in annotation, ignoring
```

Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
Instead of importing like this can we namespace it? I'm a little wary of how confusing this is with the c type
Is that the general way we should be import Py_ssize_t? We import like this in a few places

```bash
% grep -R --include="*.pyx" "from cython cimport Py_ssize_t"
./pandas/_libs/sas.pyx:from cython cimport Py_ssize_t
./pandas/_libs/interval.pyx:from cython cimport Py_ssize_t
./pandas/_libs/join.pyx:from cython cimport Py_ssize_t
./pandas/_libs/byteswap.pyx:from cython cimport Py_ssize_t
./pandas/_libs/properties.pyx:from cython cimport Py_ssize_t
./pandas/_libs/reshape.pyx:from cython cimport Py_ssize_t
./pandas/_libs/missing.pyx:from cython cimport Py_ssize_t
./pandas/_libs/tslibs/fields.pyx:from cython cimport Py_ssize_t
./pandas/_libs/tslibs/tzconversion.pyx:from cython cimport Py_ssize_t
./pandas/_libs/tslibs/parsing.pyx:from cython cimport Py_ssize_t
./pandas/_libs/algos.pyx:from cython cimport Py_ssize_t
./pandas/_libs/internals.pyx:from cython cimport Py_ssize_t
./pandas/_libs/ops.pyx:from cython cimport Py_ssize_t
./pandas/_libs/parsers.pyx:from cython cimport Py_ssize_t
```

(if so might be good for a follow up?)
Ah I see. Carry on then - lgtm
DOC: added docstring for `storage_options` in `read_html`
State: closed

- [x] closes #54772
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.

Labels:
name: Docs

Comments:
@mroeschke docstring validation failing with

> 
> Redefining name from outer scope.........................................Failed
> - hook id: pylint
> - duration: 45.12s
> - exit code: 4
> 
> ************* Module pandas.io.html
> pandas/io/html.py:368:28: W0621: Redefining name 'doc' from outer scope (line 26) (redefined-outer-name)
> pandas/io/html.py:599:28: W0621: Redefining name 'doc' from outer scope (line 26) (redefined-outer-name)
> pandas/io/html.py:731:28: W0621: Redefining name 'doc' from outer scope (line 26) (redefined-outer-name)

one such function which shares the same name as the decorator is this:
`    def _parse_tables(self, doc, match, attrs):`

Any suggestions? should I rename the parameter name?
> should I rename the parameter name?

Yes this sounds reasonable
@mroeschke can you please review this PR?
Thanks @rsm-23
ERR: improve setitem error message for DatetimeArray
State: closed

This is a small detail, but currently we see this kind of error message:

```python
>>> ser = pd.Series(pd.arrays.DatetimeArray(np.arange(10, dtype="i8") * 24 * 3600 * 10**9))
>>> ser.searchsorted(["foo", "baz"])
TypeError: value should be a 'Timestamp', 'NaT', or array of those. Got 'StringArray' instead.
>>> ser.searchsorted(np.array([1, 2, 3]))
TypeError: value should be a 'Timestamp', 'NaT', or array of those. Got 'IntegerArray' instead.
>>> pd.options.mode.string_storage = "pyarrow_numpy"
>>> ser.searchsorted(["foo", "baz"])
TypeError: value should be a 'Timestamp', 'NaT', or array of those. Got 'ArrowStringArrayNumpySemantics' instead.
```
(and same error message is used for the array's `__setitem__`)

I personally don't like that the error message uses the array class names (those are not exactly private, but still not something users typically have to work with directly), and I think in general we should try to write our messages in a more generic, user-friendly way (what the user wants to know here is that they provided data of a wrong data type).

With this small PR, the above message become the following, respectively:

```
TypeError: value should be a 'Timestamp', 'NaT', or array of those. Got string array instead.
TypeError: value should be a 'Timestamp', 'NaT', or array of those. Got Int64 array instead.
TypeError: value should be a 'Timestamp', 'NaT', or array of those. Got string array instead.
```

(and as a bonus, the testing actually became easier)

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Thanks @jorisvandenbossche
Removed whitespace in README.md
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
This destroys line breaks