BrandonFors/pandas-github-issues huggingface dataset:
Removed GNU strdup extension from JSON
State: closed

Came up during some of the meson work @lithomas1 is doing. Looks like we allow non-portable GNU extensions in JSON, but these aren't upstream in ultrajson. Couldn't find a history of why we have this.


Labels:
name: IO JSON
description: read_json, to_json, json_normalize
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
See also https://stackoverflow.com/questions/5582211/what-does-define-gnu-source-imply/5583764#5583764
We are pretty far diverged. Unfortunately when we vendored stuff in the past we would edit in place rather than bolt on to it, so it's hard to sync up with more recent versions
OK looks like this goes back to https://github.com/pandas-dev/pandas/issues/4918 though I think the strdup calls are unnecessary. Let's see...
Interestingly enough this becomes part of the standard in C2X
FYI, the docs build seems to be segfaulting from a quick glance, since it looks like the same thing that happened to me. If you run it with a single worker, you'll be able to see the exit code, I think (245 is segfault). You can also run it with catchsegv, which'll print out more details if it's actually a segfault.
Super strange. Cool thanks for the heads up - didn't know about that feature
Looks like I was freeing the wrong variable. Surprised it didn't segfault on other parts of CI - its possible the locale test isn't really even running but will leave that to another effort

cpplint also complains about strcpy but is fine with memcpy...I think the former would be more clear in this case. Worth reconsidering if we want to continue using cpplint given its not well maintained by Google
Nice cleanup @WillAyd
CI: Limit pip extras check to Build tag
State: closed

xref https://github.com/pandas-dev/pandas/pull/49241#issuecomment-1320485994

cc @lithomas1 


Labels:
name: Build
description: Library building on various platforms
name: CI
description: Continuous Integration

Comments:
This doesn't seem to be triggering, but the logic looks correct to me(pretty sure its identical to the working sdist trigger).

I'll try again later, we might be hitting some sort of concurrency limit. 
Oh, I figured it out. You need to add "labeled" to the "on" tag like https://github.com/pandas-dev/pandas/blob/main/.github/workflows/sdist.yml#L12, for the labelling to trigger the runs.
> Oh, I figured it out. You need to add "labeled" to the "on" tag like https://github.com/pandas-dev/pandas/blob/main/.github/workflows/sdist.yml#L12, for the labelling to trigger the runs.

Ah cool thanks. Added the on tags to the workflow
Thanks @mroeschke.
DEPR: enforce Series constructor with int dtype deprs
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jbrockmendel 
the original test fails on 1.5.x with numpy 1.24.0 - we OK to backport?
This would enforce a deprecation early in 1.5 so that would be an API change?
sure, but it's already erroring with numpy 1.24.0

alternatively, do we just pin numpy in the 1.5.x branch and let numpy error? https://github.com/pandas-dev/pandas/pull/50384
For maximal backwards compat, users should still be able to use pandas 1.5.3 with numpy < 1.24 and not see an API change while being able to use numpy=1.24.0 and not see failures, but that's a lot of effort I presume

I would be supportive of pinning numpy in the 1.5.x branch and possibly also including an upper pin for numpy in `pyproject.toml`? Thoughts @pandas-dev/pandas-core 


would an upper pin not be pretty limiting, especially on such a widely-used package as numpy, if it's just for the sake of a single test?
That's fair, this isn't a showstopping issue. 

If users address the pandas deprecation they shouldn't hit the numpy error correct? If so then I would opt to pin numpy on the 1.5.x build
I would prefer not to pin numpy as a whole, since pinning tends to break pip's solver. There's also the risk that pip will just pull the version without the pin.
would the simplest thing be to just conditionally xfail the affected test?
sure, I've gone for that in https://github.com/pandas-dev/pandas/pull/50384
REGR: arithmetic ops recursion error with midx with different dtypes
State: closed

- [x] closes #49769 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


We don't actually care about the dtype here, but equals does so we have to check this differently


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
cc @mroeschke could you have a look? would be nice to get into 1.5.2
Thanks @phofl
DEPR: enforce inplaceness for df.loc[:, foo]=bar
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @jorisvandenbossche i expect the CoW-case edits in the copy_view tests are unwanted.  pls advise.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Deprecate
description: Functionality to remove in pandas

Comments:
Everything here looks mostly address so going to merge. Can create follow up PRs if there are any followups needed
DOC: Update start/end parameter for BusinessHour and CustomBusinessHour
State: closed

These are all valid usages. Examples from pandas-stubs here:

https://github.com/pandas-dev/pandas-stubs/blob/main/tests/test_timefuncs.py#L601-L639

Labels:
name: Docs
name: Frequency
description: DateOffsets

Comments:
Thanks @gandhis1
STYLE: Enable Pylint useless-parent-delegation warning
State: closed

Associated with #48855.  Enables the Pylint type "W" warning `useless-parent-delegation`.

I have questions about the changes that I made to `test_decimal.py` and `test_numpy.py`.  I'll elaborate in comments below.

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
you'll need to run the pre-commit checks too, see the contributing guide for how to do this
PERF: first try inplace setitem for .at indexer
State: closed

This is from the commit that I originally also tested in the PR that caused the perf regression: https://github.com/pandas-dev/pandas/pull/47074/files#r878902106 (but eventually didn't include because it didn't seem needed for getting the correct behaviour, but didn't consider performance at the time)

Using the example from https://github.com/pandas-dev/pandas/issues/49729:

```
import pandas as pd

def foo(df):
    for idx in df.index:
        df.at[idx, "bar"] = 3

df = pd.DataFrame(range(10000))
df["bar"] = 0

%timeit foo(df)
# 442 ms ± 16.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  <-- main
# 202 ms ± 3.13 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)` <-- PR
```

This is not yet as fast as on 1.4.x, but https://github.com/pandas-dev/pandas/pull/49771 further helps with this as well. With both together, I get to down to 119 ms (vs 50 ms on 1.4.3). 
The remaining overhead is mostly from Manager.iget to get the SingleBlockManager


- [x] closes https://github.com/pandas-dev/pandas/issues/49729
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance

Comments:
@jbrockmendel any other comment, or OK to merge?
Good point. Unless I am missing something, it seems we actually don't have _any_ benchmarks at all for `.at` .. 
(the underlying `_get_value` was previously still benchmarked through the `lookup` benchmarks, but those have been removed since that method had been removed)
https://github.com/pandas-dev/pandas/pull/49772/files#r1026928855
Added a benchmark case for at (both getitem and setitem) to the existing DataFrame indexing benchmark class. Using a code snippet mimicking the benchmark setup:

```
import pandas._testing as tm

index = tm.makeStringIndex(1000)
columns = tm.makeStringIndex(30)
df = DataFrame(np.random.randn(1000, 30), index=index, columns=columns)
idx_scalar = index[100]
col_scalar = columns[10]

In [3]: %timeit df.at[idx_scalar, col_scalar] = 0.0
100 µs ± 18.8 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)  # <-- main
25.7 µs ± 2.54 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)  # <-- PR
```
PERF: Use fastpath for accessing option value in internals
State: closed

Related to https://github.com/pandas-dev/pandas/issues/49729

Accessing an option value is in general quite fast, but when it is in code that is called repeatedly in a for loop, it still adds up it seems. 

Using the example from https://github.com/pandas-dev/pandas/issues/49729:

```python
import pandas as pd

def foo(df):
    for idx in df.index:
        df.at[idx, "bar"] = 3

df = pd.DataFrame(range(10000))
df["bar"] = 0

%timeit foo(df)
# 442 ms ± 16.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  <-- main
# 316 ms ± 10.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  <-- PR
```

I think this should be a safe way to access the option (but will add a test that ensures you can switch back and forth with CoW within a session).

Are we OK with this pattern? 

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature. -> added in https://github.com/pandas-dev/pandas/pull/49772


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
hmm at one point we had a code check that checked for private imports.  guess that got lost somewhere along the way
Curious if you've looked at making `get_option` a class and using `__slots__` instead for this? Seems like it would be the idiomatic approach to speeding up attribute access
@WillAyd you mean the `DictWrapper` class that `get_option`/`options` uses? That is currently using a dynamic `__getattr__` to look up attributes in a global dictionary that actually stores the option values. 
What you suggest might be an option (but not directly an idea, didn't look into that), but that seems a bigger change in the internals of the options machinery. While what I am doing here is just using the fact that the current options internals is just a plain dictionary. 
Best Practices-wise, the best-case would be to try to optimize get_option.  Next-best would be to make _global_config public-for-inernal-usage and canonize this as how we do option checks within pandas
> best-cast would be to try to optimize get_option

That would require a change of (or addition to) how `get_option` works, I think. Currently that takes a string (eg `get_option("mode.copy_on_write")`), and thus that requires parsing the string, and also has some overhead. 

It's easy to rename `_global_config` to `global_config` (its module is already private) if that is preferred.
yah even if we did optimize get_option a dict lookup will always be way faster
STYLE enable pylint: typevar-name-incorrect-variance
State: closed

Issue #48855. This PR enables pylint type "C" warning: `typevar-name-incorrect-variance`
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @natmokval
BUG: loc raising error for MultiIndex with bool indexer
State: closed

- [x] closes #47687 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Thanks @phofl
STYLE: fixing linting issues (redefined-outer-name)
State: closed

Fixing pylint issues in xml.py, style.py, and _json.py

- [ ] Adds Fixes to #49656
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I am a new contributor to Pandas, and still learning the ropes, I apologize ahead of time for making any rookie mistakes. Any advice would be highly appreciated! 

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
This code causes the tests to fail, will need to troubleshoot why.
CLN: Cleanups in groupby due to numeric_only deprecations
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Summary:

 - For ops that call `_op_via_apply`, numeric_only being None or lib.no_default is the same as False since these just call the corresponding DataFrame methods where the deprecation has already been enforced
 - The above remark also applies for idxmin/idxmax
 - Where we set `ignore_failures=numeric_only is lib.no_default`, across groupby numeric_only=lib.no_defualt is internally treated as numeric_only=True, so there is no reason to ignore failures.
 - In the one test change, this is where groupby attempts to default to `numeric_only=True` but as this results in an empty frame the op instead defaults to `numeric_only=False`. For the whatsnew, this will be covered by the general message that groupby will no longer automatically drop nuisance columns.

Labels:
name: Groupby
name: Clean
name: Nuisance Columns
description: Identifying/Dropping nuisance columns in reductions, groupby.add, DataFrame.apply

Comments:
Thanks @rhshadrach
BUG/TST: Include sem & count in all_numeric_reductions
State: closed

The `sem` bug with Arrow types could be backported to 1.5.x but given the larger testing changes IMO it's fine for 2.0


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Arrow
description: pyarrow functionality

Comments:
Going to push this one in since the pyarrow sem implementation matches the `nanops.sem` one. If both are wrong at least we have a test now that said both should be aligned.
TST: Refactor sql test classes.
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

After #49531 , the `sql.SQLDatabase` class only accepts SQLAlchemy Connections and not Engines.  As a result, any test class that would have passed an Engine had to be turned off in that PR, even though it was still needed to support the test classes which passed Connections.  The `_EngineToConnMixin` class bridged between the Engine test classes and the Connection test classes.

In this refactor, I remove the `_EngineToConnMixin` class and edit the setup and teardown methods so that the number of classes is reduced.  The number of tests in `test_sql.py` and their outcomes remain the same.  With sqlalchemy installed, 330 pass, 6 xfail, and 2 are skipped.  Without sqlalchemy, 83 pass, and 255 are skipped.  The tests seem to be slightly faster after the refactor.

This refactor will simplify the PR that I'm ultimately working towards (#48576).

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
Unfortunately, that would take a while for me to figure out.
Thanks @cdcadman
TST: Use stronger assertion in test_regression_allowlist_methods
State: closed

- [ ] closes #49629
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby

Comments:
Awesome work @isaac-chung 
Thanks for your help! @mroeschke
Backport PR Revert "Add color and size to arguments (#44856)"
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Visualization
description: plotting

Comments:
Thanks
STYLE ignore no-else-* checks, + assorted cleanups
State: closed

@jbrockmendel 's mentioned he's not keen on the`no-else-*` pylint checks, and to be fair, they are kinda nitpicky. Hard to imagine that they could ever help uncover a bug

Let's keep them turned off then. Have added a section of "intentionally turned off checks"

Labels:
name: Code Style
description: Code style, linting, code_checks
name: Clean

Comments:
can we add this stuff to the pre-commit hook so i can catch it locally instead of in the CI?
could do, but @jorisvandenbossche had brought up that it's a pretty slow hook (esp. compared with the other ones)

I'll see if there's a way to speed it up
@jbrockmendel @jorisvandenbossche I don't there's much that can be done to speed up pylint - they say it's expected that it's slow because it's thorough.

And to be fair, it has been helping find actual issues.

To not disrupt local development, but to also help keep catch issues, I'd suggest:
- keep pylint only running in CI with `stages: [manual]` (though note that you can  still trigger it locally if you wish, with `pre-commit run pylint --hook-stages manual --all-files`)
- turn off the more nitpicky pylint checks such as the `no-else-*` ones, which probably don't help find issues anyway
> pre-commit run pylint --hook-stages manual --all-files

this will help me out, thanks.  The pain point for me is having stuff pass locally and then fail on the CI for some nitpicky reason.
cool, so if we remove the nitpicky checks, then local development should still be fast and CI should only fail for reasons which are potential sources of bugs (e.g. `redefined-outer-name`), but you can still run these checks manually if you wish

and if the are failures because of other reasons deemed to be nitpicky, we can add them back to `pyproject.toml` in the `# intentionally turned off` section

any objections to moving along with this?
> any objections to moving along with this?

nope
> pre-commit run pylint --hook-stages manual --all-files

Trying this I'm getting `pre-commit: error: unrecognized arguments: --hook-stages manual`
sorry, typo, I meant `--hook-stage`
> > any objections to moving along with this?
> 
> nope

cool, thanks - if someone approves I'll merge then
thanks for reviewing!