BrandonFors/pandas-github-issues huggingface dataset:
updating df.query and df.eval docstrings. resolves #16283
State: closed

- [x] closes #16283
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This is a docstring improvement suggested in #16283 issue discussion. `df.eval` and `df.query` are updated for consistency and adding exemples with `@` usage for local variables

Labels:
name: Docs
name: expressions
description: pd.eval, query

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/58749/
Thanks @mutricyl
DOC: Update orc.py to say orc instead of parquet
State: closed

Looks like the [filesystem implementation was initially copy pasted](https://github.com/pandas-dev/pandas/pull/51623/files#diff-dcfc078a101eee58a0d665a67ac19730f1e6b68d3bf7760ca8bbede2746a4c39R72) from the parquet one, and a few edits were missed to refer to orc instead.

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @mjperrone 
I appreciate the fast response!

I discovered this error while investigating this issue with the implementation. I would _really_ appreciate 👀 on that as well!

https://github.com/pandas-dev/pandas/issues/58746
DOC: Whatsnew for DataFrameGroupBy.__len__ with NA values bugfix
State: closed

- [x] closes #58644 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Thanks @rhshadrach 
Thanks for the fixup @mroeschke
DOC: fix formatting of date_format special options
State: closed

- ~~[ ] closes #xxxx (Replace xxxx with the GitHub issue number)~~
- ~~[ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature~~
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- ~~[ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.~~
- ~~[ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.~~


Labels:
name: Docs

Comments:
Thanks @wjandrea
PERF: Eliminate circular references in accessor attributes
State: closed

- [ ] closes #47667 (Replace xxxx with the GitHub issue number)
- [ ] closes #41357 (Replace xxxx with the GitHub issue number)

Given the discussion in those issues, appears users are OK with the tradeoff of accessor initialization not being cached in favor of eliminating a circular references that could block garbage collection


Labels:
name: Performance
description: Memory or execution speed performance
name: Accessors
description: accessor registration mechanism (not .str, .dt, .cat)

Comments:
While this is considered private and inside `pandas.core`, this seems to be used quite a bit in the wild in practice, so maybe we could keep the existing `CachedAccessor` name as an alias for now? (and either deprecate that specifically, or handle that through the generic pandas.core deprecation)

This came up in geopandas, but from a quick github search it seems quite some other projects are using this as well like pyspark, modin, etc

- https://github.com/apache/spark/blob/3edd6c7e1d504860fefc9921208ec47ab562ae41/python/pyspark/pandas/series.py#L48
- https://github.com/snowflakedb/snowpark-python/blob/fbf193e163308eb83d6fc182f333d9f648a18e5b/src/snowflake/snowpark/modin/pandas/accessor.py#L190
- https://github.com/modin-project/modin/blob/43eff5bea3862213135acda5fa65f3172279df57/modin/pandas/accessor.py#L193
- https://github.com/PatrikHlobil/Pandas-Bokeh/blob/bb4181c2f563e1b83d9853a2c344848d8006b32d/pandas_bokeh/__init__.py#L18
- https://github.com/jseabold/pandas-selectable/blob/fbfd0e86eec29b7404845d21f2cea2bf4f8a4b43/pandas_selectable/__init__.py#L6
Sure added the alias back in https://github.com/pandas-dev/pandas/pull/58754
Dep groupby corrwith
State: closed

- [x] closes #57158 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby
name: Deprecate
description: Functionality to remove in pandas

Comments:
@rhshadrach 
Thanks @pedrocariellof - and for the additional cleanups too!
ENH: Processing of .mask() for pd.NA #56844
State: closed

- [ ] closes #56844
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This commit addresses an issue where using NA in conjunction with the mask() method that resulted in unexpected behavior. The problem arose when comparing a Series containing NA as a condition within mask(), causing inconsistencies in the output. This fix ensures that NA behaves consistently as False in logical operations within mask().


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Stale

Comments:
@phofl @jorisvandenbossche this seems related to part of the ice cream agreement. Are we ready to move forward with that part? I’d expect to do the indexing bits all at once
@jbrockmendel I think we can simply treat this as a bug fix of the current implementation. We do handle the nullable boolean dtype specifically in indexing already (considering NA as False, a behaviour which has been discussed in the past already, https://github.com/pandas-dev/pandas/pull/31591), so fixing that for `mask` as well seems like a bug fix to me / fix to make things consistent.
Hi,
I've addressed the initial feedback on mask() handling pd.NA values. However, I'm unsure about the next steps, particularly regarding pd.NA equality semantics and indexing consistency. Any specific guidance would be appreciated!
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
DOC: Added clarification for the input to pandas.core.series.Series
State: closed

- [x] closes #55842 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @averagejoy
BUG: Fixed ChainedAssignmentError link to documentation will break? #58716
State: closed

- [x] closes #58716
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @SiddheshBangar
DOC: Fix Docstring validation for pandas.Series.floordiv and others
State: closed

ref: #58504 

fixes PR07 in:
```
pandas.Series.floordiv
pandas.Series.pow
pandas.Series.rmod
pandas.Series.rmod
pandas.Series.rtruediv
pandas.Series.sub
pandas.Series.eq
pandas.Series.ge
pandas.Series.gt
pandas.Series.le
pandas.Series.lt
pandas.Series.ne
```


Labels:
name: Docs

Comments:
Also fixes PR07 in Series.:

radd, rsub, rmul, rdiv, rfloordiv, rpow
Thanks @sam-baumann
DEPR: Deprecate method argument of reindex_like
State: closed

- [x] closes #58667 
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @tomytp
DOC: Fix typo in indexing.rst
State: closed

Fix typo in "Returning a view versus a copy" section
#58653

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @m-ahmadi
DEPR: revert enforcing the deprecation of exposing blocks in core.internals and deprecate with FutureWarning
State: closed

xref #58467

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Deprecate
description: Functionality to remove in pandas

Comments:
I reverted the removal of `Block` and `ExtensionBlock` from core.internals, and replaced `DeprecationWarning` with `FutureWarning`.  I think CI failures are unrelated to my changes.

@jorisvandenbossche could you please take a look at my changes? Is that what you meant in your comment in #58467?
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Looks like this PR is fairly stale so closing to clear the queue, but happy to reopen at a later time
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
(accidentally added this to the 2.3 milestone instead of 3.0, this change of course does not need to be backported because the related change is only on main)
Sentence restructured in compare methods's doc
State: closed

A sentence from section [Merge, join, concatenate and compare](https://pandas.pydata.org/docs/user_guide/merging.html#) in user guide doc -specifically in [DataFrame.compare()](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.compare.html#pandas.DataFrame.compare) method's  docs,has been restructured so that it is more easy to comprehend ,earlier more obscure sentence has now been simplified by just adding few words more 


Labels:
name: Docs

Comments:
Thanks @imTejasRajput
DOC: Grammar improvements in getting started tutorials
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @Aloqeely
DOC: fix PR02 for pandas.tseries.offsets.MonthEnd methods
State: closed

- [ ] xref #58536

fixes
```
pandas.tseries.offsets.BMonthEnd PR02
pandas.tseries.offsets.BusinessMonthEnd PR02
pandas.tseries.offsets.MonthEnd PR02
pandas.tseries.offsets.SemiMonthEnd PR02
```


Labels:
name: Docs

Comments:
Thanks @tuhinsharma121
DOC: Fix PR02 for pandas.tseries.offsets.BYearBegin and pandas.tseries.offsets.YearBegin
State: closed

- [ ] xref #58536

fixes
```
pandas.tseries.offsets.BYearBegin PR02
pandas.tseries.offsets.YearBegin PR02
```

Labels:
name: Docs

Comments:
Thanks @tuhinsharma121
DOC: Document Remote Code Execution risk for Dataframe.query and computation.eval
State: closed

## TLDR
This diff adds documentation to the Remote Code Execution risk for `Dataframe.query` and `computation.eval` with the same wording used in [Dataframe.eval](https://github.com/pandas-dev/pandas/blob/v2.2.2/pandas/core/frame.py#L4850-L4852)

## Details
The Dataframe.query function is internally calling Dataframe.eval [here](https://github.com/pandas-dev/pandas/blob/b552dc95c9fa50e9ca2a0c9f9cdb8757f794fedb/pandas/core/frame.py#L4547) passing the `expr` parameter. 
Dataframe.eval is correctly warning in the [documentation](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.eval.html#pandas.DataFrame.eval) that this could lead to Code execution risk.
This is possible because using the `@` syntax we can access local variables and by referencing the imported pandas library we can get the `os.system` function thanks to `os` being imported inside [compat/__init__.py](https://github.com/pandas-dev/pandas/blob/main/pandas/compat/__init__.py#L13). In this way passing something like `@pd.compat.os.system('whoami')` to the `expr` parameter of `Dataframe.query` we can achieve code execution.

However in Dataframe.query [documentation](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.query.html) we do not surface this risk we only point out that pandas.eval is called but also [there](https://pandas.pydata.org/docs/reference/api/pandas.eval.html#pandas.eval) we do not warn about any code execution risks. 

The vulnerable code I found and that also surprised the developer who wrote it of the code execution risk can be summarized in UserControlled input flowing into the DataFrame.query expr value similarly to
```
import pandas as pd

def api_whatever(request: HTTPRequest):
  filter = request.GET["filter"]
  data = {
    "test": [125, 230, 412],
  }
  df = pd.DataFrame(data)
  df.query(filter)
   ... 

```
Calling this api with data:
https://www.example.com?filter=@pd.compat.os.system('whoami')
will allow code execution.



Labels:
name: Docs
name: expressions
description: pd.eval, query

Comments:
Thanks @r0rshark!
DEPR: Change stacklevel to 2 in DataFrame(mgr) deprecation
State: closed

For reference: #55591

@jorisvandenbossche is this the only thing I'm supposed to change? or am I supposed to also change all `stacklevel=2` to `stacklevel=find_stack_level()`


Labels:
name: Clean
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
> is this the only thing I'm supposed to change? or am I supposed to also change all `stacklevel=2` to `stacklevel=find_stack_level()`

This should be sufficient, and we explicitly want to keep it hard-coded at 2 and not use `find_stack_level()` for this case.
And thanks for the PR! If you merge in latest main, that should fix some of the CI failures.
> And thanks for the PR! If you merge in latest main, that should fix some of the CI failures.

My pleasure - green now.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks @Aloqeely!
TST: Fix CI failures (don't xfail postgresql / don't xfail for pyarrow=16)
State: closed

Using `tm.assert_produces_warning` for the first test gives:
```
AssertionError: Warning not set with correct stacklevel
```
It seems the stacklevel is set incorrectly and should be changed (see #55591), I just temporarily fixed it by using `pytest.mark.filterwarnings`

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
pre-commit.ci autofix
`xarray` just released a new version which seems to be breaking some tests. I will take a look at it tomorrow if no one else fixes it.
Thanks @Aloqeely
TST: Test searchsorted for monotonic decreasing index
State: closed

- [x] ref #58641 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Modified the tests to ensure `searchsorted` works for monotonic decreasing indices

Labels:
name: Docs

Comments:
Thanks @Aloqeely
A  typo in user guide doc 
State: closed

There was a typo in user guide doc specifically in [Merge, join, concatenate and compare](https://pandas.pydata.org/docs/dev/user_guide/merging.html#) ,after fixing ,it is now making more sense while reading 

Labels:
name: Docs

Comments:
A major typo has been fixed now ,earlier it was not making any sense while reading that particular line of docs but now it is more intuitive and making more sense while reading
Thanks @imTejasRajput
typo fixed
State: closed

There was typo in merging documentation ,fixed it now

Labels:

Comments:
A typo is fixed in  pandas documentation  particulary in user guide of  [Merge, join, concatenate and compare](https://pandas.pydata.org/docs/dev/user_guide/merging.html#) section   . Quoting a line from docs where a minor change has been done for clarity

**Earlier:**  

You can [Series](https://pandas.pydata.org/docs/dev/reference/api/pandas.Series.html#pandas.Series) and a [DataFrame](https://pandas.pydata.org/docs/dev/reference/api/pandas.DataFrame.html#pandas.DataFrame) with a [MultiIndex](https://pandas.pydata.org/docs/dev/reference/api/pandas.MultiIndex.html#pandas.MultiIndex) if the names of the [MultiIndex](https://pandas.pydata.org/docs/dev/reference/api/pandas.MultiIndex.html#pandas.MultiIndex) correspond to the columns from the [DataFrame](https://pandas.pydata.org/docs/dev/reference/api/pandas.DataFrame.html#pandas.DataFrame).

**Now:**

You can **merge** [Series](https://pandas.pydata.org/docs/dev/reference/api/pandas.Series.html#pandas.Series) and a [DataFrame](https://pandas.pydata.org/docs/dev/reference/api/pandas.DataFrame.html#pandas.DataFrame) with a [MultiIndex](https://pandas.pydata.org/docs/dev/reference/api/pandas.MultiIndex.html#pandas.MultiIndex) if the names of the [MultiIndex](https://pandas.pydata.org/docs/dev/reference/api/pandas.MultiIndex.html#pandas.MultiIndex) correspond to the columns from the [DataFrame](https://pandas.pydata.org/docs/dev/reference/api/pandas.DataFrame.html#pandas.DataFrame).

without **merge** word sentence is not making any sense
DOC: Update Python version support
State: closed

The supported versions in https://pandas.pydata.org/docs/getting_started/install.html#python-version-support are hard-coded and the reasoning/criteria behind that is ambiguous.

The policy in the [version policy](https://pandas.pydata.org/docs/development/policies.html#python-support) document is clearer, so I just copied it.

Labels:
name: Docs

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/58680/
Thanks @Aloqeely
DOC: Enforce Numpy Docstring Validation for pandas.IntervalIndex.to_tuples and pandas.arrays.IntervalArray.to_tuples
State: closed

- [ ] xref #58498 

fixes
```
pandas.IntervalIndex.to_tuples RT03,SA01
pandas.arrays.IntervalArray.to_tuples RT03,SA01
```

Labels:
name: Docs

Comments:
Thanks @tuhinsharma121
DOC: Enforce Numpy Docstring Validation for pandas.IntervalIndex.is_non_overlapping_monotonic and pandas.arrays.IntervalArray.is_non_overlapping_monotonic
State: closed

- [ ] xref #58498 

fixes
```
pandas.IntervalIndex.is_non_overlapping_monotonic SA01
pandas.arrays.IntervalArray.is_non_overlapping_monotonic SA01
```


Labels:
name: Docs

Comments:
Thanks @tuhinsharma121