BrandonFors/pandas-github-issues huggingface dataset:
numpy ravel with dataframe test GH#26247
State: closed


- [ ] closes #26247 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Needs Tests
description: Unit test(s) needed to prevent regressions

Comments:
Thanks @liang3zy22
TST: move DataFrame.applymap tests to tests/frame/methods/test_map.py
State: closed

Currently the tests for `DataFrame.applymap` are mixed together with the tests for `DataFrame.apply`. Keeping them separate makes it easier to find tests for one method so I've moved the `applymap` ones to tests/frame/methods/test_map.py.

Note there is also a proposal to rename `DataFrame.applymap` to `DataFrame.map` (#52364). The one who gets merged later of this PR or 52364 will need to be rebased.

xref #52384.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Thanks @topper-123
DOC: Fix link points to other organization
State: closed

- [x] closes #51625 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks for the pull request, are you still working on this or can it be taken out of draft mode?
> Thanks for the pull request, are you still working on this or can it be taken out of draft mode?

Hi @mroeschke 
Yes, I'm done with the changes. 
Moving it to `ready for review`

Thanks for approving!
Thanks @srkds
TST: `Groupby.agg` with mutliples ignores as_index False
State: closed

- [x] closes #50724  (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Thanks @luke396
fix #52389 : typo in "It row raises"
State: closed

changed row to now in the statement "It row raises a NotImplementedError".

- [x] closes #52389 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @yonashub
BUG: dt.days for timedelta non-nano overflows int32
State: closed

- [ ] closes #52386 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Timedelta
description: Timedelta data type
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Going to merge as the checks are green, but can address any follow ups
PERF: slightly faster dtype.kind checks
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
Thanks @jbrockmendel
CLN: remove unused nan comparison functions, ensure_float
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Clean

Comments:
Thanks @jbrockmendel
TST: move Series.map tests to tests/series/methods/test_map.py
State: closed

Currently the tests for `Series.map` are mixed together with the tests for `Series.apply`. Keeping them separate makes it easier to find tests for one method and I've moved the map ones to tests/series/methods/test_map.py.

Also fixed a bug that I found in this proces:

```python
>>> import pandas as pd
>>> ser = pd.Series([], dtype=int)
>>> ser.map(lambda: x + 1).dtype
object  # main
int64  # this PR
```


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Thanks @topper-123
PERF: Series.any/all
State: closed

- [x] closes #26032 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Example based on OP from #26032
```
import numpy as np
import pandas as pd
ser = pd.Series(np.random.randint(0, 2, 100000)).astype(bool)

In [2]: %timeit s.any(skipna=True)
5.79 µs ± 719 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)  # <- PR
8.44 µs ± 473 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)  # <- main
```

We'll be hard-pressed to get trim any more overhead from this, so I'm declaring this as closing #26032.

Labels:
name: Performance
description: Memory or execution speed performance
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
Thanks @jbrockmendel
Add a test for agg std
State: closed

- [X] closes #18734 
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


I believe that this test ensures the expected output. Let me know if you have any questions or concerns.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Can you move the test to `pandas/tests/apply/test_frame_apply.py`?
I believe that this will fulfill your request, but please let me know if there is anything else I can do. @mroeschke 
Looks like a few of the code check builds are failing
@mroeschke , I have fixed the code checks that were failing, and I have constructed the tests  as  your instructions. 
Awesome! Thanks @steliospetrakis02 
> Awesome! Thanks @steliospetrakis02

You're welcome! I thank you for your guidance and assistance as I take my first steps into contributing to pandas
PERF: call np.errstate less
State: closed

These add a surprising amount of overhead.

Labels:
name: Performance
description: Memory or execution speed performance
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
> So now float point errors from numpy will warn correct?

This only removes np.errstate in places where doing so didn't produce any new warnings in the tests.
Thanks @jbrockmendel
Bug: DataFrame.clip fixed for lower > upper
State: closed

Fixed behavior of DataFrame.clip for `lower` > `upper` arguments. Changed to be consistent with numpy - all equal to `upper`. Main reason of bug was undocumented swap of arguments for scalars and for list-like arguments comparison with initial DataFrame for both bounds.

- [x] closes #52147 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Looks like some tests are failing after merging in main branch. I will look at this.
I personally think raising is the most informative behavior, but open to thoughts from other core team members
It is worth mentioning that `Series.clip` also works in the same way as `numpy.clip`. I think it is rather logical result, in first step we are clipping lower numbers, in second step upper numbers. We end up with all numbers equal to `upper`.

Another similar example from pandas api, `pandas.date_range` - when we pass `start` > `end` there is no exception, but empty `DatetimeIndex`.
> Another similar example from pandas api, pandas.date_range - when we pass start > end there is no exception, but empty DatetimeIndex.

`pd.date_range` makes sense to be more flexible in this sense because the `freq` can be negative allowing start > end to produce a non-empty date range
Ok, so I guess we also need to change `Series.clip` method to raise exception. 

I am not sure which exception it should raise, some existing pandas exception? I guess `ValueError` might be appropriate.
I have started implementation and I think there is too much to make validation for scalars, lists and DataFrame arguments. 

So if you insist on doing this, I can close this PR and unassign from issue.
I would suggest closing this PR now until there is some consensus on the main issue on how to handle this scenario
Thanks for the PR, but I think this scenario needs a little more discussion before moving forward so closing
API/DEPR: Rename DataFrame.applymap to DataFrame.map
State: closed

- [x] closes #52353
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This PR builds on #52355, so could make sense to look at that first or ignore the first commit in this PR.

Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
In the file: pandas/doc/source/user_guide/style.ipynb there is a linked reference to DataFrame.applymap in a cell, under "Methods to add styles". It is probably best to also relink this, now, to avoid it being forgotten and breaking later.
Ok, I’ve done that @attack68.
Shouldn't we deprecate this before outright removing? Wouldn't this break user code for 2.1?
> Shouldn't we deprecate this before outright removing? Wouldn't this break user code for 2.1?

That's what the current code does I think
My mistake - totally misread that
Yes this PR just deprecates `DataFrame.applymap`.
```
2023-04-14T22:44:55.0688278Z /home/runner/work/pandas/pandas/doc/source/whatsnew/v2.1.0.rst:249: WARNING: Inline interpreted text or phrase reference start-string without end-string.
```
Thanks,, I’ve updated.
Updated.
Thanks @topper-123
TYP: Fix type hint for params passed to read_sql_query
State: closed

- [x] closes #49716
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
CI failed due to a GitHub issue. Can someone rerun the checks?
Thanks @Tolker-KU
PERF: extract_array
State: closed

```
import numpy as np
import pandas as pd

arr = np.arange(5)
ser = pd.Series(arr)

%timeit pd.core.construction.extract_array(arr, extract_numpy=True)
924 ns ± 23.5 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)  # <- main
205 ns ± 15.6 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)  # <- PR

%timeit pd.core.construction.extract_array(ser, extract_numpy=True)
1.34 µs ± 20.8 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)  # <- main
471 ns ± 9.53 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)  # <- PR
```

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
thx @jbrockmendel
CI: Test development container
State: closed

- [x] closes #48537
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
> the pull request, but I am -0.5 on fully running the tests on the development container. We already run the test suite for other platforms on our CI which can take a while and the container is more of a development tool than a CI runner

I agree with you point. Feature description says "The Docker dev environment test should be that it passes at least tests ran in the test_fast.sh file.". In issue 48537, test_fast.sh was used but the file was deleted in https://github.com/pandas-dev/pandas/pull/48958. Shall i close this pr or do you see an alternative?

I think we can close this PR and original issue as won't do.
PERF: DataFrame.values for pyarrow-backed numeric types
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Perf improvement for `DataFrame.values` when backed by a _single pyarrow numeric dtype without any nulls_. I realize this is a narrow use case, so happy to close this PR if it isn't worth special casing. The current slowness is due to `DataFrame.values` always casting to `object` dtype for EA-backed frames. Unfortunately, a single null anywhere in the dataframe misses this optimization since `pd.NA` is used as the null representation in the `ndarray`.

```
import pandas as pd
import numpy as np

data = np.random.randn(100_000, 20)
df = pd.DataFrame(data, dtype="float64[pyarrow]")

%timeit df.values

# 98.7 ms ± 11.8 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)   <- main
# 3.56 ms ± 96.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  <- PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Arrow
description: pyarrow functionality

Comments:
This also changes behavior (e.g. getting float instead of object). Personally, I think this is fine but we have an unresolved discussion about this somewhere. We should decide there first before special casing here I'd say
> This also changes behavior (e.g. getting float instead of object). 

Yes, the performance improvement is due to avoiding the cast to object. Note, this behavior actually already exists on `main` for a `DataFrame` with a single column:

Behavior on `main`:
```
import pandas as pd

df1 = pd.DataFrame({"a": [1.0, 2.0, 3.0]}, dtype="float64[pyarrow]")
df2 = pd.DataFrame({"a": [1.0, 2.0, pd.NA]}, dtype="float64[pyarrow]")

print(df1.values.dtype)  # float64
print(df2.values.dtype)  # object
```

> Personally, I think this is fine but we have an unresolved discussion about this somewhere. We should decide there first before special casing here I'd say

Sure, I think you might be referring to https://github.com/pandas-dev/pandas/issues/22791
closing for now pending further discussion in https://github.com/pandas-dev/pandas/issues/22791
REF: Use _python_apply_general internally
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

For #7155, avoiding calling the public `.apply` to raise deprecation warnings.

Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Thanks @rhshadrach
PERF: Series.any
State: closed

```
import numpy as np
import pandas as pd

s = pd.Series(np.random.randint(0, 2, 100000)).astype(bool) 

In [8]: %timeit s.any(skipna=True)                                                                     
19.5 µs ± 757 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)  # <- main
7.71 µs ± 142 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)  # <- PR

In [10]: %timeit s.values.any() 
3.14 µs ± 47.4 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
```

In the OP for #26032 the Series.any was 223x slower than the ndarray.any.  In main it is 6.2x and this gets it down to 2.5x.  Let's look at what's left:

```
In [21]: %prun -s cumtime for n in range(10000): s.any(skipna=True)
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    10000    0.010    0.000    0.182    0.000 generic.py:11509(any)
    10000    0.009    0.000    0.171    0.000 generic.py:11208(any)
    10000    0.026    0.000    0.163    0.000 generic.py:11165(_logical_func)
    10000    0.020    0.000    0.126    0.000 series.py:4530(_reduce)
    10000    0.016    0.000    0.094    0.000 nanops.py:487(nanany)
    10000    0.006    0.000    0.046    0.000 {method 'any' of 'numpy.ndarray' objects}
    10000    0.004    0.000    0.040    0.000 _methods.py:55(_any)
    10000    0.036    0.000    0.036    0.000 {method 'reduce' of 'numpy.ufunc' objects}
    10000    0.012    0.000    0.022    0.000 nanops.py:259(_get_values)
    20000    0.010    0.000    0.014    0.000 common.py:1093(needs_i8_conversion)
    10000    0.005    0.000    0.008    0.000 series.py:726(_values)
    30000    0.006    0.000    0.006    0.000 {built-in method builtins.isinstance}
    10000    0.004    0.000    0.006    0.000 _validators.py:224(validate_bool_kwarg)
```

We have a few layers in between the Series.any call and nanops.nanany that we could refactor away.  These add up to .01+.009+.026+.02=.065 seconds here, about 34% of the total runtime.


The ufunc_config+seterr calls add up to .177s here.  Disabling the `with np.errstate(all="ignore")` in `Series._reduce` cuts the timeit result down to 9us.  It also doesn't break any tests or surface any warnings in the tests.
Update: did this after determining that we already set errstate within the relevant nanops functions.

_get_values is still 
more than half
update:a quarter of nanany, 
and more than half of _get_values is in extract_array.  According to the annotation the extract_array is unnecessary (though it is necessary for the doctests and a handful of tests in test_nanops).
update: the extract_array has now been removed.

So there is still some room for optimization, but some of it would be more invasive than this.  I would be +1 on making all these optimizations, will wait for a little buy-in first though.

Labels:
name: Performance
description: Memory or execution speed performance
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
Thanks @jbrockmendel