BrandonFors/pandas-github-issues huggingface dataset:
CLN: memory_map do not silently fail
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: IO CSV
description: read_csv, to_csv
name: Clean

Comments:
thanks @twoertwein
TST: pytest.mark.parameterize some tests
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them



Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks, small comment maybe worth doing in followup
CI: Filter any SSLSocket ResouceWarnings from dependencies
State: closed

- [x] closes #44732
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them



Labels:
name: CI
description: Continuous Integration

Comments:
thanks @mroeschke 
We should be able to get rid of [this](https://github.com/pandas-dev/pandas/blob/c58d8ee6e93fc1ba7aa4a40dc71488cc9807ed43/pandas/tests/io/parser/common/test_chunksize.py#L210) xfail now right ?
> We should be able to get rid of [this](https://github.com/pandas-dev/pandas/blob/c58d8ee6e93fc1ba7aa4a40dc71488cc9807ed43/pandas/tests/io/parser/common/test_chunksize.py#L210) xfail now right ?

Yup should be good to remove
TST: Add regression test for groupby rolling
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them



Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @mroeschke
fix PR06 failing CI
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


xref https://github.com/pandas-dev/pandas/pull/43511#issuecomment-986314251

Labels:
name: Docs

Comments:
hah, should have had him rebase.

thanks @MarcoGorelli 

merge on green
CLN: FIXMEs
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean

Comments:
CI fail is from the lint issue fixed on master a little while ago
One that i'll need some help figuring out how to make actionable: https://github.com/pandas-dev/pandas/blob/master/pandas/plotting/_matplotlib/timeseries.py#L231
TYP: Create DeepChainMap from ChainMap.new_child in scope.py
State: closed

xref #37715

scope is created as a DeepChainMap but on lines 139, 145, 262 call to new_child was returning ChainMap. Created a new DeepChainMap from the resulting ChainMap.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Hello @kashifkhan! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-12-11 15:17:11 UTC
@kashifkhan seems to be failing precommit; you can run this locally (and fix up)
CI should be fixed if you merge master
> CI should be fixed if you merge master

Sure Ill take care of this tomorrow !!
thanks @kashifkhan
Fix mypy error in testing
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

Fixup to get ci green again

Labels:
name: CI
description: Continuous Integration
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
great merge in green
can i suggest the approach taken in #44744?  (also fixes a FutureWarning that is failing the npdev build)
i think closed by #44744
yeah i think fixed on master already, thanks @phofl 
No worries
CLN: memory-mapping code
State: closed

Rebased on #44761

No need for `codecs.getincrementaldecoder` as `io.TextWrapperIO` will do that (and we can use `io.TextWrapperIO` because mmap is wrapped inside `_IOWrapper`). `io.TextWrapperIO` also provides `__next__` for us :)

Probably will need some benchmarking with utf-8/non-utf8 files.

Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: Clean

Comments:
Hello @twoertwein! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-12-05 23:25:04 UTC
~~This PR should speed up all `read_csv(..., engine="c", encoding="utf-8")` calls when a binary handle is used (this should now be faster than using text handles), by applying the same shortcut as done for `memory_map` in #43787.~~

Using TextIOWrapper seems to be slower than the previous solution :( Will revert most of the changes in this PR.
BUG: Raise TypeError for scalar indexer in take
State: closed

- [x] closes #42875
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Could you rebase master? We had a few ci failures
Thanks @gfkang
BUG: MultiIndex union setting wrong sortorder parameter
State: closed

- [x] closes #44752
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
thanks @phofl 
thanks very much!
ENH: support SpooledTemporaryFile
State: closed

- [x] closes #44748
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
A trimmed down version of this PR could go to 1.3.* as it technically fixes a regression. Let me know whether it should be back-ported or not, based on that I will create a whatsnew for either 1.3 or 1.4.
let's just put in 1.4, trying to avoid backporting anything unless its critical
Thank you for rebasing @jreback, green except for (unrelated?) codecov error.
thanks @twoertwein this looks great (yeah a bit longer but way more readable). waiting to merge as i hope some CI patches get us back to green and then can rebase here.
are the windows checks related?
It should be unrelated (the test case doesn't seem to open any files). Will rebase and force push to test it again. Cannot reproduce it on linux.
The error disappeared but I can imagine that it might be related: if mmap fails, opened handles will not be released.

edit: not from this PR but as a result from https://github.com/pandas-dev/pandas/pull/44777
thanks @twoertwein
PERF: unstack
State: closed

```
from asv_bench.benchmarks.reshape import *
self = Unstack()
self.setup("category")

%timeit self.time_without_last_row("category")
77.3 ms ± 1.97 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  # <- master
21.2 ms ± 788 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  # <- PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
nice!
DOC: Uniform block directive syntax.
State: closed

block directive syntax technically does not have space before the
double-colon.

While this seem to be accepted by docutils – but not documented,
this can throw off other RST parser/syntax highlighter.
Sometime they can see that as comments, sometime they give garbage out.

Plus this is really uncommon with 3 occurrences in pandas codebase with
space and 4800 without:

    pandas[master]  $ rg '\.\. \w+::' | wc -l
    4853

Numpy and scipy used to have a few space in their block directive, but
current master/main branch of both should also be free of directive with
space before the double colon.

---

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Note that this issue can be seen in the syntax highlighting of the diff of this PR itself, where the deleted lines are gray (comment), and added ones are blue/purple (directives)



Labels:
name: Docs
name: Code Style
description: Code style, linting, code_checks

Comments:
cc @MarcoGorelli 
> could u add as a doc check rule as well (can be a follow up)

Yeah, I'm unsure how to currently check for that. I detected it automatically but the project that made me discover that is way too far from prime time to be added as a check. I could just grep for it and check the status code, but I'm afraid it would be a bit brittle.
Could you add it as a pygrep check to the precommit config file? 
Done. 
lgtm. cc @MarcoGorelli 
thanks @Carreau
CI/DOC: warning in docbuild
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
merge in green
```
----> 1 pd.read_csv(io.StringIO(data), true_values=["Yes"], false_values=["No"])

TypeError: initial_value must be str or None, not DataFrame
```

Maybe we need to freeze the old example so it doesnt get evaluated?  i dont know how that works
use a ioython block and paste the example in
Thx for fixing
@jreback green but im really not confident when it comes to code-block stuff
don't need the okwarning but ok
ENH: implement tm.shares_memory
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

xref #44152

Dedicated implementation based on https://github.com/pandas-dev/pandas/pull/43930/files#diff-6676eb1e10942179c11b3ef5a8a59d835a05873934580a52139117ebd1f2d7c3R236


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Meant to implement tests before merging; will do so in follow-up(s)
I think this is causing mypy to complain on master:

```
pandas/_testing/__init__.py:1086: error: "ExtensionArray" has no attribute "_data"  [attr-defined]
pandas/_testing/__init__.py:1087: error: "ExtensionArray" has no attribute "_data"  [attr-defined]
```
ENH: IntervalArray.min/max
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Enhancement
name: Interval
description: Interval data type

Comments:
looks fine, can you merge master and ping on green
rebased + green-ish (same lint complaints as on master)
CLN/TST: skips -> xfails
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean

Comments:
@jreback gentle ping; gets us to green
TST: stricten xfails
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
if you can rebase
rebased + greenish (including fix for mypy complaints on master)
PERF: reducing dtype checking overhead in groupby
State: closed

This is not to be merged as is, but rather to illustrate how the generic dtype checking methods we have cause some overhead in certain operations (or at least in the benchmark cases we have), and how this is also relatively easy to solve.

Using the `GroupManyLabels.time_sum` case (a wide dataframe):

```python
ncols = 1000
N = 1000
data = np.random.randn(N, ncols)
labels = np.random.randint(0, 100, size=N)
df = pd.DataFrame(data)
df_am = df._as_manager('array').copy()

In [3]: %timeit df_am.groupby(labels).sum()
38.2 ms ± 861 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  # <-- master
26.6 ms ± 486 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  # <-- PR
```

So a combination of 1) more specialized dtype checks and 2) caching some repeated dtype+op-dependent helpers quickly gives a decent speedup (and there is some more room for improvement with similar changes). 

So at some point I think we should look a bit more into our `is_..` checks and have eg specialized versions that eg assume you already have a dtype (I know we discussed this before, not directly finding a relevant issue), and use those throughout the codebase where we know we have a dtype object (might be a good issue for someone starting to dive into the code base).

Labels:
name: Performance
description: Memory or execution speed performance
name: Stale

Comments:
yeah looks nice, i agree should try to refactor is_* to make them more strict is a good idea
The dtype.kind checks i definitely like.  How big a difference does the caching make?
prob could work if you can rebase @jorisvandenbossche 
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
this would be nice to do
> this would be nice to do

closing as stale. but feel free to reopen when ready.
REF: share _reduce method
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
lgtm ex @jorisvandenbossche comment
thanks
[ArrayManager] Array version of interpolate logic
State: closed

Similar to what I did in https://github.com/pandas-dev/pandas/pull/41104 for `fillna` (xref https://github.com/pandas-dev/pandas/issues/39146).

This also improves the performance for some of the fillna/interpolate benchmarks (mostly due to avoiding the overhead of creating a block for each column). For example (from ` frame_methods.Fillna.time_frame_fillna(True, 'bfill', 'datetime64[ns]')`:

```python
N, M = 10000, 100
dtype = "datetime64[ns]"
data = pd.date_range("2011-01-01", freq="H", periods=N)
df = pd.DataFrame({f"col_{i}": data for i in range(M)})
df[::2] = None
df_am = df._as_manager("array").copy()

In [2]: %timeit df_am.fillna(inplace=True, method="bfill")
7.31 ms ± 313 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <-- master
3.57 ms ± 370 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <-- PR
```



Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Stale

Comments:
Can any of this be shared with the Block method?
> Can any of this be shared with the Block method?

I think the actual interpolate function is already shared (i.e. `interpolate_array_2d`). The rest is mostly some boilerplate that is slightly different (eg the Block method has an additional check to split the block). I think the `try/except` for `clean_fill_method` could be shared, but I am not sure it is worth creating another function for those few lines.
I increasingly think that this type of change should be left until right before BlockManager is ripped out. Until then, this just duplicates code and makes it easy for behavior to accidentally drift apart.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
DOC: Improve code example for DataFrame.join
State: closed

The modified code example does not have unique values in the 'key' column in the calling DataFrame `df`. With unique keys the special behavior of `join` has not been highlighted, since it just added a new column. With the modified values the example shows that the 'key' column is really used as key.

Labels:
name: Docs
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
OK, should I add it for all calls or just with the last and maybe most interesting call, i. e. `df.join(other.set_index('key'), on='key')`?
I think the last one is enough
Hello @ChristofKaufmann! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-12-04 00:58:08 UTC
thanks @ChristofKaufmann 
Thank you!
CI: Run sdist build less often
State: closed

- [ ] closes #43745
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Need to test all possible combos of 
- Run on merged commits [tested]
- Run when labeled [tested]
- Skip when no "Build" label [tested]
- Run when commit pushed to labeled PR [tested]
- Run when labeled PR closed/reopened [tested]

Labels:
name: Build
description: Library building on various platforms
name: CI
description: Continuous Integration

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@lithomas1 is this actionable?
thanks @lithomas1
TST: remove non-strict xfails
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

If/when this gets to green, I'll want to re-run it a few times to make sure it stays that way.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
FWIW these tests shouldn't be running in the CI currently since we switched to a moto service. This might be only applicable for local runs?
if they're not being run on the CI then aren't they kind of pointless?
These tests should be useful for those who run the test suite locally in which case these tests are still useful.

In theory, if/once CI platforms allows moto service images for non-Linux OSs these tests should be run-able.
Makes sense, thanks.
PERF: slow tests
State: closed

- [x] closes #44624
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Biggest difference is in test_get_loc_tuple_monotonic_above_size_cutoff going from 12.92 seconds t 4.72, all in improving DatetimeLikeArray.astype.

Labels:

Comments:
Thanks!
TST: Fix local run on s3 tests
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

xref https://github.com/pandas-dev/pandas/pull/44612#issuecomment-984548056

cc @simonjayhawkins I ran this locally on some test and it was working. Let me know if this works for you as well


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @mroeschke 
> cc @simonjayhawkins I ran this locally on some test and it was working. Let me know if this works for you as well

All good now. Thanks @mroeschke
PERF: DataFrame.T
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Perf regression caused by #44593 (not yet released, so no whatsnew)

```
from asv_bench.benchmarks.reshape import *
cls = ReshapeExtensionDtype
dtype = cls.params[0]
self = cls()
self.setup(dtype)

%timeit self.df.T
91.1 ms ± 1.75 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  # <- master
163 µs ± 3.8 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <- PR

```

Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
thanks @jbrockmendel
Backport PR #44658 on branch 1.3.x (BUG: groupby.sum with timedelta64 and NaT)
State: closed

Backport PR #44658

Labels:
name: Groupby
name: Timedelta
description: Timedelta data type

Comments:
```
=========================== short test summary info ============================
FAILED pandas/tests/frame/methods/test_rename.py::TestRename::test_rename_inplace
= 1 failed, 169202 passed, 6163 skipped, 962 xfailed, 3 xpassed, 4 warnings in 2144.77s (0:35:44) =
```

is a known flaky test
Added the note of class DataFrame
State: closed

Added the note of class DataFrame, if the data type is torch.Tensor when creating DataFrame, it should be converted to Numpy Array for better performance.
More More detailed description in https://github.com/pandas-dev/pandas/issues/44616.
Originally, the judgment was added directly in the init method of DataFrame: whether data is torch.Tensor
if isinstance(data,torch.Tensor):
     data = data.numpy()
This can solve the time-consuming problem of DataFrame (tensor), but need to import torch, which may affect other uses of pandas, so it is more appropriate to add an explanation in the class comment. 

Labels:
name: Docs

Comments:
Hello @YeahNew! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-12-02 07:36:46 UTC
closing as won't fix. I don't find this a useful doc addition; discussed in the original issue.