BrandonFors/pandas-github-issues huggingface dataset:
DOC : DataFrame.to_parquet doesn't round-trip pyarrow StringDtype #42664
State: closed

- [ ] closes #42664 

Added a small note, both in _user_guide/text.rst_ and _user_guide/io.rst_ and example.


Labels:
name: Docs
name: Strings
description: String extension data type and string data

Comments:
This is my first contribution, please guide me for the required changes that are necessary, would be grateful.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Can you fix-up the pre-commit checks and merge master? Thanks.
> Can you fix-up the pre-commit checks and merge master? Thanks.

Yes I will try to do, Thanks

Thanks for the pull request, but appears this PR has gotten stale. If you'd like to continue please merge master, address the pre-commit failures and we'd be happy to reopen.
Date time index.indexer between time
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Please take a look and let me know if things look fine, I'll make a whatsnew entry then :)

This PR has been made as per [this](https://github.com/pandas-dev/pandas/pull/43248#discussion_r701003907) suggestion by @attack68.

Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Stale

Comments:
can you merge master and address comments
I'm sorry for the absence! I'd been away due to some work.

@jreback I did leave a [comment in my OLD PR ](https://github.com/pandas-dev/pandas/pull/43248/) addressing why the tests are failing in THIS PR. Can you please take a look at it? I'm not sure what my course of action should be going ahead.
> I'm sorry for the absence! I'd been away due to some work.
> 
> @jreback I did leave a [comment in my OLD PR ](https://github.com/pandas-dev/pandas/pull/43248/) addressing why the tests are failing in THIS PR. Can you please take a look at it? I'm not sure what my course of action should be going ahead.
you should be able to run this locally and reproduce
Yes, indeed I can reproduce the failing test case locally. 

However, what I'm confused about is the absence of a code block from `pandas/core/generic.py` in the `master` branch. 

Can you please let me know why I can see the below code block in [this PR](https://github.com/pandas-dev/pandas/pull/43248/files#diff-1a2e3df0db7dd8bddc2ec4bff9de8a7a55e328e6c32e2cecde761dc9549fcd46), but NOT on the [master branch](https://github.com/pandas-dev/pandas/blob/master/pandas/core/generic.py).?

```
            elif inclusive not in ["both", "neither", "left", "right"]:
                raise ValueError(
                    f"Inclusive has to be either string of 'both', "
                    f"'left', 'right', or 'neither'. Got {inclusive}."
                )
```

Edit: I'm not sure if this is a bad question but can you please address this @jreback
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Yes I am willing.

> On Nov 5, 2021, at 1:04 AM, github-actions[bot] ***@***.***> wrote:
> 
> ﻿
> This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.
> Triage notifications on the go with GitHub Mobile for iOS or Android. 

@suyashgupta01 can you merge master and update to comments
Thanks for the PR, but it appears to have gone stale. If you could merge master and address the reviews we would be happy to reopen. Closing.
Fix mistake in cut() docstring
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
thanks @benibienz
BUG: .cat changing dtype inplace
State: closed

- [x] closes #43232
- [x] closes #43334
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Categorical
description: Categorical Data Type

Comments:
@meeseeksdev backport 1.3.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.3.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 26064f0f7ea01607a85b399ee4f3de29a864f338
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #43597: BUG: .cat changing dtype inplace'
```

4. Push to a named branch :

```
git push YOURFORK 1.3.x:auto-backport-of-pr-43597-on-1.3.x
```

5. Create a PR against branch 1.3.x, I would have named this PR:

> "Backport PR #43597 on branch 1.3.x (BUG: .cat changing dtype inplace)"

And apply the correct labels and milestones.

Congratulation you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove `Still Needs Manual Backport` label once the PR gets merged.

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
BUG: concat fails if indexes are all the same and keys are not unique
State: closed

- [x] closes #43595
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
https://github.com/pandas-dev/pandas/pull/43596/checks?check_run_id=3617031474 showing a warning
thanks @rhshadrach
TST: Adding unit tests for named indexes resulting from empty datafra…
State: closed

…me shifts

- [x] closes #41264
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Tests added for issue: https://github.com/pandas-dev/pandas/issues/41264

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Index
description: Related to the Index class or subclasses

Comments:
Hello @IamJasonBian! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-10-31 04:18:30 UTC
@jreback 
* merged with test_groupby_shift_diff
* removed third test as issue did not exist in 1.3.0.dev0+1510.gfea6799dba
@jreback, ok to push? Not sure why pre-commit black is failing and PEP8 linter passed locally. 
no if precommit is failing then there is a problem
@jreback pinging on green
@IamJasonBian could you merge master one more time? Will merge after all green again
thanks @IamJasonBian
REF: pre-compute JoinUnit.needs_filling
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
can you rebase
PERF: MultiIndex.equals
State: closed

- [x] closes #43549
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

```
from dateutil import tz
import pandas as pd
from time import time

dates = pd.date_range('2010-01-01', periods=1000, tz=tz.tzutc())
index = pd.MultiIndex.from_product([range(100), dates])
index2 = index.copy()

%timeit index.equals(index2)
330 ms ± 6.44 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  # <- master
1.79 ms ± 7.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <- PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
prob worth backporting this, can you add a whatsnew note @jbrockmendel ping on green
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
PERF: GroupBy.apply
State: closed

```
from asv_bench.benchmarks.groupby import *
self = Apply()
self.setup(4)

%timeit self.time_copy_function_multi_col(4)
451 ms ± 2.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  # <- PR
559 ms ± 1.35 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  # <- master
```

Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
There's an issue #38358 about the commit that added this `filter` causing a perf regression, would this fix that?
I think so yes.
CLN: typos
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Clean

Comments:
> ValueError: 'unique' is not a valid function name for transform(name) (in asvs)
REF: Move rest of aggregation into apply
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

Purely a move. I have a PR just about ready to start tackling #41112, but it needs to touch some of the renamer code. It'd be nice to have it all in `core.apply`, and logically makes sense to go there.


Labels:
name: Refactor
description: Internal refactoring of code
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
thanks @rhshadrach
Backport PR #43553 on branch 1.3.x (REGR: Outer merge failing with integer and NaN keys)
State: closed

Backport PR #43553: REGR: Outer merge failing with integer and NaN keys

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
> intersphinx inventory 'https://www.statsmodels.org/devel/objects.inv' not fetchable due to <class 'requests.exceptions.ConnectionError'>: HTTPSConnectionPool(host='www.statsmodels.org', port=443): Max retries exceeded with url: /devel/objects.inv (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f7e2df00e50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'


showing up again but ignoring.
TST: Add test for masking timedelta (#39548)
State: closed

- [x] closes #39548
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Timedelta
description: Timedelta data type

Comments:
thanks @el-g-1
REF: Share Index comparison and arithmetic methods
State: closed

- [x] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Index
description: Related to the Index class or subclasses

Comments:
wow, very nice!
REGR: Outer merge failing with integer and NaN keys
State: closed

- [x] closes #43550 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
thx @debnathshoham
ENH: `Styler.format_index` split PR base components
State: closed

Splits #43101 to debug failing tests

All this PR does is add a method and fix failing tests because of a minor change to jinja2 templates.

Labels:

Comments:
This PR suffers the same undiagnosable fault as the mentioned PR.. 
If anyone fancies having a crack at solving the mysterious failure, please shout out.
try renaming this to format_index2 as a test and see if it works
> try renaming this to format_index2 as a test and see if it works

so that worked? whats the scoop?
> > try renaming this to format_index2 as a test and see if it works
> 
> so that worked? whats the scoop?

wow. so was thinking that `.format_index` shadows something, but we don't have a method by that name so that's obviously bunk, but maybe we somehow autogenerate something. 
> > > try renaming this to format_index2 as a test and see if it works
> > 
> > 
> > so that worked? whats the scoop?
> 
> wow. so was thinking that `.format_index` shadows something, but we don't have a method by that name so that's obviously bunk, but maybe we somehow autogenerate something.

so you wanna keep the method named like this or something else. i've not faced issue like this before so bit clueless.
must be some sphinx thing 

ideally let's keep format_index though could call format_axis
OK well this PR now uses `format_axis` and the original PR still has `format_index`, which I prefer since it complements `apply_index` and `applymap_index`, downside being it breaks the CI and I can't fix it! So you can choose either! (I'll fix up the tests and add the doc examples if you go with this one)
yeah might try to google this
i suspect we can set some sphinx option in conf.py to avoid this
Refactor/validate utils
State: closed

- [x] refactoring for #40245
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
thanks @zyc09 very nice!
CI Use pytest randomly
State: closed

- [ ] closes #43534
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:

Comments:
xref #34373
right, thanks - I'll close the new issue in favour of that then (this PR doesn't solve anything, it was just a test to see if pytest-randomly did indeed introduce failures)
CI: network tests running even with "not network"
State: closed

While looking into windows timeouts, noticed network tests were still being run with -m "not network". `tm.network` adds the keyword `network`, but does not add a mark, so -m "not network" was still running network tests.  While a mark could probably be set somehow in that decorator, I couldn't find a `pytest` documented way to do that.

Instead, this pushes `pytest_runtest_setup` into `pytest_collection_modifyitems` (which I think is more intuitive since we know if a test should be skipped or not at collection time, and shouldn't need these skip checks at setup-time for each test).

The other reason for this change was that `pytest_runtest_setup` is too late in the pipeline to add the network marker. By moving the skip logic to `pytest_collection_modifyitems`, we can consolidate the skip checks with adding the corresponding marker if the "network" keyword is found.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
can you confirm that this doesn't change the number of tests run (e.g. compare a couple of builds which uses these keywords vs master)
Some different builds and results in the details block below. These look as expected I think - the patterns with "not network" run slightly fewer depending on the subset of tests included since "not network" is now respected.

The one with the "slow" pattern (the only one not specifying "not network") has an exactly matching number of tests

<details>

```
macOS py38_macos_1
Cmd: pytest -m "not slow and not network
69160 passed, 4425 skipped, 646 xfailed  # pr
69161 passed, 4425 skipped, 645 xfailed  # master
```

```
Windows py38_np18_2
Cmd: pytest -m "not slow and not network
53929 passed, 2511 skipped, 400 xfailed, 8 xpassed  # pr
53966 passed, 2523 skipped, 399 xfailed, 8 xpassed  # master
```

```
Linux_py38_IO
Cmd: pytest -m "((not slow and not network and not clipboard) or (single and db))
9808 passed, 1313 skipped, 214 xfailed  # pr
9845 passed, 1321 skipped, 214 xfailed  # master
```

```
pytest (actions-38-minimum_versions
Cmd: -m "not slow and not network and not clipboard"
170922 passed, 7056 skipped, 1047 xfailed, 5 xpassed  # pr
170948 passed, 7079 skipped, 1046 xfailed, 5 xpassed  # master
```

```
actions-39-slow.yaml
Cmd: -m "slow"
2436 passed, 11 skipped, 6 xfailed, 5 xpassed  # pr
2436 passed, 11 skipped, 6 xfailed, 5 xpassed  # master
```

</details>
thanks @mzeitlin11
DOC: add plt.show so that the plot will show, along with note about p…
State: closed

…lt.show, plt.savefig and Jupyter Notebook.

- [x] closes #43560
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs
name: Visualization
description: plotting

Comments:
![image](https://user-images.githubusercontent.com/992869/133195394-bfa2396f-d643-4406-95e5-82c6bbc7437a.png)
have added issue #43560 for this, and referenced this PR
@mzeitlin11 over to you
CI: mark user-agent tests as network
State: closed

cc @jbrockmendel, inspired by https://github.com/pandas-dev/pandas/pull/43517#issuecomment-917464806. This being the source of the timeout is also supported by starting to see timeouts on windows2 after changing the split such that the io tests were included in windows2.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
Hmm, actually these tests don't touch network (only use localhost) and timeout still happened, so must not be the cause
Good thought!  Are you confident that the timeout(s?) are from these tests?
> Good thought! Are you confident that the timeout(s?) are from these tests?

Now I am confident they are _not_ from the user-agent tests since this pr skipped them, but a timeout still occurred :) But seems likely the issue is somewhere in tests/io based on the timeouts moving to windows2 after the new split. Hoping #43545 helps
BUG/PERF: sparse min/max don't densify
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

This now follows  the structure of other `SparseArray` reductions like `sum` more closely, though there is some added complexity here to avoid taking the `min/max` of an empty array, which would raise when we want to return the corresponding missing value instead.

Benchmarks:
```
       before           after         ratio
     [5deec13d]       [f3eea313]
     <master>         <perf_sparse_min_max>
-      4.05±0.3ms       11.6±0.3μs     0.00  sparse.MinMax.time_min_max('max', 0.0)
-      4.49±0.4ms       11.6±0.2μs     0.00  sparse.MinMax.time_min_max('min', 0.0)
-      4.62±0.7ms       10.6±0.2μs     0.00  sparse.MinMax.time_min_max('max', nan)
-      5.37±0.7ms       10.3±0.3μs     0.00  sparse.MinMax.time_min_max('min', nan)
```

Labels:
name: Bug
name: Performance
description: Memory or execution speed performance
name: Sparse
description: Sparse Data Type
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
thanks @mzeitlin11
DEPR: index argument to SparseArray
State: closed

- [x] closes #23089
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Sparse
description: Sparse Data Type
name: Deprecate
description: Functionality to remove in pandas
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
thanks !
PERF: groupby.pad/backfill follow-up
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Follow-ups include the big perf bump discussed in #43478

```
import pandas as pd
import numpy as np

np.random.seed(23446365)
arr = np.random.randn(10**5, 10)
mask = arr < -1
arr[mask] = np.nan

df = pd.DataFrame(arr)

gb = df.groupby(df.index % 7)

%timeit res = gb.pad()
29.8 ms ± 193 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  # <- master
14.6 ms ± 2.41 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <- PR
```

Labels:
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Performance
description: Memory or execution speed performance

Comments:
prob should do some kind of whatsnew note for significant improvements like this (followon)
BUG: Groupby.agg with timegrouper, nat, and numba
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Resample
description: resample method
name: numba
description: numba-accelerated operations

Comments:
IIRC you did a similar bug fix quite recently (for non-numba?) can you add this pr number to that note.
> can you add this pr number to that note

done + green
TYP: sparse.pyi
State: closed

A start to typing sparse stuff, tried to do this without touching the cython code, hopefully some cleanups possible once typing somewhat specified.

cc @jbrockmendel, one annoying piece going on here is the abstract base class `SparseIndex` - it functions entirely like an interface, and so has nothing but abstract methods. Currently, in `cython` only `__init__` is defined for the base class, so the current typing with multiple methods in the base class is a bit of a white lie. We could just add all the methods to the base class and have them raise `NotImplementedError`, but not sure if that's just unnecessary code bloat since it works as is.

Labels:
name: Sparse
description: Sparse Data Type
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
LGTM; i think we've gone the white lie route elsewhere but not sure where off the top of my head
thanks @mzeitlin11