BrandonFors/pandas-github-issues huggingface dataset:
CI: Add mambaforge to ci builds
State: closed


Labels:
name: CI
description: Continuous Integration

Comments:
Nice! Any sense of how much faster the environment building is?
The conda incubator takes around 3 minutes I think. With conda it takes around 4 in a best case scenario. Unfortunately we have a lot of builds where this takes more than 15, while mamba should stay at around 3 for them. It will keep our build times consistent.

Unfortunately we have 2 failing pipelines which do seem relevant, will check what this is about
Also add here https://github.com/pandas-dev/pandas/blob/master/.github/workflows/sdist.yml#L54 and here https://github.com/pandas-dev/pandas/blob/master/.github/workflows/asv-bot.yml#L44?
This is not finished yet. I will have to figure out how to fix the installation issues before adding this somewhere else. 
FYI, on my machine (macOS) those tests run fine when using the script. 
This is not a test issue per se. Mamba resolves the dependencies differently, which results in different versions. The 3.9 build installs arrow 2.0.1 instead of 6.0.0, causing the tests to fail. I think I found a solution for this, but have to look a bit further
What I meant to say is that for me Mamba seems to resolve correctly/in a way that doesn't break the tests.
With the 3.9 dependency file? Could you post the output of conda info?
@phofl see #44979, seems to work ok
Yeah, would want to avoid the pin, but we'll see. Will finish this tomorrow after the setuptools pin is merged
This is ready for review.

We have a few builds where creating the environment regularly takes more than one hour. Mamba does speed up the process a bit in general, but more important: it does not take forever in some instances.

The 3.9 envs are solved a bit weird. Hence I added the pyarrow pin, which avoids this
do we still have pyarrow 1.* testing?
Yep, for example in Database / Linux_py38_IO (ci/deps/actions-38-db-min.yaml) 
thanks @phofl
Bug: to_csv coercing datetime columns to same format
State: closed

- [x] closes #21734
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
@phofl the web & docs build failed but hard to tell why, can you rebase
The web and docs failure was unrelated. An inventory was not available. greenish
CLN: tests/window/moments
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

Reduces a lot of no-op parameterizations by creating more specific fixtures


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Window
description: rolling, ewma, expanding

Comments:
One comment, otherwise LGTM
BTW just curious, is there a gameplan for the xfails in test_online and test_expanding?
> BTW just curious, is there a gameplan for the xfails in test_online and test_expanding?

Hmm those xfails are just not implemented features with associated issues I think. I don't have a timeline for tackling those yet.
Test added to check if MultiIndex is unique
State: closed

- [X] closes #44786
-  1 tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- New test added to check if MultiIndex is unique


Labels:
name: MultiIndex

Comments:
closing in favor of #45822
DOC: update documentation of type annotation (PEP 585)
State: closed


Labels:
name: Docs
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
pre commit failing
Without the CSS:
![image](https://user-images.githubusercontent.com/6618166/146617012-a67d98de-2419-4608-ba7c-e36d608e5ebb.png)
BUG: iterating on a subset of columns in a GroupBy object (#44821)
State: closed

Fixes issue #44821.

When trying to iterate on a subset of columns in a GroupBy object,
it returned all columns, instead of the selected subset.

GroupBy.__iter__ used self.obj instead of self._selected_obj (see
PR #6570).

- [x] closes #44821
- [x] 1 test added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Groupby

Comments:
Hi.

Only one test is broken, which I mentioned here: https://github.com/pandas-dev/pandas/issues/44821#issuecomment-997037279

As mentioned there, I'm not sure how to continue, as users may rely on current behavior.

By the way, should I could continue the discussion in this PR?

Well, I decided to fix that test. While this PR changes `GroupBy.__iter__`, it didn't actually broke any test that tested its behavior. The test that was previously failing concerned the behaviour of `GroupBy.rolling`, and the fix was in the "manual computation" to compare against.

So, although this PR might break somebody's code, this `GroupBy` behavior wasn't "promised" anywhere, and it could have been changed as part of #6570.
cc @rhshadrach for comments
thanks @maurosilber
BUG: groupby agg raising when column was selected twice
State: closed

- [x] closes #44924
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Not exactly sure where to put the test


Labels:
name: Groupby
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
> Not exactly sure where to put the test

is fine where you have it (that said that file is getting kind of long)
thanks @phofl
PERF: Performance improvement in to_csv with unused levels in multiindex
State: closed

- [x] closes #37484
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

```
before 1.97 s ± 102 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
after 39.6 ms ± 3.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

Labels:
name: Performance
description: Memory or execution speed performance
name: IO CSV
description: read_csv, to_csv

Comments:
wow!
ASV: Add additional to_csv benchmark
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

This is the only one which makes sens and is not already there


Labels:
name: IO CSV
description: read_csv, to_csv
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
thanks!
BUG: nullable dtypes not preserved in Series.replace
State: closed

- [x] closes #33484
- [x] closes #40732
- [x] closes #31644
- [x] closes #41215
- [x] closes #25438
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

Sits on top of #44932

test_replace_dtype is cribbed from #37512

Improves inconsistent Categorical behavior while removing a special-case kludge for Categorical in replace_list.

Fixes an inconsistent int->float behavior introduced (intentionally, the inconsistency wasn't noticed) in #15742

Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
lgtm. let's get a few more eyes on this.

cc @pandas-dev/pandas-core  if any comments
updated with tests for some more closed issues
thanks followup comments
TST: Add any_string_dtype parameterization to test
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

xref https://github.com/pandas-dev/pandas/pull/44876#discussion_r770374539

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @mroeschke
Fix faulty logic with apply_empty_result()
State: closed

Corresponds to issue 44090 (https://github.com/pandas-dev/pandas/issues/44090)

- [x] closes #44090
- [] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

There was some illogical code flow in this method, and it was adjusted accordingly. The progression should be the same and no adjustments were made to the actual underpinnings of the method.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Apply
description: Apply, Aggregate, Transform, Map
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears that this broke existing tests and we would need a pull request that also includes unit tests. If you could address the comments and merge the main branch, we can reopen. Closing.
TYP: Upgrade to mypy 0.920
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

new mypy was released


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Converting to draft till mypy is on conda-forge
> Converting to draft till mypy is on conda-forge

seems to be available: https://anaconda.org/conda-forge/mypy
yep, ci is green, was failing before
one more rebase if you don't mind (as just merged your typing PR)
rebased and greenish @jreback
thanks @phofl
BUG: to_csv casting datetimes in categorical to int
State: closed

- [x] closes #40754
- [x] closes #18024
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: IO CSV
description: read_csv, to_csv
name: Categorical
description: Categorical Data Type

Comments:
i think the underlying problem is in `Categorical.astype(object)`
Was not sure if this was intentional in the astype call. Will have a look again
So I think I figured the issue out now.

``Categorical.astype`` was incorrect for object dtype as you suggested. But this does not directly help us in the ``to_csv`` case, becaause this has to respect a ``date_format`` in ``to_native_types``. We have to convert this to a DatetimeArray before doing the actual conversion. 
thanks @phofl
Added dtype test concat function
State: closed

- [ ] closes #32934 
- [X] tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Stale

Comments:
Hello @GustavoGB! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-12-16 19:37:58 UTC
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in main, address the comments and we can reopen
ENH: Implemented a check to test if filepath_or_buffer is a valid JSON string or a valid filepath and raises an error in the case that it is neither.
State: closed

- [x] closes #29104 thread.
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

ENH: Implemented a check to test if filepath_or_buffer is a valid JSON string or a valid filepath and raises an error in the case that it is neither. Per the thread in #29102 and related thread in #29104, there should be a way to differentiate between a json string passed as input and a file_path. We cannot just say that since a string is not a valid path, raise a FileNotFoundException. So this PR attempts to check intent.

Labels:

Comments:
Hello @cheungje! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/io/json/_json.py`](https://github.com/pandas-dev/pandas/blob/56c338ee4ddc43584c34e5a901ec32e0986ada6c/pandas/io/json/_json.py):

> [Line 693:1](https://github.com/pandas-dev/pandas/blob/56c338ee4ddc43584c34e5a901ec32e0986ada6c/pandas/io/json/_json.py#L693): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace
> [Line 730:89](https://github.com/pandas-dev/pandas/blob/56c338ee4ddc43584c34e5a901ec32e0986ada6c/pandas/io/json/_json.py#L730): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (89 > 88 characters)
> [Line 739:89](https://github.com/pandas-dev/pandas/blob/56c338ee4ddc43584c34e5a901ec32e0986ada6c/pandas/io/json/_json.py#L739): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (96 > 88 characters)




Please open only one pr. You can easily push this to your other pr
TST: Added tests to check if file exists for read_json.
State: closed

- [x] closes 29102 and the thread in #29104. 
- [x ] tests added / passed
- [x ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Per the thread in #29104, there should be a differentiation between a json string passed as input and a filepath (from filepath_or_buffer variable passed into _get_data_from_filepath function). If it is not a valid string, then we cannot just raise a FileNotFound Exception error. Therefore, the variable must be checked to see if it is a valid JSON string, if it is a valid file path, and suggest a course of action or raise an appropriate error if it is a faulty string. 


Labels:

Comments:
Please open only one pr per topic
WIP: Debug stalling azure windows/macos builds
State: closed

xref https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=70734&view=logs&jobId=054c7612-f473-5cf0-0a65-77b54574553a&j=054c7612-f473-5cf0-0a65-77b54574553a&t=7218990d-7c28-52cc-5797-2fee2b0645b4


Labels:

Comments:
The builtin pytest faulthandler doesn't seem to be compatable with xdist due to https://github.com/pytest-dev/pytest-xdist/issues/205
I am not sure what else to try. Closing
DEPR: ‘names’ param in Index.copy
State: closed

Deprecates this parameter. `names` is not used in `Index.__init__`, so no reason to be used in `Index.copy`.

Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Index
description: Related to the Index class or subclasses

Comments:
@jbrockmendel ok here?
No objections
thanks @topper-123 
@topper-123 shouldnt this be done in RangeIndex.copy too? should `name` be deprecated in MultiIndex.copy?
String whitespace
State: closed

Added a function that would remove whitespaces

Labels:

Comments:
Hello @minnaau! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/tests/strings/test_split_partition.py`](https://github.com/pandas-dev/pandas/blob/d429458c61a1e82392549fff1a14bf003dae2250/pandas/tests/strings/test_split_partition.py):

> [Line 111:1](https://github.com/pandas-dev/pandas/blob/d429458c61a1e82392549fff1a14bf003dae2250/pandas/tests/strings/test_split_partition.py#L111): [E302](https://duckduckgo.com/?q=pep8%20E302) expected 2 blank lines, found 1
> [Line 126:1](https://github.com/pandas-dev/pandas/blob/d429458c61a1e82392549fff1a14bf003dae2250/pandas/tests/strings/test_split_partition.py#L126): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace
> [Line 131:1](https://github.com/pandas-dev/pandas/blob/d429458c61a1e82392549fff1a14bf003dae2250/pandas/tests/strings/test_split_partition.py#L131): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace
> [Line 132:25](https://github.com/pandas-dev/pandas/blob/d429458c61a1e82392549fff1a14bf003dae2250/pandas/tests/strings/test_split_partition.py#L132): [W291](https://duckduckgo.com/?q=pep8%20W291) trailing whitespace
> [Line 136:1](https://github.com/pandas-dev/pandas/blob/d429458c61a1e82392549fff1a14bf003dae2250/pandas/tests/strings/test_split_partition.py#L136): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace
> [Line 141:1](https://github.com/pandas-dev/pandas/blob/d429458c61a1e82392549fff1a14bf003dae2250/pandas/tests/strings/test_split_partition.py#L141): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace
> [Line 146:1](https://github.com/pandas-dev/pandas/blob/d429458c61a1e82392549fff1a14bf003dae2250/pandas/tests/strings/test_split_partition.py#L146): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace


Please open only on pr for the same change
TST: Add test to Series.str.strip()
State: closed

- [x] closes #44867
- [x] tests added
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:

Comments:
Hello @minnaau! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/tests/strings/test_split_partition.py`](https://github.com/pandas-dev/pandas/blob/101de9f476fbe7743736d43558cf52d33c2180c2/pandas/tests/strings/test_split_partition.py):

> [Line 111:1](https://github.com/pandas-dev/pandas/blob/101de9f476fbe7743736d43558cf52d33c2180c2/pandas/tests/strings/test_split_partition.py#L111): [E302](https://duckduckgo.com/?q=pep8%20E302) expected 2 blank lines, found 1
> [Line 126:1](https://github.com/pandas-dev/pandas/blob/101de9f476fbe7743736d43558cf52d33c2180c2/pandas/tests/strings/test_split_partition.py#L126): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace
> [Line 131:1](https://github.com/pandas-dev/pandas/blob/101de9f476fbe7743736d43558cf52d33c2180c2/pandas/tests/strings/test_split_partition.py#L131): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace
> [Line 136:1](https://github.com/pandas-dev/pandas/blob/101de9f476fbe7743736d43558cf52d33c2180c2/pandas/tests/strings/test_split_partition.py#L136): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace
> [Line 141:1](https://github.com/pandas-dev/pandas/blob/101de9f476fbe7743736d43558cf52d33c2180c2/pandas/tests/strings/test_split_partition.py#L141): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace
> [Line 146:1](https://github.com/pandas-dev/pandas/blob/101de9f476fbe7743736d43558cf52d33c2180c2/pandas/tests/strings/test_split_partition.py#L146): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace


Thanks @minnaau for the PR.

IIRC we have plenty of tests for the string methods on mixed object Series.

In the issue there is a suggestion that the docs could be updated to clarify the behavior.

So closing this PR as there is another PR open #44885 that is updating the docs and IMO we don't really need any additional tests.
TST: Added tests to check if file exists for read_json.
State: closed

- [x] closes #29102 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

#29102 already added code to fix the original issue of fixing misleading error messages when trying to open a nonexistent json file. This commit adds test cases. 

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO JSON
description: read_json, to_json, json_normalize
name: Stale

Comments:
@cheungje if you'd merge master and address comments
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
closing as stale
PERF: Improve performance in to_csv with date format for index
State: closed

- [x] closes #39413
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

Before:
```
%timeit df.to_csv(date_format='%Y-%m-%d %H:%M:%S')
1.05 s ± 11.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
afterwards:
```
%timeit df.to_csv(date_format='%Y-%m-%d %H:%M:%S')
392 ms ± 4.31 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

Labels:
name: Performance
description: Memory or execution speed performance
name: IO CSV
description: read_csv, to_csv

Comments:
thanks @phofl wouldn't object to addtitional benchmarks as specified in the OP
We have some of them already, will check and add as necessary
BUG: np.divmod with Sparse
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Bug
name: Sparse
description: Sparse Data Type

Comments:
thanks, if this is user facing can you add a note in a followup
Bug in concat casting all na levels to float
State: closed

- [x] closes #44900
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
thanks @phofl
BUG: read_csv converting nans to 1 when casting bools to float
State: closed

- [x] closes #42808
- [x] closes #34120
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

This is mostly based on #42944, but would like to get this into 1.4

cc @joelgibson if you would like to continue your initial pr

The xfail pyarrow case is not really related to the pyarrow engine, but to df.astype("bool") casting NaN to False

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO CSV
description: read_csv, to_csv

Comments:
cc @gfyoung if any comments
Sorry about dropping out on this, got swamped by work. Thanks for bringing it forward @phofl.
thanks @phofl