BrandonFors/pandas-github-issues huggingface dataset:
Bump s3fs to 2021.08.00
State: closed

- [x] closes #48271 (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


cc @mroeschke @lithomas1 

Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
Have to bump the others too
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.5.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 6559eda304d9a8b23e52043c6b540b0302d9ca20
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #48299: Bump s3fs to 2021.08.00'
```

4. Push to a named branch:

```
git push YOURFORK 1.5.x:auto-backport-of-pr-48299-on-1.5.x
```

5. Create a PR against branch 1.5.x, I would have named this PR:

> "Backport PR #48299 on branch 1.5.x (Bump s3fs to 2021.08.00)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
thanks @phofl 
@meeseeksdev backport 1.5.x
The indexes of DataFrame.describe(percentiles=[0.29, 0.57, 0.58]) are incorrect
State: closed

- [ ] closes #46362
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
Thanks @weikhor!
CI: Check enforcing min botocore due to PY310 six ImportWarning
State: closed

xref https://github.com/pandas-dev/pandas/pull/48291 https://github.com/pandas-dev/pandas/pull/48278

Checking if botocore's vendored six fixes allow environments to solve and not raise this `ImportWarning`


Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
Ah yeah, if you remove the aiobotocore pin then this causes failures
Got it. I think we should move forward with bumping s3fs then.

If we bump s3fs to `2021.08.0` would we still also need the aibotocore pin?
Don't think so, opened #48299 to find out
Passing tolerance
State: closed

- [x] closes #43913 
- [x] continued from #47947 
- [x] passed tolerance down in place of creating an entirely new function


Labels:

Comments:
As mentioned in https://github.com/pandas-dev/pandas/pull/47947#discussion_r957583805, please continue this work in  https://github.com/pandas-dev/pandas/pull/47947 to keep the discussion in one place, thanks. Closing.
Run GC in file leak check
State: closed

This fixes a problem with `file_leak_context` (xref https://github.com/pandas-dev/pandas/pull/30096#issuecomment-1222094485).

Demo:

```py
import gc
import pandas as pd
import pandas.util._test_decorators as td

with td.file_leak_context():
    reader = pd.read_csv("t.csv", iterator=True)
    reader.cycle = reader
    del reader
    # gc.collect()
```

Here we have a reference cycle `reader -> .cycle -> reader` that is only cleared when GC is run. On my machine, without the `gc.collect()` statement, `file_leak_context` detects a file leak where there is none.

The PR fixes this by explicitly calling `gc.collect()` before collecting the leaked files. I also changed an equality to a subset check to make the code robust against additional closed files (due to the `gc.collect()` call, or other reasons that I don't have any examples for).

cc @jbrockmendel 

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
does this merit a test?
Happy to add one, but I think it is difficult to come up with one that is both not too contrived and guaranteed to work (the one above scores good on the former but bad on the latter)
I'm not sure I conceptually agree with explicitly running the garbage collector. Is this not something we should just handle via weak references?
That would require all references that might indirectly reference any open files to be weak references. Actually I think it’s likely not even the standard library guarantees this. The standard idiom is to use `with` to make sure everything is closed immediately when you’re done but that’s not possible with Pandas’ idiom.
Ah sorry didn't notice the full context to this. Your explanation makes sense, though this seems symptomatic of other problems in the code base that ideally _should_ be addressed
Thanks @jonashaag
BUG: DataFrame.pivot not respecting None as column name
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Columns can never be empty when called by the user, should we deprecate positional arguments here and then remove the default? avoids the check on our side and simplifies things a bit

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
> Columns can never be empty when called by the user, should we deprecate positional arguments here and then remove the default? avoids the check on our side and simplifies things a bit

+1 for this. 
Will open a pr to backport to 1.5rc then, will convert this to draft till the other pr is merged then to simplify the backport 
cc @mroeschke  I think this is ready for another look now
Thanks @phofl
ENH: Allow to skip series type check in assert_frame_equal (#48287)
State: closed

Add an option check_series_type that is passed to assert_series_equal
in compare by columns.

- [x] closes #48287
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Needs Discussion
description: Requires discussion from core team before further action

Comments:
> Please always add tests. In general, it is better to wait for consensus on enhancement requests

Thank you for your feedback, @phofl.  I submitted a draft PR to show that the enhancement that I proposed is very easy to implement.  I was planning to add tests, but will need guidance from more experienced contributors to find a place to put the tests.  I was not able to find exiting tests for the `assert_frame_equal()` function.

With respect to reaching the consensus, I am new to the pandas contributors community and will appreciate your advice on the appropriate forum to bring my proposal to.
Opening an issue is the correct approach (as you already did). Discussion should happen there.
Thanks for the pull request, but it appears to have gone stale. Additionally new features usually need buy-in from the core team in the discussion of the issue before moving forward. Closing until there has been sufficient discussion, but can reopen later.
CI: Force newer S3FS
State: closed

- [ ] closes #48271  (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Could we enforce `botocore>=1.23.0` instead? Looks like the six changes occured there: https://github.com/boto/botocore/commit/5d4eccaa700556ed279bd243b9b7688748e2aad8

Not totally ideal IMO that this s3fs pinning in these files is higher than our minimum version.
Is there a reason to pin down s3fs? 

mamba is pulling really old versions of s3fs by chance(I think it's happening because old s3fs are not pinning down aiobotocore, allowing for mamba to pull down the newest aiobotocore versions which are not actually compatible with s3fs). Really old s3fs working with the newest aiobotocore is more luck than anything else.

This PR is just forcing mamba not to do this by setting a lower bound to a s3fs that pins down aiobotocore to a version it can support. The minimum version is still getting tested in the min version build so I don't think we're losing coverage.

> Could we enforce botocore>=1.23.0 instead? Looks like the six changes occured there: https://github.com/boto/botocore/commit/5d4eccaa700556ed279bd243b9b7688748e2aad8

I haven't tested this, but I'm pretty sure aiobotocore pins/controls the botocore version(IIUC it messes around with botocore internals so they have to pin down to a specific botocore version), so doing that would probably just result in an unsolvable environment. 

(As for why we care about aiobotocore, its a dependency of s3fs)

I agree with you on the point of bumping s3fs though(I think we should probably do it ASAP for 2.0).
Yeah the actual issue is that our minimum version of s3fs is not pinning aiobotocore, causing failures, hence we have to pin aiobotocore too causing the python 3.10 issues…
Yeah the actual issue is that our minimum version of s3fs is not pinning aiobotocore, causing failures, hence we have to pin aiobotocore too causing the python 3.10 issues…
I'll check the environment solving by lower pinning `botocore` quickly in https://github.com/pandas-dev/pandas/pull/48297

> Is there a reason to pin down s3fs?

Not really. I think it was May when I updated the min version of several packages using a 1-year-release-prior rule of thumb. IMO I think it would be okay to bump s3fs for 1.5 to `2021.08.0` since we advertise our min versions of optional dependencies as the ones we test with, and lower versions may very well still work.
Yeah then let’s bump this for the rc, if the solve fails
#48299, if this works out then we can close here
CLN, STYLE: remove unused variables and imports in Cython files
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Tried putting together a little Cython linter, here's the unused variables/imports it finds. More to come, but this already cleans up quite a few things

Labels:
name: Code Style
description: Code style, linting, code_checks
name: Clean

Comments:
Great cleanup @MarcoGorelli! 
It would be interesting to know why a lot of these unused variables don't get picked up via build warnings
I think the build warnings just pick up unreachable code
Gcc/llvm should warn for unused variables with the flags we have (-Wunused-variable us part of -Wall). My guess is Cython is generating code that still "uses" these variables and prevents that from happening. Likely hard to fix but in an ideal would that wouldn't happen in the first place
TYP: Specify how in dropna
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @phofl. Can follow up with defining `any/all` in `_typing.py`
BUG: MultiIndex.append not checking names for equality
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: MultiIndex

Comments:
Added test and also fixed an internal bug in pivot_table creating missmatches in Index names
Thanks @phofl
TST: add tests for using .at() with DateTime indexes
State: closed

- [x] closes #29709 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


added the tests described in issue #29709 but I feel like more around this subject could be written. let me know what you think! thank you for your time


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Datetime
description: Datetime data dtype

Comments:
Hello @nealxm! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-08-31 02:15:58 UTC
Hi Patrick, thanks so much for the feedback hopefully this is better. Just aside, I did not ignore your note on creating `df` just with passing `data=1` I just removed the the explicit column names to make comparing two full data frames a bit easier so the dimensions were no longer given. 
hello! I never left a comment after I fixed what your comments were referring to, just wanted to update you that I took care of it. 
Thanks @nealxm
WEB: Unpin pydata sphinx theme
State: closed

- [x] closes #46615 (Replace xxxx with the Github issue number)
- [x] closes #42744 (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Is there a way to test the docs served by the website locally?


Labels:
name: Dependencies
description: Required and optional dependencies
name: Web
description: pandas website

Comments:
Not sure if you've seen that this needs to be updated if we unpin the theme version:
```
WARNING: `generate_nav_html` is deprecated and will be removed.Use `generate_toctree_html` instead.
```

Also, I think it's a better practice to not include the domain in internal links (in a link in a page hosted at pandas.pydata.org, use `/foo/bar.html` instead of `https://pandas.pydata.org/foo/bar.html`). If for whatever reason we host this under a different domain (a dev server, change of domain we use, a reverse proxy...), then having the domain make things way more difficult.
@phofl did you check if we're getting the dark theme out of the box with the new theme? If we do, feel free to add #42744 to the list of issues this closes in the description.

Btw, I think it'd be nice to get this in for the release. I added the 1.5 milestone for now. If you want me to continue the work on this and see what's wrong with the CI, just let me know.
Changed the link, is this what you were referring to?

Yes, we get the dark theme ootb
Not sure if the docs are building, the last one was cancelled
Thanks @phofl and @datapythonista for the review. I guess we'll need to update the `versions.json` when 1.5 is released.
@datapythonista @phofl please see #49811 as a potential solution for a issue raised due to this we think (@tehunter).
DOC: fix a typo: relaying -> relying
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: Visualization
description: plotting

Comments:
Thanks @partev
DOC: Add none as a default argument for orient
State: closed

- [x] closes #48252 

Add None as an acceptable argument for orient in read_json over [here](https://pandas.pydata.org/docs/reference/api/pandas.read_json.html)


Labels:
name: Docs
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
Based on my understanding, None does nothing. It is just an acceptable value. 
The value of 'orient' depends on the value of "typ" parameter as mentioned later in the docstring over [here](https://pandas.pydata.org/docs/reference/api/pandas.read_json.html):

> The allowed and default values depend on the value of the typ parameter.

> For Series: default is index and For Frame: default is columns

Since, it is already mentioned, I am not quite sure how shall I improve it. Shall I change the default value of it being 'columns' instead of None since the default value of 'typ' is 'Series'?
thx @ambujpawar
BUG: array ufuncs with more than 2 inputs fails with 2 or more pd.Series and scalar arguments
State: closed

- [ ] closes #39853
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: ufuncs
description: __array_ufunc__ and __array_function__

Comments:
Ok Thank for review

@phofl @mroeschke 

Can please have review. Thank. :)
Please be patient, we will have a look when we have time
> Please be patient, we will have a look when we have time

Ok. Understand. Thank
And the tests?
> And the tests?
I have add the test in `test_array_ufuncs_for_many_arguments` inside this `pandas/tests/frame/test_ufunc.py`

> I think you had a test for ufunc(ser, ser, 1) with an earlier commit?

?
> And the tests?

@phofl yes. Earlier commit has test ufunc(ser, ser, 1) and ufunc(df, df, 1). Already remove test ufunc(ser, ser, 1) because it almost the same.  Do you think it is better I add ufunc(ser, ser, 1) test?
We should cover all cases, so yes please add back in. It's easy to construct an implementation that allows ufunc(df, df, 1) but rejects ufunc(ser, ser, 1)
> We should cover all cases, so yes please add back in. It's easy to construct an implementation that allows ufunc(df, df, 1) but rejects ufunc(ser, ser, 1)

Ok. Done
thx @weikhor
Backport PR #48272 on branch 1.5.x (CI: Require s3fs greater than minumum version in builds)
State: closed

Backport PR #48272: CI: Require s3fs greater than minumum version in builds

Labels:
name: CI
description: Continuous Integration
name: IO Network
description: Local or Cloud (AWS, GCS, etc.) IO Issues
name: Dependencies
description: Required and optional dependencies

Comments:
#48272 is unavoidable, but we should at least get #48276 in before merging here
Waiting for #48299