BrandonFors/pandas-github-issues huggingface dataset:
CI/CLN: update travis
State: closed

update and cleanup Travis 

closes #36601 


Labels:
name: CI
description: Continuous Integration

Comments:
> try rebasing and we'll see if travis fixes itself

This particular Travis failure has been affecting everything all day.
any idea where this is coming from?
> any idea where this is coming from?

I set the number of workers to 1, and it works now. It's almost identical to what I saw on Raspberry Pi. I think the root cause is pytest-xdist. It probably has a problem specific to ARM architecture.
> > any idea where this is coming from?
> 
> I set the number of workers to 1, and it works now. It's almost identical to what I saw on Raspberry Pi. I think the root cause is pytest-xdist. It probably has a problem specific to ARM architecture.

oh wow, ok! ping on green.
thanks @fangchenli
CI: troubleshoot segfault
State: closed

I can't reproduce the CI failures locally, so just rolling back a few commits to narrow it down.

Labels:
name: CI
description: Continuous Integration

Comments:
Looking at https://github.com/pandas-dev/pandas/issues/31922#issuecomment-585999883 it seems the test reverted in https://github.com/pandas-dev/pandas/pull/36511/commits/4679970b2d64d3c55150e0953d04a15d8ae23a48 wasn't supposed to run across all builds?
> wasn't supposed to run across all builds?

Looks like, yah
hmm this is failing older builds?
ok so merging this and will re-open #31922 as this is supposed to run only on numpy master? (are the older build too old)?
DOC: Update generate_pip_deps_from_conda.py #36494
State: closed

- [x] closes #36494
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Sorry @dishak331 , it looks like #36496 was already opened first - lots of other open issues here to take though if you want to contribute :)
okay
CLN: Unify Series case in _wrap_applied_output
State: closed

One block of _wrap_applied_output is only executed when `first_not_none` is a Series; moved it to the else clause.

Labels:
name: Groupby
name: Clean

Comments:
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
sure
BUG: alignment changing index on input series
State: closed

- [x] closes #33671
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
also i think this needs a whatsnew bug fix reshaping note
whatsnew added + green
updated + greenasitgets
Modify doc/source/whatsnew7
State: closed

- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] modify files in doc/source/whatsnew 'v0.25.0.rst' 'v0.25.1.rst' 'v0.25.2.rst' 'v0.25.3.rst' 'v1.0.0.rst' 'v1.0.1.rst' 'v1.0.2.rst' 'v1.0.3.rst' 'v1.0.4.rst' 'v1.05.rst' 'v1.1.0.rst'

Labels:
name: Docs
name: Clean

Comments:
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
LGTM; only ambiguous one is IO vs I/O
thanks @cleconte987
PERF: DataFrame initialisation with dict with single key/value pair
State: closed

Faster initialisation of DataFrames, when the data is a single-item dict:

```python
>>> x = np.arange(1_000_000)
>>> %timeit pd.DataFrame(dict(a=x))
2.16 ms ± 66.7 µs per loop  # master
282 µs ± 4.63 µs per loop  # this PR
>>> x = np.array([str(u) for u in range(1_000_000)], dtype=object)
>>> %timeit pd.DataFrame(dict(a=x))
29.5 ms ± 131 µs per loop  # master
12.6 ms ± 55.8 µs per loop  # this PR
```

I don't think it's possible to optimize the case with more than 1 key/value pair, as concatenation will always take some time to perform.

Labels:

Comments:
Some issue found. Closing for now.
Update outdated instructions in scripts/generate_pip_deps_from_conda.py #36494
State: closed

- [x] closes #36494 
- [x] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Build
description: Library building on various platforms
name: Docs

Comments:
PR in response to https://github.com/pandas-dev/pandas/issues/36494
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
thanks @sharonwoo
CI: Update version of 'black'
State: closed

closes #35925 

Not many files left so am making a PR to close this off

Labels:
name: CI
description: Continuous Integration
name: Code Style
description: Code style, linting, code_checks

Comments:
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
Thanks @MarcoGorelli !
@pandas-dev/pandas-core @pandas-dev/pandas-triage reminder/notification to update `black`, in case you've not already done so
TST: DataFrame.to_parquet accepts pathlib.Path with partition_cols defined
State: closed

- [x] closes #35902
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Parquet
description: parquet, feather

Comments:
Does this require bumping the minimum pyarrow version? @jorisvandenbossche 
hmm, how / why does this pass on older pyarrow?
code lgtm just unclear on pyarrow versioning and why this would be working now - can you comment @arw2019 ?
> code lgtm just unclear on pyarrow versioning and why this would be working now - can you comment @arw2019 ?

It was fixed on the pandas side in #35681. 

So it will work in 1.2 with pyarrow 0.15+ (I have checked current master against 0.15, 0.17 and 1.0 pyarrow)
Thanks @arw2019 !
thanks @jorisvandenbossche @WillAyd @jreback for reviewing!
Commas Edited
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Hi @Noahlq2657 - what's this file you've added? Is there an open issue it references?
Hi @Noahlq2657 - the comma cleanup issue is now closed, so am closing this, sorry - thanks anyway for your PR!
Don't unlabel stale PR on update
State: closed

Apparently the bot unlabels on any activity (not just activity from the owner), e.g., https://github.com/pandas-dev/pandas/pull/34584#issuecomment-693107104, so if you ask the person for an update and they don't respond, the PR is no longer stale. Probably better to do this manually for now.

Labels:
name: Admin
description: Administrative tasks related to the pandas project

Comments:
sure
add a test for loc method; check if a warning raise when replacing a …
State: closed

…subframe

- [x] closes #36424
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Hello @samilAyoub! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-09-22 11:20:12 UTC
@arw2019 I need a help here, why Linting step fail here?
So I don't think this is actually testable with the current setup. Apparently code run in tests is configured to raise an error instead of a warning on chained assignment (hence all the test failures):

https://github.com/pandas-dev/pandas/blob/d38dc0652fd97b2ea56cff6696261bd628eb2f96/pandas/conftest.py#L151
@dsaxton So we can modify the test to capture raising instead of warning when chained assignment is used.
> @dsaxton So we can modify the test to capture raising instead of warning when chained assignment is used.

That seems reasonable
Also if you can now merge master that should fix the CI
@dsaxton did you mean merging master branch into this branch (add_test_subframe)?
> @dsaxton did you mean merging master branch into this branch (add_test_subframe)?

Yes, actually it seems CI is fine so not really necessary
@dsaxton Thank you for your review !
thanks @samilAyoub
TYP: core.missing; PERF for needs_i8_conversion
State: closed

Split off from a branch that shares `fillna` code between some of our EAs.

Preliminary to a PR that allows for an actually-inplace fillna implementation.

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
looks fine. assume the failures are transient.
ENH: PandasArray ops use core.ops functions
State: closed

Needs tests, hopefully i can tweak fixture usage in tests.arithmetic

Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
> Needs tests, hopefully i can tweak fixture usage in tests.arithmetic

you want to do this as a followon?
> you want to do this as a followon?

No, i'll add them here before long
closing to clear the queue while i figure out tests locally
updated with tests
thanks
CLN: Use https for @tm.network
State: closed

- [x] ref #36467

 AssertionError: Caused unexpected warning(s): [('ResourceWarning', ResourceWarning("unclosed <ssl.SSLSocket fd=18, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.20.0.43', 54722), raddr=('**172.217.204.95**', 443)>"), ), 

**This is a google address** -> which leads me to believe this resource warning is coming from our `network` function in `_testing.py` since it calls `urllib.request.urlopen("https://www.google.com")` 

Note: this doesn't solve the problem but we should use https. (Avoids a redirect)

Labels:
name: IO Network
description: Local or Cloud (AWS, GCS, etc.) IO Issues
name: Clean

Comments:
thanks @alimcmaster1
CLN: aggregation.transform
State: closed

- [x] closes #36330
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Followup to #35964 

- Moved whatsnew to reshaping
- Use is_list_like/is_dict_like in aggregate.transform
- Broke out dict-like and str/callable computations to own functions to split up the transform function
- Added/refined typing

Labels:
name: Clean
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
@jreback Tests added and pass; responses to your questions are above.
Thanks for reviewing @ivanovmg. I don't necessarily disagree with your suggestions, but I think they are outside the scope of this PR.
@jreback The diff on this is getting to be too complex (and honestly started that way). I've opened #36618 as a precursor.
@jreback - This is ready for another review. Failure is unrelated.
thanks @rhshadrach very nice
Backport PR #36385 on branch 1.1.x (BUG: Always cast to Categorical in lexsort_indexer)
State: closed

Backport PR #36385: BUG: Always cast to Categorical in lexsort_indexer

Labels:
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: Categorical
description: Categorical Data Type

Comments:
failure unrelated. see https://github.com/pandas-dev/pandas/pull/36396#issuecomment-695195148
Turn on stale PR GitHub action
State: closed

I think it makes sense to turn this on now (it's only labeling stale PRs for now, and after doing a lot of this manually it seems to have calmed down). Later can add some sort of friendly reminder to update the PR. Also rather than closing these stalled PRs I'm wondering if they could serve as starting points for new contributors who want to pick up where someone else left off.

Some recent workflows here (you can search for the string "Marking" in the logs to find PRs that would have been labeled): https://github.com/pandas-dev/pandas/actions?query=workflow%3A%22Stale+PRs%22

@simonjayhawkins You'd mentioned doing this more frequently in debug mode, would you prefer longer intervals when running live?

Labels:
name: Admin
description: Administrative tasks related to the pandas project

Comments:
> @simonjayhawkins You'd mentioned doing this more frequently in debug mode, would you prefer longer intervals when running live?

no strong preference. if you want to revert to daily, fine with me.
thanks @dsaxton
Backport PR #36266:: BUG: fix isin with nans and large arrays
State: closed

#36266

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
failure unrelated. see https://github.com/pandas-dev/pandas/pull/36396#issuecomment-695195148
Backport PR #36440 on branch 1.1.x (REGR: Series[numeric] comparison with str raising on numexpr path)
State: closed

Backport PR #36440: REGR: Series[numeric] comparison with str raising on numexpr path

Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
failure unrelated. see https://github.com/pandas-dev/pandas/pull/36396#issuecomment-695195148
CI: xfail html test #36467
State: closed

Part of #36467



Labels:
name: CI
description: Continuous Integration
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
I'm fairly certain that i've seen this on the backports too. so can either backport this or merge to master and I could backport if I see it again.
are we sure this is not coming from something inside read_html? eg something we are doing 
> are we sure this is not coming from something inside read_html? eg something we are doing

I guess we will be keeping the issue open until we are sure, but the problem will be less visible of course.
> are we sure this is not coming from something inside read_html? eg something we are doing

Agree - I think I diagnosed the root cause here: https://github.com/pandas-dev/pandas/pull/36480 (doesn't actually solve the problem though)
yeah i think we should NOT suppress these as they tend to get ignored
> yeah i think we should NOT suppress these as they tend to get ignored

agree since it appears issues can get closed and the xfails not removed. eg. #35660 (have opened #36524)
BUG: Fix astype from float32 to string
State: closed

- [x] closes #36451
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

It seems both bugs were caused by the same behavior from numpy:

```python
[ins] In [1]: import numpy as np

[ins] In [2]: arr = np.array([0.1], dtype=np.float32)

[ins] In [3]: arr
Out[3]: array([0.1], dtype=float32)

[ins] In [4]: np.asarray(arr, dtype="object")
Out[4]: array([0.10000000149011612], dtype=object)
```

Labels:
name: Bug
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Regression
description: Functionality that used to work in a prior pandas version
name: Strings
description: String extension data type and string data

Comments:
Thanks @dsaxton !
@MeeseeksDev backport to 1.1.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.1.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 a0d6d061de42c54b8071d28a4bf60cd69853c388
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #36464: BUG: Fix astype from float32 to string'
```

4. Push to a named branch :

```
git push YOURFORK 1.1.x:auto-backport-of-pr-36464-on-1.1.x
```

5. Create a PR against branch 1.1.x, I would have named this PR:

> "Backport PR #36464 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
[BUG]: Rolling.sum() calculated wrong values when axis is one and dtypes are mixed
State: closed

- [x] closes #20649
- [x] closes #35596
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

In case of ``axis=1`` and mixed dtypes, the ``_apply_blockwise`` did not calculate the sum for a complete row. Through converting the obj and consolidating the blocks we can avoid this.

Labels:
name: Bug
name: Window
description: rolling, ewma, expanding

Comments:
Does this also potentially close https://github.com/pandas-dev/pandas/issues/35596?
Thanks, I missed that issue. When the output you posted is the correct one, then yes, this fixes the issue. I will add a test.
@jreback so we should raise if we get dtypes except float and int?
I think we should just drop columns that are not coercible to float like `axis=0` behavior

```
In [1]: df = pd.DataFrame({'a': range(5), 'b': ['a'] * 5})

In [2]: df
Out[2]:
   a  b
0  0  a
1  1  a
2  2  a
3  3  a
4  4  a

In [3]: df.rolling(2).sum()
Out[3]:
     a
0  NaN
1  1.0
2  3.0
3  5.0
4  7.0
```
Thanks @mroeschke. Implemented it.
lgtm ping on green.
@jreback green 
thanks @phofl very nice!
CLN: Update files (as per #36450) to Python 3.7+ syntax
State: closed

Ran `pyupgrade` on all files unchecked in #36450. formatter pre-commit 
hook had no complaints; removed unnecessary imports from the following
to satisfy `flake8` hook:

```
pandas/tests/io/test_sql.py
pandas/test/series/test_io.py
```

- [x] closes #36450
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @ahgamut very nice!
DOC: replaced link to the documentation about setting a value on a co…
State: closed

…py of a slice from a DataFrame

- [x] closes #36448
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
After checking I realized that the link should not be changed in the release notes file - in the remaining references the link is correct.
CLN: Clean some formatting
State: closed

ref https://github.com/pandas-dev/pandas/issues/36450

Labels:
name: Clean

Comments:
Some of this seems to overlap with #36453, e.g. pandas/core/generic.py
> Some of this seems to overlap with #36453, e.g. pandas/core/generic.py

Ah you're right (maybe we're using the same tool 😄). This may find some different things so could always merge that one in and look at the (hopefully non-zero) diff between master and this branch afterwards. Or if we're doing literally the exact same thing I could just close this.
:rofl: looks like we had the same idea, sorry!

Looks like the rest of the files are covered by #36457 
NP, I can just close this
[TST]: Wrong Corr with Timedelta index
State: closed

- [x] xref #31286 (closes the issue, if there is no way around the numerical issues)
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`



Labels:
name: Window
description: rolling, ewma, expanding

Comments:
When using var instead of std, we can solve the numerical issues for the cases where var is still within floating point precision while std would not.
ok i'll buy this, why don't you add a bug fix note (in combination with the warning should be sufficient)
Thanks, should we close this issue then?
Looks like this is still worthwhile. Can you merge in master and fix the merge conflicts?
cc @mroeschke Conflicts resolved
Sorry could you merge master one more time just to be sure. The travis failure doesn't look related to this PR
cc @mroeschke 

Merged master, checks failure unlrelated. Opened #37404 to clean it up
cc @mroeschke Travis successful. Merged master to show ci/checks is green now.
Looks like there's one more merge conflict
Done
thanks @phofl
ASV: added benchamark tests for DataFrame.to_numpy() and .values
State: closed

- [x] closes #35023
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Adds benchmarks with single dtype and mixed frames of varying sizes.
xref #34999

Labels:
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
can you run these so we can see (and post here) in dev mode is fine as there is nothing to compare
@jreback, here are the benchmark results produced by `asv dev`. hope this is what we are looking for.

```
[  6.25%] ··· frame_methods.ToNumpy.time_to_numpy_mixed_tall                                      18.5±0ms
[ 12.50%] ··· frame_methods.ToNumpy.time_to_numpy_mixed_wide                                      3.97±0ms
[ 18.75%] ··· frame_methods.ToNumpy.time_to_numpy_tall                                            40.9±0μs
[ 25.00%] ··· frame_methods.ToNumpy.time_to_numpy_wide                                            95.4±0μs
[ 31.25%] ··· frame_methods.ToNumpy.time_values_mixed_tall                                        21.6±0ms
[ 37.50%] ··· frame_methods.ToNumpy.time_values_mixed_wide                                        5.70±0ms
[ 43.75%] ··· frame_methods.ToNumpy.time_values_tall                                              56.4±0μs
[ 50.00%] ··· frame_methods.ToNumpy.time_values_wide                                              86.0±0μs
```
This would be my first opensource contribution. I was not sure in what section of `whatsnew` file should I do an entry for this. Or, if I need to do it at all for benchmark tests, would appreciate some notes in that direction. Thank you.
thanks @hardikpnsp no whatsnew necessary for this.
CLN: remove trailing commas for black update
State: closed

File edited: 
pandas/tests/io/test_sql.py

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Hello @Noahlq! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-09-19 18:15:20 UTC
Hi @Noahlq 

Workflow here should be:

- update black to new version
- run black on files
- for the lines that have changed, try removing trailing commas
- run black on files again
- downgrade black to current version, check nothing changes
CLN: 35925 rm trailing commas
State: closed

#35925
Checked the following files to make sure they are ready for black :

- /pandas/tests/indexes/base_class/test_indexing.py
- /pandas/tests/indexes/test_common.py
- /pandas/tests/indexes/test_numeric.py


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @lacrosse91
CI: fix gbq test #36436
State: closed

- [x] closes #36436



Labels:
name: CI
description: Continuous Integration

Comments:
Guessing but would pinning google-cloud-core==1.3.0 work as in the comments from the google-cloud-python issue?

https://github.com/googleapis/google-cloud-python/issues/10471#issuecomment-669252150
> Guessing but would pinning google-cloud-core==1.3.0 work as in the comments from the google-cloud-python issue?
> 
> [googleapis/google-cloud-python#10471 (comment)](https://github.com/googleapis/google-cloud-python/issues/10471#issuecomment-669252150)

I haven't tried that yet. But it seems like google fixed it in a recent version.
The previous failure was unrealated to gbq.
Thanks @VirosaLi for tracking that down