BrandonFors/pandas-github-issues huggingface dataset:
STYLE: Use f-string in io/parsers
State: closed

xref #29547

Fixed many of the strings in pandas/io/parsers.py except for the string at line 1501 as that will require more time to make it function properly.  Please note that this is my first contribution ever so please double check my work!

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Hello @naomi172839! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-12-25 22:34:39 UTC
Thanks for the work on this @naomi172839 

If you check your proposed changes in https://github.com/pandas-dev/pandas/pull/30466/files you'll see that there are many unrelated changes (changes in the indentation).

To keep things simple, we don't merge PRs with side changes. And in this case, they are also incorrect based on Python standards.

I guess the changes were done automatically by your editor, or a different version of black than you should have. In any case, please have a look and make sure that you're only proposing the intended changes.

Let me know if you need help.
I believe that pycharm automatically formats the code.  I will correct the issue and submit a new pull request.
> I believe that pycharm automatically formats the code. I will correct the issue and submit a new pull request.

I was guessing is the editor. I don't know anything about PyCharm, but you may want to disable whatever is modifying the code automatically.

You don't need to create a new PR. Make the changes locally in your branch, commit them, and push them.

For the next time, you should create a git branch before starting working on something you want to propose. Working on `master` directly means that now you can't work on something in parallel now (may not be a problem right now, but will be in the future).
@datapythonista  Thank you for the advice! I ran "black" on the file and recommitted it.  Hopefully that is all that I needed to do.  Also thank you for your time! 
Thanks again @datapythonista!  Now that everything passes is there anything else that I am supposed to do?
> Now that everything passes is there anything else that I am supposed to do?

No, now someone else will have a look, and either have comments, or will merge this. We like to have two people from the team reviewing all the changes, that's why I didn't merge this myself (I approved, so next maintainer can merge if they are happy).
thanks @naomi172839
STY: Removed unconcatenated strings
State: closed

- [x] ref #30454
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
@MomIsBestFriend do you mind opening a PR for the script and the call to the script in `ci/code_checks.sh`? And just leave here the fixes.

The fixes should be non-trivial to get merged, but not the script (I think we should try using regex for it). Thanks!
> @MomIsBestFriend do you mind opening a PR for the script and the call to the script in `ci/code_checks.sh`? And just leave here the fixes.
> 
> The fixes should be non-trivial to get merged, but not the script (I think we should try using regex for it). Thanks!

Sure
thanks @MomIsBestFriend
BUG/ENH: groupby quantile arraylike fails with integer columns
State: closed

When columns are integers, `df.groupby(label).quantile(<arraylike>)` fails.
This PR fixes this issue by telling `MultiIndex` to only look for positional levels during an reordering operation.
Two new internal methods in `MultiIndex` class, and one new parameter for `DataFrame.reorder_levels` and `MultiIndex.reorder_levels` are added during the debugging process.

- [ ] closes #30289
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
> this is not the right solution as we are not adding a confusing at best parameter to a public function
> 
> this entire routine needs to be refactored to avoid the current method which is highly suspect now

hi Jeff, thanks for review.
Now that I think about it, I agree with you this solution is too rough.
The idea of `positional` parameter came from the discussions in the issue (#30289).
By "this entire routine needs to be refactored" I assume you mean `GroupBy.quantile` should not rely on `DataFrame.reorder_levels` at all? Could you let me know a bit more details about the design you are thinking of? 
Anyway, should I close this PR and start another one when ready? Or do you think maybe some of it is useful and I should keep this?
group by quantile needs the refactor to avoid the need to reindex the multiindex which is causing the issue now)

you can close or push to this PR up to u
Sorry I messed up the different commits and reverts in this branch. Will close this PR and open a new one.
TYP: Annotations in pandas/core/nanops.py
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
In the CI logs:
```
>           from scipy.stats import kendalltau, spearmanr
E           ModuleNotFoundError: No module named 'scipy'
```

For this to work you need to first do `from scipy import stats`, then do the specific imports.  I think a recent change in the test imports may be the reason why this PR was affected, but the import here definitely needs to be updated.
apparently a test in test_nanops is missing a td.skip_if_no_scipy.  have you rebased recently?
> have you rebased recently?

Yes, I have


looks ok, @MomIsBestFriend there is the return type of nanprod that is incorrect (and not erroring) ?

> looks ok, @MomIsBestFriend there is the return type of nanprod that is incorrect (and not erroring) ?

On my PC the type is

```python
<class 'numpy.float64'>
```

I think that's is because my CPU is 64-bit architecture, should I annotate it as
```python
Union[np.float32, np.float64]
```
to add support for 32-bit architecture CPUs as well?
or just to annotate it as ```float``` or ```np.float```?


cc @jreback @simonjayhawkins 
> or just to annotate it as `float` or `np.float`?

just `float` for now, since np.float resolves to `Any` without stubs for numpy.
> pls rebase as well

Rebased.
lgtm. @simonjayhawkins or @WillAyd if any comments. let's merge if good rather than add things to this one. ideally we keep basic typing PRs non-controversial and separate out the controversial to other PRs.
Thanks @MomIsBestFriend
BUG/Compat: dt64/td64 cummin, cummax on np 1.18
State: closed

- [x] closes #29058
- [x] closes #15553
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This also fixes+tests Series cummin/cummax with dt64tz, will have to look to see if there are any GH issues for that.

Following this and #29872 I think we should refactor a bunch of this out into core.nanops.

Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Timedelta
description: Timedelta data type
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Fixes at least one of the cases in #15553, will see if the others are already fixed+tested
Removed dead intervaltree code
State: closed

While looking to reduce build warnings I came across these methods and couldn't find any public use to them. The only one that seems to get hit is `get_loc` and that is in tests only so I think dead code

Labels:
name: Interval
description: Interval data type
name: Clean

Comments:
It looks like IntervalIndex.get_loc doesn't use `self._engine.get_loc`, which seems kind of weird.  Also looks like there's a good bit of either untested or dead code in indexes.interval: https://codecov.io/gh/pandas-dev/pandas/src/master/pandas/core/indexes/interval.py
@jschendel 
> It looks like `IntervalIndex.get_loc` doesn't use `self._engine.get_loc`, which seems kind of weird.

This is a result of the interval indexing behavior changes where we no longer allow partial interval overlaps, which was the main purpose of `IntervalTree.get_loc`.  The logic is now a bit simpler and much more straightforward to vectorize in numpy.

> Also looks like there's a good bit of either untested or dead code in indexes.interval

Yes, some of these can be removed for the same reason as above.  However, others are simply helper functions that are poorly tested.  After a quick glance:
- `_get_next_label` is used but poorly tested
- `_get_interval_closed_bounds` can be removed
- `_maybe_cast_indexed` can be removed
- `_can_reindex` is used and overrides a parent class definition that's not valid for `IntervalIndex`
- `_find_non_overlapping_monotonic_bounds` can be removed

Can look into the above more next week.

Thanks @WillAyd!
CI: Code checks ensure return status is handled
State: closed

Follow up from discussion here. https://github.com/pandas-dev/pandas/pull/30370#discussion_r361213525

Seems like we are not return the return code of all these code checks - unless I am missing something?

Expected this to fail on master until https://github.com/pandas-dev/pandas/pull/30455 is merged

cc. @WillAyd @datapythonista 

Labels:
name: CI
description: Continuous Integration
name: Clean

Comments:
Hmm OK - so I guess none of these were doing anything before? Do they all return the same value on failure or is it possible for a -1 return to clash with a +1 return now?
> Hmm OK - so I guess none of these were doing anything before? Do they all return the same value on failure or is it possible for a -1 return to clash with a +1 return now?

Yeah I believe these were not doing anything before.. all our other uses of invgrep update RET afterwards.

Believe our invgrep only returns [0,1] in `code_checks.sh`, so no need to worry about return values < 0 unless i'm missing something?

```
function invgrep {
    # grep with inverse exist status and formatting for azure-pipelines
    #
    # This function works exactly as grep, but with opposite exit status:
    # - 0 (success) when no patterns are found
    # - 1 (fail) when the patterns are found
```
OK lgtm then. @datapythonista care to take a look?
Will merge master once Marc has had a look. (Code checks should then pass since #30455 is now merged)

Example with CI failing as expected: https://github.com/pandas-dev/pandas/runs/362659323#step:4:10
> Strange that we missed this before, but yes, this seems to make sense.

Great thanks for taking a look - much appreciated! 
Thanks for spotting and fixing those @alimcmaster1 

For reference, this was introduced here: #29318

Not sure if for consistency/clarity we could have a separate error message for each of those, like everywhere else in the file. But not really important.
TST/REF: fixturize so as to avoid 500+ skips
State: closed

Re-order the fixturizatin of engine and read_ext so as to yield only valid combinations, avoiding the need to pytest.skip the invalid combinations.  This cuts out 500+ pytest.skips, which is about a third of the skips I see when i test with `--not-slow`.

The idea is that by cutting down on skips we can keep a closer eye on them, and things like [this](https://github.com/pandas-dev/pandas/pull/30441#discussion_r361222254) wont fall through the cracks quite as easily. 

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks, happy to remove these skips!
CLN: Fix abc imports 
State: closed

invgrep/Github actions exit status doesn't seem to be working correctly.

Currently on master we get these errors in the code_checks stage:
https://github.com/pandas-dev/pandas/pull/30452/checks#step:4:7

As per discussion here:
https://github.com/pandas-dev/pandas/pull/30370#discussion_r361213525





Labels:
name: CI
description: Continuous Integration
name: Clean

Comments:
> lgtm ping on green
> 
> If you have interest in dissecting invgrep later would also be welcome

Aha I think i've found the cause - see https://github.com/pandas-dev/pandas/pull/30457
Code checks now runs without error for non standard imports
```
Run ci/code_checks.sh patterns
Check for non-standard imports
Check for non-standard imports DONE
```

green cc. @WillAyd 
Thanks for the quick turnaround @alimcmaster1!
Parallel build for CI
State: closed

Letting this sit in draft state for a while to see if issues pop up

Note that this doesn't affect Windows (not sure yet if it can). Also previous attempts had `-j4` but I believe we only get 2 virtual cores for the CI machines so that was probably overkill. 

Labels:
name: CI
description: Continuous Integration

Comments:
lgtm pending green
Nice! - I was just about to ask on https://github.com/pandas-dev/pandas/issues/30356 why we don't do this in CI
@alimcmaster1 tried previously in #30214 but had to revert as the Py36 builds would intermittently fail thereafter.

Certainly open to any input you have on this
can you rebase again so we have a fresh comparison vs master. can you show the build or CI times (just see a representative sample)
Overall build time tends to fluctuate but this looks to chop off 2-5 minutes on average from non-Windows
lgtm. thanks @WillAyd 
Nice improvement, thanks @WillAyd 

xref #24028
Fix EX03 errors in generic.py
State: closed

Fixes EX03 errors in the examples of some docstrings in the script `pandas/core/ generic.py`.

Specifically, flake8 errors in the docstrings of the following functions were corrected:

```
pandas.DataFrame.add_prefix
pandas.DataFrame.add_suffix
pandas.DataFrame.head
pandas.DataFrame.last
pandas.DataFrame.pop
pandas.DataFrame.reindex
pandas.DataFrame.reindex_like
pandas.DataFrame.tail
pandas.DataFrame.take
pandas.DataFrame.to_xarray 
pandas.DataFrame.truncate
pandas.DataFrame.tz_localize
```

Related to #27977 cc @datapythonista 

- [ ] closes #xxxx
- [ ] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Thanks @martinagvilas . This error still occurs in other modules right? Would certainly take more PRs to fix up if interested
TST: Add tests for if_exists keyword argument in df.to_gbq
State: closed

- [x] closes #29598
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
ping
@WillAyd
Thanks @charlesdong1991 
Minor comment: Potentially might neater to parameterize the test above `test_roundtrip` since the method is very similar?
thanks for your comment @alimcmaster1 

sorry that i just noticed it. I quickly looked at the `test_round_trip` method, but I am not sure if it is need to parametrize. Could you clarify for which part you think to parametrize?

thanks!
BUG: Fix wrong error in df drop with non unique datetime index and invalid keys
State: closed

- [x] closes #30399
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
thanks @fujiaxiang
TST: Add tests for fixed issues
State: closed

- [x] closes #5647
- [x] closes #5586
- [x] closes #5286
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Hello @mroeschke! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-12-24 22:18:54 UTC
looks like a int32 vs int64 issue?
@jbrockmendel looks like that was the issue. All green
the other 3 LGTM
thanks @mroeschke
CLN .format to f-strings for several files
State: closed

updated to f-strings for:
pandas/tests/scalar/interval/test_ops.py
 pandas/tests/computation/test_eval.py
 pandas/tests/dtypes/*
      pandas/tests/dtypes/cast/test_infer_dtype.py
      pandas/tests/dtypes/test_dtypes.py
      pandas/tests/dtypes/test_inference.py

held off on implementing for regex match strings pending PEP 536

- [x] contributes to fstring corrections in #29547
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`



Labels:
name: Code Style
description: Code style, linting, code_checks
name: Clean

Comments:
Tests failed, updating with fix.

thanks @JMBurley
TST: clean up skips and xfails
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
sure
Thanks @jbrockmendel
Replaced .format{}/% with f-strings in core/tools/datetime.py
State: closed

Modified to python3 format strings
ref: https://github.com/pandas-dev/pandas/issues/29547


- [x] contributes to fstring corrections in  #29547
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
~~whatsnew entry~~ (will not add now. I think the full set of changes in #29547 should get a single update entry rather than a piecemeal one)


Labels:
name: Code Style
description: Code style, linting, code_checks
name: Clean

Comments:
suggest labels clean & style for this pull request, & part of the 1.0 milestone.
thanks @JMBurley
ENH: Allow scatter plot to plot objects and datetime type data
State: closed

closes #18755 
closes #30391 

xref #8113

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Because this is a plot PR, so I put the figures gotten by the two examples in those two PRs in this description to facilitate seeing output, and more cases are tested in test files.
![Screen Shot 2019-12-23 at 7 34 21 PM](https://user-images.githubusercontent.com/9269816/71374603-91982200-25bb-11ea-8ffd-4328fb1ea882.png)




Labels:
name: Visualization
description: plotting

Comments:
thanks @charlesdong1991
CI: Remove powershell scripts (Azure)
State: closed

- [x] closes #26344

Ref @jbrockmendel comment here https://github.com/pandas-dev/pandas/pull/23454#discussion_r360933924

CI: fails as expected example [here](https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=23715&view=logs&j=fd89438c-1fd3-5981-222e-115d06c89c2d)

Labels:
name: CI
description: Continuous Integration

Comments:
Nice! Thanks for taking a look at this.  LGTM pending green.
is the net effect of this that azure will run all steps but fail the build if anyone of them fails?
> is the net effect of this that azure will run all steps but fail the build if anyone of them fails?

Goal behind this is https://github.com/pandas-dev/pandas/issues/26344 to get rid of the powershell scripts. (Azure step conditions are unchanged - apart from Publish test result - which will run regardless of the outcome of previous)
cool, thanks @alimcmaster1
CLN: use f-string for JSON related files
State: closed

- [x] ref #29547 
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Labels:
name: Code Style
description: Code style, linting, code_checks
name: Clean

Comments:
Thanks @jyscao
CLN: Use of Iterable from collections.abc instead of typing.Iterable
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
looks fine, @WillAyd, @simonjayhawkins did we need something else besides this?
I _think_ the intention here was to use typing.collection https://docs.python.org/3/library/typing.html#typing.Collection which was added in 3.6 as opposed to using Iterable from collections.
Thanks @MomIsBestFriend
CLN/TYP: Removed redundant 'int' from pandas/_typing.py
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

As @simonjayhawkins says [here](https://github.com/pandas-dev/pandas/pull/30403#discussion_r360828949) (comment), having ```Union``` with both ```int``` and ```float``` is redundant.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Whether or not this changes anything for the type checker I think should keep both for the sake of explicitness
closing as per comments
CLN: changed .format to fstring in pandas/io/formats/csvs.py
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

https://github.com/pandas-dev/pandas/issues/29547

My first contribution.

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Test failures related to @MomIsBestFriend comments above:

```
 with pytest.raises(ValueError, match=msg):
>               float_frame.to_csv(path, header=["AA", "X"])
E               AssertionError: Pattern 'Writing 4 cols but got 2 aliases' not found in 'Writing 4 cols but got 
```
Issue to ref here is https://github.com/pandas-dev/pandas/issues/29547
Hello @sardonick! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-12-24 01:38:19 UTC
thanks @sardonick
ISORT: Add _typing to pre_core isort block
State: closed

- [x] closes #27534

Labels:
name: Code Style
description: Code style, linting, code_checks
name: Clean

Comments:
can you rebase and ping on green. as just merged some more things.
@jreback done and green - thanks!
thanks @alimcmaster1 

is this automatically pickuped up in the pre-commit hooks? (I think yes, just confirming)
fixed pandas/io/formats/printing.py formatting: replaced %/{}.format …
State: closed

Updated pandas/io/formats/printing.py formatting.  Replaced % and {}.format() examples to python3 formatting. Ran tests and they passed.  

ref [#29886](https://github.com/pandas-dev/pandas/issues/29886) and [#29547](https://github.com/pandas-dev/pandas/issues/29547).

Labels:
name: Code Style
description: Code style, linting, code_checks
name: Clean

Comments:
Thanks @AncientRickles
Fixing pandas/tests/indexes/datetimes/test_ops.py to utilize python3 …
State: closed

Updated pandas/tests/indexes/datetimes/test_ops.py to utilize python3 format strings.  Ran the test and it seems good.  I moved the msg into the for loop because it seems cleaner with the new format string logic. 
ref [#29886](https://github.com/pandas-dev/pandas/issues/29886) and [#29547](https://github.com/pandas-dev/pandas/issues/29547).

Labels:
name: Code Style
description: Code style, linting, code_checks
name: Clean

Comments:
Thanks @AncientRickles