BrandonFors/pandas-github-issues huggingface dataset:
STY: x.__class__ to type(x) #batch-5
State: closed

- [x] ref https://github.com/pandas-dev/pandas/pull/29816
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks for the PR @MomIsBestFriend ! Do we have an issue for this btw? Ref just seems to be to a PR? 
> Thanks for the PR @MomIsBestFriend ! Do we have an issue for this btw? Ref just seems to be to a PR?

No we don't have an issue for that, it's something that was mentioned [here](https://github.com/pandas-dev/pandas/pull/29775#discussion_r349407678), And I decided to take care of it.
Convert core/indexes/base.py to f-strings
State: closed


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
@topper-123 need to xref #29547 so @MomIsBestFriend can maintain to-do list
I've renamed self.__class__ to type(self).

I'm not sure I agree with a policy of strictly requiring ``repr`` in f-string. f-strings and str.format have their own formatting mini-language with lots of other options than just !r, and !r is not an unusual modifer itself.
> I'm not sure I agree with a policy of strictly requiring `repr` in f-string. f-strings and str.format have their own formatting mini-language with lots of other options than just !r, and !r is not an unusual modifer itself.

I agree. !r is more concise and would be my choice, but we should be consistent.
@topper-123 looks like a merge conflict - can you merge master?

> I'm not sure I agree with a policy of strictly requiring `repr` in f-string. f-strings and str.format have their own formatting mini-language with lots of other options than just !r, and !r is not an unusual modifer itself.

Not a strong opinion but I do find `{repr(some_var)}` clearer than `{some_var!r}`. I always know what `repr` does don't always remember the format language specs. Ditto for `{str(some_var)}` versus `{some_var!s}`
DEPR: Categorical.ravel, get_dtype_counts, dtype_str, to_dense
State: closed

Starting in on the 0.25.0 section in #6581.

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
sure
@datapythonista im having trouble telling what the checks failure is about.  one thing suggests it is in asvs, but another suggests annotations
I just see this error in the benckmarks: https://github.com/pandas-dev/pandas/pull/29900/checks?check_run_id=323732409#step:13:4913

I guess it means that one of the 3 runs was too slow, but you'll know better.

It took me forever to find the error. I think it shouldn't be difficult to highlight failures in the benchmarks, so it's immediate to know what failed. I'll see if I can open a PR for that.
Thanks for taking a look.  I agree itd be nice to have an easier way to find when there is a broken asv.  I'm hopeful that #29906 will fix this particular problem
Thanks @jbrockmendel
TYP: some types for pandas/_config/config.py
State: closed

broken off #28339



Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
small comments, non-blockers, LGTM
LGTM
thanks @simonjayhawkins, @WillAyd comment might be worth a followup
> thanks @simonjayhawkins, @WillAyd comment might be worth a followup

The ignore was removed in #36012, since Name was added to the stub for tokenize.

I don't think a follow-up now needed or we need to change our usage.
STY: x.__class__ to type(x) #batch-3
State: closed

- [x] ref https://github.com/pandas-dev/pandas/pull/29816
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks!
STY: x.__class__ to type(x) #batch-2
State: closed

- [x] ref https://github.com/pandas-dev/pandas/pull/29816
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
LGTM
Thanks @MomIsBestFriend
String formatting > fstring
State: closed

xref #29547
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

More fstring changes

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
can you merge master; ping on green.
thanks @ForTimeBeing
CI: fix mypy complaint introduced in #29873
State: closed

@jreback the pertinent `delete` methods are never hit in tests.  any chance we can rip them out?

Labels:
name: CI
description: Continuous Integration
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
> the pertinent delete methods are never hit in tests. any chance we can rip them out?

If `delete` and `remove` are considered private, then the answer should be yes.  `delete` is only ever called by `remove`, and `remove` is only ever called with kwargs such that it never calls `delete`

the ``.delete`` method you just annotated is likely abstract so yes.

``.remove`` is public and calls the ``.delete`` method on the storer
> .remove is public and calls the .delete method on the storer

Darn. If we could remove `where, start, stop` from the signature (in tests these are never passed) we could rule out `delete` ever getting called.

Anyway, this should fix the CI
k merge on green.
py36 failure is due to a 0.1 microsecond lag between calls, should be ignored
> py36 failure is due to a 0.1 microsecond lag between calls, should be ignored

this is the now comparison test? let's just change that test to be better
STY: x.__class__ to type(x) #batch-1
State: closed

- [x] ref https://github.com/pandas-dev/pandas/pull/29816
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @MomIsBestFriend
BUG: .count() raises if use_inf_as_na is enabled
State: closed

- [x] closes #29478
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Thanks @thy09 
It's my pleasure!

When will the next version release?
This issue caused some problems in our project now.
>  When will the next version release?

Pandas 1.0 is planned for beginning next year

Thanks a lot!
ENH: warn Series.interpolate(method='index') with unsorted index
State: closed

`Series.interpolate(method='index')` use `numpy.interp` under the hood, which, from numpy doc, _Does not check that the x-coordinate sequence xp is increasing. If xp is not increasing, the results are nonsense._ [(link)](https://docs.scipy.org/doc/numpy/reference/generated/numpy.interp.html)
Here adds a warning to warn the user about this.

- [x] closes #21037
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:

Comments:
Can this be fixed instead of producing a warning?
Yes it can, in the cost of introducing dependency on scipy. (like other interpolate methods)
We can't use `pd.where(..., df.index)` to fill from the index?
You mean we sort it, interpolate, then fill it back?  this may leads to undefined behavior for duplicated indexes, better leaves the responsibility to scipy IMHO. :joy: 
> You mean we sort it, interpolate, then fill it back? this may leads to undefined behavior for duplicated indexes, better leaves the responsibility to scipy IMHO. 😂

then would just raise rather than a warning. 
So the preferred approach would be stick to `numpy.interp` but we do the sort for it right? No problem. I will close this one and do it in a new branch. Thanks for your suggestions.
i think either we sort then reorder to get the original 
or raise
CLN: Clean up of locale testing
State: closed

- [X] xref #23923, https://github.com/pandas-dev/pandas/pull/29852#discussion_r351228283

There are couple of things I don't understand from the locale:

- For what I see, we test with different locales (e.g. `it_IT.UTF-8`) but pandas never uses the language (`it_IT`) only the encoding (`UTF-8`). So, not sure if testing with different locales is being useful.
- The variable `LOCALE_OVERRIDE` seems to add more complexity than value. Unless `LC_ALL`/`LANG` are being set afterwards, but I don't think so.
- There is a test that is using `LOCALE_OVERRIDE` but doesn't seem to depend on the actual locale of the system. It probably makes more sense to parametrize and test all the locales we want, than use the variable value (when testing locally this test will be more useful).
- There is a "test" being skipped in `run_tests.sh` that is probably worth converting to an actual test.

Addressing these things here, but I may be missing something here. Please let me know if that's the case.

Labels:
name: Clean

Comments:
> these builds were added a few years ago to ensure that we run correctly in other locales. does this PR change this?

With this PR we still run with the same exact locales as before. Nothing less is being tested, I'm making sure the same works, but also that pandas works when not only the locale is different but also the encoding.
ok looks fine, ping on green.
I couldn't find a way to change the terminal encoding in Ubuntu to one different than UTF-8. Leaving this out out this PR. But I think this means that there are no tests in the CI to know if the option `display.encoding` is working or not.

This PR seems to properly change the locale (I think the locale stuff currently in master is not actually doing anything). The failures in the CI are tests that only work with an English locale (the error messages are localized, and don't match the ones in tests):
```
AssertionError: Pattern "(File (b')?.+does_not_exist\\.xlsx'? does not exist|\\[Errno 2\\] No such file or directory: '.+does_not_exist\\.xlsx'|Expected object or value|path_or_buf needs to be a string file path or file-like|\\[Errno 2\\] File .+does_not_exist\\.xlsx does not exist: '.+does_not_exist\\.xlsx')" not found in "[Errno 2] 没有那个文件或目录: '/home/vsts/work/1/s/pandas/tests/io/data/does_not_exist.xlsx'"
pandas/tests/io/test_common.py:166: AssertionError
```

If I'm not wrong, the exceptions come from other libraries (xlrd for example), that localize them. Not sure what's the best solution.

@jreback any preference?
@jreback got this working. This makes the locale builds actually test with different locales. Currently the locale builds don't really change the system locale.
very nice @datapythonista
TYP: privatize type alias in pandas/util/_decorators.py
State: closed

broken off #28339



Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
why do you think privatizing this is warranted? (other parts are ok)
It was a mistake when first added. Type aliases added directly in modules are added or the purpose of type checking only and shouldn't be public.

They should be considered an implementation detail and subject to change without notice.

All aliases in typeshed are prefixed with an underscore.

> It was a mistake when first added. Type aliases added directly in modules are added or the purpose of type checking only and shouldn't be public.
> 
> They should be considered an implementation detail and subject to change without notice.
> 
> All aliases in typeshed are prefixed with an underscore.

this entire module is private and this is inconsistent with the rest of the codebase where a type annotation is just a name; private just makes this harder to read.
so should I close this then?
> so should I close this then?

i don't see any other changes? if so, yes

do we have this pattern elsewhere? we should be consistent about it (I think we *mostly* are already)
> i don't see any other changes? if so, yes

one other minor change, i'll include in future PR instead.

> do we have this pattern elsewhere?

just https://github.com/pandas-dev/pandas/pull/29755#discussion_r349210957, could be reverted.

used heavily in #28339, but will change as break bits off


> we should be consistent about it

agreed. so this module being private is probably not relevant. We should not privatize aliases in any modules

> (I think we _mostly_ are already)

not really used many aliases outside of pd._typing yet
CI: Fix tests broken by np 1.18 sorting change
State: closed

Longer-term, do we want to use numpy's new behavior?

cc @TomAugspurger 

<b>Update</b> A little more background: consider two ndarrays in np<1.18

```
arr1 = np.array([1, 2, np.nan, 3, 4])
arr2 = np.array([1, 2, np.datetime64("NaT"), 3, 4], dtype="datetime64[ns]")

>>> np.sort(arr1)
array([ 1.,  2.,  3.,  4., nan])
>>> np.sort(arr2)
array([                          'NaT', '1970-01-01T00:00:00.000000001',
       '1970-01-01T00:00:00.000000002', '1970-01-01T00:00:00.000000003',
       '1970-01-01T00:00:00.000000004'], dtype='datetime64[ns]')
```

In numpy 1.18, the behavior of `np.sort(arr2)` is changing to put the NaT at the end instead of at the beginning, to behave more like NaN.  This breaks a few tests, which this PR fixes.

Side-note: as of now, 1.18 changes the behavior for datetime64, but not timedelta64.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Dependencies
description: Required and optional dependencies

Comments:
> Longer-term, do we want to use numpy's new behavior?

We already do? (put NaT last)

```
In [5]: pd.Series(np.array([1, 2, np.datetime64("NaT"), 3, 4], dtype="datetime64[ns]")).sort_values()  
Out[5]: 
0   1970-01-01 00:00:00.000000001
1   1970-01-01 00:00:00.000000002
3   1970-01-01 00:00:00.000000003
4   1970-01-01 00:00:00.000000004
2                             NaT
dtype: datetime64[ns]

In [6]: np.sort(np.array([1, 2, np.datetime64("NaT"), 3, 4], dtype="datetime64[ns]")) 
Out[6]: 
array([                          'NaT', '1970-01-01T00:00:00.000000001',
       '1970-01-01T00:00:00.000000002', '1970-01-01T00:00:00.000000003',
       '1970-01-01T00:00:00.000000004'], dtype='datetime64[ns]')
```
> We already do? (put NaT last)

Ah, I tested for Series. But for Index we indeed follow numpy's behaviour and put it first.


Going to merge this to get CI green again.

> Longer-term, do we want to use numpy's new behavior?

@jbrockmendel can you open an issue for this?

> Side-note: as of now, 1.18 changes the behavior for datetime64, but not timedelta64.

Known, or something to report to numpy?
DEPR: dropna multiple axes, fillna int for td64, from_codes with floats, Series.nonzero
State: closed


Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Clean

Comments:
Hello @jbrockmendel! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-12-01 18:23:00 UTC
ping
one more rebase and ping on green.
ping
rebased+green
thanks @jbrockmendel
CLN: remove never-used kwargs, make kwargs explicit
State: closed


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
Nice PR - always on board with these types of cleanups
thanks
ci is failing. was already failing here. revert or push fix? @jbrockmendel 
the 37npdev build should have been fixed.  the mypy complaint ill push a PR for in a few minutes
REF: implement cumulative ops block-wise
State: closed

- [x] closes #19296
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

cc @TomAugspurger 

Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
ok with moving this later, does this need a whatsnew note / additional tests as @TomAugspurger asks?
> does this need a whatsnew note / additional tests as

So far i haven't been able to write tests that fail without this.  Haven't given up.
do @TomAugspurger original tests work on master?
> do @TomAugspurger original tests work on master?

good point!  no, but they work here.  will update
Hello @jbrockmendel! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-12-27 20:23:03 UTC
same unrelated-looking docbuild failure as #29847.  im guessing this is going to be affecting all PRs about now, will take a look
thanks, looking forward to the cleanups!
REF: io.pytables operate on DataFrames instead of Blocks
State: closed

Wouldn't be surprised if there is a perf hit here, will run asvs.

Labels:
name: Refactor
description: Internal refactoring of code
name: IO HDF5
description: read_hdf, HDFStore
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
asvs seem like noise:

```
       before           after         ratio
-      10.7±0.1ms      9.26±0.07ms     0.86  io.hdf.HDFStoreDataFrame.time_query_store_table
-         186±3ms          159±1ms     0.86  io.hdf.HDFStoreDataFrame.time_write_store_table_dc
-      7.59±0.2ms      6.34±0.07ms     0.84  io.hdf.HDFStoreDataFrame.time_store_info

       before           after         ratio
+        30.6±2ms         42.6±2ms     1.39  io.hdf.HDF.time_read_hdf('fixed')
-      7.55±0.2ms      6.80±0.01ms     0.90  io.hdf.HDFStoreDataFrame.time_store_info

       before           after         ratio
+        29.3±1ms         41.3±5ms     1.41  io.hdf.HDF.time_read_hdf('fixed')
+     4.00±0.04ms      4.40±0.07ms     1.10  io.hdf.HDFStoreDataFrame.time_read_store_table

```
rebased+green
thanks
STY: F-strings
State: closed

- [x] xref #29547 
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @MomIsBestFriend
CI: Removing Checks job form Azure pipelines
State: closed

The `Checks` job was copied to GitHub actions, and looks like it's been working all right. Removing it from pipelines, so it's not diuplicated anymore.

Labels:
name: CI
description: Continuous Integration
name: Clean

Comments:
sure
LGTM.  Any reason to wait for the npdev/doc build?
LGTM as well.

Given that the failures are most certainly not related to the build.  Merging this.
Thanks @datapythonista !
Changed string formatting to fstring
State: closed

- [x] xref #29547
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

I am new to contributing to open source projects bare with me. I normally only push to my own 
master branch. I have never used git diff before(I know it is supposed to compare changed) and I just keep getting the error: fatal: bad revision 'upstream/master' if someone could tell me how to properly use it that would be great! It passed the black pandas test though. Once I figure out how to properly submit a PR abiding by pandas standards I will do trying a lot more PR. This is only a sample


Labels:
name: Code Style
description: Code style, linting, code_checks
name: Clean

Comments:
> Thanks for the changes @ForTimeBeing
> 
> You should have git remotes for:
> 
>     * `origin`: your fork
> 
>     * `upstream`: this repo (pandas-dev/pandas)
> 
> 
> Something like `git remote add upstream git@github.com:pandas-dev/pandas.git` should leave this correct if you cloned your fork, and the diff should work.

I now have upstreams as well:

origin  https://github.com/ForTimeBeing/pandas.git (fetch)
origin  https://github.com/ForTimeBeing/pandas.git (push)
upstream        git@github.com:pandas-dev/pandas.git (fetch)
upstream        git@github.com:pandas-dev/pandas.git (push)

I am still getting the bad revision though
run `git fetch upstream` first
@ForTimeBeing can you merge master to fix ci issues.
@ForTimeBeing can you try one more merge of master?
GH is showing me a zero-file diff.  Have these changes already been merged?
I guess this has been addressed elsewhere. Closing.
I apologize, I have not been around because of the holiday

Happy holidays everyone
CLN: remove unsupported sparse code from io.pytables
State: closed

AFAICT this is a legacy of SparseDataFrame

Labels:
name: Clean

Comments:
I think we're good removing these. There's a brief conversation in https://github.com/pandas-dev/pandas/pull/28425/files/b6fb1aabc19515672e840c7d0b30fafee1b88844#diff-1b15d1477da3a0548d2dd72a5d023d00 (where I removed SparseDataFrame & Series). IIRC, I had some hacks to try supporting it, but things looked generally broken for `DataFrame[sparse]`. I think I just forgot to remove this methods.
rebased+green
thanks
CI: Fix version openpyxl
State: closed

- [ ] closes #29854 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
https://bitbucket.org/openpyxl/openpyxl/issues/1373/broken-writer-with-lxml-defusedxml

Looks like there is a breaking change on openpyxl, so could limit it to `<=3.0.1` to avoid upgrading to the latest version until they fix the issue.
Thanks for the fix @charlesdong1991 

I don't think we want this. This will fix the error in the documentation, but `.to_excel()` will keep broken. I think we want an actual fix.
oh, sorry, did not notice your comment when i was committing @datapythonista 

thanks for your review, i thought this was only a doc error since I only observed this doc error in my another failed PR, I will take a another look, or you mean you want an actual fix from `openpyxl` side?
> or you mean you want an actual fix from openpyxl side?

The behavior that is failing in the docbuild should be made into a test (which will currently fail with new openpyxl) which we then need to fix on our end.
that said, we could add a test and xfail it to avoid blocking for now (while keeping a fixed version of openpyxl).
> that said, we could add a test and xfail it to avoid blocking for now (while keeping a fixed version of openpyxl).

sounds good
Thanks for both of your review @jreback and @jbrockmendel 

I made a new test and xfail if version of openpyxl if newer than 3.0.1, pls take a look to see if it is something we want now to avoid this blocker for other PRs.
LGTM.  Merging so we can get the CI back up
Is there an issue for the openpyxl regression? (here in pandas or upstream)
> Is there an issue for the openpyxl regression? (here in pandas or upstream)

Potentially this: 
https://bitbucket.org/openpyxl/openpyxl/issues/1373/broken-writer-with-lxml-defusedxml
CI: Fix npdev build
State: closed


Labels:
name: CI
description: Continuous Integration

Comments:
@jreback @TomAugspurger npdev build is green here
@jreback gentle ping.  combined with #29862 (just merged), this should get the CI back to green
thanks!
@jbrockmendel do you know if this are deliberate changes on numpy's side, or is this something to report upstream?
not that im aware of
BUG: loc-indexing on a CategoricalIndex with integer categories
State: closed

- [x] closes #17569
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Bug when indexing with ``.loc`` and the index is a CateggoricalIndex with integer categories.

Labels:

Comments:
Got some problems with this, closing temporarily.