BrandonFors/pandas-github-issues huggingface dataset:
REF: Simplify _cython_functions lookup
State: closed

The remaining two non-trivial cases are ("aggregate", "first") and ("aggregate", "median").  The "first" case I think is straightforward.  The "median" case I'm reticent to change because it isn't clear to me why it is a dict at all (cc @WillAyd if you have any ideas)

Labels:
name: Groupby
name: Clean

Comments:
Hello @jbrockmendel! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-10-30 23:16:54 UTC
So I don't know off the top of my head why this design started; I know I personally added the `rank` dict following in the footsteps of `first` and `median`

My hesitation with this change is that it only seems to solve for `rank` and leaves the other two in their current state. I could be convinced otherwise but would rather seem them address altogether if we are going to make a change
> My hesitation with this change is that it only seems to solve for rank and leaves the other two in their current state. I could be convinced otherwise but would rather seem them address altogether if we are going to make a change

That makes sense.  `first` is easy to "fix", just need to define default values for `rank` in group_nth (though AFAICT we never call group_nth with anything _but_ those default values, so we may want to just hard-code it and make group_nth into group_first)

For `median` the tests pass if we  change `{"name": "group_median"}` to just `"group_median"`, but I expect its a dict for some reason (ideas @jreback?)
> can we just add ngroups to the rank signatures and then this would just work?

That would work, yes
Updated per comments.  group_rank gets a new unused parameter, and group_nth we're punting on for the time being
thanks @jbrockmendel
ENH: Styler.background_gradient to accept vmin vmax and dtype Int64
State: closed

For `vmin` `vmax` use the same implementation as `Styler.bar`
For dtype `Int64` issue, deprecated `.values` and use `.to_numpy` instead
Here explicitly assign the dtype to float since we are doing normalize

- [x] closes #12145 and #28869
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @immaxchen!
REF: use fused types in hashtables
State: closed

This just gets the helper functions.  Getting the rest is more difficult.

Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
does this have any perf impact?
> does this have any perf impact?

will let you know in ~8 hours
This appears to cause a significant slowdown in `duplicated`.  Will look into this.
> This appears to cause a significant slowdown in `duplicated`. Will look into this.

k great thanks for running the asvs
Closing to clear the queue while i figure this out locally
Follow up PR: #28097 Simplify branch statement
State: closed


xref #28097

This is follow up pr. eliminate branch statement. "grouper" variable is assigned  when handle NAs.

Labels:
name: Groupby
name: MultiIndex
name: Clean

Comments:
can you merge master and see if you can get passing
thanks @proost this is not exactly what I was looking for, but its ok. thanks.
@MeeseeksDev backport to 1.0.x
Replace _has_complex_internals #29227
State: closed

closes #29227

Hi, it my first issues within hacktoberfest. 
I replaced _has_complex_internals with isinstance/
Is it possible to import ABCMultiIndex?


Labels:
name: Clean

Comments:
You're going to need to run `isort -rc pandas` to fix the linting errors.  In the reduction.pyx we don't allow imports from `core`, so the check will need to be something like `index.nlevels > 1`
also pls merge master, we just update the codebase with another small usage of  this (which needs to be changed)
pushed rebased and isort fixup
@jbrockmendel this needed or did you already remove?
still needed
ok @Aivengoe can you merge master. 

@jbrockmendel feel free to push here if its faster
thanks @Aivengoe and @jbrockmendel
API/DOC: an ExtensionDtype.__from_arrow__ method to convert pyarrow.Array into ExtensionArray
State: closed

xref the discussion in https://github.com/pandas-dev/pandas/issues/20612, and a companion to my PR in Arrow: https://github.com/apache/arrow/pull/5512

Summary: to support ExtensionArrays in the conversion of an arrow table into a pandas DataFrame, we need some way to convert pyarrow Arrays into pandas ExtensionArrays, given a certain pandas dtype (in pyarrow, we can for example know the resulting dtype from the stored metadata).

For that, I propose to add the `ExtensionDtype.__from_arrow__` method, with the following signature:

```
class ExtensionDtype:

    def __from_arrow__(self, array: pyarrow.Array) -> pandas.ExtensionArray:
        ...
```

Note: I only added documentation about it (which should still be expanded) for now, and not a method in the base class (eg a NotImplementedError), because in pyarrow we use `hasattr` to see if this is supported (see the linked arrow PR).

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Would this make more sense to be defined on the `ExtensionArray` itself rather than the `Dtype`?
@TomAugspurger asked the same question on the issue: https://github.com/pandas-dev/pandas/issues/20612#issuecomment-530818356, where I gave some arguments.

I am certainly not tied to it, but reasons I prefer the dtype:

- It's the dtype object that pyarrow has access to (and not the array class). So if this is defined on the array class, there needs to be another step of indirection `pandas_dtype(dtype).construct_array_type().__from_arow__(..)` instead of `pandas_dtype(dtype).__from_arrow__(..)`.
- We don't require a 1-to-1 mapping between dtype and array class, so you can have multiple dtypes using the same array class. We in fact do this ourselves for `IntegerArray` and `Int64Dtype`/`Int32Dtype`/..., and once the array class is constructed, it does not know by which dtype it was constructed. Now in practice for this example it is not a problem, as the `IntegerArray` will know from the incoming arrow array type which pandas dtype to use, but in theory this is not guaranteed to always be the case.


lgtm - we already construct from the dtype internally so this is appropriate 
Note: this PR is ready from my part, but let's leave it open until the feature is actually merged in Arrow (based on feedback there, details might still change).
OK, this is merged in arrow now. So merging here as well then.
Remove TestData in frame tests in multiple files - part 3
State: closed

Part of #22471 

This is removing/replacing TestData usage in the following files:
* `test_query_eval.py`
* `test_indexing.py`

Also removes `TestData` from `pandas/tests/frame/common.py` 

--------
- [x] closes #22471
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @SaturnFromTitan nice removals!
Period index 29204
State: closed

- [ ] closes #29204 


Labels:
name: Docs

Comments:
Hello @HughKelley! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-10-29 14:20:10 UTC
Remove TestData in frame tests in multiple files - part2
State: closed

Part of #22471 

This is removing/replacing TestData usage in the following files:
* `test_reshape.py`
* `test_sorting.py`
* `test_subclass.py`
* `test_timeseries.py`
* `test_to_csv.py`

--------
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks!
Remove TestData in series tests indexing
State: closed

Part of #22550

* Replaced test_data fixture in `pandas/tests/series/indexing/` with more granular fixtures
* Removed pandas/tests/series/indexing/conftest.py as the test_data fixture is not needed anymore
* Removed pandas/tests/series/common.py as TestData isn't used anymore

- [x] closes #22550
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean

Comments:
lgtm. Can you remove the common module altogether now?
As the other series related PRs are merged now - yes! I'll add that another commit for that
thanks @SaturnFromTitan very nice!
ENH: Show column name in assert_frame_equal
State: closed

The output of the AssertionError raised by assert_frame_equal is changed:

- old: `AssertionError: Attributes of DataFrame.iloc[:, 1] are different`
- new: `AssertionError: Attributes of DataFrame.loc[:, 'b'] are different`

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
hmm, the only issue is this doesn't quite work for duplicated names, but i guess this is friendlier for readers of the error message.
cc @TomAugspurger @jorisvandenbossche 
Yeah duplicates labels was my initial concern.

Can we update the error message to keep using `.iloc[:, i]`, but also include `column name=" "` somewhere?
Thanks for the feedback!
I see, I didn't think about duplicate names (and didn't know that they are allowed at all).
Hello @moi90! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-06 20:45:56 UTC
Sorry for the multiple commits. Feel free to squash.
@moi90 no need to worry about the squashing, but we do need to get the tests passing.  looks like there are a handful of places where you need to update the expected error message
I will look into this.
@moi90 any luck? If you have questions feel free to post them on there :)
@moi90 if you can use f-strings and ping on green.
@moi90 can you rebase
the CI error in the linter usually means there is an unused import somehwere
No, it's an error in flake8:

```
   File "/home/runner/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/flake8/formatting/default.py", line 39, in format
    "col": error.column_number,
ValueError: unsupported format character ':' (0x3a) at index 41
```

I'm beginning to give up... Can't someone of the core team bring it to a successful conclusion?
```
ValueError: unsupported format character ':' (0x3a) at index 41
```

The last few times I've seen this, it turned out there was an unused import somewhere.
can you merge master
@moi90 I merged master and fixed some flake8 issues. Not sure why it failed before, but know that you can always run the linting checks locally (I recommend installing git commit hooks for this: https://dev.pandas.io/docs/development/contributing.html#python-pep8-black).

All green now, so will merge. Thanks for the PR!
Wonderful, thanks! And you're welcome :)
# noqa removal in code base
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks
name: Clean

Comments:
As I removed all the `# noqa: F401`, Build failed "error F401: '*' imported but unused" @WillAyd 
Some of these are needed to pass our linting.  See the output in the "Checks" CI build.

That said, there are some of these (particularly non-specific "noqa"s) that are leftover from olden days and cleaning them up is definitely helpful.
Where is this coming from? Do you have an issue number, or a discussion in some other PR or issue that you want to reference in the description, so we know the context?
I don't think we can remove all of these - looks to be some failing CI you can keep around. I'm surprised to only see 10 in CI but maybe that's a good thing...
@datapythonista this is a follow up to #29209 and particularly my comment https://github.com/pandas-dev/pandas/pull/29209#issuecomment-546045353
@kaathi524 can this be closed in favor of several smaller PRs?
> @kaathi524 can this be closed in favor of several smaller PRs?

Yeah sure. I tried running flake8 in my local setup without removing any 'noqa' but it throwing error. 
Thanks @kaathi524, looking forward to the fixes in the smaller PRs. Please reference this PR when you open them, so we can track the discussion.
Changing logo in README and removing old logo
State: closed

Not sure if anyone prefers to publish the new logo when we release pandas 1.0 and the new website is life. I don't have a strong preference, but I'd say let's change the logo now if nobody cares, so one thing less to think about. I'll change the twitter logo and tweet about it once this is merged.

Labels:
name: Docs

Comments:
I'd slightly prefer to wait until the new site goes up as it would be cooler to release at once, but not blocking if others want to move forward with this
As said, no strong preference, and also thought it'd be cool to change everything together. But I see advantages on moving forward:
- NumFOCUS has the stickers, the shop...
- This announcement separately will allow a bit more visibility to Indeed for sponsoring the logo (and besides the deserved credit, I think it may be good that other companies see that open source projects are happy to see help from them)
- Releasing is enough work, having to synchronize these with the release is adding some more things to keep in mind

Nothing is that important, but that's why I thought of opening this now.

CC: @TomAugspurger 
+1 in this

doing anythig non completely standardized at release time is a no no
Either works for me. Will's points are good, but I may lean slightly towards just updating this now. 1.0 will still be a large enough celebration I think :)

The CI failure can likely be ignored.
@WillAyd happy to get this merged, based on previous comments?
Yea totally - merge away if ready
CI: Drop Python 3.5 support
State: closed

- [X] closes #29034
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration

Comments:
I suggest leave it to community for follow ups as they are kind of easy to do. I can think of issues to:

  1. Rip out any Py35 compat
  2. Replace OrderedDict with just dict (maybe?)
  3. Convert annotation comments to 36 inline syntax
  4. Use f-strings instead of .format (maybe?)

SGTM.

On Thu, Oct 24, 2019 at 2:22 PM William Ayd <notifications@github.com>
wrote:

> I suggest leave it to community for follow ups as they are kind of easy to
> do. I can think of issues to:
>
>    1. Rip out any Py35 compat
>    2. Replace OrderedDict with just dict (maybe?)
>    3. Convert annotation comments to 36 inline syntax
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/29212?email_source=notifications&email_token=AAKAOITYTX4Q6EL7CXGFCG3QQHYYXA5CNFSM4JEYZLM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECGEXQY#issuecomment-546065347>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKAOISFLC7MNDF5IEVCOP3QQHYYXANCNFSM4JEYZLMQ>
> .
>

@datapythonista : For the MacOS build (`ci/deps/azure-macos-35.yaml → ci/deps/azure-macos-36.yaml`), I think you need to re-specify the Python version to `3.6.*`.
Thanks for the review and the comments @TomAugspurger and @gfyoung , addressed them.

Good point @WillAyd, I included 1 here, but makes sense to create issues for 2 and 3, and not sure if we could create some for f-strings too. We don't want to get crazy with that, but in many cases that can be a nice clean up.
Not sure on the 36 failure here but I think @TomAugspurger cleaned up the solvability of the environment. Merge master and try again?
Seems like Python 3.6 has something (I guess a bug) causing the error `TypeError: only integer scalar arrays can be converted to a scalar index` when converting strings to bytes in some of our cases.

Checked locally, and seems like tests pass again in Python 3.6.1. Updated this PR, so the minimum supported version of Python is 3.6.1.
Interesting. Not totally against it, though I wonder if we should also up our minimum numpy version as part of this. 

Per NEP 29 I think they only support 1.15 with Python 3.6:

https://numpy.org/neps/nep-0029-deprecation_policy.html

So maybe worth bumping from 13.3 to that instead?
I don't have an opinion on bumping numpy version. But I think this PR is complex enough, and dropping old numpy versions is unrelated to this. I'd discuss and implement that separately.

NEP-29 is recent. Not sure when Python 3.5 is expected to be dropped by numpy based on it, but 1.15 supports even Python 3.4.
thanks @datapythonista very nice!

very small comment for possible followup.

Also would take a bump in numpy version I think we can go to min 1.15, but open an issue for it first.
Adding test for assert_equal_index with empty iterables
State: closed

- [x] closes #16844 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
You will need to run `black pandas`
Thanks @VijayantSoni 
Thank you guys @mroeschke @gfyoung
CLN: Remove unnecessary sys.version_info checks
State: closed

Title is self-explanatory.

Labels:
name: Clean

Comments:
I think versioneer is vendored
@jbrockmendel : Given the [history of changes](https://github.com/pandas-dev/pandas/commits/master/versioneer.py) we have made to it, I think it's fine to modify.
thanks @gfyoung
Clean up noqa E241 #29207
State: closed

- [X] closes #29207
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks
name: Clean

Comments:
lgtm. cc @TomAugspurger in case he knows any reason why maybe we should keep these, otherwise feel free to merge

By the way @kaathi524 if you are looking to contribute more auditing all of the `# noqa` comments in our code base might be a good place to start. Especially for any stylistic ones they might just not be applicable any more now that we have black formatting our code base
Looks good, thanks @kaathi524!
@WillAyd Yeah sure.
CLN: simplify core.algorithms
State: closed


Labels:
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: Clean

Comments:
thanks
This seems to have caused a small but consistent performance regression: https://pandas.pydata.org/speed/pandas/#index_object.IntervalIndexMethod.time_intersection?p-param1=1000&commits=f58b07e0 (might be worth investigating)

(to be clear: just reporting here what ASV reports, didn't do any sanity check of the validity of the report)
I'll take a look, thanks
Added io Annotation
State: closed

This is a quick follow up to https://github.com/pandas-dev/pandas/pull/26024/files#r317413254

The last remaining argument still has a slew of issues with it (provided below for reference) so leaving that for now

```sh
pandas/io/common.py:439: error: Incompatible types in assignment (expression has type "GzipFile", variable has type "Union[str, Path, IO[Any]]")
pandas/io/common.py:439: error: Argument "fileobj" to "GzipFile" has incompatible type "Union[str, Path, IO[Any]]"; expected "Optional[IO[bytes]]"
pandas/io/common.py:478: error: Argument 1 to "append" of "list" has incompatible type "Union[str, Path, IO[Any]]"; expected "IO[Any]"
pandas/io/common.py:483: error: Argument 1 to "open" has incompatible type "Union[str, Path, IO[Any]]"; expected "Union[str, bytes, int, _PathLike[Any]]"
pandas/io/common.py:486: error: Argument 1 to "open" has incompatible type "Union[str, Path, IO[Any]]"; expected "Union[str, bytes, int, _PathLike[Any]]"
pandas/io/common.py:489: error: Argument 1 to "open" has incompatible type "Union[str, Path, IO[Any]]"; expected "Union[str, bytes, int, _PathLike[Any]]"
pandas/io/common.py:496: error: Argument 1 to "TextIOWrapper" has incompatible type "Union[str, Path, IO[Any]]"; expected "IO[bytes]"
pandas/io/common.py:503: error: Argument 1 to "MMapWrapper" has incompatible type "Union[str, Path, IO[Any]]"; expected "IO[Any]"
pandas/io/common.py:504: error: Item "str" of "Union[str, Path, IO[Any]]" has no attribute "close"
pandas/io/common.py:504: error: Item "Path" of "Union[str, Path, IO[Any]]" has no attribute "close"
pandas/io/common.py:505: error: Incompatible types in assignment (expression has type "MMapWrapper", variable has type "Union[str, Path, IO[Any]]")
```


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Hmm I think this must be a 0.73 thing then. Will wait for that release
Test For MultiIndex not contained item
State: closed

- [x] closes #21094
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

I just made the test that was requested in this issue but in fact it isn't actually passing like mentioned there. Maybe my master is not updated for this. Should tests be added to whatsnew entry too?


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: MultiIndex

Comments:
Hello @gabriellm1! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-11-05 11:30:28 UTC
Looks like the new test is failing, can you take a look
@gabriellm1 can you rebase and try to figure out why the CI was failing
@gabriellm1 is this still active?
closing as stale, ping if you want to continue
Removed generate_bins_generic
State: closed

appears to be dead groupby code

Labels:
name: Clean

Comments:
nice!
@WillAyd can you rebase; hopefully azure will go through this time
Just restarted build instead to see. These intermittent failures have been rather frustrating lately :-X
one more push to restart CI?
Looks like that one worked
Thanks @jbrockmendel and @simonjayhawkins for review