BrandonFors/pandas-github-issues huggingface dataset:
Add pandarallel to ecosystem documentation
State: closed

As proposed by datapythonista here: https://github.com/nalepae/pandarallel/issues/28


Labels:
name: Docs

Comments:
@nalepae is this this active? Mind merging master the CI issues should now be fixed.
@alimcmaster1: Yes this PR (and `pandarallel` itself) are both active.

I'll be happy to be part of the `pandas` ecosystem!


Thanks @nalepae
TST: filter warnings for is_extension_type deprecation
State: closed

Follow up on https://github.com/pandas-dev/pandas/pull/29457

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Deprecate
description: Functionality to remove in pandas

Comments:
cc @jbrockmendel 
I'm surprised we dont have better tests for is_extension_array_dtype.  LGTM.
> What's this referring to?

the need for `test_is_extension_array_dtype` this implements
I assume it's verifying that `is_extension_array_dtype` is indeed a drop-in replacement for all the (tested) cases of `is_extension_dtype`. Some are probably duplicative.
> I assume it's verifying that is_extension_array_dtype is indeed a drop-in replacement for all the (tested) cases of is_extension_dtype.

Yes, but Brock's comment was about that this was needed, because `is_extension_array_dtype` did not yet have its own dedicated tests in `test_common.py` (it has some indirect tests in the extension array tests)
CI: GitHub action for checks (linting, docstrings...)
State: closed

xref #29472

Labels:
name: CI
description: Continuous Integration

Comments:
Is this supposed to take the place of something currently in travis/azure?  or totally new?
This is replicating part of the `Web_and_Docs` build in azure. I expected this PR to create a new check, next to the travis and azure builds, to see how it looks like. But seems like it needs to be committed to run (or I am missing something).

I guess in some time we'll want to replace both azure pipelines and travis with GitHub actions, but it's early to start discussing about it.
@pandas-dev/pandas-core here you can see how GitHub actions will look like:
- https://github.com/pandas-dev/pandas/pull/29546/checks?check_run_id=306452705

The main advantage over azure-pipelines is that clicking on `Details` takes you directly to the list of steps/errors. I also find the yaml syntax of actions a bit clearer than pipelines.

If we're happy to move forward, we can merge this, and leave the `Checks` build for a while in both systems, until we're confident with actions, and then we keep moving other things.
lgtm. can you rebase
@jreback rebased and all green
thanks
https://github.com/pandas-dev/pandas/runs/306852441#step:3:36

@datapythonista @jreback : I think this might have been merged too soon.  What's going on here?
ahh it’s not actually failing

thanks for the catch @gfyoung 

@datapythonista can u follow up
Wow, it's really amazing that we can link to a specific line in the log. :) I don't think we had this before, right?

Thanks for the heads up @gfyoung. I don't quite understand what's going on, but let me have a look.
> Wow, it's really amazing that we can link to a specific line in the log.

Travis CI has had this functionality for quite some time.  Why Azure does not escapes me 🙂.
Just to deliver some feedback we've just developed an extensive GHA configuration in Apache Arrow and we are pleased so far (see https://github.com/apache/arrow/tree/master/.github/workflows), if you have any questions @kszucs would be a good person to talk to, there's probably plenty we can learn from our respective experiences
Thanks for the info @wesm. Looks quite interesting, need to check in more detail.

I just opened a PR for a draft of our CI in actions: #29715

@kszucs happy to discuss and share experiences.
@datapythonista we keep our linux builds in docker containers, which is CI agnostic, makes local reproducibility possible and speeds up the builds in case of many dependencies. I'll try to take a closer look on your current CI scripts and your PR in the upcoming days.
[BUG] Validate dtype when Int64Index, UInt64Index, or Float64Index are cons…
State: closed

…tructed

- [x] closes #29539 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
Done, but I don't think it's right yet, need to understand what goes on in test_coercion (shouldn't be necessary to allow Float64 index with integer dtype?)

UPDATE: fixed
@jschendel thanks for your review :) I've updated my PR as per your suggestions
Thanks for your reviews, @jreback and @jschendel , really appreciate it. Have updated the PR accordingly.
@jschendel sure, done!
Can you merge master to resolve the conflict in the whatsnew?
Done - thanks for the approval!
thanks @MarcoGorelli
Add type hinting to public-facing API  in pandas/core/generic.py #26792
State: closed

- [x] Part of #26792
- [x] tests passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
you need to run `isort pandas/core/generic.py`
closing as stale, some of this has been done, welcome for PRs to continue
Fixed more SS03 errors
State: closed

Fixed SS03 errors in the docstrings for `pandas.tseries.frequencies.to_offset`; `pandas.Int64Index`; `pandas.UInt64Index`; `pandas.Float64Index`; `pandas.IntervalIndex`; `pandas.plotting.plot_params`; `pandas.io.formats.style.Styler.loader`.

- [x] xref #29315
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This is part of the docstring errors in #27977.


Labels:
name: Docs

Comments:
thanks @SebStrug 
does this close *all* of #29315 ?
@jreback 
Two still remain:
- [ ] pandas.Timestamp.isoweekday: Summary does not end with a period
- [ ] pandas.Timestamp.weekday: Summary does not end with a period
I couldn't find these in the codebase.

More SS03 errors exist for `pandas.tseries.offsets` but these are false-positives according to @datapythonista. I haven't investigated this, but could open an issue after I've looked into it.
We had some cases where the validation script was validating a Python function we were wrapping, and not our function. That's probably the case for `isoweekday` and `weekday`. I think the new version of the script in numpydoc should fix that problem, so probably better to forget about them for now, and when we use the new version of the script, we'll see what's pending.
REF: remove unnecessary exception in groupby
State: closed

in a previous pass we made this unnecessary be pre-checking for the condition under which a ValueError was raised.  This removes the try/except.

cc @jreback orthonal to other groupby PRs

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Clean

Comments:
@gfyoung 

the fact that this passes current groupby tests is good enough for the refactoring here. I think @jbrockmendel is trying to simplify this code overall.
REF: separate out _get_cython_func_and_vals
State: closed

Refactor this out of an overgrown method.  I think there may be a small speedup available by using a check that will short-circuit infer_dtype, will look at this in an upcoming pass.

Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
thanks @jbrockmendel
TST: add test for df comparing strings to numbers raises ValueError
State: closed

- [x] closes #11565
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
thanks @ganevgv
Clean Up Case Insensitive Comps in Tokenizer
State: closed

Some of this is unused; other aspects were reinventing the wheel

to_boolean was doing a lot of extra work to have a case insensitive comparison. This PR removes that code (and heap memory management) while using a cross-platform "strcasecmp" function as defined in `portable.h` already


Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
azure fail looks unrelated, can you re-push
LGTM.  cc @chris-b1
thanks @WillAyd
TST: add test for .loc indexing Index type preservation
State: closed

- [x] closes #15166
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
thanks @ganevgv
TST: add tests for Series/DF logical operations nan propagation
State: closed

- [x] closes #13896
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
thanks @ganevgv
Makes NumericIndex constructor dtype aware
State: closed

- [x] closes #29526
- [x] enables #29420
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Index
description: Related to the Index class or subclasses

Comments:
I added the what's new entry but I must admit I haven't yet managed to set up building docs in a performant way. I'll check that soon and let you know.
The failing tast was [this](https://github.com/pandas-dev/pandas/blob/master/pandas/tests/indexes/test_numeric.py#L565). Now there are more of them :) Perhaps it's too late now, I'll take a fresh look over the weekend.

I had some questions as well...

2. `Index._coerce_to_ndarray()` has practically become `Index._check_types_and_coerce_to_ndarray()`. Does it make sense to make an issue/PR to split these?
3. The change I made is in the method of an abstract Index class. This method has effectively become a "numeric-only" method. However, CategoricalIndex also inherits from Index. Does it make sense to change the conditional added in 458c25e to reflect this, something like

```
# use _typ instead of inferred_type property since this method, if ever used, will be used before the instance exists
if cls._typ != 'categoricalindex' and lib.infer_dtype(data) == 'string':
```
Hello @oguzhanogreden! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-11-17 21:30:54 UTC
Sorry about a bit of back and forth here.

Finally:

1. The solution depends on explicit passing of dtype.
2. @jschendel 's point about argument validation seems to be addressed centrally [here](https://github.com/pandas-dev/pandas/blob/master/pandas/core/dtypes/common.py#L1841).
thanks @oguzhanogreden
CLN: F-string formatting in pandas/_libs/*.pyx
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Super lost,
the tests that failed before are now passing, and the test that passed before are now failing, all I changed was 5 spaces before a string to make pep8 happy.

I have no clue what am I doing wrong.

can anyone help?
@jbrockmendel You legend!!!
thanks @MomIsBestFriend 

if you can make a master issue for this; would like to do the entire codebase but in multiple PRs, then add a check that we don't regress these changes.
ENH: Support for column specific col_space argument for frame.to_html…
State: closed

…() method (#28917)

- [ ] closes #28917
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] what's new: the col_space argument of to_html() can also take a dict or list to specify different col_space for different columns. 


Labels:
name: Enhancement
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
@AakankshaAshok is this still active? Can you add test(s)?
> @AakankshaAshok is this still active? Can you add test(s)?

@WillAyd Yes. Will do. Any other suggestions?
Not for now; can file more comments after test(s) are added
@AakankshaAshok still active? If so please merge master
Hi @AakankshaAshok  - sorry to chase you up, just wanted to ask whether you're still working on this :)
Closing as stale; ping if you'd like to pick back up
TST: Check error raised when inserting wrong length categorical column
State: closed

- [x] closes #9374
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Categorical
description: Categorical Data Type
name: DataFrame
description: DataFrame data structure

Comments:
Any ides why CI is failing on Linux Py36 ?  Doesn't seem related to my additions - having similar issue in #29522

```
=================================== FAILURES ===================================
______________ test_chunks_have_consistent_numerical_type[python] ______________
[gw0] linux -- Python 3.6.1 /home/vsts/miniconda3/envs/pandas-dev/bin/python

all_parsers = <pandas.tests.io.parser.conftest.PythonParser object at 0x7fb4c86cb5f8>

    def test_chunks_have_consistent_numerical_type(all_parsers):
        parser = all_parsers
        integers = [str(i) for i in range(499999)]
        data = "a\n" + "\n".join(integers + ["1.0", "2.0"] + integers)
    
        # Coercions should work without warnings.
        with tm.assert_produces_warning(None):
>           result = parser.read_csv(StringIO(data))

pandas/tests/io/parser/test_common.py:1204: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <contextlib._GeneratorContextManager object at 0x7fb48ad70390>
type = None, value = None, traceback = None

    def __exit__(self, type, value, traceback):
        if type is None:
            try:
>               next(self.gen)
E               AssertionError: Caused unexpected warning(s): [('ResourceWarning', ResourceWarning("unclosed file <_io.BufferedReader name='test1.xlsx'>",), '/home/vsts/work/1/s/pandas/io/parsers.py', 2927)].

```
Just a flakey test: https://github.com/pandas-dev/pandas/issues/29514

Could you try pushing again?
Ok just repushed - waiting for CI
CI failures still related to that test #29514 
Thanks - will try repushing again and see if it works  now.
thanks @ryankarlos
TST: Merge multiple cols with mixed columns/index
State: closed

- [x] closes #11354
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
@gfyoung  all green
TST:Add test to check single category col returns series with single row slice
State: closed

- [x] closes #14011
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Categorical
description: Categorical Data Type

Comments:
Hello @ryankarlos! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-11-10 13:34:58 UTC
Thanks @ryankarlos !
TST: Add tests for single level indexing with loc(axis=1)
State: closed

- [x] closes #13655
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: DataFrame
description: DataFrame data structure

Comments:
@mroeschke all green now
thanks @ryankarlos
CLN: String formatting % -> f-strings
State: closed

- Follow up from https://github.com/pandas-dev/pandas/pull/29212
- Towards https://github.com/pandas-dev/pandas/issues/29547

- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Old style % formatting to f-strings

Labels:
name: Clean

Comments:
looks like black complaints
@jbrockmendel - ready to re-review whenever you have a moment 
A handful of comments, general rule about minimizing the code that is executed inside f-space.  If its viable to have the black-version-related stuff not in this diff that'll make it a little easier.  cc @gfyoung for another look
Looks pretty good!  I agree with the comments here.
Updated as per comments and green cc. @jbrockmendel and @gfyoung 
thanks @alimcmaster1
TST: add test for .unique() dtype preserving
State: closed

- [x] closes #15442
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @ganevgv
Make subdirs in tests/io/data
State: closed

- [x] closes https://github.com/pandas-dev/pandas/issues/29439
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

A few of the "network" tests are failing locally as they require the data on master:
e.g
```
  @tm.network
    def test_spam_url(self):
        url = (
            "https://raw.githubusercontent.com/pandas-dev/pandas/master/"
            "pandas/tests/io/data/html/spam.html"
        )
        df1 = self.read_html(url, ".*Water.*")
        df2 = self.read_html(url, "Unit")

        assert_framelist_equal(df1, df2)
```

^ Will change to this branch to get the test passing on this PR. Then in a follow up PR will revert to master unless any core devs have a different idea


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean

Comments:
Hello @alimcmaster1! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:























There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-11-13 14:52:13 UTC
Yea the network stuff is a little wonky to manage when doing this, but note there is also non-network stuff mixed in. Out of the azure stuff I think only `py36_locale_slow_old_np` actually runs the network tests, so rest should be green
Test failures now just 1 test related to - https://github.com/pandas-dev/pandas/issues/29514
i think if you merge master this will be green.
thanks !
No worries - I’ll attempt to fix that flakey test mentioned above
API: Remove kwargs from GroupBy
State: closed

Remove kwargs from ``_GroupBy`.__init__`` and adds an explicit ``mutated`` parameter name instead. Also makes related changes. 

Avoiding kwargs when possible is better for typing ergonomics.

Labels:
name: Groupby
name: API Design

Comments:
generally +1.  im also unclear on the mutated thing.  I epxect we dont want users passing it directly, wondering if it can be kept out of the top-level method signature
> wondering if it can be kept out of the top-level method signature.

It is already allowed in the top-level signature, it's just hidden, so it would have to go through a deprecation process. But else I agree.
I am actually -1 on this change; unfortunate that we accept kwargs here but mutated is entirely an internal construction. I don't think we should expose it out as part of the GroupBy API
If mutated is internal, I think it should not be in the ``NDFrame.groupby`` signature, becuase that is a public signature. It should just call ``pandas.core.groupby.groupby.groupby`` directly, bypassing NDFrame.groupby.
I've made a few changes:

* the ``mutated`` parameter in NDFrame.groupby has been formally deprecated.
* I've renamed ``grouper.groupby.groupby`` to ``grouper.grouper.get_groupby`` to better signal it's a function
* I've typed out the parameters of ``get_groupby`` to make its signature explicit.

It didn't seem that any internal functionality actually used ``mutated``. At least no tests that don't explicitly use ``mutated`` failed.

why would you deprecate? it is not public 
> It didn't seem that any internal functionality actually used mutated. At least no tests that don't explicitly use mutated failed.

If this is correct, we could also directly remove it? (it has never been publicly exposed, only hidden in the kwargs, so I don't think a user could ever have known to pass a `mutated` keyword?) 


Ok, I've removed ``mutated`` from ``NDFrame.groupby``.
thanks @topper-123 nice clean
TST: add test for pd.melt category preservation
State: closed

- [x] closes #15853
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Categorical
description: Categorical Data Type

Comments:
Thanks @ganevgv
CLN: Added annotations.
State: closed

- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
I'm really confused about how it failed. 
Don't think the test failures are related to this change - I see the same on the latest PR that was merged -
https://github.com/pandas-dev/pandas/runs/295247988
I'm now even more confused, about the fact that it passed.
> I'm now even more confused, about the fact that it passed.

The test is unreliable ive raised an issue here: https://github.com/pandas-dev/pandas/issues/29514
Thanks @MomIsBestFriend
TST: Add tests for MultiIndex columns cases in aggregate relabelling
State: closed

- [x] closes #29422 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: MultiIndex

Comments:
the error isn't related to the change in this PR
many thanks for your review! @gfyoung 
changed!

tests still fail, but i don't think it is relevant to the change in this PR. 🤔 
@charlesdong1991 : Can you try rebasing OR merging with `master` and see what happens?
yeah, it works, probably a flaky test error, I also saw the same error in other PRs. @gfyoung 
thanks @charlesdong1991