BrandonFors/pandas-github-issues huggingface dataset:
REF: directory for method-specific series/frame tests
State: closed

Per the docstring in each of the new directories:

```
Test files dedicated to individual (stand-alone) DataFrame methods

Ideally these files/tests should correspond 1-to-1 with tests.series.methods

These may also present opportunities for sharing/de-duplicating test code. 
```

If this approach is approved, I'll then go through the remaining test files (in particular test_analytics) for things that have become "misc" piles.


Labels:
name: Refactor
description: Internal refactoring of code
name: Testing
description: pandas testing functions or related to the test suite

Comments:
sure
REF: refactor cumulative op tests from test_analytics
State: closed

The removal of `ix` highlighted how difficult it is to tell how thorough tests are for some funcs/methods.  This is one of several PRs intended to clarify where the tests for X live, and ideally to make that follow a standard pattern across DataFrame/Series/Index subclasses (and EA where relevant)

Labels:
name: Refactor
description: Internal refactoring of code
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks. i kind of like the classes as well, useful for organizing, but equally can just make a directory and a no-class structure. prob not worth the churn though.
[ENH] Add to_markdown method
State: closed

- [ ] closes #11052 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
Hello @MarcoGorelli! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:











There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-12-27 16:16:15 UTC
should Series support this too?  does `tabulate` have something that could be used for a `read_markdown` to round-trip?
> should Series support this too? does `tabulate` have something that could be used for a `read_markdown` to round-trip?

Sure, I've added it to Series.

I haven't found anything in the `tabulate` source code that can be easily used for `read_markdown`, but it would be a nice enhancement to work on.

Thank you @TomAugspurger and @jbrockmendel for your reviews!
> I haven't found anything in the `tabulate` source code that can be easily used for `read_markdown`, but it would be a nice enhancement to work on.

Maybe CSV code with `|` separator could work. One just has to ignore the header separation line. Maybe empty columns will be added because of the outer border separators and should be removed in post-processing automatically.
@jreback I've mentioned tabulate and added a link to its docs in `install.rst`, then rebased

@datapythonista Thanks! I've read through the thread you linked to. There seems to still be some disagreement among the core team about the issue though, so I'm not sure what to do and am in no position to pick sides
> Thanks @MarcoGorelli, this is quite useful.
> 
> But personally I think this is a great opportunity to implement this as a third party module, and start to work on making it easier to extend pandas with I/O adapters like this (see #26804)

@datapythonista that's orthogonal, and to be honest *this* IS almost entirely a third-party module.

It is *much* *much* easier to simply refactor later than to re-factor while adding something new.
thanks @MarcoGorelli very nice!
DOC: "Next" link from user_guide/io.rst goes to read_sql_table API pa…
State: closed

…ge (#30332)

Labels:
name: Docs

Comments:
Can you post a screenshot of what the "Next" text looks like when you build it locally?
![Screenshot from 2019-12-19 22-07-11](https://user-images.githubusercontent.com/24843773/71191316-31983780-22ac-11ea-8679-d311a755a34b.png)
Here is the screenshot.
Thanks @souvik3333
COMPAT: numpy test warnings
State: closed

Fixes #30334.

Some of the warnings are coming from within pandas (like the FutureWarning on truediv), some are numpy warnings (like the nan-warning on sign). All in all changes are minor.

Tests are passing with `-Werror` for me now.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Hello @nschloe! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-20 19:27:10 UTC
The truediv ones look perfect, thanks
needs `black pandas/tests/plotting/test_datetimelike.py`
@nschloe if you make a PR for just the truediv part id be comfortable merging that
The truediv part alone won't help me much, but feel free to just take the code and include it. I'd rather keep working on this one here until it's in shape.
can you merge master; we merged a patch which partially overlaps with this
@jreback Done.
@jbrockmendel if any comments
can you rebase
> can you rebase

There are conflicts now. Perhaps merge then?
Merged master.
Hi @nschloe  - sorry to chase you up, just wanted to ask whether you're still working on this :)
Seems that there's some test failures, but I can't check them as it says "build not found"
Just pushed a tentative fix.
cc @jbrockmendel 
LGTM
Thanks @nschloe
REF: pytables prepare to make _create_axes return a new object
State: closed

in the next pass we'll make create_axes return a new object, and instead of altering a bunch of attributes in create_axes, we'll pass them to the constructor for our new object just before returning

Doing all that makes for a pretty big diff, so this splits out some pieces.  The biggest one is implementing _get_blocks_and_items, which I find helpful since its a big chunk of code that i dont have to keep re-checking if it alters state.

Labels:
name: IO HDF5
description: read_hdf, HDFStore

Comments:
sure
Fix typos, via a Levenshtein-style corrector
State: closed

Should be non-semantic.

Labels:
name: Docs
name: Clean

Comments:
LGTM
Thanks @bwignall. Mind sharing how you do this? I’m familiar with Levenstein, but do you have a program that scans source codes for spelling errors?
Certainly, @topper-123 : I use https://github.com/bwignall/typochecker . It's a basic script (that's already in need of some refactoring), but the first version used https://en.wikipedia.org/wiki/Wikipedia:Lists_of_common_misspellings/For_machines to find likely typos, and it has bootstrapped the list of likely typos by looking at a few heuristics (such as a subset of Levenshtein edits) that have been run against some non-trivial codebases, including pandas.
Nice thanks. Can be nice to keep in mind when cleaning code bases.
DEPR: camelCase in offsets, get_offset
State: closed

- [x] closes #4205
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Frequency
description: DateOffsets
name: Deprecate
description: Functionality to remove in pandas

Comments:
@datapythonista the good news is im able to reproduce this locally with `make doc` like you suggested, the bad is that I have no idea why I'm getting a bunch of these complaints:

```
/home/vsts/work/1/s/doc/source/reference/api/pandas.tseries.offsets.BQuarterBegin.isAnchored.rst: WARNING: document isn't included in any toctree
```

ideas?

We fixed a lot of these in the past, but I'm not sure if we ever understood the problem. At least I don't think I did.

I think what we do is to add the docs being reported to a hidden autosummary. See: https://github.com/pandas-dev/pandas/blame/master/doc/source/reference/index.rst#L44

@jorisvandenbossche probably knows more
> BQuarterBegin.isAnchored.rst: WARNING: document isn't included in any toctree

It's because you removed those from the api doc pages, while they are still generated automatically from the class docstring. But so they also need to be included somewhere in another autosummary that adds them into the toctree. So like Marc said, we use hidden autosummaries to solve this. There should be some other examples in the api docs of that (look at the bottom of those pages).
See the "hidden" text in `doc/source/reference/index.rst` (https://github.com/pandas-dev/pandas/blob/master/doc/source/reference/index.rst)
thank you both
@jorisvandenbossche my attempts at mimicing the examples in offset_frequency.rst haven't worked.  in this commit i used "api.offsets.Foo.onOffset" but itt also didn't work with "Foo.onOffset", "api/Foo.onOffset", "pandas.tseries.offsets.Foo.onOffset"
The toctree needs the path (which becomes the url, eg https://dev.pandas.io/docs/reference/api/pandas.tseries.offsets.DateOffset.isAnchored.html), so you need something like `api/pandas.tseries.offsets.DateOffset.isAnchored`. However, if you do it on this page, it might need to be `../api/pandas...` since it is one level down in the hierarchy.
@jorisvandenbossche see newest commit where i tried just about every variant I could think of, including putting them in a different file where we already use this pattern.
>  is there any difference between get_offset and to_offset that the user should be aware of?

I dont think get_offset is supported in a meaningful enough way for this to make sense.  e.g. i just found the example in the docstring was wrong
> did you add a deprecation note in whatsnew?

yes
great thanks.
BUG: Fix infer_dtype_from_scalar to infer IntervalDtype
State: closed

- [X] closes #30337
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Interval
description: Interval data type

Comments:
LGTM
Thanks!
REF: change parameter name fname -> path
State: closed

- [x] closes #23574
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Deprecate
description: Functionality to remove in pandas
name: IO Parquet
description: parquet, feather

Comments:
@datapythonista i think we discussed this recently: is there a way to make the docbuild output easier to identify the part that went wrong?  Also would be nice to have a way to test locally instead of having to flood the CI
I think for the docs is not trivial to generate the output in a GutHub Actions friendly way. I guess we can do the `sed` hack in the bash script, if the sphinx output has a consistent format.

But a better fix would be to implement a Sphinx extension, less hacky, and it could be reused by other projects. 

Not sure what do you mean by testing locally. What can't we test locally? 
> Not sure what do you mean by testing locally. What can't we test locally?

I mean doing the docbuild locally so i can troubleshoot it without clogging the CI.

You've already gone above-and-beyond in making the CI easier to use, don't worry about this nitpicking.
if you installed the dependencies in the root `environment.yml` file, building the docs locally should be as easy as running `. /doc/make.py html`. I don't think anything else is missing, not sure if I'm missing something.

I'll have a look at making the errors in the output of the docs easier to find. I think those things are worth, since they save all us a significant amount of time. But I'll check for GitHub actions, since we're moving the docs build there. So it'll have to wait until #29874 is merged.
BUG: raise on non-hashable Index name, closes #29069
State: closed

- [x] closes #29069
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Index
description: Related to the Index class or subclasses

Comments:
To close https://github.com/pandas-dev/pandas/issues/29069, Index.name should probably be a settable property, and the setter should validate that the name is hashable.

Any perf concerns here? Things should be OK right?
huh, looks like i got confused by `names`, which is a property.  will update.
Are we OK with being able to set MultiIndex.name?  When I tried disallowing it, some tests broke
xref #11979 should we try to address that here while were at it?
I think #19171 becomes easier after this.
thanks
BUG: Integer Overflow in read_json with big number in string
State: closed

- [x] closes #30320
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
The result is only float if all inputs in a column are int/float. Actually, I've used a wrong testcase here, the Error only occurs when both int and str are used for one column. I will rectify that. As for the floats, I will look into keeping the whole column as object instead of float when parsing big numbers.
can you merge master and will look again
@rohitkg98 if you'd rebase and update for @WillAyd comment.
Sorry for the late reply. I'll do it ASAP @jreback @WillAyd 
Hello @rohitkg98! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-21 03:17:05 UTC
lgtm. @WillAyd if you had further questions.
Thanks @rohitkg98
CLN: Old string formatting: .format() -> f""
State: closed

- [x] xref #29547
- [x] passes `black pandas`
Old style % formatting to f-strings

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Hello @baevpetr! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:











There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-12-18 22:33:08 UTC
Couple of small issues, otherwise this is a nice cleanup, thanks
So there're some more `" f" ` typos. Fixed them. Sry for commit ddos.
thanks @baevpetr
TST: tests for needs-test issues #12857 #12689
State: closed

- [x] closes #12689
- [x] closes #9232
- [x] closes #6051
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Hello @jbrockmendel! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-12-20 21:34:58 UTC
thanks
Switching %r for repr(op_str) in test-expressions.py for issue 29886
State: closed

Swaps the following in pandas/tests/test_expressions.py:
From:
    `msg = "operator %r not implemented for bool dtypes"`
    `f = getattr(operator, opname)`
    `err_msg = re.escape(msg % op_str)`
To:
    `msg = f"operator {repr(op_str)} not implemented for bool dtypes"`
    `f = getattr(operator, opname)`
    `err_msg = re.escape(msg)`

This is for [#29886](https://github.com/pandas-dev/pandas/issues/29886)

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @AncientRickles!
DOC/CLN: move NDFrame.groupby to (DataFrame|Series).groupby
State: closed

By moving this method up to dataFrame/Series, we get better doc strings and more precise type hints.


Labels:
name: Groupby
name: Clean

Comments:
Makes sense. Can we get rid of the `get_grouper` function altogether as part of this?
get_groupby is used in ``Resampler``, so its used elsewhere.
@topper-123 IIUC the `get_groupby` usage in Resample sends the object as an argument so with this refactor wouldn't we just be able to change code like:

```python
grouped = get_groupby(subset, by=None, grouper=grouper, axis=self.axis)
```

To 

```python
subset.groupby(by=None, grouper=grouper, axis=self.axis)
```

? I think would be good to go all the way on that cleanup if so
@WillAyd , good observation, I've removed ``get_groupby`` from the code base in the newest version.

I've also renamed ``grp_generic`` to ``groupby_generic`` to make it's content a bit clearer. 
@WillAyd , looking more into it, that doesn't work after all, because the signatures to ``get_groupby`` and ``(DataFrame|Series).groupby`` are different. E.g. ``.groupby`` does not have parameters ``grouper`` and ``exclusions`` etc.

I'll revert the removal of ``get_groupby``.
Is this ok to merge? Notice that ``DataFrame.groupby`` doesn't have a grouper argument, while ``DataFrame.GroupBy`` does, so calling the the two are not equivalent.
Hmm so I think would be nice if we replaced `get_grouper` and aligned signatures; as it currently stands I'm not really sure this adds much though by moving the functions from the base class to subclasses so +/-0 on the change but let's see what @jreback thinks

ok with aligning signatures but in a dedicated PR
I've rebased the PR.

I'm not sure I understand how the signatures can be aligned: e.g. ``GroupBy`` has a ``grouper`` argument, which ``NDFrame.groupby`` doesn't have. I don't think it's a good idea to add the internal parameters of ``GroupBy`` to the public-facing ``NDFrame.groupby``.
> I've rebased the PR.
> 
> I'm not sure I understand how the signatures can be aligned: e.g. `GroupBy` has a `grouper` argument, which `NDFrame.groupby` doesn't have. I don't think it's a good idea to add the internal parameters of `GroupBy` to the public-facing `NDFrame.groupby`.

yeah I am now confused on what was said about aligning things. The Grouper and GroupBy are different things here.

thanks for the PR.
CI: Fixed travis-36-cov build
State: closed

Closes #30309 

Just debugging right now. Can't reproduce locally yet.

Labels:
name: CI
description: Continuous Integration

Comments:
In theory, setting `LD_PRELOAD=/opt/conda/lib/libgomp.sov` will fix this. Trying to understand the issue a bit more first.
Rather than updating that may also just want to get off of the trusty service since may have been patched in glibc

https://github.com/pytorch/pytorch/issues/2575#issuecomment-523667875


Mmm good to know.

I think we had issues with databases last time I looked. Will see if I can find that thread.
The LD_LIBRARY_PRELOAD thing did the trick, but I'd prefer to solve it more cleanly. Trying Will's suggestion next.
I think also need to add the appropriate DB services for CI to be green:

https://docs.travis-ci.com/user/database-setup/#mysql
Whoops sorry I see you did that one some already, but I guess that's what is causing the CI failure for the ones currently red
Yep, I have a commit to fix that, just waiting for the running job to see if the openmp thing is fixed. 
OK that one passed. Took 34 minutes though, compared to 27 in https://travis-ci.org/pandas-dev/pandas/builds/626386764 :/
Hmm yea something watch, though I think the runs normally fluctuate in that range
Thanks for the help Will! I'm glad to have avoided the LD_LIBRARY_PRELOAD stuff.

This last run took 32 minutes. I recall some ongoing work to balance these builds a bit, will see if I can track that down.
Groupby agg works with pd.Series.nunique, but groupby nunique fails with axis=1
State: closed

- [x] closes #30253
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Groupby

Comments:
PR has been updated accordingly to your remarks
Great thanks @nlepleux !
What performance are you looking for from the axis=1 case? Wonder if time is better spent optimizing that than reverting

Sent from my iPhone

> On Dec 18, 2019, at 4:17 PM, Jeff Reback <notifications@github.com> wrote:
> 
> ﻿
> @jreback commented on this pull request.
> 
> In pandas/core/groupby/generic.py:
> 
> > @@ -1813,9 +1813,20 @@ def groupby_series(obj, col=None):
>              # Try to consolidate with normal wrapping functions
>              from pandas.core.reshape.concat import concat
>  
> -            results = [groupby_series(content, label) for label, content in obj.items()]
> +            axis_number = obj._get_axis_number(self.axis)
> @WillAyd this is adding quite a bit of very hard and bespoke code
> 
> pls don’t merge non trivial things like this
> 
> this is also likely non performant
> 
> pls revert
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

i am way more concerned about the code smell this introduces
w/o much review
Fixings fstrings in pandas/io/stata.py & pandas/io/sql.py
State: closed

- [x] ref #29547
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks for the PR! CI failure is unrelated a few comments on first glance
This is going to produce a lot of conflicts with #30285 
While the changes here look cool, I think we should close this PR, and leave the changes here to #30285 (and a follow up of if needed).
agreed with @datapythonista.  Will revisit after #30285.  Closing.
CLN: changed .format to f string in pandas/core/arrays/base.py
State: closed

- [ ] ref #29547
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Sounds good thanks @Jcole429 !
@WillAyd , What is the best practice for maintaining my fork? Should I delete and make a new fork next time I want to contribute?

> This action cannot be undone. This will permanently delete the Jcole429/pandas repository, wiki, issues, and comments, and **remove all collaborator associations**.

If I delete my fork, will I lose any evidence of collaboration from my profile?

Thank you


Set pandas as your upstream if you haven't already (noted in contributor guide) then just always pull updates from `upstream/master` to your local master. Then just branch locally off master to the branch you want and push that to your fork before making a PR

I wouldn't worry about keeping master on your fork in sync with pandas unless you have another reason to
Okay, I do have it set as my upstream. I'll do that, thank you!

Just for clarification, if I were to delete my fork, all evidence would be removed from my profile?
You can delete the branch on github after merge but keep your fork. You will push future branches to your fork

Sent from my iPhone

> On Dec 17, 2019, at 10:42 AM, Justin Cole <notifications@github.com> wrote:
> 
> ﻿
> Okay, I do have it set as my upstream. I'll do that, thank you!
> 
> Just for clarification, if I were to delete my fork, all evidence would be removed from my profile?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.
ENH: Add support for DataFrame(ExtensionArray)
State: closed

- [x] closes #11363
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Problem:
In core.frame
```
        elif isinstance(data, abc.Iterable) and not isinstance(data, (str, bytes)):
            if not isinstance(data, (abc.Sequence)):
                data = list(data)
```
Categorical Object cast as list. So In core.construction it is not identified as Categorical. 


Labels:
name: DataFrame
description: DataFrame data structure
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
Thanks, @proost