BrandonFors/pandas-github-issues huggingface dataset:
BUG: Fix pd.json_normalize to not skip the first element of a generator input
State: closed



- [x] closes #35923
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
Consuming the generator seemed the easiest way to me. Since we also went for this approach in other location of the codebase I used it here as well. 

Please LMK if we want to solve this is another way, i.e. not consuming the whole generator at once.
Failures in CI caused by #38703 
Fair point. I might be misremembering but I feel like we do something similar for other IO methods, so a common way of doing that would be nice. But agreed for scope of this PR converting to list is fine

> On Dec 29, 2020, at 10:39 AM, Jeff Reback <notifications@github.com> wrote:
> 
> 
> @jreback commented on this pull request.
> 
> In pandas/io/json/_normalize.py <https://github.com/pandas-dev/pandas/pull/38698#discussion_r549807766>:
> 
> > @@ -261,9 +261,12 @@ def _pull_records(js: Dict[str, Any], spec: Union[List, str]) -> List:
>  
>      if isinstance(data, list) and not data:
>          return DataFrame()
> -
> -    # A bit of a hackjob
> -    if isinstance(data, dict):
> +    elif isinstance(data, abc.Iterator):
> +        # GH35923 Fix pd.json_normalize to not skip the first element of a
> +        # generator input
> +        data = list(data)
> we barely support generators (its not even documented), so -1 if this adds any complexity.
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub <https://github.com/pandas-dev/pandas/pull/38698#discussion_r549807766>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEU4UOCSISA6VXUMZEHTMLSXIO4RANCNFSM4VJFUZVA>.
> 


thanks @avinashpancham
TST: GH30999 Add match=msg to all pytest.raises in tests/indexes
State: closed

This pull request partially addresses xref #30999 to remove bare pytest.raises by adding match=msg. It doesn't close that issue as I have only addressed modules in pandas/tests/indexes

I made a few of the raised errors more specific types. There are two other changes that are a bit more complex but I will put specific comments on the code lines to explain those.

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
I am confused by the failing tests: 
FAILED pandas/tests/arrays/categorical/test_warnings.py::TestCategoricalWarnings::test_tab_complete_warning
FAILED pandas/tests/frame/test_api.py::TestDataFrameMisc::test_tab_complete_warning[DataFrame]
FAILED pandas/tests/frame/test_api.py::TestDataFrameMisc::test_tab_complete_warning[Series]
FAILED pandas/tests/indexes/test_base.py::TestIndex::test_tab_complete_warning
FAILED pandas/tests/resample/test_resampler_grouper.py::test_tab_complete_ipython6_warning

None of which I touched? Any idea what is going on here?
Ah, it's not just this PR. Several recent PRs have those same failures.
Jedi update causing problems with the CI raised in #38703 
cc @jbrockmendel if any comments, go ahead and merge if good
Hi @moink - can you fetch and merge upstream/master please and resolve the conflict?
> Hi @moink - can you fetch and merge upstream/master please and resolve the conflict?

@MarcoGorelli ok done and green except travis
I am not sure why I closed this before - must have been accidental.
thanks @moink
Updated PULL_REQUEST_TEMPLATE.md
State: closed

- [ ] closes #38696
- [ ] tests added / passed
- [ ] passes `pre-commit run --from-ref=upstream/master --to-ref=HEAD --all-files`
- [ ] whatsnew entry



Labels:
name: Docs

Comments:
- [ ] closes [#38696](https://github.com/pandas-dev/pandas/issues/38694)
- [ ]  tests added / passed
- [ ]  passes `pre-commit run --from-ref=upstream/master --to-ref=HEAD --all-files`
- [ ] whatsnew entry
> Do you want to update this in the template and also in the docs? Feel free to ask if you want/need help with any of this

I would love to contribute 😁
> it's been suggested to **instead** link to the docs and just have "Ensure all linting tests pass, see here how to run them" here

did you mean insert ?

* [ ]  closes [#38696](https://github.com/pandas-dev/pandas/issues/38694)
* [ ]   tests added / passed
* [ ]   passes `pre-commit run --from-ref=upstream/master --to-ref=HEAD --all-files`
* [ ]  whatsnew entry
* [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) how to run them

Should the new PR include the above details ☝
@MarcoGorelli 
what should be the documentation problem for the new issue mentioned above
#### DOC: update contributing.rst                

#### Location of the documentation

doc/source/development/contributing.rst

#### Documentation problem

Need additional information about the command that verifies the linting of code files.

#### Suggested fix for documentation

`pre-commit run --from-ref=upstream/master --to-ref=HEAD --all-files`

The command verifies the linting of code files, it looks for common mistake patterns
*NOTE: An additional paragraph is preferred for this*

☝ Is this issue description appropriate for the new issue?
take
thanks @BobinMathew 
Thankyou @MarcoGorelli and others for the support 🤝
TST: GH30999 Add match=msg to all pytest.raises in tests/generic/methods/test_sample.py
State: closed

This pull request partially addresses xref #30999  to remove bare `pytest.raises` by adding `match=msg`. It doesn't close that issue as I have only addressed one test module: pandas/tests/generic/methods/test_sample.py

The only other thing I did was move a few lines out of the `pytest.raises` context as they were setup lines that did not raise the error, in order to make it clear which pandas method was raising the error.

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
thanks @moink
TST: implement tm.check_setitem_equivalents
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
can you merge master again
Add match=msg to pytest.raises or convert to external_error_raised for modules in tests/plotting
State: closed

This pull request xref #30999 to remove bare `pytest.raises`. It doesn't close that issue as I have only addressed files within `pandas/tests/plotting/`.

Things I did that were not simple adding `match = msg` to `pytest.raises` or converting them to `tm.external_error_raised`:

- In `pandas/plotting/_matplotlib/_core.py` I found two error messages somewhat unclear. I changed the error message in the `ValueError` raised by `_get_stacked_values` to be, imo, a little bit clearer. In addition, in the constructor for `PiePlot`, the previous error message for the `ValueError` was "None doesn't allow negative values" which I changed to "pie plot doesn't allow negative values" because I believe that was the original intent.
- In `test_frame.py`, in the `test_errorbar_plot` method, the `pytest.raises` was allowing matplotlib to raise either a `TypeError` or a `ValueError`. I changed it to an `external_error_raised` but also specified it to be a `TypeError` because that is what it is, and I didn't see the reason it was allowed to be either error.
- Also in `test_frame.py`, the test `test_invalid_xy_args` was pytest-parameterized to test two different kinds of errors: x not being a label or position, or the label being bad. I split it into two tests, calling the new test `test_bad_label`
- Also in `test_frame.py`, the test `test_partially_invalid_plot_data` looped over a list with one element in it. I guessed that this was a remnant of an older test strategy that is no longer employed. I removed the list and made the kind of plot being tested more explicit.
- In a couple of test modules I moved a few lines out of the context of the `pytest.raises` because they were setup lines that didn't raise the error: I wanted to make it clear exactly which pandas method was raising the error.

I will note that another developer, [DylanBrdt](https://github.com/DylanBrdt) commented on the issue that they would address these files but they haven't been active on github since February.

- [ ] xref #30999
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
thanks @moink 

FYI on an issue like this use xref rather than closes in the PR header, otherwise GH will acually close the issue (which you normally want, but this issue is not closed yet).
> thanks @moink
> 
> FYI on an issue like this use xref rather than closes in the PR header, otherwise GH will acually close the issue (which you normally want, but this issue is not closed yet).

Ok thanks
BUG: DataFrame.rank with np.inf and np.nan
State: closed

- [x] closes #32593
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Seemed like two distinct issues here - first `np.inf` wasn't distinguished from `np.nan` (https://github.com/pandas-dev/pandas/issues/32593#issuecomment-597376199).  The changes to `skip_condition` logic handles that, but doesn't fix issue from OP. Rest of diff is to handle when `np.inf` and `np.nan` are both included.

Labels:
name: Bug
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
can you add the 2 examples from the OP as tests just to have them.
Added the tests from the issue
cc @jbrockmendel if any comments.
thanks @mzeitlin11
Revert "Deprecated casting an object-dtype index of datetime objects to DatetimeIndex in the Series constructor (23598)"
State: closed

This **partially** reverts commit 90a61350b1be7a200238212bd8d118a25a4d3d95.

- [ ] closes #38182
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
fine by me. cc @jbrockmendel 
If the idea is to un-revert following 1.2, sure.
> If the idea is to un-revert following 1.2, sure.

from https://github.com/pandas-dev/pandas/issues/38182#issue-753330794

> Users should not see unrelated deprecation warnings from the internals of pandas that they can't do anything about themselves, so IMO we should either fix those for 1.2, or delay the deprecationg for 1.3

not fixed, so delaying to 1.3
ok great, thanks @simonjayhawkins 

yeah we can unrevert this for 1.3, but i think ok to release once this hits.
@meeseeksdev backport 1.2.x
REF: share Index.difference
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry



Labels:
name: Refactor
description: Internal refactoring of code
name: Index
description: Related to the Index class or subclasses

Comments:
nice!
BUG: `scatter_matrix` raising with 2d axis passed
State: closed

- [x] closes #16253
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Visualization
description: plotting

Comments:
thanks @mzeitlin11 very nice!
CLN: Rolling _prep_values
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Clean
name: Window
description: rolling, ewma, expanding

Comments:
lgtm merge on green
CLN: Rolling helper attributes
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

`is_datetimelike` was just an internal check and can be in-lined 

`_on` can just be calculated once in the init

Labels:
name: Clean
name: Window
description: rolling, ewma, expanding

Comments:
sure merge on green-ish
Develop pre check
State: closed

- [ x] closes #37957
- [ ] tests added / passed
- [ x passes `black pandas`
- [ x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Hey @MarcoGorell and @jreback I added my changes to the existing check, implementing a wildcard. This PR should be ready for final review.

In addition, I think a great approach to implementing the pre-commit library would be to not only identify problems like we do now, but to fix them automatically. We can do this using the search and replace plugin - https://github.com/mattlqx/pre-commit-search-and-replace.
CLN: numba.prange usage
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

xref https://github.com/pandas-dev/pandas/pull/38417#discussion_r547925255

Labels:
name: Clean
name: numba
description: numba-accelerated operations

Comments:
Travis failure unrelated
DOC/CI: fix class alias docstrings for sphinx 3.4 + unpin sphinx
State: closed

Closes #38600


Labels:
name: Docs

Comments:
(we can also wait a bit to see what they answer on the sphinx side)
looks fine. but agreed looks like sphinx broke this.
prob need to backport this
@meeseeksdev backport 1.2.x
Something went wrong ... Please have  a look at my logs.
BUG: MultiIndex, IntervalIndex intersection with Categorical
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: MultiIndex
name: Categorical
description: Categorical Data Type

Comments:
what is the bug here? need a whatsnew? can you rebase
> can you rebase

just did, and managed to fully share the intersection methods!

> need a whatsnew?

added

> what is the bug here? 

```
mi = pd.MultiIndex.from_product([range(2), range(3)])
cat = mi.to_flat_index().astype('category')

>>> mi.intersection(cat)
MultiIndex([], )
```
nice ping on greenish
ping
TST/BUG: _gen_two_subplots always adding subplot even with passed axis
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

`kwargs.get("ax", fig.add_subplot(211))` was always adding a subplot (since the argument to `.get()` will be evaluated even if it doesn't need to be used). Assuming the intended behavior was for `fig.add_subplot(211)` to be called only if `ax` is not passed. This change also gets rid of half of remaining `matplotlib` warnings from 
`pandas/tests/plotting/test_hist_method.py::TestSeriesPlots::test_hist_legacy`

Labels:
name: Visualization
description: plotting

Comments:
this a user facing change? if so pls add a whatsnew (bug fix plotting in 1.3)
can you see if we have any issues open that this would close 
> this a user facing change? if so pls add a whatsnew (bug fix plotting in 1.3)
> can you see if we have any issues open that this would close

Don't think so, function is only used for testing plots (called from `_check_plot_works`)
thanks @mzeitlin11
ENH/ERR: More consistent error reporting with invalid win_type in rolling
State: closed

- [x] closes #15969
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Also in-lining methods that were only used once and moving a BaseIndexer test that was using `win_type`

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Window
description: rolling, ewma, expanding

Comments:
thanks @mroeschke 
@mroeschke this seems to have broken dask (this PR, or one of the other recent rolling-related PRs, but from a quick look at the diff, I suspect this one)

AFAIU, the issue is that the `Rolling` object (returned from eg `df.rolling("1s")`, which *is* a public object) its `win_type` attribute changed in case of time-based windows. Before, this was set to `r.win_type == "freq"`, but that's no longer the case.
Gotcha, do you have a reference issue how dask was using `win_type` attribute?
Not yet opened an issue about it, but so this is the code where it is failing: https://github.com/dask/dask/blob/4a7a2438219c4ee493434042e50f4cdb67b6ec9f/dask/dataframe/rolling.py#L317-L326. So basically they used `win_type == "freq"` (`self._win_type` in dask is pandas' `Rolling.win_type`) to check if they have time-based rolling, and because that changed they now end up in a different if/else branch

The failure it is causing is eg https://github.com/dask/dask/runs/1644840945#step:5:4998
BUG: Fix regression in pandas merge
State: closed

- [x] closes #38616
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Since this is a regression on master I think a whatsnew is not necessary?


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
thanks @phofl 
@meeseeksdev backport 1.2.x
REF: simplify logic in assert_produces_warning
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Simplified logic in ``assert_produces_warning`` by extracting functions ``_assert_caught_expected_warning``, ``_assert_caught_no_extra_warnings`` and ``_is_unexpected_warning``.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @ivanovmg
TYP: Extend typing to allow passing Index to_datetime
State: closed

ArrayConvertible should be able to accept Index or Series type as well
This extends it to use AnyArrayLike instead of ArrayLike.

- [ x] passes `black pandas`
- [ x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Ah sorry, it failed `isort` because the import I added was out of order (should be alphabetical: `AnyArrayLike, ArrayLike`).
The motivation for this is that a pretty common action
`df.index = pd.to_datetime(df.index)` 
is marked by my IDE (PyCharm 2020.3.1 Professional Edition) as a type error, since `to_datetime` accepts a list/tuple/Series/DateTimeScalar, but not an Index. The change doesn't seem to break anything else.
thanks @josh-friedlander-kando
DOC: capitalize NumPy as proper noun
State: closed

In a similar vein to https://github.com/pandas-dev/pandas/pull/37808

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
is it possible to modify the docs checkers to make sure this in enforced?
We could add a spelling checker when building Sphinx https://sphinxcontrib-spelling.readthedocs.io/en/latest/index.html. I don't see this extension used in the current `conf.py` file (https://github.com/pandas-dev/pandas/blob/master/doc/source/conf.py#L52-L69). But of course, this would add to the number of dependencies, which may or may not be wanted.

If there's another place you're referring to, @jreback, I'd be interested to learn.
i think this needs to be a pre-commit hook or maybe we discussed as a code_check. cc @MarcoGorelli

can do later for that i guess.
I think a `pygrep` hook to check that we don't use `Numpy` (with only the `N` capitalised) would be OK - @erictleung if you check the other local pygrep hooks in `.pre-commit-config.yaml` they can provide a guide, feel free to reach out to me for help (here or on the gitter channel) if you want to do this and need help
thanks @erictleung
TST/REF: Delete duplicate hist tests
State: closed

- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

The tests deleted in this pr are also found starting here: https://github.com/pandas-dev/pandas/blob/e4f425dddda6b299fe5f5e05f18f128302f8604c/pandas/tests/plotting/test_hist_method.py#L26-L27


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Visualization
description: plotting

Comments:
thanks @mzeitlin11 very nice
TYP: pd.eval, DataFrame.(eval, query) 
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

xref https://github.com/pandas-dev/pandas/pull/38416#discussion_r546736072

Labels:
name: Typing
description: type annotations, mypy/pyright type checking
name: expressions
description: pd.eval, query
name: Stale

Comments:
@arw2019 can you address @jbrockmendel comments.
Needs rebase and a couple of comments are outstanding, but otherwise I think this is pretty close.
@jbrockmendel over to you.
@MarcoGorelli feel free to push or otherwise close. I was going to close as stale but @jbrockmendel commented it was pretty close. https://github.com/pandas-dev/pandas/pull/38625#issuecomment-808916410
closing as stale.
BUG: join not working correctly with MultiIndex and one dimension categorical
State: closed

- [x] closes #38502
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Categorical
description: Categorical Data Type

Comments:
thanks @phofl
CI: move arm64 build to azure #38617
State: closed

xref #38617


Labels:
name: CI
description: Continuous Integration

Comments:
I'm not sure why azure won't allow me to run a script. It says permission denied.
The aarch64 build timed out. 
should investigate how many cores that this chip has. i think we need to make this *much* faster. hasn't even compiled after 30 min

maybe look at numpy recipe for aarch64
> should investigate how many cores that this chip has. i think we need to make this _much_ faster. hasn't even compiled after 30 min
> 
> maybe look at numpy recipe for aarch64

I think It's this slow due to that we need to use qemu to emulate arm64 in the container. NumPy uses Travis for arm64 build. I wonder how they are going to deal with this Travis credit issue. 
> > should investigate how many cores that this chip has. i think we need to make this _much_ faster. hasn't even compiled after 30 min
> > maybe look at numpy recipe for aarch64
> 
> I think It's this slow due to that we need to use qemu to emulate arm64 in the container. NumPy uses Travis for arm64 build. I wonder how they are going to deal with this Travis credit issue.

so we *can* use travis.com and if its only for a couple of builds its fine. though that build also was taking > 1 hr which seems crazy.
i guess we can just do this on the new travis.com
close for now.