BrandonFors/pandas-github-issues huggingface dataset:
PERF: op(frame, series) when series is not EA
State: closed

We have the same optimization in place for the flex ops.

ipython results for the asv this adds:

```
In [3]: arr = np.arange(10 ** 6).reshape(100, -1)                                                                                                                                                                   
In [4]: df = pd.DataFrame(arr)                                                                                                                                                                                         
In [5]: df["C"] = 1.0                                                                                                                                                                                               
In [6]: row = df.iloc[0]                                                                                                                                                                                            

In [11]: %timeit df + row                                                                                                                                                                                           
1.92 s ± 71.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)   # <-- master
1 ms ± 19.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <-- PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
@WillAyd any guesses why the benchmark check would be exiting at 50%?
hmm when i run  `asv dev` locally it stops at 50% but with exit code 0.  any guesses @datapythonista ?
> hmm when i run `asv dev` locally it stops at 50% but with exit code 0. any guesses @datapythonista ?

I tried `asv run` locally with master, and it worked well until the end. Is it just `asv dev`, or just this PR that is stopping half way? I don't see why this should happen.
Tried `asv dev` locally, and also stopped at 50%. Is there an issue for it? I see that last release of `asv` was a year ago, so I guess it's something in our side.
Looks like the asv issue is actually that this causes an unrelated benchmark to fail.  Am troubleshooting now and will add an appropriate test.
> lgtm. can you add a whatsnew note & an asv that covers this.

done
Update _core.py with missing parameters
State: closed

added missing parameters:
 - ax
 - subplots
 - sharex
 - sharey
 - secondary_y
 - sort_columns
 - stacked

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs
name: Visualization
description: plotting

Comments:
Hello @loudlemon! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-05-03 03:42:43 UTC
Not super well versed in this area so @datapythonista merge when happy
Thanks @loudlemon
Added a check for standard documentation
State: closed

- [ ] closes #33213 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

I created a check to see which files contain Pandas or *pandas* which will be of use for individuals aiming to resolve issue #32316. When the codecheck script is run, it will output the files that contain those strings.

Labels:
name: Docs
name: Code Style
description: Code style, linting, code_checks

Comments:
The CI is failing because of trailing whitespace in the code check file
@willie3838 can you address comments https://github.com/pandas-dev/pandas/pull/33592#discussion_r413938010 and https://github.com/pandas-dev/pandas/pull/33592#discussion_r413938641
@willie3838 can you merge master and address comments
closing as stale
pls ping if u want to continue
TST: added test for GH28597
State: closed

Added test to ensure that categories stay ordered when grouping
with missing values.

- [ ] closes #28597
- [X] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Categorical
description: Categorical Data Type

Comments:
Hello @mojones! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-04-17 08:44:02 UTC
thanks @mojones very nice.

happy to take a test for #23865 as well
BUG: Raise a TypeError when record_path doesn't point to an array
State: closed

When `record_path` points to something that is Iterable but is not
a sequence in JSON world we will receive odd results.

```
>>> json_normalize([{'key': 'value'}], record_path='key')
0
0  v
1  a
2  l
3  u
4  e
```

Based on RFC 8259 (https://tools.ietf.org/html/rfc8259) a JSON value MUST be
object, array, number, or string, false, null, true. But only two of them
should be treated as Iterable.

```
An object is an unordered *collection* of zero or more name/value
pairs, where a name is a string and a value is a string, number,
boolean, null, object, or array.

An array is an ordered *sequence* of zero or more values.

--
https://tools.ietf.org/html/rfc8259#page-3
```

Based on that `[{'key': 'value'}]` and `{'key': 'value'}` should not be
treated in the same way. In `json_normalize` documentation `record_path`
is described as `Path in each object to list of records`.

So when we want to translate JSON to Python like an object we need to take
into consideration a list (sequence). Based on that `record_path` should
point out to `list`, not `Iterable`.

In specs I added all possibile values that are allowed in JSON and
should not be treated as a collection. There is a special case for null
value that is already implemented.

|  type  |  value  | Iterable | Should be treated as list |
|--------|---------|----------|---------------------------|
| object | {}      | Yes      | No (unordered list)       |
| array  | []      | Yes      | Yes                       |
| number | 1       | No       | No                        |
| string | "value" | Yes      | No                        |
| false  | False   | No       | No                        |
| null   | Null    | No       | No (Check #30148)         |
| true   | True    | No       | No                        |

- [x] closes #26284
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
cc @WillAyd 
Thanks @LTe
TST: add message check to pytest.raises (tests/arrays/boolean/test_arithmetic.py)
State: closed

- [ ] xref #30999
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Hello @Dxin-code! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/tests/arrays/boolean/test_arithmetic.py`](https://github.com/pandas-dev/pandas/blob/27a882dd4bc55e2c86b6ffb298522f8e37478f2b/pandas/tests/arrays/boolean/test_arithmetic.py):

> [Line 26:10](https://github.com/pandas-dev/pandas/blob/27a882dd4bc55e2c86b6ffb298522f8e37478f2b/pandas/tests/arrays/boolean/test_arithmetic.py#L26): [E111](https://duckduckgo.com/?q=pep8%20E111) indentation is not a multiple of four
> [Line 26:10](https://github.com/pandas-dev/pandas/blob/27a882dd4bc55e2c86b6ffb298522f8e37478f2b/pandas/tests/arrays/boolean/test_arithmetic.py#L26): [E113](https://duckduckgo.com/?q=pep8%20E113) unexpected indentation
> [Line 26:13](https://github.com/pandas-dev/pandas/blob/27a882dd4bc55e2c86b6ffb298522f8e37478f2b/pandas/tests/arrays/boolean/test_arithmetic.py#L26): [E225](https://duckduckgo.com/?q=pep8%20E225) missing whitespace around operator
> [Line 34:8](https://github.com/pandas-dev/pandas/blob/27a882dd4bc55e2c86b6ffb298522f8e37478f2b/pandas/tests/arrays/boolean/test_arithmetic.py#L34): [E901](https://duckduckgo.com/?q=pep8%20E901) IndentationError: unindent does not match any outer indentation level


is this a duplicate of #33581? please make changes to the original PR instead.
Issue 33428 fix
State: closed

- [ ] closes #33428
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Hello @Dxin-code! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/plotting/_misc.py`](https://github.com/pandas-dev/pandas/blob/9f88a29b51e044e8d75b0dfeccdbfca7efd433d4/pandas/plotting/_misc.py):

> [Line 362:89](https://github.com/pandas-dev/pandas/blob/9f88a29b51e044e8d75b0dfeccdbfca7efd433d4/pandas/plotting/_misc.py#L362): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (102 > 88 characters)




Thanks @Dxin-code for the PR. #33428 is already fixed? issue is closed.
Thanks for the PR but looks like this was already solved in #33099
> Thanks for the PR but looks like this was already solved in #33099

Got it! Thanks for the reminder!
fixup some f-strings
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry
xref #29547 

Labels:
name: Code Style
description: Code style, linting, code_checks
name: Clean

Comments:
agreed with the comments here - I don't think there is anything to be done with this one
Issue 30999 fix
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Hello @Dxin-code! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/tests/arrays/boolean/test_arithmetic.py`](https://github.com/pandas-dev/pandas/blob/27a882dd4bc55e2c86b6ffb298522f8e37478f2b/pandas/tests/arrays/boolean/test_arithmetic.py):

> [Line 26:10](https://github.com/pandas-dev/pandas/blob/27a882dd4bc55e2c86b6ffb298522f8e37478f2b/pandas/tests/arrays/boolean/test_arithmetic.py#L26): [E111](https://duckduckgo.com/?q=pep8%20E111) indentation is not a multiple of four
> [Line 26:10](https://github.com/pandas-dev/pandas/blob/27a882dd4bc55e2c86b6ffb298522f8e37478f2b/pandas/tests/arrays/boolean/test_arithmetic.py#L26): [E113](https://duckduckgo.com/?q=pep8%20E113) unexpected indentation
> [Line 26:13](https://github.com/pandas-dev/pandas/blob/27a882dd4bc55e2c86b6ffb298522f8e37478f2b/pandas/tests/arrays/boolean/test_arithmetic.py#L26): [E225](https://duckduckgo.com/?q=pep8%20E225) missing whitespace around operator
> [Line 34:8](https://github.com/pandas-dev/pandas/blob/27a882dd4bc55e2c86b6ffb298522f8e37478f2b/pandas/tests/arrays/boolean/test_arithmetic.py#L34): [E901](https://duckduckgo.com/?q=pep8%20E901) IndentationError: unindent does not match any outer indentation level


Please can you give all these descriptive titles + fix up pep8? Thanks 
@Dxin-code closing as stale.
DOC: Added check for standard
State: closed

- [ ] xref #32316
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This is my first time contributing, if I made any errors please tell me! I tried to continue what @joybhallaa  was working on. I made a check to see which files are not using the standard documentation.

Labels:

Comments:
@willie3838 Thanks for the PR.

> I tried to continue what @joybhallaa was working on.

#33213 is still open. closing this to avoid conflicting PRs.
TST: Added test case for DataFrame.at
State: closed

- [x] closes #33544
- [x] tests added
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Timezones
description: Timezone data dtype

Comments:
@simonjayhawkins brings up a good point. This is possibly not fully fixed as the expected dtype should not be object but `datetime64[ns, UTC]`.

Interested in fixing this bug @EdAbati? 
Hi @mroeschke and @simonjayhawkins ,

I could try to to give it a go, but I am not super familiar with every internal class and I may need some guidance/a little bit more time.

I have tried to debug the `__setitem__` during `result.at[0, "foo"] = datetime(2000, 1, 2, tzinfo=timezone.utc)`. 
I have noticed that at some point it is checked if the timezones are the same/compatible.
In the specific example of the test, even if the tz are not the same/compatible (one is `None` and the other is 'UTC'), it does not raise the TypeError [here](https://github.com/pandas-dev/pandas/blob/b630cdbc33c6e766cc1bfec857b44b04e9db08f8/pandas/core/arrays/datetimes.py#L644) because TypeError is part of a exception statement.

The item is set using `iLocIndexer` and the function `find_common_type` ([link](https://github.com/pandas-dev/pandas/blob/0570237c663434e09fed5bc61712ead07bb14692/pandas/core/dtypes/cast.py#L1425)) is used to translate the dtype to `object`.

What do you think, would changing `find_common_type` be a good approach in this case?
Or would it be better to raise an error for the timezones mismatch?

I think that behavior is working as expected. The odd part is that essentially the entire column is being changed, so I am not sure why we need to do this data type check.
thanks, `.at` like `.loc` does not do coercing of the entire column, unlike `[]` which does.
my take-away for this is that the behaviour reported in original issue was perhaps the correct behaviour.

The loss of the timezone a result of the scalar being coerced. This is similar to #22740. Agreed in both cases the user 'incorrectly' expected a different behaviour, because the scalar could be coerced.
For this case, numpy will raise an error in the future and would therefore raise. This will be consistent with https://github.com/pandas-dev/pandas/pull/33577#discussion_r410415299

```
>>> df = pd.DataFrame({"foo": [dt.datetime(2000, 1, 1)]})
>>> df.foo
0   2000-01-01
Name: foo, dtype: datetime64[ns]
>>>
>>> df.foo.dtype.type(dt.datetime(2000, 1, 2, tzinfo=dt.timezone.utc))
<stdin>:1: DeprecationWarning: parsing timezone aware datetimes is deprecated; this will raise an error in the future
numpy.datetime64('2000-01-02T00:00:00.000000')
>>>
```

The 'fixed' issue now coerces the entire column from `datetime64[ns]` to `object`. This is an incorrect behaviour for `.at` IMO
ENH: Add index to output of assert_series_equal on category and datetime values
State: closed

- [ ] closes #xxxx
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This is really an extension of #31435, which added index output to the messages produced when series (and DataFrames) are compared.  I failed to notice that categorical values and datetime values were evaluated through a different code path and didn't add the index output for these data types.  Since the original problem (index reordering) could just as easily happen for these types the functionality should be there as well.  Also, it makes the output more consistent.

I added a couple of new tests to surface and check the additional output.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @amilbourne
BUG: set_levels set wrong order levels for MutiIndex
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Added parameter that allows user to reset codes

Labels:

Comments:
If there's a better way to update the codes, please let me know!
BUG: Debug grouped quantile with NA values
State: closed

- [x] closes #33569
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Groupby

Comments:
Closing as this is basically identical to https://github.com/pandas-dev/pandas/pull/33644 now
Reverted cython pin
State: closed

reverts #33534 and closes #33507 

Labels:
name: CI
description: Continuous Integration

Comments:
This might fail until tomorrow, after a new Cython wheel has been uploaded to that server.
Yea makes sense. I'll refresh overnight
--pre in this build is picking up this [Cython alpha](https://pypi.org/project/Cython/3.0a1/) so will have to wait new for alpha before this is potentially fixed.
Yea thanks @alimcmaster1 - let’s come back to this when that happens
DOC: Fix heading capitalization in doc/source/whatsnew - part5 (#32550)
State: closed

-Quite a lot of complicated issues here, need reviewing. Such as 'Groupby' transformed to 'GroupBy', 'I/O' to 'IO', and some others.

- [ ] Modify files v0.18.0.rst, v0.18.1.rst, v0.19.0.rst, v0.19.1.rst, v0.19.2.rst, v0.20.0.rst, v0.20.2.rst, v0.20.3.rst, v0.21.0.rst,  v0.21.1.rst

-File v0.20.0.rst has bad file name as it is actually talking about version v0.20.1.rst

Labels:
name: Docs

Comments:
> Looks good; a few minor nits

Ok changes made as requested.
<samp>
Commenter does not have sufficient privileges for PR 33568 in repo pandas-dev/pandas<br>

</samp>
@jbrockmendel @WillAyd  I made the change
I don't understand what is going wrong during the checks
> Thanks @cleconte987 nice work. The CI is green, I guess you already fixed the problems you mentioned.

No I just merged master and pushed, updated PR, and it worked this time. It happened several times that checks failed and I have no idea why it happens. Maybe server issues, server load or server busy I don't understand why is it failing.
Thanks @cleconte987
CI: Fix jedi deprecation warning for 0.17.0 on IPython
State: closed

- [ ] closes #33567 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration

Comments:
similar to #31323, a fix needs to take place in IPython.
ex small comment, LGTM pending green
Thanks @charlesdong1991 
@Meeseeksdev backport to 1.0.x
PERF: operate on arrays instead of Series in DataFrame/DataFrame ops
State: closed

xref https://github.com/pandas-dev/pandas/pull/32779

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
While you're doing this, want to get the other `.iloc[:, i]` cases within `dispatch_to_series`?
Yes, will do. 

(note, I would also be fine with you including this in #33561, *if* we decide to have a switch between column/blockwise. But it's mainly because you seemed quite hesitant to include something like this, that I thought to do this PR separately) 
can you point to some asv's that change with this? (nice cleanup anyhow)
> can you point to some asv's that change with this? (nice cleanup anyhow)

There is already a `MixedFrameWithSeriesAxis` benchmark that captures this partly:

Running that benchmark specifically, on master:

```
[100.00%] ··· arithmetic.MixedFrameWithSeriesAxis.time_frame_op_with_series_axis1                                                                                                                                ok
[100.00%] ··· ========== ==========
                opname             
              ---------- ----------
                  eq      194±4ms  
                  ne      191±3ms  
                  lt      189±3ms  
                  le      190±4ms  
                  ge      209±50ms 
                  gt      225±20ms 
                 add      4.31±1ms 
                 sub      4.68±1ms 
               truediv    4.64±1ms 
               floordiv   54.8±2ms 
                 mul      4.73±1ms 
                 pow      15.4±2ms 
              ========== ==========
```

and with this branch:

```
[100.00%] ··· arithmetic.MixedFrameWithSeriesAxis.time_frame_op_with_series_axis1                                                                                                                                ok
[100.00%] ··· ========== ============
                opname               
              ---------- ------------
                  eq       68.5±9ms  
                  ne       70.3±9ms  
                  lt       62.7±6ms  
                  le      68.1±10ms  
                  ge       70.5±8ms  
                  gt       70.7±9ms  
                 add       4.44±1ms  
                 sub       5.52±1ms  
               truediv    5.45±0.8ms 
               floordiv    60.7±3ms  
                 mul       4.42±1ms  
                 pow       15.1±4ms  
              ========== ============
```

The comparison ops take the code path I changed here, and those show a consistent speed-up (the arithmetic ones actually take a different code path)
fix
State: closed

- [x] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:

Comments:
f strings
@csaoma can you give your PRs informative titles in the future
> @csaoma can you give your PRs informative titles in the future


Closing - you have a PR open with these changes
CLN: x-array test warnings
State: closed

Remove these warnings:

```
pandas/tests/generic/test_to_xarray.py:102
  /home/travis/build/pandas-dev/pandas/pandas/tests/generic/test_to_xarray.py:102: DeprecationWarning: The default dtype for empty Series will be 'object' instead of 'float64' in a future version. Specify a dtype explicitly to silence this warning.
    s = Series(range(len(indices)), index=indices)
```

https://travis-ci.org/github/pandas-dev/pandas/jobs/675056013

Labels:
name: Clean

Comments:
thanks @alimcmaster1
BUG: tz_localize needs to invalidate freq
State: closed

- [x] closes #30511
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Identified when trying to add `freq` check to `assert_index_equal` for DatetimeIndex and TimedeltaIndex.

Labels:
name: Bug
name: Frequency
description: DateOffsets
name: Timezones
description: Timezone data dtype

Comments:
Think this would close https://github.com/pandas-dev/pandas/issues/30511

Also what about tz_convert?
> Also what about tz_convert?

I'm pretty sure that for Tick-like freqs, tz_convert should always retain freq.  Others are going to be trickier.
Yes, closes #30511
yep thanks. can `tz_convert` change freq? (I think no)
> can tz_convert change freq? (I think no)

I'm pretty sure that for Tick-like freqs, tz_convert should always retain freq. Others are going to be trickier.