BrandonFors/pandas-github-issues huggingface dataset:
Remove Scatter and Hexbin from Series plot documentation
State: closed

- [x] closes #38976
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] Remove scatter and hexbin entry from https://github.com/pandas-dev/pandas/blob/v1.2.0/pandas/plotting/_core.py#L603-L1708


Labels:
name: Docs
name: Visualization
description: plotting

Comments:
this is fine. there *is* a more comprehensive soln to this but it is more involed in not showing the access for those plot types for the underlying object (and then dynamically generating the doc-string), but out of scope.
> this is fine. there _is_ a more comprehensive soln to this but it is more involed in not showing the access for those plot types for the underlying object (and then dynamically generating the doc-string), but out of scope.

Can you provide a reference for this?
> > this is fine. there _is_ a more comprehensive soln to this but it is more involed in not showing the access for those plot types for the underlying object (and then dynamically generating the doc-string), but out of scope.
> 
> Can you provide a reference for this?

you would to look in pandas/plotting/_core and see how the accessor is registered, basically trace it.
thanks @1nF0rmed
REGR: errors='replace' when encoding/errors are not specified
State: closed

- [x] closes #38989
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Should 1.3 use `errors='replace'` when no `encoding/errors` are specified or use `errors=None` (strict)?

Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
I would say replace only when encoding is Not set and inferred to utf-8. 

This should go back to 1.2.x
Should a follow-up PR intentionally revert this PR for 1.3? Personally, I would want to know when `read_csv` fails to read certain characters (`errors = None` by default).
I don't think that content from skipped rows should matter and it should raise no error if the rest of the file is ok. Maybe you could add a test withouth skiprows, this raises on 1.1.5 which is good and should keep raising
I thought `errors` is exposed in `read_csv`. It isn't. In that case it makes sense to keep `'replace'` as the default.

I don't think it is feasible to ignore encoding errors only for skipped rows. (We could read everything in binary mode but we would need to decode it to determine line endings.) 
This errors parameter is Not exposed 
yes, it isn't exposed. Sorry, that is what I meant to say :)
thanks @twoertwein 
@meeseeksdev backport 1.2.x
Something went wrong ... Please have  a look at my logs.
Thanks
ENH: Add table-wise numba rolling to other agg funcions
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Timings for a wide table
```
import pandas as pd
import numpy as np

df = pd.DataFrame(np.random.rand(10, 10**5))

roll_single = df.rolling(2, method="single")
roll_table = df.rolling(2, method="table")

%timeit roll_single.mean()

4.92 s ± 463 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

roll_single.mean(engine="numba", engine_kwargs={"nopython": True, "nogil": True, "parallel": True})

%timeit roll_single.mean(engine="numba", engine_kwargs={"nopython": True, "nogil": True, "parallel": True})

5.72 s ± 430 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

roll_table.mean(engine="numba", engine_kwargs={"nopython": True, "nogil": True, "parallel": True})

%timeit roll_table.mean(engine="numba", engine_kwargs={"nopython": True, "nogil": True, "parallel": True})

10.3 ms ± 1.23 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)	
```

Labels:
name: Enhancement
name: Window
description: rolling, ewma, expanding

Comments:
lgtm. test are failing :-< ping on green
All green @jreback. Needed to mark some of the numba tests as slow. 
Sorry if this is not the appropriate place for asking this, but will this be usable with `.aggregate`? Like  `df.rolling('1y', on='Timestamp').aggregate([np.mean, np.std, np.sum], engine='numba', engine_kwargs={'nopython': True, 'parallel': True})`. Thanks in advance.
@8549 unfortunately no. PR's to support `aggregate` would be appreciated!
DOC: elaborate on copies vs in place operations in comparison docs
State: closed

<img width="782" alt="Screen Shot 2021-01-06 at 12 15 05 AM" src="https://user-images.githubusercontent.com/86842/103731907-437e7280-4fb4-11eb-9e6d-e702f117656c.png">

- [ ] ~~closes #xxxx~~
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] ~~whatsnew entry~~


Labels:
name: Docs

Comments:
also pls merge master
thanks !
DOC: add more sections to spreadsheet comparison
State: closed

[Preview (link to PDF on Google Drive)](https://drive.google.com/file/d/1uUyTQyEAX3F6h4EJqKz7Mz4KehGBnAig/view?usp=sharing)

This pull request gets closer to full parity with [SAS](https://pandas.pydata.org/pandas-docs/stable/getting_started/comparison/comparison_with_sas.html)/[STATA](https://pandas.pydata.org/pandas-docs/stable/getting_started/comparison/comparison_with_stata.html) comparison pages by adding the Data Input/Output through Merging sections. It still needs Missing Data and GroupBy, but wanted to get this in while I was at a good stopping place. Each section was done in its own commit, if it's easier to review that way.

---

- [x] ~~closes~~ part of https://github.com/pandas-dev/pandas/issues/38990
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] ~~whatsnew entry~~

Labels:
name: Docs

Comments:
thanks @afeld love the documentation! keep em coming
Fix bug on master
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

The banklist file was removed and replaced with banklist.csv. This causes failures on master.

This is just a temporary fix for the user guide.

cc @jreback


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
thanks for fixing @phofl 
@meeseeksdev backport 1.2.x
BUG: Datetimelike equality comparisons with Categorical 
State: closed

- [ ] closes #30699 
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Datetime
description: Datetime data dtype
name: Categorical
description: Categorical Data Type

Comments:
Hello @ftrihardjo! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-01-08 01:13:09 UTC
thanks @ftrihardjo
BUG: MultiIndex.intersection duplicating nans in result
State: closed

- [x] xref #38623
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

This also aligns the multiindex implementation with the base implementation a bit more.

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: MultiIndex

Comments:
excellent!
REGR: Bug fix for ExtensionArray groupby aggregation on non-numeric types
State: closed

- [X] closes #38980 
- [x] tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Groupby
name: Regression
description: Functionality that used to work in a prior pandas version
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Exception handling that checks for this particular message in order to fallback to non-cython impl is here https://github.com/pandas-dev/pandas/blob/master/pandas/core/groupby/groupby.py#L1047

I can work on adding a unit test for this
need tests
@BryanCutler this is a small example that I was testing that worked before and is failing now, so you can use that as a base for writing tests if you like

```
In [1]: df = pd.DataFrame({'key': ['a', 'a', 'b', 'b'], 'val': ['a', 'b', 'c', 'd']}, dtype="string")

In [2]: df.groupby("key").agg({'val': 'first'})  # or 'min'
Out[2]: 
    val
key    
a     a
b     c
```

Specifying `'sum'` doesn't work in case of the "string" dtype, since it raises an error (because for now we don't allow "sum" on a string type, something we probably should). But "sum" apparently takes a slightly different route and doesn't try the cython_agg.

Also with the decimal test extension type that is included in our tests for such testing purposes, it can be reproduced:

```
from pandas.tests.extension.decimal import DecimalArray, make_data
df = pd.DataFrame({'key': ['a', 'a', 'b', 'b'], 'val': DecimalArray(make_data()[:4])})
df.groupby("key").agg({'val': 'min'})
```

Thanks for the code samples @jorisvandenbossche.  I tried to add a base extension test to cover more types and looks good for all except decimal and json, which give the following error:

```
E           AssertionError: ExtensionArray are different
E           
E           Attribute "dtype" are different
E           [left]:  object
E           [right]: decimal
```
https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=51924&view=logs&j=2d7fb38a-2053-50f3-a67c-09f6e91d3121&t=449937cc-3d50-56b5-5662-e489f41f1268&l=177

Not sure if it's a problem with wrapping back up the extension array, or maybe these tests need a specialization. I'll look into it a little later..
@jreback I addressed the feeback and tests have passed. Please take another look when you can, thanks!
thanks @BryanCutler very nice!
@meeseeksdev backport 1.2.x
Thanks @BryanCutler !
Thank you all for reviewing!
BUG: MultiIndex.union dropping duplicates from result
State: closed

- [x] xref #38745
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

This more or less sits on top of #36299. The current base-_union implementation works only for sorted indexes correctly if indexes containing duplicates. Hence I've only added tests for sorted indexes, 

Labels:
name: MultiIndex

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Thx, code itself is good, but this depends on #40862, after this is merged we can remove the sort_values in the tests. Will run asvs afterwards to check for performance implications
wait for #40862 right?
Yes 
Unfortunately this looks like a pretty high performance penalty

```
       before           after         ratio
     [84d9c5ed]       [f77120cc]
     <multiindex_union~1^2>       <multiindex_union>
+        77.4±2ms          159±4ms     2.05  multiindex_object.SetOperations.time_operation('non_monotonic', 'int', 'union')
+      33.4±0.5ms         57.9±1ms     1.73  multiindex_object.GetLoc.time_large_get_loc_warm
+         102±1ms          152±2ms     1.49  multiindex_object.SetOperations.time_operation('non_monotonic', 'string', 'union')
+      99.4±0.9ms          144±1ms     1.45  multiindex_object.SetOperations.time_operation('monotonic', 'string', 'union')
+         520±4ms         640±20ms     1.23  multiindex_object.SetOperations.time_operation('non_monotonic', 'datetime', 'union')

```
where is the perf diff coming from now? 
The Index union for duplicates is not that fast
ok this is fine if this slowdown only occurs when we have duplicates (which i think is the case)
I'll check the benchmarks
Unfortunately the slowdown is for also for non monotonic indexes. We could do it with only dispatching to Index.union in case of duplicates or nans, but this is makes it more complicated unfortunately.

```
       before           after         ratio
     [0158382a]       [692272cb]
                      <multiindex_union>
+         685±5ms         887±20ms     1.30  multiindex_object.SetOperations.time_operation('non_monotonic', 'datetime', 'intersection')
```
@phofl can you merge master
Done
thanks @phofl
DOC: remove is_lexsorted from MultiIndex docstring
State: closed

`is_lexsorted` has been deprecated (https://github.com/pandas-dev/pandas/pull/38701)

- [x] closes https://github.com/pandas-dev/pandas/issues/38953
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: MultiIndex
name: Deprecate
description: Functionality to remove in pandas

Comments:
thanks @kylekeppler
TST: update pre-commit config to only exclude extension from bare pytest.raises check
State: closed

With #38920 I eliminated all instances of `pytest.raise` without `match=msg` in pandas/tests/computation and pandas/tests/io. #38799 was happening around the same time and missed that they were fixed. So this closes the loop and now only pandas/tests/extension needs to be excluded from the linting check.

I don't think the bare `pytest.raise`s in pandas/tests/extensions will be removed. They are in a pretty complex inheritance hierarchy and reused for many different types of errors and error messages. So I propose that this PR closes #30999.

- [x] closes #30999 
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks!
TST: add missing iloc label indexing tests
State: closed

- [x] closes #38967
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
thanks @vangorade
BUG: Timedelta(td64_out_of_bounds) silently overflowing
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Timedelta
description: Timedelta data type

Comments:
Updated to avoid using cast_from_unit (which goes through python-space)

working on a separate branch to implement tslibs.units that will, among other things, try to share the ndarray/scalar ensure-nanos functions
updated + green
REF: move functions out of _testing/__init__
State: closed


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
BC we don’t have __all__

On Mon, Jan 4, 2021 at 4:18 PM Jeff Reback <notifications@github.com> wrote:

> *@jreback* commented on this pull request.
> ------------------------------
>
> In pandas/_testing/__init__.py
> <https://github.com/pandas-dev/pandas/pull/38961#discussion_r551645232>:
>
> > @@ -55,6 +39,22 @@
>      Series,
>      bdate_range,
>  )
> +from pandas._testing._io import (  # noqa:F401
>
> umm why are these ignored?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/38961#pullrequestreview-561436961>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB5UM6CAIQ743IMYOMEU353SYJLFBANCNFSM4VT2H3HA>
> .
>

Pet peeve for overlap with stdlib names

On Mon, Jan 4, 2021 at 4:17 PM Jeff Reback <notifications@github.com> wrote:

> *@jreback* requested changes on this pull request.
> ------------------------------
>
> In pandas/_testing/_io.py
> <https://github.com/pandas-dev/pandas/pull/38961#discussion_r551644894>:
>
> > @@ -0,0 +1,405 @@
> +import bz2
>
> why _io.py? this is *already* a private module.
> ------------------------------
>
> In pandas/_testing/_random.py
> <https://github.com/pandas-dev/pandas/pull/38961#discussion_r551644965>:
>
> > @@ -0,0 +1,48 @@
> +import string
> +
> +import numpy as np
>
> same comment
> ------------------------------
>
> In pandas/_testing/_warnings.py
> <https://github.com/pandas-dev/pandas/pull/38961#discussion_r551644987>:
>
> > @@ -0,0 +1,174 @@
> +from contextlib import contextmanager
> +import re
>
> same
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/38961#pullrequestreview-561436538>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB5UM6HNABZZN22RJKZIIDTSYJLCFANCNFSM4VT2H3HA>
> .
>

> BC we don’t have __all__
> […](#)
> On Mon, Jan 4, 2021 at 4:18 PM Jeff Reback ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In pandas/_testing/__init__.py <[#38961 (comment)](https://github.com/pandas-dev/pandas/pull/38961#discussion_r551645232)>: > @@ -55,6 +39,22 @@ Series, bdate_range, ) +from pandas._testing._io import ( # noqa:F401 umm why are these ignored? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <[#38961 (review)](https://github.com/pandas-dev/pandas/pull/38961#pullrequestreview-561436961)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5UM6CAIQ743IMYOMEU353SYJLFBANCNFSM4VT2H3HA> .

should add :->
> should add

i think once we get the implementations out of this file
thanks followon comment
DOC: move API breaking "check_freq" section from v1.2.1rst to v1.1.0.rst
State: closed

and add reference to 1.1.0 whats new update in v1.2.1.rst

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

This moves the doc notes for `assert_series_equal` and `assert_frame_equal` from the 1.2.1 whats new to the 1.1.0 whats new and adds a small reference to the section in the 1.1.0 page in the 1.2.1 page. This came up in #38471 and makes more sense to me than the current approach, but happy to close if current state is preferred.


Labels:
name: Docs

Comments:
yeah this is better. of course there is no visibility of this 'change' as it happened in the past, but now documented.

thanks @kylekeppler 
@meeseeksdev backport 1.2.x
Something went wrong ... Please have  a look at my logs.
CLN: Unify number recognition tests for all parsers
State: closed

- [x] closes #38926
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Minor regex improvements.
Is a fixture the right thing to do here? 


Labels:
name: IO CSV
description: read_csv, to_csv
name: Clean

Comments:
very nice @phofl
DOC: minor tweaks to formatting on SQL comparison page
State: closed

Adding code formatting, missing punctuation, etc. No changes of substance.

- [ ] ~~closes #xxxx~~
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] ~~whatsnew entry~~

Labels:
name: Docs
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
thanks @afeld
API: honor copy=True when passing dict to DataFrame
State: closed

- [x] closes #32960
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

xref #34872 cc @TomAugspurger used the test_dict_nocopy you wrote but it ended up pretty mangled

Labels:
name: API - Consistency
description: Internal Consistency of API/Behavior
name: Copy / view semantics

Comments:
I am leary of this consolidation (or rather non-consolidation). it is going to be quite fragile. why is consolidation tied to the dict copy at all?
> I am leary of this consolidation (or rather non-consolidation). it is going to be quite fragile. why is consolidation tied to the dict copy at all?


@TomAugspurger can you weigh in on why this is worthwhile
I think just having the `copy` keyword apply to all `data` inputs, rather than just ndarrays.
ok i think fine to merge. any comments @pandas-dev/pandas-core 
@jbrockmendel conflict if you can merge master
rebased + green
lgtm. unless comment by @pandas-dev/pandas-core will merge tom.
@jbrockmendel can you rebase. i think this is ok to merge.
rebased + green
Sorry for the late reaction here (I looked at it earlier but at that point I thought it was just about honoring the keyword, not changing the default).

To be clear, I am fully on board with the code changes to honor the keyword also for dict input (it's very useful to be able to create a DataFrame from a dict of arrays without making a copy). But why can't we keep the current default of copying (for dict input), and then people can now actually use `copy=False` to avoid a copy? (in practice that means the default would become `copy=None`, so we can know the user passed ``copy=False`` themselves)

That allows the behaviour we want to make possible, but without being a breaking change. 

> That allows the behaviour we want to make possible, but without being a breaking change.

works for me.  id also be OK with calling this a bugfix as its clearly wrong behavior, but it has been there for a while.  either way.  cc @TomAugspurger @jreback?
Yeah, if possible (and IIRC it is possible), we'd want to have `pd.DataFrame(data)` not change behavior for dict or ndarray. But `pd.DataFrame(data, copy=False)` would change for dict `data` (that's the bug fix)`.
Changing the default breaks a handful of tests, so ive got some troubleshooting to do.

I'm not wild about de-facto having "copy=False is the default, except for if data is a dict"
@jbrockmendel can you indicate when this is ready for a new look? (you pushed a commit with "requested edits", but it's not clear whether it's ready or not)
updated with a much nicer handling of copy in `__init__`
@jbrockmendel can you do a final merge of master here to be sure it's all passing with latest master?

@jreback all good for you? (your "approve" is from before the behaviour changed to preserve the copy=False behaviour for dicts)
hmm, this looks to be failing a bunch of things
thanks @jbrockmendel this has been a long road, and @TomAugspurger for the original version.