BrandonFors/pandas-github-issues huggingface dataset:
CLN/REF: de-duplicate DatetimeTZBlock.setitem
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Move towards avoiding special treatment for pandas-internal EAs.

xref #24020

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
great, eliminating these special cases would be great (assume already on your list)
> great, eliminating these special cases would be great (assume already on your list)

yah, basically all of my current core.internals PRs are aimed towards getting us down to just Block+ExtensionBlock
REF/TYP: define NDFrame numeric methods non-dynamically
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

cc @simonjayhawkins along with #36989 this gets most of the generic.py mypy complaints (notwithstanding complaints about `cls.foo = foo` that can be addressed by changing to `setattr(cls, "foo", foo)`.

it'd be nice to have a less-verbose way of pinning the docstrings.  i haven't had any luck so far at making that work.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
> it'd be nice to have a less-verbose way of pinning the docstrings. i haven't had any luck so far at making that work.

i've spent more time than i'd like to admit looking into this.

tried altering the doc decorator to allow templates but it turns out easier to pass the actually docstring to the decorator instead of the function.

https://github.com/jbrockmendel/pandas/compare/typ-generic-3...simonjayhawkins:docs?expand=1 for a potential solution to any and all. (i've merged master so the diff is not so clear) that sits on top of this PR.

of course, now that the methods are in both DataFrame and Series, don't need the common examples.

> https://github.com/jbrockmendel/pandas/compare/typ-generic-3...simonjayhawkins:docs?expand=1 for a potential solution to any and all. (i've merged master so the diff is not so clear) that sits on top of this PR.

thats a nice touch putting the shared bits into `_logical_func.__doc__`

Did the `axis_descr` get lost somewhere though?
> Did the `axis_descr` get lost somewhere though?

not used in any/all, `.format` does not raise KeyError with extra parameters, only when missing.
merging to unblock a doc-issue, pls followup as needed
REF/TYP: consistent return type for Block.replace
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

move some convert logic out of _replace_coerce to avoid an otherwise-duplicated method

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
very nice
CLN: standardize values coercion in Blocks
State: closed

- [x] closes #19492
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
nice!
Add pandas-genomics to the ecosystem list
State: closed

- [x] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Added [as suggested here](https://github.com/pandas-dev/pandas/pull/36987#pullrequestreview-505157583).

Labels:
name: Docs

Comments:
thanks @jrm5100
BLD: remove blockslider #34014
State: closed

Part of #34014

xref:
https://github.com/ivirshup/pandas/commit/2c17e72a9deffd8e0d8999b88a2e31c0a58c746c
#34997

Impact on groupby
```
       before           after         ratio
     [97877442]       [a82d3773]
     <master^2>       <libreduction-cython>
+     5.07±0.06ms         43.8±1ms     8.65  groupby.Apply.time_scalar_function_single_col
+      15.5±0.2ms        108±0.6ms     6.97  groupby.Apply.time_scalar_function_multi_col
+         453±3ms          508±1ms     1.12  groupby.Apply.time_copy_overhead_single_col
+      1.44±0.03s       1.61±0.01s     1.12  groupby.Apply.time_copy_function_multi_col
-      26.0±0.7ms       23.4±0.2ms     0.90  groupby.Nth.time_frame_nth_any('float64')
```

Impact on index_cached_properties
```
       before           after         ratio
     [97877442]       [a82d3773]
     <master^2>       <libreduction-cython>
+        401±10ns        862±400ns     2.15  index_cached_properties.IndexCache.time_inferred_type('Int64Index')
+        397±10ns        649±200ns     1.64  index_cached_properties.IndexCache.time_is_unique('Int64Index')
+     1.01±0.03μs       1.31±0.2μs     1.30  index_cached_properties.IndexCache.time_values('DatetimeIndex')
+      3.51±0.2μs       4.55±0.8μs     1.29  index_cached_properties.IndexCache.time_shape('CategoricalIndex')
+      1.97±0.2μs       2.30±0.2μs     1.17  index_cached_properties.IndexCache.time_inferred_type('MultiIndex')
+     1.75±0.04μs       2.02±0.2μs     1.15  index_cached_properties.IndexCache.time_shape('DatetimeIndex')
-      1.99±0.3μs       1.58±0.2μs     0.79  index_cached_properties.IndexCache.time_values('UInt64Index')
-      2.33±0.6μs       1.77±0.5μs     0.76  index_cached_properties.IndexCache.time_inferred_type('Float64Index')
```

No significant changes on frame_methods


Labels:

Comments:
hmm, yeah i guess this perf regression is why BlockSlider exists in the first place. 
> hmm, yeah i guess this perf regression is why BlockSlider exists in the first place.

yes... this is tricky
xref #34997
close for now.
CI: isort fixup on 1.1.x
State: closed

isort-5.6.1

Labels:
name: CI
description: Continuous Integration

Comments:
i'll reopen this after some other backports and include black fixups for older black version
Hello @simonjayhawkins! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-10-09 13:23:34 UTC
CLN: Move _aggregate and _aggregate_multiple_funcs to core.aggregation
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Renamed functions by removing underscores and renamed `self` argument to `obj`.

Labels:
name: Clean
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
nice, comment for followon.
REF/BUG/TYP: read_csv shouldn't close user-provided file handles
State: closed

- [x] closes #36980
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

REF/BUG: de-duplicate all file handling in `TextReader` by calling `get_handle` in `CParserWrapper`. ~~When `TextReader` gets a string it uses memory mapping (it is given a file object in all other cases).~~

REF/TYP: The second commit adds a new return value to `get_handle` (whether the buffer is wrapped inside a TextIOWrapper: in that case we cannot close it, we need to detach it (and flush it if we wrote to it)). I made `get_handle` return a typed dataclass `HandleArgs` and made sure that all created handles are in `HandleArgs.created_handles` there is no need to close `HandleArgs.handle` (unless it is created by `get_filename_or_buffer`).

I used asserts for mypy when I'm 100% certain about the type, otherwise I added mypy ignore statements.

In the future it might be good to merge `get_handle` and `get_filename_or_buffer`.

Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
cc @gfyoung 
the failure on windows when reading ./pandas/tests/io/sas/data/test_sas7bdat_2.csv was caused when `encoding` was `None`. `TextIOWrapper` on windows seems to default to charmap instead of utf-8.
Hello @twoertwein! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:













































There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-11-04 02:11:22 UTC
@twoertwein can you merge master and ill take another look
@jbrockmendel rebased. Sorry for the large diff. Most code changes are from changing the return value of `get_handle`.
```
pandas/_typing.py:1:1: F401 'dataclasses' imported but unused
pandas/_typing.py:6:1: F401 'typing.Generic' imported but unused
```

from pre-commit checks
lgtm ping on green.
@jreback  green'ish
i restarted the pre commit
@jreback pre-commit is green
thanks @twoertwein really nice!
CI: xfail intermittently-failing tests
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Usually I avoid `strict=False` like the plague, but these are a PITA recently.

Labels:
name: CI
description: Continuous Integration

Comments:
lgtm. merge on green (also if you can make an issue to revert this)
TYP: sas, stata, style
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
fair, merge on green-ish
followups appreciated here
DOC: Added indexing views to roadmap
State: closed

This adds the indexing copy vs. view proposal to the roadmap (#36195).

Given the length of the proposal, I've included a summary in the main `roadmap.rst`, with a link to the full proposal in a second document. Here's the summary:

The major change from the Google doc shared earlier is the recommendation for "Error on Write" over "Copy on Write". I think the proposal should recommend one over the other, and I think "error on write" is probably the way to go, but I'm fairly uncertain on this point.

cc @pandas-dev/pandas-core 

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Roadmap
description: A proposal for the roadmap.

Comments:
hmm ci/checks are failing
General question about error / copy on write (forgive me if this has been answered elsewhere), but does the same hold when we have a parent / view but modify the parent instead of the view? For example, supposing there's error on write and we have `s = df["a"]`; does setting on `s` raise but changes to `df["a"]` would still silently propagate?
> General question about error / copy on write (forgive me if this has been answered elsewhere), but does the same hold when we have a parent / view but modify the parent instead of the view? For example, supposing there's error on write and we have `s = df["a"]`; does setting on `s` raise but changes to `df["a"]` would still silently propagate?

I'm not sure that we have discussed that???  Today the propagation is both ways, when using `df["a"]`, but doesn't happen with `df[["A"]]`.  I think the proposal is that for a view that is a `Series`, it remains both ways, but for a view that is a `DataFrame`, modifying the underlying `DataFrame` would silently propagate to the view.
Let's try to keep the discussion in the issue. This PR was opened prematurely.

The "modifying the parent" issue is mentioned in the proposal at https://github.com/pandas-dev/pandas/pull/36988/files#diff-8a818e230eb272faebe3cf4b7b164a29e0311a42a09a8bd79d17d43d3249f4d3R231.
Fix test_unstack
State: closed

The test was failing if the ExtensionDtype had an na_value that wasn't equivalent to nan

- [x] closes #36986 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Hello @jrm5100! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-10-08 19:24:05 UTC
I restarted the failed numpy dev build (timed out)
thanks @jrm5100 
> Thanks, I guess it came down to this passing
> 
> ```python
> In [45]: a = pd.Series([1, np.nan], dtype=object)
> 
> In [46]: b = pd.Series([1, pd.NA], dtype=object)
> 
> In [47]: pd.testing.assert_series_equal(a, b)
> ```
> 
> FYI, we're making a list of 3rd-party extension types at https://pandas.pydata.org/pandas-docs/stable/ecosystem.html#extension-data-types, if you want to add pandas-genomics to ecosystem.rst

shouldn't this raise as these are *not* equal
> shouldn't this raise as these are not equal

I think so. Looks like it's come up in https://github.com/pandas-dev/pandas/issues/18463 / https://github.com/pandas-dev/pandas/issues/27613
CLN: move maybe_casted_values from pandas/core/frame.py to pandas/core/dtype/cast.py
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

xref #36876, #27370 (stale PR)

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Clean

Comments:
For something thats only used once, im not sure it is worth moving.  Definitely worth seeing if this can be simplified though, per a couple of TODO comments in there
> Definitely worth seeing if this can be simplified though, per a couple of TODO comments in there

@jbrockmendel Will do! (in a followon)
ping on green-ish (meaning either green or a specific buidl is failing on completely unrelated; though I think @jbrockmendel may have fixed this).
thanks @arw2019 

if you can rebase the typing one on master would be great.
> thanks @arw2019

Thanks @jreback @jbrockmendel for reviewing!

> if you can rebase the typing one on master would be great.

Done
DOC: improve description of the example which dataframe has two rows
State: closed

improve description of the example which dataframe has two rows

- [ ] closes #36970
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Hello @yonashub! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-10-11 18:32:15 UTC
Thanks @yonashub 

In the future, it would be better to create a new branch when submitting PRs, see the contributing guide
REF: de-duplicate skipna logic
State: closed

De-duplicate logic in handling skipna
inside methods any() and all().

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Closing as the logic was actually slightly different in the both methods.
CLN: share to_native_types
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Clean

Comments:
```
Performing static analysis using mypy
pandas/core/internals/blocks.py:2119: error: Definition of "to_native_types" in base class "DatetimeLikeBlockMixin" is incompatible with definition in base class "Block"  [misc]
pandas/core/internals/blocks.py:2365: error: Definition of "to_native_types" in base class "DatetimeLikeBlockMixin" is incompatible with definition in base class "Block"  [misc]
Found 2 errors in 1 file (checked 1057 source files)
```
mypy complaints fixed; unrelated timeout
Looks like this is a blocker (for mypy reasons) for the otherwise-ready branch fixing #32441
REF/TYP: use OpsMixin for logical methods
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
Thanks @jbrockmendel
BUG/API: tighter checks on DTI/TDI.equals
State: closed

- [x] closes #33531
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Also fixes `TimedeltaIndex(Categorical(tdi))` (which DatetimeIndex already gets right)

Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
ci / checks are failing
mypy complaints fixed; unrelated windows+plotting failure
TYP: use OpsMixin for DecimalArray
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
i guess should disable this for now
```
ERROR: /home/runner/work/pandas/pandas/pandas/_libs/tslibs/timezones.pxd Imports are incorrectly sorted and/or formatted.
ERROR: /home/runner/work/pandas/pandas/pandas/_libs/tslibs/offsets.pxd Imports are incorrectly sorted and/or formatted.
ERROR: /home/runner/work/pandas/pandas/pandas/_libs/tslibs/timestamps.pxd Imports are incorrectly sorted and/or formatted.
ERROR: /home/runner/work/pandas/pandas/pandas/_libs/tslibs/ccalendar.pxd Imports are incorrectly sorted and/or formatted.
ERROR: /home/runner/work/pandas/pandas/pandas/_libs/tslibs/conversion.pxd Imports are incorrectly sorted and/or formatted.
ERROR: /home/runner/work/pandas/pandas/pandas/_libs/tslibs/np_datetime.pxd Imports are incorrectly sorted and/or formatted.
ERROR: /home/runner/work/pandas/pandas/pandas/_libs/tslibs/timedeltas.pxd Imports are incorrectly sorted and/or formatted.
ERROR: /home/runner/work/pandas/pandas/pandas/_libs/tslibs/period.pxd Imports are incorrectly sorted and/or formatted.
Check import format using isort DONE
```
> i guess should disable this for now

I thought i saw a PR that addressed that.  if i dont find it ill open one shortly
yah looks like someone already got to the isort thing, so this is green
TYP: core.missing, ops.docstrings, internals.ops, internals.managers, io.html
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @jbrockmendel
DOC: Add GCS as supported filesystem for read_parquet
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: IO Parquet
description: parquet, feather

Comments:
does this have an issue reference? 
No it doesn’t, just based on my observation of the docs.

On Thu, Oct 8, 2020 at 17:14 Jeff Reback <notifications@github.com> wrote:

> does this have an issue reference?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/36958#issuecomment-705827289>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACCFHKZD457WPX63QV7XZZDSJYTUVANCNFSM4SH4RU6Q>
> .
>

thanks @joshtemple
REGR: Allow positional arguments in DataFrame.agg
State: closed

Although `DataFrame.agg` is documented as accepting `func, axis, *args, **kwargs` with `*args` and `**kwargs` passed to `func`, passing positional arguments raises a TypeError.

The reason for this is that the internal call to `self._aggregate` uses a keyword argument (axis) before passing *args and `**kwargs`, and as such the first positional argument is always interpreted as a second specification of `axis`, which raises TypeError.

Prior to commit 433c9007781080658553fbef1a4d0c2813b404c0, TypeErrors were being suppressed, falling back to `self.apply`, which in v1.1.0 turned into an error.

This fixes issue GH-36948.

- [x] closes #36948
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
seems reasonable; the ci / checks are failing.
The failures look unrelated. I've rebased against the current master and I think #36978 might have been the cause of the check failure.
@pganssle Conflict needs resolved.
Thanks @pganssle 
@meeseeksdev backport 1.1.x
REGR: Make DateOffset immutable
State: closed

- [x] closes #36940
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

I'm assuming this is a regression and not behavior we want?

Labels:
name: Datetime
description: Datetime data dtype
name: Regression
description: Functionality that used to work in a prior pandas version
name: Frequency
description: DateOffsets

Comments:
LGTM
thanks @dsaxton 

any ideas about the windows failures ? (obviously not related here)
> any ideas about the windows failures ? (obviously not related here)

Hmm, I'm not sure to be honest
DOC: Update dependency for to_markdown documentation
State: closed

Adding to the documentation that tabulate is required for to_markdown to be executed

- [x] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Docs
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Dependencies
description: Required and optional dependencies

Comments:
Thanks @eyaltrabelsi , but we already raise an import error if `tabulate` isn't installed

```
ImportError: Missing optional dependency 'tabulate'.  Use pip or conda to install tabulate.
```

, does this need to be added to the docstring?
@MarcoGorelli In my opinion it should be part of the documentation as well, but if you think its unneeded we can close this PR :)