BrandonFors/pandas-github-issues huggingface dataset:
[#38390] Bug fix for Datetime64Formatter with values of ndim > 1
State: closed

- [x] closes #38390
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This change fixes a bug in `Datetime64Formatter` that errors with values of ndim > 1. This was discovered when using an ExtensionArray with 2D values of datetime64 and attempting to format/display. The formatter currently allows for 2D values, but will end up returning nested strings instead of a flat list of string. The same error occurs when using the `Datetime64Formatter` directly on 2D values.

The fix is to flatten the values to get a flat list of formatted datetime64 strings, then reshape, and then format a second time over the outer dimension to produce the properly nested list format and the final flat list of strings.

Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Stale

Comments:
Thanks for the review @jbrockmendel , sorry for the delay. I updated with a fix and tests for Datetime64TZFormatter as well, since the same bug happens.
The remaining test failures look like they are unrelated from a network timeout
more importantly, shouldn't this actually be handled on the EA itself?
Thanks for reviewing @jreback , I just had a couple questions.

> more importantly, shouldn't this actually be handled on the EA itself?

This would be preferred, but I could not find a way to do this. When calling `repr()` on a `Series` wrapping my `ExtensionArray`, the formatting goes like this: 
1. `Series` sees the dtype is an `ExtensionType` and ends up in  `ExtensionArrayFormatter._format_strings()`
2. `ExtensionArrayFormatter._format_strings()` converts array to `numpy.ndarray`
3. `Datetime64Formatter` formats the `ndarray` into a list of strings and returns

Is there a way for the ExtensionArray to tie into this somehow?
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
> This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

I am still planning on working this. I noticed there is a similar problem with extension arrays with floating point values of ndim > 1, so I will work on a fix that will hopefully handle both of these cases soon.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
I haven't had the time to get back to this, will close for now and reopen when I can update
REF: consolidate NDFrame._iset_item, _set_item with DataFrame methods
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
nice, yeah we shouldn't have *any* internal function calls like that outside of internals
Updated docstring for DataFrame.plot.bar graph clearify the use of color list
State: closed

- [X] closes #31691 
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
@MarcoGorelli Some of the tests are timing out for some reason. I am not sure why this is the case.
.
CI failure is unrelated

I'll leave this open for a bit in case others have comments
Thanks @ankushduacodes
CLN: C408 Unnecessary dict call, last files #38138
State: closed

- [x] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Hello @UrielMaD! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:































There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-12-09 03:41:56 UTC
BUG: item_cache invalidation on DataFrame.insert
State: closed

- [x] closes #26985
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Something wonky is going on with tm.assert_produces_warning, will see if it shows up on the CI.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance

Comments:
whatsnew added + green
this just puts a warning up
was the bug already fixed?
The bug is fixed by removing the _consolidate_inplace, which is replaced with the warning
ok!
CLN: lighter-weight casting
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
> are there other places we can replace this function?

none as easily as this, but im whittling away
REF: share IntervalIndex.intersection with Index.intersection
State: closed

- [x] closes #38299
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This changes the return dtype of IntervalIndex.intersection when that intersection is empty. 

Labels:
name: Refactor
description: Internal refactoring of code
name: Index
description: Related to the Index class or subclasses

Comments:
nice!
[WIP] ENH: add Pyarrow csv engine
State: closed

- [x] closes #23697
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Picking up #31817. 

I'm putting this up to use CI for debugging. There are a bunch of comments in #31817 that I haven't addressed yet (working on them) - but I'll take feedback at any time

Labels:
name: Enhancement
name: Performance
description: Memory or execution speed performance
name: IO CSV
description: read_csv, to_csv
name: Stale

Comments:
CI green.
@arw2019 Thanks for picking up where I left off. Can you go and rerun the benchmarks to ensure that no performance regressions occurred in the refactor, when this is ready? (Note: speedup should be based on number of cores).

Regarding the other comments, I think it'll be easier to xfail tests if we put all the error checking tests into one file, since those are the ones mostly causing the failures I think.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@arw2019 i think the precursors were all merged, would be nice to get this in.
ok - will rebase (sometime in the next few days - it's a little bit of work now) and ping when ready 
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Just a heads up that I'm going to try to revive this PR sometime soon-ish. 
> Just a heads up that I'm going to try to revive this PR sometime soon-ish.

OK. closing this one as stale.
Working on this rn, currently rebasing. Will open new PR.
BUG FIX- C408
State: closed

- [ ] closes #c408
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Hello @shrutiguptabu! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-12-08 16:25:50 UTC
TST: a weird bug with numpy + DataFrame with duplicate columns
State: closed

- [x] closes #31954
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Picking up #37125

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
thanks @arw2019
PERF: IntervalIndex.isin
State: closed

```
In [1]: import pandas as pd

In [2]: ii = pd.IntervalIndex.from_breaks(range(100000))

In [3]: values = ii[:100]

In [4]: from pandas.core.algorithms import isin

In [5]: %timeit isin(ii, values)
91.3 ms ± 5.24 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  # <-- master
3.84 ms ± 57.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <-- PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Interval
description: Interval data type

Comments:
ok this is fine. can you add a whatsnew note. ping on green.
ping
ENH: column-wise DataFrame.fillna with Series/Dict value
State: closed

- [x] closes #4514
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Picking up #30922

Labels:
name: Enhancement
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Green + addressed comments
Thanks for the PR but it appears this has gone stale. Looks fairly close but let us know if you'd like to continue by merging master and targeting 1.4
BUG: item_cache invalidation
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Refactor
description: Internal refactoring of code

Comments:
lgtm. whats new for 1.3 (just merged) and ping on green 
ping
TST: add message matches to pytest.raises in various tests GH30999
State: closed

- [x] ref [#30999 ](https://github.com/pandas-dev/pandas/issues/30999)
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Files changed: 
pandas/tests/io/pytables/test_complex.py
pandas/tests/io/test_clipboard.py
pandas/tests/plotting/frame/test_frame_subplots.py

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
I tried to follow the contributing guidelines as best as I could. If there's any problem, let me know.

Edit: Looks like some tests didn't pass. @MarcoGorelli can you give me some pointers on how to fix this? 
thanks @JoseNavy
PERF: isin casting integers to object
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

xref https://github.com/pandas-dev/pandas/pull/38279#issuecomment-739973166 cc @jorisvandenbossche 

Labels:
name: Performance
description: Memory or execution speed performance
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
lgtm. do we have / need asv's? do we need a whatsnew note (not sure if this was working in 1.1 at all)
No, this was catched by an asv we already have, and only a regression on master
TYP: pandas/core/missing.py
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@arw2019 this is somewhat old, so maybe simpler to start a new PR, or merge master and update; as always on typing if can do the unambiguous ones first
> @arw2019 this is somewhat old, so maybe simpler to start a new PR, or merge master and update; as always on typing if can do the unambiguous ones first

I'm thinking I'll factor out the easy ones into a separate PR so this is easier to look at and maybe get that merged first

(FWIW this got a lot of review so i'm confident the hard ones here are mostly right at this point)
@arw2019 can you merge master and fixup mypy failures
> @arw2019 this is somewhat old, so maybe simpler to start a new PR, or merge master and update; as always on typing if can do the unambiguous ones first

@arw2019 closing this. can you break this up and open several new smaller PRs
@arw2019 if you add an ignore, cast or assert, or change/delete an existing annotation or change any actual code, put those changes a minimally scoped PR.
REF: avoid accessing private iloc methods
State: closed

I'm pretty sure this is made possible by @phofl's recent fix to only call `_align_series` in `_setitem_with_indexer` when `name != "iloc"`

Labels:
name: Refactor
description: Internal refactoring of code
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Nice!
rewrite dict as literal in tests/io/parser/test_index_col.py
State: closed

for issue # 38138 
rewrite unnecessary dict call as a literal in pandas/tests/io/parser/test_index_col.py

Please let me know if anything needs to be amended, changed or fixed. I'm new to contributing.

- [ ] issue #38138 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Hello @pillemer! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-12-07 04:33:22 UTC
Thanks. Could you fix up the linting errors?
@mroeschke, no problem, done! Is there anything else?
> @mroeschke, no problem, done! Is there anything else?

There's still some failures, could you try again please? 
```
pre-commit run --files pandas/tests/io/parser/test_index_col.py
```
closing as it looks like this one's already been done, sorry about that
BUG: read_csv not converting to float for python engine with decimal sep, usecols and parse_dates
State: closed

- [x] closes #35873
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

self.columns is alread resticted to usecols, while data is not restricted yet, so we have use the initial column indices

Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
ci/checks is failing
Fixed the mypy issues. If we do it this way, we are more type safe than before
this looked good last time, can you merge master
Done
> pandas/tests/io/parser/usecols/__init__.py 

seems to have gotten commited?
These folders were missing __init__ files, so I have added them. Shall I remove them?
> These folders were missing **init** files, so I have added them. Shall I remove them?

nope this is fine, didnt' realize they weren't there
REGR: Fix Index construction from Sparse["datetime64[ns]"]
State: closed

- [ ] closes #35843
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype
name: Sparse
description: Sparse Data Type
name: Index
description: Related to the Index class or subclasses

Comments:
yep later in the day here, so if all backports ok, will start the release in the morning.
@simonjayhawkins great on your schedule of course!
BUG: first("1M") returning two months when first day is last day of month
State: closed

- [x] closes #29623
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Frequency
description: DateOffsets

Comments:
thanks @phofl 
@meeseeksdev backport 1.2.x
@phofl this didn't fix the "2M" case from the issue?

```
>>> import pandas as pd
>>>
>>> pd.__version__
'1.3.0.dev0+42.g36c4d5ccf1'
>>>
>>> x = pd.Series(1, index=pd.bdate_range("2010-03-31", periods=100))
>>>
>>> print(x.first("1M"))
2010-03-31    1
Freq: B, dtype: int64
>>> print(x.first("2M"))
2010-03-31    1
Freq: B, dtype: int64
>>>
```
Unfortunately you are right. Opened #38446 to adress this.
Forgot to add the test concerning this case unfortunately
REF/TYP: Rename ABCIndexClass->ABCIndex and use cast
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Labels:
name: Refactor
description: Internal refactoring of code
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@rhshadrach can you merge master
yep this looks good (just merge master as indicated) and ping on green.
@jreback green.
thanks @rhshadrach 

IIRC there was an issue for this? or maybe just a comment
To my knowledge this https://github.com/pandas-dev/pandas/pull/38055#issuecomment-733966956
CLN: dtypes.cast
State: closed


Labels:
name: Clean
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
reminder @pandas-dev/pandas-core don't explicity tag 1.2 any more, can do right before 1.2 to avoid issues that are not ready for this milestone. if you think it *really* should be included in the rc then tag, but provide a comment.
CLN: use with instead of try finally in `io/stata.py`
State: closed

This is reported by lgtm.com.

Labels:
name: IO Stata
description: read_stata, to_stata
name: Code Style
description: Code style, linting, code_checks

Comments:
hmm do you know why lgtm is reporting this? this is a context manager, maybe not picking it up somehow?

cc @twoertwein 
thank you, looks good!
sorry i was looking at the old. thanks @fangchenli
BUG: read_csv raising IndexError with multiple header cols, specified index_col and no data rows
State: closed

- [x] closes #38292
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Parser returned an empty list instead of a list filled with None, which caused the IndexError.

Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
@jreback green
lgtm, resolved conflict, though there is another PR that might conflict on whatsnew, but will see.
Yeah, thanks for merging. As soon as we got a few messages in there we can vary the order a bit to avoid this
thanks
BUG: BDay() offsetting not behaving as expected
State: closed

- [x] closes  #38046
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Frequency
description: DateOffsets

Comments:
Hello @arw2019, 
I could make the expected have less timestamp by adjusting the date_range to smaller range. Would that be enough? |
Do you have any specific suggestion? This is my first contribution in this project so my proposed solution might not be
the most optimized.
There is only a failing travis check but it does not seem to come from the pr changes. Is there a way to re-rerun only this check?
Also, note before editing the import in the file all the checks were green.
thanks @theoniko
CLN: Implement multiindex handling for get_op_result_name
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

@jbrockmendel 

This would solve the inconsistency in ``get_op_result_name`` for MultiIndexes, but adds a bit more complexity here.
I think this would allow us to remove the now duplicated code in ``intersection``

Labels:
name: MultiIndex
name: Clean

Comments:
Can we do this by overriding get_reconciled_name_object and not get_op_result_name?  id like to avoid the small-but-frequent additional overhead there
pls merge master
Done and Done
LGTM ex @jreback comments
also pls merge master
looks good. is this user visible in any way? e.g you mention: `This would solve the inconsistency in get_op_result_name for MultiIndexes`
No, we did use rename previously which avoided this. But now we are consistent between the intersection methods
thanks @phofl
DEPR: xlwt for writing excel files
State: closed

- [x] closes #26552
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Deprecates both specifying `engine="xlwt"` and the option `io.excel.xls.writer`. Note that a warning is emitted when a user sets the `io.excel.xls.writer` option, but not via the context manager. Whatsnew now reads:

![image](https://user-images.githubusercontent.com/45562402/101409318-bd371980-38ab-11eb-80f4-f64afc259438.png)



Labels:
name: IO Excel
description: read_excel, to_excel
name: Deprecate
description: Functionality to remove in pandas

Comments:
looks ok to me (ex text comments). ping when revised.
Thanks @jorisvandenbossche and @jreback, changes made. I also updated the screenshot in the OP.
TYP: datetimelike
State: closed

use np.putmask in a few places, as it is appreciably more performant than `__setitem__`

Labels:
name: Datetime
description: Datetime data dtype
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
might actually be some perf boosts here due to using putmask
> might actually be some perf boosts here due to using putmask

yep thats the idea
thanks
TYP/BUG: fix passing incorrect type to interpolate_1d
State: closed


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
> any additional tests?

i dont think so.  i checked that we get here with dt64tz but thats about it.
wow typing actually is helping
BUG: Clean intersection and fix resulting names when MultiIndex are equal
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Names were wrong when MultiIndexes where equal and unique with different names. Additional cleanups

cc @jbrockmendel 

Labels:
name: MultiIndex
name: Clean

Comments:
Do not need a whatsnew, cause this was introduced with 7e2aa42c93852bf505c2e34c41ddc8874f04decd
PeriodIndex?
Missed that. Added it and added a test
LGTM cc @jreback
> Do not need a whatsnew, cause this was introduced with [7e2aa42](https://github.com/pandas-dev/pandas/commit/7e2aa42c93852bf505c2e34c41ddc8874f04decd)

nope if in the same version (which this is)
thanks @phofl
BUG: Fix typo in melt value_name futurewarning
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Just removing the duplicated 'i' in "resultiing" of the futurewarning in melt about using `value_name` with the same name as a column the in the original dataframe.  Filing an issue or a whatsnew entry for such a minor change seemed excessive.

Labels:

Comments:
thanks @fmaguire
New dict call fixings #38138
State: closed

xref #38138

File modified: pandas/core/strings/accesory.py

Changed unnecessary dict calls for flake8 (C408) standard:

Rewrite dict() as {}


- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @UrielMaD