BrandonFors/pandas-github-issues huggingface dataset:
BUG: json.decode fails for nums larger than sys.maxsize
State: closed

- [x] closes #20599 
- [x] tests added
- [ ] tests passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Follow-up to #34473 

Labels:
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
@WillAyd I've made some headway on this. I've fixed `decode_numeric`
https://github.com/pandas-dev/pandas/blob/f5b2e5a9a2431165b1e8002219c5d3962d7bc920/pandas/_libs/src/ujson/lib/ultrajsondec.c#L117
so that for `intValue`, the variable into which we read the incoming int,  when it overflows it's reset rather than throwing a `ValueError`. If I understand this we'd want to store these pre-reset `intValues` in some way and then combine them into a PyLong object in `Object_newBigNum`.

Is that right, and if yes what's the best way to go forward? For example, would we want to use a `wide_char` similarly to what's done for string?
https://github.com/pandas-dev/pandas/blob/f5b2e5a9a2431165b1e8002219c5d3962d7bc920/pandas/_libs/src/ujson/python/JSONtoObj.c#L452
Closing for now. I'll open a new request when I've got something ready for review
@arw2019 Did you ever create a new PR, by any chance? Thanks to your MR #34473, I'm able to write out large numbers to JSON. But, I can't read them back in.
> @arw2019 Did you ever create a new PR, by any chance? Thanks to your MR #34473, I'm able to write out large numbers to JSON. But, I can't read them back in.

I have an unfinished PR in the works, planning to finish it soon-ish (1.2 is probably a stretch, next release more likely) - unless you'd like to do a PR?
BUG: HDFStore unable to create colindex w/o error thrown
State: closed

- [x] closes #28156 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

I found this discussion of the `data_columns` argument to `pd.HDFStore.create_table_index` really useful:
https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#query-via-data-columns

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO HDF5
description: read_hdf, HDFStore

Comments:
thanks @arw2019 very nice!
REF: simplify advance/move/set_length in libreduction
State: closed

Make it so that we set `buf.data` in fewer places.  I think from here its pretty straightforward to avoid setting buf.data at all.

cc @WillAyd.

Labels:
name: Refactor
description: Internal refactoring of code
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
thanks
DOC: Add notes about M and Y to to_timedelata documentation.  (#34968)
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Not sure it closes these issues, but using "M" in the arg argument of to_timedelta caused confusion in #34968 and #27285.  #33094 is somewhat related. I came across another issue I can't find now where there was confusion about "Y" returning a days with times delta

Labels:
name: Docs

Comments:
Can you also link to the relevant issue in the description?
Since MS is valid for milliseconds and contains the character M my PR should probably be changed to something like

> The data to be converted to timedelta.  The character M by itself, e.g. '1M', is treated as minute, not month.  The characters Y and y are treated as the mean length of the Gregorian calendar year - 365.2425 days or 365 days 5 hours 49 minutes 12 seconds.
> Since MS is valid for milliseconds and contains the character M my PR should probably be changed to something like
> 
> > The data to be converted to timedelta.  The character M by itself, e.g. '1M', is treated as minute, not month.  The characters Y and y are treated as the mean length of the Gregorian calendar year - 365.2425 days or 365 days 5 hours 49 minutes 12 seconds.

Makes sense to me - do you want to push such an update?
Hello @selasley! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-09-01 16:05:04 UTC
I think I updated it correctly ( https://xkcd.com/1597/ ).  If not, please let me know what I should have done to update the PR.  Thanks.
Does this close https://github.com/pandas-dev/pandas/issues/34968? If not, can you suggest what needs to be done there?
As far as I can tell, this closes #34968 - @selasley , seeing as you wrote "Not sure it closes these issues", do you agree?
Yes, I believe documenting the current behavior addresses the issue.  Thanks.
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
```python
>>> (pd.Timestamp.today() + pd.to_timedelta('1 Y')) - pd.Timestamp.today()
Timedelta('365 days 05:49:11.999916')
>>> (pd.Timestamp.today() + pd.to_timedelta('1 M')) - pd.Timestamp.today()
Timedelta('0 days 00:00:59.999766')
```

Looks right to me...thanks @selasley !
hmm
did we not fully deprecate M and Y for timedelta conversion? i don't remember if these were actually removed

this added note is very misleading then

if the above is true then pls revert this
@jreback `pd.to_timedelta(1, unit='M')` is deprecated, but not `pd.to_timedelta('1 M')`.

In issue #34968 you'd commented

> I'll repurpose this issue to be a doc one if you'd like to submit a PR to update.
i c

pls create a new issue to deprecate these as strings as well

we do not support these as they are very confusing

yes they work but they are fixed offsets and almost never what you want
sure, done - #36666
TYP: make the type annotations of read_csv & read_table discoverable
State: closed

- [x] closes #25648
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

In master the type of ``read_csv`` and ``read_table`` is 'Any'.  This PR makes the functions signatures discoverable by mypy. Also fixes the error message when wrong kwarg is passed.

Labels:
name: Refactor
description: Internal refactoring of code
name: IO CSV
description: read_csv, to_csv

Comments:
I am on board with this. This probably also closes #25648
So maybe worth pulling the test case from #33023
@gfyoung 
I'm a bit torn on this TBH

I see the benefits to this change (mypy + #25648, #33023), but on the other hand, it does mean we will need to maintain two copies of essentially the same signature.  This function generator setup was meant to alleviate that.

If we're mostly okay with maintaining two copies of the signature, then go for it.  @pandas-dev/pandas-core 
@WillAyd , good idea taking in that test from #33023. I'll do that in the next commit.

@gfyoung , that was the reason I added the test ``test_read_table_same_signature_as_read_csv``, to ensure the signatures don't drift apart :-)
@topper-123 : Ah, that's a fair point.  Okay...it still feels little hacky, but the test does make me feel better about this.
No comments.

On Thu, Jun 25, 2020 at 9:37 AM Jeff Reback <notifications@github.com>
wrote:

> *@jreback* approved this pull request.
>
> any comments @TomAugspurger <https://github.com/TomAugspurger> or
> @jorisvandenbossche <https://github.com/jorisvandenbossche>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/34976#pullrequestreview-437559627>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKAOIVCUXXW5OVFSKHFMHTRYNOJDANCNFSM4OHB6EIQ>
> .
>

Thanks @topper-123
TST: update incorrectly defined fixtures
State: closed

handles the deprecationwarning messages and allows "pytest pandas" to proceed. (see issue #34970)

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Hello @pizzathief! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/tests/extension/json/array.py`](https://github.com/pandas-dev/pandas/blob/ebd36cdd66e8d91cf9530626464296668c0bb9c3/pandas/tests/extension/json/array.py):

> [Line 14:42](https://github.com/pandas-dev/pandas/blob/ebd36cdd66e8d91cf9530626464296668c0bb9c3/pandas/tests/extension/json/array.py#L14): [E241](https://duckduckgo.com/?q=pep8%20E241) multiple spaces after ','
> [Line 16:38](https://github.com/pandas-dev/pandas/blob/ebd36cdd66e8d91cf9530626464296668c0bb9c3/pandas/tests/extension/json/array.py#L16): [E241](https://duckduckgo.com/?q=pep8%20E241) multiple spaces after ','

* In the file [`pandas/tests/groupby/test_whitelist.py`](https://github.com/pandas-dev/pandas/blob/ebd36cdd66e8d91cf9530626464296668c0bb9c3/pandas/tests/groupby/test_whitelist.py):

> [Line 94:1](https://github.com/pandas-dev/pandas/blob/ebd36cdd66e8d91cf9530626464296668c0bb9c3/pandas/tests/groupby/test_whitelist.py#L94): [E302](https://duckduckgo.com/?q=pep8%20E302) expected 2 blank lines, found 1
> [Line 102:1](https://github.com/pandas-dev/pandas/blob/ebd36cdd66e8d91cf9530626464296668c0bb9c3/pandas/tests/groupby/test_whitelist.py#L102): [E305](https://duckduckgo.com/?q=pep8%20E305) expected 2 blank lines after class or function definition, found 1

* In the file [`pandas/tests/indexes/datetimes/test_tools.py`](https://github.com/pandas-dev/pandas/blob/ebd36cdd66e8d91cf9530626464296668c0bb9c3/pandas/tests/indexes/datetimes/test_tools.py):

> [Line 1544:1](https://github.com/pandas-dev/pandas/blob/ebd36cdd66e8d91cf9530626464296668c0bb9c3/pandas/tests/indexes/datetimes/test_tools.py#L1544): [E302](https://duckduckgo.com/?q=pep8%20E302) expected 2 blank lines, found 1
> [Line 1549:1](https://github.com/pandas-dev/pandas/blob/ebd36cdd66e8d91cf9530626464296668c0bb9c3/pandas/tests/indexes/datetimes/test_tools.py#L1549): [E302](https://duckduckgo.com/?q=pep8%20E302) expected 2 blank lines, found 1

* In the file [`pandas/tests/series/test_analytics.py`](https://github.com/pandas-dev/pandas/blob/ebd36cdd66e8d91cf9530626464296668c0bb9c3/pandas/tests/series/test_analytics.py):

> [Line 1794:14](https://github.com/pandas-dev/pandas/blob/ebd36cdd66e8d91cf9530626464296668c0bb9c3/pandas/tests/series/test_analytics.py#L1794): [W291](https://duckduckgo.com/?q=pep8%20W291) trailing whitespace
> [Line 1796:1](https://github.com/pandas-dev/pandas/blob/ebd36cdd66e8d91cf9530626464296668c0bb9c3/pandas/tests/series/test_analytics.py#L1796): [E302](https://duckduckgo.com/?q=pep8%20E302) expected 2 blank lines, found 1
REF: dont consolidate in BlockManager.equals
State: closed

Avoid side-effects.

Side-note: with EA.equals recently added to the interface, should we update array_equivalent to use it?  ATM array_equivalent starts by casting both inputs to ndarray.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
> Side-note: with EA.equals recently added to the interface, should we update array_equivalent to use it?

Probably, yes
> With this change, we no longer use the Block.equals methods?

I think this is correct, will double-check and update.
gentle ping
This had a large performance regression in https://pandas.pydata.org/speed/pandas/index.html#frame_methods.Equals.time_frame_float_equal?commits=77f4a4d673e2d2b3f7396a94818771d332bd913c. Is that expected?
Some slowdown can probably expected for this case since it went from block-wise to column-wise (while having a consolidated block). 

But I think a large part of the slowdown is due to `array_equivalent` being much slower for certain dtypes right now. For example, the FloatBlock had a custom `equals` method not using array_equivalent but:

```    
def equals(self, other) -> bool:
        if self.dtype != other.dtype or self.shape != other.shape:
            return False
        left, right = self.values, other.values
        return ((left == right) | (np.isnan(left) & np.isnan(right))).all()
```

and this is quite a bit faster:

```
In [5]: a = np.random.randn(N)      

In [6]: %timeit ((a == a) | (np.isnan(a) & np.isnan(a))).all()    
5.28 µs ± 366 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

In [8]: %timeit pd.core.dtypes.missing.array_equivalent(a, a)    
57.7 µs ± 2.2 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
```
I suppose that a similar "fastpath" could be added to `array_equivalent`, though?

Also for DatetimeTZBlock, there was an override of `equals` for performance:

```
    def equals(self, other) -> bool:
        # override for significant performance improvement
        if self.dtype != other.dtype or self.shape != other.shape:
            return False
        return (self.values.view("i8") == other.values.view("i8")).all()
```
I think the thing to do here is optimize array_equivalent.
see also #32339
ENH: GH34946 Check type of names argument to `read_csv`, `read_table`…
State: closed


- [x] closes #34946 
- [x] tests passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

- ~~One test failed (TestTSPlot.test_ts_plot_with_tz) but it's not related to what I changed.~~
- ~~Does it need a whatsnew entry?~~

Labels:
name: Bug
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO CSV
description: read_csv, to_csv

Comments:
I used `is_list_like` to check the type which does not consider strings as lists, would that be a problem? 
In the tests whereever `names` argument was passed, it was a list. The doc kind of implies it expects a list too, but the type is specified as `array-like` which I don't think is accurate because a python list is a perfectly valid argument for that but it isn't `array-like`.
Can you add a test for this? I think can go in pandas/tests/io/parser/test_common.py
Will do
Provided tests pass, lgtm
thanks @MJafarMashhadi
DOC: Clarify in pandas.DataFrame.drop that a copy is returned when inplace = False.
State: closed

Clarify in `pandas.DataFrame.drop` that a copy is returned when `inplace = False`.

- [x] closes #33451  
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Thanks @marinomaria
PERF: Fixed cut regression, improve Categorical
State: closed

This has two changes to address a performance regression in
cut / qcut.

~1. Avoid an unnecessary `set` conversion in cut.~ (reverted for now)
2. Aviod a costly conversion to object in the Categoriacl constructor
   for dtypes we don't have a hashtable for.

```python
In [2]: idx = pd.interval_range(0, 1, periods=10000)
In [3]: %timeit pd.Categorical(idx, idx)
```

```
# 1.0.4
10.4 ms ± 351 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

# master
256 ms ± 5.85 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

# HEAD
53.2 µs ± 1.26 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
```

And for the qcut ASV

```
# 1.0.4
58.5 ms ± 3.13 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

# master
134 ms ± 9.5 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

# HEAD
53.6 ms ± 1.06 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

Closes https://github.com/pandas-dev/pandas/issues/33921

Labels:
name: Performance
description: Memory or execution speed performance
name: Regression
description: Functionality that used to work in a prior pandas version
name: Categorical
description: Categorical Data Type
name: cut
description: cut, qcut

Comments:
thanks @TomAugspurger
CLN: Removed unnecessary variable call
State: closed

- [x] closes #34934
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Clean

Comments:
> Can you prepend `CLN` to the pull request title?

Yes done, just for my learning, why are we doing that?
> > Can you prepend `CLN` to the pull request title?
> 
> Yes done, just for my learning, why are we doing that?

It's just to make it clear what a PR does (e.g. this one cleans up some code)
Thanks!
DOC: Demonstrate custom rolling indexer with Businessday
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Docs
name: Window
description: rolling, ewma, expanding

Comments:
thanks @mroeschke 

possibly an example in the docs proper .rst using this would be nice as a followup.
BUG: exponential moving window covariance fails for multiIndexed DataFrame
State: closed

- [x] closes #34440 
- [x] tests added\tests passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: MultiIndex
name: Window
description: rolling, ewma, expanding

Comments:
Thanks @arw2019 nice PR!
BUG: reset_index doesn't preserve dtype on empty frame with MultiIndex
State: closed

- [x] closes #19602
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: MultiIndex

Comments:
Hello @rhshadrach! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-06-23 11:23:20 UTC
very nice @rhshadrach keep em coming!
ENH: Add NDFrame.format for easier conversion to string dtype
State: closed

- [x] closes #17211
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This adds a ``format`` method to DataFrame and Series. This is useful for data transformation.

This method allows/makes it easier to do more complex conversion from arbitrary dtypes to ``string`` series, including combining several columns in a DataFrame to make the string series. For example we can now do this conversion quite easily:

```python
>>> df = pd.DataFrame({
...     'state_name': ['California', 'Texas', 'Florida'],
...     'state_abbreviation': ['CA', 'TX', 'FL'],
...     'population': [39_512_223, 28_995_881, 21_477_737],
...     }, index=[1, 2, 3])
>>> df
   state_name state_abbreviation  population
1  California                 CA    39512223
2       Texas                 TX    28995881
3     Florida                 FL    21477737

>>>  df.format("{state_name:<10} ({state_abbreviation}): {population:,}")
1    California (CA): 39,512,223
2    Texas      (TX): 28,995,881
3    Florida    (FL): 21,477,737
dtype: string
```

I still need to update text.rst, but would like feedback on this first, as this is a bit different than discussed in #17211. In that issue we e.g. only discussed a ``format`` method for ``Series``, while this also adds it for ``DataFrame``. In #17211 I also aired the idea of allowing series methods in the format string. I think that is technically quite difficult, so is not part of this PR.

Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: API Design

Comments:
@topper-123 can you indicate a bit what is different compared to the discussions on the issue? 
@jorisvandenbossche , I've added to the beginning comment, so the the rationale is collected in one place.
Thanks for the update

> Just thinking out loud but would it not be more useful to apply this formatting to the columns returning a like-shaped data frame instead of a series?

Although I like the example you show with combining multiple columns, on second thought I think I agree with @WillAyd that applying the formatting to each column separately might be what users would expect by default. 

In any case, I think it can certainly also be useful. For example if you have a numerical DataFrame and want to format all columns with `"{:,.2f}"`. 

So a keyword to toggle between both might be a good option (detecting it from the format specification is probably a bit brittle?)

@topper-123 can you resolve conflicts and address comments
Closing as I think this is stale but ping @topper-123 if you'd like to pick back up
TST: disallow bare pytest raises
State: closed

xref #30999 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Changes for:
-    pandas/tests/frame/methods/test_assign.py
-    pandas/tests/frame/methods/test_at_time.py
-    pandas/tests/frame/methods/test_between_time.py
-    pandas/tests/frame/methods/test_first_and_last.py
-    pandas/tests/frame/methods/test_interpolate.py
-    pandas/tests/frame/methods/test_replace.py
-    pandas/tests/frame/test_query_eval.py


Belongs to https://github.com/pandas-dev/pandas/issues/30999

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean

Comments:
thanks @boweyism
HDFStore append_to_multiple with min_itemsize
State: closed

- [x] closes #11238
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: IO HDF5
description: read_hdf, HDFStore

Comments:
thanks @VirosaLi very nice
Remove exploded variable
State: closed

- [x] closes #34934
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:

Comments:
@TomAugspurger would you mind to trigger a rerun here as well? Btw, I'm not sure why my commits of the other branch are shown here..

> Btw, I'm not sure why my commits of the other branch are shown here..

This is probably because you created this new branch from the other one. In general, I usually do
```
git checkout -b <new-pr-branch>
git fetch upstream master
git reset --hard upstream/master
```
before starting to work on a new PR to make sure this doesn't happen

> would you mind to trigger a rerun here as well?

If you rebase onto upstream/master, drop the unrelated commits, and force push, it should re-run
@MarcoGorelli I created a new PR, because this branche was messed up. I will close this one, see: https://github.com/pandas-dev/pandas/pull/34949