BrandonFors/pandas-github-issues huggingface dataset:
FIX: Added small corrections to the test for interpolate limit_area
State: closed

- [ ] closes # nothing to close here
- [ ] tests added / passed
- [ ] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry # not required for this minor fix

Some `tm.assert_series_equal()` have been forgotten in https://github.com/pandas-dev/pandas/blob/2baf788819cd1073d9c15444ebcefcfab8e4b9c8/pandas/tests/series/test_missing.py#L1338

There was also a typo in https://github.com/pandas-dev/pandas/blob/2baf788819cd1073d9c15444ebcefcfab8e4b9c8/pandas/tests/series/test_missing.py#L1374 where `direction` has to be `limit_direction`.

These things have been fixed in this small PR.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @cchwala
API: Disallow NaN in StringArray constructor
State: closed

Closes https://github.com/pandas-dev/pandas/issues/30966

~There were a few ways we could have done this. I opted for this way since it still only requires a single pass over the data to validate the values. Other ways like checking for `np.isnan` after checking `is_string_array` would have required a second pass.~

I changed the implementation in subsequent commits. The basic idea is the same, don't allow NaN in the array passed to StringArray, so that we only make a single pass over the data. We do this by changing `StringValidator.is_valid_null` to only allow NA. This required a small change to `PandasArray._from_sequence`, since previously we relied on creating a temporarily invalid StringArray before doing an inplace `__setitem__` to replace NaNs with NA.

cc @tsvikas.


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Strings
description: String extension data type and string data
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Actually, we don't use `is_string_array` in all that many places, and I don't think it's public, so perhaps I can just make the change to exclude NaN / None. Will explore.
thanks @TomAugspurger
STY: concat strings
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @MomIsBestFriend
BUG: pd.crosstab(s1, s2) handle column index incorrectly when both series have tuple names
State: closed

- [ ] closes #18321
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
ping @jreback 
Thanks @charlesdong1991
BUG: ensure_datetime64ns with bigendian array
State: closed

- [x] closes #29684
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
I think a test that directly hits the user-facing behavior of `pd.Series(big_endian_array)` would be valuable to ensure we don't regress in a future refactor.
What do people think about backporting this? IMO it's appropriate for 1.0.1, but we don't have backports to that setup yet. So 1.0.0 is probably best.

We also need a whatsnew for this, but that depends on the decision for where this should go.
I guess I view 1.0.1 as reserved for regressions introduced in 1.0 which this wouldn't fall under, so very soft preference for 1.1 but not losing sleep in either case
-1 on backporting this; we have very little support for big-endian at all and no testing. In fact, @jbrockmendel how do you know this actually works?
> In fact, @jbrockmendel how do you know this actually works?

Because you can pass a big-endian dtype even on a little-endian system.  The added test does this.
> > In fact, @jbrockmendel how do you know this actually works?
> 
> Because you can pass a big-endian dtype even on a little-endian system. The added test does this.

disagree; we cannot reproduce the error. It would be ok to have the OP try to repro with this patch though. Not objecting to the change, but we don't have *any* testing whatsoever.
> disagree; we cannot reproduce the error

We can.  Try the tests this adds on master.
> > disagree; we cannot reproduce the error
> 
> We can. Try the tests this adds on master.

we don't have tests on a big-endian CI.
> we don't have tests on a big-endian CI.

I'm aware.  I'm saying that when I run the test on my little-endian computer, it fails in master and passes after this PR.
k thanks @jbrockmendel 

not harassing you about testing, just skeptical of big-endian testing generally
BUG: reductions for nullable dtypes should return pd.NA for skipna=False
State: closed

Closes https://github.com/pandas-dev/pandas/issues/30958

Just realized that I should probably check boolean dtype as well

Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
We have a section for NA-related API breaking changes to IntegerArray that this can fall under.
sure
DOC: Move couple of deprecations whatsnew to correct section
State: closed

- [x] closes #30927
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Docs

Comments:
Also corrected issue number to #30610 for `pd.datetime` deprecation which was a new issue created and linked to original issue but not updated in the whats new
Thanks @ryankarlos
[BUG] -1 to the power of pd.NA was returning -1
State: closed

- [x] closes #30956 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
> ##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

I don't know what to make of this error, sorry
@jorisvandenbossche in these situations is a whatsnew note needed?
That's not needed in this case, since it is a new feature, not a bugfix compared to a previous release.
although it's a user reported bug from the rc release. could be worth considering in future having a short section in the whatsnew listing changes since the rc.
I think we're OK without one.
Thanks @MarcoGorelli!
@MeeseeksDev backport 1.0.x
For some reason this got not backported (we will need to check it a bit ...)
ENH: Add Stata 119 writer
State: closed

Add support for writing Stata 119 format files
Rename new writer to StataWriterUTF8 since no longer version-specific
Improve exception message for unsupported files
Fix small issues in to_stata missed in 118

- [X] xref #23573 (already closed)
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry


Labels:
name: Enhancement
name: IO Stata
description: read_stata, to_stata

Comments:
The rename should be easy now since it isn't released yet.
I can squash/rebase if there is a consensus that targetting 1.0.0 is a good idea. 
I think ok to do this for 1.0.0 
is there an issue that this would close? if so can you update the top 
> do you have sufficient tests coverage for all of the added cases?

Coverage of new code (`StataWriterUTF8`) introduced in 1.0.0 is 100%.
> is there an issue that this would close? if so can you update the top

There is no issue open for this.  I cross-referenced #23573 which is the closest, but is already closed. 
Thanks @bashtage
Backport PR #30926 on branch 1.0.x (DOC: Fix whatsnew contributors section)
State: closed

Backport PR #30926: DOC: Fix whatsnew contributors section

Labels:

Comments:
#30952 should be merged and backported and ci re-run before merging this?
That should indeed be backported as well. 
But already merging this, we know the failure, so fine to fix it after merging this (everything else is passing).
REF: handle searchsorted casting within DatetimeLikeArray
State: closed


Labels:
name: Refactor
description: Internal refactoring of code
name: Index
description: Related to the Index class or subclasses

Comments:
> I think changing exception types and error messages should have a whatsnew. also in the case of Exception type, I would regard a change as a breaking change. not sure what others think. Should we have a deprecation for this?

If it were on the Index subclasses I would agree, but for the DTA/TDA/PA (which we dont expect users to use directly) I think we're OK
> If it were on the Index subclasses I would agree, but for the DTA/TDA/PA (which we dont expect users to use directly) I think we're OK

https://dev.pandas.io/docs/reference/api/pandas.array.html has examples on how to construct these.



ping
thanks
DOC: Clarified documentation for convert_dates and use_default_dates params
State: closed

This hopefully clarifies the documentation for the behaviour of the convert_dates and keep_default_dates parameters on the read_json function.

I recently had to use this function and found the existing docs a bit confusing.  I actually created a truth table (below), but this is probably a bit much for the docs.

| convert_dates | keep_default_dates | Result                      |
| ------------- | ------------------ | --------------------------- |
| True          | True               | Convert default date fields |
| True          | False              | Don’t convert anything |
| False         | True               | Don’t convert anything |
| False         | False              | Don’t convert anything |
| List          | True               | Convert listed and default date fields |
| List          | False              | Convert just listed date fields |

I hope you agree that my changes make the behaviour clearer - but perhaps people will disagree...

Labels:
name: Docs
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
Great thanks @amilbourne
TYP: NDFrame.resample
State: closed

This PR types up the method ``NDFrame.resample``.

Also renames the func ``resample`` to ``get_resampler`` and import the groupby types used for type checking in frame.py/series.py undir the ``if TYPE_CHECKING:`` clause.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @topper-123
STY: spaces in wrong place (DO NOT MERGE)
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Hello @MomIsBestFriend! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/tests/io/formats/test_format.py`](https://github.com/pandas-dev/pandas/blob/2269f31daf4a3f097522a5f8ca98cb91a99bcc58/pandas/tests/io/formats/test_format.py):

> [Line 544:69](https://github.com/pandas-dev/pandas/blob/2269f31daf4a3f097522a5f8ca98cb91a99bcc58/pandas/tests/io/formats/test_format.py#L544): [E261](https://duckduckgo.com/?q=pep8%20E261) at least two spaces before inline comment
BUG: Use self._constructor_sliced in df._reduce to respect subclassed…
State: closed

… types. (GH25596)

- [x] closes #25596
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
> Thanks @EmilianoJordan for the PR. will need whatsnew once #30907 is merged.

Subscribed to #30907 to know when I can do this. Also, updated the initial pull request comment to reflect that this has not been completed.
thanks @EmilianoJordan
BUG: Fix MutliIndexed unstack failures at tuple names
State: closed

- [x] closes #19966
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex

Comments:
Hello @charlesdong1991! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-20 20:12:00 UTC
sure! @simonjayhawkins I will keep this in mind and wait for that PR to be merged together with reviews from other maintainers
comment added and tests moved, and also rebased

ping @jreback 
some followup reviews will be appreciated ^^
gentle ping @jreback ^^ thanks 😅 
thanks @charlesdong1991 

very nice!
STY: concat strings that should not be seperated
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
>  best in general to leave cython files untouched and wait for black to support formatting cython.

@simonjayhawkins should I close this PR then?

> > best in general to leave cython files untouched and wait for black to support formatting cython.
> 
> @simonjayhawkins should I close this PR then?

I think OK to merge this once the changed message https://github.com/pandas-dev/pandas/pull/30942#discussion_r365592016 is reverted
ping @simonjayhawkins 
Thanks @MomIsBestFriend
DOC: Fix SS03 docstring error
State: closed

Fixes SS03 errors. 

```
None:None:SS03:pandas.DatetimeIndex.freqstr:Summary does not end with a period
None:None:SS03:pandas.PeriodIndex.freqstr:Summary does not end with a period
pandas/pandas/core/window/indexers.py:34:SS03:pandas.api.indexers.BaseIndexer:Summary does not end with a period
```

Related to #27977, #30733 cc @datapythonista

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes black pandas
- [x] passes git diff upstream/master -u -- "*.py" | flake8 --diff
- [ ] whatsnew entry

When I ran the script, there are others left, but they might be false positives since I can't find them in the codebase.

Labels:
name: Docs

Comments:
Thanks @galuhsahid !
ENH: Add engine keyword to expanding.apply to utilize Numba
State: closed

- [x] closes #30936
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Enhancement
name: Window
description: rolling, ewma, expanding

Comments:
actually ok for 1.0.0; can you add this issues / PR number to the whatsnew note that already existis for rolling.apply
PERF: RangeIndex.get_loc
State: closed

Small simplification gives a small speedup

```
In [2]: rng = pd.Index(range(10**5)) 

In [3]: %timeit rng.get_loc(5.0)                                                
6.38 µs ± 381 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  <-- master
1.15 µs ± 40.9 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  <-- PR

In [5]: %timeit rng.get_loc(5)                                                  
845 ns ± 5.71 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  <--master
In [5]: %timeit rng.get_loc(5)                                                                                                                                                                            
860 ns ± 16.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  <-- PR

In [9]: def foo(): 
   ...:     try: 
   ...:         return rng.get_loc(None) 
   ...:     except KeyError: 
   ...:         pass 
   ...:                                                                         

In [10]: %timeit foo()                                                          
6.72 µs ± 656 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  <-- master
1.11 µs ± 94 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  <-- PR
```

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
How does this change affect `__contains__` ?
> How does this change affect __contains__ ?

Typo, will update title
The timings you showed were then also not relevant? 
> The timings you showed were then also not relevant?

Huh, not sure how that happened.  Will update with get_loc timings, which make a much bigger difference:

```
In [2]: rng = pd.Index(range(10**5)) 

In [3]: %timeit rng.get_loc(5.0)                                                
6.38 µs ± 381 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  <-- master
1.15 µs ± 40.9 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  <-- PR

In [5]: %timeit rng.get_loc(5)                                                  
845 ns ± 5.71 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  <--master
In [5]: %timeit rng.get_loc(5)                                                                                                                                                                            
860 ns ± 16.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  <-- PR

In [9]: def foo(): 
   ...:     try: 
   ...:         return rng.get_loc(None) 
   ...:     except KeyError: 
   ...:         pass 
   ...:                                                                         

In [10]: %timeit foo()                                                          
6.72 µs ± 656 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  <-- master
1.11 µs ± 94 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  <-- PR
```
That looks better :)
thanks @jbrockmendel
ENH: Allow multi values for index and columns in df.pivot
State: closed

- [x] closes #21425
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Hello @charlesdong1991! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-02-02 19:24:06 UTC
would be nice if someone could take a look here, i just rebased ^^
just in case if you'd like to take a look too @WillAyd thanks! ^^
HI, @jreback  and @WillAyd thanks for both of yours reviews, appreciate a lot!!

One thing to mention: i'd like to have signature added as a follow-up to make this clearer, since we have PRs addressing TYPING issues only if you agree. Otherwise, I think I have addressed both of your reviews in the change ^^

Feel free for further reviews! Thanks! 
just updated with latest master
gentle ping @WillAyd for feedback? ^^
gentle ping @jreback ^^
sorry for pushing for a review, I know you are so busy with so many PRs. This PR has been rebased several times, so some feedbacks would be appreciated @jreback 👍 
thanks @charlesdong1991 
very nice
BUG: correct wrong error message in df.pivot when columns=None
State: closed

- [x] closes #30924 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Not at a computer to check but why is this not optional? Wouldn't not specifying it just return a Series?
emm, not sure about this behaviour, but from documentation, it is supposed to return a reshaped `DataFrame`, so probably this is why this is not optional @WillAyd 
OK I think I see the point here. If you can address @simonjayhawkins comments and put a note in for 1.0.1 when available I think should be good
> OK I think I see the point here. If you can address @simonjayhawkins comments and put a note in for 1.0.1 when available I think should be good

@WillAyd maybe i missed the discussion, but the first patch release following 1.0.0 will be for regressions only. This should be 1.1.0?
Yea makes sense. Let's do 1.1 instead (comment still applies)
thanks @WillAyd and @simonjayhawkins I will add a whatsnew note once 1.1 is there
ping @WillAyd @simonjayhawkins 
Thanks @charlesdong1991 lgtm. @WillAyd 
ping @jreback 
ping @WillAyd 
thanks @charlesdong1991
ENH:column-wise DataFrame.fillna and duplicated DataFrame.fillna with Series and Dict 
State: closed

- [x] closes #4514
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

description:
Access "DataFrame" each column, fills up NA values using "Series.fillna" 
and what i found is if dataframe is duplicated, ".fillna" not guarantee filling NA. So i change them can fill NA.

Q. Why access column base not index base?
A. problem of Index base f".illna" is not preserve column's dtype. so i use column base.

Q. Why assign in column new values not use inplace=True ?
A. To avoid chained indexing. If column and index are both duplicated, chained indexing happens, i want to avoid this situation.


Labels:
name: Enhancement
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Stale

Comments:
Hello @proost! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-08-10 17:20:16 UTC
@proost can you fix merge conflict and try to fix CI failure?
@proost is this still active? If yes can you merge master and fix the CI failure?
Closing in favor of  #38352
BUG: loc setitem with missing integer slices does not raise
State: closed

- [ ] closes #26412
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
There are two reasons for updating an old test case `test_loc_setitem_empty_append_raises` in `test_loc.py`:

1. To raise the same error as `.loc.__getitem__` when give slice does not exist

1. The current error says `cannot copy sequence with size 2 to array axis with dimension 0`. This is not in line with what the user is trying to do: to add 2 new rows or columns. The error should be a TypeError because we don't current support (not yet implemented) using integer slice to add new rows / columns.

Hope that sounds convincing.
If not, let me know.
This change has a huge impact on existing test cases. Let me investigate further.
closing this PR in favor of #31840.
BUG30787 fixed unexpected behaviour by removing nullable values
State: closed

- [done ] closes #30787 
Added a simple check not to allow nan values for calculation. Numpy seems to throw error when nan values are passed to searchsorted function.


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Closing as I think stale but ping if you want to pick back up and can address comments / red CI