BrandonFors/pandas-github-issues huggingface dataset:
BUG: Fix replacing in `string` series with NA (pandas-dev#32621)
State: closed

The pd.NA values are replaced with np.nan before comparing the arrays/scalars

- [X] closes #32621 
- [X] tests passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
I think the problem here is actually that the element wise comparison reduces to a scalar:

```python
>>> np.array(["one", "two", np.nan]) == "one"
array([ True, False, False])
>>> np.array(["one", "two", pd.NA]) == "one"
__main__:1: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
False
```

Which then gets caught in some of the functions here

@TomAugspurger or @jorisvandenbossche do you know the backstory to that behavior?
> I think the problem here is actually that the element wise comparison reduces to a scalar:
> 
> ```python
> >>> np.array(["one", "two", np.nan]) == "one"
> array([ True, False, False])
> >>> np.array(["one", "two", pd.NA]) == "one"
> __main__:1: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
> False
> ```
> 
> Which then gets caught in some of the functions here
> 
> @TomAugspurger or @jorisvandenbossche do you know the backstory to that behavior?

Hi @WillAyd, please check my comment https://github.com/pandas-dev/pandas/pull/32890#discussion_r396038591 which further explains why probably that warning is raised.
Hello @chrispe92! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-04-10 11:09:11 UTC
Hi @jreback,

Can you have a look my latest changes? I'm curious to know what you think.
Hi @jreback, I've made all the changes that you suggested.
Other than those, I've also made the following changes:
- Added expected types in both of the functions
- Removed the is_array variables in order to pass the mypy tests.

Can you please check them and let me know what you think? Did I miss anything? Thanks.
Hi @jreback, a note for the bug is now included to the whatsnew 1.1 bug fixes missing section. Can you have a look? Thanks.
thanks @chrispe92 

if you could open a new issue for the dtype preservation on `.replace`, this PR solves the current issue so can address the other later.
> thanks @chrispe92
> 
> if you could open a new issue for the dtype preservation on `.replace`, this PR solves the current issue so can address the other later.

Hi @jreback, related issue is now created (https://github.com/pandas-dev/pandas/issues/33484)
CLN: pandas/_libs/tslibs/nattype.pyx
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

---

This PR is doing basicly two things:

* Getting rid of ```noqa: E128``` comments.

* Unifying use of```util.foo```

in some places there was a use of  ```is_integer_object(foo)``` and in some places there was a use of ```util.is_integer_object(foo)```, now ```util.is_integer_object(foo)``` is being in use instead.

---

Benchmarks:

```
In [1]: from pandas._libs.tslibs.nattype import _make_nat_func 

In [2]: %timeit _make_nat_func("foo", "bar")
120 ns ± 1.01 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each) # Master
117 ns ± 1.47 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each) # PR
```

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Clean

Comments:
thanks @MomIsBestFriend
TST: Replace tm.all_index_generator with indices fixture
State: closed

Inspired by #30999
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`



Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Originally I wanted to implement
> create the indices fixture using `tm.all_index_generator` as suggested [by jbrockmendel in this comment](https://github.com/pandas-dev/pandas/pull/31701#discussion_r378532760)

But then I realised that all places where `tm.all_index_generator` is used would benefit from using `indices` instead.
@jreback CI is green
thanks @SaturnFromTitan
fix bare pytest raises in indexes/datetimes
State: closed

- [ ] ref #30999
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @quangngd
PERF/REF: MultiIndex.copy
State: closed

Makes ``MultiIndex.copy`` call ``MultiIndex._shallow_copy`` rather than the other way around. This is cleaner and let's us copy the existing  ``.cache``, so may give performance boost when operating on copied MultiIndexes:

```python
>>> n = 100_000
>>> df = pd.DataFrame({'a': range(n), 'b': range(1, n+1)})
>>> mi = pd.MultiIndex.from_frame(df)
>>> mi.get_loc(mi[0])  # also sets up the cache
>>> %timeit mi.copy().get_loc(mi[0])
8.57 ms ± 157 µs per loop  # master
57.9 µs ± 798 ns per loop  # this PR
```

Also cleans kwargs from the ``MultiIndex._shallow_copy`` signature. This PR is somewhat related to #32669.


Labels:
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
thanks.
DOC: Fix errors in pandas.DataFrame.melt
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Related to #27977. 

output of `python scripts/validate_docstrings.py pandas.DataFrame.melt`:
```
################################################################################
################################## Validation ##################################
################################################################################


Labels:
name: Docs

Comments:
looks good ping on green
thanks @farhanreynaldo
DOC: Fix errors in pandas.DataFrame.sort_index
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Related to #27977. 

output of `python scripts/validate_docstrings.py pandas.DataFrame.sort_index`:
```
################################################################################
################################## Validation ##################################
################################################################################
```



Labels:
name: Docs

Comments:
ping on green
> 
> 
> ping on green

Sure will do
Thanks @farhanreynaldo
TST: Avoid bare pytest.raises in test_series.py
State: closed

* [x]  ref #30999
 
* [x]  tests added / passed
 
* [x]  passes `black pandas`

* [x]  passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Please note that this file had an error message that was pretty ugly, so I changed
it to only include the class name as I have done for other tests.

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
this is a very tiny api change, but ok.
thanks
REF: collect to_xarray tests
State: closed

I'm split on where to put test_to_xarray.  Since its testing one method, tests/generic/methods/ makes sense.  BUT we could try to reserve that directory for tests that are nicely parametrized over Series/DataFrame, which these are not.  Thoughts?

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
I would create pandas/tests/downstream and move pandas/tests/test_downstream.py to there along with this test file (name is good), maybe even move the xarray test to that (inside test_downstream.py)
ping on green.
> I would create pandas/tests/downstream and move pandas/tests/test_downstream.py to there along with this test file (name is good), maybe even move the xarray test to that (inside test_downstream.py)

I'll give this some thought.  I kind of like test_downstream.py for the specific "dont break downstream" purpose it has no (just asked xarray folks to open a PR with a regression test thatll go there)
hmm needs a rebase
thanks
CLN: Period tests
State: closed

Pretty much re-wrote the comparison tests to match the patterns we use elsewhere, avoided bare pytest.raises.

A bunch of tests use `Period("NaT", freq)` which ends up being very duplicative.  Cut that down a bit and marked a couple tests that belong elsewhere, will move in a separate pass.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Period
description: Period data type

Comments:
thanks
BUG/TST: Add searchsorted tests
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

xref: https://github.com/pandas-dev/pandas/issues/32845

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
LGTM
Carried over to https://github.com/pandas-dev/pandas/pull/32764 since there's overlap
Modification of validate_rst_title_capitalization.py script
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Hello @cleconte987! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`scripts/validate_rst_title_capitalization.py`](https://github.com/pandas-dev/pandas/blob/048fb783f299d3238dbe113744fa2d20b12ec3c3/scripts/validate_rst_title_capitalization.py):

> [Line 77:5](https://github.com/pandas-dev/pandas/blob/048fb783f299d3238dbe113744fa2d20b12ec3c3/scripts/validate_rst_title_capitalization.py#L77): [E265](https://duckduckgo.com/?q=pep8%20E265) block comment should start with '# '
> [Line 77:65](https://github.com/pandas-dev/pandas/blob/048fb783f299d3238dbe113744fa2d20b12ec3c3/scripts/validate_rst_title_capitalization.py#L77): [W291](https://duckduckgo.com/?q=pep8%20W291) trailing whitespace
> [Line 78:5](https://github.com/pandas-dev/pandas/blob/048fb783f299d3238dbe113744fa2d20b12ec3c3/scripts/validate_rst_title_capitalization.py#L78): [E265](https://duckduckgo.com/?q=pep8%20E265) block comment should start with '# '
> [Line 79:5](https://github.com/pandas-dev/pandas/blob/048fb783f299d3238dbe113744fa2d20b12ec3c3/scripts/validate_rst_title_capitalization.py#L79): [E265](https://duckduckgo.com/?q=pep8%20E265) block comment should start with '# '
DOC: Remove latest whatsnew from header
State: closed

In favor of going via the "Release notes" which stays in the header.

Closes https://github.com/pandas-dev/pandas/issues/32748

Labels:
name: Docs
name: Clean

Comments:
Nice, thanks @TomAugspurger 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.0.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 f76763fe741cf5c7e0a4c69faef319e437f475e5
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #32870: DOC: Remove latest whatsnew from header'
```

4. Push to a named branch :

```
git push YOURFORK 1.0.x:auto-backport-of-pr-32870-on-1.0.x
```

5. Create a PR against branch 1.0.x, I would have named this PR:

> "Backport PR #32870 on branch 1.0.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
Update 03_subset_data.rst
State: closed

IIUC, type() will report pandas.core.series.Series to me

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Great thanks for the fix!
TST: add test for non UTC datetime split #14042
State: closed

- [x] closes #14042
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @JDkuba
Fix wrong type checking in concat
State: closed

While the documentation of ``pandas.DataFrame.concat`` specifies that
any mapping of ``Label`` to ``FrameOrSeries`` is acceptable as the first
argument, but the code actually checks for an instance of ``dict``.
This changeset fixes that restriction.

- [x] closes #32863 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
also pls add a test for this (in the OP)
Hello @pbourguignon! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-03-24 17:07:11 UTC
One last question: should I add an entry to the existing whatsnew for 1.1.0?
yes
closing in favor of #32953 
@pbourguignon you have 2 PR for the same issue;. I closed tihs one as the other is more recent (and has the changes here too). If this is not correct pls comments.
> @pbourguignon you have 2 PR for the same issue;. I closed tihs one as the other is more recent (and has the changes here too). If this is not correct pls comments.

That's alright, I just couldn't find how to create the later PR against the earlier one.
PERF: allow to skip validation/sanitization in DataFrame._from_arrays
State: closed

For cases where you know to have valid data (eg you just created them yourself, or they are already validated), it can be useful to skip the validation checks when creating a DataFrame from arrays.

Use case is for example https://github.com/pandas-dev/pandas/pull/32825

From investigating https://github.com/pandas-dev/pandas/issues/32196#issuecomment-600824238

@rth this gives another 20% improvement on the dataframe creation part. Together with https://github.com/pandas-dev/pandas/pull/32856, it gives a bit more than a 2x improvement on the dataframe creation part (once the sparse arrays are created)

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
```
In [1]: arrays = [pd.arrays.SparseArray(np.random.randint(0, 2, 1000), dtype="float64") for _ in range(10000)] 
   ...: index = pd.Index(range(len(arrays[0])))   
   ...: columns = pd.Index(range(len(arrays)))

In [2]: %timeit pd.DataFrame._from_arrays(arrays, index=index, columns=columns)   
119 ms ± 3.52 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [3]: %timeit pd.DataFrame._from_arrays(arrays, index=index, columns=columns, verify_integrity=False)    
98.1 ms ± 713 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```
Very nice, thanks @jorisvandenbossche ! It's good that is applies to extension arrays in general not just sparse frames. Though I guess the use case of a very large number of columns (>10k) is less common outside of sparse.
Actually my comment was more about https://github.com/pandas-dev/pandas/pull/32856, got confused in your multiple performance improvement PRs :) This is nice too! 
I think your comment applies to both ;) 
Nice docstring. 😄 
Doesn't matter too much, can be added either here or there. 
And I would also like to add `verify_integrity` to the benchmark that I am adding in https://github.com/pandas-dev/pandas/pull/32856
SGTM. I think this can go in since CI is passing :)
BUG: Improved error msg
State: closed

- [x] closes #9259
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Hopefully this is a more informative message. The original bug report mentions that the index key also fails for Series but didn't report the error type, so I took my best guess at the reporter's intent in the test.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Error Reporting
description: Incorrect or improved errors from pandas
name: MultiIndex

Comments:
I made some changes. Please let me know if this is what you had mind. Thanks!
I am not sure this is needed any longer as we merged #32520, but pls @a-y-khan if you can test and see if the original issue now works, or we still need something.
closing as stale, if you'd like to update as above, pls ping / open a new PR.
BUG: is_scalar_indexer
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

ATM we go through the wrong path when setting a listlike entry in a single position.

The broader goal: separate out the cases where setitem is inplace vs copying.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Clean

Comments:
Is there an actual bug that this fixes? (do you have an example with indexing that now did the wrong thing?)
> Is there an actual bug that this fixes? (do you have an example with indexing that now did the wrong thing?)

Nothing user-facing, just an unnecessary failed-cast:

```
df = DataFrame(index=[0, 1], columns=[0])
df.iloc[1, 0] = [1, 2, 3]
```

ATM this incorrectly doesnt see the indexer as a scalar-indexer, but does see this (also incorrectly) as an `exact_match`, so goes through the try/except in L906-909.  This is why the try/except is needed in the first place.
REF: implement _with_freq, use _from_sequence less
State: closed

This makes method-chaining a little bit nicer.  

Moving away from using _from_sequence and towards using the constructors where feasible.  This will make it easier to stricten what we accept in these _from_sequence methods.

We also should be inferring freq _less_ in arithmetic methods, but that is a behavior change that will be done separately.

Labels:
name: Refactor
description: Internal refactoring of code
name: Frequency
description: DateOffsets

Comments:
thanks
Backport PR #32840 on branch 1.0.x (DOC: use new pydata-sphinx-theme name)
State: closed

Manual Backport of #32840

Labels:

Comments:
I have basically merged #32833 and #32840, because #32840 allows the build to be built, and #32833 makes the build to be passed, I can't see an option to apply those changes in separate PRs
Whoops, I already merged the other backport PR, sorry (in the assumption it didn't matter much, we will see it being green after it is merged)
@MomIsBestFriend and thanks for the manual backport!
DOC: Fix capitalization among headings in documentation files (#32550)
State: closed

- [x] closes #32550 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Docs
name: Clean

Comments:
@datapythonista this is all of the `.rst` files. Let me know if you notice that some heading in not capitalised properly.
Thanks a lot for working on this @themien.

You will have to split this PR in smaller parts. We can't review 47 files. Every time there is a comment and you fix it, we'd need to review 47 files again, which is too much time consuming for us.

Also, you already got conflicts, which will keep happening often with the size of this PR.

Four or five files at a time is probably reasonable, depending on how long they are.
BUG: Fix segfault in GroupBy.count and DataFrame.count
State: closed

- [x] closes #32841 
- [x] closes #21824
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Groupby
name: Segfault
description: Non-Recoverable Error

Comments:
Hello @tv3141! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-03-29 16:22:27 UTC
So far, this PR does not fix the related https://github.com/pandas-dev/pandas/issues/21824.

One option would be to mask rows `-1` as well in the calling code as well. But I think it would be good to prevent SegFaults in `count_level_2d` itself, especially if bound checking should be disabled for performance. 
> Thanks for taking a look; on first pass I think this code is essentially the same thing though?
> 
> Do you know which array access / index is actually causing the OOB error?

`labels` contains `-1`s when the groupby column contains `NaN`s. 

The calling function `count()` creates `mask` for `-1`s, and `count_level_2d` would ignore masked values. This changed with https://github.com/pandas-dev/pandas/commit/d968aab3da13f68fabbb05f8eba12927d8bc5836#diff-8fa7422077eecd07a4006c11c78fc93aL1265-L1268, since then out-of-bounds access on `counts` can happen.
ping @jreback 
@tv3141 can you fix merge conflict in whatsnew? Otherwise lgtm as well
@jbrockmendel any comments outstanding on this?
this looks good. i think ok to merge.
no objections here
Great thanks @tv3141
DOC: use new pydata-sphinx-theme name
State: closed

Following https://github.com/pandas-dev/pydata-sphinx-theme/issues/102, need to use the new package name

We could actually also start using the released version. But going to merge this soon when CI passes (as I think doc build in other PRs will start to fail)

Labels:
name: Docs

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.0.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 bfa29f10d4115fada208848e6e7756040525e42d
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #32840: DOC: use new pydata-sphinx-theme name'
```

4. Push to a named branch :

```
git push YOURFORK 1.0.x:auto-backport-of-pr-32840-on-1.0.x
```

5. Create a PR against branch 1.0.x, I would have named this PR:

> "Backport PR #32840 on branch 1.0.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
BUG: Fix read_csv IndexError crash for c engine with header=None and 2 (or more) extra columns
State: closed

- [x] closes #26218
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
thanks @roberthdevries