BrandonFors/pandas-github-issues huggingface dataset:
BUG: _nsorted incorrect with duplicated values in index (#13412)
State: closed

 - [X] closes #13412
 - [X] tests added / passed
 - [X] passes ``git diff upstream/master | flake8 --diff``
 - [X] whatsnew entry

`DataFrame.nlargest` and `DataFrame.nsmallest` does not behave correctly when there's duplicate values in the index. 

Any feedback is appreciated!

Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
I am not sold on this impl. Can you see if there are perf tests (asv) for this, and give a check.
There wasn't an existing benchmark for `DataFrame.nlargest`. I wrote my own and I got an `after/before` ratio of 1.37, so there is some speed regression.

Do you have any performance recommendations @jreback if I use the `merge` method? Or should I look into a new solution? 
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14707?src=pr) is 85.27% (diff: 100%)
> Merging [#14707](https://codecov.io/gh/pandas-dev/pandas/pull/14707?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

```diff
@@             master     #14707   diff @@
==========================================
  Files           144        144          
  Lines         50946      50949     +3   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43445      43447     +2   
- Misses         7501       7502     +1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [c0e13d1...d03be72](https://codecov.io/gh/pandas-dev/pandas/compare/c0e13d1bccd4a783486eba8cc769db48a7875de8...d03be72c56a8eea3754121b98c9dae70a45a9b0c?src=pr)
can you add similar tests for series when duplicates are there the algo might work already, idk.

further I would move ``_nsorted`` to ``algorithms``. maybe rename ``select_n`` -> ``select_n_series`` and call this one ``select_n_frame``
the perf is fine. this is a pretty cheap operation.
Tests passed for Series & reorganized the code @jreback 
(Looks like a Travis-CI job failed due to connectivity)
ping @jreback. Thanks for your help! I'll also look into reorganizing the nlargest/nsmallest code.
trivial change (and needs rebase). ping on green.
ping @jreback. Thanks!
thanks!
BUG: Categorical.searchsorted(): use provided categorical order
State: closed

 - [x] closes #14522
 - [x] tests added / passed
 - [x] passes ``git diff upstream/master | flake8 --diff``
 - [x] whatsnew entry

Previously, it used lexical order instead of the provided categorical
order.

Tests updated accordingly.

Closes https://github.com/pandas-dev/pandas/issues/14522

Documentation also needs to be updated. 
Could you please review? If it's OK I'll append doc update to this pull request.
Thanks!

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Categorical
description: Categorical Data Type

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14697?src=pr) is 84.57% (diff: 100%)
> Merging [#14697](https://codecov.io/gh/pandas-dev/pandas/pull/14697?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **<.01%**

```diff
@@             master     #14697   diff @@
==========================================
  Files           144        144          
  Lines         51057      51059     +2   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43180      43183     +3   
+ Misses         7877       7876     -1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [f1cfe5b...86b42d0](https://codecov.io/gh/pandas-dev/pandas/compare/f1cfe5b4eb614f9b4e201e764e9bd25f2389b355...86b42d08b9da389d05a83495c7d3059c9cc81b2c?src=pr)
add a whatsnew note in 0.19.2
can you rebase
lgtm. minor doc-changes. pls also review categorical.rst if any changes are needed.
thanks!
Sorry I was unavailable for the last two weeks..
Thank you!
BUG: Fix move_into_mutable_buffer for python 3.6.
State: closed

In python 3.6, the CALL_FUNCTION handling was updated. One change is
that when calling a C function from a python function python now counts
the reference owned by the argument tuple. This means that move was
always seeing objects with two references instead of the expected
one.

Python 3.6 also removed a copy in the argument tuple when *unpacking
functions. This means that if a user does:

tuple = (create_string(),)
move_into_mutable_buffer(*tuple)

where create_string() creates a string object with one reference then we
will fail to raise a BadMove even though the user could later retrieve
that string with tuple[0]. There is no way to detect this case so this
patch adds a warning to the docstring advising against star unpacking.

xref: #14679

I played around with removing the extra reference that was added in 3.6 but it looks like playing with borrowed refs everywhere will be a bit tricky. This change should clear things up for 3.6 while continuing to work for older versions. In 3.6 you __could__ get a shared mutable string from this but you need to try pretty hard for it.

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14695?src=pr) is 85.20% (diff: 100%)
> Merging [#14695](https://codecov.io/gh/pandas-dev/pandas/pull/14695?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **<.01%**

```diff
@@             master     #14695   diff @@
==========================================
  Files           143        143          
  Lines         50787      50787          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43273      43274     +1   
+ Misses         7514       7513     -1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [f26b049...d1e8b1b](https://codecov.io/gh/pandas-dev/pandas/compare/f26b049786624ed983f2718687c23e3f1adbb670...d1e8b1b2654b878619638003aa1f90677675725f?src=pr)
ping: tests passing now.
thanks @llllllllll
ENH: Allow updating GBQ table schemas
State: closed

 - [x] tests added / passed
 - [x] passes ``git diff upstream/master | flake8 --diff``
 - [x] whatsnew entry


Labels:

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14694?src=pr) is 85.24% (diff: 12.00%)
> Merging [#14694](https://codecov.io/gh/pandas-dev/pandas/pull/14694?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **0.04%**

```diff
@@             master     #14694   diff @@
==========================================
  Files           143        144     +1   
  Lines         50787      50941   +154   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43273      43426   +153   
- Misses         7514       7515     +1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [f26b049...7056a7a](https://codecov.io/gh/pandas-dev/pandas/compare/f26b049786624ed983f2718687c23e3f1adbb670...7056a7ac9546dd81ce983ae845e231e5a0b7f550?src=pr)
Looks like the 1 failing test was because of a conda repo issue.

what is the purpose of this? is there an issue for this change?
@jreback There isn't a tracked issue.  I was running into an issue where I wanted to append to existing GBQ tables but either had a new column or were missing a few legacy columns and that caused the append to fail.
cc @parthea thoughts?
status of this?
I haven't had a chance to make the changes.  I'll try to get them done this week.
@joshowen this has migrated to https://github.com/pydata/pandas-gbq, feel free to open issue / PR there.
BUG: Fix for json lines issue with backslashed quotes
State: closed

 - [x] tests added / passed
 - [x] passes ``git diff upstream/master | flake8 --diff``
 - [x] whatsnew entry

This is an additional fix to:
https://github.com/pandas-dev/pandas/pull/14429
https://github.com/pandas-dev/pandas/pull/14391

Labels:
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14693?src=pr) is 85.20% (diff: 100%)
> Merging [#14693](https://codecov.io/gh/pandas-dev/pandas/pull/14693?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **<.01%**

```diff
@@             master     #14693   diff @@
==========================================
  Files           143        143          
  Lines         50787      50787          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43273      43274     +1   
+ Misses         7514       7513     -1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [f26b049...9908a7c](https://codecov.io/gh/pandas-dev/pandas/compare/f26b049786624ed983f2718687c23e3f1adbb670...9908a7c1ecbf0329f54025a200ca1c74fc88d62c?src=pr)
This fixed an edge case, but ultimately there are a bunch more.  I'm moving away from using this to the following code:
```
        from io import StringIO
        import jsonlines
        with StringIO() as buf:
            writer = jsonlines.Writer(buf)
            writer.write_all(ujson.loads(s))
            writer.close()
            buf.seek(0)
            return buf.read()
```

Their solution is fairly simple, but I'm not comfortable enough to update the vendored ujson package.
https://github.com/wbolster/jsonlines/blob/master/jsonlines/jsonlines.py

I've added the following PR to speed it up a bit:
https://github.com/wbolster/jsonlines/pull/24
@jreback I think the right way to do this is to use jsonlines or to build its functionality into ujson rather than trying to transform the json formatted output.  What do you think?
jfyi, `StringIO.getvalue()` is typically used to obtain the value as a a string, instead of `.seek(0)` and `.read()`.
also, there is no need to use `StringIO` as a context manager, while it does make sense to use `jsonlines.Writer` as a context manager:

``` python
import io
import jsonlines

buf = io.StringIO()
with jsonlines.Writer(buf) as writer:
    writer.write_all(ujson.loads(s))
return buf.getvalue()
```
@joshowen 

> @jreback I think the right way to do this is to use jsonlines or to build its functionality into ujson rather than trying to transform the json formatted output. What do you think?

IIRC from the original issue, @aterrel and I had discussed this. Though its pretty performant now, the correct approach is to put it in the custom ujson code that pandas uses. That is somewhat more involved (though probably pretty straightforward).
DEPR: Patch to_dense behaviour for sparse.
State: closed

Patches the following for `to_dense`:

1) Fix `SparseArray.to_dense` documentation to refer to `SparseArray` and not `SparseSeries`.

2) Deprecate the fill parameter in `SparseArray.to_dense`, as that parameter was not being respected.

3) Deprecate the sparse_only parameter in `SparseSeries.to_dense`, as that parameter is inconsistent with the `to_dense` API we want, which is no parameters.

Closes #14647.

Labels:
name: API Design
name: Sparse
description: Sparse Data Type
name: Deprecate
description: Functionality to remove in pandas

Comments:
lgtm

## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14686?src=pr) is 85.21% (diff: 100%)
> Merging [#14686](https://codecov.io/gh/pandas-dev/pandas/pull/14686?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

```diff
@@             master     #14686   diff @@
==========================================
  Files           143        143          
  Lines         50796      50800     +4   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43285      43288     +3   
- Misses         7511       7512     +1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [880de30...ad7da32](https://codecov.io/gh/pandas-dev/pandas/compare/880de3054c7ce3a6c48e295837207787dc73350f...ad7da322303251dcd06c0531d39f37cbfd1d5171?src=pr)
lgtm. tough travis is not passing.
@jreback : One of the build machines appears to have broke for a reason unrelated to this PR.  Are you able to restart the build on your end?
@jreback : Everything is passing.  Ready to merge if there is nothing else.
thanks!
BUG: fix pickling of Custom offsets in 3.6
State: closed

 xref #14679

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14685?src=pr) is 85.29% (diff: 100%)

> Merging [#14685](https://codecov.io/gh/pandas-dev/pandas/pull/14685?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #14685   diff @@
==========================================
  Files           140        140          
  Lines         50706      50708     +2   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43247      43250     +3   
+ Misses         7459       7458     -1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [748000d...8ad212c](https://codecov.io/gh/pandas-dev/pandas/compare/748000d220c3c7d4daddf5811e10a5acdfce49ea...8ad212c92a86aadffe0fb3b2af394775c221310d?src=pr)
TST: more 3.6 test fixing
State: closed

xref #14679 

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14684?src=pr) is 85.28% (diff: 80.00%)

> Merging [#14684](https://codecov.io/gh/pandas-dev/pandas/pull/14684?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

``` diff
@@             master     #14684   diff @@
==========================================
  Files           140        140          
  Lines         50706      50717    +11   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43247      43256     +9   
- Misses         7459       7461     +2   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [748000d...66aca46](https://codecov.io/gh/pandas-dev/pandas/compare/748000d220c3c7d4daddf5811e10a5acdfce49ea...66aca467b3aa3eaee8d89fc7f2a4a7838e2ae92c?src=pr)
BUG: Avoid AmbiguousTimeError on groupby
State: closed

closes #14682 



Labels:
name: Bug
name: Groupby
name: Timezones
description: Timezone data dtype

Comments:
Could you add your example from the issue as a test? Also a release note in the whatsnew file.

it seems I've broken something.... I'll try to fix.

## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14683?src=pr) is 85.20% (diff: 100%)
> Merging [#14683](https://codecov.io/gh/pandas-dev/pandas/pull/14683?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will decrease coverage by **0.08%**

```diff
@@             master     #14683   diff @@
==========================================
  Files           140        143     +3   
  Lines         50706      50790    +84   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43248      43275    +27   
- Misses         7458       7515    +57   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [45543ec...d90afaf](https://codecov.io/gh/pandas-dev/pandas/compare/45543ec22bb01d5c46eff9384491d218cffddd64...d90afaf01a148a5cea15250dbe28617095a84163?src=pr)
lgtm.

cc @rockg 
Looks fine.
thanks @j-santander 

keep em' coming!
ERR: fix exception propogation for datetime parsing functions, noted in python 3.6
State: closed

 closes #14561

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
so after this PR: https://travis-ci.org/jreback/pandas/jobs/176738690

get a small number of failures (and bunch of deprecation warnings). but looks like this fixes the xref issue.

@jreback Thanks!

Just a question, what is the reason for the change in setup.py?

changes to that file (.pxd) were not in the build chain - it's a dependency so should have been
TST: Test aggregation over arrays (#3788)
State: closed

 - [x] closes #3788
 - [x] tests added / passed
 - [x] passes ``git diff upstream/master | flake8 --diff``


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby

Comments:
you are picking up the commit for the pivot table PR. this should be separate

@mroeschke specifically, the workflow should be
- sync your master with upstream `git fetch upstream; git rebase upstream/master` from your master branch
- checkout a new branch for each change `git co -b array-agg`

You should just be able to `git rebase -i upstream/master` and delete the commit with the pivot table changes and force push.

Ah didn't notice this picked up the commit from my other PR. Thanks for the guidance! I'll decouple it later tonight. 

## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14675?src=pr) is 85.21% (diff: 100%)

> Merging [#14675](https://codecov.io/gh/pandas-dev/pandas/pull/14675?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #14675   diff @@
==========================================
  Files           143        143          
  Lines         50787      50787          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43273      43276     +3   
+ Misses         7514       7511     -3   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [f26b049...01dce9d](https://codecov.io/gh/pandas-dev/pandas/compare/f26b049786624ed983f2718687c23e3f1adbb670...01dce9d82e7fa751d24355b9c6eb9aa6a548cb88?src=pr)

I checked out the travis-ci build logs and it doesn't look like my change directly caused the build to fail (to my knowledge). Is there anything I need to address on my end?

EDIT: Reran the tests, and they are now passing 

thanks!
raise ValueError if usecols has same size as but doesn't exist in headers (#14671)
State: closed

fixes #14671 

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO CSV
description: read_csv, to_csv

Comments:
- needs tests
- add whatsnew note in 0.19.2

## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14674?src=pr) is 85.29% (diff: 100%)
> Merging [#14674](https://codecov.io/gh/pandas-dev/pandas/pull/14674?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **0.51%**

```diff
@@             master     #14674   diff @@
==========================================
  Files           145        140     -5   
  Lines         51090      50723   -367   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          43315      43263    -52   
+ Misses         7775       7460   -315   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [0d3ecfa...0b15cd1](https://codecov.io/gh/pandas-dev/pandas/compare/0d3ecfa8ba3930b688a6781c7f84138bc80da41a...0b15cd179ceb33794da82224929a8a0b9a7585df?src=pr)
can you rebase
can you rebase
@GGordonGordon : 

1) Rebase onto `master` - your patch will need to be changed for this be merge-able.

2) I agree with @jorisvandenbossche that you do not need `self._reader.header`.  You can use `self.names` AFAICT, so apply your current logic with `self.names`.  The reason is that `self.names` should IINM contain the column names of the final output.  Thus, you should be able to use that regardless of whether you read the `header` OR provide `names`.

3) Note that the Python parser does not raise if you provide out of bounds indices:
~~~python
>>> data = 'a,b\n1,2'
>>> read_csv(StringIO(data), usecols=[0, 2], engine='python')
  a
0 1
~~~
Given the scope of your patch (which applies to both out-of-bounds indices and non-existent names), you will have to also make sure the Python engine raises similarly in this situation for consistency.
pls comment if youd like to update / rebase and continue
BUG: Columns and Index share the same numpy object underneath when pd.DataFrame.cov is used 
State: closed

 - [ ] closes #14617 
 - [ ] tests added / passed
 - [ ] passes ``git diff upstream/master | flake8 --diff``
 - [ ] Index now copies the columns array instead of using the same object in
covariance and correlations stats methods of dataframe


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
- needs tests; you can put next to the other .cov tests
- add a whatsnew note in 0.19.2

## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/14667?src=pr) is 85.28% (diff: 100%)

> Merging [#14667](https://codecov.io/gh/pandas-dev/pandas/pull/14667?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #14667   diff @@
==========================================
  Files           140        140          
  Lines         50693      50695     +2   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43234      43236     +2   
  Misses         7459       7459          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [3f523f3...428cb37](https://codecov.io/gh/pandas-dev/pandas/compare/3f523f3652602c7105194b19b72ff30bbff47258...428cb374f0686313749a2841b76b866c3e9982b0?src=pr)

can you update (whatsnew in 0.20.0)
can you add a test and whatsnew for 0.20.0 (bug fix)
closing as stale. please ping if you'd like to continue.