BrandonFors/pandas-github-issues huggingface dataset:
ERR: fix error message for to_datetime
State: closed

- [x] ~~closes #xxxx~~ There is no issue for this.
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

Fixes the error message when the argument to `to_datetime` doesn't match the format specified.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13450?src=pr) is **84.28%**

> Merging [#13450](https://codecov.io/gh/pydata/pandas/pull/13450?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #13450   diff @@
==========================================
  Files           138        138          
  Lines         50929      50929          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42925      42926     +1   
+ Misses         8004       8003     -1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [f752886...988048d](https://codecov.io/gh/pydata/pandas/compare/f7528866e07048f051f46f621ca76947730c7d32...988048dcfd2a50b29677c255dd0fd2deacf238c9)

thanks!
BUG: Rolling negative window issue fix #13383
State: closed

- [x] closes #13383 
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

Added functionality in validate function of Rolling to ensure that window size is non-negative.


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13441?src=pr) is **84.32%**

> Merging [#13441](https://codecov.io/gh/pydata/pandas/pull/13441?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #13441   diff @@
==========================================
  Files           138        138          
  Lines         51068      51072     +4   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43065      43069     +4   
  Misses         8003       8003          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [b06bc7a...26c9b2d](https://codecov.io/gh/pydata/pandas/compare/b06bc7ab10552079b5300f53ae9458b5b1583402...26c9b2d2f57a3d474ea33baba28e6e83509baf22)

thanks!
PERF: float hash slow in py3
State: closed

closes #13166, closes #13335

Using exactly the approach suggested by @ruoyu0088

significant changes in asv below

```
     before     after       ratio
-     8.88s    78.12ms      0.01  indexing.float_loc.time_float_loc
-    13.11s    78.12ms      0.01  groupby.groupby_float32.time_groupby_sum
```

Factor of 10 smaller benches

```
     before     after       ratio
-  171.88ms    43.29ms      0.25  indexing.float_loc.time_float_loc
-     1.42s    11.23ms      0.01  groupby.groupby_float32.time_groupby_sum
```


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
wow those were slow before, can you make 10x smaller in the asv (so the slow ones don't take as much time). you should still have the same ratio.

@jreback - I've updated this.  It took a while to figure out, but the issue here isn't actually the time of the python hash function, it's that in py3 it returns a `Py_ssize_t` (was `long`), and our khash keys are 32 bit, so the truncation is causing the collisions.  I updated the asv at the top with the smaller benches - because it's a collision issue the ratios aren't both the same, but still show the issue.

``` python
In [50]: a = np.arange(1000000)
    ...: ind = pd.Float64Index(a * 4.8000000418824129e-08)

In [51]: hashes = np.array([hash(x) for x in ind])

In [52]: len(hashes), len(pd.unique(hashes))
Out[52]: (1000000, 1000000)

In [53]: truncated = hashes.view('int32')[::2]

In [54]: len(truncated), len(pd.unique(truncated))
Out[54]: (1000000, 524288)
```

thanks @chris-b1 yeah, I briefly looked at the hash code in python itself and it was doing a lot of things to guarantee for example hashes of fractions and such. Yeah this is simpler.

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13436?src=pr) is **84.28%**

> Merging [#13436](https://codecov.io/gh/pydata/pandas/pull/13436?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **0.05%**

``` diff
@@             master     #13436   diff @@
==========================================
  Files           138        138          
  Lines         50805      50929   +124   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42796      42926   +130   
+ Misses         8009       8003     -6   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [62b4327...3aec078](https://codecov.io/gh/pydata/pandas/compare/62b4327c9b60d890b1d404cddf3a4244254346a8...3aec078d48670c6e3db6838f8b41667bf8b1088f)

ty sir!
BUG: df.pivot_table: margins_name ignored when aggfunc is a list
State: closed

- [ ] closes #13354 
- [ ] tests added / passed
- [ ] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Could you also add a test ensuring that the margins have the correct names? Thanks.

Sure thing!

Chris Mazzullo
On Jun 13, 2016 11:25 PM, "Tom Augspurger" notifications@github.com wrote:

> Could you also add a test ensuring that the margins have the correct
> names? Thanks.
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> https://github.com/pydata/pandas/pull/13435#issuecomment-225770371, or mute
> the thread
> https://github.com/notifications/unsubscribe/AF-H8__ES0GttIBx9u98G2eGa8LqbHalks5qLh84gaJpZM4I01I4
> .

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13435?src=pr) is **84.32%**

> Merging [#13435](https://codecov.io/gh/pydata/pandas/pull/13435?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **0.09%**

``` diff
@@             master     #13435   diff @@
==========================================
  Files           138        138          
  Lines         50805      51072   +267   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42796      43069   +273   
+ Misses         8009       8003     -6   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [62b4327...6017e25](https://codecov.io/gh/pydata/pandas/compare/62b4327c9b60d890b1d404cddf3a4244254346a8...6017e252e81e2754e21b04e71d6c4226bf410016)

thanks!
ENH: PeriodIndex now accepts pd.NaT
State: closed

- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

Related to #12759. `PeirodIndex` can be created from list/array which contains `pd.NaT`.

currently it raises:

```
pd.PeriodIndex([pd.NaT])
# ValueError: freq not specified and cannot be inferred from first element

pd.PeriodIndex([pd.Period('2011-01', freq='M'), pd.NaT])
# AttributeError: 'NaTType' object has no attribute 'ordinal'
```


Labels:
name: API Design
name: Period
description: Period data type

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13430?src=pr) is **84.25%**

> Merging [#13430](https://codecov.io/gh/pydata/pandas/pull/13430?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #13430   diff @@
==========================================
  Files           138        138          
  Lines         50810      50805     -5   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          42809      42805     -4   
+ Misses         8001       8000     -1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [07761c5...5990de6](https://codecov.io/gh/pydata/pandas/compare/07761c519154a6ed8a9345512476152226499ba6...5990de6ae425dc6b59495418c8b38daeceaa5285)

lgtm. ping on green.

ty sir!
BUG: in _nsorted for frame with duplicated values index
State: closed

- [x] closes #13412
- [ ] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug

Comments:
what doesn't pass tests ?

You must fix test to pass `flake8` check. 

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13428?src=pr) is **84.23%**

> Merging [#13428](https://codecov.io/gh/pydata/pandas/pull/13428?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #13428   diff @@
==========================================
  Files           138        138          
  Lines         50805      50810     +5   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42796      42801     +5   
  Misses         8009       8009          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [62b4327...9e47bbe](https://codecov.io/gh/pydata/pandas/compare/62b4327c9b60d890b1d404cddf3a4244254346a8...9e47bbe9afffa7e7fb1a539352114bfa81127b44)

`git diff upstream/master | flake8 --diff` will help you track down the changes you have to make.

I think a simple soln will work here

```
(Pdb) p self.loc[ser.index].head(n).sort_values(columns, ascending=ascending,kind='mergesort')
   a  b
1  3  2
(Pdb) p self.loc[ser.index]
   a  b
1  3  2
1  4  1
```

~~@jreback what do you mean ? in `_nsorted` ? I don't think so~~

Testing. I think you're right

maybe the `.head()` should go at the end. I don't recall the exact guarantees of this.

I tested and your soln doesn't work and doesn't pass the test with this case :
`df = pd.DataFrame({'a': [1, 2, 3, 4],
                           'b': [4, 3, 2, 1]}, index=[0, 0, 1, 1])`

Any other suggestion ?

@Tux1 well then play around with it. What you are doing is WAY too complicated for a simple take.

can you rebase / update?
BUG: categorical unpickle to use _coerce_indexer_dtype
State: closed

- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

follow up for #13080 to use `_coerce_indexer_dtype`.


Labels:
name: Bug
name: Categorical
description: Categorical Data Type

Comments:
this _should_ show up in tests I think for an older pickle (I don't recall when we started coercing the codes), but not really a big deal.

can you add a test, or do we have coverage?

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13426?src=pr) is **84.34%**

> Merging [#13426](https://codecov.io/gh/pydata/pandas/pull/13426?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

``` diff
@@             master     #13426   diff @@
==========================================
  Files           138        138          
  Lines         51126      51126          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          43122      43121     -1   
- Misses         8004       8005     +1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [f20b41e...e8702f8](https://codecov.io/gh/pydata/pandas/compare/f20b41e73fdc84089289cd3999762892f85ec4b6...e8702f80f6c100352e23d4168bc6c81f16718aa9)

No, added some pickles to test it.

thanks!
CLN: Remove the engine parameter in CSVFormatter and to_csv
State: closed

Title is self-explanatory.

Internal code and #6581 indicate that this was long overdue.


Labels:
name: IO CSV
description: read_csv, to_csv
name: Deprecate
description: Functionality to remove in pandas

Comments:
see #11274 this was actually just deprecated in 0.17.1 (even though it _said_ 0.13).

@gfyoung Master is now targeted for 0.19.0, so this can go in. Can you move the whatsnew note to the correct file?

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13419?src=pr) is **84.32%**

> Merging [#13419](https://codecov.io/gh/pydata/pandas/pull/13419?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **0.01%**

``` diff
@@             master     #13419   diff @@
==========================================
  Files           138        138          
  Lines         51157      51083    -74   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          43131      43076    -55   
+ Misses         8026       8007    -19   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [5701c69...ddd6cd2](https://codecov.io/gh/pydata/pandas/compare/5701c69369264f3aa6f571384602ceec1133dabc...ddd6cd2e367543ec3fc77a3cf0b34d8b7290e21e?src=pr)

@jreback , @jorisvandenbossche : Travis is passing, so ready to merge if there are no other concerns.

lgtm. ping on green.

@jreback : Travis is giving the green light.  Ready to merge if there are no other concerns.

@gfyoung yes, merging shortly. 

thanks @gfyoung
BUG: Fix csv.QUOTE_NONNUMERIC quoting in to_csv
State: closed

Closes #12922: "bug" traced to #12194 via bisection, where the float formatting was unconditionally casting everything to string.

I say "bug" (with quotations) because the changes to `get_result_as_array` did **correctly** cast everything to string as per the documentation (i.e. it had inadvertently patched a bug itself even though it was just a cleaning PR).  However, the changes had overlooked the impact it would have on `to_csv`.


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO CSV
description: read_csv, to_csv

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13418?src=pr) is **84.23%**

> Merging [#13418](https://codecov.io/gh/pydata/pandas/pull/13418?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will decrease coverage by **0.04%**

``` diff
@@             master     #13418   diff @@
==========================================
  Files           138        138          
  Lines         50929      50813   -116   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          42925      42803   -122   
- Misses         8004       8010     +6   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [f752886...3f212e4](https://codecov.io/gh/pydata/pandas/compare/f7528866e07048f051f46f621ca76947730c7d32...3f212e4e4424cd3f96833860fa2c42d05078cd66)

@jreback : Moved `engine` parameter removal into separate PR and removed the `_get_formatted_data` function and move its code in `to_native_types`.  Travis is still happy, so ready to merge if there are no other concerns.

@jreback : Modified the relevant doc-strings and comments as requested.  Ready to merge if there are no other concerns.

ty
DOC- typo fix and adding correct command for environment deactivation…
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry

… for windows and linux


Labels:
name: Docs

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13413?src=pr) is **84.23%**

> Merging [#13413](https://codecov.io/gh/pydata/pandas/pull/13413?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will not change coverage

``` diff
@@             master     #13413   diff @@
==========================================
  Files           138        138          
  Lines         50805      50805          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          42796      42796          
  Misses         8009       8009          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [3eb4784...6108138](https://codecov.io/gh/pydata/pandas/compare/3eb4784dbf1757f2c3383beff2f70c5da47a93fe...6108138a94334c2863592d6eac3984824ac11b34)

thanks!

No Problem :)
ENH: implement 'reverse' for grouping
State: closed

when there are lots of columns of index levels it is more convenient to specify the column or index level that should be merged rather than all of the other columns or index levels

for example a multiindex dataframe

```
df.mean(level=['year', 'month', 'day', 'hour', 'min', 'trial'])
```

can be replaced with

```
df.mean(level='rep', reverse_level=True)
```

or for a flat index dataframe (where the data is in a column named 'result')

```
df.groupby(['year', 'month', 'day', 'hour', 'min', 'trial']).mean()
```

can be replaced with

```
df.groupby(['rep', 'result'], reverse=True)
```

http://stackoverflow.com/questions/16808682/is-there-in-pandas-operation-complementary-opposite-to-groupby


Labels:
name: Groupby
name: API Design

Comments:
aside from the fact there there are no tests, use case quoted in the SO

`df.groupby(df.columns.drop(['col6']))['vals'].mean()` seems just fine to me.

would take an example of doing this in the cookbook / docs. If you want to repurpose as that pls reopen.

typically these things are better brought up as an issue. Like hey what do people think about this. Then after comments / design a PR can be brought.

Sure, the main use case is really for multiindexing where the index names are a fixed list so you can't do names.drop('somename'), however this 'solution' is really a hack because it only address the case where a level name is provided.  I'll submit an issue next time.
BUG: don't raise on empty usecols
State: closed

Title is self-explanatory.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO CSV
description: read_csv, to_csv

Comments:
yep this fine. ping on green.

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13402?src=pr) is **84.23%**

> Merging [#13402](https://codecov.io/gh/pydata/pandas/pull/13402?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #13402   diff @@
==========================================
  Files           138        138          
  Lines         50805      50805          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42795      42796     +1   
+ Misses         8010       8009     -1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [d405bf2...8eed8d1](https://codecov.io/gh/pydata/pandas/compare/d405bf26a03b3ecdaa30b62c4b1dd5cb22930329...8eed8d13bfc93883bd5b6e259e460693b419d1b4)

@jreback: Travis is giving the green light.  Ready to merge if there are no concerns.
BUG: Fix for .extractall (single group with quantifier) #13382
State: closed

- [X] closes #13382
- [X] tests added / passed
- [X] passes `git diff upstream/master | flake8 --diff`
- [X] whatsnew entry

Note that I had to fix it based on this [thread](http://stackoverflow.com/questions/2111759/whats-the-best-practice-for-handling-single-value-tuples-in-python), rather than directly with `[x for x in ['ab']]` as this broke previous tests. 


Labels:
name: Bug
name: Strings
description: String extension data type and string data

Comments:
minor changes, looks good. ping when green.

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13397?src=pr) is **84.23%**

> Merging [#13397](https://codecov.io/gh/pydata/pandas/pull/13397?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #13397   diff @@
==========================================
  Files           138        138          
  Lines         50803      50805     +2   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42793      42795     +2   
  Misses         8010       8010          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [d5bea25...2afae62](https://codecov.io/gh/pydata/pandas/compare/d5bea25274b5faad5d2014b279a1da6451ebe1c7...2afae6272783f0fd3de76ae520b6bdf1e593eeca)

@jreback Done. 

thanks!
COMPAT, TST: allow numpy array comparisons with complex dtypes
State: closed

Traces back to bug in NumPy `v1.7.1` in which the `array_equivalent` method could not compare NumPy arrays with complicated dtypes. As `pandas` relies on this function to check NumPy array equality during testing, this commit adds a fallback method for doing so.

Closes #13388.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Didn't really think I needed a `whatsnew` in this case since this does seem rather internal to `pandas`.  If I do need one, not sure what I would write exactly.

this needs perf testing, not sure how much that path is hit (so I don't think its an issue). but pls run them.

@sinhrks 

@jreback : Will quickly run benchmarks, but as this is just a compatibility issue (bug exists until `v.1.8.0` of `numpy`), this path will most certainly never be hit for the more recent versions.

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13392?src=pr) is **84.22%**

> Merging [#13392](https://codecov.io/gh/pydata/pandas/pull/13392?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

``` diff
@@             master     #13392   diff @@
==========================================
  Files           138        138          
  Lines         50756      50763     +7   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42756      42757     +1   
- Misses         8000       8006     +6   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [67b72e3...230616e](https://codecov.io/gh/pydata/pandas/compare/67b72e3cbbaeb89a5b9c780b2fe1c8d5eaa9c505...230616eebe7055c116b6822a2a294fd8571a6361)

@jreback : Travis is passing on all machines again.  Ready to merge if there are no other concerns.

benchmarks ok?

No obvious changes in the benchmarks AFAICT and as expected.

ty