BrandonFors/pandas-github-issues huggingface dataset:
ENH: support MultiIndex and tuple hashing
State: closed

on top of #15216 

closes #15227 

Labels:
name: Enhancement
name: MultiIndex

Comments:
cc @mikegraham 

I separated this from the prior PR.

yes my goal here is to effectively emulate ``hash(tuple)`` but doing it efficiently (e.g. we already have the arrays in a packed format, turning into python objects, just to compute a tuple hash is very inefficient).
@mikegraham 
ok I incorporated your commit (with a slight modification) and now passes tests!
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/15224?src=pr) is 86.32% (diff: 100%)
> Merging [#15224](https://codecov.io/gh/pandas-dev/pandas/pull/15224?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **0.01%**

```diff
@@             master     #15224   diff @@
==========================================
  Files           139        139          
  Lines         51096      51140    +44   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          44103      44148    +45   
+ Misses         6993       6992     -1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [be32852...8b1d3f9](https://codecov.io/gh/pandas-dev/pandas/compare/be328521b3a5525175b08cb740bb970ccf75727e...8b1d3f9c5718799b5b0f2e1d8069d16fab01324a?src=pr)
There were a couple memory improvements in 187573bd490027078cb3d9b4bd9af349675a6a9d over the commit as you cherry-picked it, I think.
@mikegraham ahh ok, I will pick-again. thanks!
@mikegraham ok fixed.

Note that this obviously returns ``uint64``, so it is not quite compat with the tuplehashing in python?
I stole the algorithm, but it's not compatible. I also iterate over the members in forward error rather than reverse...I don't think that is substantive.
cc @mikegraham any final comments?

@jorisvandenbossche 
We might want to do something about this behavior.

```
>>> pandas.tools.hashing.hash_pandas_object(pandas.DataFrame(), index=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pandas/tools/hashing.py", line 91, in hash_pandas_object
    h = _combine_hash_arrays(hashes, num_items)
  File "pandas/tools/hashing.py", line 19, in _combine_hash_arrays
    first = next(arrays)
StopIteration```
@mikegraham updated
CLN: remove deprecated google-analytics module (GH11308)
State: closed


Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Clean

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/15223?src=pr) is 86.31% (diff: 100%)
> Merging [#15223](https://codecov.io/gh/pandas-dev/pandas/pull/15223?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **0.32%**

```diff
@@             master     #15223   diff @@
==========================================
  Files           140        139     -1   
  Lines         51272      51082   -190   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          44087      44089     +2   
+ Misses         7185       6993   -192   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [64d7670...7c6d5da](https://codecov.io/gh/pandas-dev/pandas/compare/64d7670d99a11ec4e263da73f2bb1335cb4290d3...7c6d5da20396a99e8fefb6f7f45dfebc73c7dd17?src=pr)
DOC: Merge FAQ and gotcha (rebase of GH13768)
State: closed

Rebase and clean-up of https://github.com/pandas-dev/pandas/pull/13768

closes #9809

Labels:
name: Docs

Comments:
lgtm!
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/15222?src=pr) is 85.98% (diff: 100%)
> Merging [#15222](https://codecov.io/gh/pandas-dev/pandas/pull/15222?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

```diff
@@             master     #15222   diff @@
==========================================
  Files           140        140          
  Lines         51272      51275     +3   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          44087      44089     +2   
- Misses         7185       7186     +1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [64d7670...7abb65b](https://codecov.io/gh/pandas-dev/pandas/compare/64d7670d99a11ec4e263da73f2bb1335cb4290d3...7abb65bcf56d5924feceb6a91cc9d77cd9408074?src=pr)
thanks!
BUG: TimedeltaIndex raising ValueError when boolean indexing (#14946)
State: closed

 - [x] closes #14946
 - [x] tests added / passed
 - [x] passes ``git diff upstream/master | flake8 --diff``
 - [x] whatsnew entry

Implementing patch suggested by @jreback [here](https://github.com/pandas-dev/pandas/issues/14946#issuecomment-268666453) that prevents raising a `ValueError` when boolean indexing with a `TimedeltaIndex`

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Timedelta
description: Timedelta data type

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/15221?src=pr) is 86.31% (diff: 100%)
> Merging [#15221](https://codecov.io/gh/pandas-dev/pandas/pull/15221?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **<.01%**

```diff
@@             master     #15221   diff @@
==========================================
  Files           139        139          
  Lines         51091      51093     +2   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          44099      44101     +2   
  Misses         6992       6992          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [0c8442c...b8ac04e](https://codecov.io/gh/pandas-dev/pandas/compare/0c8442c3f5c76c8e5b36ca313d3c38527e9f4b0c...b8ac04ebc84086b6dbe59f02384a9fd00a2b5ecf?src=pr)
Moved the tests to `pandas/tests/indexing/test_indexing.py`, created class for TimedeltaIndex, and all green. I didn't find any tests here that tested TimedeltaIndex exclusively, so I didn't group any more under this class. 
thanks @mroeschke keep em coming!
ENH union_categoricals supports ignore_order GH13410
State: closed

xref #13410 (ignore_order portion)


Labels:
name: Enhancement
name: Categorical
description: Categorical Data Type

Comments:
Thanks for the comments.  Pull request has been updated.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15219?src=pr&el=h1) Report
> Merging [#15219](https://codecov.io/gh/pandas-dev/pandas/pull/15219?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/be4a63fe791e27c2f8a9ae4f3a419ccc255c1b5b?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.


```diff
@@            Coverage Diff             @@
##           master   #15219      +/-   ##
==========================================
+ Coverage   90.37%   90.37%   +<.01%     
==========================================
  Files         135      135              
  Lines       49464    49466       +2     
==========================================
+ Hits        44702    44705       +3     
+ Misses       4762     4761       -1
```


| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15219?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/types/concat.py](https://codecov.io/gh/pandas-dev/pandas/compare/be4a63fe791e27c2f8a9ae4f3a419ccc255c1b5b...d278d623b7ed00317a870b3649c5692c284d2864?src=pr&el=tree#diff-cGFuZGFzL3R5cGVzL2NvbmNhdC5weQ==) | `98.06% <100%> (+0.01%)` | :white_check_mark: |
| [pandas/core/common.py](https://codecov.io/gh/pandas-dev/pandas/compare/be4a63fe791e27c2f8a9ae4f3a419ccc255c1b5b...d278d623b7ed00317a870b3649c5692c284d2864?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tbW9uLnB5) | `91.36% <ø> (+0.33%)` | :white_check_mark: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15219?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15219?src=pr&el=footer). Last update [be4a63f...e9d00de](https://codecov.io/gh/pandas-dev/pandas/compare/be4a63fe791e27c2f8a9ae4f3a419ccc255c1b5b...e9d00dec17bd6952048a46ae130ab8e79cba286b?el=footer&src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
@jorisvandenbossche Any other actions on my part?
@js3711 minor corrections and a flake issue you can see

``git diff master | flake8 --diff`` to see locally

ping when pushed / green.
can you rebase / update 
I'll update this weekend.  Apologies for the delay.
@jreback  thanks for the feedback.  I made the suggested changes and submitted a new PR.
thanks for the PR @js3711
BLD: hashtable.pxd is a pxd depends for _join, hashtable, and index
State: closed


Labels:
name: Build
description: Library building on various platforms

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/15218?src=pr) is 85.98% (diff: 100%)
> Merging [#15218](https://codecov.io/gh/pandas-dev/pandas/pull/15218?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

```diff
@@             master     #15218   diff @@
==========================================
  Files           140        140          
  Lines         51272      51272          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          44087      44086     -1   
- Misses         7185       7186     +1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [64d7670...5322245](https://codecov.io/gh/pandas-dev/pandas/compare/64d7670d99a11ec4e263da73f2bb1335cb4290d3...53222457316c2e9a61845fbc7ab33ecb18105b22?src=pr)
ENH: add MultiIndex.to_dataframe
State: closed

ENH: allow hashing of MultiIndex

closes #12397


Labels:
name: Enhancement
name: MultiIndex

Comments:
cc @mrocklin
cc @jcrist 

also adds hashing of MultiIndex.
```
In [52]: i = pd.MultiIndex.from_tuples([(118, 472), (236, 118), (51, 204), (102, 51)])

In [53]: i
Out[53]: 
MultiIndex(levels=[[51, 102, 118, 236], [51, 118, 204, 472]],
           labels=[[2, 3, 0, 1], [3, 1, 2, 0]])

In [55]: i.to_dataframe(index=False)
Out[55]: 
     0    1
0  118  472
1  236  118
2   51  204
3  102   51

In [56]: from pandas.tools.hashing import hash_pandas_object

In [57]: hash_pandas_object(i.to_dataframe(index=False), index=False)
Out[57]: 
0    11950414010286087598
1    11950414010286087598
2    10472907816967777234
3    10472907816967777234
dtype: uint64

In [58]: hash_pandas_object(i.to_dataframe(index=False), index=True)
Out[58]: 
0    17404497957148711178
1     5195826631379738351
2    10365020365066803200
3    15157173997208611942
dtype: uint64
```

odd that [57] can produce duplicates. any thoughts @mikegraham 
(in practice I don't think this matters as we normally *also* include the index, which then makes these unique)

but I have a case where I just want to uniquely hash values (no index)
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/15216?src=pr) is 86.30% (diff: 100%)


> No coverage report found for **master** at ba05744.

> Powered by [Codecov](https://codecov.io?src=pr). Last update [ba05744...4a151c6](https://codecov.io/gh/pandas-dev/pandas/compare/ba057443d1b69bb3735a4b18a18e4e4897231867...4a151c62baf87a5b33c70a96efee163e57ff2315?src=pr)
@jorisvandenbossche ok with this? (I need to build on top of this for other things).....
Yes, looks good. Only thing I am wondering is the name. We already have `Series.to_frame` as well, so it would be nice to be consistent here (although I think I like `to_dataframe` more ..). 

Also wondering if this should be restricted to MultiIndex and not just general for Index (but that can certainly go in a follow-up PR if we want that)
sure will change ``.to_frame()``

not sure if we should add this to ``Series``, it doesn't really make senses there :>
> not sure if we should add this to ``Series``, it doesn't really make senses there :>

I wrote Index, not Series (but maybe that's what you meant :-)). Index already has a `to_series` method, which is more logical for an Index, but the main motivation would be to just not create more distinction in api between single/multi index  (eg to not always have to check the number of levels of your index when writing generic code, cfr #3268)
@jorisvandenbossche yeah that is a fair point, we can revisit, i'll create an issue.
BLD: add locale to 34_nslow build
State: closed


Labels:
name: Build
description: Library building on various platforms

Comments:
You might need

```
      addons:
        apt:
          packages:
          - language-pack-zh-hans
```
I don't know how job name and cache name work, but there are 2 with the name `34_nslow` and the same cache.
@bashtage there are always 2 when you have an allowed failure.
Python 2/3 Unicode challenges.
yep I just put it upagain using ZH on 2.7, so will see.
Just did a quick local check and it looks like my patch for the date problem fixes Python 3 issues in `stata`.
@bashtage do we have a test for this? (IOW, the above shows no issues in master on py3).
All of these failures

https://travis-ci.org/jreback/pandas/jobs/194872508

are the same bug with locale being used for `strftime`.  There didn't seem to be a specific test needed since it was already being hit in lots of existing tests.
hmm, I must have cancelled it too soon (or was looking at the wrong build). ok so this fails on py3. great, going to merge the original change then (running ZH on 3.4).
this surfaced an addition (non-stata error)

```
======================================================================
ERROR: test_to_datetime_format_microsecond (pandas.tseries.tests.test_timeseries.TimeConversionFormats)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/jreback/pandas/pandas/tseries/tools.py", line 407, in _convert_listlike
    values, tz = tslib.datetime_to_datetime64(arg)
  File "pandas/tslib.pyx", line 1756, in pandas.tslib.datetime_to_datetime64 (pandas/tslib.c:30671)
    raise TypeError('Unrecognized value type: %s' % type(val))
TypeError: Unrecognized value type: <class 'str'>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/travis/build/jreback/pandas/pandas/tseries/tests/test_timeseries.py", line 5374, in test_to_datetime_format_microsecond
    result = to_datetime(val, format=format)
  File "/home/travis/build/jreback/pandas/pandas/tseries/tools.py", line 427, in to_datetime
    return _convert_listlike(np.array([arg]), box, format)[0]
  File "/home/travis/build/jreback/pandas/pandas/tseries/tools.py", line 410, in _convert_listlike
    raise e
  File "/home/travis/build/jreback/pandas/pandas/tseries/tools.py", line 377, in _convert_listlike
    errors=errors)
  File "pandas/tslib.pyx", line 3600, in pandas.tslib.array_strptime (pandas/tslib.c:58664)
    raise ValueError("time data %r does not match "
ValueError: time data '01-Apr-2011 00:00:01.978' does not match format '%d-%b-%Y %H:%M:%S.%f' (match)
```
@bashtage going to merge this one: https://travis-ci.org/jreback/pandas/jobs/194920988

which your PR should fix all but 1 of these I think (the one above needs more looking), so i'll create an issue.
TST: fix excel tests when openpyxl is not installed
State: closed

Some tests fail when you don't have openpyxl installed, so explicitly skipping those (they need it for certain exts because they write a frame in the test)

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Excel
description: read_excel, to_excel

Comments:
+1 I thought I had fixed these........
Is it worth to have a build without openpyxl for this?
Although it should be a build without openpyxl, but with xlrd/xlwt, to have this surface, and it's also not a bug, only a testing thing, so probably not worth it
prob not worth it, but no objection if u want to change one (say 3.4_nslow)
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/15209?src=pr) is 85.98% (diff: 100%)
> Merging [#15209](https://codecov.io/gh/pandas-dev/pandas/pull/15209?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will decrease coverage by **0.01%**

```diff
@@             master     #15209   diff @@
==========================================
  Files           140        140          
  Lines         51273      51272     -1   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          44091      44085     -6   
- Misses         7182       7187     +5   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [dc40058...f1cbe8f](https://codecov.io/gh/pandas-dev/pandas/compare/dc400587a3e792d90f5db0be6caff7a4e12574c0...f1cbe8f393c413c53560eb7463f7a306bb7895eb?src=pr)
thanks!
BUG: Remove locale conversion from Stata file date
State: closed

Prevent locale from affecting Stata file date creation, which must
be en_US.

xref #13856

 - [x] tests added / passed
 - [x] passes ``git diff upstream/master | flake8 --diff``
 - [x] whatsnew entry


Labels:
name: Bug
name: IO Stata
description: read_stata, to_stata

Comments:
Does not close any issue since this was only identified as part of #13856.  

Passes all tests (not just slow) when locale is `zh_CN.utf8`.  This config is not enabled by default.
https://travis-ci.org/jreback/pandas/jobs/194872508

oddly this didn't fail on py3. 
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/15208?src=pr) is 85.99% (diff: 100%)


> No coverage report found for **master** at 9309eba.

> Powered by [Codecov](https://codecov.io?src=pr). Last update [9309eba...a23a92f](https://codecov.io/gh/pandas-dev/pandas/compare/9309ebae91ca580d54a7d3d8d2b9db7df5e0d72b...a23a92f3eba48fa85a2bb680382c751521708927?src=pr)
can you rebase and push again, just to make sure this passes? ping on green.
@jreback Green - one test fails in another module that has the same problem due to locales.

https://travis-ci.org/pandas-dev/pandas/jobs/194970098


thanks for the fix @bashtage
BUG: groupby.cummin/cummax nans inf values for int64
State: closed

 - [x] closes #15109 
 - [x] tests added / passed
 - [x] passes ``git diff upstream/master | flake8 --diff``

I migrated the `accum` portions of the code into cython and attempted to implement your suggested method to fix this issue @jreback. Let me know if this is along the right track. 


Labels:
name: Bug
name: Groupby

Comments:
linting issue showing up & small comment, but otherwise  lgtm. ping when pushed and green.
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/15205?src=pr) is 85.97% (diff: 100%)


> No coverage report found for **master** at 64d7670.

> Powered by [Codecov](https://codecov.io?src=pr). Last update [64d7670...b390fdb](https://codecov.io/gh/pandas-dev/pandas/compare/64d7670d99a11ec4e263da73f2bb1335cb4290d3...b390fdbe8bb2d89d0a687efe6bb5a2886d9fe499?src=pr)
Ping all green. 
thanks @mroeschke
BUG: GH14233 resample().median() failed if duplicate column names wer…
State: closed

Simple fix for  median issue.  Should use cython implementation.

 - [X] closes #14233
 - [X] tests added / passed
 - [X] passes ``git diff upstream/master | flake8 --diff``
 - [X] whatsnew entry


Labels:
name: Bug
name: Resample
description: resample method

Comments:
lgtm. ping on green.
thanks!
COMPAT: don't rely on dtype being False after numpy >= 1.13
State: closed

closes #15199



Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/15200?src=pr) is 85.97% (diff: 100%)
> Merging [#15200](https://codecov.io/gh/pandas-dev/pandas/pull/15200?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will decrease coverage by **0.01%**

```diff
@@             master     #15200   diff @@
==========================================
  Files           140        140          
  Lines         51273      51263    -10   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          44091      44075    -16   
- Misses         7182       7188     +6   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [dc40058...69f3fa4](https://codecov.io/gh/pandas-dev/pandas/compare/dc400587a3e792d90f5db0be6caff7a4e12574c0...69f3fa44c37f1edec3b2ac2f7ef00e52fa1eda2c?src=pr)
Fix bug in window function count should count anything non-null
State: closed

 - [x] closes #12541 
 - [x] tests added / passed
 - [x] passes ``git diff upstream/master | flake8 --diff``
 - [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
this looks right, pls add a note in whatsnew, bug fix section.

note that I did try to change this a while back and got some tests failures (which I think you mentioned), so these need to be investigated / fixed. I think the 'floatifying' is slightly different that checking for non-null, but need to drill down and see exactly.
## [Current coverage](https://codecov.io/gh/pandas-dev/pandas/pull/15196?src=pr) is 85.98% (diff: 100%)
> Merging [#15196](https://codecov.io/gh/pandas-dev/pandas/pull/15196?src=pr) into [master](https://codecov.io/gh/pandas-dev/pandas/branch/master?src=pr) will increase coverage by **<.01%**

```diff
@@             master     #15196   diff @@
==========================================
  Files           140        140          
  Lines         51263      51267     +4   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          44076      44084     +8   
+ Misses         7187       7183     -4   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [be3f2ae...65d70eb](https://codecov.io/gh/pandas-dev/pandas/compare/be3f2aea4a8ad3f6bf4eb84e6ca3269abff5fcfb...65d70eb614d8b799f5c155768448763ceaa64e26?src=pr)
lgtm. ping on green (I will fix the minor typo on merge)
thanks @mralgos