Repository: pandas-dev/pandas

Issue:
Merge on single level of MultiIndex
I don't think we can do this now, but it potentially sounds useful...

http://stackoverflow.com/questions/16650945/merge-on-single-level-of-multiindex
https://groups.google.com/forum/#!topic/pydata/LBSFq6of8ao

Example:

```
In [10]: df = pd.DataFrame(randn(10,5))

In [11]: df1 = df[range(0, 3)].set_index([0, 2])

In [12]: df2 = df[range(2, 5)].set_index([2, 4])
```

join/merge on 2.



Discussion:
is `df1.join(df2, how='outer')` not what ur looking for?

No, I don't think so...

```
In [13]: df1.join(df2, how='outer')
Out[13]:
                            1         3
-3.987006 -0.534963       NaN -0.521078
-1.808632 -0.227492       NaN -0.880865
-1.188311 -1.808632 -1.018368       NaN
-0.985607 -0.151162  0.815265       NaN
-0.528617  1.369224 -0.642621       NaN
-0.151162 -0.462676       NaN  0.388309
-0.115006 -3.987006  0.548307       NaN
 0.066250  0.590389  0.275410       NaN
 0.590389 -0.926148       NaN -0.999415
 0.763442  0.618186       NaN  1.212661
 0.930736 -0.856200       NaN -0.861088
 0.975342  1.824005 -0.162082       NaN
 1.006033  0.340764       NaN -1.809402
 1.169599  1.006033 -2.006705       NaN
 1.189407  0.930736  0.645642       NaN
 1.243792  0.763442 -0.972110       NaN
 1.369224  0.547792       NaN  1.515713
 1.375035  2.192195       NaN -0.728762
 1.824005 -1.364679       NaN -0.231887
 2.151640  1.375035 -0.656242       NaN
```

Hoped for:

```
In [21]: df3 = pd.DataFrame({1: df[1], 3: df[3]})

In [22]: df3.index = df[2]

In [23]: df3
Out[23]:
                  1         3
2
 0.930736  0.645642 -0.861088
 0.590389  0.275410 -0.999415
 1.375035 -0.656242 -0.728762
 0.763442 -0.972110  1.212661
 1.824005 -0.162082 -0.231887
 1.369224 -0.642621  1.515713
-0.151162  0.815265  0.388309
 1.006033 -2.006705 -1.809402
-3.987006  0.548307 -0.521078
-1.808632 -1.018368 -0.880865
```

yes this would most def be useful IMHO

@hayd I think that 

``` python
df1.join(df2, how='inner')
```

actually should do what you want. Right now it _doesn't_ do this and in fact `join` doesn't do what it claims to do in this regard, that is, with `how='inner'` return the intersection of the indices. The intersection of the indices of `df1` and `df2` is not the empty set, yet the empty `DataFrame` is returned, so this might be considered a bug...

OT: should the empty ndframe be a singleton? i haven't really checked, but these objects return a different id every time they are instantiated...

Down in the guts of this there's a function called `_join_level` defined on `Index`. Currently it throws an error saying that joining 2 multiindex objects on a level is ambiguous, why? If I have two `MultiIndex`s with two levels with the same name shouldn't I be able to compute the intersection of those indices? It should only be empty if they have no elements in common (by definition).

from some comments on the mailing list: https://groups.google.com/forum/#!topic/pydata/LBSFq6of8ao

What would help move this along would be:

1) some self-contained test cases, esp corner / cases where this would fail
2) the 'quick' and dirty soln can prob be implemented pretty easily (reset indexes and merge, reset the indexes). This may actually be a pretty good soln as their is not a lot of cost (in memory or speed) for resetting / setting indexes - and if memory is an issue that's a whole other problem (http://stackoverflow.com/questions/14614512/merging-two-tables-with-millions-of-rows-in-python/14617925#14617925)
3) an acutal indexing merge (where right now it raises in core/index.py : again test cases are the most important here.

would love for someone to attempt 2) as then can have a speed/memory benchmark and see even if 3) is worthwhile (I don't know how much gain this would really have - so not sure how much effort it needs).

I'm not quite sure I understand the difference between 2) and 3) - is this just the underlying implementation or do you mean something else by "an actual indexing merge)? 

As for 2), are you saying that all that needs to be done would be to hide the SO solution from the user and allow specifying the index?

Testcases: In principle this should amount to re-using test cases for joins with regular columns, no? I had a very brief look into that but couldn't find much -- any pointers?

Looking at the expected behavior above, I am not sure I'd quite agree (at least for outer joins, which seem to be the more natural use case to me) -- I would expect a MultiIndex on the resulting dataframe with the index being the union of all index constituents of the two original tables.

2) is like the SO question, but exactly, hiding from the user.
3) is potentially much faster / better user of memory as its done internal in the index module (e.g. right where the raise happens now)

Maybe give me a small example of what you want to do, e.g. input and output (e.g. the normal use case)

Here's a number of issues relating to this (and why we need specific test cases):
- normal case is number of levels match up and names on those levels match
- what happens if the right can only match up on certain level, (e.g. right has 1 level and left has 2; match on which level?
  - the name matches then its clear (I think this is handled now)
  - what do do if their are NO matching levels

Here comes an example, see [these slides](http://www.wiwi.uni-bonn.de/gaudecker/teaching/prog_econ/2011/15_data_management.pdf) for some context (p. 19+).

```
In [49]: import io
In [50]: household = pd.read_table(
    io.StringIO(
"""household_id,male,wealth
1,0,196087.3
2,1,316478.7
3,0,294750
"""
    ), 
    sep=',', index_col='household_id'
)

In [51]: household
Out[51]: 
              male    wealth
household_id                
1                0  196087.3
2                1  316478.7
3                0  294750.0

[3 rows x 2 columns]

In [52]: portfolio = pd.read_table(
    io.StringIO(
        """"household_id","asset_id","name","share"
"1","nl0000301109","ABN Amro","1.0"
"2","nl0000289783","Robeco","0.4"
"2","gb00b03mlx29","Royal Dutch Shell","0.6"
"3","gb00b03mlx29","Royal Dutch Shell","0.15"
"3","lu0197800237","AAB Eastern Europe Equity Fund","0.6"
"3","nl0000289965","Postbank BioTech Fonds","0.25"
"4",,,"1.0"
"""
    ),
    sep=',', index_col=['household_id', 'asset_id']
)

In [53]: portfolio
Out[53]: 
                                                     name  share
household_id asset_id                                           
1            nl0000301109                        ABN Amro   1.00
2            nl0000289783                          Robeco   0.40
             gb00b03mlx29               Royal Dutch Shell   0.60
3            gb00b03mlx29               Royal Dutch Shell   0.15
             lu0197800237  AAB Eastern Europe Equity Fund   0.60
             nl0000289965          Postbank BioTech Fonds   0.25
4            NaN                                      NaN   1.00

[7 rows x 2 columns]

In [54]: household.join(portfolio, how='inner')
Out[54]: 
Empty DataFrame
Columns: [male, wealth, name, share]
Index: []

[0 rows x 4 columns]
```

(So I would argue this is not handled)

I would expect:

```
                             male    wealth                            name  share
household_id asset_id                                                           
1            nl0000301109       0  196087.3                        ABN Amro   1.00
2            nl0000289783       1  316478.7                          Robeco   0.40
             gb00b03mlx29       1  316478.7               Royal Dutch Shell   0.60
3            gb00b03mlx29       0  294750.0               Royal Dutch Shell   0.15
             lu0197800237       0  294750.0  AAB Eastern Europe Equity Fund   0.60
             nl0000289965       0  294750.0          Postbank BioTech Fonds   0.25
```

> What do do if their are NO matching levels?

I would be fine requiring the user to name the levels consistently. Else you can always fall back on the existing solution based on regular columns.


Resolving patch:
diff --git a/pandas/core/index.py b/pandas/core/index.py
--- a/pandas/core/index.py
+++ b/pandas/core/index.py
@@ -1265,8 +1265,21 @@ def join(self, other, how='left', level=None, return_indexers=False):
         -------
         join_index, (left_indexer, right_indexer)
         """
-        if (level is not None and (isinstance(self, MultiIndex) or
-                                   isinstance(other, MultiIndex))):
+        self_is_mi = isinstance(self, MultiIndex)
+        other_is_mi = isinstance(other, MultiIndex)
+
+        # try to figure out the join level
+        # GH3662
+        if (level is None and (self_is_mi or other_is_mi)):
+
+            # have the same levels/names so a simple join
+            if self.names == other.names:
+                pass
+            else:
+                return self._join_multi(other, how=how, return_indexers=return_indexers)
+
+        # join on the level
+        if (level is not None and (self_is_mi or other_is_mi)):
             return self._join_level(other, level, how=how,
                                     return_indexers=return_indexers)
 
@@ -1344,6 +1357,43 @@ def join(self, other, how='left', level=None, return_indexers=False):
         else:
             return join_index
 
+    def _join_multi(self, other, how, return_indexers=True):
+
+        self_is_mi = isinstance(self, MultiIndex)
+        other_is_mi = isinstance(other, MultiIndex)
+
+        # figure out join names
+        self_names = [ n for n in self.names if n is not None ]
+        other_names = [ n for n in other.names if n is not None ]
+        overlap = list(set(self_names) & set(other_names))
+
+        # need at least 1 in common, but not more than 1
+        if not len(overlap):
+            raise ValueError("cannot join with no level specified and no overlapping names")
+        if len(overlap) > 1:
+            raise NotImplementedError("merging with more than one level overlap on a multi-index is not implemented")
+        jl = overlap[0]
+
+        # make the indices into mi's that match
+        if not (self_is_mi and other_is_mi):
+
+            flip_order = False
+            if self_is_mi:
+                self, other = other, self
+                flip_order = True
+
+            level = other.names.index(jl)
+            result = self._join_level(other, level, how=how,
+                                      return_indexers=return_indexers)
+
+            if flip_order:
+                if isinstance(result, tuple):
+                    return result[0], result[2], result[1]
+            return result
+
+        # 2 multi-indexes
+        raise NotImplementedError("merging with both multi-indexes is not implemented")
+
     def _join_non_unique(self, other, how='left', return_indexers=False):
         from pandas.tools.merge import _get_join_indexers
 


Test patch:
diff --git a/pandas/tools/tests/test_merge.py b/pandas/tools/tests/test_merge.py
--- a/pandas/tools/tests/test_merge.py
+++ b/pandas/tools/tests/test_merge.py
@@ -8,7 +8,7 @@
 import numpy as np
 import random
 
-from pandas.compat import range, lrange, lzip, zip
+from pandas.compat import range, lrange, lzip, zip, StringIO
 from pandas import compat, _np_version_under1p7
 from pandas.tseries.index import DatetimeIndex
 from pandas.tools.merge import merge, concat, ordered_merge, MergeError
@@ -16,7 +16,7 @@
                                  assert_almost_equal, rands,
                                  makeCustomDataframe as mkdf,
                                  assertRaisesRegexp)
-from pandas import isnull, DataFrame, Index, MultiIndex, Panel, Series, date_range
+from pandas import isnull, DataFrame, Index, MultiIndex, Panel, Series, date_range, read_table
 import pandas.algos as algos
 import pandas.util.testing as tm
 
@@ -1025,6 +1025,98 @@ def test_int64_overflow_issues(self):
         result = merge(df1, df2, how='outer')
         self.assertTrue(len(result) == 2000)
 
+    def test_join_multi_levels(self):
+
+        # GH 3662
+        # merge multi-levels
+
+        household = DataFrame(dict(household_id = [1,2,3],
+                                   male = [0,1,0],
+                                   wealth = [196087.3,316478.7,294750]),
+                              columns = ['household_id','male','wealth']).set_index('household_id')
+        portfolio = DataFrame(dict(household_id = [1,2,2,3,3,3,4],
+                                   asset_id = ["nl0000301109","nl0000289783","gb00b03mlx29","gb00b03mlx29","lu0197800237","nl0000289965",np.nan],
+                                   name = ["ABN Amro","Robeco","Royal Dutch Shell","Royal Dutch Shell","AAB Eastern Europe Equity Fund","Postbank BioTech Fonds",np.nan],
+                                   share = [1.0,0.4,0.6,0.15,0.6,0.25,1.0]),
+                              columns = ['household_id','asset_id','name','share']).set_index(['household_id','asset_id'])
+        result = household.join(portfolio, how='inner')
+        expected = DataFrame(dict(male = [0,1,1,0,0,0],
+                                  wealth = [ 196087.3, 316478.7, 316478.7, 294750.0, 294750.0, 294750.0 ],
+                                  name = ['ABN Amro','Robeco','Royal Dutch Shell','Royal Dutch Shell','AAB Eastern Europe Equity Fund','Postbank BioTech Fonds'],
+                                  share = [1.00,0.40,0.60,0.15,0.60,0.25],
+                                  household_id = [1,2,2,3,3,3],
+                                  asset_id = ['nl0000301109','nl0000289783','gb00b03mlx29','gb00b03mlx29','lu0197800237','nl0000289965']),
+                             ).set_index(['household_id','asset_id']).reindex(columns=['male','wealth','name','share'])
+        assert_frame_equal(result,expected)
+
+        assert_frame_equal(result,expected)
+
+        # equivalency
+        result2 = merge(household.reset_index(),portfolio.reset_index(),on=['household_id'],how='inner').set_index(['household_id','asset_id'])
+        assert_frame_equal(result2,expected)
+
+        result = household.join(portfolio, how='outer')
+        expected = concat([expected,DataFrame(dict(share = [1.00]),
+                                              index=MultiIndex.from_tuples([(4,np.nan)],
+                                                                           names=['household_id','asset_id']))],
+                          axis=0).reindex(columns=expected.columns)
+        assert_frame_equal(result,expected)
+
+        # invalid cases
+        household.index.name = 'foo'
+        def f():
+            household.join(portfolio, how='inner')
+        self.assertRaises(ValueError, f)
+
+        portfolio2 = portfolio.copy()
+        portfolio2.index.set_names(['household_id','foo'])
+        def f():
+            portfolio2.join(portfolio, how='inner')
+        self.assertRaises(ValueError, f)
+
+    def test_join_multi_levels2(self):
+
+        # some more advanced merges
+        # GH6360
+        household = DataFrame(dict(household_id = [1,2,2,3,3,3,4],
+                                   asset_id = ["nl0000301109","nl0000301109","gb00b03mlx29","gb00b03mlx29","lu0197800237","nl0000289965",np.nan],
+                                   share = [1.0,0.4,0.6,0.15,0.6,0.25,1.0]),
+                              columns = ['household_id','asset_id','share']).set_index(['household_id','asset_id'])
+
+        log_return = DataFrame(dict(
+            asset_id = ["gb00b03mlx29", "gb00b03mlx29", "gb00b03mlx29", "lu0197800237", "lu0197800237"],
+            t = [233, 234, 235, 180, 181],
+            log_return = [.09604978, -.06524096, .03532373, .03025441, .036997]
+                )).set_index(["asset_id","t"])
+
+        expected = DataFrame(dict(
+            household_id = [2, 2, 2, 3, 3, 3, 3, 3],
+            asset_id = ["gb00b03mlx29", "gb00b03mlx29", "gb00b03mlx29", "gb00b03mlx29", "gb00b03mlx29", "gb00b03mlx29", "lu0197800237", "lu0197800237"],
+            t = [233, 234, 235, 233, 234, 235, 180, 181],
+            share = [0.6, 0.6, 0.6, 0.15, 0.15, 0.15, 0.6, 0.6],
+            log_return = [.09604978, -.06524096, .03532373, .09604978, -.06524096, .03532373, .03025441, .036997]
+            )).set_index(["household_id", "asset_id", "t"]).reindex(columns=['share','log_return'])
+
+        def f():
+            household.join(log_return, how='inner')
+        self.assertRaises(NotImplementedError, f)
+
+        # this is the equivalency
+        result = merge(household.reset_index(),log_return.reset_index(),on=['asset_id'],how='inner').set_index(['household_id','asset_id','t'])
+        assert_frame_equal(result,expected)
+
+        expected = DataFrame(dict(
+            household_id = [1, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 4],
+            asset_id = ["nl0000301109", "nl0000289783", "gb00b03mlx29", "gb00b03mlx29", "gb00b03mlx29", "gb00b03mlx29", "gb00b03mlx29", "gb00b03mlx29", "lu0197800237", "lu0197800237", "nl0000289965", None],
+            t = [None, None, 233, 234, 235, 233, 234, 235, 180, 181, None, None],
+            share = [1.0, 0.4, 0.6, 0.6, 0.6, 0.15, 0.15, 0.15, 0.6, 0.6, 0.25, 1.0],
+            log_return = [None, None, .09604978, -.06524096, .03532373, .09604978, -.06524096, .03532373, .03025441, .036997, None, None]
+            )).set_index(["household_id", "asset_id", "t"])
+
+        def f():
+            household.join(log_return, how='outer')
+        self.assertRaises(NotImplementedError, f)
+
 def _check_join(left, right, result, join_col, how='left',
                 lsuffix='_x', rsuffix='_y'):