ENH: Add sym_diff for index
State: closed

Close https://github.com/pydata/pandas/issues/5543

If there's any interest for this in 0.14, here's the code.

Thoughts on the method name? A regular difference is `diff` so I want to be consistent with that. I figured it would be a bit weird to truncate `difference` to `diff` but not truncate `symmetric` to `sym` (or `symm`) so I went with `sym_diff` (python sets use `symmetric_difference`).

Also when writing the tests, I discovered that NaNs are _even_ weirder than I thought:

``` python
In [20]: idx = pd.Index([np.nan, 1, np.nan])

In [21]: sorted(set(idx))
Out[21]: [nan, 1.0, nan]
```

Seems like neither `set` nor `sorted` did anything. Anyway my PR should be consistent with this weirdness.


Labels:
name: Enhancement
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: API Design

Comments:
`Index` has set algebra operations, including union and difference. We could do this.
at least `diff` does not assume `other` is an `Index`, this shouldn't either.

Good call on not requiring `other` to be an `Index`. I've also updated the `diff` docstring to say that it accepts an array like object.

Yahoo network errors are causing the failure. I'll retry later.

@TomAugspurger if you rebase on master the network errors should go away...@y-p fixed this

I guess I forgot to fetch before rebasing. I'll throw in a release notes entry once the .14 cycle starts.

@TomAugspurger Two ideas for the documentation of someone not so familiar with set terminology:
- Could you add short explanation on what the function does, apart from "(symmetric) set difference". Eg something like "the elements in the index that are not in `other`" and "the elements which are in both sets but not in their intersection"
- And maybe you could also add a small example with output in the docstring, so you can see in a glance what it does.

@TomAugspurger FYI, you should enable the operator '^'

e.g. `s ^ t` is a standard symmetric diff op

see here: http://docs.python.org/2/library/sets.html (you define `__xor__` to do this)

Thanks. @jorisvandenbossche added more to notes and an example.

@jreback I didn't know that python used `^` at all. Cool.

@TomAugspurger Thanks for the clarifications!

@TomAugspurger looks good...pls add release notes / maybe add that you can do this in indexing section (where the descriptin of the Index internal methods are - near the bottom)

@jreback rebased and added notes and docs. Should be good to go.

gr8!

pls review the release notes/v0.14.0 after this is built this is built when master processes this PR: http://pandas-docs.github.io/pandas-docs-travis/

thanks!

Will do. Thanks.

@TomAugspurger 

this test fails on python 3.4 (only tested on windows atm)

any ideas?

```
 ======================================================================
FAIL: test_symmetric_diff (pandas.tests.test_index.TestIndex)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\Users\Jeff Reback\Documents\GitHub\pandas\build\lib.win32-3.4\pandas\tests\test_index.py", line 501, in test_symmetric_diff
    self.assert_(pd.isnull(result[nans]).all())
AssertionError: False is not true

----------------------------------------------------------------------
Ran 4771 tests in 139.262s

FAILED (SKIP=174, failures=1)

C:\Users\Jeff Reback\Documents\GitHub\pandas\build\lib.win32-3.4 [master]> cd ..
C:\Users\Jeff Reback\Documents\GitHub\pandas\build [master]> cd ..
C:\Users\Jeff Reback\Documents\GitHub\pandas [master]> C:\python34-32\Scripts\nosetests.exe .\build\lib.win32-3.4\pandas\tests\test_index.py --pdb --p
db-failure
..........................................................> c:\python34-32\lib\unittest\case.py(651)assertTrue()
-> raise self.failureException(msg)
(Pdb) u
> c:\python34-32\lib\unittest\case.py(1287)deprecated_func()
-> return original_func(*args, **kwargs)
(Pdb) u
> c:\users\jeff reback\documents\github\pandas\build\lib.win32-3.4\pandas\tests\test_index.py(501)test_symmetric_diff()
-> self.assert_(pd.isnull(result[nans]).all())
(Pdb) l
496             idx1 = Index([1, 2, np.nan])
497             idx2 = Index([0, 1, np.nan])
498             result = idx1.sym_diff(idx2)
499             expected = Index([0.0, np.nan, 2.0, np.nan])  # oddness with nans
500             nans = pd.isnull(expected)
501  ->         self.assert_(pd.isnull(result[nans]).all())
502             self.assert_(tm.equalContents(result[~nans], expected[~nans]))
503
504             # other not an Index:
505             idx1 = Index([1, 2, 3, 4], name='idx1')
506             idx2 = np.array([2, 3, 4, 5])
(Pdb) p nans
array([False,  True, False,  True], dtype=bool)
(Pdb) p result
Float64Index([0.0, nan, nan, 2.0], dtype='object')
(Pdb) p result[nans]
Float64Index([nan, 2.0], dtype='object')
(Pdb) p idx1
Float64Index([1.0, 2.0, nan], dtype='object')
(Pdb) p idx2
Float64Index([0.0, 1.0, nan], dtype='object')
(Pdb) p idx1.sym_diff(idx2)
Float64Index([0.0, nan, nan, 2.0], dtype='object')
(Pdb)
Float64Index([0.0, nan, nan, 2.0], dtype='object')
(Pdb)
Float64Index([0.0, nan, nan, 2.0], dtype='object')
(Pdb)
Float64Index([0.0, nan, nan, 2.0], dtype='object')
(Pdb)
Float64Index([0.0, nan, nan, 2.0], dtype='object')
(Pdb) p idx1
Float64Index([1.0, 2.0, nan], dtype='object')
(Pdb) p idx2
Float64Index([0.0, 1.0, nan], dtype='object')
(Pdb) p idx1.sym_diff(idx2)
Float64Index([0.0, nan, nan, 2.0], dtype='object')
(Pdb)
```

Strange. I'll get a 3.4 env setup and see what's going on.

note...that we don't test on travis 3.4

Here's my setup of 34-32 (34-64 fails too)
ths is just FYI...I think key is numpy 1.8/3.4

```
C:\Builds>c:/python27-64/python.exe c:/Builds/check_and_build.py -b 34-32 -v
2014-02-17 14:46:53,487: 34-32 :
2014-02-17 14:46:53,487: 34-32 : INSTALLED VERSIONS
2014-02-17 14:46:53,489: 34-32 : ------------------
2014-02-17 14:46:53,489: 34-32 : commit: None
2014-02-17 14:46:53,489: 34-32 : python: 3.4.0.candidate.1
2014-02-17 14:46:53,489: 34-32 : python-bits: 32
2014-02-17 14:46:53,489: 34-32 : OS: Windows
2014-02-17 14:46:53,489: 34-32 : OS-release: 7
2014-02-17 14:46:53,489: 34-32 : machine: AMD64
2014-02-17 14:46:53,490: 34-32 : processor: Intel64 Family 6 Model 42 Stepping 7, GenuineIntel
2014-02-17 14:46:53,490: 34-32 : byteorder: little
2014-02-17 14:46:53,490: 34-32 : LC_ALL: None
2014-02-17 14:46:53,490: 34-32 : LANG: None
2014-02-17 14:46:53,490: 34-32 :
2014-02-17 14:46:53,490: 34-32 : pandas: None
2014-02-17 14:46:53,490: 34-32 : Cython: 0.20.1
2014-02-17 14:46:53,490: 34-32 : numpy: 1.8.0
2014-02-17 14:46:53,490: 34-32 : scipy: 0.13.3
2014-02-17 14:46:53,490: 34-32 : statsmodels: 0.5.0
2014-02-17 14:46:53,490: 34-32 : IPython: None
2014-02-17 14:46:53,490: 34-32 : sphinx: None
2014-02-17 14:46:53,490: 34-32 : patsy: 0.2.1
2014-02-17 14:46:53,490: 34-32 : scikits.timeseries: None
2014-02-17 14:46:53,490: 34-32 : dateutil: 2.2
2014-02-17 14:46:53,492: 34-32 : pytz: 2013.9
2014-02-17 14:46:53,492: 34-32 : bottleneck: 0.8.0
2014-02-17 14:46:53,492: 34-32 : tables: 3.1.0
2014-02-17 14:46:53,492: 34-32 : numexpr: 2.3
2014-02-17 14:46:53,492: 34-32 : matplotlib: 1.3.1
2014-02-17 14:46:53,492: 34-32 : openpyxl: 1.8.3
2014-02-17 14:46:53,492: 34-32 : xlrd: 0.9.2
2014-02-17 14:46:53,493: 34-32 : xlwt: None
2014-02-17 14:46:53,493: 34-32 : xlsxwriter: 0.5.2
2014-02-17 14:46:53,493: 34-32 : lxml: None
2014-02-17 14:46:53,493: 34-32 : bs4: None
2014-02-17 14:46:53,493: 34-32 : html5lib: None
2014-02-17 14:46:53,493: 34-32 : bq: None
2014-02-17 14:46:53,493: 34-32 : apiclient: None
2014-02-17 14:46:53,493: 34-32 : rpy2: None
2014-02-17 14:46:53,493: 34-32 : sqlalchemy: 0.9.2
2014-02-17 14:46:53,494: 34-32 : pymysql: None
2014-02-17 14:46:53,494: 34-32 : psycopg2: None
```

@TomAugspurger Do you remember the background of adding `result_name` kw? Other set ops doesn't have the kw. 

Should be added to others if it's useful, otherwise deprecate? 

Huh, I have no idea. I don't know that I've ever used it, so I'm not sure why I included that.