Repository: pandas-dev/pandas

Issue:
Groupby all levels except the specified ones
Inspired by 

> http://stackoverflow.com/questions/25606478/grouping-by-everything-except-for-one-index-column-in-pandas

 I was thinking if it would be useful to add a parameter to the groupby function (e.g. allbut=True) so that pandas will group all levels except the specified ones) .

I suggest so because in a multilevel index with several levels, eliminating just one level requires a quite verbose approach 
Added FrozenList subtraction
 - [x] closes #15475
 - [x] tests added / passed
 - [x] passes ``git diff upstream/master | flake8 --diff``
 - [x] whatsnew entry



Discussion:
Actually this would be quite trivial to implement. We currently support a union (set ops) for ``FrozenList``, e.g. ``df.index.names + ['foo']``, so supporting difference is easy.

This might break other things, so need some tests. Also would be nice to support intersection, ``&`` as well.

```
In [3]: df.groupby(level=df.index.names-['name']).mean()
Out[3]: 
                 accuracy  velocity
condition trial                    
a         1      0.661597  0.277299
          2      0.953187 -0.875767
b         1     -0.836886  0.944629
          2     -0.251443  0.799134

In [4]: quit()
(pandas) bash-3.2$ git diff
diff --git a/pandas/core/base.py b/pandas/core/base.py
index 92ec6bb..b8a4c1d 100644
--- a/pandas/core/base.py
+++ b/pandas/core/base.py
@@ -711,6 +711,12 @@ class FrozenList(PandasObject, list):
 
     __iadd__ = __add__
 
+    def __sub__(self, other):
+        if isinstance(other, tuple):
+            other = list(other)
+        return self.__class__([ e for e in self if e not in set(other)])
+
+
     # Python 2 compat
     def __getslice__(self, i, j):
         return self.__class__(super(FrozenList, self).__getslice__(i, j))
```
Great idea. Operations on Frozenlists would be really helpful indeed. 
Thanks for the quick reply
going to merge this soon: https://github.com/pandas-dev/pandas/pull/15477

should provide a bit more runway for expanding / testing these structures.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15506?src=pr&el=h1) Report
> Merging [#15506](https://codecov.io/gh/pandas-dev/pandas/pull/15506?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/2340fb8b97a3f65ce4f630075849e42fb256e3be?src=pr&el=desc) will **decrease** coverage by `-0.03%`.
> The diff coverage is `100%`.


```diff
@@            Coverage Diff             @@
##           master   #15506      +/-   ##
==========================================
- Coverage   91.06%   91.04%   -0.03%     
==========================================
  Files         136      136              
  Lines       49099    49182      +83     
==========================================
+ Hits        44714    44779      +65     
- Misses       4385     4403      +18
```


| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15506?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/indexes/frozen.py](https://codecov.io/gh/pandas-dev/pandas/compare/2340fb8b97a3f65ce4f630075849e42fb256e3be...428a1b33f5a48bce94dccebb47949193610a749d?src=pr&el=tree#diff-cGFuZGFzL2luZGV4ZXMvZnJvemVuLnB5) | `93.24% <100%> (-0.31%)` | :x: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/compare/2340fb8b97a3f65ce4f630075849e42fb256e3be...428a1b33f5a48bce94dccebb47949193610a749d?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `40% <0%> (-46.67%)` | :x: |
| [pandas/util/decorators.py](https://codecov.io/gh/pandas-dev/pandas/compare/2340fb8b97a3f65ce4f630075849e42fb256e3be...428a1b33f5a48bce94dccebb47949193610a749d?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvZGVjb3JhdG9ycy5weQ==) | `67.27% <0%> (-1.96%)` | :x: |
| [pandas/tools/merge.py](https://codecov.io/gh/pandas-dev/pandas/compare/2340fb8b97a3f65ce4f630075849e42fb256e3be...428a1b33f5a48bce94dccebb47949193610a749d?src=pr&el=tree#diff-cGFuZGFzL3Rvb2xzL21lcmdlLnB5) | `91.78% <0%> (-0.35%)` | :x: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/compare/2340fb8b97a3f65ce4f630075849e42fb256e3be...428a1b33f5a48bce94dccebb47949193610a749d?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `81.87% <0%> (-0.19%)` | :x: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/compare/2340fb8b97a3f65ce4f630075849e42fb256e3be...428a1b33f5a48bce94dccebb47949193610a749d?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.74% <0%> (-0.1%)` | :x: |
| [pandas/core/algorithms.py](https://codecov.io/gh/pandas-dev/pandas/compare/2340fb8b97a3f65ce4f630075849e42fb256e3be...428a1b33f5a48bce94dccebb47949193610a749d?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYWxnb3JpdGhtcy5weQ==) | `94.48% <0%> (ø)` | :white_check_mark: |
| [pandas/core/categorical.py](https://codecov.io/gh/pandas-dev/pandas/compare/2340fb8b97a3f65ce4f630075849e42fb256e3be...428a1b33f5a48bce94dccebb47949193610a749d?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY2F0ZWdvcmljYWwucHk=) | `96.92% <0%> (+0.01%)` | :white_check_mark: |
| [pandas/formats/style.py](https://codecov.io/gh/pandas-dev/pandas/compare/2340fb8b97a3f65ce4f630075849e42fb256e3be...428a1b33f5a48bce94dccebb47949193610a749d?src=pr&el=tree#diff-cGFuZGFzL2Zvcm1hdHMvc3R5bGUucHk=) | `96.8% <0%> (+0.63%)` | :white_check_mark: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15506?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15506?src=pr&el=footer). Last update [2340fb8...428a1b3](https://codecov.io/gh/pandas-dev/pandas/compare/2340fb8b97a3f65ce4f630075849e42fb256e3be...428a1b33f5a48bce94dccebb47949193610a749d?el=footer&src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
I want to raise some concern regarding adding this feature (without wanting to dismiss your effort @bthayer2365 !). I think we should think about this some more. 

Adding this would introduce a discrepancy between index level names and column names, while we actually try to lessen the gap (eg by being able to specify an index level name in places where you can specify a column name). This is actually a feature we *removed* from column names, in favor of the `difference` method. 
So that is maybe also an alternative? Putting this in a method instead of overloading subtraction? (although I certainly agree what we try to obtain here is useful, and as a method it is more verbose)


no conceptual problem with making these real setops, e.g. ``.union``, ``.difference`` as @jorisvandenbossche suggests. Publicly these are *only* used for names of levels anyhow.
Would you like me to change `__add__` to `union` as well?
yes; u will have to deprecate the add though 
To do that, do I just decorate it with `@deprecated`, or is there more?
that should work (and add in whatsnew as well)
iadd too
I tried resolving the conflicts and merging manually, but the check failed, so I reverted back to before I resolved the conflict. Am I supposed to resolve the merge conflicts and merge master?
@bthayer2365 Yep, you need to resolve conflicts on your local branch. You should do:

```
git pull upstream master --rebase --tags
```
And resolve the conflicts one by one, doing `git rebase --continue` once each one has been resolved. Then push to your fork.
yes the idea is to rebase on master. But don't worry about that as much as passing tests (of course I think travis / amazon is having issues atm).
Okay, should we just wait a bit to see if the tests work when everything
starts working again? Everything worked on the command line

On Feb 28, 2017 7:59 PM, "Jeff Reback" <notifications@github.com> wrote:

> yes the idea is to rebase on master. But don't worry about that as much as
> passing tests (of course I think travis / amazon is having issues atm).
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/15506#issuecomment-283221620>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AF0RY4QdddVNhcFqKprMdBJcVTc45us2ks5rhNEegaJpZM4MMIo2>
> .
>

I'll leave it like this and see if it passes
you'll want to rebase on master as travis was having some issues
Anything else? All tests passed 👍 

Resolving patch:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -1529,6 +1529,8 @@ def to_feather(self, fname):
         from pandas.io.feather_format import to_feather
         to_feather(self, fname)
 
+    @Substitution(header='Write out column names. If a list of string is given, \
+it is assumed to be aliases for the column names')
     @Appender(fmt.docstring_to_string, indents=1)
     def to_string(self, buf=None, columns=None, col_space=None, header=True,
                   index=True, na_rep='NaN', formatters=None, float_format=None,
@@ -1556,6 +1558,7 @@ def to_string(self, buf=None, columns=None, col_space=None, header=True,
             result = formatter.buf.getvalue()
             return result
 
+    @Substitution(header='whether to print column labels, default True')
     @Appender(fmt.docstring_to_string, indents=1)
     def to_html(self, buf=None, columns=None, col_space=None, header=True,
                 index=True, na_rep='NaN', formatters=None, float_format=None,
@@ -1609,6 +1612,8 @@ def to_html(self, buf=None, columns=None, col_space=None, header=True,
         if buf is None:
             return formatter.buf.getvalue()
 
+    @Substitution(header='Write out column names. If a list of string is given, \
+it is assumed to be aliases for the column names.')
     @Appender(fmt.common_docstring + fmt.return_docstring, indents=1)
     def to_latex(self, buf=None, columns=None, col_space=None, header=True,
                  index=True, na_rep='NaN', formatters=None, float_format=None,
diff --git a/pandas/formats/format.py b/pandas/formats/format.py
--- a/pandas/formats/format.py
+++ b/pandas/formats/format.py
@@ -20,9 +20,9 @@
                                  is_float,
                                  is_numeric_dtype,
                                  is_datetime64_dtype,
-                                 is_timedelta64_dtype)
+                                 is_timedelta64_dtype,
+                                 is_list_like)
 from pandas.types.generic import ABCSparseArray
-
 from pandas.core.base import PandasObject
 from pandas.core.index import Index, MultiIndex, _ensure_index
 from pandas import compat
@@ -53,7 +53,7 @@
     col_space : int, optional
         the minimum width of each column
     header : bool, optional
-        whether to print column labels, default True
+        %(header)s
     index : bool, optional
         whether to print index (row) labels, default True
     na_rep : string, optional
@@ -487,32 +487,38 @@ def _to_str_columns(self):
         # may include levels names also
 
         str_index = self._get_formatted_index(frame)
-        str_columns = self._get_formatted_column_labels(frame)
 
-        if self.header:
+        if not is_list_like(self.header) and not self.header:
             stringified = []
             for i, c in enumerate(frame):
-                cheader = str_columns[i]
-                max_colwidth = max(self.col_space or 0, *(self.adj.len(x)
-                                                          for x in cheader))
                 fmt_values = self._format_col(i)
                 fmt_values = _make_fixed_width(fmt_values, self.justify,
-                                               minimum=max_colwidth,
+                                               minimum=(self.col_space or 0),
                                                adj=self.adj)
-
-                max_len = max(np.max([self.adj.len(x) for x in fmt_values]),
-                              max_colwidth)
-                cheader = self.adj.justify(cheader, max_len, mode=self.justify)
-                stringified.append(cheader + fmt_values)
+                stringified.append(fmt_values)
         else:
+            if is_list_like(self.header):
+                if len(self.header) != len(self.columns):
+                    raise ValueError(('Writing %d cols but got %d aliases'
+                                      % (len(self.columns), len(self.header))))
+                str_columns = [[label] for label in self.header]
+            else:
+                str_columns = self._get_formatted_column_labels(frame)
+
             stringified = []
             for i, c in enumerate(frame):
+                cheader = str_columns[i]
+                header_colwidth = max(self.col_space or 0,
+                                      *(self.adj.len(x) for x in cheader))
                 fmt_values = self._format_col(i)
                 fmt_values = _make_fixed_width(fmt_values, self.justify,
-                                               minimum=(self.col_space or 0),
+                                               minimum=header_colwidth,
                                                adj=self.adj)
 
-                stringified.append(fmt_values)
+                max_len = max(np.max([self.adj.len(x) for x in fmt_values]),
+                              header_colwidth)
+                cheader = self.adj.justify(cheader, max_len, mode=self.justify)
+                stringified.append(cheader + fmt_values)
 
         strcols = stringified
         if self.index:


Test patch:
diff --git a/pandas/tests/formats/test_format.py b/pandas/tests/formats/test_format.py
--- a/pandas/tests/formats/test_format.py
+++ b/pandas/tests/formats/test_format.py
@@ -1125,6 +1125,17 @@ def test_to_string_no_header(self):
 
         self.assertEqual(df_s, expected)
 
+    def test_to_string_specified_header(self):
+        df = DataFrame({'x': [1, 2, 3], 'y': [4, 5, 6]})
+
+        df_s = df.to_string(header=['X', 'Y'])
+        expected = '   X  Y\n0  1  4\n1  2  5\n2  3  6'
+
+        self.assertEqual(df_s, expected)
+
+        with tm.assertRaises(ValueError):
+            df.to_string(header=['X'])
+
     def test_to_string_no_index(self):
         df = DataFrame({'x': [1, 2, 3], 'y': [4, 5, 6]})
 
diff --git a/pandas/tests/formats/test_to_latex.py b/pandas/tests/formats/test_to_latex.py
--- a/pandas/tests/formats/test_to_latex.py
+++ b/pandas/tests/formats/test_to_latex.py
@@ -428,6 +428,51 @@ def test_to_latex_no_header(self):
 
         assert withoutindex_result == withoutindex_expected
 
+    def test_to_latex_specified_header(self):
+        # GH 7124
+        df = DataFrame({'a': [1, 2], 'b': ['b1', 'b2']})
+        withindex_result = df.to_latex(header=['AA', 'BB'])
+        withindex_expected = r"""\begin{tabular}{lrl}
+\toprule
+{} & AA &  BB \\
+\midrule
+0 &  1 &  b1 \\
+1 &  2 &  b2 \\
+\bottomrule
+\end{tabular}
+"""
+
+        assert withindex_result == withindex_expected
+
+        withoutindex_result = df.to_latex(header=['AA', 'BB'], index=False)
+        withoutindex_expected = r"""\begin{tabular}{rl}
+\toprule
+AA &  BB \\
+\midrule
+ 1 &  b1 \\
+ 2 &  b2 \\
+\bottomrule
+\end{tabular}
+"""
+
+        assert withoutindex_result == withoutindex_expected
+
+        withoutescape_result = df.to_latex(header=['$A$', '$B$'], escape=False)
+        withoutescape_expected = r"""\begin{tabular}{lrl}
+\toprule
+{} & $A$ & $B$ \\
+\midrule
+0 &   1 &  b1 \\
+1 &   2 &  b2 \\
+\bottomrule
+\end{tabular}
+"""
+
+        assert withoutescape_result == withoutescape_expected
+
+        with tm.assertRaises(ValueError):
+            df.to_latex(header=['A'])
+
     def test_to_latex_decimal(self, frame):
         # GH 12031
         frame.to_latex()