Question: TL;DR: How can we achieve something similar to Group By Roll Up with any kind of aggregates in pandas? (Credit to @Scott Boston for this term) I have following dataframe:  P Q R S T 0 PLAC NR F HOL F 1 PLAC NR F NHOL F 2 TRTB NR M NHOL M 3 PLAC NR M NHOL M 4 PLAC NR F NHOL F 5 PLAC R M NHOL M 6 TRTA R F HOL F 7 TRTA NR F HOL F 8 TRTB NR F NHOL F 9 PLAC NR F NHOL F 10 TRTB NR F NHOL F 11 TRTB NR M NHOL M 12 TRTA NR F HOL F 13 PLAC NR F HOL F 14 PLAC R F NHOL F  For a list of columns ['Q', 'R', 'S', 'T'], I want to calculate some aggregates on P column on following 4 list of grouping columns:  ['Q'] ['Q', 'R'] ['Q', 'R', 'S'] ['Q', 'R', 'S', 'T']  I've already written the code to group above dataframes in an increasing number of columns, and calculate the aggregate (using count for the shake of simplicity) on each of the groupby object, and finally concatenate them: cols = list('QRST') aggCol = 'P' groupCols = [] result = [] for col in cols: groupCols.append(col) result.append(df.groupby(groupCols)[aggCol].agg(count='count').reset_index()) result = pd.concat(result)[groupCols+['count']]  However, I've strong feeling that above method is not so efficient in terms of CPU time. Is there a more efficient way to apply aggregate on such continuously increasing number of columns for grouping? Why I think it is not so efficient is because: For above values, in first iteration, it groups the dataframe on Q column then calculates aggregate. Then in next iteration it groups the dataframe on Q and R, that means it again needs to group it by Q then R, but it was already grouped by Q in the first iteration, so the same operation is repeating. If there is some way to utilize the previously created groups, I think it'll be efficient. OUTPUT:  Q R S T count 0 NR NaN NaN NaN 12 1 R NaN NaN NaN 3 0 NR F NaN NaN 9 1 NR M NaN NaN 3 2 R F NaN NaN 2 3 R M NaN NaN 1 0 NR F HOL NaN 4 1 NR F NHOL NaN 5 2 NR M NHOL NaN 3 3 R F HOL NaN 1 4 R F NHOL NaN 1 5 R M NHOL NaN 1 0 NR F HOL F 4 1 NR F NHOL F 5 2 NR M NHOL M 3 3 R F HOL F 1 4 R F NHOL F 1 5 R M NHOL M 1  I already looked into Is there an equivalent of SQL GROUP BY ROLLUP in Python pandas? and Pandas Pivot tables row subtotals, they don't work in my case, I already tried them i.e. These method can be used to get the count only, and immediately fail even for unique counts when the same identifier appears for more than one values: pd.pivot_table(df, aggCol, columns=cols, aggfunc='count', margins=True).T.reset_index() Q R S T P 0 NR F HOL F 4 1 NR F NHOL F 5 2 NR M NHOL M 3 3 NR All 3 4 R F HOL F 1 5 R F NHOL F 1 6 R M NHOL M 1 7 R All 3  UPDATE In order to avoid any unnecessary confusion with just getting the count as per suggestion in the comment, I have added it for the mean as aggregate, changing P column to a numeric type:  P Q R S T 0 9 NR F HOL F 1 7 NR F NHOL F 2 3 NR M NHOL M 3 9 NR M NHOL M 4 1 NR F NHOL F 5 0 R M NHOL M 6 1 R F HOL F 7 7 NR F HOL F 8 2 NR F NHOL F 9 2 NR F NHOL F 10 1 NR F NHOL F 11 2 NR M NHOL M 12 3 NR F HOL F 13 6 NR F HOL F 14 0 R F NHOL F cols = list('QRST') cols = list('QRST') aggCol = 'P' groupCols = [] result = [] for col in cols: groupCols.append(col) result.append(df.groupby(groupCols)[aggCol] .agg(agg=np.mean) .round(2).reset_index()) result = pd.concat(result)[groupCols+['agg']]  >>> result Q R S T agg 0 NR NaN NaN NaN 4.33 1 R NaN NaN NaN 0.33 0 NR F NaN NaN 4.22 1 NR M NaN NaN 4.67 2 R F NaN NaN 0.50 3 R M NaN NaN 0.00 0 NR F HOL NaN 6.25 1 NR F NHOL NaN 2.60 2 NR M NHOL NaN 4.67 3 R F HOL NaN 1.00 4 R F NHOL NaN 0.00 5 R M NHOL NaN 0.00 0 NR F HOL F 6.25 1 NR F NHOL F 2.60 2 NR M NHOL M 4.67 3 R F HOL F 1.00 4 R F NHOL F 0.00 5 R M NHOL M 0.00 

Answer: Building on the idea of @ScottBoston (progressive aggregation, i.e., repeatedly aggregating on the previous aggregate result), we can do something that is relatively generic with regard to the aggregation function, if that function can be expressed as a composition of functions ((f3 ∘ f2 ∘ f2 ∘ ... ∘ f1)(x), or in other words: f3(f2(f2(...(f1(x)))))). For example, sum works fine as is, because sum is associative, so the sum of group sums is the sum of the whole. For count, the initial function (f1) is indeed count, but f2 has to be sum, and the final f3 has to be identity. For mean, the initial function (f1) has to produce two quantities: sum and count. The intermediary function f2 can be sum, and the final function (f3) has then to be the ratio of the two quantities. Here is a rough template, with a few functions defined. As an added bonus, the function also optionally produces a grand total: # think map-reduce: first map, then reduce (arbitrary number of times), then map to result myfuncs = { 'sum': [sum, sum], 'prod': ['prod', 'prod'], 'count': ['count', sum], 'set': [set, lambda g: set.union(*g)], 'list': [list, sum], 'mean': [[sum, 'count'], sum, lambda r: r[0]/r[1]], 'var': [ [lambda x: (x**2).sum(), sum, 'count'], sum, lambda r: (r[0].sum() - r[1].sum()**2 / r[2]) / (r[2] - 1)], 'std': [ [lambda x: (x**2).sum(), sum, 'count'], sum, lambda r: np.sqrt((r[0].sum() - r[1].sum()**2 / r[2]) / (r[2] - 1))], } totalCol = '__total__' def agg(df, cols, aggCol, fun, total=True): if total: cols = [totalCol] + cols df = df.assign(__total__=0) funs = myfuncs[fun] b = df.groupby(cols).agg({aggCol: funs[0]}) frames = [b.reset_index()] for k in range(1, len(cols)): b = b.groupby(cols[:-k]).agg(funs[1]) frames.append(b.reset_index()) result = pd.concat(frames).reset_index(drop=True) result = result[frames[0].columns] if len(funs) > 2: s = result[aggCol].apply(funs[2], axis=1) result = result.drop(aggCol, axis=1, level=0) result[aggCol] = s result.columns = result.columns.droplevel(-1) if total: result = result.drop(columns=[totalCol]) return result  Examples cols = list('QRST') aggCol = 'P' >>> agg(df, cols, aggCol, 'count') Q R S T P 0 NR F HOL F 4 1 NR F NHOL F 5 2 NR M NHOL M 3 3 R F HOL F 1 .. ... ... ... ... .. 15 R M NaN NaN 1 16 NR NaN NaN NaN 12 17 R NaN NaN NaN 3 18 NaN NaN NaN NaN 15  >>> agg(df, cols, aggCol, 'mean') Q R S T P 0 NR F HOL F 6.250000 1 NR F NHOL F 2.600000 2 NR M NHOL M 4.666667 3 R F HOL F 1.000000 .. ... ... ... ... ... 15 R M NaN NaN 0.000000 16 NR NaN NaN NaN 4.333333 17 R NaN NaN NaN 0.333333 18 NaN NaN NaN NaN 3.533333  >>> agg(df, cols, aggCol, 'sum') Q R S T P 0 NR F HOL F 25 1 NR F NHOL F 13 2 NR M NHOL M 14 3 R F HOL F 1 .. ... ... ... ... .. 15 R M NaN NaN 0 16 NR NaN NaN NaN 52 17 R NaN NaN NaN 1 18 NaN NaN NaN NaN 53  >>> agg(df, cols, aggCol, 'set') Q R S T P 0 NR F HOL F {9, 3, 6, 7} 1 NR F NHOL F {1, 2, 7} 2 NR M NHOL M {9, 2, 3} 3 R F HOL F {1} .. ... ... ... ... ... 15 R M NaN NaN {0} 16 NR NaN NaN NaN {1, 2, 3, 6, 7, 9} 17 R NaN NaN NaN {0, 1} 18 NaN NaN NaN NaN {0, 1, 2, 3, 6, 7, 9}  >>> agg(df, cols, aggCol, 'std') Q R S T P 0 NR F HOL F 2.500000 1 NR F NHOL F 2.509980 2 NR M NHOL M 3.785939 3 R F HOL F NaN .. ... ... ... ... ... 15 R M NaN NaN NaN 16 NR NaN NaN NaN 3.055050 17 R NaN NaN NaN 0.577350 18 NaN NaN NaN NaN 3.181793  Notes  The code is not as 'pure' as I would like it to be. There are two reasons for that:  groupby likes to do some magic to the shape of the result. For example, in some cases (but not always, strangely enough), if there is only one resulting group, the output is sometimes squeezed to a Series.  the pandas arithmetic on set seems sometimes bogus, or finicky at best. My initial definition had: 'set': [set, sum] and this was working reasonably well (pandas seems to sometimes understand that .agg(sum) on a Series of set objects, it is desirable to apply set.union), except that, weirdly enough, in some conditions we'd get a NaN result instead.    This only works for a single aggCol.  The expressions for std and var are relatively naive. For improved numerical stability, see Standard Deviation: Rapid calculation methods.   Speed Since the original posting of this answer, another solution has been proposed by @U12-Forward. After a bit of cleaning (e.g. not using recursion, and changing the agg dtype to whatever it needs to be, instead of object, this solution becomes: def v_u12(df, cols, aggCol, fun): newdf = pd.DataFrame(columns=cols) for count in range(1, len(cols)+1): groupcols = cols[:count] newdf = newdf.append( df.groupby(groupcols)[aggCol].agg(fun).reset_index().reindex(columns=groupcols + [aggCol]), ignore_index=True, ) return newdf  To compare speed, let's generate DataFrames of arbitrary sizes: def gen_example(n, m=4, seed=-1): if seed >= 0: np.random.seed(seed) aggCol = 'v' cols = list(ascii_uppercase)[:m] choices = [['R', 'NR'], ['F', 'M'], ['HOL', 'NHOL']] df = pd.DataFrame({ aggCol: np.random.uniform(size=n), **{ k: np.random.choice(choices[np.random.randint(0, len(choices))], n) for k in cols }}) return df # example >>> gen_example(8, 5, 0) v A B C D E 0 0.548814 R M F M NR 1 0.715189 R M M M R 2 0.602763 NR M M F R 3 0.544883 R F F M NR 4 0.423655 NR M F F NR 5 0.645894 NR F M M R 6 0.437587 R M F M NR 7 0.891773 R F M M R  We can now compare speed over a range of sizes, using the excellent perfplot package, plus a few definitions: m = 4 aggCol, *cols = gen_example(2, m).columns fun = 'mean' def ours(df): funname = fun if isinstance(fun, str) else fun.__name__ return agg(df, cols, aggCol, funname, total=False) def u12(df): return v_u12(df, cols, aggCol, fun) def equality_check(a, b): a = a.sort_values(cols).reset_index(drop=True) b = b.sort_values(cols).reset_index(drop=True) non_numeric = a[aggCol].dtype == 'object' if non_numeric: return a[cols+[aggCol]].equals(b[cols+[aggCol]]) return a[cols].equals(b[cols]) and np.allclose(a[aggCol], b[aggCol]) perfplot.show( time_unit='auto', setup=lambda n: gen_example(n, m), kernels=[ours, u12], n_range=[2 ** k for k in range(4, 21)], equality_check=equality_check, xlabel=f'n rows\n(m={m} columns, fun={fun})' )  The comparisons for a few aggregation functions and m values are below (y-axis is average time: lower is better):     m fun perfplot     4 'mean'    10 'mean'    10 'sum'    4 'set'      For functions that are not associative (e.g. 'mean'), our "progressive re-aggregation" needs to keep track of multiple values (e.g., for mean: sum and count), so for relatively small DataFrames, the speed is about twice as slow as u12. But as the size grows, the gain of the re-aggregation overcomes that and ours becomes faster.