Question: What I am trying to achieve I have a pandas DataFrame in a long format, containing values for different groups. I want to compute and apply a quantile based-binning (e.g. quintiles in this example) to each group of the DataFrame. I also need to be able to keep the bins edges for each group and apply the same labelling (via pd.cut) to a new DataFrame. E.g. for each of group, find the quintiles and assign them to a new column value_label. import numpy as np import pandas as pd df1 = pd.DataFrame({"group": "A", "val": np.random.normal(loc=10, scale=5, size=100)}) df2 = pd.DataFrame({"group": "B", "val": np.random.normal(loc=5, scale=3, size=100)}) df = pd.concat([df1, df2], ignore_index=True) # apply qcut labels_and_bins = df.groupby("group")["val"].apply( lambda x: pd.qcut(x, q=5, duplicates="drop", retbins=True) ) # where e.g. labels_and_bins["A"][0] # are the applied labels to all the rows in group "A" labels_and_bins["A"][1] # are the bin edges to apply the same segmentation going forward for group in df.group.unique(): df.loc[df["group"] == group, "value_label"] = labels_and_bins[group][0]  When I try to run this, at the second iteration I get the following error: TypeError: Cannot set a Categorical with another, without identical categories So, essentially I need Pandas to accept extending the categories belonging to the column dtype. What I have tried/considered Transform Using .transform() would probably solve the issue of assigning the label on the first DataFrame, but it's not clear to me how I could re-use the identified bins in future iterations Unioning the categorical dtype I tried two approaches: add_categories() labels_and_bins['A'][0].cat.add_categories(labels_and_bins['B'][0].cat.as_unordered()) This results in ValueError: Categorical categories must be unique union_categoricals() pd.api.types.union_categoricals( [labels_and_bins["A"][0].cat.as_unordered(), labels_and_bins["B"][0].cat.as_unordered()].get_inde )  This results in InvalidIndexError: cannot handle overlapping indices; use IntervalIndex.get_indexer_non_unique One solution Get rid of the Interval object by calling qcut without labels, e.g.: labels_and_bins = df.groupby("group")["val"].apply( lambda x: pd.qcut(x, q=5, duplicates="drop", retbins=True, labels=False) )  However I would be interested in a way to keep the Interval if possible for better interpretability  Overall this feels like a big anti-pattern, so I'm confident I'm missing a much more basic solution to this problem! Thanks in advance for your inputs!

Answer: You can use groupby(...).quantile to get your bins. Getting the labels is the tricky part, if you want to have the same type of labels that cut and qcut returns you can convert this result to an pandas.arrays.IntervalArray and grab the left edges from there. bins = ( df.groupby('group')['val'] .quantile([0, .2, .4, .6, .8, 1]) # all of the below is to get those silly labels .groupby('group').apply(lambda s: pd.Series(pd.arrays.IntervalArray.from_breaks(s)) ) .to_frame('labels') .assign(breaks=lambda d: d['labels'].array.left) ) print(bins) labels breaks group A 0 (1.0537779249398262, 6.012235091686201] 1.053778 1 (6.012235091686201, 8.290140550780626] 6.012235 2 (8.290140550780626, 10.486391253124191] 8.290141 3 (10.486391253124191, 14.187164706356482] 10.486391 4 (14.187164706356482, 30.41871778451676] 14.187165 B 0 (-2.223397003597773, 2.929644956582803] -2.223397 1 (2.929644956582803, 4.205189382479166] 2.929645 2 (4.205189382479166, 5.512744579486252] 4.205189 3 (5.512744579486252, 6.9496607395781504] 5.512745 4 (6.9496607395781504, 11.247907758459675] 6.949661 # Note that the 'labels' columns are not actually a bunch of strings, # but is a `Series` wrapped around the underlying `IntervalArray` # you can access the `IntervalArray` via the `.array` property  From here you can use pd.merge_asof to align your bins to other datasets or even back to your original DataFrame! # all of the sorting is required for `merge_asof` result = pd.merge_asof( df.sort_values('val'), bins.sort_values('breaks'), left_on='val', right_on='breaks', by='group' ) print(result.sample(8, random_state=0)) group val labels breaks 18 B 2.323406 (-2.223397003597773, 2.929644956582803] -2.223397 170 A 12.238075 (10.486391253124191, 14.187164706356482] 10.486391 107 A 6.680556 (6.012235091686201, 8.290140550780626] 6.012235 98 A 6.499656 (6.012235091686201, 8.290140550780626] 6.012235 177 A 13.156868 (10.486391253124191, 14.187164706356482] 10.486391 182 A 14.734745 (14.187164706356482, 30.41871778451676] 14.187165 5 B 0.227872 (-2.223397003597773, 2.929644956582803] -2.223397 146 A 9.562343 (8.290140550780626, 10.486391253124191] 8.290141