SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Marina Knittel Max Springer John Dickerson MohammadTaghi Hajiaghayi
{mknittel,mss423}@umd.edu
{john,hajiagha}@cs.umd.edu
Research in fair machine learning, and particularly clustering, has been crucial in recent years given the many ethical controversies that modern intelligent systems have posed. Ahmadian et al. [2020] established the study of fairness in hierarchical clustering, a stronger, more structured variant of its well-known flat counterpart, though their proposed algorithm that optimizes for Dasgupta's [2016] famous cost function was highly theoretical. Knittel et al. [2023] then proposed the first practical fair approximation for cost, however they were unable to break the polynomial-approximate barrier they posed as a hurdle of interest. We break this barrier, proposing the first truly polylogarithmic-approximate low-cost fair hierarchical clustering, thus greatly bridging the gap between the best fair and vanilla hierarchical clustering approximations.
Clustering is a pervasive machine learning technique which has filled a vital niche in every day computer systems. Extending upon this, a hierarchical clustering is a recursively defined clustering where each cluster is partitioned into two or more clusters, and so on. This adds structure to flat clustering, giving an algorithm the ability to depict data similarity at different resolutions as well as an ancestral relationship between data points, as in the phylogenetic tree [Kriskov et al. 2003].
On top of computational biology, hierarchical clustering has found various uses across computer imaging [Chen et al., 2021b, Selvan et al., 2005], computer security [Chen et al., 2020, 2021a], natural language processing [Ramanath et al., 2013], and much more. Moreover, it can be applied to any flat clustering problem where the number of desired clusters is not given. Specifically, a hierarchical clustering can be viewed as a structure of clusterings at different resolutions that all agree with each other (i.e., two points clustered together in a higher resolution clustering will also be together in a lower resolution clustering). Generally, hierarchical clustering techniques are quite impactful on modern technology, and it is important to guarantee they are both effective and unharmful.
Researchers have recognized the harmful biases unchecked machine learning programs pose. A few examples depicting racial discrimination include allocation of health care [Ledford, 2019], presentation of ads suggestive of arrest records [Sweeney, 2013], prediction of hiring success [Bogen and Rieke, 2018], and estimation of recidivism risk [Angwin et al., 2016]. A popular solution that has been extensively studied in the past decade is fair machine learning. Here, fairness concerns the mitigation of bias, particularly against protected classes. Most often, fairness is an additional constraint on the allowed solution space; we optimize for problems in light of this constraint. For instance, the notion of individual fairness introduced by the foundational work of [Dwork et al., 2012] deems that an output must guarantee that any two individuals who are similar are classified similarly.
Figure 1: A hierarchical clustering of news articles. Red articles are conservative, blue are liberal. On the left is the optimal unfair hierarchy. We alter the hierarchy slightly on the right to achieve fairness. Now, the user's query for global warming will yield both liberal and conservative articles.
In line with previous work in clustering and hierarchical clustering, this paper utilizes the notion of group fairness, which enforces that different protected classes receive a proportionally similar distribution of classifications (in our case, cluster placement). Chierichetti et al. [2017] first introduced this as a constraint for the flat clustering problem, arguing that it mitigates a system's disparate impact, or non-proportional impact on different demographics. This notion of fair clustering has been similarly leveraged and extended by a vast range of works in both flat [Ahmadian et al., 2019], Bera et al., 2019], Bercea et al., 2019] and hierarchical [Ahmadian et al., 2020], Knittel et al., 2023] clustering.
To illustrate our fairness concept, consider the following application (Figure 1): a news database is structured as a hierarchical clustering of search terms, where a search term is associated with a cluster of news articles to output to the reader, and more specific search terms access finer-resolution clusters. When a user searches for a term, we simply identify the corresponding cluster and output the contained articles. However, as is, the system does not account for the political skew of articles. In Figure 1 we label conservative-leaning articles in red and liberal-leaning articles in blue. We can see that in this example, when the user searches for global warming articles, they will only see liberal articles. To resolve this, we add a group fairness constraint on our cluster: for example, require at least  $1/3$  of the articles in each cluster to be of each political skew. This guarantees (as depicted on the right) that the outputted articles will always be at least  $1/3$  liberal and  $1/3$  conservative, thus guaranteeing the user is exposed to a range of perspectives along this political axis. This notion of fairness, which we formally define in Definition 3, has been explored in the context of hierarchical clustering in both [Ahmadian et al. 2020] and Knittel et al. 2023].
This paper is concerned with approximations for fair low-cost hierarchical clustering. Cost, first defined by Dasgupta [2016] and formally stated in Definition 2 is perhaps the most natural and well-motivated optimization metric for hierarchical clustering evaluation. Effectively, every pair of points contributes an additive cost of similarity  $\times$  clust_size, where the latter is the size of the smallest cluster in the hierarchy containing both points. Unfortunately, it is quite difficult to optimize for (the best being  $O(\sqrt{\log n})$ -approximations [Charikar and Chatziafratis, 2017; Dasgupta, 2016]; it hypothesized to not be  $O(1)$ -approximable [Charikar and Chatziafratis, 2017]). This appears to be even more difficult in the hierarchical clustering literature. The first work to attempt this problem, Ahmadian et al. [2020], achieved a highly impractical  $O(n^{5/6} \log^{3/2} n)$ -approximation (not too far from the trivial  $O(n)$  upper bound), posing fair low-cost hierarchical clustering as an interesting and inherently difficult problem. Knittel et al. [2023] greatly improved this to a near-polylog approximation factor of  $O(n^{\delta} \log(n))$ , where  $\delta$  can be arbitrarily close to 0, and parameterizes a trade-off between approximation factor and degree of fairness. Still, a true polylog approximation was left as an open problem, one which we solve in this paper.
This work proposes the first polylogarithmic approximation for fair, low-cost hierarchical clustering. We leverage the work of Knittel et al. [2023] as a starting inspiration and create something much simpler, more direct, and better in both fairness and approximation. Like their algorithm, our algorithm starts with a low-cost unfair hierarchical clustering and then alters it with multiple well-defined and limited tree operators. This gives it a degree of explainability, in that the user can understand exactly the steps the algorithm took to achieve its result and why. In addition, our
algorithm achieves both relative cluster balance (i.e., clusters who are children of the same cluster have similar size) and fairness, along with a parameterizable trade-off between the two.
On top of the benefits of Knittel et al. [2023]’s techniques, we propose a greatly simplified algorithm. They initially proposed an algorithm that required four tree operators, however, we only require two of the four, and we greatly simplify the more complicated operator. This makes the algorithm simpler to understand and more implementable. We show that even with this reduced functionality, we can cleverly achieve both a better approximation and degree of fairness:
Theorem 1 (High-Level). Given a  $\gamma$ -approximation to the cost objective for a hierarchical clustering on set of  $n$  points, Algorithm 2 yields an  $O(\log^2 n)$  approximation to cost which is relatively fair in time  $O(n \log^2 n)$ . Moreover, all clusters are  $O(1 / \log n)$  balanced, i.e., any cluster's children clusters are within  $O(1 / \log n)$  size different.
To put this in perspective, previously, the best approximation for fair hierarchical clustering previously was  $O(n^{\delta}\mathrm{polylog}(n))$ , whereas the best unfair approximation is  $O(\sqrt{\log n})$ . Our work greatly reduces this gap by providing a true  $O(\mathrm{polylog}(n))$  approximation, moreover with a low degree of 2. Note that this is not the most general and rigorous form of our result--for that, we defer to Theorem 1 as stated in the body of our paper. This simpler version is found by setting  $k = O(1)$ ,  $h = O(\log n)$ , and  $\epsilon = O(1 / \log n)$  in the more general theorem (note we assume  $\lambda = O(1)$ ).
2.1 The Vanilla Problem
Fair clustering literature refers to the original problem variant, without fairness, as the "vanilla" problem. We define the vanilla problem of finding a low-cost hierarchical clustering here using our specific notation.
In this problem, we are given a complete graph  $G = (V, E, w)$  with a weight function  $w: E \to \mathbb{R}^+$  is a measure of the similarity between datapoints. Note the data is encoded as a complete tree because we require knowledge of all point-point relationships. We must construct a hierarchical clustering, represented by its dendrogram,  $T$ , with root denoted  $\mathrm{root}(T)$ .  $T$  is a tree with vertices corresponding to all clusters of the hierarchical clustering. Leaves of  $T$ , denoted  $\mathrm{leaves}(\mathrm{root}(T))$  correspond to the points in the dataset (i.e., singleton clusters). An internal node  $v$  corresponds to the cluster containing all leaf-data of the maximal subtree (i.e., contains all its descendants) rooted at  $v$ ,  $T[v]$ . In addition, we let  $u \wedge v$  denote the lowest common ancestor of  $u$  and  $v$  in  $T$ .
In order to define Dasgupta [2016]’s cost function, we use the same notational simplifications as Knittel et al. [2023]. For an edge  $e = (x,y) \in E$ , we say  $n_T(e) = |\mathrm{leaves}(T[x \wedge y])|$  is the size of the smallest cluster in the hierarchy containing  $e$ . Similarly, for a hierarchy node  $v$ ,  $n_T(v_i) = |\mathrm{leaves}(T[v_i])|$  is the size of the corresponding cluster. This is sufficient to introduce the notion of cost.
Definition 1 (Knittel et al. [2023]). The cost of  $e \in E$  in a graph  $G = (V, E, w)$  in a hierarchy  $T$  is  $\mathrm{cost}_T(e) = w(e) \cdot n_T(e)$ .
Dasgupta's cost function can then be written as a sum over the costs of all edges.
Definition 2 (Dasgupta [2016]). The cost of a hierarchy  $T$  on graph  $G = (V, E, w)$  is:
$$
\operatorname {c o s t} (T) = \sum_ {e \in E} \operatorname {c o s t} _ {T} (e)
$$
Our algorithm begins by assuming we have some approximate vanilla hierarchy,  $T$ . That is, if  $OPT$  is the optimal hierarchy tree, then  $\mathrm{cost}(T) \leq \alpha \cdot \mathrm{cost}(OPT)$  for some approximation factor  $\alpha$ . According to Dasgupta [2016], we can transform this hierarchy to be binary without increasing the cost. Our paper simply assumes our input is binary. We then produce a modified hierarchy  $T'$  which similar structure to  $T$  that guarantees fairness, i.e.,  $\mathrm{cost}(T') \leq \alpha' \cdot \mathrm{cost}(OPT)$  for some approximation factor  $\alpha' \geq \alpha$ . Note this comparison is being made to the vanilla  $OPT$ , as we are unsure, at this time, how to classify the optimal fair hierarchy. Note that the binary assumption may not hold when we consider adding a fairness constraint.
2.2 Fairness and Balance Constraints
We consider the fairness constraints based off those introduced by Chierichetti et al. [2017] and extended by Bercea et al. [2019]. On a graph  $G$  with colored vertices, let  $\ell(C)$  count the number of  $\ell$ -colored points in cluster  $C$ .
Definition 3 (Knittel et al. [2023]). Consider a graph  $G = (V, E, w)$  with vertices colored one of  $\lambda$  colors, and two vectors of parameters  $\alpha, \beta \in (0,1)^{\lambda}$  with  $\alpha_{\ell} \leq \beta_{\ell}$  for all  $\ell \in [\lambda]$ . A hierarchy  $T$  on  $G$  is fair if for any non-singleton cluster  $C$  in  $T$  and for every  $\ell \in [\lambda]$ ,  $\alpha_{\ell}|C| \leq \ell(C) \leq \beta_{\ell}|C|$ . Additionally, any cluster with a leaf child has only leaf children.
Effectively, we are given bounds  $\alpha_{\ell}$  and  $\beta_{\ell}$  for each color  $\ell$ . Every non-singleton cluster must have at least an  $\alpha_{\ell}$  fraction and at most a  $\beta_{\ell}$  fraction of color  $\ell$ . This guarantees proportional representational fairness of each color in each cluster.
As an intermediate step in achieving fairness, we will create splits in our hierarchy that achieve relative balance in terms of subcluster size. Thus, the following definition will come in handy.
Definition 4. In a hierarchy, a vertex  $v$  (corresponding to cluster  $C$ ) with  $c_v$  children is  $\epsilon$ -relatively balanced if for every cluster  $\{C_i\}_{i \in [c_v]}$  that corresponds to a child of  $v$ ,  $(\frac{1}{c_v} - \epsilon)|C| \leq |C_i| \leq (\frac{1}{c_v} + \epsilon)|C|$ .
While this definition is quite similar to that from Knittel et al. [2023], it deviates in two ways: 1) we only define it on a single split as opposed to the entire hierarchy and 2) we allow splits to be non-binary. If we apply it to the entire hierarchy and constrain it to be binary, it is equivalent to the former definition.
2.3 Tree Operators
Our work simplifies the work of Knittel et al. [2023]. In doing so, we follow the same framework, using tree operators to make well-defined and limited alterations to a given hierarchical clustering (Figure 2). In addition, our algorithm simplifies operator use in two ways: 1) we only utilize two of their four tree operators, and 2) we greatly simplified their most complicated operator and show that it can still be used to create a fair hierarchy.
First off, we utilize the same subtree deletion and insertion operator. The main difference is how we use it, which will be discussed in Section 3. At a high level, this operator removes a subtree from one part of the hierarchy and reinserts it elsewhere, adding and removing
Figure 2: Our operators: subtree deletion and insertion and shallow tree folding.
Definition 5 (Knittel et al. [2023]). Consider a binary tree  $T$  with internal nodes  $u$ , some non-ancecestor  $v$ ,  $u$ 's sibling  $s$ , and  $v$ 's parent  $g$ . Subtree deletion at  $u$  removes  $T[u]$  from  $T$  and contracts  $s$  into its parent. Subtree insertion of  $T[u]$  at  $v$  inserts a new parent  $p$  between  $v$  and  $g$  and adds  $u$  as a second child of  $p$ . The operator  $\operatorname{del\_ins}(u, v)$  deletes  $u$  and inserts  $T[u]$  at  $v$ .
The other operator we leverage is their tree folding operator, however we greatly simplify it. In the previous work, tree folding took two or more isomorphic trees and mapped the internal nodes to each other. Instead, we simply take two or more subtrees and merge their roots. The new root then directly splits into all children of the roots of all folded trees. In a way, this is an implementation of their folding operator but only at a single vertex in the tree topology. This is why we call it a shallow tree fold.
Definition 6. Consider a set of subtrees  $T_{1}, \ldots, T_{k}$  of  $T$  such that all root  $(T_{i})$  have the same parent  $p$  in  $T$ . A shallow tree folding of trees  $T_{1}, \ldots, T_{k}$  (shallow_fold  $(T_{1}, \ldots, T_{k})$ ) modifies  $T$  such that all  $T_{1}, \ldots, T_{k}$  are replaced by a single tree  $T_{f}$  whose root root  $(T_{f})$  is made a child of  $p$ , and whose root-children are  $\cup_{i \in [k]}$  children  $(\text{root}(T_{i}))$ .
In addition, we assume the subtree  $T_f$  is then arbitrarily binarized [Dasgupta, 2016] after folding. Since our algorithm works top-bottom, creating balanced vertices as it goes, we don't yet care about the fairness of the descendants of  $T_f$ . Moreover, we will then recursively call our algorithm on  $T_f$  to do precisely this.
3 Main Algorithm
In this section, we present our fair, low-cost, hierarchical clustering algorithm along with its analysis. Ultimately, we achieve the following (for a more intuitive explanation, see Section I):
Theorem 1. When  $T$  is a  $\gamma$ -approximate low-cost vanilla hierarchical clustering over  $\ell(V) = c_{\ell}n = O(n)$  vertices of each color  $\ell \in [\lambda]$ , MakeFair (Algorithm[2]), for any constants  $\epsilon, h, k$  with  $h \gg k^{\lambda}$  and  $n \gg h$ , runs in  $O(n\log n(h + \lambda\log n))$  time and yields a hierarchy  $T'$  satisfying:
1.  $T^{\prime}$  is an  $O\left(\frac{(h - 1)}{\epsilon} + \frac{1 + \epsilon}{1 - \epsilon} k^{\lambda}\right)\gamma$ -approximation for cost.
2.  $T^{\prime}$  is fair for any parameters for all  $i\in [\lambda ]\colon \alpha_i\leq \frac{\lambda_i}{n}\left(\frac{1 - \epsilon}{(1 + \epsilon)^2}\left(1 - \frac{k(1 + \epsilon)}{c_ih}\right)\right)^{O(\log (n))}$  and  $\beta_{i}\geq \frac{\lambda_{i}}{n}\left(\frac{1 + \epsilon}{(1 - \epsilon)^{2}}\left(1 + \frac{1 - \epsilon}{c_{i}k}\right)\right)^{O(\log (n))},$  where  $\lambda_{i} = c_{i}n.$
3. All internal nodes in  $T'$  are  $\epsilon$ -relatively balanced.
The main idea of our algorithm is to leverage similar tree operators to that of Knittel et al. [2023], but greatly simplify their usage and apply them in a more direct, careful manner. Specifically, the previous work processes the tree four times: once to achieve 1/6-relative balance everywhere, next to achieve  $\epsilon$ -relative balance, next to remove the bottom of the hierarchy, and finally to achieve fairness. The problem is that this causes proportional cost increases to grow in an exponential manner, particularly because the relative balance significantly degrades as you descend the hierarchy. Our solution is to instead do a single top to bottom pass of the tree, rebalancing and folding to achieve fairness as we go. We describe this in detail now.
First, we assume our input is some given hierarchical clustering tree. Ideally, this will be a good approximation for the vanilla problem, but our results do work as a black box on top of any hierarchical clustering algorithm. Second, we apply SplitRoot in order to balance the root (Section 3.1). And finally, we apply shallow tree folding on the children of the root to achieve fairness (Section 3.2). This gives us the first layer of our output, and then we recurse.
3.1 Root Splitting and Balancing
SplitRoot is depicted in Algorithm [1]. This fills the role of Knittel et al. [2023]’s Refine Rebalance Tree algorithm (and skips their Rebalance Tree algorithm), but it functions differently in that it only rebalances the root and it immediately splits the root into  $h$  children, according to our input parameter  $h$ .
We start SplitRoot by adding dummy children to  $v$  until it has  $h$  children (recall we can assume the input is binary). A dummy or null child is just a placeholder for a child to be constructed, or alternatively simply a zero-sized tree (note: this does not add any leaves to the tree). None of these children will be left empty in the end. Next, we define  $v_{max}$  and  $v_{min}$ , the maximal subtrees rooted at children  $(\mathrm{root}(T'))$  which have the most and fewest leaves, respectively.
As long as the root is not  $\epsilon$ -relatively balanced (which is equivalent to  $n_{T'}(v_{max})$  or  $n_{T'}(v_{min})$  deviating from the target  $n/h$  by over  $n\epsilon$ , as they are extreme points), we will attempt to rebalance. We define  $\delta_1$  and  $\delta_2$  to be the proportional deviation of  $n_{T'}(v_{min})$  and  $n_{T'}(v_{max})$  from the target size  $n/h$  respectively, and  $\delta$  to be the minimum of the two. In effect,  $\delta$  measures the maximum number of leaves we can move from the large subtree to the small subtree without causing  $n_{T'}(v_{max})$  to dip below  $n/h$  or  $n_{T'}(v_{min})$  to peak above  $n/h$ . This is important to guarantee our runtime: as an accounting scheme, we show that clusters monotonically approach size  $n/h$ , and thus we can quantify how fast our algorithm completes. We fully analyze this later, in Lemma 2.
Now we must attempt exactly this procedure: move a large subtree from  $v_{max}$  to  $v_{min}$ , though this subtree can have no more than  $\delta n$  leaves. To do this, we simply start at  $v_{max}$  and traverse down its right children (recall below  $v_{max}$ , the tree is still binary). We halt on the first child that is of size  $\delta n$  or smaller. We then remove it and find a place to reinsert it under  $v_{min}$ .
The insertion spot is found similarly by descending down  $v_{min}$ 's left children until the right child of the current vertex has fewer leaves in its subtree than the tree we are inserting. Thus, we have
completed our insertion and deletion operations. We repeat until the tree is relatively balanced as desired.
Algorithm 1 SplitRoot
Input: A binary hierarchy tree  $T$  of size  $n \geq 1/2\epsilon$  over a graph  $G = (V, E, w)$ , with smaller cluster always on the left, and parameters  $h \in [n]$  and  $\epsilon \in (0, \min(1/6, 1/h))$ .
Output: A hierarchical clustering  $T'$  with an  $\epsilon$ -relatively balanced root that has  $k$  children.
6: while  $n_{T'}(v_{\text{max}}) > n(1/h + \epsilon)$  or  $n_{T'}(v_{\text{min}}) < n(1/h - \epsilon)$  do
1: Initialize  $T' = T$
2:  $v = \operatorname{root}(T')$
3: Add null children to  $v$  until it has  $h$  children
4: Let  $v_{min} = \mathrm{argmin}_{v' \in \mathrm{children}(v)} n_{T'}(v')$
5: Let  $v_{max} = \operatorname{argmax}_{v' \in \text{children}(v)} n_{T'}(v')$
7:  $\delta_{1} = 1 / h - n_{T^{\prime}}(v_{min}) / n$
8:  $\delta_{2} = n_{T^{\prime}}(v_{max}) / n - 1 / h$
9:  $\delta = \min (\delta_1,\delta_2)$
10: Let  $v = v_{max}$
11:
12: while  $n_{T'}(v) > \delta n$  do
13:  $v\gets \mathrm{right}_{T^{\prime}}(v)$
14: end while
15:
16:  $u\gets v_{min}$
17: while  $n_{T'}(\mathrm{right}_{T'}(u)) \geq n_{T'}(v)$  do
18:  $u\gets \mathrm{left}_{T^{\prime}}(u)$
19: end while
20:  $T^{\prime}\gets T^{\prime}.\mathrm{del\_ins}(u,v)$
21: Reset  $v_{min}$  and  $v_{max}$
22: end while
We now analyze this part of the algorithm. The full proofs can be found in Appendix B and we proceed to give the intuition here. To start, consider the tree we are deleting and reinserting,  $T'[v]$ . Ideally, we want this to have many leaves, but no more than  $\delta n$ . We demonstrate that:
Lemma 1. For a subtree  $T'[v]$  that is deleted and reinserted in SplitRoot (Algorithm [7]), we must have that  $\epsilon n / (2(h - 1)) < n_T(v) \leq \delta n$ .