Title: BanditPAM++: Faster $k$-medoids Clustering
Venue: NeurIPS 2023
Accepted: yes
Keywords: multi-armed bandits

clustering

k-medoids

best-arm identification
Abstract: Clustering is a fundamental task in data science with wide-ranging applications. In $k$-medoids clustering, cluster centers must be actual datapoints and arbitrary distance metrics may be used; these features allow for greater interpretability of the cluster centers and the clustering of exotic objects in $k$-medoids clustering, respectively. $k$-medoids clustering has recently grown in popularity due to the discovery of more efficient $k$-medoids algorithms. In particular, recent research has proposed BanditPAM, a randomized $k$-medoids algorithm with state-of-the-art complexity and clustering accuracy. In this paper, we present BanditPAM++, which accelerates BanditPAM via two algorithmic improvements, and is $O(k)$ faster than BanditPAM in complexity and substantially faster than BanditPAM in wall-clock runtime. First, we demonstrate that BanditPAM has a special structure that allows the reuse of clustering information $\textit{within}$ each iteration. Second, we demonstrate that BanditPAM has additional structure that permits the reuse of information $\textit{across}$ different iterations. These observations inspire our proposed algorithm, BanditPAM++, which returns the same clustering solutions as BanditPAM but often several times faster. For example, on the CIFAR10 dataset, BanditPAM++ returns the same results as BanditPAM but runs over 10$\times$ faster. Finally, we provide a high-performance C++ implementation of BanditPAM++, callable from Python and R, that may be of interest to practitioners at https://github.com/motiwari/BanditPAM. Auxiliary code to reproduce all of our experiments via a one-line script is available at https://github.com/ThrunGroup/BanditPAM_plusplus_experiments.

BanditPAM++: Faster k -medoids Clustering

Abstract

Clustering is a fundamental task in data science with wide-ranging applications. In k -medoids clustering, cluster centers must be actual datapoints and arbitrary distance metrics may be used; these features allow for greater interpretability of the cluster centers and the clustering of exotic objects in k -medoids clustering, respectively. k -medoids clustering has recently grown in popularity due to the discovery of more efficient k -medoids algorithms. In particular, recent research has proposed BanditPAM, a randomized k -medoids algorithm with state-of-the-art complexity and clustering accuracy. In this paper, we present BanditPAM++, which accelerates BanditPAM via two algorithmic improvements, and is O ( k ) faster than BanditPAM in complexity and substantially faster than BanditPAM in wall-clock runtime. First, we demonstrate that BanditPAM has a special structure that allows the reuse of clustering information within each iteration. Second, we demonstrate that BanditPAM has additional structure that permits the reuse of information across different iterations. These observations inspire our proposed algorithm, BanditPAM++, which returns the same clustering solutions as BanditPAM but often several times faster. For example, on the CIFAR10 dataset, BanditPAM++ returns the same results as BanditPAM but runs over 10 × faster. Finally, we provide a high-performance C++ implementation of BanditPAM++, callable from Python and R, that may be of interest to practitioners at https://github.com/motiwari/BanditPAM . Auxiliary code to reproduce all of our experiments via a one-line script is available at https://github.com/ThrunGroup/BanditPAM_plusplus_experiments/ .

1 Introduction

Clustering is a critical and ubiquitous task in data science and machine learning. Clustering aims to separate a dataset X of n datapoints into k disjoint sets that form a partition of the original dataset. Intuitively, datapoints within a cluster are similar and datapoints in different clusters are dissimilar. Clustering problems and algorithms have found numerous applications in textual data [20], social network analysis [18], biology [28], and education [28].

Donghyun Lee*

Chris Piech

Martin Jinye Zhang ‡

A common objective function used in clustering is Equation 1:

Under this loss function, the goal becomes to minimize the distance, defined by the distance function d , from each datapoint to its nearest cluster center c among the set of cluster centers C . Note that this formulation is general and does not require the datapoints to be vectors or assume a specific functional form of the distance function d .

Specific choices of C , dataset X , and distance function d give rise to different clustering problems. Perhaps one of the most commonly used clustering methods is k -means clustering [17, 16]. In k -means clustering, each datapoint is a vector in R p and the distance function d is usually taken to be squared L 2 distance; there are no constraints on C other than that it is a subset of R p . Mathematically, the k -means objective function is

subject to |C| = k .

The most common algorithm for the k -means problem is Lloyd iteration [16], which has been improved by other algorithms such as k -means++ [2]. These algorithms are widely used in practice due to their simplicity and computational efficiency. Despite widespread use in practice, k -means clustering suffers from several limitations. Perhaps its most significant restriction is the choice of d as the squared L 2 distance. This choice of d is for computational efficiency - as the mean of many points can be efficiently computed under squared L 2 distance - but prevents clustering with other distance metrics that may be preferred in other contexts [22, 8, 5]. For example, k -means is difficult to use on textual data that necessitates string edit distance [20], social network analyses using graph metrics [18], or sparse datasets (such as those found in recommendation systems [15]) that lend themselves to other distance functions. While k -means algorithms have been adapted to specific other metrics, e.g., cosine distance [23], these methods are bespoke to the metric and not readily generalizable. Another limitation of k -means clustering is that the set of cluster centers C may often be uninterpretable, as each cluster center is (generally) a linear combination of datapoints. This limitation can be especially problematic when dealing with structured data, such as parse trees in context-free grammars, where the mean of trees is not necessarily well-defined, or images in computer vision, where the mean image can appear as random noise [28, 15].

In contrast with k -means clustering, k -medoids clustering [10, 11] requires the cluster centers to be actual datapoints, i.e., requires C ⊂ X . More formally, the objective is to find a set of medoids M⊂X (versus R p in k -means) that minimizes

subject to |M| = k . Note that there is no restriction on the distance function d .

k -medoids clustering has several advantages over k -means. Crucially, the requirement that each cluster center is a datapoint leads to greater interpretability of the cluster centers because each cluster center can be inspected. Furthermore, k -medoids supports arbitrary dissimilarity measures; the distance function d in Equation (3) need not be a proper metric, i.e., may be negative, asymmetric, or violate the triangle inequality. Because k -medoids supports arbitrary dissimilarity measures, it can also be used to cluster 'exotic' objects that are not vectors in R p , such as trees and strings [28], without embedding them in R p .

The k -medoids clustering problem in Equation (3) is a combinatorial optimization algorithm that is NP-hard in general [25]; as such, algorithms for k -medoids clustering are restricted to heuristic solutions. A popular early heuristic solution for the k -medoids problem was the Partitioning Around Medoids (PAM) algorithm [11]; however, PAM is quadratic in dataset size n in each iteration, which

is prohibitively expensive on large dataset. Improving the computational efficiency of these heuristic solutions is an active area of research. Recently, [28] proposed BanditPAM, the first subquadratic algorithm for the k -medoids problem that matched prior state-of-the-art solutions in clustering quality. In this work, we propose BanditPAM++, which improves the computational efficiency of BanditPAM while maintaining the same results. We anticipate these computational improvements will be important in the era of big data, when k -medoids clustering is used on huge datasets.

Contributions: We propose a new algorithm, BanditPAM++, that is significantly more computationally efficient than PAM and BanditPAM but returns the same clustering results with high probability. BanditPAM++ is O ( k ) faster than BanditPAM in complexity and substantially faster than BanditPAM in actual runtime wall-clock runtime. Consequently, BanditPAM++ is faster than prior state-of-the-art k -medoids algorithms while maintaining the same clustering quality. BanditPAM++ is based on two observations about the structure of BanditPAM and the k -medoids problem, described in Section 4. The first observation leads to a technique that we call Virtual Arms (VA) . The second observation leads to a technique that we refer to as Permutation-Invariant Caching (PIC) . We combine these techniques in BanditPAM++ and prove (in Section 5) and experimentally validate (in Section 6) that BanditPAM++ returns the same solution to the k -medoids problem as PAM and BanditPAM with high probability, but is more computationally efficient. In some instances, BanditPAM++ is over 10 × faster than BanditPAM. Additionally, we provide a highly optimized implementation of BanditPAM++ in C++ that is callable from Python and R and may be of interest to practitioners.

2 Related Work

As discussed in Section 1, global optimization of the k -medoids problem (Equation (3)) is NP-hard in general [25]. Recent work attempts to perform attempts global optimization and is able to achieve an optimality gap of 0.1% on one million datapoints, but is restricted to L 2 distance and takes several hours to run on commodity hardware [24].

Because of the difficulty of global optimization of the k -medoids problem, many heuristic algorithms have been developed for the k -medoids problem that scale polynomially with the dataset size and number of clusters. The complexity of these algorithms is measured by their sample complexity, i.e., the number of pairwise distance computations that are computed; these computations have been observed to dominate runtime costs and, as such, sample complexity translates to wall-clock time via an approximately constant factor [28] (this is also consistent with our experiments in Section 6 and Appendix 3).

Among the heuristic solutions for the k -medoids problem, the algorithm with the best clustering loss is Partitioning Around Medoids (PAM) [10, 11], which consists of two phases: the BUILD phase and the SWAP phase. However, the BUILD phase and each SWAP iteration of PAM perform O ( kn 2 ) distance computations, which can be impractical for large datasets or when distance computations are expensive. We provide greater details about the PAM algorithm in Section 3 because it is an important baseline against which we assess the clustering quality of new algorithms.

Though PAM achieves the best clustering loss among heuristic algorithms, the era of huge data has necessitated the development of faster k -medoids algorithms in recent years. These algorithms have typically been divided into two categories: those that agree with PAM and recover the same solution to the k -medoids problem but scale quadratically in n , and those that sacrifice clustering quality for runtime improvements. In the former category, [25] proposed a deterministic algorithm called FastPAM1, which maintains the same output as PAM but reduces the computational complexity of each SWAP iteration from O ( kn 2 ) to O ( n 2 ) . However, this algorithm still scales quadratically in n in every iteration, which is prohibitively expensive on large datasets.

Faster heuristic algorithms have been proposed but these usually sacrifice clustering quality; such algorithms include CLARA [11], CLARANS [21], and FastPAM [25]. While these algorithms scale subquadratically in n , they return substantially worse solutions than PAM [28]. Other algorithms with better sample complexity, such as optimizations for Euclidean space and those based on tabu search heuristics [6] also return worse solutions. Finally, [1] attempts to minimize the number of unique pairwise distances or adaptively estimate these distances or coordinate-wise distances in specific settings [14, 3], but all these approaches sacrifice clustering quality for runtime.

Recently, [28] proposed BanditPAM, a state-of-the-art k -medoids algorithm that arrives at the same solution as PAM with high probability in O ( kn log n ) time. BanditPAM borrows techniques from the multi-armed bandit literature to sample pairwise distance computations rather than compute all O ( n 2 ) . In this work, we show that BanditPAM can be made more efficient by reusing distance computations both within iterations and across iterations.

We note that the use of adaptive sampling techniques and multi-armed bandits to accelerate algorithms has also had recent successes in other work, e.g., to accelerate the training of Random Forests [27], solve the Maximum Inner Product Search problem [27], and more [26].

3 Preliminaries and Background

Notation: We consider a dataset X of size n (that may contain vectors in R p or other objects). Our goal is to find a solution to the k -medoids problem, Equation (3). We are also given a dissimilarity function d that measures the dissimilarity between two objects in X . Note that we do not assume a specific functional form of d . We use [ n ] to denote the set { 1 , . . . , n } , and a ∧ b (respectively, a ∨ b ) to denote the minimum (respectively, maximum) of a and b .

Partitioning Around Medoids (PAM): The original Partitioning Around Medoids (PAM) algorithm [10, 11] consists of two main phases: BUILD and SWAP. In the BUILD phase, PAM iteratively initializes each medoid in a greedy, one-by-one fashion: in each iteration, it selects the next medoid that would reduce the k -medoids clustering loss (Equation (3)) the most, given the prior choices of medoids. More precisely, given the current set of l medoids M l = { m 1 , · · · , m l } , the next point to add as a medoid is:

The output of the BUILD step is an initial set of the k medoids, around which a local search is performed by the SWAP phase. The SWAP phase involves iteratively examining all k ( n -k ) medoidnonmedoid pairs and performs the swap that would lower the total loss the most. More precisely, with M the current set of k medoids, PAM finds the best medoid-nonmedoid pair to swap:

PAMrequires O ( kn 2 ) distance computations for the k greedy searches in the BUILD step and O ( kn 2 ) distance computations for each SWAP iteration [28]. The quadratic complexity of PAM makes it prohibitively expensive to run on large datasets. Nonetheless, we describe the PAM algorithm because it has been observed to have the best clustering loss among heuristic solutions to the k -medoids problem. More recent algorithms, such as BanditPAM [28], achieve the same clustering loss but have a significantly improved complexity of O ( kn log n ) in each step. Our proposed algorithm, BanditPAM++, improves upon the computational complexity of BanditPAM by a factor of O ( k ) .

Sequential Multi-Armed Bandits: BanditPAM [28] improves the computational complexity of PAM by converting each step of PAM to a multi-armed bandit problem. A multi-armed bandit problem (MAB) is defined as a collection of random variables { R 1 , . . . , R n } , called actions or arms. We are commonly interested in the best-arm identification problem, which is to identify the arm with the highest mean, i.e., arg max i E [ R i ] , with a given probability of possible error δ . Many algorithms for this problem exist, each of which make distributional assumptions about the random variables { R 1 , . . . , R n } ; popular ones include the upper confidence bound (UCB) algorithm and successive elimination. For an overview of common algorithms, we refer the reader to [9].

We define a sequential multi-armed bandit problem to be an ordered sequence of multi-armed bandit problems Q = { B 1 , . . . , B T } where each individual multi-armed bandit problem B t = { R t 1 , . . . , R t n } has the same number of arms n , with respective, timestep-dependent means µ t 1 , . . . , µ t n . At each timestep t , our goal is to determine (and take) the best action a t = arg max i E [ R t i ] . Crucially, our choices of a t will affect the rewards at future timesteps, i.e., the R t ′ i for t ′ > t . Our definition of a sequential multi-armed bandit problem is similar to non-stationary multi-armed bandit problems, with the added restriction that the only non-stationarity in the problem comes from our previous actions.

We now make a few assumptions for tractability. We assume that each R t i is observed by sampling an element from a set S with S possible values. We refer to the values of S as the reference points , where each possible reference point is sampled with equal probability and determines the observed reward. With some abuse of notation, we write R t i ( x s ) for the reward observed from arm R t i when the latent variable from S is observed to be x s . We refer to a sequential multi-armed bandit as permutation-invariant , or as a SPIMAB (for Sequential, Permutation-Invariant Multi-Armed Bandit), if the following conditions hold:

For every arm i and timestep t , R t i = f ( D i , { a 0 , a 1 , . . . , a t -1 } ) for some known function f and some random variable D i with mean µ i := E [ D i ] = 1 S ∑ S s =1 D i ( x s ) ,

There exists a common set of reference points, S , shared amongst each D i ,

It is possible to sample each D i in O (1) time by drawing from the points in S without replacement, and

f is computable in O (1) time given its inputs.

Intuitively, the conditions above require that at each timestep, each random variable R t i is expressible as a known function of another random variable D i and the prior actions taken in the sequential multiarmed bandit problem. Crucially, D i does not depend on the timestep ; R t i is only permitted to depend on the timestep through the agent's previously taken actions { a 0 , a 1 , . . . , a t -1 } . The SPIMAB conditions imply that if E [ D i ] = µ i is known for each i , then µ t i := E [ R t i ] is also computable in O (1) time for each i and t , i.e., for each arm and timestep.

BanditPAM: BanditPAM [28] reduces the scaling with n of each step of the PAM algorithm by reformulating each step as a best-arm identification problem. In PAM, each of the k BUILD steps has complexity O ( n 2 ) and each SWAP iteration has complexity O ( kn 2 ) . In contrast, the complexity of BanditPAM is O ( n log n ) for each of the k BUILD steps and O ( kn log n ) for each SWAP iteration. Fundamentally, BanditPAM achieves this reduction in complexity by sampling distance computations instead of using all O ( n 2 ) pairwise distances in each iteration. We note that all k BUILD steps of BanditPAM (respectively, PAM) have the same complexity as each SWAP iteration of BanditPAM (respectively, PAM). Since the number of SWAP iterations is usually O ( k ) ([28]; see also our experiments in the Appendix 3), most of BanditPAM's runtime is spent in the SWAP iterations; this suggests improvements to BanditPAM should focus on expediting its SWAP phase.

4 BanditPAM++: Algorithmic Improvements to BanditPAM

In this section, we discuss two improvements to the BanditPAM algorithm. We first show how each SWAP iteration of BanditPAM can be improved via a technique we call Virtual Arms (VA). With this improvement, the modified algorithm can be cast as a SPIMAB. The conversion to a SPIMAB permits a second improvement via a technique we call the Permutation-Invariant Cache (PIC). Whereas the VA technique improves only the SWAP phase, the PIC technique improves both the BUILD and SWAP phases. The VA technique improves the complexity of each SWAP iteration by a factor of O ( k ) , whereas the PIC improves the wall-clock runtime of both the BUILD and SWAP phases.

4.1 Virtual Arms (VA)

As discussed in Section 3, most of the runtime of BanditPAM is spent in the SWAP iterations. When evaluating a medoid-nonmedoid pair ( m,x i ) to potentially swap, BanditPAM estimates the quantity:

for each medoid m and candidate nonmedoid x i , where

is the change in clustering loss (Equation (3)) induced on point x s for swapping medoid m with nonmedoid x i in the set of medoids M . Crucially, we will find that for a given x s , each ∆ l m,x i ( x s ) for m = 1 , . . . , k , except possibly one, is equal. We state this observation formally in Theorem 1.

Theorem 1. Let ∆ l m,x i ( x s ) be the change in clustering loss induced on point x s by swapping medoid m with nonmedoid x i , given in Equation (7) , with x s and x i fixed. Then the values ∆ l m,x i ( x s ) for m = 1 , . . . , k are equal, except possibly where m is the medoid for reference point x s .

Theorem 1 is proven in Appendix 2. Crucially, Theorem 1 tells us that when estimating ∆ L m,x i in Equation (6) for fixed x i and various values of m , we can reuse a significant number of the summands across different indices m (across k medoids). We note that Theorem 1 has been observed in alternate forms, e.g., as the 'FastPAM1' trick, in prior work [28]. However, to the best of our knowledge, we are the first to provide a formal statement and proof of Theorem 1 and demonstrate its use in an adaptive sampling scheme inspired by multi-armed bandits.

Motivated by Theorem 1, we present the SWAP step of our algorithm, BanditPAM++, in Algorithm 1. BanditPAM++ uses the VA technique to improve the complexity of each SWAP iteration by a factor of O ( k ) . We call the technique 'virtual arms' because it uses only a constant number of distance computations to update each of the k 'virtual' arms for each of the 'real' arms, where a 'real' arm corresponds to a datapoint.

4.2 Permutation-Invariant Caching (PIC):

The original BanditPAM algorithms considers each of the k ( n -k ) medoid-nonmedoid pairs as arms. With the VA technique described in Section 4.1, BanditPAM++ instead considers each of the n datapoints (including existing medoids) as arms in each SWAP iteration. Crucially, this implies that the BUILD phase and each SWAP iteration of BanditPAM++ consider the same set of arms. It is this observation, induced by the VA technique, that allows us to cast BanditPAM++ as a SPIMAB and implement a second improvement. We call this second improvement the Permutation-Invariant Cache (PIC).

We formalize the reduction of BanditPAM++ to a SPIMAB in Table 1. In the SPIMAB formulation of BanditPAM++, the set of reference points S is the same as the set of datapoints X . Each D i is a random variable representing the distance from point x i to one of the sampled reference points and can be sampled in O (1) time without replacement. Each µ i = E [ D i ] corresponds to the average distance from point x i to all the points in the dataset X . Each arm R t i corresponds to the point that we would add to the set of medoids (for t ≤ k ) or swap in to the set of medoids (for t > k ), of which there are n at each possible timestep. Similarly, the actions { a 0 , . . . , a p , . . . , a t } correspond to points added to the set of medoids (for t ≤ k ) or swaps performed (for t > k ). Equation (8) provides the functional forms of f for each of the BUILD and SWAP steps.

where M is a set of medoids. For the BUILD step, A is a sequence of t actions that results in a set of medoids of size t ≤ k , and, for the SWAP step, A is a set of actions that results in a set of medoids M of size k .

The observation that BanditPAM++ is a SPIMAB allows us to develop an intelligent cache design, which we call a permutation-invariant cache (PIC) . We may choose a permutation π of the reference points S = X and sample distance computations to these reference points in the order of the permutation. Since we only need to sample some of the reference points, and not all of them, we do not need to compute all O ( n 2 ) pairwise distances. Crucially, we can also reuse distance computations across different steps of BanditPAM++ to save on computational cost and runtime.

The full BanditPAM++ algorithm is given in Algorithm 1. Crucially, for each candidate point x i to swap into the set of medoids on Line 6, we only perform 3 distance computations (not k ) to update all k arms, each of which has a mean and confidence interval (CI), on Lines 13-15. This is permitted by Theorem 1 and the VA technique which says that k -1 'virtual' arms for a fixed i will get the same update. The PIC technique allows us to choose a permutation of reference points (the x s 's) and reuse those x s 's across the BUILD and SWAP steps; as such, many values of d ( x i , x s ) can be cached.

We emphasize that BanditPAM++ uses the same BUILD step as the original BanditPAM algorithm, but with the PIC. The PIC is also used during the SWAP step of BanditPAM++, as is the V A technique. We prove that the full BanditPAM++ algorithm returns the same results as BanditPAM and PAM in Section 5 and demonstrate the empirical benefits of both the PIC and V A techniques in Section 6.

5 Analysis of the Algorithm

In this section, we demonstrate that, with high probability, BanditPAM++ returns the same answer to the k -medoids clustering problem as PAM and BanditPAM while improving the SWAP complexity of BanditPAM by O ( k ) and substantially decreasing its runtime. Since the BUILD step of BanditPAM is the same as the BUILD step of BanditPAM++, it is sufficient to show that each SWAP step of BanditPAM++ returns the same swap as the corresponding step of BanditPAM (and PAM). All of the following theorems are proven in Appendix 2.

First, we demonstrate that PIC does not affect the results of BanditPAM++ in Theorem 2:

Theorem 2. Let X = { x 1 , . . . , x S } be the reference points of D i , and let π be a random permutation of { 1 , . . . , S } . Then for any c ≤ S , ∑ c q =1 D i ( x π ( p q ) ) has the same distribution as ∑ c q =1 D i ( x p q ) , where each p q is drawn uniformly without replacement from { 1 , . . . , S } .

Intuitively, Theorem 2 says that instead of randomly sampling new reference points at each iteration of BanditPAM++, we may choose a fixed permutation π in advance and sample in permutation order at each step of the algorithm. This allows us to reuse computation across different steps of the algorithm.

̸

We now show that BanditPAM++ returns the same result as BanditPAM (and PAM) in every SWAP iteration and has the same complexity in n as BanditPAM. First, we consider a single call to Algorithm 1. Let µ i := min j ∈ [ k ] µ i,j and let i ∗ := arg min i ∈ [ n ] µ i be the optimal point to swap in to the set of medoids, so that the medoid to swap out is j ∗ := arg min j ∈ [ k ] µ i ∗ ,j . For another candidate point i ∈ [ n ] with i = i ∗ , let ∆ i := µ i -µ i ∗ , and for i = i ∗ , let ∆ i := min (2) j µ i,j -min j µ i,j , where min (2) j denotes the second smallest value over the indices j . To state the following results, we will assume that, for a fixed candidate point i and a randomly sampled reference point x s , the random variable f ( D i ( x s ) , A ) is σ i -sub-Gaussian for some known parameter σ i (which, in practice, can be estimated from the data [28]):

Theorem 3. For δ = 1 /kn 3 , with probability at least 1 -2 n , Algorithm 1 returns the optimal swap to perform using a total of M distance computations, where

Intuitively, Theorem 3 states that with high probability, each SWAP iteration of BanditPAM++ returns the same result as BanditPAM and PAM. Since the BUILD step of BanditPAM++ is the same as the BUILD step of BanditPAM, this implies that BanditPAM++ follows the exact same optimization trajectories as BanditPAM and PAM over the entire course of the algorithm with high probability. We formalize this observation in Theorem 4:

Theorem 4. If BanditPAM++ is run on a dataset X with δ = 1 /kn 3 , then it returns the same set of k medoids as PAM with probability 1 -o (1) . Furthermore, the total number of distance computations M total required satisfies

Note on assumptions: For Theorem 3, we assumed that the data is generated in a way such that the observations f ( D i ( x s ) , A ) follow a sub-Gaussian distribution. Furthermore, for Theorem 4, we assume that the ∆ i 's are not all close to 0 , i.e., that we are not in the degenerate arm setting where many of the swaps are equally optimal, and assume that the σ i 's are bounded (we formalize these assumptions in Appendix 2). These assumptions have been found to hold in many real-world datasets [28]; see Section 7 and Appendices 1.1, and 2 for more formal discussions.

Additionally, we assume that both BanditPAM and BanditPAM++ place a hard constraint T on the maximum number of SWAP iterations that are allowed. While the limit on the maximum number of swap steps T may seem restrictive, it is not uncommon to place a maximum number of iterations on iterative algorithms. Furthermore, T has been observed empirically to be O ( k ) [25], consistent with our experiments in Section 6 and Appendix 3.

We note that statements similar to Theorems 3 and 4 can be proven for other values of δ . We provide additional experiments to understand the effects of the hyperparameters T and δ in Appendix 3.

Complexity in k : The original BanditPAM algorithm scales as O ( kc ( k ) n log kn ) , where c ( k ) is a problem-dependent function of k . Intuitively, c ( k ) governs the 'hardness' of the problem as a function of k ; as more medoids are added, the average distance from each point to its closest medoid will decrease and the arm gaps (the ∆ i 's) will decrease, increasing the sample complexity in Theorem 3. With the VA technique, BanditPAM++ removes the explicit factor of k : each SWAP iteration has complexity O ( c ( k ) n log kn ) . The implicit dependence on k may still enter through the term c ( k ) for a fixed dataset, which we observe in our experiments in Section 6.

6 Empirical Results

Setup: BanditPAM++ consists of two improvements upon the original BanditPAM algorithm: the VA and PIC techniques. We measure the gains of each technique by presenting an ablation study in which we compare the original BanditPAM algorithm (BP), BanditPAM with only the VA technique (BP+VA), BanditPAM with only the PIC (BP+PIC), and BanditPAM with both the VA and PIC techniques (BP++, the final BanditPAM++ algorithm).

First, we demonstrate that all algorithms achieve the same clustering solution and loss as BanditPAM across a variety of datasets and dataset sizes. In particular, this implies that BanditPAM++ matches

prior state-of-the-art in clustering quality. Next, we investigate the scaling of all four algorithms in both n and k across a variety of datasets and metrics. We present our results in both sample-complexity and wall-clock runtime. BanditPAM++ outperforms BanditPAM by up to 10 × . Furthermore, our results demonstrate that each of the VA and PIC techniques improves the runtime of the original BanditPAM algorithm.

For an experiment on a dataset of size n , we sampled n datapoints from the original dataset with replacement. In all experiments using the PIC technique, we allowed the algorithm to store up to 1 , 000 distance computations per point. For the wall-clock runtime and sample complexity metrics, we divide the result of each experiment by the number of swap iterations +1 , where the +1 accounts for the complexity of the BUILD step.

Datasets: We conduct experiments on several public, real-world datasets to evaluate BanditPAM++'s performance: the MNIST dataset, the CIFAR10 dataset, and the 20 Newsgroups dataset. The MNIST dataset [13] contains 70,000 black-and-white images of handwritten digits. The CIFAR10 dataset [12] comprises 60,000 images, where each image consists of 32 × 32 pixels and each pixel has 3 colors. The 20 Newsgroups dataset [19] consist of approximately 18,000 posts on 20 topics split in two subsets: train and test. We used a fixed subsample of 10,000 training posts and embedding them into 385 -dimensional vectors using a sentence transformer from HuggingFace [7]. We use the L 2 , L 1 , and cosine distances across the MNIST, CIFAR10, and 20 Newsgroups datasets, respectively.

6.1 Clustering/loss quality

First, we assess the solution quality all four algorithms across various datasets, metrics, and dataset sizes. Table 2 shows the relative losses of BanditPAM++ with respect to the loss of BanditPAM; the results for BP+PIC and BP+VA are identical and omitted for clarity. All four algorithms return identical solutions; this demonstrates that neither the VA nor the PIC technique affect solution quality. In particular, this implies that BanditPAM++ matches the prior state-of-the-art algorithms, BanditPAM and PAM, in clustering quality.

6.2 Scaling with k

Figure 1 compares the wall-clock runtime scaling with k of BP, BP+PIC, BP+VA, and BP++ on same datasets as Figure 1. Across all data subset sizes, metrics, and values of k , BP++ outperforms each of BP+VA and BP+PIC, both of which in turn outperform BP. As k increases, the performance gap between algorithms using the VA technique and the other algorithms increases. For example, on the CIFAR10 dataset with k = 15 , BanditPAM++ is over 10 × faster than BanditPAM. This provides empirical evidence for our claims in Section 5 that the VA technique improves the scaling of the BanditPAM algorithm with k .

We provide similar experiments that demonstrate the scaling with n of BanditPAM++ and each of the baseline algorithms in Appendix 3. The results are qualitiatively similar to those shown here; in particular, BanditPAM++ outperforms BP+PIC and BP+VA, which both outperform the original BanditPAM algorithm.

7 Conclusions and Limitations

We proposed BanditPAM++, an improvement upon BanditPAM that produces state-of-the-art results for the k -medoids problem. BanditPAM++ improves upon BanditPAM using the Virtual Arms

6

8

10

12

14

Number of medoids (

k

)

0.5

1.0

1.5

2.0

2.5

3.0

Time per step (s)

1e2

MNIST,

L

2,

n

=20000

BP++

BP+PIC

BP+VA

BP

(a)

6

8

10

12

14

Number of medoids (

k

)

0.0

0.2

0.4

0.6

0.8

1.0

Time per step (s)

1e3

CIFAR10,

L

1,

n

=20000

BP++

BP+PIC

BP+VA

BP

(b)

6

8

10

12

14

Number of medoids (

k

)

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Time per step (s)

1e2

20 Newsgroups, Cosine,

n

=10000

BP++

BP+PIC

BP+VA

BP

(c)

technique, which improves the complexity of each SWAP iteration by O ( k ) , and the PermutationInvariant Cache, which allows the reuse of computation across different phases of the algorithm. We prove that BanditPAM++ returns the same results as BanditPAM (and therefore PAM) with high probability. Furthermore, our experimental evidence demonstrates the superiority of BanditPAM++ over baseline algorithms; across a variety of datasets, BanditPAM++ is up to 10 × faster than prior state-of-the-art while returning the same results. While the assumptions of BanditPAM and BanditPAM++ are likely to hold in many practical scenarios, it is important to acknowledge that these assumptions can impose limitations on our approach. Specifically, when numerous arm gaps are narrow, BanditPAM++ might employ a naïve and fall back to brute force computation. Similarly, if the distributional assumptions on arm returns are violated, the complexity of BanditPAM++ may be no better than PAM. We discuss these settings in greater detail in Appendix 1.1.

References

Amin Aghaee, Mehrdad Ghadiri, and Mahdieh Soleymani Baghshah. Active distance-based clustering using k-medoids. Pacific-Asia Conference on Knowledge Discovery and Data Mining , 9651:253-264, 2016.

David Arthur and Sergei Vassilvitskii. K-means++ the advantages of careful seeding. In Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms , pages 1027-1035, 2007.

Vivek Bagaria, Tavor Z. Baharav, Govinda M. Kamath, and David Tse. Bandit-based monte carlo optimization for nearest neighbors. In Advances in Neural Information Processing Systems , pages 3650-3659, 2019.

Vivek Bagaria, Govinda M. Kamath, Vasilis Ntranos, Martin J. Zhang, and David Tse. Medoids in almost-linear time via multi-armed bandits. In International Conference on Artificial Intelligence and Statistics , pages 500-509, 2018.

Paul S. Bradley, Olvi L. Mangasarian, and W. N. Street. Clustering via concave minimization. In Advances in Neural Information Processing Systems , pages 368-374, 1997.

Vladimir Estivill-Castro and Michael E. Houle. Robust distance-based clustering with applications to spatial data mining. Algorithmica , 30(2):216-242, 2001.

HuggingFace. all-minilm-l6-v2 model card. https://huggingface.co/ sentence-transformers/all-MiniLM-L6-v2 . Accessed: 2023-10-28.

Anil K. Jain and Richard C. Dubes. Algorithms for clustering data . Prentice-Hall, 1988.

Kevin Jamieson and Robert Nowak. Best-arm identification algorithms for multi-armed bandits in the fixed confidence setting. In Annual Conference on Information Sciences and Systems , pages 1-6, 2014.

Leonard Kaufman and Peter J. Rousseeuw. Clustering by means of medoids. Statistical Data Analysis based on the L1 Norm and Related Methods , pages 405-416, 1987.

Leonard Kaufman and Peter J. Rousseeuw. Partitioning around medoids (program pam). Finding groups in data: an introduction to cluster analysis , pages 68-125, 1990.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Daniel LeJeune, Richard G. Baraniuk, and Reinhard Heckel. Adaptive estimation for approximate k-nearest-neighbor computations. In International Conference on Artificial Intelligence and Statistics , 2019.

Jure Leskovec, Anand Rajaraman, and Jeffrey D. Ullman. Mining of massive data sets . Cambridge university press, 2020.

Stuart Lloyd. Least squares quantization in pcm. IEEE transactions on information theory , 28(2):129-137, 1982.

James MacQueen. Some methods for classification and analysis of multivariate observations. In Berkeley Symposium on Mathematical Statistics and Probability , volume 1, pages 281-297, 1967.

Nina Mishra, Robert Schreiber, Isabelle Stanton, and Robert E. Tarjan. Clustering social networks. In International Workshop on Algorithms and Models for the Web-Graph , pages 56-67. Springer, 2007.

Tom Mitchell. Twenty Newsgroups. UCI Machine Learning Repository, 1999. DOI: https://doi.org/10.24432/C5C323.

Gonzalo Navarro. A guided tour to approximate string matching. ACM computing surveys , 33(1):31-88, 2001.

Raymond T. Ng and Jiawei Han. Clarans: A method for clustering objects for spatial data mining. IEEE transactions on knowledge and data engineering , 14(5):1003-1016, 2002.

Michael L Overton. A quadratically convergent method for minimizing a sum of euclidean norms. Mathematical Programming , 27(1):34-63, 1983.

Rameshwar Pratap, Anup Deshmukh, Pratheeksha Nair, and Tarun Dutt. A faster sampling algorithm for spherical k -means. In Asian Conference on Machine Learning , pages 343-358. PMLR, 2018.

Jiayang Ren, Kaixun Hua, and Yankai Cao. Global optimal k-medoids clustering of one million samples. Advances in Neural Information Processing Systems , 35:982-994, 2022.

Erich Schubert and Peter J Rousseeuw. Faster k-medoids clustering: improving the pam, clara, and clarans algorithms. In International Conference on Similarity Search and Applications , pages 171-187. Springer, 2019.

Mo Tiwari. Accelerating machine learning algorithms with adaptive sampling. arXiv preprint arXiv:2309.14221 , 2023.

Mo Tiwari, Ryan Kang, Jaeyong Lee, Chris Piech, Ilan Shomorony, Sebastian Thrun, and Martin J Zhang. Mabsplit: Faster forest training using multi-armed bandits. Advances in Neural Information Processing Systems , 35:1223-1237, 2022.

Mo Tiwari, Martin Jinye Zhang, James Mayclin, Sebastian Thrun, Chris Piech, and Ilan Shomorony. Banditpam: Almost linear time k-medoids clustering via multi-armed bandits. In Advances in Neural Information Processing Systems , 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a clustering problem (k-medoid) in which we have to select k centers among the datapoints in order to minimize the sum of distances of points to their closest center. For this problem, there is an intuitive algorithm which works well empirically (the PAM algorithm): (1) add greedily centers one by one by selected the next center to be the one which maximizes the decrease in cost (BUILD phase), (2) once we have selected k-centers, we try to swap one center for another datapoint again greedily (SWAP phase). We stop after some iterations (usually we stop after O(k) iterations). This algorithm works well in practice but is quite expensive in terms of computation. Therefore, several works have designed variants of the PAM algorithm to lower the computational cost without sacrificing too much of the quality of the solution. BanditPAM is one of these algorithm which returns the same solution as PAM (with high probability) but performs much less distance computations. \n\nThis paper further improve BanditPAM to propose BanditPAM++. They show that BanditPAM++ also returns the same solution as PAM with high probability, and uses less comparisons compared to BanditPAM. On some real-world datasets, BanditPAM++ can be up to 10x faster than BanditPAM.\n\nSTRENGTHS:\nI think this is an interesting and useful result. The experimental improvements seem substantial.\n\nWEAKNESSES:\nThe techniques used to obtain this algorithm did not seem very innovative (I am not an expert in bandit algorithms however).\n\ntypos:\nLine 118: fundamentally\nLine 161: missing Delta I think\nPseudocode page 6: I guess m_r^(2) the second closest medoid ? I am not sure this is defined in this page.\n\nQUESTIONS:\nIn the beginning of intro, I got confused by the remark that k-means does not generalize to other metrics. What do you mean by that? Isn’t equation (1) such a generalization?\nAbout the PAM algorithm in general, it seems that there is no worst-case guarantee for this method. Is there any practical algorithm that also gives some kind of worst-case guarantee in terms of clustering cost?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces BanditPAM++, a faster version of the BanditPAM algorithm for k-medoids clustering. It incorporates two improvements: the Virtual Arms technique, which reuses loss changes within each SWAP iteration, and the Permutation-Invariant Caching method, which samples reference points and reuses distance computations across steps. The authors provide theoretical guarantees that BanditPAM++ produces the same solution as BanditPAM and showcase its significant speedup on multiple datasets.\n\nSTRENGTHS:\nThe originality of the paper lies in the combination of techniques from the multi-armed bandit literature with k-medoids clustering, resulting in a novel algorithm that achieves significant speedup. Most part of the paper is well written and organized.\n\nWEAKNESSES:\n1. The presentation of empirical results is somewhat confusing. Here are a few specific concerns:\n\n1.1 The names of methods are not consistent between the text and figures. For example, \"BanditPAM\" in Line 234 is referred to as \"BanditPAM Original with caching\" in the figures. It would be helpful to ensure consistency in naming.\n\n1.2 It seems that the results in Figure 1 and Figure 3 are intended to reflect the same experimental setting, but there are inconsistencies. For instance, in the MNIST case with n=70000, the runtime of each method differs between the two plots. Additionally, in the CIFAR10 case, the runtime for k=10 in Figure 3 is significantly smaller than that in Figure 1. It would be beneficial for the author to review and revise the results or provide an explanation for these discrepancies.\n\n1.3 Since the specific speedup of BanditPAM++ can only be evaluated based on the empirical results, it would be clearer to include a table presenting the specific values of speedup. For example, adding a table of average speedup over all dataset sizes on different k. This approach would make it easier for readers to grasp the key points compared to the current figures, which require a non-trivial comparison between several plots.\n    \n2. In addition to heuristic methods, there is currently a growing trend in solving clustering problems, including K-Medoids, using global optimization approaches. It would be valuable for the author to discuss these methods in the related work section and provide a comparison between global and heuristic methods. This discussion could highlight the advantages and limitations of each approach, shedding light on the current state-of-the-art in clustering techniques.\n\nQUESTIONS:\nSee the weakness\n\nLIMITATIONS:\nThe exact speedup of this method cannot be theoretically derived; only empirical speedup can be obtained. Furthermore, the authors have highlighted cases in which the VA technique fails, as mentioned in section 6.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the $k$-mediods clustering problem. Given an instance $P$, the goal of the $k$-mediods problem is to find a subset $C \\subseteq P$ of size $k$ such that the sum of the defined distances from data points to $P$ to the centers in $C$ is minimized. In previous heuristic solutions for the $k$-mediods problem, the state-of-the-art $k$-mediods algorithm is the BanditPAM method based on the multi-armed bandit strategy which can return the same solution as PAM (the best heuristic algorithm using exhaustive searching) with high probability and time complexity $O(nlogn)$ and $O(knlogn)$ for each BUILD and SWAP iteration, respectively. In this paper, an improved algorithm is proposed (called BanditPAM++) with faster running time and can return the same solution to BanditPAM with high probability. By designing Virtual Arms (VA) technique and using a permutation-invariant cache (PIC) structure, this paper shows that the total number of distance computations in expectation can be reduced to $O(nlogn)$, which is much faster than BanditPAM method. Experimental results show that the proposed BanditPAM++ method runs 10 times faster than BanditPAM method with the same clustering results on some of the commonly used datasets for clustering tasks.\n\nSTRENGTHS:\n1. This paper proposes a new heuristic method for the $k$-mediods clustering problem. The authors show that  the proposed method runs much faster than the current SOTA (BanditPAM method) and can give the same solution as the current SOTA with high probability. \n\n2. This paper proposed two novel techniques and structures, namely the Virtual Arms (VA) technique and the permutation-invariant cache (PIC), which is proved to be efficient for accelerating the BanditPAM method in both theory and practice.\n\n3. The experimental results show that the proposed BanditPAM++ method has significant advantages on the running time compared with BanditPAM method, while the clustering quality of BanditPAM++ matches that of BanditPAM\n\nWEAKNESSES:\n1. The theoretical proofs of this paper is mainly based on the assumptions that (1): during the SWAP process, the samples are drawn from a sub-Gaussian distribution. (2) the maximum number of swap steps is limited. However, there is no detailed discussion about the assumptions made (especially the sub-Gaussian distribution assumption). Although the authors mentioned in the conclusions and limitations part that \"while the assumptions of BanditPAM and BanditPAM++ are likely to hold in many practical scenarios, it is important to acknowledge that these assumptions can impose limitations on our approach\", there is no clear explanation about the assumptions made and it is unclear why the Gaussian distribution is likely to hold in many practical scenarios. Besides, in a clustering algorithm, the iteration rounds needed to reach the convergence is not usually a constant $T$. For example, in local search algorithms for clustering, the iteration rounds could be as large as $O(klog\\Delta)$ if the initialization is bad, where $\\Delta$ is the aspect ratio of the given instance. Thus, it is unlcear how the number of swap steps could affect the results of BanditPAM++\n\n2. For the proposed BanditPAM++ method, this paper claims that it runs much faster than BanditPAM. However, there is no direct comparison between each phase (i.e., the BUILD and SWAP iteration). It is unclear how the time complexity is improved compared with BanditPAM method where each BUILD iteration requires $O(nlogn)$ running time and each SWAP iteration requires $O(nklogn)$ running time.\n\nQUESTIONS:\n1. Can the authors provide detailed comparison between the running time of BanditPAM++ and BandiPAM for both BUILD iteration and SWAP iteration.\n\n2. Does the maximum iteration $T$ required for BanditPAM and BanditPAM++ to reach a good convergence the same?\n\n3. Why the authors claim that the assumptions of BanditPAM and BanditPAM++ are likely to hold in many practical scenarios?\n\n4. If the sub-gaussian distribution assumptions do not hold, what will the guarantee be like (maybe some worse guarantee)?\n\nLIMITATIONS:\nI don't think this paper has potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors developed an algorithm (called BanditPAM++) which is claimed to be more efficient than banditPAM, the state-of-the-art algorithm for k-medioid (which is the version of the k-means problem where the centers must be input points). They prove that with high probability the two algorithms produce the same results, with BanditPAM++ being more efficient. An experimental evaluation on two datasets complements their results.\n\nSTRENGTHS:\n- the work focuses on improving the scalability of an effective algorithm for clustering\n- theoretical results support their claims, which are validated empirically on two datasets\n\nWEAKNESSES:\n- it is not clear how some of the parameters (in particular T=max number of swaps and delta) affect the results.\n- only two datasets are included in the experimental evaluation. I recommend a more extensive evaluation, given that it is not clear what is the advantage of their approach from a theoretical point of view\n- the analysis focuses on the case when the input points follow a subgaussian distribution, while for example k-means++ guarantees hold for any distribution\n- the presentation of the paper could be improved (e.g. the role of delta is not discussed, not immediately clear which curve in the experiments correspond to BanditPAM++, some claims are inaccurate, e.g. there is a version of k-means that can deal with cosine similarity)\n\nQUESTIONS:\nQuestions:\n- from a theoretical point of view, what is the advantage of BanditPAM++ over BanditPAM? (both have running time O(n logn) as far I could check). Update after rebuttal: I had missed that BanditPAM++ is faster than BanditPAM by a factor of k. This should be stressed in the text. Perhaps a short paragraph right after Theorem 4 might help. \n- discuss the role of delta and the max number of swaps T and how they might affect the results\n\nother comments:\n- Figure 4 does not offer many insights, all approaches are superimposed and we just see a straight line.\n- there is a version of k-means with cosine similarity, in contrast with what is claimed in the paper (references [4,6,18] are old and not up to date)\n\nA Faster Sampling Algorithm for Spherical k-means, Journal of machine learning 2018. \n\n- I recommend to evaluate the impact of delta and T in the experiments, for future work\n- in the experiments it should be immediately clear which curves correspond to BanditPAM++ (all of the variants? some?)\n\nLIMITATIONS:\nNo potential negative societal impact, as far as I could check.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \n * presents a new algorithm for the k-medoids problem called BandidPAM++, which is an extension of BanditPAM\n * analyzes the algorithm theoretically showing that it produces essentially the same output as BanditPAM and uses O(n log n) distance computations\n * evaluates the algorithm empirically comparing it to BanditPAM\n\nSTRENGTHS:\n* The paper new algorithm outperforms the baseline in experiments, delivering significantly better running times\n* While I'm not absolutely certain, it appears to me that some of the ideas involved are nontrivial\n\nWEAKNESSES:\nThe main issue with the paper is the presentation. In the current form the description of the proposed algorithm consists of a short, high-level description + a nontrivial pseudocode. This is especially disappointing given that it would be easy to fit in more text in the first 9 pages (there are many subsection headers, figures could be made smaller, Algorithm 1 could potentially go to the appendix, etc). This, combined with the fact that I'm not familiar with the area of bandit algorithms, makes it difficult for me to make a confident decision.\n\nMy second (smaller) concern is about the empirical evaluation: while the motivation for using k-medoids is clustering points in non-Euclidean spaces, the experiments use two datasets, both of which define Euclidean space, which is non-ideal (but I guess acceptable for a paper whose main goal is to demonstrate speedups)\n\n[Edit: thank you for addressing this point in the rebuttal]\n\nQUESTIONS:\n1. Is it true that BanditPAM and BanditPAM++ come with the same running time bound on the BUILD iterations?\n1. Is the running time of each SWAP iteration O(k n log n) for BanditPAM vs O(n log n) for BanditPAM++? (Theorem 4 bounds the number of comparisons which is not the same as the running time)\n1. Do you use any results from the multi-arm bandit literature in any of the proofs?\n1. Since Theorem 1 is a reformulation of a known fact, what is the novelty in the virtual arms optimization?\n1. How similar is Theorem 3 to what was previously done for BanditPAM?\n1. What is the high-level idea behind Theorem 3?\n1. Could you also elaborate on the main idea behind permutation-invariant cache? I find the current description (lines 196-201) to be too compact to be understandable.\n1. Line 131 suggests that each $D_j$ is a random variable which represents a *uniform* distribution over $\\mathcal{S}$. Could you formally define $D_j$?\n1. Does Table 1 describe BanditPAM or BanditPAM++? The caption vs line 183 seem to contradict each other.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}