Clustering and Ordering Variable-Sized Sets: The Catalog Problem
Authors: Mateusz Maria Jurewicz
Graham W. Taylor
Leon Derczynski
Venue: Submitted to ICLR 2023 (2023)
Keywords: neural clustering
set-to-sequence
supervised clustering
structure prediction
set representation
learning to order

Abstract: Prediction of a varying number of ordered clusters from sets of any cardinality is a challenging task for neural networks, combining elements of set representation, clustering and learning to order. This task arises in many diverse areas, ranging from medical triage, through multi-channel signal analysis for petroleum exploration to product catalog structure prediction. This paper focuses on the latter, which exemplifies a number of challenges inherent to adaptive ordered clustering, referred to further as the eponymous Catalog Problem. These include learning variable cluster constraints, exhibiting relational reasoning and managing combinatorial complexity. Despite progress in both neural clustering and set-to-sequence methods, no joint, fully differentiable model exists to-date. We develop such a modular architecture, referred to further as Neural Ordered Clusters (NOC), enhance it with a specific mechanism for learning cluster-level cardinality constraints, and provide a robust comparison of its performance in relation to alternative models. We test our method on three datasets, including synthetic catalog structures and PROCAT, a dataset of real-world catalogs consisting of over 1.5M products, achieving state-of-the-art results on a new, more challenging formulation of the underlying problem, which has not been addressed before. Additionally, we examine the network's ability to learn higher-order interactions and investigate its capacity to learn both compositional and structural rulesets.

Under review as a conference paper at ICLR 2023
CLUSTERING AND ORDERING VARIABLE-SIZED SETS: THE CATALOG PROBLEM
Anonymous authors Paper under double-blind review
ABSTRACT
Prediction of a varying number of ordered clusters from sets of any cardinality is a challenging task for neural networks, combining elements of set representa- tion, clustering and learning to order. This task arises in many diverse areas, ranging from medical triage, through multi-channel signal analysis for petroleum exploration to product catalog structure prediction. This paper focuses on the latter, which exemplifies a number of challenges inherent to adaptive ordered clustering, referred to further as the eponymous Catalog Problem. These include learning variable cluster constraints, exhibiting relational reasoning and managing combinatorial complexity. Despite progress both neural clustering and set-to- sequence methods, no joint, fully differentiable model exists to-date. We develop such a modular architecture, referred to further as Neural Ordered Clusters (NOC), enhance it with specific mechanism for learning cluster-level cardinality constraints, and provide a robust comparison of its performance relation alternative models. We test our method on three datasets, including synthetic catalog structures and PROCAT, a dataset of real-world catalogs consisting of over 1.5 M products, achieving ofate-of-the-arr results formulation of the underlying problem, which has not been addressed before. Additionally, we examine the network's s ability to learn higher-order interactions and investigate its capacity to learn both compositional and structural rulesets.
new, more challenging
INTRODUCTION
The ability to group members of set and order these groups is key to many important real-world caaision-making processes. It finds applications ranging from supply chain management (Wenzel al., 2019) to prioritization in medical triage (Miles et al., 2020). Other application domains include petroleum exploration (Rabiller et al., 2010), business process analytics (Iee et al., 2014), and also product catalog structuring (Jurewicz & Derczynski, 2022), where the goal is to take set of products and work out how to group them together and order these groups to form coherent product catalog. We term this problem of simultancously grouping and ordering
of items the Catalog Problem.
This paper defines the Catalog Problem and presents an investigation into neural network approaches it. To this end we introduce addresses the Catalog Problem. In it, sets of items are clustered into groups, and an ordering between groups is established. All of this achieved in a supervised manner. While clustering methods are often unsupervised (Aljalbout et al., 2018; Ronen et al., 2022), the meaningful ordering of clusters often requires more knowledge than is available from the instance representation alone.
dillyydifferentiabbl, deep learning (DL) model architecture that
Similarly, learning to order is often framed et al., 2020; Shi, 2022). Referred to further as set-to-sequence (S2S), this area and its corresponding methods inspire the cluster-ordering aspect of our proposed Neural Ordered Clusters (NOC) model. Both neural clustering and set-to-sequence models have limitations. Element-wise neural clustering methods require Onn) passes over the input set of cardinality n.' Cluster-wise and attention based models are more computationally efficient, but exhibit a limited ability to learn cluster cardinality constraints (Pakman al., 2020), integral to both the prototypical Catalog Problem and its practical
supervised learning task (Vinyals al., 2015; Yin
(O(n) can be prohibitive with large input sets (n >= 1000), which is often the case in many interesting set-input problems such as 3D point cloud tasks (Qi al., 2017; Ge al., 2018; Zhao al., 2021).

---

Under review as a conference paper at ICLR 2023
X
C
Figure 1: The Catalog Problem. From left right: set of imput elements (X); clustering of those elements (C); and target ordering over those clustered elements (y). left to right. The model has to perform all these tasks using information about inter-element relations and intra-cluster relations in order to characterise cluster, and inter-cluster relations to generate the final, ordered clustering.
instantiations. Set-to-sequence methods, on the other hand, are effective learning constraints (Zhu et al., 2021) and generalizing unseen distributions (Wen, 2022). However, they are limited by their inability to predict an adaptive number of clusters without major adjustments (Fernández- González, 2022), two of which proposed in Section 5.1. Nonetheless, these S2S variants suffer from noisy in-cluster order and cascading first-choice costs (Gan et al., 2020; Vial et al., 2022).
To address these challenges, implement unified clustering and cluster ordering method. NOC is capable of predicting ordered, partitional cluster assignments for elements of sets of varying car- dinality. It infers flexible, input-dependent number of diverse clusters, maintains O(k) complexity and utilizes jointly learned representation of set elements to find the target cluster order. Unlike ex- isting neural clustering methods, it exhibits the ability to learn cluster cardinality constraints through supervision. To our knowledge, no other neural-bare based method exists to address such challenges in an end-to-end, jointly trainable way. instead performing clustering and ordering as two separate tasks, sometimes with the separate addition of representation learning step (Aljalbout et al., 2018). 8). code, hyper-parametent and datasets required for reproducing our results are made available and detailed via the appendix. To summarize, our contributions are as follows:
Firstly, introduce the Catalog Problem, novel joint clustering and cluster ordering problem over sets of elements, which is a challenging variant the set-to-sequence domain with multiple aspects that are not handled by existing neural methods. We exemplify and tackle this problem on three datasets, including a real-world dataset of over 1.5 M products grouped and ordered into product catalogs by human experts.
Secondly, we propose novel, fully differentiable, joint neural clustering and cluster or- dering model, Neural Ordered Clusters (NOC), capable of predicting an adaptive, input- dependent number of ordered, partitional clusters from sets of varying cardinality.
Thirdly, we provide robust comparison existing and proposed neural methods on the Catalog Problem using synthetic & real-world datasets, providing insights into the models' capacity to learn higher-order relational rules of cluster composition and ordered structure.
THE CATALOG PROBLEM
Many problems require predicting an adaptive, input-dependent number of partitional clusters from sets of varying cardinality and consequently ordering these clusters according to target preference. We refer to this as the Catalog Problem. In the Catalog Problem, the input is an unordered set of unique elements. The output is clustering of these elements, with suitable cluster cardinalities, and an ordering over the clusters (Figure 1). The input may be of any cardinality. Candidate approaches to the problem have determine how many clusters create, choose which items to assign which clusters and also order the clusters. This is a general problem that, as is shown by experiments later in this paper, is non-trivial.
2

---

Under review as a conference paper at ICLR 2023
Although the Catalog Problem is named because it models the task of creating catalog of items, e.g. products, no specific application is prescribed; the problem only defines imput and output types and a relation between these two. The difficulty lies in learning the relationships between both input elements and groups thereof. This difficulty can be compounded by the uniqueness of input elements, making learning representations difficult, due the scarcity of distributional information.
2.1 RELATED WORK
There have been many machine learning (MI.) approaches to clustering with some notion of order, albeit often aimed at preventing the impact of this order on the clusters (Fisher al., 1992). In the more common, unsupervised setting these range from hierarchical clustering (Johnson, 1967: Chu, 1974), through ordinal clustering (Janowitz, 1978) and incremental conceptual clustering (Fisher, 1987) to Markov clustering (Van Dongen, 2000) and other, more recent methods (Ankerst et al., 1999; Turowski et al., 2020). Certain unsupervised clustering methods without the ordering element, like affinity propagation (Frey & Dueck, 2007: Vlasblom & Wodak, 2009), are also capable of outputting an adaptive, input-dependent number of partitional clusters.
Closer to the supervised setting of interest, there have been attempts to leverage instance labels to augment k-means (Ergun et al., 2022), improve the interpretability thereof (Peng et al., 2022) and cluster labelled data to facilitate permutation learning (Lee & Kim, 2020). Similarly, contrastive clustering utilizes soft labels to maximize the similarities of positive pairs while minimizing those of negative ones (Li al., 2021), in an approach reminiscent the pairwise order prediction modules that resulted in increased performance on strictly set-to-sequence tasks (Yin et al., 2020). However, these supervised clustering methods do not yield an ordering of clusters.
BACKGROUND
We identify three classes of neural approaches to solving aspects of the Catalog Problem: set rep- resentation; neural clustering; and ordering through pointer attention. Firstly, learning permutation invariant set representations that can encode higher-order interactions is vital, due to the complex relational factors among set elements that determine the target output. Deep learning advances in set representation focus primarily on being able to effectively learn such relations, starting with Deep Sets (Zaheer et al., 2017), through the Set Transformer (Lee et al., 2019) modifications thereof (Girgis et al., 2021; Jurewicz & Derczynski, 2022). These methods can be used for both encoding elements and representing clusters. In the Set Transformer, given an unordered set (X), obtain the representations of set elements (Ez) and subsequently the entire input set (8) via:
E MAB(X, LN(H + 6 (H)), where H = =N(XX + MHA(X, Xx, X.)),
(1)
PMA(EE) MAB(r,E, Ex).
Here, multihead, intra-set attention (denoted MHA) is performed by casting the input set X to query, key, and value matrices X,.XA. X,,X, X, according to an arbitrary permutation π, and adding residual connection defined by Vaswani et al. (2017), without positional encoding. This operation incorporated into multihead self-attention block (MAB) by the inclusion a row-wise feed-forward neural network (NN) Φ. with layer normalization (LN) after each block (Ba et al., 2016), resulting in permutation equivariant matrix of per-element representations (E,). These then aggregated into permutation invariant representation of the entire set (s) by performing pooling by multihead attention (PMA) between the per-element representations and a learned seed vector These operations used extensively in method for encoding both the initial input set and the predicted clusters.
Secondly, supervised neural clustering obtains per-element cluster assignments (ê:) through num- ber of modular functions parameterized by NNs. These networks leverage set representation meth- ods to encode the set of currently available, unassigned elements (U,), each previously completed cluster (g,) and consequently all clustered elements jointly (G,), at each step j. This is paired
Foor formal proof, Section 3.1 and supplementary material of Lee et al. (2019).
3

---

Under review as a conference paper at ICLR 2023
with algorithm for selecting the next j-th cluster (if clusterwise) or element (if pointwise) to be considered until nothing remains to be assigned.
In the O(k) clusterwise formulation each cluster assignment is the output of another NN (p), in the form of a binary choice (c+) per encoded element (xz), conditioned on these representations and trained in a teacher-forced manner, with loss calculated only for the elements belonging to the current cluster. In the attention-based, clusterwise framework of the Attentive Clustering Pro- cess (Pakman al., 2020) a random anchor element (xxa) is selected at each step j. along with latent variable (zj) sampled from a Normal distribution via learned mean and standard deviation, on which the final, per-element predictions are conditioned for the current j-th cluster:
p(z; x3) Nzz xa U,. Gj),
(3)
Po, = x,) = sigmoid (p(x;, Ra. 2j. Uj G2)).
(4)
Thirdly, pointer attention can be used to select a single element from a set of any cardinality n, common in set-to-sequence NNs. At each step E (I,..., k} it outputs an attention vector (am) over all obtained clusters C. As the clusters are selected sequentially. this represents their predicted order, with highest attention value pointing to the index of the m-th cluster in that order:
am softmax(o' tanh(Wa W2h
(5)
where W1 and W2 are model parameters, tanh is the hyperbolic tangent nonlinearity, and h customarily the hidden state of the pointer network at current selection step m. The first hidden state hg can be initialized from the permutation invariant set representation 8. In our context, this in principle enables us to sequentially select predicted clusters according to their learned target order.
4 THE NEURAL ORDERED CLUSTERS MODEL
Existing methods do not, to the best of our knowledge, directly address the catalog problem of joint clustering and cluster ordering. To this end, we investigate set of novel and adapted methods to apply to this problem. This section introduces the proposed Neural Ordered Clusters (NOC) model. NOC consists of three modular parts, each with a corresponding loss factor. These components take the form of partitional neural clustering, per-cluster cardinality prediction, and cluster ordering via pointer attention. The learned representations of elements and the set in its entirety are transformed by each of these modules and continuously adjusted during training in a fully differentiable way. For an overview of the NOC architecture, we refer the reader to Figure
The first step is to obtain a partitional clustering (NOC1). We propose to achieve this through an adjusted neural clustering module. building on the process described in equations and 4. First, we utilize the Set Interdependence Transformer, or SIT (Jurewicz & Derczynski, 2022), to obtain both the representations of the individual elements (E+) and the permutation-invan invariant representation of the entire set (s). SIT consists of stack of MAB layers described in equations 1 and 2, except that the second layer's input takes the form of an augmented matrix, which the vector representation of the set is concatenated to E,, as if it was additional set element e,, This is intended to enable learning of higher-orde interactions in fewer layers. At each cluster prediction step j the representa- tions of unassigned elements (U,) and each previously completed cluster (g1:5) are adjusted through a stack of SIT transformations and used to make cluster assignments é per unassigned element i:
(e, = po.: = x2) = (ei egg, zj. SIT(U,),, SIT(
(6)
The second step (NOC2) to adjust the cluster assignments via the predicted cardinality t, of the j-th cluster. At each step a function, parameterized by a fully-connected neural network Φ2. is used to predict the cardinality of the current cluster as a regression task. The obtained cardinality, conditioned the available elements and previously predicted clusters, is used as a threshold for the maximum number of elements to assign to current cluster. If the number of elements assigned by NOC, exceeds this threshold, the elements with lower values of é excluded from cluster Cj-
4

---

Under review as a conference paper at ICLR 2023
CLUSTERING STEP
CARDINALITY PREDICTION
CLUSTER ORDERING
UNASSOND EMINIS
PRENIAUS CLUSTERS
POOLING BY MULLEBBAAAAEENIIO
(MM
AVTUNIRIN
U
V
V
M.
RANDOR ANTHOR
i $ 1051
M,
(U),
GG))
6g
6
62
C4
in
C
ESRANCED POINTR AUTENTION
MOBs q.)
(IT,(C))
9
B
8.
LOCAL COHERENCE
Gj+1
CENITRNGSTT #+1
M
CJ+1
GLOBALORRIETAARE ORIENTATION
GLORAL
0/11
MILP
SIT,
MIPP
61
02
CLUSTER ASSIGIMENTS
C= (C...... C,+1}
CLUSTER ORDER
Figure 2: NOC. Starting the top of the leftmost panel, at clustering step j the representations of unassigned elements (U,), previously created clusters (Gy) and a random anchor element (ef) used to make initial cluster assignments (c1-6). In the middle panel the current cardinality (tz)) is predicted and used to adjust the juh cluster, which is then transformed via SIT,(C,) into its embedded representation 9j. which becomes part of the Gj+1 matrix and used during the remaining clustering steps. In the rightmost panel, after A iterations of the NOC; and NOC, steps, the predicted clusters are ordered via NOCs's Enhanced Pointer attention (A.2).
t, NOC, (
C - c
C Ci
iF otherwise
(8)
Steps and two repeated until have obtained & partitional clusters (Ci.... Sih....o C.) with indi- vidual cardinalities. Set-to-sequence methods expect fixed-length vector representations, therefore SIT and PMA to obtain them (C, (cı,-... Cs. where Cj PMA (STT((()))) In the third and final stage of NOC3 an Enhanced Pointer Network (Yin et al., 2020) is used to output an attention vector am each step E {l...., .,. k). The highest attention value points the cluster to be placed at m-th position in the output sequence of ordered clusters:
am softmax(v Lanh(W,nn Mm W2h
h LSTM( LSTM( Cmmm))
This largely resembles the process outlined in Equation 5, with the exception matrix Mm, spe- cific to the Enhanced Pointer Network, explained in more detail in appendix A.2. Together, these three elements of NOC allow for the prediction an adaptive number & of partitional clusters with varying. learned cardinalities. This learning is achieved through weighted sum of the loss factors from each of the three stages NOC, with reacher-forcing (Williams & Zipser, 1989). The full algorithm is provided in Appendix A.
5

---

Under review as a conference paper at ICLR 2023
Table Clustering and permutation results on all three datasets
2D Gaussians
Procedural Catalogs
PROCAT
Method
V-Measure
Kendall's
V-Measure
Kendall's
V-Measure
Kendall's T
NCP + S2S
91.52 3.30
75.31 t 4.5
63.12 + 4.12
74.82 + 5.1
25,42 + 5.14
21.94 4.3 25.24 4.0
CCP + $2S
93.94 + 2.13
83.88 = 4.2
79.41 + 3.76
81,10
37,41 + 3.10
ACP + S2S
96.63 + 1.82
90.13 + 3.7
87.66 + 3.91
85.73 3.2
41.38 # 3.88
31.73 3.1
S2S-B
89.37 + 4.21
95.89 + 2.3
78.39 + 1.64
92.13 + 2.0
39.01 + 3.35
44.39 3.7
S2S-C
92.45 + 2.01
93.41 = 2.1
75.83 + 4.91
91.55 # 3.3
36.71 = 4.26
40.22 + 4.2
NOC
97.81 + 0.92
98,40 + 0.5
96.13 + 1.28
95.84 = 0.9
52.84 + 3.15
56.67 2.8
5
EXPERIMENTS
The Catalog Problem presents an interesting type signature, where while the input-as as in S2S - is an unordered set, the output is more complex structure that is the result of clustering and ordering. In this section we examine multiple approaches to the Catalog Problem, including baseline methods adapted to this output structure as well the NOC model, evaluating over both synthetic and real- world datasets. All datasets, hyperparameters and code are freely available and described in detail in Appendix A.4. The provided code includes all data pre- pre-processing and generation steps.
The models' exact layer dimensions are given in Appendix A.4, with the number of learnable pa- rameters of each model varying by less than 5% per task. The Adam (Loshchilov & Hutter, 2017) optimizer was used with weight decay coefficient le-3, learning rate (ax) le-4, dropout rate of 0.05 and batch size 64, for 50-100 epochs. Experiments were performed on cloud-based GPU instances, with NVIDIA Quadro P6000 graphics cards (24 GB) and 8 CPU cores. To represent natural lan- guage entities in Section 5.4 we use the concatenated and averaged output of the last 4 layers of the cased, large version of BERT (Devlin al., 2019), frozen during training isolate the effect of compared clustering and permutation methods on the final performance.
The best performance is reported in bold and second best is underlined. Reported results are aver- aged over three full training runs, standard deviation is reported after the + sign. We use V-Measure (Rosenberg & Hirschberg, 2007) and Kendall's Rank Correlation Coefficient (r) the primary clus- tering and permutation metrics respectively. scaled by factor of a hundred for readability, following convention (Wang& & Wan, 2019; Pandey & Chowdary, 2020).
5.1 BASELINES
We present two groups of baselines for addressing the Catalog Problem. i) Neural clustering meth- ods with an added set-to-sequence module: the module takes the predicted clusters and outputs their order via attention-based pointing. These methods include the pointwise Neural Clustering Process (NCP), the Clusterwise Clustering Process (CCP) and the Attentive Clustering Process (ACP) developed by Pakman et al. (2020) and Wang et al. (2021). ). ii) Proposed variants of the set-to-sequence architecture: these S2S variants enhance the pointer mechanism with the notion of predicting ordered clusters, as opposed to ordered elements. The first variant, called S2S-B (for "break"), adds secondary decision of whether or not start new cluster in parallel to the selec- tion of the set element to be placed next in the predicted sequence. The second variant, called S2S-C (for "clusterwise"), uses a threshold mechanism to select multiple elements forming single cluster at each step. For details, see Appendix A.3.
5.2 ORDERED MIXTURES OF GAUSSIANS
This dataset consists of 2D coordinates for number of points, generated from mixture of fi- nite number Gaussian distributions. The points should be clustered and the clusters ordered by distance from origin. Following convention from probabilistic models for clustering (McLachlan & Basford, 1988), we introduce a random variable c, signifying the cluster to which each data point x; is assigned. The generation process creates random number of clusters k. each with their

---

Under review as a conference paper at ICLR 2023
Distance from Origin
Target (supervision)
ACP
S25-8 NOC
Figure 3: Example of predicted ordered clusters. Target (supervision) shows clusters and their order through colour, red being closest to the origin point (marked with a gray x), dark blue and violet being furthest. Heat map (leftmost) indicates distance for individual points. The ACP prediction exhibits good clustering, but errs in the ordering (mistaken red and orange clusters). S2S-B exhibits good ordering, but incorrect clustering in the bottom-left quarter. NOC (ours) is closest to the target.
own parameter vector μ4j controlling the distribution of the j-th cluster. For comparison with prior work (Pakman et al., 2020), we use a Chinese Restaurant Process with a single modification - the addition of a target order of the clusters, based the Euclidean distance of their centroids from the origin point, An example of the joint prediction of per-element cluster assignments and predicted cluster order can be seen in Figure 3. The predicted order is denoted through colour gradient, with a bright red to deep blue and violet scale. the figure, three separate predictions are displayed, one from the ACP model, one from modification of set-to-sequence methods in the form of S2S-B and finally one from the proposed NOC model.
As shown in the rightmost column of Table 1, NOC outperforms other methods on both the cluster- ing task, according to V-Measure, and the cluster ordering task, measured with Kendall's 7. Specif- ically it improves by +1.18 points over the second-best clustering method (ACP) and +2.51 over the second-best set-to-sequence method (S2S-B). Its performance appears relatively consistent, showing a smaller standard deviation over three full training runs.
5.3 PROCEDURALLY GENERATED CATALOGS
The second experiment uses synthetic catalogs. These catalogs consist of varying-length sequences of clusters of elements, with repetition. Elements are colour-coded. These catalogs form the super- vised training targets yı: with the unordered set of available atomic elements forming the inputs X,- The correct composition of individual sections and the structure of the overall catalog, in the form of the order of its sections, depends on n-th order interactions between the input elements.
For procedural generation, these interactions are formalized as compositional (intra-cluster) and structural (inter-cluster) rules. A simplified example of a compositional rule would be: "if the inpur set contains only red, blue and yellow elements, section containing red and yellow elements in 1:1 ratio is a valid section". An example of a structural rule could specify that (given the same input) the catalog has begin with all-red section end Compositional rules also include upper cardinality constraints for valid sections. We use the tool provided by Jurewicz & Derezynski (2022) to generate catalogs.
all-blue one (top row of Figure 4).
As shown in Table 2, neural clustering methods appear to be better at composing valid catalog sec- tions but struggle with ordering sections into valid catalogs. This is indicated through two metrics- a compositional score, which is the percentage of predicted sections that were valid in accordance with the applicable n-th order ruleset, and a structural score, which the percentage of valid pre- dicted catalog structures (i.e. section ordering). By contrast, the adapted S2S models outperform neural clustering methods at correctly ordering sections, as measured via the structural score. NOC outperforms both methods each of the two scores. This improvement is also reflected on the same test set in the more general but related V-Measure and Kendall's т, shown in Table 1, where NOC surpasses the next-best models by +8.47 and +3.71 percentage points.
Among the sections predicted by neural clustering methods (NCP, CCP, ACP). the predominant error (present in 74% of invalid sections) stemmed from incorrect cardinality, even though the models correctly predict the composition (15%) and ratio (11%) of elements to include. This error occurs despite the presence of mechanisms that could, in principle, allow for the learning of max-cardinality
7

---

Under review as a conference paper at ICLR 2023
INPUT SET (NTERACTIONS))
RULESETS
OUTPUT STRUCTURE (VALID EXAMPLE)
COMPOSITIONAL
STRUCTURAL.
Figure 4: Procedurally generated catalogs. Relations between elements of the input set define the compositional and structural rules, which inform the generation of these synthetic datasets. A suc- cessful model should learn these rules from supervised exposure to the resulting synthetic datasets, and then be able to order new sets of elements according the learned rules. One valid example is given for each input set (wrapped over 2 lines). See the second paragraph of Section 5.3 for written description of the compositional and structural ruleset portrayed in the top row.
constraints: NCP constructs clusters element-by-element. further transforming the candidate cluster each element's addition; CCP and ACP obtain representation of the current candidate cluster before assigning candidate elements.
NOC overcomes this limitation through the addition of cluster-level cardinality prediction mecha- nism and corresponding loss. It outperforms the second best method on the section composition task by +11.96, +13.01 and +15.65 percentage points with regards to the 3rd 4th and 5th order relational ruleset respectively. It also performs better with regards to the structural score, offering smaller but consistent improvement over the S2S-C and S2S-B methods by +4.54, +3.59 and +3.61 points, with respect to increasing n-th order rulesets.
Table 2: Results over procedurally generated catalogs, by n-th order relational ruleset
Compositional score
Structural score
Method
3
n
NCP S2S
64.13 + 3.9
55.81 + 4.6
51.82 + 5.2
56.49 + 4.0
51.87 + 5.1
49.70 = 6,8
CCP+ S2S
75.40 3.2
71.49 4.3
65.11 + 4.5
70.21 3.5
68.39 4.7
66.55 + 5.4
ACP S2S
87.05 1.7
81.33
76.83 2.2
81.09 + 2.2
76.34 +.4
+ 73.86
S2S-B
84.99 0.5
82.90
74.82 + 0.6
92.33 1.5
87.31 2.0
S2S-C
82.03 1.8
2.1
72.13+2..
92.49 +
87.41 + 2.2
85,05 2.3
NOC
99.01 0.3
95.91 0.4
92.48 + 0.4
97.03 # 0.9
93.42 # 1.0
90.92 1.2
5.4 PROCAT
The final experiment was performed on the PROCAT dataset (Jurewicz & Derczynski, 2021), using its provided training and testing split. All models were trained on approximately 9K product catalogs and tested on a separate set of 2K catalogs. Unlike the benchmarks provided with the PROCAT dataset, this formulation of the task mirrors the Catalog Problem exactly, with no information about the target number of sections (clusters) being available to the models. Individual elements were transformed into vector representations via pre-trained, frozen language model as described the beginning of Section 5, removing its effect on the variation in performance on the downstream task. Figure displays sample catalog predicted by NOC from PROCAT input set of product offers.
8

---

Under review as a conference paper at ICLR 2023
Wave
19.-
12.-
10,-
Coca-Colaa
59,
HONTEZBAAS
WMM
24,-
BLACK
265
Figure 5; PROCAT. An example of three sequential sections predicted by the NOC as part of larger catalog, from an input set of products from the PROCAT dataset. The prediction groups elements into complementary sections (the three pages shown above) and orders them into a rendered catalog.
As the two rightmost columns of Table show, the PROCAT structure prediction task is more diffi cult than the previous tasks. The best results in terms of both the clustering quality (via V-Measure) and section order (measured indirectly via Kendall's with regards to element order) are approxi- mately 40% below the corresponding scores on the procedural task in its default configuration. One possible explanation stems from the existence of a higher number of reasonable substitutions for each element in any given section from the entire input set of initially available products. While also present in the procedural catalogs, this challenge becomes harder to overcome as the cardinality of the input increases from tens in the procedural case hundreds in PROCAT.
NOC outperforms both neural clustering methods and the adjusted set-to set-to-sequence models. While the overall pattern of neural clustering methods outperforming S2S-B and S2S-C in V-Measure is upheld, it is less pronounced (+2.37 points between ACP and S2S-B on PROCAT compared to +9.27 and +4.18 on the procedural and 2D Gaussian task respectively). The adjusted set-to-sequence models continue to outperform NCP, CCP and ACP on the ordering aspect of the task, with a margin of +12.66 points. NOC yields the best performance in terms of both partitional clustering and ordering, exceeding the relevant second-best methods by +11.46% and + +12.28% respectively.
6 CONCLUSION
The posited Catalog Problem consists of learning to group elements and to order the groups It poses a more difficult challenge than its individual components. Our work defined benchmark tasks representing this problem and presented approaches for them, including both adjusted baselines and a candidate approach, Neural Ordered Clusters (NOC). Existing neural clustering methods appear ineffective at learning cluster-level cardinality constraints. Our method offers an improvement in this area through its cardinality- preeiccttno module. NOC outperforms adjusted $2S methods in terms of both clustering quality and accuracy of the predicted cluster order, indicating that structuring models to address adaptive ordered clustering leads to improved performance over standard S2S prediction.
Nevertheless, the complexity and fluidity of intra- and inter-cluster relations result in the Catalog Problem remaining significantly more challenging than S2S processing. We considered a predictive solution to the catalog problem, where trained the model to yield single "ground truth" human- generated catalog given a set of products. Future work could consider a fully generative formulation of the problem that respects unlimited number of valid solutions for both clustering and ordering.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
Given the e-commerce context of the third presented dataset, we must highlight the wider problem of endless scroll user interfaces product presentation apps and social media (Lupinacci Amaral, 2020). Although the PROCAT dataset is tailored to the prediction of cluster sequences of finite lengths, we cannot rule out the possibility of extending the proposed adaptive clustering and cluster ordering models to non-finite sets. It is also in principle possible to retrain the proposed models with additional inputs such as embedded personal preferences, making the predicted catalogs tai- lored to specific individuals, which has previously been linked to mental health issues in relation to smartphone addiction (Noë al., 2019).
As with many machine learning systems, the results are not perfect, and sub-optimal predictions from NOC could silently disadvantage an end-user; for example a business may produce catalogs that don't 't make it easier for the reader to discover relevant, cost-saving offers, or individual may receive an inaccurate medical analysis (in the case of the hypothesized medical triage use case).
Applying this tool may impact the employment of people performing creative eutalog-related tasks, and further, might not even do the task as well as them. Product catalog design is considered some- thing of art among its practitioners, and there may be deep interactions not clearly evinced in training data that are lost by transiting the ownership of the catalog construction task from human subject matter experts to a machine learning model. Attempting completely replace human at this task may lead to both unsatisfactory and marginalizing results Birhane (2021).
We do not see any direct way for the presented methods to exacerbate bias against people of certain gender, race, sexuality, or who have other protected characteristics. However, bias inherent to the marketing decisions made by people who have designed the catalogues contained in the PROCAT dataset, will be propagated by models trained on it. Negative biases in this particular scenario include as the pink tax (Stevens Shanahan, 2017). In general, learning from socially-biased data and making predictions based on it will propagate those biases Buolamwini (2017); Raji (2020).
REPRODUCIBILITY STATEMENT
In order to ensure reproducibility all code and datasets needed for repeated experiments have been made freely available, as described detail in Appendix A.4 as part the provided supplementary materials. The anonymized code repository includes a comprehensive readme.md md file describing the necessary steps to set up the execution environment, download, generate and preprocess the datasets and run each of the experiments discussed in Section 5. The exact hyperparameters per experiment are stated both in the Appendix (A.4.1, A.4.2, A.4.3) and in the provided configuration files in the linked code repository. Additionally, detailed description of the NOC algorithm is provided (1) to ensure that the method can be reimplemented, necessary.
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Elie Aljalbout, Vladimir Golkov, Yawar Siddiqui, Maximilian Strobel, and Daniel Cremers. Clus- tering with deep learning: Taxonomy and new methods. arXiv preprint arXiv:1801.07644 2018.
Mihael Ankerst, Markus M Breunig, Hans-Peter Kriegel, and Jörg Sander. Optics: Ordering points to identify the clustering structure. ACM Sigmod record, (8(2):49-60) 1999.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06455 2016.
Abeba Birhane. The impossibility of automating ambiguity. Artificial Life, 27(1):44-1,, 2021.
J Buolamwini. Limited vision: The undersampled majority. In AI Now. MIT Media Lab, 2017.
Kenneth C Chu. Applications of artificial intelligence to chemistry. use of pattern recognition and cluster analysis determine the pharmacological activity of some organic compounds. Analytical chemistry, (669))1111-11983 118811177 1974.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL, 2019.
Jon Ergun, Zhili Feng. Sandeep Silwal, David P Woodruff, and Samson Zhou. bearning-augment k-means clustering, arXiv preprint wrXiv:2110.14099 2022.
Daniel Fernández-Cronzállezz Multitask pointer network for multi-represennationaa parsing- Knowledge-Bond Based Systems, 236:107766 107760, 2022.
Douglas Fisher, Ling Xu, and Nazih Zard. Ordering effects in clustering. In Machine Learning Proceedings 1992, pp. 163-168. Elsevier, 1992.
Douglas H Fisher. Knowledge acquisition via incremental conceptual clustering. Machine learning, 2(2):139- 172, 1987.
Brendan J Frey and Delbert Dueck. Clustering by passing messages between data points. science, 315(5814):922-976. 2007.
Chao Gan, Ruida Zhou, Jing Yang, and Cong Shen. Cost-aware cascading bandits. IEEE Transac- fions Signal Processing, 68:3692-3706, 2020.
Liuhao Ge, Yujun Cai, Junwu Weng, and Junsong Yuan. Hand pointnet: 3d hand pose estima- tion using point sets. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8417-8426, 2018
Roger Girgis, Florian Golemo, Felipe Codevilla, Martin Weiss, Jim Aldon D' D'uuuz, Samira Ebrahimi Kahou, Felix Heide, and Christopher Pal. Latent variable sequential set trans- formers for joint multi-agent motion prediction. International Conference on Learning Repre- sentations, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1024-1034, 2015. 2015.
Melvin Janowitz. An order theoretic model for cluster analysis. SIAM Journal on Applied Math- ematics, 34(1):55-72, 1978.
Stephen Johnson. Hierarchical clustering schemes. Psychometrika, 32(3):241-2544 1967.
Mateusz Jurewiez and Leon Derczynski. PROCAT: Product catalogue dataset for implicit cluster- ing, permutation learning and structure prediction. Proceedings of the conference on Neural Information Processing Systems: Datasets and Benchmarks Track, 2021.
Mateusz Jurewiez and Leon Derezynski. Set interdependence transformer: Set-to-sequence neural networks for permutation learning and structure prediction. In Proceedings of the International Joint Conference on Artificial Intelligence, 2022.

---

Under review as a conference paper at ICLR 2023
Mai Le, Detlef Nauck, Bogdan Gabrys, and Trevor Martin. Sequential clustering for event sequences and its impact on next process step prediction. In International Conference on Information Pro- cessing and Management of Uncertainty in Knowledge Based Systems, pp. 168-178. Springer, 2014.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam R Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer. In International Conference on Machine Learning, 2019.
Seon-Ho Lee and Chang-Su Kim. Deep repulsive clustering of ordered data based on order-identity decomposition. In International Conference on Learning Representations, 2020.
Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive cluster- ing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8547-8555, 2021.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. In CoRR, abs/1711.05101, 2017., 2017.
Ludmila Lupinacci Amaral. "absentmindedly scrolling through nothing': liveness and compulsory continuous connectedness in social media. Media, Culture & Society, 43:016344372093945, 07 2020. doi: 10.1177/016344772099955
Geoffrey McLachlan and Kaye E Basford. Mixture models: Inference and applications to cluster- ing. volume 38. M. Dekker New York, 1988.
Jamie Miles, Janette Turner, Richard Jacques, Julia Williams, and Suzanne Mason. Using machine- learning risk prediction models to triage the acuity of undifferentiated patients entering the emer- gency care system: systematic review. Diagnostic and prognostic research, 4(1):1-12, 2020.
Beryl Noë, Liam D Turner, David EJ Linden, Stuart M Allen, Bjor Winkens, and Roger M Whitaker. Identifying indicators of smartphone addiction through user-app interaction. Com- puters in human behavior, 99:56-65, 2019.
Ari Pakman, Yueqi Wang, Catalin Mitelut, JinHyung Lee, and Liam Paninski. Neural clustering processes. In International Conference on Machine Learning, pp. 7455-7465. PMLR, 2020.
Divesh Pandey and Ravindranath Chowdary. Modeling coherence by ordering paragraphs using pointer networks. Neural Networks, 126:36-41 41, 2020.
Xi Peng, Yunfan Li, Ivor W Tsang, Hongyuan Zhu, Jiancheng Lv, and Joey Tianyi Zhou. Xai beyond classification: Interpretable neural clustering. Journal of Machine Learning Research, 23 (6):1-28, 2022.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 652-660, 2017.
Philippe Rabiller, Peter Boles, and Boualem Boashash. Ordered clustering: A way to simplify analysis of multichannel signals. In 10th International Conference on Information Science, Signal Processing and their Applications (ISSPA 2010), pp. 237-242. IEEE, 2010.
Deborah Raji. How our data encodes systematic racism. MIT Technology Review, 10, dec 2020.
Meitar Ronen, Shahaf E Finder, and Oren Freifeld. Deepdpm: Deep clustering with an unknown number of clusters. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Andrew Rosenberg and Julia Hirschberg. V-measure: A conditional entropy-based external cluster evaluation measure. In Proceedings of the 2007 joint conference empirical methods in natural language processing and computational natural language learning 420, 2007.
pp. 410-
12

---

Under review as a conference paper at ICLR 2023
Chenxu Shi. Pointer network solution pool: Combining pointer networks and heuristics to solve tsp problems. In 2022 3rd International Conference on Computer Vision, Image and Deep Learning & International Conference on Computer Engineering and Applications (CVIDL & ICCEA), pp- 1236-1242. IEEE, 2022.
Jennifer L Stevens and Kevin Shanahan. Structured abstract: Anger, willingness, or clueless? understanding why women pay a pink tax on the products they consume. In Creating Marketing Magic and Innovative Future Marketing Trends, pp. 571-575, Springer, 2017.
2017.
Krzysztof Turowski, Jithin K Sreedharan, and Wojciech Szpankowski. Temporal ordered clustering in dynamic networks. In 2020 IEEE International Symposium on Information Theory (ISIT), pp- 1349-1354 IEEE, 2020.
Stijn Marinus Van Dongen. Graph clustering by flow simulation. PhD thesis, Utrecht University, 2000.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Daniel Vial, Sujay Sanghavi, Sanjay Shakkottai, and R Srikant. Minimax regret for cascading bandits. arXiv preprint vrXiv:2203.12577 2022.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. Advances in neural informa- fion processing systems, 28, 2015.
James Vlasblom and Shoshana Wodak. Markov clustering versus affinity propagation for the partitioning of protein interaction graphs. BMC bioinformatics, 10(1):1-14, 2009.
Tianming Wang and Xiaojun Wan. Hierarchical Attention Networks for Sentence Ordering. AAAI, 33:7184-7191, 2019. ISSN 2159-5399. doi: 10. 10.1669/aaai.v33312 333001.33
Yueqi Wang. Yoonho Lee, Pallab Basu, Juho Lee, Yee Whye Teh, Liam Paninski, and Ari Pakman. Amortized probabilistic detection of communities in graphs. arXiv preprint arXiv:2010.1572) 2021.
Jiqqi Wen. Spectral-pointer network: Pre-sort leads the pointer network to elude the tsp vortex. In 2022 3rd International Conference on Computer Vision, Image and Deep Learning Interna- tional Conference on Computer Engineering and Applications (CVIDL & ICCEA), pp. 578-586. IEEE, 2022.
Hannah Wenzel, Daniel Smit, and Saskia Sardesai. A literature review on machine learning in supply chain management. In Artificial Intelligence and Digital Transformation in Supply Chain Man- agement: Innovative Approaches for Supply Chains. Proceedings of the Hamburg International Conference of Logistics (HICL), Vol. 27, pp. 413-441, Berlin: epubli GmbH, 2019.
Ronald Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2):270-280, 1989.
Yongjing Yin, Fandong Meng, Jinsong Su, Yubin Ge, Lingeng Song, Jie Zhou, and Jiebo Luo. Enhancing pointer network for sentence ordering with pairwise ordering predictions. In AAAI, volume 34, pp. 9482-9489, 2020.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16259-16268, 2021.
Yutao Zhu, Kun Zhou, Jian- Nie, Shengchao Liu, and Zhicheng Dou. Neural sentence ordering based on constraint graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 14656-14664, 2021.
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.I NOC ALGORITHM
In this section of the appendix we outline the progression over all three stages of the proposed Neural Ordered Clusters method. Steps 1-23 jointly describe the processing within NOC₁ and NOC₂ (steps 12-16 specifically for the latter), as presented in Section 4. The third module, NOC3 is shown in steps 24-30. We begin with unordered set X R (of any cardinality), and assume that it has at least two elements, which can then potentially belong to separate clusters. This set is represented as matrix of d-dimensional elements, ordered according some arbitrary permutation π into x The initial, intermediate output comes in the form of individual clusters of elements at each j-th iteration, which ultimately form the set of all predicted clusters (C- (Ci..... C.).) C+ Each candidate cluster goes through final cardinality prediction step, resulting in the threshold value of Lj through which some elements may excluded from their original cluster. Finally, an Enhanced Pointer Network (EPN) performs & iterations, selecting a single cluster to be placed next in the final output sequence ý by the index of the highest value in the predicted attention vector am.
Algorithm Neural Ordered Clusters
Require:
[
b At least two elements, otherwise single cluster No repeated elements
Ensure:
+ x; i,j + i (n....,m) 1: Ex - x x) j+1
2: < -
Track number of unassigned elements > Randomly chosen anchor for initial cluster Initialize unassigned representations > No previous clusters
3: e f- E,
4: U,+ SIT(E Me(d)
5: a f+2 Ø
6: while
do
7:
2j ~ N NSIITe (g))
8:
fori+l... do
9:
& ea 2j. SIT(U)), SIT(G))) end for
>> j-th cluster assignments per element
11:
C₁ - E s4=1
Cluster j from assignments (sorted)
12:
t (PMA(c (IT(g))
Predict cluster cardinality
13:
if C S then
Cj Cj
15:
else
C + C1a
> Adjust j-th cluster's cardinality
17:
end if
j j+1
19:
ei + E,
D Randomly chosen anchor for next cluster
20:
U, + SIT(EE C2)
P Update unassigned representations
21:
4 f C- --C=(C (CI...., C₁
P Update preceding clusters' representations Adjust number of unassigned elements
22:
r 1C,1 -1
23: end while
24: h + SITT(C ~ C)
> First hidden state from all clusters Final prediction placeholder
25: ÿ )0
10
26: for m +
& do
am, hd + EPN(Cs, h
Enhanced pointer attention over k predicted clusters
28: t arg max (m))
29: Im C,
>> Next cluster by highest attention index
30: end for
A.2 ENIIANCED POINTER NETWORK
In all reported experiments the same set-to-sequence module, the Enhanced Pointer Network Yin et al. (2020), which a pointer-attention based method inspired by the popular Pointer Network Vinyals al. (2015). It offers performance improvement by leveraging additional mecha-
14

---

Under review as a conference paper at ICLR 2023
nisms for pairwise ordering predictions towards improved global and local coherence of the output sequence. Formally, the conditional probability predicted order ÿ is calculated as:
K
pa C) = II
( شマン 8o); = PMA(SIT(Cx.)
(10)
po(K) Øejs C) = softmax(w tan W2M,))
(11)
h LSTM( hg 8c
(12)
where W1 and W2 are model parameters, K is the total number of clusters, tanh is the hyperbolic tangent nonlincarity, Cy- is the fixed-length embedding of the cluster selected at the preceding step j and hy is the hidden state of the permutation module at current step i. The first hidden state hg is initialized from the permutation invariant set representation of all previously predicted clusters 8c, obtained via SIT and PMA. The M, matrix provides additional context consisting of 2 kinds of information. The first is global orientation relating all remaining unordered clusters to one another. The second is local coherence between previously selected clusters and remaining candidates. This contextual information is obtained via history and future sub-modules from the original matrix all cluster representations (Cr x C). These two sub-modules output pairwise ordering predictions in relation to each candidate cluster, which then concatenated to form M,. For exact implementation details, we refer the reader to Yin et al. (2020).
A.3
BASELINES
In this subsection, more detailed description of the proposed S2S variants is given. The S2S-B variant utilizes pointer attention to select individual remaining set elements each step, following the convention of Pointer Networks Vinyals et al. (2015) and their enhancements ( (Yin et al., 2020). What distinguishes S2S-B from these models an added prediction target which requires making binary decisions, where is the cardinality of the input set. At each step of the predicted permutation sequence, S2S-B indicates whether the currently selected element should be the last one of the current, open cluster. If so, this would indicate "break" " in the sequence, reminiscent of a page break in product catalog. Once the last available element is reached, any remaining opened clusters are closed by default, hence are considered members of the current open cluster.
All previously pointed-to elements since the last break
The S2S-B model thus capable of predicting clustering where each element is assigned its own cluster and one where all elements belong to a single cluster. It is guaranteed to assign a cluster to every single element and can handle varying cardinality input sets, like all pointer networks. The first difficulty faced due to this particular modification stems from highly skewed class distribution. Namely, we never complete (or break) cluster after each element. This is mitigated via class- weighted binary cross-entropy loss function:
m
LBCE-W (0)
(wb x ym x log(gm)) + (1- ym) x log(1 (m))
(13)
m
Where is the number of training examples, wa is the adjusted weight for the positive class, and Y: and iJ are the target and prediction respectably. This loss factor then scaled and added to the negative loss likelihood loss used to train the pointer selection mechanism. The main disadvantage of this model is that predicts meaningless in-cluster order, making the loss signal noisy. The order of elements within each cluster is meaningless within the confines of the presented Catalog Problem.
To mitigate this disadvantage, a second variant was developed. Referred to as S2S-C (for "cluster- wise"), this model predicts the entire next cluster of elements each step, instead of pointing to single next element in the output sequence. Instead of performing n transformation steps in a loop over the entire input set, it outputs attention vector over all available elements until there none
15

---

Under review as a conference paper at ICLR 2023
left. Thus it is also bound between assigning all elements to single cluster every element to its own cluster, much like S2S-B, guaranteeing cluster assignment for each element of the input set.
In order to predict clusters of adaptive, input-dependent cardinality, the formula for obtaining the pointer-attention vector over available elements had to be adjusted. The softmax operator was replaced with the sigmoid function (o) and threshold (f.)) of 0.5 was adopted. At each step j {1,2... : ; every element with corresponding attention value (a:) above the threshold is thus assigned to the next cluster:
a; a tanh( W2E, + W(h)))
(14)
[0, 1, otherwise
a < to
v -1,
(15)
During training, the S2S-C model was teacher-forced (Williams & Zipser, 1989) to prevent the cascading impact of incorrect initial cluster assignment subsequent computation steps, which is known challenge in certain areas machine learning, such as the multi-armed bandit problem (Gan al., 2020). This is departure from the other tested models (with the exception of S2S-B), as all neural clustering baselines are also teacher-forced during training, as per author implementations of the papers that originally introduced them.
A.4 CODE, DATASETS AND PARAMETERS
The code required for all three of the main experiments can be found fully anonymized reposi- tory under the following link:
https://github.c con/ wwwoo///noooouuuuoooooooo http://www.....cmooooooooooaaaaa wtw.aapprrrsmmmssissssssss neural-ordered-cllee
Follow the instructions provided in the readme. md document set up the necessary environment locally, via the requirements.tx file listing all necessary packages and their versions.
In the following sections we describe each dataset in more detail, including how download or generate it. All datasets are freely available under publicly accessible links. Additionally, each section contains the specific hyperparameters used for repeated experiments well as the exact number of layers and parameters per tested NOC model.
A.4.1 ORDERED MIXTURES OF 2D GAUSSIANS
Data. The dataset for predicting ordered clusters of 2D Gaussians (based on their distance from the origin point) synthetically generated when running the experiment via the linked run run- mun_gauss2D, py file. The full, default configuration is given in the parser arguments (nothing should require ad- justment to run the equivalent experiment). This includes default seed, which should help ensure repeatability. In the provided experiments generate 30K batches of 64 examples each, for total of just under 2 million individual training examples for full run. Each example is a set of 5 to 100 individual points characterized by their coordinates, generated through the Chinese Restaurant Process with dispersion parameter a set to 0.7 for all experiments. Unlike the batch generation pro- cess used by Pakman al. (2020), we generate batches with diverse number of clusters and cluster cardinalities in each example.
Hy prrpaaaaressss.. The training regimen includes a learning rate adjustment from le-4 to 5e-5 the 15 batch and le-5 at the 20K-th batch. The AdamW (Loshchilov & Hutter, 2017) optimizer was used with weight decay coefficient of 1e-3. Additionally, the default weights per loss factor provided. The main clustering loss factor X is equal to 1.0. the cluster ordering loss factor is set to X, = 4.0 and the cardinality prediction loss factor A = 3e-3. A 100 inferences samples is generated by default during validation, final metrics being calculated for the clustering prediction with the highest probability.
Model parameters. The NOC model with reported performance had over 12mil trainable param- eters. The element and cluster encoding functions, each consisting three stacked ISAB layers had the input and hidden dimensions of 128. The set pooling functions consisted of two stacked
16

---

Under review as a conference paper at ICLR 2023
ISAB layers followed by a PMA layer, also with 128 dimensions. The NOC₁ clustering module consistently uses a Parametric Rectified Linear Unit (PReLU) as the nonlinearity He et al. (2015).
A.4.2 PROCEDURALLY GENERATED CATALOGS
Data. The dataset for predicting the cluster composition (sections of offer tokens) and structure (order of these sections) of synthetic catalogs is automatically generated when running the linked run uun_synthttic.y experiment script with default parser arguments. This script loads the provided configuration file rynthetic.rulesets which the generated synthetic catalogs will adhere. In all reported experiments we refer to this default set of rulesets, but encourage researchers to treat it as easy-to-edit, flexible configuration that can be adjusted for other exploratory experiments.
which specifies all compositional and structural rulesets to
For the experiments, we generate 300K synthetic catalogs for the training and 75K for the vali- dation and test sets (split into 15 data-loaders). Each example consists of 35-50 offer tokens, each batch consists of 64 examples with varied number of clusters and cluster cardinalities in each batch. The NOC model is trained over 250K batch iterations, the equivalent of 50 epochs.
Hyperparameters. The procedurally generated catalog training regimen includes a learning rate adjustment from le-4 to 5e-5 at the 100K-th batch iteration and le-5 at the 200K-th. The AdamW (Loshchilov & Hutter, 2017) optimizer was used with a weight decay coefficient of Ie- 3. Additionally, the default weights per loss factor are provided. The main clustering loss factor d equal to 1.0, the cluster ordering loss factor set to A = 15.0 and the cardinality prediction loss factor A = 0.1. A hundred inferences samples is generated by default during validation, final metries being calculated for the clustering prediction with the highest probability.
Model parameters. The NOC model with reported performance had over 18mil trainable param- eters. The element and cluster encoding functions, each consisting of four stacked ISAB layers had the input and hidden dimensions of 128. The set pooling functions consisted of three stacked ISAB layers followed by a PMA layer, also with 128 dimensions. The NOC₁ clustering module consistently uses Parametric Rectified Linear Unit (PReLJU) as the nonlinearity He et al. (2015).
A.4.3 PROCAT
Data. The PROCAT dataset is freely available under the following link: /
htt.://cw......eee.eeee/ee/etttt CGructaree_Pred..tffomm//
We follow the provided train test split of 8K 2K catalogs and all pre-processing steps from the original paper (Jurewicz & Derczynski, 2021). The provided section break tokens are removed in the pre-processing to enable the prediction of input-dependent number of sections. Elements are by default truncated to 512 dictionary tokens for the language-specift, BERT model, available in the linked hugging face repository and the suggested max-offer threshold of 200 per catalog is followed. Batches of 64 catalogs are used. The proposed NOC model is trained for 12.5K batch-iterations, the equivalent of 100 epochs.
Hyperparameters. The PROCAT training regimen includes a learning rate adjustment from 1e-4 to 5e-5 at the 5K-th batch iteration and le-5 at the 10K-th. The AdamW (Loshchilov & Hutter, 2017) optimizer was used with a weight decay coefficient of le-3. Additionally, the default weights per loss factor are provided. The main clustering loss factor d equal 1.0, the cluster ordering loss factor is set to X = 10.0 and the cardinality prediction loss factor A 0.5. A hundred inferences samples is generated by default during validation, final metrics being calculated for the clustering prediction with the highest probability.
Model parameters. The NOC model with reported performance had 23mil trainable parameters (not including the BERT model, which was frozen during training). The element and cluster encod- ing functions, each consisting of stacked ISAB layers had the input and hidden dimensions of 128. The set pooling functions consisted of 4 stacked ISAB layers followed by PMA layer, also with 128 dimensions. The NOC, clustering module consistently uses a Parametric Rectified Linear Unit (PReLU) as the nonlinearity (He et al., 2015).
17


Reviews:
[{"review_id": "tU-vqeKCYP", "reviewer": "Reviewer_PMJN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Neural Ordered Clusters (NOC), a supervised approach for set-to-sequence prediction. The resulting model learns to cluster data *and* sort elements within each resulting cluster. NOC was evaluated on two synthetic datasets and a real-world dataset (PROCAT, Jurewickz and Derczynski 2021).", "summary_of_the_review": "Unfortunately I do not consider the manuscript fit for publication at this stage.\n\nThe paper's main shortcoming is its severe lack of clarity which, in my opinion, necessitates a major revision to be addressed.\n\nThe paper's main contribution is arguably dealing with a rare setup: \"The Catalog Problem\", where items need to be *both* clustered *and* sorted within each cluster. In spite of this, I do not believe the manuscript to be sufficiently convincing with respect to the relevance and importance of this problem.\n\nFinally, and perhaps less importantly, the manuscript's methodological contributions also appears to be limited, consisting of a novel application of existing components (c.f. Attentive Clustering Processes, Set Interdependence Transformers,  Enhanced Pointer Networks). However, this would be in my opinion only a minor problem if the authors provided sufficiently compelling evidence that this novel application is truly relevant for the field.", "strength_and_weaknesses": "Strengths:\n+ Proposed approach is reported to compare favourably to baselines in two synthetic datasets and a real-world dataset (PROCAT). \n\nWeaknesses:\n+ Severe lack of clarity throughout the paper: problem statement, method and experiments.\n+ Insufficient motivation: not enough compelling evidence that \"The Catalog Problem\" is relevant.\n+ Incremental methodological contribution: proposed approach seems a novel application of existing methods (disclaimer: this judgement may be impaired by the lack of clarity).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ogJ_QPZAHAt", "reviewer": "Reviewer_W2N9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a neural architecture designed to jointly cluster observations and order the clusters. The approach is demonstrated on three data sets; two synthetic and one real-world. The approach is compared to step-wise models which first cluster and then order, as well as some additional method proposed by the authors which extend the set-to-sequence architecture. Performance is measured using V-measure and Kendall's rank correlation. \n", "summary_of_the_review": "The authors address a difficult problem, yet complexity of the method and allusions to details that are not explicitly stated detract from the overall presentation.\n", "strength_and_weaknesses": "The proposed method builds on state-of-the art methods for reasoning over sets using neural architectures. \nHowever, the end result has quite a lot of machinery, as illustrated in Figure 2. This makes it unclear which components may or may not be important for overall performance. Some ablation experiments are warranted. Can some elements of the model be removed or replaced with trivial stand-ins? How would this effect performance?\n\nThe notion of previous clusters are mentioned a few times. Where do these come from? Are they the training data? Is this method iterative? There is some weakness in the explanation here. \n\nI have a hard time thinking of examples of the catalog problem. How common is it to need to order clusters? I did not follow the references in the first paragraph, which are cited as exemplars of application areas. Is it common to have supervision for the problems in these applications. This is a weakness in the presentation and extends into the demonstration data sets. A 2-dimensional mixture of Gaussians is a toy demonstration. The description of the procedurally generated catalog is vague and difficult to appreciate. What is the ground truth ordering? Is it the order with which each cluster is generated? Similarly, there appears to be no explicit statement of what constitutes a cluster or under what principle the clusters are ordered for the PROCAT experiment. I found it difficult to appreciate the goal of the ordering on PROCAT. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OU1qcacViC", "reviewer": "Reviewer_Nkbm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the catalog problem that jointly separates items into groups and orders these groups. The authors proposed a novel method called Neural Ordered Clusters (NOC) to solve the problem. The proposed model is tested on synthetic and real-world datasets.\n", "summary_of_the_review": "I believe the authors introduced a new problem and provided a novel method for solving the problem. However, I am not convinced that the method is helpful in solving a real-world problem. The clarity of the paper can be further improved. ", "strength_and_weaknesses": "Strengths\nThe authors introduced a novel problem called the catalog problem and proposed a novel method for solving this problem.\nThe proposed method outperforms other methods in terms of clustering and ordering qualities.\n\nWeaknesses\nThe motivation for the problem is not well described. I am not sure whether the proposed problem is realistic.\nThe descriptions of the methods and the experiments are not clear enough.\n\nI have included the details in the next section.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HlbjXCIzHf", "reviewer": "Reviewer_ZYNQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies an apparently new problem, called Catalog Problem, which looks for grouping of items and ordering of groups.  The paper further proposes a neural network-based model called Neural Ordered Clusters for solving the problem.  The empirical results show apparently superior performance of the proposed method over several baseline methods on two synthetic data sets and one large real-world data set.\n", "summary_of_the_review": "The paper proposes a new problem and a new model for solving the problem.  However, the proposed problem is not clearly defined and it cannot demonstrate the strengths of the proposed method.", "strength_and_weaknesses": "Strengths:\n\n- The paper attempts a new problem to study.\n- A model based on neural network is proposed to solve the catalog problem.\n- The empirical results apparently show that the proposed method performed better than other baseline methods.\n\nWeaknesses:\n\n- The Catalog Problem is not clearly defined in the paper.  In particular, it is not clear what information is needed for the problem.  The identified clusters are supposed to be ranked, but it is unclear if it needs any additional information for ordering the items.\n- It is not clear what value the the Catalog Problem problem discussed in the paper has.  For example, products where grouped into meaning clusters in Figure 5, but it is not obvious what meaning the ordering can provide.  The proposed method may appear to be less significant if the value of underlying problem cannot be justified.\n- The related work section looks a bit brief and cannot position the proposed method well among other works.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject