Match prediction from group comparison data using neural networks
Authors: Sunghyun Kim
Minje jang
Changho Suh
Venue:  (2020)
Keywords: Neural networks
Group comparison
Match prediction
Rank aggregation

Abstract: We explore the match prediction problem where one seeks to estimate the likelihood of a group of M items preferred over another, based on partial group comparison data. Challenges arise in practice. As existing state-of-the-art algorithms are tailored to certain statistical models, we have different best algorithms across distinct scenarios. Worse yet, we have no prior knowledge on the underlying model for a given scenario. These call for a unified approach that can be universally applied to a wide range of scenarios and achieve consistently high performances. To this end, we incorporate deep learning architectures so as to reflect the key structural features that most state-of-the-art algorithms, some of which are optimal in certain settings, share in common. This enables us to infer hidden models underlying a given dataset, which govern in-group interactions and statistical patterns of comparisons, and hence to devise the best algorithm tailored to the dataset at hand. Through extensive experiments on synthetic and real-world datasets, we evaluate our framework in comparison to state-of-the-art algorithms. It turns out that our framework consistently leads to the best performance across all datasets in terms of cross entropy loss and prediction accuracy, while the state-of-the-art algorithms suffer from inconsistent performances across different datasets. Furthermore, we show that it can be easily extended to attain satisfactory performances in rank aggregation tasks, suggesting that it can be adaptable for other tasks as well.

Under review as a conference paper at ICLR 2020
MATCH PREDICTION FROM GROUP COMPARISON DATA USING NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
We explore the match prediction problem where seeks estimate the likeli- hood of a group of M items preferred over another, based on partial group compar- ison data. Challenges arise in practice. As existing state-of-the-an algorithms are tailored to certain statistical models, we have different best algorithms across dis- tinct scenarios. Worse yet, we have no prior knowledge on the underlying model for given scenario. These call for unified approach that can be universally ap- plied a wide range of scenarios and achieve consistently high performances. To this end, we incorporate deep learning architectures so as to reflect the key struc- tural features that most state-of-the-art algorithms, some of which are optimal certain settings, share in common. This enables us infer hidden models under- lying a given dataset, which govern in- in-group interactions and statistical patterns comparisons, and hence devise the best algorithm tailored to the dataset at hand. Through extensive experiments on synthetic and real-world datasets, we evaluate our framework in comparison to state-of-the-an algorithms. It turns that framework consistently leads to the best performance across all datasets terms of cross entropy loss and prediction accuracy, while the state-of-the-ee algorithms suffer from inconsistent performances across different datasets. Furthermore, we show that it can be easily extended to attain satisfactory performances rank aggregation tasks, suggesting that it can be adaptable for other tasks as well.
INTRODUCTION
The most elementary form of comparisons pairwise: we often compare a pair of items and make judgments to which one is of higher utility simply preferable over the other. With large amount of such comparison data, one can consider various interesting tasks. One may wish to predict future outcomes of unseen matches, and also to rank alternatives in order of utility preference.
Challenges arise in carrying out these tasks. Almost all existing state-of ofeee----te-rrt algorithms have been developed under the assumption that given scenario, there exists certain underlying model which governs statistical patterns of comparison data (see Section for details). As such, we have different best-perforning algorithms across distinct scenarios. This traditional approach, which begins by assuming certain models to develop algorithms, comes with limitations in practice.
First, it gives rise to inconsistent performances. No single algorithm can perform consistently well in wide range of scenarios, since it has been tailored to specific model. Second, it is hard to know the underlying model without expert domain knowledge. In its absence, we have little choice but to find an appropriate algorithm via trial- -ril--and-error. Third, the model can be inherently complex for any existing algorithm be effective. Sometimes groups of items are compared, thus the effects of interactions among in-group items come into play, further complicating the model.
In this work, we propose a unified algorithmic framework aimed to overcome these barriers. We focus on the match prediction problem where one wishes to estimate the likelihood of group of M items preferred over another, based on partially observed group comparison data among a collection of n items. One can imagine that such group comparison data may bear complex statistical patterns due to a combination of two underlying models: the interaction model which governs the effects of in group interactions in determining the utility preference of group; and the comparison model which governs the statistical patterns of pairwise group comparison data. Hence, designing a novel framework hinges heavily upon accurate inference of these underlying models.

---

Under review as a conference paper at ICLR 2020
Main contribution. We incorporate deep learning techniques into our framework design. This enables us to infer the underlying models from real-world data obtained from a given application, and thus to achieve consistently high performances a variety of datasets from diverse real-world applications where match prediction tasks are of interest.
To this end, we build on progress made through analysis in well-defined statistical models. We gain insights instrumental to the progress by looking into existing state-of-the-att algorithms in related and long-studied tasks such as rank aggregation (Negahban al., (0166] [uunter] 2004] Huang al., 2006 2008). We find that most of them share a key element. They all exhibit so-called reward-and- penalry mechanisms in estimating the utilities of individual items.
To be more specific, they reward an item more greatly for winning (or being more preferred) in a dis- advantageous comparison where its group is weaker than the counterpart, Likewise, they penalize more greatly for losing (or being less preferred) in an advantageous one. In addition, the magnitudes of rewards and penalties are proportional to the contribution of the individual item to its group.
This structural similarity across the state-of oftie--f--heaaree art algorithms has attracted our attention. Through some manipulation. we find that they employ the same basic rule for estimating individual utilities (see (6) in Section 4 for details). The terms corresponding to rewards and penalties turn out to vary as either one of the two underlying models changes. This observation has inspired us to incorporate neural networks into our framework design.
The novelty of our design is salient in an ablation study where we compare it with a simple design. As an initial effort, a single-layer neural network has been employed to predict winning probabilities of unseen group matches (Menke & Martinez 2008). It has shown promising result, demonstrat- ing prediction accuracy be improved on real-world online game dataset, but also exhibited scalability issue. requires one input node per item, making it prohibitive to be extended to real- world applications with large number of items. Leveraging more advanced architectures (see Figures [ and 2 motivated by observant analysis as emphasized, our design not only addresses such scalability issue by design, but also outperforms the single-layer neural network. The merits of our design are evaluated against the single-layer neural network and other state-of-the-arr algorithms through extensive experiments on a variety of synthetic and real-world datasets (see Section 5
Using synthetic datasets, we demonstrate that our approach can achieve the performances of the state-of-the-as algorithms in the models for which they have been specifically developed. We inves- tigate four models. Three consider various extensions of the Bradley-Terry--Lun Terry 1952) to the group comparison scenario. The other is generalized version of the Thurstone model (Herbrich et al., 2007) widely used in skill rating systems of online games. As a result, we show that our framework consistently yields the best performances across all these datasets (near- best in some cases), while the other of-te-of-the-ar algorithms suffer from inconsistent performances across different models.
model (Bradley &
Using rorl-world datasets, also demonstrate that framework performs consistently well across diverse real-world applications. We investigate five real-world datasets (sources in Footnote 6) One is crowd-sourced image classification dataset, another is collection of movie ratings, and the other three are match records from online games. We consider, in addition to the cross entropy loss, the prediction accuracy as another metric (detined in ( As a result, we show that our framework consistently yields almost the best performances across all of these datasets in terms of both metrics.
We also show that our framework can be easily extended to achieve the best performance in rank aggregation tasks where one seeks to rank items in order of utility or preference. Using real- world dataset of movie ratings, we demonstrate that our framework yields the best performances in terms of two well-known metrics (see Footnote 10): Kendall tau distance (Kendall) 1938) and normalized discounted eumulative gain (Järvelin & Kekäläinen 2002). This result suggests that can potentially be adaptable for other tasks as well.
RELATED WORK
The most related prior works are (Huang et al. 2006 2008 Li et al., 2018 Herbrich et al., 2007)
where plausible statistical models (some long-established and some widely used in practice) for in-
2

---

Under review as a conference paper at ICLR 2020
group interactions and group comparisons assumed, and statistical analysis is carried out to great extent. We use the algorithms developed in these models and their variants for main baselines.
The problem of estimating individual utilities from group comparison data has been investigated in (Huang et al.; 2006; 2008). They considered extensions of the BTL model where the group utility is either the sum the product of individual utilities, and group comparison data follow the BTL model in terms two group utilities (which we call the BTL-sum and BTL-product models respectively).
A more advanced in- in-group interaction model has been explored in (Li et al. 2018). They considered scenario where pair of individuals in group leads synergy, which contributes to the group. The group utility is represented as the sum of two quantities (the HOI model): (1) the sum of individual utilities and (2) the sum of the products of all pairs of individual utilities. general scenario, where any k-tuple of individuals in group leads to synergy, has been considered in (DeLong et al. 2011).
In (Herbrich et al. 2007). they assumed individual utilities to be centered around a mean following Gaussian distribution and viewed the group utility as their sum (the Thurstone model). The algorithm therein is widely used in skill rating systems of online games where groups of users compete.
Employing neural networks has been considered as an initial effort to predict winning probabilities of unseen group matches (Menke & Martinez, 2008). It has been shown that single-layer neu- ral network can some variants of the BTL model (Huang et al. 2008) and improve prediction accuracy through experiments
real-world online game dataset.
Some other works have also employed neural networks to exploit hidden models, for example, in information retrieval (Burges et al. 2005 Richardson et al. 2006 Guo et al. 2016) and community
detection (Chen et al. 2018b) in graph domains (Scarselli et al. 2009 Schlichtkrul et al. 2018).
PROBLEM SETUP
We investigate the match prediction problem where we seek to predict comparison outcomes for unobserved pairs of groups given collection of pairwise group comparison data. We consider the setting where each group consists of M individual items.
We are given comparison observations between two groups of size M. To be more specific, each comparison consists of (A,, B3 BAB).. A and B are groups of M individuals, respectively. yAB indicates which group wins (or is preferred):
1
if A B;
MAB
otherwise,
(1)
where B indicates that group A wins over group B. We denote by Dobs the of observed comparisons, so all information we are given is {(A, B, )AB)) The unobserved set of comparisons, which we wish predict, denoted by Dunobs.
We consider the cross entropy loss as our metric, as it serves to quantify the discrepancy between two probability distributions. We define iAAB the estimate Pr [YLS = 1]. Our goal is to minimize
Σ
BLA log LAB + +1 y.AB) log(1 - -.AB))
(2)
Dobs
(A.
We develop our algorithm primarily based on the cross entropy loss, but for evaluation purposes, also consider other metrics, such as prediction accuracy, the Kendall tau distance, and normalized discounted cumulative gain (see Footnote I0).
Notation. We denote by [n] {1,2..... n the set all individual items. We use lowercase letters such as and represent individual items, and caligraphic letters such A and B to represent sets of items. We denote by [w: heA the set of w:'s for all i E A. We denote by [ws]ees a vector of w;'s for all A and its ordering information is provided in the context. We denote by an estimate of y. We use subscripts as in MAS when concerns a comparison between A and B. We use superscripts as w(()) when w is updated iteratively. We use boldsymbols as w to represent the set of w; 's for all E [m).
3

---

Under review as a conference paper at ICLR 2020
4 PROPOSED ALGORITHM
stage
2 stage
R
33)
R
(2)
R
G
LAs
P
P
B)
Figure Overall architecture our proposed algorithmic framework.
R
G
[
hea
eC
2M
L.M
RACU
ReCO
ReCU
aigmond
ReL.U
ReL.d
ReLU
sigineed
[
bes
Input layer
fully connected layers
output layer
RRI Lico
10
Input layer
fully connected layers
Figure 2: Detailed architectures of the key modules: module (or P) (left) and module G (right).
We propose a neural network architecture that learns from observed group comparison data to predict unobserved group comparison outcomes. As presented in Figure L it consists of three modules, presentt which we denote by R, P and G. Figure presents the detailed architecture of R and (left) which have the same structure but are separate modules with different weights, and that of G (right).
4.1 MOTIVATION
Our decision to incorporate two modules R and P into our architecture has been inspired by state-of- the-art algorithms that have been shown optimal (either achieving the minimal sample complexity or the global minima of cross entropy loss; details presented soon) under extensions of the well- established BTL model. Our main contribution lies in this design choice. Examining the algorithms in detail, we discover that they all share a similar mechanism in estimating individual utilities:
(a) They all exhibit "reward" and "penalty" terms in the estimation process (details in and They update individual's utility estimate by rewarding the individual for contributing to its group's 's winning and likewise penalizing it for contributing to its group's 's losing.
(b) These reward and penalty terms vary in form as the underlying models change.
(c) The magnitudes of rewards and penalties depend on the power dynamies between groups. A greater reward is given to an individual when its group is relatively weaker compared to the opponent group, and likewise greater penalty is given when its group is relatively stronger.
(d) Their magnitudes also depend the portion of individual's contribution within its group. Suppose an individual's group wins (or loses) in a group comparison. The individual is given a greater reward (or penalty) when its contribution among others in the group is relatively greater.
Our algorithm design principles are centered around these key observations. We introduce two separate modules to represent rewards and penalties respectively. We employ deep neural networks into the modules so that they can serve to infer underlying models, which are unknown in practice.
Reward and Penalty in State-of-the-art Algorithms. We present details of the state-of oftte------rrttt algorithms developed under extensions of the BTL model. They illustrate all of the key observations.
To begin with a simple case, Rank Centrality (Negabban et al. 2016) has been developed under the well-known BTL model where individual items compared in pairs. It has been shown to achieve the minimal sample complexity for top-K rank aggregation, whose task is to estimate the of top-K items, in certain regimes (Jang et al. 2017 Chen al.

---

Under review as a conference paper at ICLR 2020
2017 As in I. we define May 1 if > and 0 otherwise, given a pair of individual
items and j. Then, its individual utility update rule is
((+1)
Wi (1)
a
E (V - y)5) (11-)))u )((r))DDo
(3)
(++1) For item i, one can consider w (next to Biy) as the reward because it increases wi w
when > j (yi) = 1), and w ( (next to (1 (1-)k)) )) as the penalty because it decreases w (++1) when < j (Vey 0). One can consider a as a step size the update. Note that the reward is large when the opponent's utility estimate is large, since it can be considered that it has won tough match. Likewise, the penalty is large when its own utility estimate is large, since it has lost in easy match. We can see the reward and penalty mechanisms, and the magnitudes of their influences based on the dynamics between groups. These correspond to observations (a) and (c).
(MM) for the BTL-sum model has been developed in (Hunter 2004 Huang al. 2006 We define " ΣiEA w! Then, its individual utility update rule
1++1) t W
Σ
(BAB: RAB, (1 MAB) P(AB.)
wi
+
(4)
where
(t)
(8) w
R()) AB,
W
W
p(t) AB,
w
(5)
(C)
W
+ WB
+ w
A
Note that the update rule 44) is similar to 3 of Rank Centrality in form, but the reward and penalty terms (5) are different. The interpretation is similar. The reward for item is large when the opponent group's utility estimate is large (wg)) (t) Note another factor: the larger the contribution of item i within its own group (w (.). the greater the reward. The same holds for the penalty. In addition to observations (a) and (c), we can also see the varying reward and penalty terms due to a different underlying model, and the effect of an individual's contribution within its group for rewards and penalties. These correspond to observations (b) and (d).
MM for the BTL-product model has been developed in (Huang et al. 2008) and shown to achieve global minima in terms of cross entropy loss. Its individual utility update rule is described as in (4) but the reward and penalty terms are differen 3 Here, similar interpretation applies again, and we can also see observations (a)-(d) at play.
Our decision to incorporate another module G is as follows. To perform the match prediction task, need not only individual utility estimates, but also in in-group interaction and group comparison models that use such estimates to determine winning probabilities for pairs of groups. The role of G is to fit these underlying models during training. It takes input the individual utility estimates from a pair of groups, which and P help quantify, and predicts as output the probability of one group preferred over the other. The three modules interact as whole perform the task.
4.2 MODULES R AND P
The input and output of modules R (or P) are of dimension 2M (see Figure 2 arbitrary integer greater than unity (M 1). We set value according to the dataset at hand.
M can be an
lAs (egahbaa al. 2016), a 1/dmax where dmxx := max; di and is the number of distinct items to which item is compared. Also, we describe Rank Centrality as an iterative algorithm (one way to obtain the stationary distribution of the empirical pairwise preference matrix) in order to highlight its inherent reward-and aade--nnd-ppllltt penalty mechanisms.
(w)()
as
+ wm
wwth (t)
Illed 10 a;
Σ
w
(0)
R(O) AB,
w
(6)
and p(t)
wil wf+ +w (0
200
ABA
5

---

Under review as a conference paper at ICLR 2020
They take the current utilityy estimates of the individuals in a given group comparison (A, B) as input, and produce the current R (or P) estimates for the individuals as outpu I Note that the input and output dimensions are independent of the total number of items. Hence. work does not suffer from scalability issues in contrast to prior approach (Menke & Martinez, 2008) also based on employing neural networks. To make our algorithm robust against arbi- trary orderings of the items within a group, we apply data augmentation. Given a sample, we create extra samples which represent the same outcome but have different item orderings. For example, given a sample (A = (1, B = (3, 4), VAB 1), we create an extra sample such as (A'' (2, 1) (4, 3), VA'B = 1). We also make our algorithm robust against arbitrary order- ings of the two groups. That is, we create extra samples by changing the order of two sets A and B as well as A' and B'. This technique helps train modules R and
frame-
such a way that they become robust against two kinds of arbitrary orderings: item orderings within a group and group orderings.
All layers are fully connected. The activation functions between two layers are rectified linear units. The final activation function is the sigmoid function whose output ranges between 0 and 1.
Starting with an initial vector (()) E R" (reflecting the current utility estimates), we finally obtain (T) E R by applying R and repeatedly T times. Each iteration is described as follows:
w w
(1+1) t w + a
Σ
(JAB: RAB, (1 - VAB) P.B.)
(6)
(A.B)
wherd a c/maxx d, and d; = ((A, : AUB)
:iE
At the end of each iteration, we transform w(++1) to be zero-mean as in w ((+1) Ei= (1+11) /n, and unity-norm as in (++1) f w, ((+1) ( At iteration t, given w(t))
wi ((+1)
R and P produce positive real values in [0, 1]:
R()) ABB R (i, 1uu5"11e. (ung IEB):
((t) AB, p(( (u IEA
(w IEEB)))))))))))))))
Modules R and P take as input concatenation of two vectors [w]] [tn( jeA and [urg kEg. 8. As they are vectors, not sets, ordering matters. Recall that given a sample, we create multiple additional samples by employing data augmentation techniques. In doing so, we randomly mix the item ordering within group and also the group ordering between the two groups. We preserve the resulting orderings in the created samples for the input to and P. Resorting to the previous example, if the created sample by data augmentation is (AA' = (2, 1), B' = (4, 3), MAvg = 1), the first element of the input (+) vector for R and P concerns W2 the second (") the third wa and the fourth (0)
4.3 MODULE G
The input and output of module G are of dimension 2M and scalar respectively (see Figure 2. As in modules R and P, we set the value of M according to the dataset at hand. Since the dimensions are independent of the total number of items, the module does not suffer from scalability issues. The module takes the final utility estimates of the individuals in a given group comparison (A,, B)) as input (see Footnote 4 and produces the winning probability estimate of one group preferred over the other in the given group comparison as output:
AA G (Iu! JiEA,
(2) IiA, (w] 1EE3))
(8)
Similarly as in module G takes input concatenation of two vectors [u] | The item and group orderings used for R and P are preserved through the input to G.
ljes- Jie and [u]))
All layers are fully connected. The activation functions between two layers are rectified linear units. The final activation function is the sigmoid function whose output ranges between 0 and 1.
"To be accurate, the produced output is mix of the individual utility estimates, resulting from being passed through fully-connected layers. Conceptually speaking, we refer to them simply as individual utility estimates.
Sprior work (see Footnote I motivates the choice of and hyperparameter tuning determines its scaling in the numerator.

---

Under review as a conference paper at ICLR 2020
We now describe training procedure. We first split available data Dabs randomly into Derain and Dval- We let Daal a small fraction (1% of Dobs and use for validation purposes.
Training Procedure.
(I) Initialize w(() randomly using a Gaussian distribution whose mean is 0 and variance is the normalized identity matrix. Also, initialize the parameters of R, P and G using the Xavier initialization (Glorot &
(2) Obtain T through T iterations in each of which modules R and P. and also group comparison samples in Drrain-
(3) Obtain (A,B)
for each group comparison sample in Derain by using w()) obtained in (2) above and module G.
(4) Update the parameters of R, P. and G via the Adam optimizer (Kingma & Ba 2014) to min- imize the cross entropy loss (2) replacing Dobs therein by Derain- We apply weight decay regularization with a factor of 0.01.
For each training epoch, we repeat (2)(4) above. We use 500 epochs, in each of which we calculate validation loss using Dual. We apply early stopping, choosing the model parameters in the epoch with the lowest validation loss. To avoid terminological confusion, we make it clear that we use batch gradient descent. That is, we update the model parameters once at the end of each epoch. Hence, the T iterations in (2) do not mean the number of the model parameters are updated, as in the conventional way. They are our architectural constructs intended obtain well-refined estimates (T"), from which we fit the underlying model.
per epoch, in each of which
EXPERIMENT RESULTS
To verify the broad applicability of our algorithm, we conduct extensive experiments using synthetic and real-world datasets. We compare it with five other algorithms: MM-sum (Huang et al.) 2006), MM-prod (Huang et al., 2008), SGD-HOI (Li et al. 2018), TrueSkill (Herbrich et al. 2007) and Rank Centrality (Negahban et al. 2016). SGD-HOI has been developed for the factorization HOI model in (Li et al. 2018 18) and Skill for the Thurstone model in (Herbrich et al. 2007). As Rank Centrality has been developed for the case of comparing two individual items, consider its natural extensions depending on the dataset. For example, in the sum model, we replace the summation in
by E(A.B)EP (AAB ΣkeB w(( + (1 YAB) EREA w."" (t)
5.1 SYNTHETIC DATA EXPERIMENTS
We four synthetic datasets: BTL-sum, BTL-product, HOI and a generalized Thurstone. In all, we set n = 300 and M = 5. In the HOI model, we generate the ground truth utilities and dimension-7 features using Gaussian distributions. In the others, we generate the ground truth utilities uniformly at random. We generate 5n log distinct paired groups and each pair is compared 10 times.
We split generated datasets randomly into Dobs (90%) and Punobs (10%). All algorithms use Dobs to predict unobserved group comparisons in Dunobs. They use fraction (1 (1%-2%) of Dots for validation purposes if necessary. We use 20 and the learning rate of 10 We use four hidden layers with 7M nodes each for modules R and P, and four hidden layers with 9M nodes each for module G.
As our performance metric, we consider the cross entropy loss in (2) MM-sum and MM-prod have been developed to achieve maximum likelihood, which can be shown to be equivalent to minimizing the cross entropy loss, and SGD-HOI is tailored for the cross entropy loss as it can adopt an arbitrary loss function. It may seem somewhat unfair for Rank Centrality and Trullkill, which have not been developed to minimize the cross entropy loss. In Section 5.2 some of our experiment results real-world datasets compare the algorithms in terms of other metrics, for which none of them are tailored. They include prediction accuracy, the Kendall tau distance, and normalized discounted cumulative gain (see Footnote 10), which may be considered more relevant for practical use.
Figure shows our result. The algorithms that underperform by large gaps not presented.

---

Under review as a conference paper at ICLR 2020
0.75
0.35
5624101
His pivd
3.72
Hel prx)
5CD415
HMt SCD4101
Bark Cansaliy
Durt Cenvelyy
Bank Canculay
Prousses
29.55
165
176
286
36
286
515
[Dwa) Fraticne [Diod)
Uin Trctlons of [Dos]]
[Dus]] fractarn (Diad)
(in frcelorn of
Figure 3: Results of experiments. From left right: BTL-sum model, BTL-product model, HOI model, and a generalized Thurstone model, Off-the-scale curves are not shown. Amount of data used for training described in fractions of Dobs, 90% of the entire dataset for training (x-axis). Performance is obtained by using Dunobs 10% of the entire dataset for testing (y-axis).
BTL-sum model: In most settings where the amount of data samples is sufficient, our algorithm achieves the performance promised by MM-sum, which has been shown in (Huang al. 2006 to achieve local minima in terms of cross entropy loss.
BTL- model: Our algorithm achieves the optimal performance promised by MM-prod, which has been shown in (Huang et al. 2008) to achieve global minima in terms of cross entropy loss.
HOI model: SGD-HOI performs best in most settings. Our algorithm is second-best with slight gap to the best performance in those settings, but performs best when the amount of data samples ample. This is because our algorithm using neural networks is affected by overfitting when the amount of data is insufficient.
Thurstone model: MM-prod and our algorithm perform best. Trul comes next with a gap, but it clearly outperforms the others. It is interesting to observe that Truelkill, which has been developed specifically for the Thurstone model, does not lead to the best performance. However, this result does not run counter to theory. as its optimality has not been shown in the literature.
Our algorithm performs consistently best (or near-best with a negligible gap) across all datasets, while the others perform inconsistently across them. Some perform well in one, but poorly in the others (for example, MM-sum performs best only in the BTL-sum model but poorly in all others).
This result has an important implication. Our algorithm is shown to achieve consistently the best performances, matching those of the state-of-the-arr algorithms specifically developed for the mod- els that underlie the synthetic datasets. This implies that our algorithm can be universally applied to achieve consistently high performances in a wide range of real-world match prediction applications. We corroborate its universality further in the following extensive real-world data experiments.
5.2 REAL- DATA EXPERIMENTS
split real-world datasets randomly into Dobs (90%) and D Dunobs (10%), and use a fraction (%-2%) of Dobs for validation purposes if necessary. We use five different real- world datasets GIFGIF, HOTS, DOTA 2, LoL, IMDb 5000. We use T = 30, 15, 15, 20, 20 and the 10 10 2, 10 respectively. We use four hidden layers with 7M nodes each for modules R and P, and four hidden layers with 9.M nodes each for module G.
As in Section 5.1
learning rates of 10 10-3,10 10
GIFGIF: A crowd-sourcing projec E We use the dataset pre- processed in (Maystre & Grossglauser, 2017 A participant presented with two images and asked to choose one which better describes a given emotior]] This dataset belongs to special case of our interest as individual comparisons are concerned. We consider the emotion of happiness. We have 6, images and 106,886 samples.
HOTS: A collection of HOTS match records from 10/26/17 to 1 11/26/17 collected by HOTS log Each match consists of two groups with five players each. The players choose heroes (characters) for each match out of a pool of 84. We choose high-quality matches only where all players are highly-skilled according to some available statistics. There are 26,486 match records.
"Source: gifgif. media.mit.ed. (GIFGIF): hww.slooos..oo/ntoo///aa (HOTS); kaggle saggle..commeeiinaazeemooooaa 2-matches (DOTA 2); http../.cccccccpppo//eeegcccccee movie-dataset (JMDb 5000).
(Lol.); kaggle. baggg..com/cmm//aantttnnnnnnn-nn
"One is allowed choose "neither" but we exclude such data.
8

---

Under review as a conference paper at ICLR 2020
Table I: Match prediction results in terms of cross entropy loss and prediction accuracy for experi-
ments with five real-world datasets (N/A: non available due to scalability issues)
GIFII
HOYS
DOIN2
LoL
IMDD 2000
CE Loss
ACL
CE Loss
CE Loss
ACe
CE Loss
ACC
CL Loss
Ace
FROPOSED
(13
0.8729 (3)
[L0790
0.5973 (1)
64:59 (1)
0.6091 [2]
0.8036 (1)
0.5411
(1)07711
Musse & MARTINEZ 2008
N/A
N/A
(.6
0.5455 175
0.6001 13)
66127 (1)
0.6937 (2)
0,5255 (5)
N/A
NíA
MM--SUM
0.3114 (3)
018758:0 (1)
0.6795 (4)
0. 0.564510 (4)
0.6610 (4)
0.0049 (5)
0. 0.2033 (3)
0.S165: (7)
1.0719 (6)
0.58 (2)
MM--RR00
0.3126 (4)
(18758:0 (1)
0.6295 (3)
0. 13566011 135
0.6403 (3)
0.6083 (7)
0.7070 (5)
0.
0.5249 (3)
8.7455 (5)
0.5428 (5)
SGD-HOI
0.4056 (6)
0.8614455 (5)
0.6795 (5)
0.5563 (5)
0.6673 [6]
0.5955 (6)
0.7128 (6)
0.5171 (6)
0.9471 (3)
0.5780 (5)
TRUESKELL
0.3063 (2)
0.8728 (4)
0.6857 (7)
0.5499 (6)
0.6683 (7)
0.5863 (7)
0.7055 (4)
0.5245 (4)
0.8409 (2)
0.5736 (4)
RANK CENIKALITY
0.3761 (5)
0.8555 (6)
0.6799 (2)
0.50099123
0.6067
0.0051 (4)
0.7258 (7)
0.5275
3,7455 (5)
0.5096 (6)
DOTA 2: A collection of DOTA 2 match record Each match consists of two groups with five players each, and they choose heroes out of a pool of 113. There are 50,000 match records.
LoL: A collection of LoL professional match record Two groups with five players each compete. The players choose heroes for each match out of pool of 140. There are 7,610 match records.
IMDb 5000: A collection of meta-data for 5,000 movie Each movie has a score and is associated with keywords. To fit our purpose, we generate match records for movie pairs from the collection. We consider each movie as group and its five associated keywords as its items. Given a pair, we declare a win for the one with higher score. We have 8,021 keywords and 123,420 samples.
In addition to the cross entropy loss, we consider another metric relevant in the real-world: prediction accuracy. In real-world data experiments, we measure cross entropy loss and prediction accuracy. We declare it win for a group if the estimate of its winning probability is above certain threshold. which we set as 0.5 (Delaleeuu et al. 2012). Thus, the prediction accuracy is expressed as follows
VAST2o.5( + (1 - To.ss)o).
Dunobs
(A.B)
The rationale behind using both cross entropy loss and prediction accuracy that they serve as com- plementary metrics. To see this, let us consider toy example. Suppose we have three group compar- isons: (A, (B,C) and (C,A). The ground-truth -ruth assumed (y,A8, yBC: (C.A) (0.6, 0.7, 0.8). Consider two algorithms: Algorithm I estimates (DAB. Dzc,, (CAA) (0.55, 0.6, 0.7), and Algo- rithm 2 yields (fAB, UBC, (c.A)= = (0.45, 0.7, 0.8). Using ② and (9) we can check that Algorithm achieves a cross entropy loss of 0.6122 and a prediction accuracy of 0.7, and Algorithm 2 achieves 0.6098 and 0.6333. Note that better cross entropy losses do not necessarily translate into better pre- diction accuracies, and vice versa. This is because the cross entropy loss measures the closeness be- tween the estimation and the ground-truth, whereas the prediction accuracy measures the frequency which the estimation predicts the same group being more preferable as the ground-truth.
We also conduct an ablation study where we consider an algorithm based on a single-layer neural network (Menke & Martinez, 2008). It has scalability problem, however, as it requires at least one input node per item. This prevents us from measuring its performance in our setur for some datasets with a large number of items such as GIFGIF and IMDb 5000.
19
Table D shows our result. The best performances are boldfaced, and the second-best are underlined. The numbers in the parentheses indicate the ranks among the algorithms being compared in a given setup of dataset and metric. Our algorithm consistently yields the top performances in all cases. In contrast, the state-of-the-art algorithms suffer from inconsistent performances across different datasets and/or metrics.
Here lies the value of our algorithm. In practice, it is difficult to choose which algorithm to deploy since we do not know the underlying model for a given scenario without expert domain knowledge. Even when we have a reasonably accurate estimate of the underlying model, multiple algorithms known to perform equally well in one model can lead to noticeably different performances, making it difficult to choose one given multiple alternatives. This is demonstrated in the GIFGIF case where MM and Rank Centrality, known to perform well in the BTL model, show different performances. More importantly, models in a variety of real-world applications can potentially be so complicated that all existing algorithms tailored to specific models may perform poorly. As demonstrated in the extensive real-world data experiments, our algorithm has the potential to be universally applicable.
*We define T (z) as 1 if x 2 0.5 and as 0 otherwise. (2) equals to if x < 0.5 and to O otherwise. Intel Core 17-6850K @ 3.6GHz (CPU) and GeForce GTX 1080 Ti (Single GPU).

---

Under review as a conference paper at ICLR 2020
Table 2: Rank aggregation results in terms of Kendall tau distance and normalized discounted cu- mulative gain for experiments with IMDb 5000 movie ratings data
RANK CENTRALITY
PROPOSED
MM-SUM
MM-PROD
SGD-HOI
TRUESKILL
KENDALL TAU DISTANCE
0.4065
0.4171
0.4727
0.4727
0,4285
0.4882
N
0.3933
0.3348
0.2555
0.2555
0.2932
0.1220
Extension to Rank Aggregation. We have so far focused on the match prediction problem. How- ever, we believe that our algorithm can be easily extended for other tasks as well. In an attempt to demonstrate its potential flexibility, we present preliminary results in rank aggregation tasks.
In rank aggregation tasks, one seeks to rank all items order of utility (Rajkumar Agarwal 2014 Agarwal et al. 2018), or to identify only few top-ranked items (Chen & Suh] 2015 Chen et al., 2018a) as in top-K ranking. In both tasks, one needs to obtain a collection of individual utility estimates inferred from comparison data, which is close to the ground truth that one postulates.
We use the IMDb 5000 dataset as it comes with IMDb movie scores which can be regarded as the ground-truth. The other algorithms produce individual utility estimates and use them to compute group utility estimates based on the models they assume. In extending our algorithm, we let R, P and G stay the same. We use its final winning probabilities for pairs of groups to compute group utility estimates. In doing so, we adopt associated scores proposed in (Shah & Wainwright/ (015)) group's score is the probability that it is preferred over another group chosen uniformly random.
We measure the performance terms of two well-known metrics T0 the Kendall tau distance (Kendall, 1938) and normalized discounted cumulative gain (NDCG@ K) (Järvelin & Kekäläinen 2002). Table shows our result. It turns out that our algorithm performs best in both metrics. This result suggests that with slight adjustments of our framework to fit the purpose, it can potentially lead to satisfactory performances for other related tasks as well.
6 CONCLUSION AND FUTURE WORK
We investigate the match prediction problem where the task is to predict the preference of one group over the other given an unseen pair of groups based on the observed group comparison data. Facing with real-world challenges that underlying models that govern in-group interactions and group comparisons are unknown and complex, we develop an algorithm that employs deep neural networks to infer such latent models from data specific to a given application. As a result, we show that our algorithm can show consistently best prediction performances compared to other state-of- the-art algorithms on multiple datasets across various domains. We also demonstrate that it can be applicd to the rank aggregation task, which implics its potentially broader application to other tasks.
In view of it, we consider the following task as possible direction for future work. The task is to predict whether multiple items that constitute a group would make an effective combination producing positive synergies, and thus lead to desired outcome. Bundling strategies in e-commerce can be a real-world example: multiple items are bundled as a package and offered to the potential buyer with a discount. The goal is to figure out which of items would appeal most to the buyer given past sales data. We expect that our current architecture can be extended to this task. Among a number of items, some will contribute positively to the group (rewards) and some negatively (penalties). Our modules R and P can be applied to measure them. Our module G can be applied to govern how these rewards and penalties manifest collectively as group outcome. We also expect that our work can appear frequently in other tasks as well where in- in-group interactions are critically concerned, but their statistical patterns are unknown practice.
10 Kendall tau distance is defined as [((i.)): i < j. (n < ( A 72 > Tz(i)) V (1) > ( A T2()) < r2()))H where TI (i) and (1) are the rankings of iitm in and r2. In words, it counts the number of item pairs that ranked reversely in the two rankings. In NDCG, items associated with relevance scores. In case, items ranked higher in the ground-truth ranking have higher scores. Let rel, be the score of the item ranked i-th in ranking. NDCG discounts rel: by log, (i + 1) to "penalyze" the quality a ranking for placing high-relevance item low rank. NDCG@ K is normalized DCG@ K defined as SK rel:/logz(t (i 1).
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
A. Agarwal, P. Patil, and S. Agarwal. Accelerated spectral ranking. In International Conference on Machine Learning, pp. 70-79, 2018.
R. A. Bradley and M. Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3-)):3243455 1952.
C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning, pp. 39-96. ACM, 2005.
X. Chen, S. Gopi, J. Mao, and Schneider. Optimal instance adaptive algorithm for the top-K ranking problem. IEEE Transactions on Information Theory, 64(9):6139-611 6160, 2018a.
Y. Chen and C. Suh. Spectral MLE: Top-K rank aggregation from pairwise comparisons. In Inter- national Conference on Machine Learning, pp. 371-380, 2015.
Y. Chen, Fan, C. Ma, and K. Wang. Spectral method and regularized MLE are both optimal for top-K K ranking. arXiv preprint arXiv: 2017.
Z. Chen, X. Li, and J. Bruna. Supervised community detection with line graph neural networks. arXiv preprint haff::7705.8000 2018b.
O. Delalleau, E. Contal, E. Sabbodeau-Laufees R. C. Ferrari, Y. Bengio, and Zhang. Beyond skill rating: Advanced matchmaking in ghost recon online. IEEE Transactions Computational Intelligence and Al in Games, 4(3):677-777, 2012.
C. DeLong, N. Pathak, K. Erickson, E. Perrino, K. Shim, and J. Srivastava. Teamskill: Modeling team chemistry online multi-player games. In Pacific-Asia Conference Knowledge Discov- ery and Data Mining, pp. 519-531- Springer, 2011.
X. Glorot and Y Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Machine Learning, pp. 249-256, 2010.
J. Guo, Fan, Q. Ai, and W. B. Croft. A deep relevance matching model for ad-hoc retrieval. In ACM International on Conference on Information and Knowledge Management, pp. 55-64. ACM, 2016.
R. Herbrich, T. Minka, and T. Graepel. Trueskil[ A bayesian skill rating system. In Advances in neural information processing systems, pp. 569-576, 2007.
Huang, C. J. Lin, and R. C. Weng. Generalized Bradley- models and multi-class proba- bility estimates. Journal of Machine Learning Research, (JJa)):55-115, 2006.
T. K. Huang, C. J. Lin, and R. C. Weng. Ranking individuals by group comparisons. Journal of Machine Learning Research, 00c)):21877) 2008.
D. R. Hunter. MM algorithms for generalized Bradlcy-Terry models. Annals of Statistics, pp. 384- 406, 2004.
M. Jang, S. Kim, C. Suh, and Oh. Optimal sample complexity of M-wise data for top-K ranking. In Advances in Neural Information Processing Systems, pp. 1686-1696, 2017.
K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Transac- tions on Information Systems, 20(4):422 446, 2002.
M. G. Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81 1938.
D. Kingma and J. Ba. 1rXiv:1412.6980, 2014.
Adam: method for stochastic optimization. arXiv preprint
Y. Li, M. Cheng, K. Fujii, F. Hsieh, and Cho-Jui Hsich. Learning from group comparisons: Ex- ploiting higher order interactions. In Advances in Neural Information Processing Systems, pp- 4981-4990, 2018.
11

---

Under review as a conference paper at ICLR 2020
L. Maystre and M. Grossglauser. Just sort it! A simple and effective approach to active preference learning. International Conference on Machine Learning, pp. 2344-24442222 2353,, 2017.
J. E. Menke and T. R. Martinez. A Bradley-Terry artificial neural network model for individual ratings in group competitions. Neural computing and Applications, 17(2):175-186, 2008
Negahban, S. Oh, and D. Shah. Rank centrality: Ranking from pair-wise comparisons. Operations Research, 65:266-2880 287,, 2016.
A. Rajkumar and S. Agarwal. A statistical convergence perspective of algorithms for rank aggrega- tion from pairwise data. In International Conference on Machine Learning, pp. 118-126, 2014.
M. Richardson, A. Prakash, and E. Brill. Beyond PageRank: Machine learning for static ranking. In International conference World Wide Web, pp. 707-715, ACM, 2006.
Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20( (0(())661880) 2009.
M. Schlichtkrull, T. N. Kipf, P. Bloem, R. Van den Berg. I. Titov, and M. Welling, Modeling relational data with graph convolutional networks. European Semantic Web Conference, pp. 593-607. Springer, 2018.
N. B. Shah and M. J. Wainwright. Simple, robust and optimal ranking from pairwise comparisons. arXiv preprint 1rX19.151..08949, 2015.
12


Reviews:
[{"review_id": "r1xPqmwoqr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a novel architecture to tackle the match prediction problem.  There are two/three modules in the architecture, the R/P modules and the G module. R/P modules take the current utility estimates of the individuals in a given group comparison as input and produce the current R/P estimates for the individuals as output. The G module takes the final utility estimates of the individuals in a given group comparison as input and produces the winning probability estimate of one group preferred over the other in the given group comparison as output.\n\nI would recommend a weak accept for this paper based on the following reasons:\n* Both the R/P and G modules' input and output dimensions are independent of the number of items. This keeps the architecture scalable when facing large data sets in the real world.\n* The empirical result looks satisfactory on several data sets across different domains.\n* The theoretical foundation is sound.\n\nI would hope that the authors will make some effort in making the paper more approachable and  practical:\n* A more detailed motivation section for the architecture will make it much easier for the readers to understand.\n* A conclusion plus future work section could come in handy for future researchers.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SJg58BU6Fr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a technique to solve match prediction problem -- the problem of estimating likelihood of preference between a pair of M-sized sets. The paper replaces the previously proposed conventional statistical models with a deep learning architecture and achieve superior performance than some of the baselines. The experiments show the efficacy of the proposed methods.\n\nI have some major concerns with this paper. These are described below:\n\n1.  The paper presentation is not very clear. The paper contains many imprecise statements. The problem is not setup very well. \n(a) The abstract and the introduction  contains vague statements with no proper description of what the technique is about. \n(b) In introduction, the authors start with discussing 1-sized pairwise comparisons, and then suddenly are discussing in-group items effects at the end of the third paragraph, so that means they are talking about M-sized comparisons with M > 1. This adds to the confusion.\n(c) The same things holds true for the first paragraph in \"Main Contribution\". Since the authors are using deep learning frameworks, this does not mean the authors \"can infer the underlying models accurately.\" I am not sure what the authors wanted to convey with this statement. \n(d) The reason I am being very stringent with impreciseness of the statements is majorly after reading the first statement in the Motivation section. It says, \"Our decision to incorporate two modules R and P into our architecture has been inspired by SOME state-of-the-art algorithms developed under CERTAIN statistical models, which have been shown\ntherein to be OPTIMAL.\" Please avoid using some, certain, optimal when they are not defined properly in the paper yet.\n \n 2. The paper mentions multiple times that it does not use statistical models tailored for a dataset or application but instead use deep neural networks. In my opinion, NN is just another form of statistical model that captures statistical patterns of comparisons and in-group interaction.  They are not following the same modeling assumptions as others but have created their own in some sense. The authors may want to rephrase those statements.\n\n3. The evaluation metric for the datasets that author consider is the prediction accuracy. I am not sure why the authors evaluate on cross entropy as well in Table 1, since both are closely related (CE is a consistent surrogate of accuracy). Can you please explain? However, I liked that they compared the methods with other metrics in Table 2. \n\n4. In problem setup, please mention M > 1. For M = 1, the problem is similar to [1] and many solutions have been proposed for that. \n\n5. I am not sure how the R and P modules capture what the authors want them to capture. I would suggest the authors to include that when they first discuss R and P modules.\n\n6. In equations 7, it is not clear how the function R(.,.,.,) and P(.,.,.) defined? Is it similar to what is described in equation 5?\n\n7. The training procedure contains the standard details; however, it is not clear to me how the modules R, P, and G interact during training.\n\nOverall, I believe the paper has a descent idea and contains satisfactory experimental results; however, the presentation of the paper is very weak at this moment. \n\n[1] Joachims, Thorsten. \"Optimizing search engines using clickthrough data.\" Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2002.\n\n---- After Rebuttal ---\n\nI thank the authors for providing response to my questions and making edits to the paper. My clarity on boxes R and G has become better; however, I am still not totally convinced. Also, the presentation of the paper could be further improved. Therefore, I would keep the same score.  ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1xrIzbC5H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper attempts to solve match prediction problem, i.e., whether a group is preferred over the other. The key challenge is \"consistency\" since it's hard to find the universal pattern over tasks. Instead, this paper propose to learn reward and penalty modules and both vary when the underlying model changes. Experiment results show that the proposed method consistently works the best. \n\nMy comments:\n[1] The paper is well written\n[2] This paper tries to solve an interesting problem but the application is a bit limited\n[3] It would be great to conduct ablation study, e.g., analyze efforts of different components (R, P, G)\n\n\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "ByeF2bR9YB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a deep neural network solution to the set ranking problem. The authors design a special architecture for this task inspired by previous manually designed algorithms. The authors show empirical superior performance. \n\nThe idea seems potentially interesting. I think if the authors can convincingly show that they incorporate the inductive bias of previous manually designed algorithms, and sprinkle in some trainable parameters and optimization, this could be a good paper. However, I have several concerns about the paper. \n\nI am confused about the argument about scalability. Given the input sets A and B, how are the “current utility” computed. Some function from a set to a fixed dimensional vector is still necessary. How is this computed? In Eq(7) it seems that the input to P is a set, but this contradicts what is claimed (it is a fixed length vector?). \n\nIn Eq.(6) it seems that only one of R and P is multiplied by a non-zero coefficient. Does this basically mean we use R network when label is 0 and P network when label is 1. How is this design choice justified? \n\nIn the synthetic experiments, a cross entropy loss of 0.5+ seem very bad for binary classification. For example if the model just predicts p=0.5 (random guess) the loss should be better. It seems that the synthetic data labels are generated randomly. These dataset decisions should be better explained and justified. I think deriving the conclusion “This implies that our algorithm can be universally applied to achieve consistently high performances in a wide range of real-world match prediction applications.” from four synthetic datasets seem far-fetched. \n\nFor the real world experiments, the method seems to perform marginal better on average. I think one small issue I have is as before, these accuracies (and cross entropy) are not much better than random guess, and sometimes worse. Also note that the baselines are not deep neural networks, so do not leverage the capability of automatic differentiation and optimization. \n\nThe authors’ adaptation for Rank Centrality involves summing up the weights; this does not model their interaction and seem to be a weak baseline. \n\nWriting: I had some difficulty following the paper. \n\nI was confused about the notation. I didn’t find the definition for many symbols such as the ones in Figure 2. \n\nA very large fraction of the paper (2 pages) describe existing work in detail, and explain which component of the prior work is manifested (philosophically) in the current work. I think this is unnecessarily cumbersome because the current work seems to be a natural instantiation of a modern neural network to this task. Alternatively if the authors can explain the connection more concretely (i.e some parameters learned by the neural network recovers models in prior work) the arguments can be convincing.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates neural networks for group comparison -- i.e., deciding if one group of objects would be preferred over another. The paper received 4 reviews (we requested an emergency review because of a late review that eventually did arrive). R1 recommends Weak Reject, based primarily on unclear presentation, missing details, and concerns about experiments. R2 recommends Reject, also based on concerns about writing, unclear notation, weak baselines, and unclear technical details. In a short review, R3 recommends Weak Accept and suggests some additional experiments, but also indicates that their familiarity with this area is not strong. R4 also recommends Weak Accept and suggests some clarifications in the writing (e.g. additional motivation future work). The authors submitted a response and revision that addresses many of these concerns. Given the split decision, the AC also read the paper; while we see that it has significant merit, we agree with R1 and R2's concerns, and feel the paper needs another round of peer review to address the remaining concerns.
Decision: Reject