Match prediction from group comparison data using neural networks

1 INTRODUCTION . The most elementary form of comparisons is pairwise : we often compare a pair of items and make judgments as to which one is of higher utility or simply preferable over the other . With a large amount of such comparison data , one can consider various interesting tasks . One may wish to predict future outcomes of unseen matches , and also to rank alternatives in order of utility or preference . Challenges arise in carrying out these tasks . Almost all existing state-of-the-art algorithms have been developed under the assumption that given a scenario , there exists a certain underlying model which governs statistical patterns of comparison data ( see Section 2 for details ) . As such , we have different best-performing algorithms across distinct scenarios . This traditional approach , which begins by assuming certain models to develop algorithms , comes with limitations in practice . First , it gives rise to inconsistent performances . No single algorithm can perform consistently well in a wide range of scenarios , since it has been tailored to a specific model . Second , it is hard to know the underlying model without expert domain knowledge . In its absence , we have little choice but to find an appropriate algorithm via trial-and-error . Third , the model can be inherently complex for any existing algorithm to be effective . Sometimes groups of items are compared , thus the effects of interactions among in-group items come into play , further complicating the model . In this work , we propose a unified algorithmic framework aimed to overcome these barriers . We focus on the match prediction problem where one wishes to estimate the likelihood of a group of M items preferred over another , based on partially observed group comparison data among a collection of n items . One can imagine that such group comparison data may bear complex statistical patterns due to a combination of two underlying models : the interaction model which governs the effects of in-group interactions in determining the utility or preference of a group ; and the comparison model which governs the statistical patterns of pairwise group comparison data . Hence , designing a novel framework hinges heavily upon accurate inference of these underlying models . Main contribution . We incorporate deep learning techniques into our framework design . This enables us to infer the underlying models from real-world data obtained from a given application , and thus to achieve consistently high performances on a variety of datasets from diverse real-world applications where match prediction tasks are of interest . To this end , we build on progress made through analysis in well-defined statistical models . We gain insights instrumental to the progress by looking into existing state-of-the-art algorithms in related and long-studied tasks such as rank aggregation ( Negahban et al. , 2016 ; Hunter , 2004 ; Huang et al. , 2006 ; 2008 ) . We find that most of them share a key element . They all exhibit so-called reward-andpenalty mechanisms in estimating the utilities of individual items . To be more specific , they reward an item more greatly for winning ( or being more preferred ) in a disadvantageous comparison where its group is weaker than the counterpart . Likewise , they penalize it more greatly for losing ( or being less preferred ) in an advantageous one . In addition , the magnitudes of rewards and penalties are proportional to the contribution of the individual item to its group . This structural similarity across the state-of-the-art algorithms has attracted our attention . Through some manipulation , we find that they all employ the same basic rule for estimating individual utilities ( see ( 6 ) in Section 4 for details ) . The terms corresponding to rewards and penalties turn out to vary as either one of the two underlying models changes . This observation has inspired us to incorporate neural networks into our framework design . The novelty of our design is salient in an ablation study where we compare it with a simple design . As an initial effort , a single-layer neural network has been employed to predict winning probabilities of unseen group matches ( Menke & Martinez , 2008 ) . It has shown a promising result , demonstrating prediction accuracy to be improved on a real-world online game dataset , but also exhibited a scalability issue . It requires one input node per item , making it prohibitive to be extended to realworld applications with a large number of items . Leveraging more advanced architectures ( see Figures 1 and 2 ) motivated by observant analysis as emphasized , our design not only addresses such a scalability issue by design , but also outperforms the single-layer neural network . The merits of our design are evaluated against the single-layer neural network and other state-of-the-art algorithms through extensive experiments on a variety of synthetic and real-world datasets ( see Section 5 ) . Using synthetic datasets , we demonstrate that our approach can achieve the performances of the state-of-the-art algorithms in the models for which they have been specifically developed . We investigate four models . Three consider various extensions of the Bradley-Terry-Luce model ( Bradley & Terry , 1952 ) to the group comparison scenario . The other is a generalized version of the Thurstone model ( Herbrich et al. , 2007 ) widely used in skill rating systems of online games . As a result , we show that our framework consistently yields the best performances across all of these datasets ( nearbest in some cases ) , while the other state-of-the-art algorithms suffer from inconsistent performances across different models . Using real-world datasets , we also demonstrate that our framework performs consistently well across diverse real-world applications . We investigate five real-world datasets ( sources in Footnote 6 ) . One is a crowd-sourced image classification dataset , another is a collection of movie ratings , and the other three are match records from online games . We consider , in addition to the cross entropy loss , the prediction accuracy as another metric ( defined in ( 9 ) ) . As a result , we show that our framework consistently yields almost the best performances across all of these datasets in terms of both metrics . We also show that our framework can be easily extended to achieve the best performance in rank aggregation tasks where one seeks to rank items in order of utility or preference . Using a realworld dataset of movie ratings , we demonstrate that our framework yields the best performances in terms of two well-known metrics ( see Footnote 10 ) : Kendall tau distance ( Kendall , 1938 ) and normalized discounted cumulative gain ( Järvelin & Kekäläinen , 2002 ) . This result suggests that it can potentially be adaptable for other tasks as well . 2 RELATED WORK . The most related prior works are ( Huang et al. , 2006 ; 2008 ; Li et al. , 2018 ; Herbrich et al. , 2007 ) where plausible statistical models ( some long-established and some widely used in practice ) for in- group interactions and group comparisons are assumed , and statistical analysis is carried out to a great extent . We use the algorithms developed in these models and their variants for main baselines . The problem of estimating individual utilities from group comparison data has been investigated in ( Huang et al. , 2006 ; 2008 ) . They considered extensions of the BTL model where the group utility is either the sum or the product of individual utilities , and group comparison data follow the BTL model in terms of two group utilities ( which we call the BTL-sum and BTL-product models respectively ) . A more advanced in-group interaction model has been explored in ( Li et al. , 2018 ) . They considered a scenario where a pair of individuals in a group leads to a synergy , which contributes to the group . The group utility is represented as the sum of two quantities ( the HOI model ) : ( 1 ) the sum of individual utilities and ( 2 ) the sum of the products of all pairs of individual utilities . A general scenario , where any k-tuple of individuals in a group leads to a synergy , has been considered in ( DeLong et al. , 2011 ) . In ( Herbrich et al. , 2007 ) , they assumed individual utilities to be centered around a mean following a Gaussian distribution and viewed the group utility as their sum ( the Thurstone model ) . The algorithm therein is widely used in skill rating systems of online games where groups of users compete . Employing neural networks has been considered as an initial effort to predict winning probabilities of unseen group matches ( Menke & Martinez , 2008 ) . It has been shown that a single-layer neural network can fit some variants of the BTL model ( Huang et al. , 2008 ) and improve prediction accuracy through experiments on a real-world online game dataset . Some other works have also employed neural networks to exploit hidden models , for example , in information retrieval ( Burges et al. , 2005 ; Richardson et al. , 2006 ; Guo et al. , 2016 ) and community detection ( Chen et al. , 2018b ) in graph domains ( Scarselli et al. , 2009 ; Schlichtkrull et al. , 2018 ) . 3 PROBLEM SETUP . We investigate the match prediction problem where we seek to predict comparison outcomes for unobserved pairs of groups given a collection of pairwise group comparison data . We consider the setting where each group consists of M individual items . We are given comparison observations between two groups of size M . To be more specific , each comparison consists of ( A , B , yAB ) . A and B are groups of M individuals , respectively . yAB indicates which group wins ( or is preferred ) : yAB = { 1 if A B ; 0 otherwise , ( 1 ) where A B indicates that group A wins over group B . We denote by Dobs the set of observed comparisons , so all information we are given is { ( A , B , yAB ) } ( A , B ) ∈Dobs . The unobserved set of comparisons , which we wish to predict , is denoted by Dunobs . We consider the cross entropy loss as our metric , as it serves to quantify the discrepancy between two probability distributions . We define ŷAB as the estimate of Pr [ yAB = 1 ] . Our goal is to minimize −1 |Dobs| ∑ ( A , B ) ∈Dobs yAB log ŷAB + ( 1− yAB ) log ( 1− ŷAB ) . ( 2 ) We develop our algorithm primarily based on the cross entropy loss , but for evaluation purposes , we also consider other metrics , such as prediction accuracy , the Kendall tau distance , and normalized discounted cumulative gain ( see Footnote 10 ) . Notation . We denote by [ n ] = { 1 , 2 , . . . , n } the set of all individual items . We use lowercase letters such as i and j to represent individual items , and caligraphic letters such as A and B to represent sets of items . We denote by { wi } i∈A the set of wi ’ s for all i ∈ A . We denote by [ wi ] i∈A a vector of wi ’ s for all i ∈ A and its ordering information is provided in the context . We denote by ŷ an estimate of y . We use subscripts as in yAB when y concerns a comparison between A and B . We use superscripts as in w ( t ) when w is updated iteratively . We use boldsymbols as in w to represent the set of wi ’ s for all i ∈ [ n ] .

Summary: This paper proposed a novel architecture to tackle the match prediction problem.  There are two/three modules in the architecture, the R/P modules and the G module. R/P modules take the current utility estimates of the individuals in a given group comparison as input and produce the current R/P estimates for the individuals as output. The G module takes the final utility estimates of the individuals in a given group comparison as input and produces the winning probability estimate of one group preferred over the other in the given group comparison as output.