Match prediction from group comparison data using neural networks

1 INTRODUCTION . The most elementary form of comparisons is pairwise : we often compare a pair of items and make judgments as to which one is of higher utility or simply preferable over the other . With a large amount of such comparison data , one can consider various interesting tasks . One may wish to predict future outcomes of unseen matches , and also to rank alternatives in order of utility or preference . Challenges arise in carrying out these tasks . Almost all existing state-of-the-art algorithms have been developed under the assumption that given a scenario , there exists a certain underlying model which governs statistical patterns of comparison data ( see Section 2 for details ) . As such , we have different best-performing algorithms across distinct scenarios . This traditional approach , which begins by assuming certain models to develop algorithms , comes with limitations in practice . First , it gives rise to inconsistent performances . No single algorithm can perform consistently well in a wide range of scenarios , since it has been tailored to a specific model . Second , it is hard to know the underlying model without expert domain knowledge . In its absence , we have little choice but to find an appropriate algorithm via trial-and-error . Third , the model can be inherently complex for any existing algorithm to be effective . Sometimes groups of items are compared , thus the effects of interactions among in-group items come into play , further complicating the model . In this work , we propose a unified algorithmic framework aimed to overcome these barriers . We focus on the match prediction problem where one wishes to estimate the likelihood of a group of M items preferred over another , based on partially observed group comparison data among a collection of n items . One can imagine that such group comparison data may bear complex statistical patterns due to a combination of two underlying models : the interaction model which governs the effects of in-group interactions in determining the utility or preference of a group ; and the comparison model which governs the statistical patterns of pairwise group comparison data . Hence , designing a novel framework hinges heavily upon accurate inference of these underlying models . Main contribution . We incorporate deep learning techniques into our framework design . This enables us to infer the underlying models from real-world data obtained from a given application , and thus to achieve consistently high performances on a variety of datasets from diverse real-world applications where match prediction tasks are of interest . To this end , we build on progress made through analysis in well-defined statistical models . We gain insights instrumental to the progress by looking into existing state-of-the-art algorithms in related and long-studied tasks such as rank aggregation ( Negahban et al. , 2016 ; Hunter , 2004 ; Huang et al. , 2006 ; 2008 ) . We find that most of them share a key element . They all exhibit so-called reward-andpenalty mechanisms in estimating the utilities of individual items . To be more specific , they reward an item more greatly for winning ( or being more preferred ) in a disadvantageous comparison where its group is weaker than the counterpart . Likewise , they penalize it more greatly for losing ( or being less preferred ) in an advantageous one . In addition , the magnitudes of rewards and penalties are proportional to the contribution of the individual item to its group . This structural similarity across the state-of-the-art algorithms has attracted our attention . Through some manipulation , we find that they all employ the same basic rule for estimating individual utilities ( see ( 6 ) in Section 4 for details ) . The terms corresponding to rewards and penalties turn out to vary as either one of the two underlying models changes . This observation has inspired us to incorporate neural networks into our framework design . The novelty of our design is salient in an ablation study where we compare it with a simple design . As an initial effort , a single-layer neural network has been employed to predict winning probabilities of unseen group matches ( Menke & Martinez , 2008 ) . It has shown a promising result , demonstrating prediction accuracy to be improved on a real-world online game dataset , but also exhibited a scalability issue . It requires one input node per item , making it prohibitive to be extended to realworld applications with a large number of items . Leveraging more advanced architectures ( see Figures 1 and 2 ) motivated by observant analysis as emphasized , our design not only addresses such a scalability issue by design , but also outperforms the single-layer neural network . The merits of our design are evaluated against the single-layer neural network and other state-of-the-art algorithms through extensive experiments on a variety of synthetic and real-world datasets ( see Section 5 ) . Using synthetic datasets , we demonstrate that our approach can achieve the performances of the state-of-the-art algorithms in the models for which they have been specifically developed . We investigate four models . Three consider various extensions of the Bradley-Terry-Luce model ( Bradley & Terry , 1952 ) to the group comparison scenario . The other is a generalized version of the Thurstone model ( Herbrich et al. , 2007 ) widely used in skill rating systems of online games . As a result , we show that our framework consistently yields the best performances across all of these datasets ( nearbest in some cases ) , while the other state-of-the-art algorithms suffer from inconsistent performances across different models . Using real-world datasets , we also demonstrate that our framework performs consistently well across diverse real-world applications . We investigate five real-world datasets ( sources in Footnote 6 ) . One is a crowd-sourced image classification dataset , another is a collection of movie ratings , and the other three are match records from online games . We consider , in addition to the cross entropy loss , the prediction accuracy as another metric ( defined in ( 9 ) ) . As a result , we show that our framework consistently yields almost the best performances across all of these datasets in terms of both metrics . We also show that our framework can be easily extended to achieve the best performance in rank aggregation tasks where one seeks to rank items in order of utility or preference . Using a realworld dataset of movie ratings , we demonstrate that our framework yields the best performances in terms of two well-known metrics ( see Footnote 10 ) : Kendall tau distance ( Kendall , 1938 ) and normalized discounted cumulative gain ( Järvelin & Kekäläinen , 2002 ) . This result suggests that it can potentially be adaptable for other tasks as well . 2 RELATED WORK . The most related prior works are ( Huang et al. , 2006 ; 2008 ; Li et al. , 2018 ; Herbrich et al. , 2007 ) where plausible statistical models ( some long-established and some widely used in practice ) for in- group interactions and group comparisons are assumed , and statistical analysis is carried out to a great extent . We use the algorithms developed in these models and their variants for main baselines . The problem of estimating individual utilities from group comparison data has been investigated in ( Huang et al. , 2006 ; 2008 ) . They considered extensions of the BTL model where the group utility is either the sum or the product of individual utilities , and group comparison data follow the BTL model in terms of two group utilities ( which we call the BTL-sum and BTL-product models respectively ) . A more advanced in-group interaction model has been explored in ( Li et al. , 2018 ) . They considered a scenario where a pair of individuals in a group leads to a synergy , which contributes to the group . The group utility is represented as the sum of two quantities ( the HOI model ) : ( 1 ) the sum of individual utilities and ( 2 ) the sum of the products of all pairs of individual utilities . A general scenario , where any k-tuple of individuals in a group leads to a synergy , has been considered in ( DeLong et al. , 2011 ) . In ( Herbrich et al. , 2007 ) , they assumed individual utilities to be centered around a mean following a Gaussian distribution and viewed the group utility as their sum ( the Thurstone model ) . The algorithm therein is widely used in skill rating systems of online games where groups of users compete . Employing neural networks has been considered as an initial effort to predict winning probabilities of unseen group matches ( Menke & Martinez , 2008 ) . It has been shown that a single-layer neural network can fit some variants of the BTL model ( Huang et al. , 2008 ) and improve prediction accuracy through experiments on a real-world online game dataset . Some other works have also employed neural networks to exploit hidden models , for example , in information retrieval ( Burges et al. , 2005 ; Richardson et al. , 2006 ; Guo et al. , 2016 ) and community detection ( Chen et al. , 2018b ) in graph domains ( Scarselli et al. , 2009 ; Schlichtkrull et al. , 2018 ) . 3 PROBLEM SETUP . We investigate the match prediction problem where we seek to predict comparison outcomes for unobserved pairs of groups given a collection of pairwise group comparison data . We consider the setting where each group consists of M individual items . We are given comparison observations between two groups of size M . To be more specific , each comparison consists of ( A , B , yAB ) . A and B are groups of M individuals , respectively . yAB indicates which group wins ( or is preferred ) : yAB = { 1 if A B ; 0 otherwise , ( 1 ) where A B indicates that group A wins over group B . We denote by Dobs the set of observed comparisons , so all information we are given is { ( A , B , yAB ) } ( A , B ) ∈Dobs . The unobserved set of comparisons , which we wish to predict , is denoted by Dunobs . We consider the cross entropy loss as our metric , as it serves to quantify the discrepancy between two probability distributions . We define ŷAB as the estimate of Pr [ yAB = 1 ] . Our goal is to minimize −1 |Dobs| ∑ ( A , B ) ∈Dobs yAB log ŷAB + ( 1− yAB ) log ( 1− ŷAB ) . ( 2 ) We develop our algorithm primarily based on the cross entropy loss , but for evaluation purposes , we also consider other metrics , such as prediction accuracy , the Kendall tau distance , and normalized discounted cumulative gain ( see Footnote 10 ) . Notation . We denote by [ n ] = { 1 , 2 , . . . , n } the set of all individual items . We use lowercase letters such as i and j to represent individual items , and caligraphic letters such as A and B to represent sets of items . We denote by { wi } i∈A the set of wi ’ s for all i ∈ A . We denote by [ wi ] i∈A a vector of wi ’ s for all i ∈ A and its ordering information is provided in the context . We denote by ŷ an estimate of y . We use subscripts as in yAB when y concerns a comparison between A and B . We use superscripts as in w ( t ) when w is updated iteratively . We use boldsymbols as in w to represent the set of wi ’ s for all i ∈ [ n ] .

Summary: This paper attempts to solve match prediction problem, i.e., whether a group is preferred over the other. The key challenge is "consistency" since it's hard to find the universal pattern over tasks. Instead, this paper propose to learn reward and penalty modules and both vary when the underlying model changes. Experiment results show that the proposed method consistently works the best.