Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper attempts to solve match prediction problem, i.e., whether a group is preferred over the other. The key challenge is "consistency" since it's hard to find the universal pattern over tasks. Instead, this paper propose to learn reward and penalty modules and both vary when the underlying model changes. Experiment results show that the proposed method consistently works the best. 

My comments:
[1] The paper is well written
[2] This paper tries to solve an interesting problem but the application is a bit limited
[3] It would be great to conduct ablation study, e.g., analyze efforts of different components (R, P, G)




review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposed a novel architecture to tackle the match prediction problem.  There are two/three modules in the architecture, the R/P modules and the G module. R/P modules take the current utility estimates of the individuals in a given group comparison as input and produce the current R/P estimates for the individuals as output. The G module takes the final utility estimates of the individuals in a given group comparison as input and produces the winning probability estimate of one group preferred over the other in the given group comparison as output.

I would recommend a weak accept for this paper based on the following reasons:
* Both the R/P and G modules' input and output dimensions are independent of the number of items. This keeps the architecture scalable when facing large data sets in the real world.
* The empirical result looks satisfactory on several data sets across different domains.
* The theoretical foundation is sound.

I would hope that the authors will make some effort in making the paper more approachable and  practical:
* A more detailed motivation section for the architecture will make it much easier for the readers to understand.
* A conclusion plus future work section could come in handy for future researchers.
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper provides a technique to solve match prediction problem -- the problem of estimating likelihood of preference between a pair of M-sized sets. The paper replaces the previously proposed conventional statistical models with a deep learning architecture and achieve superior performance than some of the baselines. The experiments show the efficacy of the proposed methods.

I have some major concerns with this paper. These are described below:

1.  The paper presentation is not very clear. The paper contains many imprecise statements. The problem is not setup very well. 
(a) The abstract and the introduction  contains vague statements with no proper description of what the technique is about. 
(b) In introduction, the authors start with discussing 1-sized pairwise comparisons, and then suddenly are discussing in-group items effects at the end of the third paragraph, so that means they are talking about M-sized comparisons with M > 1. This adds to the confusion.
(c) The same things holds true for the first paragraph in "Main Contribution". Since the authors are using deep learning frameworks, this does not mean the authors "can infer the underlying models accurately." I am not sure what the authors wanted to convey with this statement. 
(d) The reason I am being very stringent with impreciseness of the statements is majorly after reading the first statement in the Motivation section. It says, "Our decision to incorporate two modules R and P into our architecture has been inspired by SOME state-of-the-art algorithms developed under CERTAIN statistical models, which have been shown
therein to be OPTIMAL." Please avoid using some, certain, optimal when they are not defined properly in the paper yet.
 
 2. The paper mentions multiple times that it does not use statistical models tailored for a dataset or application but instead use deep neural networks. In my opinion, NN is just another form of statistical model that captures statistical patterns of comparisons and in-group interaction.  They are not following the same modeling assumptions as others but have created their own in some sense. The authors may want to rephrase those statements.

3. The evaluation metric for the datasets that author consider is the prediction accuracy. I am not sure why the authors evaluate on cross entropy as well in Table 1, since both are closely related (CE is a consistent surrogate of accuracy). Can you please explain? However, I liked that they compared the methods with other metrics in Table 2. 

4. In problem setup, please mention M > 1. For M = 1, the problem is similar to [1] and many solutions have been proposed for that. 

5. I am not sure how the R and P modules capture what the authors want them to capture. I would suggest the authors to include that when they first discuss R and P modules.

6. In equations 7, it is not clear how the function R(.,.,.,) and P(.,.,.) defined? Is it similar to what is described in equation 5?

7. The training procedure contains the standard details; however, it is not clear to me how the modules R, P, and G interact during training.

Overall, I believe the paper has a descent idea and contains satisfactory experimental results; however, the presentation of the paper is very weak at this moment. 

[1] Joachims, Thorsten. "Optimizing search engines using clickthrough data." Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2002.

---- After Rebuttal ---

I thank the authors for providing response to my questions and making edits to the paper. My clarity on boxes R and G has become better; however, I am still not totally convinced. Also, the presentation of the paper could be further improved. Therefore, I would keep the same score.  
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: This paper proposes a deep neural network solution to the set ranking problem. The authors design a special architecture for this task inspired by previous manually designed algorithms. The authors show empirical superior performance. 

The idea seems potentially interesting. I think if the authors can convincingly show that they incorporate the inductive bias of previous manually designed algorithms, and sprinkle in some trainable parameters and optimization, this could be a good paper. However, I have several concerns about the paper. 

I am confused about the argument about scalability. Given the input sets A and B, how are the “current utility” computed. Some function from a set to a fixed dimensional vector is still necessary. How is this computed? In Eq(7) it seems that the input to P is a set, but this contradicts what is claimed (it is a fixed length vector?). 

In Eq.(6) it seems that only one of R and P is multiplied by a non-zero coefficient. Does this basically mean we use R network when label is 0 and P network when label is 1. How is this design choice justified? 

In the synthetic experiments, a cross entropy loss of 0.5+ seem very bad for binary classification. For example if the model just predicts p=0.5 (random guess) the loss should be better. It seems that the synthetic data labels are generated randomly. These dataset decisions should be better explained and justified. I think deriving the conclusion “This implies that our algorithm can be universally applied to achieve consistently high performances in a wide range of real-world match prediction applications.” from four synthetic datasets seem far-fetched. 

For the real world experiments, the method seems to perform marginal better on average. I think one small issue I have is as before, these accuracies (and cross entropy) are not much better than random guess, and sometimes worse. Also note that the baselines are not deep neural networks, so do not leverage the capability of automatic differentiation and optimization. 

The authors’ adaptation for Rank Centrality involves summing up the weights; this does not model their interaction and seem to be a weak baseline. 

Writing: I had some difficulty following the paper. 

I was confused about the notation. I didn’t find the definition for many symbols such as the ones in Figure 2. 

A very large fraction of the paper (2 pages) describe existing work in detail, and explain which component of the prior work is manifested (philosophically) in the current work. I think this is unnecessarily cumbersome because the current work seems to be a natural instantiation of a modern neural network to this task. Alternatively if the authors can explain the connection more concretely (i.e some parameters learned by the neural network recovers models in prior work) the arguments can be convincing.

review_title: Official Blind Review #2

Meta-review: This paper investigates neural networks for group comparison -- i.e., deciding if one group of objects would be preferred over another. The paper received 4 reviews (we requested an emergency review because of a late review that eventually did arrive). R1 recommends Weak Reject, based primarily on unclear presentation, missing details, and concerns about experiments. R2 recommends Reject, also based on concerns about writing, unclear notation, weak baselines, and unclear technical details. In a short review, R3 recommends Weak Accept and suggests some additional experiments, but also indicates that their familiarity with this area is not strong. R4 also recommends Weak Accept and suggests some clarifications in the writing (e.g. additional motivation future work). The authors submitted a response and revision that addresses many of these concerns. Given the split decision, the AC also read the paper; while we see that it has significant merit, we agree with R1 and R2's concerns, and feel the paper needs another round of peer review to address the remaining concerns.
Decision: Reject