Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper tackles the problem of calibrating ratings, focusing on the application of scientific peer review. The idea is to normalize reviewers who interpret the rating scale differently. If one reviewer tends to use low ratings and another tends to use generous high ratings, the calibration step will equilibrate them. The calibration requires the two reviewers to have some papers in common, or at least both have papers in common with a chain of other reviewers. The idea is that each paper has an inherent quality, each reviewer perceives that quality with independent zero-mean noise, and each reviewer translates that perceived quality into a rating using a monotone function. Given this model of how ratings are formed, the authors propose an optimization problem to find the hidden qualities, function parameters, and error rates that minimize the squared errors that reviewers perceive. They call this Least Square Calibration (line 133). The authors show how to do this for linear functions with and without noise, and prove some conditions under which the exact qualities can be recovered, including an if and only if connection to doubly connected review graphs with no noise. The authors also show how to do run the optimization for non-linear functions. The authors discuss similarities with linear regression and matrix seriation. The authors provide simulations to show that their method works.
main_review: The authors address a real problem: how to calibrate ratings. The problem is hard because if one reviewer uses low ratings and another uses high ratings, it's hard to disentangle reviewer tendencies from paper qualities. If the papers the two reviewers rate are disjoint, it's especially hard, but not impossible, because the first and second reviewers may each have papers in common with a third reviewer or a chain of reviewers. The authors formulate the problem in a reasonable way. The solution is straightforward. I am not fully satisfied with the results for three reasons:

First, the authors don't discuss enough the problem of overfitting. If the function hypothesis class is too large, the optimization will almost certainly overfit. For example, in the definition of LSC at line 133, if the function f is a complex non-linear function with many parameters, the optimization will be able to find a solution with all the epsilon error terms zero, all the x values arbitrary, and the function f memorizing a mapping from x to y. When the function is linear, overfitting is somewhat controlled. When the function is non-linear, the authors will need some sort of regularization.

AFTER AUTHOR RESPONSE: the following second half of the previous sentence was not correct in my original review, as the authors' pointed out in their response: ", and the tests will need to be done in the usual ML way with the optimization being solved on different data than the tests are run."

Second, the empirical tests are simulations. For this application, testing on real data will be much more persuasive. Some peer review data is here: https://github.com/allenai/PeerRead/tree/master/data described in the paper https://arxiv.org/abs/1804.09635 . The data does include some numerical scores and includes accept/reject decisions that can be thought of as ground truth. There are other ratings data sets the authors can use. For example, data from peer grading systems should work, for example http://www.tml.cs.uni-tuebingen.de/team/luxburg/code_and_data/peer_grading_data_request_new.php . Here is another peer grading paper: https://www.cs.ubc.ca/~jrwright/mta/ . Data from review web sites might work too. Here is some review data: https://www.preflib.org/data/CD . There is no ground truth, so this may or may not work. One idea would be to use the average of all ratings as the ground truth and then try to recover the ground truth by calibrating a small subset of the users. A Mechanical Turk experiment would also be revealing and can be engineering to have a ground truth.

AFTER AUTHOR RESPONSE: I am impressed with the new empirical results. Thank you for doing that with amazing speed. I am glad that the review process led to a better paper with both synthetic and real results. Still, this is only one test and it is very hard to evaluate a new test produced during the review process.

Third, the empirical tests do not split the data into training set and test set, as far as I can tell. Perhaps I missed that. Without a proper split of training and testing, the authors cannot say whether their optimization overfit to the data.

AFTER AUTHOR RESPONSE: Thank you for your response and clearing up my misunderstanding. I looked back at the paper again at the main optimization problem on line 133. This IS an unsupervised problem. So, my third complaint above about no train-test split in the experiments is not valid, as the authors point out. Sorry about that. 
However, I feel that my first argument is still valid. I don't agree that an unsupervised learning algorithm cannot overfit. For example, a clustering algorithm can get perfect results (by some metrics like purity or mutual information) by placing every data point in its own cluster: https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-clustering-1.html This is a form of overfitting. Specifically, for this paper, in the definition of LSC at line 133, I don't see why some complex non-linear function f with many parameters couldn't be optimized to find a solution with all the epsilon error terms zero, all the x values arbitrary, and the function f memorizing a mapping from x to y. If the function class for f is too large (the analogy of setting k=n in k-means clustering), the optimization can memorize an answer that has all error terms zero.


I had one more question:

Is Definition 2.1 (Linear Scoring Function) only for the linear function without noise? What happened to the independent zero-mean noise? Is it zero? What is the linear scoring function with noise? Should it be a(x+epsilon)+b?

Here are some minor writing suggestions/corrections:

there exists cardinal scores estimators perform better than any ordinal ranking estimators
there exist cardinal score estimators that perform better than any ordinal ranking estimators

320: the same number of reviews
320: the same number of reviewers

assigned scoring function from pre-specified family
assigned a scoring function from a pre-specified family

bayesian model
Bayesian model

Reference [24] has no publication venue
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper has proposed a new flexible framework, least square calibration (LSC), for selecting top candidates from peer ratings. The major contributions are summarized as follows:
a.	Proposed a model which could handle different types of miscalibration in peer reviews, and also incorporate various levels of prior knowledge
b.	Self-prove the effectiveness of LSC by both theoretically and empirically 

main_review: Originality
It is a good try to use LSC and other classes of linear scoring function for selecting top candidates for peer review, to the best of our knowledge, we do not see any previous papers have used this technique

Quality
The paper has provided rigorous mathematical proof for the LSC and other stuff, however, there are not many experiments done by the author, for example one more ablation experiment on how prior knowledge is incorporated and how that will impact the results. And this is actually facing a chicken-and-egg problem, if you know the quality, peer review is not needed anymore, but if you do not know the quality, you will not know how the performances will be in the true data. It might be also good to make experiments on real data. It outperforms all previous systems by a lot, which is really impressive. The paper has been too long (9 pages) and exceeded the maximum allowed length (8 pages).

Clarity
The paper is well-organized, used well-selected notations, and has provided rigorous proof of the method, but sometimes it is hard for the readers to follow.

Significance
The paper has proposed LSC to address miscalibration problem for peer review which outperforms all previous models.

limitations_and_societal_impact: None.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies calibration in the peer review process. Unlike most previous work, in this paper, the structure of reviewer miscalibration needs to only be monotone (in the true score of the paper being reviewed).  The authors also study the cases where: miscalibration is linear, and there is added noise.  The main contribution is framing the calibration problem as a (specific) functional optimization problem. Theoretically, they prove that optimal solutions to their optimization problems yield the correct underlying ordering of the papers (by true score). In comparison to 3 baseline algorithms the proposed approach tends to perform best in simulated experiments.
main_review: *Originality*:
Framing the calibration problem as functional optimization seems novel to me and is interesting.  In experiments it also tends to yield superior results (i.e., precision/GAP) than other competing methods. Similarly, while subtle, I think the idea of putting the noise term _inside_ the score generation (equation 1), is quite useful and makes the problem easier to solve (than putting the noise term outside). I don’t see a problem with this approach immediately; assuming it is not problematic, this could be used in further research on calibration. 

Additionally, I found remark 3.5 especially interesting. I was unaware that additional reviews are requested in order to achieve better calibration (I thought additional reviews were requested for other reasons).  Authors: can you give a citation or anecdotal evidence of this? Either way, the remark implies a strategy of holding out some reviewers and assigning them after an initial set of reviews for the sake of better calibrating the reviewer scores.


*Quality*: 
The work seems technically sound: the “if” direction of Theorem 3.4 seems correct; the “only if” direction I’m a bit confused about. Theorem 3.6 also seems correct.  All functional optimization problems outlined seem correct as well.

In the experimental setup: M=1000 and N=1000 seems unrealistic. Why not use the exact parameters from ICLR? I’m curious about whether/how these parameters affect the results.  Moreover, I’m interested in how the distribution of paper (true) scores affect the results (in terms of precision/gap). I’m also curious about other evaluation metrics: how many papers would be wrongly accepted/rejected when using each of the baselines? The precision/gap get at this, but it would be illuminating to calculate statistics like this for each method. 

Finally, in the experimental section, it would be useful to describe why the proposed method outperforms the other baselines in each case, especially in the case of linear functions (with/without noise).

*Clarity*: 
Overall, the paper was clear. However, some details related to clarity could be improved. For example: what is the range of true scores, $x*$? Of the scores x in the LP?  In the LSC function with linear scores and no noise: it was at first confusing to me that the scoring functions do not appear; consider adding them or explaining this in the text. In definition 3.1: I’m a bit confused if $x(i)$ is a linear function of the true score and $y(i)$ is a linear function of $x(i)$.  Also, can you explain why the first constraint in CP 4 is necessary? Why does Theorem 3.6 only imply that the true item ordering is recoverable but not the scores?

At first glance, Section 4 seems a bit superfluous.  That being said, I found 4.1 helpful. Perhaps some of this information could be condensed and added into previous sections. This would give room for more experimental results.

In the experiments section, k is referenced before it is defined.

In Figure 1: why does the precision of Average go down with increasing fraction of linear functions?  Moreover, it seems like EQ performs much worse when noise is added; can you explain why?

There are grammatical errors in the proof sketch of Theorem 3.4 (there are extra “the”s).

Algorithm box 1 may not be necessary (just describe it in words and put the algorithm in the appendix). 


*Significance*:
I found the result from 3.4 interesting, however, I’m unsure of how technically novel it is. Moreover, I’m curious as to how often we’re likely to get a doubly-connected component that covers all papers in practice.  I think it would be worth it to check this using real data and standard (max-similarity) matching. 

Applying function optimization to reviewer calibration is reasonable and I’d be very curious to see how this works in practice.  Similarly, I think it would be worthwhile for program chairs to experiment with this style of calibration in practice.


EDIT AFTER AUTHOR RESPONSE:  Thank you authors for clearing up some of my confusion.  Implementing the changes you identified in your response (e.g., writing changes, more realistic simulation and extrapolation of ethical concerns) will improve the paper. 
limitations_and_societal_impact: There are a handful of limitations with this work.  First and foremost, the monotonicity requirements on the score generation. While this is a significant limitation, a) this could be reasonable in practice and b) virtually all related work makes the same assumption.

Another limitation is the assumptions required on the type of scoring functions (linear/monotonic).  This seems to be difficult to judge a-priori and important: e.g., the QP approach outperforms the proposed LSC-monotone approach in the noiseless case.  That being said, the noiseless case is perhaps a bad approximation of reality. 

As mentioned above, the expectation of doubly connectedness in the review graph is a limitation. Do all the results require this? Do all the simulations guarantee that the review graphs are doubly connected? And, if so, what happens if the review graphs break this assumption?

Finally, I’d like to better understand the design choice of applying noise to the papers rather than the reviewers.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper addresses the problem of review calibration, i.e., to
estimate true paper qualities from reviewer ratings, if they
are monotonic functions in the true quantities. The authors model
the problem as an optimization problem that aims to minimize
input noise on the reviewer functions. The authors derive explicit
linear programs for the case of linear reviewer functions with
and without noise and sketch further variants. They discuss differences
of their approach to linear regression and matrix seriation.
In an experiment with synthetic data (and thus given ground truth),
they compare their method against two baselines from the literature
and show improvements both in the case of linear reviewer functions
being the groundtruth as also for more general reviewer functions.

main_review: The paper is very well written and a joy to read. The proposed method
is plausible and the experimental results look very good, though only
synthetic data has been used.

I have only three minor points with the paper:
1. The proposed method is a pure optimization solution, thus the
    paper maybe will be better positioned in an optimization conference?
2. It might be practically interesting only for a small, special audience (PC chairs).  
3. If there is no way to observe the groundtruth as the authors allude to
   ("if we already know the true qualities, peer reviews would not be needed
   anymore"), this would raise the question what exactly these "true qualities"
   then are. Maybe the field of peer review calibration needs to find
   some way to make them observable. 

limitations_and_societal_impact: yes
review_title: null

Meta-review: This paper addresses miscalibration in peer review. Miscalibration can cause problems with the fairness of the peer review process, but so far not much has been achieved in terms of theoretical algorithms that also have practical appeal. Some previous papers study simple linear models and another study considers arbitrary or adversarial miscalibration. This paper fills in this void by considering general classes of miscalibration models. It proposes an interesting "LSC" method, provides theoretical results to back it up, and simulations to investigate it further. The rebuttal, in response to some reviewers' comments, also provides experiments on real-world data and their algorithms work well here. This is a good contribution to the field and I recommend acceptance.

Important: In their camera ready version, the authors should incorporate the various points discussed with the reviewers or with me during the review process. We hope that this review process has helped the authors improve the paper.

Summary of reviews:
- Reviewer o3ak provides a detailed review, regards the paper as novel and interesting
- Reviewer zLc7 makes some initial criticisms in their initial review. The rebuttal addresses the objective parts of these criticisms but the reviewer did not show up after the rebuttal (I had sent multiple pings).
- Reviewer PoXu provides a detailed review. They make several criticisms in their initial review, and the objective parts of these criticisms are addressed in a 3-way discussion between me, the authors, and the reviewer.
- Reviewer Emh1's review has only minor points of concern

Decision: Accept (Poster)