SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Bias of standard matrix estimators due to dependence. Note that the sensing matrix (4) depends on the noise  $\eta$ . Quantitatively, we have  $\mathbb{E}\left[\eta A^{\mathrm{inv}}\right] \neq 0$  (see Appendix D.1). Standard trace regression analyses require that this quantity be zero, typically assuming (at least) that  $\eta$  is zero-mean conditioned on the sensing matrix  $A$ . The failure of this to hold in our case introduces a bias that does not decrease with the sample size  $N$ .
- Heavy-tailed sensing matrix. The factor  $\frac{1}{a^{\top}\Sigma^{\star}a}$  in  $A^{\mathrm{inv}}$  (see Equation (4)) makes  $A^{\mathrm{inv}}$  heavy-tailed in general. When  $a$  is Gaussian, the term  $\frac{1}{a^{\top}\Sigma^{\star}a}$  is an inverse weighted chi-square random variable, whose higher-order moments are infinite (and the number of finite moments depends on the rank of  $\Sigma^{\star}$ ). This makes error analysis more difficult, as standard analyses require the sensing matrix  $A$  to concentrate well (e.g., be sub-exponential).
To overcome these challenges, we make two key modifications to the procedure (5).
Step 1: Bias reduction via averaging. First, we want to mitigate the bias due to the dependence between the sensing matrix  $A^{\mathrm{inv}}$  and the noise  $\eta$ . The bias term  $\mathbb{E}\left[\eta A^{\mathrm{inv}}\right]$  scales proportionally to  $\mathbb{E}\left[\eta (y + \eta)\right] = \mathbb{E}\left[\eta^{2}\right]$ . Therefore, to reduce this bias in the least-squares estimator (5), we need to reduce the noise variance. We reduce the effective noise variance (and hence the bias) by averaging i.i.d. samples. Operationally, instead of obtaining  $N$  measurements from  $N$  distinct sensing vectors  $\{a_i\}_{i = 1}^N$ , we draw  $n$  sensing vectors  $\{a_i\}_{i = 1}^n$ , and collect  $m$  measurements, denoted by  $\{\gamma_i^{(j)}\}_{j = 1}^m$  corresponding to each sensing vector  $a_{i}$ . We refer to  $n$  as the number of (distinct) sensing vectors.
Algorithm 1 Inverted measurement sensing, averaging, and truncation.
Input: number of total measurements  $N$ , averaging parameter  $m$  (that divides  $N$ ), truncation threshold  $\tau$ , measurement value  $y$
1: Compute the number of sensing vectors  $n = \frac{N}{m}$
2: for  $i = 1$  to  $n$  do
3: Draw sensing vector  $\mathbf{a}_i$  from standard multivariate normal distribution
4: Obtain  $m$  PAQ measurements  $(\gamma_i^{(1)})^2, \ldots, (\gamma_i^{(m)})^2$  with  $\mathbf{a}_i$  and  $y$ .
5: Bias elimination via averaging: compute averaged response  $\bar{\gamma}_i^2 = \frac{1}{m}\sum_{j = 1}^{m}(\gamma_i^{(j)})^2$
6: Heavy tail mitigation via truncation: compute truncated response  $\widetilde{\gamma}_i^2 = \bar{\gamma}_i^2\wedge \tau$
7: end for
Output: truncated responses  $\widetilde{\gamma}_1^2, \ldots, \widetilde{\gamma}_n^2$
To keep the total number of measurements constant, we set  $n = \frac{N}{m}$ , where the value of  $m$  is specified later. For each sensing vector  $\mathbf{a}_i$ , we compute the empirical mean of the  $m$  measurements:
$$
\bar {\gamma} _ {i} ^ {2} := \frac {1}{m} \sum_ {j = 1} ^ {m} \left(\gamma_ {i} ^ {(j)}\right) ^ {2} = \frac {1}{m} \sum_ {j = 1} ^ {m} \frac {y + \eta_ {i} ^ {(j)}}{\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a} _ {i}} = \frac {y + \bar {\eta} _ {i}}{\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a} _ {i}}, \tag {6}
$$
where we define the average noise by  $\bar{\eta}_i\coloneqq \frac{1}{m}\sum_{j = 1}^{m}\eta_i^{(j)}$ . This averaging operation reduces the effective noise variance from  $\mathrm{var}(\eta_i) = \nu_\eta^2$  to  $\mathrm{var}(\bar{\eta}_i) = \frac{\nu_\eta^2}{m}$ . If  $n$  is small, we may have large error due to an insufficient number of query vectors  $\pmb{a}_i$ . On the other hand, a small  $m$  leads to a large bias. Therefore, we set the value of  $m$  carefully to balance these two effects. This is studied theoretically in Section 4 and demonstrated empirically in Section 5.
Step 2: Heavy tail mitigation via truncation. Next, we need to control the heavy-tailed behavior introduced by the  $\frac{1}{a^{\top}\Sigma^{*}a}$  term in the sensing matrix  $A^{\mathrm{inv}}$ . Note that the sample averaging procedure (6) does not mitigate this problem. We adopt the approach in [39] and truncate the observations. Specifically, we truncate the averaged measurements  $\bar{\gamma}_i^2$  to  $\widetilde{\gamma}^2 \coloneqq \bar{\gamma}^2 \wedge \tau$ , where  $\tau > 0$  is a truncation threshold that we specify later. We then construct the truncated sensing matrices
$$
\widetilde {\boldsymbol {A}} _ {i} = \widetilde {\gamma} _ {i} ^ {2} \boldsymbol {a} _ {i} \boldsymbol {a} _ {i} ^ {\top} = \left(\frac {y + \bar {\eta} _ {i}}{\boldsymbol {a} _ {i} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a} _ {i}} \wedge \tau\right) \boldsymbol {a} _ {i} \boldsymbol {a} _ {i} ^ {\top}, \quad i = 1, \dots , n. \tag {7}
$$
While truncation mitigates heavy-tailed behavior, it also introduces additional bias in our estimate. The truncation threshold  $\tau$  therefore gives us another tradeoff, and in our analysis to follow, we carefully set the value of  $\tau$  to balance the effects of heavy-tailedness and bias.
Final algorithm. Before presenting our final optimization program, we summarize our assumptions and sensing model below.
Assumption 1. The noise values  $\eta_{i}$  are i.i.d copies of the random variable  $\eta$ , which is independent of the random sensing vector  $\mathbf{a}$ . The random noise is (1) zero-mean:  $\mathbb{E}[\eta] = 0$ , and (2) bounded: There exists a positive constant  $\eta^{\uparrow}$  such that  $-y \leq \eta \leq \eta^{\uparrow}$  with probability 1.
We choose the sensing vector distribution to be the standard multivariate normal distribution and collect, average, and truncate  $N$  PAQ responses following Algorithm 1. This process yields  $n$  truncated responses  $\widetilde{\gamma}_{\underline{1}}^{2},\ldots \widetilde{\gamma}_{n}^{2}$ . We then use these truncated responses to form the averaged and truncated matrices  $\{\pmb {A}_i\}_{i = 1}^n$ , which we substitute into the original least-squares problem (5). To estimate  $\Sigma^{\star}$ , we solve
$$
\widehat {\boldsymbol {\Sigma}} \in \underset {\boldsymbol {\Sigma} \succeq \mathbf {0}} {\arg \min } \frac {1}{n} \sum_ {i = 1} ^ {n} \left(y - \langle \widetilde {\boldsymbol {A}} _ {i}, \boldsymbol {\Sigma} \rangle\right) ^ {2} + \lambda_ {n} \| \boldsymbol {\Sigma} \| _ {*}, \tag {8}
$$
where, again,  $\lambda_{n}$  is a regularization parameter that we specify later.
Practical considerations. In the averaging step, we collect  $m$  measurements for each sensing vector  $\pmb{a}_i$ . These measurements could be collected from  $m$  different users. Furthermore, recall from Section 2.2 that the measurements do not depend on the reference item  $x$ . As a result, one may
also collect multiple responses from the same user by presenting the same query vector  $\mathbf{a}_i$  with different reference items. In addition, recall from Section 2.1 that user responses are scale-invariant. Practitioners are hence free to set the boundary  $y$  to be any positive value of their choice without loss of generality, and the noise  $\eta$  scales accordingly with  $y$ . The user interface does not depend on the value of  $y$ .
We now present our main theoretical result, which is a finite-sample error bound for estimating a low-rank metric from inverted measurements with the nuclear norm regularized estimator (8). Our error bound is generally stated, and depends on the averaging parameter  $m$  and the truncation threshold  $\tau$ . Recall that  $\nu_{\eta}^{2}$  denotes the variance of  $\eta$ . We define the quantities  $y^{\uparrow} := y + \eta^{\uparrow}$  and  $\mu_y = y + \mathrm{median}(\eta)$ . We further denote by  $\sigma_1 \geq \dots \geq \sigma_r > 0$  the non-zero singular values of  $\Sigma^{\star}$ .
Theorem 1. Suppose  $\Sigma^{\star}$  is rank  $r$ , with  $r > 8$ . Assume that we choose the sensing vector distribution the be the standard multivariate normal distribution, that Assumption 1 holds on the noise, and that we collect, average, and truncate measurements following Algorithm 1. Further, assume that the truncation threshold  $\tau$  satisfies  $\tau \geq \frac{\mu_y}{\mathrm{tr}(\Sigma^{\star})}$ . Then there are positive constants  $c, C, C_1$ , and  $C_2$ , such that if the regularization parameter and the number of sensing vectors satisfy
$$
\lambda_ {n} \geq C _ {1} \left[ y ^ {\uparrow} \left(\frac {y ^ {\uparrow}}{\sigma_ {r} r} \sqrt {\frac {d}{n}} + \frac {d}{n} \tau + \left(\frac {y ^ {\uparrow}}{\sigma_ {r} r}\right) ^ {2} \frac {1}{\tau}\right) + \frac {1}{\sigma_ {r} r} \frac {\nu_ {\eta} ^ {2}}{m} \right] \quad a n d \quad n \geq C _ {2} r d, \tag {9}
$$
then any solution  $\widehat{\Sigma}$  to the optimization program (8) satisfies
$$
\left\| \widehat {\boldsymbol {\Sigma}} - \boldsymbol {\Sigma} ^ {\star} \right\| _ {F} \leq C \left(\frac {\operatorname {t r} \left(\boldsymbol {\Sigma} ^ {\star}\right)}{\mu_ {y}}\right) ^ {2} \sqrt {r} \lambda_ {n} \tag {10}
$$
with probability at least  $1 - 4e^{-d} - e^{-cn}$ .
The proof of Theorem 1 is presented in Appendix E. The two sources of bias discussed in Section 3.2 appear in the expression (9) for the regularization parameter  $\lambda_{n}$  (and consequently in the error bound (10)). The term scaling as  $1 / \tau$  corresponds to the bias induced by truncation, and decreases as the truncation gets milder (i.e., as the threshold  $\tau$  gets larger). The term scaling as  $\nu_{\eta}^{2} / m$  corresponds to the bias arising from dependence between the noise and sensing matrix. As discussed in Section 3.2, in this model,  $m$ -averaging results in a bias that scales like  $1 / m$ . Given the dependence of the estimation error bound on the parameters  $m$  and  $\tau$ , we carefully set these parameters to obtain a tight bound as a function of the number of total measurements  $N = mn$ . These choices for  $m$  and  $\tau$ , along with the final estimation error, are presented in the following corollary, proved in Appendix F.
Corollary 1. Recall that  $N = mn$ . Assume that the conditions of Theorem 1 hold, and set the values of the constants  $(c, C, C_1, C_2)$  according to Theorem 1. Suppose that the number of total measurements satisfies
$$
N \geq \left\{2 C _ {2} ^ {3 / 2} \frac {\nu_ {\eta} ^ {2}}{(y ^ {\uparrow}) ^ {2}} r ^ {3 / 2} d \right\} \vee \left\{C _ {2} r d \right\}. \tag {11}
$$
Set the averaging parameter  $m$  and truncation threshold  $\tau$  to be
$$
m = \left\lceil \left(\frac {\nu_ {\eta} ^ {2}}{(y ^ {\uparrow}) ^ {2}}\right) ^ {2 / 3} \left(\frac {N}{d}\right) ^ {1 / 3} \right\rceil \quad a n d \quad \tau = \frac {y ^ {\uparrow}}{\sigma_ {r} r} \sqrt {\frac {N}{m d}}, \tag {12}
$$
and set  $\lambda_{n}$  equal to its lower bound in (9). With probability at least  $1 - 4e^{-d} - e^{-cN / m}$ , we have:
(a) If  $\frac{\nu_{\eta}^{2}}{(y^{\dagger})^{2}} > \sqrt{\frac{d}{N}}$ , then any solution  $\widehat{\Sigma}$  to the optimization program (8) satisfies
$$
\left\| \widehat {\boldsymbol {\Sigma}} - \boldsymbol {\Sigma} ^ {\star} \right\| _ {F} \leq C ^ {\prime} \frac {\sigma_ {1} ^ {2}}{\sigma_ {r}} \frac {\left(y ^ {\uparrow}\right) ^ {4 / 3} \left(v _ {\eta} ^ {2}\right) ^ {1 / 3}}{\mu_ {y} ^ {2}} r ^ {3 / 2} \left(\frac {d}{N}\right) ^ {1 / 3}. \tag {13}
$$
(b) If  $\frac{\nu_{\eta}^{2}}{(y^{\uparrow})^{2}} \leq \sqrt{\frac{d}{N}}$ , then any solution  $\widehat{\Sigma}$  to the optimization program (8) satisfies
$$
\left\| \widehat {\boldsymbol {\Sigma}} - \boldsymbol {\Sigma} ^ {\star} \right\| _ {F} \leq C ^ {\prime} \frac {\sigma_ {1} ^ {2}}{\sigma_ {r}} \left(\frac {y ^ {\uparrow}}{\mu_ {y}}\right) ^ {2} r ^ {3 / 2} \left(\frac {d}{N}\right) ^ {1 / 2}. \tag {14}
$$
In both cases,  $C' = 3C \cdot C_1$ .
A few comments are warranted about our error bounds (13) and (14):
Error rates and noise regimes. Under the standard trace measurement model, it is known that if the measurement matrices are i.i.d. according to some sub-Gaussian distribution and the number of measurements satisfies  $N \gtrsim rd$ , then nuclear norm regularized estimators achieve an error that scales like  $\sqrt{\frac{rd}{N}}$  (e.g., [24, 25]). Such a result is also known to be minimax optimal [25]. Allowing heavier-tailed assumptions on the sensing matrices, such as sub-exponential [32, 42] or bounded fourth moment [39], typically results in additional log  $d$  factors but does not impact the exponent  $1/2$  in the error rate. However, a crucial assumption in these results is that  $\mathbb{E}\left[\eta A^{\mathrm{inv}}\right] = 0$ , and thus there is no bias due to measurement noise. Our inverted measurement sensing matrix is not only heavy-tailed but also leads to bias (see Lemma 1 in Appendix D.1). Nevertheless, we are able to reduce the bias and trade it for variance, ensuring consistent estimation in all regimes.
In Corollary 1, there are two distinct cases for error rate which correspond to two different noise regimes induced by the quantity  $\nu_{\eta}^{2} / (y^{\uparrow})^{2}$ , which captures the noise level in our measurements. In particular, the two cases in Corollary 1 correspond to two regimes with distinct bias behavior:
(a) High-noise regime: In this setting, the bias due to measurement noise is non-negligible. As a result, we employ averaging with large  $m$ , which results in the rate scaling as  $(d / N)^{1 / 3}$ .
(b) Low-noise regime: In this setting, the measurement noise bias is dominated by the variance, and thus has negligible impact on the estimation error. As a result, we are able to achieve a rate of order  $(d / N)^{1 / 2}$ , which is consistent with established results for low-rank matrix estimation.
Sample complexity. Since the degrees of freedom in a rank-  $r$  matrix of size  $d \times d$  is of order  $rd$ , one expects that the minimum number of measurements to identify a rank-  $r$  matrix is of order  $rd$ . This is reflected in Theorem 1, which assumes that the number of distinct sensing vectors  $\{\pmb{a}_i\}$  satisfies  $n \gtrsim rd$ . In the high-noise regime, from (12) in Corollary 1, we have that  $m$  scales like  $(N / d)^{1/3}$ . Thus, the total number of measurements is  $N = mn \gtrsim (N / d)^{1/3} \cdot rd \gtrsim N^{1/3}d^{2/3}r$ , and hence  $N \gtrsim r^{3/2}d$ . Given that the rank is assumed to be relatively small compared to the dimension, the extra factor of  $\sqrt{r}$  is a relatively small price to pay to obtain consistent estimation. In the low-noise regime, it can be verified that  $m = 1$  in (12) due to the low-noise condition  $\nu_{\eta}^2 / (y^\dagger)^2 \leq \sqrt{d/N}$ . No averaging is needed, and we only require  $N = n \gtrsim rd$ .
Dependence on rank. When compared to standard results, Corollary 1 differs in its dependence on rank. First, the matrix  $\pmb{\Sigma}^{\star}$  is assumed to have rank  $r > 8$ . This prevents the term  $\frac{1}{a^{\top}\Sigma^{\star}a}$  from making the sensing matrices so heavy-tailed that even truncation does not help. We empirically show that the assumption of  $r > 8$  is necessary in Section 5. Second, there is an additional factor of  $r$  in our rate for both noise regimes. To interpret this, note that if  $\pmb{\Sigma}^{\star}$  has non-zero singular values in a fixed range, then  $\mathbb{E}\left[a^{\top}\pmb{\Sigma}^{\star}\pmb{a}\right] = \mathrm{tr}\left(\pmb{\Sigma}^{\star}\right)\approx r$ . Since the "magnitude" of the sensing matrix  $\pmb{A}^{\mathrm{inv}}$  is inversely proportional to  $a^\top \Sigma^\star a$ , increasing  $r$  decreases the magnitude of  $\pmb{A}^{\mathrm{inv}}$  and thus also (for a fixed noise level) the signal-to-noise ratio.
Scale equivariance. As discussed in Section 2.1, the metric learning from PAQs problem aims to find an equivalence class  $\{(c_{\mathrm{scale}}\pmb {\Sigma},c_{\mathrm{scale}}y):c_{\mathrm{scale}} > 0\}$ , and the ground-truth  $\pmb{\Sigma}^{\star}$  is defined with respect to a particular choice of  $y$ . Accordingly, our error bounds are scale-equivariant: If we instead replaced  $y$  with  $c_{\mathrm{scale}}y$ , the bounds (13) and (14) would scale linearly in  $c_{\mathrm{scale}}$ . This fact is precisely verified in Appendix C and relies on the fact that the noise also scales appropriately in  $c_{\mathrm{scale}}$ . Thus practitioners may simply set  $y$  to be any positive number.
(b)
Figure 4: Simulations quantifying the effect of dimension  $d$ , rank  $r$ , and averaging parameter  $m$  on estimation error. Shaded areas correspond to standard error of the mean but sometimes not visible.
5 Numerical simulations
In this section, we provide numerical simulations investigating the effects of the various problem and estimation parameters. For all results, we report the normalized estimation error  $\| \widehat{\pmb{\Sigma}} -\pmb{\Sigma}^{\star}\|_{F} / \| \pmb{\Sigma}^{\star}\|_{F}$  averaged over 20 trials. Shaded areas (sometimes not visible) represent standard error of the mean. For all experiments, we follow [19] and generate the ground-truth metric matrix as  $\pmb{\Sigma}^{\star} = \frac{d}{\sqrt{r}}\pmb{U}\pmb{U}^{\top}$ , where  $\pmb{U}\in \mathbb{R}^{d\times r}$  is a randomly generated matrix with orthonormal columns. The noise  $\eta$  is sampled from a uniform distribution on  $[- \eta^{\uparrow},\eta^{\uparrow}]$  (where  $\eta^{\uparrow}\leq y$ ). We set the regularization parameter, truncation threshold, and averaging parameter in a manner consistent with our theoretical results (see Eqs. (9) and (12)), cross-validating to choose the constant factors. We solve the optimization problem using cvxpy [43, 44]. Code for all simulations is provided at https://github.com/austinxu87/paq.
Effects of dimension and rank. Our first set of experiments characterizes the effects of dimension  $d$  and matrix rank  $r$ . For all experiments, unless we are sweeping a specific parameter, we set  $y = 200$ ,  $d = 50$ ,  $r = 15$ , and  $\eta^{\uparrow} = 10$ . Fig. 4a shows the performance for varying values of  $d$  plotted against the normalized sample size  $N / d$ . For all dimensions  $d$ , the error decays to zero as the total number of measurements  $N$  increases. Furthermore, the error curves are well-aligned when the sample size is normalized by  $d$  with fixed  $r$ , empirically aligning with Corollary 1. Fig. 4b shows the performance for varying values of rank  $r$ . Recall that for our theoretical results we assume  $r > 8$  to ensure that the quadratic term  $\boldsymbol{a}^{\top}\boldsymbol{\Sigma}^{\star}\boldsymbol{a}$  in the denominator of our sensing matrices does not lead to excessively heavy-tailed behavior. When  $r > 8$ , the number of measurements required for the same estimation error increases as the rank increases. A clear phase transition occurs at  $r = 8$ . The error still decreases with  $N$  for  $r \leq 8$ , but at a markedly slower rate than when  $r > 8$ . This empirically demonstrates that when  $r \leq 8$ , the sensing matrix tails are too heavy to be mitigated by truncation.
Effect of averaging parameter  $m$ . Equation (12) suggests that the averaging parameter  $m$  should scale proportionally to  $(N / d)^{1/3}$ . To test this, we set  $y = 200$ ,  $d = 50$ ,  $r = 9$ , and  $\eta^{\uparrow} = 200$ . We vary values of  $m$  for different choices of the  $(N, d)$  pair, as shown in Fig. 4c. The empirically optimal choice of  $m$  is observed to be the same when  $N / d$  is fixed, regardless of the particular choices of  $N$  or  $d$  (the green and red curves overlap, and the blue and orange curves overlap). Moreover, the optimal  $m$  is smaller when  $N / d = 400$  compared to when  $N / d = 1000$ .
In this paper we introduced the perceptual adjustment query, a cognitively lightweight way to obtain expressive responses from humans. We specifically investigate using PAQs for human perceptual similarity learning. We use a Mahalanobis distance-based model for human similarity perception and use PAQs to estimate the unknown metric. Using random measurements to learn an unknown Mahalanobis metric gives rise to the new inverted measurement scheme for high dimensional low rank matrix estimation which violates commonly held assumptions for existing estimators. We developed a two-stage estimator and provide corresponding sample complexity guarantees. This work opens up many interesting lines of future work in inverted measurements. One key direction is to characterize their fundamental limits with information-theoretic lower bounds.
AX was supported by National Science Foundation grant CCF-2107455. JW was supported by the Ronald J. and Carol T. Beerman President's Postdoctoral Fellowship and the Algorithms and Randomness Center Postdoctoral Fellowship at Georgia Tech. MD was supported by National Science Foundation grants CCF-2107455, IIS-2212182, and DMS-2134037. AP was supported in part by the National Science Foundation grants CCF-2107455 and DMS-2210734, and by research awards/gifts from Adobe, Amazon and Mathworks.
[1] William L. Rankin and Joel W. Grube. A comparison of ranking and rating procedures for value system measurement. European Journal of Social Psychology, 10(3):233-246, 1980.
[2] Anne-Wil Harzing, Joyce Baldueza, Wilhelm Barner-Rasmussen, Cordula Barzantny, Anne Canabal, Anabella Davila, Alvaro Espejo, Rita Ferreira, Axele Giroud, Kathrin Koester, Yung-Kuei Liang, Audra Mockaitis, Michael J. Morley, Barbara Myloni, Joseph O.T. Odusanya, Sharon Leiba O'Sullivan, Ananda Kumar Palaniappan, Paulo Prochno, Srabani Roy Choudhury, Ayse Saka-Helmhout, Sununta Siengthai, Linda Viswat, Ayda Uzuncarsili Soydas, and Lena Zander. Rating versus ranking: What is the best way to reduce response and language bias in cross-national research? International Business Review, 18(4):417-432, 2009. ISSN 0969-5931.
[3] Georgios N. Yannakakis and John Hallam. Ranking vs. preference: A comparative study of self-reporting. In Sidney D'Mello, Arthur Graesser, Björn Schuller, and Jean-Claude Martin, editors, Affective Computing and Intelligent Interaction, pages 437-446, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg. ISBN 978-3-642-24600-5.
[4] Nihar B. Shah, Joseph K Bradley, Abhay Parekh, Martin Wainwright, and Kannan Ramchandran. A case for ordinal peer-evaluation in MOOCs. In NIPS Workshop on Data Driven Education, 2013.
[5] Karthik Raman and Thorsten Joachims. Methods for ordinal peer grading. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, volume 20, 2014.
[6] Richard D. Goffin and James M. Olson. Is it all relative? Comparative judgments and the possible improvement of self-ratings and ratings of others. Perspectives on Psychological Science, 6(1):48-60, 2011.
[7] William Barnett. The modern theory of consumer behavior: Ordinal or cardinal? The Quarterly Journal of Austrian Economics, 6(1):41-65, 2003.
[8] Richard Batley. On ordinal utility, cardinal utility and random utility. Theory and Decision, 64: 37-63, 2008.
[9] Nihar B. Shah, Sivaraman Balakrishnan, Joseph Bradley, Abhay Parekh, Kannan Ramchandran, and Martin J. Wainwright. Estimation from pairwise comparisons: Sharp minimax bounds with topology dependence. Journal of Machine Learning Research, 17(58):1-47, 2016.
[10] Jingyan Wang and Nihar B. Shah. Your 2 is my 1, your 3 is my 9: Handling arbitrary miscalibrations in ratings. In Proceedings of the 18th International Conference on Autonomous Agents and Multiagent Systems, 2019.
[11] Dale Griffin and Lyle Brenner. Perspectives on Probability Judgment Calibration, chapter 9. Wiley-Blackwell, 2008. ISBN 9780470752937.
[12] Polina Harik, Brian Clauser, Irina Grabovsky, Ronald Nungester, David Swanson, and Ratna Nandakumar. An examination of rater drift within a generalizability theory framework. Journal of Educational Measurement, 46:43-58, 2009.
[13] Carol M. Myford and Edward W. Wolfe. Monitoring rater performance over time: A framework for detecting differential accuracy and differential scale category use. Journal of Educational Measurement, 46(4):371-389, 2009.
A Broader impacts and limitations
In this section, we discuss the broader impacts and limitations of our work.