SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Austin  $\mathbf{X}\mathbf{u}^{*}$
Andrew D. McRae
Jingyan Wang
Mark A. Davenport
Ashwin Pananjady
We introduce a new type of query mechanism for collecting human feedback, called the perceptual adjustment query (PAQ). Being both informative and cognitively lightweight, the PAQ adopts an inverted measurement scheme, and combines advantages from both cardinal and ordinal queries. We showcase the PAQ in the metric learning problem, where we collect PAQ measurements to learn an unknown Mahalanobis distance. This gives rise to a high-dimensional, low-rank matrix estimation problem to which standard matrix estimators cannot be applied. Consequently, we develop a two-stage estimator for metric learning from PAQs, and provide sample complexity guarantees for this estimator. We present numerical simulations demonstrating the performance of the estimator and its notable properties.
Should we query cardinal or ordinal data from people? This question arises in a broad range of applications, such as in conducting surveys [1-3], grading assignments [4, 5], evaluating employees [6], and comparing or rating products [7, 8], to name a few. Cardinal data are numerical scores. For example, teachers score writing assignments in the range of 0-100, and survey respondents express their agreement with a statement on a scale of 1 to 7. Ordinal data are relations between items, such as pairwise comparisons (choosing the better item in a pair) and rankings (ordering all or a subset of items). There is no free lunch, and both cardinal and ordinal queries have pros and cons.
On the one hand, collecting ordinal data is typically more efficient in terms of worker time and cognitive load [9], and surprisingly often matches or exceeds the accuracy of cardinal data [1, 9]. The information contained in ordinal queries, however, is fundamentally limited and lacks expressiveness. For example, pairwise comparisons elicit binary responses where two items are compared against each other, but the absolute placement of these items with respect to the entire pool is lost. On the other hand, cardinal data are more expressive [10]. For example, assigning two items scores of 1 and 2 conveys a very different message from assigning them scores of 9 and 10, or 1 and 10, although all yield the same pairwise comparison outcome. However, the expressiveness of cardinal data often comes at the cost of miscalibration: Prior work has shown that different people have different scales [11], and even a single person's scale can drift over time (e.g., [12, 13]). These inter-person and intra-person discrepancies make it challenging to interpret and aggregate raw scores effectively.
The goal of this paper is to study whether one can combine the advantages of cardinal and ordinal queries to achieve the best of both worlds. Specifically, we pose the research question:
*Contact: axu@gatech.edu. The full version of this work can be found on arXiv.
Figure 1: The user interface for perceptual adjustment query (PAQ) for preference learning (top) and similarity learning (bottom).
Can we develop a new paradigm for human data elicitation that is expressive, accurate, and cognitively lightweight?
Towards this goal, we extract key features of both cardinal and ordinal queries, and we propose a new type of query scheme that we term the perceptual adjustment query (PAQ). As a thought experiment, consider the task of learning an individual's preferences between modes of transport. The query can take the following forms:
- Ordinal: Do you prefer a $2 bus ride that takes 40 minutes or a $25 taxi that takes 10 minutes?
- Cardinal: On a scale of 0 to 1, how much do you value a $2 bus ride that takes 40 minutes?
- Proposed approach: To reach the same level of preference for a  $2 bus trip that takes 40 minutes, a taxi that takes 10 minutes would cost$ x.
A user interface for the proposed approach is shown in Figure 1 (top). We present the user a reference item (a \(2 bus ride that takes 40 minutes) and a sliding bar representing the number of dollars \((x)\) for the 10-minute taxi cost. As the user adjusts the slider, the value of \(x\) starts with 0 and gradually increases on a continuous scale. The user is instructed to place the slider at a point where they equally prefer a \)2 bus ride and a taxi ride of \(x\) dollars. The PAQ thus combines ordinal and cardinal elicitation in an intuitive fashion: We obtain ordinal information by asking the user to make cognitive judgments in a relative sense by comparing items, and cardinal information can be extracted from the location of the slider. The ordinal reasoning endows the query with accuracy and efficiency, while the cardinal output enables a more expressive response. Moreover, this cardinal output mitigates miscalibration, because instead of asking the user to rate on a subjective and ambiguous notion (i.e., preference), we provide the user a reference object (i.e., the \)2 bus ride) to anchor their rating scale.
Beyond combining the strengths of cardinal and ordinal queries, PAQs have additional advantages that are well illustrated with the example in Figure 1 (bottom). First, PAQs provide users with the context of a specific (continuous) dimension along which items vary. For example, consider a pairwise comparison between the reference item and the "yellow apple" selected in Figure 1. They have similar shapes, but different colors. If these two items are shown to the user in isolation, the user lacks context to judge whether they should be considered similar or dissimilar. In contrast, the full spectrum provided in PAQs tells the user that the similarity judgment is apples vs. pears. The access to such context improves self-consistency in user responses [14]. Second, PAQs provide "hard examples" by design and thus enable effective learning. Consider Figure 1 (bottom): Items on the left of the spectrum are apples (clearly similar to the reference), and items on the right are pears (clearly dissimilar to the reference), and only a small subset of items in the middle appear ambiguous. PAQs collect information precisely about "confusing" items in this ambiguous region. On the other hand, if ordinal queries are constructed by selecting uniformly at random from the items shown, an item in the ambiguous region will rarely be presented to the user.
The ordinal component is crucial in our proposed perceptual adjustment query—we provide a reference item and instruct people to make a relational judgment of the target item compared to the reference item. Hence, the perceptual adjustment query is distinct from sliding survey questions that elicit purely cardinal responses.
The advantages of PAQs make their deployment appealing in a variety of problem settings. For example, practitioners may deploy PAQs to learn human preferences, like in the taxi and bus example in Figure 1 or more complex settings such as housing preferences, where multiple features (price, square footage, proximity to employment etc.) vary as the slider is moved. PAQs can also be used to learn models for human perception, such as characterizing the extent of color blindness for a given user. One can present a user with red-green color blindness a sequence of colors that slowly transitions between red and green, and ask them to drag the slider until they perceive a difference in colors. An extended discussion of applications is provided in the full version of this paper on arXiv. In this paper, we focus on learning metrics to characterize human perception. In this problem, items are represented by points in a (possibly high-dimensional) space, and the goal is to learn a distance metric such that a smaller distance between a pair of items means that they are semantically and perceptually closer, and vice versa. Figure 1 (bottom) presents a PAQ for collecting similarity data for metric learning, where the user is instructed to place the slider at the precise point where the object appears to transition from being similar to dissimilar. To construct a sequence of images as shown in Figure 1 (bottom), one can traverse a path in the latent space of a generative model — given a latent feature vector, the generative model synthesizes a corresponding image.
1.1 Do PAQs improve upon ordinal queries? A simulation vignette
Consider the problem of Mahalanobis metric learning, which forms the focus of this paper. In this setting, items are represented as points in the vector space  $\mathbb{R}^d$ , which is in turn endowed with a Mahalanobis metric parametrized by a symmetric positive semidefinite matrix  $\boldsymbol{\Sigma}^{\star} \in \mathbb{R}^{d \times d}$ . The (dis-)similarity of two items is determined by their distance under the metric: The larger the (squared) distance  $\| \pmb{x} - \pmb{x}' \|_{\pmb{\Sigma}^{\star}}^2 = (\pmb{x} - \pmb{x}')^\top \pmb{\Sigma}^{\star} (\pmb{x} - \pmb{x}')$  between two items  $\pmb{x}$  and  $\pmb{x}'$  is, the more dissimilar the items are. We are particularly interested in the setting in which  $\boldsymbol{\Sigma}^{\star}$  is low-rank. Established approaches in metric learning use ordinal queries, such as pairwise comparisons ("Are items  $\pmb{x}$  and  $\pmb{x}'$  similar?") [15-18], triplet comparisons [19] ("Which of the two items  $\pmb{x}_1$  and  $\pmb{x}_2$  is closer to reference item  $\pmb{x}_0$ ?"), and ranking- $k$  queries ("Given a reference item  $\pmb{x}_0$ , rank the set of items  $\pmb{x}_1, \dots, \pmb{x}_k$  in terms of similarity to  $\pmb{x}_0$ ) [14].
We compare the performance of such queries against PAQs in a toy metric learning setup. In particular, we choose a random low-rank matrix  $\pmb{\Sigma}^{\star}$  in dimension 50 with rank 10 (see Appendix B for our precise construction) and use the models and state-of-the-art algorithms of [19, 14] to produce pairwise, triplet, and ranking- $k$  queries and estimate the low-rank metric. In addition to these ordinal queries, we simulate PAQ responses under the model presented in Section 2 and use our algorithm (see Section 3) for estimation. To simplify the example, all query responses are generated in a noiseless fashion—for example, the triplet comparison always returns the closer item to the reference. We present our results in Figure 2, which illustrates a significant gap in information richness between PAQs and a variety of ordinal queries. The number of PAQ responses needed to attain a reasonable normalized error is dramatically lower than those of typical ordinal queries, illustrating that PAQs can greatly improve upon the performance of existing ordinal queries for metric learning. The rest of our paper explores this opportunity: It aims to make the deployment of PAQs theoretically grounded by designing provable methodology for learning a low-rank metric from PAQ responses.
1.2 Our contributions and organization
In addition to introducing the perceptual adjustment query (PAQ), we demonstrate its applicability to metric learning under a Mahalanobis metric. We first present a mathematical formulation of this estimation problem in Section 2. We then show that the sliding bar response can be viewed as an inverted measurement of the metric matrix that we want to estimate, which allows us to restate our problem as that of estimating a low-rank matrix from a specific type of trace measurement (Section 3). However, our PAQ formulation differs from classical matrix estimation due to two technical challenges: (a) the sensing matrices and noise are correlated, and (b) the sensing matrices are heavy-tailed. As a result, standard matrix estimation algorithms give rise to biased estimators. We propose a query procedure and an estimator that overcome these two challenges, and we prove statistical error bounds on the estimation error (Section 4). The unconventional nature of the sensing model and estimator causes unexpected behaviors in our error bounds; in Section 5, we present simulations verifying that these behaviors also appear in practice.
Figure 2: Performance of noiseless PAQs and various ordinal queries for low-rank metric learning. We plot the mean and standard error of the mean (shaded regions, not visible) of the normalized estimation error over 10 trials.
Figure 3: The perceptual adjustment query. Given a reference item  $\pmb{x}$  and a query vector  $\pmb{a}$ , a continuous path of items is formed  $\{\pmb{x} + \gamma \pmb{a} : \gamma \in [0, \infty)\}$ . Then, a user is asked to pick the first item along this path that is dissimilar to the reference item, denoted by  $\pmb{x} + \gamma \pmb{a}$ .
We discuss related work on metric learning and the statistical techniques that we use.
Metric learning. As discussed in Section 1.1, prior work in metric learning [20] considers a wide variety of ordinal queries. PAQ can be viewed as extending a discrete set of items presented to users to a continuous spectrum, which is natural when one uses a generative model such as a GAN [21, 22]. However, the goal of tuple queries is to rank the items, whereas in PAQ the ranking is provided by the feature space and we ask people to identify a transition point (similar vs. dissimilar) in this ranking.
Statistical techniques. In our theoretical results, we apply techniques from the high-dimensional statistics literature. Our theoretical formulation (presented in Section 3) resembles the problem of low-rank matrix estimation from trace measurements (e.g., [23-28]; see [29] for a more complete overview), and, in particular, when the sensing matrix is of rank one [30-33]. However, as discussed in Section 3, our model presents two important distinctions from prior literature. In our case, the sensing matrices are both heavy-tailed and correlated with the measurement noise. The heavy-tailed matrices violate the assumptions of much prior work that relies on sub-Gaussian or sub-exponential assumptions on the sensing matrices. Prior work has attempted to address the challenge of heavy tails with methods such as robust loss functions [34, 35] or the "median-of-means" approach [36-38]. We draw particular inspiration from Fan et al. [39], which considers truncation to control heavy-tailed behavior for a variety of problems. However, in the low-rank matrix estimation setting, Fan et al. [39] only analyze the case of heavy-tailed noise under a sub-Gaussian design, meaning their methodology and results are not applicable to our problem setting.
In this section, we present our model for the perceptual adjustment query (PAQ) in the context of its application to metric learning.
2.1 Mahalanobis metric learning
We consider a  $d$ -dimensional feature space where each item is represented by a point in  $\mathbb{R}^d$ . The distance metric model for human similarity perception posits that there is a metric on  $\mathbb{R}^d$  that measures how dissimilar items are perceived to be. A recent line of work [40, 41] has modeled the distance metric as a Mahalanobis metric. If  $\boldsymbol{\Sigma}^{\star} \in \mathbb{R}^{d \times d}$  is a symmetric positive semidefinite (PSD) matrix, the squared Mahalanobis distance with respect to  $\boldsymbol{\Sigma}^{\star}$  between items  $\boldsymbol{x}$  and  $\boldsymbol{x}' \in \mathbb{R}^d$  is  $\| \boldsymbol{x} - \boldsymbol{x}' \|_{\Sigma^{\star}}^2 \coloneqq (\boldsymbol{x} - \boldsymbol{x}')^\top \boldsymbol{\Sigma}^{\star} (\boldsymbol{x} - \boldsymbol{x}')$ . The distance represents the extent of dissimilarity between items  $\boldsymbol{x}$  and  $\boldsymbol{x}'$ : If we further have a perceptual boundary value  $y > 0$ , this model posits that items  $\boldsymbol{x}, \boldsymbol{x}'$  are perceived as similar if  $\| \boldsymbol{x} - \boldsymbol{x}' \|_{\Sigma^{\star}}^2 < y$  and dissimilar if  $\| \boldsymbol{x} - \boldsymbol{x}' \|_{\Sigma^{\star}}^2 \geq y$ . We adopt a high-dimensional framework and, following [19, 41], assume that the matrix  $\boldsymbol{\Sigma}^{\star}$  is low-rank.
Note that if the goal is to predict whether two items are similar or dissimilar via computing the relation  $\| x - x' \|_{\Sigma^{\star}}^{2} \geqslant y$ , then this problem is scale-invariant, in the sense that two items are predicted as similar (or dissimilar) according to  $(\Sigma^{\star}, y)$ , if and only if they are predicted as similar (or dissimilar)
according to  $(c_{\mathrm{scale}}\pmb{\Sigma}^{\star}, c_{\mathrm{scale}}y)$  for any scaling factor  $c_{\mathrm{scale}} > 0$ . We are thus interested in finding the equivalence class of solutions  $\{(c_{\mathrm{scale}}\pmb{\Sigma}^{\star}, c_{\mathrm{scale}}y): c_{\mathrm{scale}} > 0\}$ . Therefore, in practice, one can set  $y$  to be any positive scalar and then estimate the corresponding  $\pmb{\Sigma}^{\star}$ . Indeed, our theoretical error bounds on  $\|\widehat{\pmb{\Sigma}} - \pmb{\Sigma}^{\star}\|_F$  exhibit a natural scale-equivariant property (see Section 4, Scale Equivariance).
2.2 The perceptual adjustment query (PAQ)
We assume that every point in our feature space  $\mathbb{R}^d$  corresponds to some item. Recall from Figure 1 that a PAQ collects similarity data between a pair of items, where a reference item is fixed, and a spectrum of target items is generated from a one-dimensional path in the feature space. Denote the reference item by  $\pmb{x} \in \mathbb{R}^d$ . The target items can be generated by any path in  $\mathbb{R}^d$ , but, for simplicity, we consider straight lines. For any vector  $\pmb{a} \in \mathbb{R}^d$ , we construct the line  $\{\pmb{x} + \gamma \pmb{a} : \gamma \in [0, \infty)\}$ . We call this vector  $\pmb{a}$  the query vector. As shown in Figure 3, the user moves the slider from left to right, and the value of  $\gamma$  increases proportionally to the distance traversed by the slider. Note that the value  $\gamma$  is dimensionless.
The user is instructed to stop the slider at the transition point where the target item transitions from being similar to dissimilar with the reference item. According to our model, this transition point occurs when the  $\Sigma^{\star}$ -Mahalanobis distance between the target item and the reference item is  $y$ . The (noiseless) transition point, denoted by  $\gamma_{\star}$ , satisfies the equation
$$
y = \| \boldsymbol {x} - (\boldsymbol {x} + \gamma_ {\star} \boldsymbol {a}) \| _ {\boldsymbol {\Sigma} ^ {\star}} ^ {2} = \gamma_ {\star} ^ {2} \boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}. \tag {1}
$$
Note that the ideal PAQ response  $\gamma_{\star}$  does not depend on the specific reference item  $x$  but rather only on the query direction  $a$  and the (unknown) metric matrix  $\Sigma^{\star}$ . When querying users with PAQs, the practitioner has control over how the query vectors  $a$  are selected, which we discuss in Section 3.2.
2.3 Noise model
We model the noise in human responses as follows: In the PAQ response (1), we replace the boundary value  $y$  by  $y + \eta$ , where  $\eta \in \mathbb{R}$  represents noise. Thus the user provides a noisy response  $\gamma$  whose value satisfies  $\gamma^2\pmb{a}^\top \pmb{\Sigma}^\star \pmb{a} = y + \eta$ . Substituting in (1), we have  $\gamma^2 = \gamma_\star^2 + \frac{\eta}{\pmb{a}^\top \pmb{\Sigma}^\star \pmb{a}}$ . This model captures qualitatively how we would expect the variance of  $\gamma$  due to noise to scale. To see why, recall that  $\gamma$  is proportional to the distance traversed by the slider in the user interface Figure 1 (bottom). If  $\pmb{a}^\top \pmb{\Sigma}^\star \pmb{a}$  is large, then the semantic meaning of the item changes rapidly as the user moves the slider, and the slider will stop at a position that is close to the true transition point. On the other hand, if  $\pmb{a}^\top \pmb{\Sigma}^\star \pmb{a}$  is small, then the item changes slowly as the user moves the slider. It is then hard to determine where exactly the transition occurs, so the slider may stop in a larger interval around the transition point.
In this section, we formally present the statistical estimation problem for metric learning from noisy PAQ data, and we develop our algorithm for estimating the true metric matrix  $\pmb{\Sigma}^{\star}$ .
3.1 Statistical estimation
Assume we collect  $N$  PAQ responses, using  $N$  query vectors  $\{a_{i}\}_{i = 1}^{N}$  that we select. Denote the noise associated with these queries by random variables  $\eta_1,\dots ,\eta_N\in \mathbb{R}$ . We obtain PAQ responses, denoted by  $\gamma_{1},\ldots ,\gamma_{N}$ , that satisfy
$$
\gamma_ {i} ^ {2} \boldsymbol {a} _ {i} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a} _ {i} = y + \eta_ {i}, i = 1, \dots , N. \tag {2}
$$
We assume the noise variable  $\eta$  is independent of the query  $a$ , has zero mean and variance  $\nu_{\eta}^{2}$ , and is bounded, with  $-y \leq \eta \leq \eta^{\uparrow}$  for some constant  $\eta^{\uparrow} \geq 0$ . Note that we must have  $\eta + y \geq 0$  since  $\gamma^{2} \geq 0$ ; in addition, we place an upper bound  $\eta^{\uparrow}$  on the noise.
3In what follows, we use the terms "responses"/"measurements" interchangeably for  $\gamma$ , and the terms "query vector"/"sensing vector" interchangeably for  $a$ .
This could be relaxed by placing conditions on the conditional distributions of  $\eta$  given  $\pmb{a}$  (and even the reference point  $\pmb{x}$ ), but we omit this for simplicity.
Given the query directions  $\{\pmb{a}_i\}_{i=1}^N$  and the PAQ responses  $\{\gamma_i\}_{i=1}^N$ , we want to estimate the matrix  $\pmb{\Sigma}^{\star}$ . We first rewrite our measurement model as follows: Recall that the matrix inner product is denoted by  $\langle A, B \rangle \coloneqq \operatorname{tr}(A^\top B)$  for any two matrices  $A$  and  $B$  of compatible dimension. Then from (2), we write
$$
\gamma^ {2} = \frac {y + \eta}{\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {\star} \boldsymbol {a}}. \tag {3}
$$
Plugging (3) once more into (2), we have
$$
y + \eta = \langle \mathbf {A} ^ {\mathrm {i n v}}, \boldsymbol {\Sigma} ^ {\star} \rangle ,
$$
$$
\boldsymbol {A} ^ {\text {i n v}} := \gamma^ {2} \boldsymbol {a} \boldsymbol {a} ^ {\top} = \frac {y + \eta}{\boldsymbol {a} ^ {\top} \boldsymbol {\Sigma} ^ {*} \boldsymbol {a}} \boldsymbol {a} \boldsymbol {a} ^ {\top}. \tag {4}
$$
Hence, our problem resembles trace regression, and, in particular, low-rank matrix estimation from rank-one measurements (because the matrix  $A^{\mathrm{inv}}$  has rank 1) [31, 30, 32, 33]. We call  $A^{\mathrm{inv}}$  the sensing matrix, and  $a$  the sensing vector. Classical trace regression assumes that we make (noisy) observations of the form  $y = \langle A, \Sigma^{\star} \rangle + \epsilon$  where  $A$  is fixed before we make the measurement; in our problem, the sensing matrix  $A^{\mathrm{inv}}$  depends on our observed response  $\gamma$  and associated sensing vector  $a$ . Hence, the process of obtaining a PAQ response can be viewed as an inversion of the standard trace measurement process. The inverse nature of our problem makes estimator design more challenging, as we discuss in the following section.
As our first attempt at a procedure to estimate  $\pmb{\Sigma}^{\star}$ , we follow the literature [24, 33] and consider randomly sampling i.i.d. vectors  $\pmb{a}_i\sim \mathcal{N}(\pmb {0},\pmb {I}_d)$ . We then use standard least-squares estimation of  $\pmb{\Sigma}^{\star}$ . Since we expect  $\pmb{\Sigma}^{\star}$  to be low-rank, we add nuclear-norm regularization to promote low rank. In particular, we solve the following program:
$$
\min  _ {\boldsymbol {\Sigma} \succeq \mathbf {0}} \frac {1}{N} \sum_ {i = 1} ^ {N} \left(y - \left\langle \boldsymbol {A} _ {i} ^ {\text {i n v}}, \boldsymbol {\Sigma} \right\rangle\right) ^ {2} + \lambda_ {N} \| \boldsymbol {\Sigma} \| _ {*}, \tag {5}
$$
where  $\lambda_N > 0$  is a regularization parameter. This is a convex semidefinite program and can be solved with standard off-the-shelf solvers. However, the inverted form of our measurement model creates two critical issues when naively using (5):