Title: Perceptual adjustment queries and an inverted measurement paradigm for low-rank metric learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: human querying

high dimensional low rank matrix estimation

metric learning
Abstract: We introduce a new type of query mechanism for collecting human feedback, called the perceptual adjustment query (PAQ). Being both informative and cognitively lightweight, the PAQ adopts an inverted measurement scheme, and combines advantages from both cardinal and ordinal queries. We showcase the PAQ in the metric learning problem, where we collect PAQ measurements to learn an unknown Mahalanobis distance. This gives rise to a high-dimensional, low-rank matrix estimation problem to which standard matrix estimators cannot be applied. Consequently, we develop a two-stage estimator for metric learning from PAQs, and provide sample complexity guarantees for this estimator. We present numerical simulations demonstrating the performance of the estimator and its notable properties.

Perceptual adjustment queries and an inverted measurement paradigm for low-rank metric learning

Abstract

Weintroduce a new type of query mechanism for collecting human feedback, called the perceptual adjustment query (PAQ). Being both informative and cognitively lightweight, the PAQ adopts an inverted measurement scheme, and combines advantages from both cardinal and ordinal queries. We showcase the PAQ in the metric learning problem, where we collect PAQ measurements to learn an unknown Mahalanobis distance. This gives rise to a high-dimensional, low-rank matrix estimation problem to which standard matrix estimators cannot be applied. Consequently, we develop a two-stage estimator for metric learning from PAQs, and provide sample complexity guarantees for this estimator. We present numerical simulations demonstrating the performance of the estimator and its notable properties.

1 Introduction

Should we query cardinal or ordinal data from people? This question arises in a broad range of applications, such as in conducting surveys [1-3], grading assignments [4, 5], evaluating employees [6], and comparing or rating products [7, 8], to name a few. Cardinal data are numerical scores. For example, teachers score writing assignments in the range of 0 -100 , and survey respondents express their agreement with a statement on a scale of 1 to 7 . Ordinal data are relations between items, such as pairwise comparisons (choosing the better item in a pair) and rankings (ordering all or a subset of items). There is no free lunch, and both cardinal and ordinal queries have pros and cons.

On the one hand, collecting ordinal data is typically more efficient in terms of worker time and cognitive load [9], and surprisingly often matches or exceeds the accuracy of cardinal data [1, 9]. The information contained in ordinal queries, however, is fundamentally limited and lacks expressiveness. For example, pairwise comparisons elicit binary responses where two items are compared against each other, but the absolute placement of these items with respect to the entire pool is lost. On the other hand, cardinal data are more expressive [10]. For example, assigning two items scores of 1 and 2 conveys a very different message from assigning them scores of 9 and 10 , or 1 and 10 , although all yield the same pairwise comparison outcome. However, the expressiveness of cardinal data often comes at the cost of miscalibration: Prior work has shown that different people have different scales [11], and even a single person's scale can drift over time (e.g., [12, 13]). These inter-person and intra-person discrepancies make it challenging to interpret and aggregate raw scores effectively.

The goal of this paper is to study whether one can combine the advantages of cardinal and ordinal queries to achieve the best of both worlds. Specifically, we pose the research question:

A $2 bus ride that

Adjust slider until you no longer prefer a 10 minute taxi ride

takes40 minutes

$0

$19

Referenceitem

Adjust slider until the fruit is dissimilar to the reference

Referenceitem

Can we develop a new paradigm for human data elicitation that is expressive, accurate, and cognitively lightweight?

Towards this goal, we extract key features of both cardinal and ordinal queries, and we propose a new type of query scheme that we term the perceptual adjustment query (PAQ). As a thought experiment, consider the task of learning an individual's preferences between modes of transport. The query can take the following forms:

Ordinal: Do you prefer a $2 bus ride that takes 40 minutes or a $25 taxi that takes 10 minutes?

Cardinal: On a scale of 0 to 1 , how much do you value a $2 bus ride that takes 40 minutes?

Proposed approach: To reach the same level of preference for a $2 bus trip that takes 40 minutes, a taxi that takes 10 minutes would cost $ x .

A user interface for the proposed approach is shown in Figure 1 (top). We present the user a reference item (a $2 bus ride that takes 40 minutes) and a sliding bar representing the number of dollars ( x ) for the 10-minute taxi cost. As the user adjusts the slider, the value of x starts with 0 and gradually increases on a continuous scale. The user is instructed to place the slider at a point where they equally prefer a $2 bus ride and a taxi ride of x dollars. 2 The PAQ thus combines ordinal and cardinal elicitation in an intuitive fashion: We obtain ordinal information by asking the user to make cognitive judgments in a relative sense by comparing items, and cardinal information can be extracted from the location of the slider. The ordinal reasoning endows the query with accuracy and efficiency, while the cardinal output enables a more expressive response. Moreover, this cardinal output mitigates miscalibration, because instead of asking the user to rate on a subjective and ambiguous notion (i.e., preference), we provide the user a reference object (i.e., the $2 bus ride) to anchor their rating scale.

Beyond combining the strengths of cardinal and ordinal queries, PAQs have additional advantages that are well illustrated with the example in Figure 1 (bottom). First, PAQs provide users with the context of a specific (continuous) dimension along which items vary. For example, consider a pairwise comparison between the reference item and the 'yellow apple ' selected in Figure 1. They have similar shapes, but different colors. If these two items are shown to the user in isolation, the user lacks context to judge whether they should be considered similar or dissimilar. In contrast, the full spectrum provided in PAQs tells the user that the similarity judgment is apples vs. pears. The access to such context improves self-consistency in user responses [14]. Second, PAQs provide 'hard examples' by design and thus enable effective learning. Consider Figure 1 (bottom): Items on the left of the spectrum are apples (clearly similar to the reference), and items on the right are pears (clearly dissimilar to the reference), and only a small subset of items in the middle appear ambiguous. PAQs collect information precisely about 'confusing" items in this ambiguous region. On the other hand, if ordinal queries are constructed by selecting uniformly at random from the items shown, an item in the ambiguous region will rarely be presented to the user.

The advantages of PAQs makes their deployment appealing in a variety of problem settings. For example, practitioners may deploy PAQs to learn human preferences, like in the taxi and bus example in Figure 1 or more complex settings such as housing preferences, where multiple features (price, square footage, proximity to employment etc.) vary as the slider is moved. PAQs can also be used to learn models for human perception, such as characterizing the extent of color blindness for a given user. One can present a user with red-green color blindness a sequence of colors that slowly transitions between red and green, and ask them to drag the slider until they perceive a difference in colors. An extended discussion of applications is provided in the full version of this paper on arXiv. In this paper, we focus on learning metrics to characterize human perception. In this problem, items are represented by points in a (possibly high-dimensional) space, and the goal is to learn a distance metric such that a smaller distance between a pair of items means that they are semantically and perceptually closer, and vice versa. Figure 1 (bottom) presents a PAQ for collecting similarity data for metric learning, where the user is instructed to place the slider at the precise point where the object appears to transition from being similar to dissimilar. To construct a sequence of images as shown in Figure 1 (bottom), one can traverse a path in the latent space of a generative model - given a latent feature vector, the generative model synthesizes a corresponding image.

1.1 Do PAQs improve upon ordinal queries? A simulation vignette

Consider the problem of Mahalanobis metric learning, which forms the focus of this paper. In this setting, items are represented as points in the vector space R d , which is in turn endowed with a Mahalanobis metric parametrized by a symmetric positive semidefinite matrix Σ ⋆ ∈ R d × d . The (dis-)similarity of two items is determined by their distance under the metric: The larger the (squared) distance ∥ x -x ′ ∥ 2 Σ ⋆ = ( x -x ′ ) ⊤ Σ ⋆ ( x -x ′ ) between two items x and x ′ is, the more dissimilar the items are. We are particularly interested in the setting in which Σ ⋆ is low-rank . Established approaches in metric learning use ordinal queries, such as pairwise comparisons ('Are items x and x ′ similar?') [15-18], triplet comparisons [19] ('Which of the two items x 1 and x 2 is closer to reference item x 0 ?'), and ranking -k queries ('Given a reference item x 0 , rank the set of items x 1 , . . . , x k in terms of similarity to x 0 ') [14].

Wecompare the performance of such queries against PAQs in a toy metric learning setup. In particular, we choose a random low-rank matrix Σ ⋆ in dimension 50 with rank 10 (see Appendix B for our precise construction) and use the models and state-of-the-art algorithms of [19, 14] to produce pairwise, triplet, and rankingk queries and estimate the low-rank metric. In addition to these ordinal queries, we simulate PAQ responses under the model presented in Section 2 and use our algorithm (see Section 3) for estimation. To simplify the example, all query responses are generated in a noiseless fashion-for example, the triplet comparison always returns the closer item to the reference. We present our results in Figure 2, which illustrates a significant gap in information richness between PAQs and a variety of ordinal queries. The number of PAQ responses needed to attain a reasonable normalized error is dramatically lower than those of typical ordinal queries, illustrating that PAQs can greatly improve upon the performance of existing ordinal queries for metric learning. The rest of our paper explores this opportunity: It aims to make the deployment of PAQs theoretically grounded by designing provable methodology for learning a low-rank metric from PAQ responses.

1.2 Our contributions and organization

In addition to introducing the perceptual adjustment query (PAQ), we demonstrate its applicability to metric learning under a Mahalanobis metric. We first present a mathematical formulation of this estimation problem in Section 2. We then show that the sliding bar response can be viewed as an inverted measurement of the metric matrix that we want to estimate, which allows us to restate our problem as that of estimating a low-rank matrix from a specific type of trace measurement (Section 3). However, our PAQ formulation differs from classical matrix estimation due to two technical challenges: (a) the sensing matrices and noise are correlated, and (b) the sensing matrices are heavy-tailed. As a result, standard matrix estimation algorithms give rise to biased estimators . We propose a query procedure and an estimator that overcome these two challenges, and we prove statistical error bounds on the estimation error (Section 4). The unconventional nature of the sensing model and estimator causes unexpected behaviors in our error bounds; in Section 5, we present simulations verifying that these behaviors also appear in practice.

Errorvs.numberofnoiselessresponses

PAQS

二

Ranking-4

estimationerror

1.2

Pairwise

comparisons

Ranking-5

1.0

Triplets

0.8

Normalized

0.6

0.4

0.2

0.0

0

500

1000

1500

2000

2500

Numberofresponses

1.3 Related work

We discuss related work on metric learning and the statistical techniques that we use.

Metric learning. As discussed in Section 1.1, prior work in metric learning [20] considers a wide variety of ordinal queries. PAQ can be viewed as extending a discrete set of items presented to users to a continuous spectrum, which is natural when one uses a generative model such as a GAN [21, 22]. However, the goal of tuple queries is to rank the items, whereas in PAQ the ranking is provided by the feature space and we ask people to identify a transition point (similar vs. dissimilar) in this ranking.

Statistical techniques. In our theoretical results, we apply techniques from the high-dimensional statistics literature. Our theoretical formulation (presented in Section 3) resembles the problem of low-rank matrix estimation from trace measurements (e.g., [23-28]; see [29] for a more complete overview), and, in particular, when the sensing matrix is of rank one [30-33]. However, as discussed in Section 3, our model presents two important distinctions from prior literature. In our case, the sensing matrices are both heavy-tailed and correlated with the measurement noise. The heavy-tailed matrices violate the assumptions of much prior work that relies on sub-Gaussian or sub-exponential assumptions on the sensing matrices. Prior work has attempted to address the challenge of heavy tails with methods such as robust loss functions [34, 35] or the 'median-of-means' approach [36-38]. We draw particular inspiration from Fan et al. [39], which considers truncation to control heavy-tailed behavior for a variety of problems. However, in the low-rank matrix estimation setting, Fan et al. [39] only analyze the case of heavy-tailed noise under a sub-Gaussian design, meaning their methodology and results are not applicable to our problem setting.

2 Model

In this section, we present our model for the perceptual adjustment query (PAQ) in the context of its application to metric learning.

2.1 Mahalanobis metric learning

We consider a d -dimensional feature space where each item is represented by a point in R d . The distance metric model for human similarity perception posits that there is a metric on R d that measures how dissimilar items are perceived to be. A recent line of work [40, 41] has modeled the distance metric as a Mahalanobis metric. If Σ ⋆ ∈ R d × d is a symmetric positive semidefinite (PSD) matrix, the squared Mahalanobis distance with respect to Σ ⋆ between items x and x ′ ∈ R d is ∥ x -x ′ ∥ 2 Σ ⋆ := ( x -x ′ ) ⊤ Σ ⋆ ( x -x ′ ) . The distance represents the extent of dissimilarity between items x and x ′ : If we further have a perceptual boundary value y > 0 , this model posits that items x , x ′ are perceived as similar if ∥ x -x ′ ∥ 2 Σ ⋆ < y and dissimilar if ∥ x -x ′ ∥ 2 Σ ⋆ ≥ y . We adopt a high-dimensional framework and, following [19, 41], assume that the matrix Σ ⋆ is low-rank.

Note that if the goal is to predict whether two items are similar or dissimilar via computing the relation ∥ x -x ′ ∥ 2 Σ ⋆ ≷ y , then this problem is scale-invariant, in the sense that two items are predicted as similar (or dissimilar) according to ( Σ ⋆ , y ) , if and only if they are predicted as similar (or dissimilar)

PAQ

response

Query

vectora

Reference

DL+

item

according to ( c scale Σ ⋆ , c scale y ) for any scaling factor c scale > 0 . We are thus interested in finding the equivalence class of solutions { ( c scale Σ ⋆ , c scale y ) : c scale > 0 } . Therefore, in practice, one can set y to be any positive scalar and then estimate the corresponding Σ ⋆ . Indeed, our theoretical error bounds on ∥ ̂ Σ -Σ ⋆ ∥ F exhibit a natural scale-equivariant property (see Section 4, Scale Equivariance).

2.2 The perceptual adjustment query (PAQ)

We assume that every point in our feature space R d corresponds to some item. Recall from Figure 1 that a PAQ collects similarity data between a pair of items, where a reference item is fixed, and a spectrum of target items is generated from a one-dimensional path in the feature space. Denote the reference item by x ∈ R d . The target items can be generated by any path in R d , but, for simplicity, we consider straight lines. For any vector a ∈ R d , we construct the line { x + γ a : γ ∈ [0 , ∞ ) } . We call this vector a the query vector . As shown in Figure 3, the user moves the slider from left to right, and the value of γ increases proportionally to the distance traversed by the slider. Note that the value γ is dimensionless .

The user is instructed to stop the slider at the transition point where the target item transitions from being similar to dissimilar with the reference item. According to our model, this transition point occurs when the Σ ⋆ -Mahalanobis distance between the target item and the reference item is y . The (noiseless) transition point, denoted by γ ⋆ , satisfies the equation

Note that the ideal PAQ response γ ⋆ does not depend on the specific reference item x but rather only on the query direction a and the (unknown) metric matrix Σ ⋆ . When querying users with PAQs, the practitioner has control over how the query vectors a are selected, which we discuss in Section 3.2.

2.3 Noise model

We model the noise in human responses as follows: In the PAQ response (1), we replace the boundary value y by y + η , where η ∈ R represents noise. Thus the user provides a noisy response γ whose value satisfies γ 2 a ⊤ Σ ⋆ a = y + η . Substituting in (1), we have γ 2 = γ 2 ⋆ + η a ⊤ Σ ⋆ a . This model captures qualitatively how we would expect the variance of γ due to noise to scale. To see why, recall that γ is proportional to the distance traversed by the slider in the user interface Figure 1 (bottom). If a ⊤ Σ ⋆ a is large, then the semantic meaning of the item changes rapidly as the user moves the slider, and the slider will stop at a position that is close to the true transition point. On the other hand, if a ⊤ Σ ⋆ a is small, then the item changes slowly as the user moves the slider. It is then hard to determine where exactly the transition occurs, so the slider may stop in a larger interval around the transition point.

3 Methodology

In this section, we formally present the statistical estimation problem for metric learning from noisy PAQ data, and we develop our algorithm for estimating the true metric matrix Σ ⋆ .

3.1 Statistical estimation

Assume we collect N PAQ responses, using N query vectors { a i } N i =1 that we select 3 . Denote the noise associated with these queries by random variables η 1 , . . . , η N ∈ R . We obtain PAQ responses, denoted by γ 1 , . . . , γ N , that satisfy

We assume the noise variable η is independent 4 of the query a , has zero mean and variance ν 2 η , and is bounded, with -y ≤ η ≤ η ↑ for some constant η ↑ ≥ 0 . Note that we must have η + y ≥ 0 since γ 2 ≥ 0 ; in addition, we place an upper bound η ↑ on the noise.

Given the query directions { a i } N i =1 and the PAQ responses { γ i } N i =1 , we want to estimate the matrix Σ ⋆ . We first rewrite our measurement model as follows: Recall that the matrix inner product is denoted by ⟨ A , B ⟩ := tr ( A ⊤ B ) for any two matrices A and B of compatible dimension. Then from (2), we write

Plugging (3) once more into (2), we have

Hence, our problem resembles trace regression, and, in particular, low-rank matrix estimation from rank-one measurements (because the matrix A inv has rank 1 ) [31, 30, 32, 33]. We call A inv the sensing matrix, and a the sensing vector. Classical trace regression assumes that we make (noisy) observations of the form y = ⟨ A , Σ ⋆ ⟩ + ϵ where A is fixed before we make the measurement; in our problem, the sensing matrix A inv depends on our observed response γ and associated sensing vector a . Hence, the process of obtaining a PAQ response can be viewed as an inversion of the standard trace measurement process. The inverse nature of our problem makes estimator design more challenging, as we discuss in the following section.

3.2 Algorithm

As our first attempt at a procedure to estimate Σ ⋆ , we follow the literature [24, 33] and consider randomly sampling i.i.d. vectors a i ∼ N ( 0 , I d ) . We then use standard least-squares estimation of Σ ⋆ . Since we expect Σ ⋆ to be low-rank, we add nuclear-norm regularization to promote low rank. In particular, we solve the following program:

where λ N > 0 is a regularization parameter. This is a convex semidefinite program and can be solved with standard off-the-shelf solvers. However, the inverted form of our measurement model creates two critical issues when naïvely using (5):

̸

Bias of standard matrix estimators due to dependence. Note that the sensing matrix (4) depends on the noise η . Quantitatively, we have E [ η A inv ] = 0 (see Appendix D.1). Standard trace regression analyses require that this quantity be zero, typically assuming (at least) that η is zeromean conditioned on the sensing matrix A . The failure of this to hold in our case introduces a bias that does not decrease with the sample size N .

Heavy-tailed sensing matrix. The factor 1 a ⊤ Σ ⋆ a in A inv (see Equation (4)) makes A inv heavytailed in general. When a is Gaussian, the term 1 a ⊤ Σ ⋆ a is an inverse weighted chi-square random variable, whose higher-order moments are infinite (and the number of finite moments depends on the rank of Σ ⋆ ). This makes error analysis more difficult, as standard analyses require the sensing matrix A to concentrate well (e.g., be sub-exponential).

To overcome these challenges, we make two key modifications to the procedure (5).

Step 1: Bias reduction via averaging. First, we want to mitigate the bias due to the dependence between the sensing matrix A inv and the noise η . The bias term E [ η A inv ] scales proportionally to E [ η ( y + η )] = E [ η 2 ] . Therefore, to reduce this bias in the least-squares estimator (5), we need to reduce the noise variance. We reduce the effective noise variance (and hence the bias) by averaging i.i.d. samples. Operationally, instead of obtaining N measurements from N distinct sensing vectors { a i } N i =1 , we draw n sensing vectors { a i } n i =1 , and collect m measurements, denoted by { γ ( j ) i } j m =1 , corresponding to each sensing vector a i . We refer to n as the number of (distinct) sensing vectors.

where

Algorithm 1 Inverted measurement sensing, averaging, and truncation.

Input: number of total measurements N , averaging parameter m (that divides N ), truncation threshold τ , measurement value y

1: Compute the number of sensing vectors n = N m

2: for i = 1 to n do

3: Draw sensing vector a i from standard multivariate normal distribution

4: Obtain m PAQ measurements ( γ (1) i ) 2 , . . . , ( γ ( m ) i ) 2 with a i and y .

5: Bias elimination via averaging: compute averaged response ¯ γ 2 i = 1 m ∑ m j =1 ( γ ( j ) i ) 2 .

6: Heavy tail mitigation via truncation: compute truncated response ˜ γ 2 i = ¯ γ 2 i ∧ τ.

7: end for

Output: truncated responses ˜ γ 2 1 , . . . ˜ γ 2 n

To keep the total number of measurements constant, we set n = N m , where the value of m is specified later. For each sensing vector a i , we compute the empirical mean of the m measurements:

where we define the average noise by ¯ η i := 1 m ∑ m j =1 η ( j ) i . This averaging operation reduces the effective noise variance from var( η i ) = ν 2 η to var(¯ η i ) = ν 2 η m . If n is small, we may have large error due to an insufficient number of query vectors a i . On the other hand, a small m leads to a large bias. Therefore, we set the value of m carefully to balance these two effects. This is studied theoretically in Section 4 and demonstrated empirically in Section 5.

Step 2: Heavy tail mitigation via truncation. Next, we need to control the heavy-tailed behavior introduced by the 1 a ⊤ Σ ⋆ a term in the sensing matrix A inv . Note that the sample averaging procedure (6) does not mitigate this problem. We adopt the approach in [39] and truncate the observations. Specifically, we truncate the averaged measurements ¯ γ 2 i to ˜ γ 2 := ¯ γ 2 ∧ τ , where τ > 0 is a truncation threshold that we specify later. We then construct the truncated sensing matrices

While truncation mitigates heavy-tailed behavior, it also introduces additional bias in our estimate. The truncation threshold τ therefore gives us another tradeoff, and in our analysis to follow, we carefully set the value of τ to balance the effects of heavy-tailedness and bias.

Final algorithm. Before presenting our final optimization program, we summarize our assumptions and sensing model below.

Assumption 1. The noise values η i are i.i.d copies of the random variable η , which is independent of the random sensing vector a . The random noise is (1) zero-mean: E [ η ] = 0 , and (2) bounded: There exists a positive constant η ↑ such that -y ≤ η ≤ η ↑ with probability 1 .

We choose the sensing vector distribution to be the standard multivariate normal distribution and collect, average, and truncate N PAQ responses following Algorithm 1. This process yields n truncated responses ˜ γ 2 1 , . . . ˜ γ 2 n . We then use these truncated responses to form the averaged and truncated matrices { ˜ A i } n i =1 , which we substitute into the original least-squares problem (5). To estimate Σ ⋆ , we solve

where, again, λ n is a regularization parameter that we specify later.

Practical considerations. In the averaging step, we collect m measurements for each sensing vector a i . These measurements could be collected from m different users. Furthermore, recall from Section 2.2 that the measurements do not depend on the reference item x . As a result, one may

also collect multiple responses from the same user by presenting the same query vector a i with different reference items. In addition, recall from Section 2.1 that user responses are scale-invariant. Practitioners are hence free to set the boundary y to be any positive value of their choice without loss of generality, and the noise η scales accordingly with y . The user interface does not depend on the value of y .

4 Theoretical results

We now present our main theoretical result, which is a finite-sample error bound for estimating a low-rank metric from inverted measurements with the nuclear norm regularized estimator (8). Our error bound is generally stated, and depends on the averaging parameter m and the truncation threshold τ . Recall that ν 2 η denotes the variance of η . We define the quantities y ↑ := y + η ↑ and µ y = y + median ( η ) . We further denote by σ 1 ≥ · · · ≥ σ r > 0 the non-zero singular values of Σ ⋆ .

Theorem 1. Suppose Σ ⋆ is rank r , with r > 8 . Assume that we choose the sensing vector distribution the be the standard multivariate normal distribution, that Assumption 1 holds on the noise, and that we collect, average, and truncate measurements following Algorithm 1. Further, assume that the truncation threshold τ satisfies τ ≥ µ y tr( Σ ⋆ ) . Then there are positive constants c, C, C 1 , and C 2 , such that if the regularization parameter and the number of sensing vectors satisfy

then any solution ̂ Σ to the optimization program (8) satisfies

with probability at least 1 -4 e -d -e -cn .

The proof of Theorem 1 is presented in Appendix E. The two sources of bias discussed in Section 3.2 appear in the expression (9) for the regularization parameter λ n (and consequently in the error bound (10)). The term scaling as 1 /τ corresponds to the bias induced by truncation, and decreases as the truncation gets milder (i.e., as the threshold τ gets larger). The term scaling as ν 2 η /m corresponds to the bias arising from dependence between the noise and sensing matrix. As discussed in Section 3.2, in this model, m -averaging results in a bias that scales like 1 /m . Given the dependence of the estimation error bound on the parameters m and τ , we carefully set these parameters to obtain a tight bound as a function of the number of total measurements N = mn . These choices for m and τ , along with the final estimation error, are presented in the following corollary, proved in Appendix F.

Corollary 1. Recall that N = mn . Assume that the conditions of Theorem 1 hold, and set the values of the constants ( c, C, C 1 , C 2 ) according to Theorem 1. Suppose that the number of total measurements satisfies

Set the averaging parameter m and truncation threshold τ to be

and set λ n equal to its lower bound in (9) . With probability at least 1 -4 e -d -e -cN/m , we have:

(a) If ν 2 η ( y ↑ ) 2 > √ d N , then any solution ̂ Σ to the optimization program (8) satisfies

(b) If ν 2 η ( y ↑ ) 2 ≤ √ d N , then any solution ̂ Σ to the optimization program (8) satisfies

In both cases, C ′ = 3 C · C 1 .

A few comments are warranted about our error bounds (13) and (14):

Error rates and noise regimes. Under the standard trace measurement model, it is known that if the measurement matrices are i.i.d. according to some sub-Gaussian distribution and the number of measurements satisfies N ≳ rd , then nuclear norm regularized estimators achieve an error that scales like √ rd N (e.g., [24, 25]). Such a result is also known to be minimax optimal [25]. Allowing heavier-tailed assumptions on the sensing matrices, such as sub-exponential [32, 42] or bounded fourth moment [39], typically results in additional log d factors but does not impact the exponent 1 / 2 in the error rate. However, a crucial assumption in these results is that E [ η A inv ] = 0 , and thus there is no bias due to measurement noise. Our inverted measurement sensing matrix is not only heavy-tailed but also leads to bias (see Lemma 1 in Appendix D.1). Nevertheless, we are able to reduce the bias and trade it for variance, ensuring consistent estimation in all regimes.

In Corollary 1, there are two distinct cases for error rate which correspond to two different noise regimes induced by the quantity ν 2 η / ( y ↑ ) 2 , which captures the noise level in our measurements. In particular, the two cases in Corollary 1 correspond to two regimes with distinct bias behavior:

(a) High-noise regime: In this setting, the bias due to measurement noise is non-negligible. As a result, we employ averaging with large m , which results in the rate scaling as ( d/N ) 1 / 3 .

(b) Low-noise regime: In this setting, the measurement noise bias is dominated by the variance, and thus has negligible impact on the estimation error. As a result, we are able to achieve a rate of order ( d/N ) 1 / 2 , which is consistent with established results for low-rank matrix estimation.

Sample complexity. Since the degrees of freedom in a rankr matrix of size d × d is of order rd , one expects that the minimum number of measurements to identify a rankr matrix is of order rd . This is reflected in Theorem 1, which assumes that the number of distinct sensing vectors { a i } satisfies n ≳ rd . In the high-noise regime, from (12) in Corollary 1, we have that m scales like ( N/d ) 1 / 3 . Thus, the total number of measurements is N = mn ≳ ( N/d ) 1 / 3 · rd ≳ N 1 / 3 d 2 / 3 r , and hence N ≳ r 3 / 2 d . Given that the rank is assumed to be relatively small compared to the dimension, the extra factor of √ r is a relatively small price to pay to obtain consistent estimation. In the low-noise regime, it can be verified that m = 1 in (12) due to the low-noise condition ν 2 η / ( y ↑ ) 2 ≤ √ d/N . No averaging is needed, and we only require N = n ≳ rd .

Dependence on rank. When compared to standard results, Corollary 1 differs in its dependence on rank. First, the matrix Σ ⋆ is assumed to have rank r > 8 . This prevents the term 1 a ⊤ Σ ⋆ a from making the sensing matrices so heavy-tailed that even truncation does not help. We empirically show that the assumption of r > 8 is necessary in Section 5. Second, there is an additional factor of r in our rate for both noise regimes. To interpret this, note that if Σ ⋆ has non-zero singular values in a fixed range, then E [ a ⊤ Σ ⋆ a ] = tr ( Σ ⋆ ) ≈ r . Since the 'magnitude" of the sensing matrix A inv is inversely proportional to a ⊤ Σ ⋆ a , increasing r decreases the magnitude of A inv and thus also (for a fixed noise level) the signal-to-noise ratio.

Scale equivariance. As discussed in Section 2.1, the metric learning from PAQs problem aims to find an equivalence class { ( c scale Σ , c scale y ) : c scale > 0 } , and the ground-truth Σ ⋆ is defined with respect to a particular choice of y . Accordingly, our error bounds are scale-equivariant: If we instead replaced y with c scale y , the bounds (13) and (14) would scale linearly in c scale . This fact is precisely verified in Appendix C and relies on the fact that the noise also scales appropriately in c scale . Thus practitioners may simply set y to be any positive number.

(a)

(b)

(c)

g10)

[601)

-0.5

0.0

d = 40

d = 50

d = 60

(01601)

0.5

0.0

r=8

S = J

6 = J

r = 10

r = 15

r = 20

0.8

二

(N, d) = (16k, 40), N/d = 400

(N, d) = (40k, 40), N/d =1000

(N, d) = (20k, 50), N/d = 400

(N, d) = (50k, 50), N/d = 1000

error

-1.0

error

-1.0

est.

Min. error, N/d = 400

est.

1.5

est.

一

1.5

Normalized

0.4

Normalized

-2.0

zed

N

-2.0

Min. error, N/d = 1000

2.5

Norma

一

2.5

0.2

3.003.253.503.754.004.254.50

-3.0

5.00

5.25

5.50

5.75

6.00

6.25

0.0

0

10

20

30

40

50

Normalized total measurements N/d (log10)

Number of total measurements(log10)

Averagingparameterm

5 Numerical simulations

In this section, we provide numerical simulations investigating the effects of the various problem and estimation parameters. For all results, we report the normalized estimation error ∥ ̂ Σ -Σ ⋆ ∥ F / ∥ Σ ⋆ ∥ F averaged over 20 trials. Shaded areas (sometimes not visible) represent standard error of the mean. For all experiments, we follow [19] and generate the ground-truth metric matrix as Σ ⋆ = d √ r UU ⊤ , where U ∈ R d × r is a randomly generated matrix with orthonormal columns. The noise η is sampled from a uniform distribution on [ -η ↑ , η ↑ ] (where η ↑ ≤ y ). We set the regularization parameter, truncation threshold, and averaging parameter in a manner consistent with our theoretical results (see Eqs. (9) and (12)), cross-validating to choose the constant factors. We solve the optimization problem using cvxpy [43, 44]. Code for all simulations is provided at https://github.com/austinxu87/paq .

Effects of dimension and rank. Our first set of experiments characterizes the effects of dimension d and matrix rank r . For all experiments, unless we are sweeping a specific parameter, we set y = 200 , d = 50 , r = 15 , and η ↑ = 10 . Fig. 4a shows the performance for varying values of d plotted against the normalized sample size N/d . For all dimensions d , the error decays to zero as the total number of measurements N increases. Furthermore, the error curves are well-aligned when the sample size is normalized by d with fixed r , empirically aligning with Corollary 1. Fig. 4b shows the performance for varying values of rank r . Recall that for our theoretical results we assume r > 8 to ensure that the quadratic term a ⊤ Σ ⋆ a in the denominator of our sensing matrices does not lead to excessively heavy-tailed behavior. When r > 8 , the number of measurements required for the same estimation error increases as the rank increases. A clear phase transition occurs at r = 8 . The error still decreases with N for r ≤ 8 , but at a markedly slower rate than when r > 8 . This empirically demonstrates that when r ≤ 8 , the sensing matrix tails are too heavy to be mitigated by truncation.

Effect of averaging parameter m . Equation (12) suggests that the averaging parameter m should scale proportionally to ( N/d ) 1 / 3 . To test this, we set y = 200 , d = 50 , r = 9 , and η ↑ = 200 . We vary values of m for different choices of the ( N,d ) pair, as shown in Fig. 4c. The empirically optimal choice of m is observed to be the same when N/d is fixed, regardless of the particular choices of N or d (the green and red curves overlap, and the blue and orange curves overlap). Moreover, the optimal m is smaller when N/d = 400 compared to when N/d = 1000 .

6 Discussion

In this paper we introduced the perceptual adjustment query, a cognitively lightweight way to obtain expressive responses from humans. We specifically investigate using PAQs for human perceptual similarity learning. We use a Mahalanobis distance-based model for human similarity perception and use PAQs to estimate the unknown metric. Using random measurements to learn an unknown Mahalanobis metric gives rise to the new inverted measurement scheme for high dimensional low rank matrix estimation which violates commonly held assumptions for existing estimators. We developed a two-stage estimator and provide corresponding sample complexity guarantees. This work opens up many interesting lines of future work in inverted measurements. One key direction is to characterize their fundamental limits with information-theoretic lower bounds.

Acknowledgments

AX was supported by National Science Foundation grant CCF-2107455. JW was supported by the Ronald J. and Carol T. Beerman President's Postdoctoral Fellowship and the Algorithms and Randomness Center Postdoctoral Fellowship at Georgia Tech. MD was supported by National Science Foundation grants CCF-2107455, IIS-2212182, and DMS-2134037. AP was supported in part by the National Science Foundation grants CCF-2107455 and DMS-2210734, and by research awards/gifts from Adobe, Amazon and Mathworks.

References

William L. Rankin and Joel W. Grube. A comparison of ranking and rating procedures for value system measurement. European Journal of Social Psychology , 10(3):233-246, 1980.

Anne-Wil Harzing, Joyce Baldueza, Wilhelm Barner-Rasmussen, Cordula Barzantny, Anne Canabal, Anabella Davila, Alvaro Espejo, Rita Ferreira, Axele Giroud, Kathrin Koester, YungKuei Liang, Audra Mockaitis, Michael J. Morley, Barbara Myloni, Joseph O.T. Odusanya, Sharon Leiba O'Sullivan, Ananda Kumar Palaniappan, Paulo Prochno, Srabani Roy Choudhury, Ayse Saka-Helmhout, Sununta Siengthai, Linda Viswat, Ayda Uzuncarsili Soydas, and Lena Zander. Rating versus ranking: What is the best way to reduce response and language bias in cross-national research? International Business Review , 18(4):417-432, 2009. ISSN 0969-5931.

Georgios N. Yannakakis and John Hallam. Ranking vs. preference: A comparative study of self-reporting. In Sidney D'Mello, Arthur Graesser, Björn Schuller, and Jean-Claude Martin, editors, Affective Computing and Intelligent Interaction , pages 437-446, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg. ISBN 978-3-642-24600-5.

Nihar B. Shah, Joseph K Bradley, Abhay Parekh, Martin Wainwright, and Kannan Ramchandran. A case for ordinal peer-evaluation in MOOCs. In NIPS Workshop on Data Driven Education , 2013.

Karthik Raman and Thorsten Joachims. Methods for ordinal peer grading. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , volume 20, 2014.

Richard D. Goffin and James M. Olson. Is it all relative? Comparative judgments and the possible improvement of self-ratings and ratings of others. Perspectives on Psychological Science , 6(1):48-60, 2011.

William Barnett. The modern theory of consumer behavior: Ordinal or cardinal? The Quarterly Journal of Austrian Economics , 6(1):41-65, 2003.

Richard Batley. On ordinal utility, cardinal utility and random utility. Theory and Decision , 64: 37-63, 2008.

Nihar B. Shah, Sivaraman Balakrishnan, Joseph Bradley, Abhay Parekh, Kannan Ramchandran, and Martin J. Wainwright. Estimation from pairwise comparisons: Sharp minimax bounds with topology dependence. Journal of Machine Learning Research , 17(58):1-47, 2016.

Jingyan Wang and Nihar B. Shah. Your 2 is my 1, your 3 is my 9: Handling arbitrary miscalibrations in ratings. In Proceedings of the 18th International Conference on Autonomous Agents and Multiagent Systems , 2019.

Dale Griffin and Lyle Brenner. Perspectives on Probability Judgment Calibration , chapter 9. Wiley-Blackwell, 2008. ISBN 9780470752937.

Polina Harik, Brian Clauser, Irina Grabovsky, Ronald Nungester, David Swanson, and Ratna Nandakumar. An examination of rater drift within a generalizability theory framework. Journal of Educational Measurement , 46:43-58, 2009.

Carol M. Myford and Edward W. Wolfe. Monitoring rater performance over time: A framework for detecting differential accuracy and differential scale category use. Journal of Educational Measurement , 46(4):371-389, 2009.

Gregory Canal, Stefano Fenu, and Christopher Rozell. Active ordinal querying for tuplewise similarity learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, 2020.

Yiming Ying, Kaizhu Huang, and Colin Campbell. Sparse metric learning via smooth optimization. Advances in Neural Information Processing Systems , 22, 2009.

Wei Bian and Dacheng Tao. Constrained empirical risk minimization framework for distance metric learning. IEEE Transactions on Neural Networks and Learning Systems , 23(8):11941205, 2012.

Zheng-Chu Guo and Yiming Ying. Guaranteed classification via regularized similarity learning. Neural Computation , 26(3):497-522, 2014.

Aurélien Bellet and Amaury Habrard. Robustness and generalization for metric learning. Neurocomputing , 151:259-267, 2015.

Blake Mason, Lalit Jain, and Robert Nowak. Learning low-dimensional metrics. In Advances in Neural Information Processing Systems , volume 30, 2017.

Aurélien Bellet, Amaury Habrard, and Marc Sebban. Metric Learning . Springer Nature, 2022.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems (NeurIPS) , 27, 2014.

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8110-8119, 2020.

Benjamin Recht, Maryam Fazel, and Pablo A Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM review , 52(3):471-501, 2010.

Sahand Negahban and Martin J Wainwright. Estimation of (near) low-rank matrices with noise and high-dimensional scaling. The Annals of Statistics , 39(2):1069-1097, 2011.

A Tsybakov and A Rohde. Estimation of high-dimensional low-rank matrices. The Annals of Statistics , 39(2):887-930, 2011.

Emmanuel J Candes and Yaniv Plan. Tight oracle inequalities for low-rank matrix recovery from a minimal number of noisy random measurements. IEEE Transactions on Information Theory , 57(4):2342-2359, 2011.

Sahand N Negahban, Pradeep Ravikumar, Martin J Wainwright, and Bin Yu. A unified framework for high-dimensional analysis of M -estimators with decomposable regularizers. Statistical Science , 27(4):538-557, 2012.

T Tony Cai and Anru Zhang. Sparse representation of a polytope and recovery of sparse signals and low-rank matrices. IEEE Transactions on Information Theory , 60(1):122-132, 2013.

Mark A Davenport and Justin Romberg. An overview of low-rank matrix recovery from incomplete observations. IEEE Journal of Selected Topics in Signal Processing , 10(4):608-622, 2016.

Yuxin Chen, Yuejie Chi, and Andrea J Goldsmith. Exact and stable covariance estimation from quadratic sampling via convex programming. IEEE Transactions on Information Theory , 61(7): 4034-4059, 2015.

T Tony Cai and Anru Zhang. ROP: Matrix recovery via rank-one projections. The Annals of Statistics , 43(1):102-138, 2015.

Richard Kueng, Holger Rauhut, and Ulrich Terstiege. Low rank matrix recovery from rank one measurements. Applied and Computational Harmonic Analysis , 42(1):88-116, 2017.

Andrew D McRae, Justin Romberg, and Mark A Davenport. Optimal convex lifted sparse phase retrieval and PCA with an atomic matrix norm regularizer. IEEE Transactions on Information Theory , 69(3):1866-1882, 2022.

Po-Ling Loh. Statistical consistency and asymptotic normality for high-dimensional robust M -estimators. The Annals of Statistics , 45(2):866-896, 2017. doi: 10.1214/16-AOS1471.

Jianqing Fan, Quefeng Li, and Yuyan Wang. Estimation of high dimensional mean regression in the absence of symmetry and light tail assumptions. Journal of the Royal Statistical Society Series B, Statistical methodology , 79(1):247-265, 2016.

Arkadij Semenoviˇ c Nemirovskij and David Borisovich Yudin. Problem complexity and method efficiency in optimization . Wiley-Interscience, 1983.

Stanislav Minsker. Geometric median and robust estimation in Banach spaces. Bernoulli , 21(4): 2308-2335, 2015.

Daniel Hsu and Sivan Sabato. Loss minimization and parameter estimation with heavy tails. Journal of Machine Learning Research , 17(1):543-582, 2016.

Jianqing Fan, Weichen Wang, and Ziwei Zhu. A shrinkage principle for heavy-tailed data: High-dimensional robust low-rank matrix recovery. The Annals of Statistics , 49(3):1239-1266, 2021.

Austin Xu and Mark Davenport. Simultaneous preference and metric learning from paired comparisons. Advances in Neural Information Processing Systems , 33, 2020.

Gregory Canal, Blake Mason, Ramya Korlakai Vinayak, and Robert Nowak. One for all: Simultaneous metric and preference learning over multiple users. In Advances in Neural Information Processing Systems , volume 35, 2022.

Sahand Negahban and Martin J Wainwright. Restricted strong convexity and weighted matrix completion: Optimal bounds with noise. Journal of Machine Learning Research , 13(1):16651697, 2012.

Steven Diamond and Stephen Boyd. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research , 17(83):1-5, 2016.

Akshay Agrawal, Robin Verschueren, Steven Diamond, and Stephen Boyd. A rewriting system for convex optimization problems. Journal of Control and Decision , 5(1):42-60, 2018.

Andrew K Massimino and Mark A Davenport. As you like it: Localization via paired comparisons. Journal of Machine Learning Research , 22(1):8357-8395, 2021.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration Inequalities: A Nonasymptotic Theory of Independence . Oxford University Press, 2013. ISBN 9780199535255. doi: 10.1093/acprof:oso/9780199535255.001.0001.

Yong Bao and Raymond Kan. On the moments of ratios of quadratic forms in normal random variables. Journal of Multivariate Analysis , 117:229-245, 2013.

Shahar Mendelson. Learning without concentration. Journal of the ACM (JACM) , 62(3):1-25, 2015.

Joel A Tropp. Convex recovery of a structured signal from independent random linear measurements. Sampling Theory, a Renaissance: Compressive Sensing and Other Developments , pages 67-101, 2015.

Mark Rudelson and Roman Vershynin. Hanson-Wright inequality and sub-gaussian concentration. Electronic Communications in Probability , 18:1-9, 2013.

Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027 , 2010.

A Broader impacts and limitations

In this section, we discuss the broader impacts and limitations of our work.

A.1 Broader impacts

Automated systems that use human feedback are being used in an increasing number of contexts, spanning everything from predicting user preferences to finetuning language models. It is important to ensure that such systems are as accurate as possible; this naturally requires humans to respond in an accurate and consistent manner. Using the perceptual adjustment query to collect data in such settings could lead to more expressive responses without heavy cognitive burdens on users. Furthermore, providing a user the additional context of a continuous spectrum of items may result in more selfconsistent responses. The downstream effects of collecting more expressive and self-consistent human responses could lead to improved models or entirely new paradigms of model development for a myriad of problem settings. With these advantages come associated risks as well. Due to how expressive the responses to PAQs are, the effects of adversarial responses may be magnified. That is, if an adversary purposely chooses to respond in an antagonistic manner, models trained with PAQs may be trained poorly or in opposition to the stated goal. Mitigating such effects likely requires a holistic approach from both the query design and robust model design perspectives.

A.2 Limitations

From a data collection perspective, the perceptual adjustment query requires access to a continuous space where each point corresponds to an item. In many applications, assuming access to this continuous space is reasonable. For example, if we use PAQs to characterize color blindness, then a natural continuous space is the RGB color space. In general, we situate our data collection within the latent space of a generative model, such as a GAN. While GANs are capable of producing extremely high fidelity images, these images are not always free of semantically meaningful artifacts. Our query design and modeling assumptions do not explicitly consider the case where a portion of the continuous spectrum of items may be corrupted. Furthermore, because our work is an initial exploration into low-rank matrix estimation from inverted measurements, we have not considered scenarios such as unbounded noise or heavier-tailed sensing vectors, and we have not established information-theoretic lower bounds for the inverted measurement paradigm. We hope that further exploration of the inverted measurement paradigm will lead to a rich line of follow-up work.

B Simulation details

In this section, we provide details for the simulation results presented in Figure 2. For our experiments, we adopt a normalized version of the setup of [30] and form the metric Σ ⋆ by Σ ⋆ = LL ⊤ / ∥ LL ⊤ ∥ F , where L is a 50 × 10 matrix with i.i.d. Gaussian entries. We sweep the number of query responses N , estimate the metric with ̂ Σ , and report the normalized estimation error ∥ Σ ⋆ -̂ Σ ∥ F / ∥ Σ ⋆ ∥ F averaged over 10 independent trials. For each query response, items are drawn i.i.d. from a standard multivariate normal distribution, similar to [45].

Pairwise comparison setup. For pairwise comparisons, we use value of y = 10 to denote the squared distance at which items become dissimilar, following our distance-based model for human perception (see Section 2.1). For the i -th pairwise comparisons, we draw two items x ( i ) 1 , x ( i ) 2 i.i.d. from a standard multivariate normal distribution. We record the pairwise comparison outcome ϵ i ∈ {-1 , +1 } as ϵ i = sign ( ∥ x ( i ) 1 -x ( i ) 2 ∥ 2 Σ ⋆ -y ) . To estimate the metric from pairwise comparisons, we utilize a nuclear-norm regularized hinge loss and solve the following optimization problem:

Triplet setup. For the i -th triplet, we draw three items x ( i ) 1 , x ( i ) 2 , x ( i ) 3 i.i.d. from a standard multivariate normal distribution and record the outcome ϵ i ∈ {-1 , +1 } as ϵ i = sign ( ∥ x ( i ) 1 -

x ( i ) 2 ∥ 2 Σ ⋆ -∥ x ( i ) 1 -x ( i ) 3 ∥ 2 Σ ⋆ ) . To estimate the metric from triplet responses, we follow [19] and utilize a nuclear-norm regularized hinge loss and solve the following optimization problem:

Rankingk query setup. For the i -th ranking query with a reference item x 0 and k items x 1 , . . . , x k to be ranked, we draw all items i.i.d. from a standard multivariate normal distribution. For each item x k , we compute the squared distance ∥ x 0 -x k ∥ 2 Σ ⋆ . To determine the ranking of items, we sort the items based on this squared distance. To estimate the metric, we follow the approach of [14] and decompose the full ranking into its constituent triplets. A ranking consisting of k items can equivalently be decomposed into k ( k -1) / 2 triplet responses. To estimate the metric, we decompose each ranking query and use the triplet estimator presented above with regularization parameter λ R to obtain estimate ̂ Σ Rk .

PAQ setup. For the i -th PAQ response, we draw the reference item x i and query vector a i i.i.d. from the standard multivariate normal distribution. We then receive a scaling γ 2 i satisfying γ 2 i = y/ a ⊤ i Σ ⋆ a i , with y = 10 . To perform estimation, we leverage our method presented in Section 3. Our theoretical results indicate that the averaging parameter m should be set to 1 in the noiseless setting. Furthermore, the truncation threshold τ is large relative to our responses γ 2 i , meaning no truncation is employed. As a result, we solve the nuclear-norm regularized trace regression problem

In all cases above, we solve all optimization problems with cvxpy and normalize the estimated metric ̂ Σ { PC, T, Rk , PAQ } to be unit Frobenius norm to ensure consistent scaling when compared against the true metric Σ ⋆ . We use a value of 0 . 05 for all regularization parameters λ { PC, T, Rk , PAQ } and observe similar performance trends for other choices of regularization parameter.

C Scale equivariance

In this section, we verify the scale-equivariance of our derived theoretical bounds (13) and (14). Specifically, we denote by Σ ⋆ and ̂ Σ the ground-truth and the estimated matrices corresponding to value y . We denote by Σ ⋆ c and ̂ Σ c the ground-truth and estimated matrices corresponding to value c scale y for any c scale > 0 . By definition, we have Σ ⋆ c = c scale Σ ⋆ , and it can be verified that solving the optimization program (8) yields ̂ Σ c = c scale ̂ Σ . Hence, one expects the error bound to scale as c scale . To verify this linear scaling in c scale , we confirm that the noise η scales as c scale .

Under the ground-truth metric Σ ⋆ , if the user responds with an item that is a distance y + η away from the reference item, then that same item is a distance c scale ( y + η ) away from the reference under the scaled setting. As a result, the noise scales as a result of the choice of y . Therefore, the following values in the upper bound (13) can be written as scaled versions of their corresponding 'ground-truth' values.

Substituting these scaled expressions into the upper bounds (13) and (14), we have

in the high-noise regime and

in the low-noise regime. Note that the constant C ′ is independent of c scale .

D Preliminaries and notation

In this section, we provide an overview of the key tools that are utilized in our proofs. We first introduce notation which is used throughout our proofs.

Notation. For two real numbers a and b , let a ∧ b = min { a, b } . Given a vector x ∈ R d , denote ∥ x ∥ 1 and ∥ x ∥ 2 as the ℓ 1 and ℓ 2 norm, respectively. Denote S d -1 := { x ∈ R d : ∥ x ∥ 2 = 1 } to be the set of vectors with unit ℓ 2 norm. Given a matrix A ∈ R d 1 × d 2 , denote ∥ A ∥ F , ∥ A ∥ ∗ , and ∥ A ∥ op as the Frobenius norm, nuclear norm, and operator norm, respectively. Denote S d × d = { A ∈ R d × d : A = A ⊤ } to be the set of symmetric d × d matrices. Denote A ⪰ 0 to mean A is symmetric positive semi-definite. For A ⪰ 0 , define the (pseudo-) inner product ⟨ x , y ⟩ A = x ⊤ Ay and the associated (pseudo-) norm ∥ x ∥ A = √ x ⊤ Ax . For matrices A , B ∈ R d 1 × d 2 , denote ⟨ A , B ⟩ = tr ( A ⊤ B ) as the Frobenius inner product.

We use the notation f ( x ) ≲ g ( x ) to denote that there exists some universal positive constant c > 0 , such that f ( x ) ≤ c · g ( x ) , and use the notation f ( x ) ≳ g ( x ) when g ( x ) ≲ f ( x ) .

We define random matrices and

as the sensing matrix formed with the m -averaged responses ¯ γ and truncated responses ˜ γ , respectively.

D.1 Inverted measurement sensing matrices result in estimation bias

Recall from Equation (4) that the random sensing matrix A inv takes the form

Standard trace regression analysis assumes that for some sensing matrix A and measurement noise η , E [ η A ] = 0 . Specifically, it is often typically assumed that η is zero-mean conditioned on the sensing matrix A . The following lemma shows that for the inverted measurements, we have E [ η A inv ] = 0 , resulting in bias in estimation.

̸

Lemma 1. Let A inv be the random matrix defined in Eq. (4) and η be the measurement noise. Then

̸

The proof of Lemma 1 is provided in Appendix D.6.1. Hence, utilizing established low-rank matrix estimators for inverted measurements results in biased estimation.

D.2 Sub-exponential random variables

Our analysis utilizes properties of sub-exponential random variables, a class of random variables with heavier tails than the Gaussian distribution.

Lemma 2 (Moment bounds for sub-exponential random variables [46, Proposition 2.7.1(b)]) . If X is a sub-exponential random variable, then there exists some constant c (only dependent on the distribution of the random variable X ) such that for all integers p ≥ 1 ,

D.3 Bernstein's inequality

In our proofs, we use Bernstein's inequality to bound the sums of independent sub-exponential random variables.

Lemma3 (Bernstein's inequality, adapted from [47, Theorem 2.10]) . Let X 1 , . . . , X n be independent real-valued random variables. Assume there exist positive numbers u 1 and u 2 such that

Then for all t > 0 ,

D.4 Moments of the ratios of quadratic forms

The quadratic term a ⊤ Σ ⋆ a appears in the denominator of our sensing matrices, so we use the following result to quantify the moments of the ratios of quadratic forms.

Lemma 4. There exists an absolute constant c > 0 such that the following is true. Let a ∼ N ( 0 , I d ) , Σ ⋆ ∈ R d × d be any PSD matrix with rank r , and U ∈ R d × d be an arbitrary symmetric matrix.

(a) Suppose that r > 8 . Then we have

(b) Suppose that r > 2 . Then we have

The proof of Lemma 4 is presented in Appendix D.6.2.

D.5 A fourth moment bound for ¯ γ 2

Recall from Equation (6) that the averaged measurement ¯ γ 2 takes the form

Throughout our analysis, we utilize the fact that ¯ γ 2 has a bounded fourth moment, as characterized in the following lemma.

Lemma 5. Assume r > 8 . Then there exists a universal constant c > 0 , such that

where σ r is the smallest non-zero singular value of Σ ⋆ .

The proof of Lemma 5 is presented in Appendix D.6.3. For notational simplicity of the proofs, we denote M = c ( y + η ↑ σ r r ) 4 .

D.6 Proofs of preliminary lemmas

In this section, we present proofs for preliminary lemmas from Appendices D.1, D.4, and D.5.

D.6.1 Proof of Lemma 1

Using the independence of the noise η and the sensing vector a , and the assumption that η is zero mean, we have

̸

The expectation in (17) is non-zero, because the random matrix 1 a ⊤ Σ ⋆ a aa ⊤ is symmetric positive definite almost surely. Therefore, we have E [ η A inv ] = 0 , as desired.

D.6.2 Proof of Lemma 4

Since Σ ⋆ is symmetric positive semidefinite, it be decomposed as Q Σ Q ⊤ , where Q is a square orthonormal matrix and Σ is a diagonal matrix with non-negative entries. Multiplying a by any square orthonormal matrix does not change its distribution. Therefore, without loss of generality, we assume that Σ ⋆ is diagonal with all non-negative diagonal entries. We first note that the moments of the ratios in both parts of Lemma 4 exist, because by [48, Proposition 1], for non-negative integers p and q , the quantity E ( a ⊤ Ua ) p ( a ⊤ Σ ⋆ a ) q exists if r 2 > q . Furthermore, we use the following expression from [48, Proposition 2]:

where ∆ t = ( I d +2 t Σ ⋆ ) -1 / 2 and | ∆ t | is the determinant of ∆ t . To characterize the determinant | ∆ t | , we note that ∆ t is a diagonal matrix whose d diagonal entries are

Hence, the determinant is the product | ∆ t | = ∏ r i =1 1 (1+2 tσ i ) 1 / 2 . Furthermore, this product can be bounded as:

We now prove parts (a) and (b) separately.

Part (a). Using the integral expression (18) with p = 0 and q = 4 , and the upper bound (19) on the determinant, we have

Denoting s := 1 + 2 tσ r , we have

as desired.

Part (b). Using the integral expression (18) p = q = 1 and the upper bound (19) on the determinant, we have

We now bound the expectation term in (20). Note that for a ∼ N ( 0 , I d ) , we have E [ a ⊤ Ba ] = tr ( B ) for any symmetric matrix B . Therefore, we have

where (i) the fact that tr ( B ) ≤ ∥ B ∥ ∗ for any symmetric matrix B . Furthermore, (ii) follows from Hölder's inequality for Schattenp norms, where we have that ∥ ∆ t U ∆ t ∥ ∗ ≤ ∥ ∆ t ∥ 2 op · ∥ U ∥ ∗ . Because ∆ t is diagonal and the entries of ∆ t are bounded between 0 and 1 , we bound the operator norm as ∥ ∆ t ∥ op ≤ 1 . Substituting (21) to (20), we obtain

as desired.

D.6.3 Proof of Lemma 5

By the assumption that the noise is upper bounded by η ↑ , we have y + ¯ η ≤ y + η ↑ . Therefore, we have

where step (i) applies part (a) of Lemma 4.

E Proof of Theorem 1

Recall that we assume we collect N measurements under the inverted measurements sensing model presented in Algorithm 1 with standard Gaussian sensing vectors and bounded noise, mean-zero noise (Assumption 1).

We first introduce a restricted strong convexity (RSC) condition that our proof relies on. Since the matrix Σ ⋆ is assumed to be symmetric positive semidefinite matrices and of rank r , we follow [24] and consider a restricted set on which we analyze the behavior of the sensing matrices ˜ A i . We call this set the 'error set', defined by:

where recall that S d × d denotes the set of symmetric d × d matrices. We say that our shrunken sensing matrices { ˜ A i } n i =1 satisfy a restricted strong convexity (RSC) condition over the error set E , if there exists some positive constant κ > 0 such that

The following proposition shows that the estimation error, when the sensing matrices satisfy the RSC condition and the regularization parameter is sufficiently large.

Proposition 1 ([39, Theorem 1] with q = 0 ) . Suppose that Σ ⋆ has rank r and the shrunken sensing matrices satisfy the restricted strong convexity condition (23) with positive constant κ > 0 . Then if the regularization parameter satisfies

any optimal solution ̂ Σ of the optimization program (8) satisfies

This theorem is a special case of Theorem 1 in [39], which is in turn adapted from Theorem 1 in [24] (see [24] or [39] for the proof). Proposition 1 is a deterministic and nonasymptotic result and provides a roadmap for proving our desired upper bound. First, we show that the operator norm (24) can be upper bounded with high probability, allowing us to set the regularization parameter λ n accordingly. Second, we show that the RSC condition (23) is satisfied with high probability. We begin by bounding the operator norm (24) in the following proposition.

Proposition 2. Let y ↑ = y + η ↑ . Suppose that Σ ⋆ has rank r , with r > 8 . Then there exists a positive absolute constant C 1 such that

with probability at least 1 -4 exp ( -d ) .

The proof of Proposition 2 is provided in Appendix E.1. Next, we show that the RSC condition (23) is satisfied with high probability, as is done in the following proposition.

Proposition 3. Let µ y be the median of y + ¯ η . Suppose that the truncation threshold τ satisfies τ ≥ µ y tr( Σ ⋆ ) . Then there exist positive absolute constants κ L , c , and C such that if the number of sensing vectors satisfy

simultaneously for all matrices U ∈ E with probability at least 1 -exp( -cn ) , where E is the error set defined in Equation (22) .

The proof of Proposition 3 is provided in Appendix E.2. We now utilize the results of Propositions 1, 2 and 3 to derive our final error bound. By Proposition 2, the operator norm (24) can be upper bounded with high probability. We set the regularization parameter λ n to satisfy

where C 1 is the constant in Proposition 2. Furthermore, by Proposition 3, we have that there exist universal constant C 2 > 0 such that if the number of sensing vectors satisfies n ≥ C 2 rd , the RSC condition (23) holds for constant κ = κ L ( µ y tr( Σ ⋆ ) ) 2 with high probability. Taking a union bound, we have that Proposition 2 and Proposition 3 hold simultaneously with probability at least 1 -4 exp( -d ) -exp( -cn ) . Invoking Proposition 1, we have

with probability at least 1 -4 exp( -d ) -exp( -cn ) , as desired.

E.1 Proof of Proposition 2

In the proof, we decompose the operator norm ∥ ∥ ∥ ∥ op

individual terms and bound them separately. Recall the definitions of random matrices ¯ A from Equation 15 and ˜ A from Equation 16.

Step 1: decompose the error into five terms. We begin by adding and subtracting multiple quantities as follows:

then we have

where step (i) is true by substituting y = ⟨ ¯ A , Σ ⋆ ⟩ -¯ η to the term of E [ y ¯ A ] , and the fact that the noise term ¯ η is zero-mean. By triangle inequality, we group the terms in (27) and bound the operator norm by

In the remaining proof, we bound the five terms in (28) individually. We first discuss the nature of these five terms.

Terms 1 and 4: These two terms characterize the difference between the empirical mean of quantities involving ˜ A and their true expectation (see Lemma 6 and Lemma 9). In the proof, we show that the empirical mean concentrates around the expectation with high probability, as a function of the number of sensing vectors n .

Terms 2 and 3: These two terms characterize the difference in expectation introduced by truncating ¯ A to ˜ A (see Lemma 7 and Lemma 8). Hence, these two terms characterize biases that arise from truncation. They diminish as τ →∞ , because setting τ to ∞ is equivalent to no thresholding, and hence ˜ A becomes identical to ¯ A . Since expectations are considered, these two terms depend on the threshold τ , but not the number of sensing vectors n or the averaging parameter m .

Term 5: Term 5 is a bias term that arises from the fact that the mean of the noise η conditioned on sensing matrix ¯ A is non-zero. We show that this bias scales like 1 m (see Lemma 10) in terms of the averaging parameter m .

Putting these terms together, Terms 1 and 4 depend on n , Terms 2 and 3 depend on τ , and Term 5 depends on m . In Corollary 1, we set the values of τ , n and m to balance these terms.

Step 2: bound the five terms individually. In what follows, we provide five lemmas to bound each of the five terms individually. In the proofs of the five lemmas, we rely on an upper bound on the fourth moment of the m -sample averaged measurements ¯ γ 2 . As shown in Lemma 5 in Appendix D.5, for some absolute constant c , this fourth moment can be upper bounded by a quantity that we denote M :

We also rely heavily on the following truncation properties relating the m -sample averaged measurements ¯ γ 2 and truncated measurements ˜ γ 2 :

The following lemma provides a bound for Term 1.

Lemma 6. Let ˜ A 1 , . . . , ˜ A n be i.i.d copies of a random matrix ˜ A as defined in Equation (16) . There exists an absolute constant c > 0 such that for any t > 0 , we have

with probability at least 1 -2 · 9 d · exp( -t ) .

The proof of Lemma 6 is provided in Appendix E.1.1. The next lemma provides an upper bound for Term 2.

Lemma 7. Let ¯ A and ˜ A be the random matrices defined in Equation (15) and Equation (16) , respectively. Then there exists an absolute constant c > 0 such that

The proof of Lemma 7 is provided in Appendix E.1.2. The following lemma provides an upper bound for Term 3. Recall that the quantity y ↑ denotes y + η ↑ .

Lemma 8. Let ¯ A and ˜ A be the random matrices defined in Equation (15) and Equation (16) , respectively. Then there exists an absolute constant c > 0 such that

The proof of Lemma 8 is provided in Appendix E.1.3. The following lemma provides an upper bound for Term 4.

Lemma 9. Let ˜ A 1 , . . . , ˜ A n be i.i.d copies of a random matrix ˜ A defined in Equation (16) . There exists an absolute constant c > 0 such that for any t > 0 , we have

with probability at least 1 -2 · 9 d · exp( -t ) .

The proof of Lemma 9 is provided in Appendix E.1.4. We note that Terms 2 and 3 are bias that result from shrinkage, but crucially are inversely dependent on the shrinkage threshold τ . This fact allows us to set τ so that the order of Terms 2 and 3 match the order of Terms 1 and 4.

The final lemma bounds Term 5, which is a bias that arises from the dependence of the sensing matrix ¯ A on the noise η .

Lemma 10. Let ¯ A be the random matrix defined in Equation (15) . Suppose that Σ ⋆ has rank r with r > 2 . Then there exists an absolute constant c > 0 such that

The proof of Lemma 10 is provided in Appendix E.1.5. We note that the bias scales with the variance of the m -sample averaged noise ¯ η , which scales inversely with m .

Step 3: combine the five terms. We set t = (log 9 + 1) d . Substituting the bounds from Lemmas 610 back to (28) and taking a union bound, we have that with probability at least 1 -4 exp( -d ) ,

where step (i) is true by substituting in the expression (29) for M .

E.1.1 Proof of Lemma 6.

Let A 1 4 ⊆ S d -1 be a 1 4 -covering of the d -dimensional unit sphere S d -1 := { x ∈ R d : ∥ x ∥ 2 = 1 } . By a covering argument [46, Exercise 4.4.3], for any symmetric matrix U ∈ S d × d , its operator norm is bounded by ∥ U ∥ op ≤ 2 sup v ∈A 1 4 ∣ ∣ v ⊤ Uv ∣ ∣ . Hence, we have

We invoke Bernstein's inequality. We first show that the Bernstein condition holds. Namely, we show that for each integer p ≥ 2 , we have that for any unit vector v ∈ R d ,

where u 1 = c 1 M 1 2 and u 2 = c 2 τ for some universal positive constants c 1 and c 2 . Given the Bernstein condition (31), we then apply Bernstein's inequality to bound (30).

Proving the Bernstein condition (31) . We fix any unit vector v ∈ R d . Since ˜ A = ˜ γ 2 aa ⊤ , we have v ⊤ ˜ Av = ˜ γ 2 ( v ⊤ a ) 2 . Recall that the random vector a is distributed as a ∼ N ( 0 , I d ) . Since v is a unit vector, it follows that v ⊤ a ∼ N (0 , 1) . Denote by G ∼ N (0 , 1) a standard normal random variable. For any integer p ≥ 2 , we have

where steps (i) and (ii) follow from (TP1) and (TP2), respectively; step (iii) follows from CauchySchwarz inequality; and step (iv) follows upper bounding the fourth moment of ¯ γ 2 with the quantity M from Equation (29).

Note that since G is standard normal, by definition G 2 follows a Chi-Square distribution with 1 degree of freedom, and hence sub-exponential. By Lemma 2 in Appendix D.2, there exists some constant c > 0 such that we have ( E [ ( G 2 ) p ]) 1 / p ≤ cp for all p ≥ 1 . Hence, we have ( E [ G 4 p ]) 1 / 2 p ≤ 2 cp and

where step (i) is true by Stirling's inequality that for all p ≥ 1 ,

Substituting (33) back to (32) and rearranging terms completes the proof of the Bernstein condition (31).

Applying Bernstein's inequality to bound (30) . By Bernstein's inequality (see Lemma 3), given condition (31), we have that for any unit vector v ∈ R d and any t > 0 ,

By [46, Corollary 4.2.13], the cardinality of the covering set A 1 4 is bounded above by 9 d . Therefore, taking a union bound on (34), we have

Substituting in (30) to (35), for any t > 0 , we have

as desired.

E.1.2 Proof of Lemma 7

By definition of the operator norm, we have

We fix any v ∈ S d -1 , and bound v ⊤ ( E [ ¯ A ] -E [ ˜ A ]) v . Similar to the proof of Lemma 6, we note that v ⊤ a ∼ N (0 , 1) and denote the random variable G ∼ N (0 , 1) . Substituting in the expression for sensing matrices ¯ A and ˜ A , we have

where where step (i) is true because ¯ γ 2 ≥ ˜ γ 2 from to (TP2), step (ii) is true due to (TP3), and steps (iii) and (iv) follow from Cauchy-Schwarz inequality. We proceed by bounding each of the terms in (36) separately. First, we can upper bound the fourth moment E [ | ¯ γ 2 | 4 ] by the quantity M from Equation (29). Second, G 2 is a sub-exponential random variable. By Lemma 2 in Appendix D.2, we have that E [ | G 2 | 4 ] 1 / 4 ≤ c for some constant c . It remains to bound the term ( P ( ¯ γ 2 ≥ τ ) ) 1 / 2 . We have

where step (i) follows from Markov's inequality, step (ii) follows from Cauchy-Schwarz inequality, and step (iii) follows from the fourth moment bound on the averaged scaling ¯ γ 2 . Putting everything together back to (36), we have

for any vector v ∈ S d -1 . Therefore,

as desired.

E.1.3 Proof of Lemma 8

Substituting in the definitions ¯ A = ¯ γ 2 aa ⊤ and ˜ A = ˜ γ 2 aa ⊤ , we have

Therefore, our goal is to bound the operator norm

Similar to the proof of Lemma 7, we fix any vector v ∈ S d -1 . Again, note that v ⊤ a ∼ N (0 , 1) and denote G ∼ N (0 , 1) . We have

where steps (i) and (ii) are true because ¯ γ 2 ≥ ˜ γ 2 from (TP2), step (iii) follows from the definition ¯ γ 2 = y +¯ η a ⊤ Σ ⋆ a , and step (iv) follows from (TP3) and the definition of η ↑ as the upper bound on the noise η .

The rest of the proof follows the exact steps of the proof of Lemma 7 in Appendix E.1.2. Therefore, we have the bound

as desired.

E.1.4 Proof of Lemma 9

The proof follows the steps as in the proof of Lemma 6, and we describe the difference of the two proofs. We again apply Bernstein's inequality.

Proving a Bernstein condition. We prove a Bernstein condition with u 1 = c 1 ( y + η ↑ ) 2 and u 2 = c 2 ( y + η ↑ ) τ . Namely, for every integer p ≥ 2 , we have (cf. (31) in Lemma 6)

To show (37), we plug in ˜ A = ˜ γ 2 aa ⊤ and have

where step (i) follows from (TP2), step (ii) follows from the definition ¯ γ 2 = y +¯ η a ⊤ Σ ⋆ a , and step (iii) follows from the definition of η ↑ as the upper bound on the noise η . Substituting in (31) from Lemma 6 to bound the term E ∣ ∣ ∣ v ⊤ ˜ Av ∣ ∣ ∣ p in (38) completes the proof of the Bernstein condition (37).

Applying Bernstein's inequality. The rest of the proof follows in the same manner as the proof of Lemma 6 in Appendix E.1.1, with an additional factor of ( y + η ↑ ) . We have

with probability at least 1 -2 · 9 d · exp( -t ) , as desired.

E.1.5 Proof of Lemma 10

Recall that by definition ¯ A = ¯ γ 2 aa ⊤ = y +¯ η a ⊤ Σ ⋆ a aa ⊤ . We have

To bound the operator norm term in (39), we apply Lemma 4(b) in Appendix D.4. For any matrix U , we have

Note that aa ⊤ a ⊤ Σ ⋆ a is symmetric positive semidefinite, so we have

where step (i) is true by substituting in (40) with U = vv T , and step (ii) is true because v is unit norm, and hence ∥ vv ⊤ ∥ ∗ = 1 . Substituting (41) back to (39), we have

as desired.

E.2 Proof of Proposition 3

We analyze the term 1 n n ∑ i =1 ⟨ ˜ A i , U ⟩ 2 from (26). Recall from the definition of ˜ A that for any i = 1 , . . . , n ,

so we have

From (42), we have that for any matrix U , the term n ∑ i =1 ⟨ ˜ A i , U ⟩ 2 is nondecreasing in τ when τ > 0 . Defining a random matrix

where for every i = 1 , . . . , n , matrix ˜ A τ ′ i is formed with the same realizations of random quantities a i and ¯ η i as ˜ A i . The two matrices only differ in choice of truncation threshold: τ ′ instead of τ . As a result, for the rest of the proof, we lower bound 1 n n ∑ i =1 ⟨ ˜ A τ ′ i , U ⟩ 2 for an appropriate choice of τ ′ to be specified later. To proceed, we use a small-ball argument [49, 50] based on the following lemma.

Lemma 11 ([50, Proposition 5.1], adapted to our notation) . Let X 1 , . . . , X n ∈ R d × d be i.i.d. copies of a random matrix X ∈ R d × d . Let E ⊂ R d × d be a subset of matrices. Let ξ > 0 and Q > 0 be real values such that for every matrix U ∈ E , the marginal tail condition holds:

for any τ ′ ∈ (0 , τ ] , we have

Define the Rademacher width as

where ε 1 , . . . , ε n are i.i.d. Rademacher random variables independent of { X i } i ∈ [ n ] . Then for any t > 0 , we have

with probability at least 1 -exp ( -nt 2 2 ) .

Recall the error set E defined in Equation (22). Because the claim (26) is invariant to scaling, it suffices to prove it for ∥ U ∥ F = 1 . Correspondingly, we define the set E as

We invoke Lemma 11 with set E defined above, X i = ˜ A τ ′ i , ξ = c 1 2 ( µ y tr( Σ ⋆ ) ∧ τ ′ ) , and Q = c 2 , where µ y is the median of ¯ η and c 1 and c 2 , are constants to be specified later. The rest of the proof is comprised of two steps. We first verify that our choices for ξ and Q are valid for establishing the marginal tail condition (45). We then bound the Rademacher width W above. The following lemma verifies our choices for ξ and Q .

Lemma 12. Consider any τ ′ ∈ (0 , τ ] . There exist absolute constants c 1 , c 2 > 0 such that for every U ∈ E , we have

The proof of Lemma 12 is presented in Appendix E.2.1. We now turn to the second step of the proof, which is bounding the Rademacher width W . The next lemma characterizes this width.

Lemma 13. Consider any τ ′ ∈ (0 , τ ] . Let ˜ A τ ′ 1 , . . . , ˜ A τ ′ n ∈ R d × d be i.i.d. copies of the random matrix ˜ A τ ′ ∈ R d × d defined in Equation (43) . Let E be the set defined in Equation (46) . Then, there exists some absolute constants c 1 and c 2 such that if n ≥ c 1 d , then we have

The proof of Lemma 13 is presented in Appendix E.2.2. Lemma 12 establishes the marginal tail condition for Lemma 11, and Lemma 13 upper bounds the Rademacher width. We now invoke Lemma 11 and substitute in the upper bound for the Rademacher width W . For some constant c 4 , if n ≥ c 4 d , we have that with probability at least 1 -exp ( -nt 2 2 ) ,

where step (i) is true due to the monotonicity property (44). We set τ ′ = µ y tr( Σ ⋆ ) , where recall that µ y is the median of the random quantity y + ¯ η . By the assumption τ ≥ µ y tr( Σ ⋆ ) , this choice of τ ′ satisfies τ ′ ≤ τ . Setting t = c 2 2 , we have that with probability at least 1 -exp ( -c 2 2 n 8 ) ,

Recall from the definition of E (46) that ∥ U ∥ F = 1 . As a result, if n ≥ max { ( 4 c 3 c 1 c 2 ) 2 , c 4 } rd , we have

with probability at least 1 -exp ( -c 2 2 n 8 ) . We conclude by setting κ L = ( c 1 c 2 4 ) 2 , c = c 2 2 8 , and C = max { ( 4 c 3 c 1 c 2 ) 2 , c 4 } in Proposition 3.

E.2.1 Proof of Lemma 12

We fix any U ∈ E . Recall that µ y denotes the median of y + ¯ η . Let G be the event that y + ¯ η ≥ µ y , which occurs with probability 1 2 . For any ξ > 0 , because the averaged noise ¯ η and sensing vector a are independent, we have

where step (i) is true by plugging in the definition of ˜ A τ ′ , and step (ii) is true by the definition of the event G . We proceed by bounding the terms in (47) separately.

Lower bound on ∣ ∣ ⟨ aa ⊤ , U ⟩ ∣ ∣ . We use the approach from [32, Section 4.1]. By Paley-Zygmund inequality,

We now analyze the terms in (48). As noted in [32, Section 4.1], there exists some constant c 1 > 0 such that for any matrix U with ∥ U ∥ F = 1 ,

Note that by the definition of the set E , every matrix U ∈ E satisfies ∥ U ∥ F = 1 . Utilizing inequalities (48) and (49), there exists positive constant c 2 > 0 such that

Upper bound on a ⊤ Σ ⋆ a . By Hanson-Wright inequality [51, Theorem 1.1], there exist some positive absolute constants c 3 and c 4 such that for any t > 0 , we have

We set t = -1 c 4 log( c 2 4 ) so that 2 exp ( -c 4 t ) = c 2 2 . Since Σ ⋆ is symmetric positive semidefinite, we have

As a result, we have that there exists some constant c 5 > 0 such that

Substituting the two bounds back to (47) . By a union bound of (50) and (51), we have

Combining (52) and (47), and redefining constant c 2 appropriately, we have

as desired.

E.2.2 Proof of Lemma 13

We begin by noting that for any matrix U ∈ E ,

where step (i) follows from Hölder's inequality, and step (ii) follows from the fact that ∥ U ∥ ∗ ≤ 4 √ 2 r from the definition of the set E . It remains to bound the expectation of the operator norm in (53). We follow the standard covering arguments in [52, Section 5.4.1], [50, Section 8.6], [32, Section 4.1], with a slight modification to accommodate the bounded term ( y +¯ η i a ⊤ i Σ ⋆ a i ∧ τ ′ ) that appears in each of the matrices ˜ A τ ′ i . As a result, there exist universal constants c 1 and c 2 such that if n satisfies n ≥ c 1 d , then we have

We conclude by re-defining c 2 appropriately.

F Proof of Corollary 1

We proceed by considering two cases. For each case, the proof consists of two steps. We first verify that the choices of the averaging parameter m and truncation threshold τ ,

satisfy the assumptions of Theorem 1, namely n ≥ C 2 rd and τ ≥ µ y tr( Σ ⋆ ) . We then invoke Theorem 1.

F.1 Case 1: high-noise regime

In this case, we have ν 2 η ( y ↑ ) 2 > √ d N , which means by setting m according to Equation (54), we have m ≥ 2 . As a result, the bound

holds in the high-noise regime.

Verifying the condition on

where step (i) is true by plugging in the choice of m from (54) and applying the bound (55), and step (ii) is true by substituting in the assumption N ≥ 2 C 3 / 2 2 ( ν 2 η ( y ↑ ) 2 ) 2 r 3 / 2 d . Thus the condition n ≥ C 2 rd of Theorem 1 is satisfied.

Verifying the condition on τ . For the term √ N dm in the expression of τ in (54), note that, by the previous point, N m = n ≳ rd (with a constant that is greater than 1). Thus √ N dm ≥ √ r > 1 . Therefore, to verify the condition τ ≥ µ y tr( Σ ⋆ ) , it suffices to verify that

By definition, we have y ↑ ≥ µ y . Furthermore, since Σ ⋆ is symmetric positive semidefinite, its eigenvalues are all non-negative and are identical to its singular values, and hence tr ( Σ ⋆ ) ≥ σ r r , verifying the condition (56).

Invoking Theorem 1. By setting λ n to its lower bound in (9) and substituting in n = N m and our choice of τ from (54), we have

Substituting this expression of λ n to the error bound (10), then substituting in our choice of m from (54) to (57) and defining C ′ = 3 C · C 1 , we have

Using the fact that tr ( Σ ⋆ ) ≤ σ 1 r , we have

as desired.

F.2 Case 2: low-noise regime

In this case, we have ν 2 η ( y ↑ ) 2 ≤ √ d N , which means by setting m according to Equation (54), we have m = 1 . As a result, no averaging occurs.

Verifying the condition on n . Because m = 1 in this case, we have that n = N . By assumption, we have that N ≥ C 2 rd , satisfying the condition n ≥ C 2 rd in Theorem 1.

Verifying the condition on τ . By the same analysis as in Case 1, we have that the condition τ ≥ µ y tr Σ ⋆ in Theorem 1.

Invoking Theorem 1. By setting λ n to its lower bound in (9), substituting in our choice of τ from (54) and noting m = 1 , we have

We define C ′ = 3 C · C 1 and note that n = N under Case 2. Substituting this expression of λ n in (58) to the error bound (10), then using the fact that under Case 2, the bound ν 2 η ≤ ( y ↑ ) 2 √ d N holds, we have

Using the fact that tr ( Σ ⋆ ) ≤ σ 1 r , we have

as desired.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces perceptual adjustment query (PAQ), which combines cardinal reporting and ordinal reasoning using a sliding bar for relational questions through identifying the transition point (similar vs. dissimilar) along a continuum to the reference item. Formulating the problem as Mahalanobis distance learning in the feature space, this gives rise to a high-dimensional, low-rank matrix estimation problem. The authors propose two techniques including sample averaging and truncation to overcome challenges not present in classical low-rank matrix estimation problems and prove statistical bounds on the estimator error.\n\nSTRENGTHS:\n1. This paper studies the interesting problem of collecting more expressive and self-consistent human responses without heavy cognitive burden on users. The proposed PAQ method appears to achieve this by combining the advantages of both cardinal and ordinal queries. This gives a continuous way of constructing the tuplewise similarity queries [1]. And instead of asking for similarity, it asks for the transition point from similar to dissimilar. \n\n2. Thanks to the new approach for collecting human responses, the formulated Mahalanobis metric learning problem encounters several new challenges, including dependent noise and a heavy-tailed sensing matrix. To address these issues, the authors propose to leverage the averaging and truncation techniques. Both theoretical and simulation analyses demonstrate the efficacy of these methods.\n\n3. This paper presents a coherent structure and is easy to follow.\n\n[1] G. Canal, S. Fenu, and C. Rozell. Active ordinal querying for tuplewise similarity learning. AAAI 2020.\n\nWEAKNESSES:\n1. One major concern regarding this work is the lack of empirical validation for the proposed PAQ in real-world applications. The authors offer a conceptual explanation of its functionality and user interface design but fail to provide any implementation guidance. For instance, the practical implementation of the continuum remains unclear - should it be continuous or discretized? Moreover, the authors do not provide any synthetic tests or real human responses to support the claimed benefits or practical applicability of the proposed PAQ. The current validation relies solely on simulation results, which assess the theoretical analysis in relatively low-dimensional settings instead of truly high-dimensional scenarios.\n\n2. Another concern is the dependence of the proposed method on a pre-trained generative model that exhibits a linear latent space. This constraint limits the method's broad applicability compared to existing methods, such as tuplewise queries [1]. Also, many items generated along the line may be invalid in specific domains, such as molecules and proteins.\n\n3. The proposed solutions to the two challenges seem somewhat expected: averaging to reduce the variance and truncation to mitigate the heavy-tail issue have been commonly used in similar situations.\n\nQUESTIONS:\n1. What are the main contributions of this work? The authors claim both the proposal of the PAQ for the Mahalanobis metric learning and the solution to the associated matrix estimation problem in Section 1.1. Therefore, both should be validated quantitatively to make this work coherent and complete.\n\n2. The definition of the perceptual adjustment query in Section 2.2 does not seem to be dependent on a latent space associated with a item decoder/generator but only a feature space. Could the PAQ be applied to items solely within a feature space where a decoder/generator is not feasible, such as in the case of housing pricing similarity?\n\n3. The authors mention in Section 5 that they \"cross-validate to chose the constant factor.\" How to perform cross-validation in practice, given that only human responses and query vectors are available? Also, the boundary value y is set at 200, which seems to be an unusually large number. Although the actual value of y may not be critical due to the scale invariance property, the noise variance scales with y.\n\n4. The authors mention that they \"develop a two-stage estimator for metric learning from PAQs\". What are those two stages? From Algorithm 1, there seems to be only one stage.\n\nLIMITATIONS:\nThe authors provide a discussion of limitations and potential societal impact in Appendix A.1 and A.2.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPeople evaluate things in two ways: scoring and comparing, which is the process of generating cardinal data and ordinal data. However, since different people have different scales, the cardinal data are more influenced by people's subjective perception, while the ordinal data are fundamentally limited and laces expressiveness because the comparison results are only for several things. Therefore, the paper introduces a new type of informative and yet cognitively lightweight evaluation mechanism for collecting human feedback. Specifically, by generating a series of consecutive items between two things, one progressively compares the reference item with other items, making cognitive judgments in a relative sense while identifying a transition point (similar vs. dissimilar) in the ranking.  Subsequently, the Mahalanobis distance metric learning model is proposed based on the above evaluation mechanism, where the model requires that the learned metric makes each reference sample as close as possible to its generated transition point. Moreover, the generalized error bound analysis of the proposed model is given.\n\nSTRENGTHS:\nThe idea of the paper is novel. The paper presents a novel query mechanism and uses it to propose an efficient and easily solvable metric learning model.\n\nWEAKNESSES:\nThe idea of the paper is novel, however the experiments are very inadequate and not at all compared with any known metric learning algorithms. It is suggested to add comparative experiments with existing metric learning algorithms under different perspectives to fully illustrate the value of the proposed algorithm. Meanwhile, there are some expression and formatting problems with the paper. The details are as follows:\n1. Lack of description of existing relevant metric learning.\n2. There are major problems with the format of references, such as the lack of journal names and page numbers in some citations, as well as the problem of inconsistent cases between the paper name and the journal name.\n3 There are inconsistencies in expression or format. For example, “metric model” and “measurement model”, “In metric learning” and “In our theoretical results”.\n4 The usage of some symbols is inconsistent with the conventions of metric learning. For example, y often indicates a label in metric learning, there is a symbol with an arrow to indicate a constant that has not seen such usage, and there is a greater than or equal symbol with a wavy line that does not understand its meaning.\n5. There are some grammatical errors, please proofread carefully. For example, “such as in conducting surveys, grading assignments,…”should remove “in”, inconsistent case after the initial letter of the colon.\n\nQUESTIONS:\n1.  While novel ideas and theories are important, it is sufficient experimentation that will prove the value of the proposed method. It is recommended that sufficient experiments be added to the text to fully demonstrate the significance of the proposed model.\n2.  Comparison with existing metric learning and discussion are lacking, and it is suggested to add relevant literature descriptions to highlight the contribution of the proposed method.\n\nLIMITATIONS:\nThe idea is novel, the model is simple, and it is suggested that the proposed method be extended to more complex multi-metric and deep metric learning models to enhance the usefulness of the proposed theory and model.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nthis paper introduces the Perceptual Adjustment Query (PAQ) mechanism for collecting human feedback in a cognitively lightweight manner, which combines advantages from both ordinal and cardinal queries. \nThe authors apply PAQs to the problem of  metric learning under a Mahalanobis metric.\n\nSTRENGTHS:\n- The introduction of the Perceptual Adjustment Query (PAQ) as a hybrid of ordinal and cardinal queries is a novel contribution. This mechanism offers the advantages of both query types.\n\n- The paper applies the PAQ scheme to the framework of metric learning for human perception. This application demonstrates the utility of PAQs in learning distance metrics and capturing similarities and dissimilarities in perceptual data.\n\n- The authors provide sample complexity guarantees and numerical experiments for the proposed estimator, which offer insights into the understanding of the PAQ-based approach.\n\nWEAKNESSES:\n1. The paper is based on the idea that the PAQ style of queries is more useful than ordinal or cardinal queries. But it is not clear how this can be tested. The paper would benefit from discussing potential real-world applications where the PAQ mechanism could be better than other relatedapproaches. This discussion would provide insights into the impact of the proposed approach.\n2. Similarly, I think the paper would benefit from exploring additional tasks other than Mahalanobis metric learning, \n3. Limited Evaluation: While the paper mentions ”extensive numerical simulations” to demonstrate the performance and properties of the estimator, it would be beneficial to provide more experimental results. Additional evaluation metrics and comparisons with alternative approaches would strengthen the paper's contributions. For example:\n- Apply this approach to tasks other than Mahalanobis metric learning ? \n- Comparison: The experiments focus solely on evaluating the PAQ-based metric learning method. Including a comparative analysis with existing approaches or baselines would provide a more comprehensive evaluation and further validate the proposed approach.\n- How is the data collected different from ordinal/cardinal queries? Can this difference be demonstrated experimentally?\n\nQUESTIONS:\nsame as above.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes to collect users' perceptual adjustment queries (PAQ) to capture change points in a semantic space. In contrast to ordinal and cardinal data, PAQ better aligns with human perception and is cheap to collect. PAQ can be used to learn the semantic space manifold by solving a row-rank optimization algorithm. The paper proposes theoretical guarantees accompanied by numerical results.

STRENGTHS:
1. PAQ is conceptually novel, easy to collect, and aligns with human perception of concepts.
2. The metric learning algorithm is straightforward and is backed with theoretical guarantees.

WEAKNESSES:
1. The paper does not provide a downstream use case for PAQ.
2. Selecting the direction $a$ is challenging in high dimensions. 
3. The truncation trick in line 219 seems ad hoc to me.

QUESTIONS:
The demonstration in Figure 2. looks very similar to embedding spaces of images. Have you considered using PAQ to learn the underlying embedding spaces for languages and images?

LIMITATIONS:
See "weaknesses."
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10