Title: Carrot and Stick: Eliciting Comparison Data and Beyond
Venue: NeurIPS 2024
Accepted: yes
Keywords: information elicitation

mechanism design

crowdsourcing

ranking data

social network
Abstract: Comparison data elicited from people are fundamental to many machine learning tasks, including reinforcement learning from human feedback for large language models and estimating ranking models. They are typically subjective and not directly verifiable. How to truthfully elicit such comparison data from rational individuals? We design peer prediction mechanisms for eliciting comparison data using a bonus-penalty payment. Our design leverages on the strong stochastic transitivity for comparison data to create symmetrically strongly truthful mechanisms such that truth-telling 1) forms a strict Bayesian Nash equilibrium, and 2) yields the highest payment among all symmetric equilibria. Each individual only needs to evaluate one pair of items and report her comparison in our mechanism.

We further extend the bonus-penalty payment concept to eliciting networked data, designing a symmetrically strongly truthful mechanism when agents’ private signals are sampled according to the Ising models. We provide the necessary and sufficient conditions for our bonus-penalty payment to have truth-telling as a strict Bayesian Nash equilibrium. Experiments on two real-world datasets further support our theoretical discoveries.

Carrot and Stick:

Abstract

Comparison data elicited from people are fundamental to many machine learning tasks, including reinforcement learning from human feedback for large language models and estimating ranking models. They are typically subjective and not directly verifiable. How to truthfully elicit such comparison data from rational individuals? We design peer prediction mechanisms for eliciting comparison data using a bonus-penalty payment [11]. Our design leverages on the strong stochastic transitivity for comparison data [60, 13] to create symmetrically strongly truthful mechanisms such that truth-telling 1) forms a strict Bayesian Nash equilibrium, and 2) yields the highest payment among all symmetric equilibria. Each individual only needs to evaluate one pair of items and report her comparison in our mechanism.

We further extend the bonus-penalty payment concept to eliciting networked data, designing a symmetrically strongly truthful mechanism when agents' private signals are sampled according to the Ising models. We provide the necessary and sufficient conditions for our bonus-penalty payment to have truth-telling as a strict Bayesian Nash equilibrium. Experiments on two real-world datasets further support our theoretical discoveries.

1 Introduction

In the past two decades, researchers have been embracing the challenge of eliciting private information from individuals when there is no ground truth available to evaluate the quality of elicited contributions, and have made amazing progress. Many mechanisms, collectively called peer prediction [42], have been developed to incentivize individuals to strictly truthfully report their information at a Bayesian Nash equilibrium (BNE), by artful design of payment functions that only depend on reports from individuals. Moreover, in multi-task peer prediction mechanisms, the truthful BNE gives each individual the highest expected payoff among all BNEs (i.e. it's a strongly truthful BNE). [11, 57, 30]

However, all prior multi-task peer prediction mechanisms require tasks being ex-ante identical, and hence individuals' private information is independently and identically distributed (iid) for each task. Multi-task peer prediction leverages this structure of information to succeed at truthful elicitation. But what if such structure of information doesn't hold for an information elicitation problem?

One notable application is to elicit pair-wise comparisons of multiple alternatives, such as preferences for consumer products [53], translation [34], peer grading [55], and relevance of language model outputs [9, 10]. Such pair-wise comparison data are crucial for estimating a ranking of the alternatives and for devising reward functions for reinforcement learning. Comparison tasks for different pairs are clearly not ex-ante identical - answers to the tasks demonstrate a certain degree of transitivity (e.g. if 𝑎 is preferred to 𝑎 ′ and 𝑎 ′ is preferred to 𝑎 ′′ , then it's more likely that 𝑎 is preferred to 𝑎 ′′ ), rendering existing peer prediction mechanisms not applicable.

In this paper, we design a peer prediction mechanism for eliciting comparison data. We model individuals' private information of pair-wise comparisons as Bayesian strongly stochastically transitive (Bayesian SST), which takes many widely used models (e.g. Thurstone [58], Bradley-TerryLuce [4, 38], and Mallows [39]) as special cases. Our mechanism uses a simple bonus-penalty payment [11] (hence carrot and stick) that takes three reports as inputs and admits a strongly truthful symmetric BNE. The key insight that we develop is a condition of information structure that we call uniform dominance . When uniform dominance is satisfied, the bonus-penalty payment is the only type of payment that induces a strictly truthful BNE. Information of individuals, 𝑖 , 𝑗 , and 𝑘 , satisfies uniform dominance if, conditioned on any realization of agent 𝑖 's information, the probability for 𝑗 's information to agree with 𝑖 's is higher than the probability for 𝑘 's information to agree with 𝑖 's. Bayesian SST allows us to group three pairwise comparisons, ( 𝑎, 𝑎 ′ ) , ( 𝑎 ′′ , 𝑎 ′ ) and ( 𝑎 ′′ , 𝑎 ) , together such that private information about these pairs satisfies uniform dominance. After identifying uniform dominance as a central structure for incentivizing truthful elicitation, we further generalize the bonus-penalty payment to truthfully elicit private information over social networks that demonstrate homophily (i.e. friends tend to have similar opinions than non-friends) [40], and our mechanism can be integrated with common survey techniques such as snowball sampling [24].

Our contributions. Our work is a leap forward for designing mechanisms for complex information elicitation settings where ground truth verification is not available.

We are the first to design mechanisms to truthfully elicit pairwise comparison data under Bayesian SST and networked data under Ising models. In our mechanisms, truthful reporting forms a BNE and yields a strictly higher payoff than any symmetric non-permutation equilibrium.

We identify a key structure of information, uniform dominance, as a lever such that the simple bonus-penalty payment is the unique payment inducing a strictly truthful BNE. This identification may offer a path for developing truthful elicitation mechanisms for other settings in the future.

We use Griffiths' inequality and Weitz's self-avoiding walk [65] to prove the uniform dominance property in the Ising model. The resulting correlation bounds may be of independent interest.

We test our mechanisms on real-world data (sushi preference dataset [26, 27] and Last.fm dataset [8]). Even though these datasets do not perfectly satisfy our theoretical assumptions, our mechanisms still provide a stronger incentive for truthful reporting compared to misreporting.

Related work. Information elicitation has two settings according to whether verification is possible. Our paper focuses on elicitation without verification.

For information elicitation without verification, Miller et al. [41] introduce the first mechanism for single task signal elicitation that has truth-telling as a strict Bayesian Nash equilibrium but requires full knowledge of the common prior. Bayesian truth serum (BTS) [47] is the first strongly truthful peer prediction mechanism, but requires complicated reports from agents (their private signal and predictions on other's reports). A series of works [48, 49, 67, 66, 68, 3, 51, 31] relax certain assumptions of BTS but still require complicated reports from agents. Dasgupta and Ghosh [11] introduces the multi-task setting where agents are assigned batch iid tasks and only report their signals. Several works extend this to multiple-choice questions [29, 33, 56, 11], predictions [37], or even continuous value [50], and investigate the limitation and robustness [52, 6, 70, 17, 70]. Another related line of work is co-training and federated learning, which wants to elicit models [32, 36], or samples [64] when multiple iid data or feature of data are available. For more related works, see Faltings [16].

One popular line of work considers information elicitation when verification is possible. Spotchecking requires direct verification of the agent's report [21]. Recent work on comparison data elicitation [19] utilizes spot-checking concepts and focuses on incentivizing effort. Another form of verification involves using additional samples to evaluate how the agent's reports improve model performance [1, 28]. Additionally, the verification may have a general relation to the agent's signal, e.g., proper scoring rules [23, 46, 35, 20].

2 Problem Formulation

We discuss our model for eliciting comparison data in this section and defer the extensions to Section 5. Given a collection of items A and a set of strategic agents N . Agents privately observe

noisy comparisons between pairs of items. Our goal is to design mechanisms to truthfully elicit agents' private information. We will first introduce the information structure of agents' private information of pairwise comparisons in Section 2.1 and then define the information elicitation problem in Section 2.2.

2.1 Bayesian SST Models for Comparison Data

We introduce Bayesian Strong Stochastic Transitivity (Bayesian SST) models to capture the structure of agents' private information for comparison data.

Given the set of items A , the underlying unknown state about the items is 𝜃 ∈ Θ . 𝜃 can be the vector of quality scores for the items (Example 2.2) or a reference ranking (Example 2.4). 𝜃 is drawn according to a common prior 𝑃 Θ : 𝜃 ∼ 𝑃 Θ . Any realized 𝜃 has an associated stochastic comparison function 𝑇 𝜃 : A 2 → {-1 , 1 } . For comparisons of two items 𝑎 and 𝑎 ′ , 𝑇 𝜃 ( 𝑎, 𝑎 ′ ) and 𝑇 𝜃 ( 𝑎 ′ , 𝑎 ) stochastically take value 1 or -1 , with Pr [ 𝑇 𝜃 ( 𝑎, 𝑎 ′ ) = 1 ] = 1 -Pr [ 𝑇 𝜃 ( 𝑎 ′ , 𝑎 ) = -1 ] . For any 𝜃 , 𝑇 𝜃 is strongly stochastically transitive as defined below.

Definition 2.1 ([60, 13]) . A stochastic comparison function, 𝑇 : A 2 → {-1 , 1 } , is strongly stochastically transitive (SST) if for all 𝑎, 𝑎 ′ , 𝑎 ′′ ∈ A with Pr [ 𝑇 ( 𝑎, 𝑎 ′ ) = 1 ] > 1 / 2 and Pr [ 𝑇 ( 𝑎 ′ , 𝑎 ′′ ) = 1 ] > 1 / 2 , we have

Intuitively, a comparison function is SST when for any three items 𝑎, 𝑎 ′ , 𝑎 ′′ , if 𝑎 is more favorable than 𝑎 ′ and 𝑎 ′ is more favorable than 𝑎 ′′ , then 𝑎 is even more favorable than 𝑎 ′′ . The concept of SST is a well-established property of comparisons in social science and psychology [18].

Each agent 𝑖 ∈ N has the knowledge of ( 𝑇 𝜃 ) 𝜃 ∈ Θ and 𝑃 Θ . When asked to compare a pair of items ( 𝑎, 𝑎 ′ ) , the agent observes an independent draw according to the stochastic comparison function: 𝑆 𝑖 = 𝑇 𝜃 ( 𝑎, 𝑎 ′ ) , where realization 𝑠 𝑖 = 1 represent item 𝑎 is preferred over item 𝑎 ′ by agent 𝑖 . We assume items are a priori similar but ex-post distinct so that for all 𝑎, 𝑎 ′ ∈ A , E [ 𝑇 𝜃 ( 𝑎, 𝑎 ′ )] = E [ E [ 𝑇 𝜃 ( 𝑎, 𝑎 ′ ) | 𝜃 ]] = 0 and E [ 𝑇 𝜃 ( 𝑎, 𝑎 ′ ) | 𝜃 ] ≠ 0 for all 𝜃 .

Examples of Bayesian SST models. Bayesian SST models are a general family of models that take many classical parametric ranking models, including Bradley-Terry Luce [4, 38], Thurstone (Case V) [58], and Mallows 𝜂 -model [39], as special cases.

Example 2.2 (Bradley-Terry-Luce, Thurstone model, and more [59]) . Let 𝜃 ∈ R A = Θ where each coordinate is independently and identically sampled from a fixed non-atomic distribution 𝜈 on R , and each item 𝑎 have a scalar quality 𝜃 𝑎 ∈ R . Let 𝐹 : R → [ 0 , 1 ] be any strictly increasing function such that 𝐹 ( 𝑡 ) = 1 -𝐹 (-𝑡 ) for all 𝑡 ∈ R . Conditional on a fixed 𝜃 ,

This model recovers the Thurstone model [58] by setting 𝐹 ( 𝑡 ) = Φ ( 𝑡 ) where Φ is the Gaussian CDF, and the Bradley-Terry-Luce model [4] by setting 𝐹 ( 𝑡 ) = 𝑒 𝑡 1 + 𝑒 𝑡 , the sigmoid function. Moreover, this model also contains any additive random utility model [2] where 𝑇 ( 𝑎, 𝑎 ′ ) = 1 if 𝜃 𝑎 + 𝑍 > 𝜃 𝑎 ′ + 𝑍 ′ with iid noise 𝑍 and 𝑍 ′ , because we can set 𝐹 to be the CDF of the difference of two iid noise.

Proposition 2.3. For any strictly increasing 𝐹 and non-atomic 𝜈 on R , the parametric model in Example 2.2 is a Bayesian SST model.

Example 2.4 (Mallows 𝜂 -model [39]) . Let Θ be the set of rankings on A and 𝜂 > 0 be a dispersion parameter. Given a reference ranking 𝜃 ∈ Θ , the Mallows 𝜂 -model generate a ranking 𝜙 ∈ Θ with probability Pr ( 𝜙 ) ∝ exp (-𝜂𝑑 ( 𝜃, 𝜙 )) where 𝑑 ( 𝜃, 𝜙 ) = GLYPH<12> GLYPH<12> {( 𝑎, 𝑎 ′ ) ∈ A 2 : 𝜃 ( 𝑎 ) < 𝜃 ( 𝑎 ′ ) and 𝜙 ( 𝑎 ) > 𝜙 ( 𝑎 ′ )} GLYPH<12> GLYPH<12> is Kendall's tau distance, and 𝜃 ( 𝑎 ) is the rank of item 𝑎 . Therefore, to generate comparisons, we first sample a uniform 𝜃 and

Proposition 2.5. For any 𝜂 > 0 , Mallows 𝜂 -model in Example 2.4 with uniform distribution on reference ranking is an Bayesian SST model.

The proofs for propositions 2.3 and 2.5 are closely related to strong stochastic transitivity [54, 7], but are provided in the appendix for completeness.

2.2 Peer Prediction Mechanism Design

To truthfully elicit comparison data from agents, a peer prediction mechanism creates a game between the agents outlined below: First, we choose an assignment E = { 𝑒 𝑖 = ( 𝑎 𝑢 𝑖 , 𝑎 𝑣 𝑖 ) : 𝑖 ∈ N} where agent 𝑖 ∈ N gets a pair of items 𝑒 𝑖 = ( 𝑎 𝑢 𝑖 , 𝑎 𝑣 𝑖 ) ∈ A 2 to compare. Then each agent 𝑖 ∈ N privately observes the realization of the comparison (signal) 𝑠 𝑖 ∈ {-1 , 1 } , which is an independent realization of 𝑇 𝜃 ( 𝑎 𝑢 𝑖 , 𝑎 𝑣 𝑖 ) , and reports ˆ 𝑠 𝑖 ∈ {-1 , 1 } potentially different from her signal. We use 𝑆 𝑖 = 𝑆 ( 𝑎 𝑢 𝑖 , 𝑎 𝑣 𝑖 ) to denote the random variable of agent 𝑖 's signal, where the randomness of 𝑆 (· , ·) comes from both 𝜃 and 𝑇 𝜃 . Let S represent the random vector of all agents' signals, s = ( 𝑠 𝑖 ) 𝑖 ∈N be all agents' realized private signals and ˆ s = ( ˆ 𝑠 𝑖 ) 𝑖 ∈N be all agents' reports. Finally, a peer prediction mechanism ( 𝑀 𝑖 ) 𝑖 ∈N takes all agents' reports ˆ s and pays agent 𝑖 with 𝑀 𝑖 ( ˆ s ) ∈ R .

Each agent 𝑖 's strategy is a random function from her signal to a report 𝜎 𝑖 : 𝑠 𝑖 ↦→ ˆ 𝑠 𝑖 , and the randomness of their strategies is independent of each other's and all signals. With slight abuse of notation, we write 𝜎 𝑖 ( 𝑠 𝑖 , ˆ 𝑠 𝑖 ) = Pr [ ˆ 𝑆 𝑖 = ˆ 𝑠 𝑖 | 𝑆 𝑖 = 𝑠 𝑖 ] as the conditional probability of reporting ˆ 𝑠 𝑖 given private signal 𝑠 𝑖 . A strategy profile 𝝈 is a collection of all agent's strategies. All agents are rational and risk-neutral, so they want to maximize their expected payments. Thus, given prior 𝑃 Θ , randomness of 𝑇 𝜃 and a strategy profile 𝝈 , agent 𝑖 wants to maximize her ex-ante payment denoted as E 𝝈 , 𝜃 ,𝑇 𝜃 [ 𝑀 𝑖 ( ˆ S )] where ˆ S is the random vector of all agents' report that depends on the signals S and strategy profile 𝝈 .

We introduce three families of strategies, truth-telling, permutation, and uninformed strategy profiles, which are central to understanding effective peer prediction mechanisms.

A strategy 𝜎 𝑖 is truthful (or truth-telling) if it is a deterministic identity map, 𝜎 𝑖 ( 𝑠 𝑖 ) = 𝑠 𝑖 . A strategy profile is truthful if all agents' strategies are truthful.

A permutation strategy profile is where agents simultaneously relabel their signals and then report the relabeled ones. A permutation strategy is indistinguishable from truth-telling unless the peer prediction mechanism has additional knowledge about the prior signal distribution.[33]

Finally, a strategy is uninformed if it has the same report distribution across all signals, and it is informed otherwise. Common examples include consistently reporting all signals as a constant value, such as 1 or -1 , or using a random report regardless of the signal. Uninformed strategies are undesirable as the reports bear no relationship to the private signals.

A strategy profile is symmetric if all agents use the same strategy. For example, both truth-telling and permutation strategy profiles are symmetric.

We now introduce goals for a peer prediction mechanism that favors truth-telling more than other strategies. First, we want the truth-telling (strategy profile) to be a strict Bayesian Nash equilibrium (BNE) so that any agent's payment would strictly decrease if she unilaterally changes to any nontruthful strategy. Moreover, there may be multiple equilibria, and a desirable mechanism should ensure that truth-telling is better than all other equilibria. In this paper, we aim for symmetrically strongly truthful mechanisms defined below.

Definition 2.6. A peer prediction mechanism is symmetrically strongly truthful if truth-telling is a BNE, and each agent's expected payment in truth-telling is no less than the payment in any other symmetric equilibrium with equality for the equilibrium with a permutation strategy profile. 1

3 Bonus-penalty Payment Mechanism for Comparison Data

We now propose a bonus-penalty payment mechanism for eliciting comparison data. The mechanism makes use of a bonus-penalty payment function, which can be seen as an agreement payment and introduced by Dasgupta and Ghosh [11] in a different context (see discussion in appendix A). Formally, for any ˆ 𝑠 𝑖 , ˆ 𝑠 𝑗 , ˆ 𝑠 𝑘 ∈ {-1 , 1 } , the bonus-penalty payment function is

which rewards when the first input agrees with the second but punishes when it agrees with the third.

Mechanism 1 uses the bonus-penalty payment eq. (1) for each agent 𝑖 by carefully choosing agent 𝑗 and 𝑘 such that agent 𝑗 's signal is more likely to agree with agent 𝑖 's than agent 𝑘 's signal is. The crux of finding such pair of agents is to show that if agent 𝑖 prefers item 𝑎 over 𝑎 ′ , she would expect that others will prefer any third item 𝑎 ′′ over 𝑎 ′ , and prefer 𝑎 over 𝑎 ′′ . Thus, if agent 𝑗 has pair ( 𝑎 ′′ , 𝑎 ′ ) and agent 𝑘 has pair ( 𝑎 ′′ , 𝑎 ) , then agent 𝑗 's signal is more likely to take the same value as 𝑖 's than agent 𝑘 's signal is. This is the main idea behind the proof of Theorem 3.1, where we establish the symmetrically strongly truthfulness of Mechanism 1. To ensure the existence of such pairs are assigned, we require the assignment E to be admissible where for all ( 𝑎, 𝑎 ′ ) ∈ E , there exists 𝑎 ′′ ∈ A so that ( 𝑎 ′′ , 𝑎 ′ ) and ( 𝑎 ′′ , 𝑎 ) ∈ E .

Mechanism 1: BPP mechanism for comparison data

Input: Let A be a collection of items, E be an admissible assignment, and ˆ s be agents' reports. for agent 𝑖 ∈ N with pair 𝑒 𝑖 = ( 𝑎 𝑢 𝑖 , 𝑎 𝑣 𝑖 ) = ( 𝑎, 𝑎 ′ ) do

Find 𝑎 ′′ ∈ A and two agents 𝑗 and 𝑘 so that 𝑒 𝑗 = ( 𝑎 ′′ , 𝑎 ′ ) and 𝑒 𝑘 = ( 𝑎 ′′ , 𝑎 ) , and pay agent 𝑖

Theorem 3.1. Given a collection of items A and a set of agents N with |A| , |N| ≥ 3 , for any admissible assignment matrix E and Bayesian SST model with ( 𝑇 𝜃 ) 𝜃 ∈ Θ and 𝑃 Θ , the BPP mechanism for comparison (Mechanism 1) is symmetrically strongly truthful.

We defer the proof of theorem 3.1 to section 4. The admissible condition imposes little overhead on downstream learning problems, including rank recovery [25] and identification of the top 𝑘 items [15]. Specifically, the size of assignment E is the number of comparisons and corresponds to the sample complexity for these learning problems. If a learning algorithm requires a set of pairs to compare E 𝑀𝐿 , we can construct an admissible superset E that introduces a constant factor overhead and can recover E 𝑀𝐿 ⊆ E . 2

We remark that the bonus-penalty payment function eq. (2) can be seen as a boolean function for transitivity [45]; see remark 3.2 for a formal statement. Hence, theorem 3.1 implies that agents' manipulations can only decrease the probability of transitivity among their reports.

Remark 3.2. Note that a deterministic comparison function 𝑡 : A×A → {1 , 1 } satisfies transitivity on three items 𝑎, 𝑎 ′ , 𝑎 ′′ ∈ A if and only if 𝑡 ( 𝑎, 𝑎 ′ ) , 𝑡 ( 𝑎 ′ , 𝑎 ′′ ) , 𝑡 ( 𝑎 ′′ , 𝑎 ) are not all equal, that is 𝑁𝐴𝐸 ( 𝑡 ( 𝑎, 𝑎 ′ ) , 𝑡 ( 𝑎 ′ , 𝑎 ′′ ) , 𝑡 ( 𝑎 ′′ , 𝑎 )) = 1 where

The agent's random noisy comparisons may or may not satisfy transitivity. The probability of transitivity is the probability that they do.

We can show that the bonus-penalty payment in eq. (2) is equivalent to the above transitivity test when agents are truth-telling. Formally,

Therefore, arg max ˆ 𝑠 𝑖 E [ 𝑁𝐴𝐸 ( ˆ 𝑠 𝑖 , -𝑆 𝑗 , 𝑆 𝑘 )| 𝑆 𝑖 = 𝑠 𝑖 ] = arg max ˆ 𝑠 𝑖 E GLYPH<2> 1 4 𝑀 𝑖 ( ˆ 𝑠 𝑖 , 𝑆 𝑗 , 𝑆 𝑘 ) + 3 4 + 1 4 𝑆 𝑗 𝑆 𝑘 | 𝑆 𝑖 = 𝑠 𝑖 GLYPH<3> = arg max ˆ 𝑠 𝑖 E [ 𝑀 𝑖 ( ˆ 𝑠 𝑖 , 𝑆 𝑗 , 𝑆 𝑘 )| 𝑆 𝑖 = 𝑠 𝑖 ] .

4 Proof of Theorem 3.1: from Bayesian SST Model to Uniform Dominance

To prove theorem 3.1, we formalize the idea that agent 𝑗 's signal is more likely to agree with agent 𝑖 's than agent 𝑘 's is as what we call uniform dominance in definition 4.1. We'll show that any Bayesian SST model satisfies this property. Then, we'll prove that BPP mechanism is symmetrically strongly truthful when agents' private signals satisfy uniform dominance.

Definition 4.1. Given a random vector ( 𝑆 𝑖 , 𝑆 𝑗 , 𝑆 𝑘 ) ∈ {-1 , 1 } 3 with joint distribution 𝑃 , 𝑆 𝑗 uniformly dominates 𝑆 𝑘 for 𝑆 𝑖 if Pr [ 𝑆 𝑗 = 1 | 𝑆 𝑖 = 1 ] > Pr [ 𝑆 𝑘 = 1 | 𝑆 𝑖 = 1 ] and Pr [ 𝑆 𝑗 = -1 | 𝑆 𝑖 = -1 ] > Pr [ 𝑆 𝑘 = -1 | 𝑆 𝑖 = -1 ] . We call such an ordered tuple ⟨ 𝑆 𝑖 , 𝑆 𝑗 , 𝑆 𝑘 ⟩ a uniformly dominant tuple. 3

Lemma 4.2 shows how to identify uniformly dominant tuples under Bayesian SST models.

Lemma 4.2. Under any Bayesian SST model, for any agent 𝑖 and items 𝑎 , 𝑎 ′ and 𝑎 ′′ , agent 𝑗 's signal 𝑆 𝑗 = 𝑆 ( 𝑎 ′′ , 𝑎 ′ ) uniformly dominates agent 𝑘 's signal 𝑆 𝑘 = 𝑆 ( 𝑎 ′′ , 𝑎 ) for signal 𝑆 𝑖 = 𝑆 ( 𝑎, 𝑎 ′ ) .

In other words, under any Bayesian SST model, the distribution of 𝑆 ( 𝑎, 𝑎 ′ ) , 𝑆 ( 𝑎 ′′ , 𝑎 ′ ) , 𝑆 ( 𝑎 ′′ , 𝑎 ) satisfies uniform dominance for any 𝑎, 𝑎 ′ , 𝑎 ′′ . In the rest of this section, we can view ( 𝑆 𝑖 , 𝑆 𝑗 , 𝑆 𝑘 ) as an abstract random vector with some joint distribution 𝑃 .

We now establish some implications of uniform dominance on the bonus-penalty payment. Lemma 4.3 shows that truth-telling is the best response if other signals are reported truthfully. Lemma 4.4 states that the expected payment is zero if everyone uses uninformed strategies (random functions independent of input). Lemma 4.5 characterizes the best response under symmetric strategy profiles (the same random function on each coordinate).

Lemma 4.3 (Truthfulness) . Given a uniformly dominant tuple ⟨ 𝑆 𝑖 , 𝑆 𝑗 , 𝑆 𝑘 ⟩ with distribution 𝑃 , for all 𝑠 𝑖 ∈ {-1 , 1 } , 𝑠 𝑖 = arg max ˆ 𝑠 𝑖 ∈{-1 , 1 } E 𝑃 GLYPH<2> 𝑈 𝐵𝑃𝑃 ( ˆ 𝑠 𝑖 , 𝑆 𝑗 , 𝑆 𝑘 ) | 𝑆 𝑖 = 𝑠 𝑖 GLYPH<3> and E 𝑃 GLYPH<2> 𝑈 𝐵𝑃𝑃 ( 𝑆 𝑖 , 𝑆 𝑗 , 𝑆 𝑘 ) GLYPH<3> > 0 .

Lemma 4.4. Given a uniformly dominant tuple ⟨ 𝑆 𝑖 , 𝑆 𝑗 , 𝑆 𝑘 ⟩ , with joint distribution 𝑃 if agent 𝑗 and 𝑘 both use an uninformed strategy 𝜎 so that ˆ 𝑆 𝑗 = 𝜎 ( 𝑆 𝑗 ) and ˆ 𝑆 𝑘 = 𝜎 ( 𝑆 𝑘 ) , for all 𝑠 𝑖 and ˆ 𝑠 𝑖 in {-1 , 1 } , E 𝜎,𝑃 GLYPH<2> 𝑈 𝐵𝑃𝑃 ( ˆ 𝑠 𝑖 , ˆ 𝑆 𝑗 , ˆ 𝑆 𝑘 ) | 𝑆 𝑖 = 𝑠 𝑖 GLYPH<3> = 0 .

Lemma4.5. Given a uniformly dominant tuple ⟨ 𝑆 𝑖 , 𝑆 𝑗 , 𝑆 𝑘 ⟩ with distribution 𝑃 , for any strategy 𝜎 and 𝑠 𝑖 ∈ {-1 , 1 } when agent 𝑗 and 𝑘 both use 𝜎 , arg max ˆ 𝑠 𝑖 ∈{-1 , 1 } E 𝜎,𝑃 GLYPH<2> 𝑈 𝐵𝑃𝑃 ( ˆ 𝑠 𝑖 , ˆ 𝑆 𝑗 , ˆ 𝑆 𝑘 ) | 𝑆 𝑖 = 𝑠 𝑖 GLYPH<3> = arg max ˆ 𝑠 𝑖 ∈{-1 , 1 } { 𝜎 ( 𝑠 𝑖 , ˆ 𝑠 𝑖 ) -𝜎 (-𝑠 𝑖 , ˆ 𝑠 𝑖 )} .

We'd like to highlight that lemmas 4.3 to 4.5 as well as the proof of theorem 3.1 below hold for any uniformly dominant tuple ⟨ 𝑆 𝑖 , 𝑆 𝑗 , 𝑆 𝑘 ⟩ , not necessarily derived from the Bayesian SST model. This offers a path to generalize our mechanism for comparison data to other settings.

Proof of theorem 3.1. By lemma 4.2, for any agent 𝑖 , the associated agent 𝑗 's signal 𝑆 𝑗 = 𝑆 ( 𝑎 ′′ , 𝑎 ′ ) uniformly dominates the associated 𝑘 's signal 𝑆 𝑘 = 𝑆 ( 𝑎 ′′ , 𝑎 ) for signal 𝑆 𝑖 = 𝑆 ( 𝑎, 𝑎 ′ ) . By lemma 4.3, if agent 𝑗 and 𝑘 are truthful, agent 𝑖 's best response is truthful reporting, so truth-telling is a BNE.

Now we show that all other symmetric equilibria are permutation or uninformed equilibria. For any symmetric equilibrium 𝝈 = ( 𝜎 𝜄 ) 𝜄 ∈N so that everyone uses the same strategy 𝜎 𝜄 = 𝜎 for all 𝜄 ∈ N . If 𝜎 is not deterministic so that 𝜎 ( 𝑠, 𝑠 ) , 𝜎 ( 𝑠, -𝑠 ) > 0 for some 𝑠 ∈ {-1 , 1 } , agent 𝑖 must be indifferent between reporting 𝑠 and -𝑠 when getting 𝑆 𝑖 = 𝑠 . 𝜎 ( 𝑠, 𝑠 ) -𝜎 (-𝑠, 𝑠 ) = 𝜎 ( 𝑠, -𝑠 ) -𝜎 (-𝑠, -𝑠 ) by lemma 4.5. This means 𝜎 ( 𝑠, 𝑠 ) = 𝜎 (-𝑠, 𝑠 ) and 𝜎 ( 𝑠, -𝑠 ) = 𝜎 (-𝑠, -𝑠 ) , and 𝜎 is an uninformed strategy. If the strategy is deterministic, there are two cases. If 𝜎 ( 𝑠 ) = 𝜎 (-𝑠 ) , the strategy is also uninformed. If 𝜎 ( 𝑠 ) ≠ 𝜎 (-𝑠 ) , 𝜎 is either truth-telling 𝑠 ↦→ 𝑠 or flipping 𝑠 ↦→-𝑠 for all 𝑠 .

Finally, by lemma 4.4, any uninformed equilibrium's expectation is zero. Additionally, because eq. (1) is invariant when all inputs are flipped, the truth-telling and flipping/permutation equilibria has the same expected payment which is positive by lemma 4.3. □

5 Generalization of Bonus-penalty Payment Mechanisms

We now leverage the key idea of uniform dominance to design peer prediction mechanisms for networked data in section 5.1. In section 5.2, we summarize our design approach as a general scheme that first identifies uniform dominance structures and then engages the bonus-penalty payment. We prove the uniqueness of bonus-penalty payment: it is the only payment function, up to some positive affine transformation, that induces truth-telling as a strict BNE for all uniform dominant tuples.

5.1 Bonus-penalty Payment Mechanisms for Networked Data

Uniform dominance implies agent 𝑖 's signal is more likely to agree with agent 𝑗 's than with agent 𝑘 's. Social networks are another natural domain exhibiting this property, as homophily [40] suggests that agents' opinions or signals in a social network are more likely to agree with their friends than with non-friends. Leveraging this insight, we use a bonus-penalty payment scheme to elicit binary networked data.

Mechanism 2: BPP mechanism for networked data

Input: Let ( 𝑉, 𝐸 ) be a graph of agents in 𝑉 , ˆ s ∈ {-1 , 1 } 𝑉 for agent 𝑖 ∈ 𝑉 do

Find agents 𝑗 (friend) and 𝑘 (non-friend) so that ( 𝑖, 𝑗 ) ∈ 𝐸 but ( 𝑖, 𝑘 ) ∉ 𝐸 , and pay agent 𝑖

Below, we provide a theoretical guarantee for our mechanism under a popular graphical model for social network data, Ising model [14, 43], which captures the correlation between agents and their friends. Formally, an Ising model consists of an undirected graph ( 𝑉, 𝐸 ) and correlation parameter 𝛽 𝑖, 𝑗 ≥ 0 for each edge ( 𝑖, 𝑗 ) ∈ 𝐸 . Each agent is a node in the graph, N = 𝑉 , and has a binary private signal ( 1 or -1 ) jointly distributed as the following: For all s = ( 𝑠 𝑖 ) 𝑖 ∈ 𝑉 ∈ {-1 , 1 } 𝑉 , Pr 𝜷 [ S = s ] ∝ exp ( 𝐻 ( s )) where the energy function is 𝐻 ( s ) = ˝ ( 𝑖, 𝑗 ) ∈ 𝐸 𝛽 𝑖, 𝑗 𝑠 𝑖 𝑠 𝑗 .

Theorem 5.1. If agents' signals are sampled from an Ising model on undirected graph ( 𝑉, 𝐸 ) with correlation parameters 𝜷 , Mechanism 2 is symmetrically strongly truthful, when 2 𝛽 𝑑 > ln 𝑒 2 ( 𝑑 + 1 ) 𝛽 + 1 𝑒 2 𝛽 + 𝑒 2 𝑑𝛽 where 𝛽 = min ( 𝑖, 𝑗 ) ∈ 𝐸 𝛽 𝑖, 𝑗 , 𝛽 = max ( 𝑖, 𝑗 ) ∈ 𝐸 𝛽 𝑖, 𝑗 , and 𝑑 is the maximal degree of graph ( 𝑉, 𝐸 ) .

Mechanism 2 does not require knowledge about parameters of the Ising model, but only the connection of the network ( 𝑉, 𝐸 ) . Social network platforms, which already possess this knowledge, can easily integrate our mechanism when conducting surveys. Additionally, snowball sampling [24], which relies on participants referring their friends, is also naturally compatible with our mechanism.

The complete proof of theorem 5.1 is quite technical and is deferred to the appendix, where we also explain why the bound between 𝜷 and 𝑑 is necessary. Below, we provide a sketch of the proof.

Proof sketch for theorem 5.1. As discussed in section 4, we only need to show that for any agent 𝑖 , for all agent 𝑗 with ( 𝑖, 𝑗 ) ∈ 𝐸 and 𝑘 with ( 𝑖, 𝑘 ) ∉ 𝐸 , 𝑗 's signal uniformly dominates 𝑘 's signal for 𝑖 's signal. Because the energy function 𝐻 ( s ) above remains invariant when the signs are flipped, Pr [ 𝑆 𝑖 = 1 ] = Pr [ 𝑆 𝑗 = 1 ] = Pr [ 𝑆 𝑘 = 1 ] = 1 / 2 , it is sufficient to prove that

We then prove a lower bound for the left-hand side and an upper bound for the right-hand side separately. For the left-hand side, we use the Griffiths' inequality [44] to show that the minimum value of Pr [ 𝑆 𝑖 = 1 | 𝑆 𝑗 = 1 ] happens when 𝑗 is the only friend of 𝑖 . For the right-hand side, we use Weitz's self-avoiding walk [65] and reduce any graph with maximum degree 𝑑 into a 𝑑 -ary tree. □

5.2 General Design Scheme and Uniqueness

The design of BPP mechanisms for comparison data and networked data has suggested a general design scheme for other elicitation settings. That is, if one can identify a uniformly dominant tuple for

Mechanism 3: General design scheme using BPP

Input: Let ˆ s be reports from agents in N .

for agent 𝑖 ∈ N do

Find two agents 𝑗 and 𝑘 so that 𝑗 's signal uniformly dominates 𝑘 's for 𝑖 's, and pay agent 𝑖

each agent , adopting the bonus-penalty payment gives a symmetrically strongly truthful mechanism. We further show that the bonus-penalty payment is in some sense unique.

Theorem 5.2. If for each agent 𝑖 the associated agent 𝑗 's signal uniformly dominates 𝑘 's signal for 𝑖 's signal, the above scheme is symmetrically strongly truthful.

When an agent 𝑖 has multiple pairs of ( 𝑗 1 , 𝑘 1 ) , . . . , ( 𝑗 ℓ , 𝑘 ℓ ) so that 𝑗 𝑙 's signal uniformly dominates 𝑘 𝑙 's for 𝑖 's for each 𝑙 = 1 , . . . , ℓ , we may pay agent 𝑖 the average of bonus-penalty payment on all pairs 𝑀 𝑖 ( ˆ s ) = 1 ℓ ˝ ℓ 𝑙 = 1 𝑈 𝐵𝑃𝑃 ( ˆ 𝑠 𝑖 , ˆ 𝑠 𝑗 𝑙 , ˆ 𝑠 𝑘 𝑙 ) . This average maintains our symmetrically strongly truthful guarantee while potentially reducing the variance in payments.

Theorem 5.2 shows that bonus-penalty payment is a sufficient condition for designing good elicitation mechanisms for information structures with uniform dominance. We now prove it is also a necessary condition: any payment that induces truth-telling as a strict BNE under all uniformly dominant tuples must be an affine transformation of the bonus-penalty payment.

Theorem 5.3 (Uniqueness) . A payment 𝑈 : {-1 , 1 } 3 → R satisfies that, for all uniformly dominant tuples ⟨ 𝑆 𝑖 , 𝑆 𝑗 , 𝑆 𝑘 ⟩ , 𝑠 𝑖 = arg max ˆ 𝑠 𝑖 ∈{-1 , 1 } E GLYPH<2> 𝑈 ( ˆ 𝑠 𝑖 , 𝑆 𝑗 , 𝑆 𝑘 ) | 𝑆 𝑖 = 𝑠 𝑖 GLYPH<3> , if and only if there exist 𝜆 > 0 and 𝜇 : {-1 , 1 } 2 → R so that

where choice of 𝜇 does not affect the set of equilibria.

6 Experiments

We present experiments on real-world data to evaluate our models and mechanisms. We hope to cast insights on two questions empirically. Does our mechanism provide better rewards when all agents report truthfully than when all agents report randomly? Does our mechanism incentivize truth-telling for each agent if all other agents are truthful? We evaluate Mechanism 1 and 2 by comparing three settings, truth-telling, uninformed, and unilateral deviation, using empirical cumulative distribution functions (ECDF) on agents' payments. Each point on ECDF denotes the fraction of agents who get paid less than a particular value.

For both comparison and networked datasets (figs. 1 and 2), we find our mechanisms provide better payments to agents under truthful settings than the other two settings. The ECDF under truth-telling lies below the other two ECDFs, which is known as first-order stochastic dominance. This implies that the truth-telling strategy results in higher average, quantiles (e.g., first quartile, median, and third quartile), and a greater expectation of any monotone function on the empirical distribution than the other two settings. We provide additional

6.1 SUSHI Preference Dataset for Comparison Data

We consider preference data for a collection of 10 sushi items (item set A) [26, 27], and focus on a group of 249 agents. Each agent provides a complete ranking of all 10 types of sushi in the dataset. These agents are female, aged thirty to forty-nine, who took more than three hundred seconds to rank the items and mostly lived in Kanto and Shizuoka until age fifteen. We restrict the set of agents to avoid significant violations of transitivity across different agents and to better align with our model assumptions. In the appendix, we will present the experimental results for other groups of users and further test whether the dataset satisfies transitivity.

For the first question, we use Mechanism 1 to compute each agent's payment under the truth-telling or uninformed strategy profile. For each agent 𝑖 , we 1) randomly sample three items 𝑎, 𝑎 ′ , 𝑎 ′′ and

two agents 𝑗 , 𝑘 , 2) derive agent 𝑖 's comparison on the first two items ( 𝑎, 𝑎 ′ ) from her ranking, (and similarly for agent 𝑗 's comparison on ( 𝑎 ′ , 𝑎 ′′ ) , and agent 𝑘 's comparison on ( 𝑎, 𝑎 ′′ ) ), 3) compute bonus-penalty payment on these three comparisons, 4) repeat the above procedure 100 times and pay agent 𝑖 with the average of those 100 trials. For the uninformed strategy setting, we replace every agent's comparisons with uniform random bits and compute the payment. The left column of fig. 1 presents the ECDF of payments for the agents in both settings. The figure shows that in the uninformed random strategy setting only about 50% of the agents receive positive payments, while in the original dataset (truthful strategy setting) over 75% of the users receive positive payments. The right column of fig. 1 tests the second question if the agent has the incentive to deviate when every other agent is truthful. The truth-telling curve is identical to the left column of fig. 1. For unilateral deviation, each agent gets the above bonus-penalty payment when her comparisons are replaced by uniform random bits. We plot the ECDFs of payments for both settings in the right column of fig. 1. The figure shows that the ECDF of the unilateral deviation payments is above the ECDF of human users' payments, indicating that our mechanism pays more to the truth-telling agents.

ECDF of Payments

ECDFof Payments

1.0

truth-telling

1.0

truth-telling

uninformed

unilateral deviation

0.8

0.8

0.6

0.6

ECDF

ECDF

0.4

0.4

0.2

0.2

0.0

0.0

0.4

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

6.2 Last.fm Dataset for Networked Data

We test our BPP mechanism on the Last.fm dataset from Cantador et al. [8]. This dataset consists of 1892 agents on Last.fm, forming a social network with 12704 edges and an average degree of 6 . 71 . It records agents' top fifty favorite artists whom they have listened to the most. We note that, in the dataset, listener fractions for all artists are much smaller than non-listener fractions. This bias differs from our Ising model in section 5.1 where every agent has the same chance to get both signals. Thus, the result can be seen as a stress test for our mechanism even when the data deviate from the assumption of our theoretical results.

Figure 2 focuses on the most popular artist in the dataset, Lady Gaga, who has a listener fraction of 32 . 3% . The results for additional artists are presented in the supplementary material. The left column of fig. 2 tests the first question. Each agent has a binary signal about whether or not she listens to a particular artist (Lady Gaga in this section). For the truth-telling setting, everyone reports her signal truthfully and gets payment by the bonus-penalty payment (formally defined in section 5.1). For the uninformed setting, everyone gets the bonus-penalty payment when all reports are iid according to the prior ( 0 . 322 for Lady Gaga). When everyone is truthful, more than 76% of agents get positive payments and have an average payment of 0 . 37 for Lady Gaga, while when agents report randomly, only half get positive payments, and have a near zero average payment. These results suggest that agents got more incentive to choose the truth-telling equilibrium than the uninformed equilibrium. The right column of fig. 2 tests the second question. The truth-telling curve is identical to the left column of fig. 2. For the unilateral deviation setting, each agent gets the bonus-penalty payment when she reports listener/non-listener uniformly at random. The unilateral deviation's payment is worse than the payments for truth-telling, decreasing from 0 . 37 to near zero for Lady Gaga.

Payment Amount

Payment Amount

ECDFforLadyGaga

ECDFforLadyGaga

1.0

truth-telling

1.0

truth-telling

uninformed

unilateral deviation

0.8

0.8

0.6

0.6

ECDF

ECDF

0.4

0.4

0.2

0.2

0.0

0.0

1.0

0.5

0.0

0.5

1.0

1.5

1.0

0.5

0.0

0.5

1.0

1.5

Payment

Payment

7 Conclusion and Discussion

We introduce a symmetrically strongly truthful peer prediction mechanism for eliciting comparison data without verification and extend it to eliciting networked data under Ising models. Our mechanisms are evaluated using real-world data. A key insight from our work is the identification of a structure we term 'uniform dominance,' which suggests a path for designing mechanisms in more complex elicitation settings. For example, in time-series data, adjacent points tend to be more related than distant ones, and in contextual settings, feedback from similar contexts is typically more related than from different contexts.

A central assumption in this study is that agents are a priori similar . Hence, noisy comparisons of item pairs are independent of the assigned agent's identity. This assumption is reasonable for items with widely agreed-upon rankings, such as quality assessments of large language model (LLM) outputs. However, it may break down in settings where preferences are highly polarized, such as political opinions or social choice problems 4 . Despite this, our additional experiments in appendix F, which relax the selection rule used in obtaining fig. 1, show that the mechanism remains robust even when some dissimilarities among agents exist.

Agents in our model are assumed to focus solely on maximizing their payments, without accounting for efforts or external incentives such as minimizing others' rewards or intentionally distorting rankings. While our mechanism may be extended to handle binary effort as suggested in previous work [11, 57], accommodating more than two effort levels would require additional assumptions [69]. Moreover, one may hope to incorporate the designer's utility, by factoring in downstream learning problems along with elicitation payments. This would necessitate a significant overhaul of the existing learning framework.

Our mechanisms achieve a symmetric, strongly truthful equilibrium. This does not rule out the existence of non-symmetric equilibria with potentially higher utility. However, such equilibria would require complex coordination among agents, making them less likely to arise naturally.

From a technical standpoint, our approach involves several assumptions that can be generalized or relaxed. Our Bayesian SST model, which relies on strong stochastic transitivity, serves as a non-parametric extension of several widely used parametric ranking models. In appendix C.2, we present both positive and negative results regarding weaker notions of transitivity (e.g., [5]). While we assume admissible assignments, this can be relaxed to random assignments with full support. Additionally, limited liability can be ensured in our mechanism. For example, adding a constant of 1 to the payment function in eq. (2) ensures that the payment is either 2 or 0 .

Acknowledgments

This research was partially supported by the National Science Foundation under grant no. IIS2147187.

References

Jacob D Abernethy and Rafael Frongillo. A collaborative mechanism for crowdsourcing prediction problems. Advances in neural information processing systems , 24, 2011.

Hossein Azari, David Parks, and Lirong Xia. Random utility theory for social choice. Advances in Neural Information Processing Systems , 25, 2012.

Aurélien Baillon. Bayesian markets to elicit private information. Proceedings of the National Academy of Sciences , 114(30):7958-7962, 2017.

Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324-345, 1952.

Mark Braverman and Elchanan Mossel. Noisy sorting without resampling. In Proceedings of the Nineteenth Annual ACM-SIAM Symposium on Discrete Algorithms , SODA '08, page 268-276, USA, 2008. Society for Industrial and Applied Mathematics.

Noah Burrell and Grant Schoenebeck. Measurement integrity in peer prediction: A peer assessment case study. In Proceedings of the 24th ACM Conference on Economics and Computation , pages 369-389, 2023.

Róbert Busa-Fekete, Eyke Hüllermeier, and Balázs Szörényi. Preference-based rank elicitation using statistical models: The case of mallows. In International conference on machine learning , pages 1071-1079. PMLR, 2014.

Iván Cantador, Peter Brusilovsky, and Tsvi Kuflik. Second workshop on information heterogeneity and fusion in recommender systems. In Proceedings of the fifth ACM conference on Recommender systems , pages 387-388, 2011.

Mingyue Cheng, Hao Zhang, Jiqian Yang, Qi Liu, Li Li, Xin Huang, Liwei Song, Zhi Li, Zhenya Huang, and Enhong Chen. Towards personalized evaluation of large language models with an anonymous crowd-sourcing platform. In Companion Proceedings of the ACM on Web Conference 2024 , pages 1035-1038, 2024.

Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132 , 2024.

Anirban Dasgupta and Arpita Ghosh. Crowdsourced judgement elicitation with endogenous proficiency. In Proceedings of the 22nd international conference on World Wide Web , pages 319-330, 2013.

Constantinos Daskalakis, Nishanth Dikkala, and Gautam Kamath. Concentration of multilinear functions of the ising model with applications to network data. Advances in Neural Information Processing Systems , 30, 2017.

Donald Davidson and Jacob Marschak. Experimental tests of a stochastic decision theory. Measurement: Definitions and theories , 17(2), 1959.

Glenn Ellison. Learning, local interaction, and coordination. Econometrica: Journal of the Econometric Society , pages 1047-1071, 1993.

Brian Eriksson. Learning to top-k search using pairwise comparisons. In Artificial Intelligence and Statistics , pages 265-273. PMLR, 2013.

Boi Faltings. Game-theoretic mechanisms for eliciting accurate information. In IJCAI , 2022.

Shi Feng, Fang-Yi Yu, and Yiling Chen. Peer prediction for learning agents. Advances in Neural Information Processing Systems , 35:17276-17286, 2022.

Peter C Fishburn. Binary choice probabilities: on the varieties of stochastic transitivity. Journal of Mathematical psychology , 10(4):327-352, 1973.

Kiriaki Frangias, Andrew Lin, Ellen Vitercik, and Manolis Zampetakis. Algorithmic contract design for crowdsourced ranking. arXiv preprint arXiv:2310.09974 , 2023.

Rafael Frongillo and Ian A Kash. Vector-valued property elicitation. In Conference on Learning Theory , pages 710-727. PMLR, 2015.

Xi Alice Gao, James R Wright, and Kevin Leyton-Brown. Incentivizing evaluation with peer prediction and limited access to ground truth. Artificial Intelligence , 275:618-638, 2019.

Hans-Otto Georgii. Gibbs measures and phase transitions . Walter de Gruyter GmbH & Co. KG, Berlin, 2011.

Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association , 102(477):359-378, 2007.

Leo A Goodman. Snowball sampling. The annals of mathematical statistics , pages 148-170, 1961.

David R Hunter. Mm algorithms for generalized bradley-terry models. The annals of statistics , 32(1):384-406, 2004.

Toshihiro Kamishima. Nantonac collaborative filtering: recommendation based on order responses. In Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining , pages 583-588, 2003.

Toshihiro Kamishima and Shotaro Akaho. Efficient clustering for orders. In Sixth IEEE International Conference on Data Mining-Workshops (ICDMW'06) , pages 274-278. IEEE, 2006.

Sai Praneeth Karimireddy, Wenshuo Guo, and Michael I Jordan. Mechanisms that incentivize data sharing in federated learning. arXiv preprint arXiv:2207.04557 , 2022.

Yuqing Kong. Dominantly truthful multi-task peer prediction with a constant number of tasks. In Proceedings of the fourteenth annual acm-siam symposium on discrete algorithms , pages 2398-2411. SIAM, 2020.

Yuqing Kong and Grant Schoenebeck. A framework for designing information elicitation mechanisms that reward truth-telling. CoRR , abs/1605.01021, 2016. URL http://arxiv. org/abs/1605.01021 .

Yuqing Kong and Grant Schoenebeck. Equilibrium selection in information elicitation without verification via information monotonicity. In 9th Innovations in Theoretical Computer Science Conference (ITCS 2018) . Schloss-Dagstuhl-Leibniz Zentrum für Informatik, 2018.

Yuqing Kong and Grant Schoenebeck. Water from two rocks: Maximizing the mutual information. In Proceedings of the 2018 ACM Conference on Economics and Computation , pages 177-194, 2018.

Yuqing Kong and Grant Schoenebeck. An information theoretic framework for designing information elicitation mechanisms that reward truth-telling. ACM Transactions on Economics and Computation (TEAC) , 7(1):1-33, 2019.

Julia Kreutzer, Joshua Uyheng, and Stefan Riezler. Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning. arXiv preprint arXiv:1805.10627 , 2018.

Nicolas S Lambert, David M Pennock, and Yoav Shoham. Eliciting properties of probability distributions. In Proceedings of the 9th ACM Conference on Electronic Commerce , pages 129-138, 2008.

Yang Liu, Rixing Lou, and Jiaheng Wei. Auditing for federated learning: A model elicitation approach. In Proceedings of the Fifth International Conference on Distributed Artificial Intelligence , DAI '23, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400708480. doi: 10.1145/3627676.3627683. URL https://doi.org/10.1145/ 3627676.3627683 .

Yang Liu, Juntao Wang, and Yiling Chen. Surrogate scoring rules. ACM Transactions on Economics and Computation , 10(3):1-36, 2023.

R Duncan Luce. Individual choice behavior: A theoretical analysis . Courier Corporation, 2005.

C. L. Mallows. Non-null ranking models. i. Biometrika , 44(1/2):114-130, 1957. ISSN 00063444. URL http://www.jstor.org/stable/2333244 .

Miller McPherson, Lynn Smith-Lovin, and James M Cook. Birds of a feather: Homophily in social networks. Annual review of sociology , 27(1):415-444, 2001.

N. Miller, P. Resnick, and R. Zeckhauser. Eliciting informative feedback: The peer-prediction method. Management Science , pages 1359-1373, 2005.

Nolan Miller, Paul Resnick, and Richard Zeckhauser. Eliciting informative feedback: The peer-prediction method. Management Science , 51(9):1359-1373, 2005.

Andrea Montanari and Amin Saberi. The spread of innovations in social networks. Proceedings of the National Academy of Sciences , 107(47):20196-20201, 2010.

Marc Mézard and Andrea Montanari. Information, Physics, and Computation . Oxford University Press, 01 2009. ISBN 9780198570837. doi: 10.1093/acprof:oso/9780198570837.001.0001. URL https://doi.org/10.1093/acprof:oso/9780198570837.001.0001 .

Ryan O'Donnell. Analysis of boolean functions . Cambridge University Press, 2014.

Kent Harold Osband. Providing Incentives for Better Cost Forecasting (Prediction, Uncertainty Elicitation) . University of California, Berkeley, 1985.

Drazen Prelec. A bayesian truth serum for subjective data. science , 306(5695):462-466, 2004.

Goran Radanovic and Boi Faltings. A robust bayesian truth serum for non-binary signals. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 27, pages 833-839, 2013.

Goran Radanovic and Boi Faltings. Incentives for truthful information elicitation of continuous signals. In Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence , pages 770-776, 2014.

Grant Schoenebeck and Fang-Yi Yu. Learning and strongly truthful multi-task peer prediction: A variational approach. In 12th Innovations in Theoretical Computer Science Conference (ITCS 2021) . Schloss Dagstuhl-Leibniz-Zentrum für Informatik, 2021.

Grant Schoenebeck and Fang-Yi Yu. Two strongly truthful mechanisms for three heterogeneous agents answering one question. ACM Transactions on Economics and Computation , 10(4):1-26, 2023.

Grant Schoenebeck, Fang-Yi Yu, and Yichi Zhang. Information elicitation from rowdy crowds. In Proceedings of the Web Conference 2021 , pages 3974-3986, 2021.

Sören W Scholz, Martin Meissner, and Reinhold Decker. Measuring consumer preferences for complex products: A compositional approach basedonpaired comparisons. Journal of Marketing Research , 47(4):685-698, 2010.

Nihar Shah, Sivaraman Balakrishnan, Aditya Guntuboyina, and Martin Wainwright. Stochastically transitive models for pairwise comparisons: Statistical and computational issues. In International Conference on Machine Learning , pages 11-20. PMLR, 2016.

Nihar B Shah, Joseph K Bradley, Abhay Parekh, Martin Wainwright, and Kannan Ramchandran. A case for ordinal peer-evaluation in moocs. In NIPS workshop on data driven education , volume 15, page 67, 2013.

Victor Shnayder, Arpit Agarwal, Rafael Frongillo, and David C. Parkes. Informed truthfulness in multi-task peer prediction. In Proceedings of the 2016 ACM Conference on Economics and Computation , EC '16, pages 179-196, New York, NY, USA, 2016. ACM. ISBN 978-1-45033936-0.

Victor Shnayder, Arpit Agarwal, Rafael M. Frongillo, and David C. Parkes. Informed truthfulness in multi-task peer prediction. CoRR , abs/1603.03151, 2016. URL http: //arxiv.org/abs/1603.03151 .

Louis L Thurstone. A law of comparative judgment. In Scaling , pages 81-92. Routledge, 2017.

Amos Tversky and J Edward Russo. Substitutability and similarity in binary choices. Journal of Mathematical psychology , 6(1):1-12, 1969.

Stephan Vail. A stochastic model for utilities. Unpublished manuscript , 1953.

Luis Von Ahn and Laura Dabbish. Labeling images with a computer game. In Proceedings of the SIGCHI conference on Human factors in computing systems , pages 319-326, 2004.

Luis von Ahn and Laura Dabbish. Designing games with a purpose. Commun. ACM , 51(8): 58-67, aug 2008. ISSN 0001-0782. doi: 10.1145/1378704.1378719. URL https://doi. org/10.1145/1378704.1378719 .

Bo Waggoner and Yiling Chen. Output agreement mechanisms and common knowledge. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing , volume 2, pages 220-226, 2014.

Jiaheng Wei, Zuyue Fu, Yang Liu, Xingyu Li, Zhuoran Yang, and Zhaoran Wang. Sample elicitation. In International Conference on Artificial Intelligence and Statistics , pages 26922700. PMLR, 2021.

Dror Weitz. Counting independent sets up to the tree threshold. In Proceedings of the thirtyeighth annual ACM symposium on Theory of computing , pages 140-149, 2006.

Jens Witkowski and David C. Parkes. A Robust Bayesian Truth Serum for Small Populations. In Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI 2012) , 2011.

Jens Witkowski and David C. Parkes. Peer prediction without a common prior. In Proceedings of the 13th ACM Conference on Electronic Commerce, EC 2012, Valencia, Spain, June 4-8, 2012 , pages 964-981. ACM, 2012.

Peter Zhang and Yiling Chen. Elicitability and knowledge-free elicitation with peer prediction. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems , pages 245-252. International Foundation for Autonomous Agents and Multiagent Systems, 2014.

Yichi Zhang and Grant Schoenebeck. High-effort crowds: Limited liability via tournaments. In Proceedings of the ACM Web Conference 2023 , WWW'23, page 3467-3477, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450394161. doi: 10.1145/ 3543507.3583334. URL https://doi.org/10.1145/3543507.3583334 .

Shuran Zheng, Fang-Yi Yu, and Yiling Chen. The limits of multi-task peer prediction. In Proceedings of the 22nd ACM Conference on Economics and Computation , pages 907-926, 2021.

A Further discussion on BPP payment

In this section, we discuss the connection of bonus-penalty payment and existing peer prediction mechanisms. First, if we substitute the third input with a uniformly random bit, denoted as ˆ 𝑠 𝑘 = 𝑍 ∼ 𝑢 {-1 , 1 } , the bonus-penalty payment simplifies to the agreement mechanism [62, 61, 63], one of the most basic peer prediction mechanisms,

However, the agreement mechanism is not symmetrically strongly truthful, as all agents always reporting 1 and -1 can result in higher payments than truth-telling.

The bonus-penalty payment eq. (1) is originally proposed by [11, 57] for the multi-task setting. Our BPP mechanism in Mechanism 3 can be seen as a generalization of multi-task setting. In the multi-task setting, agents works on multiple tasks and for each task the private signals are jointly identically and independently (iid) sampled from a fixed distribution and the each agent's strategy also are iid. Take two agents (Isabel and Julia) and two tasks as an example: Isabel has a private signal ( 𝑠 1 𝑖 , 𝑠 2 𝑖 ) and reports ( ˆ 𝑠 1 𝑖 , ˆ 𝑠 2 𝑖 ) and Julia has ( 𝑠 1 𝑗 , 𝑠 2 𝑗 ) and reports ( ˆ 𝑠 1 𝑗 , ˆ 𝑠 2 𝑗 ) where ( 𝑠 𝑙 𝑖 , 𝑠 𝑙 𝑗 ) are iid from random vector ( 𝑆 𝑖 , 𝑆 𝑗 ) . Isabel and Julia decide their reports on each task using random function 𝜎 𝑖 , 𝜎 𝑗 : {-1 , 1 } ↦→ {-1 , 1 } respectively. Dasgupta and Ghosh [11] use the following payments for Isabel

The payment is a special case of Mechanism 3 by taking the second input as ˆ 𝑠 1 𝑗 and the third input as ˆ 𝑠 2 𝑗 . Additionally, 𝑆 1 𝑗 uniform dominates 𝑆 2 𝑗 for 𝑆 1 𝑖 if and only if

which is called categorical signal distributions [57].

Finally, similar to Shnayder et al. [57], we may extend to non-binary signal setting by extending the payment to

and the definition of uniform dominance to the following.

Definition A.1. Given a random vector ( 𝑆 𝑖 , 𝑆 𝑗 , 𝑆 𝑘 ) ∈ Ω 3 on a discrete domain, we say 𝑆 𝑗 uniformly dominates 𝑆 𝑘 for 𝑆 𝑖 if

for all 𝑠, 𝑠 ′ ∈ Ω with 𝑠 ≠ 𝑠 ′ .

However, the guarantee for truth-telling ( informed truthfulness ) is weaker than the binary setting.

Theorem A.2. Given any discrete domain Ω , if for each agent 𝑖 the associated agent 𝑗 's signal uniformly dominates 𝑘 's signal for 𝑖 's signal (definition A.1), Mechanism 3's scheme is symmetrically informed truthful so that

truth-telling is a strict equilibrium, and

each agent's expected payment in truth-telling is no less than the payment in any other symmetric equilibria and strictly better than any uninformed equilibrium's.

Proof. First truth-telling is a strict equilibrium, because if 𝑆 𝑖 = 𝑠 ,

(by definition A.1)

Additionally, because Pr [ 𝑆 𝑗 = 𝑠 | 𝑆 𝑖 = 𝑠 ] -Pr [ 𝑆 𝑘 = 𝑠 | 𝑆 𝑖 = 𝑠 ] > Pr [ 𝑆 𝑗 = 𝑠 ′ | 𝑆 𝑖 = 𝑠 ] -Pr [ 𝑆 𝑘 = 𝑠 ′ | 𝑆 𝑖 = 𝑠 ] for all 𝑠 ′ ≠ 𝑠 , summing over all possible 𝑠 ′ ∈ Ω on both sides gets Pr [ 𝑆 𝑗 = 𝑠 | 𝑆 𝑖 = 𝑠 ] -Pr [ 𝑆 𝑘 = 𝑠 | 𝑆 𝑖 = 𝑠 ] > 0 and

For any informed equilibrium, by a direct computation E GLYPH<2> 𝑈 𝐵𝑃𝑃 ( ˆ 𝑆 𝑖 , ˆ 𝑆 𝑗 , ˆ 𝑆 𝑘 ) GLYPH<3> = 0 .

Finally, we show that the truth-telling has the maximum expected payment for each agents. When all agent use a strategy 𝜎 : Ω → Ω , agent 𝑖 's expected payment is

Let 𝑓 𝑠 𝑖 ( 𝑠 ) : = ˝ ˆ 𝑠 𝑖 ∈ Ω 𝜎 ( 𝑠 𝑖 , ˆ 𝑠 𝑖 ) 𝜎 ( 𝑠, ˆ 𝑠 𝑖 ) which is between 0 and 1 , because 𝑓 𝑠 𝑖 ( 𝑠 ) ≤ ˝ ˆ 𝑠 𝑖 ∈ Ω 𝜎 ( 𝑠 𝑖 , ˆ 𝑠 𝑖 ) ˝ ˆ 𝑠 𝑖 ∈ Ω 𝜎 ( 𝑠, ˆ 𝑠 𝑖 ) = 1 . Then the expectation becomes

The inequality holds because 𝑓 𝑠 𝑖 ∈ [ 0 , 1 ] and definition A.1. Therefore, we complete the proof. □

B Proofs in Section 2: Bayesian SST model and other models

The proofs of propositions 2.3 and 2.5 are standard, and variations can be found in related literature. We include proofs here for completeness.

Proof of proposition 2.3. First given 𝜃 ∈ R A , for all distinct 𝑎, 𝑎 ′ , 𝑎 ′′ ∈ A , Pr [ 𝑇 𝜃 ( 𝑎, 𝑎 ′ ) = 1 ] , Pr [ 𝑇 𝜃 ( 𝑎 ′ , 𝑎 ′′ ) = 1 ] > 1 / 2 implies that 𝜃 𝑎 -𝜃 𝑎 ′ > 0 and 𝜃 𝑎 ′ -𝜃 𝑎 ′′ > 0 becuase 𝐹 is strictly increasing and 𝐹 ( 0 ) = 1 / 2 . Because 𝜃 𝑎 -𝜃 𝑎 ′′ = 𝜃 𝑎 -𝜃 𝑎 ′ + 𝜃 𝑎 ′ -𝜃 𝑎 ′′ > max ( 𝜃 𝑎 -𝜃 𝑎 ′ , 𝜃 𝑎 ′ -𝜃 𝑎 ′′ ) , we have

and thus 𝑇 𝜃 is strongly stochastically transitive for all 𝜃 with distinct coordinates which happens surely as 𝜈 is non-atomic. Finally, since the distribution on 𝜃 is exchangeable on each coordinate, E [ E [ 𝑇 𝜃 ( 𝑎, 𝑎 ′ )]] = 0 for all 𝑎, 𝑎 ′ . □

Proof of proposition 2.5. First given 𝜃 ∈ Θ , for all distinct 𝑎, 𝑎 ′ ∈ A , if the rank of 𝑎 is higher than 𝑎 ′ ,

where ℎ 𝜂 ( 𝑥 ) = 𝑥 1 -exp (-𝜂𝑥 ) by Busa-Fekete et al. [7].

Claim B.1. For any 𝜂 > 0 and 𝑥 ∈ Z > 0 , the difference ℎ 𝜂 ( 𝑥 + 1 ) -ℎ 𝜂 ( 𝑥 ) is increasing and larger than 1 / 2 where ℎ 𝜂 ( 𝑥 ) = 𝑥 1 -exp (-𝜂𝑥 ) .

By claim B.1, Pr [ 𝑇 𝜃 ( 𝑎, 𝑎 ′ ) = 1 ] , Pr [ 𝑇 𝜃 ( 𝑎 ′ , 𝑎 ′′ ) = 1 ] > 1 / 2 implies that 𝜃 ( 𝑎 ′ ) -𝜃 ( 𝑎 ) > 0 and 𝜃 ( 𝑎 ′′ ) -𝜃 ( 𝑎 ′ ) > 0 . Thus, 𝜃 ( 𝑎 ′′ ) -𝜃 ( 𝑎 ) > max ( 𝜃 ( 𝑎 ′′ ) -𝜃 ( 𝑎 ′ ) , 𝜃 ( 𝑎 ′′ ) -𝜃 ( 𝑎 ′ )) , and

where the second inequality is due to claim B.1. Therefore, 𝑇 𝜃 is strongly stochastically transitive for all 𝜃 . Finally, E [ E [ 𝑇 𝜃 ( 𝑎, 𝑎 ′ )]] = 0 for all 𝑎, 𝑎 ′ since 𝜃 is an uniform distribution on rankings. □

Proof of claim B.1. We first prove that the function ℎ 𝜂 ( 𝑥 ) = 𝑥 1 -exp (-𝜂𝑥 ) is increasing and strictly convex on 𝑥 ≥ 0 . Because ℎ 𝜂 ( 𝑥 ) = 1 𝜂 ℎ 1 ( 𝜂𝑥 ) , for all 𝜂, 𝑥 , it is sufficient to consider 𝜂 = 1 . First, ℎ ′ 1 ( 𝑥 ) = 1 -( 𝑥 + 1 ) 𝑒 -𝑥 ( 1 -𝑒 -𝑥 ) 2 > 0 , so ℎ 1 is increasing. Second, as ℎ ′′ 1 ( 𝑥 ) = 𝑒 -𝑥 ( ( 𝑥 -2 )+( 𝑥 + 2 ) 𝑒 -𝑥 ) ( 1 -𝑒 -𝑥 ) 3 , to show ℎ ′′ 1 ( 𝑥 ) > 0 for all 𝑥 > 0 , it is sufficient to show that 𝑔 ( 𝑥 ) = ( 𝑥 -2 ) + ( 𝑥 + 2 ) 𝑒 -𝑥 > 0 . Because 𝑔 ( 0 ) = 0 and 𝑔 ′ ( 𝑥 ) = 1 - ( 𝑥 + 1 ) 𝑒 -𝑥 > 0 , 𝑔 ( 𝑥 ) > 0 for all 𝑥 > 0 . Therefore, ℎ 1 is strictly convex.

Onthe other hand, ℎ 𝜂 ( 𝑥 + 2 )-ℎ 𝜂 ( 𝑥 + 1 ) > ℎ 𝜂 ( 𝑥 + 1 )-ℎ 𝜂 ( 𝑥 ) for all 𝑥 by convexity, and ℎ 𝜂 ( 2 )-ℎ 𝜂 ( 1 ) = 1 1 + 𝑒 -𝜂 > 1 2 which completes the proof. □

C Proofs in Section 3 and 4

C.1 Uniform dominance from Bayesian SST

Proof of lemma 4.2. With a prior similar assumption for Bayesian SST model, we only need to show

and the other case Pr [ 𝑆 ( 𝑎 ′′ , 𝑎 ′ ) = -1 | 𝑆 ( 𝑎, 𝑎 ′ ) = -1 ] > Pr [ 𝑆 ( 𝑎 ′′ , 𝑎 ) = -1 | 𝑆 ( 𝑎, 𝑎 ′ ) = -1 ] follows by symmetry. To prove eq. (5), we can rewrite the conditional probability in expectations of 𝑇 𝜃 .

Claim C.1. For any strongly stochastically transitive 𝑇 𝜃 on A , and distinct 𝑎, 𝑎 ′ , 𝑎 ′′ ∈ A

With claim C.1, we have

This completes the proof of eq. (5), and thus the uniform dominance.

□

Proof of claim C.1. We let 𝑄 ( 𝛼, 𝛼 ′ ) : = E [ 𝑇 𝜃 ( 𝛼, 𝛼 ′ ) | 𝜃 ] = 2 Pr [ 𝑇 𝜃 ( 𝛼, 𝛼 ′ ) = 1 | 𝜃 ] -1 for all 𝛼, 𝛼 ′ . Note that 𝑄 ( 𝛼, 𝛼 ′ ) > 0 if and only if Pr [ 𝑇 𝜃 ( 𝛼, 𝛼 ′ ) = 1 | 𝜃 ] > 1 / 2 and 𝑄 ( 𝛼, 𝛼 ′ ) = -𝑄 ( 𝛼 ′ , 𝛼 ) .

By symmetry, let 𝑄 ( 𝑎, 𝑎 ′ ) > 0 . It is sufficient to show that

If 𝑄 ( 𝑎 ′ , 𝑎 ′′ ) > 0 , by definition 2.1 𝑄 ( 𝑎, 𝑎 ′′ ) > 𝑄 ( 𝑎 ′ , 𝑎 ′′ ) > 0 so 𝑄 ( 𝑎 ′′ , 𝑎 ′ ) > 𝑄 ( 𝑎 ′′ , 𝑎 ) . Now consider 𝑄 ( 𝑎 ′ , 𝑎 ′′ ) < 0 . If 𝑄 ( 𝑎 ′′ , 𝑎 ) < 0 , 𝑄 ( 𝑎 ′′ , 𝑎 ′ ) > 0 > 𝑄 ( 𝑎 ′′ , 𝑎 ) . If 𝑄 ( 𝑎 ′′ , 𝑎 ) > 0 , we have 𝑄 ( 𝑎 ′′ , 𝑎 ) > 0 , 𝑄 ( 𝑎, 𝑎 ′ ) > 0 , and thus 𝑄 ( 𝑎 ′′ , 𝑎 ′ ) > 𝑄 ( 𝑎 ′′ , 𝑎 ) by definition 2.1 □

C.2 Uniform dominance and weak notions of stochastic transitivity

There are weaker forms of stochastic transitivity, raising the question of whether they are sufficient for uniform dominance as in lemma 4.2. We show that general weak stochastic transitivity is not sufficient. Additionally, we show that although the noisy sorting model from [5] is only weakly stochastically transitive but does not satisfy definition 2.1, it exhibits uniform dominance.

Definition C.2 ([13]) . A stochastic comparison function, 𝑇 : A 2 →{-1 , 1 } , is weakly stochastically transitive if for all 𝑎, 𝑎 ′ , 𝑎 ′′ ∈ A with Pr [ 𝑇 ( 𝑎, 𝑎 ′ ) = 1 ] > 1 / 2 and Pr [ 𝑇 ( 𝑎 ′ , 𝑎 ′′ ) = 1 ] > 1 / 2 ,

Compared to definition 2.1, the weak stochastic transitivity only require the item 𝑎 is favorable than 𝑎 ′′ . Below we provide a simple weakly stochastically transitive example with a prior similar property that does not satisfy the uniform dominance in eq. (5).

Example C.3. Consider the set of three items and Θ consists of all ranking on A with uniform prior where 𝜃 maps each items to its value. Given 𝜃 ∈ Θ so that if 𝜃 ( 𝑎 ) > 𝜃 ( 𝑎 ′ ) > 𝜃 ( 𝑎 ′′ ) ,

Note that the model is weakly stochastically transitive, because an item with a larger value is more favorable and the weak stochastic transitivity is reduced to transitivity on the values. However, the model is not strongly stochastically transitive, because Pr [ 𝑇 𝜃 ( 𝑎, 𝑎 ′′ ) = 1 ] = 0 . 6 < max { Pr [( 𝑇 ( 𝑎, 𝑎 ′ ) = 1 ] , Pr [( 𝑇 ( 𝑎 ′ , 𝑎 ′′ ) = 1 ]]} = 0 . 9 . Finally, as the rank 𝜃 has a uniform prior, the model satisfies a prior similar assumption.

To conclude the example, we show that eq. (5) does not hold for the above model. By direct computation over all six possible ranking 𝜃 , we have

but Pr [ 𝑆 ( 𝑎 ′′ , 𝑎 ) = 1 | 𝑆 ( 𝑎, 𝑎 ′ ) = 1 ] = 1 2 GLYPH<16> 1 + 64 6 GLYPH<17> . Therefore, we have Pr [ 𝑆 ( 𝑎 ′′ , 𝑎 ′ ) = 1 | 𝑆 ( 𝑎, 𝑎 ′ ) = 1 ] < Pr [ 𝑆 ( 𝑎 ′′ , 𝑎 ) = 1 | 𝑆 ( 𝑎, 𝑎 ′ ) = 1 ] , and show that eq. (5) does not hold.

Though the above example shows that weak stochastic transitivity is not sufficient. 5 Below we show a popular weakly stochastically transitive model in Braverman and Mossel [5] has uniform dominance as in lemma 4.2.

Example C.4. Let Θ be the set of rankings on A and 𝜂 > 0 be a parameter. Given a uniformly distributed reference ranking 𝜃 ∈ Θ , the noise ranking model [5] ensures that for all 𝜃 ( 𝑎 ) > 𝜃 ( 𝑎 ′ )

Note that the above model does not satisfy the strict inequality in definition 2.1, but by direct computation, Pr [ 𝑆 ( 𝑎 ′′ , 𝑎 ′ ) = 1 | 𝑆 ( 𝑎, 𝑎 ′ ) = 1 ] = 1 2 GLYPH<16> 1 + 4 𝛾 2 3 GLYPH<17> and Pr [ 𝑆 ( 𝑎 ′′ , 𝑎 ) = 1 | 𝑆 ( 𝑎, 𝑎 ′ ) = 1 ] =

5 In the above example, we can also decrease 0 . 9 to a smaller number that satisfies both uniform dominance and weak stochastic transitivity.

C.3 Symmetrically strongly truthful from uniform dominance

Proof of lemma 4.3. Suppose 𝑆 𝑖 = 1 . Because Pr [ 𝑆 𝑗 = 1 | 𝑆 𝑖 = 1 ] > Pr [ 𝑆 𝑘 = 1 | 𝑆 𝑖 = 1 ] , Pr [ 𝑆 𝑗 = -1 | 𝑆 𝑖 = 1 ] < Pr [ 𝑆 𝑘 = -1 | 𝑆 𝑖 = 1 ] . Therefore, arg max ˆ 𝑠 ∈{-1 , 1 } Pr [ 𝑆 𝑗 = ˆ 𝑠 | 𝑆 𝑖 = 1 ] -Pr [ 𝑆 𝑘 = ˆ 𝑠 𝑖 | 𝑆 𝑖 = 1 ] = 1 . Identical argument holds for the case of 𝑆 𝑖 = -1 which completes the proof.

Additionally, the expected payment of truth-telling is

>

The last inequality holds due to definition 4.1.

□

Proof of lemma 4.4. As 𝜎 is uninformed, let 𝜇 ( 𝑠 ) = 𝜎 ( 𝑠, 𝑠 ) = 𝜎 (-𝑠, 𝑠 ) and 𝜇 (-𝑠 ) = 𝜎 ( 𝑠, -𝑠 ) = 𝜎 (-𝑠, -𝑠 ) for all 𝑠 .

The first equality holds as the reports are independent of signals.

Let 𝛿 = Pr [ 𝑆 𝑗 = 𝑠 𝑖 | 𝑆 𝑖 = 𝑠 𝑖 ] -Pr [ 𝑆 𝑘 = 𝑠 𝑖 | 𝑆 𝑖 = 𝑠 𝑖 ] > 0 , because 𝑆 𝑗 uniformly dominates 𝑆 𝑘 for 𝑆 𝑖 . Additionally, Pr [ 𝑆 𝑗 = -𝑠 𝑖 | 𝑆 𝑖 = 𝑠 𝑖 ] -Pr [ 𝑆 𝑘 = -𝑠 𝑖 | 𝑆 𝑖 = 𝑠 𝑖 ] = 1 -Pr [ 𝑆 𝑗 = 𝑠 𝑖 | 𝑆 𝑖 = 𝑠 𝑖 ] -1 + Pr [ 𝑆 𝑘 = 𝑠 𝑖 | 𝑆 𝑖 = 𝑠 𝑖 ] = -𝛿 . We have

=

2

𝛿

(

𝜎

(

𝑠

𝑖

,

ˆ

𝑠

𝑖

) -

𝜎

(-

𝑎,

ˆ

𝑠

𝑖

))

,

so arg max ˆ 𝑠 𝑖 ∈{-1 , 1 } E 𝑃,𝜎 GLYPH<2> 𝑈 𝐵𝑃𝑃 ( ˆ 𝑠 𝑖 , ˆ 𝑆 𝑗 , ˆ 𝑆 𝑘 ) | 𝑆 𝑖 = 𝑠 𝑖 GLYPH<3> = arg max ˆ 𝑠 𝑖 ∈{-1 , 1 } { 𝜎 ( 𝑠 𝑖 , ˆ 𝑠 𝑖 ) -𝜎 (-𝑠 𝑖 , ˆ 𝑠 𝑖 )} which completes the proof. □

D Proofs in Section 5.1

Before diving into the proof, we introduce some notations. We further introduce Ising models with bias parameter 𝜶 ∈ R 𝑉 ≥ 0 in addition to 𝜷 where

□

and Pr 𝜶 , 𝜷 [ S = s ] ∝ exp ( 𝐻 ( s )) , for all configuration s . Given 𝑖 ∈ 𝑉 , let the expectation and ratio be

respectively which are monotone to each other. We will omit 𝜶 , 𝜷 when clear. Given a subset 𝑈 ⊆ 𝑉 , s 𝑈 ∈ {-1 , 1 } 𝑈 is a configuration over the nodes in 𝑈 , and s 𝑈 = 1 if 𝑥 𝜄 = 1 for all 𝜄 ∈ 𝑈 . We write Pr [·| S 𝑈 = s 𝑈 ] , 𝜈 𝑖 | S 𝑈 = s 𝑈 , and 𝜌 𝑖 | S 𝑈 = s 𝑈 for the conditional probability, expectation and ratio when the configuration in 𝑈 is fixed as specified by s 𝑈 .

Alower bound for LHS Informally, we want to lower bound the correlation between adjacent 𝑖 and 𝑗 (friends). Note that as we remove edges (setting coordinates of 𝜷 to zeros), the correlation should decrease, and the smallest correlation between neighboring nodes 𝑖 and 𝑗 happens when 𝐸 = {( 𝑖, 𝑗 )} . Lemma D.2 formalizes this idea using the following monotone inequality [44, Theorem 17.2].

Theorem D.1 (Griffiths' inequality) . For any 𝑖 ∈ 𝑉 , 𝜈 𝑖 ( 𝜶 , 𝜷 ) = E 𝜶 , 𝜷 [ 𝑆 𝑖 ] is non-negative and non-decreasing in all 𝛽 𝑗,𝑘 ≥ 0 and 𝛼 𝑗 ≥ 0 with 𝑗 , 𝑘 ∈ 𝑉 .

Lemma D.2. Given 𝑉 and 𝑖, 𝑗 ∈ 𝑉 , for all 𝜶 , 𝜷 , and 𝜷 ′ , if 𝛽 ′ 𝑒 = 𝛽 𝑒 when 𝑒 = ( 𝑖, 𝑗 ) and 𝛽 ′ 𝑒 = 0 otherwise, we have

Proof. First, note that we can write the conditional expectation E 𝜶 , 𝜷 GLYPH<2> 𝑆 𝑖 | 𝑆 𝑗 = 1 GLYPH<3> as marginal expectation. Formally, consider 𝜶 𝜂 so that 𝛼 𝜂 𝜄 = 𝛼 𝜄 if 𝜄 ≠ 𝑗 and 𝛼 𝜂 𝑗 = 𝛼 𝑗 + 𝜂 . Because 𝜂 → 𝜶 𝜂 is non-decreasing, 𝜂 → 𝜈 𝑖 ( 𝜶 𝜂 , 𝜷 ) is non-decreasing by theorem D.1. In addition, Pr 𝜶 𝜂 , 𝜷 [ 𝑆 𝑖 | 𝑆 𝑗 = 𝑠 ] = Pr 𝜶 , 𝜷 [ 𝑆 𝑖 | 𝑆 𝑗 = 𝑠 ] for all 𝑠 , and Pr 𝜶 𝜂 , 𝜷 [ 𝑆 𝑗 = -1 ] = 𝑂 ( 𝑒 -2 𝜂 ) , so

Similarly,

On the other hand, consider 𝜷 𝜆 so that 𝛽 𝜆 𝑒 = 𝛽 𝑒 if 𝑒 ≠ ( 𝑖, 𝑗 ) and 𝛽 𝜆 𝑖, 𝑗 = 𝛽 𝑖, 𝑗 + 𝜆 . By theorem D.1, 𝜈 𝑖 ( 𝜶 𝜂 , 𝜷 𝜆 ) is non-decreasing in 𝜆 for all 𝜂 . Because 𝜷 0 = 𝜷 ′ and 𝜷 1 = 𝜷 , we have

Because 𝜌 𝑖 = 1 + 𝜈 𝑖 1 -𝜈 𝑖 is monotone in 𝜈 𝑖 , the second part follows.

Given 𝜷 ′ defined in lemma D.2, by some direct computation with 𝜶 = 0

An upper bound for RHS Now, we need to upper bound the correlation between non-adjacent 𝑖 and 𝑘 (non-friends). We will use Weitz's self-avoiding walks reduction [65] to upper bound the correlation on general graph 𝐺 by the correlation on trees.

Given a general graph 𝐺 , and an arbitrary node 𝑖 , we can construct the Self Avoiding Walk Tree of 𝐺 rooted at 𝑖 , denoted 𝑇 𝑆𝐴𝑊 ( 𝐺,𝑖 ) , so that Pr [ 𝑆 𝑖 = 1 | S 𝑈 = s 𝑈 ] is the same in 𝐺 as in the tree. We outline the construction. 𝑇 𝑆𝐴𝑊 ( 𝐺,𝑖 ) enumerates all self-avoiding walks in 𝐺 starting at 𝑖 which terminates when it revisits a previous node (closes a cycle). Then, 𝑇 𝑆𝐴𝑊 ( 𝐺,𝑖 ) introduces a leaf with a certain boundary condition. The self-avoiding walk never revisits a node immediately, so there all the leaves with fixed boundary conditions are at least three hops away from node 𝑖 . Note that if 𝐺 has maximum degree 𝑑 , 𝑇 𝑆𝐴𝑊 is a 𝑑 -ary tree.

Theorem D.3 ([65]) . For any 𝜶 , 𝜷 , node 𝑖 ∈ 𝑉 , and configuration s 𝑈 on 𝑈 ⊂ 𝑉 ,

□

First, with the above theorem, we have 𝜈 𝑖 | 𝑆 𝑘 = 1 ( 𝜶 , 𝜷 ) = E 𝜶 , 𝜷 [ 𝑆 𝑖 | 𝑆 𝑘 = 1 ] = E 𝑇 𝑆𝐴𝑊 ( 𝐺,𝑖 ) [ 𝑆 𝑖 | 𝑆 𝑘 = 1 ] . By the monotone property in theorem D.1, setting all two-hop neighbors 𝑈 in 𝑇 𝑆𝐴𝑊 ( 𝐺,𝑖 ) to 1 (recalled that any boundary conditions for 𝑇 𝑆𝐴𝑊 ( 𝐺,𝑖 ) being at least three hops away) increases the conditional expectation,

Let 𝑇 be the tree by truncating 𝑇 𝑆𝐴𝑊 ( 𝐺,𝑖 ) at level 2 . By the Markov property of Ising models, the expectation is equal to the expectation on 𝑇 .

Finally, we can recursively compute the probability ratio 𝜌 𝑖 (and thus expectation 𝜈 𝑖 ) on trees. Specifically, given a rooted tree 𝑇 ′ , we define 𝜌 𝑇 ′ as the ratio of probabilities for the root to be + 1 and -1 respectively, and 𝜌 𝑇 ′ | S 𝑈 = s 𝑈 for the ratio of conditional probabilities. As stated in the following lemma, it is well known (see, for example, [22]) that the ratio of each node can be computed recursively over the children's ratio.

Lemma D.4. Given a tree 𝑇 rooted at 𝑖 with parameter ( 𝜶 , 𝜷 ) and boundary condition s 𝑈 ,

where 𝑗 1 , . . . , 𝑗 𝑑 are children of 𝑖 and 𝑇 𝑙 is the subtree rooted at 𝑗 𝑙 for all 𝑙 .

By the monotone property in theorem D.1, the maximum of right-hand side of eq. (7) happens when 𝑇 is a complete 𝑑 -ary tree with 𝜷 = 𝛽 . Therefore,

Finally, with eqs. (6) and (8), we have 𝜌 𝑖 | 𝑆 𝑗 = 1 ( 𝜶 , 𝜷 ) ≥ 𝑒 2 𝛽 ≥ GLYPH<16> 𝑒 2 ( 𝑑 + 1 ) 𝛽 + 1 𝑒 2 𝛽 + 𝑒 2 𝑑𝛽 GLYPH<17> 𝑑 ≥ 𝜌 𝑖 | 𝑆 𝑘 = 1 ( 𝜶 , 𝜷 ) which implies eq. (4).

Remark D.5. Note that for any graph 𝐺 there exists small enough 𝛽, 𝛽 so that the condition in theorem 5.1 is satisfied, because the inequality become equality when 𝛽 = 𝛽 = 0 , and 𝜕 𝜕𝛽 2 𝛽 𝑑 > 0 = 𝜕 𝜕𝛽 ln 𝑒 2 ( 𝑑 + 1 ) 𝛽 + 1 𝑒 2 𝛽 + 𝑒 2 𝑑𝛽 .

The bound between 𝜷 and 𝑑 is necessary as shown in fig. 3. On the other hand, by the Markov property of the Ising model, the majority of all neighbor's signals is a sufficient statistic, and we can show the majority of all neighbor's signals are uniformly dominant to a non-neighbor's signal. Therefore, we can get a symmetrically strongly truthful mechanism by replacing 𝑗 's reports with the majority of reports from 𝑖 's neighbors.

E Proof of Theorem 5.3

The sufficient condition is done by lemma 4.3, because

For the necessary, given 𝑈 , define 𝐷 ( 𝑠 𝑗 , 𝑠 𝑘 ) = 1 2 GLYPH<0> 𝑈 ( 1 , 𝑠 𝑗 , 𝑠 𝑘 ) -𝑈 (-1 , 𝑠 𝑗 , 𝑠 𝑘 ) GLYPH<1> and 𝜇 ( 𝑠 𝑗 , 𝑠 𝑘 ) = 1 2 ( 𝑈 ( 1 , 𝑠 𝑗 , 𝑠 𝑘 ) + 𝑈 (-1 , 𝑠 𝑗 , 𝑠 𝑘 )) for all 𝑠 𝑗 and 𝑠 𝑘 in {-1 , 1 } . Hence

V1

V2

V3

Vo

Vn1

V4

Vn2

Given a joint distribution satisfying definition 4.1, we let 𝑝 𝑠 𝑖 ( 𝑠 𝑗 , 𝑠 𝑘 ) = Pr [ 𝑆 𝑗 = 𝑠 𝑗 , 𝑆 𝑘 = 𝑠 𝑘 | 𝑆 𝑖 = 𝑠 𝑖 ] and additionally write 𝑝 𝑠 𝑖 = GLYPH<20> 𝑝 𝑠 𝑖 ( 1 , 1 ) 𝑝 𝑠 𝑖 ( 1 , -1 ) 𝑝 𝑠 𝑖 (-1 , 1 ) 𝑝 𝑠 𝑖 (-1 , -1 ) GLYPH<21> . Then definition 4.1 ensures that

Because 𝑈 is truthful for all uniformly dominant tuples, we have

Suppose the following are true

Let 𝜆 = 𝐷 ( 1 , -1 ) > 0 . By eqs. (11) and (12), we have

which completes the proof. Thus, we will construct three joint distributions satisfying definition 4.1 to prove eqs. (11) and (12).

The first joint distribution 𝑝 𝑠 𝑖 1 ( 𝑠 𝑗 , 𝑠 𝑘 ) with 0 < 𝛿 ≤ 1 / 2

Then eq. (10) on the first distribution reduces to

As we take 𝛿 to zero, we prove 𝐷 ( 1 , -1 ) = -𝐷 (-1 , 1 ) . Then plugging in with nonzero 𝛿 , we have 𝐷 ( 1 , -1 ) > 0 and complete the proof of eq. (11).

The second joint distribution 𝑝 𝑠 𝑖 2 ( 𝑠 𝑗 , 𝑠 𝑘 ) with 0 ≤ 𝜖 ≤ 1 is

With eq. (11), eq. (10) reduces to

By taking 𝜖 to zero, we prove 𝐷 ( 1 , 1 ) = 0 . We can prove 𝐷 (-1 , -1 ) = 0 using the similar trick and complete the proof of eq. (12).

F Additional empirical results

F.1 Comparison data

Here we test if the dataset satisfy transitivity property. We denote the proportion of rankings such that item 𝑎 is higher than item 𝑎 ′ in the dataset by 𝑝 𝑎>𝑎 ′ . If 𝑝 𝑎>𝑎 ′ > 1 / 2 , 𝑝 𝑎 ′ >𝑎 ′′ > 1 / 2 , and 𝑝 𝑎>𝑎 ′′ > 1 / 2 , we say the triple of items { 𝑎, 𝑎 ′ , 𝑎 ′′ } empirically satisfies transitivity. If 𝑝 𝑎>𝑎 ′ > 1 / 2 , 𝑝 𝑎 ′ >𝑎 ′′ > 1 / 2 , and 𝑝 𝑎>𝑎 ′′ > max { 𝑝 𝑎>𝑎 ′ , 𝑝 𝑎 ′ >𝑎 ′′ } , we say the triple of items { 𝑎, 𝑎 ′ , 𝑎 ′′ } empirically satisfies strong transitivity. We first test the transitivity of the SUSHI subdataset selected in section 6.1. We find that 100% of the item triples empirically satisfy transitivity, and 69 . 17% of the item triples empirically satisfy strong transitivity. This suggests that our transitivity assumption for the comparison data is mostly aligned.

Moreover, we conducted an experiment on the entire SUSHI dataset without any selection criteria and demonstrated the results in fig. 4. Observe that the ECDF of payments from original human users also dominates the payments under the uninformed strategy and the unilateral deviating strategy. This is consistent with our experimental results in section 6.1. However, there are two minor difference. First the separation of truth-telling from the other two is slightly less prominent than fig. 1 with the selection criteria. This may be due to a slightly lower degree of transitivity across agents with different backgrounds. In particular, we found the average value of 𝑝 𝑎>𝑎 ′′ -max { 𝑝 𝑎>𝑎 ′ , 𝑝 𝑎 ′ >𝑎 ′′ } is 0 . 0559 without the selection criteria which is less than 0 . 0604 with the selection criteria in fig. 1. Second, the fraction of agents receiving positive payments is slightly higher than in fig. 1 ( 0 . 785 and 0 . 763 respectively). This aligned with or empirical (strong) transitively which are 1 and 0 . 7667 compared to the above 1 and 0 . 69117 . Furthermore, we also conducted experiments on other groups of users by changing the selection criteria. Those interested can refer to fig. 5, fig. 6 and table 1 for the results, which further verify the effectiveness of our mechanism.

F.2 Networked data

Alongside fig. 2, Figure 7 and table 2 present empirical results for the top five popular artists in the dataset, excluding Lady Gaga, who are Britney Spears, Rihanna, The Beatles, and Katy Perry. All these settings show similar results. However, the Beatles' data is less conclusive as the payment distribution under the uninformed strategy profile is close to the truth-telling. This observation is

Payment Amount

ECDFof Payments

1.0

truth-telling

1.0

truth-telling

uninformed

unilateral deviation

0.8

0.8

0.6

0.6

ECDF

ECDF

0.4

0.4

0.2

0.2

0.0

0.0

0.4

0.2

0.0

0.2

0.4

0.6

-0.8

0.6

0.4

0.2

0.0

0.2

0.4

0.6

ECDFof Payments

Payment Amount

also documented in Daskalakis et al. [12] which notes that the Ising model performs much better for rock artists than for pop artists. The authors conjecture that this may be due to the highly divisive popularity of pop artists like Lady Gaga and Britney Spears, whose listeners may form dense cliques within the graph.

Note that there is a buck of agent with a payment of around 0 . 5 under the truth-telling. This is because many non-listeners have no listener friends, and payment is 1 - [( 1 -𝑝 ) -𝑝 ] = 2 𝑝 is twice the popularity 𝑝 ≈ 0 . 25 . Moreover, the jump is most minor for the Beatles, and indicates less agreement between non-listeners. Additionally, by the definition of bonus-penalty payment, we can see the payment of deviation is the minus of the truthful payment, so that the ECDF is symmetric around ( 0 , 0 . 5 ) .

Figure 8 further shows the scatter plot of average payment and fraction of agents with positive payments across the top fifty popular artists where all settings have more than 60% percent of agents get positive payment. However, for less popular artists, the performance of our mechanism declines. This is expected, as we cannot provide effective incentives when only one agent listens to an artist.

ECDF of Payments

ECDF of Payments

1.0

truth-telling

1.0

truth-telling

uninformed

unilateral deviation

0.8

0.8

0.6

0.6

ECDF

ECDF

0.4

0.4

0.2

0.2

0.0

00

0.4

0.2

0.0

0.2

0.4

0.6

0.2

0.0

0.2

0.4

0.6

Payment Amount

Payment Amount

ECDF of Payments

ECDF of Payments

1.0

truth-telling

1.0

truth-telling

uninformed

unilateral deviation

0.8

0.8

0.6

0.6

ECDF

ECDF

0.4

0.4

0.2

0.2

00

00

0.2

0.0

0.2

0.4

0.6

0.4

0.2

0.0

0.2

0.4

0.6

Payment Amount

Payment Amount

ECDF of Payments

ECDF of Payments

1.0

truth-telling

1.0

truth-telling

uninformed

unilateral deviation

0.8

0.8

0.6

0.6

ECDF

ECDF

0.4

0.4

0.2

0.2

0.0

0.0

0.4

0.3

0.2

0.1

0.0

0.1

0.2

0.3

0.4

0.3

0.2

0.1

0.0

0.1

0.2

0.3

0.4

Payment Amount

Payment Amount

ECDF of Payments

ECDF of Payments

1.0

users

1.0

users

uninformed

unilateral deviation

0.8

0.8

0.6

0.6

ECDF

ECDF

0.4

0.4

0.2

0.2

0.0七

0.0

0.4

0.2

0.0

0.2

0.4

0.2

0.0

0.2

0.4

Payment Amount

Payment Amount

ECDF of Payments

ECDF of Payments

1.0

truth-telling

1.0

truth-telling

uninformed

unilateral deviation

0.8

0.8

0.6

0.6 -

ECDF

ECDF

0.4

0.4

0.2

0.2

00

00

0.2

0.1

0.0

0.1

0.2

0.3

0.4

0.4

0.2

0.0

0.2

0.4

Payment Amount

Payment Amount

1.0

truth-telling

1.0

truth-telling

uninformed

unilateral deviation

0.8

0.8

0.6

0.6

ECDF

ECDF

0.4

0.4

0.2

0.2

0.0

0.0

1.5

1.0

0.5

0.0

0.5

1.0

1.5

1.5

1.0

0.5

0.0

0.5

1.0

1.5

Payment

Payment

1.0

truth-telling

1.0

truth-telling

uninformed

unilateral deviation

0.8

0.8

0.6

0.6

ECDF

ECDF

0.4

0.4

0.2

0.2

0.0

00

1.5

1.0

0.5

0.0

0.5

1.0

1.5

1.5

-1.0

0.5

0.0

0.5

1.0

1.5

Payment

Payment

ECDF for The Beatles

ECDF for The Beatles

1.0

truth-telling

1.0

truth-telling

uninformed

unilateral deviation

0.8

0.8

0.6

0.6

ECDF

ECDF

0.4

0.4

0.2

0.2

0.0

+0°0

1.5

1.0

0.5

0.0

0.5

1.0

1.5

1.5

1.0

0.5

0.0

0.5

1.0

1.5

Payment

Payment

1.0

truth-telling

1.0

truth-telling

uninformed

unilateral deviation

0.8

0.8

0.6

0.6

ECDF

ECDF

0.4

0.4

0.2

0.2

00

00

1.5

1.0

0.5

00

0.5

1.0

1.5

1.5

1.0

0.5

0.0

0.5

1.0

1.5

Payment

Payment

Truthfulpaymentaccrosstopfiftyartists

1.0

30

fraction of non-negative payment

0.9

25

Popularity (percent)

0.8

20

0.7

15

0.6

0.5

10

0.000.05 0.100.15 0.200.250.300.350.40

averagepayment

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our claims accurately reflect the contributions and scope of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: In section 7, we discuss potential future research directions, which are the limitations of our current work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We present all the assumptions. The complete proofs are provided in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The code is uploaded in the supplementary material. All the information required to reproduce the experimental results is provided.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is uploaded in the supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Most of these details are explained in section 6 and in the provided code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We believe the error bars are not relevant to our empirical metrics.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We believe the computer resources are not relevant to our main contributions.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in our paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our work contributes to the theory of information elicitation. We discussed the applicability and limitations for elicitation settings.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We believe this paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The datasets used in this paper are mentioned with URLs and the licenses and terms of use are properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies elicting comparison data with truthfulness guarantees, specifically strict Bayesian Nash equilbrium. The authors utilize strong stochastic transitivity to define a Bayesian strong stochastic transitivity model and generalize several existing models. Under this model (i.e., problem setting), the authors propose a peer prediction mechanism based on bonus-penalty and show that it achieves symmetrically strong truthfulness, for both comparison data. Moreover, the authors also identify the conditions (in theoretical results) and the corresponding mechanism that achieves symmetric strong truthfulness for networked data and the general setting. Empirical results on two real-world datasets show that the proposed mechanism achieves better truthfulness or incentive, by showing that being truthful leads to a higher payment.\n\nSTRENGTHS:\n- The studied problem is important and relevant.\n- The theoretical rigor is appreciated.\n- The proposed approach seems sound, and the definition of Bayesian SST to generalize several existing models is interesting.\n\nWEAKNESSES:\n- An important requirement or assumption is the admissible condition to ensure that the assignment contains the necessary triplets. While the authors describe that one can create a superset, with a bounded size, to ensure the admissible condition. It appears that in doing so, one may need the agents to observe and report additional pairs of items (correct me if I am wrong). If so, in practice, one may not always be able to ask the agents to make addtional observations on required pairs of items (e.g., customers' reviews of products of services).\n\n- The abstract and introduction motivates the importance of eliciting comparison data by citing several applications and use cases. However, the covered use cases by the empirical results seem to be less extensive than these.\n\nQUESTIONS:\nIn lines 132-133,\n\n> ... where the randomness of $S(\\cdot, \\cdot)$ comes from both $\\theta$ and $T_{\\theta}$\nThe randomness of $\\theta$ is due to $\\theta \\sim P_{\\Theta}$. What is the randomness of $T_{\\theta}$? How should one interpret this, and its implications?\n\nWhat other families of strategies are useful? Can strategically mis-behaving agents be accounted for, for instance to exploit this system to minimize someone's expected payment or intentionally aiming to obfuscate identification of the true comparison or ranking? In other words, are these considered families of strategies sufficient to consider some robustness to mis-behaving agents (and which types)?\n\nIn lines 173-174,\n\n> ... she would expect that others will ... prefer $a$ over $a''$\n\nWhat is the intuition behind this? Why would $a$ be preferred over $a''$ when the agent itself only has information that $a$ is preferred over $a'$?\n\nIn line 190-191,\n> Hence, theorem 3.1 implies that agents’ manipulations can only decrease the probability of transitivity among their reports.\n\nWhat is the implication of \"decreasing the probability of transitivity among their reports\"?\n\nLIMITATIONS:\nThe method requires a specific problem setting or model called the Bayesian strong stochastic transitivity. But it is shown to generalize several other existing models.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of eliciting pairwise comparisons truthfully from strategic agents in the absence of ground truth. It considers the popular framework of peer prediction which is a class of mechanisms where reports from \"peer\" agents are used as a proxy for ground truth in order to design a payment scheme. The problem with using existing peer prediction mechanisms is that they mainly work for settings where tasks are drawn from the same distribution. However, this does not apply to the case of pairwise comparisons, as the two items being compared can be very different. \n\nThis paper designs a new peer prediction mechanism that is based on the Dasgupta-Ghosh mechanism where the agents also suffer a penalty in addition to rewards so as to avoid spurious \"agree to agree\" equilibria. The main novelty of the paper comes from the connection it makes to the well-known strong stochastic transitivity (SST) condition for pairwise comparison data. Under the assumption that pairwise comparisons satisfy SST condition, the mechanism cleverly utilizes this condition to ensure that the reward term exceeds the penalty term for truthful agents which is one of the main requirements for truthfulness. The paper also considers other elicitation tasks beyond pairwise comparisons, such as eliciting networked data, and give a general framework for extending.\n\nSTRENGTHS:\nIn my opinion, the paper makes a good contribution to the literature on peer prediction. While the proposed mechanism is mainly based on the Dasgupta-Ghosh mechanism, it is a principled way of extending the Dasgupta-Ghosh mechanism to heterogenous tasks and utilizing task specific properties (such as SST) for incentive design. I really like that the paper connects the peer prediction literature to the ranking literature in a meaningful manner.\n\nWEAKNESSES:\nI am only concerned about the relevance of this paper to NeurIPS as this is mainly a mechanism design paper. But given the usefulness of pairwise comparisons in training LLMs, there is definitely need for truthful elicitation of comparison data.\n\nThe results on networked elicitation seem a bit disconnected from the results on elicitation of comparison data.\n\nQUESTIONS:\nPerhaps the framework also works for weak stochastic transitivity?\n\nLIMITATIONS:\nI did not find adequate discussion about limitations. Perhaps a section on limitations of the mechanism and peer prediction in-general might be useful.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n* This paper proposes a peer prediction mechanism for elicitation of comparison data.\n* In the model, there is a collection of items $A$, and a set of agents $N$ which privately observe noisy comparisons between items. Comparisons are characterized by a stochastic comparison function $T_\\\\theta$, parameterized by $\\\\theta \\\\in \\\\Theta$ and drawn from a common prior $P_\\\\Theta$.\n* A peer prediction mechanism assigns a pair of items for each agent $i$ to compare. Each agent then observes a noisy comparison signal $s_i$ and strategically reports a possibly different value $\\\\hat{s}_i$ to maximize their ex-ante payment $\\\\mathbb{E} [M_i (\\\\hat{\\\\mathbf{s}})]$. The goal is to design a mechanism $M$ which elicits truthful reporting.\n* The authors present Mechanism 1 for elicitation of comparison data, and prove that it is symmetrically strongly truthful (Theorem 3.1). The mechanism is based on a bonus-penalty payment function, which for agent $i$ awards agreement with some agent $j$, and penalizes agreement with some agent $k$.\n* Section 5.1 presents a generalization of the mechanism to networked data, proving that it is symmetrically strongly truthful for signals sampled from the graph Ising model (Theorem 5.1). Section 5.2 further generalizes by showing a mechanism which is symmetrically strongly truthful for data with uniform dominance.\n* Finally, empirical evaluation is performed on two datasets (Sushi preferences and Last.fm). Results seem to show stochastic dominance of payout in truthful reporting vs. random reporting.\n\nSTRENGTHS:\n* Presentation is clear and concise. Intuition is given for formal proofs.\n* The concept of uniform dominance is interesting and seems to be generalizable.\n* Theoretical results seem to be quite general, and applicable to many pairwise choice models.\n\nWEAKNESSES:\n* Empirical evaluation compares truthful reporting against benchmarks which seem relatively “simple” - uninformed random reporting, as opposed to more sophisticated forms of strategic behavior.\n* Empirical evaluation procedure is not sufficiently clear. Method of constructing the graphs in the empirical section is not presented sufficiently formally. Graph legends and captions are uninformative.\n* Implications of assumptions are not discussed in detail. Examples for such assumptions - assuming symmetric deviation, the ability to penalize users, assuming that items are a priori similar but ex-post distinct. See below.\n\nQUESTIONS:\n* What is the relation between Mechanism 3 and Mechanisms 1,2? Is it possible to think about Theorem 3.1 and Theorem 5.1 as corollaries of Theorem 5.2? If not, how do they fundamentally differ?\n* “We assume items are a priori similar but ex-post distinct” L106 - What are the implications of this assumption? When does it apply, and does it restrict generality?\n* For the given mechanism, is it possible to have a better BNE which is not symmetric? (e.g, a beneficial equilibrium where only some agents report truthfully?)\n* What would be the implications of agents having limited liability? (e.g if the agents can choose not to participate in the game when they may get negative reward)\n* Possible typo in L318: “ The figure shows that only about 50% of the agents using an uninformed random strategy receive positive payments, while over 75% of the users in the original dataset receive positive payments” - Percentages seem to add to more than 100%.\n\nLIMITATIONS:\nResults seem to rely on relatively strong assumptions, such as symmetric deviation, and the ability to penalize users. Despite that, such limitations don’t seem to be discussed in detail.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}