Title: Stochastically Dominant Peer Prediction
Venue: NeurIPS 2025
Accepted: yes
Keywords: Information elicitation

mechanism design

crowdsourcing
Abstract: Eliciting reliable human feedback is essential for many machine learning tasks, such as learning from noisy labels and aligning AI systems with human preferences. Peer prediction mechanisms incentivize truthful reporting without ground truth verification by scoring agents based on correlations with peers. Traditional mechanisms, which ensure that truth-telling maximizes the \textbf{expected scores} in equilibrium, can elicit honest information while assuming agents' utilities are \textbf{linear functions} of their scores. However, in practice, non-linear payment rules are usually preferred, or agents' utilities are inherently non-linear.

We propose \emph{stochastically dominant truthfulness (SD-truthfulness)} as a stronger guarantee: the score distribution of truth-telling stochastically dominates all other strategies, incentivizing truthful reporting for a wide range of monotone utility functions. Our first observation is that no existing peer prediction mechanism naturally satisfies this criterion without strong assumptions. A simple solution - rounding scores into binary lotteries — can enforce SD-truthfulness, but often degrades \emph{sensitivity}, a key property related to fairness and statistical efficiency. We demonstrate how a more careful application of rounding can better preserve sensitivity. Furthermore, we introduce a new enforced agreement (EA) mechanism that is theoretically guaranteed to be SD-truthful in binary-signal settings and, under mild assumptions, empirically achieves the highest sensitivity among all known SD-truthful mechanisms.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Enforcement Agreement, a new mechanism for the peer prediction problem that satisfies the novel SD-truthfulness criteria. Whereas other mechanisms rely on linear utility functions in order to satisfy Truthfulness, SD-truthful mecahnisms incentivize truth-telling for general increasing utility functions. It is shown that partition rounding can translate any truthful mechanism with bounded scores into a SD-truthful one (Prop 4.4). Furthermore, EA is shown to be SD-truthful for the binary setting when signals are self-predicting (Thm 5.1), and demonstrated to have high sensitivity. Empirical results compare EA to other mechanisms, including OA, PTS, CA, and MA.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper does a good job at introducing the peer prediction problem, explaining how the new proposed concepts (SD-truthfulness, partial rounding, and EA) compare to prior literature (OA, PTS, and CA), and demonstrating why EA satisfies the SD-truthful criteria. The authors do well explaining how and why EA (which incorporates OA with partial rounding) achieves a much better sensitivity ratio than OA with direct rounding. An empirical study compares the results. Overall I found the paper clear and convincing. While I did not check the proofs in the appendix thoroughly, they appear generally correct.\n\nStill, peer prediction is a complicated concept and the paper could use several edits to clean up the presentation. Here are some major comments:\n- Can you explain the assumption about Bob's reports being truthful on Line 109? You introduce this relationship with Bayes Nash Equilibrium, which makes sense, but don't refer back to it throughout the paper. It's not clear why this assumption, or being in equilibrium, makes sense across these mechanisms.\n- The paper could really make use of a running example to demonstrate these concepts. I think your discussion of Direct and Parititon rounding is clear. But the discussion breaks down when you introduce $\\Phi$ on line 278. It seems like the discussion in Section 5 is significiantly detached from the model introduced in Section 2, and it's immediately clear to the reader how to build this relationship. The paper loses its relationship to the applications in lines 37-40 very quickly, so I struggle to understand how it could be directly applied.\n- Lines 139-143: You introduce two new concepts here -- budget efficiency and measurement integrity. BE is only used in Appendix E.3; MI is not used at all in the paper. I would recommend reframing this paragraph to suggest that these are \"useful to know\" concepts related to sensitivity, as opposed to highlighting them as important for the reader to remember later on.\n- Although you demonstrate EA has better sensitivity, due to partial rounding, you don't explain this in context. It's not clear how good EA is on this measure dimension. \n- The discussion on Lines 304-308 could use significant elaboration in the main text, since they appear to be significant parts of your paper's contribution.\n- It's not clear: (1) what $\\Phi$ is relative to the model introduced in Section 2, (2) what \"information structure\" means, and (3) how the joint distribution $Pr(X_a, X_b)$ affects your results -- or if this is just a modeling necessity.\n\n\nOthers:\n- Fix \\citet -> \\citep citation format on lines 26, 38, 54; also in Appendix A\n- Line 29: fix to \"subjective assessment tasks.\" Otherwise subjectivity is in contrast with \"ground truth\"\n- Not sure if colored variables in Line 298-299 are acceptable for NeurIPS. Please double check.\n- Lines 62-64 high-level description of \"sensitivity\" could be improved.\n- Line 78 -- please explain \"enforced count\" for readers not familiar with the peer prediction literature\n- Line 88 -- why is assessing budget efficiency a \"robustness check\"?\n- Line 130: is $X_a$ the random variable over $\\Sigma$? So that if $e=1$ then this reduces to the model discussed in Sec 2.1?\n- Citation for Prop 3.2? Is there a corresponding necessary condition for OA to be truthful?\n- Line 198 - clarify \"depends on implementation\": either say \"not necessarily\" or also mention which implementation would be SD-truthful. Does this contradict Table 1? Are lines 210-211 important enough to be bolded?\n- Line 227 - how is sensitivity related to practicality? Since this concept is important throughout the paper, it may be worthwhile on elaborating some of the intuitive findings from [Zhang and Schoenebeck, 2023a] on lines 136-138 (not just the technical relationships).\n- Line 334: please include a one sentence description of EA-optimal, and then point to the appendix for the full description\n- Inconsistent reference style in bibliography\n\nQUESTIONS:\n- The K partitions seems to be analogous to Bootstrapping. Do you need the partition assumption in order to get your results -- instead of, i.i.d sampling with replacement?\n- Is line 122-124 the only explanation for why your mechanism can handle monotonic but non-linear utility functions? Also, it's not clear what a utility function is, relative to what is discussed in Section 2.\n- Why are single task mechanisms different than multi-task mechanisms? Line 443 in Appendix A refers to Prelec (2004) but doesn't suggest why ideas from that literature doesn't seem to translate to the problem addressed in this paper.\n\nLIMITATIONS:\nThe main result (Thm 5.1) is limited to binary and self-predicating signals. The authors demonstrate that SD-truthfulness does not hold for 3+ elements. I would really like a better understanding in this paper why self-predicting (Def 3.3) is a useful or important concept with respect to either PTS (in Sec 3.2) or in the proof of Thm 5.1. This is introduced as a sufficient condition, but no context is provided for why this condition is important, or in what types of real-world decision problems this condition holds.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new notion of truthfulness for peer prediction mechanisms called stochastically dominant truthfulness (SD-truthfulness). They then provide several contributions connected to this definition:\n(1) They propose a modification of the (known) rounding/normalization strategy that converts an existing peer-prediction mechanism into one that is SD-truthful, such that the resulting modified peer prediction mechanism additionally has higher sensitivity compared to vanilla rounding.\n(2) They then propose a novel peer prediction mechanism, called Enforced Agreement (EA), that instead of performing using a rounding strategy proceeds via a different method on top of Output Agreement --- this method provably achieves SD-truthfulness in the binary setting (though not generally/necessarily SD-truthful beyond that), and establish several further properties of EA.\n(3) They perform experiments establishing the favorable performance of the SD-truthful methods they have developed, as compared to prior methods with that property --- establishing that SD-truthfulness via their constructions (EA and the \"bucketed rounding\") leads to better sensitivity curves (i.e. to better sensitivity across various levels of effort).\n\nSTRENGTHS AND WEAKNESSES:\nAs the Strengths of this paper, the following stand out:\n\n++ The idea of defining SD-truthfulness is quite simple and neat: due to the canonical link between stochastic dominance and monotonic transformations of random variables, it implies strictly stronger truthfulness guarantees than the linear-utility ones explored before, and thus naturally cover a variety of natural settings where handling a generic monotonic transform of the scores is necessary (such as the given example of students targeting grades A, B, C, and many other conceivable structured settings).\n\n++ The constructions for both the new rounding scheme and for EA are pleasantly simple --- almost to the point of being elegant (being based in particular on clever constructions that permit the use of stochastic coupling to enable meaningful stochastic dominance comparisons), and in that sense are in my opinion a perfect starting point for further, more sophisticated studies/follow-ups that would further explore SD-truthfulness.\n\n++ The empirics do a fairly good job exposing the benefits afforded by the proposed methods, from directly providing sensitivity plots to (in the Appendix) exploring positive consequences on budgeting.\n\nAs far as the weaknesses:\n\n-- Why is the sensitivity notion, as defined, the important/the right one to consider? First off, budgeting and fairness that are mentioned existing applications, are a fair point, but they are not elaborated on sufficiently in the main part for a first-time reader's benefit (nor, unfortunately, in the appendix modulo the extra experiments). More fundamentally, I like the sensitivity definition as-is in the standard context of linear utilities (that are modulo offset proportional to scores) --- but since you are considering general monotonic utilities here as your main contribution, a sensitivity notion that would strike me as far more natural would measure e.g. the worst-case (or average case, etc) sensitivity with respect to the utility functions in a certain class of monotonic utilities. In other words, why should I care more about how sensitive the scores themselves are, rather than the utilities generated from those scores? The moment things become nonlinear these are two different notions.\n\n-- One important technical point got swiped under the rug (despite being mentioned twice: in the intro and in the appendix): For a multitude of mechanisms that are more \"complex\" than the ones discussed in this manuscript, in particular ones based on mutual information etc., it is claimed without further elaboration that due to this complexity, such mechanisms would be \"unlikely\" to be SD-truthful due to the strategic manipulations that are permitted through these complexities. For the readers' benefit, I believe it would be quite critical to get more concrete on this point. If you believe that a general and simple to state class of manipulations would prevent SD-truthfulness from holding, I'd like to request that you please give such a construction in your response/in the paper. On my end, while I completely agree that exact SD-truthfulness is elusive, in my opinion just citing the complexity of scoring rules as a major reason sounds a bit peculiar/non-fundamental. Moreover, when it comes to looking at approximate rather than exact SD-truthfulness, the logic of saying \"the scoring is way too complex, clearly can be exploited\" would seem to not be robust to that (as such complexity could perfectly well hide inside the approximation parameter in that case, and possibly in a pretty inconspicuous way).\n\n-- The paper may not be 100% clearly written from the perspective of a general ML/GT reader at times, which is because the paper was written with a specific audience in mind (more so for a generic EC attendee than a generic NeurIPS attendee); in particular, some notions --- like Bayes Nash equilibrium --- get thrown around quite causally throughout the text, and while it's helpful for the seasoned expert reader, I would have like to see further effort on the accessibility front (i.e. either explain such terms at least informally at least in footnotes, or try to avoid them unless necessary). Moreover, even from the experienced reader's perspective there is some lack of clarity in places, see the above Weaknesses.\n\nQUESTIONS:\nSee more details in the Weaknesses section, but the following are distilled important questions to resolve:\n\n(1) Please concretely discuss why/how SD-truthfulness would get violated (or not) in complex mechanisms (such as mutual information-based).\n\n(2) Please concretely discuss in substantially more detail why sensitivity is an important metric (currently it's a few lines in the middle of the paper, which is not enough for a key notion), and why one shouldn't worry about defining sensitivity with respect to the utility functions in this case.\n\n(3) Please make general clarity-inducing improvements targeting less experienced readers in terms of extra terminology, context from the elicitation literature, etc.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers peer prediction mechanisms for eliciting truthful behavior.\n\nIt proposes a new type of truthfulness guarantee, stochastic dominance-truthful (SD-truthful), in which the probability that the mechanism assigns a score above some threshold $t$ to the truth-telling strategy is greater than the probability that it assigns a score above that threshold for any other strategy, for any threshold $t \\in \\mathbb{R}$.\n\nIt also discusses mechanism sensitivity, a measure of the extent to which the mechanism gives higher scores to more informative reports. The goal of the paper is \"to design SD-truthful mechanisms that also achieve high sensitivity.\"\n\nThe paper makes the following claims:\n\n1. No existing peer prediction mechanism is naturally SD-truthful under general information structures.\n2. Achieving SD-truthfulness is easy in principle but challenging in practice.\n3. The proposed _enforcement agreement_ mechanism is more sensitive than the _direct rounding_ or _partition rounding_ reductions, and requires weaker assumptions (signals are self-predicting) for SD-truthfulness than _output agreement_ (signals are self-dominating).\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The new SD-truthfulness property is presented clearly.\n- The paper does a good job of motivating both SD-truthfulness and sensitivity.\n- The paper reviews several prior mechanisms and well situates the new SD-truthfulness property within that context.\n    - It shows that SD-truthfulness is only achieved (by output agreement) under the strong assumption of self-dominating signals. \n    - It also shows that peer truth serum (PTS) is truthful, but not SD-truthful, under the weaker assumption of self-predicting signals. The justification feels right, although I did not check the details of the proof carefully.\n- The new mechanism, enforced agreement (EA) is clearly presented. It is also SD-truthful for binary, self-predicting signals (which is a weaker assumption than OA requires), while being more sensitive under the partition rounding reduction than correlated agreement (CA), matching agreement (MA), and (usually) PTS.\n\nOverall, this feels like a valuable contribution that future work can build on, both for the SD-truthfulness property and for the novel EA mechanism.\n\n### Weaknesses\n\nThe proposed mechanism, enforced agreement (EA), is not generally SD-truthful in the non-binary setting. The paper does a good job of pointing out this limitation in the main text, and discusses it in depth in Appendix C. It also points out that EA is still truthful as long as the information structure is known. I am not an expert, so I am not sure what this information structure assumption means or whether it is reasonable.\n\nThe explanation of \"effort\" in the definition of sensitivity was hard for me to follow. I get it at an intuitive level, but the clarity of this section could definitely be improved.\n\nQUESTIONS:\n1. In Figure 1, the paper provides a comparison of the sensitivity of different mechanisms. Is there a way to also evaluate and visualize truthfulness?\n\n2. The legend for Figure 1 (a) and (b) includes CA-partition-round, but I don't see the red line. Is it missing from the plots?\n\nLIMITATIONS:\nSee weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study mechanism design for peer prediction. Whereas theoretically scoring rules that guarantee truthfulness in expectation is sufficient, in practice non-linear payment rules are preferred for incentivizing truthful behavior. To address this discrepancy, the authors introduce a new stronger notion of truthfulness namely stochastically dominant (SD) truthfulness where the truth-telling reward distribution must first-order dominate the reward distributions of all other strategies.\n\nThe authors observe that SD-truthfulness does not hold for many mechanisms but can be trivially be attained by rounding all rewards to two extreme values so that the expectation is preserved. This directly yields SD-truthful mechanisms but introduces significantly higher variance in rewards. To address this, they seek to optimize an additional objective, namely sensitivity. They propose an optimized SD-truthful mechanism called Enforced Agreement (EA) that reduces sensitivity and empirically show that its levels of sensitivity closely follow that of other peer-prediction mechanisms while also being SD-truthful.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper clearly motivates why expected utility truthfulness is insufficient in practice. It introduces SD-truthfulness as a more robust guarantee for arbitrary monotonic preferences. It proves that classical mechanisms (Output Agreement, PTS, Correlated Agreement) do not meet this stronger criterion and designs EA, a novel and creative mechanism to achieve SD-truthfulness in binary domains.\n\nWeaknesses\nAn important weakness with the proposed mechanism is the lack of theoretical exploration. The paper mainly proposes the EA mechanism as a novel alternative but does not establish its theoretical performance or optimality in some domain. There could be other mechanisms that are better and even not-restricted to give binary outputs.\nMoreover, while the proposed mechanism aims to be more practically relevant, its limitation to binary-signals feels substantial in practice.\n\nFinally, the main body of the paper seems very imbalanced. It spends many pages describing the limitations of other mechanisms and spends less than a page to the description of the proposed alternative which very dense, hard to parse and only focused on the SD-truthfulness of EA without motivation.\n\nQUESTIONS:\n- Is it possible to theoretically establish optimal mechanisms that satisfy SD-truthfulness for any domain. Is EA optimal in any regard?\n- Is there a characterization for SD-truthful mechanisms?\n\nLIMITATIONS:\nThe authors clearly state the limitations of their proposed mechanism.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-12"}