Evaluating LLM-contaminated Crowdsourcing Data Without Ground Truth
Authors: Yichi Zhang
Jinlong Pang
Zhaowei Zhu
Yang Liu
Venue: NeurIPS 2025 poster ()
Keywords: LLM-contamination
crowdsourcing
peer prediction
data evaluation
Primary area: other
TL;DR: A peer prediction-based automatic evaluator for scoring human values in crowdsourcing datasets contaminated by LLM-generated responses.

Abstract: The recent success of generative AI highlights the crucial role of high-quality human feedback in building trustworthy AI systems. However, the increasing use of large language models (LLMs) by crowdsourcing workers poses a significant challenge: datasets intended to reflect human input may be compromised by LLM-generated responses. Existing LLM detection approaches often rely on high-dimensional training data such as text, making them unsuitable for structured annotation tasks like multiple-choice labeling. In this work, we investigate the potential of peer prediction --- a mechanism that evaluates the information within workers' responses --- to mitigate LLM-assisted cheating in crowdsourcing with a focus on annotation tasks. Our method quantifies the correlations between worker answers while conditioning on (a subset of) LLM-generated labels available to the requester. Building on prior research, we propose a training-free scoring mechanism with theoretical guarantees under a novel model that accounts for LLM collusion. We establish conditions under which our method is effective and empirically demonstrate its robustness in detecting low-effort cheating on real-world crowdsourcing datasets.

Evaluating LLM-Contaminated Crowdsourcing Data Without Ground Truth
Yichi Zhang* DIMACS, Rutgers University yz1636&dimacs. matteodderass..uteeee.e rutgers......
Jinlong Pang"
University of California, Santa Cruz ppang14@ucsc.com edu
Zhaowei Zhu
Yang Liu
University of California, Santa Cruz zuzhuβucsc.edu
University of California, Santa Cruz yangliuQucsc.
Abstract
The recent success of generative Al highlights the crucial role high-quality human feedback in building trustworthy Al systems. However, the increasing large language models (LLMs) by crowdsourcing workers poses a significant challenge: datasets intended to reflect human input may be compromised by LLM-generated responses. Existing LLM detection approaches often rely on high-dimensional training data such text, making them unsuitable for structured annotation tasks like multiple-choice labeling. In this work, we investigate the potential of peer prediction- mechanism that evaluates the information within workers' responses-to mitigate LLM-assisted cheating in crowdsoureing with focus on annotation tasks. Our approach quantifies the correlations between worker answers while conditioning on (a subset of) LLM-generatee labels available the requester. Building on prior research, we propose a training-free free scoring mechanism with theoretical guarantees under novel model that accounts for LLM collusion. We establish conditions under which our method effective and empirically demonstrate its robustness in detecting low-effort cheating on real-world crowdsoureing datasets.
Introduction
High-quality human feedback plays crucial role in shaping modern Al systems, such ChatGPT; to be trustworthy, safe, and aligned with human preference Ouyang et al. [202)). This feedback is usually collected through crowdsourcing, where requester posts tasks and recruits trained experts or general participants to complete them. For example, platforms like Amazon Mechanical Turk and Prolific are widely used by researchers for tasks such as data labeling and survey administration.
However, the integrity of this data collection process is increasingly threatened by the very Al models it seeks to improve. Evidence suggests that many agents rely large language models (LLMs) to complete tasks such as abstract summarization [Veselovsky et al. 2023] and peer review [iing et al. 2024]. leading to datasets that longer reflect genuine human input, phenomenon known as L.IM contamination. While M-asiisede annotations or responses may sometimes be more accurate or coherent than human's 's GGlardi al. 2023 Törnberg 2023 Kaikaus al.) 2023) their widespread use undermines the fundamental goal of gathering diverse and authentic human opinions. This issue particularly concerning applications that depend on human subjectivity or expertise, such as preference labeling and toxicity detection, where human judgment remains the gold standard.
Equul contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

How to detect potential LLM-misuse and evaluate the quality of a human-contributed dataset? Existing black-box methods for detecting LLM-generated content primarily focus on distinguishing human-written text from Al-generated text by analyzing statistical differences in writing style [Gehrmann al., 2019 Mao et al., 2024 Koike et al.) [024 Hu et al., 2023]. While effective in open-ended text generation tasks, these training-based approaches are not well-suited for structured annotation tasks like multiple-choice labeling, where the discrete responses lack the rich textual features needed for classification.
In this paper, we develop a theoretically robust scoring metric to evaluate the quality of human contributions in addition to existing LLM answers, effectively distinguishing workers with less informative responses. Our method builds on peer prediction. a scoring mechanism designed to incentivize truthful reporting in the absence of ground truth Miller et al. (2005]. [Shnayder et al. 2016). Traditional peer prediction methods evaluate a worker based on the correlation between her responses and those of her peers. In the standard model, every worker independently observes only high-effort signals, denoted as X; for worker i, after working on set of i.i.d. questions. Peer prediction mechanisms shown to be information monotone, meaning that workers maximize their expected score by truthfully reporting their high-effort signals.
However, these methods are ineffective when workers can coordinate on cheap signals, denoted as Z; for worker that are highly correlated and require minimal effort to produce. In our settings, LLM responses serve as such cheap signals. To address this limitation, we extend peer prediction by measuring correlations between workers' responses while conditioning on a signal Z that is observable to the requester. For example, Z; and Z could represent responses to the questions independently generated by the same L.M or different L.MSs. Intuitively, by conditioning on Z, our method scores the information within workers' responses beyond what can be obtained from LLMs alone, ensuring that human effort is properly rewarded while Al-reliant is penalized.
Theoretically, we identify the conditions under which our mechanism maintains information mono- tonicity under two regimes: (1) for any form of manipulation strategy and (2) for a subset of practical cheating strategies, which we refer to as lazy-reporting strategies. A lazy-reportig strategy, for instance, includes blindly relying on LLM on fraction of the tasks while answering the rest truthfully. To achieve information monotonicity in the general case, for an arbitrary pair of workers, i's cheap signal Z, has to be approximately uncorrelated with j's high-effort signal X, and their cheap signal Zj. conditioned on Z (A (Assumptions] 4.1 and 4.2 For lazy-reporting strategies, weaker condition suffices: conditioning on Z, the correlation between high-effort signals (X,, Xy) is stronger than the correlation between both (Z,, X,) and (Z,, 2,) (Assumption 4.5).
Empirically, we test our assumptions using two subjective labeling datasets and five types of com- monly used LLMs. Our findings show that human responses consistently exhibit stronger correlations than --eeerrrrtttdd responses when samples of and Z; originate from the same model, supporting the validity of Assumption 4.5, However, the correlations between independent samples of LLMs responses are usually non-zero, suggesting that Assumptions 4.1 and 4.2 are too strong to hold in practice. Finally, under scenarios where Assumption 4.5 holds, we evaluate the effectiveness of our method in detecting low-effort agents. Echoing our theoretical insights, we show that our approach has the most robust performance on mixed crowd compared with all the baselines.
Related Work
Our study relates to the studies on LLMs usage in crowdsoureing. Cegin et al. (2023) find that ChatGPT can partially replace human workers in paraphrase generation, and Kaikaus et al. 2023) report that GPT labels are more consistent than humans in sentiment annotations, Ashhtorab et al. [222] demonstrate that "batch labeling", framework of using AI to assist human labeling by allowing a single labeling action to apply to multiple records, can increase the overall labeling accuracy and increase the speed. However, several studies also highlight potential downsides: del Rio-Chanona et al. 2023 report a 16% decline in Stack Overflow activity following the rise of LLMs, Ashwin et al. 2023 find that LLMs exhibit greater bias than human annotators in qualitative analysis. In this context, our paper introduces a scoring mechanism that quantitatively measures the informativeness of human reports relative to the information already provided by Al.
Our work builds on advancements in information elicitation, particularly peer prediction. As a generalization of the correlated agreement (CA) mechanism (Shnayder et al. 2016), our method

---

can score the quality of human responses conditioned on an external information source. We chose the CA mechanism for its simplicity and robust empirical performance [Burrell and Schoenebeck 2021 Zhang and Schoenebeck [023a]. Our approach is also inspired by Kong and Schoenebeck [2018] who proposed using conditioned mutual information to elicit hierarehical information. Their framework assumes an information hierarchy where more effort yields strictly more informative signals, so low-effort signals can be elicited first, which are used to further elicit high-effort signals. In contrast, our method assumes that the principal has access to samples of low-effort signals, which is more straightforward and practical in the LLM-influenced crowdsourcing settings. Lastly, we propose heuristic generalization of our method to handle textual data (Appendix[F) which is related to methods that elicit textual information using LLMs [Lu et al. 2024 Wu and Hartline 2024).
The studies on LLM-content detection are also related. Common approaches for LLM content detection include watermarking [Gu et al. 2022 Lee et al. 2023 Zhao al. 2023 Yang et al. 2023
which embeds identifiable markers into data, models, or generated text; statistical detection, which
leverage differences in output by analyzing logits statistics (Su et al. 2023 Vasilatos et al. 2023 or
vocabulary statistics [Gehrmann et al. 2019 Mao et al. 2024]: and deep learning-based methods, which train models to distinguish Al-generated content from numan-written text [Koike et al. 2024
Hu et al., 2023] Shaah et al. 2023 However, these methods typically rely on abundant textual data for training and detection. In contrast, our approach training-fret free and addresses LoM-oontamination in non-textual responses scenarios, such as answering multi-choice questions.
3 Model
Consider truth-discovery setting where there are i.i.d. questions. The principal (requester) distributes these questions to n agents (workers) such that each agent answers multiple questions and each question is answered by more than one agent. We first present the information structure when two agents, denoted as and j. answer an arbitrary question. Because questions are i.i.d., the same information structure will be applied to any question.
Signals Suppose the underlying ground truth of the question is Y E Σ. For example, if the answer to the question is either "yes" or "no", then ΣΣ| = 2. Suppose agents and j both answer this question. After exerting effort, they each observe signal X; and X, respectively. For simplicity, we assume the signal space is the same as the ground truth space, i.e. X;, X, Σ. We can interpret X; and X, as the full-effort signals that the agents can obtain which are their best guesses of Y.
In addition to full-effort signals, agents can obtain cheap signals Z,, L, E D that may may not depend on Y. Cheap signals are usually mutually correlated with each other, sometimes even more correlated than agents' full-effort signals. For example, these can be the responses of two (potentially different) LLMs.
Figure 1: The causal relationship among key variables.
We present the causal relationship among these variables in Figure [ Note that we assume X, and Z; are independent conditioned on Y. meaning that the full-effort signal is only correlated with the cheap signal via the ground truth. However, Z4 and Z, may have additional correlations conditioned on Y. This may happen when agents rely on LLMs trained on similar data.
Strategies The principal's goal to recover the ground truth accurately as possible. We con- sider scenarios where Al underperforms human agents, motivating the principal to seek full-effort information from human agents. Such situations arise when the notion of ground truth inherently subjective, shaped by human preferences such as when labelers are asked to rank LLM responses according to personal preference, a common practice reinforcement learning with human feedback.
However, agents can strategically report based on their high-effort signal X; and the cheap signal Z. Mathematically, the report X; is (random) function of X, and Z;, denoted as X; = 0x(,, Z.) where 0; O. Let F be the truth-telling strategy, i.e. T(X,, Z,) X,, and let μq denote no-effort strategy such that (x(X, 7,) is independent of X, conditioned on Z,. The argument for X, and 0, analogous.
3

---

We highlight special type of strategy, called the lazy-reporting strategy, which is more practical and common on crowdsoureing platforms.
Definition 3.1. A laz-rreportig strategy v E Oy convex combination between truth-telling and no-effort strategy jr, i.e.
[Xi (X,,, 2.) = [u(X,, Z,)
with probability ps < 1,
otherwise.
In particular, v does not make X; depend jointly on both X; and Zi on the same question. Examples of lazy-reporting strategies include: exerting effort and reporting X, for some questions while randomly selecting answers for the rest; or fully relying on an LLM and always reporting Z,. In contrast, non-lazy reporting strategies are typically complex and impractical, often resembling malicious rather than mere low-effort behavior. For example, an agent may first exert effort to obtain X; for each question and then mix it with Z, by reporting "yes" if either X; Ze is "yes", and "no" otherwise.
Principal Information The principal can observe the reports X; and Xj. but not the underlying signals X, X, and Z; Zz- This implies that the principal cannot directly evaluate human responses using the ground truth Y. Without knowing Y, the principal cannot distinguish the real information X; and the cheap signal Z; without further assumptions. Our method assumes that the principal can observe signal Z that (strongly) correlated with the agents' cheap signals. In the context of L.MM contamination, for example, the principal can generate answers to a fraction of the questions using a popular LLM.
The effectiveness of our method necessarily depends on the quality Z-how well the principal's information block the correlation between agents' cheap signals. We will introduce method and the assumptions in Section 4 and empirically verify the assumptions Section 5.2
3.1 Objectives and Problem Statement
In practice, the principal may collect responses from multiple agents, resulting in an n X m response matrix X where each entry Xt,* E Σ if agent answers question k and X, = otherwise. Given X and the principal's s samples of the cheap signal Z, a scoring mechanism designs score for each agent. Our goal is to design a scoring mechanism that satisfies information monotonicity. Let S, X_, 7) denote the expected score of agent i, where X-, is the reports of all agents but Definition 3.2. A mechanism is information monotone for lazy-reporting strategies for any agent i, X_; Z) > S,,,, () for any lazy-reporting strategy 1 E O, with X; = (x, Z.). A mechanism is e-information monotone if it is information monotone for lazy-reporting strategies and, additionally, (,,,, 2) 2 S, Z) - holds for any strategy 0, θ with X; 0x,X, Z.).
At the individual level, information monotone scores can be used to identify low-effort agents. At the dataset level, the score distribution across agents can help interpret the quality of the crowdsourced dataset.
We note that the same concept is termed informed truthfulnees informed trrategy-rroofine in the peer prediction literature, where the objective is to incentivize agents to report truthful information Shnayder al. 2016).
Our Method
Classic peer prediction methods often struggle with cheap signals, which are highly correlated with one another but only weakly correlated with the ground truth. In this section, we generalize prior work-the correlated agreement (CA) mechanism (introduced in Appendix and propose mechanism that achieves information monotonicity when the principal can obtain some noisy samples of the cheap signal. the context of L.LM contamination, these samples can be the labels generated by popular LLM fraction of the crowdsourced questions.

---

We call our generalization the conditioned CA mechanism, which scores an agent based on the correlation between her responses and her peers' conditioned on the principal's samples of Z. Our method has two main steps.
Learn the scoring function. A scoring function determines whether pair of signals "agree" or not, which is learned based on data. We first estimate the empirical conditioned joint distribution P(X,, Xj 2), computed using the frequency of observing a pair of signals on the same question conditioned on Z F Next, we compute the delta tensor
P P X: h X;
z k) P X h z k)
X; - - 1
which is the difference between the joint distribution and the product of marginal distributions of the reports, conditioned on Z. This gives us the scoring function T3
(.. Ta (h, k,k) = ÂAIIK > 0 and O otherwise. This step becomes unnecessary if the true scoring function T = sign(A) () is already known, where A is defined analogously to Á, but based on the true signals X; and X,- For example, for binary questions with positively correlated signals, A,,, an identity matrix for any k E E.
Compute the bonus/penalty score. Fixing a M, we then repeatedly sample a bonus question g and penalty question g' while ensuring Zy = Zo' = k. For each pair of samples, the score for agent i is:
T T g. X k) Tá (Xi.g Xz.g', k),
where j is a randomly selected agent who answers both q and q'. The final score for agent is obtained by averaging over all & E Σ and the sampled pairs of and q'.
Intuitively, the conditioned CA mechanism encourages agents to agree the same question and penalizes agents when they agree on two distinct questions, given that the principal's ' sampled signal is identical on both questions. If agents rely purely on the cheap signal respond independently of the true signal, the bonus score and the penalty score cancel out in expectation. This results a score that is close zero. We defer more details Appendix C
4.1 Theoretical Insights
We demonstrate the effectiveness of our method and clarify the conditions under which the mechanism performs reliably. Before we start, we first present an intuitive way to interpret the score computed by our mechanism.
prior work, Kong and Schoenebeck [2019a] show that the score of the CA mechanism can be interpreted as a type of mutual information between agents' reports called the total variation distance (TVD) mutual information. Inspired by this prior work, we show that when agents report the high-effort signal and the underlying scoring function T's is applied, the expected score computed by the conditioned CA mechanism is essentially the conditional TVD mutual information:
(Xv X, 2)
Σ Pr(Z=z)
Σ
- = = = 12) o Pr(X, =a a =) Pr(XX = |
ZE
The proof of this observation can be found in Appendix D.1
4.1.1 Conditions for c-Information Monotonicity
We first present the following pair of assumptions which require that the principal's 's information 7 can approximately block any correlation between agent's cheap signal and the ground truth (Assumption and the correlation between agents' cheap signals Assuumption44.2)
Assumption 4.1. We assume Z; and Y are approximately independent conditioned on Z, iee. [Pr(Z,, Y () Pr(Z4 Z) Pr(Y Z) S 6, for any i E [n).
Assumption 4.2. We assume Z, and Z, are approximately independent conditioned on Y and Z, i... Pr Z Y, Z) Pr(Z, Y, Z) Pr(Z, (Z, Y. Y,2)| < for any i.j E [n].
We first show that these two assumptions are sufficient to guarantee (approximate) information monotonicity.
We can use all agents' reports to estimate the same distribution because we assume agents are homogeneous and questions are i.i.d.. This assumption is primarily made for practical concerns, as there is usually shortage of data to estimate the distribution accurately for each pair of agents.
5

---

Theorem 4.3. I is known and [xv 2) > é ((e+ + ey)1 + (c; + ())) (
then under Assumption 4.1 [nnd 4.2 the conditioned CA mechanism is é-information monotone.
We defer the proof to Appendix D.1 When agents report the high-effort signals truthfully, the expected score is the TVD mutual information between X; and X,. Theorem 4.3 suggests that if Z approximately blocks the correlation between Ze with X, and Z then any manpulation can only decrease the TVD mutual information between X; and X, up to an error.
We further show the necessity of the assumptions for information monotonicity. The following proposition suggests that if Z, and Z, have some correlation that is not captured by Z, then there exists signal structure following the relationship in Figure such that the score computed by the conditioned CA mechanism is not maximized by reporting X; and Xj-
Proposition 4.4. Suppose TA is known. If there exist (zi, Zjz V, 2) such that Pr((z: -,, Zz
= for some 0 < S 4. then there exists a joint distribution Pr(X,, X,, Zi, 2,, Y, Z) that satisfies
2j Y y, 2 = -rrzz = 2i Y = y, = 2) Pr(Z, = 2j Y = y, 2 [[=]]==
the causal relationship described in Figure [ and strategy profile (0,, B3) * (T,T) such that
(,,,,, X, Z) 2 (X,,,X Z)
&df where X; (X, Z.) and X, (X, Zz).
Before we provide the full proof (deferred to Appendix D.2), we briefly introduce the high-level idea. Consider signal structure with a binary signal space Σ = 1). Suppose agent i's high- effort signal X; is perfectly accurate at predicting X; when X; = 0 but is noisy when X; = 1, i.e. Pr(X; = 0) X; 0, Z) = 1 and Pr(X;
1|X; 1, < 1. On the other hand, the cheap signal Z, is perfectly accurate at predicting Zj when Z; 1 but is noisy when Z; = O, Le. Pr(Z, 1| 1, Z) and Pr(Z, O|Z; = O, 2) < 1. In this case, it is intuitive that agent d can be better off if both agents combine their signals-reporting reporting 0 if and only if X; = Z, 0 and reporting 1 otherwise. Note that although this example only indicates that Assumption 4.2 is necessary, the same recipe can be straightforwardly applied to show the necessity of Assumption
4.1.2 Conditions for Information Monotonicity Under Lazy-reporting Strategies
We know Assumption 4.1 and 4.2 are necessary for the conditioned CA mechanism to be e-information monotone. However, these assumptions are designed to guard against complex, unrealistic strategies, such as those where agents first obtain the full-effort signal and then blend it with the cheap signal. If we instead restrict our focus to lazy-reporting strategies, these assumptions on Z can be significantly relaxed.
Assumption 4.5. For any pair of agents and j. we assume the conditional TVD mutual information between X,, Xj, Z., Zj satisfies that
TTvD(X;; X, Z) > max (TTvD (Z; Zz 2), (TD Xx,
Proposition 4.6. Ta is known and Assumption 4.5 holds, the conditioned CA mechanism is information monotone for lazy-reporting strategies.
We defer the proof to Appendix D.3 Intuitively, a lazy-reporting strategy is essentially a mixture of truth-telling and no-effort strategies. We show that when both agents follow lazy-reporting strategies, the acore-maximiziee full effort, achieving an expected score of (Xv X; Z); 2) both agents report cheap signals, with expected score /TVD (Z;; Z, 2); 3) one agent exerts full effort while the other reports a cheap signal, yielding expected score /TVD (Z; X, Z). Therefore, the inequality in Assumption 4.5 ensures that full effort strictly dominates lazy-reporting strategies, establishing the proposition.
equilibrium must fall into one of the following three cases: 1) both agents exert
Note that Assumption 4.5 is significantly weaker than Assumption 4.1 and 4.2 which require both LTV Lz; Z, Z) and TTDD (2; X, Z) to be close to zero.
Remark 4.7. Thus far, we have assumed that the true scoring function Ta is known. In Appendix E we show that the theoretical guarantees still hold even when we have to learn the scoring function from the noisy and potentially corrupted data.
5 Experiments
We evaluate the effectiveness of our proposed method using real-world crowdsoureing datasets. We focus on labeling tasks where human judgment remains the gold standard, given that current LLMs

---

still fall short of perfect performance. We first empirically examine the conditions under which the key assumptions underlying our method are likely hold, as well as when they may fail. Next, we simulate mixed, noisy crowd of labelers and assess whether our method can detect unreliable labelers better than several well-established baselines. The datasets and code of our experiments are available at
5.1 Datasets
Labeling We first consider a toxicity labeling dataset which includes 3459 social media user comments posted in response to political news posts and videos on Twitter, YouTube, and Reddit in August 2021 Schöpke-Gonzzller
al., 2025]. MTurk workers assigned these comments and are requested to annotate them as hateful, offensive, and/or toxic. The definitions of hatefulness, offensiveness, and toxicity are selected from highly cited previous literature. Every question is answered by 5 workers and each worker answers 73 questions on average.
alignment dataset where each question compares two -LMeggnerated answers to the same prompt. Labelers rate their preference on a 0-4 scale (0: A clearly better, 4: B clearly better, 2: tie). The comparison is scored from several more detailed angles, while we only foeus on the overall preference. The dataset contains 10 461 questions, each labeled by two crowd workers and two experts. On average, each labeler answers about 180 questions. For more details, refer to Miranda et al. 2024].
Preference Alignment We further
5.2 Assumption Verification
We empirically test the following hypotheses while varying the LLMs that generate Zi Zj. and Z.
1. Z4 and Xj are approximately independent given Z, i.e. TTvD Z4; X; Z) 0.
2. Z; and Zj are approximately independent given Z, i.c. ITvD Z5| Z) x 0.
3. X; and Xj are more correlated than Z; and Xj given Z, i.e. [VD(XX,;; | () >
/TVD Ze; X, Z).
4. X, and X2 are more correlated than Z, and Zy given Z, i.c I 2) ITVD Z5; Z, ZZ)
The first two hypotheses correspond Assumption and Assumption 4.2 respectively and the other two hypotheses correspond to Assumption 4.5 More broadly, our results in this section provide valuable insights into the correlations between different LLMs and their correlations compared with human agents in the context of subjective labeling questions.
For each considered LLM (as detailed in Appendix G.I.. we prompt it to independently generate responses to each question in the dataset three times using the default temperature. These responses treated samples of the cheap signals Z, Zj, and Z. Furthermore, human responses in the original dataset as samples of X, and Xj, respectively. Using these samples across all tasks, we estimate the conditioned joint distributions and then compute the conditional mutual information. We report the results for FivD(Z;: X, Z) in Figure 2 while results for TivD (Z;; Zj Z) are deferred to Appendix G.3
randomly select two
Hypotheses and generally do not hold. In Figure 2 we observe that when Z, and are sampled from different models, the mutual information between Z: and X; is often significantly higher than that expected from the random reporting (red bar). Even when Z; and Z are independently sampled by the same model, conditioning on Z reduces but does not eliminate this correlation. This indicates that even independently generated responses from the same LLM can remain dependent after conditioning on another independent sample, contradicting Hypothesis 1. As presented in Appendix G.3 this pattern continues to hold for TTVD Li; Z, Z), rejecting hypothesis 2.
Hypothesis 3 and 4 generally hold when conditioning on the "right" model. Encouragingly. the conditioned correlation between an LLM and a human agent is consistently weaker than that between two human agents when Z; and Z come from the same model. For example, in Figure 2a the unconditional mutual information between GPT-4-generated labels and human labels exceeds that of two expert labels. However, after conditioning on GPT-4 responses, /TvD frvD(x;; (X;; X, 2) becomes significantly higher than JTvD( X, Z). This pattern is consistent across all tested datasets, providing strong support for hypothesis 3. Results for Hypothesis 4 are deferred to the appendix.
7

---

Tundbee Deelte
fundbmn Chuute
6.25
0.175
0.150
Hitrrl
Phi
hw//2.
PM
6.109
6199
643
Uncoo noitioed GPT 3.5
GPT4 Conditonce
Gomma
Motal
Phi
Uncondiooned GPT 3.5
GFT
Gomma
Mistral
Ph1
(a) Preference Alignment
(b) Hatefulness
Humen
Hamer
07733
0FF35
6.25
0.3
0.20
6.1
Conditoned LLM
Conditionesi LLM
(c) Toxicity
(d) Offensiveness
Figure The TVD mutual information between X; and X, (for "Human", "Normal worker", and "expert") or Z; and Xj (for remaining bars) conditioned on Z, i.e. TTv Z) or (Zv X; Z). LLM names on the x-axis denote the models used to sample while each bar in the same group represents the model used to sample Z. We use "Random Agents" reference, who randomly selects an answer to each question according to the prior.
Our empirical results suggest that as of the time of this study and based on the subjective labeling tasks we tested, human agents exhibit distinct correlation patterns that no LLM replicates.
5.3
Detecting Low-effort Agents
In this subsection, we focus on identifying low-effort agents with lazy-reporting strategies and comparing various methods based on their area under the ROC curve (AUC).
5.3.1 Simulating Noisy Crowds
Real-world crowdsourcing datasets do not include labels identifying whether an agent is exerting low effort or working diligently. To enable evaluation, we simulate low-effort agents and treat the original dataset labels as representing "full-effort" responses. Specifically, we replace a randomly selected fraction of agents in the original dataset with simulated low-effort agents adopting lazy-reporting strategies. To reflect the noisy nature of real-world crowdsoureing data, we simulate three types of low-effort agents, each corresponding to a common lazy-reporting strategy observed in practice Yang ctt al. 2019). Becker et al. 2021]:
1. An Ojim fraction are LLM-reliant agents, who report the LLM-generated labels on all assigned questions.
2. An a, fraction are random agents, who generate labels by independently sampling from the marginal distribution over labels for each assigned question.
3. An as fraction are biased agents, who report the dataset's majority label on 90% of questions and choose uniformly at random on the rest.
shows the distribution of conditioned CA scores, where both the LLM-reliant agents' responses and the principal's 's signals are independently sampled from separate GPT-4 outputs. As shown, human workers achieve higher scores than any type of low-effort agent on average. However, some workers score below the average low-effort agent. One possible explanation is that subset of workers in the dataset may also have exerted low effort. Due these factors, no detection method can achieve perfect accuracy.
Figure 3a
8

---

1,0
20.0
GFTA-cheater
17.5
Random cheater
Blaced cheater
0.
0.8
12.5
0.7
0.5
0.0
-0.10
-0.03
0.03
CAg score
0.10
0.35
0.20
Fraction LLM-Cheater
(a) Score distribution
(b) AUC
Figure 3: (a) Distribution the conditioned CA scores with (Otm Oy,; 0b) = (0.1, 0.05, 0.05). (b) AUC scores of various methods the fraction of I.LM-reliant agents increases. In both panels, GPT-4 generates samples for Z,, Zz. and Z using the preference alignment dataset.
5.3.2 Baseline Measurements
We introduce the key ideas of the following four baselines and defer the details to Appendix G.2
1. We consider the Output Agreement (OA) mechanism that directly counts the frequency of agreements between two agents' responses.
2. We use the Correlated Agreement (CA) mechanism as a baseline, which introduced in Appendix B The expected CA score is the unconditional mutual information /TVD (wD(X,;: X2).
3. Another common method to estimate the reliability of the noisy agents is the Expectation- Maximization (EM) algorithm under the DS model [Dawid and Skene 1979]. The algo- rithm iteratively updates a confusion matrix that captures the noise level each agent. After convergence,
the trace of each agent's confusion matrix reliability score.
4. Lastly, consider the Conditioned Agreement (OAz) z) which counts the frequency of agreements only when the agreement is different from the principal's sample on that question.
We note that there are many more heuristic methods to estimate the reliability of crowdsourcing agents Yin et al. 20088. EEbreetoo and Reise [2013), Piech al. [0133. Many of these methods variants of the EM algorithm under a different context- deeeeeee-deendeent model. We nonetheless only use the EM algorithm with the DS model represent this line of methods because they all ignore the principal's 's information Z and thus are unlikely to perform well with L.MM-corrupted datasets.
5.3.3 Results
Given a dataset and samples the cheap signal Z, we use our method as well as the baselines to score each agent. We use the AUC score measure the ability of each discussed metric distinguish the simulated low-efforrt agents from the original human agents. In particular, let be the set of human agents and /- be the of simulated low-effort agents. We compute the AUC score for each method as follows:
AUC |+|
ΣΣ (1S S,) + 1[S =S,))-
KEI-
Figure 3b presents a comparison of different methods. For each value dumm ranging from to 0.2, we randomly sample o,, and os uniformly from [0, and report the average and error bars over 50 such trials. Our result suggests that classic methods, such as the EM algorithm and the CA mechanism, perform well when the number of LLM-reliant agents is low. However, their AUCs degrade significantly as the fraction LLM --rliat agents increases. On the other hand, the performance of OAz, which scores agents based on the similarity between their reports and Z, tends to have large variance and performs poorly when there are few LLM-rrliant agents. In contrast, our proposed conditioned CA mechanism has the most robust performance for the mixed crowds and has the lowest variance.
Table I summarizes the results from Figure 3b
reporting the average AUC score and the bottom 10% quantile for each method on each dataset, where the average w.r.t. every tested combination of (Oum, Cr, ab). A high average AUC reflects strong overall performance, while high 10% quantile

---

indicates robustness to the worst-case scenarios. The results support our earlier claim that our method consistently demonstrates the highest robustness across all tested settings.
Table I: Average AUC (and bottom 10% quantiles) across datasets and methods with GPT-4-reliant agents.
Dataset
OA
CA
OAz
EM
CAx (ours)
Hatefulness
0.66 (0.45)
0.84 (0.75)
0.75 (0.64)
0.81 (0.70)
0.85 (0.81)
Offensiveness
0.80 (0.68)
0.92 (0.88)
0.92 (0.85)
0.89 (0.82)
0.94 (0.91)
Toxicity
0.60 (0,40)
0.89 (0.83)
0.89 (0.82)
0.85 (0.79)
0.91 (0.87)
Preference Alignment
0.65 (0.40)
0.70 (0.43)
0.70 (0.45)
0.68 (0.41)
0.85 (0.77)
We note that our experiments do not claim that our method outperforms all baselines in every scenario. Traditional methods like EM often perform better when the crowd consists entirely of random or biased agents. However, our method demonstrates the most robust performance across all settings due its theoretical guarantees. This is particularly useful as, in practice, the designer usually has little clue about the composition of the crowds. In contrast, the baselines exhibit cases where the AUC drops below 0.5, meaning the algorithm completely misclassifies good workers as low-effort agents.
presents analogous figures and tables for additional datasets and LLMs, further supporting the claims made in the main body. Second, we investigate how many questions the principal needs to sample for effective detection of low-effort agents (Appendix G.4.2). The results suggest that sampling LLM labels on 50% of the questions is typically sufficient to outperform the CA baseline. In cases where human and Al labels are well separated, sampling just 20% can suffice. Third, when the principal is uncertain about which LLMs the agents use, we propose a method that computes the CA score conditioned on each candidate LLM's outputs and uses the minimum score across them as the final score (Appendix G.4.3). Finally, in Appendix F
Additional results in the appendix. First, Appendix G
and Appendix G.5] we present heuristic extension of our method to open-ended textual responses. This serves as a preliminary step toward generalizing our approach beyond labeling tasks, with further refinement and sarge-scale validation left for future work.
6 Conclusion
This work addresses the challenge of LLM contamination in crowdsourcing by developing a theoret- ically grounded quality measure for annotation tasks. Leveraging an extension of peer prediction, our method evaluates worker responses beyond what LLMs can generate, ensuring reliable human contributions. We establish theoretical guarantees under different low-effort strategies and validate our approach empirically on real-world datasets. Our results demonstrate that the proposed method effectively detects low-effort agents while maintaining robustness across various conditions. Future work can focus on refining our method to better differentiate between harmful LLM usage and productive human-Al collaboration. This includes optimizing scoring mechanisms recognize when LLMs are used as assistive tools rather than substitutes for human effort and developing adaptive detection techniques that adjust based on task characteristics and worker behavior.
Broader Impact
On the positive side, our work can improve the robustness and fairness of data collection pipelines used in training AI systems, thereby enhancing the reliability of downstream decision-making or other usage of AI. However, there is potential risk that insights into reporting strategies or detection weaknesses could be misused evade quality control. We believe this risk mitigated by focusing on defensive mechanisms and open discussion of vulnerabilities to promote transparency and better system design. Furthermore, like many crowdsoureing algorithms for label-quality control, our method operates without access to ground truth. Consequently, low score does not always indicate a low-effort worker, while may also reflect a diligent agent holding a minority opinion. Therefore, while such verification-free algorithms are highly scalable, they should be used with human-in- toe- allowing the principal to carefully review low-scoring agents and prevent potential ethical concerns.
10

---

Acknowledgment
This work was partly carried out the DIMACS Center Rutgers University.
References
Arpit Agarwal, Debmalya Mandal, David C Parkes, and Nisarg Shah. Peer prediction with heteroge- neous users. In Proceedings of the 2017 ACM Conference on Economics and Computation, pages 81-98. ACM, June 2017.
Zahra Ashktorab, Michael Desmond, Josh Andres, Michael Muller, Narendra Nath Joshi, Michelle Brachman, Aabhas Sharma, Kristina Brimijoin, Qian Pan, Christine T Wolf, et al. Ai-assisted human labeling: Batching for efficiency without overreliance. Proceedings of the ACM on Human- Computer Interaction, 5(CSCW 1):1-27, 2021.
Julian Ashwin, Aditya Chhabra, and Vijayendra Rao. Using large language models for qualitative analysis can introduce serious bias. arXiv preprint arXiv:2309. 17.3009..7777. 2023.
Joshua Becker, Douglas Guilbeault, and Ned Smith. The crowd classification problem: Social dynamics of binary choice accuracy, 2021. URL http:///aaxxxi...aabresse/ee
11300
Tolga Bolukbasi, Kai- Rei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. Man to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29, 2016.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Noah Burrell and Grant Schoenebeck. Measurement integrity in peer prediction: A peer assessment case study, arXiv preprint arXiv:2108.05522 2021.
Jan Cegin, Jakub Simko, and Peter Brusilovsky. Chatgpt to replace crowdsourcing of paraphrases for intent classification: Higher diversity and comparable model robustness. arXiv preprint arXiv:2305.1294 2023.
Anirban Dasgupta and Arpita Ghosh. Crowdsourced judgement elicitation with endogenous profi- ciency. In Proceedings of the 22nd international conference on World Wide Web, pages 319-330.
ACM, 2013.
Alexander Philip Dawid and Allan M Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28 (1):20-28, 1979.
Maria del Rio-Che Kio-Chanona, Nadzeya Laurentsyeva, and Johannes Wachs. Are large language mod- els a threat to digital public goods? evidence from activity on stack overflow. arXiv preprint arXiv:2307.07337, 2023.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04880 2018
Susan E Embretson and Steven P Reise. Item response theory. Psychology Press, 2013.
Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. Gltr: Statistical detection and visualization of generated text. arXiv preprint arXiv: 2019.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences. 120(30), July 2023. ISSN 1091-6490. doi: 10. 000073/paa.2305016111 URL http:://xxdooiiooooooooooo 2305016120
doi 1ttp://oo..rgg10111//////////
Chenxi Gu, Chengsong Huang, Xiaoqing Zheng, Kai- Wei Chang, and Cho-Jui Hsieh. Watermarking pre-trained language models with backdooring, arXiv preprint arXiv:2210.07544 2022.

---

Xiaomeng Hu, Pin- Yu Chen, and Tsung-Y Yi Ho. Radar: Robust ai-text detection via adversarial learning, Advances Neural Information Processing Systems, 36: 15077-1099, 2023.
Daniel P. Jeong, Zachary C. Lipton, and Pradeep Ravikumar. LIm-select: Feature selection with large language models, 2024. URL httph: http://axx.iioggrbsss/// 222924269799944
Jamshed Kaikaus, Haoen Li, and Robert J. Brunner. Humans vs. chatgpt: Evaluating annotation
methods for financial corpora. In 2023 IEEE International Conference on Big Data (BigData),
pages 2831-2838, 2023. doi: 10. 10//BiigDDuSSS9442222...... 10386425..
Ryuto Koike, Masahiro Kaneko, and Naoaki Okazaki. Outfox:
LIm-generated essay detection through in-context learning with adversarially generated examples. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 258-21166,, 2024.
Yuqing Kong. Dominantly truthful multi-task peer prediction with a constant number of tasks. In Proceedings of the Thirty-Firss Anmual ACM-SIAM Symposium on Discrete Algorithms, SODA '20, page 2398-2411, USA, 2020a. Society for Industrial and Applied Mathematics.
Yuqing Kong. Dominantly truthful multi-task peer prediction with a constant number of tasks. In Proceedings of the fourteenth annual acm-siam symposium discrete algorithms, pages 2398 2411. SIAM, 2020b.
Yuqing Kong and Grant Schoenebeck. Eliciting expertise without verification. In Proceedings of the 2018 ACM Conference Economics and Computation, pages 195-212, 2018.
Yuqing Kong and Grant Schoenebeck. An information theoretic framework for designing information elicitation mechanisms that reward truth-telling. ACM Truns. Econ. Comput., 7(1,, jan 2019a. ISSN 2167-8375 doi: 10. 00.1145/3296670. URL https://doi. www.oogg.1... 3296660
Yuqing Kong and Grant Schoenebeck. An information theoretic framework for designing information elicitation mechanisms that reward truth-telling. ACM Transactions on Economics and Computation (TEAC), 7(1):1-33, 2019b.
Taehyun Lee, Seokhee Hong, Jaewoo Ahn, llgee Hong, Hwaran Lee, Sangdoo Yun, Jamin Shin, and Gunhee Kim. Who wrote this code? watermarking for code generation. arXiv preprint 2:X.5:2305.15060 2023.
Weixin Liang, Yaohui Zhang, Zhengxuan Wu, Haley Lepp. Wenlong Ji, Xuandong Zhao, Hancheng Cao. Sheng Liu, Siyu He, Zhi Huang, et Mapping the increasing use of Ilms scientific papers. arXiv preprint 2404::2404.00200 2024.
Minghao Liu, Zonglin Di, Jiaheng Wei, Zhongruo Wang, Hengxiang Zhang, Ruixuan Xiao, Haoyu Wang, Jinlong Pang, Hao Chen, Ankit Shah, et al. Automatic dataset construction (ade): Sample collection, data curation, and beyond. arXiv preprint 1xXiv:2408.113333 2024.
Yuxuan Lu, Shengwei Xu, Yichi Zhang, Yuqing Kong, and Grant Schoenebeck. Eliciting informative text evaluations with large language models. In Proceedings of the 25th ACM Conference on Ecomomics and Computation, EC '24 page 582-612, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400707049. httpps://oi..... org/1ooggg11 10. 1145/3670865 0111377786666363332222
doi: 10. 10.11457277886.337333.. URL
Christopher D Manning. An introduction to information retrieval. 2009.
Chengzhi Mao, Carl Vondrick, Hao Wang, and Junfeng Yang. Raidar: generative ai detection via rewriting. arXiv preprint arXiv:2401. /2970, 2024.
Tomas Mikolov. Efficient estimation of word representations in vector space. arXiv preprint 1rXiv: 3781, 2013.
Nolan Miller, Paul Resnick, and Richard Zeckhauser. Eliciting informative feedback: The peer- prediction method. Management Science, 51(9): 2005.
Lester James V. Miranda, Yizhong Wang. Yanai Elazar, Sachin Kumar, Valentina Pyatkin, Faeze Brahman, Noah A. Smith, Hannaneh Hajishirzi, and Pradeep Dasigi. Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback. arXiv, abs/2410.19133,, October 2024.
12

---

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Jinlong Pang, Jiaheng Wei, Ankit Parag Shah, Zhaowei Zhu, Yaxuan Wang, Chen Qian, Yang Liu, Yujia Bao, and Wei Wei. Improving data efficiency via curating lmm-driven rating systems. arXiv preprint arXiv:2410.10877 2024.
Jinlong Pang, Na Di, Zhaowei Zhu, Jiaheng Wei, Hao Cheng, Chen Qian, and Yang Liu. Token clean- ing: Fine-grained data selection for Ilm supervised fine-tuning, arXiv preprint arXiv22502.01968, 2025.
Chris Piech, Jonathan Huang, Zhenghao Chen, Chuong Do, Andrew Ng. and Daphne Koller. Tuned models peer assessment in MOOCs. 6th International Conference on Educational Data Mining, Memphis, TN, 2013. arXiv (307.25797
Sanh. Distilbert, a distilled version bert: smaller, faster, cheaper and lighter. arXiv preprint 10Xiv:1910.011088 2019.
Grant Schoenebeck and Fang-Yi Yi Yu. Learning and strongly truthful multi-task peer prediction: A variational approach. arXiv preprint arXiv:2009.147330 2020.
Grant Schoenebeck, Fang-Yi Yu, and Yichi Zhang. Information elicitation from rowdy crowds. In Proceedings of the Web Conference 202/, WWW '21, page 9977-3986, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383127. doi: 10.1 10.1145/344238 011134444381.3444444 1.3499800.
URL https:://ooi org 10. 1141/1444231 3449840
Angela Schöpke-Gonzalez Siqi Wu, Sagar Kumar, and Libby Hemphill. Using off-the-shelf harmful content detection models: Best practices for model reuse. Proceedings of the ACM on Human- Computer Interaction, (CSCW), 2025.
Aditya Shah, Prateek Ranka, Urmi Dedhia, Shruti Prasad, Siddhi Muni, and Kiran Bhowmick. Detecting and unmasking ai-generated texts through explainable artificial intelligence using stylistic features. International Journal of Advanced Computer Science and Applications, 14(10), 2023.
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-422 1948.
Victor Shnayder, Arpit Agarwal, Rafael Frongillo, and David C. Parkes. Informed truthfulness in multi-task peer prediction, 2016. URL https://arxi:: orgg/bs//603..oom 03151
Jinyan Su, Teny Yue Zhuo, Di Wang, and Preslav Nakov. Detectllm: Leveraging log rank information for zero-shot detection of machine-generaile text. arXiv preprint 20w::2306.055400, 2023.
Petter Törnberg. Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning. 2023.
Christoforos Vasilatos, Manaar Alam, Talal Rahwan, Yasir Zaki, and Michail Maniatakos. Howkgpt: Investigating the detection of chatgpt-generater university student homework through context- aware perplexity analysis. arXiv preprint arXiv:2305. 2023.
Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. Artificial artificial artificial intelli- gence: Crowd workers widely use large language models for text production tasks. arXiv preprint arXiv:2306.07899 2023.
Yifan Wu and Jason Hartline. Elicitationgpt: Text elicitation mechanisms via language models, 2024. 09363
URL
Xi Yang, Kejiang Chen, Weiming Zhang, Chang Liu, Yuang Qi, Jie Zhang, Han Fang, and Nenghai Yu. Watermarking text generated by black-box language models. arXiv preprint ar0iv:2305. 23888330.. 2023.
13

---

Yi Yang, Quan Bai, and Qing Liu. Modeling random guessing and task difficulty for truth inference in crowdsoureing, In Proceedings of the 18th international conference on autonomous agents and multiagent systems, pages 2288-2290, 2019.
Xiaoxin Yin, Jiawei Han, and Philip S. Yu. Truth discovery with multiple conflicting information providers on the web. IEEE Transactions on Knowledge and Data Engineering. 20(6):796-88) 808, 2008. doi: 10.1 10.1199/KDDE2200.19900 197744
Xianlong Zeng, Fanghao Song, and Ang Liu. Similar data points identification with Ilm: A human- in-the-loop strategy using summarization and hidden state insights, 2024.
Yichi Zhang and Grant Schoenebeck. High-effort crowds: Limited liability via tournaments. In Proceedings of the ACM Web Conference 2023, WWW '23, page 3467-3477, New York, NY. USA, 2023a. Association for Computing Machinery. ISBN 9781450394161. doi: 10.1145/3543507, 3583334. URL http:tt//o...oog///// htt.//....../..... 1145/35435071 3583334
Yichi Zhang and Grant Schoenebeck. Multitask peer prediction with task-dependent strategies. In Proceedings of the ACM Web Conference 2023, WWW '23, page 3436-3446, New York, NY, USA, 2023b. Association for Computing Machinery, ISBN 9781450394161. doi: 10.1145/3543577 3583292. URL https://ddii. org 10. 1145/3543500 3583292
Xuandong Zhao, Prabhanjan Ananth, Lei Li, and Yu-Xiang Wang. Provable robust watermarking for ai-generated text. arXiv preprint arXiv:2306. /7439, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please see abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Please see Section and Section 4.1 for detailed assumptions and theory. Proofs are left in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We carefully present the datasets, hyperparameters, and experimental design in Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
We provide the code for our experiments in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pspps.cc/pullic/uuii (ttpi///guudee///eeeeeeee Cct//eubeeebiaa ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please see our figures in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our paper does not include extensive GPU training. All experiments are lightweight and can be efficiently run on local machines, such as a standard MacBook.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httt::::/eeuripp..ccpppbl//////
Answer:
[Yes]
Justification:
We believe there is no violation of the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please see Section G
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
This paper does not release any models datasets with high risk of misuse. The study involves simulated crowdsourcing settings and analysis M-assissted low- effort behavior using synthetic existing benchmark data.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets and LLMs used in this paper are properly cited,
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, thcoe com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper releases the code used for running the experiments, which is provided alongside the submission to ensure reproducibility. No new datasets models are introduced.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
The paper does not collect data from human subjects. All experiments are based on existing and public crowdsourcing datasets.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not collect data from human subjects. All experiments are based on existing and public crowdsoureing datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: http://///arrrssscc/////eeeeee/e for what should or should not be described.
21

---

A Limitations
We acknowledge several limitations of our method. First, our method is primarily designed for crowdsoureing datasets comprising subjective questions or tasks where human experts are considered the gold standard. If LLMs significantly surpass human performance, the correlations between human responses may no longer reflect the ground truth Y. This violates the assumption that, conditioned on Y. agents' high-effort signals are independent. In such cases, applying our method could mistakenly reward incorrect correlations.
Second, as discussed in our experiment section, our method does not outperform all baselines in every scenario. In particular, when LLM responses diverse, e.g. when each agent relies different model or only a few agents use LLM assistance, heuristic methods such as the EM algorithm work well. This is because diverse sparse low-effort agents are less correlated with the majority of the crowd and can be easily flagged as outliers using the classic methods. Despite this, our method demonstrates the most robust performance across wide range of settings. Future work could incorporate the principal's sampled LLM signals into heuristic methods such as EM to improve their performance and enhance detection under LLM contamination.
Third, our experiments evaluate only five language models, as some models, such as LLaMA and Gemini, refuse respond to toxicity-related prompts. While this is sufficient to demonstrate the method's effectiveness, additional evaluations across more diverse models and task types would provide more comprehensive assessment.
B Preliminary
What do we know about the problem? Our work builds on advancements in information elicitation, particularly peer prediction. First introduced by Miller al. [2005], peer prediction mechanisms aim to incentivize truthful reporting without requiring ground truth verification. The core idea is to reward agents based on how well their responses predict those of others. The original mechanism relied on the principal's 's knowledge of the correlation between agents' signals (the information they observe). Later, Dasgupta and Ghosh| (2013) extended this idea, removing the need for such knowledge by assuming agents answer multiple i.i.d. binary questions. Subsequent works have expanded this framework to non-binary (but finite) signals (Shnayder et al., 2016 Kong and Schoenebeck) 2019b),
infinite signals [Schoenebeck and Yu 2020). truthful guarantees with fewer questions Kong 2020b),
heterogencous agents [Agarwal et al. 2017). adversarial agents (Schoenebeck et al. 2021]. ], and more general reporting strategies (Zhang and Schoenebeck 2023b).
Under the assumption that there are no cheap signals, the peer prediction literature suggests that estimating the mutual information between X; and X, is an information monotone metric Kong and Schoenebeck [2019a]. There are various ways to estimate the mutual information using agents responses to multiple questions Kong
and Yu [2020) and many information monotone metrics that can be explained using mutual information Dasgupta and Ghosh (2013), Shnayder et al.] [2016]. In this section, we discuss simple yet intuitive mechanism called the correlated agreement (CA) mechanism (Shnayder et al. (016).
The CA mechanism rewards agents when they "agree" on the same question and penalizes them when they "agree" on distinct questions. Instead of directly checking agreement, the mechanism redefines agreement based on the correlation between signals. A pair of signals a' E Σ is correlated agreed Pr(X, = o, X, d') Pr(X, = .) Pr(X, = a') > O. meaning that a and o' are more likely to appear on the same question rather than on distinct questions. In particular, the mechanism utilizes the delta matrix A,, Pr(X, o. X, = o') Pr(X, = σ) Pr(X, = o') to design the scoring function Ta- sign A) where sign(x) = if x > 0 and 0 otherwise.
In practice, the mechanism can be implemented in two ways depending on the information available to the principal. First, if T is known, the mechanism can be directly implemented, as shown in Algorithm|] [] This case happens if the information structure is simple, e.g. when questions have binary answers, T'A is likely to be a 2 x 2 identical matrix. Second, if I'' is unknown, the mechanism can first estimate the joint distribution, denoted as Pr Xyy) using the report matrix X. Then, an estimate of T's can be computed using the estimated Delta matrix A = Pr(X; = o, X, = o') Pr = Pr(X, ='). Next, Algorithm can be applied.
22

---

We emphasize that penalizing agreements on distinct questions is crucial for the CA mechanism to maintain information monotonicity. Without such penalization, over-reporting the majority signal could result in high score. This form of manipulation is particularly problematic when the dataset biased. For instance, task to label messages as toxic not, if only 5% of the data points are toxic, consistently reporting "not toxic" could yield a higher score than truthfully reporting the actual signal.
Therefore, the score computed by the CA mechanism can be interpreted in the following way. Let the marginal distribution of agent i's reports be Pr(X.). Imagine an agent k who reports randomly for each question according to Pr(X,). Then, the CA score for agent i can be understood as the average number of questions on which agent agrees with another randomly selected agent , more than agent k agrees with j.
Algorithm I: The Correlated Agreement Mechanism (Shnayder et al.) 2016)
Input: The crowdsourced dataset X, the (estimated) scoring function T3
Output: A score vector s
for agent i [n]] do
Let M, be the set of questions answered by agent i
Initialize <-0
for question E M; do
Randomly select peer j who answered question 9
x - x + T3 iX X,.) Randomly select question q' * answered by agent j
x Tá (Xi, i,g: X (
- I//M
return 8
C Details of the Conditioned Correlated Agreement Mechanism
After learning the scoring function, the conditioned CA mechanism scores agents according to Algorithm 2
At high level, the mechanism will iteratively fix signal k Σ and look at the questions on which Z, k. Let Mz (k) be the set of questions on which Z = k. Then, an agent i will receive a higher score if she agrees with another agent j on the same question more often than on two distinct questions conditioning that the questions are drawn from Mz (k). This measures the additional correlation between two independent agents conditioned on Z.
Algorithm 2: The conditioned CA mechanism
Input: The crowdsourced dataset X, the samples of cheap signal Z, the (estimated) scoring function T
Estimate the empirical marginal distribution Z, i.c. P(Z = σ) for o Σ. for agent E [r]] do
Let M, be the set of questions answered by agent 2;
Initialize s; = 0;
for & E Σ do
M, n Mz(k) = 0. continue;
Initialize 0:
for question E M, n M2 do
Randomly select peer j who answers
+= (2(xx, (X.q.
Xi k);
Randomly select question g' E My (k)):
x - T3 (8 Xig k)
+= P(Z
return 8 = (s1,...,8n)
23

---

D Proofs
D.I Sufficiency of the Assumptions
Intuitively, Assumption 4.1 and Assumption 4.2 imply that conditioned on Z. Z; is almost independent of X,, X,, and Zj. Therefore, any manipulation that makes X, depend on Zi will not increase the expected score. To prove this, we first write down the expected score when agent i and play strategy 0g and 03 respectively.
Proof of Theorem 4.3
S,(0;,0) 0z)
Σ
Pr(Z =)
Σ
Pr(X; = Ij, Xj Jjz 2i
= - (1)
SEE
the expected score of the bonus question
Σ P
Σ
Pr(x, 24, 2
24 2) Pr(X; 2, Z, 2j
2) Ta (0, 24)
SEC
expected the penalty question
For comparison, we next write down the expected score when both agents report truthfully.
Si(T,)) Σ Pr(Z=
Σ
Pr(X, xj X, 2 2) Ta (2r,, (T,
expected score bonus question
Σ 1E
Pr(Z = 2)
Σ
Pr(X; = 24 2) Pr(X, = 2, 2) TA 13, 2)
21
the expected score of the penalty question
Σ xEΣ
Pr(Z Pr(Z=2) =)
Σ
(Pr(si, 2) Pr(ax =) Pr(xr, =)) Ta (z.
- =E>
Pr(Z = 2)
Σ
Pr(x,, - 1 2) Pr(xz |2)
=ITVD Xx; X; 2)
We want to show that (0, 0j) < (T r) up to an error. As shown above, the expected score can be decomposed into the expected score of the bonus question and the expected score of the penalty question. Next, we provide an upper bound on the bonus score and lower bound on the penalty score.
Lower bound the penalty score. We first provide a lower bound for Pr(X,, Z, Z), which is summarized in the following lemma.
Lemma D.1. Under Assumption 4.1 = y, 2).
Pr(X,, Z, Z) 2 Pr(Z, 2) Pr(X,) Z) - Evex Pr(X,
Then, we provide lower bound of the penalty score while fixing a 2 =
Σ
Pr(c,, z 2) Pr(zy, zy =) Ta (0:(s, 2:), 0,(2,, 2;);,,,,))++ =)
Σ
Pr(z:
c, Σ
Pr(mi v.2)) Pr(x,, 23 12) (1,0, 2.), 2)),
yeE
(By Lemma D.I
By reordering the summations,
Σ
Pr(z =2) Pr(x; [=) Pr(x, 2 2) (00,,
G Σ
Σ Pr(m,, (2)
Σ Pr(m: Iv, (0
24

---

We focus on the second term. First note that T' & 1. Therefore, the second term is lower bounded by -4 EgHt.r Σz, Σxy. Pr(zy, 2g 2) En,ex Pr(z; y, 2). Marginalizing over æ;j, 2j and 2; and observing that y, 2j E E, the quality is lower-bounded by </|x)e. This means the original quantity is lower-bounded by
Σ Ty "
Pr(zi 2) Pr(x; 22) Pr(zy,2) 2) (0) -
Next, we apply the same derivation for j.
Σ
Pr(zi 2)) Pr(m;) =) Pr(z)))) 12) ej Σ
Pr(ny [1,2)
(1)
YEΣ
(4), 2z), ei/212
(By Lemma D.I
Σ
Pr(z: =) Pr(x: Pr(zz 2) Pr(ay) 2) Ta (1),
(t; + E
(2)
Upper bound the bonus score. We first present a lemma that bounds the difference between Pr(Z,, Z5 Y, Z) and Pr(Z, 2) Pr(Z, Z).
Lemma
D.2.
Under
Assumption
4.7
and
Assumption
[Pr(Z,, 2 (Y, - Pr(Zi] 7) Pr(Zg
2)
S
P2)
With this lemma, we further provide an upper bound of the expected score on the bonus question. Fixing Z = 2,
Σ
Pj 2, 2) Ta (0. 2).0,((())))))) 2y).
21, EE
Σ
Σ Pr(y 2) Pr, 21 18.2) TA (1 (z,, 2), (z(,,
(By the law of total probability)
Σ
Pr(y 2) Pr(m,, 21) Pr(zs, zj 10, 2) TA (0, 2,), ( 2)),
YEE 81 YEE
(X and Z are independent conditioned on Y.)
Σ
Σ
Pr(y| 2) Pr(a,, 11,=) Pr(z: 2) Pr(zz 22) +
4 c. Pr(ylz)
Ta (1),
Bu EEE
(By Lemma D.2
can further write this into three terms.
Σ
Σ Pr(y z) Pr(a,, 1,, Pr(z, =) Pr(zy [x)
yEX
(3)
+
Σ
Σ
Prly 2) Pr(x,, 11.2)
(4)
Piilifii EJ YE
+ (G +2))
Σ
Σ
Pr(z,, 11,z) ()))))((((,,,))))))(()))
(5)
B yEE
First note that
Equation
Σ
Pr(zi 2) Pr(z) (1)(0(((,, Σ
Pr(z,, 24
DEE
Σ
Pr(z Pr(z;) =)
Pr(mx,, 2)
25

---

Next, we upper bound the second and the third term respectively. By reordering the summations,
Equalion
Σ
Σ NE
Pr( 2)
Σ
(Ti, (z) Pr(a,, 12)
Σ
Pr(mi, 12
(TA 1)
S <
E E ED Pr(x,, |2] 1)
Equation 5 = (c+
ΣΣ
Σ
Pr(x,, 2y 1u,2)T (0. (),0)(()))))))
yEΣ Kii EL Σ
S +G)
Σ Σ
Pr(z,, By 1,,2)
(Ta S 1)
yEΣ 2113 EE K,
S (E +5)
Ennnye Pr(zi, [1,2) Pr
Combining everything,
Σ Pr(m,, 23 | 2) TA (0, 0))0,(z))))))) 2z).
Σ SATEE
Pr(z 22) Pr(zz 22) Pr(mç, 2) (2, (2)) + + + (a + e,)1213
(6)
Finally, combining Equation 2 and Equation (6)
O,)
< Σ Pr(Z= Σ Pr(z: 2) Pr(zj] 12)
Σ
(Pr(æ;, -) Pr(m, | 22) Pr(æ, | 2))
(T,0 2)) (r)z 25);
(ct (c + |E|
For simplicity, let ē (c G, + (c
( (g)
Epr(z
Pr(z= = x) Σ Pr(z: x) Pr(zj 2))
Σ
AeLiy (2(0, (,), +
21,
E
< Σ Pr(Z 2) Σ
Pr(z 2) Pr(zz 22)
Σ
Anazy TA 21, +ē
(By the design of Ta)
= =Σ Pr(Z= = Σ
Anwey (1(,,, +
ZE
=),(T,7)+ +
Lastly, we show that if agent i plays lazy-reporting strategy Mis the expected score is strictly smaller than truth-telling. Note that because a lazy-reporting strategy is a convex combination between truth-telling and no-effort strategy 14, suffices to show that the expected score of any no-effort strategy is strictly smaller than truth-telling, Because (X,, Z;) is independent of X;, reoder
26

---

the summations in Equation 7 in the following way. Let μ pa(X,, Z.).
S,414,0) 0z)
Pr(Z= Σ Pr(z: | 2)) Pr(z) 1=) Σ
Ta(u;
2E2
2..S,E2
2,62
Σ (Pr(I;, 2) - Pr(xi 2) Pr(ar 22)) +
(by Equation 7
3,E3
=Σ =E"
Pr(Z 2)
Σ A.
Pr(zi 2) Pr(z) 22) Σ
(1,,, (Pr(æ, 12) Pr(zy [a)) + é
=6.
Note that the expected score under truth-telling is exactly /TVD (X;; Z). Therefore, whenever (Xv Xj Z) é, we have Si( > Si(us, 0z) for any no-effort strategy μ4 and any strategy
0z. This completes the proof.
Proof of Lemma D.I
Pr(X,, Vi Z)
Σ
Pr(Y = 1)) 4 Y
Z,
(By the law of total probability)
3EΣ
Σ
Pr(Y y z) Y y, 2) Pr(Z, Y y,
NE
(X; and Z; are independent conditioned on Y.)
Pr(Y = 2)) Y = y. Pr(Z; Z) Pr(Y = y 7) 6
2 Σ
Pr(Y = y Z)
NEE
(By Assumption 4.1
= Pr(Z 2) Σ
Pr(Y = y P) Y = =v,2) - Σ
Pr(X, Y = y; Z)
VEΣ
Pr(Z 2) Pr(X, -)- - & Σ
Pr(X, Y Y= 2)
y
DEΣ
Proof of Lemma D.2
Prr(Z, Zy Y, Pr(Z, ( Pz, |Z]
= =IPr(Z,, K, Pr(Z,
+ Pr(Z, Pr()
2)]
< <Pr,Z,, |Y,Z) Pr(Z, Y12) Pr(Z, |Y,2) + |Pr(Z Yrz)Pr(Z, Y. Pr((, Z) PrPr(Z,) [Z]
(Triangle inequality)
S*+ |PrZ Y, 7) Pr 2. -
Z) Pr |)).
(Assumption 4.2
Therefore, we only have to bound the second quantity.
[Pr(Z Pr(z) 2, Pr(Z, Pr(2;) Z
=|Pr(Z (Pr((, Y.2) Pr(Z, 2)) + Pr(Z, (Pr((Z Y, Pr(Z, Z))l
< (PM(())Y,)(()))) |Y Pr(Z, +|][ ()(Pr(Z |YZ) - -r((Z; Z))
(Triangle inequality)
P |YZZ- Pr(Z, Z2 + [Pr(Z, |Y, Pr(Z, |Z]
1
1
Pr(z,, Y 12) Pr(Y | / Pr(Z, 2) + Pr(Y 2)
[Pr(Z,, Y 2) - Pr(( | 2) Pr(Z, 2))
Pr(Y ())
+
(By Assumption
Pr(Y | ZZ
Combining everything, we complete the proof.
27

---

D.2 Necessity of the Assumptions
Proof of Proposition 4.4 Intuitively, if 2, and 2, are correlated in different way from X, and X,, then it makes Z; and 2, no longer "cheap" signals. In this case, a strategy that combines Z; and X; will improve the expected score.
Suppose = {0.1) contains two signals. Consider the following joint distribution for corresponds (X; Zi) (0, 0), (0, 1), (1, (1,1) respectively and the columns stand for (X,, Zz). For simplicity, let's
Pr((X, 7,,, (X, Y
y,
2) where row
suppose the following joint distribution is identical for any and z.
2d
0
0
0
0
Pr((X,, 2.), (X, (x), 2)
for any v, {0,
0
2d
1
2d
2
0
Note that the example is valid when 0 < d I 4. Using this joint distribution, can further write down the joint distribution between other pairs of variables.
2d d
Pr(Z,, Zy 22 =(8
1 - 2d
Pr
21,2)
for any y E {0, 1}.
1 3d
2d
Furthermore, the product of the marginal distributions are
6d2
2d (1 - 3d)
Pr(X, 2) 2) Pr(X,
v,
3d (1 2d)
(1 - 2d) (1 - 3d)
} (1 2d)
1/2 (1 2d) d
Pr(Z; y, Pr(Zg v,
for any E {0,
d
We can further write down the difference between the joint distribution and the product of the marginal distributions.
2d(1 3d) -2d(1 - 3d)
-2d 3d)
Pr(X,, X3 2) Pr(X; y, )r v)
2d(1 3d)
d
2r 2, Pr(Z, y, Pr(Zj 11,2)
-d d
Note that in this example, Pr(Z, Z, v) Pr(Z, 2, Pr(Z, y, = d for any Z₁ Z₁ e {0, 1), satisfying the condition in the proposition. Furthermore, for any 1: {0,1), the scoring function TA (ks, h, 1) = if and only if h h- and otherwise. Now, we compute the expected score when both agents honestly report the high-effort signal.
Si(X,, X, 12)
Σ Pr(Z = 2)
=E(0.1)
Σ
(Pr(X, = I,, -I, X, xx 2) -Pr(X; = x, 2) Pr(X, = x, 22) T's (T
Σ
Pr(,, 12) -Pr(ss 2) Pr(zz |=]
8d(1 - 3d).
We show that this score is strictly smaller than the expected score when agents report 0 if and only if X, = Z, = 0 (and X, = Z, = 0) and report otherwise. Let be the strategy such that 0[k; 0 if and only if I = for any z. We write down the joint distribution and the product of
marginal distributions for X, and X,.
2d 0
Pr(X,, 1,,,),
1
4d
2d( - 2d)
Pr(f; 1 y. Pr(X' 1 2) = 2d(11 - 2d)
for any 2 E {0,
(1 - 2d)²
28

---

The difference between the above two matrices is thus
2d - 2d)
-2d(1 - 2d)
Pr(X,, X2 11, y. Pr(X) y, = (-2d(1 - 2d)
for any y, E {0,
2d(1 - 2d)
This means that the expected score under the strategy profile (0, 0) is
S, x, 2)= = Σ Pr(Z= =)
=E(0.1)
Σ
(Pr((X, = T2, X, = By 2) Pr(X; 24 2) Pr(X, 2j 12)) Ts 2g,
(,0,,(0011)
Σ
Pr(X; =
3i, X3 = 1=)= =) Pr(X, xz 12) Pr(X; 3zg =)
(0.1)
= 8d(1 - 2d)
(, () + 8d
D.3 Information Monotonicity under Lazy-reporting Strategies
Proof of Proposition 4.6 A lazy-reporting strategy is a convex combination of truth-telling and a
no-effort strategy. Therefore, it suffices to show that the expected score of truth-telling is strictly higher than that of any no-effort strategy.
A no-effort strategy is essentially mixture of two types of strategy. First, u, (X,, Z,) is independent of both X; and Z, In this case, we have shown in the proof of Theorem 4.3 that the expected score is 0, strictly less than the expected score of truth-telling, which is IIvD (X;; X, Z).
Second, s4:( (x, Z4) depends only on Z4; let (x(X,, Z,) 2, Conditioned on Z, the Markov chain Z; - 2, holds. Now, fix agent j's lazy-reporting strategy and let her report be Xy. By the data processing inequality [Shannon 1948), we have:
ITD( X, 2) s LTV((Z;; X, 2).
Next, suppose agent reporting Z,, which is the morremmximizing lazy-reporting strategy regardless of agent j's strategy. For any lazy-reporting strategy such that X, Uj (X,, Zz) that agent plays, applying the data processing inequality to agent j's report yields:
LIv( X, Z) S max (JIvD Z, 2), JIvD(Z;;; Z)).
Therefore, Assumption 4.5 is sufficient for information monotonicity under lazy-reporting strategies.
E Learning the Scoring Function
we assumed that the underlying scoring function Ta is known. For complex signal structures, especially when the signal space large, we have to learn the scoring function from the data. This introduces a concern: if a significant fraction of agents rely on LLMs, the learned scoring function TA might be manipulated in a way that benefits the LLM--gnnratedd reports. We mitigate this concern using the following proposition. Let S (0,, 0,) be the expected score computed under the scoring function T when agents' strategies are O, and 6, respectively.
In Section 4.1
hold with no error such that = = 0 for any E [n)] Then, for any scoring function T E (0, 1] (2) and any strategy profile (0., 0,), ST (0.,0) 0,) S Sf'a (T.T), where denote the truth-telling strategy.
Proposition E.1. Suppose Assumption(4.1 4.1 and Assumption 4.2ld
Intuitively, the proposition holds because when TA is known and agents are truth-telling, the expected score is /TVD(XC X,; X,, Z). However, if Ta mislearned, the expected score is upper bounded by (&,; X, Z). Since our assumptions ensure that Z; and 2, are independent given Z, the data processing inequality guarantees that (X X, 2) < ITVD (Xv Z).
29

---

Proposition E. immples that the expected score of the conditioned CA mechanism is maximized if every agent exerts full effort and reports X, while any strategy that manipulates the true scoring function will only (weakly) decrease the expected score
Proof of Proposition E.1 scoring function is T,
Note that when agent and j play strategy 0, and 02 respectively and the
ST (0., 0z)
Σ Pr(Z= =) ZEΣ
Σ (Pr(a,, Ty Ri; [2) - Pr(a; zi [2) Pr(xy, I=)) T(0: (1))))))))
Because Assumption 4.2 and Assumption 4.1 hold with no error, by Lemma D.2 and Lemma D.1 it is easy to show that Pr(xi, 25, 24, 24
Pr(a, 2) Pr(z; z) Pr(z, 2) and Pr(xi, Ri
2) Pr(x; 22) Pr(z; 2).
Σ Pr(Z =)
Σ
Pr(zi 2) Pr(zj [=)
2E
Σ
(Pr(z,, 12) - Pr(x; | 2) Pr(z, | 2)) T(0,(a, 05,
E
Pr(Z z) Σ
Pr(z: 2) Pr(zy 1))
Σ
Pr(a, 1) Pr(ara 2) Pr(x) [=]
(Because T(k, h, is either 0 or 1.) (Marginalize 24 2,)
Σ
Pr(Z =)
Σ
Pr(xg 1 Pr(aa 12) Pr(ar, | =)]
(T,T)
F A Heuristic Generalization to Complex Signals
We have focused on scenarios where the signal space is small, such as answering multi-choice questions like sentiment labeling, preference elicitation, grading. However, many real-world crowdsoureing tasks involve open-ended questions that require workers to provide more complex responses, such image or video descriptions, idea generation, feedback and review collection. These tasks introduce unique challenges for applying our method, including 1) defining meaningful representation (textual) responses, and 2) computing the information between two responses condi- tioned on the principal's 's sample Z. In this section, we outline how to address these challenges and extend our method to accommodate tasks with complex, open-ended responses. We will illustrate our idea using peer review as an example, while we note that our method can be generalized to other similar settings.
Representing responses. Suppose two agents review the same paper and each has review after reading the paper, which is denoted X, and X, respectively. Instead of working hard to obtain high-quality review, each agent can use an LLM to generate fake review, denoted as Z; and Z,. The first challenge is how to measure the level of agreement between two reviews. The idea is to represent each review with an embedding vector and compute the similarity between pair of embedding vectors. Depending on the information of interest, different embedding methods can applied. For
example, classic NLP embedding methods such as Word2Vec Mikolov [2013), TF-IDF Manning
20099. BERT and its variants Devlin (2018 ,, [anh 2019), and GPT-3 Brown et al. 2020 can be
used to represent the language-level information within the responses.
Existing embedding methods may struggle when the focus is on content-level information, such as the judgments in peer reviews. For example, at the language level, an LLM- review may be similar human-written review that is rewritten for polishing by an LLM, while the information within their judgments can be very different. To address this, we consider an alternative, interpretable
Herr assume that the principal can learn the correct 7' if everyone reports their full-effort signal.
30

---

approach for representing complex responses. The key idea is to prompt an LLM to score response based on list of pre-determined characteristics.
For instance, we can prompt an LLM to classify whether a review adopts a compromising. critical, or neutral stance on specific aspects of paper, such as writing quality, completeness of related work, novelty of the research idea, or the quality of datasets. Using this method, each textual review X can be represented
vector ((X) of length K, where each entry o(X)x encodes the attitude of the review toward the K-th aspect. The similarity between two reviews X; and X, can then be quantified using the cosine similarity between their representations ((X,) and ((X;).
The same idea can be generalized beyond peer review. Suppose the principal has pre-defined set of questions about the responses, denoted Q where Q = K. Each question requires (normalized) score between -1 and 1, ensuring that the inner product between vectors can serve as a meaningful similarity measure. Given a response X,, the principal can prompt an LLM to iteratively get an answer to each question q E Q. This process converts each response X; into a vector øf X;) of length K, and each entry is score between -1 and 1. Similar ideas have been applied identify similar data points Zeng et al. 2024 analyze peer review Liang et al. [2024] and select features Jeong et al. 2024
Computing the conditioned similarity. The key idea of the conditioned CA mechanism is to give an agent higher score for having responses that correlate with another agent j's responses beyond what both share with Z. To measure how (X;)) and ((X,) correlate conditioned on ((Z). direct application-trratiee
each vector component as a random variable-quicke becomes infeasible due to the exponential growth of the signal space in the vector dimension K. Therefore, we adopt heuristic: we compute the similarity between (Xx) and (xx,) after removing their shared direction with Z). Specifically:
1. Project each vector onto the hyperplane orthogonal to o(Z). +2 [e[2]]| be the unit vector in the direction of @(Z). Then,
Let us
((x)) = ((x,) - (á(X,) u. uz)uz.
<(X,) = ((x)) - (á(X,) u, u..
2. Compute the cosine similarity between ((X,) and @ (X,)' to score agent
By projecting the embedding vectors onto the orthogonal complement of df 6(2), we remove the information that already exists Z. This projection approach also used in NLP for "debiasing" word embeddings by eliminating similarity along "biased direction" [Bolubbasi et al. (0016).
Finally, we note that we do not assume the principal's questions in Q to be i.i.d. Thus, the theoretical guarantees in Section 4.1 do not directly apply to this heuristic generalization; we instead rely on empirical evaluation to assess performance.
G Experiments (Continue)
G.1 Large Language Models
In this work, we select five well-known LLMs generate labels or annotations for all used datasets, including GPT-3.5-turbo, GPT-4, 2e-mma-2-2biif Phi-3.5-mini-Instuuee 51 and Bestral-7B-Instruee vo.30 The corresponding prompt templates are provided in Appendix H Note that we choose only five models because other commonly used LLMs such as LLaMA and Gemini refuse to answer toxicity-related questions.
G.2 Details of the Baseline Methods
Output Agreement (OA) Perhaps the most straightforward idea to score an agent is to count the frequency of agreements between her responses and another agents' responses. Specifically, while scoring agent i, we iteratively pair her with a different agent and let Mi.j denote the set of questions
31

---

answered by both agents. Then, the score of agent is the average of the empirical probability of agreement, i.e. 1 n Eje(n]M EgeM.,, = where X is the crowdsourced dataset with |M
rows representing agents and columns representing questions A score of 1 thus indicates perfect agreement while score of O indicates no agreement at all.
Correlated Agreement (CA) We use the CA mechanism Shnayder et al. (016) a baseline which does not incorporate the principal's information Z. Details of the mechanism have been presented in Algorithm
Expeccation-Maxxiizzziion (EM) The EM algorithm a well-known approach to estimating the reliabilities of noisy agents and better recovering the ground truth considering the reliabilities of agents (Dawid and Skene, 1979]. Each agent is assumed to make mistakes according to confusion matrix I where 1,, denotes the probability that agent i reports h when the ground truth answer is I. The EM algorithm first initializes a prior of the ground truth for each question and confusion matrix for each agent. Then, it iteratively goes through an E-step and an M-step. In the E-step, the algorithm computes the posterior of the ground truth of each question given the prior and the confusion matrix. In the M-step, the confusion matrix of each agent is updated using the posterior computed in the E-step. The above two steps are iterated until the likelihood of D converges. Finally, score the reliability of agent as Ener I b. P(h) where P(h) is the empirical marginal distribution of observing l in the dataset.
Conditioned Agreement (OAz) The first three measures do not utilize the principal's infor- mation about the cheap signal, denoted as Z. Note that in our experiments, Z is the vector of labels generated by a particular LLM. The conditioned agreement metric scores a agent she can provide agreements to another agent in addition to Z. Similar to the agreement metric, the score of agent is the average of the empirical probability of agreement conditioned on Z, ie. n EEEMM EgeM.,, "x..,=
Intuitively, if agent uses an LLM whose gener-
|Mh,1 ated labels are similar to Z, then Xi, Zg will happen with small probability, meaning that the conditioned agreement score will be small.
G.3 Assumption Verifications (Continue)
Table Table Table 4 and Table 5 present the mutual information ITD (Z; Zz Z) on four datasets. Z and Z: are sampled by the same LLM indicated on the rows and Zj is sampled by the LLM indicated on the columns. The first observation is that all values displayed in black exceed 0.05, which represents the mutual information of the random reporting baseline. This implies that [2, Zz Z)= holds consistently, rejecting hypothesis 2.
Regarding hypothesis 4, we observe that the average correlations between two LLMs are generally weaker than those between two human agents, provided Z, and Z are independently generated by the same model. This trend is evident in the tables, where almost all bracketed values are positive, with one exception
G.4 Detecting Low-effort Agents (Continue)
G.4.1 Additional Figures and Tables
In Figure 4 we present additional figures on the toxicity dataset, analogous to Figure 36 for other LLMs. Analogous results to Ta for other datasets are summarized in Table Table 7 Table g and Table I These results support our main claim that, although our method is occasionally slightly outperformed by certain baselines, it consistently demonstrates the most robust performance across all tested settings.
T.ct g 0. 'The exception occurs in the preference alignment dataset, where samples of Z., Zy. and Z all generated by GPT-4.
32

---

Table 2: Conditional TVD Mutual Information Between Two LLMs on the Preference Alignment
Data
GPT-3.5
GPT-4
Gemma
Mistral
Phi
GPT-3.5
0.2255 (0.0412)
0.1973 (0.0694)
0.0865 (0.1802)
0.1224 (0.1443)
0.1311 (0.1355)
GPT-4
0.0715 (0.1537)
0.2559 (-0.0307)
0.0571 (0.1682)
0.0552 (0.1701)
0.0723 (0.1530)
Gemma
0.1013 (0.1819)
0.1767 (0.1066)
0.2453 (0.0379)
0.0655 (0.2177)
0.1174 (0.1659)
Mistral
0.0920 (0.1938)
0.1132 (0.1726)
0.0620 (0.2238)
0.2350 (0.0508)
0.0964 (0.1894)
Phi
0.0874 (0.1969) 0.1300 (0.1542)
0.0855 (9.1988) 0.0929 (0.1914)
0.2479 (0.0364)
Each entry represents /rvD(22 Z;; Z, 2) (Difference: (X; X, 2) - Twv ; Z,
2), where Zi and Zj are generated by the models in the row and column respectively, and Z uses the same model as Zμ.
Table 3: Conditional TVD Mutual Information Between Two LLMS on the Hatefulness Labeling Data
GPT-3.5
GPT-4
Gemma
Mistral
Phi
GPT-3.5
0.0317 (0.1605)
0.0362 (0.1560)
0.0385 (0.1537)
0.0426 (0.1496)
0.0353 (0.1570)
GPT-4
0.0077 (0.1575)
0.0286 (0.1366)
0.0229 (0.1424)
0.0216 (0.1437)
0.0253 (0.1400)
Gemma
0.0075 (0.1588)
0.0183 (0.1481)
0.0758 (0.0905)
0.0464 (0.1199)
0.0543 (0.1121)
Mistral
0.0040 6001)
0.0097 (0.1542)
0.0341 (0.12
0.0408 (0.1231)
0.0350 (9.12
Phi
0.0088 (0.1658)
0.0115 (0.1631)
0.0618 (0.1128)
0.0526 (0.1220)
0.0803 (0.0943)
G.4.2 Detection Performance vs. Number of Samples
We further examine how many questions the principal needs to sample to effectively detect low-effort agents. In F we report the AUC of the conditioned CA mechanism with the fraction of LLM- reliant agents fixed at 0. 0.15, while the fractions of random and biased agents are independently sampled from [0, 0.2]. We vary the sampling rate from 20% to 100%. The results vary across the datasets and the LLMs that agents use. Except for the hatefulness labeling dataset with GPT-3.5-reliam agents, sampling 50% of the questions is sufficient to outperform the baseline CA mechanism.
G.4.3 Conditioning on samples of multiple LLMs
In practice, agents may employ a variety of LLMs for low-effort response generation, and the principal lacks prior knowledge of which models they are using. To address this challenge, propose an iterative conditioning approach, systematically removing the influence of responses generated by each LLM available to the principal. The idea is to iteratively compute the CA score conditioned on each of the LLM samples the principal has, and then take the minimum as the final score. Intuitively, human agents will perform consistently well when conditioning However, M----reliant agents will perform poorly in at least one of the iterations and thus leading to lower final score.
any of the LLMs' information.
In Figure 6 we replicate Figure 3b for the setting where the LLM-reliant agents randomly pick two LLMs with half-half probability while the principal iteratively conditions out his samples for these two LLMs' answers. Our method exhibits the most robust performance across all datasets, echoing the previous conclusion.
G.S Complex Signals
We further test the idea illustrated in Appendix that generalizes our method to high-dimensional settings.
33

---

56
68
64
0.3
030
5.5
0.00
0.53
330
815
6.0
0.3
0:30
815
000
(a) GPT-4
(b) GPT-3.5-turbo
(c) Gemma-2-2b-it
69
0.7
0.5
64
03
Poutiis of MoChhaee
(d) Mistral--B-Inntruucke
(e) Phi-3.5-miiii-mee mmit-inii-tttt
Figure 4: The area under the ROC curve (AUC) for various methods while varying the fraction of LLM-reliant agents. These are the analogous figures for Figure 3b for the remaining LLMs on the toxicity labeling dataset.
0.90
0.85
0.85
0.80
CA
0.75
CA
CA
0.70
0.05
0.70
0.00
0.05
0.55
GPT-3.5
Mistral
Phi
GPT-3.5
Mistral
0.50
GPT-4
Gemma
GP1-4
Gemma
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Sample Size
Sample Size
(a) Preference
(b) Hatefulness
0.85
GPT-3.5
Mistral Gemma
Phi
0.90
GPT-4
0.80
CA2
CA2
CA
0.75
AUC
0.65
GPT-3.5
Mistral
Phi
0.80
0.60
Sample
(c) Offensiveness
(d) Toxicity
Figure 5: The area under the ROC curve (AUC) for various methods while varying the fraction of questions the principal sample.
34

---

Table Conditional TVD Mutual Information Between Two LLMs on the Offensiveness Labeling
Data
GPT-3.5
GPT-4
Gemma
Mistral
Phi
GPT-3.5
0. 1267 (0.0809)
0.0926 (0.1149)
0.0450 (0.1625)
0.0785 (0.1290)
0.0802 (0.1273)
GPT-4
0.0419 (0.1092)
0.0920 (0.0590)
0.0520 (0.0990)
0.0867 (0.0644)
0.0794 (0.0716)
Gemma
0.0302 (0.2336)
0.0602 (0.2036)
0.1 (0.1486)
0.0597 (0.2040)
0.0676 (0.1962)
Mistral
0.0390 (0.1407)
0.0573 (0.1225)
0.0295 (0.1502)
0.0836 (0.0961)
0.0516 (0.1281)
Phi
0.0320 (0.1604)
0.0597 (0.1327)
0.0334 (0.1591)
0.0604 (0.1321)
0.0651 (0.1273)
Table 5: Conditional TVD Mutual Information Between Two LLMs on the Toxicity Labeling Data
GPT-3.5
GPT-4
Gemma
Mistral
Phi
GPT-3.5
0.1611 (0.0063)
1.2328 (0.0347)
0.1199 (0.0476)
0.1419 (0.0255)
1.1300 (0.0334)
GPT-4
0.0587 (0.1399)
0. 0.1035 (0.0951)
0.0755 (0.1231)
0.0673 (0.1312)
0.0652 (0.1334)
Gemma
0.0721 (0.1650)
0.0869 (0.1502)
0.1329 (0.1042)
0.0880 (0.1491)
0.1043 (0.1328)
Mistral
0.0602 (0.1413)
0.0622 (0.1394)
0.0454 (0.1561)
0.0934 (0.1081)
0.0642 (0.1373)
Phi
0.0416 (0.1756)
0.0517 (0.1655)
0.0382 (0.1790)
0.0472 (0.1701)
0.0666 (0.1507)
In this experiment, we randomly sample 500 papers from the ICLR 2024 OpenReview data and select three human reviews per paper human responses X. For each of the same batch of papers, we further use each of the five LLMs to generate three Al reviews using three Aimilar-maanigg yet different prompts (Appendix HD. We randomly pick one of the generated reviews the principal's sample 2, and the other two generated reviews will be used to simulate the responses of LLM-reliant agents.
We prompt GPT-4 to process each human review and LLM --geerrrrtedd review into a 19-dimension vectors with the prompt shown in Appendix H We further simulate review embedding tensor with size of n X m x 19. where there are n = 188 reviewers and m = 500 papers in total. Each reviewer randomly reviews k {5,8) 8} papers. An Ouim fraction of reviewers are LLM-reliant agents whose review vectors are replaced with a particular type of L.L--eerratedd reviews. An ap fraction of reviewers prior-reporiin agents, where the score in each entry of the review vector is randomly sampled according to the marginal distribution in the dataset. An Og fraction of reviewers are random agents, where the score in each entry of the review vector is sampled uniformly random. In our experiments, we independently sample each of Ollm. Opp and a, uniformly from the interval [0, 0.1] and take average over 100 trials. While constructing the review embedding tensor, we guarantee
1.0
0.5
038
3.4
04
33
2
0.
0.0
100
0.38
0.15
220
200
0.5
0.2
6:15
820
1.00
053
0.10
035
0.2
(a) Gemma & GPT-3.5
(b) Gemma & GPT-4
(c) GPT-3.5 & GPT-4
Figure 6: Area under the ROC curve (AUC) for various methods as the fraction of L.M--reliat agents increases. In each case, LLM--reiant agents randomly choose between two LLMS to generate responses, and the score is defined as the minimum of the two conditioned CA scores, each conditioned on samples from one LLM.
35

---

Table Average AUC (and bottom 10% quantiles) across datasets and methods with GPT-3.5-reliant
agents.
Dataset
OA
CA
OA
EM
CAg (ours)
Hatefulness
0.61 (0.40)
0.90 (0.84)
0.90 (0.85)
0.86 (0.77)
0.89 (0.86)
Offensiveness
0.80 (0.69)
0.76 (0.58)
0.48 (0.27)
0.76 (0.57)
0.83 (0.77)
Toxicity
0.75 (0.62)
0.72 (0.51)
0,47 (0.24)
0.71 (0.50)
0.78 (0.71)
Preference Alignment
0.88 (0.67)
0.86 (0.69)
0.88 (0.76)
0.96 (0.91)
0.91 (0.83)
Table Average AUC (and bottom 10% quantiles) across datasets and methods with Mistral-reliant
agents.
Dataset
OA
CA
OAz
EM
CA (ours)
Hatefulness
0.82 (0.75)
0.72 (0,51)
0,62 (0.48)
0.76 (0.54)
0.84 (0.78)
Offensiveness
0.71 (0.54)
0.79 (0.62)
0.63 (0.47)
0.75 (0.57)
0.89 (0.84)
Toxicity
0.65 (0.50)
0.75 (0.56)
0.58 (0.39)
0.70 (0.49)
0.84 (0.79)
Preference Alignment
0.86 (0.66)
0.89 (0.75)
0.94 (0.87)
0.94 (0.87)
0.93 (0.88)
that each paper has approximate reviews so that the variance the number of reviewers per paper is controlled. Again, consider five types of LLMs and for each type of LLM-rrelaat agent, the principal's sample Z is an independent generation of the same LLM under a different prompt.
We consider three methods to score the reviewers.
Cosine Similarity: Scores each reviewer based on the average cosine similarity between their review vector and the review vectors of other reviewers for the same paper.
Projected Cosine Similarity: Applies the method described in Appendix Fl where each review vector is projected onto the subspace orthogonal to the principal reviewer's sampled embedding before computing similarity.
Negative Similarity Z: Scores each reviewer by taking the negative average cosine similarity between their review vector and the principal reviewer's sampled vector Z.
We present the results in Figure It is clear that conditioning out the principal's 's sampled LLM reviews is more effective than computing the cosine similarity directly. Furthermore, naively removing reviews that are similar to 7 has a large variance, whose performance greatly depends on the existence of random agents and prior-reporting agents.
We further note that there is big room for improvement. First, from methodology perspective, the 19-dimensional embedding is heuristically chosen and optimized for distinguishing human-Al reviews. Second, from the data perspective, human reviews are naturally very noisy. Our method can be potentially applied to other crowdsourcing tasks where human responses are more consistent. We view these as future directions.
H Prompts Templates
We adopt our prompt templates based on recent studies (Pang et al. 2024 Liu et al. 2025). The specific templates used for the two dataset settings are presented Table e and Table respectively.
2024 Pang et al.
36

---

Table Average AUC (and bottom 10% quantiles) across datasets and methods with Gemma-reliant
agents.
Dataset
OA
CA
OA
EM
CAz (ours)
Hatefulness
0.85 (0.79)
0.71 (0.51)
0,60 (0.42)
0.77 (0.56)
0.85 (0.80)
Offensiveness
0.75 (0.58)
0.78 (0.59)
0.61 (0.43)
0.76 (0.53)
0.90 (0.84)
Toxicity
0.70 (0.58)
0.75 (0.59)
0.56 (0.36)
0.72 (0.52)
0.85 (0.79)
Preference Alignment
0.80 (0.58)
0.87 (0.73)
0.76 (0.62)
0.92 (0.85)
0.92 (0.86)
Table 9: Average AUC (and bottom 10% quantiles) across datasets and methods with Phi-reliant
agents.
Dataset
OA
CA
OA
EM
CAg (ours)
Hatefulness
0.87 (0.81)
0.67 (0.44)
0.66 (0.47)
0.77 (0.56)
0.85 (0.79)
Offensiveness
0.76 (0.64)
0.76 (0.59)
0.58 (0.35)
0.75 (0.53)
0.88 (0.83)
Toxicity
0.73 (0.62)
0.74 (0.54)
0.54 (0.35)
0.71 (0.48)
0.83 (0.77)
Preference Alignment
0.85 (0.68)
0.82 (0.63)
0.82 (0.63)
0.97 (0.94)
0.93 (0.88)
Prompt Template for Toxicity Labeling
As a helpful system judgment assistant, your task is to answer the following questions for the given comment. You should label comment hateful if it expresses hatred towards a targeted group or is intended to be derogatory, to humiliate, or to insult the members of the group. You should label comment offensive if it contains hurtful, derogatory, obscene comments. You should label a comment as toxic if it is a rude, disrespectful, or unreasonable comment that is likely to make readers want to leave discussion.
System Prompt:
Question 1: Do you think this comment is hateful? -Yes or No. Question 2: Do you think this comment is offensive? - Yes No.
Question 3: Do you think this comment is toxic? -Yes or No.
Comment: [Comment]
User Prompt: Now, please evaluate the following comment and return the final answers in JSON format:
"question 1": {"hateful"; <yes/no>},
"question 2": ("offensive": <yes/no>),
"question 3"; ("toxic": <yes/no>>
0.8
0.8
0.6
0.6-
0.5
0.5
0.3
0.3-
0.2
0.2
Cosine Similarity
Cosine Similerity
Projected Casine Similanity
Projected Cosine Similoityy
Negative Similarity
Negative Similarity
0.0
GPT-3.5
GPT-4
Mistrel
0.0
GPT-3.5
GPT-4
Mistral
Cheater LLMs
Cheater LLMs
(a) 5 reviews per reviewer
(b) j: reviews per reviewer
Figure 7: A comparison of the area under the ROC curve (AUC) of various methods in the peer review dataset.
37

---

Prompt Template for Preference Alignment
System Prompt:
Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants the user question displayed below. You should choose the assistant that follows the user's instructions and answers the user's s question better. Your evaluation should consider factors such the helpfulness, harmlessness, truthfulness, and level of detail of their responses. Begin your evaluation by comparing the two responses and provide short explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names the assistants. Be as objective as possible.
After providing your explanation, output your final verdict by strictly following 5-point Likert scale:
A is clearly better
A is slightly befter
Tie
B is slightly better
. B is clearly better
Question: [Question]
Response A: [Response
Response B: [Response B]
User Prompt:
Now, please evaluate the responses and return the final verdict JSON format:
"final verdict"; bets--e / A-is-slightllybbtteer Tie, B-is-slightlly-better / B-is-cleerly-better>>
"short explanation": <your short explanation within
38

---

Prompt Template for Paper Review 19-Dimensional Judgment Embedding Generation
System Prompt:
You will be given a review of a computer science paper. Such a review is typically composed of summary of the corresponding paper, reasons accept, reasons to reject, and questions for the authors. Upon receiving the review, your task is to generate an embedding of the review in a 19-dimensional space according to the following instructions:
1. Identify and ignore the summary and the questions section the review (if any).
2. Separate the review into ad advantages and disadvantages.
3. Further decompose the advantages and disadvantages into independent judgments.
4. You will be given 19 judgment categories. For each category:
If positive-toned judgment falls into the category, assign
of
If a negative-toned judgment falls into the category, assign score of -1.
no judgment related to the category, assign a score O.
5. Return the scores for all 19 categories as a dictionary. Do not include any additional explanation.
The judgment categories are:
Clarity and Presentation
Related Work
Theory Soundness Novelty of Method Implications of the Research Relevance to Conference Utility and Applicability Algorithm Efficiency
Experiment Design Practicality of Method Reproducibility
Significance the Problem
Comparison to Previous Studies
Ethical Aspects Scope and Generalizability
Experiment Execution
Defense Effectiveness
Real-world Applicability
Data Generation and Augmentation
User Prompt:
Now, please evaluate the paper review and return the scores for each category in the following JSON format:
"Clarity and Presentation"; <-1/0/1>,
"Related Work": <-1/0/1>,
"Theory Soundness"; <-1/0/1>, "Experiment Design" <-1/0/1>,
"Significance of the Problem": <-1/0/1>,
"Novelty of Method": <-1/0/1>,
"Practicality Method": <-1/0/1>,
"Comparison to Previous Studies": <-1/0/1>,
"Implications of the Research": <-1/0/1>,
"Reproducibilityy:: <-1/0/1>, "Ethical Aspects"; <-1/0/1>,
"Relevance to Conference": <-1/0/1>,
"Experiment Execution": <-1/0/1>,
"Scope and Gemeralizability": <-1/0/17,
"Utility and 4pplicability"; <-1/0/1>,
"Defense Effectiveness": <-1/0/1>,
"Real-world Applicability"; <-1/0/1>,
"Data Generation and Augmentation": <-1/0/1>, "Algorithm Efficiency": <-1/0/1>
39


Reviews:
[{"review_id": "mdAG3oAleu", "reviewer": "Reviewer_CDQu", "summary": "This paper develops a theoretically robust quality measure to evaluate the quality of human contributions in addition to existing LLM answers and detect workers with less informative responses.", "questions": "What is the computational cost of the score?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper studies an important task. The method is well-motivated and well-thought-out.\n- The paper provided interesting insights regarding the pattern of human labelers and LLM labelers. \n- The paper is well-written and clear.\n\nThe method is developed with a realistic setting and moderate assumptions, a good improvement over previous methods. The authors made it very clear about the scope and how to interpret the results, which I appreciate a lot. The writing is nice and easy to follow, with sufficient explanations and motivations. The experiments are well-designed and validate the claims made earlier.\n\nOverall, I think it is a good paper.", "was_revised": true, "final_justification": "The paper is well-written with clear contribution", "rebuttal": "We’re grateful for the reviewer’s recognition of our contributions. For your question, the computational cost is very minor. Computing the scores once for the tested dataset takes less than 1 minute on a MacBook Pro (no repeating for computing error bars). We will add the computation requirement in the paper."}, {"review_id": "w7SFnhOt6m", "reviewer": "Reviewer_NyJA", "summary": "This paper proposes a peer prediction-based mechanism to detect LLM-assisted cheating in crowdsourced annotation tasks where ground truth is unavailable. The core hypothesis posits that authentic human-generated responses exhibit distinct correlational patterns irreducible to LLM outputs. The introduced Conditioned Correlated Agreement (CAz) mechanism evaluates worker responses by quantifying inter-annotator correlations conditioned on requester-accessible LLM-generated signals (Z)\nThe paper addresses a timely problem with a theoretically grounded and empirically validated method. While assumptions require relaxation and scope expansion needs further validation, the CAz mechanism represents a significant advance in crowdsourcing data integrity preservation.", "questions": "How might CAz be adapted for high-dimensional, open-ended responses (e.g., peer reviews, creative writing) while maintaining theoretical guarantees? Does the embedding projection heuristic preserve information monotonicity properties?", "limitations": "1. The major concern is still the blurring line between \"Tool Use\" and \"Cheating\": we are all becoming cyborgs in our daily work. At what point does using an LLM to assist with a task cross the line from acceptable tool use to data-corrupting cheating? Or, shouldn't we towards a more clear and inspired benchmarking that show the more interesting distribution that we want AI to be but not just reject something?\n2. Is LLM-Polished Data bad? The paper doesn't provide a framework for navigating this gray area of legitimate augmentation versus outright replacement.\n3. The risk of penalizing Neurodiversity and creative outliers. The method inherently rewards workers whose judgments align with the \"diligent majority.\" What about a worker who is a genuine human but has a unique, outlier perspective? Or someone who interprets the task instructions in an unconventional but still valid way? Their responses might not correlate well with others, leading the CAz method to incorrectly flag them as low-effort or a \"cheater.\" The system could inadvertently filter out cognitive diversity, which is often the very thing subjective tasks are trying to capture.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. Addresses the critical challenge of data integrity erosion due to undetected LLM usage in human feedback pipelines, particularly for tasks reliant on subjective human judgment (e.g., preference labeling, toxicity assessment).\n2. The CAz mechanism provides a training-free, ground-truth-agnostic framework leveraging requester-generated LLM signals to disentangle genuine human effort from LLM-derived responses.\n\n**Weaknesses**\n\n1. The core requirement of this paper is that the principal can generate labels with an LLM $z$ to use for conditioning. This hinges on a crucial assumption: the principal knows which LLMs the cheaters are likely to use. If cheaters use a different proprietary model that the principal doesn't have access to, the effectiveness of the conditioning step is questionable. And\n2. The strongest guarantees (for e-information monotonicity against any cheating strategy) rely on Assumption 4.1 and 4.2. However, the papers own empirical tests in Section 5.3 show that these assumptions \"generally do not hold in practice\". The authors then fall back on the weaker Assumption 4.5, which only provides guarantees against \"naive\" cheating strategies.\n3. When the fraction of LLM cheaters is very low, the performance of the method degraded to traditional methods. The scene of this CAz can be clarified more specifically.\n4. The further requirements of LLM data annotation would turn to more complex and open-ended text, which can exceed the tasks analyzed in this paper.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the thoughtful comments. Many of the concerns relate to the negative observations we made in the paper, and some generalizations of our method that we consider as future work. We would like to emphasize that **the goal of our paper is not to offer a panacea for the challenging problem of detecting LLM cheating WITHOUT ground truth, but rather to propose a useful solution and carefully examine when it works and when it does not.** However, we aim to show that the scenarios where our method works are fairly practical and important.\n\nBelow, we respond to each point with more details.\n* Assumption on knowing the cheating LLMs\n  * **[This is a useful message]** We view the observation that $CA_Z$ is most effective when $Z$ is generated by the same model as $Z_i$ not as a weakness, but as a useful insight. It is an important warning to both the peer prediction and LLM communities: when conditioning on a mismatched LLM, the correlation among responses generated by the same LLM can exceed that of genuine human labels. In such cases, peer prediction mechanisms may fail to effectively incentivize or identify authentic human feedback.\n  * **[This is a practical assumption]** Our method is primarily designed for settings where a non-trivial fraction of users rely on the same popular LLM(s) or report uninformatively, in which case the principal can draw samples. This is a particularly harmful and practical setting, where existing baselines have limitations. In contrast, if only a few users adopt uncommon or proprietary models, their outputs tend to be less correlated and can be effectively distinguished from human labels by $CA_Z$, as well as by existing crowdsourcing algorithms. We show that $CA_Z$ performs competitively with the baselines across both settings, whereas each baseline fails significantly in at least one of them.\n* Weaker truthful guarantee\n  * **[This is a useful message]** We view the hardness of obtaining information monotonicity under arbitrary cheating strategies as a part of our contributions, not a weakness. As AI becomes better, it is a rather challenging task to fully distinguish LLM labels from human labels. Therefore, we consider it to be good news that we are able to offer a tool that is useful to address cheating with naive strategies.\n  * **[This is a practical assumption]** Furthermore, perhaps misled by its name, naive strategies are actually particularly important and common in practice. As highlighted in lines 136-140, agents tend to either decide to work for observing $X_i$ or decide to cheat and observe $Z_i$, but often never try to observe them at the same time. Any non-naive cheating requires simultaneously observing $X_i$ and $Z_i$ and thus is more like an adversarial behaviour rather than cheating. \n* Low fraction of LLM cheaters\n  * As noted above and in lines 354–357, the key advantage of $CA_Z$ lies in its theoretical guarantees and robust performance across settings. While knowing the fraction of LLM cheaters might allow for more tailored methods, this information is rarely available in practice—underscoring the value of a robust, general-purpose approach.\n* Generalization of $CA_Z$ to Open-ended crowdsourcing tasks\n  * Unfortunately, the heuristic generalization of $CA_Z$ (shown in Appendix F) does not technically preserve the truthful guarantees. The key issue is that the dimension of the embedding vector can be very high, making it difficult to directly estimate the mutual information between two embedding vectors. Therefore, we use cosine similarity as a heuristic alternative, which preserves the spirit of mutual information but is not technically an estimate of it.\n  * Even though the theoretical guarantee does not generalize, the performance suggests that the idea still works. However, further research on (weaker) theoretical guarantees for high-dimensional embeddings and advanced detection methods is beyond the purpose of the current paper.\n* Tool-using vs cheating, and LLM-polished data\n  * As mentioned in line 376, we view further distinguishing “tool use” and “cheating” as an important future work. For the main application considered in the current paper, i.e. subjective labeling tasks, we assume the principal is primarily interested in genuine human feedback, even though AIs (or human-AT teams) can do better in some aspects. In other words, we target the crowdsourcing tasks where AI usage is forbidden by the principal.\n  * Distinguishing between LLM-polished and LLM-generated responses is particularly relevant in tasks involving textual feedback, such as peer review, rather than in labeling tasks. We provide a discussion of the generalization of $CA_Z$ for textual responses in Appendix F, where the goal is exactly to distinguish responses based on the information they have rather than the vocabulary they use. However, as noted, our theoretical results do not directly extend to this setting, and we view this as a promising direction for future work. \n* Penalizing outliers\n  * We agree that outlier responses are less likely to receive high scores under peer prediction mechanisms. However, it is fundamentally difficult—if not impossible—to distinguish low-effort cheaters from genuine outliers based solely on observed responses. This tradeoff is not unique to $CA_Z$; it applies broadly to existing crowdsourcing algorithms. Due to this tradeoff, we can't enjoy the benefits without accepting the costs.\n  * Nonetheless, we are still designing new scoring mechanisms and are actively using them in practice. This is because a primary issue in crowdsourcing is that agents are not incentivized to exert effort, often resulting non-trivial fraction of data that might be pure noise. Therefore, we need scoring mechanisms to identify potentially unreliable responses and flag agents who may warrant further scrutiny. Importantly, **low scores do not imply automatic exclusion, but rather provide the principal with a tool to prioritize which data/agent may need additional attention.**"}, {"review_id": "XqctpoG2VZ", "reviewer": "Reviewer_huUj", "summary": "This paper proposes theoretically grounded quality measure method for annotation tasks, called the conditioned correlated agreement (CA) mechanism. The method quantifies the correlations between worker answers while conditioning on (a subset of) LLM-generated labels available to the requester. Building on prior research, the proposed training-free scoring mechanism is theoretical guaranteed under a novel model that accounts for LLM collusion. Experimental results show that their method is effective in identifying LLM-assisted cheating when conditioning on the “right” model.", "questions": "Please refer to the weaknesses above.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths：\n1. Compared with existing LLM detection methods, the method proposed in this paper is based on peer prediction and is suitable for structured annotation tasks like multiple-choice labeling.\n\n2. The proposed conditioned CA mechanism is information monotone under naive strategies, which motivates workers to truthfully report their high-effort signals.\n\n3. Experiments demonstrates that the proposed conditioned CA mechanism is robust in detecting low-effort cheating on real-world crowdsourcing datasets.\n\nWeaknesses:\n1. Figure 2 only shows the results on (a) Hatefulness and (b) Preference Alignment. Why not directly show the results on (a) Hatefulness, (b) Offensiveness, (c) Toxicity and (d) Preference Alignment? As the extension of Figure 2, Figure 4 shows the results on Hatefulness again, but the result is different from Figure 2. Yet at the same time, the result on Toxicity is not shown. Please check if the figures are incorrectly labeled.\n\n2. In Figure 2's legend, “Expert”, “Normal worker”, and “Human” refer to X_i according to the “Assumption Verification” subsection, which is inconsistent with the description of Figure 2’s caption.\n\n3. In Figure 3b, it is not clear which methods the different colored lines correspond to respectively. Please provide a legend.\n\n4. In subsection 5.3, the metric chosen to measure the effectiveness of proposed method to identify cheating agents with naive strategies is AUC. How does AUC reflect the effectiveness of detecting cheating agents?", "was_revised": true, "final_justification": "The author has adopted my suggestions regarding Figures 2, 3, and 4, and provided additional explanations for the experimental metrics. Considering the feedback from other reviewers, I have decided to keep my score unchanged.", "rebuttal": "We appreciate your clarification questions about the figures. They all make great sense and will be adjusted in the future version. \n* Label mistake in Figure 4\n  * Thanks for pointing this out! Figure 4 (a) should be “toxicity” rather than “hatefulness”. We defer two figures to the appendix for space considerations.\n* Captain of Figure 2\n  * You are right. We will update the caption to “The TVD mutual information between $X_i$ and $X_j$ (for “Human”, “Normal worker”, and “expert”) or $Z_i$ and $X_j$ (for remaining bars) conditioned on $Z$.” We will also update the y-axis label.\n* Missing legend for Figure 3\n  * Thanks for pointing out the mistake. The legend is the same as Figure 5 (will update).\n* AUC curves\n  * In our context, let $I^+$ be the set of hard-working human labelers and $I^-$ be the set of simulated cheaters. Let $S_i$ be the score of labeler i. We are able to compute the AUC score as follows:\n $$ \\text{AUC} = \\frac{1}{|I^+| \\cdot |I^-|} \\sum_{i \\in I^+} \\sum_{j \\in I^-} (\\mathbb{1}[ S_i > S_j ] + 0.5 \\cdot\\mathbb{1}[ S_i = S_j ])$$\n  * Intuitively, AUC quantifies the probability that a randomly chosen good labeler is ranked higher than a randomly chosen cheater under each scoring function. It measures how well each scoring function can be used to distinguish good workers from cheaters. We will further explain this concept in future versions of our paper."}, {"review_id": "Hm3DApolaJ", "reviewer": "Reviewer_DdeM", "summary": "This research aims to address the impact of Large Language Model (LLM)-assisted cheating on the data quality of crowdsourcing. Traditional methods for dealing with low-quality workers in crowdsourcing often rely on correlations among workers' responses. However, such methods are ill-suited for situations where correlations arise from the use of LLM-generated outputs.\n\nTo overcome this, this study adopts an approach that utilizes correlations between workers' responses, conditioned on LLM-generated outputs. By doing so, the paper proposes a theoretically grounded quality measure for crowdsourced annotation tasks. The effectiveness of this approach is further evaluated through empirical experiments.", "questions": "N/A", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper appears to be very well-written in my assessment. The motivation is clear, and I believe the content is a good fit for this conference.\nA weakness of this paper is that its primary focus is on crowdsourcing datasets involving subjective questions or tasks where human experts are considered the gold standard. With the recent advancements in LLMs, the scope of applicable tasks might decrease over time.\n\nNevertheless, overall, I consider this to be a good paper.", "was_revised": true, "final_justification": "My evaluation of this paper has not changed.", "rebuttal": "Thank you for your appreciation! We agree that human-based crowdsourcing is gradually being replaced (at least reshaped) by AI, and may eventually be fully replaced. This means the value of the current paper may be timely discounted (and we hope so). However, at least for the next couple of years, we envision that protecting genuine human feedback will still be an essential and important issue, such as supervising trustworthy AI via RLHF. Now is perhaps the right time to pay more attention to this pressing problem."}]

Author rebuttal:

Meta-review: This paper addresses the emerging problem of detecting LLM-dependent contamination in crowdsourcing. 
A framework that works without ground truth is presented and both theoretical support and empirical effectiveness are shown. 
The timeliness and novelty of the problem setting, as well as the convincing experimental results, are strong points that were appreciated by the reviewers.

At the same time, there are remaining concerns about the simplicity of the model and potential risks to fairness, such as unfair treatment of outliers or minority opinions. Ethical reviews also pointed out issues related to worker rights, privacy, and the unclear boundary between LLM assistance and cheating. The authors responded by promising improvements, including revising terminology, clarifying data usage, and emphasizing human-in-the-loop oversight.

Overall, the work makes a clear and timely contribution and points to an important direction for the field.
Decision: Accept (poster)