Title: Truthful Aggregation of LLMs with an Application to Online Advertising
Venue: NeurIPS 2025
Accepted: yes
Keywords: Market Design

LLMs

Large Language Models

Auctions

Economics and Computation

LLM Aggregation

Online Advertising Auctions

Preference Aggregation
Abstract: The next frontier of online advertising is revenue generation from LLM-generated content. We consider a setting where advertisers aim to influence the responses of an LLM, while platforms seek to maximize advertiser value and ensure user satisfaction. The challenge is that advertisers' preferences generally conflict with those of the user, and advertisers may misreport their preferences. To address this, we introduce MOSAIC, an auction mechanism that ensures that truthful reporting is a dominant strategy for advertisers and that aligns the utility of each advertiser with their contribution to social welfare. Importantly, the mechanism operates without LLM fine-tuning or access to model weights and provably converges to the output of the optimally fine-tuned LLM as computational resources increase. Additionally, it can incorporate contextual information about advertisers,  which significantly improves social welfare. Via experiments with publicly available LLMs, we show that MOSAIC leads to high advertiser value and platform revenue with low computational costs. While our motivating application is online advertising, our mechanism can be applied in any setting with monetary transfers, making it a general-purpose solution for truthfully aggregating the preferences of self-interested agents over LLM-generated replies.


Peer reviews:
comments: SUMMARY:
This paper study the scenario where LLM are applied to ad auctions and they propose a mechanism that converges to optimal distribution. They conduct experiments of MOSAIC in real settings. Their payment rule ensures that there is no incentive for a participant to misreport their preference, according to the theory analysis.

STRENGTHS AND WEAKNESSES:
Strength: 

(1) Applying LLM to ad auctions is an emerging field and has own its significance in real application and requires further research. This paper provides one way of doing so.

(2) The results provided in this paper are comprehensive, in the sense of with both theory analysis and empirical results.

QUESTIONS:
How in real world that this model makes sense, i.e., the auctioneers pay to influence the output of LLM.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MOSAIC, a novel auction mechanism designed for the truthful aggregation of multiple advertisers' preferences concerning Large Language Model (LLM) outputs. The mechanism addresses the challenge of enabling advertisers to influence LLM-generated content while simultaneously allowing the platform to optimize advertiser utility without compromising user satisfaction. MOSAIC offers several key advantages: it incentivizes truthful preference reporting, provably and efficiently converges to the optimal LLM output without requiring fine-tuning or direct manipulation of model weights, and effectively integrates contextual information to enhance performance. Experimental evaluations underscore MOSAIC's effectiveness, demonstrating strong advertiser value, robust platform revenue, and notable computational efficiency.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n\n1. This paper investigates an interesting and highly practical problem with strong motivation, particularly relevant for online advertising scenarios.\n\n2. The authors provide thorough explanations for each of their results and conduct detailed comparisons with prior work across various properties, significantly improving the paper's clarity and ease of comprehension.\n\n3. The proposed mechanism is straightforward to implement, and the experimental section offers valuable insights into how the size of the candidate set ($M$) influences other critical variables.\n\n### Weaknesses:\n\nMy primary concern lies with the relative technical strength of the paper. The main theoretical result, Theorem 5.1, appears derivable by simply validating the cyclic monotonicity of the allocation rule, which in turn directly implies the payment rule. While other theoretical results pertain solely to computational aspects.\n\nRegarding the experimental section, the comparison between the Context-Aware Mechanism Output and Baseline Mechanism Output in Figure 1 is arguably meaningless. Without incorporating advertiser-specific information, it is unreasonable to expect the LLM to generate content relevant to them.\n\nQUESTIONS:\n1. How robust is the MOSAIC mechanism in scenarios where advertiser preferences significantly diverge from the user preferences encapsulated by the reference LLM?\n\n2. What implications arise for MOSAIC's performance or theoretical guarantees if the contextual policy ($\\pi_{con}$) is substantially different from the optimal policy ($\\pi_{opt}$)? In the current experiments, the observed difference between $\\pi_{con}$ and $\\pi_{opt}$ appears relatively small.\n\n3. Do the authors believe that the current allocation rule presented is the unique solution for this problem setting?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a model where multiple LLMs need to truthfully aggregate preferences, with respect to a reference LLM. The authors noted that online advertising is the most natural application setting of this model, so discussion of this paper is centered around advertising.\n\nUnder the authors' model, each advertiser has her own LLM, and the advertisers aim to influence the reference LLM (provided by the search engine to give good answers to users maximizing user experience). The authors noted a relationship between the studied model and Reinforcement Learning from Human Feedback. The core technique is described in Equation 2, which basically uses RLHF techniques. To summarize it in English, we can use the refernece LLM to generate samples, and then apply importance sampling techniques to create samples that maximize Equation 1, which considers the advertisers' preference.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, the paper's proposal of the new LLM advertising framework is, for the most part, practically sound. The algorithm is elegant. The authors showed that the allocation rule is monotonic so strategy-proof payments can be derived and different versions of IR conditions are discussed in the paper. Even though LLM advertising's main goal should be revenue in my opinion, but welfare maximizing is also a reasonable objective. Section 4.2 discusses practical considerations, which in my opinion adds a lot of strength to the paper. The proposed framework is easy to implement, unlike the earlier token-based auction, which in its current form cannot be used in practise. For example, the proposed framework uses 20 LLM forward passes and the welfare is already quite good, as shown in experiments, and the token-based auction would require a lot more to generate any long paragraph.\n\nThe downside of this paper is \"context-aware\" part. In practise, if we need to provide context like footnote 4 -- \"try to mention advertiser x, advertiser x description\", then I think it creates a free-rider situation. It will be a problem if the objective is revenue maximizing. If we need to provide context like this, then I think there is a good chance that an advertiser gets free advertisement. What I mean is that an advertiser can bid 0 and there is still chance of having the ads shown if the context is provided this way in practise. This is difficult to fix. As the authors noted, we kind of must provide the context in practise, because the beautiful theoretical result in Equation 1 and 2 cannot be used directly without the context.\n\nQUESTIONS:\nCould you please comment on whether free riding is an issue, as a result of providing context using the method in footnote 4.\n\nLIMITATIONS:\nYes, in the paper checklist, the authors emphasized that this is theoretical research.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a truthful mechanism MOSAIC for advertising with LLMs. The authors assume that each advertiser provides an LLM or reward model as the ad creation and wants to affect the auctioneer’s LLM output towards the advertiser’s benefit. This paper provides a practical mechanism to balance the advertisers’ utilities and user’s satisfaction. The mechanism includes an allocation rule that only requires API access to the advertisers’ models and approximately maximizes the social welfare, and a payment rule that incentivizes truthful reward reporting. The authors conducted experiments on real LLMs, demonstrating the effectiveness of MOSAIC.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\nTo the best of my knowledge, this work proposed one of the first practical mechanisms for LLM advertising, showing the potential to fill the gap between theory and practice. The mechanism is concise and theoretical analysis is solid.\nWeaknesses\nConsidering that the authors claim this is a practical mechanism, the experiment setup is not sufficient, leading to less convincing results. For example, the authors did not discuss the optimal M choice when various numbers of advertisers participated in. I suggest the authors provide more experimental results in various settings, including different kinds of advertiser’s model (reward model or LLMs), various ad topics and more advertisers.\nThe paper assumes advertisers can accurately encode their preferences via LLMs or reward functions, which may not be feasible for all advertisers, especially smaller or non-technical ones.\nSome of the notations are not clear. For example, what is the exact definition of $\\pi_{\\hat{r}_{-i}}$ in Lemma 5.2 (Line 287)?.\nThe paper does not deeply engage with ethical implications of allowing advertisers to influence LLM outputs, such as manipulation, bias amplification, or impacts on user trust. There is also little discussion of fairness, such as whether small advertisers get fair exposure compared to large ones with higher rewards.\n\nQUESTIONS:\nWhen the advertiser evaluates its preference of the query-answer pair, the user query is meant to be sent to the advertiser’s model. This can potentially cause privacy issues, especially in the “API access” scenario.\nHow about the cost of serving the advertisers’ models? The cost could be too high to be ignored in the mechanism especially for the cutting-edge LLMs.\n\nLIMITATIONS:\nplease refer above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-13"}