Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching

## Abstract

Abstract Nash Learning from Human Feedback ( NLHF ) is a game-theoretic framework for aligning large language models (LLMs) with human preferences by modeling learning as a two-player zero-sum game.
However, using raw preference as the payoff in the game highly limits the potential of the game-theoretic LLM alignment framework.
In this paper, we systematically study using what choices of payoff based on the pairwise human preferences can yield desirable alignment properties.
We establish necessary and sufficient conditions for Condorcet consistency, diversity through mixed strategies, and Smith consistency.
These results provide a theoretical foundation for the robustness of game-theoretic LLM alignment.
Further, we show the impossibility of preference matching—i.e., no smooth and learnable mappings of pairwise preferences can guarantee a unique Nash equilibrium that matches a target policy, even under standard assumptions like the Bradley-Terry-Luce model.
This result highlights the fundamental limitation of game-theoretic LLM alignment.

## 1 Introduction

Large language models (LLMs) such as OpenAI-o3 (OpenAI, 2025) and DeepSeek-R1 (DeepSeek-AI et al., 2025) have demonstrated impressive capabilities across a wide range of domains, including code generation, data analysis, elementary mathematics, and reasoning (Hurst et al., 2024; Anthropic, 2024; Chowdhery et al., 2023; Touvron et al., 2023; Ji et al., 2025). These models are increasingly being used to tackle previously unsolved mathematical problems, drive scientific and algorithmic discoveries, optimize complex codebases, and support decision-making processes that were once considered unlikely to be automated in the near future (Bubeck et al., 2023; Eloundou et al., 2024; Novikov et al., 2025).

A key factor behind the popularity and effectiveness of LLMs is alignment: the process by which models learn to interact with human users and accommodate diverse human opinions and values by aligning their outputs with human preferences (Christiano et al., 2017).
The traditional method for alignment, reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022; Casper et al., 2023; Dong et al., 2024), typically begins by training a reward model on preference data collected from human labelers, often using the Bradley-Terry-Luce (BTL) model (Bradley and Terry, 1952; Luce, 2012),

$$ $\mathcal{P}(y\succ y^{\prime}\mid x)=\frac{\exp(r(x,y))}{\exp(r(x,y))+\exp(r(x ,y^{\prime}))},$ (1.1) $$

where $r(x,y)$ is the reward function and $\mathcal{P}(y\succ y^{\prime}|x)$ is pairwise human preference, i.e., the fraction of individuals who prefer $y$ over $y^{\prime}$ under prompt $x$.
In this framework, a higher scalar score assigned by the reward model to an LLM-generated response indicates a stronger preference by human labelers.
The LLM is then fine-tuned through maximizing the reward to produce responses that are more likely to align with these preferences.
However, Munos et al. (2024) pointed out that the reward model cannot deal with preferences with cycles, and proposes an alternative alignment approach called Nash learning from human feedback (NLHF).
Unlike the reward-based methods, NLHF directly uses preference data to train a preference model and formulates LLM finetuning as finding Nash equilibrium in a two-player zero-sum game, also known as a von Neumann game (Myerson, 2013).
Specifically, for a given prompt $x$, the LLM’s policy $\bm{\pi}$ competes against an opposing policy $\bm{\pi}^{\prime}$ in a pairwise preference contest, where the objective is to find a policy that maximizes its worst-case preference score. Formally, NLHF solves the following min-max optimization problem:

$$ $\max_{\bm{\pi}}\min_{\bm{\pi}^{\prime}}\mathbb{E}_{x\sim\rho}\left[\mathbb{E}_ {y\sim\bm{\pi}(\cdot\mid x),y^{\prime}\sim\bm{\pi}^{\prime}(\cdot\mid x)}\left [\mathcal{P}\left(y\succ y^{\prime}\mid x\right)\right]\right],$ $$

where $\rho$ is a given distribution over prompts.
However, Munos et al. (2024) did not demonstrate the advantages of using the preference as the payoff in the game.

Recently, criteria from both social choice theory (Conitzer et al., 2024; Dai and Fleisig, 2024; Mishra, 2023) and principles related to diversity (Xiao et al., 2024; Chakraborty et al., 2024) have been increasingly employed to scrutinize the alignment of LLM with human preference.
Notably, RLHF has been shown to fail both social choice theory considerations (Noothigattu et al., 2020; Siththaranjan et al., 2024; Ge et al., 2024; Liu et al., 2025) and diversity considerations (Xiao et al., 2024; Chakraborty et al., 2024).
In contrast, NLHF has been proved to enjoy these desirable properties.
It is shown in Maura-Rivero et al. (2025) and Liu et al. (2025) that NLHF is Condorcet consistent (see Definition [3.2](https://arxiv.org/html/2505.20627v1#S3.Thmothertheorem2)), meaning that the method always outputs the Condorcet winning response, a response that beats every other alternative response in pairwise majority comparisons, whenever one exists. Further, under a no-tie assumption (see Assumption [2.1](https://arxiv.org/html/2505.20627v1#S2.Thmothertheorem1)), Liu et al. (2025) showed that NLHF is Smith consistent (see Definition [4.1](https://arxiv.org/html/2505.20627v1#S4.Thmothertheorem1)), meaning that the method always outputs responses from the Smith set, the smallest nonempty set of responses that pairwise dominate all alternatives outside the set.
Moreover, Liu et al. (2025) showed that when human preference is diverse, i.e., there does not exist a single response that beat every other alternative, NLHF avoids collapsing to a single response by adopting a mixed strategy.

Despite these advantages, there is no reason why we must use raw preference to design the payoff in the game-theoretic alignment approach.
Using alternative payoffs in the game-theoretic LLM alignment framework might also lead to desirable alignment.
In this work, we systematically investigate the fundamental limits of the game-theoretic LLM alignment framework by analyzing how various choices of payoff influence its ability to satisfy key alignment criteria. We consider the following general game-theoretic alignment problem, involving a mapping applied to the preference denoted by $\Psi$:

$$ $\max_{\bm{\pi}}\min_{\bm{\pi}^{\prime}}\mathbb{E}_{x\sim\rho}\left[\mathbb{E}_ {y\sim\bm{\pi}(\cdot\mid x)}\mathbb{E}_{y^{\prime}\sim\bm{\pi}^{\prime}(\cdot \mid x)}\left[\Psi\left(\mathcal{P}(y\succ y^{\prime}\mid x)\right)\right] \right].$ (1.2) $$

The general problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) encompasses a range of games.
When $\Psi(t)=t$ is the identity mapping, the objective in Equation ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) is equivalent to the standard NLHF objective. When $\Psi(t)=\log(t/(1-t))$ and the preference is generated by a BTL model, Equation ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) recovers the standard RLHF objective. More importantly, the preference model $\mathcal{P}_{\theta}$ used in practice is an estimation of the true human preference, which can be regarded as a noisy mapping of the ground-truth preference $\mathcal{P}$. Allowing $\Psi$ to be stochastic provides a way to account for the uncertainty and noise inherent in estimating human preferences.
It is worth noting that a similar formalism has been proposed for non game-theoretic approach in Azar et al. (2024), which used an non-decreasing mapping to process the preference.

In this paper, we first discuss when the solution to problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) is Condorcet consistent and Smith consistent in Section [3](https://arxiv.org/html/2505.20627v1#S3) and [4](https://arxiv.org/html/2505.20627v1#S4) respectively. Our results show that these desirable properties are insensitive to the exact value of the payoff, revealing the robustness of game-theoretic alignment approaches. As a special case, we discover a natural generalization of RLHF objective that satisfy all these desirable properties. Technically, we develop novel proof techniques that can tackle a general non-symmetric game directly, instead of relying crucially on the symmetric nature of NLHF as in Liu et al. (2025).

In addition, we examine the diversity of the solution by investigating whether the model produces a mixed strategy (Liu et al., 2025), and whether its output can satisfy the criterion of preference matching (Xiao et al., 2024), meaning that the model output exactly matches a target policy which fully accounts for the diversity of human preference.
Our findings suggest diversity can be ensured by mixed strategies, but exactly matching a target is difficult for any game-theoretic alignment approach.
This reveals a fundamental limitation of game-theoretic alignment approaches.

### 1.1 Summary of Contributions

We summarize our contributions as follows:

- •
We show that Condorcet consistency is insensitive to the exact value of the payoff (Theorem [3.1](https://arxiv.org/html/2505.20627v1#S3.Thmtheorem1)), revealing the robustness of game-theoretic alignment approaches.
- •
We show that Smith consistency can be ensured by further maintaining the symmetry of the game (Theorem [4.2](https://arxiv.org/html/2505.20627v1#S4.Thmtheorem2)). Moreover, Smith consistent methods automatically preserve the diversity in human preferences by adopting mixed strategies (Corollary [4.2](https://arxiv.org/html/2505.20627v1#S4.Thmothertheorem2)).
- •
We show that preserving the diversity in human preference strictly, in the sense of preference matching, is impossible in general (Theorem [5.1](https://arxiv.org/html/2505.20627v1#S5.Thmtheorem1)). This reveals a fundamental limitation of game-theoretic alignment approaches.

Assuming $\Psi$ is continuous, Table [1](https://arxiv.org/html/2505.20627v1#S1.T1) provides a concise summary of our mathematical results.

**Table 1: Summary of our mathematical results: the necessary and sufficient conditions on $\Psi$ to guarantee certain desirable alignment properties.**
| Condorcet consistency | $\Psi(t)\geqslant\Psi(1/2)\,,\forall\,1/2\leqslant t\leqslant 1$ and $\Psi(t)<\Psi(1/2)\,,\forall\,0\leqslant t<1/2$ |
| --- | --- |
| Mixed & Condorcet consistency | $\Psi(t)+\Psi(1-t)\geqslant 2\Psi(1/2)\,,\forall\,1/2\leqslant t\leqslant 1$ and $\Psi(t)<\Psi(1/2)\,,\forall\,0\leqslant t<1/2$ |
| Smith consistency | $\Psi(t)+\Psi(1-t)=2\Psi(1/2)\,,\forall\,1/2\leqslant t\leqslant 1$ and $\Psi(t)<\Psi(1/2)\,,\forall\,0\leqslant t<1/2$ |

### 1.2 Related Works

A general mapping $\Psi$ was first introduced in Azar et al. (2024) to facilitate the analysis of traditional non game-theoretic LLM alignment methodologies. Their objective function, called $\Psi$PO, applies a general mapping $\Psi$ to the original human preference.
In this way, they were able to treat RLHF and DPO as special cases of $\Psi$PO under BTL model and argue that these methods are prone to overfitting. To avoid overfitting, they took $\Psi$ to be identity and arrived at a new efficient algorithm called IPO.
Our problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) can be regarded as the analogy of $\Psi$PO in the context of game-theoretic LLM alignment.
Another difference is that rather than focusing on statistical properties like overfitting, our focus is on the alignment properties such as Smith consistency and preference matching.
Moreover, they restricted $\Psi$ to be a non-decreasing map, while we allow $\Psi$ to be arbitrary, even stochastic.

Condorcet consistency is one of the dominant concept in the theory of voting (Gehrlein, 2006; Balinski and Laraki, 2010), and Smith consistency is its natural generalization (Shoham and Leyton-Brown, 2008; Börgers, 2010).
They have not been studied in the context of LLM alignment until recently (Maura-Rivero et al., 2025; Liu et al., 2025).
In Maura-Rivero et al. (2025), the authors showed that NLHF with a selection probability that deals with ties is Condorcet consistent. Under a no-tie assumption, Liu et al. (2025) showed that NLHF is Condorcet consistent and Smith consistent, whereas RLHF is not unless the preference satisfies a BTL model. Further, the paper showed that the probability that the preference satisfies a BTL model is vanishing under the impartial culture assumption, highlighting a key advantage of the NLHF framework.

Several recent works also focus on aligning LLMs with the diverse human preference (Chakraborty et al., 2024; Xiao et al., 2024; Liu et al., 2025).
In Chakraborty et al. (2024), the authors introduced a mixture model to account for the opinion of minority group and arrive at the MaxMin-RLHF method. In Xiao et al. (2024), the authors introduced the concept of preference matching and develop the PM-RLHF objective to pursue this goal.
Liu et al. (2025) demonstrated that the original NLHF yields a mixed strategy when no Condorcet winning response exists, whereas standard RLHF produces a deterministic strategy, highlighting a potential advantage of NLHF in preserving the diversity of human preferences.

## 2 Preliminaries

Consider a general mapping $\Psi:[0,1]\to\mathbb{R}$. We apply $\Psi$ to the preference and study the max-min problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) with this generalized payoff.
Any solution $\bm{\pi}$ employed by the first player at the Nash equilibrium,

$$ $\bm{\pi}\in\arg\max_{\bm{\pi}}\min_{\bm{\pi}^{\prime}}\mathbb{E}_{x\sim\rho} \left[\mathbb{E}_{y\sim\bm{\pi}(\cdot\mid x)}\mathbb{E}_{y^{\prime}\sim\bm{\pi }^{\prime}(\cdot\mid x)}\left[\Psi\left(\mathcal{P}(y\succ y^{\prime}\mid x) \right)\right]\right]\,,$ (2.1) $$

is called a Nash solution to the problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)). The Nash solution is the policy which fully aligned LLMs will perform.
Note that the set of Nash solutions remain the same after an overall shift of payoff, that is, changing $\Psi$ to $\Psi+C$ for any constant $C$ will not affect the problem.
The original NLHF objective (Munos et al., 2024) corresponds to the special case where $\Psi(t)=t$, equivalent to $\Psi(t)=t-1/2$, and the resulting game is symmetric (Duersch et al., 2012), meaning that the two players are the same. However, for an arbitrary mapping $\Psi$, the game is usually not symmetric, and we only focus on the Nash solution employed by the first player.

Given a prompt $x$, we consider the set of all possible responses generated by the LLM: $\{y_{1},\ldots,y_{n}\}$, where $n$ is the total number of possible responses.
Without any loss of generality, we drop the dependence on the prompt $x$ from now on. For any two distinct response $y$ and $y^{\prime}$, recall that $\mathcal{P}(y\succ y^{\prime})$ denote the preference of $y$ over $y^{\prime}$, defined as the expected proportion of individuals who prefer $y$ over $y^{\prime}$.
By definition, human preference satisfies the condition $\mathcal{P}(y\succ y^{\prime})+\mathcal{P}(y^{\prime}\succ y)=1$ and naturally we let $\mathcal{P}(y\succ y)=1/2$ (Munos et al., 2024).
For any distinct pair of responses $y$ and $y^{\prime}$, we say that $y$ beats $y^{\prime}$ if $\mathcal{P}(y\succ y^{\prime})>1/2$.
Additionally, following Liu et al. (2025), we adopt the No-Tie assumption throughout this paper.

###### Assumption 2.1 (No-Tie) .

For any distinct responses $y$ and $y^{\prime}$, we assume that $\mathcal{P}(y\succ y^{\prime})\neq 1/2$.

This assumption is both minimal and practically reasonable. First, if the number of labelers is odd, it automatically holds. Even in cases where a tie occurs, it can always be resolved through a more precise comparison.

###### Assumption 2.1 (No-Tie) .

#### Notation.

For any set $A$, we denote its cardinality by $|A|$. For any $n\in\mathbb{N}_{+}$, we define $[n]:=\{1,\ldots,n\}$. We use $\delta_{ij}:=\mathds{1}\{i=j\}$ for $1\leqslant i,j\leqslant n$.
We represent high-dimensional vectors using bold symbols. Any policy $\bm{\pi}$ over the set of possible responses $\{y_{1},\ldots,y_{n}\}$ can be identified with a vector in $\mathbb{R}^{n}$, where each entry $\pi_{i}$ corresponds to the probability assigned to $y_{i}$ for $i\in[n]$.
We then define the support of a policy $\bm{\pi}$ as $\operatorname{supp}(\bm{\pi}):=\{y_{i}\mid\pi_{i}>0\,,i\in[n]\}$. We write $\bm{\pi}>0$ if $\pi_{i}>0$ for all $i\in[n]$, and similarly, $\bm{\pi}\geqslant 0$ if $\pi_{i}\geqslant 0$ for all $i\in[n]$.
For simplicity, we denote $\Psi(\mathcal{P}(y_{i}\succ y_{j}))$ as $\Psi_{ij}$ for any $1\leqslant i,j\leqslant n$, and define the payoff matrix as $\bm{\Psi}:=\{\Psi_{ij}\}_{1\leqslant i,j\leqslant n}$.
We then define the total payoff by:

$$ $\mathcal{P}_{\Psi}(\bm{\pi}_{1},\bm{\pi}_{2}):=\sum_{i=1}^{n}\sum_{j=1}^{n}\pi _{1,i}\pi_{2,j}\Psi_{ij}\,.$ $$

We denote by $\bm{\delta}_{i}$ the policy supported solely on $y_{i}$, i.e., $\operatorname{supp}(\bm{\delta}_{i})=\{y_{i}\}$. Furthermore, the mixed policy $(\bm{\delta}_{i_{1}}+\ldots+\bm{\delta}_{i_{k}})/k$ is defined as the policy $\bm{\pi}$ such that

$$ $\pi_{i}=\begin{cases}1/k\,,&i\in\{i_{1},\cdots,i_{k}\}\\ 0\,,&\text{otherwise}\end{cases}\,,$ $$

for any subset $\{i_{1},\ldots,i_{k}\}\subseteq[n]$.

## 3 Condorcet Consistency

In this section, we examine Condorcet consistency—a desirable property for LLM alignment inspired by social choice theory—within the generalized game-theoretic LLM fine-tuning framework ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)). We begin by defining the Condorcet winning response and Condorcet consistency. We then present Theorem [3.1](https://arxiv.org/html/2505.20627v1#S3.Thmtheorem1), which characterizes the necessary and sufficient conditions on the mapping $\Psi$ to guarantee Condorcet consistency. Next, we examine the conditions under which $\Psi$ preserves human preference diversity when no Condorcet winner exists and introduce Theorem [3.2](https://arxiv.org/html/2505.20627v1#S3.Thmtheorem2). Finally, we discuss the continuity assumption underlying Theorem [3.2](https://arxiv.org/html/2505.20627v1#S3.Thmtheorem2). We provide the proofs of Theorems [3.1](https://arxiv.org/html/2505.20627v1#S3.Thmtheorem1) and [3.2](https://arxiv.org/html/2505.20627v1#S3.Thmtheorem2) in Section [3.1](https://arxiv.org/html/2505.20627v1#S3.SS1) and Section [3.2](https://arxiv.org/html/2505.20627v1#S3.SS2), respectively.

Following Liu et al. (2025), a response that is preferred over all others in pairwise comparisons by the preference model is referred to as the Condorcet winning response.

###### Definition 3.1 (Condorcet Winning Response) .

A response $y^{\star}$ is called a Condorcet winning response if $\mathcal{P}(y^{\star}\succ y)>1/2$ for all $y\neq y^{\star}$.

It is clear that there can be at most one Condorcet winning response. When such a response exists, a natural requirement for LLM alignment is that this response should be the output. This property is known as Condorcet consistency.

###### Definition 3.2 (Condorcet Consistency) .

Problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) is Condorcet consistent if
when there exists a Condorcet winning response, the Nash solution to ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) is unique and corresponds to this Condorcet winning response.

Liu et al. (2025) and Maura-Rivero et al. (2025) showed that the original NLHF objective, which corresponds to the case where $\Psi(\cdot)$ is identity, is Condorcet consistent. In this paper, we proceed further and investigate the following question:

*Which choices of $\Psi$ ensure Condorcet consistency?*

We answer this question in Theorem [3.1](https://arxiv.org/html/2505.20627v1#S3.Thmtheorem1) and the proof is deferred to Section [3.1](https://arxiv.org/html/2505.20627v1#S3.SS1).

###### Theorem 3.1 .

Problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) is Condorcet consistent if and only if $\Psi(\cdot)$ satisfies

$$ $\begin{cases}\Psi(t)\geqslant\Psi(1/2)\,,1\geqslant t\geqslant 1/2\\ \Psi(t)<\Psi(1/2)\,,1/2>t\geqslant 0\end{cases}\,.$ (3.1) $$

Note that this condition is much weaker than requiring $\Psi$ to be increasing. It only demands that $\Psi$ maps any value greater than 1/2 to some value larger than $\Psi(1/2)$, and any value less than $1/2$ to some value smaller than $\Psi(1/2)$. This implies that a wide range of mapping functions can be used within the game-theoretic LLM alignment framework ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) to ensure Condorcet consistency.
Furthermore, in practice, we do not have access to the ground-truth preference model.
Instead, we parameterize the preference model using a deep neural network, $\mathcal{P}_{\theta}(y\succ y^{\prime})$, trained on large-scale preference datasets (Munos et al., 2024). Due to the limitations of the datasets and the optimization process, the learned model only approximates the true human preferences. We can view this approximation as $\Psi(\mathcal{P}(y\succ y^{\prime}))$ in our framework. In practice, we can enforce the parameterized preference model to satisfy $\mathcal{P}_{\theta}(y\succ y)=1/2$, then our results show that as long as this approximation yields the correct pairwise majority comparisons—specifically, that $\mathcal{P}_{\theta}(y\succ y^{\prime})\geqslant 1/2>\mathcal{P}_{\theta}(y^{
\prime}\succ y)$ whenever $y$ beats $y^{\prime}$—then the LLM alignment remains Condorcet consistent. This strongly highlights the robustness of the game-theoretic LLM alignment approach in achieving Condorcet consistency.

When a Condorcet winning response does not exist, human preferences are diverse and there is no single response that is better than others. Therefore, in order to preserve the diversity inherent in human preferences, it is natural to require the Nash solution not to collapse to a single response. This motivation leads to the following characterization of diversity through mixed strategies.

###### Definition 3.3 (Mixed Strategies) .

A Nash solution $\bm{\pi}$ is called a mixed strategy if $|\operatorname{supp}(\bm{\pi})|>1$.

Liu et al. (2025) demonstrated that the original NLHF, which corresponds to the case where $\Psi(\cdot)$ is identity, yields a mixed strategy when no Condorcet winning response exists. Assuming that problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) is Condorcet consistent, we proceed further and investigate:

*Which choices of $\Psi$ lead to a mixed strategy in the absence of a Condorcet winning response?*

We now focus on mappings $\Psi$ that are continuous at 1/2, a condition commonly encountered in practical learning setups. Under this mild assumption, we answer this question in Theorem [3.2](https://arxiv.org/html/2505.20627v1#S3.Thmtheorem2) and provide its proof in Section [3.2](https://arxiv.org/html/2505.20627v1#S3.SS2).

###### Theorem 3.2 .

Assume that the mapping $\Psi(\cdot)$ is continuous at $1/2$.
Assuming the Condorcet consistency of problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)), then any Nash solution is mixed when there is no Condorcet winning response if and only if $\Psi(\cdot)$ satisfies

$$ $\Psi(t)+\Psi(1-t)\geqslant 2\Psi(1/2)\,,\forall\,0\leqslant t\leqslant 1\text{ and }\Psi(t)<\Psi(1/2)\,,\forall\,0\leqslant t<1/2\,.$ (3.2) $$

The first condition arises from the requirement of mixed strategies, while the second condition is a reduction of the condition inherited from Theorem [3.1](https://arxiv.org/html/2505.20627v1#S3.Thmtheorem1) under the assumption of Condorcet consistency and the first condition.

Choices of payoff functions are harder to characterize when we relax the continuity assumption. The following example investigate a special piece-wise constant mapping, which does not satisfy the first condition in Theorem [3.2](https://arxiv.org/html/2505.20627v1#S3.Thmtheorem2).

###### Example 3.4 .

Let $M_{-}<\Psi(1/2)\leqslant M_{+}$ and take

$$ $\Psi(t)=\begin{cases}M_{-}\,,&0\leqslant t<1/2\\ \Psi(1/2)\,,&t=1/2\\ M_{+}\,,&1/2<t\leqslant 1\end{cases}\,.$ $$

Then, any Nash solution is mixed when there is no Condorcet winning response.

###### Proof of Example 3.4 .

We prove this conclusion by contradiction. Suppose that the Nash solution is $\bm{\delta}_{i^{\star}}$ for some $i^{\star}\in[n]$, and the Nash equilibrium is $(\bm{\delta}_{i^{\star}},\bm{\pi}^{\star})$. As there is no Condorcet winning response, by definition, there exists $j^{\prime}$ such that $\mathcal{P}(y_{i^{\star}}\succ y_{j^{\prime}})<1/2$. Then we have

$$ $\mathcal{P}_{\Psi}(\bm{\delta}_{i^{\star}},\bm{\pi}^{\star})=\min_{\bm{\pi}} \mathcal{P}_{\Psi}(\bm{\delta}_{i^{\star}},\bm{\pi})\leqslant\mathcal{P}_{\Psi }(\bm{\delta}_{i^{\star}},\bm{\delta}_{j^{\prime}})=\Psi\left(\mathcal{P}\left (y_{i^{\star}}\succ y_{j^{\prime}}\right)\right)=M_{-}\,.$ (3.3) $$

However, choosing $i^{\prime}$ such that $\pi^{\star}_{i^{\prime}}>0$, we have

$$ $\mathcal{P}_{\Psi}(\bm{\delta}_{i^{\star}},\bm{\pi}^{\star})=\max_{\bm{\pi}} \mathcal{P}_{\Psi}(\bm{\pi},\bm{\pi}^{\star})\geqslant\mathcal{P}_{\Psi}(\bm{ \delta}_{i^{\prime}},\bm{\pi}^{\star})=\sum_{i=1}^{n}\pi_{i}^{\star}\Psi\left( \mathcal{P}\left(y_{i^{\prime}}\succ y_{i}\right)\right)>M_{-}\,,$ $$

which causes a contradiction to ([3.3](https://arxiv.org/html/2505.20627v1#S3.E3)). Hence, we conclude our proof.
∎

This example implies that choices of payoff functions are considerably richer when we relax the continuity assumption.

###### Definition 3.1 (Condorcet Winning Response) .

###### Definition 3.2 (Condorcet Consistency) .

###### Theorem 3.1 .

###### Definition 3.3 (Mixed Strategies) .

###### Theorem 3.2 .

###### Example 3.4 .

###### Proof of Example 3.4 .

### 3.1 Proof of Theorem 3.1

Without any loss of generality, we assume that $y_{1}$ is the Condorcet winning response. First, we show that a necessary condition that ensures the Condorcet consistency of problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) is:

$$ $\begin{cases}\Psi(t)\geqslant\Psi(1/2)\,,1\geqslant t\geqslant 1/2\\ \Psi(t)<\Psi(1/2)\,,0\leqslant t<1/2\end{cases}\,.$ (3.4) $$

To show this, we examine the case where $n=2$. For any $1\geqslant t>1/2$, we consider the game with the payoff in Table [2](https://arxiv.org/html/2505.20627v1#S3.T2). By the definition of Condorcet consistency, all Nash equilibrium of this game is of the form $(\bm{\delta}_{1},\bm{\pi}^{\star})$ for some $\bm{\pi}^{\star}$.

**Table 2: Payoff matrix with two responses $\{y_{1},y_{2}\}$.**
| $\Psi(\mathcal{P}(y\succ y^{\prime}))$ | $y^{\prime}=y_{1}$ | $y^{\prime}=y_{2}$ |
| --- | --- | --- |
| $y=y_{1}$ | $\Psi(1/2)$ | $\Psi(t)$ |
| $y=y_{2}$ | $\Psi(1-t)$ | $\Psi(1/2)$ |

#### Case 1.

If $\Psi(t)>\Psi(1/2)$, we have

$$ $\bm{\pi}^{\star}=\arg\min_{\bm{\pi}}\mathcal{P}_{\Psi}(\bm{\delta}_{1},\bm{\pi })=\arg\min_{\bm{\pi}}\left\{\pi_{1}\Psi(1/2)+\pi_{2}\Psi(t)\right\}=\bm{ \delta}_{1}\,.$ $$

Therefore, we have

$$ $\displaystyle\Psi(1/2)=\mathcal{P}_{\Psi}(\bm{\delta}_{1},\bm{\delta}_{1})= \max_{\bm{\pi}}\mathcal{P}_{\Psi}(\bm{\pi},\bm{\delta}_{1})\geqslant\mathcal{P }_{\Psi}(\bm{\delta}_{2},\bm{\delta}_{1})=\Psi_{21}=\Psi(1-t)\,,$ $\displaystyle\Psi(1/2)=\mathcal{P}_{\Psi}(\bm{\delta}_{1},\bm{\delta}_{1})= \min_{\bm{\pi}}\mathcal{P}_{\Psi}(\bm{\delta}_{1},\bm{\pi})\leqslant\mathcal{P }_{\Psi}(\bm{\delta}_{1},\bm{\delta}_{2})=\Psi_{12}=\Psi(t)\,.$ $$

Hence, we have $\Psi(1-t)\leqslant\Psi(1/2)<\Psi(t)$. If $\Psi(1/2)=\Psi(1-t)$, notice that

$$ $\displaystyle\mathcal{P}_{\Psi}(\bm{\pi},\bm{\delta}_{1})=\pi_{1}\Psi(1/2)+\pi _{2}\Psi(1-t)=\Psi(1/2)\Longrightarrow\bm{\delta}_{2}\in\arg\max_{\bm{\pi}} \mathcal{P}_{\Psi}(\bm{\pi},\bm{\delta}_{1})\,,$ $\displaystyle\mathcal{P}_{\Psi}(\bm{\delta}_{2},\bm{\pi})=\pi_{1}\Psi(1-t)+\pi _{2}\Psi(1/2)=\Psi(1/2)\Longrightarrow\bm{\delta}_{1}\in\arg\min_{\bm{\pi}} \mathcal{P}_{\Psi}(\bm{\delta}_{2},\bm{\pi})\,.$ $$

Therefore, $(\bm{\delta}_{2},\bm{\delta}_{1})$ is also a Nash equilibrium, which causes a contradiction to the fact that problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) is Condorcet consistent. Therefore, we have $\Psi(t)>\Psi(1/2)>\Psi(1-t)$ for any $1\geqslant t>1/2$.

#### Case 2.

If $\Psi(t)<\Psi(1/2)$, we have

$$ $\bm{\pi}^{\star}=\arg\min_{\bm{\pi}}\mathcal{P}_{\Psi}(\bm{\delta}_{1},\bm{\pi })=\arg\min_{\bm{\pi}}\left\{\pi_{1}\Psi(1/2)+\pi_{2}\Psi(t)\right\}=\bm{ \delta}_{2}\,.$ $$

However, notice that

$$ $\Psi(1/2)=\mathcal{P}_{\Psi}(\bm{\delta}_{2},\bm{\delta}_{2})\leqslant\max_{ \bm{\pi}}\mathcal{P}_{\Psi}(\bm{\pi},\bm{\delta}_{2})=\mathcal{P}_{\Psi}(\bm{ \delta}_{1},\bm{\delta}_{2})=\Psi(t)<\Psi(1/2)\,,$ $$

which causes a contradiction.

#### Case 3.

If $\Psi(t)=\Psi(1/2)$. When $\Psi(1-t)=\Psi(1/2)$, any $(\bm{\pi}_{1},\bm{\pi}_{2})$ is a Nash equilibrium, which causes a contradiction to the fact that problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) is Condorcet consistent. When $\Psi(1-t)>\Psi(1/2)$, note that

$$ $\displaystyle\mathcal{P}_{\Psi}(\bm{\delta}_{2},\bm{\pi})=\pi_{1}\Psi(1-t)+\pi _{2}\Psi(1/2)\geqslant\Psi(1/2)\Longrightarrow$ $\displaystyle\ \bm{\delta}_{2}\in\arg\min_{\bm{\pi}}\mathcal{P}_{\Psi}(\bm{ \delta}_{2},\bm{\pi})\,,$ $\displaystyle\mathcal{P}_{\Psi}(\bm{\pi},\bm{\delta}_{2})=\pi_{1}\Psi(t)+\pi_{ 2}\Psi(1/2)=\Psi(1/2)\Longrightarrow$ $\displaystyle\ \bm{\delta}_{2}\in\arg\max_{\bm{\pi}}\mathcal{P}_{\Psi}(\bm{\pi },\bm{\delta}_{2})\,.$ $$

Therefore, $(\bm{\delta}_{2},\bm{\delta}_{2})$ is a Nash equilibrium, which also causes a contradiction to the fact problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) is Condorcet consistent. Hence, we have $\Psi(1-t)<\Psi(1/2)$.

In summary, for any $1\geqslant t>1/2$, we have $\Psi(1-t)<\Psi(1/2)\leqslant\Psi(t)$.
Hence, ([3.4](https://arxiv.org/html/2505.20627v1#S3.E4)) holds if problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) is Condorcet consistent.
Next, we prove that ([3.4](https://arxiv.org/html/2505.20627v1#S3.E4)) is also sufficient for the Condorcet consistency of problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)). Recall that $\Psi_{i1}=\Psi(\mathcal{P}(y_{i}\succ y_{1}))<\Psi(1/2)$, and $\Psi_{1i}=\Psi(\mathcal{P}(y_{1}\succ y_{i}))\geqslant\Psi(1/2)$ for any $i\neq 1$. If $(\bm{\pi}_{1}^{\star},\bm{\pi}_{2}^{\star})$ is a Nash equilibrium. Notice that

$$ $\displaystyle\mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\bm{\pi}_{2}^{\star})= \max_{\bm{\pi}}\mathcal{P}_{\Psi}(\bm{\pi},\bm{\pi}_{2}^{\star})\geqslant \mathcal{P}_{\Psi}(\bm{\delta}_{1},\bm{\pi}_{2}^{\star})=\sum_{i=1}^{n}\pi^{ \star}_{2,i}\Psi_{1i}\,,$ $\displaystyle\mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\bm{\pi}_{2}^{\star})= \min_{\bm{\pi}}\mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\bm{\pi})\leqslant \mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\bm{\delta}_{1})=\sum_{i=1}^{n}\pi_{1, i}^{\star}\Psi_{i1}\,.$ $$

Therefore, if $\bm{\pi}_{1}^{\star}\neq\bm{\delta}_{1}$, we have

$$ $\Psi(1/2)\leqslant\sum_{i=1}^{n}\pi^{\star}_{2,i}\Psi_{1i}\leqslant\mathcal{P} _{\Psi}(\bm{\pi}_{1}^{\star},\bm{\pi}_{2}^{\star})\leqslant\sum_{i=1}^{n}\pi_{ 1,i}^{\star}\Psi_{i1}<\Psi(1/2)\,,$ (3.5) $$

which causes a contradiction. Therefore, $\bm{\pi}_{1}^{\star}=\bm{\delta}_{1}$, i.e., problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) is Condorcet consistent.

### 3.2 Proof of Theorem 3.2

First, according to Theorem [3.1](https://arxiv.org/html/2505.20627v1#S3.Thmtheorem1), when the Nash solution is Condorcet consistent, we have

$$ $\begin{cases}\Psi(t)\geqslant\Psi(1/2)\,,1\geqslant t\geqslant 1/2\\ \Psi(t)<\Psi(1/2)\,,1/2>t\geqslant 0\end{cases}\,.$ (3.6) $$

In addition, we show that $\Psi(\cdot)$ must satisfy $\Psi(t)+\Psi(1-t)\geqslant 2\Psi(1/2)\,,\forall t\in[0,1]$ for ensuring that the Nash solution is mixed when there is no Condorcet winning response. We consider the case where $n=4$ and the game with the payoff in Table [3](https://arxiv.org/html/2505.20627v1#S3.T3) for any $t_{1},t_{2}>1/2$.

**Table 3: Payoff matrix with four responses $\{y_{1},y_{2},y_{3},y_{4}\}$.**
| $\Psi(\mathcal{P}(y\succ y^{\prime}))$ | $y^{\prime}=y_{1}$ | $y^{\prime}=y_{2}$ | $y^{\prime}=y_{3}$ | $y^{\prime}=y_{4}$ |
| --- | --- | --- | --- | --- |
| $y=y_{1}$ | $\Psi(1/2)$ | $\Psi(t_{1})$ | $\Psi(1-t_{1})$ | $\Psi(t_{2})$ |
| $y=y_{2}$ | $\Psi(1-t_{1})$ | $\Psi(1/2)$ | $\Psi(t_{1})$ | $\Psi(t_{2})$ |
| $y=y_{3}$ | $\Psi(t_{1})$ | $\Psi(1-t_{1})$ | $\Psi(1/2)$ | $\Psi(t_{2})$ |
| $y=y_{4}$ | $\Psi(1-t_{2})$ | $\Psi(1-t_{2})$ | $\Psi(1-t_{2})$ | $\Psi(1/2)$ |

Notice that if $\Psi(t_{1})+\Psi(1-t_{1})+\Psi(1/2)\leqslant 3\Psi(1-t_{2})$, we have

$$ $\mathcal{P}_{\Psi}(\bm{\delta}_{4},\bm{\pi})=(\pi_{1}+\pi_{2}+\pi_{3})\Psi(1-t _{2})+\pi_{4}\Psi(1/2)\Longrightarrow\frac{\bm{\delta}_{1}+\bm{\delta}_{2}+\bm {\delta}_{3}}{3}\in\arg\min_{\bm{\pi}}\mathcal{P}_{\Psi}(\bm{\delta}_{4},\bm{ \pi})\,,$ $$

and

$$ $\displaystyle\mathcal{P}_{\Psi}\left(\bm{\pi},\frac{\bm{\delta}_{1}+\bm{\delta }_{2}+\bm{\delta}_{3}}{3}\right)=$ $\displaystyle\ (\pi_{1}+\pi_{2}+\pi_{3})\cdot\frac{\Psi(1/2)+\Psi(t_{1})+\Psi( 1-t_{1})}{3}+\pi_{4}\Psi(1-t_{2})$ $\displaystyle\ \Longrightarrow\bm{\delta}_{4}\in\arg\max_{\bm{\pi}}\mathcal{P} _{\Psi}\left(\bm{\pi},\frac{\bm{\delta}_{1}+\bm{\delta}_{2}+\bm{\delta}_{3}}{3 }\right)\,.$ $$

Therefore, $(\bm{\delta}_{4},(\bm{\delta}_{1}+\bm{\delta}_{2}+\bm{\delta}_{3})/3)$ is a Nash equilibrium, which causes a contradiction to the fact that the Nash solution is mixed. Hence, we have $\Psi(t_{1})+\Psi(1-t_{1})+\Psi(1/2)>3\Psi(1-t_{2})$ for any $t_{1},t_{2}>1/2$. Let $t_{2}\to 1/2$, we have $\Psi(t)+\Psi(1-t)\geqslant 2\Psi(1/2)$ for any $t\in[0,1]$. Hence, combining ([3.6](https://arxiv.org/html/2505.20627v1#S3.E6)), we have shown that the necessary condition for ensuring that the Nash solution is mixed is:

$$ $\Psi(t)+\Psi(1-t)\geqslant 2\Psi(1/2)\,,\forall t\in[0,1]\text{ and }\begin{ cases}\Psi(t)\geqslant\Psi(1/2)\,,1\geqslant t\geqslant 1/2\\ \Psi(t)<\Psi(1/2)\,,1/2>t\geqslant 0\end{cases}\,.$ (3.7) $$

Next, we prove that the condition ([3.7](https://arxiv.org/html/2505.20627v1#S3.E7)) is also sufficient. Suppose that $(\bm{\delta}_{i^{\star}},\bm{\pi}^{\star})$ is a Nash equilibrium, then we have

$$ $\displaystyle\mathcal{P}_{\Psi}(\bm{\delta}_{i^{\star}},\bm{\pi}^{\star})=\max _{\bm{\pi}}\mathcal{P}_{\Psi}(\bm{\pi},\bm{\pi}^{\star})\geqslant$ $\displaystyle\ \mathcal{P}_{\Psi}(\bm{\pi}^{\star},\bm{\pi}^{\star})$ $\displaystyle=$ $\displaystyle\ \sum_{i=1}^{n}\sum_{j=1}^{n}\pi^{\star}_{i}\pi^{\star}_{j}\Psi_ {ij}=\frac{1}{2}\sum_{i=1}^{n}\sum_{j=1}^{n}\pi_{i}^{\star}\pi_{j}^{\star} \left(\Psi_{ij}+\Psi_{ji}\right)\geqslant\Psi(1/2)\,.$ $$

However, notice that for any $j$, we have

$$ $\mathcal{P}_{\Psi}(\bm{\delta}_{i^{\star}},\bm{\pi}^{\star})=\min_{\bm{\pi}} \mathcal{P}_{\Psi}(\bm{\delta}_{i^{\star}},\bm{\pi})\leqslant\mathcal{P}_{\Psi }(\bm{\delta}_{i^{\star}},\bm{\delta}_{j})=\Psi_{i^{\star}j}\,.$ $$

As there is no Condorcet winning response, there must exist $j^{\star}$ such that $\mathcal{P}(y_{i^{\star}}\succ y_{j^{\star}})<1/2$, thus $\Psi_{i^{\star}j^{\star}}<\Psi(1/2)$. Hence, $\Psi(1/2)\leqslant\mathcal{P}_{\Psi}(\bm{\delta}_{i^{\star}},\bm{\pi}^{\star})
\leqslant\Psi_{i^{\star}j^{\star}}<\Psi(1/2)$, which causes a contradiction. Therefore, the Nash solution must be mixed.

## 4 Smith Consistency

In this section, we extend the discussion of Condorcet consistency to Smith consistency. First, we define the Smith set and Smith consistency. Next, we present Theorem [4.2](https://arxiv.org/html/2505.20627v1#S4.Thmtheorem2), which provides the necessary and sufficient condition for the mapping $\Psi$ to ensure Smith consistency. Finally, we highlight that Smith-consistent methods inherently preserve the diversity present in human preferences and discuss the continuity assumption in Theorem [4.2](https://arxiv.org/html/2505.20627v1#S4.Thmtheorem2). We also discuss the continuity assumption used in Theorem [4.2](https://arxiv.org/html/2505.20627v1#S4.Thmtheorem2) through Example [4.3](https://arxiv.org/html/2505.20627v1#S4.Thmothertheorem3). The proofs of Theorem [4.2](https://arxiv.org/html/2505.20627v1#S4.Thmtheorem2) and Example [4.3](https://arxiv.org/html/2505.20627v1#S4.Thmothertheorem3) are provided in Section [4.1](https://arxiv.org/html/2505.20627v1#S4.SS1) and Section [4.2](https://arxiv.org/html/2505.20627v1#S4.SS2), respectively.

Condorcet consistency only ensures the method capture the right response when there exists a Condorcet winning response. In general, when there is no Condorcet winning response, we can expect that there might be a set of responses satisfying similar property, generalizing Definition [3.1](https://arxiv.org/html/2505.20627v1#S3.Thmothertheorem1).
Under Assumption [2.1](https://arxiv.org/html/2505.20627v1#S2.Thmothertheorem1),
Liu et al. (2025) revealed a more detailed decomposition of the preference structure.
Specifically, the set of responses can be partitioned into distinct groups $S_{1},\ldots,S_{k}$, where every response in $S_{i}$ is preferred over all responses in $S_{j}$ for $i<j$, summarized in the following theorem.

###### Theorem 4.1 ( Liu et al. ( 2025 ) ) .

Under Assumption [2.1](https://arxiv.org/html/2505.20627v1#S2.Thmothertheorem1), the set of responses can be partitioned into disjoint subsets $S_{1},\ldots,S_{k}$ such that:

- 1.
Each $S_{i}$ either forms a Condorcet cycle or is a single response.
- 2.
For any $j>i$, any response $y\in S_{i}$ and $y^{\prime}\in S_{j}$, $\mathcal{P}(y\succ y^{\prime})>\frac{1}{2}$.

Moreover, this decomposition is unique.

When $|S_{1}|=1$, the response in $S_{1}$ is exactly the Condorcet winning response.
Thus, $S_{1}$ is the generalization of Condorcet winning response, and is referred as the Condorcet winning set in Liu et al. (2025).
Traditionally, a subset with such property is also known as the Smith set in the literature of social choice theory (Shoham and Leyton-Brown, 2008).
Here we choose to adopt the name Smith set to distinguish with the concept of Condorcet winning response.
Given this decomposition, it is natural to desire that an aligned LLM adopts a strategy supported exclusively on the top group $S_{1}$, as any response outside $S_{1}$ is strictly less preferred than any response inside $S_{1}$.
This desirable property is referred to as Smith consistency:

###### Definition 4.1 (Smith Consistency) .

Problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) is Smith consistent if the support of any Nash solution is contained in the Smith set $S_{1}$.

Liu et al. (2025) showed that the original NLHF payoff, which corresponds to the case where $\Psi(t)=t$, is Smith consistent. Here, we investigate this question for a general mapping $\Psi$:

*Which choices of $\Psi$ ensure Smith consistency?*

Here, similar to Theorem [3.2](https://arxiv.org/html/2505.20627v1#S3.Thmtheorem2), we answer this question in Theorem [4.2](https://arxiv.org/html/2505.20627v1#S4.Thmtheorem2) for mappings that is continuous at $1/2$. The proof is provided in Section [4.1](https://arxiv.org/html/2505.20627v1#S4.SS1).

###### Theorem 4.2 .

Suppose that the mapping $\Psi(\cdot)$ is continuous at $1/2$, problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) is Smith consistent if and only if $\Psi(\cdot)$ satisfies

$$ $\Psi(t)+\Psi(1-t)=2\Psi(1/2)\,,\forall\,t\in[0,1]\text{ and }\Psi(t)<\Psi(1/2) \,,\forall\,0\leqslant t<1/2\,.$ $$

The first condition $\Psi(t)+\Psi(1-t)=2\Psi(1/2)$ says nothing but the zero-sum game formed by problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) is equivalent to a symmetric two-player zero-sum game(^1^11This can be seen by shifting the payoff by $\Psi(1/2)$, which leaves the Nash solution unchanged.) (Duersch et al., 2012).
By definition, Smith consistency implies Condorcet consistency because when there is a Condorcet winning response, $S_{1}$ is exactly the set whose only element is the Condorcet winning response. Thus, the second condition is just a reduction of the condition in Theorem [3.1](https://arxiv.org/html/2505.20627v1#S3.Thmtheorem1) under the first condition.
It is easy to see $\Psi(t)=t$ satisfies these conditions, and thus our result generalize Theorem 3.6 in Liu et al. (2025).
More interestingly, $\Psi(t)=\log(t/(1-t))$ also satisfies these conditions.
This implies that

$$ $\max_{\bm{\pi}}\min_{\bm{\pi}^{\prime}}\mathbb{E}_{x\sim\rho}\left[\mathbb{E}_ {y\sim\bm{\pi}(\cdot\mid x)}\mathbb{E}_{y^{\prime}\sim\bm{\pi}^{\prime}(\cdot \mid x)}\left[\log\left(\frac{\mathcal{P}(y\succ y^{\prime}\mid x)}{\mathcal{P }(y^{\prime}\succ y\mid x)}\right)\right]\right],$ $$

which is a natural generalization of standard RLHF when human preferences does not satisfy BTL model, is also Smith consistent.

The set of choices for $\Psi$ that ensure Smith consistency is quite broad. We can easily construct such a $\Psi$ by first defining $\Psi(t)$ on $[0,1/2]$ to satisfy $\Psi(t)<\Psi(1/2)$ for all $t\in[0,1/2)$, and then extending it to $[0,1]$ by setting $\Psi(t)=2\Psi(1/2)-\Psi(1-t)$ for all $t\in(1/2,1]$.
Moreover, as discussed in Section [3](https://arxiv.org/html/2505.20627v1#S3), a practical preference model $\mathcal{P}_{\theta}(y\succ y^{\prime})$ can be seen as a mapping of the ground truth preference via $\Psi$, i.e., $\Psi(\mathcal{P}(y\succ y^{\prime}))$. Thus, the first condition in Theorem [4.2](https://arxiv.org/html/2505.20627v1#S4.Thmtheorem2) requires the preference model to satisfy $\mathcal{P}_{\theta}(y\succ y^{\prime})+\mathcal{P}_{\theta}(y^{\prime}\succ y
)=1$, with $\mathcal{P}_{\theta}(y\succ y)=1/2$ enforced.
However, several practically used preference models (Munos et al., 2024; Jiang et al., 2023; Wu et al., 2024) do not guarantee this condition, which may cause the aligned LLM strategy to fail to satisfy Smith consistency.

As any mapping satisfying the condition in Theorem [4.2](https://arxiv.org/html/2505.20627v1#S4.Thmtheorem2) also satisfies the condition in Theorem [3.2](https://arxiv.org/html/2505.20627v1#S3.Thmtheorem2), we obtain the following corollary:

###### Corollary 4.2 .

Suppose that the mapping $\Psi(\cdot)$ is continuous at $1/2$. Then if problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) is Smith consistent, any Nash solution is also mixed.

This shows that when $|S_{1}|>1$, the Nash solution to problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) with any $\Psi$ such that Smith consistency holds will not only support on $S_{1}$ but also be a mixed strategy on $S_{1}$ without collapsing to a single response. As a conclusion, a Smith consistent method can preserve the diversity inherent in human preferences, at least partially.

Lastly, we discuss what happens if $\Psi$ is not continuous at $1/2$. Choices of mappings $\Psi$ are considerably richer and consequently harder to characterize when we relax the continuity assumption.
The following example shows that the piece-wise constant mapping in Example [3.4](https://arxiv.org/html/2505.20627v1#S3.Thmothertheorem4) also ensures Smith consistency.

###### Example 4.3 .

Let $M_{-}<\Psi(\frac{1}{2})<M_{+}$, and we take

$$ $\Psi(t)=\begin{cases}M_{-}&0\leqslant t<1/2\\ \Psi(1/2)&t=1/2\\ M_{+}&1/2<t\leqslant 1\end{cases}\,.$ $$

Then problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) is Smith consistent. The proof is provided in Section [4.2](https://arxiv.org/html/2505.20627v1#S4.SS2).

###### Theorem 4.1 ( Liu et al. ( 2025 ) ) .

###### Definition 4.1 (Smith Consistency) .

###### Theorem 4.2 .

###### Corollary 4.2 .

###### Example 4.3 .

### 4.1 Proof of Theorem 4.2

First, we show that the necessary condition for ensuring that problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) is Smith consistent is:

$$ $\Psi(t)+\Psi(1-t)=2\Psi(1/2)\,,\forall t\in[0,1]\text{ and }\begin{cases}\Psi( t)\geqslant\Psi(1/2)\,,1\geqslant t\geqslant 1/2\\ \Psi(t)<\Psi(1/2)\,,0\leqslant t<1/2\end{cases}\,.$ (4.1) $$

First, Condorcet consistency must hold when Smith consistency holds. According to Theorem [3.1](https://arxiv.org/html/2505.20627v1#S3.Thmtheorem1), we have

$$ $\begin{cases}\Psi(t)\geqslant\Psi(1/2)\,,1\geqslant t\geqslant 1/2\\ \Psi(t)<\Psi(1/2)\,,0\leqslant t<1/2\end{cases}$ $$

Next, we show that when $\Psi(\cdot)$ is continuous at $1/2$, $\Psi(\cdot)$ must satisfy $\Psi(t)+\Psi(1-t)\geqslant 2\Psi(1/2)$ (Lemma [4.4](https://arxiv.org/html/2505.20627v1#S4.Thmothertheorem4)) and $\Psi(t)+\Psi(1-t)\leqslant 2\Psi(1/2)$ (Lemma [4.5](https://arxiv.org/html/2505.20627v1#S4.Thmothertheorem5)) for any $t\in[0,1]$. Therefore, combining the two results together, we obtain the condition ([4.1](https://arxiv.org/html/2505.20627v1#S4.E1)).

###### Lemma 4.4 .

When $\Psi(\cdot)$ is continuous at $1/2$. Achieving Smith consistency only if

$$ $\Psi(t)+\Psi(1-t)\geqslant 2\Psi(1/2)\,,\forall t\in[0,1]\,.$ $$

###### Proof of Lemma 4.4 .

We consider the case where $n=4$ and the game with the payoff in Table [4](https://arxiv.org/html/2505.20627v1#S4.T4) for any $t_{1},t_{2}>1/2$. Notice that if $\Psi(t_{1})+\Psi(1-t_{1})+\Psi(1/2)\leqslant 3\Psi(1-t_{2})$, we have

$$ $\mathcal{P}_{\Psi}(\bm{\delta}_{4},\bm{\pi})=(\pi_{1}+\pi_{2}+\pi_{3})\Psi(1-t _{2})+\pi_{4}\Psi(1/2)\Longrightarrow\frac{\bm{\delta}_{1}+\bm{\delta}_{2}+\bm {\delta}_{3}}{3}\in\arg\min_{\bm{\pi}}\mathcal{P}_{\Psi}(\bm{\delta}_{4},\bm{ \pi})\,,$ $$

and

$$ $\displaystyle\mathcal{P}_{\Psi}\left(\bm{\pi},\frac{\bm{\delta}_{1}+\bm{\delta }_{2}+\bm{\delta}_{3}}{3}\right)=$ $\displaystyle\ (\pi_{1}+\pi_{2}+\pi_{3})\cdot\frac{\Psi(1/2)+\Psi(t_{1})+\Psi( 1-t_{1})}{3}+\pi_{4}\Psi(1-t_{2})$ $\displaystyle\ \Longrightarrow\bm{\delta}_{4}\in\arg\max_{\bm{\pi}}\mathcal{P} _{\Psi}\left(\bm{\pi},\frac{\bm{\delta}_{1}+\bm{\delta}_{2}+\bm{\delta}_{3}}{3 }\right)\,.$ $$

Therefore, $(\bm{\delta}_{4},(\bm{\delta}_{1}+\bm{\delta}_{2}+\bm{\delta}_{3})/3)$ is a Nash equilibrium, which causes a contradiction to the fact that the Nash solution supports on $S_{1}:=\{y_{1},y_{2},y_{3}\}$. Hence, we have $\Psi(t_{1})+\Psi(1-t_{1})+\Psi(1/2)>3\Psi(1-t_{2})$ for any $t_{1},t_{2}>1/2$. Let $t_{2}\to 1/2$, we have $\Psi(t)+\Psi(1-t)\geqslant 2\Psi(1/2)$ for any $t\in[0,1]$.

**Table 4: Payoff matrix with four responses $\{y_{1},y_{2},y_{3},y_{4}\}$.**
| $\Psi(\mathcal{P}(y\succ y^{\prime}))$ | $y^{\prime}=y_{1}$ | $y^{\prime}=y_{2}$ | $y^{\prime}=y_{3}$ | $y^{\prime}=y_{4}$ |
| --- | --- | --- | --- | --- |
| $y=y_{1}$ | $\Psi(1/2)$ | $\Psi(t_{1})$ | $\Psi(1-t_{1})$ | $\Psi(t_{2})$ |
| $y=y_{2}$ | $\Psi(1-t_{1})$ | $\Psi(1/2)$ | $\Psi(t_{1})$ | $\Psi(t_{2})$ |
| $y=y_{3}$ | $\Psi(t_{1})$ | $\Psi(1-t_{1})$ | $\Psi(1/2)$ | $\Psi(t_{2})$ |
| $y=y_{4}$ | $\Psi(1-t_{2})$ | $\Psi(1-t_{2})$ | $\Psi(1-t_{2})$ | $\Psi(1/2)$ |

∎

###### Lemma 4.5 .

When $\Psi(\cdot)$ is continuous at $1/2$. Achieving Smith consistency only if

$$ $\Psi(t)+\Psi(1-t)\leqslant 2\Psi(1/2)\,,\forall t\in[0,1]\,.$ $$

###### Proof of Lemma 4.5 .

We consider the case where $n=6$ and the game with the payoff in Table [5](https://arxiv.org/html/2505.20627v1#S4.T5) for any $t_{1},t_{2}>1/2$.
Notice that if $\Psi(t_{1})+\Psi(1/2)+\Psi(1-t_{1})>3\Psi(t_{2})(\geqslant 3\Psi(1/2)>3\Psi(1-
t_{2}))$, there exists positive $\bm{\mu}=(\mu_{1}/3,\mu_{1}/3,\mu_{1}/3,\mu_{2}/3,\mu_{2}/3,\mu_{2}/3)$ and $\bm{\mu}^{\prime}=(\mu_{1}^{\prime}/3,\mu_{1}^{\prime}/3,\mu_{1}^{\prime}/3\mu
_{2}^{\prime}/3,\mu_{2}^{\prime}/3,\mu_{2}^{\prime}/3)$ such that $\mu_{1}+\mu_{2}=\mu^{\prime}_{1}+\mu^{\prime}_{2}=1$, and

$$ $\displaystyle\mu_{1}\left[\Psi(1/2)+\Psi(t_{1})+\Psi(1-t_{1})-3\Psi(t_{2}) \right]=\mu_{2}\left[\Psi(1/2)+\Psi(t_{1})+\Psi(1-t_{1})-3\Psi(1-t_{2})\right]\,,$ $\displaystyle\mu^{\prime}_{1}\left[\Psi(1/2)+\Psi(t_{1})+\Psi(1-t_{1})-3\Psi(1 -t_{2})\right]=\mu^{\prime}_{2}\left[\Psi(1/2)+\Psi(t_{1})+\Psi(1-t_{1})-3\Psi (t_{2})\right]\,.$ $$

Hence, we have

$$ $\displaystyle\ \mu_{1}(\Psi(1/2)+\Psi(t_{1})+\Psi(1-t_{1}))+3\mu_{2}\Psi(1-t_{ 2})$ $\displaystyle=$ $\displaystyle\ \mu_{2}(\Psi(1/2)+\Psi(t_{1})+\Psi(1-t_{1}))+3\mu_{1}\Psi(t_{2} ):=3A\,,$ $\displaystyle\ \mu^{\prime}_{1}(\Psi(1/2)+\Psi(t_{1})+\Psi(1-t_{1}))+3\mu^{ \prime}_{2}\Psi(t_{2})$ $\displaystyle=$ $\displaystyle\ \mu^{\prime}_{2}(\Psi(1/2)+\Psi(t_{1})+\Psi(1-t_{1}))+3\mu^{ \prime}_{1}\Psi(1-t_{2}):=3B\,.$ $$

Thus, we have

$$ $\displaystyle\mathcal{P}_{\Psi}(\bm{\pi},\bm{\mu}^{\prime})=$ $\displaystyle\ \left(\pi_{1}+\pi_{2}+\pi_{3}\right)\left[\frac{\mu_{1}^{\prime }}{3}\left(\Psi(1/2)+\Psi(t_{1})+\Psi(1-t_{1})\right)+\mu_{2}^{\prime}\Psi(t_{ 2})\right]$ $\displaystyle\ \,+\left(\pi_{4}+\pi_{5}+\pi_{6}\right)\left[\mu_{1}^{\prime} \Psi(1-t_{2})+\frac{\mu_{2}^{\prime}}{3}\left(\Psi(1/2)+\Psi(t_{1})+\Psi(1-t_{ 1})\right)\right]=B\,,$ $$

and

$$ $\displaystyle\mathcal{P}_{\Psi}(\bm{\mu},\bm{\pi})=$ $\displaystyle\ \left(\pi_{1}+\pi_{2}+\pi_{3}\right)\left[\frac{\mu_{1}}{3} \left(\Psi(1/2)+\Psi(t_{1})+\Psi(1-t_{1})\right)+\mu_{2}\Psi(1-t_{2})\right]$ $\displaystyle\ \,+\left(\pi_{4}+\pi_{5}+\pi_{6}\right)\left[\mu_{1}\Psi(t_{2}) +\frac{\mu_{2}}{3}\left(\Psi(1/2)+\Psi(t_{1})+\Psi(1-t_{1})\right)\right]=A\,.$ $$

Therefore, $\bm{\mu}\in\arg\max_{\bm{\pi}}\mathcal{P}_{\Psi}(\bm{\pi},\bm{\mu}^{\prime})\,
,\,\bm{\mu}^{\prime}\in\arg\min_{\bm{\pi}}\mathcal{P}_{\Psi}(\bm{\mu},\bm{\pi})$, which provides that $(\bm{\mu},\bm{\mu}^{\prime})$ is a Nash equilibrium. However, this causes a contradiction to the fact that the Nash solution supports on $S_{1}:=\{y_{1},y_{2},y_{3}\}$. Thus, it must hold that $\Psi(t_{1})+\Psi(1/2)+\Psi(1-t_{1})\leqslant 3\Psi(t_{2})$ for any $t_{1},t_{2}>1/2$. Let $t_{2}\to 1/2$, we obtain $\Psi(t)+\Psi(1-t)\leqslant 2\Psi(1/2)$ for any $t\in[0,1]$.

**Table 5: Payoff matrix with six responses $\{y_{1},y_{2},y_{3},y_{4},y_{5},y_{6}\}$.**
| $\Psi(\mathcal{P}(y\succ y^{\prime}))$ | $y^{\prime}=y_{1}$ | $y^{\prime}=y_{2}$ | $y^{\prime}=y_{3}$ | $y^{\prime}=y_{4}$ | $y^{\prime}=y_{5}$ | $y^{\prime}=y_{6}$ |
| --- | --- | --- | --- | --- | --- | --- |
| $y=y_{1}$ | $\Psi(1/2)$ | $\Psi(t_{1})$ | $\Psi(1-t_{1})$ | $\Psi(t_{2})$ | $\Psi(t_{2})$ | $\Psi(t_{2})$ |
| $y=y_{2}$ | $\Psi(1-t_{1})$ | $\Psi(1/2)$ | $\Psi(t_{1})$ | $\Psi(t_{2})$ | $\Psi(t_{2})$ | $\Psi(t_{2})$ |
| $y=y_{3}$ | $\Psi(t_{1})$ | $\Psi(1-t_{1})$ | $\Psi(1/2)$ | $\Psi(t_{2})$ | $\Psi(t_{2})$ | $\Psi(t_{2})$ |
| $y=y_{4}$ | $\Psi(1-t_{2})$ | $\Psi(1-t_{2})$ | $\Psi(1-t_{2})$ | $\Psi(1/2)$ | $\Psi(t_{1})$ | $\Psi(1-t_{1})$ |
| $y=y_{5}$ | $\Psi(1-t_{2})$ | $\Psi(1-t_{2})$ | $\Psi(1-t_{2})$ | $\Psi(1-t_{1})$ | $\Psi(1/2)$ | $\Psi(t_{1})$ |
| $y=y_{6}$ | $\Psi(1-t_{2})$ | $\Psi(1-t_{2})$ | $\Psi(1-t_{2})$ | $\Psi(t_{1})$ | $\Psi(1-t_{1})$ | $\Psi(1/2)$ |

∎

Finally, we prove that the condition ([4.1](https://arxiv.org/html/2505.20627v1#S4.E1)) is also sufficient for Smith consistency. Suppose that $(\bm{\pi}_{1}^{\star},\bm{\pi}_{2}^{\star})$ is a Nash equilibrium, notice that

$$ $\displaystyle\mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\bm{\pi}_{2}^{\star})= \max_{\bm{\pi}}\mathcal{P}_{\Psi}(\bm{\pi},\bm{\pi}_{2}^{\star})\geqslant \mathcal{P}_{\Psi}(\bm{\pi}_{2}^{\star},\bm{\pi}_{2}^{\star})=\Psi(1/2)\,,$ (4.2) $\displaystyle\mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\bm{\pi}_{2}^{\star})= \min_{\bm{\pi}}\mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\bm{\pi})\leqslant \mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\bm{\pi}_{1}^{\star})=\Psi(1/2)\,,$ $$

which follows from the following fact: for any $\bm{\pi}$,

$$ $\mathcal{P}_{\Psi}(\bm{\pi},\bm{\pi})=\sum_{i=1}^{n}\sum_{j=1}^{n}\pi_{i}\pi_{ j}\Psi_{ij}=\frac{1}{2}\sum_{i=1}^{n}\sum_{j=1}^{n}\pi_{i}\pi_{j}\left(\Psi_{ ij}+\Psi_{ji}\right)=\Psi(1/2)\sum_{i=1}^{n}\sum_{j=1}^{n}\pi_{i}\pi_{j}=\Psi( 1/2)\,.$ $$

Thus, from ([4.2](https://arxiv.org/html/2505.20627v1#S4.E2)), we have $\mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\bm{\pi}_{2}^{\star})=\Psi(1/2)$. Then we prove $\operatorname{supp}(\bm{\pi}_{1}^{\star})\subseteq S_{1}$. Hence, the Nash solution is Smith consistent, i.e., only supports on $S_{1}$.

###### Lemma 4.4 .

###### Proof of Lemma 4.4 .

###### Lemma 4.5 .

###### Proof of Lemma 4.5 .

#### Case 1.

If $\operatorname{supp}(\bm{\pi}_{1}^{\star})\bigcap S_{1}=\emptyset$, taking any $j\in S_{1}$, we have

$$ $\mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\bm{\pi}_{2}^{\star})=\min_{\bm{\pi}} \mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\bm{\pi})\leqslant\mathcal{P}_{\Psi}( \bm{\pi}_{1}^{\star},\bm{\delta}_{j})=\sum_{i=1}^{n}\pi_{1,i}^{\star}\Psi_{ij} =\sum_{i\in S_{1}^{c}}\pi_{1,i}^{\star}\Psi_{ij}<\Psi(1/2)\,,$ $$

which causes a contradiction to the fact that $\mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\bm{\pi}_{2}^{\star})=\Psi(1/2)$.

#### Case 2.

If $\operatorname{supp}(\bm{\pi}_{1}^{\star})\bigcap S_{1}\neq\emptyset$, and $\operatorname{supp}(\bm{\pi}_{1}^{\star})\bigcap S_{1}^{c}\neq\emptyset$, taking $\widetilde{\bm{\pi}}_{2}^{\star}$ as:

$$ $\widetilde{\pi}^{\star}_{2,j}=\mathds{1}\left\{j\in S_{1}\right\}\cdot\frac{ \pi^{\star}_{1,j}}{\sum_{j\in S_{1}}\pi^{\star}_{1,j}}\,.$ $$

Then we have

$$ $\displaystyle\mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\bm{\pi}_{2}^{\star})=$ $\displaystyle\ \min_{\bm{\pi}}\mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\bm{\pi} )\leqslant\mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\widetilde{\bm{\pi}}^{\star} _{2})$ (4.3) $\displaystyle=$ $\displaystyle\ \sum_{i\in S_{1}}\sum_{j\in S_{1}}\pi^{\star}_{1,i}\widetilde{ \pi}^{\star}_{2,j}\Psi_{ij}+\sum_{i\in S_{1}^{c}}\sum_{j\in S_{1}}\pi^{\star}_ {1,i}\widetilde{\pi}^{\star}_{2,j}\Psi_{ij}$ $\displaystyle<$ $\displaystyle\ \frac{\sum_{i\in S_{1}}\sum_{j\in S_{1}}\pi^{\star}_{1,i}\pi^{ \star}_{1,j}\Psi_{ij}}{\sum_{j\in S_{1}}\pi^{\star}_{1,j}}+\Psi(1/2)\sum_{i\in S _{1}^{c}}\sum_{j\in S_{1}}\pi^{\star}_{1,i}\widetilde{\pi}^{\star}_{2,j}$ $\displaystyle=$ $\displaystyle\ \Psi(1/2)\sum_{i\in S_{1}}\pi_{1,i}^{\star}+\Psi(1/2)\sum_{i\in S _{1}^{c}}\pi^{\star}_{1,i}=\Psi(1/2)\,,$ $$

which follows from the following fact:

$$ $\displaystyle\sum_{i\in S_{1}}\sum_{j\in S_{1}}\pi_{1,i}^{\star}\pi_{1,j}^{ \star}\Psi_{ij}=$ $\displaystyle\ \frac{1}{2}\sum_{i\in S_{1}}\sum_{j\in S_{1}}\pi_{1,i}^{\star} \pi_{1,j}^{\star}\left(\Psi_{ij}+\Psi_{ji}\right)$ $\displaystyle=$ $\displaystyle\ \Psi(1/2)\sum_{i\in S_{1}}\sum_{j\in S_{1}}\pi_{1,i}^{\star}\pi _{1,j}^{\star}=\Psi(1/2)\left(\sum_{i\in S_{1}}\pi_{1,i}^{\star}\right)\left( \sum_{j\in S_{1}}\pi_{1,j}^{\star}\right)$ $$

However, ([4.3](https://arxiv.org/html/2505.20627v1#S4.E3)) also causes a contradiction to the fact that $\mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\bm{\pi}_{2}^{\star})=\Psi(1/2)$.

Therefore, it must hold that $\operatorname{supp}(\bm{\pi}_{1}^{\star})\bigcap S_{1}^{c}=\emptyset$, i.e., $\operatorname{supp}(\bm{\pi}_{1}^{\star})\subseteq S_{1}$.

### 4.2 Proof of Example 4.3

We prove this conclusion by contradiction. Suppose that the Nash solution is $\bm{\pi}_{1}^{\star}$ that satisfies $\operatorname{supp}(\bm{\pi}_{1}^{\star})\cap S_{1}^{c}\neq\emptyset$, and the Nash equilibrium is $(\bm{\pi}_{1}^{\star},\bm{\pi}_{2}^{\star})$.

#### Case 1.

If $\operatorname{supp}(\bm{\pi}_{1}^{\star})\cap S_{1}=\emptyset$, taking $j^{\prime}\in S_{1}$, we have

$$ $\mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\bm{\pi}_{2}^{\star})=\min_{\bm{\pi}} \mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\bm{\pi})\leqslant\mathcal{P}_{\Psi}( \bm{\pi}_{1}^{\star},\bm{\delta}_{j^{\prime}})=\sum_{i\in S_{1}^{c}}\pi^{\star }_{1,i}\Psi_{ij^{\prime}}=M_{-}\,.$ $$

However, we have

$$ $\mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\bm{\pi}_{2}^{\star})=\max_{\bm{\pi}} \mathcal{P}_{\Psi}(\bm{\pi},\bm{\pi}_{2}^{\star})\geqslant\mathcal{P}_{\Psi}( \operatorname{Unif}(S_{1}),\bm{\pi}_{2}^{\star})=\sum_{i\in S_{1}}\sum_{j=1}^{ n}\frac{\pi^{\star}_{2,j}}{|S_{1}|}\Psi_{ij}>M_{-}\,,$ $$

which causes a contradiction.

#### Case 2.

If $\operatorname{supp}(\bm{\pi}_{2}^{\star})\cap S_{1}=\emptyset$ and $\operatorname{supp}(\bm{\pi}_{1}^{\star})\cap S_{1}\neq\emptyset$, taking $i^{\prime}\in\operatorname{supp}(\bm{\pi}_{1}^{\star})\cap S_{1}$, we have

$$ $\mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\bm{\pi}_{2}^{\star})=\max_{\bm{\pi}} \mathcal{P}_{\Psi}(\bm{\pi},\bm{\pi}_{2}^{\star})\geqslant\mathcal{P}_{\Psi}( \bm{\delta}_{i^{\prime}},\bm{\pi}_{2}^{\star})=\sum_{j\in S_{1}^{c}}\pi_{2,j}^ {\star}\Psi_{i^{\prime}j}=M_{+}\,.$ $$

However, we have

$$ $\mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\bm{\pi}_{2}^{\star})=\min_{\bm{\pi}} \mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\bm{\pi})\leqslant\mathcal{P}_{\Psi}( \bm{\pi}_{1}^{\star},\operatorname{Unif}(S_{1}))=\sum_{i=1}^{n}\sum_{j\in S_{1 }}\frac{\pi^{\star}_{1,i}}{|S_{1}|}\Psi_{ij}<M_{+}\,,$ $$

which cause a contradiction.

#### Case 3.

If If $\operatorname{supp}(\bm{\pi}_{2}^{\star})\cap S_{1}\neq\emptyset$ and $\operatorname{supp}(\bm{\pi}_{1}^{\star})\cap S_{1}\neq\emptyset$, taking $i_{2}^{\star}\in\operatorname{supp}(\bm{\pi}_{2}^{\star})\cap S_{1}$, we consider the following strategy $\bm{\pi}^{\prime}_{1}$:

$$ $\begin{cases}\pi^{\prime}_{1,i}=0\,,&i\in S_{1}^{c}\\ \pi^{\prime}_{1,i}=\pi^{\star}_{1,i}\,,&i\in S_{1}\backslash\{i_{2}^{\star}\} \\ \pi^{\prime}_{1,i_{2}^{\star}}=\pi^{\star}_{1,i_{2}^{\star}}+\sum_{i\in S_{1}^ {c}}\pi^{\star}_{1,i}\,,&i=i_{2}^{\star}\end{cases}\,.$ $$

Then we have

$$ $\displaystyle\mathcal{P}_{\Psi}(\bm{\pi}^{\prime}_{1},\bm{\pi}_{2}^{\star})- \mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\star},\bm{\pi}_{2}^{\star})=$ $\displaystyle\ \sum_{i=1}^{n}\sum_{j=1}^{n}\left(\pi^{\prime}_{1,i}-\pi^{\star }_{1,i}\right)\pi^{\star}_{2,j}\Psi_{ij}$ (4.4) $\displaystyle=$ $\displaystyle\ -\sum_{i\in S_{1}^{c}}\sum_{j=1}^{n}\pi^{\star}_{1,i}\pi^{\star }_{2,j}\Psi_{ij}+\sum_{j=1}^{n}\sum_{i\in S_{1}^{c}}\pi_{1,i}^{\star}\pi^{ \star}_{2,j}\Psi_{i_{2}^{\star}j}$ $\displaystyle=$ $\displaystyle\ \sum_{j=1}^{n}\pi^{\star}_{2,j}\left[\sum_{i\in S_{1}^{c}}\pi_{ 1,i}^{\star}\left(\Psi_{i_{2}^{\star}j}-\Psi_{ij}\right)\right]>0\,.$ $$

where the last inequality follows from the following two facts: for any $i\in S_{1}^{c}$,

$$ $\Psi_{i_{2}^{\star}j}-\Psi_{ij}=\begin{cases}M_{+}-\Psi_{ij}\geqslant 0\,,&j \in S_{1}^{c}\\ \Psi_{i_{2}^{\star}j}-M_{-}\geqslant 0\,,&j\in S_{1}\end{cases}\,,$ $$

and when $j=i_{2}^{\star}$,

$$ $\pi^{\star}_{2,i_{2}^{\star}}\left[\sum_{i\in S_{1}^{c}}\pi_{1,i}^{\star}\left (\Psi_{i_{2}^{\star}i_{2}^{\star}}-\Psi_{ii_{2}^{\star}}\right)\right]=\pi^{ \star}_{2,i_{2}^{\star}}\left(\Psi(1/2)-M_{-}\right)\sum_{i\in S_{1}^{c}}\pi^{ \star}_{1,i}>0\,.$ $$

However, ([4.4](https://arxiv.org/html/2505.20627v1#S4.E4)) causes a contradiction to the fact that $\mathcal{P}_{\Psi}(\bm{\pi}_{1}^{\prime},\bm{\pi}_{2}^{\star})\leqslant\max_{
\bm{\pi}}\mathcal{P}_{\Psi}(\bm{\pi},\bm{\pi}^{\star}_{2})=\mathcal{P}_{\Psi}(
\bm{\pi}_{1}^{\star},\bm{\pi}_{2}^{\star})$.

Hence, in summary, it must hold that $\operatorname{supp}(\bm{\pi}_{1}^{\star})\cap S_{1}^{c}=\emptyset$, i.e., $\operatorname{supp}(\bm{\pi}_{1}^{\star})\subseteq S_{1}$.

## 5 Impossibility of Preference Matching

In this section, we first revisit the definition of preference matching in the BTL model (Xiao et al., 2024). Then we introduce a general theoretical framework of preference matching within the context of game-theoretic LLM alignment, and establish a general impossibility result, as stated in Theorem [5.1](https://arxiv.org/html/2505.20627v1#S5.Thmtheorem1). Finally we apply this general result to problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)), concluding that preference matching is impossible. The proof of Theorem [5.1](https://arxiv.org/html/2505.20627v1#S5.Thmtheorem1) is provided in Section [5.1](https://arxiv.org/html/2505.20627v1#S5.SS1).

In previous sections, we have characterized the diversity of alignment result via mixed strategies. In Xiao et al. (2024), the authors proposed a more refined criterion for diversity when the preference $\mathcal{P}(y\succ y^{\prime}\mid x)$ follows a BTL model,

$$ $\mathcal{P}(y\succ y^{\prime}\mid x)=\frac{\exp(r(x,y))}{\exp(r(x,y))+\exp(r(x ,y^{\prime}))}\,,$ $$

as given by ([1.1](https://arxiv.org/html/2505.20627v1#S1.E1)).
They pointed out that it is unwise to completely disregard any minority opinions in the case that 51% of human labelers prefer $y_{1}$ over $y_{2}$ for a binary comparison.
They suggested that the policy ([5.1](https://arxiv.org/html/2505.20627v1#S5.E1)),

$$ $\bm{\pi}^{*}(y\mid x)=\frac{\exp{(r(x,y))}}{\sum_{y^{\prime}}\exp{(r(x,y^{ \prime}))}}\,,$ (5.1) $$

referred to as the preference-matching policy, fully accounts for the diversity in human preferences.

It is easy to see that there exists a Condorcet winning response under BTL model. According to Theorem [3.1](https://arxiv.org/html/2505.20627v1#S3.Thmtheorem1), using preference $\Psi(\mathcal{P}(y\succ y^{\prime}\mid x))$ as payoff with $\Psi(t)=t$ or $\Psi(t)=\log(t/(1-t))$ will lead the Nash solution to collapse to a single response instead of matching with $\bm{\pi}^{*}$. This shows that both RLHF and NLHF do not accounts for the diversity inherent in human preferences from the perspective of preference matching (Xiao et al., 2024; Liu et al., 2025), even under BTL model.

To achieve alignment fully accounting for diversity, we would like to match the Nash solution with the desired policy $\bm{\pi}^{*}$.
In Xiao et al. (2024), the authors answered this question for RLHF. They proposed the preference matching RLHF (PM-RLHF) method which successfully achieves preference matching, by slightly modifying the RLHF objective.
Here, we aim to explore the possibility of designing a new learnable payoff matrix that aligns with the desired strategy in a game-theoretic framework for LLM alignment:

*Which choices of $\Psi$ ensure preference matching?*

Although it is currently unknown how to generalize the notion of preference matching policy to a general non-BTL preference, to maintain the generality of the discussion and drop the BTL model assumption, we suppose there exists an ideal policy, denoted by $\bm{\pi}^{*}$, which captures the diversity of human preferences perfectly.

Given a prompt $x$, we consider the set of all possible responses generated by the LLM: $\{y_{1},\ldots,y_{n}\}$. We further suppose that the policy $\bm{\pi}^{*}$ has full support over these $n$ responses, meaning $\bm{\pi}^{*}>0$, as we exclude responses not supported by $\bm{\pi}^{*}$ from consideration. Then our goal is to construct a game, represented by a payoff matrix $\{\alpha_{ij}\}_{i,j=1}^{n}$, with its Nash solution the given policy $\bm{\pi}^{*}$, i.e.,

$$ $\bm{\pi}^{*}=\arg\max_{\bm{\pi}}\min_{\bm{\pi}^{\prime}}\sum_{i=1}^{n}\sum_{j= 1}^{n}\alpha_{ij}\pi_{i}\pi^{\prime}_{j}\,.$ $$

To answer this question, we characterize the Nash solution under the given payoff matrix $\{\alpha_{ij}\}_{i,j=1}^{n}$, which is summarized by Lemma [5.1](https://arxiv.org/html/2505.20627v1#S5.Thmothertheorem1). The proof is deferred to Appendix [A](https://arxiv.org/html/2505.20627v1#A1).

###### Lemma 5.1 (KKT Condition) .

Consider a game with payoff matrix $\{\alpha_{ij}\}_{i,j=1}^{n}$. Then $\bm{\pi}^{*}>0$ is a Nash solution to the game if and only if there exists $\bm{u}^{*}\in\mathbb{R}^{n}$ with $\bm{u}^{*}\geqslant 0$ and $\sum_{i=1}^{n}u_{i}^{*}=1$, and $t^{*}\in\mathbb{R}$ such that the following KKT conditions hold:

$$ $\begin{cases}\sum_{i=1}^{n}\pi_{i}^{*}\alpha_{ij}-t^{*}\leqslant 0&j=1,\cdots, n\\ u_{j}^{*}\left(\sum_{i=1}^{n}\pi_{i}^{*}\alpha_{ij}-t^{*}\right)=0&j=1,\cdots, n\\ \sum_{j=1}^{n}\alpha_{ij}u_{j}^{*}=t^{*}&i=1,\cdots,n\end{cases}\,.$ $$

According to Lemma [5.1](https://arxiv.org/html/2505.20627v1#S5.Thmothertheorem1), it is easy to verify that the payoff matrix

$$ $\alpha_{ij}=\pi_{i}^{*}+\pi_{j}^{*}-\delta_{ij}\,,\forall\,1\leqslant i,j \leqslant n\,,$ (5.2) $$

and the payoff matrix

$$ $\alpha_{ij}=-\frac{\pi_{j}^{*}}{\pi_{i}^{*}}+n\delta_{ij}\,,\forall\,1 \leqslant i,j\leqslant n\,,$ (5.3) $$

both guarantee that $\bm{\pi}^{*}$ is a Nash solution (the details are provided in Appendix [B](https://arxiv.org/html/2505.20627v1#A2)).
However, these payoff matrices do not depend on the given policy $\bm{\pi}^{*}$ in a reasonable way. The payoff matrix in Equation ([5.2](https://arxiv.org/html/2505.20627v1#S5.E2)) is symmetric, making it difficult to interpret. Even worse, it depends on the raw value of $\bm{\pi}^{*}$. In practice, $\bm{\pi}^{*}$ is often only known up to a normalizing constant. For instance, the preference matching policy ([5.1](https://arxiv.org/html/2505.20627v1#S5.E1)) includes a normalizing constant in the denominator that involves summing over $n$ terms. This constant is hard to determine when $n$ is large and unknown, as is often the case in LLMs. The payoff matrix in Equation ([5.3](https://arxiv.org/html/2505.20627v1#S5.E3)) faces a similar issue as it explicitly depends on $n$, which is an extremely large and unknown value in practice.

In summary, the above two payoff matrices rely on information that is often unavailable in practice, such as $n$ and the raw value of $\bm{\pi}^{*}$.
What we can obtain in practice for the design of $\alpha_{ij}$ is the preference information between two responses $y_{i}$ and $y_{j}$, which we assume depends solely on the ratio between $\pi_{i}^{*}$ and $\pi_{j}^{*}$.
When the preference satisfies the BTL model ([1.1](https://arxiv.org/html/2505.20627v1#S1.E1)), this assumption is justified by the fact that the preference between any two responses depends solely on the ratio of the values assigned by their corresponding preference matching policies ([5.1](https://arxiv.org/html/2505.20627v1#S5.E1)).
From this practical consideration, we assume that the payoff matrix satisfies the following assumptions:

###### Assumption 5.2 .

Given any $\bm{\pi}^{*}>0$, the payoff matrix $\{\alpha_{ij}\}_{i,j=1}^{n}$ satisfies the following conditions:

- 1.
For all $i\in[n]$, $\alpha_{ii}=C$ where $C$ is a constant independent of $\bm{\pi}^{*}$ and $n$. In other words, the diagonal elements are the same constant.
- 2.
For all $i,j\in[n]$ with $i\neq j$, $\alpha_{ij}=f\left(\frac{\pi^{*}_{i}}{\pi^{*}_{j}}\right)$ for some smooth function $f$ that is independent of $\bm{\pi}^{*}$ and $n$. In other words, the off-diagonal elements depend on the ratio $\frac{\pi^{*}_{i}}{\pi^{*}_{j}}$ in the same way for all pairs $(i,j)$ with $i\neq j$.

We emphasize that the above two assumptions are crucial for constructing a meaningful and practically learnable payoff matrix. Furthermore, for effective alignment, the payoff matrix should not only ensure $\bm{\pi}^{*}$ to be a Nash solution, but $\bm{\pi}^{*}$ must be the only Nash solution. The uniqueness requirement excludes trivial payoff matrices such as $\alpha_{ij}=C$, where every $\bm{\pi}^{*}>0$ is a Nash solution.

Unfortunately, in Theorem [5.1](https://arxiv.org/html/2505.20627v1#S5.Thmtheorem1), we prove that such a payoff matrix $\{\alpha_{ij}\}_{i,j=1}^{n}$ does not exist generally. The proof can be found in Section [5.1](https://arxiv.org/html/2505.20627v1#S5.SS1).

###### Theorem 5.1 (Impossibility of Preference Matching for General Payoffs) .

There does not exist a payoff matrix $\{\alpha_{ij}\}_{i,j=1}^{n}$ satisfying Assumption [5.2](https://arxiv.org/html/2505.20627v1#S5.Thmothertheorem2) such that for any given $\bm{\pi}^{*}>0$, the Nash solution to the game is unique and equals to $\bm{\pi}^{*}$.

###### Remark 5.3 .

If we relax Assumption [5.2](https://arxiv.org/html/2505.20627v1#S5.Thmothertheorem2) and allow the entries of the payoff matrix to depend on $n$, then the design ([5.3](https://arxiv.org/html/2505.20627v1#S5.E3)) is actually eligible for preference matching.

Theorem [5.1](https://arxiv.org/html/2505.20627v1#S5.Thmtheorem1) implies that no simple mapping of the preference can yield a payoff that leads to preference matching.
As a special case of Theorem [5.1](https://arxiv.org/html/2505.20627v1#S5.Thmtheorem1), under BTL model, the generalized game in Equation ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) with a smooth mapping $\Psi$,

$$ $\max_{\bm{\pi}}\min_{\bm{\pi}^{\prime}}\mathbb{E}_{x\sim\rho}\left[\mathbb{E}_ {y\sim\bm{\pi}(\cdot\mid x)}\mathbb{E}_{y^{\prime}\sim\bm{\pi}^{\prime}(\cdot \mid x)}\left[\Psi\left(\mathcal{P}(y\succ y^{\prime}\mid x)\right)\right] \right]\,,$ $$

cannot achieve preference matching. Therefore, we obtain the following corollary.

###### Corollary 5.4 .

Problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)) with smooth mapping $\Psi$ cannot achieve preference matching.

###### Lemma 5.1 (KKT Condition) .

###### Assumption 5.2 .

###### Theorem 5.1 (Impossibility of Preference Matching for General Payoffs) .

###### Remark 5.3 .

###### Corollary 5.4 .

### 5.1 Proof of Theorem 5.1

We first present a useful lemma (Lemma [5.5](https://arxiv.org/html/2505.20627v1#S5.Thmothertheorem5)) that further investigates the KKT conditions (Lemma [5.1](https://arxiv.org/html/2505.20627v1#S5.Thmothertheorem1)) when the payoff matrix induces a unique Nash equilibrium.

###### Lemma 5.5 .

If a game with the payoff matrix $\{\alpha_{ij}\}_{i,j=1}^{n}$ has a unique Nash solution $\bm{\pi}^{*}$, then for any $j\in[n]$, it must hold $u_{j}^{*}>0$ in the KKT conditions, and

$$ $\sum_{i=1}^{n}\pi_{i}^{*}\alpha_{ij}=t^{*}.$ $$

###### Proof of Lemma 5.5 .

Suppose that the KKT conditions provide the unique Nash solution $(\bm{\pi}^{*},\bm{u}^{*},t^{*})$. Then we define:

$$ $\mathcal{J}_{0}:=\left\{j\in[n]:u_{j}^{*}\neq 0\right\},\text{ and }\tilde{ \mathcal{J}}_{0}:=\left\{j\in[n]:u_{j}^{*}=0\right\},$ $$

with $\mathcal{J}_{0}\cup\tilde{\mathcal{J}}_{0}=[n]$.
Since $\bm{u}^{*}\geqslant 0$ and $\sum u_{j}^{*}=1$, there exists $j\in[n]$, such that $u_{j}^{*}\neq 0$, i.e., $\mathcal{J}_{0}\neq\emptyset$.
Now, we aim to show $\tilde{\mathcal{J}}_{0}=\emptyset$.
We prove by contradiction.
Suppose $\tilde{\mathcal{J}}_{0}\neq\emptyset$, taking $j_{0}\in\mathcal{J}_{0}$, we consider two spaces

$$ $\displaystyle\ V_{1}:=\left\{\bm{\pi}\in\mathbb{R}^{n}:\sum_{i=1}^{n}\pi_{i} \left(\alpha_{ij}-\alpha_{ij_{0}}\right)=0,\ \forall j\in\mathcal{J}_{0} \backslash\{j_{0}\}\right\},$ $\displaystyle\ V_{2}:=V_{1}\bigcap\left\{\bm{\pi}\in\mathbb{R}^{n}:\sum_{i=1}^ {n}\pi_{i}(\alpha_{ij}-\alpha_{ij_{0}})\leqslant 0,\ \forall j\in\tilde{ \mathcal{J}}_{0}\right\}.$ $$

Then we claim that $\bm{\pi}^{*}\in V_{2}$ and $\operatorname{dim}(V_{2})\geqslant 2$.
For the first claim, by the KKT conditions in Lemma [5.1](https://arxiv.org/html/2505.20627v1#S5.Thmothertheorem1), for any $j\in\mathcal{J}_{0}$, we obtain

$$ $\sum_{i=1}^{n}\pi^{*}_{i}\alpha_{ij}=t^{*}=\sum_{i=1}^{n}\pi^{*}_{i}\alpha_{ij _{0}},$ $$

thus $\bm{\pi}^{*}\in V_{1}$.
Moreover, again by the KKT conditions, for any $j\in\tilde{\mathcal{J}}_{0}$, we have

$$ $\sum_{i=1}^{n}\pi_{i}^{*}\alpha_{ij}\leqslant t^{*}=\sum_{i=1}^{n}\pi^{*}_{i} \alpha_{ij_{0}},$ $$

which shows that $\bm{\pi}^{*}\in V_{2}$.
For the second claim, take $\tilde{j}_{0}\in\tilde{\mathcal{J}}_{0}$ and consider

$$ $\displaystyle\ V_{3}:=\left\{\bm{\pi}\in\mathbb{R}^{n}:\sum_{i=1}^{n}\pi_{i} \left(\alpha_{ij}-\alpha_{ij_{0}}\right)=0,\ \forall j\in[n]\backslash\{j_{0}, \tilde{j}_{0}\}\right\},$ $\displaystyle\ V_{4}:=V_{3}\bigcap\left\{\bm{\pi}\in\mathbb{R}^{n}:\sum_{i=1}^ {n}\pi_{i}(\alpha_{i\tilde{j}_{0}}-\alpha_{ij_{0}})\leqslant 0\right\}.$ $$

We can easily see $V_{4}\subseteq V_{2}$.
Note that $V_{3}$ can be regarded as a kernel space of a linear transformation from $\mathbb{R}^{n}$ to $\mathbb{R}^{n-2}$.
By the dimension theorem in linear algebra, we obtain $\operatorname{dim}(V_{3})=n-\operatorname{dim}(\operatorname{Im}(A))\geqslant n
-(n-2)=2$.
For any $\bm{\pi}\in V_{3}$, it must hold that $\bm{\pi}\in V_{4}$ or $-\bm{\pi}\in V_{4}$, so $\operatorname{dim}(V_{4})=\operatorname{dim}(V_{3})\geqslant 2$.
Therefore, we have $\operatorname{dim}(V_{1})\geqslant\operatorname{dim}(V_{2})\geqslant
\operatorname{dim}(V_{4})\geqslant 2$.

Thus, we can take another $\tilde{\bm{\pi}}^{*}\in V_{2}$ which is linear independent with $\bm{\pi}^{*}$.
Note that for any $a,b\in\mathbb{R}_{+}$, $a\bm{\pi}^{*}+b\tilde{\bm{\pi}}^{*}\in V_{2}$.
Taking large $a\in\mathbb{R}_{+}$, we have $a\bm{\pi}^{*}+b\tilde{\bm{\pi}}^{*}\in V_{2}$ and $a\bm{\pi}^{*}+b\tilde{\bm{\pi}}^{*}>0$, since $\pi^{*}>0$.
Therefore, there exists $a_{1}\in\mathbb{R}_{+}$, such that $\bm{\pi}_{2}^{*}:=\frac{a\bm{\pi}^{*}+\tilde{\bm{\pi}}^{*}}{a_{1}}\in V_{2}$ that satisfies $\bm{\pi}_{2}^{*}\neq\bm{\pi}^{*}$, $\bm{\pi}_{2}^{*}>0$, and $\sum_{i}\pi_{2,i}^{*}=1$.
Thus, we obtain another Nash equilibrium $(\bm{\pi}_{2}^{*},\bm{u}^{*},t^{*})$, causing contradiction to the uniqueness of Nash solution.
Hence, it must hold that $\tilde{\mathcal{J}}_{0}=\emptyset$.
∎

Next we provide the proof of Theorem [5.1](https://arxiv.org/html/2505.20627v1#S5.Thmtheorem1). Using Lemma [5.5](https://arxiv.org/html/2505.20627v1#S5.Thmothertheorem5), uniqueness requires us to seek solutions that satisfies

$$ $\sum_{i=1}^{n}\pi^{*}_{i}\alpha_{ij}=t^{*}$ $$

for all $j\in[n]$, where $t^{*}$ is a constant that may depend on $\bm{\pi}^{*}$.
Consider $n\geqslant 5$, for any four distinct indices $j_{1},j_{2},k_{1},k_{2}$, we have

$$ $\displaystyle\ \sum_{i\neq j_{1},k_{1},k_{2}}\pi_{i}f\left(\frac{\pi_{i}}{\pi_ {j_{1}}}\right)+\pi_{k_{1}}f\left(\frac{\pi_{k_{1}}}{\pi_{j_{1}}}\right)+\pi_{ k_{2}}f\left(\frac{\pi_{k_{2}}}{\pi_{j_{1}}}\right)+C\pi_{j_{1}}$ (5.4) $\displaystyle=$ $\displaystyle\ \sum_{i\neq j_{2},k_{1},k_{2}}\pi_{i}f\left(\frac{\pi_{i}}{\pi_ {j_{2}}}\right)+\pi_{k_{1}}f\left(\frac{\pi_{k_{1}}}{\pi_{j_{2}}}\right)+\pi_{ k_{2}}f\left(\frac{\pi_{k_{2}}}{\pi_{j_{2}}}\right)+C\pi_{j_{2}}$ $$

Let us consider the infinitesimal variation $\pi_{k_{1}}\to\pi_{k_{1}}+\delta$ and $\pi_{k_{2}}\to\pi_{k_{2}}-\delta$, keeping others still. We obtain that

$$ $\displaystyle\ \sum_{i\neq j_{1},k_{1},k_{2}}\pi_{i}f\left(\frac{\pi_{i}}{\pi_ {j_{1}}}\right)+(\pi_{k_{1}}+\delta)f\left(\frac{\pi_{k_{1}}+\delta}{\pi_{j_{1 }}}\right)+(\pi_{k_{2}}-\delta)f\left(\frac{\pi_{k_{2}}-\delta}{\pi_{j_{1}}} \right)+C\pi_{j_{1}}$ (5.5) $\displaystyle=$ $\displaystyle\ \sum_{i\neq j_{2},k_{1},k_{2}}\pi_{i}f\left(\frac{\pi_{i}}{\pi_ {j_{2}}}\right)+(\pi_{k_{1}}+\delta)f\left(\frac{\pi_{k_{1}}+\delta}{\pi_{j_{2 }}}\right)+(\pi_{k_{2}}-\delta)f\left(\frac{\pi_{k_{2}}-\delta}{\pi_{j_{2}}} \right)+C\pi_{j_{2}}$ $$

Subtracting both sides of ([5.4](https://arxiv.org/html/2505.20627v1#S5.E4)) from ([5.5](https://arxiv.org/html/2505.20627v1#S5.E5)), we obtain that

$$ $\displaystyle\ (\pi_{k_{1}}+\delta)f\left(\frac{\pi_{k_{1}}+\delta}{\pi_{j_{1} }}\right)+(\pi_{k_{2}}-\delta)f\left(\frac{\pi_{k_{2}}-\delta}{\pi_{j_{1}}} \right)-\pi_{k_{1}}f\left(\frac{\pi_{k_{1}}}{\pi_{j_{1}}}\right)-\pi_{k_{2}}f \left(\frac{\pi_{k_{2}}}{\pi_{j_{1}}}\right)$ (5.6) $\displaystyle=$ $\displaystyle\ (\pi_{k_{1}}+\delta)f\left(\frac{\pi_{k_{1}}+\delta}{\pi_{j_{2} }}\right)+(\pi_{k_{2}}-\delta)f\left(\frac{\pi_{k_{2}}-\delta}{\pi_{j_{2}}} \right)-\pi_{k_{1}}f\left(\frac{\pi_{k_{1}}}{\pi_{j_{2}}}\right)-\pi_{k_{2}}f \left(\frac{\pi_{k_{2}}}{\pi_{j_{2}}}\right)\,,$ $$

i.e., we have

$$ $\displaystyle\ (\pi_{k_{1}}+\delta)\left(f\left(\frac{\pi_{k_{1}}+\delta}{\pi_ {j_{1}}}\right)-f\left(\frac{\pi_{k_{1}}}{\pi_{j_{1}}}\right)\right)+\delta f \left(\frac{\pi_{k_{1}}}{\pi_{j_{1}}}\right)$ (5.7) $\displaystyle\ \ +(\pi_{k_{2}}-\delta)\left(f\left(\frac{\pi_{k_{2}}-\delta}{ \pi_{j_{1}}}\right)-f\left(\frac{\pi_{k_{2}}}{\pi_{j_{1}}}\right)\right)- \delta f\left(\frac{\pi_{k_{2}}}{\pi_{j_{1}}}\right)$ $\displaystyle=$ $\displaystyle\ (\pi_{k_{1}}+\delta)\left(f\left(\frac{\pi_{k_{1}}+\delta}{\pi_ {j_{2}}}\right)-f\left(\frac{\pi_{k_{1}}}{\pi_{j_{2}}}\right)\right)+\delta f \left(\frac{\pi_{k_{1}}}{\pi_{j_{2}}}\right)$ $\displaystyle\ \ +(\pi_{k_{2}}-\delta)\left(f\left(\frac{\pi_{k_{2}}-\delta}{ \pi_{j_{2}}}\right)-f\left(\frac{\pi_{k_{2}}}{\pi_{j_{2}}}\right)\right)- \delta f\left(\frac{\pi_{k_{2}}}{\pi_{j_{2}}}\right).$ $$

As $f$ is smooth, using

$$ $\lim_{\delta\to 0}\frac{f\left(\frac{x+\delta}{\pi_{j}}\right)-f\left(\frac{x} {\pi_{j}}\right)}{\delta}=\frac{1}{\pi_{j}}f^{\prime}\left(\frac{x}{\pi_{j}} \right),$ $$

and taking $\delta\to 0$, we obtain the following identity from ([5.7](https://arxiv.org/html/2505.20627v1#S5.E7)),

$$ $\displaystyle\ f\left(\frac{\pi_{k_{1}}}{\pi_{j_{1}}}\right)+\frac{\pi_{k_{1}} }{\pi_{j_{1}}}f^{\prime}\left(\frac{\pi_{k_{1}}}{\pi_{j_{1}}}\right)-f\left( \frac{\pi_{k_{2}}}{\pi_{j_{1}}}\right)-\frac{\pi_{k_{2}}}{\pi_{j_{1}}}f^{ \prime}\left(\frac{\pi_{k_{2}}}{\pi_{j_{1}}}\right)$ (5.8) $\displaystyle=$ $\displaystyle\ f\left(\frac{\pi_{k_{1}}}{\pi_{j_{2}}}\right)+\frac{\pi_{k_{1}} }{\pi_{j_{2}}}f^{\prime}\left(\frac{\pi_{k_{1}}}{\pi_{j_{2}}}\right)-f\left( \frac{\pi_{k_{2}}}{\pi_{j_{2}}}\right)-\frac{\pi_{k_{2}}}{\pi_{j_{2}}}f^{ \prime}\left(\frac{\pi_{k_{2}}}{\pi_{j_{2}}}\right).$ $$

Thus, we obtain that

$$ $\displaystyle\ f\left(\frac{\pi_{k_{1}}}{\pi_{j_{1}}}\right)+\frac{\pi_{k_{1}} }{\pi_{j_{1}}}f^{\prime}\left(\frac{\pi_{k_{1}}}{\pi_{j_{1}}}\right)-f\left( \frac{\pi_{k_{1}}}{\pi_{j_{2}}}\right)-\frac{\pi_{k_{1}}}{\pi_{j_{2}}}f^{ \prime}\left(\frac{\pi_{k_{1}}}{\pi_{j_{2}}}\right)$ (5.9) $\displaystyle=$ $\displaystyle\ f\left(\frac{\pi_{k_{2}}}{\pi_{j_{1}}}\right)+\frac{\pi_{k_{2}} }{\pi_{j_{1}}}f^{\prime}\left(\frac{\pi_{k_{2}}}{\pi_{j_{1}}}\right)-f\left( \frac{\pi_{k_{2}}}{\pi_{j_{2}}}\right)-\frac{\pi_{k_{2}}}{\pi_{j_{2}}}f^{ \prime}\left(\frac{\pi_{k_{2}}}{\pi_{j_{2}}}\right).$ $$

Since ([5.9](https://arxiv.org/html/2505.20627v1#S5.E9)) holds for any $\pi>0$, given any $\pi_{j_{1}}\neq\pi_{j_{2}}$, for any $x_{1},x_{2}\in(0,1-\pi_{j_{1}}-\pi_{j_{2}})$, we have

$$ $\displaystyle\ f\left(\frac{x_{1}}{\pi_{j_{1}}}\right)+\frac{x_{1}}{\pi_{j_{1} }}f^{\prime}\left(\frac{x_{1}}{\pi_{j_{1}}}\right)-f\left(\frac{x_{1}}{\pi_{j_ {2}}}\right)-\frac{x_{1}}{\pi_{j_{2}}}f^{\prime}\left(\frac{x_{1}}{\pi_{j_{2}} }\right)$ $\displaystyle=$ $\displaystyle\ f\left(\frac{x_{2}}{\pi_{j_{1}}}\right)+\frac{x_{2}}{\pi_{j_{1} }}f^{\prime}\left(\frac{x_{2}}{\pi_{j_{1}}}\right)-f\left(\frac{x_{2}}{\pi_{j_ {2}}}\right)-\frac{x_{2}}{\pi_{j_{2}}}f^{\prime}\left(\frac{x_{2}}{\pi_{j_{2}} }\right),$ $$

which induces the following for any $x\in(0,1-\pi_{j_{1}}-\pi_{j_{2}})$,

$$ $f\left(\frac{x}{\pi_{j_{1}}}\right)+\frac{x}{\pi_{j_{1}}}f^{\prime}\left(\frac {x}{\pi_{j_{1}}}\right)-f\left(\frac{x}{\pi_{j_{2}}}\right)-\frac{x}{\pi_{j_{2 }}}f^{\prime}\left(\frac{x}{\pi_{j_{2}}}\right)=C(\pi_{j_{1}},\pi_{j_{2}})\,,$ (5.10) $$

i.e., we have

$$ $f\left(\frac{x}{\pi_{j_{1}}}\right)+\frac{x}{\pi_{j_{1}}}f^{\prime}\left(\frac {x}{\pi_{j_{1}}}\right)=C(\pi_{j_{1}},\pi_{j_{2}})+f\left(\frac{x}{\pi_{j_{2}} }\right)+\frac{x}{\pi_{j_{2}}}f^{\prime}\left(\frac{x}{\pi_{j_{2}}}\right).$ (5.11) $$

Without any loss of generality, we assume $\pi_{j_{1}}<\pi_{j_{2}}$, then we obtain

$$ $\displaystyle\ f\left(\frac{x}{\pi_{j_{1}}}\right)+\frac{x}{\pi_{j_{1}}}f^{ \prime}\left(\frac{x}{\pi_{j_{1}}}\right)$ $\displaystyle=$ $\displaystyle\ C(\pi_{j_{1}},\pi_{j_{2}})+f\left(\frac{x}{\pi_{j_{2}}}\right)+ \frac{x}{\pi_{j_{2}}}f^{\prime}\left(\frac{x}{\pi_{j_{2}}}\right)$ $\displaystyle=$ $\displaystyle\ 2C(\pi_{j_{1}},\pi_{j_{2}})+f\left(\frac{\pi_{j_{1}}x}{\pi^{2}_ {j_{2}}}\right)+\frac{\pi_{j_{1}}x}{\pi^{2}_{j_{2}}}f^{\prime}\left(\frac{\pi_ {j_{1}}x}{\pi^{2}_{j_{2}}}\right)$ $\displaystyle=$ $\displaystyle\ \cdots\cdots$ $\displaystyle=$ $\displaystyle\ nC(\pi_{j_{1}},\pi_{j_{2}})+f\left(\frac{\pi^{n-1}_{j_{1}}x}{ \pi^{n}_{j_{2}}}\right)+\frac{\pi^{n-1}_{j_{1}}x}{\pi^{n}_{j_{2}}}f^{\prime} \left(\frac{\pi^{n-1}_{j_{1}}x}{\pi^{n}_{j_{2}}}\right)$ $\displaystyle=$ $\displaystyle\ \cdots\cdots.$ $$

Taking limit, it must hold $C(\pi_{j_{1}},\pi_{j_{2}})=0$, i.e., we have

$$ $f\left(\frac{x}{\pi_{j_{1}}}\right)+\frac{x}{\pi_{j_{1}}}f^{\prime}\left(\frac {x}{\pi_{j_{1}}}\right)=f\left(\frac{x}{\pi_{j_{2}}}\right)+\frac{x}{\pi_{j_{2 }}}f^{\prime}\left(\frac{x}{\pi_{j_{2}}}\right).$ (5.12) $$

Since ([5.12](https://arxiv.org/html/2505.20627v1#S5.E12)) holds for any $\pi>0$, for any $x_{1},x_{2}\in\mathbb{R}_{+}$, we have

$$ $f(x_{1})+x_{1}f^{\prime}(x_{1})=f^{\prime}(x_{2})+x_{2}f^{\prime}(x_{2}),$ $$

thus, for any $x\in\mathbb{R}_{+}$, we have

$$ $f\left(x\right)+xf^{\prime}\left(x\right)=C_{1}\,.$ (5.13) $$

Solving ([5.13](https://arxiv.org/html/2505.20627v1#S5.E13)), we obtain that

$$ $f(x)=\frac{C_{2}}{x}+C_{3}\,.$ $$

Then we obtain that

$$ $\sum_{i=1}^{n}\pi_{i}\alpha_{ij}=C\pi_{j}+\sum_{i\neq j}\pi_{i}\left(\frac{C_{ 2}\pi_{j}}{\pi_{i}}+C_{3}\right)=C_{3}+(C+(n-1)C_{2}-C_{3})\pi_{j}\,,$ $$

yielding $C_{3}=C+(n-1)C_{2}$, and

$$ $f(x)=C+C_{2}\left(\frac{1}{x}+n-1\right),$ $$

which is contradictory to our assumptions.

###### Lemma 5.5 .

###### Proof of Lemma 5.5 .

## 6 Conclusions

We have investigated several properties motivated by social choice theory and diversity considerations within the general game-theoretic LLM alignment framework ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)), where the payoff is designed as a mapping $\Psi$ of the original preference. We have identified the necessary and sufficient conditions on $\Psi$ to guarantee Condorcet consistency and Smith consistency. These conditions allow for a considerably broad class of choices for $\Psi$, demonstrating that these desirable alignment properties are not sensitive to the exact values of the payoff, thereby providing a theoretical foundation for the robustness of the game-theoretic LLM alignment approach. Additionally, we have examined conditions on $\Psi$ that ensure the resulting policy is a mixed strategy, preserving diversity in human preferences. Finally, we have proved that achieving exact preference matching is impossible under the general game-theoretic alignment framework with a smooth mapping, revealing fundamental limitations of this approach.

Our findings suggest several promising directions for future research on LLM alignment. First, while we establish an impossibility result for preference matching under the assumption that $\Psi$ is smooth, it remains an open question whether preference matching can be achieved when $\Psi$ is merely continuous. Second, in practical settings, regularization terms based on the reference model are often added to problem ([1.2](https://arxiv.org/html/2505.20627v1#S1.E2)). Regularization may be crucial for preference matching, for example, Xiao et al. (2024) modified the regularization term in RLHF to achieve preference matching. Analyzing the alignment properties of game-theoretic methods with such regularization is another interesting avenue for future work. Furthermore, how to explicitly define a preference-matching policy for general preferences that do not satisfy the BTL model, and how to develop alignment approaches capable of learning such a policy, remain open problems. Finally, our results highlight that practical preference models must satisfy certain anti-symmetry conditions to ensure Smith consistency—conditions that are not guaranteed by several currently used models. Thus, designing preference model architectures that enforce anti-symmetry is an important and interesting future direction.

## Acknowledgments

We are grateful to Yuhe Qin for providing helpful discussions.
This work was supported in part by NIH grant U01CA274576, ARPA-H Award D24AC00253, NSF grant DMS-2310679, a Meta Faculty Research Award, and Wharton AI for Business.

## References

- Anthropic (2024)
A. Anthropic.
The claude 3 model family: Opus, sonnet, haiku.
*Claude-3 Model Card*, 2024.
- Azar et al. (2024)
M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello.
A general theoretical paradigm to understand learning from human preferences.
In *International Conference on Artificial Intelligence and Statistics*, pages 4447–4455. PMLR, 2024.
- Balinski and Laraki (2010)
M. L. Balinski and R. Laraki.
*Majority judgment : measuring, ranking, and electing*.
MIT Press, 2010.
- Börgers (2010)
C. Börgers.
*Mathematics of social choice: voting, compensation, and division*.
Society for Industrial and Applied Mathematics, 2010.
- Bradley and Terry (1952)
R. A. Bradley and M. E. Terry.
Rank analysis of incomplete block designs: I. the method of paired comparisons.
*Biometrika*, 39(3/4):324–345, 1952.
- Bubeck et al. (2023)
S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang.
Sparks of artificial general intelligence: Early experiments with gpt-4.
*arXiv preprint arXiv:2303.12712*, 2023.
- Casper et al. (2023)
S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, T. T. Wang, S. Marks, C.-R. Segerie, M. Carroll, A. Peng, P. J. Christoffersen, M. Damani, S. Slocum, U. Anwar, A. Siththaranjan, M. Nadeau, E. J. Michaud, J. Pfau, D. Krasheninnikov, X. Chen, L. Langosco, P. Hase, E. Biyik, A. Dragan, D. Krueger, D. Sadigh, and D. Hadfield-Menell.
Open problems and fundamental limitations of reinforcement learning from human feedback.
*Transactions on Machine Learning Research*, 2023.
ISSN 2835-8856.
URL [https://openreview.net/forum?id=bx24KpJ4Eb](https://openreview.net/forum?id=bx24KpJ4Eb).
- Chakraborty et al. (2024)
S. Chakraborty, J. Qiu, H. Yuan, A. Koppel, F. Huang, D. Manocha, A. S. Bedi, and M. Wang.
Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences.
*arXiv preprint arXiv:2402.08925*, 2024.
- Chowdhery et al. (2023)
A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. Dev, H. Michalewski, X. Garcia, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Dohan, S. Agrawal, M. Omernick, A. M. Dai, T. S. Pillai, M. Pellat, A. Lewkowycz, E. Moreira, R. Child, O. Polozov, K. Lee, Z. Zhou, X. Wang, B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, K. Meier-Hellstern, D. Eck, J. Dean, S. Petrov, and N. Fiedel.
Palm: Scaling language modeling with pathways.
*Journal of Machine Learning Research*, 24(240):1–113, 2023.
- Christiano et al. (2017)
P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei.
Deep reinforcement learning from human preferences.
In *Advances in Neural Information Processing Systems*, volume 30, 2017.
- Conitzer et al. (2024)
V. Conitzer, R. Freedman, J. Heitzig, W. H. Holliday, B. M. Jacobs, N. Lambert, M. Mossé, E. Pacuit, S. Russell, H. Schoelkopf, E. Tewolde, and W. S. Zwicker.
Position: social choice should guide ai alignment in dealing with diverse human feedback.
In *Forty-first International Conference on Machine Learning*, 2024.
- Dai and Fleisig (2024)
J. Dai and E. Fleisig.
Mapping social choice theory to RLHF.
In *ICLR 2024 Workshop on Reliable and Responsible Foundation Models*, 2024.
- DeepSeek-AI et al. (2025)
DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J.-M. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B.-L. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D.-L. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. Wang, H. Ding, H. Xin, H. Gao, H. Qu, H. Li, J. Guo, J. Li, J. Wang, J. Chen, J. Yuan, J. Qiu, J. Li, J. Cai, J. Ni, J. Liang, J. Chen, K. Dong, K. Hu, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang, L. Zhang, L. Zhao, L. Wang, L. Zhang, L. Xu, L. Xia, M. Zhang, M. Zhang, M. Tang, M. Li, M. Wang, M. Li, N. Tian, P. Huang, P. Zhang, Q. Wang, Q. Chen, Q. Du, R. Ge, R. Zhang, R. Pan, R. Wang, R. J. Chen, R. Jin, R. Chen, S. Lu, S. Zhou, S. Chen, S. Ye, S. Wang, S. Yu, S. Zhou, S. Pan, S. S. Li, S. Zhou, S.-K. Wu, T. Yun, T. Pei, T. Sun, T. Wang, W. Zeng, W. Zhao, W. Liu, W. Liang, W. Gao, W.-X. Yu, W. Zhang, W. L. Xiao, W. An, X. Liu, X. Wang, X. Chen, X. Nie,
X. Cheng, X. Liu, X. Xie, X. Liu, X. Yang, X. Li, X. Su, X. Lin, X. Q. Li, X. Jin, X.-C. Shen, X. Chen, X. Sun, X. Wang, X. Song, X. Zhou, X. Wang, X. Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. Zhang, Y. Xu, Y. Li, Y. Zhao, Y. Sun, Y. Wang, Y. Yu, Y. Zhang, Y. Shi, Y. Xiong, Y. He, Y. Piao, Y. Wang, Y. Tan, Y. Ma, Y. Liu, Y. Guo, Y. Ou, Y. Wang, Y. Gong, Y.-J. Zou, Y. He, Y. Xiong, Y.-W. Luo, Y. mei You, Y. Liu, Y. Zhou, Y. X. Zhu, Y. Huang, Y. Li, Y. Zheng, Y. Zhu, Y. Ma, Y. Tang, Y. Zha, Y. Yan, Z. Ren, Z. Ren, Z. Sha, Z. Fu, Z. Xu, Z. Xie, Z. guo Zhang, Z. Hao, Z. Ma, Z. Yan, Z. Wu, Z. Gu, Z. Zhu, Z. Liu, Z.-A. Li, Z. Xie, Z. Song, Z. Pan, Z. Huang, Z. Xu, Z. Zhang, and Z. Zhang.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- Dong et al. (2024)
H. Dong, W. Xiong, B. Pang, H. Wang, H. Zhao, Y. Zhou, N. Jiang, D. Sahoo, C. Xiong, and T. Zhang.
RLHF workflow: From reward modeling to online RLHF.
*Transactions on Machine Learning Research*, 2024.
ISSN 2835-8856.
URL [https://openreview.net/forum?id=a13aYUU9eU](https://openreview.net/forum?id=a13aYUU9eU).
- Duersch et al. (2012)
P. Duersch, J. Oechssler, and B. C. Schipper.
Pure strategy equilibria in symmetric two-player zero-sum games.
*International Journal of Game Theory*, 41:553–564, 2012.
ISSN 0020-7276.
- Eloundou et al. (2024)
T. Eloundou, S. Manning, P. Mishkin, and D. Rock.
GPTs are GPTs: Labor market impact potential of LLMs.
*Science*, 384(6702):1306–1308, 2024.
- Ge et al. (2024)
L. Ge, D. Halpern, E. Micha, A. D. Procaccia, I. Shapira, Y. Vorobeychik, and J. Wu.
Axioms for ai alignment from human feedback.
In *Advances in Neural Information Processing Systems*, volume 37, pages 80439–80465, 2024.
- Gehrlein (2006)
W. V. Gehrlein.
*Condorcet’s paradox*.
Springer, 2006.
- Hurst et al. (2024)
A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, A. Mądry, A. Baker-Whitcomb, A. Beutel, A. Borzunov, A. Carney, A. Chow, A. Kirillov, A. Nichol, A. Paino, A. Renzin, A. T. Passos, A. Kirillov, A. Christakis, A. Conneau, A. Kamali, A. Jabri, A. Moyer, A. Tam, A. Crookes, A. Tootoochian, A. Tootoonchian, A. Kumar, A. Vallone, A. Karpathy, A. Braunstein, A. Cann, A. Codispoti, A. Galu, A. Kondrich, A. Tulloch, A. Mishchenko, A. Baek, A. Jiang, A. Pelisse, A. Woodford, A. Gosalia, A. Dhar, A. Pantuliano, A. Nayak, A. Oliver, B. Zoph, B. Ghorbani, B. Leimberger, B. Rossen, B. Sokolowsky, B. Wang, B. Zweig, B. Hoover, B. Samic, B. McGrew, B. Spero, B. Giertler, B. Cheng, B. Lightcap, B. Walkin, B. Quinn, B. Guarraci, B. Hsu, B. Kellogg, B. Eastman, C. Lugaresi, C. Wainwright, C. Bassin, C. Hudson, C. Chu, C. Nelson, C. Li, C. J. Shern, C. Conger, C. Barette, C. Voss, C. Ding, C. Lu, C. Zhang, C. Beaumont, C. Hallacy, C. Koch, C. Gibson, C. Kim,
C. Choi, C. McLeavey, C. Hesse, C. Fischer, C. Winter, C. Czarnecki, C. Jarvis, C. Wei, C. Koumouzelis, D. Sherburn, D. Kappler, D. Levin, D. Levy, D. Carr, D. Farhi, D. Mely, D. Robinson, D. Sasaki, D. Jin, D. Valladares, D. Tsipras, D. Li, D. P. Nguyen, D. Findlay, E. Oiwoh, E. Wong, E. Asdar, E. Proehl, E. Yang, E. Antonow, E. Kramer, E. Peterson, E. Sigler, E. Wallace, E. Brevdo, E. Mays, F. Khorasani, F. P. Such, F. Raso, F. Zhang, F. von Lohmann, F. Sulit, G. Goh, G. Oden, G. Salmon, G. Starace, G. Brockman, H. Salman, H. Bao, H. Hu, H. Wong, H. Wang, H. Schmidt, H. Whitney, H. Jun, H. Kirchner, H. P. de Oliveira Pinto, H. Ren, H. Chang, H. W. Chung, I. Kivlichan, I. O’Connell, I. O’Connell, I. Osband, I. Silber, I. Sohl, I. Okuyucu, I. Lan, I. Kostrikov, I. Sutskever, I. Kanitscheider, I. Gulrajani, J. Coxon, J. Menick, J. Pachocki, J. Aung, J. Betker, J. Crooks, J. Lennon, J. Kiros, J. Leike, J. Park, J. Kwon, J. Phang, J. Teplitz, J. Wei, J. Wolfe, J. Chen, J. Harris, J. Varavva, J. G. Lee,
J. Shieh, J. Lin, J. Yu, J. Weng, J. Tang, J. Yu, J. Jang, J. Q. Candela, J. Beutler, J. Landers, J. Parish, J. Heidecke, J. Schulman, J. Lachman, J. McKay, J. Uesato, J. Ward, J. W. Kim, J. Huizinga, J. Sitkin, J. Kraaijeveld, J. Gross, J. Kaplan, J. Snyder, J. Achiam, J. Jiao, J. Lee, J. Zhuang, J. Harriman, K. Fricke, K. Hayashi, K. Singhal, K. Shi, K. Karthik, K. Wood, K. Rimbach, K. Hsu, K. Nguyen, K. Gu-Lemberg, K. Button, K. Liu, K. Howe, K. Muthukumar, K. Luther, L. Ahmad, L. Kai, L. Itow, L. Workman, L. Pathak, L. Chen, L. Jing, L. Guy, L. Fedus, L. Zhou, L. Mamitsuka, L. Weng, L. McCallum, L. Held, L. Ouyang, L. Feuvrier, L. Zhang, L. Kondraciuk, L. Kaiser, L. Hewitt, L. Metz, L. Doshi, M. Aflak, M. Simens, M. Boyd, M. Thompson, M. Dukhan, M. Chen, M. Gray, M. Hudnall, M. Zhang, M. Aljubeh, M. Litwin, M. Zeng, M. Johnson, M. Shetty, M. Gupta, M. Shah, M. Yatbaz, M. J. Yang, M. Zhong, M. Glaese, M. Chen, M. Janner, M. Lampe, M. Petrov, M. Wu, M. Wang, M. Fradin, M. Pokrass, M. Castro, M. O. T.
de Castro, M. Pavlov, M. Brundage, M. Wang, M. Khan, M. Murati, M. Bavarian, M. Lin, M. Yesildal, N. Soto, N. Gimelshein, N. Cone, N. Staudacher, N. Summers, N. LaFontaine, N. Chowdhury, N. Ryder, N. Stathas, N. Turley, N. Tezak, N. Felix, N. Kudige, N. Keskar, N. Deutsch, N. Bundick, N. Puckett, O. Nachum, O. Okelola, O. Boiko, O. Murk, O. Jaffe, O. Watkins, O. Godement, O. Campbell-Moore, P. Chao, P. McMillan, P. Belov, P. Su, P. Bak, P. Bakkum, P. Deng, P. Dolan, P. Hoeschele, P. Welinder, P. Tillet, P. Pronin, P. Tillet, P. Dhariwal, Q. Yuan, R. Dias, R. Lim, R. Arora, R. Troll, R. Lin, R. G. Lopes, R. Puri, R. Miyara, R. Leike, R. Gaubert, R. Zamani, R. Wang, R. Donnelly, R. Honsby, R. Smith, R. Sahai, R. Ramchandani, R. Huet, R. Carmichael, R. Zellers, R. Chen, R. Chen, R. Nigmatullin, R. Cheu, S. Jain, S. Altman, S. Schoenholz, S. Toizer, S. Miserendino, S. Agarwal, S. Culver, S. Ethersmith, S. Gray, S. Grove, S. Metzger, S. Hermani, S. Jain, S. Zhao, S. Wu, S. Jomoto, S. Wu, Shuaiqi, Xia, S. Phene,
S. Papay, S. Narayanan, S. Coffey, S. Lee, S. Hall, S. Balaji, T. Broda, T. Stramer, T. Xu, T. Gogineni, T. Christianson, T. Sanders, T. Patwardhan, T. Cunninghman, T. Degry, T. Dimson, T. Raoux, T. Shadwell, T. Zheng, T. Underwood, T. Markov, T. Sherbakov, T. Rubin, T. Stasi, T. Kaftan, T. Heywood, T. Peterson, T. Walters, T. Eloundou, V. Qi, V. Moeller, V. Monaco, V. Kuo, V. Fomenko, W. Chang, W. Zheng, W. Zhou, W. Manassra, W. Sheu, W. Zaremba, Y. Patil, Y. Qian, Y. Kim, Y. Cheng, Y. Zhang, Y. He, Y. Zhang, Y. Jin, Y. Dai, and Y. Malkov.
Gpt-4o system card.
*arXiv preprint arXiv:2410.21276*, 2024.
- Ji et al. (2025)
W. Ji, W. Yuan, E. Getzen, K. Cho, M. I. Jordan, S. Mei, J. E. Weston, W. J. Su, J. Xu, and L. Zhang.
An overview of large language models for statisticians.
*arXiv preprint arXiv:2502.17814*, 2025.
- Jiang et al. (2023)
D. Jiang, X. Ren, and B. Y. Lin.
LLM-blender: Ensembling large language models with pairwise ranking and generative fusion.
In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 14165–14178. Association for Computational Linguistics, 2023.
doi: 10.18653/v1/2023.acl-long.792.
URL [https://aclanthology.org/2023.acl-long.792/](https://aclanthology.org/2023.acl-long.792/).
- Liu et al. (2025)
K. Liu, Q. Long, Z. Shi, W. J. Su, and J. Xiao.
Statistical impossibility and possibility of aligning llms with human preferences: From condorcet paradox to nash equilibrium.
*arXiv preprint arXiv:2503.10990*, 2025.
- Luce (2012)
R. D. Luce.
*Individual choice behavior: A theoretical analysis*.
Courier Corporation, 2012.
- Maura-Rivero et al. (2025)
R.-R. Maura-Rivero, M. Lanctot, F. Visin, and K. Larson.
Jackpot! alignment as a maximal lottery.
*arXiv preprint arXiv:2501.19266*, 2025.
- Mishra (2023)
A. Mishra.
Ai alignment and social choice: Fundamental limitations and policy implications.
*arXiv preprint arXiv:2310.16048*, 2023.
- Munos et al. (2024)
R. Munos, M. Valko, D. Calandriello, M. Gheshlaghi Azar, M. Rowland, Z. D. Guo, Y. Tang, M. Geist, T. Mesnard, C. Fiegel, A. Michi, M. Selvi, S. Girgin, N. Momchev, O. Bachem, D. J. Mankowitz, D. Precup, and B. Piot.
Nash learning from human feedback.
In *Forty-first International Conference on Machine Learning*, 2024.
- Myerson (2013)
R. B. Myerson.
*Game theory*.
Harvard university press, 2013.
- Noothigattu et al. (2020)
R. Noothigattu, D. Peters, and A. D. Procaccia.
Axioms for learning from pairwise comparisons.
In *Advances in Neural Information Processing Systems*, volume 33, pages 17745–17754, 2020.
- Novikov et al. (2025)
A. Novikov, N. Vũ, M. Eisenberger, E. Dupont, P.-S. Huang, A. Z. Wagner, S. Shirobokov, B. Kozlovskii, F. J. R. Ruiz, A. Mehrabian, M. P. Kumar, A. See, S. Chaudhuri, G. Holland, A. Davies, S. Nowozin, P. Kohli, and M. Balog.
AlphaEvolve: A coding agent for scientific and algorithmic discovery.
Technical report, Google DeepMind, 05 2025.
URL [https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/AlphaEvolve.pdf](https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/AlphaEvolve.pdf).
- OpenAI (2025)
OpenAI.
Openai o3 and o4-mini system card.
Technical report, OpenAI, 2025.
- Ouyang et al. (2022)
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe.
Training language models to follow instructions with human feedback.
In *Advances in Neural Information Processing Systems*, volume 35, pages 27730–27744, 2022.
- Shoham and Leyton-Brown (2008)
Y. Shoham and K. Leyton-Brown.
*Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations*.
Cambridge University Press, 2008.
- Siththaranjan et al. (2024)
A. Siththaranjan, C. Laidlaw, and D. Hadfield-Menell.
Distributional preference learning: Understanding and accounting for hidden context in RLHF.
In *The Twelfth International Conference on Learning Representations*, 2024.
- Touvron et al. (2023)
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample.
Llama: Open and efficient foundation language models.
*arXiv preprint arXiv:2302.13971*, 2023.
- Wu et al. (2024)
Y. Wu, Z. Sun, H. Yuan, K. Ji, Y. Yang, and Q. Gu.
Self-play preference optimization for language model alignment.
In *Adaptive Foundation Models: Evolving AI for Personalized and Efficient Learning*, 2024.
URL [https://openreview.net/forum?id=Z1PDdGekgn](https://openreview.net/forum?id=Z1PDdGekgn).
- Xiao et al. (2024)
J. Xiao, Z. Li, X. Xie, E. Getzen, C. Fang, Q. Long, and W. J. Su.
On the algorithmic bias of aligning large language models with rlhf: Preference collapse and matching regularization.
*arXiv preprint arXiv:2405.16455*, 2024.

## Appendix A Proof of Lemma 5.1

Suppose each player has $n$ policies and the payoff matrix is $\{\alpha_{ij}\}_{i=1}^{n}$. Then,

$$ $\displaystyle\max_{\bm{\pi}}\min_{\bm{\pi}^{\prime}}\left\{\sum_{i=1}^{n}\sum_ {j=1}^{n}\alpha_{ij}\pi_{i}\pi^{\prime}_{j}\right\}=$ $\displaystyle\max_{\bm{\pi}}\min_{\bm{\pi}^{\prime}}\left\{\sum_{j=1}^{n}\left (\sum_{i=1}^{n}\alpha_{ij}\pi_{i}\right)\pi^{\prime}_{j}\right\}=\max_{\bm{\pi }}\min_{j}\left\{\sum_{i=1}^{n}\alpha_{ij}\pi_{i}\right\}.$ $$

Let us reformulated it into a convex optimization problem.

$$ $\displaystyle\min_{\bm{\pi}}$ $\displaystyle\max_{j}\sum_{i=1}^{n}\alpha_{ij}\pi_{i}$ ( $P$ ) subject to $\displaystyle-\pi_{i}\leqslant 0,\quad i=1,\ldots,n$ $\displaystyle\sum_{i=1}^{n}\pi_{i}-1=0$ $$

Let us further reformulate this problem into the epigraph form by introducing a single variable $t\in\mathbb{R}$:

$$ $\displaystyle\min_{\bm{\pi},t}$ $\displaystyle t$ ( $P^{\prime}$ ) subject to $\displaystyle\sum_{i=1}^{n}\alpha_{ij}\pi_{i}-t\leqslant 0,\quad j=1,\ldots,n$ $\displaystyle-\pi_{i}\leqslant 0,\quad i=1,\ldots,n$ $\displaystyle\sum_{i=1}^{n}\pi_{i}-1=0$ $$

By introducing the dual variables $\bm{u}^{*}\in\mathbb{R}^{n}$, $\tilde{\bm{u}}^{*}\in\mathbb{R}^{n}$ and $v^{*}\in\mathbb{R}$, the KKT conditions is:

- •
stationary condition:
$\sum_{j=1}^{n}\alpha_{ij}u_{j}^{*}-\tilde{u}_{i}^{*}=-v^{*}\quad i=1,\cdots,n$
- •
complementary slackness:
$u_{j}^{*}\left(\sum_{i=1}^{n}\pi_{i}^{*}\alpha_{ij}-t^{*}\right)=0\quad j=1,
\cdots,n$
$\tilde{u}^{*}_{i}\pi^{*}_{i}=0\quad i=1,\cdots,n$
- •
primal feasibility:
$\sum_{i=1}^{n}\pi_{i}^{*}\alpha_{ij}-t^{*}\leqslant 0$
$\bm{\pi}^{*}\geqslant 0$
$\sum_{i=1}^{n}\pi_{i}^{*}=1$
- •
dual feasibility:
$\bm{u}^{*}\geqslant 0$
$\sum_{i=1}^{n}u^{*}_{i}=1$
$\tilde{\bm{u}}^{*}\geqslant 0$

We can easily see that Slater’s condition is satisfied for this problem, so the KKT points are equivalent to primal and dual solutions.
Then taking $\bm{\pi}^{*}>0$ into account, we have $\tilde{u}_{i}^{*}=0$ by the second complementary slackness condition, and the above equations can be simplified to the following system of equations:

$$ $\begin{cases}\bm{u}^{*}\geqslant 0&\\ \sum_{i=1}^{n}u^{*}_{i}=1&\\ \sum_{i=1}^{n}\pi_{i}^{*}\alpha_{ij}-t^{*}\leqslant 0&j=1,\cdots,n\\ u_{j}^{*}\left(\sum_{i=1}^{n}\pi_{i}^{*}\alpha_{ij}-t^{*}\right)=0&j=1,\cdots, n\\ \sum_{j=1}^{n}\alpha_{ij}u_{j}^{*}=-v^{*}&i=1,\cdots,n\end{cases}$ $$

Moreover, notice that

$$ $0=\sum_{j=1}^{n}u_{j}^{*}\left(\sum_{i=1}^{n}\pi_{i}^{*}\alpha_{ij}-t^{*} \right)=\sum_{i=1}^{n}\pi_{i}^{*}\sum_{j=1}^{n}\alpha_{ij}u_{j}^{*}-t^{*}=-v^{ *}-t^{*}\,,$ $$

thus $v^{*}=-t^{*}$. Hence, we conclude our proof.

## Appendix B Verifying Equation ( 5.2 ) and Equation ( 5.3 )

For ([5.2](https://arxiv.org/html/2505.20627v1#S5.E2)), choosing $t^{*}=-v^{*}=\sum_{i=1}^{n}(\pi^{*}_{i})^{2}$ and $u_{i}^{*}=v_{i}^{*}$, $\bm{\pi}^{*}$ is a Nash solution.
For ([5.3](https://arxiv.org/html/2505.20627v1#S5.E3)), choosing $t^{*}=-v^{*}=0$ and $u_{j}^{*}=\frac{(\pi_{j}^{*})^{-1}}{\sum_{j=1}^{n}(\pi_{j}^{*})^{-1}}$, $\bm{\pi}^{*}$ is a Nash solution.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 2
presentation: 2
questions: Here is a line of works [4-5] doesn't require the full payoff matrix for preference optimization. Will this make the proposed issue of "the ground-truth preference matrix is often unavailable" less concerning?


Add missing related works:

[1] Wang, Mingzhi, et al. "Magnetic preference optimization: Achieving last-iterate convergence for language model alignment." arXiv preprint arXiv:2410.16714 (2024).

[2] Zhang, Yuheng, et al. "Improving LLM general preference alignment via optimistic online mirror descent." arXiv preprint arXiv:2502.16852 (2025).

[3] Tang, Xiaohang, et al. "Game-Theoretic Regularized Self-Play Alignment of Large Language Models." arXiv preprint arXiv:2503.00030 (2025).

[4] Zhang, Yuheng, et al. "Iterative nash policy optimization: Aligning llms with general preferences via no-regret learning." arXiv preprint arXiv:2407.00617 (2024).

[5] Rosset, Corby, et al. "Direct nash optimization: Teaching language models to self-improve with general preferences." arXiv preprint arXiv:2404.03715 (2024).
soundness: 3
strengths: The results of Smith Consistency shared with both RLHF and game-theoretic alignment is interesting, also intuitive.
summary: The paper investigates the theoretical aspects of game-theoretic alignment. The authors discuss from three aspects, revealing game-theoretic alignment's superiority and limitations, as well as common property shared with RLHF.
weaknesses: The Presentation is problematic, making the paper less clear. The results might be interesting but the writings without emphasizing the critical results make me hard to follow.
 
- What is continuity assumption specifically in the paper? The authors didn't define it.

- Axiom 4.1 is not a proper axiom. Smith set is not defined. A proper definition is needed for Smith set. Do not assume the readers know it. And I also tend to not use Axiom in writing as these so called axioms are more like definitions.

- The authors should demonstrate more details and analysis on the meanings of the theoretical results. For example, what can theorem 3.2 imply and how does such theoretical results help understand the game-theoretic alignment field? Does that imply learning from noisy preference data or lack of preference data to learn the general preference model in theory wouldn't harm too much? To make some space, not all the theoretical details are needed in the paper, the authors chouls wrap them into a theorem, leaving the insights and analysis in the main text.

- I think the key concern is that it is unclear the practical meaning of these theoretical results. Some writings are also confusing such as "preserving the diversity in human preference strictly, in the sense of preference matching". Do the authors mean game-theoretic alignment cannot preserve diversity? Or just doesn't satisfy preference matching? What's the practical consequence of not satisfying preference mathcing?

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: ## Questions
1. in line 115-117, the authors say "Smith consistency can be ensured by further maintaining the symmetry of the game", and Theorem 4.2 seems to say the same. However, in line 720-723, it seems to say the opposite: "... preference models must satisfy certain anti-symmetry conditions to ensure Smith consistency". To me there seem to be a contradiction. Am I missing something?
2. In line 380-382, the paper mentions Xiao et al. (2025) which showed that a preference matching policy exists under the BTL model. In line 467-471, however, the paper says the generalized game in line 469-470 cannot achieve preference matching. Could you perhaps explain the differences in the settings between these two that caused the difference?

## Typos and Small Suggestions
1. line 381: “such a policy **exist** -> **exists**”
2. line 1063: “Let us **reformulated** -> **reformulate**”
3. line 1064: there seems to be unnecessary space between “min_pi” and “max_j” operators?
soundness: 3
strengths: 1. The paper discusses properties of game-theoretical LLM alignment methods that are highly relevant to many researchers, as well as extending the scope of analysis from the previous works.
2. The paper is well organized, presenting conceptual extensions gradually so that it becomes easier for the readers to follow up (ex. line 220, 254, 313, 396).
summary: This paper provides a theoretical investigation of game-theoretical LLM alignment framework, focusing on how the mapping $\Psi$ of the original preference affects the characteristics of the Nash solution. 

The paper identifies the necessary and sufficient conditions on the mapping to guarantee Condorcet consistency and Smith consistency, where the latter extends the former and considers disjoint subsets of the responses. Conditions on $\Psi$ that ensure the Nash solution is a mixed strategy and preserve diversity are also examined.

While the investigations of consistencies provide a foundation for the robustness of the game-theoretic LLM alignment approaches, the paper also discusses the impossibility of achieving exact preference matching under the general game-theoretic alignment framework where the mapping is assumed to be smooth.
weaknesses: While I understand that the paper mainly focuses on theoretical analysis of the algorithms and preference mapping $\Psi$, I think the paper could have benefitted from providing visual examples (ex. mapping functions and the derived properties) and experiments for better understanding.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Why would $\Psi$ ever be such that $\Psi(\mathcal{P})$ is not inducing a symmetric game? In practice, we could always enforce the symmetry, couldn't we? In this case, i.e., when the game is symmetric, isn't Theorem 3.2 trivial when there exists no Condorcet winner? 
2. The notation of Axiom 5.1 (Preference Matching) is unusual. I'd like to suggest to you to not use "$a:b$" but stick to \frac or "/". 


Overall, I consider the paper borderline at this stage. I'd be happy to further discuss with the authors and potentially raise my score after the rebuttal.
soundness: 4
strengths: - The paper is well-written and particularly the first four sections coherent, well-explained and well-contextualized within the existing literature. 
- The topic of Condorcet and Smith consistency under a learned preference model and the robustness of the solution concept under estimation errors of the preference model is an interesting and generally important research direction.
summary: The paper studies a generalization of the NLHF objective under a mapping $\Psi$, i.e., the game-theoretic optimization of a preference function $\Psi (\mathcal{P})$ where $\mathcal{P}$ is the true preference model. The authors focus on necessary and sufficient conditions for desirable alignment properites. The authors show that Condorcet and Smith consistency hold under intuitive, general conditions. It is then also shown that similary results do not hold for preference matching.
weaknesses: - Some of the results are incremental. For example, I personally view Theorem 3.1 as folklore (particularly in the dueling bandit community). As long as we map probabilities $\geq 1/2$ again to probabilities $\geq 1/2$ and similarly for probabilities $< 1/2$, then the ranking among candidate actions does not change. Hence, if there exists a Condorcet winner under $\mathcal{P}$ it is still a Condorcet winner under $\Psi(\mathcal{P})$, and the NE is naturally equal to playing that action. That this is true becomes quite obvious when simply using the notation $y \succ_{\mathcal{P}} y'$ if $\mathcal{P}(y \succ y') > 1/2$ which is typically used when talking about the Condorcet winner and we don't care about the preference strength (the $\Psi$ you define leaves $y \succ y'$ unchanged). 
- While "preference matching" is conceptually interesting, I don't see why it is desirable or a reasonable thing to aim for, especially, in the context of LLMs and NLHF. Firstly, it only exists if the preferences follow a BTL model, so we are in the RLHF framework automatically even though you're primiarly motivated by NLHF. Secondly, you motivated your work with the fact that learned preference models can be biased and suffer from large estimation errors. Aiming for preference matching w.r.t. this estimated preference model that appears exactly the opposite of what we are interested in. We are interested in robustness against the errors in the preference model and do not want to propagate / mirror them. 
- This is theoretical work and I do not believe that theoretical work generally needs experiments. Though, partly due to the lack of empirical validation, the practical relevance of the work is limited in my opinion, and the contributions fairly incremental.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: I don't have other questions than those raised in the Weaknesses part.
soundness: 3
strengths: 1. **Theoretical Rigor**: The paper provides complete characterizations (necessary and sufficient conditions) for multiple alignment axioms, going beyond previous work that only verified specific cases (e.g., Liu et al., 2025 for $\Psi(t)=t$).

2. **Robustness Results**: Theorem 3.1 and Corollary 3.2 show that Condorcet consistency is preserved under approximation errors, providing theoretical insights for practical implementations with noisy preference models.

3. **Novel Technical Contributions**: The authors develop techniques for non-symmetric games, extending beyond the symmetric game analysis in Liu et al. (2025).

4. **Impossibility Result**: Theorem 5.1 provides a fundamental limitation result, clarifying that game-theoretic methods cannot achieve preference matching.
summary: This paper investigates the theoretical foundations of game-theoretic LLM alignment by studying a generalized NLHF framework where preferences $\mathcal{P}$ are transformed by a mapping $\Psi$. The authors characterize necessary and sufficient conditions on $\Psi$ for three desirable properties:

1. **Condorcet consistency**: Selecting the majority-preferred response when one exists
2. **Smith consistency**: Restricting output to the Smith set (top preference cycle)
3. **Preference matching**: Exactly matching a target distribution reflecting preference diversity

The main contributions are:
- Showing Condorcet and Smith consistency are robust to the exact payoff values (only ordinal properties matter)
- Proving that Smith-consistent methods automatically preserve diversity through mixed strategies
- Establishing an impossibility result for exact preference matching under practical constraints
weaknesses: 1. **Limited Practical Motivation**: The transformation $\Psi$, while mathematically tractable, lacks clear practical motivation. Practitioners don't typically apply non-linear transformations to collected preference data. Even the $\Psi$PO framework (Azar et al., 2024) that motivated this work focuses on identity mapping (IPO) in practice.

2. **Lack of Empirical Validation**: The paper is entirely theoretical with no experiments—neither numerical simulations on synthetic games nor actual LLM alignment tasks. This makes it difficult to assess:
   - Which preference models learned from real data satisfy which conditions
   - The practical impact of violations of these conditions
   - Whether the impossibility results matter in practice

3. **Questionable Claims about Existing Models**: The claim (above Corollary 4.2) that "several popular preference models do not satisfy" the skew-symmetry condition on $\mathcal{P}$ is problematic. This can be trivially enforced by setting $\mathcal{P}\_\theta (y \succ y') = \frac{1}{2} (M_\theta (y,y') + 1 - M_\theta (y',y))$ when $M_\theta$ is the imperfect (which may be influenced by the response order) preference model. The cited papers may not explicitly do this, but it's a straightforward fix.

4. **Disconnect Between Title and Content**: Most results are about general two-player zero-sum games and don't fundamentally require the LLM alignment context. The connection to LLMs appears mostly in framing rather than in the mathematical substance.

5. **Section 5 Unclear Connection**: The preference matching section (Section 5) studies payoff matrices $\\{\alpha_{i, j}\\}$ under Assumption 5.3, but the connection to the $\Psi(\mathcal{P})$ framework from earlier sections is unclear. How does composing preferences with $\Psi$ relate to matrices satisfying Assumption 5.3? This disconnect undermines the unified narrative.

6. **Restrictive Assumptions**: Assumption 5.3 prohibits dependence on $n$, but this seems overly restrictive. Allowing $\mathsf{poly}(\log n)$ dependence would be more realistic for LLMs ($\log n \approx$ generation length). It is not clear the influence of such relaxation.

7. **Missing Related Work**: The paper omits a large body of relevant NLHF papers including https://arxiv.org/abs/2401.04056, https://arxiv.org/abs/2410.16714, https://arxiv.org/abs/2503.08942, etc.


Decision: Reject